Target detection method and device, electronic equipment and storage medium

Abstract

The invention relates to a target detection method and device, electronic equipment and a storage medium, and is applied to the technical field of automatic driving, and the method comprises the steps: obtaining image data and 4D millimeter wave radar point cloud data for a to-be-detected region; extracting image features from the acquired image data, and projecting the image features as first bird's-eye view (BEV) features; extracting point cloud features from the obtained 4D millimeter wave radar point cloud data, and extracting second BEV features from the point cloud features; performing feature fusion on the first BEV feature and the second BEV feature to obtain a third BEV feature; and performing 3D feature extraction on the third BEV feature by using a pre-trained transfomer fusion model to obtain a fused 3D feature, and obtaining feature information of the detected target according to the fused 3D feature. The accuracy of target detection can be improved.

Description

Technical Field

[0001] The present application relates to the field of autonomous driving technology, and in particular to a target detection method, device, electronic device and storage medium. Background Art

[0002] In intelligent connected vehicles, cameras have problems such as dependence on lighting conditions, poor real-time performance, and depth information cannot be directly obtained. In order to overcome this limitation, cameras can be combined with other sensors (such as lidar, etc.) to perform target detection.

[0003] However, in some scenarios, such as for weakly reflective objects, objects with surfaces of different reflectivity, etc., the environmental information obtained by the fusion of the camera and other sensors is insufficient, resulting in low accuracy of target detection. Summary of the invention

[0004] In order to solve the above technical problems, the present application provides a target detection method, device, electronic device and storage medium.

[0005] According to a first aspect of the present application, a target detection method is provided, comprising:

[0006] Obtain image data and 4D millimeter wave radar point cloud data for the area to be detected;

[0007] Extracting image features from the acquired image data, and projecting the image features into first bird's-eye view BEV features;

[0008] Extracting point cloud features from the acquired 4D millimeter-wave radar point cloud data, and extracting a second BEV feature from the point cloud features;

[0009] Fusing the first BEV feature and the second BEV feature to obtain a third BEV feature;

[0010] The pre-trained transfomer fusion model is used to perform 3D feature extraction on the third BEV feature to obtain a fused 3D feature, and feature information of the detected target is obtained based on the fused 3D feature.

[0011] Optionally, the 4D millimeter wave radar point cloud data includes: position data and speed data;

[0012] The step of extracting point cloud features from the acquired 4D millimeter wave radar point cloud data, and extracting a second BEV feature from the point cloud features, includes:

[0013] Extracting location features from the location data, and obtaining location BEV features from the location features;

[0014] extracting a speed feature from the speed data, and obtaining a speed BEV feature from the speed feature;

[0015] The position BEV feature and the speed BEV feature are fused to obtain a second BEV feature.

[0016] Optionally, the method further comprises:

[0017] In the case where the 4D millimeter wave radar point cloud data cannot be obtained, a pre-trained transfomer image model is used to perform 3D feature extraction on the first BEV feature to obtain an image 3D feature;

[0018] Obtaining feature information of the detected target according to the 3D features of the image;

[0019] or,

[0020] In the case where image data cannot be obtained, a pre-trained transfomer point cloud model is used to extract 3D features from the second BEV feature to obtain a point cloud 3D feature;

[0021] According to the point cloud 3D features, feature information of the detected target is obtained.

[0022] Optionally, extracting point cloud features from the acquired 4D millimeter wave radar point cloud data includes:

[0023] Dividing the 4D millimeter-wave radar point cloud data into a plurality of grids according to the position coordinates of the acquired 4D millimeter-wave radar point cloud data;

[0024] According to the 4D millimeter wave radar point cloud data contained in each grid, determine the 4D millimeter wave radar point cloud data corresponding to the grid;

[0025] The point cloud features corresponding to each grid are extracted from the 4D millimeter-wave radar point cloud data corresponding to the grid.

[0026] Optionally, extracting image features from the acquired image data includes:

[0027] Image features are extracted from the acquired image data using an image feature extraction network built based on a 2D convolutional network.

[0028] Optionally, extracting point cloud features from the acquired 4D millimeter wave radar point cloud data includes:

[0029] A point cloud feature extraction network built based on a 3D convolutional network is used to extract point cloud features from the acquired 4D millimeter-wave radar point cloud data.

[0030] According to a second aspect of the present application, there is provided a target detection device, comprising:

[0031] A data acquisition module is used to acquire image data and 4D millimeter wave radar point cloud data for the area to be detected;

[0032] A first BEV feature determination module, configured to extract image features from the acquired image data, and project the image features into a first bird's-eye view BEV feature;

[0033] A second BEV feature determination module, used to extract point cloud features from the acquired 4D millimeter wave radar point cloud data, and extract a second BEV feature from the point cloud features;

[0034] A BEV feature fusion module, used for fusing the first BEV feature and the second BEV feature to obtain a third BEV feature;

[0035] A fused 3D feature extraction module, used for performing 3D feature extraction on the third BEV feature using a pre-trained transfomer fusion model to obtain a fused 3D feature;

[0036] The feature information determination module is used to obtain the feature information of the detected target according to the fused 3D features.

[0037] Optionally, the 4D millimeter wave radar point cloud data includes: position data and speed data;

[0038] The second BEV feature determination module is specifically used to extract position features from the position data and obtain position BEV features from the position features; extract speed features from the speed data and obtain speed BEV features from the speed features; and fuse the position BEV features and the speed BEV features to obtain a second BEV feature.

[0039] Optionally, the target detection device further includes:

[0040] An image 3D feature extraction module, used for performing 3D feature extraction on the first BEV feature using a pre-trained transfomer image model to obtain an image 3D feature when 4D millimeter wave radar point cloud data cannot be obtained;

[0041] The feature information determination module is further used to obtain feature information of the detected target according to the 3D features of the image;

[0042] or,

[0043] A point cloud 3D feature extraction module, used for performing 3D feature extraction on the second BEV feature using a pre-trained transfomer point cloud model to obtain a point cloud 3D feature when image data cannot be obtained;

[0044] The feature information determination module is also used to obtain feature information of the detected target based on the point cloud 3D features.

[0045] Optionally, the second BEV feature determination module is specifically configured to extract point cloud features from the acquired 4D millimeter wave radar point cloud data through the following steps:

[0046] Dividing the 4D millimeter-wave radar point cloud data into a plurality of grids according to the position coordinates of the acquired 4D millimeter-wave radar point cloud data;

[0047] According to the 4D millimeter wave radar point cloud data contained in each grid, determine the 4D millimeter wave radar point cloud data corresponding to the grid;

[0048] The point cloud features corresponding to each grid are extracted from the 4D millimeter-wave radar point cloud data corresponding to the grid.

[0049] Optionally, the first BEV feature determination module is specifically configured to extract image features from the acquired image data through the following steps:

[0050] Image features are extracted from the acquired image data using an image feature extraction network built based on a 2D convolutional network.

[0051] Optionally, the second BEV feature determination module is specifically configured to extract point cloud features from the acquired 4D millimeter wave radar point cloud data through the following steps:

[0052] A point cloud feature extraction network built based on a 3D convolutional network is used to extract point cloud features from the acquired 4D millimeter-wave radar point cloud data.

[0053] According to a third aspect of the present application, an electronic device is provided, comprising: a processor, wherein the processor is configured to execute a computer program stored in a memory, wherein the computer program implements the method described in the first aspect when executed by the processor.

[0054] According to a fourth aspect of the present application, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the method described in the first aspect is implemented.

[0055] According to a fifth aspect of the present application, a computer program product is provided. When the computer program product is run on a computer, the computer is enabled to execute the method described in the first aspect.

[0056] Compared with the prior art, the technical solution provided by the embodiments of the present application has the following advantages:

[0057] Cameras can provide visual information, and 4D millimeter-wave radars can provide 3D spatial information and more accurate target motion information, which helps to achieve more stable and accurate target tracking. Therefore, combining cameras and 4D millimeter-wave radars can make up for their respective shortcomings. Since the BEV perspective can provide more comprehensive and intuitive scene information, the corresponding BEV features are extracted from the image data and the 4D millimeter-wave radar point cloud data respectively, and the two BEV features are fused to associate the overall features of the camera and 4D millimeter-wave field of view to associate the overall features of the camera and 4D millimeter-wave radar field of view. The fused BEV features are extracted using the pre-trained transfomer fusion model to obtain fused 3D features, and the feature information of the detected target is obtained based on the fused 3D features. Combining cameras with 4D millimeter-wave radars can give full play to the advantages of both. Multimodal perception can provide richer information, help the autonomous driving system make more accurate decisions, and provide more comprehensive, accurate, and robust environmental perception capabilities, thereby achieving better performance and safety in areas such as autonomous driving. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0059] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0060] Figure 1 A flow chart of a target detection method in an embodiment of the present application;

[0061] Figure 2 A schematic diagram of a target detection method in an embodiment of the present application;

[0062] Figure 3 A schematic diagram of the structure of a target detection device in an embodiment of the present application;

[0063] Figure 4 A schematic diagram of the structure of an electronic device in an embodiment of the present application. DETAILED DESCRIPTION

[0064] In order to more clearly understand the above-mentioned purposes, features and advantages of the present application, the scheme of the present application will be further described below. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0065] In the following description, many specific details are set forth to facilitate a full understanding of the present application, but the present application may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only part of the embodiments of the present application, rather than all of the embodiments.

[0066] See also Figure 1 , Figure 1 A flow chart of a target detection method in an embodiment of the present application may include the following steps:

[0067] Step S110, acquiring image data and 4D millimeter wave radar point cloud data for the area to be detected.

[0068] In an embodiment of the present application, a multimodal sensor fusion method can be used for target detection. Multimodal sensor fusion refers to combining information from multiple different types of sensors to obtain more comprehensive environmental information. The multimodal sensor includes a camera and a 4D millimeter-wave radar. The camera is used to collect image data in the area to be detected to provide information such as the appearance characteristics, color, texture, etc. of the object, while the 4D millimeter-wave radar is used to collect point cloud data in the area to be detected, that is, 4D millimeter-wave radar point cloud data. The 4D millimeter-wave radar point cloud data includes position coordinates and the reflection intensity corresponding to each position coordinate, etc. The 4D millimeter-wave radar can provide detailed object information such as distance, speed, and angle, that is, it can provide more accurate target motion information, which helps to achieve more stable and accurate target tracking, especially in high-speed and complex traffic environments.

[0069] The area to be detected can change with the change of the position of the camera and the 4D millimeter wave radar, and the target can be tracked when the position of the camera and the 4D millimeter wave radar changes. The execution subject of the embodiment of the present application is a controller, and the controller can be, for example, a CPU (central processing unit) or the like.

[0070] Step S120: extracting image features from the acquired image data, and projecting the image features as first BEV features.

[0071] Image data refers to the image captured by the camera, and the image features extracted from the image data refer to feature vectors. In some embodiments, an image feature extraction network constructed based on a 2D convolutional network can be used to extract image features from the acquired image data, and the image features are feature vectors extracted by the 2D convolutional network. This application does not limit the network structure of the image feature extraction network, and it can also be a network of other structures.

[0072] Compared with the traditional 2D perspective, the BEV (bird's eye view) perspective can provide more comprehensive and intuitive scene information. Therefore, the image features can be projected from the forward 2D to the bird's eye view using the camera calibration parameters. Figure 3 D perspective.

[0073] Step S130: extracting point cloud features from the acquired 4D millimeter-wave radar point cloud data, and extracting second BEV features from the point cloud features.

[0074] Similarly, point cloud features are extracted from 4D millimeter wave radar point cloud data, and the point cloud features are feature vectors. In some embodiments, a point cloud feature extraction network constructed based on a 3D convolutional network can be used to extract point cloud features from the acquired 4D millimeter wave radar point cloud data. Alternatively, other types of network structures can be used to construct a point cloud feature extraction network, which is not limited in this application.

[0075] In an embodiment of the present application, in order to reduce the amount of data processing, the point cloud data can be divided into different grids, and one grid corresponds to one position coordinate and one point cloud data. Optionally, the 4D millimeter-wave radar point cloud data is divided into multiple grids according to the position coordinates of the acquired 4D millimeter-wave radar point cloud data. For example, the length, width, and height of the grid can be set, and the 4D millimeter-wave radar point cloud data located in the same grid can be obtained according to the position coordinates of the 4D millimeter-wave radar point cloud data. Optionally, the 4D millimeter-wave radar point cloud data in each grid is set to a fixed number N, where N is a suitable positive integer. If the number of 4D millimeter-wave radar point cloud data contained in the grid exceeds N, then N 4D millimeter-wave radar point cloud data can be randomly sampled. If the number of 4D millimeter-wave radar point cloud data in the grid is less than N, the less part can be filled with 0.

[0076] Afterwards, the 4D millimeter-wave radar point cloud data corresponding to the grid can be determined based on the 4D millimeter-wave radar point cloud data contained in each grid. For example, the corresponding center position coordinates can be set for each grid, and the average value of the 4D millimeter-wave radar point cloud data in the grid can be used as the 4D millimeter-wave radar point cloud data corresponding to the grid. For example, the average value of all 4D millimeter-wave radar point cloud data contained in the grid can be directly used as the 4D millimeter-wave radar point cloud data corresponding to the grid. Alternatively, after setting the value N, the average value of N 4D millimeter-wave radar point cloud data in the grid can be used as the 4D millimeter-wave radar point cloud data corresponding to the grid.

[0077] In this way, multiple 4D millimeter-wave radar point cloud data in a grid can be simplified into one 4D millimeter-wave radar point cloud data, that is, the original 4D millimeter-wave radar point cloud data can be simplified into 4D millimeter-wave radar point cloud data in units of grids. The point cloud features corresponding to each grid are extracted from the 4D millimeter-wave radar point cloud data corresponding to the grid.

[0078] The 4D millimeter wave radar point cloud data includes: position data and speed data, wherein the position data includes: the position coordinates of the 4D millimeter wave radar point cloud data (coordinates in three dimensions: x-axis, y-axis, and z-axis), reflection intensity, the position coordinates of the grid where the 4D millimeter wave radar point cloud data is located, and the relative position between the position coordinates of the 4D millimeter wave radar point cloud data and the position coordinates of the grid where the 4D millimeter wave radar point cloud data is located. The speed data includes: the absolute speed of the 4D millimeter wave radar point cloud data (relative to the ground), the relative speed of the 4D millimeter wave radar point cloud data (relative to the vehicle body), and the average speed of the 4D millimeter wave radar point cloud data.

[0079] Since the image data collected by the camera does not have speed information, in some embodiments, in order to better extract speed information and improve the accuracy of speed and direction, the position data and speed data can be processed separately as two different channels, that is, the corresponding BEV features are extracted separately and then fused. Specifically, the position feature is extracted from the position data, and the position BEV feature is obtained from the position feature; the speed feature is extracted from the speed data, and the speed BEV feature is obtained from the speed feature; the position BEV feature and the speed BEV feature are fused to obtain a second BEV feature. In this way, the speed BEV feature and the position BEV feature can be better distinguished from the second BEV feature.

[0080] Step S140: Fusing the first BEV feature and the second BEV feature to obtain a third BEV feature.

[0081] By fusing the first BEV feature extracted from the image data and the second BEV feature extracted from the 4D millimeter-wave radar point cloud data, that is, splicing the first BEV feature and the second BEV feature, richer BEV features can be obtained, thereby creating a more accurate environmental model and map, providing a more reliable reference.

[0082] Step S150, using the pre-trained transfomer fusion model to perform 3D feature extraction on the third BEV feature to obtain a fused 3D feature, and obtain feature information of the detected target based on the fused 3D feature.

[0083] In the embodiment of the present application, the basic structure of the transfomer fusion model is a transfomer network structure, including an encoder part and a decoder part. The encoder part and the decoder part encode and decode the third BEV feature to obtain the corresponding fused 3D feature. The input of the transfomer fusion model is the BEV feature, and the output is the 3D feature. The 3D feature is processed by the 3D detection head to obtain the feature information of the detected target. Among them, the feature information of the detected target includes: the 3D size, center coordinates, direction, category and confidence of the target, etc.

[0084] In the target detection method of the embodiment of the present application, the camera can provide visual information, and the 4D millimeter-wave radar can provide 3D spatial information and more accurate target motion information, which is helpful to achieve more stable and accurate target tracking. Therefore, combining the camera and the 4D millimeter-wave radar can make up for their respective shortcomings. For example, the camera may be affected by obstructions, while the 4D millimeter-wave radar can penetrate certain obstructions; the performance of the camera degrades under insufficient light or bad weather conditions, while the 4D millimeter-wave radar is not affected by these factors. Since the BEV perspective can provide more comprehensive and intuitive scene information, the corresponding BEV features are extracted from the image data and the 4D millimeter-wave radar point cloud data respectively, and the two BEV features are fused to associate the overall features of the camera and 4D millimeter-wave field of view to associate the overall features of the camera and 4D millimeter-wave radar field of view. The fused BEV features are extracted using a pre-trained transfomer fusion model to obtain fused 3D features, and the feature information of the detected target is obtained based on the fused 3D features. Combining cameras with 4D millimeter-wave radars can give full play to the advantages of both. Multimodal perception can provide richer information, help autonomous driving systems make decisions more accurately, and provide more comprehensive, accurate, and robust environmental perception capabilities, thereby achieving better performance and safety in areas such as autonomous driving. In addition, by processing the position data and speed data in the 4D millimeter-wave radar point cloud data separately, that is, extracting the corresponding BEV features and then fusing them, the speed information can be better extracted and the accuracy of speed and direction can be improved. Due to the different characteristics of cameras and 4D millimeter-wave radars, the embodiments of the present application can better adapt to different scenarios, such as cities, highways, etc.

[0085] See also Figure 2 , Figure 2 A schematic diagram of a target detection method in an embodiment of the present application. When image data and 4D millimeter wave radar point cloud data are obtained, the target detection method can be Figure 1The method shown in the embodiment performs target detection. When 4D millimeter wave radar point cloud data cannot be obtained, a pre-trained transfomer image model can be used to extract 3D features from the first BEV feature to obtain image 3D features; and based on the image 3D features, feature information of the detected target can be obtained.

[0086] Alternatively, when image data cannot be obtained, a pre-trained transfomer point cloud model can be used to perform 3D feature extraction on the second BEV feature to obtain a point cloud 3D feature; based on the point cloud 3D feature, feature information of the detected target can be obtained.

[0087] Similar to the aforementioned transfomer fusion model, the network structures of the transfomer image model and the transfomer point cloud model are also transfomer network structures. The input of the three models are all BEV features, and the output is all 3D features. The difference is that the types of BEV features input to the three models are different.

[0088] It can be seen that the embodiment of the present application can also achieve data isolation between the 4D millimeter-wave radar and the camera. Even if one of the sensors fails, it will not affect the operation of the other sensor, thereby improving the stability and reliability of the perception performance.

[0089] It should be noted that although the steps of the method in the present application are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc.

[0090] Corresponding to the above method embodiment, the present application embodiment also provides a target detection device, see Figure 3 , the target detection device 300 includes:

[0091] The data acquisition module 310 is used to acquire image data and 4D millimeter wave radar point cloud data for the area to be detected;

[0092] A first BEV feature determination module 320, configured to extract image features from the acquired image data and project the image features into a first bird's-eye view BEV feature;

[0093] A second BEV feature determination module 330 is used to extract point cloud features from the acquired 4D millimeter wave radar point cloud data, and extract a second BEV feature from the point cloud features;

[0094] A BEV feature fusion module 340 is used to fuse the first BEV feature and the second BEV feature to obtain a third BEV feature;

[0095] A fused 3D feature extraction module 350 is used to perform 3D feature extraction on the third BEV feature using a pre-trained transfomer fusion model to obtain a fused 3D feature;

[0096] The feature information determination module 360 ​​is used to obtain feature information of the detected target according to the fused 3D features.

[0097] Optionally, the 4D millimeter wave radar point cloud data includes: position data and speed data;

[0098] The second BEV feature determination module 330 is specifically used to extract position features from the position data and obtain position BEV features from the position features; extract speed features from the speed data and obtain speed BEV features from the speed features; and fuse the position BEV features and the speed BEV features to obtain the second BEV features.

[0099] Optionally, the target detection device 300 further includes:

[0100] An image 3D feature extraction module is used to extract 3D features from the first BEV feature using a pre-trained transfomer image model to obtain image 3D features when 4D millimeter wave radar point cloud data cannot be obtained;

[0101] The feature information determination module is also used to obtain feature information of the detected target based on the 3D features of the image;

[0102] or,

[0103] A point cloud 3D feature extraction module is used to extract 3D features from the second BEV feature using a pre-trained transfomer point cloud model to obtain a point cloud 3D feature when image data cannot be obtained;

[0104] The feature information determination module is also used to obtain feature information of the detected target based on the point cloud 3D features.

[0105] Optionally, the second BEV feature determination module 330 is specifically configured to extract point cloud features from the acquired 4D millimeter wave radar point cloud data through the following steps:

[0106] Divide the 4D millimeter-wave radar point cloud data into multiple grids according to the position coordinates of the acquired 4D millimeter-wave radar point cloud data;

[0107] According to the 4D millimeter wave radar point cloud data contained in each grid, determine the 4D millimeter wave radar point cloud data corresponding to the grid;

[0108] The point cloud features corresponding to each grid are extracted from the 4D millimeter-wave radar point cloud data corresponding to the grid.

[0109] Optionally, the first BEV feature determination module 320 is specifically configured to extract image features from the acquired image data through the following steps:

[0110] Image features are extracted from the acquired image data using an image feature extraction network built based on a 2D convolutional network.

[0111] Optionally, the second BEV feature determination module 330 is specifically configured to extract point cloud features from the acquired 4D millimeter wave radar point cloud data through the following steps:

[0112] A point cloud feature extraction network built based on a 3D convolutional network is used to extract point cloud features from the acquired 4D millimeter-wave radar point cloud data.

[0113] The specific details of each module or unit in the above device have been described in detail in the corresponding method, so they will not be repeated here.

[0114] It should be noted that, although several modules or units of the equipment for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into being embodied by multiple modules or units.

[0115] In an exemplary embodiment of the present application, an electronic device is also provided, comprising: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to execute the above-mentioned target detection method in this exemplary embodiment.

[0116] Figure 4 This is a schematic diagram of the structure of an electronic device in an embodiment of the present application. It should be noted that: Figure 4 The electronic device 400 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0117] like Figure 4As shown, electronic device 400 includes (CPU) 401, which can perform various appropriate actions and processes according to the program stored in read-only memory (ROM) 402 or the program loaded from storage part 408 into random access memory (RAM) 403. In RAM 403, various programs and data required for system operation are also stored. Central processing unit 401, ROM 402 and RAM 403 are connected to each other through bus 404. Input / output (I / O) interface 405 is also connected to bus 404.

[0118] The following components are connected to the I / O interface 405: an input section 406 including a keyboard, a mouse, etc.; an output section 407 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 408 including a hard disk, etc.; and a communication section 409 including a network interface card such as a local area network (LAN) card, a modem, etc. The communication section 409 performs communication processing via a network such as the Internet. A drive 410 is also connected to the I / O interface 405 as needed. A removable medium 411, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 410 as needed so that a computer program read therefrom is installed into the storage section 408 as needed.

[0119] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through the communication part 409, and / or installed from a removable medium 411. When the computer program is executed by the central processing unit 401, various functions defined in the device of the present application are executed.

[0120] In an embodiment of the present application, a computer-readable storage medium is further provided, on which a computer program is stored, and when the computer program is executed by a processor, the above-mentioned target detection method is implemented.

[0121] It should be noted that the computer-readable storage medium shown in the present application may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory, a read-only memory, an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, radio frequency, etc., or any suitable combination of the above.

[0122] In an embodiment of the present application, a computer program product is also provided. When the computer program product is run on a computer, the computer executes the above-mentioned target detection method.

[0123] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0124] The above description is only a specific implementation of the present application, so that those skilled in the art can understand or implement the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments described herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A target detection method, characterized in that: include: Obtain image data and 4D millimeter wave radar point cloud data for the area to be detected; Extracting image features from the acquired image data, and projecting the image features into first bird's-eye view BEV features; Extracting point cloud features from the acquired 4D millimeter-wave radar point cloud data, and extracting a second BEV feature from the point cloud features; Fusing the first BEV feature and the second BEV feature to obtain a third BEV feature; The pre-trained transfomer fusion model is used to perform 3D feature extraction on the third BEV feature to obtain a fused 3D feature, and feature information of the detected target is obtained based on the fused 3D feature.

2. The method according to claim 1, characterized in that The 4D millimeter wave radar point cloud data includes: position data and speed data; The step of extracting point cloud features from the acquired 4D millimeter wave radar point cloud data, and extracting a second BEV feature from the point cloud features, includes: Extracting location features from the location data, and obtaining location BEV features from the location features; extracting a speed feature from the speed data, and obtaining a speed BEV feature from the speed feature; The position BEV feature and the speed BEV feature are fused to obtain a second BEV feature.

3. The method according to claim 1, characterized in that The method further comprises: In the case where the 4D millimeter wave radar point cloud data cannot be obtained, a pre-trained transfomer image model is used to perform 3D feature extraction on the first BEV feature to obtain an image 3D feature; Obtaining feature information of the detected target according to the 3D features of the image; or, In the case where image data cannot be obtained, a pre-trained transfomer point cloud model is used to extract 3D features from the second BEV feature to obtain a point cloud 3D feature; According to the point cloud 3D features, feature information of the detected target is obtained.

4. The method according to claim 1, characterized in that: The step of extracting point cloud features from the acquired 4D millimeter wave radar point cloud data includes: Dividing the 4D millimeter-wave radar point cloud data into a plurality of grids according to the position coordinates of the acquired 4D millimeter-wave radar point cloud data; According to the 4D millimeter wave radar point cloud data contained in each grid, determine the 4D millimeter wave radar point cloud data corresponding to the grid; The point cloud features corresponding to each grid are extracted from the 4D millimeter-wave radar point cloud data corresponding to the grid.

5. The method according to claim 1, characterized in that The step of extracting image features from the acquired image data comprises: Image features are extracted from the acquired image data using an image feature extraction network built based on a 2D convolutional network.

6. The method according to claim 1, characterized in that The step of extracting point cloud features from the acquired 4D millimeter wave radar point cloud data includes: A point cloud feature extraction network built based on a 3D convolutional network is used to extract point cloud features from the acquired 4D millimeter-wave radar point cloud data.

7. A target detection device, characterized in that: The device comprises: A data acquisition module is used to acquire image data and 4D millimeter wave radar point cloud data for the area to be detected; A first BEV feature determination module, configured to extract image features from the acquired image data, and project the image features into a first bird's-eye view BEV feature; A second BEV feature determination module, used to extract point cloud features from the acquired 4D millimeter wave radar point cloud data, and extract a second BEV feature from the point cloud features; A BEV feature fusion module, used for fusing the first BEV feature and the second BEV feature to obtain a third BEV feature; A 3D feature extraction module, used for performing 3D feature extraction on the third BEV feature using a pre-trained transfomer fusion model to obtain a fused 3D feature; The feature information determination module is used to obtain the feature information of the detected target according to the fused 3D features.

8. The device according to claim 7, characterized in that The 4D millimeter wave radar point cloud data includes: position data and speed data; The second BEV feature determination module is specifically used to extract position features from the position data and obtain position BEV features from the position features; extract speed features from the speed data and obtain speed BEV features from the speed features; and fuse the position BEV features and the speed BEV features to obtain a second BEV feature.

9. An electronic device, characterized in that: include: A processor, wherein the processor is used to execute a computer program stored in a memory, wherein the computer program, when executed by the processor, implements the method according to any one of claims 1 to 6.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images