Three-dimensional target detection method, device, storage medium and electronic device

Abstract

The present application discloses a three-dimensional target detection method, device, storage medium and electronic device, including: obtaining 4D millimeter wave radar point cloud data and image acquisition data corresponding to the same scene; performing feature extraction and feature conversion on the image acquisition data to obtain image features; using a first backbone model and a second backbone model to extract point cloud features from the point cloud data respectively, and correspondingly obtaining two radar features; using a cross-attention mechanism to fuse the two radar features, and splicing the obtained first fusion result with the radar feature obtained by the first backbone model to obtain a fused point cloud feature; cross-modally fusing the fused point cloud feature and image feature, and performing three-dimensional target detection based on the obtained second fusion result. The application can effectively improve the accuracy of three-dimensional target detection.

Description

Technical Field

[0001] The present application relates to target detection technology, and in particular to a three-dimensional target detection method, device, storage medium and electronic device. Background Art

[0002] With the popularization and promotion of various sensors, the complementary characteristics of various sensors can be used to achieve target detection by comprehensively processing the collected data of various sensors. At present, for three-dimensional target detection, three-dimensional target detection can be performed based on the point cloud data collected by 4D millimeter-wave radar combined with image data collected by image sensors such as cameras. For example, in an autonomous driving system, the data collected by the on-board 4D millimeter-wave radar and the on-board camera can be combined for comprehensive processing to achieve three-dimensional target detection.

[0003] The existing solution of using 4D millimeter-wave radar and image sensor for three-dimensional target detection has been found through practical application that the accuracy of target detection needs to be improved. Summary of the invention

[0004] The present application provides a three-dimensional object detection method, device, storage medium and electronic device, which can effectively improve the accuracy of three-dimensional object detection.

[0005] To achieve the above objectives, this application adopts the following technical solutions:

[0006] A three-dimensional target detection method, comprising:

[0007] Obtain 4D millimeter wave radar point cloud data and image acquisition data corresponding to the same scene;

[0008] Performing feature extraction and feature conversion on the image acquisition data to obtain image features;

[0009] The first backbone model and the second backbone model are used to extract point cloud features from the point cloud data, and the corresponding radar features are obtained. f radar-pillar and f radar-transformer ; Wherein, the input of the first backbone model includes the radar space coordinates, radar speed and point cloud signal-to-noise ratio corresponding to the point cloud data;

[0010] Use the cross attention mechanism to f radar-pillar and stated f radar-transformer The first fusion result is combined with the f radar-pillar splicing to obtain the fused point cloud features; wherein, when performing the fusion, based on the fradar-pillar Determine the query matrix Q based on the f radar-transformer Determine the key matrix K and the value matrix V;

[0011] The fused point cloud features and the image features are cross-modally fused, and three-dimensional target detection is performed based on the obtained second fusion result.

[0012] Preferably, the cross-modal fusion is performed using a cross-attention mechanism or a deformable cross-attention mechanism.

[0013] Preferably, the method further comprises:

[0014] Regarding the f radar-transformer Perform feature extraction and predict occupancy information to indicate whether there is a target at each location;

[0015] The performing three-dimensional target detection based on the obtained second fusion result includes:

[0016] The second fusion result is spliced ​​with the occupancy information, and the three-dimensional target detection is performed based on the splicing result.

[0017] Preferably, the method further comprises: performing BEV feature extraction on the image features, and using the corresponding feature extraction results as new image features.

[0018] Preferably, the method further comprises: performing BEV feature extraction on the fused point cloud features, and using the corresponding feature extraction results as new fused point cloud features.

[0019] A three-dimensional target detection device, comprising: a data acquisition unit, an image data processing unit, a point cloud data processing unit, a cross-modal fusion unit and a detection unit;

[0020] The data acquisition unit is used to acquire 4D millimeter wave radar point cloud data and image acquisition data corresponding to the same scene;

[0021] The image data processing unit is used to perform feature extraction and feature conversion on the image acquisition data to obtain image features;

[0022] The point cloud data processing unit is used to extract point cloud features from the point cloud data using the first backbone model and the second backbone model, and obtain radar features accordingly. f radar-pillar and f radar-transformer ; It is also used to use the cross attention mechanism to f radar-pillar and stated fradar-transformer The first fusion result is combined with the f radar-pillar splicing to obtain fused point cloud features; wherein the input of the first backbone model includes the radar space coordinates, radar speed and point cloud signal-to-noise ratio corresponding to the point cloud data; when performing the fusion, based on the f radar-pillar Determine the query matrix Q based on the f radar-transformer Determine the key matrix K and the value matrix V;

[0023] The cross-modal fusion unit is used to perform cross-modal fusion on the fused point cloud features and the image features to obtain a second fusion result;

[0024] The detection unit is used to perform three-dimensional target detection based on the second fusion result.

[0025] Preferably, in the cross-modal fusion unit, a cross-attention mechanism or a deformable cross-attention mechanism is used to perform the cross-modal fusion.

[0026] Preferably, the point cloud data processing unit is further used to f radar-transformer Perform feature extraction and predict occupancy information to indicate whether there is a target at each location;

[0027] In the detection unit, performing three-dimensional target detection based on the second fusion result includes:

[0028] The second fusion result is spliced ​​with the occupancy information, and the three-dimensional target detection is performed based on the splicing result.

[0029] Preferably, the image data processing unit is further used to perform BEV feature extraction on the image features, and use the corresponding feature extraction results as new image features.

[0030] Preferably, the point cloud data unit is further used to perform BEV feature extraction on the fused point cloud features, and use the corresponding feature extraction results as new fused point cloud features.

[0031] The present application provides a computer-readable storage medium having computer instructions stored thereon, and when the instructions are executed by a processor, any of the above-described three-dimensional target detection methods can be implemented.

[0032] The present application provides an electronic device, which includes at least a computer-readable storage medium and a processor;

[0033] The processor is used to read executable instructions from the computer-readable storage medium and execute the instructions to implement any of the three-dimensional object detection methods described above.

[0034] It can be seen from the above technical solution that in this application, the 4D millimeter wave radar point cloud data and image acquisition data corresponding to the same scene are first obtained to perform 3D target detection on the corresponding scene. Next, on the one hand, the image acquisition data is feature extracted and converted to obtain image features in the BEV space; on the other hand, the point cloud data is respectively extracted using the first backbone model and the second backbone model to obtain radar features. f radar-pillar and f radar-transformer , and use the cross attention mechanism to fuse the two radar features, and then combine the fusion result with the radar feature f radar-pillar The fused point cloud features are obtained by splicing. Thus, the multi-dimensional information of the point cloud data is effectively utilized through the characteristics of the first backbone model to improve the accuracy of the point cloud features and make up for the low accuracy of the height information in the 4D millimeter wave radar. At the same time, the peripheral information of the point cloud is effectively utilized through the second backbone model to improve the accuracy of the point cloud features and f radar-pillar and f radar-transformer To reflect the distribution characteristics of point cloud f radar-pillar is the main feature, and the fusion result is combined with f radar-pillar Splicing is performed to ensure that the fused point cloud features can reflect the point cloud distribution characteristics more importantly while reflecting the surrounding point cloud information. Finally, the fused point cloud features determined based on the point cloud data and the image features of the BEV space determined based on the image acquisition data are cross-modally fused, which can effectively combine the respective feature information of the two types of data, and then perform three-dimensional target detection based on this. Since the special processing of point cloud data can effectively improve the accuracy of point cloud features, the accuracy of three-dimensional target detection can be improved accordingly. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 A schematic diagram of the basic process of the three-dimensional object detection method in this application;

[0036] Figure 2 A block diagram of the implementation of the three-dimensional target detection method in the embodiment of the present application;

[0037] Figure 3 A schematic diagram of a specific process of a three-dimensional target detection method in an embodiment of the present application;

[0038] Figure 4Schematic diagram of the processing flow of the second backbone model in the embodiment of the present application;

[0039] Figure 5 This is a schematic diagram of the fusion of two types of radar features in an embodiment of the present application;

[0040] Figure 6 This is an example diagram of the misalignment between the millimeter wave radar point cloud and the image sensor space in the embodiment of the present application;

[0041] Figure 7 Schematic diagram of cross-modal fusion in an embodiment of the present application;

[0042] Figure 8 This is a schematic diagram of the sparse point cloud of a long-distance 4D millimeter-wave radar in an embodiment of the present application;

[0043] Fig. 9 Schematic diagram of the basic structure of the three-dimensional object detection device in this application;

[0044] Fig.10 It is a schematic diagram of the basic structure of the electronic device in this application. DETAILED DESCRIPTION

[0045] In order to make the objectives, technical means and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings.

[0046] First, the existing schemes for 3D target detection using 4D millimeter-wave radar and image sensors are analyzed. In the existing schemes, the feature extraction method of laser radar data is directly used to extract the radar backbone of 4D millimeter-wave radar data in voxels to obtain the point cloud features of 4D millimeter-wave radar data; however, compared with laser radar, 4D millimeter-wave radar has fewer point clouds and there are certain errors in height information. Therefore, the point cloud features obtained after feature extraction in voxel format are not delicate and accurate enough. Using such point cloud features for 3D target detection will also affect the accuracy of 3D target detection.

[0047] Based on the above analysis, the applicant provides a three-dimensional target detection method in this application, which combines the first backbone model and the second backbone model to perform point cloud feature extraction of 4D millimeter-wave radar, thereby obtaining more delicate and accurate point cloud features, thereby effectively improving the accuracy of three-dimensional target detection.

[0048] Figure 1 Schematic diagram of the basic process of the three-dimensional target detection method in this application. Figure 1 As shown, the method includes:

[0049] Step 101, obtaining 4D millimeter wave radar point cloud data and image acquisition data corresponding to the same scene.

[0050] This step is used to collect data using 4D millimeter wave radar and image sensor for scenes that require target detection, and obtain point cloud data through 4D millimeter wave radar and image data through image sensor. The specific processing method is the same as the existing one, so it will not be repeated here.

[0051] Step 102: extract and transform the image data to obtain image features.

[0052] This step is used to process the image acquisition data, specifically including: extracting features from the data, then performing feature conversion, predicting image depth, and determining image features including depth information. The specific processing method can be the same as the existing method, which will not be repeated here.

[0053] Step 103: extract point cloud features from the point cloud data using the first backbone model and the second backbone model, and obtain radar features f radar-pillar and f radar-transformer .

[0054] Steps 103-104 are used to process the point cloud data obtained in step 101. First, point cloud feature extraction is performed in step 103. In this application, in order to effectively improve the accuracy and delicacy of point cloud features, a first backbone model and a second backbone model are selected to perform point cloud feature extraction respectively, wherein the first backbone model can be a pillar-base network, and the second backbone model can be a Transformer-base network.

[0055] Specifically, the first backbone model is a point cloud-based 3D target detection model that processes spatial data by converting point cloud data into pillars. Compared with the existing voxel format processing method, it can more effectively utilize multi-dimensional spatial data and overcome the problem that inaccurate height information in 4D millimeter-wave radar point cloud data affects the accuracy of point cloud features. Among them, the input of the first backbone model includes the radar spatial coordinates corresponding to the point cloud data, radar speed, and point cloud signal-to-noise ratio.

[0056] When the second backbone model performs point cloud feature extraction, it can not only extract the features of the point cloud itself, but also learn the surrounding point cloud information. For large targets, it can effectively improve the accuracy of point cloud features. And because the amount of point cloud data acquired by 4D millimeter-wave radar is limited, using the second backbone model for feature extraction will not bring too much processing burden to target detection.

[0057] Step 104, using the cross attention mechanism f radar-pillar and fradar-transformer Fusion is performed, and the first fusion result is obtained with f radar-pillar The fused point cloud features are obtained by splicing.

[0058] In step 103, the two models are used to extract features from the point cloud data respectively, thereby overcoming the problem of inaccurate height information of the 4D millimeter wave radar point cloud on the one hand, and expanding the learning of surrounding point cloud information on the other hand, which can effectively improve the accuracy of point cloud features. This step fuses the point cloud features extracted by the above two methods, taking into account the radar features f radar-pillar represents the distribution feature of the point cloud, so this feature is used as the main feature for fusion processing. Specifically, when fusion is performed, based on f radar-pillar Determine the query matrix Q based on f radar-transformer Determine the key matrix K and the value matrix V, and then fuse them through the cross attention mechanism to obtain the fusion result. In order to distinguish it from other subsequent fusion results, the above f radar-pillar and f radar-transformer The fusion result of is called the first fusion result. Next, also considering the feature f radar-pillar Represents the point cloud distribution characteristics, and the first fusion result is combined with f radar-pillar Then perform another stitching to obtain the fused point cloud features. In this way, more accurate and delicate point cloud features can be obtained.

[0059] Step 105: cross-modally fuse the fused point cloud features and image features, and perform three-dimensional target detection based on the obtained second fusion result.

[0060] In the above process, the image acquisition data is processed to obtain image features through step 102, and the point cloud data is processed to obtain fused point cloud features through steps 103-104. The processing of the image acquisition data and the processing of the point cloud data can be performed in parallel or in series in any order.

[0061] In this step, the features obtained after processing the above two types of data are cross-modally fused to obtain the second fusion result. The specific cross-modal fusion can be simply spliced, or it can be fused using a cross-attention mechanism or a deformed cross-attention mechanism, and then the second fusion result is used for 3D object detection.

[0062] Since the aforementioned processing of point cloud data can obtain more accurate and delicate point cloud features, the three-dimensional target detection performed in this step can also improve the accuracy accordingly.

[0063] At this point, the three-dimensional target detection process in this application ends.

[0064] The specific implementation of the three-dimensional object detection method in this application is described below through specific embodiments.

[0065] Figure 2 is a block diagram of an implementation method of a three-dimensional target detection method in an embodiment of the present application, Figure 3 Schematic diagram of the specific process of the three-dimensional target detection method in the embodiment of the present application. Figure 2 and Figure 3 As shown, the three-dimensional target detection method process in the embodiment includes:

[0066] Step 301, obtaining 4D millimeter wave radar point cloud data and image acquisition data corresponding to the same scene.

[0067] The processing of this step is the same as step 201, and will not be repeated here.

[0068] Step 302: Use an image feature encoder to extract features from the image acquisition data.

[0069] The image feature encoder can use a convolutional neural network to extract features from the image acquisition data, providing a basis for subsequent processing. In this embodiment, the image feature encoder first passes through a standard ResNet-101 network (excluding the last fully connected layer) to obtain W img / 32× H img / 32×2048 dimensional features, where W img and H img They are the width and height of the original image respectively; then the above features are upsampled through 3 layers of deconvolution operations to form the required image features ,in, W f = W img / 4, H f = H img / 4.

[0070] Step 303 : The feature conversion module performs feature conversion on the feature extraction result of step 302 .

[0071] In this embodiment, the feature conversion module can adopt a depth estimation algorithm such as that in LSS, and take the features based on the image view (i.e., the feature extraction result of step 302) as input, first construct a viewing cone by only up-dimensionalizing the image to predict the depth of the image, and then generate a point cloud based on the predicted image depth and the image feature extraction result, and finally pool in the longitudinal direction to obtain the image features, that is, the features of the image in the BEV space.

[0072] Step 304: Use the image BEV encoder to perform BEV feature extraction on the image features to obtain updated image features.

[0073] Considering that there may be errors in the image depth estimation in step 303, the image BEV encoder is introduced in this embodiment to extract BEV features from the image features, so that the BEV features are more obvious and updated image features are obtained. When the fused point cloud features obtained after point cloud data processing are subsequently fused, the updated image features are used for fusion processing.

[0074] Step 305: extract point cloud features from the point cloud data using the first backbone model and the second backbone model, and obtain radar features f radar-pillar and f radar-transformer .

[0075] 1) Use the first backbone network to extract features from point cloud data

[0076] Since the height information of the 4D millimeter wave radar point cloud is not accurate enough, the first backbone model is used in this application for feature extraction. In this embodiment, the modified PointPillars is used for feature extraction. The specific input is N r ×6-dimensional matrix, N r is the number of 4D millimeter wave radar point clouds, each point is a 6-dimensional vector ( x , y , z , v x , v y , snr ), x , y , z represents the radar space coordinates, v x , v y represents the radar speed, snrRepresents the signal-to-noise ratio of the point cloud. Except for the input dimension, all structures are consistent with the standard PointPillar. After feature extraction of PointPillars, radar features are obtained. f radar-pillar .

[0077] 2) Use the second backbone network to extract features from point cloud data

[0078] Since the number of 4D millimeter-wave radar point clouds is smaller than that of laser radar, with about 4,000 points in a single frame (the number of point clouds in a single laser radar frame is about tens of thousands), the second backbone model can be used for feature extraction, which is time-saving and does not cause additional burden; and by extracting point cloud features through Transformer, the information of surrounding point clouds can be learned, which is of great benefit for large targets. Figure 4 As shown in the figure, the input Embedding layer in the Transformer is used to transform the coordinates of the point cloud, and the point cloud coordinates are mapped to a higher-dimensional space. Then, the cascaded Attention network is used for processing, and the outputs of the Attention networks at each level are spliced ​​together to learn the features of the surrounding point clouds, making up for the fact that the first backbone model can only extract features within the Pillar. Then, the global point cloud features are obtained through MLP. f radar-transformer .

[0079] Step 306, using the cross attention mechanism f radar-pillar and f radar-transformer Fusion is performed, and the first fusion result is obtained with f radar-pillar The points are spliced ​​to obtain the fused point cloud features.

[0080] In order to obtain the radar characteristics of the first backbone f radar-pillar and the radar signature obtained by the second backbone f radar-transformer The fusion is performed to obtain more accurate and delicate point cloud features, and the cross-attention mechanism is used for fusion. The specific fusion method is as follows: Figure 5 As shown. Radar characteristics obtained due to the first backbone f radar-pillar It can reflect the distribution characteristics of point cloud, so it is used as the main radar feature to fuse the radar features obtained by the second backbone f radar-transformer The specific implementation structure can use the CrossAttention fusion in Transformer to obtain the first fusion result. Then the fused features (i.e. the first fusion result) are combined with the features of the first backbone.f radar-pillar After stitching, a delicate point cloud feature is finally obtained.

[0081] Step 307: Use the radar BEV encoder to extract BEV features from the fused point cloud features.

[0082] In this embodiment, a radar BEV encoder is introduced to extract BEV features from the fused point cloud features, thereby making the BEV features more obvious.

[0083] Step 308: Use the OccNet module to analyze the radar features. f radar-transformer Feature extraction is performed to predict occupancy information indicating whether there is a target at each location.

[0084] Considering the limited number of point clouds of 4D millimeter-wave radar, in order to further improve the accuracy of target detection in sparse parts of the point cloud, preferably, the OccNet module is introduced in this embodiment to predict occupancy information.

[0085] Specifically, the OccNet module obtains radar features from the second backbone process f radar-transformer Feature extraction is performed again to predict whether there is an object at each position in the BEV space, without accurately obtaining information such as the object category and size. Here, the information indicating whether there is an object at each position is called occupancy information. The network structure of OccNet can adopt ResNet-101 network or other feature extraction networks, and its main function is to extract whether there is occupancy information at the position.

[0086] because f radar-transformer It can reflect the surrounding features of the point cloud. Therefore, the occupancy information predicted based on this feature can be combined with the surrounding information to comprehensively predict whether there is a target at each position in the sparse area of ​​the point cloud. For such sparse areas, the indication of occupancy information can give a hint of whether there is a target. In this way, when combined with the fused target features in the subsequent step, the information of the sparse part of the point cloud can be more aggregated, better reflecting the characteristics of the sparse part, and improving the accuracy of target detection in this area, especially for the detection of small targets, which can effectively improve the detection accuracy.

[0087] Step 309 , cross-modally fuse the fused point cloud features and image features to obtain a second fusion result.

[0088] This step is used to perform cross-modal fusion of the image features obtained by step 304 and the fused point cloud features obtained by step 307. In this embodiment, a deformed cross-attention mechanism is used for fusion processing.

[0089] Specifically, the cross-attention mechanism is naturally suitable for cross-modal feature fusion, because it is difficult to ensure feature alignment of multi-modal features, and the cross-attention mechanism can learn local surrounding feature information to ensure that two features of the same target can be well learned together. This fusion is even more suitable for the modalities of 4D millimeter-wave radar and camera, such as Figure 6 As shown, the left picture is the visualization result of 4D millimeter-wave radar in BEV, the yellow dot is the radar point cloud (CFAR), the green is the track (that is, the clustering result based on the point cloud), and the white is the laser point (the default is the true value position of the target). Figure 6 The image in the pink circle on the right corresponds to the point cloud in the pink circle on the left, but in fact the true target box of the corresponding target is the white target box below the pink circle on the left. Obviously, the 4D millimeter-wave radar point cloud in the pink circle is outside the target box, which will cause the features under BEV to be unable to align. Therefore, the cross-attention mechanism is used for cross-modal fusion. The time complexity of the deformable cross-attention mechanism is lower, and the fusion effect can also meet the requirements. The specific fusion structure diagram of the deformable cross-attention mechanism is as follows: Figure 7 As shown. After position encoding of the image features, the first query matrix, the first key matrix and the first value matrix are determined; after position encoding of the fused point cloud features, the second query features, the second key matrix and the second value matrix are determined; then, the first query matrix, the second key matrix and the second value matrix are used to perform a deformed cross attention mechanism to obtain a first processing result; the second query matrix, the first key matrix and the first value matrix are used to perform a deformed cross attention mechanism to obtain a second processing result; finally, the first processing result and the second processing result are spliced ​​to obtain a second fusion result.

[0090] In step 310, the occupancy information obtained in step 308 is concatenated with the second fusion result obtained in step 309 to take the target feature and perform three-dimensional target detection.

[0091] In current 3D target detection projects (such as vehicle-mounted projects), the image detection performance for long-distance targets or small targets needs to be improved. Although 4D millimeter-wave radar has a long detection distance and small targets also have point clouds, the number of point clouds of such targets is often small. Figure 8As shown, after feature extraction, the features of this type of target are not obvious. In order to improve the detection performance of this type of sparse point cloud target, the occupancy information is fused to add a strong feature to indicate that the target exists, thereby improving the overall performance of target detection. In this step, the occupancy information extracted by OccNet is concat-superimposed with the second fusion result of cross-modal fusion, and finally a more accurate feature can be obtained. It is used as a three-dimensional target feature, which can effectively reflect the feature information of the three-dimensional target in the BEV space. The above three-dimensional target features are input into the three-dimensional target detection head network, and the three-dimensional target detection results can be obtained by processing.

[0092] At this point, the method flow in this embodiment ends. In the above embodiment, the image BEV encoder and radar BEV encoder are newly introduced to further obtain more obvious and accurate BEV features, and the Occ module is newly introduced to predict occupancy information and provide additional target information, thereby further improving the accuracy of three-dimensional target detection.

[0093] It can be seen from the above that in the three-dimensional target detection method provided in the present application, three-dimensional target detection can be effectively performed, and the three-dimensional information such as spatial position, size, orientation, and movement speed of the object can be estimated while detecting the object; among them, through the two point cloud feature extraction methods of the first backbone and the second backbone, compared with the traditional voxelization extraction of point cloud features, it is possible to utilize global point cloud information and local point cloud information to provide more accurate and delicate point cloud features; at the same time, not only the cross-modal image features and point cloud features are fused, but also the occupancy information of the 4D millimeter-wave radar is fused, thereby effectively improving the three-dimensional target detection performance.

[0094] The above is a specific implementation of the three-dimensional object detection method in this application. This application also provides a three-dimensional object detection device, which can be used to implement the above three-dimensional object detection method. Fig. 9 Schematic diagram of the basic structure of the three-dimensional target detection device in this application. Fig. 9 As shown, the device includes: a data acquisition unit, an image data processing unit, a point cloud data processing unit, a cross-modal fusion unit and a detection unit.

[0095] The data acquisition unit is used to acquire 4D millimeter wave radar point cloud data and image acquisition data corresponding to the same scene;

[0096] An image data processing unit, used for performing feature extraction and feature conversion on the image acquisition data to obtain image features;

[0097] The point cloud data processing unit is used to extract point cloud features from the point cloud data using the first backbone model and the second backbone model, and obtain radar features accordingly. f radar-pillar and f radar-transformer; It is also used to use the cross attention mechanism to f radar-pillar and f radar-transformer Fusion is performed, and the first fusion result is obtained with f radar-pillar splicing to obtain the fused point cloud features; the input of the first backbone model includes the radar space coordinates, radar speed and point cloud signal-to-noise ratio corresponding to the point cloud data; when fusion is performed, based on f radar-pillar Determine the query matrix Q based on f radar-transformer Determine the key matrix K and the value matrix V;

[0098] A cross-modal fusion unit, used for cross-modally fusing the fused point cloud features and image features to obtain a second fusion result;

[0099] A detection unit is used to perform three-dimensional target detection based on the second fusion result.

[0100] Optionally, in the cross-modal fusion unit, a cross-attention mechanism or a deformable cross-attention mechanism is used to perform cross-modal fusion.

[0101] Optionally, the point cloud data processing unit is further used to f radar-transformer Perform feature extraction and predict occupancy information to indicate whether there is a target at each location;

[0102] In the detection unit, specific processing of performing three-dimensional target detection based on the second fusion result may include:

[0103] The second fusion result is spliced ​​with the occupancy information, and three-dimensional target detection is performed based on the splicing result.

[0104] Optionally, the image data processing unit may be further used to perform BEV feature extraction on the image features, and use the corresponding feature extraction results as new image features.

[0105] Optionally, the point cloud data processing unit may be further configured to perform BEV feature extraction on the fused point cloud features, and use the corresponding feature extraction results as new fused point cloud features.

[0106] The present application also provides a computer-readable storage medium, which stores instructions, and when the instructions are executed by a processor, the steps in the three-dimensional object detection method described above can be executed. In practical applications, the computer-readable medium can be included in each device / apparatus / system in the above-mentioned embodiments, or it can exist independently without being assembled into the device / apparatus / system. Instructions are stored in the computer-readable storage medium, and when the instructions stored therein are executed by a processor, the steps in the three-dimensional object detection method described above can be executed.

[0107] According to the embodiments disclosed in the present application, the computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, it may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof, but it is not used to limit the scope of protection of the present application. In the embodiments disclosed in the present application, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, apparatus, or device.

[0108] Fig.10 This application also provides an electronic device. Fig.10 As shown, it shows a schematic diagram of the structure of the electronic device involved in the embodiment of the present application, specifically:

[0109] The electronic device may include a processor 1001 with one or more processing cores, a memory 1002 with one or more computer-readable storage media, and a computer program stored in the memory and executable on the processor. When executing the program in the memory 1002, a method for three-dimensional object detection may be implemented.

[0110] Specifically, in practical applications, the electronic device may further include components such as a power supply 1003, an input / output unit 1004, etc. Those skilled in the art may understand that Fig.10 The structure of the electronic device shown in the figure does not constitute a limitation on the electronic device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0111] Processor 1001 is the control center of the electronic device. It uses various interfaces and lines to connect various parts of the entire electronic device. It executes various functions of the server and processes data by running or executing software programs and / or modules stored in memory 1002 and calling data stored in memory 1002, thereby monitoring the electronic device as a whole.

[0112] The memory 1002 can be used to store software programs and modules, that is, the above-mentioned computer-readable storage medium. The processor 1001 executes various functional applications and data processing by running the software programs and modules stored in the memory 1002. The memory 1002 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function, etc.; the data storage area may store data created according to the use of the server, etc. In addition, the memory 1002 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 1002 may also include a memory controller to provide the processor 1001 with access to the memory 1002.

[0113] The electronic device also includes a power supply 1003 for supplying power to various components, which can be logically connected to the processor 1001 through a power management system, so as to manage charging, discharging, power consumption and other functions through the power management system. The power supply 1003 can also include any components such as one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, and power status indicators.

[0114] The electronic device may also include an input and output unit 1004, which may be used to receive input digital or character information and generate keyboard, mouse, joystick, and optical signal inputs related to user settings and function control. The input unit output 1004 may also be used to display information input by the user or information provided to the user and various graphical user interfaces, which may be composed of graphics, text, icons, videos, and any combination thereof.

[0115] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A three-dimensional target detection method, characterized in that: include: Obtain 4D millimeter wave radar point cloud data and image acquisition data corresponding to the same scene; Performing feature extraction and feature conversion on the image acquisition data to obtain image features; The first backbone model and the second backbone model are used to extract point cloud features from the point cloud data, and the corresponding radar features are obtained. f radar-pillar and f radar-transformer ; Wherein, the input of the first backbone model includes the radar space coordinates, radar speed and point cloud signal-to-noise ratio corresponding to the point cloud data; Use the cross attention mechanism to f radar-pillar and stated f radar-transformer The first fusion result is combined with the f radar-pillar splicing to obtain the fused point cloud features; wherein, when performing the fusion, based on the f radar-pillar Determine the query matrix Q based on the f radar-transformer Determine the key matrix K and the value matrix V; The fused point cloud features and the image features are cross-modally fused, and three-dimensional target detection is performed based on the obtained second fusion result.

2. The method according to claim 1, characterized in that The cross-modal fusion is performed using a cross-attention mechanism or a deformable cross-attention mechanism.

3. The method according to claim 1 or 2, characterized in that: The method further comprises: Regarding the f radar-transformer Perform feature extraction and predict occupancy information to indicate whether there is a target at each location; The performing three-dimensional target detection based on the obtained second fusion result includes: The second fusion result is spliced ​​with the occupancy information, and the three-dimensional target detection is performed based on the splicing result.

4. The method according to claim 1, characterized in that The method further includes: performing BEV feature extraction on the image features, and using corresponding feature extraction results as new image features.

5. The method according to claim 1, characterized in that The method further includes: performing BEV feature extraction on the fused point cloud features, and using the corresponding feature extraction results as new fused point cloud features.

6. A three-dimensional target detection device, characterized in that: include: Data acquisition unit, image data processing unit, point cloud data processing unit, cross-modal fusion unit and detection unit; The data acquisition unit is used to acquire 4D millimeter wave radar point cloud data and image acquisition data corresponding to the same scene; The image data processing unit is used to perform feature extraction and feature conversion on the image acquisition data to obtain image features; The point cloud data processing unit is used to extract point cloud features from the point cloud data using the first backbone model and the second backbone model, and obtain radar features accordingly. f radar-pillar and f radar-transformer ; It is also used to use the cross attention mechanism to f radar-pillar and stated f radar-transformer The first fusion result is combined with the f radar-pillar splicing to obtain fused point cloud features; wherein the input of the first backbone model includes the radar space coordinates, radar speed and point cloud signal-to-noise ratio corresponding to the point cloud data; when performing the fusion, based on the f radar-pillar Determine the query matrix Q based on the f radar-transformer Determine the key matrix K and the value matrix V; The cross-modal fusion unit is used to perform cross-modal fusion on the fused point cloud features and the image features to obtain a second fusion result; The detection unit is used to perform three-dimensional target detection based on the second fusion result.

7. The device according to claim 6, characterized in that In the cross-modal fusion unit, the cross-modal fusion is performed using a cross-attention mechanism or a deformable cross-attention mechanism.

8. The device according to claim 6 or 7, characterized in that The point cloud data processing unit is further used to f radar-transformer Perform feature extraction and predict occupancy information to indicate whether there is a target at each location; In the detection unit, performing three-dimensional target detection based on the second fusion result includes: The second fusion result is spliced ​​with the occupancy information, and the three-dimensional target detection is performed based on the splicing result.

9. The device according to claim 6, characterized in that The image data processing unit is further used to perform BEV feature extraction on the image features and use the corresponding feature extraction results as new image features.

10. The device according to claim 6, characterized in that The point cloud data unit is further used to perform BEV feature extraction on the fused point cloud features, and use the corresponding feature extraction results as new fused point cloud features.

11. A computer-readable storage medium having computer instructions stored thereon, characterized in that: When the instruction is executed by the processor, the three-dimensional target detection method described in any one of claims 1 to 5 can be implemented.

12. An electronic device, characterized in that: The electronic device includes at least a computer-readable storage medium and also includes a processor; The processor is used to read executable instructions from the computer-readable storage medium and execute the instructions to implement the three-dimensional target detection method described in any one of claims 1 to 5 above.

Images