3D target detection method based on millimeter wave radar and camera image double-branch mutual fusion

Abstract

The invention discloses a 3D target detection method based on millimeter wave radar and camera image double-branch mutual fusion. According to the method, specific image features of respective modes are efficiently extracted from a camera and a millimeter-wave radar, and complementary branch information is introduced. In particular, the method generates geometrically perceived camera bird's eye view (BEV) features and semantically perceived millimeter wave radar BEV features. The two types of features are fused to form a comprehensively enhanced BEV feature map, and then through decoding processing, accurate 3D target detection is realized. According to the method, the accuracy and robustness of 3D target detection are remarkably improved, and experimental results show that the method can effectively detect and position the 3D target in the environment even under the conditions of complex scenes and severe weather.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to a 3D target detection method based on dual-branch mutual fusion of millimeter-wave radar and camera images. Background Art

[0002] With the rapid development of autonomous driving technology, 3D object detection technology plays a vital role in autonomous driving systems. It helps vehicles accurately perceive the surrounding environment and provides key information such as position and speed, thereby achieving safe driving. At present, the fusion of laser radar (LiDAR) and camera is the mainstream method for 3D object detection. LiDAR can provide high-precision geometric depth information, while the camera can capture rich semantic information. The complementarity of the two provides efficient perception capabilities for 3D object detection. However, LiDAR sensors are expensive and perform poorly in bad weather conditions.

[0003] In contrast, millimeter-wave radar (4D Radar) has gradually become an alternative in the field of autonomous driving perception because of its ability to work stably in bad weather and its low cost. 4D millimeter-wave radar can not only provide distance, speed and angle information of objects, but also contains the height information of objects, giving it higher resolution in complex scenes. However, due to the sparsity and high noise of radar point cloud data, these factors limit the performance of millimeter-wave radar in target detection tasks.

[0004] Existing multimodal fusion methods use rich camera image semantic features to improve the sparse and noisy information of millimeter-wave radar to a certain extent, and thus have made good progress in 3D target detection. For example, the RCFusion method maps image features to the bird's-eye view (BEV) space through perspective transformation to better combine the geometric information of the radar. Specifically, RCFusion lifts image features to 3D space through orthographic feature transform (OFT), but due to the inherent depth ambiguity problem of 3D to 2D projection, it fails to accurately distinguish image features that exist equally along the projection ray. Although the LXL method further uses the depth estimated by monocular to distinguish image features in space, it fails to effectively use the radar depth map when optimizing the accuracy of depth estimation. The CRN method uses lift, splat, and shoot (LSS) and assists image perspective conversion through radar occupancy maps, effectively utilizing the complementary information in each modality. However, these methods have failed to fully exploit the complementary information between radar and camera, especially the deep interaction between radar geometric information and camera semantic information. They often rely on explicit geometric transformation or shallow feature fusion, and fail to fully utilize the rich information contained in each modality. Summary of the invention

[0005] Considering the lack of accuracy and robustness of existing 3D target detection methods that fuse images and radars, the present invention proposes a 3D target detection method based on dual-branch mutual fusion of millimeter-wave radar and camera images. The present invention can provide a new method for 3D target detection in autonomous driving systems, helping vehicles to accurately perceive the surrounding environment. The method of the present invention effectively extracts the unique image feature information of each modality from camera images and millimeter-wave radars, and introduces complementary branch information to generate geometrically perceived camera bird's-eye view (BEV) features and semantically perceived millimeter-wave radar BEV features. The camera BEV features and millimeter-wave radar BEV features are fused using cross-modal convolution to achieve deep interaction and fusion of multi-modal features, thereby improving the performance of 3D target detection.

[0006] On the one hand, the present invention provides a 3D target detection method based on dual-branch mutual fusion of millimeter-wave radar and camera images, and the specific steps are as follows:

[0007] S1: Implement adaptive depth completion of the image foreground area and generate geometrically aware camera BEV features. It includes the following four steps:

[0008] S11: Encode the image to obtain image features, and segment the foreground area in the image based on the image features to obtain a foreground segmentation mask. The specific method is as follows: The image features are obtained by encoding F further extracts semantic features through the semantic extraction network C is the number of channels in the feature map, H and W represent the height and width of the image respectively, and n represents the downsampling rate. The semantic features are input into the foreground segmentation network to segment the foreground area of ​​the image, thereby obtaining the foreground segmentation mask. The process can be described as:

[0009] C = ContextNet(F),

[0010] M = SegNet(C),

[0011] Among them, ContextNet(·) represents the semantic extraction network, and SegNet(·) represents the foreground segmentation network, both of which are composed of several layers of convolutional layers and linear layers.

[0012] S12: Project the millimeter-wave radar point cloud onto the camera plane to form a radar sparse depth map, and combine the radar sparse depth map with the image features obtained in step S11, and then perform depth probability estimation. The specific method is: for the millimeter-wave radar point cloud in space, project it onto the image plane to obtain a radar sparse depth map By sending it together with the image feature F into the depth completion network, we can get the depth probability estimation result Where d is the number of predefined discrete depth intervals. The depth completion network DepthNet consists of several layers of convolution Conv and the neighborhood-cross-attention (NCA) mechanism, so the process can be expressed as:

[0013] d=DepthNet(S,F)=Conv(NCA(Conv(S),Conv(F))),

[0014] S13: Calculate the loss of the depth probability estimation result obtained in step S12 and the true value of the laser radar depth, and use the foreground segmentation mask obtained in step S11 to adaptively adjust the weight of the loss, so as to optimize the depth probability estimation method and improve the accuracy of the depth probability estimation method in estimating the foreground area. When calculating the two losses, the foreground segmentation mask M is used to impose an adaptive and larger loss on the foreground area to achieve a more accurate depth probability estimate for the foreground area. The process can be expressed as:

[0015]

[0016] in, represents the loss, β represents the adaptive weight, (u,v) represents the pixel index on the image plane, Indicates that the Kullback-Leibler divergence loss is calculated for the input.

[0017] S14: Perform depth probability estimation on the image based on the depth probability estimation method optimized in step S13, perform outer product between the depth probability estimation result and the image feature, and use voxel pooling to obtain geometrically perceived camera BEV features.

[0018] S2: Accurately introduce semantic information into the radar branch and generate semantically-aware radar BEV features.

[0019] S21: Encode the millimeter-wave radar point cloud to obtain the original radar point cloud features. The process specifically includes steps such as cylinder voxelization, shallow feature extraction, and deep feature extraction. Finally, the original radar point cloud features can be obtained.

[0020] S22: Generate a virtual point cloud in the millimeter wave radar coordinate system, and fill the virtual point cloud with image features to introduce semantic information for the radar branch, which specifically includes the following steps: first, generate a series of predefined regular voxels in the millimeter wave radar coordinate system Where X and Y represent the length and width of the BEV space, respectively, and Z represents the height. The center point of each voxel is taken as the virtual point cloud, and the set of virtual point clouds of all voxels is Obviously, N = X × Y × Z. Use the camera's internal and external parameters to project N virtual point clouds onto the image plane, and use the pixel index to obtain F at the corresponding position, thereby obtaining the feature set Fill the feature set in the feature dimension to the corresponding virtual point cloud to obtain the decorated virtual point cloud Here, the present invention generates a virtual point cloud and uses image features to decorate the virtual point cloud, thereby obtaining a semantically enhanced virtual point cloud.

[0021] S23: Use the depth probability estimation result obtained in step S14 to weight the image features corresponding to the virtual point cloud, so as to convert the virtual point cloud into BEV features. Due to the projection mechanism of the virtual point cloud from 3D to 2D in S22, one image feature corresponds to multiple virtual point clouds on the projection ray. Therefore, a virtual point cloud image feature weighting mechanism is designed here to achieve accurate feature decoration. The specific method is to use the camera internal and external parameters to project N virtual point clouds onto the image plane, and use the pixel index to obtain D at the corresponding position, so as to obtain a set of depth probability estimation results. Then, according to the actual distance between the virtual point cloud and the image plane, i.e., the depth, we can index on the discrete depth space dimension D of D′ and get the probability that the virtual point cloud correctly corresponds to F′. Therefore, the present invention uses D ′′ right The image features are weighted to successfully achieve the correct image weighting of the virtual point cloud, thereby converting the virtual point cloud into BEV features.

[0022] S24: The original radar point cloud features described in step S21 are fused with the BEV features described in step S23 by convolution to obtain semantically-aware radar BEV features. Specifically, for the correctly enhanced virtual point cloud, the initial semantically enhanced radar BEV features can be obtained by encoding it in the height dimension. Next, multi-layer convolution is used to fuse R′ with the BEV feature obtained in step S23 to obtain the semantically-aware radar BEV feature.

[0023] S3: The geometrically-aware camera BEV features and the semantically-aware radar BEV features are fused using cross-modal convolution to obtain fully enhanced BEV features.

[0024] S4: Decode the fully enhanced BEV features to achieve 3D target detection and positioning.

[0025] On the other hand, the present invention provides a 3D target detection device based on the dual-branch mutual fusion of millimeter-wave radar and camera image, which includes a geometric depth completion module, a semantic radar cylinder network module, a convolution fusion module and a feature decoding module;

[0026] The geometric depth completion module is used to realize adaptive depth completion of the image foreground area and generate geometrically perceived camera BEV features;

[0027] The semantic radar column network module is used to accurately introduce semantic information into the radar branch and generate semantically-aware radar BEV features;

[0028] The convolution fusion module is used to fuse the geometrically perceived camera BEV features and the semantically perceived radar BEV features using cross-modal convolution to obtain a fully enhanced BEV feature;

[0029] The feature decoding module is used to decode the fully enhanced BEV features to achieve detection and positioning of 3D targets.

[0030] The beneficial effects of the present invention are:

[0031] The present invention discloses a 3D target detection method based on the dual-branch mutual fusion of millimeter-wave radar and camera image. The method efficiently extracts the unique features of each modality from the camera and millimeter-wave radar, and introduces complementary branch information. Specifically, in the camera branch, the method uses the geometric information of the millimeter-wave radar to generate geometrically-aware camera BEV features; in the radar branch, in view of the problem that the millimeter-wave radar point cloud lacks semantics, rich image semantic features are introduced to generate semantically-aware millimeter-wave radar BEV features. These two types of features are merged through convolution fusion to form a fully enhanced BEV feature map, which is then decoded and processed to achieve accurate 3D target detection. The method significantly improves the accuracy and robustness of 3D target detection. Experimental results show that the method can effectively detect and locate 3D targets in the environment even in complex scenes and severe weather conditions.

[0032] Compared with existing methods, the present invention not only effectively utilizes the geometric information of millimeter wave radar, but also realizes more accurate depth estimation and target detection by combining the semantic information of the camera. This method adopts the optimization process of deep learning, and can learn the essential characteristics of the 3D environment in the process of feature fusion, thereby improving the detection accuracy and generalization performance, and performs well in the categories of pedestrians, bicycles, and cars in the test data. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 It is a flowchart of a 3D target detection method based on dual-branch mutual fusion of millimeter-wave radar and camera images provided in Example 1 of the present invention.

[0034] Figure 2 It is a detection effect diagram of an embodiment of the present invention. DETAILED DESCRIPTION

[0035] The invention is further described below with reference to specific embodiments and drawings.

[0036] Example 1

[0037] Embodiment 1 of the present invention proposes a 3D target detection method based on the dual-branch mutual fusion of millimeter-wave radar and camera image. The method effectively extracts the unique feature information of each modality from the camera and millimeter-wave radar, and introduces complementary branch information to generate geometrically perceived camera BEV features and semantically perceived millimeter-wave radar BEV features; the two types of features are fused using cross-modal convolution to form a fully enhanced BEV feature map, which is then decoded to achieve accurate 3D target detection. The flowchart of the method of the present invention is as follows: Figure 1 As shown, the specific steps include:

[0038] S1: Implement adaptive depth completion of the image foreground area and generate geometry-aware camera BEV features.

[0039] S11: Encode the image to obtain image features, and segment the foreground area in the image based on the image features to obtain a foreground segmentation mask. The image features are obtained by encoding F further extracts semantic features through the semantic extraction network C is the number of channels in the feature map, H and W represent the height and width of the image respectively, and n represents the downsampling rate. The semantic features are input into the foreground segmentation network to segment the foreground area of ​​the image, thereby obtaining the foreground segmentation mask. The process can be described as:

[0040] C = ContextNet(F),

[0041] M = SegNet(C),

[0042] Among them, ContextNet(·) represents the semantic extraction network, and SegNet(·) represents the foreground segmentation network, both of which are composed of several layers of convolutional layers and linear layers.

[0043] S12: Project the millimeter-wave radar point cloud onto the camera plane to form a radar sparse depth map, and combine the radar sparse depth map with the image features obtained in step S11, and then perform depth probability estimation. Specifically, for the millimeter-wave radar point cloud in space, project it onto the image plane to obtain a radar sparse depth map By sending it together with the image feature F into the depth completion network, we can get the depth probability estimation result Where D is the number of predefined discrete depth intervals. The depth completion network DepthNet consists of several layers of convolution Conv and the neighborhood-cross-attention (NCA) mechanism, so the process can be expressed as:

[0044] D=DepthNet(S,F)=Conv(NCA(Conv(S),Conv(F))),

[0045] S13: Calculate the loss of the depth probability estimation result obtained in step S12 and the true value of the laser radar depth, and use the foreground segmentation mask obtained in step S11 to adaptively adjust the weight of the loss, so as to optimize the depth probability estimation method and improve the accuracy of the depth probability estimation method in estimating the foreground area. When calculating the two losses, the foreground segmentation mask M is used to impose an adaptive and larger loss on the foreground area to achieve a more accurate depth probability estimate for the foreground area. The process can be expressed as:

[0046]

[0047] in, represents the loss, β represents the adaptive weight, (u,v) represents the pixel index on the image plane, Indicates that the Kullback-Leibler divergence loss is calculated for the input.

[0048] S14: Perform depth probability estimation on the image based on the depth probability estimation method optimized in step S13, perform outer product between the depth probability estimation result and the image feature, and use voxel pooling to obtain geometrically perceived camera BEV features.

[0049] S2: Accurately introduce semantic information into the radar branch and generate semantically-aware radar BEV features.

[0050] S21: Encode the millimeter-wave radar point cloud to obtain the original radar point cloud features. The process includes steps such as cylinder voxelization, shallow feature extraction, and deep feature extraction. Finally, the original radar point cloud features can be obtained.

[0051] S22: Generate a virtual point cloud in the millimeter wave radar coordinate system, and fill the virtual point cloud with image features to introduce semantic information for the radar branch, which specifically includes the following steps: first, generate a series of predefined regular voxels in the millimeter wave radar coordinate system Where X and Y represent the length and width of the BEV space, respectively, and Z represents the height. The center point of each voxel is taken as the virtual point cloud, and the set of virtual point clouds of all voxels is Obviously, N = X × Y × Z. Use the camera's internal and external parameters to project N virtual point clouds onto the image plane, and use the pixel index to obtain F at the corresponding position, thereby obtaining the feature set Fill the feature set in the feature dimension to the corresponding virtual point cloud to obtain the decorated virtual point cloud

[0052] S23: Using the depth probability estimation result obtained in step S14, weight the image features corresponding to the virtual point cloud, so as to convert the virtual point cloud into BEV features. Specifically, the N virtual point clouds are projected onto the image plane using the camera internal and external parameters, and the corresponding position D can be obtained using the pixel index, so as to obtain a depth probability estimation result set Then, according to the actual distance between the virtual point cloud and the image plane, i.e., the depth, we can index on the discrete depth space dimension D of D′ and get the probability that the virtual point cloud correctly corresponds to F′. Use D″ By weighting the image features, the correct image weighting of the virtual point cloud can be successfully achieved, thereby converting the virtual point cloud into BEV features.

[0053] S24: The original radar point cloud features described in step S21 are fused with the BEV features described in step S23 by convolution to obtain semantically-aware radar BEV features. Specifically, for the correctly enhanced virtual point cloud, the initial semantically enhanced radar BEV features are obtained by encoding it in the height dimension. Next, multi-layer convolution is used to fuse R′ with the BEV feature obtained in step S23 to obtain the semantically-aware radar BEV feature.

[0054] S3: The geometrically-aware camera BEV features and the semantically-aware radar BEV features are fused using cross-modal convolution to obtain fully enhanced BEV features.

[0055] S4: Decode the fully enhanced BEV features to achieve 3D target detection and positioning.

[0056] Example 2

[0057] Embodiment 2 of the present invention provides a 3D target detection device based on the dual-branch mutual fusion of millimeter-wave radar and camera image. The device effectively extracts the unique feature information of each modality from the camera image and the millimeter-wave radar by integrating the geometric depth completion module and the semantic radar cylinder network, and introduces complementary branch information to generate geometrically perceived camera bird's-eye view (BEV) features and semantically perceived millimeter-wave radar BEV features. Deep interaction and fusion of multimodal features are achieved, thereby improving the performance of 3D target detection.

[0058] The device includes a geometric depth completion module, a semantic radar cylinder network module, a convolutional fusion module and a feature decoding module;

[0059] The geometric depth completion module is used to realize adaptive depth completion of the image foreground area and generate geometrically perceived camera BEV features;

[0060] The semantic radar column network module is used to accurately introduce semantic information into the radar branch and generate semantically-aware radar BEV features;

[0061] The convolution fusion module is used to fuse the geometrically perceived camera BEV features and the semantically perceived radar BEV features using cross-modal convolution to obtain a fully enhanced BEV feature;

[0062] The feature decoding module is used to decode the fully enhanced BEV features to achieve detection and positioning of 3D targets.

[0063] Target detection based on the above device includes the following steps:

[0064] First, the geometric depth completion module is used to achieve adaptive depth completion of the image foreground area and generate geometrically perceived camera BEV features; the specific method is as follows:

[0065] The original image is encoded using an image encoder to obtain image features. The image features are used to obtain semantic features through a semantic extraction network. The semantic features are input into a foreground segmentation network to segment the foreground area in the image and obtain a foreground segmentation mask. During the training of the foreground segmentation network, two true value labels are used for loss calculation: the foreground mask pre-segmented by Dectron2 and the foreground mask generated by the 2D box true value.

[0066] Projecting the millimeter-wave radar point cloud onto the camera plane to form a radar sparse depth map, combining the radar sparse depth map with the image features, and then performing depth probability estimation;

[0067] The obtained depth probability estimation result and the true value of the laser radar depth are subjected to loss calculation, and the obtained foreground segmentation mask is used to adaptively adjust the weight of the loss, thereby optimizing the depth probability estimation method and improving the accuracy of the depth probability estimation method in estimating the foreground area;

[0068] The depth probability of the image is estimated based on the optimized depth probability estimation method, the outer product of the depth probability estimation result and the image feature is performed, and voxel pooling is used to obtain the geometrically perceived camera BEV feature.

[0069] Then, the semantic radar cylinder network module is used to accurately introduce semantic information into the radar branch and generate semantically aware radar BEV features; the specific method is as follows:

[0070] The millimeter-wave radar point cloud is encoded using point cloud pillar encoders, and the process includes pillar voxelization, shallow feature extraction, and deep feature extraction to obtain the original radar point cloud features;

[0071] Generate a virtual point cloud in a millimeter wave radar coordinate system, and fill the virtual point cloud with image features to introduce semantic information for the radar branch;

[0072] Using the depth probability estimation result in the geometric depth completion module, weighting the image features corresponding to the virtual point cloud, thereby converting the virtual point cloud into BEV features;

[0073] The original radar point cloud features are fused with the BEV features using convolution to obtain semantically-aware radar BEV features; specifically, for the correctly enhanced virtual point cloud, a multi-layer perceptron is used to encode it in the height dimension to obtain the initial semantically-enhanced radar BEV features, and then, the initial semantically-enhanced radar BEV features are fused with the BEV features using multi-layer convolution to obtain the semantically-aware radar BEV features.

[0074] Next, the geometrically perceived camera BEV features and the semantically perceived radar BEV features are fused using cross-modal convolution to obtain fully enhanced BEV features.

[0075] Finally, the fully enhanced BEV features are sent to the detection head and decoded using the anchor-based feature decoding head to achieve 3D target detection and positioning.

[0076] Figure 2The results of running the method of the present invention on the View-of-Delft (VoD) dataset for a specific embodiment. Each image corresponds to a data frame containing images and radar points (gray), and the red triangle marks the position of the vehicle. The orange and yellow boxes represent the real boxes in the perspective view and the bird's-eye view, respectively. The green and blue boxes represent the bounding boxes predicted by the method of the present invention, and the visualization of the BEV feature map is shown in the lower left half of the figure. Figure 2 Figures (a), (b) and (c) show the detection performance of the proposed method on cars, cyclists and pedestrians in VoD. As can be seen from the figure, the proposed method has better detection effects on cars, cyclists and pedestrians, and improves the performance of 3D target detection of the millimeter wave radar-camera fusion model.

[0077] Starting from the essence of 3D target detection tasks, the method of the present invention regards the 3D target detection task of millimeter-wave radar and camera images as a deep fusion problem of geometric features and semantic features, and optimizes the fusion problem through deep learning. Specifically, this method can effectively extract and fuse the sparse geometric information of the radar and the rich semantic information of the camera. Through dual-branch mutual fusion and cross-modal feature interaction based on the attention mechanism, the network can learn the key features of 3D target detection and improve the target detection capabilities under different environments and conditions. Experimental results show that this method significantly improves the generalization and robustness of current multimodal 3D target detection methods based on deep learning in complex scenes and different data domains.

Claims

1. A 3D target detection method based on dual-branch mutual fusion of millimeter-wave radar and camera images, characterized in that: By effectively extracting the unique image feature information of each modality from the camera and millimeter-wave radar and introducing complementary branch information, geometrically aware camera BEV features and semantically aware millimeter-wave radar BEV features are generated; the two types of features are fused using cross-modal convolution to form a fully enhanced BEV feature map, which is then decoded to achieve accurate 3D target detection.

2. The 3D target detection method based on dual-branch mutual fusion of millimeter-wave radar and camera images according to claim 1 is characterized in that: The specific steps are as follows: S1: Implement adaptive depth completion of the image foreground area and generate geometry-aware camera BEV features; S11: Encode the image to obtain image features, and segment the foreground area in the image based on the image features to obtain a foreground segmentation mask; S12: Projecting the millimeter-wave radar point cloud onto the camera plane to form a radar sparse depth map, combining the radar sparse depth map with the image features obtained in step S11, and then performing depth probability estimation; S13: performing loss calculation on the depth probability estimation result obtained in step S12 and the true value of the laser radar depth, and using the foreground segmentation mask obtained in step S11 to adaptively adjust the weight of the loss, thereby optimizing the depth probability estimation method and improving the accuracy of the depth probability estimation method in estimating the foreground area; S14: performing depth probability estimation on the image based on the depth probability estimation method optimized in step S13, performing outer product between the depth probability estimation result and the image feature, and using voxel pooling to obtain geometrically perceived camera BEV features; S2: Accurately introduce semantic information into the radar branch to generate semantically aware radar BEV features; S21: Encode the millimeter wave radar point cloud to obtain the original radar point cloud features; S22: generating a virtual point cloud in a millimeter wave radar coordinate system, and filling the virtual point cloud with image features, thereby introducing semantic information for the radar branch; S23: using the depth probability estimation result obtained in step S14, weighting the image features corresponding to the virtual point cloud, thereby converting the virtual point cloud into BEV features; S24: The original radar point cloud features described in step S21 are fused with the BEV features described in step S23 by convolution to obtain semantically perceived radar BEV features; S3: using cross-modal convolution to fuse the geometrically perceived camera BEV features and the semantically perceived radar BEV features to obtain a fully enhanced BEV feature; S4: Decode the fully enhanced BEV features to achieve 3D target detection and positioning.

3. The 3D image processing system based on the dual-branch mutual fusion of millimeter-wave radar and camera images according to claim 2 The target detection method is characterized by: The S11 is specifically as follows: The image features are obtained by encoding F further extracts semantic features through the semantic extraction network C is the number of channels of the image feature, H and W represent the height and width of the image respectively, and n represents the downsampling rate; the semantic features are input into the foreground segmentation network to segment the foreground area of ​​the image, thereby obtaining the foreground segmentation mask The above process can be expressed as: C = ContextNet(F), M = SegNet(C), Among them, ContextNet(·) represents the semantic extraction network, and SegNet(·) represents the foreground segmentation network, both of which are composed of several layers of convolutional layers and linear layers.

4. The 3D image processing system based on the dual-branch mutual fusion of millimeter-wave radar and camera images according to claim 3 The target detection method is characterized by: The S12 is specifically as follows: for the millimeter wave radar point cloud in space, project it onto the image plane to obtain a radar sparse depth map Send it together with the image feature F into the depth completion network to obtain the depth probability estimation result Where D is the number of predefined discrete depth intervals; the process can be expressed as: D = DepthNet(S,F), Among them, DepthNet(·) represents a deep completion network, which consists of several layers of convolution and neighborhood attention.

5. The 3D image processing system based on the dual-branch mutual fusion of millimeter-wave radar and camera images according to claim 4 The target detection method is characterized by: In S13: for the depth probability estimation result D and the true value of the laser radar depth When calculating the two losses, the foreground segmentation mask M is used to apply adaptive loss to the foreground area to achieve a more accurate depth probability estimate for the foreground area. The process is expressed as: in, represents the loss, β represents the adaptive weight, (u,v) represents the pixel index on the image plane, Indicates that the Kullback-Leibler divergence loss is calculated for the input.

6. The 3D target detection method based on dual-branch mutual fusion of millimeter-wave radar and camera images according to claim 2 is characterized in that: The S22 is specifically as follows: firstly, a series of predefined regular voxels are generated in the millimeter wave radar coordinate system. Where X and Y represent the length and width of the BEV space, respectively, and Z represents the height. The center point of each voxel is taken as the virtual point cloud, and the set of virtual point clouds of all voxels is Obviously, N = X × Y × Z; use the camera's internal and external parameters to project N virtual point clouds onto the image plane, and use the pixel index to obtain F at the corresponding position, thereby obtaining the feature set Fill all the features in the feature set F′ into the corresponding virtual point cloud in the feature dimension to obtain the decorated virtual point cloud 7. The 3D target detection method based on dual-branch mutual fusion of millimeter-wave radar and camera images according to claim 6 is characterized in that: The S23 is specifically as follows: projecting N virtual point clouds onto the image plane using the camera internal and external parameters, and obtaining D at the corresponding position using the pixel index, thereby obtaining a depth probability estimation result set. Then, according to the actual distance between the virtual point cloud and the image plane, the probability of the virtual point cloud corresponding to F′ is obtained by indexing on the discrete depth space dimension D of D′. Using D ω′ right The image features are weighted to convert the virtual point cloud into BEV features.

8. A 3D target detection device based on dual-branch mutual fusion of millimeter-wave radar and camera images, characterized in that: It includes geometric depth completion module, semantic radar cylinder network module, convolution fusion module and feature decoding module; The geometric depth completion module is used to realize adaptive depth completion of the image foreground area and generate geometrically perceived camera BEV features; The semantic radar column network module is used to accurately introduce semantic information into the radar branch and generate semantically-aware radar BEV features; The convolution fusion module is used to fuse the geometrically perceived camera BEV features and the semantically perceived radar BEV features using cross-modal convolution to obtain a fully enhanced BEV feature; The feature decoding module is used to decode the fully enhanced BEV features to achieve detection and positioning of 3D targets.

Images