3D target detection method based on dual-branch mutual fusion of millimeter-wave radar and camera images

Abstract

The present invention discloses a 3D target detection method based on the dual-branch mutual fusion of millimeter-wave radar and camera images. This method efficiently extracts the unique image features of each modality from the camera and millimeter-wave radar, and introduces complementary branch information. Specifically, this method generates geometrically perceived camera bird's-eye view (BEV) features and semantically perceived millimeter-wave radar BEV features. These two types of features are fused to form a fully enhanced BEV feature map, which is then decoded to achieve accurate 3D target detection. This method significantly improves the accuracy and robustness of 3D target detection. Experimental results show that this method can effectively detect and locate 3D targets in the environment even in complex scenes and severe weather conditions.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to a 3D target detection method based on dual-branch mutual fusion of millimeter-wave radar and camera images. Background Art

[0002] With the rapid development of autonomous driving technology, 3D object detection technology plays a vital role in autonomous driving systems. It helps vehicles accurately perceive their surroundings and provides key information such as location and speed, thereby achieving safe driving. Currently, the fusion of laser radar (LiDAR) and cameras is the mainstream method for 3D object detection. LiDAR can provide high-precision geometric depth information, while cameras can capture rich semantic information. The complementary nature of the two provides efficient perception capabilities for 3D object detection. However, LiDAR sensors are expensive and perform poorly in adverse weather conditions.

[0003] In contrast, millimeter-wave radar (4D radar) is becoming an alternative in the field of autonomous driving perception due to its stable operation in inclement weather and low cost. 4D millimeter-wave radar not only provides distance, speed, and angle information for objects, but also includes height information, enabling higher resolution in complex scenes. However, the sparsity and high noise of radar point cloud data limit the performance of millimeter-wave radar in object detection tasks.

[0004] Existing multimodal fusion methods utilize the rich semantic features of camera images to improve the sparse and noisy information of millimeter-wave radar, thus making significant progress in 3D object detection. For example, the RCFusion method maps image features into the bird's-eye view (BEV) space through perspective transformation to better incorporate radar geometric information. Specifically, RCFusion elevates image features to 3D space through the orthographic feature transform (OFT). However, due to the inherent depth ambiguity of the 3D-to-2D projection, it fails to accurately distinguish image features that coexist along projection rays. Although the LXL method further utilizes monocular depth estimation to distinguish image features in space, it fails to effectively utilize the radar depth map to optimize depth estimation accuracy. The CRN method uses lift, splat, and shoot (LSS) and uses radar occupancy maps to assist with image perspective transformation, effectively leveraging the complementary information between the modalities. However, these methods fail to fully exploit the complementary information between radar and camera, particularly the deep interaction between radar geometric information and camera semantic information. They often rely on explicit geometric transformations or shallow feature fusion, failing to fully utilize the rich information contained in each modality. Summary of the Invention

[0005] Taking into account the insufficient accuracy and robustness of existing 3D target detection methods that fuse images and radars, the present invention proposes a 3D target detection method based on the dual-branch mutual fusion of millimeter-wave radar and camera images. The present invention can provide a new method for 3D target detection in autonomous driving systems, helping vehicles to accurately perceive the surrounding environment. The method of the present invention effectively extracts the unique image feature information of each modality from camera images and millimeter-wave radars, and introduces complementary branch information to generate geometrically perceived camera bird's-eye view (BEV) features and semantically perceived millimeter-wave radar BEV features. The camera BEV features and millimeter-wave radar BEV features are fused using cross-modal convolution to achieve deep interaction and fusion of multimodal features, thereby improving the performance of 3D target detection.

[0006] On one hand, the present invention provides a 3D target detection method based on dual-branch mutual fusion of millimeter-wave radar and camera images, and the specific steps are as follows:

[0007] S1: Implement adaptive depth completion of the image foreground area and generate geometrically aware camera BEV features. This includes the following four steps:

[0008] S11: Encode the image to obtain image features, and segment the foreground area in the image based on the image features to obtain a foreground segmentation mask. The specific method is as follows: for the image The image features are obtained by encoding F further extracts semantic features through the semantic extraction network C is the number of channels in the feature map, H and W represent the height and width of the image respectively, and n represents the downsampling rate. The semantic features are input into the foreground segmentation network to segment the foreground area of ​​the image, thereby obtaining the foreground segmentation mask. The process can be expressed as:

[0009] C=ContextNet(F),

[0010] M=SegNet(C),

[0011] Among them, ContextNet(·) represents the semantic extraction network, and SegNet(·) represents the foreground segmentation network, both of which are composed of several layers of convolutional layers and linear layers.

[0012] S12: Project the millimeter-wave radar point cloud onto the camera plane to form a radar sparse depth map, and combine the radar sparse depth map with the image features obtained in step S11, and then perform depth probability estimation. The specific method is: for the millimeter-wave radar point cloud in space, project it onto the image plane to obtain a radar sparse depth map By sending it together with the image feature F into the depth completion network, we can get the depth probability estimation result. Where d is the number of predefined discrete depth intervals. The depth completion network DepthNet consists of several layers of convolution Conv and the neighborhood-cross-attention (NCA) mechanism, so the process can be expressed as:

[0013] d=DepthNet(S,F)=Conv(NCA(Conv(S),Conv(F))),

[0014] S13: Calculate the loss between the depth probability estimation result obtained in step S12 and the true depth value of the laser radar, and use the foreground segmentation mask obtained in step S11 to adaptively adjust the weight of the loss, thereby optimizing the depth probability estimation method and improving the accuracy of the depth probability estimation method in estimating the foreground area. When calculating the two losses, the foreground segmentation mask M is used to impose an adaptive and larger loss on the foreground area to achieve a more accurate depth probability estimation of the foreground area. The process can be expressed as:

[0015]

[0016] in, represents the loss, β represents the adaptive weight, (u, v) represents the pixel index on the image plane, Indicates that the Kullback-Leibler divergence loss is calculated for the input.

[0017] S14: Perform depth probability estimation on the image based on the depth probability estimation method optimized in step S13, perform outer product between the depth probability estimation result and the image features, and use voxel pooling to obtain geometrically perceived camera BEV features.

[0018] S2: Accurately introduce semantic information into the radar branch to generate semantically aware radar BEV features.

[0019] S21: Encode the millimeter-wave radar point cloud to obtain the original radar point cloud features. The process specifically includes steps such as cylinder voxelization, shallow feature extraction, and deep feature extraction. Finally, the original radar point cloud features can be obtained.

[0020] S22: Generate a virtual point cloud in the millimeter-wave radar coordinate system, and fill the virtual point cloud with image features to introduce semantic information for the radar branch, specifically including the following steps: First, generate a series of predefined regular voxels in the millimeter-wave radar coordinate system Where X and Y represent the length and width of the BEV space respectively, and Z represents the height. The center point of each voxel is taken as the virtual point cloud, and the set of virtual point clouds of all voxels is Obviously, N = X × Y × Z. Use the camera's internal and external parameters to project N virtual point clouds onto the image plane, and use the pixel index to obtain F at the corresponding position, thereby obtaining the feature set Fill the feature set in the feature dimension to the corresponding virtual point cloud to obtain the decorated virtual point cloud Here, the present invention generates a virtual point cloud and uses image features to decorate the virtual point cloud, thereby obtaining a semantically enhanced virtual point cloud.

[0021] S23: Use the depth probability estimation result obtained in step S14 to weight the image features corresponding to the virtual point cloud, thereby converting the virtual point cloud into BEV features. Since the projection mechanism of the virtual point cloud from 3D to 2D in S22 will result in one image feature corresponding to multiple virtual point clouds on the projection ray, a virtual point cloud image feature weighting mechanism is designed here to achieve accurate feature decoration. The specific method is to use the internal and external parameters of the camera to project N virtual point clouds onto the image plane, and use the pixel index to obtain D of the corresponding position, thereby obtaining a set of depth probability estimation results. Then, according to the actual distance between the virtual point cloud and the image plane, i.e., the depth, we can index on the discrete depth space dimension D of D′ and get the probability that the virtual point cloud correctly corresponds to F′. Therefore, the present invention uses D ′′ right The image features are weighted to successfully achieve the correct image weighting of the virtual point cloud, thereby converting the virtual point cloud into BEV features.

[0022] S24: The original radar point cloud features described in step S21 are fused with the BEV features described in step S23 using convolution to obtain semantically perceived radar BEV features. Specifically, for the correctly enhanced virtual point cloud, encoding it in the height dimension can obtain the initial semantically enhanced radar BEV features. Next, multi-layer convolution is used to fuse R′ with the BEV features obtained in step S23 to obtain the semantically perceived radar BEV features.

[0023] S3: Using cross-modal convolution, the geometrically perceived camera BEV features and the semantically perceived radar BEV features are fused to obtain fully enhanced BEV features.

[0024] S4: Decode the fully enhanced BEV features to achieve 3D target detection and positioning.

[0025] On the other hand, the present invention provides a 3D target detection device based on the dual-branch mutual fusion of millimeter-wave radar and camera images, which includes a geometric depth completion module, a semantic radar cylinder network module, a convolution fusion module and a feature decoding module;

[0026] The geometric depth completion module is used to achieve adaptive depth completion of the image foreground area and generate geometrically perceived camera BEV features;

[0027] The semantic radar column network module is used to accurately introduce semantic information into the radar branch and generate semantically aware radar BEV features;

[0028] The convolution fusion module is used to fuse the geometrically perceived camera BEV features and the semantically perceived radar BEV features using cross-modal convolution to obtain fully enhanced BEV features;

[0029] The feature decoding module is used to decode the fully enhanced BEV features to achieve 3D target detection and positioning.

[0030] The beneficial effects of the present invention are:

[0031] The present invention provides a 3D target detection method based on the dual-branch mutual fusion of millimeter-wave radar and camera images. The method efficiently extracts the unique features of each modality from the camera and millimeter-wave radar, and introduces complementary branch information. Specifically, in the camera branch, the method uses the geometric information of the millimeter-wave radar to generate geometrically-aware camera BEV features; in the radar branch, to address the problem of lack of semantics in the millimeter-wave radar point cloud, rich image semantic features are introduced to generate semantically-aware millimeter-wave radar BEV features. These two types of features are merged through convolution fusion to form a fully enhanced BEV feature map, which is then decoded and processed to achieve accurate 3D target detection. This method significantly improves the accuracy and robustness of 3D target detection. Experimental results show that even in complex scenes and severe weather conditions, this method can effectively detect and locate 3D targets in the environment.

[0032] Compared to existing methods, this method not only effectively utilizes the geometric information of millimeter-wave radar but also achieves more accurate depth estimation and object detection by combining semantic information from the camera. This method utilizes a deep learning optimization process to learn the essential characteristics of the 3D environment during feature fusion, thereby improving detection accuracy and generalization performance. It performs well on pedestrians, bicycles, and cars in test data. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 This is a flowchart of a 3D target detection method based on dual-branch mutual fusion of millimeter-wave radar and camera images provided in Example 1 of the present invention.

[0034] Figure 2 2 is a diagram showing the detection effect of an embodiment of the present invention. DETAILED DESCRIPTION

[0035] The following is further described with reference to specific embodiments and drawings.

[0036] Example 1

[0037] Embodiment 1 of the present invention proposes a 3D target detection method based on the dual-branch mutual fusion of millimeter-wave radar and camera images. This method effectively extracts the unique feature information of each modality from the camera and millimeter-wave radar, and introduces complementary branch information to generate geometrically perceived camera BEV features and semantically perceived millimeter-wave radar BEV features; cross-modal convolution is used to fuse the two types of features to form a fully enhanced BEV feature map, which is then decoded to achieve accurate 3D target detection. The flowchart of the method of the present invention is as follows Figure 1 As shown, the specific steps include:

[0038] S1: Implement adaptive depth completion of the image foreground area and generate geometry-aware camera BEV features.

[0039] S11: Encode the image to obtain image features, and segment the foreground area in the image based on the image features to obtain a foreground segmentation mask. Specifically, for the image The image features are obtained by encoding F further extracts semantic features through the semantic extraction network C is the number of channels in the feature map, H and W represent the height and width of the image respectively, and n represents the downsampling rate. The semantic features are input into the foreground segmentation network to segment the foreground area of ​​the image, thereby obtaining the foreground segmentation mask. The process can be expressed as:

[0040] C=ContextNet(F),

[0041] M=SegNet(C),

[0042] Among them, ContextNet(·) represents the semantic extraction network, and SegNet(·) represents the foreground segmentation network, both of which are composed of several layers of convolutional layers and linear layers.

[0043] S12: Project the millimeter-wave radar point cloud onto the camera plane to form a radar sparse depth map, and combine the radar sparse depth map with the image features obtained in step S11, and then perform depth probability estimation. Specifically, for the millimeter-wave radar point cloud in space, project it onto the image plane to obtain a radar sparse depth map. By sending it together with the image feature F into the depth completion network, we can get the depth probability estimation result. Where D is the number of predefined discrete depth intervals. The depth completion network DepthNet consists of several layers of convolution Conv and the neighborhood-cross-attention (NCA) mechanism, so the process can be expressed as:

[0044] D=DepthNet(S,F)=Conv(NCA(Conv(S),Conv(F))),

[0045] S13: Calculate the loss between the depth probability estimation result obtained in step S12 and the true depth value of the laser radar, and use the foreground segmentation mask obtained in step S11 to adaptively adjust the weight of the loss, thereby optimizing the depth probability estimation method and improving the accuracy of the depth probability estimation method in estimating the foreground area. When calculating the two losses, the foreground segmentation mask M is used to impose an adaptive and larger loss on the foreground area to achieve a more accurate depth probability estimation of the foreground area. The process can be expressed as:

[0046]

[0047] in, represents the loss, β represents the adaptive weight, (u, v) represents the pixel index on the image plane, Indicates that the Kullback-Leibler divergence loss is calculated for the input.

[0048] S14: Perform depth probability estimation on the image based on the depth probability estimation method optimized in step S13, perform outer product between the depth probability estimation result and the image features, and use voxel pooling to obtain geometrically perceived camera BEV features.

[0049] S2: Accurately introduce semantic information into the radar branch to generate semantically aware radar BEV features.

[0050] S21: Encode the millimeter-wave radar point cloud to obtain the original radar point cloud features. The process includes steps such as cylinder voxelization, shallow feature extraction, and deep feature extraction. Finally, the original radar point cloud features can be obtained.

[0051] S22: Generate a virtual point cloud in the millimeter-wave radar coordinate system, and fill the virtual point cloud with image features to introduce semantic information for the radar branch, specifically including the following steps: First, generate a series of predefined regular voxels in the millimeter-wave radar coordinate system Where X and Y represent the length and width of the BEV space respectively, and Z represents the height. The center point of each voxel is taken as the virtual point cloud, and the set of virtual point clouds of all voxels is Obviously, N = X × Y × Z. Use the camera's internal and external parameters to project N virtual point clouds onto the image plane, and use the pixel index to obtain F at the corresponding position, thereby obtaining the feature set Fill the feature set in the feature dimension to the corresponding virtual point cloud to obtain the decorated virtual point cloud

[0052] S23: Using the depth probability estimation result obtained in step S14, weight the image features corresponding to the virtual point cloud, thereby converting the virtual point cloud into BEV features. Specifically, the N virtual point clouds are projected onto the image plane using the camera internal and external parameters, and the corresponding position D can be obtained using the pixel index, thereby obtaining a set of depth probability estimation results. Then, according to the actual distance between the virtual point cloud and the image plane, i.e., the depth, we can index on the discrete depth space dimension D of D′ and get the probability that the virtual point cloud correctly corresponds to F′. Use D″ By weighting the image features of the virtual point cloud, the correct image weighting of the virtual point cloud can be successfully achieved, thereby converting the virtual point cloud into BEV features.

[0053] S24: The original radar point cloud features described in step S21 are fused with the BEV features described in step S23 using convolution to obtain semantically perceived radar BEV features. Specifically, for the correctly enhanced virtual point cloud, the initial semantically enhanced radar BEV features can be obtained by encoding it in the height dimension. Next, multi-layer convolution is used to fuse R′ with the BEV features obtained in step S23 to obtain the semantically perceived radar BEV features.

[0054] S3: Using cross-modal convolution, the geometrically perceived camera BEV features and the semantically perceived radar BEV features are fused to obtain fully enhanced BEV features.

[0055] S4: Decode the fully enhanced BEV features to achieve 3D target detection and positioning.

[0056] Example 2

[0057] Embodiment 2 of the present invention provides a 3D object detection device based on the dual-branch mutual fusion of millimeter-wave radar and camera images. By integrating a geometric depth completion module and a semantic radar column network, the device effectively extracts the unique feature information of each modality from the camera image and millimeter-wave radar, and introduces complementary branch information to generate geometrically perceived camera bird's-eye view (BEV) features and semantically perceived millimeter-wave radar BEV features. This achieves deep interaction and fusion of multimodal features, thereby improving the performance of 3D object detection.

[0058] The device includes a geometric depth completion module, a semantic radar cylinder network module, a convolutional fusion module and a feature decoding module;

[0059] The geometric depth completion module is used to achieve adaptive depth completion of the image foreground area and generate geometrically perceived camera BEV features;

[0060] The semantic radar column network module is used to accurately introduce semantic information into the radar branch and generate semantically aware radar BEV features;

[0061] The convolution fusion module is used to fuse the geometrically perceived camera BEV features and the semantically perceived radar BEV features using cross-modal convolution to obtain fully enhanced BEV features;

[0062] The feature decoding module is used to decode the fully enhanced BEV features to achieve 3D target detection and positioning.

[0063] Target detection based on the above device includes the following steps:

[0064] First, the geometric depth completion module is used to implement adaptive depth completion of the image foreground area and generate geometrically aware camera BEV features. The specific method is as follows:

[0065] The original image is encoded using an image encoder to obtain image features. The image features are then passed through a semantic extraction network to obtain semantic features. The semantic features are then input into a foreground segmentation network to segment the foreground area in the image and obtain a foreground segmentation mask. During the training of the foreground segmentation network, two true value labels are used for loss calculation: the foreground mask pre-segmented by Dectron2 and the foreground mask generated by the 2D box true value.

[0066] Projecting the millimeter-wave radar point cloud onto the camera plane to form a radar sparse depth map, combining the radar sparse depth map with the image features, and then performing depth probability estimation;

[0067] The obtained depth probability estimation result and the true value of the lidar depth are subjected to loss calculation, and the obtained foreground segmentation mask is used to adaptively adjust the weight of the loss, thereby optimizing the depth probability estimation method and improving the accuracy of the depth probability estimation method in estimating the foreground area;

[0068] The depth probability of the image is estimated based on the optimized depth probability estimation method, the outer product of the depth probability estimation result and the image features is performed, and voxel pooling is used to obtain the geometrically perceived camera BEV features.

[0069] Then, the semantic radar column network module is used to accurately introduce semantic information into the radar branch and generate semantically aware radar BEV features. The specific method is as follows:

[0070] The millimeter-wave radar point cloud is encoded using point cloud pillar encoders. The process includes pillar voxelization, shallow feature extraction, and deep feature extraction to obtain the original radar point cloud features.

[0071] generating a virtual point cloud in a millimeter-wave radar coordinate system, and filling the virtual point cloud with image features to introduce semantic information into the radar branch;

[0072] Using the depth probability estimation result in the geometric depth completion module, weighting the image features corresponding to the virtual point cloud, thereby converting the virtual point cloud into BEV features;

[0073] The original radar point cloud features are fused with the BEV features using convolution to obtain semantically-aware radar BEV features. Specifically, the correctly enhanced virtual point cloud is encoded in the height dimension using a multi-layer perceptron to obtain initial semantically-enhanced radar BEV features. Subsequently, the initial semantically-enhanced radar BEV features are fused with BEV features using multi-layer convolution to obtain semantically-aware radar BEV features.

[0074] Then, cross-modal convolution is used to fuse the geometrically perceived camera BEV features and the semantically perceived radar BEV features to obtain a fully enhanced BEV feature.

[0075] Finally, the fully enhanced BEV features are fed into the detection head, and the anchor-based feature decoding head is used to decode the fully enhanced BEV features to achieve 3D object detection and positioning.

[0076] Figure 2Figure 1 shows the results of a specific embodiment of the method based on the present invention on the View-of-Delft (VoD) dataset. Each image corresponds to a data frame containing the image and radar points (gray). The red triangle marks the vehicle's position. The orange and yellow boxes represent the ground truth boxes in the perspective and bird's-eye views, respectively. The green and blue boxes represent the bounding boxes predicted by the method. The lower left half of the figure shows a visualization of the BEV feature map. Figure 2 Figures (a), (b), and (c) show the detection performance of our method for cars, cyclists, and pedestrians in VoD, respectively. As can be seen from the figures, our method performs well for detecting cars, cyclists, and pedestrians, improving the 3D object detection performance of the millimeter-wave radar-camera fusion model.

[0077] Starting from the essence of the 3D target detection task, the method of the present invention regards the 3D target detection task of millimeter-wave radar and camera images as a deep fusion problem of geometric features and semantic features, and optimizes this fusion problem through deep learning. Specifically, this method can effectively extract and fuse the sparse geometric information of the radar and the rich semantic information of the camera. Through dual-branch mutual fusion and cross-modal feature interaction based on the attention mechanism, the network can learn the key features of 3D target detection and improve the target detection capability in different environments and conditions. Experimental results show that this method significantly improves the generalization and robustness of current multimodal 3D target detection methods based on deep learning in complex scenes and different data domains.

Claims

1. A 3D target detection method based on dual-branch fusion of millimeter-wave radar and camera images, characterized in that: By effectively extracting unique image feature information from each modality, and introducing complementary branch information, the system generates geometrically aware camera BEV features and semantically aware millimeter-wave radar BEV features. Cross-modal convolution is used to fuse these two features to form a fully enhanced BEV feature map, which is then decoded for accurate 3D object detection. The specific steps are as follows: S1: Implement adaptive depth completion of the image foreground area and generate geometrically aware camera BEV features; S11: Encode the image to obtain image features, and segment the foreground area in the image based on the image features to obtain a foreground segmentation mask; S12: Projecting the millimeter-wave radar point cloud onto the camera plane to form a radar sparse depth map, combining the radar sparse depth map with the image features obtained in step S11, and then performing depth probability estimation; S13: performing loss calculation on the depth probability estimation result obtained in step S12 and the true value of the lidar depth, and adaptively adjusting the weight of the loss using the foreground segmentation mask obtained in step S11, thereby optimizing the depth probability estimation method and improving the accuracy of the depth probability estimation method in estimating the foreground area; S14: performing depth probability estimation on the image based on the depth probability estimation method optimized in step S13, performing outer product between the depth probability estimation result and the image feature, and using voxel pooling to obtain geometrically perceived camera BEV features; S2: Accurately introduce semantic information into the radar branch to generate semantically aware radar BEV features; S21: Encode the millimeter-wave radar point cloud to obtain the original radar point cloud features; S22: Generate a virtual point cloud in the millimeter-wave radar coordinate system, and fill the virtual point cloud with image features to introduce semantic information into the radar branch; S23: Using the depth probability estimation result obtained in step S14, weighting the image features corresponding to the virtual point cloud, thereby converting the virtual point cloud into BEV features; S24: The original radar point cloud features described in step S21 are fused with the BEV features described in step S23 using convolution to obtain semantically perceived radar BEV features; S3: fusing the geometrically perceived camera BEV features and the semantically perceived radar BEV features using cross-modal convolution to obtain fully enhanced BEV features; S4: Decode the fully enhanced BEV features to achieve 3D target detection and positioning; The S12 is specifically as follows: for the millimeter wave radar point cloud in space, project it onto the image plane to obtain a radar sparse depth map , and combine it with the image features Send them together into the depth completion network to get the depth probability estimation result ,in is the number of predefined discrete depth intervals; the process is expressed as: Among them, DepthNet represents a deep completion network consisting of several layers of convolution and neighborhood attention; The S22 is specifically as follows: first, a series of predefined regular voxels are generated in the millimeter wave radar coordinate system. ,in as well as Respectively represent the length and width of the BEV space, Represents the height size; take the center point of each voxel as the virtual point cloud, then the set of virtual point clouds of all voxels is , obviously ; Use the camera inside and outside the general The virtual point cloud is projected onto the image plane, and the pixel index is used to obtain the corresponding position , thus obtaining the feature set , the feature set All features in the feature dimension are filled into the corresponding virtual point cloud to obtain the decorated virtual point cloud .

2. The 3D target detection method based on dual-branch mutual fusion of millimeter-wave radar and camera images according to claim 1 is characterized in that: The S11 is specifically as follows: , which is encoded to obtain image features , Semantic features are further extracted through the semantic extraction network , is the number of channels of image features, and Represents the height and width of the image respectively, Indicates the downsampling rate; input the semantic features into the foreground segmentation network to segment the foreground area of ​​the image, thereby obtaining the foreground segmentation mask ; The above process can be expressed as: in, represents the semantic extraction network, Represents the foreground segmentation network, both of which are composed of several layers of convolutional layers and linear layers.

3. The 3D target detection method based on dual-branch mutual fusion of millimeter-wave radar and camera images according to claim 1 is characterized in that: In S13: for the depth probability estimation result and the true depth of the lidar , when calculating the two losses, the foreground segmentation mask is used An adaptive loss is applied to the foreground area to achieve a more accurate depth probability estimation of the foreground area. The process is expressed as: ， in, Indicates loss, represents the adaptive weight, represents the pixel index on the image plane, Indicates that the Kullback-Leibler divergence loss is calculated for the input.

4. The 3D target detection method based on dual-branch mutual fusion of millimeter-wave radar and camera images according to claim 1 is characterized in that: The S23 is specifically as follows: using the internal and external references of the camera The virtual point cloud is projected onto the image plane, and the pixel index is used to obtain the corresponding position , thus obtaining a set of depth probability estimation results , and then according to the real distance between the virtual point cloud and the image plane, The discrete depth space dimension Index on it and get the virtual point cloud corresponding to Probability ;use right The image features are weighted to convert the virtual point cloud into BEV features.

5. A 3D target detection device based on dual-branch mutual fusion of millimeter-wave radar and camera images, used to implement the method according to any one of claims 1 to 4, characterized in that: It includes geometric depth completion module, semantic radar cylinder network module, convolution fusion module and feature decoding module; The geometric depth completion module is used to achieve adaptive depth completion of the image foreground area and generate geometrically perceived camera BEV features; The semantic radar column network module is used to accurately introduce semantic information into the radar branch and generate semantically aware radar BEV features; The convolution fusion module is used to fuse the geometrically perceived camera BEV features and the semantically perceived radar BEV features using cross-modal convolution to obtain fully enhanced BEV features; The feature decoding module is used to decode the fully enhanced BEV features to achieve 3D target detection and positioning.

Images