Multimodal fusion 3D target detection method based on bidirectional guidance

Abstract

The invention discloses a multimodal fusion 3D target detection method based on bidirectional guidance, and belongs to the technical field of automatic driving and computer vision. According to the method, firstly, feature extraction is carried out on a multi-view image and LiDAR point cloud, dense depth distribution of image features is supervised and predicted by using a sparse depth map generated by LiDAR, and then the dense depth distribution is converted into depth perception 3D image features; then through a two-way guiding mechanism, using LiDAR geometric information to enhance image features, and using image semantic information to enhance LiDAR features; and finally, converting the enhanced bimodal features into a sequence, performing bidirectional interaction and fusion through a symmetric cross-modal attention module, and outputting deeply fused aerial view features for high-precision 3D target detection. According to the method, the reliability of depth estimation is effectively improved, deep complementation between modes is realized, and the detection precision and robustness in a complex scene are remarkably enhanced.

Description

Technical Field

[0001] This invention relates to the fields of autonomous driving and computer vision technology, and in particular to a multimodal fusion 3D target detection method based on bidirectional guidance. Background Technology

[0002] With the rapid development of autonomous driving technology, 3D target detection based on multi-sensor fusion has become one of the core tasks of environmental perception. To cope with complex road scenarios, a single sensor often falls short of requirements. Therefore, perception systems integrating LiDAR and camera modalities have been widely researched and applied. LiDAR sensors can provide accurate 3D geometric information, but the point cloud data they generate is relatively sparse and lacks semantic information such as texture and color; camera sensors can capture rich 2D texture and semantic information, but they themselves lack depth perception capabilities.

[0003] Existing multimodal 3D object detection methods, especially those employing fusion strategies in bird's-eye view (BEV) space, have made significant progress. However, these methods still face several key technical bottlenecks: First, the accuracy of depth estimation is crucial in the transformation of 2D image features to 3D space, but existing methods often rely on implicit learning of depth by the network, lacking explicit supervision, leading to incomplete geometric information recovery and insufficient reliability of view transformation. Second, most fusion methods only perform simple stitching or addition at the feature level, failing to fully explore and utilize the deep complementarity between LiDAR and camera modalities, particularly lacking an effective bidirectional interaction mechanism, making it difficult to use the advantages of one to compensate for the disadvantages of the other. Therefore, developing a 3D object detection method that achieves accurate depth perception, sufficient bidirectional guidance, and an adaptive fusion mechanism is of great significance. Summary of the Invention

[0004] The purpose of this invention is to provide a bidirectional guided multimodal fusion-based 3D target detection method to address the limitations in detection accuracy caused by unreliable depth estimation and insufficient modal interaction in existing technologies. This invention effectively improves the accuracy of depth perception and achieves full complementarity between modalities by introducing LiDAR-supervised depth estimation, bidirectional guided feature enhancement, and a symmetric cross-modal fusion mechanism, thereby accurately completing the 3D target detection task.

[0005] To achieve the above objectives, this invention provides a multimodal fusion 3D target detection method based on bidirectional guidance, comprising the following steps: S1. Extract features from the input multi-view image and LiDAR point cloud respectively. Use the sparse depth map generated by the LiDAR point cloud as supervision to predict the dense depth probability distribution of the image 2D features. Based on this distribution, convert the image 2D features into depth-aware 3D image features. S2. Perform bidirectional guided feature enhancement, map point cloud features to 3D space to generate occupancy weights, and fuse them with 3D image features to enhance image feature representation; generate semantic attention maps based on 3D image features, and fuse them with LiDAR point cloud bird's-eye view features to enhance LiDAR feature representation. S3. Convert the bidirectional guided enhanced LiDAR bird's-eye view features and the image bird's-eye view features into feature sequences respectively, perform bidirectional interactive attention calculation through the symmetric cross-modal attention module, and output the deeply fused bird's-eye view features.

[0006] Preferably, step S1 includes the following sub-steps: S101. Extract 2D features from multi-view images using an image backbone network. ,in , , , These are the number of cameras, the number of channels, the height and width of the feature map; multi-scale voxel features are extracted from LiDAR point clouds using a 3D sparse convolutional network. Its corresponding downsampling step size is ; S102, 3D sparse voxel features of different scales Projecting onto a unified bird's-eye view network, feature aggregation is performed along the height dimension to generate bird's-eye view features. Its expression is as follows: ; S103. Project the LiDAR point cloud onto the multi-view image plane to generate a sparse depth map. .

[0007] Preferably, step S1 further includes the following sub-steps: S104. Apply the depth prediction head in parallel at two different scales of the image feature pyramid, and output the depth probability distribution at each scale. ; S105. Using the sparse depth map obtained in step S103 As a monitoring signal, depth regression loss is calculated only at valid depth points. Its formalized expression is as follows: .

[0008] Preferably, step S1 further includes the following sub-steps: S106. Depth probability distribution at the highest resolution scale. Image features By performing a Lift operation on a weighted sum, depth-aware 3D image features are generated. ; in, ; S107. The obtained 3D image features are pooled into a predefined 3D network through the Splat operation to form standardized 3D image features. .

[0009] Preferably, step S2, "mapping point cloud features to 3D space to generate occupancy weights and fusing them with 3D image features," includes the following sub-steps: S201, Features of LiDAR bird's-eye view 3D LiDAR features are obtained by sampling to 3D space through a transposed convolutional network. ; S202. Using an occupancy detection head composed of convolutional layers and a sigmoid activation function, generate LiDAR 3D occupancy weights. ; S203, assign weight to 3D With standard 3D image features Element-wise multiplication yields .

[0010] Preferably, step S2, "generating a semantic attention map based on 3D image features and fusing it with LiDAR bird's-eye view features," includes the following sub-steps: S204, Enhanced 3D image features Max pooling is performed along the height dimension Z to obtain the bird's-eye view features of the image. ; S205. A semantic detection head composed of convolutional layers and a sigmoid activation function is used to analyze the bird's-eye view features of the image. Processing is performed to generate a semantic attention graph. ; S206, This semantic attention map Features of LiDAR bird's-eye view Element-wise multiplication yields the enhanced LiDAR bird's-eye view features. .

[0011] Preferably, step S3 includes the following sub-steps: S301, Enhanced LiDAR bird's-eye view features with bidirectional guidance and image bird's-eye view features Flattened into feature sequences and .

[0012] Preferably, step S3 further includes the following sub-steps: S302, using LiDAR feature sequences As a query, image feature sequence As keys and values, perform cross-modal attention computation and output LiDAR-enhanced features. ,in ; S303, using image feature sequences As a query, LiDAR feature sequences As keys and values, perform cross-modal attention computation and output camera view enhancement features. ,in .

[0013] Preferably, step S3 further includes the following sub-steps: S304. Sum the features output from steps S302 and S303, and apply layer normalization to obtain a preliminary fused feature sequence. ; S305, Initially fused feature sequences The final feature sequence is obtained through a feedforward neural network and residual connections. ; S306, Final feature sequence Reprojecting back to the bird's-eye view spatial dimension yields a deeply fused bird's-eye view feature map. .

[0014] Preferably, after step S3, step S4 is further included: integrating the depth-fused bird's-eye view features. Input a query-based 3D detection head and output 3D bounding boxes, category and attribute information to complete object detection.

[0015] The present invention also provides a 3D target detection system, including a processor and a memory, wherein the memory stores a computer program, and the processor executes the computer program to implement the above-described method.

[0016] Therefore, the multimodal fusion 3D target detection method based on bidirectional guidance using the above structure of the present invention has the following beneficial effects: (1) This invention utilizes sparse depth maps generated by LiDAR point cloud projection as explicit supervision signals to guide the prediction of dense depth probability distribution of image features. It effectively overcomes the shortcomings of existing methods that rely on implicit learning of depth by the network, resulting in incomplete and unreliable geometric information recovery. It provides a solid and accurate depth foundation for subsequent 2D to 3D feature transformation (Lift-Splat), fundamentally improving the reliability of view transformation.

[0017] (2) This invention proposes a unique bidirectional guided enhancement mechanism. On the one hand, it uses the precise geometric information of LiDAR to generate 3D occupancy weights, and filters and enhances the effective target regions in the image features. On the other hand, it uses the rich semantic information of the image to generate attention maps, and focuses and strengthens the key object regions in the LiDAR features. This realizes the mutual verification and enhancement of geometric and semantic information, and maximizes the advantages of the two modalities.

[0018] (3) In the feature fusion stage, this invention abandons simple splicing or addition operations and designs a symmetrical cross-modal attention module. This module allows LiDAR features and image features to perform bidirectional, interactive attention calculations, ensuring that important information from any modality can be fully focused on and absorbed by the other modality. This deep fusion mechanism enables the final feature representation to have both geometric accuracy and semantic richness, thereby significantly improving the accuracy, robustness, and generalization ability of 3D object detection in complex scenes.

[0019] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0020] Figure 1 This is a schematic diagram illustrating the steps of a multimodal fusion 3D target detection method based on bidirectional guidance according to the present invention; Figure 2 This is a flowchart illustrating a multimodal fusion 3D target detection method based on bidirectional guidance according to the present invention. Figure 3 This is a schematic diagram of the basic data for a bidirectional guided multimodal fusion 3D target detection method of the present invention. Detailed Implementation

[0021] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0022] Unless otherwise defined, the technical or scientific terms used in this invention shall have the ordinary meaning understood by one of ordinary skill in the art to which this invention pertains. The terms "first," "second," and similar terms used in this invention do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed following the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are used only to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0023] Example like Figure 1-3 As shown, this invention provides a multimodal fusion 3D target detection method based on bidirectional guidance, comprising the following steps: S1. Both multi-view images and LiDAR point cloud data can be acquired from sensors commonly found on autonomous vehicles. Feature extraction is performed on the input multi-view images and LiDAR point clouds, and depth-aware view transformation is completed. Specific steps include: S11. Assuming the input is a set of multi-view images from N cameras, use the Swin-Transformer as the image backbone network to extract multi-scale 2D image features. Simultaneously, inputting LiDAR point clouds, VoxelNet is used as a 3D sparse convolutional network to extract multi-scale voxel features. The corresponding downsampling step size is {1,2,4,8,16,32}.

[0024] S12, Multiscale voxel features Scale alignment is performed by mapping features with different sampling compensations to a unified BEV spatial scale using bilinear interpolation. Max pooling is then performed along the height dimension to aggregate features and generate LiDAR bird's-eye view features. This process can be formally represented as: ; in, This indicates that the feature space size is aligned to the specified size through upsampling. The scale, The set of voxels corresponding to the height dimension of the BEV coordinates (x, y). S13. Project the LiDAR point cloud onto the multi-view image plane using the camera intrinsic parameter matrix, retaining only the point cloud depth values ​​within the effective pixel range of the image, to generate a sparse depth map. .

[0025] S14. Apply the depth prediction head in parallel at the P3 and P4 scales of the image feature pyramid, and output the depth probability distribution at each scale. ,in This represents the number of depth intervals.

[0026] S15, with As a monitoring signal, depth regression loss is calculated only at valid depth points. Its formalized expression is as follows: ; in, It is a set of effective depth points. It is a discrete representation of the true depth label.

[0027] S16. Depth probability distribution at the P3 scale. Upsampled to the highest resolution, resulting in Combine it with image features A Lift operation is performed using weighted summation to generate an intermediate 3D camera feature point cloud. The calculation is as follows: ; S17. Use the Splat operation to transform the above intermediate 3D camera feature point cloud. Voxelization and pooling into a predefined 3D voxel mesh form canonical 3D camera voxel features. .

[0028] S2. Perform bidirectional bootstrapping feature enhancements, the specific steps of which include: S21. Features of LiDAR bird's-eye view 2D BEV features are sampled to 3D space through a transposed convolutional network, and then progressively upsampled to map the 2D BEV features to 3D voxel space to obtain 3D LiDAR features. .

[0029] S22. Using an occupancy prediction head consisting of two 3D convolutional layers and a sigmoid activation function, the prediction head is... Processing is performed to generate LiDAR 3D occupancy weights. The value of this weight ranges from [0,1]. A value of 1 indicates that the corresponding voxel contains a target, while a value of 0 indicates that the corresponding voxel is the background. This is used to filter the effective target region in the camera features.

[0030] S23, occupy the weight The 3D camera voxel features obtained in step S1 Element-wise multiplication yields geometrically enhanced 3D image features: ; S24, will Along the height dimension Max pooling is performed to compress the height dimension information while preserving the semantic features of the horizontal space, resulting in the image's bird's-eye view features. .

[0031] S25. Using a semantic prediction head consisting of two 3D convolutional layers and a sigmoid activation function, the semantic prediction is... Processing is performed to generate a semantic attention graph. This attention map focuses semantic information by highlighting the weight of target areas such as vehicles, pedestrians, and obstacles, while suppressing interference from background areas.

[0032] S26. This semantic attention map Features of LiDAR bird's-eye view Element-wise multiplication is performed, and LiDAR features are filtered and enhanced using camera semantic information. This strengthens the representation of the target region in the LiDAR features and weakens background noise with no semantic value, resulting in semantically enhanced LiDAR bird's-eye view features: ; S3. Perform symmetrical cross-modal fusion on the bidirectional guided enhanced features. Specific steps include: 31. Semantically Enhanced LiDAR Bird's-Eye View Features and enhanced image bird's-eye view features Flattened into feature sequences and ,in .

[0033] S32, using LiDAR feature sequences As a query Camera feature sequence As a key Sum Perform cross-modal attention computation and output LiDAR-enhanced features. : ; Camera feature sequences As a query LiDAR feature sequences As a key Sum Perform cross-modal attention computation and output camera-view augmented features. : ; S33. Sum the enhanced features from the two perspectives and apply layer normalization to eliminate the difference in feature distribution between the two modalities, resulting in a preliminary fused feature sequence: ; S34, will Optimization is achieved through a feedforward neural network and residual connections. The feedforward neural network consists of two fully connected layers and a GELU activation function, which is used to perform a nonlinear transformation on the fused features to obtain the final feature sequence. ; S35, will By reprojecting the image back into the spatial dimension of the bird's-eye view, the spatial structure of the BEV features is restored, resulting in a deeply fused bird's-eye view feature. .

[0034] S36, will Input a query-based 3D detector head and output the final 3D bounding box, category, and attributes.

[0035] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the technical solutions of the present invention, and these modifications or equivalent substitutions cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. A multimodal fusion 3D target detection method based on bidirectional guidance, characterized in that, Includes the following steps: S1. Extract features from the input multi-view image and LiDAR point cloud respectively. Use the sparse depth map generated by the LiDAR point cloud as supervision to predict the dense depth probability distribution of the image 2D features. Based on this distribution, convert the image 2D features into depth-aware 3D image features. S2. Perform bidirectional guided feature enhancement, map LiDAR features to 3D space to generate occupancy weights, and fuse them with 3D image features to enhance image feature representation; generate semantic attention maps based on 3D image features, and fuse them with LiDAR bird's-eye view features to enhance LiDAR feature representation. S3. The bidirectional guided enhanced LiDAR bird's-eye view features and image bird's-eye view features are converted into feature sequences respectively. The bidirectional interactive attention calculation is performed through the symmetric cross-modal attention module to output the deeply fused bird's-eye view features for 3D target detection.

2. The method for 3D target detection based on bidirectional guidance and multimodal fusion according to claim 1, characterized in that, Step S1 includes: S101. Use an image backbone network to extract 2D features from multi-view images; use a 3D sparse convolutional network to extract multi-scale voxel features from LiDAR point clouds. S102. Project 3D voxel features of different scales onto a unified bird's-eye view space and aggregate them along the height dimension to generate LiDAR bird's-eye view features. S103. Project the LiDAR point cloud onto the multi-view image plane to generate a sparse depth map.

3. The method for 3D target detection based on bidirectional guidance and multimodal fusion according to claim 2, characterized in that, Step S1 also includes: S104. Apply the depth prediction head in parallel at different scales of the image feature pyramid and output the depth probability distribution at each scale. S105. Using sparse depth maps as a supervision signal, calculate the depth regression loss at the effective depth point location.

4. The method for multimodal fusion 3D target detection based on bidirectional guidance according to claim 3, characterized in that, Step S1 also includes: S106. The depth probability distribution at the highest resolution scale is weighted and summed with the image features, and the initial 3D image features are generated by the Lift operation. S107. The initial 3D image features are pooled into a predefined 3D voxel grid through the Splat operation to form standardized 3D image features.

5. The method for multimodal fusion 3D target detection based on bidirectional guidance according to claim 1, characterized in that, Step S2, "mapping LiDAR features to 3D space to generate occupancy weights," includes: S201. Upsample the LiDAR bird's-eye view features to 3D space to obtain 3D LiDAR features; S202, Generate LiDAR 3D occupancy weights by occupying the detection head; S203. Multiply the weighted 3D image features element by element to obtain the enhanced 3D image features.

6. The method for multimodal fusion 3D target detection based on bidirectional guidance according to claim 5, characterized in that, Step S2, "Generating a semantic attention map based on 3D image features," includes: S204. Pool the enhanced 3D image features along the height dimension to obtain the bird's-eye view features of the image; S205. Generate a semantic attention map using the semantic detection head; S206. Multiply the semantic attention map element-wise with the LiDAR bird's-eye view features to obtain the enhanced LiDAR bird's-eye view features.

7. The method for 3D target detection based on bidirectional guidance and multimodal fusion according to claim 1, characterized in that, Step S3 includes: S301. Flatten the enhanced LiDAR bird's-eye view features and the image bird's-eye view features into feature sequences respectively.

8. The method for multimodal fusion 3D target detection based on bidirectional guidance according to claim 7, characterized in that, Step S3 also includes: S302. Using the LiDAR feature sequence as the query and the image feature sequence as the key and value, perform cross-modal attention calculation and output LiDAR view enhancement features; S303. Using the image feature sequence as the query and the LiDAR feature sequence as the key and value, perform cross-modal attention calculation and output image view enhancement features.

9. The method for multimodal fusion 3D target detection based on bidirectional guidance according to claim 8, characterized in that, Step S3 also includes: S304. After summing the enhanced features from the two perspectives, perform layer normalization to obtain a preliminary fused feature sequence; S305. The preliminary fused feature sequence is optimized by using a feedforward neural network and residual connections to obtain the final feature sequence; S306. Reproject the final feature sequence back into the bird's-eye view space to obtain deeply fused bird's-eye view features.

10. A 3D target detection system, characterized in that, It includes a processor and a memory, the memory storing a computer program, and the processor executing the computer program to implement the method as described in any one of claims 1-9.

Images