3D target detection method based on three-dimensional deformable attention mechanism enhancement

Abstract

The invention discloses a 3D target detection method based on three-dimensional deformable attention mechanism enhancement, and aims to improve the 3D target detection performance of multi-modal sensor fusion. According to the method, firstly, through an innovatively designed cross-modal feature fusion mechanism, radar and camera features are effectively guided and enhanced in a feature fusion process under a bird's-eye view angle, and feature initialization in a subsequent three-dimensional deformable attention mechanism is optimized; secondly, light foreground segmentation and depth estimation are combined, guiding control of a three-dimensional deformable attention mechanism in a pixel domain and a depth dimension is realized, deeper and more accurate interaction between fusion features and image semantic features is realized, and semantic information expression ability of the features is enhanced. Experimental results show that the method of the invention significantly improves the precision and robustness of 3D target detection, and especially has excellent performance in complex scenes.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to a 3D target detection method based on a three-dimensional deformable attention mechanism enhancement, aiming to improve the 3D target detection performance of multimodal sensor fusion. Background Art

[0002] With the rapid development of autonomous driving technology, 3D object detection technology plays a vital role in autonomous driving systems. It helps vehicles accurately perceive the surrounding environment and provides key information such as position and speed, thereby achieving safe driving. At present, the fusion of laser radar (LiDAR) and camera is the mainstream method for 3D object detection. LiDAR can provide high-precision geometric depth information, while the camera can capture rich semantic information. The complementarity of the two provides efficient perception capabilities for 3D object detection. However, LiDAR sensors are expensive and perform poorly in bad weather conditions.

[0003] In contrast, millimeter-wave radar (4D Radar) has gradually become an alternative in the field of autonomous driving perception because of its ability to work stably in bad weather and its low cost. 4D millimeter-wave radar can not only provide distance, speed and angle information of objects, but also contains the height information of objects, giving it higher resolution in complex scenes. However, due to the sparsity and high noise of radar point cloud data, these factors limit the performance of millimeter-wave radar in target detection tasks.

[0004] Existing multimodal fusion methods use rich camera image semantic features to improve the sparse and noisy information of millimeter-wave radar to a certain extent, and therefore have made good progress in 3D target detection. The BEVFusion method maps image features into the bird's-eye view (BEV) space through perspective transformation in order to better combine the geometric information of the radar. Specifically, BEVFusion was the first to achieve the fusion of cross-modal information in the BEV (Bird's Eye View) space, but it did not fully utilize the specific features of each modality before constructing the BEV features. The LXL method further uses the depth of monocular estimation to distinguish image features in space. However, these methods have failed to fully fuse cross-modal bird's-eye view features, nor have they used the powerful role of attention mechanisms to enhance feature expression. The BEVFormer method pioneered the use of attention mechanisms to fuse surround image features for 3D object detection, and the FUTR3D method also used sparse queries to aggregate multimodal features, but they did not consider and solve the difficulties encountered when the query finds the corresponding tag, such as the inherent depth inconsistency problem in the 3D to 2D projection process, which leads to fuzzy target features and slow convergence of the query during the interaction process. Even though the FB-BEV method and the DFA3D method have solved the problem of deep dimension target positioning to a certain extent, there is currently no method that can strengthen the association between the query and the token of interest in the pixel domain. Summary of the invention

[0005] Considering the lack of accuracy and robustness of the existing 3D target detection methods that fuse images and radars, the present invention proposes a 3D target detection method based on the enhancement of a three-dimensional deformable attention mechanism. This method uses an innovatively designed cross-modal feature fusion mechanism to effectively guide and enhance the feature fusion process of radar and camera features from a bird's-eye view, optimizing the feature initialization in the subsequent three-dimensional attention mechanism; at the same time, combined with lightweight foreground segmentation and depth estimation, it realizes the guidance control of the three-dimensional attention mechanism in the pixel domain and depth dimension, promoting a deeper and more accurate interaction between the fusion features and the image semantic features, and enhancing the semantic information expression ability of the features.

[0006] On the one hand, the present invention provides a 3D target detection method based on the enhancement of three-dimensional deformable attention mechanism. The method uses a cross-modal feature fusion mechanism to fuse the millimeter-wave radar and camera features from a bird's-eye view to obtain fused features; combined with lightweight foreground segmentation and depth estimation, the three-dimensional deformable attention mechanism is used to enhance the fused features in the pixel domain and depth dimension to obtain enhanced BEV features, thereby improving the semantic expression ability of the fused features; the enhanced BEV features are decoded to achieve 3D target detection and positioning. The specific steps of the detection method are as follows:

[0007] S1: Extract the original millimeter-wave radar and monocular image features, and obtain the BEV features from the bird's-eye view based on the corresponding features; specifically, the following steps are included:

[0008] S11: Voxelize the millimeter wave radar point cloud, extract shallow features, and extract deep features to obtain the original radar point cloud features. Use the point cloud cylinder network to extract the original radar point cloud features, and finally obtain the radar BEV features. Where X and Y represent the size of the BEV space, and C represents the number of channels of the feature map.

[0009] S12: Extract the monocular image features and segment the image foreground area based on the image features. The image features are obtained by encoding F is further extracted through the semantic extraction network to obtain semantic features C is the number of channels in the feature map, H and W represent the height and width of the image respectively, and n represents the downsampling rate. The semantic features are input into the foreground segmentation network to segment the foreground area of ​​the image and obtain the foreground segmentation mask. The process can be described as:

[0010] C = ContextNet(F),

[0011] M = SegNet(C),

[0012] Among them, ContextNet(·) represents the semantic extraction network and SegNet(·) represents the foreground segmentation network. Both are composed of several layers of convolutional layers and linear layers. During the training process, the loss calculation uses two true value labels: the foreground mask pre-segmented by Dectron2 and the foreground mask generated by the 2D box ground truth.

[0013] S13: Use the image features obtained in step S12 to perform depth probability estimation, use the depth probability estimation result and the image features to perform outer product, so as to transform the image features into a bird's-eye view, and then perform voxel pooling on the outer product result to obtain the camera BEV feature. Among them, the image features obtained in step S12 are used to perform depth probability estimation. The specific method is: send the image feature F into the depth completion network to obtain the depth probability estimation result Where D is the number of predefined discrete depth intervals. The depth completion network consists of several layers of convolution Conv and the neighborhood-cross-attention (NCA) mechanism, so the process can be expressed as:

[0014] D=DepthNet(F)=Conv(NCA(Conv(F)0),

[0015] Among them, DepthNet(·) represents the depth completion network.

[0016] S2: Based on the cross-modal feature fusion mechanism, radar BEV features and camera BEV features are fused from the bird's-eye view to obtain fused features. Specifically, the camera BEV features can be obtained from the camera branch and the radar branch respectively. and radar BEV signature Where X and Y represent the length and width of the BEV space, respectively. Due to the differences between the two sensors, the two BEV features may show different significance in the semantic description of the same space. The proposed cross-modal feature fusion mechanism can adaptively fuse the two BEV features. Specifically, for the two modality-specific BEV features, the camera BEV attention feature F can be adaptively generated by the following formula: Attn_C and radar BEV attention feature F Attn_R :

[0017] F Attn_C =Sigmoid(Conv([MP(F C ),AP(F C )])),

[0018] F Attn_R =Sigmoid(Conv([MP(F R ),AP(F R )])),

[0019] Among them, [·,·] represents the concatenation operation along the channel dimension; MP(·) and AP(·) represent the maximum pooling and average pooling operations in the channel dimension respectively; Conv represents a 3×3 convolution, and the number of input and output channels is 2 and 1 respectively. Fusion feature F Fusion Weighted generation via interactive attention:

[0020] F Fusion =CBR([F Attn_C ·F R ,F Attn_R ·F R ]),

[0021] Among them, CBR represents a combination of 3×3 convolution, batch normalization, and ReLU, which is mainly used to adjust the channel size. This step can fully utilize the advantages of cross-modality. On the one hand, the attention weight of the camera BEV can reduce the impact of millimeter-wave radar noise on the detection results. On the other hand, the radar attention weight can enhance the confidence of the camera's potential objects, thereby improving the detection performance.

[0022] S3: Combine the foreground segmentation result obtained in step S12 with the depth probability estimation result obtained in step S13, and use the three-dimensional deformable attention mechanism to enhance the fused features obtained in step S2. Specifically, this step combines lightweight foreground segmentation and depth estimation, realizes the guidance control of the three-dimensional deformable attention mechanism in the pixel domain and depth dimension, enhances the association between the query and key value of the three-dimensional deformable attention mechanism, promotes deeper and more accurate interaction between the fused features and the image semantic features, and enhances the semantic information expression ability of the features.

[0023] S31: Use the foreground segmentation result and the depth probability estimation result to locate the target in the three-dimensional feature space, so as to locate the target in the pixel domain and the depth dimension. For the extracted semantic features C and the depth probability estimation result D, firstly, the outer product (×) is performed to obtain the three-dimensional feature space. This step can locate the three-dimensional target in the depth dimension. Secondly, the foreground segmentation mask M is used to perform the dot product (·) with the three-dimensional feature space. This step can locate the three-dimensional target in the pixel domain. The process can be specifically expressed as:

[0024] F 3D =M·(D×C),

[0025] in, Represents the three-dimensional feature space after the target is positioned in the pixel domain and depth dimension. This step can highlight the target features in the three-dimensional feature space through positioning in the pixel domain and depth dimension, which is helpful for the subsequent attention mechanism.

[0026] S32: Based on the positioning in the pixel domain and the depth dimension, the fused features are enhanced using the three-dimensional deformable attention enhancement fusion mechanism to obtain the enhanced BEV features, thereby improving the semantic expression ability of the fused features. This step is specifically as follows: First, use the fused feature F Fusion Initialize the query in the attention mechanism. Then, the three-dimensional spatial feature F after positioning the target in the pixel domain and depth dimension is 3D As the value in the attention mechanism. Finally, based on the three-dimensional deformable attention mechanism, the query and value interact continuously to obtain the enhanced BEV feature F Final .

[0027] Specifically, based on the three-dimensional deformable attention mechanism, the query and value interact continuously, which can be divided into the following steps. (1) Each query is lifted from the BEV plane to the pillar according to its position in the BEV space, and 3D reference points are sampled from the pillar and assigned with the corresponding query. (2) The 3D reference point is used as the query, the three-dimensional spatial feature is used as the key and value, and deformable cross attention is performed by projecting these 3D reference points into the three-dimensional space. The above process is specifically described as follows: for the 3D query Q at position q q , the semantically enhanced

[0028]

[0029] Where N represents the number of 3D queries at position q sampled in the deformable attention mechanism; represents the camera projection function; A n ∈[0,1] is the learnable attention weight; W represents the feature projection weight; represents the predicted offset to position q; Represents trilinear interpolation.

[0030] Finally, the semantically enhanced Reintegrating into the shape of the BEV feature space, we can get Wherein, X and Y represent the length and width of the BEV space.

[0031] S4: Decode the enhanced BEV features described in step S32 to achieve detection and positioning of 3D targets.

[0032] On the other hand, the present invention provides a 3D target detection device based on a three-dimensional deformable attention mechanism enhancement, the detection device comprising an original feature extraction module, a cross-modal feature fusion module, a target-oriented attention enhancement module and a target detection module;

[0033] The original feature extraction module is used to extract original millimeter wave radar and monocular image features;

[0034] The cross-modal feature fusion module is used to obtain the BEV feature at the bird's-eye view perspective based on the original millimeter-wave radar and monocular image features, and fuse the radar BEV feature and the camera BEV feature at the bird's-eye view perspective based on the cross-modal feature fusion mechanism to obtain the fused feature;

[0035] The target-oriented attention enhancement module is used to enhance the fusion features based on the deformable attention mechanism to obtain enhanced BEV features;

[0036] The target detection module is used to decode the enhanced BEV features to achieve 3D target detection and positioning.

[0037] The beneficial effects of the present invention are:

[0038] The present invention proposes a 3D target detection method based on an enhanced three-dimensional deformable attention mechanism. First, the present invention uses an innovatively designed cross-modal feature fusion mechanism to effectively guide and enhance radar and camera features in the feature fusion process from a bird's-eye view, thereby optimizing the feature initialization in the subsequent three-dimensional attention mechanism. Secondly, the present invention combines lightweight foreground segmentation and depth estimation to achieve guidance control of the three-dimensional deformable attention mechanism in the pixel domain and depth dimension, thereby promoting a deeper and more accurate interaction between the fused features and the image semantic features, and enhancing the semantic information expression capability of the features. Experimental results show that the method of the present invention significantly improves the accuracy and robustness of 3D target detection, especially in complex scenes.

[0039] Compared with the existing methods, the present invention not only effectively integrates the cross-modal bird's-eye view features in the initial stage, but also enhances the association between the query and key value of the three-dimensional deformable attention mechanism through the guidance control in the pixel domain and depth dimension of the three-dimensional feature space, thus realizing deep interaction. This method adopts the optimization process of deep learning, and enables the neural network to learn the essential characteristics of the 3D environment during the feature fusion process, thereby improving the detection accuracy and generalization performance, and performs well in the categories of pedestrians, bicycles, and cars in the test data. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 It is a flowchart of a 3D target detection method enhanced by a three-dimensional deformable attention mechanism provided by an embodiment of the present invention.

[0041] Figure 2 This is a schematic diagram of the 3D deformable attention mechanism.

[0042] Figure 3 4 is a diagram showing the generalization enhancement effect of an embodiment of the present invention. DETAILED DESCRIPTION

[0043] The following is further described in conjunction with specific embodiments and drawings.

[0044] Example 1

[0045] This embodiment proposes a 3D target detection method based on the enhancement of three-dimensional deformable attention mechanism. This method uses a cross-modal feature fusion mechanism to fuse the features of millimeter-wave radar and camera from a bird's-eye view to obtain fused features; combined with lightweight foreground segmentation and depth estimation, the three-dimensional deformable attention mechanism is used to enhance the fused features in the pixel domain and depth dimension to obtain enhanced BEV features, thereby improving the semantic expression ability of the fused features; the enhanced BEV features are decoded to achieve 3D target detection and positioning.

[0046] like Figure 1 The flowchart of the detection method specifically includes the following steps:

[0047] S1: Extract the original millimeter-wave radar and monocular image features, and obtain the BEV features from the bird's-eye view based on the corresponding features; this step is specifically as follows:

[0048] S11: Voxelize the millimeter-wave radar point cloud and extract shallow and deep features to obtain the original radar point cloud features, and use the point cloud cylinder network to further extract radar BEV features Where X and Y represent the size of the BEV space, and C represents the number of channels of the feature map.

[0049] S12: Extract the monocular image features and segment the image foreground area based on the image features. The image features are obtained by encoding F is further extracted through the semantic extraction network to obtain semantic features C is the number of channels in the feature map, H and W represent the height and width of the image respectively, and n represents the downsampling rate. The semantic features are input into the foreground segmentation network to segment the foreground area of ​​the image and obtain the foreground segmentation mask. The process can be described as:

[0050] C = ContextNet(F),

[0051] M = SegNet(C),

[0052] Among them, ContextNet(·) represents the semantic extraction network, and SegNet(·) represents the foreground segmentation network, both of which are composed of several layers of convolutional layers and linear layers.

[0053] S13: Use the image features obtained in step S12 to perform depth probability estimation, use the depth probability estimation result and the image features to perform outer product, so as to transform the image features into a bird's-eye view, and then perform voxel pooling on the outer product result to obtain the camera BEV feature. Among them, the image features obtained in step S12 are used to perform depth probability estimation. The specific method is: send the image feature F into the depth completion network to obtain the depth probability estimation result Where D is the number of predefined discrete depth intervals. The depth completion network consists of several layers of convolution Conv and the neighborhood-cross-attention (NCA) mechanism, so the process can be expressed as:

[0054] D=DepthNet(F)=Conv(NCA(Conv(F))),

[0055] Among them, DepthNet(·) represents the depth completion network.

[0056] S2: Based on the cross-modal feature fusion mechanism, radar BEV features and camera BEV features are fused from the bird's-eye view to obtain fused features. Specifically, the camera BEV features can be obtained from the camera branch and the radar branch respectively. and radar BEV signature Where X and Y represent the length and width of the BEV space, respectively. For the two modality-specific BEV features, the camera BEV attention feature F can be adaptively generated by the following formula: Attn_C and radar BEV attention feature F Attn_R :

[0057] F Attn_c =Sigmoid(Conv([MP(F C ),AP(F C )])),

[0058] F Attn_R =Sigmoid(Conv([MP(F R ),AP(F R )])),

[0059] Among them, [·,·] represents the concatenation operation along the channel dimension; MP(·) and AP(·) represent the maximum pooling and average pooling operations in the channel dimension respectively; Conv represents a 3×3 convolution, and the number of input and output channels is 2 and 1 respectively. Fusion feature F Fusion Weighted generation via interactive attention:

[0060] F Fusion =CBR([F Attn_C ·FR ,F Attn_R ·F R ]),

[0061] Among them, CBR represents the combination of 3×3 convolution, batch normalization and ReLU, which is mainly used to adjust the channel size.

[0062] S3: Combine the foreground segmentation result obtained in step S12 with the depth probability estimation result obtained in step S13, and use the three-dimensional deformable attention mechanism to enhance the fusion feature obtained in step S2 (such as Figure 2 ). The specific steps are:

[0063] S31: Use the foreground segmentation result and the depth probability estimation result to locate the target in the three-dimensional feature space, so as to locate the target in the pixel domain and the depth dimension. For the extracted semantic features C and the depth probability estimation result D, firstly make an outer product (×) to obtain the three-dimensional feature space, and then use the foreground segmentation mask M to make a dot product (·) with the three-dimensional feature space. The process can be specifically expressed as:

[0064] F 3D =M·(D×C),

[0065] in, Represents the three-dimensional feature space after the target is positioned in the pixel domain and depth dimension. This step can highlight the target features in the three-dimensional feature space through positioning in the pixel domain and depth dimension, which is helpful for the subsequent attention mechanism.

[0066] S32: Based on the positioning in the pixel domain and the depth dimension, the fused features are enhanced using the three-dimensional deformable attention enhancement fusion mechanism to obtain the enhanced BEV features, thereby improving the semantic expression ability of the fused features. This step is specifically as follows: First, use the cross-modal fusion BEV feature F Fusion Initialize the query in the attention mechanism. Then, the three-dimensional spatial feature F after positioning the target in the pixel domain and depth dimension is 3D As the value in the attention mechanism. Finally, based on the three-dimensional deformable attention mechanism, the query and value interact continuously to obtain the enhanced BEV feature F Final .

[0067] Among them, based on the three-dimensional deformable attention mechanism, the query and value interact continuously, which can be divided into the following steps. (1) Each query is lifted from the BEV plane to the pillar according to its position in the BEV space, and 3D reference points are sampled from the pillar and assigned with the corresponding query. (2) The 3D reference point is used as the query, the three-dimensional spatial feature is used as the key and value, and deformable cross attention is performed by projecting these 3D reference points into the three-dimensional space. The above process is specifically described as follows: for the 3D query Q at position q q , the semantically enhanced

[0068]

[0069] Where N represents the number of 3D queries at position q sampled in the deformable attention mechanism; represents the camera projection function; A n ∈[0,1] is the learnable attention weight; W represents the feature projection weight; represents the predicted offset to position q; Represents trilinear interpolation.

[0070] Finally, the semantically enhanced Reintegrating into the shape of the BEV feature space, we can get Wherein, X and Y represent the length and width of the BEV space.

[0071] S4: Decode the enhanced BEV features described in step S32 to achieve detection and positioning of 3D targets.

[0072] Example 2

[0073] This embodiment provides a 3D target detection device based on a three-dimensional deformable attention mechanism enhancement. Through the innovatively designed cross-modal feature fusion module, the radar and camera features are effectively guided and enhanced in the feature fusion process from a bird's-eye view, and the feature initialization in the subsequent three-dimensional attention mechanism is optimized. At the same time, it combines a lightweight foreground segmentation network with depth estimation to achieve guidance control of the three-dimensional attention mechanism in the pixel domain and depth dimension, promote deeper and more accurate interaction between the fused features and the image semantic features, and enhance the semantic information expression ability of the features. Ultimately, deep interaction and fusion of multimodal features can be achieved, thereby improving the performance of 3D target detection.

[0074] The detection device includes an original feature extraction module, a cross-modal feature fusion module, a target-oriented attention enhancement module and a target detection module;

[0075] The original feature extraction module is used to extract original millimeter wave radar and monocular image features;

[0076] The cross-modal feature fusion module is used to obtain the BEV feature at the bird's-eye view perspective based on the original millimeter-wave radar and monocular image features, and fuse the radar BEV feature and the camera BEV feature at the bird's-eye view perspective based on the cross-modal feature fusion mechanism to obtain the fused feature;

[0077] The target-oriented attention enhancement module is used to enhance the fusion features based on the deformable attention mechanism to obtain enhanced BEV features;

[0078] The target detection module is used to decode the enhanced BEV features to achieve 3D target detection and positioning.

[0079] The detection process of the detection device can be divided into the following four stages:

[0080] In the first stage, the original feature extraction module is used to extract the original millimeter wave radar and monocular image features. Use Image Encoder to encode image features Here C is the number of channels of the feature map, H and W represent the height and width of the image respectively, and n represents the downsampling rate. For millimeter-wave radar, the point cloud encoder (Radar Encoder) is used to encode the millimeter-wave radar point cloud. The process includes voxelization, shallow feature extraction, deep feature extraction and other steps to obtain the original radar point cloud features.

[0081] In the second stage, the camera-radar dual-branch mutual fusion module is used to first obtain the BEV features of the original millimeter-wave radar and monocular image features from a bird's-eye view perspective, and based on the cross-modal feature fusion mechanism, the radar BEV features and the camera BEV features are fused from a bird's-eye view perspective to obtain fused features. Specifically, it can be divided into the camera branch (Image Branch) and the radar branch (Radar Branch) for elaboration. In the millimeter-wave radar branch of the second stage, since the original radar point cloud features of the millimeter-wave radar point cloud have been extracted in the first stage, the point cloud pillar network (RadarPillar Net) is further used here to further process the original radar point cloud features, and finally the radar point cloud BEV features (Radar-BEV) can be obtained. Here C is the number of channels in the feature map, and X and Y represent the size of the BEV space. In the second stage of the camera branch, F is further extracted through the semantic extraction network to obtain semantic features Input the semantic features into the foreground segmentation network to segment the foreground area of ​​the image and obtain the foreground segmentation mask (Mask) The process can be described as:

[0082] C = ContextNet(F),

[0083] M = SegNet(C),

[0084] Among them, ContextNet(·) represents the semantic extraction network, and SegNet(·) represents the foreground segmentation network. Both are composed of several layers of convolutional layers and linear layers. During the training process, the loss calculation uses two true value labels: the foreground mask pre-segmented by Dectron2 and the foreground mask generated by the 2D box true value. Then, the image feature F is sent to the depth completion network to obtain the depth probability estimation result Where D is the number of predefined discrete depth intervals. The depth completion network consists of several layers of convolution Conv and the neighborhood-cross-attention (NCA) mechanism, so the process can be expressed as:

[0085] D=DepthNet(F)=Conv(NCA(Conv(F))),

[0086] Among them, DepthNet(·) represents the depth completion network.

[0087] Finally, the depth probability estimation result is outer-producted with the image feature, and the outer-product result is voxel pooled (ViewTransformation) to obtain the camera BEV feature (Image-BEV) Then, the camera BEV feature F can be obtained from the camera branch and the radar branch respectively. C BEV with radar features F R Due to the differences between the two sensors, the two BEV features may show different saliency in the semantic description of the same space. The proposed cross-modality feature fusion module (Cross Modality Fusion) can adaptively fuse the two BEV features. Specifically, for the two modality-specific BEV features, the camera BEV attention feature F can be adaptively generated by the following formula: Attn_C and radar BEV attention feature F Attn_R :

[0088] F Attn_c =Sigmoid(Conv([MP(F C ),AP(F C )])),

[0089] F Attn_R =Sigmoid(Conv([MP(F R ),AP(F R )])),

[0090] Among them, [·,·] represents the concatenation operation along the channel dimension; MP(·) and AP(·) represent the maximum pooling and average pooling operations in the channel dimension respectively; Conv represents a 3×3 convolution, and the number of input and output channels is 2 and 1 respectively. Fusion feature F Fusion Weighted generation via interactive attention:

[0091] F Fusion =CBR([F Attn_C ·F R ,F Attn_R ·F R ]),

[0092] Among them, CBR represents 3×3 convolution, batch normalization and ReLU, which are mainly used to adjust the channel size. This module can fully utilize the advantages of cross-modality.

[0093] In the third stage, the object-oriented attention enhancement module (Object-Oriented Attention) enhances the fusion features based on the deformable attention mechanism to obtain the enhanced BEV features. Specifically, the enhanced BEV features can be obtained by using the location-aware three-dimensional deformable attention mechanism (Localization-Aware Cross-Attetion) and the object-oriented three-dimensional feature space (Object-Oriented 3D Feature Space Creation). First, the foreground segmentation results and the depth probability estimation results are used to locate the target in the three-dimensional feature space, thereby realizing the positioning of the target in the pixel domain and the depth dimension. The foreground segmentation results and the depth probability estimation results are used to locate the target in the three-dimensional feature space, thereby realizing the positioning of the target in the pixel domain and the depth dimension. For the extracted semantic features C and the depth probability estimation results D, firstly, the outer product (×) is performed to obtain the three-dimensional feature space. This step can locate the three-dimensional target in the depth dimension. Secondly, the foreground segmentation mask M is used to perform the dot product (·) with the three-dimensional feature space. This step can locate the three-dimensional target in the pixel domain. The process can be specifically expressed as:

[0094] F 3D =M·(D×C),

[0095] in, Represents the three-dimensional feature space after object localization in the pixel domain and depth dimension.

[0096] Then, based on the positioning of the pixel domain and the depth dimension, the fused features are enhanced using the three-dimensional deformable attention enhancement fusion mechanism to obtain the enhanced BEV features, thereby improving the semantic expression ability of the fused features. This step is specifically as follows: First, use the fused feature F Fusion Initialize the query in the attention mechanism. Then, the three-dimensional spatial feature F after positioning the target in the pixel domain and depth dimension is 3D As the value in the attention mechanism. Finally, based on the three-dimensional deformable attention mechanism, the query and value interact continuously to obtain the enhanced BEV feature F Final .

[0097] In the fourth stage, the anchor-based feature decoding head is used to decode the enhanced BEV features F Final Decoding is performed to achieve 3D target detection and positioning.

[0098] Figure 3 The results of the method of the present invention on the TJ4DRadSet dataset are shown in Figure 1. Each image corresponds to a data frame containing images and radar points (gray), and the red triangle marks the position of the vehicle. The orange and yellow boxes represent the real boxes in the perspective view and the bird's-eye view, respectively. The green and blue boxes represent the bounding boxes predicted by the method of the present invention. The lower left part of the figure shows the visualization of the BEV feature map. Figure 3 From (a), (b) and (c), it can be seen that the method of the present invention has achieved good results in the detection performance of cars and trucks in the data set.

Claims

1. A 3D object detection method based on a three-dimensional deformable attention mechanism enhancement, characterized in that: Through the cross-modal feature fusion mechanism, the millimeter-wave radar and camera features are fused from a bird's-eye view to obtain fused features; Combining lightweight foreground segmentation and depth estimation, the three-dimensional deformable attention mechanism is used to enhance the fused features in the pixel domain and depth dimension to obtain enhanced BEV features, thereby improving the semantic expression ability of the fused features; the enhanced BEV features are decoded to achieve 3D target detection and positioning.

2. A 3D object detection method based on three-dimensional deformable attention mechanism enhancement according to claim 1, characterized in that: The specific steps are as follows: S1: Extract the original millimeter-wave radar and monocular image features, and obtain the BEV features from the bird's-eye view based on the corresponding features; S11: voxelizing the millimeter-wave radar point cloud and performing feature extraction to obtain original radar point cloud features, and further extracting radar BEV features based on the original radar point cloud features; S12: extracting monocular image features and segmenting the image foreground area based on the image features; S13: Use the image features obtained in step S12 to perform depth probability estimation, use the depth probability estimation result and the image features to perform an outer product, so as to transform the image features into a bird's-eye view, and then perform voxel pooling on the outer product result to obtain the camera BEV feature; S2: Based on the cross-modal feature fusion mechanism, radar BEV features and camera BEV features are fused from the bird's-eye view perspective to obtain fused features; S3: Combining the foreground segmentation result obtained in step S12 with the depth probability estimation result obtained in step S13, the fusion feature obtained in step S2 is enhanced using a three-dimensional deformable attention mechanism; S31: Using the foreground segmentation result and the depth probability estimation result, the target is located in the three-dimensional feature space, thereby realizing the positioning of the target in the pixel domain and the depth dimension; S32: Based on the positioning in the pixel domain and depth dimension, the fused features are enhanced using the three-dimensional deformable attention enhancement fusion mechanism to obtain the enhanced BEV features, thereby improving the semantic expression ability of the fused features; S4: Decode the enhanced BEV features described in step S32 to achieve detection and positioning of 3D targets.

3. The 3D object detection method based on the three-dimensional deformable attention mechanism enhancement according to claim 2 is characterized in that: Step S12 specifically includes: The image features are obtained by encoding F is further extracted through the semantic extraction network to obtain semantic features C is the number of channels of the feature map, H and W represent the height and width of the image respectively, and n represents the downsampling rate; the semantic features are input into the foreground segmentation network to segment the foreground area of ​​the image and obtain the foreground segmentation mask The above process can be expressed as: C = ContextNet(F), M = SegNet(C), Among them, ContextNet(·) represents the semantic extraction network, and SegNet(·) represents the foreground segmentation network, both of which are composed of several layers of convolutional layers and linear layers.

4. The 3D object detection method based on the three-dimensional deformable attention mechanism enhancement according to claim 3 is characterized in that: In step S13, the image features obtained in step S12 are used to perform depth probability estimation. The specific method is as follows: the image features F are sent to the depth completion network to obtain the depth probability estimation result Where D is the number of predefined discrete depth intervals; the above process can be expressed as: D=DepthNet(F) Among them, DepthNet(·) represents a deep completion network, which consists of several layers of convolution and neighborhood attention.

5. The 3D object detection method based on the three-dimensional deformable attention mechanism enhancement according to claim 3 is characterized in that: S2 is specifically: obtain the camera BEV features from the camera branch and the radar branch respectively and radar BEV signature Among them, X and Y represent the length and width of the BEV space respectively; for the two modality-specific BEV features, the camera BEV attention feature F is adaptively generated by the following formula Attn_C and radar BEV attention feature F Attn_R : F Attn_C =Sigmoid(Conv([MP(F c ),AP(F e )])), F Attn_R =Sigmoid(Conv([MP(F R ),AP(F R )])), Where [·,·] represents the concatenation operation along the channel dimension; MP(·) and AP(·) represent the maximum pooling and average pooling operations along the channel dimension, respectively; Conv represents a 3×3 convolution, with 2 and 1 input and output channels, respectively; the fusion feature F Fusion Weighted generation via interactive attention: F Fusion =CBR([F Attn_C ·F R ,F Attn_R ·F R ]), Among them, CBR represents the combination of 3×3 convolution, batch normalization and ReLU, which is used to adjust the channel size.

6. The 3D object detection method based on the three-dimensional deformable attention mechanism enhancement according to claim 1 is characterized in that: S31 is specifically as follows: for the extracted semantic features C and the depth probability estimation result D, firstly, an outer product (×) is performed to obtain a three-dimensional feature space, and then a dot product (·) is performed between the foreground segmentation mask M and the three-dimensional feature space; the above process is expressed as follows: N 3D =M·(D×C), in, Represents the three-dimensional feature space after the target is located in the pixel domain and depth dimension.

7. The 3D object detection method based on the three-dimensional deformable attention mechanism enhancement according to claim 1 is characterized in that: S32 is specifically as follows: First, use the fusion feature F Fusion Initialize the query in the attention mechanism; then, locate the target in the pixel domain and depth dimension in the three-dimensional spatial feature F 3D As the value in the attention mechanism; finally, based on the three-dimensional deformable attention mechanism, the query and value interact continuously to obtain the enhanced BEV feature F Final .

8. The 3D object detection method based on the three-dimensional deformable attention mechanism enhancement according to claim 7 is characterized in that: Based on the three-dimensional deformable attention mechanism, the query and value interact continuously, which includes the following steps: (1) each query is lifted from the BEV plane to the column according to its position in the BEV space, and 3D reference points are sampled from the column and assigned with the corresponding query; (2) the 3D reference point is used as the query, the three-dimensional spatial feature is used as the key and value, and the deformable cross attention is performed by projecting these 3D reference points into the three-dimensional space; the above process is specifically described as: For a 3D query Q at position q q , obtained through the three-dimensional deformable attention mechanism 3DDCA after semantic enhancement Where N represents the number of 3D queries at position q sampled in the deformable attention mechanism; P(q) represents the camera projection function; A n ∈[0,1] is the learnable attention weight; W represents the feature projection weight; represents the predicted offset to position q; represents trilinear interpolation; Finally, all semantically enhanced Reintegrating into the shape of the BEV feature space, we can get Wherein, X and Y represent the length and width of the BEV space.

9. A 3D object detection device based on a three-dimensional deformable attention mechanism enhancement, characterized in that: It includes original feature extraction module, cross-modal feature fusion module, target-oriented attention enhancement module and target detection module; The original feature extraction module is used to extract original millimeter wave radar and monocular image features; The cross-modal feature fusion module is used to obtain the BEV feature at the bird's-eye view perspective based on the original millimeter-wave radar and monocular image features, and fuse the radar BEV feature and the camera BEV feature at the bird's-eye view perspective based on the cross-modal feature fusion mechanism to obtain the fused feature; The target-oriented attention enhancement module is used to enhance the fusion features based on the deformable attention mechanism to obtain enhanced BEV features; The target detection module is used to decode the enhanced BEV features to achieve 3D target detection and positioning.

Images