3D object detection method based on 3D deformable attention mechanism enhancement

Abstract

The present invention discloses a 3D target detection method based on an enhanced three-dimensional deformable attention mechanism, aiming to improve the 3D target detection performance of multimodal sensor fusion. First, the present invention uses an innovatively designed cross-modal feature fusion mechanism to effectively guide and enhance the radar and camera features in the feature fusion process under a bird's-eye view, thereby optimizing the feature initialization in the subsequent three-dimensional deformable attention mechanism. Secondly, the present invention combines lightweight foreground segmentation and depth estimation to achieve guidance control of the three-dimensional deformable attention mechanism in the pixel domain and depth dimension, promoting a deeper and more accurate interaction between the fused features and the image semantic features, and enhancing the semantic information expression capability of the features. Experimental results show that the method of the present invention significantly improves the accuracy and robustness of 3D target detection, especially in complex scenes.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to a 3D target detection method based on an enhanced three-dimensional deformable attention mechanism, aiming to improve the 3D target detection performance of multimodal sensor fusion. Background Art

[0002] With the rapid development of autonomous driving technology, 3D object detection technology plays a vital role in autonomous driving systems. It helps vehicles accurately perceive their surroundings and provides key information such as location and speed, thereby achieving safe driving. Currently, the fusion of laser radar (LiDAR) and cameras is the mainstream method for 3D object detection. LiDAR can provide high-precision geometric depth information, while cameras can capture rich semantic information. The complementary nature of the two provides efficient perception capabilities for 3D object detection. However, LiDAR sensors are expensive and perform poorly in adverse weather conditions.

[0003] In contrast, millimeter-wave radar (4D radar) is becoming an alternative in the field of autonomous driving perception due to its stable operation in inclement weather and low cost. 4D millimeter-wave radar not only provides distance, speed, and angle information for objects, but also includes height information, enabling higher resolution in complex scenes. However, the sparsity and high noise of radar point cloud data limit the performance of millimeter-wave radar in object detection tasks.

[0004] Existing multimodal fusion methods use rich semantic features of camera images to improve the sparse and noisy information of millimeter-wave radar to a certain extent, and therefore have made good progress in 3D target detection. The BEVFusion method maps image features to the bird's-eye view (BEV) space through perspective transformation to better combine the geometric information of the radar. Specifically, BEVFusion was the first to realize the fusion of cross-modal information in the BEV (Bird's Eye View) space, but it did not fully utilize the specific features of each modality before constructing the BEV features. The LXL method further uses the depth estimated by monocular to distinguish image features in space. However, these methods have failed to fully fuse cross-modal bird's-eye view features, nor have they used the powerful role of the attention mechanism to enhance feature expression. The BEVFormer method pioneered the use of an attention mechanism to fuse surround image features for 3D object detection, and the FUTR3D method also uses sparse queries to aggregate multimodal features. However, neither method considers or addresses the difficulties encountered when the query finds the corresponding token. For example, due to the inherent depth inconsistency during the 3D to 2D projection process, the query's target features are blurred and converge slowly during the interaction process. Although the FB-BEV method and the DFA3D method have addressed the depth dimension object localization problem to some extent, no method currently can strengthen the association between the query and the token of interest in the pixel domain. Summary of the Invention

[0005] Considering the inaccuracies and robustness of existing 3D target detection methods that fuse images and radar, this paper proposes a 3D target detection method based on an enhanced 3D deformable attention mechanism. This method uses an innovative cross-modal feature fusion mechanism to effectively guide and enhance radar and camera features during the bird's-eye view feature fusion process, optimizing feature initialization in the subsequent 3D attention mechanism. Simultaneously, combined with lightweight foreground segmentation and depth estimation, it achieves guided control of the 3D attention mechanism in both the pixel domain and depth dimension, promoting deeper and more accurate interaction between fused features and image semantic features, and enhancing the semantic information expression capabilities of the features.

[0006] On one hand, the present invention provides a 3D target detection method based on the enhancement of a three-dimensional deformable attention mechanism. This method uses a cross-modal feature fusion mechanism to fuse millimeter-wave radar and camera features from a bird's-eye view to obtain fused features. Combined with lightweight foreground segmentation and depth estimation, the fused features are enhanced in the pixel domain and depth dimension using a three-dimensional deformable attention mechanism to obtain enhanced BEV features, thereby improving the semantic expression ability of the fused features. The enhanced BEV features are decoded to achieve 3D target detection and positioning. The specific steps of this detection method are as follows:

[0007] S1: Extract the original millimeter-wave radar and monocular image features, and obtain the BEV features from the bird's-eye view based on the corresponding features; specifically, the following steps are included:

[0008] S11: Voxelize the millimeter-wave radar point cloud, extract shallow features, and extract deep features to obtain the original radar point cloud features. Use the point cloud cylinder network to extract the original radar point cloud features, and finally obtain the radar BEV features. Where X and Y represent the size of the BEV space, and C represents the number of channels of the feature map.

[0009] S12: Extract the monocular image features and segment the image foreground area based on the image features. The image features are obtained by encoding F is further extracted through the semantic extraction network to obtain semantic features C is the number of channels of the feature map, H and W represent the height and width of the image respectively, and n represents the downsampling rate. The semantic features are input into the foreground segmentation network to segment the foreground area of ​​the image and obtain the foreground segmentation mask. The process can be expressed as:

[0010] C=ContextNet(F),

[0011] M=SegNet(C),

[0012] ContextNet(·) represents the semantic extraction network, and SegNet(·) represents the foreground segmentation network. Both consist of several convolutional and linear layers. During training, the loss is calculated using two ground-truth labels: the foreground mask pre-segmented by Dectron2 and the foreground mask generated from the ground-truth 2D boxes.

[0013] S13: Use the image features obtained in step S12 to perform depth probability estimation, use the depth probability estimation result and the image features to perform outer product, thereby converting the image features into a bird's-eye view, and then voxel pooling the outer product result to obtain the camera BEV feature. Among them, the image features obtained in step S12 are used to perform depth probability estimation. The specific method is: send the image feature F into the depth completion network to obtain the depth probability estimation result Where D is the number of predefined discrete depth intervals. The depth completion network consists of several layers of convolution Conv and the neighborhood-cross-attention (NCA) mechanism, so the process can be expressed as:

[0014] D=DepthNet(F)=Conv(NCA(Conv(F)0),

[0015] Among them, DepthNet(·) represents the depth completion network.

[0016] S2: Based on the cross-modal feature fusion mechanism, radar BEV features and camera BEV features are fused from the bird's-eye view perspective to obtain fused features. Specifically, camera BEV features can be obtained from the camera branch and radar branch respectively. and radar BEV signatures Where X and Y represent the length and width of the BEV space, respectively. Due to the differences between the two sensors, the two BEV features may show different significance in the semantic description of the same space. The proposed cross-modal feature fusion mechanism can adaptively fuse these two BEV features. Specifically, for the two modality-specific BEV features, the camera BEV attention feature F can be adaptively generated by the following formula: Attn_C and radar BEV attention feature F Attn_R :

[0017] F Attn_C =Sigmoid(Conv([MP(F C ),AP(F C )])),

[0018] F Attn_R =Sigmoid(Conv([MP(F R ),AP(F R )])),

[0019] Among them, [·,·] represents the concatenation operation along the channel dimension; MP(·) and AP(·) represent the maximum pooling and average pooling operations in the channel dimension respectively; Conv represents a 3×3 convolution with 2 and 1 input and output channels respectively. Fusion feature F Fusion Weighted generation via interactive attention:

[0020] F Fusion =CBR([F Attn_C ·F R ,F Attn_R ·F R ]),

[0021] CBR represents a combination of 3×3 convolution, batch normalization, and ReLU, primarily used to adjust channel size. This step fully leverages the advantages of cross-modality. The camera's BEV attention weights mitigate the impact of millimeter-wave radar noise on detection results. Furthermore, the radar attention weights enhance the camera's confidence in potential objects, thereby improving detection performance.

[0022] S3: Combining the foreground segmentation results obtained in step S12 with the depth probability estimation results obtained in step S13, the fused features obtained in step S2 are enhanced using a 3D deformable attention mechanism. Specifically, this step combines lightweight foreground segmentation with depth estimation, enabling guidance and control of the 3D deformable attention mechanism in both the pixel and depth dimensions. This strengthens the association between the 3D deformable attention mechanism's query and key values, enabling deeper and more accurate interaction between the fused features and the image's semantic features, and enhancing the features' ability to express semantic information.

[0023] S31: Use the foreground segmentation result and the depth probability estimation result to locate the target in the three-dimensional feature space, thereby achieving the positioning of the target in the pixel domain and depth dimension. For the extracted semantic features C and the depth probability estimation result D, first make an outer product (×) to obtain the three-dimensional feature space. This step can locate the three-dimensional target in the depth dimension. Then, use the foreground segmentation mask M to make a dot product (·) with the three-dimensional feature space. This step can locate the three-dimensional target in the pixel domain. The process can be specifically expressed as:

[0024] F 3D =M·(D×C),

[0025] in, This represents the 3D feature space after the target is localized in the pixel and depth dimensions. This step highlights the target features in the 3D feature space through localization in the pixel and depth dimensions, which helps the subsequent attention mechanism.

[0026] S32: Based on the positioning of pixel domain and depth dimension, the fusion feature is enhanced using the three-dimensional deformable attention enhancement fusion mechanism to obtain the enhanced BEV feature, thereby improving the semantic expression ability of the fusion feature. This step is specifically as follows: First, use the fusion feature F Fusion Initialize the query in the attention mechanism. Then, the three-dimensional spatial feature F after the target is located in the pixel domain and depth dimension is 3D As the value in the attention mechanism. Finally, based on the three-dimensional deformable attention mechanism, the query and value interact continuously to obtain the enhanced BEV feature F Final .

[0027] Specifically, the three-dimensional deformable attention mechanism enables the query and value to interact continuously, which can be divided into the following steps. (1) Each query is lifted from the BEV plane to the pillar according to its position in the BEV space, and 3D reference points are sampled from the pillar and assigned with the corresponding query. (2) The 3D reference point is used as the query, the three-dimensional spatial feature is used as the key and value, and deformable cross attention is performed by projecting these 3D reference points into the three-dimensional space. The above process is specifically described as follows: for the 3D query Q at position q q , the semantically enhanced

[0028]

[0029] Where N represents the number of 3D queries at position q sampled in the deformable attention mechanism; represents the camera projection function; A n ∈[0,1] is the learnable attention weight; W represents the feature projection weight; represents the predicted offset to position q; Represents trilinear interpolation.

[0030] Finally, the semantically enhanced Reintegrating into the shape of the BEV feature space, we can get Where X and Y represent the length and width of the BEV space.

[0031] S4: Decode the enhanced BEV features described in step S32 to achieve 3D target detection and positioning.

[0032] On the other hand, the present invention provides a 3D object detection device based on a three-dimensional deformable attention mechanism enhancement, the detection device comprising an original feature extraction module, a cross-modal feature fusion module, a target-oriented attention enhancement module and a target detection module;

[0033] The original feature extraction module is used to extract original millimeter wave radar and monocular image features;

[0034] The cross-modal feature fusion module is used to obtain BEV features from a bird's-eye view perspective based on the original millimeter-wave radar and monocular image features, and to fuse the radar BEV features and the camera BEV features from a bird's-eye view perspective based on a cross-modal feature fusion mechanism to obtain a fused feature;

[0035] The goal-oriented attention enhancement module is used to enhance the fusion features based on the deformable attention mechanism to obtain enhanced BEV features;

[0036] The target detection module is used to decode the enhanced BEV features to achieve 3D target detection and positioning.

[0037] The beneficial effects of the present invention are:

[0038] The present invention proposes a 3D target detection method based on the enhancement of a three-dimensional deformable attention mechanism. First, the present invention uses an innovatively designed cross-modal feature fusion mechanism to effectively guide and enhance the radar and camera features in the feature fusion process under a bird's-eye view, thereby optimizing the feature initialization in the subsequent three-dimensional attention mechanism. Secondly, the present invention combines lightweight foreground segmentation and depth estimation to achieve guidance control of the three-dimensional deformable attention mechanism in the pixel domain and depth dimension, promoting a deeper and more accurate interaction between the fused features and the image semantic features, and enhancing the semantic information expression capability of the features. Experimental results show that the method of the present invention significantly improves the accuracy and robustness of 3D target detection, especially in complex scenes.

[0039] Compared to existing methods, this method not only effectively integrates cross-modal bird's-eye view features at the initial stage, but also enhances the query-key association of the 3D deformable attention mechanism through guidance control in the pixel domain and depth dimension of the 3D feature space, achieving deep interaction. This method utilizes a deep learning optimization process to enable the neural network to learn the essential characteristics of the 3D environment during feature fusion, thereby improving detection accuracy and generalization performance. It performs well on pedestrians, bicycles, and cars in test data. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 This is a flowchart of a 3D target detection method based on a three-dimensional deformable attention mechanism enhancement provided by an embodiment of the present invention.

[0041] Figure 2 This is a schematic diagram of the 3D deformable attention mechanism.

[0042] Figure 3 2 is a diagram showing the generalization enhancement effect of an embodiment of the present invention. DETAILED DESCRIPTION

[0043] The following is further described with reference to specific embodiments and drawings.

[0044] Example 1

[0045] This embodiment proposes a 3D object detection method based on the enhancement of a three-dimensional deformable attention mechanism. This method uses a cross-modal feature fusion mechanism to fuse millimeter-wave radar and camera features from a bird's-eye view to obtain fused features. Combined with lightweight foreground segmentation and depth estimation, the fused features are enhanced in the pixel and depth dimensions using a three-dimensional deformable attention mechanism to obtain enhanced BEV features, thereby improving the semantic expression capability of the fused features. The enhanced BEV features are then decoded to achieve 3D object detection and positioning.

[0046] like Figure 1 The flowchart of the detection method specifically includes the following steps:

[0047] S1: Extract the original millimeter-wave radar and monocular image features, and obtain the BEV features from the bird's-eye view based on the corresponding features; this step is specifically as follows:

[0048] S11: Perform voxelization and shallow feature extraction and deep feature extraction on the millimeter wave radar point cloud to obtain the original radar point cloud features, and use the point cloud cylindrical network to further extract radar BEV features Where X and Y represent the size of the BEV space, and C represents the number of channels of the feature map.

[0049] S12: Extract the monocular image features and segment the image foreground area based on the image features. The image features are obtained by encoding F is further extracted through the semantic extraction network to obtain semantic features C is the number of channels of the feature map, H and W represent the height and width of the image respectively, and n represents the downsampling rate. The semantic features are input into the foreground segmentation network to segment the foreground area of ​​the image and obtain the foreground segmentation mask. The process can be expressed as:

[0050] C=ContextNet(F),

[0051] M=SegNet(C),

[0052] Among them, ContextNet(·) represents the semantic extraction network, and SegNet(·) represents the foreground segmentation network, both of which are composed of several layers of convolutional layers and linear layers.

[0053] S13: Use the image features obtained in step S12 to perform depth probability estimation, use the depth probability estimation result and the image features to perform outer product, thereby converting the image features into a bird's-eye view, and then voxel pooling the outer product result to obtain the camera BEV feature. Among them, the image features obtained in step S12 are used to perform depth probability estimation. The specific method is: send the image feature F into the depth completion network to obtain the depth probability estimation result Where D is the number of predefined discrete depth intervals. The depth completion network consists of several layers of convolution Conv and the neighborhood-cross-attention (NCA) mechanism, so the process can be expressed as:

[0054] D=DepthNet(F)=Conv(NCA(Conv(F))),

[0055] Among them, DepthNet(·) represents the depth completion network.

[0056] S2: Based on the cross-modal feature fusion mechanism, radar BEV features and camera BEV features are fused from the bird's-eye view perspective to obtain fused features. Specifically, camera BEV features can be obtained from the camera branch and radar branch respectively. and radar BEV signatures Where X and Y represent the length and width of the BEV space respectively. For the two modality-specific BEV features, the camera BEV attention feature F can be adaptively generated by the following formula Attn_C and radar BEV attention feature F Attn_R :

[0057] F Attn_c =Sigmoid(Conv([MP(F C ),AP(F C )])),

[0058] F Attn_R =Sigmoid(Conv([MP(F R ),AP(F R )])),

[0059] Among them, [·,·] represents the concatenation operation along the channel dimension; MP(·) and AP(·) represent the maximum pooling and average pooling operations in the channel dimension respectively; Conv represents a 3×3 convolution with 2 and 1 input and output channels respectively. Fusion feature F Fusion Weighted generation via interactive attention:

[0060] F Fusion =CBR([F Attn_C ·FR ,F Attn_R ·F R ]),

[0061] Among them, CBR represents the combination of 3×3 convolution, batch normalization and ReLU, which is mainly used to adjust the channel size.

[0062] S3: Combine the foreground segmentation result obtained in step S12 with the depth probability estimation result obtained in step S13, and use the three-dimensional deformable attention mechanism to enhance the fusion feature obtained in step S2 (such as Figure 2 ). The specific steps are:

[0063] S31: Use the foreground segmentation results and the depth probability estimation results to locate the target in the three-dimensional feature space, thereby achieving the positioning of the target in the pixel domain and depth dimension. For the extracted semantic features C and the depth probability estimation results D, first perform the outer product (×) to obtain the three-dimensional feature space, and then use the foreground segmentation mask M to perform the dot product (·) with the three-dimensional feature space. This process can be specifically expressed as:

[0064] F 3D =M·(D×C),

[0065] in, This represents the 3D feature space after the target is localized in the pixel and depth dimensions. This step highlights the target features in the 3D feature space through localization in the pixel and depth dimensions, which helps the subsequent attention mechanism.

[0066] S32: Based on the positioning of pixel domain and depth dimension, the fusion feature is enhanced using the three-dimensional deformable attention enhancement fusion mechanism to obtain the enhanced BEV feature, thereby improving the semantic expression ability of the fusion feature. This step is specifically as follows: First, use the cross-modal fusion BEV feature F Fusion Initialize the query in the attention mechanism. Then, the three-dimensional spatial feature F after the target is located in the pixel domain and depth dimension is 3D As the value in the attention mechanism. Finally, based on the three-dimensional deformable attention mechanism, the query and value interact continuously to obtain the enhanced BEV feature F Final .

[0067] Among them, the three-dimensional deformable attention mechanism enables the query and value to interact continuously, which can be divided into the following steps. (1) Each query is lifted from the BEV plane to the pillar according to its position in the BEV space, and 3D reference points are sampled from the pillar and assigned with the corresponding query. (2) The 3D reference point is used as the query, the three-dimensional spatial feature is used as the key and value, and deformable cross attention is performed by projecting these 3D reference points into the three-dimensional space. The above process is specifically described as follows: for the 3D query Q at position q q , the semantically enhanced

[0068]

[0069] Where N represents the number of 3D queries at position q sampled in the deformable attention mechanism; represents the camera projection function; A n ∈[0,1] is the learnable attention weight; W represents the feature projection weight; represents the predicted offset to position q; Represents trilinear interpolation.

[0070] Finally, the semantically enhanced Reintegrating into the shape of the BEV feature space, we can get Where X and Y represent the length and width of the BEV space.

[0071] S4: Decode the enhanced BEV features described in step S32 to achieve 3D target detection and positioning.

[0072] Example 2

[0073] This embodiment provides a 3D target detection device based on an enhanced three-dimensional deformable attention mechanism. Through the innovatively designed cross-modal feature fusion module, the radar and camera features are effectively guided and enhanced in the feature fusion process from a bird's-eye view, and the feature initialization in the subsequent three-dimensional attention mechanism is optimized. At the same time, a lightweight foreground segmentation network and depth estimation are combined to realize the guidance control of the three-dimensional attention mechanism in the pixel domain and depth dimension, promoting a deeper and more accurate interaction between the fused features and the image semantic features, and enhancing the semantic information expression capability of the features. Ultimately, deep interaction and fusion of multimodal features can be achieved, thereby improving the performance of 3D target detection.

[0074] The detection device includes an original feature extraction module, a cross-modal feature fusion module, a target-oriented attention enhancement module and a target detection module;

[0075] The original feature extraction module is used to extract original millimeter wave radar and monocular image features;

[0076] The cross-modal feature fusion module is used to obtain BEV features from a bird's-eye view perspective based on the original millimeter-wave radar and monocular image features, and to fuse the radar BEV features and the camera BEV features from a bird's-eye view perspective based on a cross-modal feature fusion mechanism to obtain a fused feature;

[0077] The goal-oriented attention enhancement module is used to enhance the fusion features based on the deformable attention mechanism to obtain enhanced BEV features;

[0078] The target detection module is used to decode the enhanced BEV features to achieve 3D target detection and positioning.

[0079] The detection process of the detection device can be divided into the following four stages:

[0080] In the first stage, the original feature extraction module is used to extract the original millimeter wave radar and monocular image features. Use Image Encoder to encode image features Here, C is the number of channels in the feature map, H and W represent the image height and width, respectively, and n represents the downsampling rate. For millimeter-wave radar, a radar cloud encoder is used to encode the millimeter-wave radar point cloud. This process includes voxelization, shallow feature extraction, and deep feature extraction to obtain the original radar point cloud features.

[0081] In the second stage, the camera-radar dual-branch mutual fusion module is used to first obtain the BEV features of the original millimeter-wave radar and monocular image features from a bird's-eye view perspective, and based on the cross-modal feature fusion mechanism, the radar BEV features and camera BEV features are fused from a bird's-eye view perspective to obtain fused features. Specifically, it can be divided into the camera branch (Image Branch) and the radar branch (Radar Branch) for explanation. In the millimeter-wave radar branch of the second stage, since the original radar point cloud features have been extracted from the millimeter-wave radar point cloud in the first stage, the point cloud pillar network (RadarPillar Net) is further used here to further process the original radar point cloud features, and finally the radar point cloud BEV features (Radar-BEV) can be obtained. Here C is the number of channels in the feature map, and X and Y represent the size of the BEV space. In the second stage of the camera branch, F is further extracted through the semantic extraction network to obtain semantic features Input the semantic features into the foreground segmentation network to segment the foreground area of ​​the image and obtain the foreground segmentation mask (Mask) The process can be expressed as:

[0082] C=ContextNet(F),

[0083] M=SegNet(C),

[0084] Among them, ContextNet(·) represents the semantic extraction network and SegNet(·) represents the foreground segmentation network. Both are composed of several layers of convolutional layers and linear layers. During the training process, the loss calculation uses two ground truth labels: the foreground mask pre-segmented by Dectron2 and the foreground mask generated by the 2D box ground truth. Then, the image feature F is fed into the depth completion network to obtain the depth probability estimation result. Where D is the number of predefined discrete depth intervals. The depth completion network consists of several layers of convolution Conv and the neighborhood-cross-attention (NCA) mechanism, so the process can be expressed as:

[0085] D=DepthNet(F)=Conv(NCA(Conv(F))),

[0086] Among them, DepthNet(·) represents the depth completion network.

[0087] Finally, the depth probability estimation result is outer-producted with the image feature, and the outer-product result is voxel pooled (ViewTransformation) to obtain the camera BEV feature (Image-BEV) Then, the camera BEV feature F can be obtained from the camera branch and the radar branch respectively. C BEV with radar features F R Due to the differences between the two sensors, the two BEV features may show different saliency in the semantic description of the same space. The proposed cross-modal feature fusion module (Cross Modality Fusion) can adaptively fuse the two BEV features. Specifically, for the two modality-specific BEV features, the camera BEV attention feature F can be adaptively generated by the following formula Attn_C and radar BEV attention feature F Attn_R :

[0088] F Attn_c =Sigmoid(Conv([MP(F C ),AP(F C )])),

[0089] F Attn_R =Sigmoid(Conv([MP(F R ),AP(F R )])),

[0090] Among them, [·,·] represents the concatenation operation along the channel dimension; MP(·) and AP(·) represent the maximum pooling and average pooling operations in the channel dimension respectively; Conv represents a 3×3 convolution with 2 and 1 input and output channels respectively. Fusion feature F Fusion Weighted generation via interactive attention:

[0091] F Fusion =CBR([F Attn_C ·F R ,F Attn_R ·F R ]),

[0092] CBR stands for 3×3 convolution, batch normalization, and ReLU, which are mainly used to adjust the channel size. This module can fully utilize the advantages of cross-modality.

[0093] In the third stage, the object-oriented attention enhancement module (Object-Oriented Attention) enhances the fused features based on the deformable attention mechanism to obtain enhanced BEV features. Specifically, the enhanced BEV features are obtained by utilizing the location-aware 3D deformable attention mechanism (Localization-Aware Cross-Attetion) and the object-oriented 3D feature space (Object-Oriented 3D Feature Space Creation). First, the foreground segmentation results and the depth probability estimation results are used to locate the target in the 3D feature space, thereby achieving the location of the target in the pixel domain and depth dimension. The foreground segmentation results and the depth probability estimation results are used to locate the target in the 3D feature space, thereby achieving the location of the target in the pixel domain and depth dimension. For the extracted semantic features C and the depth probability estimation results D, the outer product (×) is first performed to obtain the 3D feature space. This step can locate the 3D target in the depth dimension. Then, the foreground segmentation mask M is used to perform the dot product (·) with the 3D feature space. This step can locate the 3D target in the pixel domain. This process can be specifically expressed as:

[0094] F 3D =M·(D×C),

[0095] in, Represents the three-dimensional feature space after target positioning in the pixel domain and depth dimension.

[0096] Then, based on the positioning of pixel domain and depth dimension, the fusion feature is enhanced using the three-dimensional deformable attention enhancement fusion mechanism to obtain the enhanced BEV feature, thereby improving the semantic expression ability of the fusion feature. This step is specifically as follows: First, use the fusion feature F Fusion Initialize the query in the attention mechanism. Then, the three-dimensional spatial feature F after the target is located in the pixel domain and depth dimension is 3D As the value in the attention mechanism. Finally, based on the three-dimensional deformable attention mechanism, the query and value interact continuously to obtain the enhanced BEV feature F Final .

[0097] In the fourth stage, the anchor-based feature decoding head is used to decode the enhanced BEV features F Final Decoding is performed to achieve 3D target detection and positioning.

[0098] Figure 3 The results of the proposed method on the TJ4DRadSet dataset are shown below. Each image corresponds to a data frame containing images and radar points (gray). The red triangle marks the vehicle position. The orange and yellow boxes represent the true boxes in the perspective view and the bird's-eye view, respectively. The green and blue boxes represent the bounding boxes predicted by the proposed method. The lower left half of the figure shows the visualization of the BEV feature map. Figure 3 From (a), (b) and (c), we can see that the method of the present invention has achieved good results in detecting cars and trucks in the data set.

Claims

1. A 3D object detection method based on a 3D deformable attention mechanism enhancement, characterized in that: Through the cross-modal feature fusion mechanism, the millimeter-wave radar and camera features are fused from a bird's-eye view to obtain fused features; Combining lightweight foreground segmentation and depth estimation, the fused features are enhanced in the pixel domain and depth dimension using a 3D deformable attention mechanism to obtain enhanced BEV features, thereby improving the semantic expression ability of the fused features. The enhanced BEV features are decoded to achieve 3D target detection and positioning. The specific steps are as follows: S1: Extract the original millimeter-wave radar and monocular image features, and obtain the BEV features from the bird's-eye view based on the corresponding features; S11: Voxelize the millimeter-wave radar point cloud and perform feature extraction to obtain raw radar point cloud features, and further extract radar BEV features based on the raw radar point cloud features; S12: extracting monocular image features and segmenting the image foreground area based on the image features; S13: Use the image features obtained in step S12 to perform depth probability estimation, use the depth probability estimation result and the image features to perform outer product, thereby converting the image features into a bird's-eye view, and then perform voxel pooling on the outer product result to obtain the camera BEV feature; S2: Based on the cross-modal feature fusion mechanism, radar BEV features and camera BEV features are fused from the bird's-eye view perspective to obtain fused features; S3: Combining the foreground segmentation result obtained in step S12 with the depth probability estimation result obtained in step S13, the fused features obtained in step S2 are enhanced using a three-dimensional deformable attention mechanism; S31: Use the foreground segmentation results and depth probability estimation results to locate the target in the three-dimensional feature space, thereby achieving the positioning of the target in the pixel domain and depth dimension; S32: Based on the positioning in the pixel domain and depth dimension, the fused features are enhanced using the 3D deformable attention enhancement fusion mechanism to obtain the enhanced BEV features, thereby improving the semantic expression ability of the fused features; S4: Decode the enhanced BEV features described in step S32 to achieve 3D target detection and positioning.

2. The 3D object detection method based on the enhanced 3D deformable attention mechanism according to claim 1, characterized in that: Step S12 is specifically as follows: , which is encoded to obtain image features , Semantic features are further extracted through the semantic extraction network , is the number of channels of the feature map, and Represents the height and width of the image respectively, Indicates the downsampling rate; input the semantic features into the foreground segmentation network to segment the foreground area of ​​the image and obtain the foreground segmentation mask ; The above process is expressed as: in, represents the semantic extraction network, Represents the foreground segmentation network, both of which are composed of several layers of convolutional layers and linear layers.

3. The 3D object detection method based on the enhanced 3D deformable attention mechanism according to claim 2, characterized in that: In step S13, the image features obtained in step S12 are used to perform depth probability estimation. The specific method is: Send it to the depth completion network to get the depth probability estimation result ,in is the number of predefined discrete depth intervals; The above process can be expressed as: ， Among them, DepthNet Represents a deep completion network consisting of several layers of convolution and neighborhood attention.

4. The 3D object detection method based on the enhanced 3D deformable attention mechanism according to claim 2, characterized in that: S2 is specifically: obtain camera BEV features from the camera branch and radar branch respectively and radar BEV signatures ,in, and Represent the length and width of the BEV space respectively; for the two modality-specific BEV features, the camera BEV attention features are adaptively generated by the following formula and radar BEV attention features : Among them, [·,·] represents the concatenation operation along the channel dimension; MP(·) and AP(·) represent the maximum pooling and average pooling operations along the channel dimension respectively; Conv represents Convolution, the number of input and output channels are 2 and 1 respectively; fusion features Weighted generation via interactive attention: Among them, CBR means A combination of convolution, batch normalization, and ReLU is used to adjust the channel size.

5. The 3D object detection method based on 3D deformable attention mechanism enhancement according to claim 1, characterized in that: S31 is specifically as follows: for the extracted semantic features And the depth probability estimation results , first do the outer product Get the three-dimensional feature space, and then use the foreground segmentation mask Dot product with the three-dimensional feature space ; The above process is expressed as follows: in, Represents the three-dimensional feature space after the target is located in the pixel domain and depth dimension.

6. The 3D object detection method based on 3D deformable attention mechanism enhancement according to claim 1, characterized in that: S32 is specifically as follows: First, use the fusion feature Initialize the query in the attention mechanism; then, locate the target in the pixel domain and depth dimension in the three-dimensional spatial features As the value in the attention mechanism; finally, based on the three-dimensional deformable attention mechanism, the query and value interact continuously to obtain the enhanced BEV feature .

7. The 3D object detection method based on the enhanced 3D deformable attention mechanism according to claim 6, characterized in that: The query and value are continuously interacted based on the three-dimensional deformable attention mechanism, which includes the following steps: (1) each query is lifted from the BEV plane to the column according to its position in the BEV space, and 3D reference points are sampled from the column and assigned with the corresponding query; (2) the 3D reference points are used as queries, the three-dimensional spatial features are used as keys and values, and deformable cross attention is performed by projecting these 3D reference points into the three-dimensional space. The above process is specifically described as follows: For the location 3D query at , through the 3D deformable attention mechanism After semantic enhancement : ， Where N represents the position The number of 3D queries at t that are sampled in the deformable attention mechanism; represents the camera projection function; is the learnable attention weight; represents the feature projection weight; Indicates position The prediction offset of represents trilinear interpolation; Finally, all semantically enhanced Reintegrating into the shape of the BEV feature space, we can get ,in, and Indicates the length and width of the BEV space.

8. A 3D object detection device based on a three-dimensional deformable attention mechanism enhancement, used to implement the method according to any one of claims 1 to 7, characterized in that: It includes original feature extraction module, cross-modal feature fusion module, target-oriented attention enhancement module and target detection module; The original feature extraction module is used to extract original millimeter wave radar and monocular image features; The cross-modal feature fusion module is used to obtain BEV features from a bird's-eye view perspective based on the original millimeter-wave radar and monocular image features, and to fuse the radar BEV features and the camera BEV features from a bird's-eye view perspective based on a cross-modal feature fusion mechanism to obtain a fused feature; The goal-oriented attention enhancement module is used to enhance the fusion features based on the deformable attention mechanism to obtain enhanced BEV features; The target detection module is used to decode the enhanced BEV features to achieve 3D target detection and positioning.

Images