Three-dimensional target detection method based on deformable attention and adaptive fusion guidance

Abstract

The invention discloses a three-dimensional target detection method guided by deformable attention and adaptive fusion, and the method comprises the steps: constructing a multi-modal feature fusion model guided by a deformable shared attention mechanism and a density threshold from the two aspects of single-modal feature quality improvement and multi-modal fusion strategy optimization; coarse-grained cross-modal fusion features are used as priori, multi-modal shared space offset and attention weight are dynamically generated through learnable parameters, and each modal is guided to adaptively adjust feature response and sampling position in a feature space; according to the method, adaptive weighted fusion is carried out on the BEV features of the point cloud and the image through an adaptive Threshold Guidance method guided by a density threshold. According to the method, the single-mode feature quality is remarkably improved, robust fusion representation with strong generalization ability to environment change is constructed, and the deployability of the model in a vehicle-mounted edge computing platform is improved; the model achieves excellent performance.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving technology, and in particular to a three-dimensional target detection method guided by deformable attention and adaptive fusion. Background Technology

[0002] Accurate detection of traffic targets such as vehicles and pedestrians is crucial for ensuring the safe and reliable operation of intelligent vehicles. Multimodal fusion, by leveraging the complementary characteristics of heterogeneous sensor data such as images and point clouds, can significantly improve target detection accuracy in complex traffic scenarios.

[0003] In recent years, multimodal fusion methods based on bird's eye view (BEV) have been widely used in the field of traffic target detection by effectively solving the problems of projection distortion and scale mismatch in traditional front view fusion by mapping cross-modal features to a unified feature space.

[0004] In research on multimodal fusion-based 3D object detection using BEV (Body Estimation Vehicle), feature enhancement and fusion strategies are two key directions. Regarding feature enhancement, BEVFix uses wavelet transform for multi-frequency decomposition and combines it with a feedforward network to achieve noise suppression and key feature preservation. Chen et al. designed a deep semantic and instance BEV feature enhancement module, significantly improving feature quality. In terms of feature fusion, existing methods can be mainly divided into two categories: implicit fusion based on depth estimation and explicit fusion based on queries. Implicit fusion methods estimate depth information from images and project image features onto the BEV space. To overcome the error accumulation problem caused by the heavy reliance on depth estimation quality in depth estimation methods, researchers proposed an explicit fusion method based on query-based Transformers. Although explicit fusion is effective, it suffers from high computational complexity, lacks discriminative power regarding feature quality in its interaction mechanism, and struggles to adapt to dynamic changes in modal reliability in open scenes.

[0005] For real-world open road scenarios, traffic target detection models still face challenges such as environmental disturbances, sensor limitations, and diverse target distances. These challenges can easily lead to local degradation of single-modal features and dynamic changes in confidence between modalities, affecting model stability and generalization ability.

[0006] While existing fusion models have made significant progress in improving target detection accuracy, they still have the following limitations under the aforementioned challenges:

[0007] First, the lack of an effective compensation mechanism for degraded features leads to limited quality of underlying representations;

[0008] Second, the static weight allocation strategy is difficult to adapt to the differences in modal confidence under different conditions, resulting in unstable model detection performance.

[0009] Therefore, in order to solve the problems existing in the prior art, a method that can improve the quality of intramodal features while optimizing intermodal fusion strategies is a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0010] In view of this, the present invention provides a three-dimensional target detection method guided by deformable attention and adaptive fusion.

[0011] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0012] A 3D target detection method guided by deformable attention and adaptive fusion includes the following steps:

[0013] Step 1: Construct a deformable shared attention mechanism and a density threshold-guided multimodal feature fusion model from two levels: single-modal feature quality improvement and multimodal fusion strategy optimization;

[0014] Step 2: Using coarse-grained cross-modal fusion features as a priori, learnable parameters are used to dynamically generate shared spatial offsets and attention weights for multiple modalities. This guides each modality to adaptively adjust its feature response and sampling position in the feature space, thereby enhancing key region features, compensating for missing features, and suppressing redundant noise. This improves the discriminativeness and robustness of single-modal features and provides high-quality feature input for multimodal fusion.

[0015] Step 3: The Density Threshold Guidance adaptive multimodal feature fusion method is used to adaptively weight and fuse point cloud and image BEV features to enhance the model’s generalization ability in open scenes. The DTG architecture mainly includes three modules: density prior estimation, gated feature generation, and fusion strategy selection.

[0016] Preferably, step 1 specifically includes the following steps:

[0017] Step 1-1: Construct parallel point cloud and image single-modal BEV feature extraction branches to achieve robust extraction of single-modal features;

[0018] Given point cloud input First, based on the PointPillars framework, the image is divided into regular columnar units (Pillars), and then mapped to pseudo-image features using a two-dimensional convolutional encoder and the Scatter operator. ,in, , , , These represent the boundary ranges of the point cloud in the X and Y directions, respectively. The dimensions of each Pillar unit on the point cloud plane are determined, and then a cascaded multi-layer 2D convolutional network is used to process pseudo-image features. Obtain point cloud BEV features ;

[0019] Step 1-2: Given Zhang resolution is Multi-view panoramic image input, each image input First, a multi-resolution feature extraction module (MFE) with an embedded channel attention mechanism (S) is used to extract hierarchical features at four scales for each input image. Furthermore, based on transposed convolution, multi-scale features are upsampled hierarchically to gradually restore spatial resolution. Simultaneously, upsampling results from different scales are stitched together along the channel dimension to generate a multi-view image feature tensor. Then, using the Fast-Ray view transformation method in Fast-BEV, the view transformation is performed. Mapping to voxel space yields voxel features. Finally, based on two-dimensional convolutional networks, Spatial aggregation and channel compression are performed to output the BEV features of the image. ;

[0020] Steps 1-3: ReLU6 and Hard-swish activation functions are used in the multi-resolution feature extraction module and the SE module. Both functions simplify computation through boundary constraints and piecewise linear approximation, respectively. Their function definitions are as follows:

[0021]

[0022]

[0023] Preferably, step 2 specifically includes the following steps:

[0024] Step 2-1: First, generate coarse-grained cross-modal features based on the following formula. ,

[0025]

[0026] in, This indicates a splicing operation. For pointwise convolution, used to compress the dimension of the concatenated feature channels, then... This feature initially integrates the BEV features of point cloud and image, and can provide global context information for single-modal feature calibration;

[0027] Step 2-2: Features Learnable weights Mapped to query features and combined with positional encoding to obtain Query vector at the location Then the deformable shared attention on the BEV features of the point cloud is defined as follows:

[0028]

[0029] Where M is the number of attention heads and K is the number of sampling points. It is obtained by linear projection and Softmax normalization of the query vector q, and corresponds to the attention weights of the m-th attention head and the k-th sampling point; The corresponding sampling offset is obtained by regression of q through a fully connected layer; These are all learnable weights corresponding to each attention point;

[0030] Steps 2-3: The deformable shared attention for BEV features of an image is defined as follows:

[0031]

[0032] At this point, all mapping operations are performed on the BEV features of the image. Execution on top, and attention weight and sampling offset It shares modes with lidar, thereby implicitly constraining the spatial consistency of the two modes during the feature sampling stage.

[0033] Preferably, step 3 specifically includes the following steps:

[0034] Step 3-1: Density Prior Estimation:

[0035] First, the number of point clouds within each Pillar is defined as the point cloud density, and the densities of all Pillars constitute a density vector. The 1024-dimensional features output by the MLP are mapped to a high-dimensional feature vector through a shared multilayer perceptron (MLP) with parameters consistent with PointNet

[19] . The MLP output is then globally pooled to capture the distribution features, and then compressed to the [0,1] interval by the Sigmoid function to generate an adaptive density threshold. Simultaneously, an unsupervised loss function based on distribution differences is designed to optimize threshold learning.

[0036]

[0037] in, For balance coefficient, Represents a low-density point set. Represents a high-density point set. This indicates calculating the mean. This means calculating the standard deviation. This loss function maximizes the difference between the means of high and low density groups while penalizing the dispersion by using the standard deviation within each group, so that the threshold can automatically locate the natural gaps in the data distribution and achieve effective division of high and low density regions.

[0038] Step 3-2: Gating Feature Generation:

[0039] Image gating weight matrix generated based on cross-modal gating mechanism. and point cloud gating weight matrix ,

[0040]

[0041] in, , , This represents the Sigmoid activation function. Represent the learnable parameters; compare the gating weights with... and Pixel-by-pixel multiplication yields image-gated enhancement features. and point cloud gating enhancement features ,

[0042]

[0043] Step 3-3: Selection of Fusion Strategy

[0044] Based on the principles of high-confidence mode dominance and complementary mode enhancement, a differentiated fusion strategy guided by point cloud density threshold is designed, and an adaptive density threshold is used. The scene is divided into high-density and low-density regions. In the high-density region, point cloud features are concatenated with image gating enhancement features, and in the low-density region, image features are concatenated with point cloud gating enhancement features. Then, the features of different regions are fused by index connection to maintain spatial topological continuity.

[0045]

[0046] in, For multimodal fusion features, Indicates index join, This indicates a splicing operation.

[0047] The present invention achieves the following technical effects compared to the prior art:

[0048] (1) The present invention designs a deformable shared attention mechanism, which generates offset vectors and attention weights through coarse-grained fusion features, guides the calibration of single-modal features, enhances the feature representation of key regions while suppressing redundant information interference, thereby significantly improving the quality of single-modal features;

[0049] (2) This invention proposes an adaptive multimodal feature fusion method guided by density threshold. By analyzing the density distribution of point cloud and scene context information, the reliability of each modality is evaluated, and its confidence and fusion strategy are adjusted. This realizes the adaptive weighted fusion of point cloud and image BEV features and constructs a robust fusion representation with strong generalization ability to environmental changes.

[0050] (3) The present invention introduces a lightweight structural design. By embedding a high-efficiency computing module, the computational complexity is significantly reduced while maintaining the detection accuracy, thereby improving the deployability of the model in the vehicle edge computing platform.

[0051] (4) The comparative experimental results of this invention on the nuScenes public dataset show that the DSAM-DTG model surpasses the state-of-the-art methods and achieves excellent performance. Attached Figure Description

[0052] Figure 1 This is the overall architecture diagram of the "3D target detection method guided by deformable attention and adaptive fusion" of the present invention;

[0053] Figure 2 This is a visualization of the results of this invention. Detailed Implementation

[0054] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0055] like Figure 1 As shown, the framework of the present invention is illustrated. The model input consists of synchronously acquired LiDAR point clouds and corresponding multi-view panoramic images, which are respectively input to parallel point cloud branches and image branches to extract BEV features.

[0056] The extracted initial single-modal features are calibrated within the modality through a deformable shared attention mechanism to obtain high-quality, high-discriminative single-modal feature representations. The calibrated features are further dynamically adjusted for modal confidence through a density threshold-guided fusion strategy to generate robust multimodal fusion features.

[0057] The fused features are output via the 3D detection head in an end-to-end manner, along with the target's 3D bounding box and category label, thus completing the 3D target detection task.

[0058] This invention discloses a 3D target detection method guided by deformable attention and adaptive fusion, comprising the following steps:

[0059] Step 1: In real-world open road scenarios, traffic target detection models still face challenges such as environmental disturbances, sensor limitations, and diverse target distances. These challenges can easily lead to local degradation of single-modal features and dynamic changes in inter-modal confidence, affecting model stability and generalization ability. Although existing fusion models have made significant progress in improving target detection accuracy, they still have many limitations under the aforementioned challenges. This invention addresses these challenges by focusing on two aspects: improving the quality of single-modal features and optimizing multimodal fusion strategies. It constructs a deformable shared attention mechanism and a density threshold-guided multimodal feature fusion model, effectively mitigating the feature representation degradation problem in open road environments.

[0060] Step 1 specifically includes the following sub-steps:

[0061] Step 1-1: Parallel point cloud and image single-modal BEV feature extraction branches were constructed to achieve robust extraction of single-modal features. Given point cloud input... First, based on the PointPillars framework, the image is divided into regular columnar units (Pillars), and then mapped to pseudo-image features using a two-dimensional convolutional encoder and the Scatter operator. ,in , , , These represent the boundary ranges of the point cloud in the X and Y directions, respectively. The dimensions of each Pillar unit on the point cloud plane are determined, and then a cascaded multi-layer 2D convolutional network is used to process pseudo-image features. Obtain point cloud BEV features .

[0062] Step 1-2: Given Zhang resolution is Multi-view panoramic image input, each image input First, a multi-resolution feature extraction (MFE) module with an embedded channel attention mechanism S is used to extract hierarchical features at four scales for each input image. Furthermore, based on transposed convolution, multi-scale features are upsampled hierarchically to gradually restore spatial resolution. Simultaneously, upsampling results from different scales are stitched together along the channel dimension to generate a multi-view image feature tensor. Then, using the Fast-Ray view transformation method in Fast-BEV, the view transformation is performed. Mapping to voxel space yields voxel features. Finally, based on two-dimensional convolutional networks, Spatial aggregation and channel compression are performed to output the BEV features of the image. .

[0063] Steps 1-3: To effectively reduce time complexity while maintaining feature representation capabilities, ReLU6 and Hard-swish activation functions are used in the multi-resolution feature extraction module and the SE module. These functions simplify computation through boundary constraints and piecewise linear approximation, respectively. Their function definitions are as follows:

[0064]

[0065]

[0066] Step 2: In open road environments, factors such as occlusion, sensor noise, and environmental changes can easily cause degradation of single-modal feature quality. If point cloud BEV features are directly fused with image BEV features, low-quality features in the degraded modality will introduce noise and errors, thereby reducing the detection accuracy and generalization ability of the multimodal fusion model.

[0067] Therefore, a Deformable Shared Attention Mechanism (DSAM) was designed. Using coarse-grained cross-modal fusion features as a prior, it dynamically generates shared spatial offsets and attention weights for multiple modalities through learnable parameters. This guides each modality to adaptively adjust its feature response and sampling position in the feature space, thereby achieving feature enhancement in key regions, compensation for missing features, and suppression of redundant noise. This significantly improves the discriminativeness and robustness of single-modal features and provides high-quality feature input for multimodal fusion.

[0068] Step 2 specifically includes the following sub-steps:

[0069] Step 2-1: First, generate coarse-grained cross-modal features based on the following formula. ,

[0070]

[0071] in, This indicates a splicing operation. For pointwise convolution, used to compress the dimension of the concatenated feature channels, then... This feature initially integrates the BEV features of point cloud and image, and can provide global contextual information for single-modal feature calibration.

[0072] Step 2-2: Features Learnable weights Mapped to query features and combined with positional encoding to obtain Query vector at the location Then the deformable shared attention on the BEV features of the point cloud is defined as:

[0073]

[0074] Where M is the number of attention heads and K is the number of sampling points. It is obtained by linear projection and Softmax normalization of the query vector q, and corresponds to the attention weights of the m-th attention head and the k-th sampling point; The corresponding sampling offset is obtained by regression of q through a fully connected layer; These are all learnable weights corresponding to each attention point.

[0075] Steps 2-3: Similarly, the deformable shared attention of BEV features in an image can be defined as:

[0076]

[0077] At this point, all mapping operations are performed on the BEV features of the image. Execution on top, and attention weight and sampling offset It shares modes with lidar, thereby implicitly constraining the spatial consistency of the two modes during the feature sampling stage.

[0078] Step 3: In open road scenarios, the degree of feature degradation varies significantly across different modalities. If a static fusion strategy is used, it is difficult to dynamically adjust the fusion weights and strategies based on the reliability of each modality, leading to a decline in model detection performance. To address this, this invention proposes a density threshold-guided adaptive multimodal feature fusion method, DTG (Density Threshold Guidance). By adaptively weighting and fusing point cloud and image BEV features, it significantly enhances the model's generalization ability in open scenarios. The proposed DTG architecture mainly includes three modules: density prior estimation, gated feature generation, and fusion strategy selection.

[0079] Step 3 specifically includes the following sub-steps:

[0080] Step 3-1: Density Prior Estimation:

[0081] First, the number of point clouds within each Pillar is defined as the point cloud density, and the densities of all Pillars constitute a density vector. Through with PointNet A shared multilayer perceptron (MLP) with consistent parameters is mapped to a high-dimensional feature vector; the 1024-dimensional features output by the MLP are globally pooled to capture distribution features, and then compressed to the [0,1] interval using the sigmoid function to generate an adaptive density threshold. Simultaneously, an unsupervised loss function based on distribution differences is designed to optimize threshold learning.

[0082]

[0083] in, For balance coefficient, Represents a low-density point set. Represents a high-density point set. This indicates calculating the mean. The loss function calculates the standard deviation. It maximizes the difference between the means of high and low density groups while penalizing the dispersion by using the standard deviation within each group. This allows the threshold to automatically locate the natural gaps in the data distribution, thus achieving effective division of high and low density regions.

[0084] Step 3-2: Gating Feature Generation:

[0085] Image gating weight matrix generated based on cross-modal gating mechanism. and point cloud gating weight matrix ,

[0086]

[0087] in, , , This represents the Sigmoid activation function. This represents the learnable parameters. The gating weights are respectively compared with... and Pixel-by-pixel multiplication yields image-gated enhancement features. and point cloud gating enhancement features ,

[0088]

[0089] Step 3-3: Selection of Fusion Strategy

[0090] Based on the principles of high-confidence mode dominance and complementary mode enhancement, a differentiated fusion strategy guided by point cloud density threshold is designed, and an adaptive density threshold is used. The scene is divided into high-density and low-density regions. In the high-density regions, point cloud features are concatenated with image-gated enhancement features, while in the low-density regions, image features are concatenated with point cloud-gated enhancement features. Then, features from different regions are fused using indexed connections to maintain spatial topological continuity.

[0091]

[0092] in, For multimodal fusion features, Indicates index join, This indicates a splicing operation.

[0093] Experimental verification:

[0094] This invention conducts comparative experiments on the nuScenes dataset;

[0095] The evaluation metrics used are mean average precision (mAP) and multiple true positive metrics, including mean translation error (mATE), mean scale error (mASE), mean angle error (mAOE), mean velocity error (mAVE), and mean attribute error (mAAE), and the nuScenes detection score (NDS) is calculated in combination.

[0096] The comparative experimental results based on the nuScenes validation set are shown in Table 1, where the bolded values ​​represent the optimal results for each index.

[0097] Experimental results show that the proposed DSAM-DTG model achieves excellent performance with mAP and NDS scores of 75.6% and 77.9%, respectively. The model's mean translation error (mATE) and mean scale error (mASE) are 0.228 and 0.225, respectively, both optimal, indicating strong geometric perception capabilities and high accuracy in target localization and size estimation. In terms of orientation perception, the model's mean orientation error (mAOE) is 0.265, slightly higher than the UniTR model (0.256), but still at a good level. Regarding motion state estimation, the model's mean velocity error (mAVE) is 0.152, indicating that the model can provide stable and reliable spatial features for velocity estimation. In terms of mean attribute error (mAAE), the proposed model, with a score of 0.116, is tied for best with SparseLIF, verifying that the features extracted by DSAM-DTG have high discriminative power and can effectively support the recognition of high-level semantic attributes.

[0098] Table 1: Comparative test results based on the nuScenes validation set

[0099]

[0100]

[0101] The nuScenes validation set was divided into four typical scenarios: sunny day (5051 frames), rainy day (968 frames), daytime (5417 frames), and nighttime (602 frames). The test results are shown in Table 2.

[0102] The analysis results show that both rainy days and nighttime scenes lead to a decrease in model performance, with the perception accuracy deteriorating particularly significantly in nighttime scenes.

[0103] The model of this invention achieves optimal mAP and NDS in sunny, daytime, and nighttime scenarios. Particularly in nighttime scenarios, the model still achieves 47.9% mAP and 49.8% NDS, significantly outperforming the comparative models, indicating its good robustness to environmental and lighting changes. In rainy scenarios, the model's mAP did not reach the optimal level. The reason for this is that rain and fog interference cause a simultaneous decrease in the modal quality of the LiDAR and camera, reducing the performance gain of the feature calibration and fusion strategies.

[0104] Overall, the DSAM-DTG model demonstrated more stable and robust detection performance under various weather and lighting conditions, validating its effectiveness and adaptability in open environments.

[0105] Table 2: Comparison Test Results of Model-Environment Robustness Based on nuScenes Validation Set

[0106]

[0107] Based on the three-dimensional Euclidean distance from the target center point to the vehicle origin, the nuScenes validation set is divided into four distance intervals: ultra-close distance (0-15m), close distance (15-30m), medium distance (30-50m), and long distance (50-80m).

[0108] Table 3 shows the detection accuracy for pedestrians and vehicles at different distances. The detection accuracy of all comparative models decreases with increasing target distance, but the DSAM-DTG model maintains optimal performance across all distance ranges. In the very close and near distance ranges, all models perform well; the LAS model experiences approximately 10.5% performance degradation in the mid-range, indicating its sensitivity to distance changes; in the long-range range, the detection accuracy of all comparative models decreases by more than 20% compared to the mid-range, while DSAM-DTG maintains a vehicle detection accuracy of 74.8% and a pedestrian detection accuracy of 70.7%, which are 20.8% and 18.2% higher than the LSA model, and 13.8% and 12.7% higher than the SparseLIF model, respectively, validating the strong robustness of the proposed model to distance changes.

[0109] Table 3: Comparison of Target Detection Accuracy at Different Distances - Test Results

[0110]

[0111] Visualized detection results based on the nuScenes dataset, as shown below Figure 2 As shown.

[0112] The central area of ​​the image shows the detection output of the point cloud modality, surrounded by the detection results of six camera modalities from different perspectives. To clearly demonstrate the model's detection performance, occluded and truncated targets are marked in light blue, and distant targets are marked in light purple.

[0113] The results show that the DSAM-DTG model can accurately estimate the 3D bounding box of the target and maintains stable and superior detection performance in challenging scenarios such as occlusion, truncation, and distant targets, fully verifying that the model has reliable 3D perception capabilities in open road scenarios.

[0114] The above description is merely a preferred embodiment of the present invention and does not constitute any limitation on the technical scope of the present invention. Therefore, any minor modifications, equivalent changes, and alterations made to the above embodiments based on the technical essence of the present invention shall still fall within the scope of the technical solution of the present invention.

Claims

1. A three-dimensional target detection method guided by deformable attention and adaptive fusion, characterized in that, Includes the following steps: Step 1: Construct a deformable shared attention mechanism and a density threshold-guided multimodal feature fusion model from two levels: single-modal feature quality improvement and multimodal fusion strategy optimization; Step 2: Using coarse-grained cross-modal fusion features as a priori, learnable parameters are used to dynamically generate shared spatial offsets and attention weights for multiple modalities. This guides each modality to adaptively adjust its feature response and sampling position in the feature space, thereby enhancing key region features, compensating for missing features, and suppressing redundant noise. This improves the discriminativeness and robustness of single-modal features and provides high-quality feature input for multimodal fusion. Step 3: The Density Threshold Guidance adaptive multimodal feature fusion method is used to adaptively weight and fuse point cloud and image BEV features to enhance the model’s generalization ability in open scenes. The DTG architecture mainly includes three modules: density prior estimation, gated feature generation, and fusion strategy selection.

2. The 3D target detection method guided by deformable attention and adaptive fusion according to claim 1, characterized in that, Step 1 specifically includes the following steps: Step 1-1: Construct parallel point cloud and image single-modal BEV feature extraction branches to achieve robust extraction of single-modal features; Given point cloud input First, based on the PointPillars framework, the image is divided into regular columnar units (Pillars), and then mapped to pseudo-image features using a two-dimensional convolutional encoder and the Scatter operator. ,in, , , , These represent the boundary ranges of the point cloud in the X and Y directions, respectively. The dimensions of each Pillar unit on the point cloud plane are determined, and then a cascaded multi-layer 2D convolutional network is used to process pseudo-image features. Obtain point cloud BEV features ; Step 1-2: Given Zhang resolution is Multi-view panoramic image input, each image input First, a multi-resolution feature extraction module (MFE) with an embedded channel attention mechanism (S) is used to extract hierarchical features at four scales for each input image. Furthermore, based on transposed convolution, multi-scale features are upsampled hierarchically to gradually restore spatial resolution. Simultaneously, upsampling results from different scales are stitched together along the channel dimension to generate a multi-view image feature tensor. Then, using the Fast-Ray view transformation method in Fast-BEV, the view transformation is performed. Mapping to voxel space yields voxel features. Finally, based on two-dimensional convolutional networks, Spatial aggregation and channel compression are performed to output the BEV features of the image. ; Steps 1-3: ReLU6 and Hard-swish activation functions are used in the multi-resolution feature extraction module and the SE module. Both functions simplify computation through boundary constraints and piecewise linear approximation, respectively. Their function definitions are as follows:

3. The three-dimensional target detection method guided by deformable attention and adaptive fusion according to claim 1, characterized in that, Step 2 specifically includes the following steps: Step 2-1: First, generate coarse-grained cross-modal features based on the following formula. , in, This indicates a splicing operation. For pointwise convolution, used to compress the dimension of the concatenated feature channels, then... This feature initially integrates the BEV features of point cloud and image, and can provide global context information for single-modal feature calibration; Step 2-2: Features Learnable weights Mapped to query features and combined with positional encoding to obtain Query vector at the location Then the deformable shared attention on the BEV features of the point cloud is defined as follows: Where M is the number of attention heads and K is the number of sampling points. It is obtained by linear projection and Softmax normalization of the query vector q, and corresponds to the attention weights of the m-th attention head and the k-th sampling point; The corresponding sampling offset is obtained by regression of q through a fully connected layer; These are all learnable weights corresponding to each attention point; Steps 2-3: The deformable shared attention for BEV features of an image is defined as follows: At this point, all mapping operations are performed on the BEV features of the image. Execution on top, and attention weight and sampling offset It shares modes with lidar, thereby implicitly constraining the spatial consistency of the two modes during the feature sampling stage.

4. The three-dimensional target detection method guided by deformable attention and adaptive fusion according to claim 1, characterized in that, Step 3 specifically includes the following steps: Step 3-1: Density Prior Estimation: First, the number of point clouds within each Pillar is defined as the point cloud density, and the densities of all Pillars constitute a density vector. The 1024-dimensional features output by the MLP are mapped to a high-dimensional feature vector through a shared multilayer perceptron (MLP) with parameters consistent with PointNet[19]. The MLP output is then globally pooled to capture the distribution features, and then compressed to the [0,1] interval by the Sigmoid function to generate an adaptive density threshold. Simultaneously, an unsupervised loss function based on distribution differences is designed to optimize threshold learning. in, For balance coefficient, Represents a low-density point set. Represents a high-density point set. This indicates calculating the mean. This means calculating the standard deviation. This loss function maximizes the difference between the means of high and low density groups while penalizing the dispersion by using the standard deviation within each group, so that the threshold can automatically locate the natural gaps in the data distribution and achieve effective division of high and low density regions. Step 3-2: Gating Feature Generation: Image gating weight matrix generated based on cross-modal gating mechanism. and point cloud gating weight matrix , in, , , This represents the Sigmoid activation function. Represent the learnable parameters; compare the gating weights with... and Pixel-by-pixel multiplication yields image-gated enhancement features. and point cloud gating enhancement features , Step 3-3: Selection of Fusion Strategy Based on the principles of high-confidence mode dominance and complementary mode enhancement, a differentiated fusion strategy guided by point cloud density threshold is designed, and an adaptive density threshold is used. The scene is divided into high-density and low-density regions. In the high-density region, point cloud features are concatenated with image gating enhancement features, and in the low-density region, image features are concatenated with point cloud gating enhancement features. Then, the features of different regions are fused by index connection to maintain spatial topological continuity. in, For multimodal fusion features, Indicates index join, This indicates a splicing operation.

Images