Three-dimensional object detection method based on multi-modal voxel image feature fusion attention

Abstract

The application discloses a three-dimensional target detection method based on multi-modal voxel image feature fusion attention, and relates to the technical field of automatic driving target detection.The application comprises the following steps: constructing a dual-domain voxel space feature coding module based on a Transformer, respectively performing corresponding feature query coding on voxel domain features and camera domain features, and maximizing the use of different modal feature information to obtain a preselection frame; and constructing a multi-scale deep pseudo-prediction module, effectively coding different modal feature information, and extracting voxel features and image space features, and concatenating the voxel features and the image space features to obtain simultaneous features.The application solves the problem that, when a current multi-modal target detection algorithm performs feature level fusion, different domain features lack accurate weight distribution, matching misplacement occurs, data loss is caused, and finally the target detection precision is reduced.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving target detection technology, specifically a three-dimensional target detection method based on multimodal voxel image feature fusion attention. Background Technology

[0002] In current autonomous driving research, 3D object detection technology has become an indispensable part. Currently, 3D object detection mainly uses two types of sensors: cameras and LiDAR (LiDAR) to acquire information about the vehicle's surrounding environment. These two sensors exhibit drastically different characteristics due to their different measurement processes. RGB images can provide dense visual information about targets, but they are not a good source of depth information. LiDAR points can capture accurate 3D spatial information, but often lack semantic information and have sparse reflective points. The complementary nature of these two sensors makes the fusion of multimodal information a recent research hotspot. Camera-LiDAR sensor fusion aims to combine the two complementary information sources to achieve robust object detection. Therefore, the key is to improve the quality of multimodal feature matching by accurately fusing features from different modalities through accurate correspondences.

[0003] Existing multimodal 3D target detection methods can be divided into two categories: decision-level fusion and feature-level fusion. The former first detects the object using the detection methods of the camera and radar respectively, and then projects the camera feature information onto the laser radar point in the three-dimensional space in the 3D space, combining the image point features with the point laser point features. However, the limitation is that the dense image information and the sparse point cloud features often cannot be effectively aligned, thereby destroying the correlation between different modal data and causing data loss. The feature-level fusion method extracts the corresponding features from the camera image and the laser radar point cloud respectively, and performs alignment fusion in the voxel domain so that the multimodal features are combined into a single representation of the detected object, so that the detector can make full use of the features of different modalities in the early inference stage. However, the existing feature-level fusion methods PointAugmenting[1] and 3D-GAF[2] transform the image features to the voxel domain using the calibration matrix and perform element feature fusion. This fusion scheme often lacks accurate weight allocation for different features, so that the key clues cannot be well highlighted. Therefore, we propose a three-dimensional target detection method based on multimodal voxel image feature fusion attention. Summary of the Invention

[0004] The purpose of this invention is to provide a three-dimensional target detection method based on multimodal voxel image feature fusion attention, which solves the problem that current multimodal target detection algorithms lack accurate weight allocation between features of different domains when performing feature-level fusion, resulting in matching misalignment, data loss, and ultimately reduced target detection accuracy.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a 3D target detection method based on multimodal voxel image feature fusion attention, specifically including the following steps:

[0006] A Transformer-based dual-domain voxel space feature encoding module is constructed to perform corresponding feature query encoding on voxel domain features and camera domain features respectively, so as to make the most of the feature information of different modalities to obtain the preselection box;

[0007] A multi-scale depth prediction module is constructed to effectively encode and extract voxel features and image space features from different modal feature information. The voxel features and image space features are then concatenated to obtain combined features.

[0008] Deep feature prediction based on joint features is used to enhance the depth interaction between features of different modalities;

[0009] A dynamic attention feature aggregation module is constructed, which uses centroids to represent voxel features more accurately, and utilizes learnable dynamic offset and cross-attention mechanisms to align different features across modalities in a fine-grained manner to generate cross-modal features.

[0010] The overall loss function is used to train the overall model constructed in the above steps, and the overall model is used for target detection.

[0011] Furthermore, a dual-domain voxel space feature encoding module is constructed to perform corresponding feature query encoding on voxel domain features and camera domain features, maximizing the use of different modal feature information to obtain preselected boxes. The specific method is as follows:

[0012] (1) Perform dual feature query on voxel domain features and camera domain features.

[0013] (1.1) By performing voxelization on the point cloud features, k non-empty voxels are obtained, and the number k is determined according to the distribution of the point cloud features.

[0014] (1.2) Voxel query Matching with corresponding non-empty voxels improves voxel domain features.

[0015] (1.3) Project the center point position of the non-empty voxel feature to the corresponding position in the camera domain, and query the camera in sequence. This is used to perform corresponding assignments and improve image domain features;

[0016] (2) Perform voxel transformer. The transformer local self-attention layer first identifies the geometric structure features within the neighborhood points, and then encodes the global target feature information through the transformer global self-attention layer, thereby obtaining the complete contextual information from local to global in the point cloud features.

[0017] (3) Perform an image transformer and use a transformer local self-attention layer to construct the positional relationship between local and global image features;

[0018] (4) Perform dual-domain feature fusion on voxel domain features and image domain features, and perform dual-domain feature fusion decoding on multiple attention layers for dual-feature query V'. q and C' q .

[0019] Further, a voxel transformer is performed, specifically as follows:

[0020] (1) The transformer's local self-attention layer queries the input voxel features. The corresponding non-empty voxels are used to obtain the foreground feature set V' through standard sampling operations. q ={V1,V2,…,V N};

[0021] (2) With V i Using a circle as the center, select feature points within a certain radius r and perform local self-attention query operations to assign different feature weights to feature points within the range, thereby obtaining a subset of geometric structures within the exploration domain.

[0022] (3) The obtained geometric structure subset is further grouped and input into the transformer encoding module that aggregates and encodes the feature information. For a given feature set V' q ={V1,V2,…,V N The output is}

[0023]

[0024] In the formula, δ=θ(V N -V i ) is the feature encoding position function, θ is an algorithmic perceptron with multiple linear layers and one fully connected layer, and λ and α are the self-attention parameters and change values, respectively. ψ is the feature point transformation parameter, and ρ is the softmax function.

[0025] Furthermore, we perform an image transformer, as follows:

[0026] (1) Where the transformer encoder is composed of an image query sequence As input, the original image features Reshaping into 2D feature sequences

[0027] In the formula, (H,W) is the resolution of the original image features, C is the number of channels, D is the resolution of the reconstructed features, and N = HW / D is the number of reconstructed features, which is also the effective input sequence length of the image transformer encoder.

[0028] (2) The Transformer uses a constant vector V in its application layer. e The reshaped features are output through a trainable linear projection using formula (2):

[0029]

[0030] The output of the transformer encoder is f i express.

[0031] (3) During pre-training, the feature classification head is implemented by a hidden MLP layer, and during fine-tuning, it is implemented by a single linear layer:

[0032]

[0033] In the formula, MSA represents the multi-head self-attention layer, and Z... * To represent a single-layer feature, MLP is a linear perceptron layer, LN is a fully connected layer, and f i This represents the output of the transformer encoder.

[0034] Furthermore, voxel domain features and image domain features are fused together, as follows:

[0035] (1) Camera features f i Given a key and a value, query C' for the camera. q Perform deformation and transformation, for a given 2D reference point. In camera domain features f i The above applies a deformable mask with adaptive offset and weights, the mask offset Δp m and mask weight Δw m :

[0036] Δp m =PFN(C′) q ),Δw m =PFN(C′) q +V′ q ),

[0037] In the formula, '+' indicates summation of elements, and PFN represents a feedforward network;

[0038] (2) Mask weight Δp m The given offset Δp is determined jointly by voxel queries and camera queries.m and weight Δw m Attention value The calculation is as follows:

[0039]

[0040] In the formula, h is the index of the attention header, k is the index of the sampling key, and K is the total number of sampling keys. and Represents the learnable projection matrix;

[0041] (3) Voxel query V' q And camera query C' q Further fusion and transformation are achieved through a gating fusion mechanism, namely V' q and C' q Blend using the different proportions given below:

[0042] C″ q =C' q +V' q ×σ(Conv1(C' q +V' q (5)

[0043] V″ q =V' q +C' q ×σ(Conv2(C' q +V' q (6)

[0044] In the formula, '×' represents element-wise multiplication, and σ(·) is the sigmoid function. Conv1(·) and Conv2(·) represent different convolutional layers.

[0045] Furthermore, a multi-scale depth prediction module is constructed to effectively encode and extract voxel features and image spatial features from different modal feature information. The voxel features and image spatial features are then concatenated to obtain combined features, as detailed below:

[0046] (1) Perform 3D voxel feature extraction:

[0047] (1.1) Given N 3D index vectors and eigenvectors The input voxels consist of c in Indicates the number of input feature channels;

[0048] (1.2) For each voxel feature x i ∈X, through 3D submanifold convolution kernel K 3D(·) Encode the number of geometric feature voxels and extract features. Specifically, based on the corresponding 3D index vector, calculate the geometric features from the non-empty voxels in the corresponding neighborhood:

[0049]

[0050] In the formula, This represents the relevant neighborhood features obtained by index S. Used to represent nonlinear activation functions;

[0051] (2) Perform 2D image spatial feature extraction:

[0052] (2.1) The 3D index is converted into a set of mesh points G(·) based on the voxelization parameter;

[0053] (2.2) Next, the grid point G(·) is transformed back to the original coordinate system and then data augmentation τ(·) is performed;

[0054] (2.3) Based on the radar calibration, the camera parameters are projected onto the 2D image plane through projection P(·), and the whole can be expressed as:

[0055]

[0056] In the formula, Represents the 2D index quantity for each voxel feature. Feature information is calculated from the corresponding domain based on the corresponding 2D index.

[0057]

[0058] in Represented by index vector The generated domain voxel features, K 2D Represents a 2D sub-flow convolution kernel;

[0059] (3) Concatenate voxel features and image spatial features to obtain combined features:

[0060] (3.1) will and By combining these equations, we can obtain the corresponding feature vectors. Where c out Indicates the number of output feature channels:

[0061]

[0062] Furthermore, deep feature prediction is performed based on the combined features, as follows:

[0063] (1) Set the extracted combined features as P = (ai ,b i ,c i This involves projecting 3D feature points onto a 2D image and recording relevant points within the foreground mask. These points serve as depth reference points to provide corresponding depth information, represented as:

[0064] In the formula u i and v i d represents the pixel coordinates of the point. i It is the acquisition of real, in-depth information.

[0065] (2) Uniformly sample a subset from each instance mask of the 2D image. Each element point in the subset will obtain the true depth information from the nearest reference point as the estimated depth;

[0066] (3) Project the sampled subset of the estimated depth back into 3D space as prediction points;

[0067] (4) Connect the 2D image features with the corresponding predicted sparse depth features of the image, and fuse them with the convolutional layer to obtain the depth-aware semantic features C. d For a predicted point M generated from the same instance mask, the corresponding pixel semantic features will have different effects along the projection ray in 3D space. Therefore, a dynamic weighting factor is calculated for each individual depth to adjust the influence of semantic features. The process for a specific predicted point is described as follows:

[0068]

[0069] In the formula, It is a subset The Mth estimated depth, This represents a dynamic weighting factor used to adjust the prediction points for the corresponding semantic features.

[0070] Furthermore, a dynamic attention feature aggregation module is constructed. By using centroids to represent voxel features more accurately, and by utilizing learnable dynamic offsets and cross-attention mechanisms, fine-grained cross-modal alignment between different features is achieved to generate cross-modal features, as detailed below:

[0071] (1) The input is a set of non-empty voxel features and image features F I S v It is a voxel index, V t It is a non-empty voxel eigenvector, N v The number of non-empty voxels is used to obtain the feature F of each voxel by averaging the spatial positions of all points within the same voxel. v The center of mass Ce :

[0072]

[0073] In the formula, (x i ,y i ,z i ) represents the location coordinate information, |p(V i )| is a voxel feature F v The number of points within;

[0074] Next, the camera projection matrix H is used as the centroid of the voxel to calculate the reference point C. i :

[0075] C i =H·C e (13)

[0076] In the formula, H is the product of the camera's intrinsic and extrinsic parameter matrices;

[0077] (2) Assign a voxel grid index to each voxel point centroid that needs to be calculated. k is the number of 3D mesh points, G is the mesh feature channel, and these are combined through a fully connected layer and a sigmoid layer. The nth... th Each feature generates a pair of centroid grid weights. The weights are applied to the non-empty voxel features and the image features respectively to obtain the weighted voxel grid features. With depth image mesh features

[0078]

[0079] (3) Next, three connection layers are used to convert the voxel grid features into query Q and the depth image grid features into Key and Value. The inner product between each query Q and the key value is performed to obtain the attention affinity matrix between the voxel grid features and the depth image grid features. The value is constant and aggregated through this matrix. Finally, the aggregated fused feature F is... ag Processed through a fully connected layer, and outputting target category and location information via standard 2D-3D cascading, the entire dynamic fusion process can be represented as follows:

[0080]

[0081] In the formula, W m and W' m These are learnable weights, M is the number of self-attention heads, K is the total number of sampling points, and Δp mik and A mik These represent the sampling offset and attention weight of the k-th sampling point in the m-th attention head, respectively.

[0082] Furthermore, the overall loss function includes a classification loss L for object and category. cls The regression loss L used to predict the bounding box reg Prediction loss L used for dual-domain feature fusion dff Depth loss L used for depth feature prediction depth The details are as follows:

[0083] L total =λ cls L cls +λ reg L reg +λ dff L dff +λ dep L dep (17)

[0084] In the formula, λ cls , λ reg , λ dff , λ dep These represent the regularization weight coefficients for classification loss, regression loss, prediction loss, and depth loss, respectively.

[0085] The depth loss for depth feature prediction is handled using focus loss:

[0086]

[0087] In the formula, |ρ| represents the 2D pixel feature region with an effective depth label. It is the accurate true value of the lidar.

[0088] According to a second aspect of the present invention, the present invention provides a three-dimensional target detection system based on multimodal voxel image feature fusion attention, comprising:

[0089] The first construction unit is used to construct a Transformer-based dual-domain voxel space feature encoding module, which performs corresponding feature query encoding on voxel domain features and camera domain features respectively, so as to make the most of different modal feature information to obtain the preselection box;

[0090] The second building unit is used to build a multi-scale depth prediction module, which effectively encodes and extracts voxel features and image space features from different modal feature information, and concatenates voxel features and image space features to obtain combined features.

[0091] The deep feature prediction unit is used to predict deep features based on combined features, thereby enhancing the degree of deep interaction between features of different modalities.

[0092] The third building unit is used to build a dynamic attention feature aggregation module. It uses centroids to represent voxel features more accurately and utilizes learnable dynamic offset and cross-attention mechanisms to align different features across modalities in a fine-grained manner, generating cross-modal features.

[0093] The training output unit is used to train the overall model constructed in the above steps using the overall loss function, and to perform object detection using the overall model.

[0094] The present invention has at least the following beneficial effects:

[0095] (1) In view of the problem of lack of accurate weight allocation between different domain features, the present invention designs a dual-domain voxel space feature encoding module based on Transformer. By performing corresponding feature query encoding on the corresponding different domain features, the preselection box is obtained by maximizing the use of different modal feature information.

[0096] (2) In order to further improve the interaction quality between features of different domains and reduce matching errors, this invention designs a multi-scale depth prediction module with depth perception function. This module enhances the interaction depth of point features at different scales, thereby further refining the features, enhancing the target detection capability, and improving the positioning accuracy of small targets at long distances.

[0097] (3) In order to more accurately fuse features from different domains, the present invention designs a dynamic attention feature aggregation module. This module uses centroids to more accurately represent voxel features and utilizes learnable dynamic offset and cross-attention mechanisms to fine-grained cross-modal alignment between different features to generate cross-modal features.

[0098] Of course, any product implementing this invention does not necessarily need to achieve all of the advantages described above at the same time. Attached Figure Description

[0099] Figure 1 This is a schematic diagram of the overall framework of the method described in this invention;

[0100] Figure 2 This is a schematic diagram of the voxel transformer framework in this invention;

[0101] Figure 3 This is a schematic diagram of the image transformer framework in this invention;

[0102] Figure 4 This is a schematic diagram of the process for extracting voxel features and image spatial features in this invention;

[0103] Figure 5 This is a schematic diagram of the framework of the dynamic attention feature aggregation module of the present invention;

[0104] Figure 6 This is an example of qualitative results from the KITTI dataset in Embodiment 1 of the present invention;

[0105] Figure 7 This is a schematic diagram comparing the precision-recall curves of vehicle detection results on the KITTI dataset in Embodiment 1 of the present invention.

[0106] Figure 8 This is an example of qualitative results on the NuScenses dataset from Embodiment 1 of the present invention;

[0107] Figure 9 This is a schematic diagram of a real vehicle platform used for actual testing in Embodiment 1 of the present invention;

[0108] Figure 10 This is a schematic diagram comparing the average accuracy of different object modules between the baseline and the method in Embodiment 1 of the present invention;

[0109] Figure 11 This is a schematic diagram illustrating the average accuracy under different standards in Embodiment 1 of the present invention;

[0110] Figure 12 This is a schematic diagram comparing the number of decoder layers on the KITTI verification set in Embodiment 1 of the present invention. Detailed Implementation

[0111] The technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0112] Currently, there are 3D object detection methods based on single-modality, multi-modality, and Transformer:

[0113] The current single-modal target detection algorithms are mainly based on lidar data or RGB images. Voxel-RCNN[5] converts the original disordered point cloud into a more structured data format through voxelization. Fan[6] et al. use the range view method to extract features from the disordered point cloud. CenterPoint[7] uses the center point mechanism to replace the original anchor frame for three-dimensional target detection. LiDAR-RCNN[8] proposes to use boundary offset and virtual point methods to solve the size ambiguity problem. SPG[9] fuses the original point cloud with the generated semantic point cloud to obtain enhanced point cloud data to improve the target detection capability. PyramidR-CNN

[10] designs a pyramid ROI head to expand the original ROI region and extract features to solve the problem of sparse and uneven distribution of point clouds in the region of interest. CT3D

[11] uses the channel attention mechanism to process 3D feature point cloud data and adaptively weights different features, thereby enhancing the representation capability of point cloud data. IA-SSD

[12] uses two learnable, instance-aware downsampling strategies to select foreground points in a hierarchical manner, thereby improving detection efficiency. Chen

[13] et al. proposed to construct pseudo-stereo images by means of image-level generation and feature-level generation, thereby improving the effect of monocular image detection. The lidar point cloud data of these algorithms are relatively sparse, or the image data lacks fine-grained information such as depth, which inevitably affects the target detection effect.

[0114] Multimodal 3D object detection

[0115] LiDAR and cameras can provide effective complementary information around the target, so fusing multimodal inputs to achieve 3D target detection is attractive. FusionPainting

[14] projects semantic segmentation masks onto LiDAR points to improve detection accuracy. VFF

[15] proposed a method called voxel field fusion for cross-modal 3D detection, which enhances contextual interaction in three-dimensional space while ensuring feature space representation. PointAugmenting

[16] effectively fuses features of different modalities through cross-modal data augmentation. 4D-Net

[17] uses a new dynamic connection learning method to better utilize motion cues and dense image information to detect distant targets while maintaining computational efficiency. 3D-GAF

[18] proposes to use a depth completion method to generate pseudo point clouds and use a parallel mesh fusion method for modal fusion and data augmentation. AutoAlignV2

[19] enables the model to reason in a more dynamic way and accelerate the aggregation of features of different modalities through a depth-aware data augmentation strategy and an image-level training scheme. CenterFusion

[20] uses a frustum-based association method to accurately associate radar detection points with targets in images. LIFT

[21] learns to align input 4D sequences across sensor data, thereby effectively utilizing multimodal data features and improving the efficiency of multimodal 3D target detection. However, the above target detection algorithms generally lack reasonable feature weights for different modal features.

[0116] Transformer-based 3D object detection

[0117] DETR

[22] is the pioneering work of applying Transformer to image object detection. DETR simplifies the detection pipeline, treating object detection as a direct set prediction problem, effectively eliminating many hand-designed components. However, the randomly initialized object query requires decoder cascade to refine the target features, which leads to DETR usually taking an extremely long time to converge during training. DeepFusion

[23] proposed two new technologies, InverseAug and LearnableAlign. InverseAug is an inverse geometric correlation enhancement technique to achieve accurate geometric alignment between LiDAR points and pixel images. LearnableAlign uses cross-attention to dynamically capture the correlation of different modal features. TransFusion designed a detection framework with a soft correlation mechanism, enabling the model to adaptively determine the information and location of target features. Monodtr

[24] introduces a depth-aware feature enhancement module, which uses assisted deep learning to enhance depth-aware features and avoids obtaining inaccurate depth prior information from pre-trained depth estimation. The latest Transformer-based methods

[25] -

[31] solve the DETR problem by designing different modules, but the current Transformer-based methods do not make good use of features from different domains.

[0118] Example 1:

[0119] Please see Figure 1-12 This invention provides a technical solution: a 3D target detection method based on multimodal voxel image feature fusion attention, specifically including the following steps:

[0120] S1. Construct a Transformer-based dual-domain voxel space feature encoding module. Employ a deformable transformer-decoder approach and utilize a deformable attention mechanism to perform corresponding feature lookup encoding on voxel domain features and camera domain features respectively, and then fuse the dual-domain features to aggregate radar and camera dual-domain feature information to generate initial candidate boxes, as detailed below:

[0121] (1) Dual-feature query: Dual-feature query refers to obtaining more accurate features by using both camera feature query and voxel feature query.

[0122] (1.1) By performing voxelization on the point cloud features, k non-empty voxels are obtained, and the number k is determined according to the distribution of the point cloud features.

[0123] (1.2) Voxel query Matching with corresponding non-empty voxels improves voxel domain features.

[0124] (1.3) Project the center point position of the non-empty voxel feature to the corresponding position in the camera domain, and query the camera in sequence. This is used to perform corresponding assignments and improve image domain features;

[0125] (2) Perform a voxel transformer. The transformer's local self-attention layer first identifies the geometric structure features within the neighborhood points, and then encodes the global target feature information through the transformer's global self-attention layer. This yields complete contextual information from local to global in the point cloud features, as detailed below:

[0126] (2.1) The transformer's local self-attention layer queries the input voxel features. The corresponding non-empty voxels are used to obtain the foreground feature set V' through standard sampling operations. q ={V1,V2,…,V N};

[0127] (2.2) with V i Using a circle as the center, local self-attention query operations are performed on feature points within a certain radius r, thereby assigning different feature weights to feature points within the range, thus obtaining a subset of geometric structures within the exploration domain and simplifying the computational burden.

[0128] (2.3) The obtained geometric structure subset is further grouped and input into the transformer encoding module that aggregates and encodes the feature information. For a given feature set V' q ={V1,V2,…,V N The output is}

[0129]

[0130] In the formula, δ=θ(V N -V i ) is the feature encoding position function, θ is an algorithmic perceptron with multiple linear layers and one fully connected layer, and γ and α are the self-attention parameters and change values, respectively. ψ is the feature point transformation parameter, and ρ is the softmax function;

[0131] (3) Perform an image transformer, using a transformer local self-attention layer to construct the positional relationship between local and global image features. A schematic diagram of the image transformer is shown below. Figure 3 As shown:

[0132] (3.1) where the transformer encoder is composed of an image query sequence As input, the original image features Reshaping into 2D feature sequences

[0133] In the formula, (H,W) is the resolution of the original image features, C is the number of channels, D is the resolution of the reconstructed features, and N = HW / D is the number of reconstructed features, which is also the effective input sequence length of the image transformer encoder.

[0134] (3.2) The Transformer uses a constant vector V in its application layer. e The reshaped features are output through a trainable linear projection using formula (2):

[0135]

[0136] The output of the transformer encoder is f i express;

[0137] (3.3) During pre-training, the feature classification head is implemented by a hidden MLP layer, and during fine-tuning, it is implemented by a single linear layer:

[0138]

[0139] In the formula, MSA represents the multi-head self-attention layer, and Z... * To represent a single-layer feature, MLP is a linear perceptron layer, LN is a fully connected layer, and f i This represents the output of the transformer encoder;

[0140] (4) Perform dual-domain feature fusion on voxel domain features and image domain features, and perform dual-domain feature fusion decoding on multiple attention layers for dual feature queries:

[0141] (4.1) Camera features f i Given a key and a value, query C' for the camera. q Perform deformation and transformation, for a given 2D reference point. In camera domain features f i The above applies a deformable mask with adaptive offset and weights, the mask offset Δp m and mask weight Δw m :

[0142] Δp m =PFN(C′) q ),Δw m =PFN(C′) q +V′ q ),

[0143] In the formula, '+' indicates summation of elements, and PFN represents a feedforward network;

[0144] (4.2) Mask weight Δp m Determined jointly by voxel queries and camera queries, the feature fusion effect can be improved by using attention weights based on voxel and camera region features, given an offset Δp. m and weight Δw m Attention value The calculation is as follows:

[0145]

[0146] In the formula, h is the index of the attention header, k is the index of the sampling key, and K is the total number of sampling keys. and Represents the learnable projection matrix;

[0147] (4.3) Voxel query V' q And camera query C' q Further fusion and transformation are achieved through a gating fusion mechanism, namely V' q and C' q Blend using the different proportions given below:

[0148] C″ q =C' q +V' q ×σ(Conv1(C' q +V' q (5)

[0149] V″ q =V' q +C' q ×σ(Conv2(C' q +V' q (6)

[0150] In the formula, '×' represents element-wise multiplication, and σ(·) is the sigmoid function. Conv1(·) and Conv2(·) represent different convolutional layers. Since the ratio is a function of the sum, the combined ratio will be adaptively adjusted according to the input features.

[0151] S2. Construct a multi-scale depth prediction module to effectively encode and extract voxel features and image spatial features from different modal feature information. Concatenate the voxel features and image spatial features to obtain joint features, which are used to enhance the interaction depth between different modal features, thereby enhancing the information features.

[0152] (1) Perform 3D voxel feature extraction:

[0153] (1.1) Given N 3D index vectors and eigenvectors The input voxels consist of c in Indicates the number of input feature channels;

[0154] (1.2) For each voxel feature x i ∈X, through 3D submanifold convolution kernel K 3D (·) Encode the number of geometric feature voxels and extract features. Specifically, based on the corresponding 3D index vector, calculate the geometric features from the non-empty voxels in the corresponding neighborhood:

[0155]

[0156] In the formula, This represents the relevant neighborhood features obtained by index S. Used to represent nonlinear activation functions;

[0157] (2) Since the semantic information of the features is mainly distributed on the boundary of the 2D target, the error interference caused by the lack of depth information will affect the detection performance. Therefore, we extend the receptive field of the convolution to the 2D image space and use 2D adjacent voxels to encode the perceptual semantic feature information to perform 2D image space feature extraction, as follows:

[0158] (2.1) The 3D index is converted into a set of mesh points G(·) based on the voxelization parameter;

[0159] (2.2) Next, the grid point G(·) is transformed back to the original coordinate system and then data augmentation τ(·) is performed;

[0160] (2.3) Based on the radar calibration, the camera parameters are projected onto the 2D image plane through projection P(·), and the whole can be expressed as:

[0161]

[0162] In the formula, Represents the 2D index quantity for each voxel feature. Feature information is calculated from the corresponding domain based on the corresponding 2D index.

[0163]

[0164] in Represented by index vector The generated domain voxel features, K 2D Represents a 2D sub-flow convolution kernel;

[0165] If there are multiple features within a single 2D neighboring voxel feature, max pooling is performed to retain the most important features before 2D convolution. By expanding the receptive field to the 2D image space and using 2D neighboring voxels to encode perceptual feature information, features of different modalities can be effectively encoded and extracted. This reduces background error while effectively utilizing foreground features. The feature extraction flowchart is as follows: Figure 4 As shown;

[0166] (3) Concatenate voxel features and image spatial features to obtain combined features:

[0167] (3.1) will and By combining these equations, we can obtain the corresponding feature vectors. Where c out Indicates the number of output feature channels:

[0168]

[0169] S3. Perform deep feature prediction based on joint features to enhance the depth interaction between features of different modalities:

[0170] The LiDAR and camera features extracted in the previous stage focus on geometric and semantic features, respectively. Next, the depth feature prediction module predicts depth feature information to improve the interaction quality between the two, thereby enabling more efficient multimodal feature fusion, as detailed below:

[0171] (1) Set the extracted combined features as P = (a i ,b i ,c i This involves projecting 3D feature points onto a 2D image and recording relevant points within the foreground mask. These points serve as depth reference points to provide corresponding depth information, represented as:

[0172] In the formula u i and v i d represents the pixel coordinates of the point. i It is the acquisition of real, in-depth information.

[0173] (2) Uniformly sample a subset from each instance mask of the 2D image. Each element point in the subset will obtain the true depth information from the nearest reference point as the estimated depth;

[0174] (3) Project the sampled subset of the estimated depth back into 3D space as prediction points;

[0175] (4) The above-mentioned predicted points are further processed to utilize the semantic information encapsulated in the image features. Unlike simple decorative processing, depth-aware semantic features are generated by pre-corresponding interaction between 2D image features and reference points from depth prediction. These depth-aware semantic features and their corresponding sampling subsets are used to adaptively modify the predicted points. Specifically:

[0176] The 2D image features are concatenated with the corresponding predicted sparse depth features, and then fused with a convolutional layer to obtain the depth-aware semantic features C. d For a predicted point M generated from the same instance mask, the corresponding pixel semantic features will have different effects along the projection ray in 3D space. Therefore, a dynamic weighting factor is calculated for each individual depth to adjust the influence of semantic features. The process for a specific predicted point is described as follows:

[0177]

[0178] In the formula, It is a subset The Mth estimated depth, This represents a dynamic weighting factor used to adjust the prediction points for the corresponding semantic features.

[0179] By predicting depth information, the degree of deep interaction between different modal features is enhanced, the domain gap between different modal data is reduced, and prediction points with dense depth features and semantic features are obtained, thereby enhancing the representation of feature information.

[0180] S4. Construct a dynamic attention feature aggregation module, which uses centroids to more accurately represent voxel features and utilizes learnable dynamic offsets and cross-attention mechanisms to fine-grained cross-modal alignment between different features, generating cross-modal features:

[0181] Regarding the technical solution of this embodiment, this embodiment utilizes a centroid dynamic fusion mechanism. By using the centroid of voxel points to represent the spatial location of target features, the problem of voxel centers ignoring the actual distribution of voxel points can be avoided. The centroid of voxel points is closer to the object scanning surface than the voxel center, and can provide original geometric shape information and scale more effectively.

[0182] This embodiment also employs a grid point dynamic fusion mechanism, which uses a 3D gridding module to crop feature information of different modalities. Compared with the original strategy of using 2D regions of interest to extract target features, it can reduce the interference of background point noise, thereby effectively utilizing semantic information and point cloud depth information in the image to improve target representation and enhance detection capabilities.

[0183] (1) The input is a set of non-empty voxel features and image features F I S v It is a voxel index, V t It is a non-empty voxel eigenvector, N v The number of non-empty voxels is used to obtain the feature F of each voxel by averaging the spatial positions of all points within the same voxel. v The center of mass C e :

[0184]

[0185] In the formula, (x i ,y i ,z i ) represents the location coordinate information, |p(V i )| is a voxel feature F v The number of points within;

[0186] Next, the camera projection matrix H is used as the centroid of the voxel to calculate the reference point C. i :

[0187] C i =H·C e (13)

[0188] In the formula, H is the product of the camera's intrinsic and extrinsic parameter matrices;

[0189] (2) Assign a voxel grid index to each voxel point centroid that needs to be calculated. k is the number of 3D mesh points, G is the mesh feature channel, and these are combined through a fully connected layer and a sigmoid layer. The nth... th Each feature generates a pair of centroid grid weights. The weights are applied to the non-empty voxel features and the image features respectively to obtain the weighted voxel grid features. With depth image mesh features

[0190]

[0191] (3) Next, three connection layers are used to convert the voxel grid features into query Q and the depth image grid features into Key and Value. The inner product between each query Q and the key value is performed to obtain the attention affinity matrix between the voxel grid features and the depth image grid features. The value is constant and aggregated through this matrix. Finally, the aggregated fused feature F is... ag Processed through a fully connected layer, and outputting target category and location information via standard 2D-3D cascading, the entire dynamic fusion process can be represented as follows:

[0192]

[0193] In the formula, W m and W' m These are learnable weights, M is the number of self-attention heads, K is the total number of sampling points, and Δp mik and A mik These represent the sampling offset and attention weight of the k-th sampling point in the m-th attention head, respectively, both of which are obtained through linear projection onto the query features;

[0194] S5. Train the overall model constructed in the above steps using the overall loss function, and then use the overall model for object detection:

[0195] The overall loss function includes classification loss L for object and category. cls The regression loss L used to predict the bounding box reg Prediction loss L used for dual-domain feature fusion dff Depth loss L used for depth feature prediction depth The overall loss function is as follows:

[0196] L total =λ cls L cls +λ reg L reg +λ dff L dff +λ dep L dep (17)

[0197] Where, λ cls , λ reg , λ dff , λ dep These represent the regularization weight coefficients for classification loss, regression loss, prediction loss, and depth loss, respectively.

[0198] To address the imbalance between positive and negative samples, focus loss is used to balance the samples for classification and prediction tasks respectively, and smoothed-L1 is used to balance the regression loss. Focus loss is also used to handle the depth loss for deep feature prediction.

[0199]

[0200] Where |ρ| is a 2D pixel feature region with an effective depth label. It is the accurate true value of the lidar.

[0201] The technical solution of the present invention will be further described in detail below with reference to specific embodiments:

[0202] Data sets and metrics

[0203] This embodiment evaluates and tests the proposed object detection method on the challenging KITTI 3D

[32] and NuScenes

[33] object detection datasets and the latest open dataset Waymo

[34] .

[0204] 1. Experimental verification

[0205] 1.1 Datasets and Metrics

[0206] This embodiment evaluates and tests the above method on the challenging KITTI 3D

[32] and NuScenes

[33] object detection datasets and the latest open dataset Waymo

[34] .

[0207] 1.2 Experimental results on the KITTI dataset

[0208] Table 1 compares the detection performance of the method in this embodiment with other target detection methods in the KITTI test set for the car category. As can be seen from Table 1, the overall detection performance of the method in this embodiment is superior to other current 3D target detection methods. Compared with the baseline CenterPoint[7] of the method in this embodiment, the average detection accuracy AP at different levels of the method in this embodiment is significantly higher. 3D The average accuracy (AP) was improved by 1.67 / 1.27 / 0.78 respectively. Compared with the high-performance SE-SSD

[41] , the average accuracy (AP) was improved on medium and hard difficulty. 3D They also increased by 0.08 and 0.66 respectively.

[0209] Table 2 shows the comparison results of the detection performance of the method in this embodiment on the KITTI test set for pedestrians and bicycles. As shown in Table 2, compared with the baseline CenterPoint[7] of the method in this embodiment, the average accuracy of the method in this embodiment for pedestrian targets improved by 0.61 / 0.07 / 0.06 at different difficulty levels, while the detection results for bicycle targets improved by 1.19 / 1.83 / 1.14 at different difficulty levels compared with the baseline. Table 3 shows the comparison results of the detection performance of the method in this embodiment on the KITTI validation set for cars. Compared with the baseline, the average accuracy improved by 0.2 / 0.58 / 1.12 at different difficulty levels. The detection diagram of the method in this embodiment on the KITTI dataset is shown below. Figure 6 As shown.

[0210] Table 1. Detection performance of car categories in the KITTI test set.

[0211]

[0212]

[0213] Table 2 shows the detection performance of pedestrians and bicycles in the KITTI test set.

[0214]

[0215] Table 3 shows the detection performance of the car category on the KITTI validation set.

[0216]

[0217] A comparison of precision-recall curves for vehicle detection results in the KITTI dataset, such as... Figure 7 As shown, compared with other target detection methods, the method in this embodiment still has high accuracy when the recall rate is high, and the method in this embodiment has good robustness and stability overall.

[0218] 1.3 Experimental results on the NuScences dataset

[0219] In order to further test the effectiveness and generality of the above method, the method was verified to have good detection performance on three-dimensional targets. The method was verified on the more challenging NuScences dataset. As shown in Table 4, the detection results of the method in this embodiment and other 3D target detection methods on NuScences for different categories of targets are compared. As can be seen from the table, the method improved by 0.1 / 1.3 compared with the best VFF

[51] on the important indicators NDS and mAP in the NuScences dataset. It can be seen that the proposed method still has good robustness in detection on different datasets. The detection results of the above method on the NuScences dataset in this embodiment are shown in the figure below. Figure 8 As shown.

[0220] Table 4 Performance comparison on the NuScenes dataset

[0221]

[0222]

[0223] 1.4 Experimental results on the Waymo dataset

[0224] This embodiment also compares the proposed method with the baseline CenterPoint[7] on the Waymo dataset. The results are shown in Table 5. The results show that the method of this embodiment performs well in different levels of tests on the Waymo dataset. Compared with CenterPoint[7], the mAP / mAPH on L1 / L2 is improved by 1.45 / 1.35 and 1.48 / 1.12 respectively. The improvement is even greater when detecting smaller targets such as pedestrians and bicycles, showing that the method of this embodiment has excellent performance for detecting small targets at a distance.

[0225] Table 5 shows the performance comparison on the Waymo dataset.

[0226]

[0227] 1.5 Experiments on a real vehicle platform

[0228] Considering that simulation experiments on datasets alone cannot prove the performance of TFIENet in real-world scenarios, this embodiment conducted a series of tests on a real vehicle platform. For example... Figure 9 As shown, in addition to the basic 32-line radar, the platform also incorporates an additional Tele-15 radar and millimeter-wave radar. The millimeter-wave radar is primarily responsible for determining the speed and distance of vehicles ahead; the Tele-15 radar serves as an additional safety redundancy to enhance vehicle safety; and the 32-line radar is responsible for short-range, wide-angle detection tasks.

[0229] To more clearly demonstrate the effectiveness of each module of the above method, the average accuracy of different target detection objects is compared between the baseline and the method of this embodiment, for example... Figure 10 As shown in Table 6, the detection results for different vehicle targets on the real-world platform are compared. Table 6 demonstrates that, in complex and ever-changing real-world environments, the addition of various modules to the method in this embodiment can effectively improve the overall performance of the model.

[0230] Table 6 Performance of vehicle categories on the real-world platform

[0231]

[0232] 1.6 Ablation Experiment

[0233] This section verifies the effectiveness of each component of the above method through numerous ablation experiments.

[0234] The effectiveness of each module is shown in Table 7. Experiment (a) is the baseline modified on CenterPoint[7] in this embodiment. The average accuracy results of 40 car recall locations calculated on the KITTI validation set are compared. As shown in Table 7, each module component can effectively help improve the accuracy of target detection results at different difficulty levels.

[0235] Table 7 shows the impact of different parts on the KITTI validation set.

[0236] The results were evaluated using APs calculated from 40 automotive recall locations.

[0237]

[0238] The dual-domain voxel space feature encoding module encodes corresponding features from different domains, fuses image and LiDAR features, and maximizes the use of different modal feature information to obtain pre-selected boxes. Through effective dynamic encoding between features, accurate feature weights are assigned to features from different domains, thereby effectively aggregating feature information from different domains. To verify the effectiveness of the dual-domain voxel space feature encoding module, this embodiment divides the module into the following parts: Dual Feature Query (DDA), Dual-Domain Transform Self-Attention Layer (DD-SAL), and Dual-Domain Feature Fusion Layer (DDFF), as shown in Table 8. Through different components of the module, DDA, DD-SAL, and DDFF provide gains of 0.12, 0.29, and 0.39, respectively.

[0239] Table 8 Validity of DOFC Modules

[0240]

[0241] Figure 11 The average 3D detection accuracy of the centerpoint was compared with that of the method in this embodiment under different 3D IoU standards and different distances. Figure 11 Constructed from real data collected by actual platforms, from Figure 11 It can be observed that the method proposed in this embodiment outperforms centerpoint at the same distance and with the same 3D IoU. Specifically, at a distance of 15m and IoU = 0.65, the performance improvement is 10.13%. Compared with other state-of-the-art methods, the method in this embodiment can output high-quality predictions with higher accuracy. As the object distance increases, the number of false alarms increases, but there are fewer false alarms compared to small-scale models. At medium to long distances, the method proposed in this embodiment can achieve an average accuracy of 68.1%.

[0242] To verify the improvement in effectiveness of the dual-query mechanism compared to the single-camera query mechanism, the variable mask offset and weights in the single-camera query mechanism are determined solely by a single camera query. In contrast, the dual-query mechanism updates and determines these values ​​jointly through feature queries from different domains, thereby obtaining more accurate variable mask offsets and feature weights. As shown in Table 9, on the KITTI validation set, the average detection accuracy of the dual-query mechanism is improved by 0.21 and 0.16 compared to the baseline and single-camera query, respectively.

[0243] Table 9 compares different query types on the KITTI validation set.

[0244]

[0245] DOFC module behavior analysis: Figure 12The performance trend of the KITTI validation set with respect to the number of decoder layers is presented. In the KITTI dataset, the performance gradually improves as the number of layers increases to 4, which indicates that the dual-query attention in this embodiment gradually refines the feature quality for 3D object detection.

[0246] The effectiveness of each component of the Dynamic Attention Feature Aggregation Module (DAFG) was verified, as shown in Table 10. Experiment (A) correlated camera features with voxel features using a naive fusion method and fused the voxel features of the LiDAR using simple summation. Experiments (B) and (C) successively added a centroid dynamic fusion mechanism and a 3D mesh fusion mechanism. The results showed that each component of the DAFG played an improvement role. Table 10 Ablation study of the multimodal feature cross-attention module. Results were evaluated using AP calculated at 40 recall locations.

[0247]

[0248] By evaluating the proposed method at different distance levels and occlusion levels, it can be determined under what conditions the proposed method improves the baseline the most. As shown in Table 11, object detection affected by different distances and occlusions shows a significant improvement. This is because the multi-scale depth prediction module enhances the interaction depth of point features at different scales, thereby refining the features.

[0249] Table 11 shows the performance at different distances and degrees of occlusion. The evaluation results are at a moderate level compared to 3D AP calculations for 40 recall locations for the vehicle class.

[0250]

[0251] In summary, this embodiment addresses the problem of inaccurate weight allocation among features in different domains in multimodal object detection methods. It proposes a Transformer-based dual-domain voxel space feature encoding module, which encodes features from different domains using corresponding feature queries and assigns appropriate feature weights to feature information, maximizing the utilization of feature information from different modalities. Secondly, a multi-scale depth prediction module with depth perception is designed to enhance the interaction depth of point features at different scales, thereby further refining features and reducing data loss. Finally, to more accurately fuse features from different domains, a dynamic attention feature aggregation module is designed. This module uses centroids to more accurately represent voxel features and utilizes learnable dynamic offsets and cross-attention mechanisms to fine-grained cross-modal alignment between different features, ultimately generating cross-modal features. Experimental results on the KITTI, NUSCENES, and WAYMO datasets demonstrate the effectiveness and universality of the proposed method. Extensive ablation experiments prove the effectiveness and efficiency of each module. Experimental results on a real-vehicle platform show that the method has good robustness in complex real-world environments.

[0252] Example 2:

[0253] This invention proposes a 3D target detection system based on multimodal voxel image feature fusion attention, comprising:

[0254] The first construction unit is used to construct a Transformer-based dual-domain voxel space feature encoding module, which performs corresponding feature query encoding on voxel domain features and camera domain features respectively, so as to make the most of different modal feature information to obtain the preselection box;

[0255] The second building unit is used to build a multi-scale depth prediction module, which effectively encodes and extracts voxel features and image space features from different modal feature information, and concatenates voxel features and image space features to obtain combined features.

[0256] The deep feature prediction unit is used to predict deep features based on combined features, thereby enhancing the degree of deep interaction between features of different modalities.

[0257] The third building unit is used to build a dynamic attention feature aggregation module. It uses centroids to represent voxel features more accurately and utilizes learnable dynamic offset and cross-attention mechanisms to align different features across modalities in a fine-grained manner, generating cross-modal features.

[0258] The training output unit is used to train the overall model constructed in the above steps using the overall loss function, and to perform object detection using the overall model.

[0259] Specifically, the first construction unit, the second construction unit, the depth feature prediction unit, and the training output unit can be embedded into a computer processing system. The computer, based on the 3D target detection method based on multimodal voxel image feature fusion attention provided above, calls each of the above units to complete the task of equivalent modeling. The first construction unit, the second construction unit, the depth feature prediction unit, and the training output unit can perform operations according to the specific steps given by the 3D target detection method based on multimodal voxel image feature fusion attention.

[0260] It should be noted that the division of the various units in the above system is merely a division of logical functions. In actual implementation, they can be fully or partially integrated into a single physical entity, or they can be physically separated. These units can be implemented entirely in software via processing element calls; they can be fully implemented in hardware; or some units can be implemented by processing element calls to software, while others are implemented in hardware. For example, the receiving unit can be a separate processing element, or it can be integrated into a chip in the aforementioned device. Alternatively, it can be stored as program code in the memory of the aforementioned device, and its signal processing unit functions can be called and executed by a processing element of the device. The implementation of other units is similar. Furthermore, these units can be fully or partially integrated together, or they can be implemented independently. The processing element mentioned here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each of the above units can be completed through integrated logic circuits in the hardware of the processor element or through software instructions.

[0261] For example, these units can be one or more integrated circuits configured to implement the above methods, such as one or more Application Specific Integrated Circuits (ASICs), one or more Digital Signal Processors (DSPs), or one or more Field Programmable Gate Arrays (FPGAs). As another example, when one of the above units is implemented in the form of processing element scheduler code, the processing element can be a general-purpose processor, such as a Central Processing Unit (CPU) or other processor capable of calling program code. Furthermore, these units can be integrated together to implement a system-on-a-chip (SOC).

[0262] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0263] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A three-dimensional object detection method based on multi-modal voxel image feature fusion attention, characterized in that, include; A Transformer-based dual-domain voxel space feature encoding module is constructed to perform corresponding feature query encoding on voxel domain features and camera domain features respectively, so as to make the most of the feature information of different modalities to obtain the preselection box; A multi-scale depth prediction module is constructed to effectively encode and extract voxel features and image space features from different modal feature information. The voxel features and image space features are then concatenated to obtain combined features. Deep feature prediction based on joint features is used to enhance the depth interaction between features of different modalities; A dynamic attention feature aggregation module is constructed, which uses centroids to represent voxel features more accurately, and utilizes learnable dynamic offset and cross-attention mechanisms to align different features across modalities with fine granularity, generating cross-modal features, as detailed below: Input is a set of non-empty voxel features and image features where is the voxel index, is the non-empty voxel feature vector, is the number of non-empty voxels, each voxel feature is obtained by averaging the spatial locations of all points within the same voxel the centroid of : (12) wherein is position coordinate information, is a point number within a voxel feature characteristic Next, use the camera projection matrix. Reference point for calculating the centroid of a voxel. : (13) In the formula, It is the product of the camera's intrinsic and extrinsic parameter matrices; Assign a voxel grid index to the centroid of each voxel point that needs to be computed. , G is the number of 3D mesh points, and G is the mesh feature channel. These are combined and passed through a fully connected layer and a sigmoid layer. Each feature generates a pair of centroid grid weights. The weights are applied to the non-empty voxel features and the image features respectively to obtain the weighted voxel grid features. With depth image mesh features : (14) (15) Next, three connection layers are used to transform the voxel grid features into queries. Transform the mesh features of the depth image into and For each query With key value By performing an inner product between the voxel mesh features and the depth image mesh features, an attention affinity matrix is ​​obtained. This matrix is ​​then used to constant and aggregate the values. Finally, the aggregated fusion features Processed through a fully connected layer, using standard... The cascaded output of target category and location information, the entire dynamic fusion process can be represented as: (16) In the formula, and These are the learnable weights, M is the number of self-attention heads, and K is the total number of sampling points. This represents the attention weight of the k-th sampling point in the m-th attention head; The overall loss function is used to train the overall model constructed in the above steps, and the overall model is used for target detection.

2. The three-dimensional target detection method based on multimodal voxel image feature fusion attention as described in claim 1, characterized in that, A dual-domain voxel space feature encoding module is constructed to perform corresponding feature query encoding on voxel domain features and camera domain features, so as to maximize the use of feature information from different modalities to obtain the preselection box. The specific method is as follows: (1) Perform dual feature query on voxel domain features and camera domain features. (1.1) By performing voxelization on the point cloud features, the following can be obtained: Number of non-empty voxels It is determined based on the distribution of point cloud features; (1.2) Voxel query Matching with corresponding non-empty voxels improves voxel domain features. (1.3) Project the center point position of the non-empty voxel feature to the corresponding position in the camera domain, and query the camera in sequence. This is used to perform corresponding assignments and improve image domain features; (2) Perform voxel transformer. The transformer local self-attention layer first identifies the geometric structure features within the neighborhood points, and then encodes the global target feature information through the transformer global self-attention layer, thereby obtaining the complete context information from local to global in the point cloud features. (3) Perform image transformer, and use transformer local self-attention layer to construct the positional relationship between local and global image features; (4) Perform dual-domain feature fusion on voxel domain features and image domain features, and perform dual-domain feature fusion decoding on multiple attention layers for dual-feature query. and .

3. The three-dimensional target detection method based on multimodal voxel image feature fusion attention according to claim 2, characterized in that, The specific method for performing a voxel transformer is as follows: (1) The transformer local self-attention layer queries the input voxel features The corresponding non-empty voxels are used to obtain the foreground feature set through standard sampling operations. ; (2) with Using a circle as the center, select feature points within a certain radius r and perform local self-attention query operations to assign different feature weights to feature points within the range, thereby obtaining a subset of geometric structures within the exploration domain. (3) The obtained geometric structure subset is further grouped and input into the transformer encoding module that aggregates and encodes the feature information. For a given feature set The output is (1) In the formula, It is a feature encoding position function. It is an algorithmic perceptron with multiple linear layers and one fully connected layer. , Separate self-attention parameters and changes, , These are the feature point transformation parameters. yes function.

4. The three-dimensional target detection method based on multimodal voxel image feature fusion attention as described in claim 2, characterized in that, To perform an image transformer, the specific method is as follows: (1) Where the transformer encoder is composed of an image query sequence As input, the original image features Reshaping into 2D feature sequences ; In the formula It is the resolution of the original image features. Where D is the number of channels, and D is the reconstructed feature resolution. It represents the number of reshaped features and also the effective input sequence length of the image transformer encoder; (2) Transformer uses constant vectors in its application layer The reshaped features are output through a trainable linear projection using formula (2): (2) The output of the transformer encoder is used for express; (3) During pre-training, the feature classification head is implemented by a hidden MLP layer, and during fine-tuning, it is implemented by a single linear layer: (3) In the formula, This indicates the multi-head self-attention layer. To represent single-layer features, MLP stands for a linear perceptron layer, and LN stands for a fully connected layer. This represents the output of the transformer encoder.

5. The three-dimensional target detection method based on multimodal voxel image feature fusion attention according to claim 4, characterized in that, The voxel domain features and image domain features are fused together, as detailed below: (1) Camera features for and Query camera Perform deformation and transformation, for a given 2D reference point. In camera domain features Apply a deformable mask with adaptive offset and weights, mask offset and mask weights : , In the formula, ' 'Indicates summation of elements, Indicates a feedforward network; (2) Mask weight The given offset is determined jointly by voxel queries and camera queries. and weight Attention value The calculation is as follows: , (4) In the formula, It is the index of the attention head. Sampling Perform indexing. It is sampling The total number, and Represents the learnable projection matrix; (3) Voxel query Camera search Further integration and transformation are achieved through a gating fusion mechanism, namely... and Blend using the different proportions given below: , (5) , (6) In the formula, ' ' indicates element-wise multiplication. It is the sigmoid function. , and These represent different convolutional layers.

6. The three-dimensional target detection method based on multimodal voxel image feature fusion attention according to claim 5, characterized in that, A multi-scale depth prediction module is constructed to effectively encode and extract voxel features and image spatial features from different modal feature information. The voxel features and image spatial features are then concatenated to obtain combined features, as detailed below: (1) Perform 3D voxel feature extraction: (1.1) Given N 3D index vectors and eigenvectors The input voxels consist of, among which Indicates the number of input feature channels; (1.2) For each voxel feature Through 3D submanifold convolution kernel The number of geometric feature voxels is encoded and features are extracted. Specifically, geometric features are calculated from non-empty voxels in the corresponding neighborhood based on the corresponding 3D index vector: (7) In the formula, Indicates by index The relevant neighborhood features obtained under guidance Used to represent nonlinear activation functions; (2) Extract spatial features from 2D images: (2.1) Convert the 3D index into a set of mesh points based on the voxelization parameter. ; (2.2) Next, the grid points Data augmentation is performed based on a backward transformation to the original coordinate system. ; (2.3) Camera parameters calibrated based on radar are projected Projected onto a 2D image plane, the whole can be represented as: (8) In the formula, Represents the 2D index quantity for each voxel feature. Feature information is calculated from the corresponding domain based on the corresponding 2D index. , (9) in Represented by index vector The generated domain voxel features, Represents a 2D sub-flow convolution kernel; (3) Concatenate voxel features and image spatial features to obtain combined features: (3.1) will and By combining these equations, we can obtain the corresponding feature vectors. ,in Indicates the number of output feature channels: 。 7. The three-dimensional target detection method based on multimodal voxel image feature fusion attention according to claim 6, characterized in that, Deep feature prediction is performed based on the combined features, as follows: (1) Set the extracted simultaneous features as The 3D feature points are projected onto the 2D image, and the relevant points within the foreground mask are recorded. These points serve as depth reference points to provide corresponding depth information, represented as follows: In the formula and These are the pixel coordinates corresponding to the point. It obtains real depth information; (2) Uniformly sample a subset from each instance mask of the 2D image. Each element point in the subset will obtain the true depth information from the nearest reference point as the estimated depth; (3) Project the sampled subset of the estimated depth back into 3D space as prediction points; (4) Connect the 2D image features with the corresponding predicted sparse depth features of the image, and fuse them with the convolutional layer to obtain depth-aware semantic features. Generated from the same instance mask For each predicted point, the corresponding pixel semantic features will have different effects along the projection ray in 3D space. Therefore, a dynamic weighting factor is calculated for each individual depth to adjust the influence of the semantic features. The process for a specific predicted point is described as follows: (10) (11) In the formula, It is a subset The An estimated depth, This represents a dynamic weighting factor used to adjust the prediction points for the corresponding semantic features. .

8. The three-dimensional target detection method based on multimodal voxel image feature fusion attention according to claim 7, characterized in that, The overall loss function includes classification losses for object and category. Regression loss used to predict bounding boxes Prediction loss for dual-domain feature fusion Depth loss used for deep feature prediction The details are as follows: (17) In the formula, , , , These represent the regularization weight coefficients for classification loss, regression loss, prediction loss, and depth loss, respectively. The depth loss for depth feature prediction is handled using focus loss: (18) In the formula, It is a 2D pixel feature region with an effective depth label. It is the accurate true value of the lidar.

9. A three-dimensional target detection system based on multimodal voxel image feature fusion attention, used to implement the three-dimensional target detection method based on multimodal voxel image feature fusion attention as described in any one of claims 1 to 8, characterized in that, include: The first construction unit is used to construct a Transformer-based dual-domain voxel space feature encoding module, which performs corresponding feature query encoding on voxel domain features and camera domain features respectively, so as to make the most of different modal feature information to obtain the preselection box; The second building unit is used to build a multi-scale depth prediction module, which effectively encodes and extracts voxel features and image space features from different modal feature information, and concatenates voxel features and image space features to obtain combined features. The deep feature prediction unit is used to predict deep features based on combined features, thereby enhancing the degree of deep interaction between features of different modalities. The third building unit is used to build a dynamic attention feature aggregation module. It uses centroids to represent voxel features more accurately and utilizes learnable dynamic offset and cross-attention mechanisms to align different features across modalities in a fine-grained manner, generating cross-modal features. The training output unit is used to train the overall model composed of the above units using the overall loss function, and to perform object detection using the overall model.

Images