Three-dimensional target detection method based on multi-modal voxel image feature fusion attention

By using Transformer's dual-domain voxel spatial feature encoding module and multi-scale depth sham prediction module in multimodal 3D object detection, combined with the dynamic attention feature aggregation module, the problem of insufficient weight allocation in feature level fusion is solved, and the problem of improved Object detection accuracy and long-distance small target positioning capability.

CN119049008AActive Publication Date: 2024-11-29SUZHOU UNIV

Patent Information

Application Number
CN202410879904.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-02
Publication Date
2024-11-29
Estimated Expiration
2044-07-02

AI Technical Summary

Technical Problem

The existing multimodal 3D object detection method lacks accurate feature weight allocation during feature-level fusion, resulting in matching misalignment and data loss, reducing object detection accuracy.

Method used

The corresponding feature query encoding is performed using the two-domain voxel spatial feature encoding module based on Transformer, combining the multi-scale depth prediction module and the dynamic attention feature aggregation module, and implementing different modes through the centroid and the learnable dynamic offset and cross attention mechanism. fine-grained alignment and deep interaction of state features.

Benefits of technology

It improves the interaction quality between different domain features, reduces matching errors, and enhances object detection capabilities, especially in the long-distance small target positioning accuracy.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119049008A_ABST
    Figure CN119049008A_ABST
Patent Text Reader

Abstract

The invention discloses a three-dimensional target detection method based on multi-modal voxel image feature fusion attention, and relates to the technical field of automatic driving target detection. The method comprises the following steps: constructing a dual-domain voxel spatial feature coding module based on Transform, respectively carrying out corresponding feature query coding on voxel domain features and camera domain features, and maximally utilizing different modal feature information to obtain a pre-selected box; a multi-scale depth quasi-prediction module is constructed, different modal feature information is effectively coded and extracted to obtain voxel features and image space features, and the voxel features and the image space features are cascaded to obtain simultaneous features; the method solves the problems that when a current multi-modal target detection algorithm carries out feature level fusion, accurate weight distribution is lacked between different domain features, so that matching dislocation occurs, data loss is caused, and finally the target detection precision is reduced.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving target detection technology, and specifically to a three-dimensional target detection method based on multimodal voxel image feature fusion attention. Background Art

[0002] In the current research on autonomous driving, 3D target detection technology has become an indispensable and important part. At present, 3D target detection mainly uses two sensors, camera and LiDAR, to obtain information about the vehicle's surrounding environment. These two sensors show completely different characteristics due to different measurement processes. RGB images can provide dense visual information of the target, but they are not a good source of depth information. LiDAR points can capture accurate 3D spatial information, but they often lack semantic information and have sparse reflection points. The complementary characteristics of the two make the fusion of multiple modal information a recent research hotspot. Camera-LiDAR sensor fusion aims to combine two complementary information to achieve robust target detection. Therefore, how to fuse the features between different modalities through accurate correspondence and improve the quality of multi-modal feature matching is the key.

[0003] Existing multimodal 3D object detection methods can be divided into two categories: decision-level fusion and feature-level fusion. The former first detects the object using the camera and radar detection methods, and then projects the camera feature information to the lidar point in the 3D space, combining the image point features with the point laser point features. However, the limitation is that the dense image information and the sparse point cloud features are often not effectively aligned, which destroys the correlation between different modal data and causes data loss. The feature-level fusion method extracts the corresponding features from the camera image and the lidar point cloud respectively, and aligns and fuses them in the voxel domain so that the multimodal features are combined into a single representation of the detected object, so that the detector can fully utilize the features of different modalities in the early reasoning stage. However, the existing feature-level fusion methods PointAugmenting[1] and 3D-GAF[2] transform the image features into the voxel domain using the calibration matrix and perform element feature fusion. This fusion scheme often lacks the accurate allocation of weights for different features, so that the key clues cannot be well highlighted. To this end, we propose a 3D object detection method based on multimodal voxel image feature fusion attention. Summary of the invention

[0004] The purpose of the present invention is to provide a three-dimensional target detection method based on multimodal voxel image feature fusion attention, which solves the problem that the current multimodal target detection algorithm lacks accurate distribution weights between different domain features when performing feature-level fusion, resulting in matching misalignment and data loss, which ultimately reduces the target detection accuracy.

[0005] To achieve the above object, the present invention provides the following technical solution: a three-dimensional target detection method based on multimodal voxel image feature fusion attention, specifically comprising the following steps:

[0006] Construct a dual-domain voxel spatial feature encoding module based on Transformer to perform corresponding feature query encoding on voxel domain features and camera domain features respectively, and make the best use of different modal feature information to obtain the pre-selected box;

[0007] Construct a multi-scale deep quasi-prediction module to effectively encode and extract the feature information of different modalities to obtain voxel features and image space features, and cascade the voxel features and image space features to obtain joint features;

[0008] Deep feature prediction based on joint features is used to enhance the depth of interaction between features of different modalities;

[0009] We build a dynamic attention feature aggregation module to more accurately represent voxel features by using centroids, and use learnable dynamic offsets and cross-attention mechanisms to align different features across modalities in a fine-grained manner to generate cross-modal features.

[0010] The overall loss function is used to train the overall model constructed in the above steps, and the overall model is used for target detection.

[0011] Furthermore, a dual-domain voxel spatial feature encoding module is constructed to perform corresponding feature query encoding on voxel domain features and camera domain features, and maximize the use of different modal feature information to obtain the pre-selected box. The specific method is as follows:

[0012] (1) Dual feature query for voxel domain features and camera domain features

[0013] (1.1) By voxelizing the point cloud features, k non-empty voxels are obtained, and the number k is determined according to the distribution of the point cloud features;

[0014] (1.2) Voxel query Matching with corresponding non-empty voxels to improve voxel domain features;

[0015] (1.3) Project the center point position of the non-empty voxel feature to the position of the corresponding camera domain, and query it by the camera in turn To perform correspondence assignment and to improve image domain features;

[0016] (2) Perform voxel transformer. The local self-attention layer of the transformer firstly extracts the geometric structure features within the field point, and encodes the global target feature information through the global self-attention layer of the transformer, thereby obtaining the complete contextual information from local to global in the point cloud features;

[0017] (3) Perform image transformer and use the transformer local self-attention layer to construct the positional relationship between local and overall image features;

[0018] (4) Perform dual-domain feature fusion on voxel domain features and image domain features, and pay attention to decoding dual feature queries V' on multiple attention layers q and C' q .

[0019] Furthermore, the voxel transformer is performed as follows:

[0020] (1) The transformer local self-attention layer queries the input voxel features The corresponding non-empty voxels obtain the foreground feature set V' through standard sampling operations q = {V1, V2, …, V N};

[0021] (2) V i With as the center of the circle, we take the feature points within a certain radius r and perform local self-attention query operations, so as to assign weights to different features of the feature points within the range, thereby obtaining a subset of the geometric structure within the exploration area points;

[0022] (3) The obtained geometric structure subsets are further grouped and input into the transformer encoding module that aggregates the feature information. For a given feature set V' q = {V1, V2, …, V N}, the output is

[0023]

[0024] In the formula, δ=θ(V N -V i ) is the feature encoding position function, θ is an algorithmic perceptron with multiple linear layers and one fully connected layer, λ, α are self-attention parameters and change values, respectively. ψ is the feature point transformation parameter, and ρ is the softmax function.

[0025] Furthermore, the image transformer is performed as follows:

[0026] (1) The transformer encoder consists of an image query sequence As input, the original image features Reshape into a 2D feature sequence

[0027] Where (H, W) is the resolution of the original image features, C is the number of channels, D is the feature resolution after reshaping, and N = HW / D is the number of features after reshaping, which is also the effective input sequence length of the image transformer encoder.

[0028] (2) Transformer uses a constant vector V in its application layer e , using formula (2), the reshaped features are output through trainable linear projection:

[0029]

[0030] The output of the transformer encoder is represented by f i express.

[0031] (3) The feature classification head is implemented by a hidden MLP layer during pre-training and a single linear layer during fine-tuning:

[0032]

[0033] In the formula, MSA represents the multi-head self-attention layer, Z * represents a single-layer feature, MLP is a linear perceptron layer, LN is a fully connected layer, and f i Represents the output of the transformer encoder.

[0034] Furthermore, the voxel domain features and the image domain features are fused in two domains, as follows:

[0035] (1) The camera feature f i Query the camera with key and value C' q Perform deformation attention transformation, for a given 2D reference point In the camera domain, the feature f i A deformable mask with adaptive offset and weight is applied on the image, with mask offset Δp m and the mask weight Δw m :

[0036] Δp m =PFN(C′ q ),Δw m =PFN(C′ q +V′ q ),

[0037] In the formula, '+' represents element summation, and PFN represents feed-forward network;

[0038] (2) Mask weight Δp m Determined by voxel query and camera query, given an offset Δpm and weight Δw m , attention value The calculation is as follows:

[0039]

[0040] Where h is the index of the attention head, k is the index of the sampling key, and K is the total number of sampling keys. and represents a learnable projection matrix;

[0041] (3) Voxel query V' q and camera query C' q The transformations are further fused through the gated fusion mechanism, i.e., V' q and C' q Fusion is performed using different ratios given below:

[0042] C″ q =C' q +V' q ×σ(Conv1(C' q +V' q )), (5)

[0043] V″ q =V' q +C' q ×σ(Conv2(C' q +V' q )), (6)

[0044] In the formula, '×' represents element-wise multiplication, and σ(·) is the sigmoid function Conv1(·) and Conv2(·) represent different convolutional layers.

[0045] Furthermore, a multi-scale deep quasi-prediction module is constructed to effectively encode and extract the feature information of different modalities to obtain voxel features and image space features, and the voxel features and image space features are cascaded to obtain joint features, as follows:

[0046] (1) Perform 3D voxel feature extraction:

[0047] (1.1) Given N 3D index vectors and the eigenvector The input voxels are composed of c in Indicates the number of input feature channels;

[0048] (1.2) For each voxel feature x i ∈X, through the 3D submanifold convolution kernel K 3D(·) Encode the number of geometric feature voxels and extract features. Specifically, based on the corresponding 3D index vector, the geometric features are calculated from the non-empty voxels in the corresponding neighborhood:

[0049]

[0050] In the formula, represents the relevant neighborhood features obtained by index S, Used to represent nonlinear activation functions;

[0051] (2) Perform 2D image spatial feature extraction:

[0052] (2.1) by converting the 3D index into a set of grid points G(·) based on the voxelization parameters;

[0053] (2.2) Then, the grid point G(·) is transformed back to the original coordinate system and data enhancement τ(·) is performed;

[0054] (2.3) Based on the radar calibration, the camera parameters are projected into the 2D image plane through projection P(·), which can be expressed as follows:

[0055]

[0056] In the formula, Represents the 2D index quantity for each voxel feature By calculating the feature information from the corresponding field based on the corresponding 2D index

[0057]

[0058] in Represented by index vector The resulting domain voxel features, K 2D Represents a 2D sub-flow convolution kernel;

[0059] (3) Cascade the voxel features and image space features to obtain joint features:

[0060] (3.1) and Combined, we get the corresponding eigenvector where c out Indicates the number of output feature channels:

[0061]

[0062] Furthermore, deep feature prediction is performed based on the joint features, as follows:

[0063] (1) Let the extracted joint features be P = (ai ,b i ,c i ), project the 3D feature points onto the 2D image and record the relevant points in the foreground mask. These points are used as depth reference points to provide the corresponding depth information, expressed as:

[0064] Where u i and v i is the pixel coordinate corresponding to the point, d i It is the real depth information obtained.

[0065] (2) Uniformly sample a subset from each instance mask of the 2D image Each element point in the subset will obtain the real depth information from the nearest reference point as the estimated depth;

[0066] (3) Project the sampled subset of estimated depth back into 3D space as prediction points;

[0067] (4) Connect the 2D image features with the sparse depth features corresponding to the image prediction, and fuse them with the convolutional layer to obtain the depth-aware semantic features C d , the corresponding pixel semantic features of the predicted points of M generated by the same instance mask will have different influences along the projection ray in 3D space. Therefore, a dynamic weight factor is calculated for each individual depth to adjust the influence of the semantic features. The process for a specific predicted point is described as:

[0068]

[0069] In the formula, Is a subset The Mth estimated depth of Represents a dynamic weight factor, which is used to adjust the prediction point of the corresponding semantic feature of the decoration

[0070] Furthermore, a dynamic attention feature aggregation module is constructed to more accurately represent voxel features by using the centroid, and to align different features across modalities in a fine-grained manner using a learnable dynamic offset and cross-attention mechanism to generate cross-modal features, as follows:

[0071] (1) The input is a set of non-empty voxel features and image features F I , where S v is the voxel index, V t is the non-empty voxel feature vector, N v is the number of non-empty voxels, and the feature F of each voxel is obtained by averaging the spatial positions of all points in the same voxel. v The centroid Ce :

[0072]

[0073] In the formula, (x i ,y i ,z i ) is the position coordinate information, |p(V i )| is the voxel feature F v The number of points within

[0074] Then use the camera projection matrix H to calculate the reference point C for the voxel point centroid i :

[0075] C i =H·C e (13)

[0076] Where H is the product of the camera's internal and external parameter matrices;

[0077] (2) Assign voxel grid index to each voxel point centroid that needs to be calculated k is the number of 3D grid points, G is the grid feature channel, which is combined through the fully connected layer and the sigmoid layer. th The feature generates a pair of centroid grid weights The weights are added to the non-empty voxel features and image features to obtain the weighted voxel grid features. With deep image mesh features

[0078]

[0079] (3) Then, three connection layers are used to convert the voxel grid features into query Q and the deep image grid features into Key and Value. The inner product between each query Q and the key value Key is performed to obtain the attention affinity matrix between the voxel grid features and the deep image grid features. The matrix is ​​used to constant and aggregate the value Value. Finally, the aggregated fusion feature F ag Processed through a fully connected layer, the target category and positioning information are output through a standard 2D-3D cascade. The entire dynamic fusion process can be expressed as:

[0080]

[0081] Where W m and W' m is the learnable weight, M is the number of self-attention heads, K is the total number of sampling points, Δp mik and A mik They represent the sampling offset and attention weight of the kth sampling point in the mth attention head respectively.

[0082] Furthermore, the overall loss function includes the classification loss L for objectness and category cls , the regression loss L for predicting the box reg , prediction loss L for dual-domain feature fusion dff , depth loss L for deep feature prediction depth , as follows:

[0083] L total =λ cls L cls +λ reg L reg +λ dff L dff +λ dep L dep (17)

[0084] In the formula, λ cls , reg , dff , dep Represent the regularization weight coefficients of classification loss, regression loss, prediction loss, and depth loss respectively;

[0085] The depth loss of deep feature prediction is processed by focal loss:

[0086]

[0087] Where |ρ| is the 2D pixel feature region with valid depth labels, is the accurate lidar true value.

[0088] According to a second aspect of the present invention, the present invention provides a three-dimensional target detection system based on multimodal voxel image feature fusion attention, comprising:

[0089] The first construction unit is used to construct a dual-domain voxel spatial feature encoding module based on Transformer, which performs corresponding feature query encoding on voxel domain features and camera domain features respectively, and maximizes the use of different modal feature information to obtain the pre-selected box;

[0090] The second construction unit is used to construct a multi-scale deep quasi-prediction module, effectively encode and extract different modal feature information to obtain voxel features and image space features, and cascade the voxel features and image space features to obtain joint features;

[0091] A deep feature prediction unit, used to predict deep features based on joint features, and to enhance the degree of deep interaction between features of different modalities;

[0092] The third building block is used to construct a dynamic attention feature aggregation module, which uses the centroid to more accurately represent the voxel features and uses a learnable dynamic offset and cross-attention mechanism to align different features across modalities in a fine-grained manner to generate cross-modal features;

[0093] The training output unit is used to train the overall model constructed in the above steps using the overall loss function, and use the overall model to perform target detection.

[0094] The present invention has at least the following beneficial effects:

[0095] (1) In order to solve the problem of lack of accurate weight allocation between different domain features, the present invention designs a dual-domain voxel spatial feature encoding module based on Transformer. By performing corresponding feature query encoding on the corresponding different domain features, the pre-selection box is obtained by making maximum use of different modal feature information.

[0096] (2) In order to further improve the interaction quality between features in different domains and reduce matching errors, the present invention designs a multi-scale depth quasi-prediction module with depth perception function, which is used to enhance the interaction depth of point features of different scales, thereby further refining the features, enhancing the target detection capability, and improving the positioning accuracy of small targets at a long distance.

[0097] (3) In order to more accurately fuse features from different domains, the present invention designs a dynamic attention feature aggregation module, which uses the centroid to more accurately represent voxel features and utilizes a learnable dynamic offset and cross-attention mechanism to align different features across modalities in a fine-grained manner to generate cross-modal features.

[0098] Of course, any product implementing the present invention does not necessarily need to achieve all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0099] Figure 1 It is a schematic diagram of the overall framework of the method of the present invention;

[0100] Figure 2 Schematic diagram of the framework of the voxel transformer in the present invention;

[0101] Figure 3 Schematic diagram of the framework of the image transformer in the present invention;

[0102] Figure 4 It is a schematic diagram of the process of extracting voxel features and image space features in the present invention;

[0103] Figure 5 Schematic diagram of the framework of the dynamic attention feature aggregation module of the present invention;

[0104] Figure 6 This is an example of qualitative results on the KITTI dataset of Embodiment 1 of the present invention;

[0105] Figure 7 This is a schematic diagram of the comparison of precision-Recall curves of vehicle detection results of the KITTI dataset in Example 1 of the present invention;

[0106] Figure 8 This is an example of qualitative results on the NuScenses dataset in Example 1 of the present invention;

[0107] Fig. 9 This is a schematic diagram of a real vehicle platform used for real testing in Example 1 of the present invention;

[0108] Fig.10 A schematic diagram showing a comparison of average accuracies of different object modules between a baseline and the method according to an embodiment of the present invention;

[0109] Fig.11 This is a schematic diagram of average accuracy under different standards in Example 1 of the present invention;

[0110] Fig.12 Schematic diagram of comparison of the number of decoder layers on the KITTI validation set according to the first embodiment of the present invention. DETAILED DESCRIPTION

[0111] The following will be combined with the drawings in the embodiments of the present disclosure to clearly and completely describe the technical solutions in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0112] Currently, there are three-dimensional object detection methods based on single modality, three-dimensional object detection methods based on multimodality, and three-dimensional object detection methods based on Transformer:

[0113] Current single-modal target detection algorithms are mainly based on LiDAR data or RGB images. Voxel-RCNN[5] converts the original unordered point cloud into a more structured data format through voxelization. Fan[6] et al. used the range view method to extract features from the unordered point cloud. CenterPoint[7] uses the center point mechanism to replace the original anchor framework for 3D target detection. LiDAR-RCNN[8] proposed to use boundary offset and virtual point methods to solve the size ambiguity problem. SPG[9] fuses the original point cloud with the generated semantic point cloud to obtain enhanced point cloud data to improve target detection capabilities. PyramidR-CNN

[10] designed a pyramid ROI head to expand the original ROI area and extract features to solve the problem of sparse and uneven distribution of point clouds in the interest area. CT3D

[11] uses the channel attention mechanism to process 3D feature point cloud data and adaptively weights different features, thereby enhancing the representation ability of point cloud data. IA-SSD

[12] uses two learnable, instance-aware downsampling strategies to hierarchically select foreground points to improve detection efficiency. Chen et al.

[13] proposed to construct pseudo-stereo images through image-level generation and feature-level generation to improve the detection effect of monocular images. The lidar point cloud data of these algorithms are relatively sparse, or the image data lacks fine-grained information such as depth, which inevitably affects the target detection effect.

[0114] 3D object detection based on multimodality

[0115] LiDAR and cameras can provide effective complementary information around the target, so it is attractive to fuse multimodal inputs to achieve 3D object detection tasks. FusionPainting

[14] projects semantic segmentation masks onto LiDAR points to improve detection accuracy. VFF

[15] proposes a method called voxel field fusion for cross-modal 3D detection, which enhances contextual interaction in three-dimensional space while ensuring feature space representation. PointAugmenting

[16] effectively fuses features of different modalities through cross-modal data augmentation. 4D-Net

[17] better uses motion cues and dense image information to detect distant targets while maintaining computational efficiency by implementing a new dynamic connection learning method. 3D-GAF

[18] proposes to use a depth completion method to generate pseudo point clouds, and uses a parallel mesh fusion method for modality fusion and data augmentation. AutoAlignV2

[19] enables the model to reason in a more dynamic way and accelerate the aggregation of features from different modalities through a depth-aware data augmentation strategy and image-level training scheme. CenterFusion

[20] uses a cone-based association method to accurately associate radar detection points with targets in images. LIFT

[21] improves the efficiency of multimodal 3D target detection by learning to align input 4D sequence cross-sensor data, thereby effectively utilizing multimodal data features. However, the above target detection algorithms generally lack reasonable feature weights for different modal features.

[0116] 3D Object Detection Based on Transformer

[0117] DETR

[22] is the pioneering work of applying Transformer to image object detection. DETR streamlines the detection pipeline and regards object detection as a direct set prediction problem, effectively eliminating many manually designed components. However, the object queries generated by random initialization require a decoder cascade to refine the target features, which causes DETR to usually take an extremely long time to converge during training. DeepFusion

[23] proposed two new technologies, InverseAug and LearnableAlign. InverseAug uses inverse geometric correlation enhancement technology to achieve precise geometric alignment between lidar points and pixel images. LearnableAlign uses cross-attention to dynamically capture the correlation of features of different modalities. TransFusion designs a detection framework with a soft association mechanism, which enables the model to adaptively determine the information and location of target features. Monodtr

[24] introduces a depth-aware feature enhancement module and uses auxiliary deep learning depth-aware features to avoid obtaining inaccurate depth prior information from pre-trained depth estimation. The latest Transformer-based methods

[25] -

[31] solve the DETR problem by designing different modules, but the current Transformer-based methods do not make effective use of features from different domains.

[0118] Embodiment 1:

[0119] See also Figure 1-12 The present invention provides a technical solution: a three-dimensional target detection method based on multimodal voxel image feature fusion attention, which specifically includes the following steps:

[0120] S1. Construct a dual-domain voxel spatial feature encoding module based on Transformer, adopt a deformable transformer-decoder, and use the deformable attention mechanism to perform corresponding feature query encoding on voxel domain features and camera domain features respectively and perform dual-domain feature fusion to achieve aggregation of radar and camera dual-domain feature information to generate initial candidate boxes, as follows:

[0121] (1) Dual feature query: Dual feature query refers to obtaining more accurate features through camera feature query and voxel feature query:

[0122] (1.1) By voxelizing the point cloud features, k non-empty voxels are obtained, and the number k is determined according to the distribution of the point cloud features;

[0123] (1.2) Voxel query Matching with corresponding non-empty voxels to improve voxel domain features;

[0124] (1.3) Project the center point position of the non-empty voxel feature to the position of the corresponding camera domain, and query it by the camera in turn To perform correspondence assignment and to improve image domain features;

[0125] (2) Perform voxel transformer. The local self-attention layer of the transformer firstly extracts the geometric structure features within the field point, and encodes the global target feature information through the global self-attention layer of the transformer, thereby obtaining the complete context information from local to global in the point cloud features, as follows:

[0126] (2.1) The transformer local self-attention layer queries the input voxel features The corresponding non-empty voxels obtain the foreground feature set V' through standard sampling operations q = {V1, V2, …, V N};

[0127] (2.2) V i With as the center of the circle, we take the feature points within a certain radius r and perform local self-attention query operations, so as to assign weights to different features of the feature points within the range, thereby obtaining a subset of the geometric structure within the exploration area and simplifying the computational burden;

[0128] (2.3) The obtained geometric structure subsets are further grouped and input into the transformer encoding module that aggregates the feature information. For a given feature set V' q = {V1, V2, …, V N}, the output is

[0129]

[0130] In the formula, δ=θ(V N -V i ) is the feature encoding position function, θ is an algorithmic perceptron with multiple linear layers and one fully connected layer, γ and α are self-attention parameters and change values, respectively. ψ is the feature point transformation parameter, ρ is the softmax function;

[0131] (3) Perform image transformer and use the transformer local self-attention layer to construct the positional relationship between the local and overall image features. The image transformer schematic diagram is shown in the figure. Figure 3 As shown:

[0132] (3.1) where the transformer encoder consists of an image query sequence As input, the original image features Reshape into a 2D feature sequence

[0133] Where (H, W) is the resolution of the original image features, C is the number of channels, D is the feature resolution after reshaping, and N = HW / D is the number of features after reshaping, which is also the effective input sequence length of the image transformer encoder.

[0134] (3.2) Transformer uses a constant vector V in its application layer e , using formula (2), the reshaped features are output through trainable linear projection:

[0135]

[0136] The output of the transformer encoder is represented by f i express;

[0137] (3.3) The feature classification head is implemented by a hidden layer MLP layer during pre-training and a single linear layer during fine-tuning:

[0138]

[0139] In the formula, MSA represents the multi-head self-attention layer, Z * represents a single-layer feature, MLP is a linear perceptron layer, LN is a fully connected layer, and f i Represents the output of the transformer encoder;

[0140] (4) Dual-domain feature fusion is performed on voxel domain features and image domain features. Dual-domain feature fusion pays attention to decoding dual-feature queries and on multiple attention layers:

[0141] (4.1) The camera feature f i Query the camera with key and value C' q Perform deformation attention transformation, for a given 2D reference point In the camera domain, the feature f i A deformable mask with adaptive offset and weight is applied on the image, with mask offset Δp m and the mask weight Δw m :

[0142] Δp m =PFN(C′ q ),Δw m =PFN(C′ q +V′ q ),

[0143] In the formula, '+' represents element summation, and PFN represents feed-forward network;

[0144] (4.2) Mask weight Δp m Determined by the voxel query and the camera query, the effect of feature fusion can be improved by the attention weights based on the voxel region and camera region features. Given an offset Δp m and weight Δw m , attention value The calculation is as follows:

[0145]

[0146] Where h is the index of the attention head, k is the index of the sampling key, and K is the total number of sampling keys. and represents a learnable projection matrix;

[0147] (4.3) Voxel query V' q and camera query C' q The transformations are further fused through the gated fusion mechanism, i.e., V' q and C' q Fusion is performed using different ratios given below:

[0148] C″ q =C' q +V' q ×σ(Conv1(C' q +V' q )), (5)

[0149] V″ q =V' q +C' q ×σ(Conv2(C' q +V' q )), (6)

[0150] In the formula, '×' represents element-wise multiplication, and σ(·) is the sigmoid function Conv1(·) and Conv2(·) represent different convolutional layers. Since the ratio is a function of and, the combined ratio will be adaptively adjusted according to the input features;

[0151] S2. Construct a multi-scale deep quasi-prediction module to effectively encode and extract voxel features and image space features from different modal feature information, and cascade the voxel features and image space features to obtain joint features, which are used to enhance the depth of interaction between different modal features, thereby enhancing information features:

[0152] (1) Perform 3D voxel feature extraction:

[0153] (1.1) Given N 3D index vectors and the eigenvector The input voxels are composed of c in Indicates the number of input feature channels;

[0154] (1.2) For each voxel feature x i ∈X, through the 3D submanifold convolution kernel K 3D (·) Encode the number of geometric feature voxels and extract features. Specifically, based on the corresponding 3D index vector, the geometric features are calculated from the non-empty voxels in the corresponding neighborhood:

[0155]

[0156] In the formula, represents the relevant neighborhood features obtained by index S, Used to represent nonlinear activation functions;

[0157] (2) Since feature semantic information is mainly distributed on the boundaries of 2D targets, the error interference caused by insufficient depth information will affect the detection performance. Therefore, we expand the convolution receptive field to the 2D image space and use 2D adjacent voxels to encode the perceived semantic feature information to extract 2D image space features. The details are as follows:

[0158] (2.1) by converting the 3D index into a set of grid points G(·) based on the voxelization parameters;

[0159] (2.2) Then, the grid point G(·) is transformed back to the original coordinate system and data enhancement τ(·) is performed;

[0160] (2.3) Based on the radar calibration, the camera parameters are projected into the 2D image plane through projection P(·), which can be expressed as follows:

[0161]

[0162] In the formula, Represents the 2D index quantity for each voxel feature By calculating the feature information from the corresponding field based on the corresponding 2D index

[0163]

[0164] in Represented by index vector The resulting domain voxel features, K 2D Represents a 2D sub-flow convolution kernel;

[0165] If there are multiple features in a single 2D adjacent voxel feature, a maximum pooling operation is performed to retain the most important feature for 2D convolution. By expanding the receptive field to the 2D image space and using 2D adjacent voxels to encode the perceived feature information, features of different modalities can be effectively encoded and extracted, while reducing background errors and effectively utilizing foreground features. The feature extraction flow chart is shown below: Figure 4 As shown;

[0166] (3) Cascade the voxel features and image space features to obtain joint features:

[0167] (3.1) and Combined, we get the corresponding eigenvector where c out Indicates the number of output feature channels:

[0168]

[0169] S3. Deep feature prediction based on joint features to enhance the depth of interaction between different modal features:

[0170] The LiDAR and camera features extracted by encoding in the previous stage focus on geometric and semantic features respectively. Next, the depth feature prediction module predicts the depth feature information to improve the interaction quality between the two, so as to perform multimodal feature fusion more efficiently, as follows:

[0171] (1) Let the extracted joint features be P = (a i ,b i ,c i ), project the 3D feature points onto the 2D image and record the relevant points in the foreground mask. These points are used as depth reference points to provide the corresponding depth information, expressed as:

[0172] Where u i and v i is the pixel coordinate corresponding to the point, d i It is the real depth information obtained.

[0173] (2) Uniformly sample a subset from each instance mask of the 2D image Each element point in the subset will obtain the real depth information from the nearest reference point as the estimated depth;

[0174] (3) Project the sampled subset of estimated depth back into 3D space as prediction points;

[0175] (4) The above prediction points are further processed to utilize the semantic information encapsulated in the image features. Different from simple decoration processing, depth-aware semantic features are generated by pre-corresponding interactions between 2D image features and reference points from depth prediction. These depth-aware semantic features and their corresponding sampling subsets modify the prediction points in an adaptive manner. Specifically:

[0176] Connect the 2D image features with the sparse depth features corresponding to the image prediction, and fuse them with the convolutional layer to obtain the depth-aware semantic features C d , the corresponding pixel semantic features of the predicted points of M generated by the same instance mask will have different influences along the projection ray in 3D space. Therefore, a dynamic weight factor is calculated for each individual depth to adjust the influence of the semantic features. The process for a specific predicted point is described as:

[0177]

[0178] In the formula, Is a subset The Mth estimated depth of Represents a dynamic weight factor, which is used to adjust the prediction point of the corresponding semantic feature of the decoration

[0179] By predicting the depth information, the depth interaction between different modal features is enhanced, the domain gap between different modal data is reduced, and the prediction points with dense depth features and semantic features are obtained, thereby enhancing the feature information representation;

[0180] S4. Construct a dynamic attention feature aggregation module to more accurately represent voxel features by using centroids, and use learnable dynamic offsets and cross-attention mechanisms to align different features across modalities in a fine-grained manner to generate cross-modal features:

[0181] With respect to the technical solution of this embodiment, this embodiment uses a centroid dynamic fusion mechanism. By using the centroid of voxel points to represent the spatial position of the target feature, the problem of the voxel center ignoring the actual distribution of the voxel points can be avoided. The centroid of the voxel point is closer to the scanned surface of the object than the voxel center, and can more effectively provide the original geometric shape information and scale.

[0182] This embodiment also adopts a dynamic fusion mechanism of grid points, and uses a 3D gridding module to respectively crop feature information of different modalities. Compared with the original strategy of using 2D regions of interest to extract target features, it can reduce the interference of related background point noise, thereby effectively utilizing the semantic information in the image and the point cloud depth information to improve the target representation and enhance the detection capability;

[0183] (1) The input is a set of non-empty voxel features and image features F I , where S v is the voxel index, V t is the non-empty voxel feature vector, N v is the number of non-empty voxels, and the feature F of each voxel is obtained by averaging the spatial positions of all points in the same voxel. v The centroid C e :

[0184]

[0185] In the formula, (x i ,y i ,z i ) is the position coordinate information, |p(V i )| is the voxel feature F v The number of points within

[0186] Then use the camera projection matrix H to calculate the reference point C for the voxel point centroid i :

[0187] C i =H·C e (13)

[0188] Where H is the product of the camera's internal and external parameter matrices;

[0189] (2) Assign voxel grid index to each voxel point centroid that needs to be calculated k is the number of 3D grid points, G is the grid feature channel, which is combined through the fully connected layer and the sigmoid layer. th The feature generates a pair of centroid grid weights The weights are added to the non-empty voxel features and image features to obtain the weighted voxel grid features. With deep image mesh features

[0190]

[0191] (3) Then, three connection layers are used to convert the voxel grid features into query Q and the deep image grid features into Key and Value. The inner product between each query Q and the key value Key is performed to obtain the attention affinity matrix between the voxel grid features and the deep image grid features. The matrix is ​​used to constant and aggregate the value Value. Finally, the aggregated fusion feature F ag Processed through a fully connected layer, the target category and positioning information are output through a standard 2D-3D cascade. The entire dynamic fusion process can be expressed as:

[0192]

[0193] Where W m and W' m is the learnable weight, M is the number of self-attention heads, K is the total number of sampling points, Δp mik and A mik They represent the sampling offset and attention weight of the kth sampling point in the mth attention head, respectively, which are obtained by linear projection on the query feature;

[0194] S5. Use the overall loss function to train the overall model constructed in the above steps, and use the overall model for target detection:

[0195] The overall loss function includes the classification loss L for objectness and category cls , the regression loss L for predicting the box reg , prediction loss L for dual-domain feature fusion dff , depth loss L for deep feature prediction depth , the overall loss function is as follows:

[0196] L total =λ cls L cls +λ reg L reg +λ dff L dff +λ dep L dep (17)

[0197] Among them, λ cls , reg , dff , dep Represent the regularization weight coefficients of classification loss, regression loss, prediction loss, and depth loss respectively;

[0198] In order to balance the imbalance of positive and negative samples, focal loss is used to balance the samples of classification tasks and prediction tasks respectively, and smoothed-L1 is used to balance the regression loss. For the depth loss of deep feature prediction, focal loss is also used to deal with it:

[0199]

[0200] where |ρ| is the 2D pixel feature region with valid depth labels, is the accurate lidar true value.

[0201] Next, the technical solution of the present invention is further described in detail with reference to specific embodiments:

[0202] Datasets and metrics

[0203] This example evaluates the proposed object detection method on the challenging KITTI 3D

[32] and NuScenes

[33] object detection datasets and the latest open dataset Waymo

[34] .

[0204] 1. Experimental Verification

[0205] 1.1 Datasets and Indicators

[0206] This example evaluates and tests the above method on the challenging KITTI 3D

[32] and NuScenes

[33] object detection datasets and the latest open dataset Waymo

[34] .

[0207] 1.2 Experimental Results on KITTI Dataset

[0208] Table 1 shows the comparison of the detection performance of the method in this embodiment and other object detection methods in the KITTI test set car category. As can be seen from Table 1, the comprehensive detection performance of the method in this embodiment is better than other current 3D object detection methods. Compared with the baseline CenterPoint[7] of the method in this embodiment, the average detection accuracy AP at different levels of the method in this embodiment is 3D The average precision AP is improved by 1.67 / 1.27 / 0.78 respectively on medium and difficult difficulty levels compared with the excellent SE-SSD

[41] . 3D They also increased by 0.08 / 0.66 respectively.

[0209] Table 2 shows the comparison results of the detection performance of the method of this embodiment for pedestrian and bicycle categories in the KITTI test set. As can be seen from Table 2, compared with the baseline CenterPoint[7] of the method of this embodiment, the average detection accuracy of the pedestrian target of the method of this embodiment is improved by 0.61 / 0.07 / 0.06 at different levels of difficulty, while the detection results of the bicycle target are improved by 1.19 / 1.83 / 1.14 at different levels of difficulty compared with the baseline. Table 3 shows the comparison results of the detection performance of the car category of the method of this embodiment in the KITTI validation set. Compared with the baseline, the average accuracy at different levels of difficulty is improved by 0.2 / 0.58 / 1.12 respectively. The detection schematic diagram of the method of this embodiment on the KITTI dataset is shown in the figure. Figure 6 shown.

[0210] Table 1 Detection performance of car category in KITTI test set

[0211]

[0212]

[0213] Table 2 Detection performance of pedestrian and bicycle categories in the KITTI test set

[0214]

[0215] Table 3 Detection performance of car category on KITTI validation set

[0216]

[0217] Comparison of precision-recall curves of some vehicle detection results in the KITTI dataset, such as Figure 7 As shown, compared with other target detection methods, the method of this embodiment still has high precision at a high recall rate, and the method of this embodiment has good robustness and stability as a whole.

[0218] 1.3 Experimental results on the NuScences dataset

[0219] In order to further test the effectiveness and versatility of the above method, the verification method has good detection performance for three-dimensional targets. The method was verified on the more challenging NuScences dataset. As shown in Table 4, the detection results of the method of this embodiment and other 3D target detection methods on different categories of targets on NuScences are compared. It can be seen from the table that in terms of the important indicators NDS and mAP in the NuScence dataset, the performance is improved by 0.1 / 1.3 compared to the optimal VFF

[51] . It can be seen that the proposed method still has good robustness in detection capabilities on different datasets. The schematic diagram of the detection results of the above method of this embodiment on the NuScences dataset is shown in the figure. Figure 8 shown.

[0220] Table 4 Performance comparison on NuScenes dataset

[0221]

[0222]

[0223] 1.4 Experimental results on the Waymo dataset

[0224] This embodiment also compares the proposed method with the baseline CenterPoint[7] on the Waymo dataset. The results are shown in Table 5. The results show that the method of this embodiment performs well in different levels of tests on the Waymo dataset. Compared with CenterPoint[7], the mAP / mAPH on L1 / L2 are improved by 1.45 / 1.35 and 1.48 / 1.12 respectively. The improvement is particularly great when detecting smaller targets such as pedestrians and bicycles, indicating that the method of this embodiment has excellent performance in detecting small targets at a long distance.

[0225] Table 5 Performance comparison on the Waymo dataset

[0226]

[0227] 1.5 Experiments on a real vehicle platform

[0228] Considering that only simulation experiments on data sets cannot prove the performance of TFIENet in actual scenarios, this embodiment conducts a series of tests on a real vehicle platform. Fig. 9 As shown in the figure, in addition to the basic 32-line radar, the platform also adds additional Tele-15 radar and millimeter-wave radar. The millimeter-wave radar is mainly responsible for the speed and distance of the vehicle in front; the Tele-15 radar increases the safety of the vehicle as its additional safety redundancy; the 32-line radar is responsible for short-distance, large-angle detection tasks.

[0229] In order to more clearly demonstrate the effectiveness of each module of the above method, for different target detection objects, the average accuracy of different object modules between the baseline and the method of this embodiment is compared. Fig.10 Table 6 is a comparison of the detection results of different vehicle targets on the real vehicle platform. As can be seen from Table 6, in a complex and changeable real environment, the addition of each module of the method of this embodiment can effectively improve the overall performance of the model.

[0230] Table 6 Performance of vehicle categories on the real vehicle platform

[0231]

[0232] 1.6 Ablation Experiment

[0233] This section verifies the effectiveness of each component of the above method through a large number of ablation experiments.

[0234] The effectiveness of each module is shown in Table 7. Experiment (a) is the baseline modified on CenterPoint[7] in this embodiment. The average precision results of 40 car recall position calculations on the KITTI validation set are compared. It can be seen from Table 7 that each module component can provide effective assistance to the accuracy of target detection results of different difficulty levels.

[0235] Table 7. The impact of different parts on the KITTI validation set.

[0236] The results are evaluated by AP calculated over 40 car-class recall locations.

[0237]

[0238] The dual-domain voxel spatial feature encoding module performs corresponding feature query encoding on the corresponding features of different domains, encodes and fuses the image and lidar features, and maximizes the use of different modal feature information to obtain the pre-selected box. In this way, accurate feature weights are assigned between different domain features through effective dynamic encoding between features, thereby effectively aggregating feature information from different domains. In order to verify the effectiveness of the dual-domain voxel spatial feature encoding module, this embodiment splits the module into the following parts: feature dual query (DDA), dual-domain transform self-attention layer (DD-SAL) and dual-domain feature fusion layer (DDFF). As shown in Table 8, through different components of the module, DDA, DD-SAL, and DDFF provide gains of 0.12, 0.29, and 0.39, respectively.

[0239] Table 8 Validity of DOFC module

[0240]

[0241] Fig.11 The average 3D detection accuracy of centerpoint and the method of this embodiment is compared under different 3D IoU standards and different distances. Fig.11 It is constructed from real data collected from actual platforms. Fig.11 It can be observed that the method proposed in this embodiment performs better than centerpoint at the same distance and the same 3D IoU. When the distance is 15m and IoU=0.65, the performance is improved by 10.13%. Compared with other state-of-the-art methods, the method in this embodiment can output high-quality predictions with higher accuracy. As the distance of the object increases, the number of false positives increases, but compared with the small model basis, there are fewer false positives. At medium and long distances, the method proposed in this embodiment can achieve an average accuracy of 68.1%.

[0242] In order to verify the improvement of the effectiveness of the dual query mechanism of this embodiment compared with the single camera query. The variable mask offset and weight of the single camera query mechanism are determined only by a single camera query. The dual query mechanism jointly updates the decision through feature queries in different domains to obtain more accurate variable mask offsets and feature weights. As shown in Table 9, compared with the baseline and single camera query on the KITTI validation set, the average detection accuracy of the dual query mechanism is improved by 0.21 and 0.16 respectively.

[0243] Table 9 Comparison of different query types on the KITTI validation set.

[0244]

[0245] DOFC module behavior analysis: Fig.12The performance trend of the KITTI validation set with respect to the number of decoder layers is given. In the KITTI dataset, as the number of layers increases to 4, the performance gradually improves, which indicates that the dual query attention of this embodiment gradually refines the feature quality for 3D object detection.

[0246] Verify the effectiveness of each part of the dynamic attention feature aggregation module (DAFG). As shown in Table 10, experiment (A) associates the camera features with the voxel features through the original naive fusion method, and fuses the voxel features of the lidar through a simple summation. Experiments (B) and (C) add the center of mass dynamic fusion mechanism and the 3D grid fusion mechanism in turn. The results show that each component of the dynamic attention feature aggregation module plays an improvement role. Table 10 Ablation study of the multimodal feature cross attention module. The results are evaluated using the AP calculated at 40 recall positions.

[0247]

[0248] By evaluating the method proposed in this embodiment at different distance levels and occlusion degrees, it can be known under what circumstances this embodiment can improve the baseline the most. As shown in Table 11, the object detection affected by different distances and occlusions has a significant improvement. This is because the multi-scale depth pseudo-prediction module enhances the interactive depth of point features of different scales, thereby refining the features.

[0249] Table 11 Performance at different distances and different degrees of occlusion. Evaluation results and 3D AP calculations are moderate for the car class at 40 recall locations.

[0250]

[0251] In summary, in order to solve the problem that multimodal target detection methods lack accurate weight allocation between different domain features, this embodiment proposes a dual-domain voxel space feature encoding module based on Transformer. This module encodes the corresponding feature query of different domain features, assigns corresponding feature weights to the feature information, and maximizes the use of different modal feature information; secondly, a multi-scale deep quasi-prediction module with depth perception function is designed, through which the interaction depth of point features of different scales is enhanced, thereby further refining the features and reducing data loss; finally, in order to be able to more accurately fuse different domain features, a dynamic attention feature aggregation module is designed, which uses the centroid to more accurately represent the voxel features, and uses the learnable dynamic offset and cross-attention mechanism to align different features across modalities in a fine-grained manner, and finally generates cross-modal features. The experimental results on the KITTI, NUSCENES and WAYMO datasets demonstrate the effectiveness and versatility of the method, and a large number of ablation experiments prove the effectiveness and efficiency of each module of the method. The experimental results on the real vehicle platform show that the method has good robustness in complex actual environments.

[0252] Embodiment 2:

[0253] The present invention proposes a three-dimensional target detection system based on multimodal voxel image feature fusion attention, comprising:

[0254] The first construction unit is used to construct a dual-domain voxel spatial feature encoding module based on Transformer, which performs corresponding feature query encoding on voxel domain features and camera domain features respectively, and maximizes the use of different modal feature information to obtain the pre-selected box;

[0255] The second construction unit is used to construct a multi-scale deep quasi-prediction module, effectively encode and extract different modal feature information to obtain voxel features and image space features, and cascade the voxel features and image space features to obtain joint features;

[0256] A deep feature prediction unit, used to predict deep features based on joint features, and to enhance the degree of deep interaction between features of different modalities;

[0257] The third building block is used to construct a dynamic attention feature aggregation module, which uses the centroid to more accurately represent the voxel features and uses a learnable dynamic offset and cross-attention mechanism to align different features across modalities in a fine-grained manner to generate cross-modal features;

[0258] The training output unit is used to train the overall model constructed in the above steps using the overall loss function, and use the overall model to perform target detection.

[0259] Specifically, the above-mentioned first construction unit, the second construction unit, the deep feature prediction unit, and the training output unit can be embedded in a computer processing system. The computer calls the above-mentioned units to complete the task of equivalent modeling based on the three-dimensional target detection method based on multimodal voxel image feature fusion and attention provided above; the above-mentioned first construction unit, the second construction unit, the deep feature prediction unit, and the training output unit can perform operations according to the specific steps given in the three-dimensional target detection method based on multimodal voxel image feature fusion and attention.

[0260] It should be noted that it should be understood that the division of the various units of the above system is only the division of logical functions. In actual implementation, they can be fully or partially integrated into one physical entity, or they can be physically separated, and these units can all be implemented in the form of software calling through processing elements; they can also be all implemented in the form of hardware; some units can also be implemented in the form of software calling through processing elements, and some units can be implemented in the form of hardware. For example, the receiving unit can be a separately established processing element, or it can be integrated in a chip of the above device. In addition, it can also be stored in the memory of the above device in the form of program code, and a processing element of the above device calls and executes the functions of the above signal processing unit. The implementation of other units is similar. In addition, these units can be fully or partially integrated together, or they can be implemented independently. The processing element described here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each of the above units can be completed by the hardware integrated logic circuit in the processor element or the instructions in the form of software.

[0261] For example, the above units may be one or more integrated circuits configured to implement the above methods, such as one or more application specific integrated circuits (ASIC), or one or more digital singnal processors (DSP), or one or more field programmable gate arrays (FPGA). For another example, when a certain unit above is implemented in the form of a processing element scheduling program code, the processing element may be a general-purpose processor, such as a central processing unit (CPU) or other processor that can call program code. For another example, these units may be integrated together and implemented in the form of a system-on-a-chip (SOC).

[0262] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0263] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A three-dimensional object detection method based on multimodal voxel image feature fusion attention, characterized in that: include; Construct a dual-domain voxel spatial feature encoding module based on Transformer to perform corresponding feature query encoding on voxel domain features and camera domain features respectively, and make the best use of different modal feature information to obtain the pre-selected box; Construct a multi-scale deep quasi-prediction module to effectively encode and extract the feature information of different modalities to obtain voxel features and image space features, and cascade the voxel features and image space features to obtain joint features; Deep feature prediction based on joint features is used to enhance the depth of interaction between features of different modalities; We build a dynamic attention feature aggregation module to more accurately represent voxel features by using centroids, and use learnable dynamic offsets and cross-attention mechanisms to align different features across modalities in a fine-grained manner to generate cross-modal features. The overall loss function is used to train the overall model constructed in the above steps, and the overall model is used for target detection.

2. The three-dimensional target detection method based on multimodal voxel image feature fusion attention according to claim 1 is characterized in that: A dual-domain voxel spatial feature encoding module is constructed to perform corresponding feature query encoding on voxel domain features and camera domain features, making maximum use of different modal feature information to obtain the pre-selected box. The specific method is as follows: (1) Dual feature query for voxel domain features and camera domain features (1.1) By voxelizing the point cloud features, k non-empty voxels are obtained, and the number k is determined according to the distribution of the point cloud features; (1.2) Voxel query Matching with corresponding non-empty voxels to improve voxel domain features; (1.3) Project the center point position of the non-empty voxel feature to the position of the corresponding camera domain, and query it by the camera in turn To perform correspondence assignment and to improve image domain features; (2) Perform voxel transformer. The local self-attention layer of the transformer firstly extracts the geometric structure features within the field point, and encodes the global target feature information through the global self-attention layer of the transformer, thereby obtaining the complete contextual information from local to global in the point cloud features; (3) Perform image transformer and use the transformer local self-attention layer to construct the positional relationship between local and overall image features; (4) Perform dual-domain feature fusion on voxel domain features and image domain features, and pay attention to decoding dual feature queries V' on multiple attention layers q and C' q .

3. The three-dimensional target detection method based on multimodal voxel image feature fusion attention according to claim 2 is characterized in that: Perform voxel transformer. The specific method is as follows: (1) The transformer local self-attention layer queries the input voxel features The corresponding non-empty voxels obtain the foreground feature set V' through standard sampling operations q = {V1, V2, …, V N }; (2) Taking Vi as the center of the circle, select feature points within a certain radius r and perform local self-attention query operations, so as to assign weights to different features of the feature points within the range, thereby obtaining a subset of the geometric structure within the exploration area points; (3) The obtained geometric structure subsets are further grouped and input into the transformer encoding module that aggregates the feature information. For a given feature set V' q = {V1, V2, …, V N }, the output is In the formula, δ=θ(V N -V i ) is the feature encoding position function, θ is an algorithmic perceptron with multiple linear layers and one fully connected layer, γ and α are self-attention parameters and change values, respectively. ψ is the feature point transformation parameter, and ρ is the softmax function.

4. The three-dimensional target detection method based on multimodal voxel image feature fusion attention according to claim 2 is characterized in that: Perform image transformer. The specific method is as follows: (1) The transformer encoder consists of an image query sequence As input, the original image features Reshape into a 2D feature sequence Where (H, W) is the resolution of the original image features, C is the number of channels, D is the feature resolution after reshaping, and N = HW / D is the number of features after reshaping, which is also the effective input sequence length of the image transformer encoder. (2) Transformer uses a constant vector V in its application layer e , using formula (2), the reshaped features are output through trainable linear projection: The output of the transformer encoder is f i express. (3) The feature classification head is implemented by a hidden MLP layer during pre-training and a single linear layer during fine-tuning: Where MSA represents the multi-head self-attention layer, Z * represents a single-layer feature, MLP is a linear perceptron layer, LN is a fully connected layer, and f i Represents the output of the transformer encoder.

5. The three-dimensional target detection method based on multimodal voxel image feature fusion attention according to claim 4 is characterized in that: The voxel domain features and image domain features are fused in two domains, as follows: (1) The camera feature f i Query the camera with key and value C' q Perform deformation attention transformation, for a given 2D reference point In the camera domain, the feature f i A deformable mask with adaptive offset and weight is applied on the image, with mask offset Δp m and the mask weight Δw m : Δp m =PFN(C′ q ),Δw m =PFN(C′ q +V′ q ), In the formula, '+' represents element summation, and PFN represents feed-forward network; (2) Mask weight Δp m Determined by voxel query and camera query, given an offset Δp m and weight Δw m , attention value The calculation is as follows: Where h is the index of the attention head, k is the index of the sampling key, and K is the total number of sampling keys. and represents a learnable projection matrix; (3) Voxel query V' q and camera query C' q The transformations are further fused through the gated fusion mechanism, i.e., V' q and C' q Fusion is performed using different ratios given below: C' q '=q' q +V' q ×σ(Conv1(C' q +V'q)), (5) V” q =V' q +C' q ×σ(Conv2(C' q +V' q )), (6) In the formula, '×' represents element-wise multiplication, and σ(·) is the sigmoid function Conv1(·) and Conv2(·) represent different convolutional layers.

6. The three-dimensional target detection method based on multimodal voxel image feature fusion attention according to claim 5, characterized in that: A multi-scale deep quasi-prediction module is constructed to effectively encode and extract the feature information of different modalities to obtain voxel features and image space features, and the voxel features and image space features are cascaded to obtain joint features, as follows: (1) Perform 3D voxel feature extraction: (1.1) Given N 3D index vectors and the eigenvector The input voxels are composed of c in Indicates the number of input feature channels; (1.2) For each voxel feature x i ∈X, through the 3D submanifold convolution kernel K 3D (·) Encode the number of geometric feature voxels and extract features. Specifically, based on the corresponding 3D index vector, the geometric features are calculated from the non-empty voxels in the corresponding neighborhood: In the formula, represents the relevant neighborhood features obtained by index S, Used to represent nonlinear activation functions; (2) Perform 2D image spatial feature extraction: (2.1) by converting the 3D index into a set of grid points G(·) based on the voxelization parameters; (2.2) Then, the grid point G(·) is transformed back to the original coordinate system and data enhancement τ(·) is performed; (2.3) Based on the radar calibration, the camera parameters are projected into the 2D image plane through projection P(·), which can be expressed as follows: In the formula, Represents the 2D index quantity for each voxel feature By calculating the feature information from the corresponding field based on the corresponding 2D index in Represented by index vector The generated domain voxel features, K 2D Represents a 2D sub-flow convolution kernel; (3) Cascade the voxel features and image space features to obtain joint features: (3.1) and Combined, we get the corresponding eigenvector where c out Indicates the number of output feature channels:

7. The three-dimensional target detection method based on multimodal voxel image feature fusion attention according to claim 6, characterized in that: The deep feature prediction is performed based on the joint features, as follows: (1) Let the extracted joint features be P = (a i ,b i ,c i ), project the 3D feature points onto the 2D image and record the relevant points in the foreground mask. These points are used as depth reference points to provide the corresponding depth information, expressed as: Where u i and v i is the pixel coordinate corresponding to the point, d i It is the real depth information obtained. (2) Uniformly sample a subset from each instance mask of the 2D image Each element point in the subset will obtain the real depth information from the nearest reference point as the estimated depth; (3) Project the sampled subset of estimated depth back into 3D space as prediction points; (4) Connect the 2D image features with the sparse depth features corresponding to the image prediction, and fuse them with the convolutional layer to obtain the depth-aware semantic features C d , the predicted points of M generated by the same instance mask, the corresponding pixel semantic features will have different influences along the projection ray in 3D space. Therefore, a dynamic weight factor is calculated for each individual depth to adjust the influence of the semantic features. The process for a specific predicted point is described as: In the formula, Is a subset The Mth estimated depth of Represents a dynamic weight factor, which is used to adjust the prediction point of the corresponding semantic feature of the decoration 8. The three-dimensional target detection method based on multimodal voxel image feature fusion attention according to claim 7, characterized in that: A dynamic attention feature aggregation module is constructed to more accurately represent voxel features by using centroids, and to align different features across modalities in a fine-grained manner using learnable dynamic offsets and cross-attention mechanisms to generate cross-modal features, as follows: (1) The input is a set of non-empty voxel features and image features F I , where S v is the voxel index, V t is the non-empty voxel feature vector, N v is the number of non-empty voxels, and the feature F of each voxel is obtained by averaging the spatial positions of all points in the same voxel. v The centroid C e : In the formula, (x i ,y i ,z i ) is the position coordinate information, |p(V i )| is the voxel feature F v The number of points within Then use the camera projection matrix H to calculate the reference point C for the voxel point centroid i : C i =H·C e (13) Where H is the product of the camera's internal and external parameter matrices; (2) Assign voxel grid index to each voxel point centroid that needs to be calculated k is the number of 3D grid points, G is the grid feature channel, which is combined through the fully connected layer and the sigmoid layer. th The feature generates a pair of centroid grid weights The weights are added to the non-empty voxel features and image features to obtain the weighted voxel grid features. With deep image mesh features (3) Then, three connection layers are used to convert the voxel grid features into query Q and the deep image grid features into Key and Value. The inner product between each query Q and the key value Key is performed to obtain the attention affinity matrix between the voxel grid features and the deep image grid features. The matrix is ​​used to constant and aggregate the value Value. Finally, the aggregated fusion feature F ag Processed through a fully connected layer, the target category and positioning information are output through a standard 2D-3D cascade. The entire dynamic fusion process can be expressed as: Where W m and W' m is the learnable weight, M is the number of self-attention heads, K is the total number of sampling points, Δp mik and A mik They represent the sampling offset and attention weight of the kth sampling point in the mth attention head respectively.

9. The three-dimensional target detection method based on multimodal voxel image feature fusion attention according to claim 8, characterized in that: The overall loss function includes the classification loss L for objectness and category cls , the regression loss L for predicting the box reg , prediction loss L for dual-domain feature fusion dff , depth loss L for deep feature prediction depth , as follows: L total =λ cls L cls +λ reg L reg +λ dff L dff +λ dep L dep (17) In the formula, λ cls , reg , dff , dep Represent the regularization weight coefficients of classification loss, regression loss, prediction loss, and depth loss respectively; The depth loss of deep feature prediction is processed by focal loss: Where |ρ| is the 2D pixel feature region with valid depth labels, is the accurate lidar true value.

10. A three-dimensional object detection system based on multimodal voxel image feature fusion attention, characterized in that: include: The first construction unit is used to construct a dual-domain voxel spatial feature encoding module based on Transformer, which performs corresponding feature query encoding on voxel domain features and camera domain features respectively, and maximizes the use of different modal feature information to obtain the pre-selected box; The second construction unit is used to construct a multi-scale deep quasi-prediction module, effectively encode and extract different modal feature information to obtain voxel features and image space features, and cascade the voxel features and image space features to obtain joint features; A deep feature prediction unit, used to predict deep features based on joint features, and to enhance the degree of deep interaction between features of different modalities; The third building block is used to construct a dynamic attention feature aggregation module, which uses the centroid to more accurately represent the voxel features and uses a learnable dynamic offset and cross-attention mechanism to align different features across modalities in a fine-grained manner to generate cross-modal features; The training output unit is used to train the overall model constructed in the above steps using the overall loss function, and use the overall model to perform target detection.

Citation Information

Patent Citations

  • Image dimension raising method and three-dimensional target detection method

    CN115063539A

  • Three-dimensional target detection method and system based on point cloud-image multi-cross mixing and storage medium

    CN116664856A

  • Multi-modal three-dimensional target detection method and system for automatic driving and storage medium

    CN116798016A

  • Multi-modal three-dimensional target detection method

    CN121281025A

Cited By

  • Target detection method and device based on large model and electronic equipment

    CN119339066A

  • Uniformity detection method for aero-engine fan blade preform

    CN120070437A

  • Multi-dimensional spatial transformation self-sensing attention mechanism image processing method and application thereof

    CN120107617A

  • Multi-modal three-dimensional target detection method and system based on virtual point cloud improvement

    CN120411489A

  • Multi-mode-based body robot control method and device, electronic equipment, readable storage medium and program product

    CN121043159A