3D target detection method based on proposal-level multi-modal data fusion

Abstract

The invention discloses a proposal-level multi-modal data fusion 3D target detection method, and the method comprises the steps: firstly obtaining a multi-modal automatic driving data set, inputting a multi-view image and a point cloud into an original feature extractor, and extracting the features of the original point cloud and the features of the original image; secondly, the original point cloud features and the original image features are input into a proposal generation module for channel and spatial dimension selection, initial proposal of the output point cloud and the image, and initial features corresponding to the proposal; then, complementary sensing features are obtained through a target sensing feature sampling module, integrated features are obtained through an adaptive feature fusion module, feature proposal level fusion is carried out, and proposal features are output. And finally, inputting the proposed features into a 3D target detection head, outputting a target detection result, constructing a loss function, and carrying out reverse training. According to the method, the complementary features of the multi-modal data are fully mined, the detection result is collaboratively optimized, and finally robust sensing in a complex scene is realized.

Description

Technical Field

[0001] The present invention relates to the field of 3D object detection, and in particular to a 3D object detection method based on proposal-level multimodal data fusion. Background Art

[0002] With the rapid development of deep learning, sensor technologies, and other technologies, autonomous driving has made significant progress in recent years and is gradually becoming part of people's daily lives. However, in complex environments, such as those with restricted fields of view, autonomous vehicles struggle to effectively capture environmental information, and accident rates remain high. Improving the fusion perception capabilities of autonomous vehicles in complex environments is key to addressing these issues. However, existing perception algorithms lack accuracy and real-time performance, particularly in scenarios with obstructions, posing a significant threat to road safety.

[0003] Existing research on multimodal fusion perception based on point clouds and images in self-driving vehicle perception suffers from the problem of excessively high proportion of invalid areas (such as static backgrounds or empty areas) when generating proposals in the point cloud and image spaces, resulting in computational redundancy and drowning out key targets; feature sampling methods face the dual challenges of insufficient geometric structure capture and reference point offset; fusion strategies often use fixed weights and ignore the dynamic characteristics of sensor reliability as the environment changes (such as nighttime image degradation or sparse point clouds in haze), resulting in a sharp drop in fusion performance in noisy scenes, making it difficult for perception accuracy and robustness to meet the requirements of complex scenes. Summary of the Invention

[0004] To address these issues, this paper investigates single-vehicle fusion perception. Addressing the challenges of low candidate target quality, inaccurate feature sampling, and insufficient utilization of multimodal information in single-vehicle multimodal fusion perception in complex environments, this paper proposes a proposal-level point cloud and image fusion detection framework for target perception. This framework leverages the three key aspects of the proposal-level fusion framework, fully exploiting the complementary features of images and point clouds to enhance the perception capabilities of autonomous driving systems and improve overall perception performance. This effectively addresses the issue of insufficient modal information utilization in single-vehicle systems and provides important support for the intelligent level of autonomous driving systems.

[0005] The present invention solves the technical problem by comprising the following steps:

[0006] S1. Obtain a large-scale multimodal autonomous driving dataset and divide it into training set, validation set, and test set data.

[0007] S2. Input the multi-view image data I and point cloud data P in the training set into the corresponding feature extractors respectively to extract the original point cloud features and original image features. The original feature extractor is specifically designed as follows:

[0008] S21. In the image branch, input multi-view image data I, and obtain the original image features through the two-dimensional image feature extraction network of the multi-scale feature fusion mechanism Among them, (H1×W1) represents the size of the image feature map, C represents the number of feature channels, and N represents the number of cameras.

[0009] S22. In the point cloud branch, input the point cloud data P and divide it into regular voxels Among them, X V 、Y V and Z V Respectively represent the number of voxels along the X, Y and Z axes. Then, the 3D sparse voxel encoder is used to extract voxel features. Among them, X, Y, Z represent the size of the 3D voxel grid, and C represents the number of feature channels. Next, the voxel feature F is calculated along the Z axis. V Compress and finally get the basic point cloud BEV features Among them, H p and W p They represent the height and width of the basic point cloud BEV feature respectively, and c is the number of channels of the basic point cloud BEV feature.

[0010] S23. The basic point cloud BEV feature F l Projected to the 2D plane coordinate system, and the original deformable attention mechanism Deformable Attention is used to aggregate the original image features F I , get the original point cloud features By interacting with the original image features, the semantic characteristics of the original point cloud features are further enhanced, improving the quality of the point cloud branch proposal.

[0011] S3. The original point cloud features output from step S2 and the original image features F I The input is the Channel-Spatial Selective Proposal Extraction (CSPE) module, which selects channels and spatial dimensions, focusing on regions with significant channel and spatial features. This module includes the Channel-Spatial Selective Proposal Extraction (CSPE) branch, which extracts initial features for the initial proposal from the original point cloud features and the original image features, respectively. The specific design of the Channel-Spatial Selective Proposal Extraction (CSPE) module is as follows:

[0012] S31. In the proposal generation based on channel-space filtering - point cloud branch, the original point cloud features are Input the channel-space attention mechanism of the Convolutional Block Attention Module (CBAM) to calculate the attention scores of the channel dimension respectively. and the attention score of the spatial dimension The specific design is as follows:

[0013] S311. Original point cloud features Average pooling and maximum pooling operations are used respectively, and then passed through two 2D convolution hidden layers to obtain intermediate features. and Finally, the intermediate features and Add and input sigmoid activation function to get the attention score of channel dimension

[0014] S312. Original point cloud features The average pooling and maximum pooling operations are used respectively, and then splicing is performed along the channel dimension. After the feature dimension is reduced by 2D convolution, the sigmoid activation function is input for nonlinear numerical mapping to obtain the attention score of the spatial dimension. Finally, through element-wise multiplication with broadcast mechanism*, the global attention map is obtained:

[0015] S313. From the global attention map Extract the N with the highest score L Positions as initial proposals for point clouds At the same time, from the original point cloud features The initial features of the point cloud of these initial proposals are sampled in, Represents the jth point cloud initial proposal point In the original point cloud features The corresponding eigenvector in .

[0016] S32. In the channel-space filtering based proposal generation - image branch, the original image feature F I Input the convolution block attention module CBAM and calculate the attention scores of the channel dimensions in the same way as step S31 and the attention score of the spatial dimension Get the global attention map:

[0017] Then, from the global attention map Extract the N with the highest score C Positions as initial image proposals At the same time, from the original image feature F I The initial features of the image are sampled from these initial proposals in, Represents the initial proposed point of the i-th image In the original image feature F I The corresponding eigenvector in .

[0018] S4. Initial proposal Q of the point cloud output from step S3 lid , image initial proposal Q cam And the original point cloud features output by step S2 and the original image features F I Input target perception feature sampling module, which includes target perception feature sampling - point cloud branch and target perception feature sampling - image branch. By extracting and integrating effective complementary features, the expressiveness of the proposed features is improved. lid Get the prediction results, project them into the 2D image feature space, and extract the complementary perception features of the point cloud The specific design of the target perception feature sampling module is as follows:

[0019] S41. In the target perception feature sampling - point cloud branch, it is necessary to sample the 2D image features, and based on the point cloud initial proposal Q lid The prediction result is in 3D space, so it is necessary to project some 3D information into the 2D image feature space. The specific design of the target perception feature sampling - point cloud branch is as follows:

[0020] S411. Initial proposal Q of input point cloud lid , after a set of feedforward neural networks (FFN) containing two hidden layers for preliminary prediction, a set of initial 3D prediction results of point clouds are obtained. in, Represents the result set corresponding to the i-th prediction target, Represents the coordinates of the center point of the i-th predicted target point cloud, Represents the depth information of the i-th predicted target point cloud, represents the size of the i-th predicted target point cloud, Indicates the direction rotation angle of the i-th predicted target point cloud, Represents the speed of the i-th predicted target point cloud, point cloud prediction category probability distribution in, Represents the category probability distribution of the i-th prediction target, and its prediction probability is the highest Corresponding categories is the point cloud prediction category, n is the number of predicted targets, and m is the number of categories.

[0021] Based on the predicted point cloud depth information And the camera intrinsic parameter matrix The rotation matrix R and translation matrix t are used to transform the coordinates of the center point of the target point cloud Project the point cloud coordinate system to the 2D plane coordinate system to obtain the target 2D center point coordinates Represents the projection matrix from the point cloud to the camera image plane. The original image features are sampled by bilinear interpolation to obtain the image sampling features in Represents the initial proposal of the i-th point cloud Extracted linear features of images.

[0022] S412. Input point cloud depth information Target 2D center coordinates and target point cloud size Each target has 8 corner points, and each corner point needs to be projected into a 2D plane coordinate system. Plus the corresponding target point cloud size With offset set o i,j The dot product result is to get the actual coordinate P of the j-th corner point of the i-th target in the 3D space. i,j , and then the actual coordinate P in the 3D space i,j and point cloud depth information Projection matrix of the input point cloud to the camera image plane Calculate the homogeneous coordinate X in 2D space i,j ,Y i,j ,W i,j .

[0023] S413. Input the homogeneous coordinate X in the 2D space output in step S412 i,j ,Y i,j ,W i,j , the 2D plane coordinates of the jth corner point of the i-th target can be calculated (u i,j ,v i,j ),in Calculate the 2D size of the i-th target where u max and u min Respectively represent the maximum and minimum values ​​of the x-axis of all corner points of the i-th target in the 2D plane coordinate system, v max and v min They represent the maximum and minimum values ​​of the y-axis of all corner points of the i-th target in the 2D plane coordinate system.

[0024] S414. Input the target 2D center point coordinates output in step S411 and the item with the highest predicted probability And the target 2D size output in step S413 Combine into a new vector cat represents the concatenation operation. After passing through the multi-layer perceptron MLP consisting of two hidden layers, the target perception vector is obtained.

[0025] S415. Target perception vector Input a multi-layer perceptron (MLP) consisting of two hidden layers to calculate the offset of the target. and weights The sampling position passes through the target 2D center point coordinates and offset Image sampling features Select K sampling points on each attention head to extract the original image features. Perform deformable attention weighting in the point cloud to calculate the complementary perception features

[0026] S42. In the target perception feature sampling - image branch, the image complementary perception feature is calculated in a similar manner to step S41.

[0027] S5. The point cloud complementary perception features output in step S4 Complementary perceptual features of images And the initial features of the point cloud output in step S3 and initial image features The input is sent to the Adaptive Feature Fusion module AFF (Adaptive Feature Fusion), which dynamically adjusts the weights of each modal feature according to its importance to obtain integrated features, thereby achieving more effective feature fusion. The specific design of the Adaptive Feature Fusion module is as follows:

[0028] S51. Complementary perception features of point clouds Complementary perceptual features of images Perform linear transformation respectively, that is, multiply by the weight matrix W and add the bias vector b, and then input the output result into the layer normalization function to obtain the transformation complementary perceptual features In this way, the complementary perceptual features and the initial features are transformed into the same modal space for spatial alignment.

[0029] S52. Initial features and transform complementary perceptual features Perform independent linear transformations to obtain the initial weight vectors and the perceptual weight vector

[0030] S53. The initial features output from step S3 and the initial weight vector output in step S52 Perform element-by-element multiplication weighting to transform the complementary perceptual features output in step S51 and the perceptual weight vector output in step S52 Perform element-by-element multiplication weighting, add the features weighted by the initial weight vector and the perceptual weight vector, and after layer normalization, use the ReLU activation function to enhance the nonlinear expression ability to obtain the fusion feature Finally, the fusion features and initial features Perform residual connection to obtain integrated features It not only retains the information of the original features, but also incorporates the enhanced features, achieving comprehensive fusion and optimization of features.

[0031] S6. The integrated features output from step S5 Splicing along the channel dimension to obtain the joint feature F concat , input proposal-level fusion module, use multi-head self-attention mechanism to focus on joint features F concat Perform proposal-level fusion to enable deep interaction between different instances, ensure sufficient information exchange between features, and improve the model's ability to understand and represent complex relationships. Then, the features output by multiple attention heads are concatenated. Finally, the concatenated results are linearly transformed and then combined with the joint feature F. concat Perform residual connection to obtain the proposed feature F enhanced .

[0032] S7. Propose feature F enhanced The input 3D object detection head predicts the target position and category score and outputs the target detection result. The 3D object detection head consists of a regression head and a classification head.

[0033] S8. Calculate loss: The loss of this model is determined by the proposed focal loss Proposed L1 norm loss Initial focus loss and the initial L1 norm loss Combination.

[0034] The beneficial effects of the present invention are as follows: the model constructs a dynamic screening mechanism, filters invalid areas through spatial semantic perception, improves the probability of generating valid proposals, and introduces a cross-modal proposal interaction mechanism to collaboratively optimize the detection results, ultimately achieving robust perception in complex scenes. The channel-space dual filtering proposal generation module selects high-quality 3D proposals by combining the importance of channel features and spatial position information, and focuses on channels with significant semantic information to avoid the introduction of irrelevant or redundant information during the fusion process; the target-aware feature sampling module accurately captures target features by dynamically adjusting the sampling area, and improves the expressive power of the proposed features by extracting and integrating effective complementary features, making it particularly suitable for complex scenes; the adaptive fusion method dynamically adjusts the weights by quantifying the importance of each modality to achieve efficient fusion of point cloud and image features. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 This is an example diagram of a method according to an embodiment of the present invention;

[0036] Figure 2 Generate schematics for channel-space filtering-based proposals;

[0037] Figure 3 Schematic diagram of adaptive feature fusion. DETAILED DESCRIPTION

[0038] The following figures further illustrate a 3D object detection framework with proposal-level multimodal data fusion. Figure 1 As shown, the following steps are included:

[0039] S1. nuScenes (a large-scale multimodal autonomous driving dataset, nuScenes, collected in complex urban traffic environments in Boston and Singapore, covering 1,000 scenes, each lasting about 20 seconds, containing about 40,000 keyframes and 1.4 million object annotations) is used as training, validation, and test data.

[0040] S2. Input the multi-view image data I and point cloud data P into the corresponding feature extractors respectively. The original feature extraction module is specifically designed as follows:

[0041] S21. In the image branch, input multi-view image data I, use ResNet50 and feature pyramid network FPN to extract multi-scale features, and obtain the original image features Among them, (H1×W1) represents the size of the image feature map, C represents the number of feature channels, and N represents the number of cameras.

[0042] S22. In the point cloud branch, input the point cloud data P and divide it into regular voxels Among them, X V 、YV and Z V Respectively represent the number of voxels along the X, Y and Z axes. Then, the voxel features are extracted using the 3D sparse voxel encoder SECOND. Among them, X, Y, Z represent the size of the 3D voxel grid, and C represents the number of feature channels. Next, the voxel feature F is calculated along the Z axis. V Compress and finally get the basic point cloud BEV features Among them H p and W p They represent the height and width of the basic point cloud BEV feature respectively, and C is the number of channels of the basic point cloud BEV feature.

[0043] S23. Basic point cloud BEV features Projected to the 2D plane coordinate system, and the original deformable attention mechanism Deformable Attention is used to aggregate the original image features F I , get the original point cloud features By interacting with the original image features, the semantic features of the point cloud are further enhanced, and the quality of the point cloud branch proposal is improved. The specific formula is as follows:

[0044]

[0045] in, Represents the basic point cloud BEV feature F l The feature at position (h,w) is Indicates the depth at that location, Indicates the position after the coordinates of the center point of the target point cloud are projected from the point cloud coordinate system to the 2D plane coordinate system. Represents the projection matrix from the point cloud to the camera image plane, which is determined by the camera intrinsic parameter matrix And the rotation matrix R and the translation matrix t. Represents the original point cloud features exist Features at the location.

[0046] S3. The original point cloud features output from step S2 and the original image features F I Input the channel-spatial filtering based proposal generation module CSPE to select the channel and spatial dimensions, focusing on the areas with significant channel and spatial features. This module contains the channel-spatial filtering based proposal generation - point cloud branch and the channel-spatial filtering based proposal generation - image branch, which extract the initial features of the initial proposal from the original point cloud features and the original image features respectively. Figure 2As shown in Figure 2, the specific design of the proposal generation module based on channel-space filtering is as follows:

[0047] S31. In the proposal generation based on channel-space filtering - point cloud branch, the original point cloud features are Input the channel-spatial attention mechanism of the convolutional block attention module CBAM and calculate the attention scores of the channel dimension respectively and the attention score of the spatial dimension The specific design is as follows:

[0048] S311. Original point cloud features Average pooling and maximum pooling operations are used respectively, and then passed through two 2D convolution hidden layers to obtain intermediate features. and Finally, the intermediate features and Add and input sigmoid activation function to get the attention score of channel dimension The specific formula is as follows:

[0049]

[0050] Among them, φ a and φ m Represent the average pooling and maximum pooling operations respectively, σ represents the sigmoid activation function, Multilayer perceptron (MLP) consisting of 2D convolution and ReLU activation function, K represents the number of target categories, and + represents the addition of elements at corresponding positions.

[0051] S312. Original point cloud features The average pooling and maximum pooling operations are used respectively, and then splicing is performed along the channel dimension. After the feature dimension is reduced by 2D convolution, the sigmoid activation function is input for nonlinear numerical mapping to obtain the attention score of the spatial dimension. Finally, through element-wise multiplication with broadcast mechanism*, the global attention map is obtained: The specific formula is as follows:

[0052]

[0053] Among them, φ a and φ m Represent the average pooling and maximum pooling operations respectively, σ represents the sigmoid activation function, and [;] represents the splicing operation along the channel dimension.

[0054] S313. From the global attention map Extract the N with the highest score L locations as initial proposals for the point cloud At the same time, from the original point cloud features The initial features of the point cloud of these initial proposals are sampled in, represents the jth BEV proposal point In the original point cloud features The corresponding eigenvector in . The specific formula is as follows:

[0055]

[0056] in, is a selection matrix based on channels and spaces, is a threshold selection function for obtaining the Select the top N L A maximum value is proposed.

[0057] S32. In the channel-space filtering based proposal generation - image branch, the original image features Input the convolution block attention module CBAM and calculate the attention scores of the channel dimensions in the same way as step S31 and the attention score of the spatial dimension Get the global attention map: That is: for the original image feature F I The average pooling and maximum pooling operations are respectively used, and then the intermediate features are obtained by passing them through a multi-layer perceptron MLP composed of two 2D convolutional hidden layers. and Finally, the intermediate features and Add and input sigmoid activation function to get the attention score of channel dimension For the original image feature F I The average pooling and maximum pooling operations are used respectively, and then splicing is performed along the channel dimension. After the feature dimension is reduced by 2D convolution, the sigmoid activation function is input for nonlinear numerical mapping to obtain the attention score of the spatial dimension. Finally, through element-wise multiplication with broadcast mechanism*, the global attention map is obtained:

[0058] Then, from the global attention map Extract the N with the highest score C locations as initial proposals for the image At the same time, from the original image features The initial features of the image are sampled from these initial proposals in, Represents the i-th image proposal point In the original image feature F I The corresponding eigenvector in .

[0059] S4. Initial proposal of the point cloud output from step S3 Image initial proposal And the original point cloud features output by step S2 and original image features Input target-aware feature sampling module, which includes target-aware feature sampling - point cloud branch and target-aware feature sampling - image branch. By extracting and integrating effective complementary features, the expressiveness of the proposed features is improved. The specific design of the target-aware feature sampling module is as follows:

[0060] S41. In the target perception feature sampling - point cloud branch, it is necessary to sample the 2D image features, and based on the point cloud initial proposal Q lid The prediction result is in 3D space, so it is necessary to project some 3D information into the 2D image feature space. The specific design of the target perception feature sampling - point cloud branch is as follows:

[0061] S411. Initial proposal Q of input point cloud lid , a set of initial 3D prediction results of point cloud are obtained through a set of feedforward neural networks FFN containing two hidden layers. in, Represents the result set corresponding to the i-th prediction target, Represents the coordinates of the center point of the i-th predicted target point cloud, Represents the depth information of the i-th predicted target point cloud, represents the size of the i-th predicted target point cloud, Indicates the direction rotation angle of the i-th predicted target point cloud, Represents the speed of the i-th predicted target point cloud, point cloud prediction category probability distribution in, Represents the category probability distribution of the i-th prediction target, and its prediction probability is the highest Corresponding categories is the point cloud prediction category, n is the number of predicted targets, and m is the number of categories. Based on the predicted point cloud depth information And the camera intrinsic parameter matrix The rotation matrix R and translation matrix t are used to transform the coordinates of the center point of the target point cloud Project the point cloud coordinate system to the 2D plane coordinate system to obtain the target 2D center point coordinates Represents the projection matrix from the point cloud to the camera image plane. The original image features are sampled by bilinear interpolation to obtain the image sampling features in Represents the initial proposal of the i-th point cloud Extracted linear features of images.

[0062] S412. Input point cloud depth information Target 2D center coordinates and target point cloud size Each target has 8 corner points, and each corner point needs to be projected into a 2D plane coordinate system. Plus the corresponding target point cloud size With offset set o i,j The dot product result is to get the actual coordinate P of the j-th corner point of the i-th target in the 3D space. i,j , and then the actual coordinate P in the 3D space i,j and point cloud depth information Projection matrix of the input point cloud to the camera image plane Calculate the homogeneous coordinate X in 2D space i,j ,Y i,j ,W i,j , the specific formula is as follows:

[0063] o i,j =[x i,j ,y j,j ,z i,j ],x i,j ,y i,j ,z i,j ∈{-0.5,0.5}

[0064]

[0065] Among them, i,j , j∈[1,8] represents the offset x of the jth corner point of the i-th target on the x-axis, y-axis and z-axis i,j ,y i,j and z i,j A collection of P i,j represents the actual coordinates of the jth corner point of the i-th target in 3D space, is the depth information, X i,j ,Y i,j ,W i,j is a homogeneous coordinate in 2D space, X i,j ,Y i,j Indicates the position of the target on the 2D plane, W i,j Represents the weight of the homogeneous coordinate.

[0066] S413. Input the homogeneous coordinate X in the 2D space output in step S412 i,j ,Y i,j ,W i,j , the 2D plane coordinates of the jth corner point of the i-th target can be calculated (u i,j ,v i,j ),in Calculate the 2D size of the i-th target where u max and u min Respectively represent the maximum and minimum values ​​of the x-axis of all corner points of the i-th target in the 2D plane coordinate system, v max and v min They represent the maximum and minimum values ​​of the y-axis of all corner points of the i-th target in the 2D plane coordinate system.

[0067] S414. Input the target 2D center point coordinates output in step S411 and the item with the highest predicted probability And the target 2D size output in step S413 Combine into a new vector cat represents the concatenation operation. After passing through the multi-layer perceptron MLP consisting of two hidden layers, the target perception vector is obtained.

[0068] S415. Target perception vector Input a multi-layer perceptron (MLP) consisting of two hidden layers to calculate the offset of the target. and weights The sampling position passes through the target 2D center point coordinates and offset Image sampling features Select K sampling points on each attention head to extract the original image features. Attention weighting is performed in the point cloud to calculate the complementary perception features Extract key features from multi-view and multi-scale image feature maps. The specific formula is as follows:

[0069]

[0070] Where DfAttention represents a deformable attention mechanism that adaptively adjusts feature weights based on the relationship between point cloud features and image features. M represents the number of attention heads, K represents the number of keys sampled in each attention head, and W m and W k is a learnable weight matrix.

[0071] S42. In the target perception feature sampling - image branch, the image complementary perception feature is calculated in a similar manner to step S41. That is: input image initial proposal Q cam After preliminary prediction by the two hidden layers of the feedforward neural network FFN, a set of initial image prediction results are obtained. in, Represents the result set corresponding to the i-th prediction target, the probability distribution of image prediction category in, Represents the category probability distribution of the i-th prediction target, and its prediction probability is the highest Corresponding categories is the point cloud prediction category, n represents the number of predicted targets. Project from the camera perspective to the point cloud plane to obtain the coordinates of the target point cloud center point represents the projection matrix from the camera image plane to the point cloud plane, where Represents the camera intrinsic parameter matrix, R represents the rotation matrix, t represents the translation matrix, and then the bilinear interpolation method is used to sample the original point cloud features to obtain the point cloud sampling features in represents the initial proposal of the i-th image The extracted linear features of the image. The coordinates of the center point of the i-th target point cloud i-th target image size and the item with the highest predicted probability Combine into a new vector After passing through the multi-layer perceptron MLP consisting of two hidden layers, the target perception vector is obtained Sampling features from point cloud and target perception vector Input the multi-layer perceptron MLP consisting of two hidden layers to calculate the offset of the query point and weights The sampling position passes through the coordinates of the center point of the target point cloud and offset Add together to get. Select K sampling points on each attention head for attention weighting and calculate the complementary perceptual features of the image The specific formula is as follows:

[0072]

[0073]

[0074] Where DfAttention represents a deformable attention mechanism that adaptively adjusts feature weights based on the relationship between point cloud features and image features. M represents the number of attention heads, K represents the number of keys sampled in each attention head, and W m and W k is a learnable weight matrix.

[0075] S5. The point cloud complementary perception features output in step S4 Complementary perceptual features of images And the initial features of the point cloud output in step S3 and initial image features The input is sent to the adaptive feature fusion module AFF to dynamically adjust the weight of each modal feature according to its importance, thereby achieving more effective feature fusion. Figure 3 As shown in Figure 2, the specific design of the adaptive feature fusion module is as follows:

[0076] S51. Complementary perception features of point clouds Complementary perceptual features of images Perform a linear transformation, that is, multiply by the weight matrix W and add the bias vector b, and then input the output result into the layer normalization function to obtain the transformed complementary perceptual features In this way, the complementary perception features and the initial features are converted into the same modal space and spatially aligned. The specific formula is as follows:

[0077]

[0078] Among them, W is the weight matrix of the linear mapping, LayerNorm represents the layer normalization operation, and b is the bias term of the linear mapping.

[0079] S52. Initial features and transform complementary perceptual features Perform independent linear transformations to obtain the initial weight vectors and the perceptual weight vector The specific formula is as follows:

[0080]

[0081] S53. The initial features output from step S3 and the initial weight vector output in step S52 Perform element-by-element multiplication weighting to transform the complementary perceptual features output in step S51 and the perceptual weight vector output in step S52 Perform element-by-element multiplication weighting, add the features weighted by the initial weight vector and the perceptual weight vector, and after layer normalization, use the ReLU activation function to enhance the nonlinear expression ability to obtain the fusion feature Finally, the fusion features and initial features Perform residual connection to obtain integrated features It not only retains the information of the original features, but also incorporates the enhanced features, achieving comprehensive fusion and optimization of features. The specific formula is as follows:

[0082]

[0083] Among them, ⊙ represents element-wise multiplication and LayerNorm represents layer normalization.

[0084] S6. The integrated features output from step S5 Splicing along the channel dimension to obtain the joint feature F concat , input proposal-level fusion module, use multi-head self-attention mechanism to focus on joint features F concat Perform proposal-level fusion to enable deep interaction between different instances, ensure sufficient information exchange between features, and improve the model's ability to understand and represent complex relationships. Then, the features output by multiple attention heads are concatenated. Finally, the concatenated results are linearly transformed and then combined with the joint feature F. concat Perform residual connection to obtain the proposed feature F enhanced The specific formula is as follows:

[0085]

[0086] F enhanced =cat(Atten1,Atten2,...,Atten h )W o +F concat

[0087] Where h represents the number of heads in the multi-head attention, W o Is the weight matrix of the linear transformation. Since the proposal Q and the initial feature are semantically related, so the joint feature F concat The deep interaction is the fusion of proposal level.

[0088] S7. Propose feature F enhanced Input the 3D object detection head to predict the target position and category score. The 3D object detection head consists of a regression head and a classification head. The specific design is as follows:

[0089] S71. Propose feature F enhancedInput to the regression head, which consists of 5 feed-forward neural networks with two hidden layers l ,l∈[1,5], each FFN outputs a prediction result. Each hidden layer consists of a fully connected layer FC i () and ReLU activation function ReLU(), where FC i () represents the fully connected layer in the i-th hidden layer. A set of prediction results is obtained, including the set of predicted target center point coordinates Predict target depth information set Predicted target size set Predicted target direction rotation angle set Predicted target speed set Where n represents the number of predicted targets. The corresponding target information is combined into a new set of prediction results in, Represents the result set corresponding to the predicted i-th target. The specific formula is as follows:

[0090] FFN l (x) = FC output (ReLU(FC2(ReLU(FC1(x)))))

[0091] pred_c=FFN1(F enhanced )

[0092] pred_d=FFN2(F enhanced )

[0093] pred_s=FFN3(F enhanced )

[0094] pred_r=FFN4(F enhanced )

[0095] pred_v=FFN5(F enganced )

[0096] Among them, FC output () represents the output layer of the feed-forward neural network FFN, which maps the feature dimension after activation by the FC2() layer and reLU() to the result quantity dimension.

[0097] S72. Propose feature F enhanced Input to the classification head, which consists of a feedforward neural network FFN consisting of three hidden layers and a Softmax layer Sofmax(), each hidden layer consists of a fully connected layer FC i () and ReLU activation function ReLU(). Among them, FC i() represents the fully connected layer in the i-th hidden layer. The predicted category probability distribution is calculated in, Represents the probability distribution of the category corresponding to the predicted i-th target, with the highest predicted probability Corresponding categories is the predicted category, n represents the number of predicted targets, and m represents the number of categories. The specific formula is as follows:

[0098] t i =Softmax(FC output (ReLU(FC3(ReLU(FC2(ReLU(FC1(F enhanced ))))))))

[0099] Among them, FC output () represents the output layer of the feedforward neural network FFN, which maps the feature dimension after the FC3() layer and ReLU() activation to the category quantity dimension.

[0100] S8. Calculate loss: The loss of this model is determined by the proposed focal loss Proposed L1 norm loss Initial focus loss and the initial L1 norm loss The loss function is specifically designed as follows:

[0101] S81. The predicted category probability distribution output in step S6 Point cloud prediction category probability distribution output in step S411 And the image prediction category probability distribution output in step S42 Input into the focal loss formula respectively (FocalLoss), the proposed focal loss is calculated Initial point cloud focal loss and initial image focus loss

[0102]

[0103] Among them, n represents the number of predicted targets, m represents the number of categories, and η v Represents the balance factor of the vth category, which is used to deal with the problem of sample imbalance. μ represents the adjustment factor, which is used to adjust the weight of the loss of easy-to-classify samples and difficult-to-classify samples. In the formula, g v represents the predicted probability of the vth category, ξ1, ξ2, and ξ3 are hyperparameters used to balance the effects of the three focal losses.

[0104] S82. Set the prediction results output in step S6 The initial 3D prediction result of the point cloud output in step S411 and the initial prediction result of the image output in step S42 Input them into the L1 norm formula respectively to calculate the proposed L1 norm loss Initial point cloud L1 norm loss and initial image L1 norm loss

[0105]

[0106] Where n represents the number of predicted targets, λ x and λ s is a weight parameter used to balance the loss of center coordinates and size, In the formula, x i and Represent the center coordinates of the real target and the predicted target respectively, s i and They represent the sizes of the true target and the predicted target respectively, and δ1, δ2, and δ3 are hyperparameters that adjust the contribution of the three regression losses to the total loss.

[0107] S9. During the model training phase, the total model loss is calculated on the training set. The total model loss is the proposed focus loss output in step S71. Initial point cloud focal loss and initial image focus loss and the proposed L1 norm loss output from step S72 Initial point cloud L1 norm loss and initial image L1 norm loss The model parameters are iteratively optimized by the adaptive moment estimation Adam method. The specific formula is as follows:

[0108]

[0109] in, Represents the total model loss, α and β are hyperparameters used to balance the impact of classification and regression losses in the overall loss function.

[0110] S10. In the inference phase of the model, the original feature extraction module, the channel-space filtering-based proposal generation module, the target perception feature sampling module, the adaptive feature fusion module, the proposal level fusion module and the 3D target detection head are used to obtain the final 3D target detection results, including the predicted target center point coordinates, predicted target depth information, predicted target size, predicted target direction rotation angle, predicted target speed, and the predicted highest probability item. And the corresponding categories First, the maximum predicted probability index value j is obtained through the maximum index function argmax(), and the final predicted category is obtained according to the index value j of the maximum predicted probability.

[0111] Example:

[0112] The steps of this embodiment are the same as those of the specific implementation method, and will not be repeated here. The implementation process and results are shown below.

[0113] We comprehensively compare the performance of our approach with other state-of-the-art solutions on the nuScenes test set, including BEVDet: High-Performance Multi-Camera Bird's-Eye 3D Object Detection, DETR3D: 3D Object Detection from Multi-View Images via 3D-2D Queries, BEVFormer: Learning Bird's-Eye Representations from Multi-Camera Images via Spatiotemporal Transformation, CenterPoint: Center-Based 3D Object Detection and Tracking, UVTR-L: Unifying Voxel-Based Representations for 3D Object Detection with Transformers, TransFusion-L: Robust LiDAR-Camera Fusion for Transformer 3D Object Detection, DeepInteraction: Modal Interaction-Based 3D Object Detection, BEVFusion(MIT): Multi-Task Multi-Sensor Fusion with Unified Bird's-Eye Representation, and SparseFusion: Fusing Multimodal Sparse Representations for Multi-Sensor 3D Object Detection. Detailed experimental results demonstrate the significant advantages of our proposed approach across multiple performance metrics. As shown in Table 1, on the nuScenes test set, our method achieves state-of-the-art performance with an NDS of 74.0% and a mAP of 72.1%, surpassing previous methods. Here, C represents the image modality and L represents the point cloud modality. Specifically, compared to the proposal-level fusion baseline, our method improves NDS by 0.2% and mAP by 0.1%, respectively. Although the improvement is small, this result demonstrates that our method further optimizes detection performance based on proposal-level fusion, particularly demonstrating greater stability when handling complex scenes and multimodal data.

[0114] Compared to BEVFusion, a leading-edge feature-level fusion solution, this patent achieves a 1.1% improvement in NDS and a 1.9% improvement in mAP. This significant performance improvement not only validates the effectiveness of our approach but also further highlights its superiority in multimodal data fusion and complex scene understanding. By introducing dual-filter proposal screening, target-aware feature sampling, and an adaptive fusion mechanism, this patented approach more effectively captures the complementary information of data from different modalities, resulting in a significant improvement in detection accuracy.

[0115] Table 1 Comparison with SOTA (State-Of-The-Art) methods on the nuScenes test set

[0116]

Claims

1. A 3D object detection method based on proposal-level multimodal data fusion, characterized in that: The following steps are involved: S1. Obtain a multimodal autonomous driving dataset and divide it into training, validation, and test sets. S2. Input the multi-view image data and point cloud data in the training set into the corresponding original feature extractors to extract the original point cloud features and original image features; S3. The original point cloud features and original image features are input into the proposal generation module based on channel-spatial filtering to select the channel and spatial dimension, and the output is the initial point cloud proposal, initial image proposal, initial point cloud features, and initial image features. S4. Input the point cloud initial proposal, image initial proposal, original point cloud features, and original image features into the target perception feature sampling module to obtain complementary perception features. After passing through the adaptive feature fusion module, the integrated features are obtained. Finally, feature proposal-level fusion is performed to output the proposed features. S5. Input the proposed features into the 3D object detection head composed of the regression head and the classification head, predict the object position and category score, and output the object detection result; S6. Construct a loss function, perform reverse training based on the training set, and complete verification and testing using the validation set and test set.

2. The 3D object detection method based on proposal-level multimodal data fusion according to claim 1, characterized in that: The specific implementation process of step S2 is as follows: Input multi-view image data I, and obtain the original image features F through the two-dimensional image feature extraction network of the multi-scale feature fusion mechanism I ; Input point cloud data P, divide it into regular voxels V, and use 3D sparse voxel encoder to extract voxel features F V ; Next, the voxel feature F is aligned along the Z axis V Compress and finally get the basic point cloud BEV feature F l ; The feature F l Project it into a 2D plane coordinate system and use a deformable attention mechanism to aggregate the original image features F I , get the original point cloud features 3. The 3D object detection method based on proposal-level multimodal data fusion according to claim 2, characterized in that: The channel-space filtering-based proposal generation module extracts initial features of the initial proposal from the original point cloud features and the original image features. The specific implementation process is as follows: S31. The original point cloud features Input the channel-spatial attention mechanism of the convolutional block attention module CBAM and calculate the attention scores of the channel dimension respectively and the attention score of the spatial dimension Get the global attention map, obtain the initial point cloud proposal and the corresponding initial point cloud features; S32. In the image branch, the original image feature F I Input the convolution block attention module CBAM and calculate the attention scores of the channel dimensions in the same way as step S31 and the attention score of the spatial dimension Get the global attention map From the global attention map Extract the N with the highest score C The positions are used as the initial image proposal Q cam , at the same time, from the original image feature F I The initial features of the image are sampled from these initial proposals 4. The 3D object detection method based on proposal-level multimodal data fusion according to claim 3, characterized in that: The step S31 is specifically implemented as follows: S311. Original point cloud features Average pooling and maximum pooling operations are used respectively, and then passed through two 2D convolution hidden layers to obtain intermediate features. and Add the two intermediate features and input them into the sigmoid activation function to get the attention score of the channel dimension S312. Original point cloud features The average pooling and maximum pooling operations are used respectively, and then splicing is performed along the channel dimension. After the feature dimension is reduced by 2D convolution, the sigmoid activation function is input for nonlinear numerical mapping to obtain the attention score of the spatial dimension. Through element-by-element multiplication with broadcasting mechanism, we get the global attention map: S313. From the global attention map Extract the N with the highest score L Positions as initial proposals for point clouds At the same time, from the original point cloud features The initial features of the point cloud of these initial proposals are sampled in, Represents the jth point cloud initial proposal point In the original point cloud features The corresponding eigenvector in .

5. The 3D object detection method based on proposal-level multimodal data fusion according to claim 4, characterized in that: The complementary perception features include point cloud complementary perception features and image complementary perception features, and the specific acquisition process is as follows: Based on the point cloud initial proposal Q lid Get the prediction results, project them into the 2D image feature space, and extract the complementary perception features of the point cloud The details are as follows: Initial proposal Q of the input point cloud lid , a set of point cloud initial 3D prediction results res is obtained through the feedforward neural network lid , where the elements Represents the result set corresponding to the i-th prediction target, Represents the coordinates of the center point of the i-th predicted target point cloud, Represents the depth information of the i-th predicted target point cloud, represents the size of the i-th predicted target point cloud, Indicates the direction rotation angle of the i-th predicted target point cloud, Represents the speed of the i-th predicted target point cloud, point cloud prediction category probability distribution Represents the category probability distribution of the i-th prediction target, and its prediction probability is the highest Corresponding categories is the point cloud prediction category, n represents the number of predicted targets, and m represents the number of categories; Based on point cloud depth information As well as the camera intrinsic matrix, rotation matrix and translation matrix, the coordinates of the center point of the target point cloud Project the point cloud coordinate system to the 2D plane coordinate system to obtain the target 2D center point coordinates; sample the original image features through bilinear interpolation to obtain image sampling features; By adding the target 2D center point coordinates to the corresponding target point cloud size With offset set o i,j The dot product result is to get the actual coordinate P of the j-th corner point of the i-th target in the 3D space. i,j , then the actual coordinate P i,j and point cloud depth information Projection matrix of the input point cloud to the camera image plane Calculate the homogeneous coordinate X in 2D space i,j ,Y i,j ,W i,j ; According to the input homogeneous coordinates, calculate the 2D plane coordinates of the jth corner point of the i-th target (u i,j ,v i,j ), and the 2D size of the i-th target Input the center point coordinates and the item with the highest predicted probability and the target 2D size Splice into vector The target perception vector is obtained through the multi-layer perceptron, and the target perception vector is input into the multi-layer perceptron to calculate the offset of the target and weights The sampling position passes through the target 2D center point coordinates and offset Add together; the image sampling feature selects K sampling points on each attention head in the original image feature F I Perform deformable attention weighting in the point cloud to calculate the complementary perception features Input image initial proposal Q cam , the image complementary perception features are calculated in the same way as the point cloud complementary perception features 6. The 3D object detection method based on proposal-level multimodal data fusion according to claim 5, characterized in that: The process of obtaining the integrated features is as follows: Complementary perception features of point clouds Complementary perceptual features of images Perform linear transformations respectively to obtain the transformation complementary perceptual features The initial features and transform complementary perceptual features Perform independent linear transformations to obtain the initial weight vectors and the perceptual weight vector The initial features and the initial weight vector Perform element-by-element multiplication weighting to transform the complementary perceptual features and step-aware weight vector Perform element-by-element multiplication weighting, add the features weighted by the initial weight vector and the perceptual weight vector respectively, and obtain the fused features through the activation function after layer normalization. The fused features and the initial features are residually connected to obtain the integrated features.

7. The 3D object detection method based on proposal-level multimodal data fusion according to claim 6, characterized in that: The feature proposal level fusion output proposal features are specifically as follows: Integrate features Splicing along the channel dimension to obtain the joint feature F concat , input proposal-level fusion module, use multi-head self-attention mechanism to focus on joint features F concat Perform proposal-level fusion, then concatenate the features output by multiple attention heads. Finally, the concatenated results are linearly transformed and then combined with the joint feature F concat Perform residual connection to obtain the proposed feature F enhanced .

Images