Multi-modal three-dimensional target detection method and system for automatic driving, and storage medium

Abstract

The application provides a kind of multi-modal three-dimensional target detection method, system and storage medium for automatic driving, comprising the following steps: step one, feature information acquisition;Step two: generate initial candidate box through the double fusion feature region proposal network of Transformer;Step three, depth feature information enhancement: through the depth information completion mechanism, dense depth information is predicted and feature semantic information is extracted;Step four, multi-modal feature fusion: adopt dynamic cross attention mechanism to obtain the correlation between different modalities, and predict the relevant weight, then obtain the fusion feature by weighting the features.The beneficial effects of the present application are: the application combines the advantages of laser radar and camera two kinds of sensors, realizes the dynamic multi-target detection technology in the field of automatic driving, can accurately identify and locate multiple targets such as vehicles, pedestrians, cyclists, etc., and the technology takes into account the real-time performance, can be applied in actual scenarios.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving technology, and in particular to a multimodal three-dimensional target detection method, system, and storage medium for autonomous driving. Background Technology

[0002] In current autonomous driving research, 3D object detection technology has become an indispensable part, primarily used for identifying and locating target objects in 3D space. Existing 3D object detection methods mainly utilize LiDAR and cameras to perceive the surrounding environment. Both LiDAR and cameras provide useful information for 3D object detection. These two sensors have significantly different characteristics. LiDAR can provide accurate 3D measurement data at close range, but the point cloud becomes sparse at long range, which reduces the system's ability to accurately detect distant objects. Cameras can provide rich color and texture features of object appearance, but are not a good source of depth information. The complementarity of their characteristics has made the fusion of LiDAR and camera sensors a topic of interest in recent years. This combination has been proven to be applicable in many application scenarios, including autonomous driving, and can achieve high-precision 3D object detection. To fully utilize the different information provided by multiple sensors for 3D object detection, effectively combining two features in different coordinate domains is a challenge. Therefore, reducing misalignment caused by different domains and improving the correlation between modal data are key during fusion. Summary of the Invention

[0003] This invention provides a multimodal three-dimensional target detection method for autonomous driving, comprising the following steps:

[0004] Step 1, Feature Information Acquisition: Using the original point cloud and color image as input, radar point cloud features and image features are obtained respectively through a standard feature extraction backbone network;

[0005] Step 2: Generate initial candidate boxes using the Transformer dual-fusion feature region proposal network;

[0006] Step 3, Deep Feature Information Enhancement: Through a deep information completion mechanism, predict dense deep information and extract feature semantic information;

[0007] Step 4, Multimodal Feature Fusion: A dynamic cross-attention mechanism is used to obtain the correlation between different modalities and predict the relevant weights. Then, the features are weighted to obtain the fused features.

[0008] The present invention provides a multimodal three-dimensional target detection system for autonomous driving, characterized in that it includes: a memory, a processor, and a computer program stored in the memory, wherein the computer program is configured to implement the steps of the multimodal three-dimensional target detection method of the present invention when called by the processor.

[0009] This invention provides a computer-readable storage medium, characterized in that: the computer-readable storage medium stores a computer program, which is configured to implement the steps of the multimodal three-dimensional target detection method of this invention when called by a processor.

[0010] The beneficial effects of this invention are: This invention integrates the advantages of both lidar and camera sensors to achieve dynamic multi-target detection technology in the field of autonomous driving. It can accurately identify and locate multiple types of targets such as vehicles, pedestrians, and cyclists. This technology takes into account real-time performance and can be applied in real-world scenarios. Attached Figure Description

[0011] Figure 1 This is a schematic diagram of the Transformer Fusion Information Augmentation Network principle framework;

[0012] Figure 2 This is a diagram of the Transformer dual-fusion feature region proposal network structure;

[0013] Figure 3 This is a schematic diagram of deep feature completion;

[0014] Figure 4 This is a schematic diagram of the multimodal feature cross-attention module. Detailed Implementation

[0015] This invention discloses a multimodal three-dimensional target detection method for autonomous driving, such as... Figure 1 As shown, constructing the Transformer Fusion Information Enhancement Network (TFIENet) can effectively reduce the misalignment phenomenon during multimodal data fusion, improve the correlation between modal data, effectively combine two features in different coordinate domains, and enhance the accuracy of 3D target detection.

[0016] The multimodal 3D target detection method includes the following steps:

[0017] Step 1, Feature Information Acquisition: The network takes the original point cloud and color image as input, and obtains radar point cloud features and image features through a standard feature extraction backbone network, respectively.

[0018] Step 2: Generate initial candidate boxes using the Transformer dual-fusion feature region proposal network. For example... Figure 2 As shown, it specifically includes:

[0019] Step A1, performing dual feature lookup: This refers to obtaining more accurate features through both camera feature lookup and voxel feature lookup. When using LiDAR features, they are voxelized to obtain... 1 non-empty voxels, of which the number It varies depending on the distribution of the input lidar features. Voxel query. Matching with corresponding non-empty voxels improves voxel-domain features. The center point of each non-empty voxel is projected onto an image pixel in the camera domain, which is then queried sequentially by the camera. Corresponding allocation, and used to improve image domain features. This represents the query for the 1st to the kth cameras. This represents the query for the first to kth voxels in the prime field.

[0020] Step A2, 3D Local Self-Attention: The self-attention layer encodes voxel queries one by one by establishing spatial relationships between them. To reduce the computational burden of global self-attention, this invention designs three-dimensional local self-attention to reduce the attention range within local regions. Specifically, by applying the farthest point sampling algorithm to the center points of non-empty voxels, they are clustered into local regions, and within a fixed radius around the centroid of each local region, the attention range is found... Individual element. Let... and They are assigned to the center of mass. A set of query features and 3D location. Then, the 3D-LSA layer performs self-focus. Indicates the location of the front-end network. It is a positional encoding function. (Through...) For two three-dimensional coordinates The interpolation is used for encoding. If the number of non-empty voxels in the radius is greater than... Then calculate Individual element. With a group Voxel queries associated with individual voxels are encoded separately by a self-attention layer. After passing through multiple self-attention layers, the input voxel queries... Updated to .

[0021] Step A3, perform dual-domain feature attention fusion: Let It is a non-empty voxel. 3D reference point at the center of each non-empty voxel , These are the coordinates of the center point of the k-th non-empty voxel. Projected onto the camera domain. And quantize it on the mesh to determine the corresponding 2D reference point. x and y are the coordinates of the corresponding 2D reference point. Camera query. From 2D reference point Indicated camera domain features Initialization is performed. Voxel queries are obtained through 3D local self-attention. Depth-aware location encoding was first applied to dual-feature queries; according to the formula and Here, z represents the depth value, and i and d are the index and dimension of the query vector, respectively. Depth-aware positional encoding is added to the dual-feature query. Dual-domain feature fusion and attention decoding are performed on multiple attention layers for dual-query processing. and First, z represents the depth value, and i and d are the index and dimension of the query vector, respectively. for and Query camera Perform deformation with attention to transformation; for a given 2D reference point Features in the camera domain A deformable mask with adaptive offset and weights is applied. Mask offset and mask weights From the formula This is given. Therefore, the feature fusion effect can be improved by using attention weights based on voxel region and camera region features. Given an offset... and weight Attention value Calculation by formula Given. Among them, It is the index of the attention header. For sampling Perform indexing. It is sampling The total number. and This represents the learnable projection matrix.

[0022] Step 3, Deep Feature Enhancement: Through a deep information completion mechanism, dense deep information is predicted and semantic information is extracted, thereby improving feature representation capabilities. Therefore, two sub-tasks need to be completed: how to predict dense deep feature information and how to effectively extract the features. Specifically, this includes:

[0023] Step B1, Depth Information Completion: The foreground points in the candidate boxes constitute features describing target cues, but due to the sparsity of radar point clouds, depth feature information is often incomplete. Feature representation is enhanced by completing dense depth feature information. Specifically, taking sparse depth and RGB images as input, the output is a completed dense depth map. To utilize effective information and reduce computation, sparse convolution and standard convolution are used to process the input depth map and RGB image, respectively. The sparse tensor in the frame... From coordinate matrix and characteristic matrix ,in, These are pixel coordinates. It is the sample index in batch processing. This is the corresponding feature vector. To reduce computational cost without sacrificing generality, the batch size is assumed to be 1, and the batch index is ignored. The coordinates and depth values ​​of effective pixels (i.e., positive points) can be used as features for sparsification. Depth map , Correspondingly, sparse tensors Its effective pixel coordinates and depth values ​​can be projected onto dense [databases]. matrix Densification in the middle, When the input is sparse depth information, it is positionally encoded using a series of Spare Residual Blocks (SRBs). Each SRB consists of three parallel branches, each with a different number of sparse convolutional blocks. After the input is processed through a max-pooling stage, it is fed into different branches, and the output of deeper features serves as the input for the next SRB. Furthermore, after each SRB, a compaction layer is used in parallel to generate a dense representation of these sparse features.

[0024] Step B2, Enhanced Feature Extraction: Specifically, for enhanced feature points... Represent it as It consists of three-dimensional geometric features Two-dimensional semantic features and coordinates Together, they form a whole. First, a fully connected layer is applied to the enhanced point features to reduce computational complexity. Second, after the fully connected layer, the feature channels are... Upgraded to Finally utilize The three-dimensional and two-dimensional positional residuals to its neighborhood make The enhanced features perceive the local correspondence between three-dimensional and two-dimensional spaces. This allows for the extraction of enhanced feature points. The corresponding three-dimensional geometric features and two-dimensional semantic features. For The Neighboring points , and The positional residuals between them can be expressed as: in, For enhancing feature points of For each set of neighborhood points, their location information is collected and the corresponding location residuals are calculated. Then, a fully connected layer is applied to the location residuals, aligning their feature channels with the features of the enhanced feature points. For a set of neighborhood features... and a set of neighborhood location residuals The corresponding neighborhood residuals are weighted with the features, and the weighted neighborhood features are concatenated to obtain the maximum information fidelity. Finally, a fully connected layer maps the aggregated feature channels back to the original features. .

[0025] Step four, multimodal feature fusion: A dynamic cross-attention mechanism is used to obtain the correlation between different modalities and predict the correlation weights. Then, the features are weighted to obtain the fused features. Specifically, this includes:

[0026] 3D Mesh Fusion Mechanism: This mechanism uses a 3D mesh module to separately crop enhanced feature data and original point cloud data. Compared to previous methods that used 2D regions of interest (ROIs) to extract image features, it reduces interference from other objects or backgrounds. Furthermore, in previous multimodal data fusion methods, domain differences exist between different modalities, and dense images and sparse point cloud data also have varying quantities and representations. Therefore, it is often impossible to effectively enhance relevant feature data. In this invention, since the enhanced data features after depth completion have the same representation as the original ROI features, the corresponding meshed features can be focused on for fusion. This effectively utilizes semantic information and point cloud depth information in the image to improve target representation and enhance detection capabilities. In practical applications, a batch of 3D mesh modules can be processed in parallel, making the 3D mesh fusion mechanism highly efficient.

[0027] Attention fusion mechanism: This mechanism introduces dependencies between features, thereby better capturing data correlations. It can effectively obtain correlations between data from different modalities and predict correlation weights. By weighting features, fused features can be obtained more accurately, thus improving the utilization efficiency of multimodal data.

[0028] Step 4: Multimodal feature fusion. Specifically: [The text abruptly ends here, likely due to an incomplete sentence or a formatting error.] and These are used to represent the original region of interest features and the enhanced data features, respectively. It is the number of 3D mesh blocks. It is a grid feature channel. and The Each feature is represented as follows: and Given a pair of mesh features Three connection layers are used to transform the enhanced data features into queries. Transform the original features of the region of interest into keys Sum For each query, the query... and key value An inner product is performed between the features to obtain the attention affinity matrix between the enhanced data features and their corresponding original features. This is achieved through... softmax After layer normalization, this matrix is ​​used to measure and aggregate values. Connect them through a fully connected layer and Layer, generating a pair of weights Finally, the weights Weighted Enhanced mesh fusion features are obtained. The aggregated feature information is processed through a fully connected layer and concatenated with deep features for output; this enhances the mesh fusion feature. as follows:

[0029] ,

[0030] Finally, the multimodal fusion features obtained by channel-by-channel cascading are then used for confidence prediction to obtain the location and classification information of the target detection box.

[0031] The beneficial effects of this invention are as follows: This invention integrates the advantages of both lidar and camera sensors to achieve dynamic multi-target detection technology in the field of autonomous driving. It can accurately identify and locate various targets such as vehicles, pedestrians, and cyclists. This technology takes into account real-time performance and can be applied in real-world scenarios.

[0032] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of the present invention, and all such modifications and substitutions should be considered within the scope of protection of the present invention.

Claims

1. A multimodal three-dimensional target detection method for autonomous driving, characterized in that, Includes the following steps: Step 1, Feature Information Acquisition: Using the original point cloud and color image as input, radar point cloud features and image features are obtained respectively through a standard feature extraction backbone network; Step 2: Generate initial candidate boxes using the Transformer dual-fusion feature region proposal network; Step 3, Deep Feature Information Enhancement: Through a deep information completion mechanism, predict dense deep information and extract feature semantic information; Step 4, Multimodal Feature Fusion: A dynamic cross-attention mechanism is used to obtain the correlation between different modalities and predict the relevant weights. Then, the fused features are obtained by weighting the features. In step four, and These are used to represent the original region of interest features and the enhanced data features, respectively. It is the number of 3D mesh blocks. It is a grid feature channel. and The Each feature is represented as follows: and Given a pair of mesh features Three connection layers are used to transform the enhanced data features into queries. Transform the original features of the region of interest into keys Sum For each query, the query and key value Perform an inner product between them to obtain the attention affinity matrix between the enhanced data features and their corresponding original features; then, through... softmax After layer normalization, this matrix is ​​used to measure and aggregate values. Connect them through a fully connected layer and Layer, generating a pair of weights Finally, the weights Weighted Enhanced mesh fusion features are obtained. The aggregated feature information is processed through a fully connected layer and concatenated with deep features for output; this enhances the mesh fusion feature. as follows: , Finally, the multimodal fusion features obtained by channel-by-channel cascading are then used to predict the confidence level again, thereby obtaining the location and classification information of the target detection box.

2. The multimodal three-dimensional target detection method according to claim 1, characterized in that, Step two includes the following steps: Step A1, performing dual feature query: This refers to obtaining more accurate features through both camera feature query and voxel feature query. When using LiDAR features, they are voxelized to obtain... 1 non-empty voxels, of which the number It varies depending on the distribution of the input lidar features; voxel query Matching with corresponding non-empty voxels improves voxel domain features; The center point of each non-empty voxel is projected onto an image pixel in the camera domain, and then queried by the camera in sequence. Corresponding allocation, and used to improve image domain features; This represents the query for the 1st to the kth cameras. This represents the query for the first to kth voxels in the voxel domain; Step A2, 3D Local Self-Attention: The self-attention layer encodes voxel queries one by one by establishing spatial relationships between them. By applying the farthest point sampling algorithm to the center points of non-empty voxels, they are clustered into local regions, and within a fixed radius around the centroid of each local region, they find... Individual element; Suppose They are assigned to the center of mass. A set of query features and 3D location; then, the 3D-LSA layer performs self-attention; Indicates the location of the front-end network. It is a position encoding function, through For two three-dimensional coordinates Encode the interpolation; If the number of non-empty voxels in the radius is greater than Then calculate Individual element; with a group Voxel queries associated with individual voxels are encoded separately by the self-attention layer; After passing through multiple self-attention layers, the input voxel query Updated to ; Step A3, perform dual-domain feature attention fusion: Let For non-empty voxels, the first 3D reference point at the center of each non-empty voxel , These are the coordinates of the center point of the k-th non-empty voxel; projected onto the camera domain. Quantization is then performed on the mesh to determine the corresponding 2D reference points. x and y are the coordinates of the corresponding 2D reference point; camera query From 2D reference point Indicated camera domain features Initialization is performed; voxel queries are obtained from 3D local self-attention. Depth-aware location encoding was first applied to dual-feature queries; according to the formula and z represents the depth value, and i and d are the index and dimension of the query vector, respectively. Depth-aware positional encoding is added to the dual-feature query; dual-domain feature fusion and attention decoding are performed on multiple attention layers for dual queries. and First, camera domain features for and Query camera Perform deformation with attention to transformation; for a given 2D reference point Features in the camera domain Apply a deformable mask with adaptive offset and weights, mask offset and mask weights From the formula Given; given mask offset and weight Attention value Calculation by formula Given, among which, It is the index of the attention head. Sampling Perform indexing. It is sampling The total number, and This represents the learnable projection matrix.

3. The multimodal three-dimensional target detection method according to claim 1, characterized in that, Step three includes the following steps: Step B1, Depth Information Completion: The foreground points in the candidate boxes constitute the features describing the target clues. The feature representation is enhanced by completing the dense depth feature information. Step B2, Enhanced Feature Extraction: For enhanced feature points Represent it as It consists of three-dimensional geometric features Two-dimensional semantic features and coordinates Together they form a whole.

4. The multimodal three-dimensional target detection method according to claim 3, characterized in that, In step B1, a sparse depth map and an RGB image are taken as input, and a completed dense depth map is output. Sparse convolution and standard convolution are used to process the input depth map and RGB image, respectively, and the sparse tensor in the frame... From coordinate matrix and characteristic matrix in, These are pixel coordinates. It is the sample index in batch processing. This is the corresponding feature vector; the batch size is assumed to be 1, and the batch index is not considered; the coordinates and depth values ​​of the effective pixels are used as features for sparsification. Depth map , Correspondingly, sparse tensors Its effective pixel coordinates and depth values ​​can be projected onto dense [databases]. matrix Densification in the middle, When the input is sparse depth information, it is positionally encoded through a series of Spare Residual Blocks (SRBs). Each residual block consists of three parallel branches, each with a different number of sparse convolutional blocks. After the input is processed by the max pooling stage, it is fed into different branches, and the output of deeper features is used as the input of the next residual block. After each residual block, a compaction layer is used in parallel to generate a compact representation of these sparse features.

5. The multimodal three-dimensional target detection method according to claim 3, characterized in that, In step B2, firstly, a fully connected layer is applied to the enhanced point features; secondly, after the fully connected layer, the feature channels are... Upgraded to Finally utilize The three-dimensional and two-dimensional positional residuals to its neighborhood make The enhanced feature perception detects the local correspondence between three-dimensional and two-dimensional spaces, thereby enabling the extraction of enhanced feature points. The corresponding three-dimensional geometric features and two-dimensional semantic features; for The Neighboring points , and The positional residuals between them can be expressed as: in, For enhanced feature points of For each set of neighborhood points, their location information is collected and the corresponding location residuals are calculated. Then, a fully connected layer is applied to the location residuals, and their feature channels are aligned with the features of the enhanced feature points. and a set of neighborhood location residuals The corresponding neighborhood residuals and features are weighted, and the weighted neighborhood features are concatenated to obtain the maximum information fidelity. Finally, a fully connected layer is used to map the aggregated feature channels back to the original features. .

6. The multimodal three-dimensional target detection method according to claim 1, characterized in that, Step four includes a 3D mesh fusion mechanism and an attention fusion mechanism. 3D mesh fusion mechanism: Use a 3D mesh module to separately crop the enhanced feature data and the original point cloud data.

7. As described in claim 1, characterized in that: The feature extraction backbone network includes the radar backbone network and the image backbone network.

8. A multimodal three-dimensional target detection system for autonomous driving, characterized in that, include: A memory, a processor, and a computer program stored on the memory, the computer program being configured to implement the steps of the multimodal three-dimensional target detection method of any one of claims 1-7 when invoked by the processor.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program configured to implement the steps of the multimodal three-dimensional target detection method according to any one of claims 1-7 when invoked by a processor.

Images