Three-dimensional target detection method and system for automatic driving, and storage medium

Abstract

The application provides a three-dimensional target detection method, system and storage medium for automatic driving, comprising the following steps: in step one, a graph convolution network with an attention mechanism is used to extract multi-scale voxel features of local neighborhood and context information in a voxel feature extraction branch; in step two, a densely connected 2D convolution network multi-level superposition aggregation is used in an image feature extraction branch to aggregate shallower and deeper layers, and a pyramid superposition structure is introduced to aggregate multi-scale image features; in step three, based on the features extracted by the voxel feature extraction branch and the image feature extraction branch, multi-modal iterative mutual attention fusion is used to fuse multi-scale image features and voxel features, and finally, 3D target detection is realized through region proposal and classification regression based on multi-modal features. The application has the advantages that: the application can accurately recognize and locate long-distance vehicles, pedestrians, cyclists and other targets, and can be applied in actual scenes.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving technology, and in particular to a multimodal three-dimensional target detection method, system, and storage medium for autonomous driving. Background Technology

[0002] In recent years, object detection technology has developed rapidly and has been widely used in autonomous driving and robotics. 3D object detection is the task of identifying and locating objects in a three-dimensional scene, but it still faces significant challenges. Recent methods mainly utilize image and LiDAR-type data. Point cloud data has rich geometric information, while images have rich semantic information. Due to the complementarity between point clouds and images, some methods first project the point cloud onto various compactly represented views, then use mature 2D convolutional neural networks to obtain features for each view, and finally fuse these features with image features. Based on this fusion, some methods use 2D detectors to filter the point cloud, performing 3D object detection only on points within the view frustum of the 2D image target. However, these methods encounter spatial information inaccuracies during projection, especially for distant targets. Another solution is to fuse image and point cloud features at different resolution levels, but this is still limited by the sparsity of points, particularly affecting distant targets. Summary of the Invention

[0003] To address the problems in the background technology, this invention proposes that multi-scale image and point cloud feature information should be fused together. A deep fusion multimodal iterative mutual attention fusion is designed, which uses an iterative mutual attention mechanism to align voxel features and image features.

[0004] This invention provides a target detection algorithm based on iterative voxel-image attention fusion.

[0005] As a further improvement of the present invention, the voxel graph feature filter (VGFF) assigns appropriate attention weights to neighboring voxels based on dynamically learned graph voxel features, enabling the features to selectively focus on the most relevant parts and extract multi-scale voxel features with more local neighborhood and contextual information.

[0006] As a further improvement of the present invention, the densely connected multi-scale image feature aggregation MHA-ResNet module obtains multi-scale aggregated features, enabling the encoder to acquire richer information, reducing the spatial information lost during input dimensionality reduction, and improving the accuracy of long-distance small target detection.

[0007] As a further improvement of the present invention, the multimodal iterative mutual attention fusion module utilizes an iterative mechanism to capture the correlation between two modal features and improves the accuracy of the alignment of the two modal features.

[0008] The beneficial effects of this invention are: This invention integrates the advantages of both lidar and camera sensors to achieve long-distance small target detection technology in the field of autonomous driving. It can accurately identify and locate targets such as vehicles, pedestrians, and cyclists at a distance, and can be applied in real-world scenarios. Attached Figure Description

[0009] Figure 1 It is a framework diagram based on iterative voxel-image attention fusion;

[0010] Figure 2 This is a diagram of the voxel image feature filtering module;

[0011] Figure 3 This is a diagram of the ResNet module in the deep aggregation pyramid. Detailed Implementation

[0012] This invention discloses a multimodal 3D target detection method for autonomous driving, or more precisely, a multimodal 3D target detection method for autonomous driving based on iterative voxel-image attention fusion. The vehicle uses multiple sensors to collect target data and iteratively fuses data from different modalities to improve the accuracy of detecting small targets at long distances.

[0013] This invention can reduce the loss of spatial information during dimensionality reduction to the feature layer, thereby enhancing the 3D target detection accuracy of small targets at long distances.

[0014] like Figures 1 to 3 As shown, this invention, based on features extracted from two branches, effectively fuses multi-scale image features and voxel features through multimodal iterative mutual attention, thereby improving the 3D target detection accuracy of small targets at long distances. The specific steps are as follows:

[0015] Step 1: In the voxel feature extraction branch, a graph convolutional network with attention mechanism is used to extract multi-scale voxel features with more local neighborhood and contextual information.

[0016] Step one includes:

[0017] Step 1: Perform voxelization. Set the depth, height, and width of the range to... The depth, height, and width of the clicked voxel Divide into voxels, and keep the number of points in each voxel from 1 to 2. The voxel features are obtained by random downsampling, where T is the algebra, referring to the number of points in each voxel.

[0018] Step 2: Constructing a spatial graph based on voxel features. A graph was constructed based on sparse voxel features. , For having Individual factors The structure of a voxel set, where vertices The set of edges is represented as .vertex The neighbors are set as Input a set of voxel features ,in ,vertex A voxel map feature filter learns a function. Input voxel features The mapping yields a new set of graph voxel features. ,in New image voxel features The spatial structural relationships between the original voxel features were preserved.

[0019] Step 3: Learn neighbor features using an attention mechanism. This invention constructs an attention mechanism. This is used to filter the learned features, and the resulting features will focus on relevant neighbor features. Attention weights for neighbor vertices. , , ,in , A multilayer perceptron is used as the feature mapping function. It helps filter out messy neighbors and obtain meaningful neighborly relationships. The guided filter allocates more attention to similar neighboring voxels. This invention relates to the attention mechanism. ,in, It's a connection operation. This represents the multilayer perceptron used in the application. Furthermore, for vertices... Attention weights are normalized to handle neighbors at different spatial scales. The feature channels are shown below: Voxel features The final output is expressed as ,in This represents the Hadamard product. It is a learnable bias. Aggregated features on the vertices of the output voxel map are achieved through graph pooling. This invention obtains novel voxel map features. . Sparse voxel features Convert to size The conventional 4D dense tensor, in which , , .

[0020] Step 4: Generate spatial attention using the spatial relationships of features and connect its global features to compensate for global information. This invention uses voxels... A combined average pooling and max pooling operation is used along the channel to obtain an effective feature space descriptor, the spatial response of which is: and The two descriptors are concatenated and convolved through a standard convolutional layer to generate the voxel 3D spatial attention of this invention. , ,in This represents the sigmoid function. Indicates the filter size is The convolution operation represents the importance of each feature channel representing the size of the filter voxel. The output of spatial attention is ,in This represents element-wise multiplication. To supplement global information, a max-pooling layer is used to aggregate the entire scene information into a global attention block. This results in a global information... Features and expand them to Size and The voxel features are then connected to obtain the final voxel map features. Finally, 3D sparse convolution is used to gradually downsample the voxel features, and the voxel features are downsampled by 4 times to obtain the 2D bird's-eye view feature map of the region proposal network.

[0021] Step 2: The image feature extraction branch uses a densely connected 2D convolutional network to aggregate shallower and deeper layers, introducing a pyramid stacking structure to aggregate multi-scale image features in order to learn more refined multi-depth features.

[0022] Step two includes:

[0023] Step A1: Perform multi-depth aggregation on the convolutional blocks. To preserve hierarchical information, this invention establishes a simple aggregation form on the basic blocks of ResNet-50, improving the depth and transmission efficiency of information. Features within each stage of the network are aggregated, and the output of an aggregation node is returned to the backbone network. After downsampling, it is then used as input back to the backbone network. This invention fuses the blocks of each stage with their subsequent blocks, and merges aggregation nodes with the same depth together. The depth is... Deep aggregation Represented as: ,in This indicates an aggregation operation, and and Defined as ,in This represents a convolution operation. These nodes are trained to select important information to maintain an output with the same scale as the input dimension. Although an aggregation node can employ arbitrary block or layer structures, this invention uses... The structure consists of a convolution followed by a BN layer and then a nonlinear activation layer, which avoids the aggregation structure from becoming too complex.

[0024] Step A2: Obtain multi-scale features of the image using a multi-level aggregation pyramid. This invention introduces the concept of a pyramid (FPN) based on multi-depth aggregation, extracting features from the bottom up. In the three-level FPN model of this invention... and These are the output feature maps of the third, fourth, and fifth stages of ResNet-50, respectively. Based on the structure of ResNet-50, yes 2 times, yes Twice as much. The top-level feature map F5 of the FPN in this invention is composed of... exist It comes from kernel convolution operations, and then each high-level feature map of the FPN is upsampled by a factor of 2 and added to it. The feature maps of the underlying ResNet-50 convolutional kernels are used to construct the lower-level feature maps of the FPN. The final output F5, F4, and F3 are calculated as follows: ,in For a 1×1 convolutional kernel, To increase the size of the feature map by four times.

[0025] Step 3: Based on the features extracted from the first two branches, multi-scale image features and voxel features are fused through multi-modal iterative mutual attention fusion. Finally, 3D object detection is achieved by performing region proposal and classification regression based on the multi-modal features.

[0026] Step three includes the following steps:

[0027] Step B1: This invention applies fusion across features at different levels, enabling point cloud features to possess high-level semantic information from the image. First, this invention extracts features from four voxel convolutional layers of different sizes and projects them onto a bird's-eye view of the corresponding size. For example, given a bird's-eye view of size... The coordinates of the point are projected onto a surface of size . The image feature map is shown. This invention combines the concept of cross-correlation in signal processing with the attention mechanism, and designs a cross-correlation attention feature fusion module to fuse information from different modal features extracted from the two branches. The point cloud bird's-eye view and image features at the corresponding scales are represented as follows: and The fusion features are obtained through bitwise addition. fusion features Attention weights are calculated by combining point cloud features and image features respectively. , Among them, the weights , The calculation is as follows , ,in To represent addition, This represents element-wise multiplication. It involves fusing features. respectively with matrix and Multiply to transform the feature space, then add a bias. and The algorithm performs a translation of the feature space and optimizes it. , , and The value aligns the fused features with the feature spaces of the point cloud bird's-eye view and the image. Then, it is obtained through the hyperbolic tangent function. and With fusion features cross-correlation value Then use The relevant function values ​​are normalized to obtain the attention weights. and The cross-correlation value after normalization Converting to real numbers between [0,1] will, on the other hand, better facilitate the extraction of deep semantics. The multimodal attention fusion representation is as follows: ,in It is a feature of fusion. This invention represents feature integration. For simplicity, this invention chooses element-wise summation as the initial integral. As input to the attention module, the initial fusion quality can significantly influence the final fusion weights, and is crucial for the fusion scheme to acquire context awareness. Therefore, this invention adopts a direct approach: using another attention module to fuse the input features. This two-stage method is termed multimodal iterative attention fusion, with the initial integral... Rewritten as .

[0028] Step B2: Implement 3D object detection based on the CenterPoint-based region proposal network and classification regression learning network.

[0029] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of the present invention, and all such modifications and substitutions should be considered within the scope of protection of the present invention.

Claims

1. A multimodal three-dimensional target detection method for autonomous driving, characterized in that, Includes the following steps: Step 1: In the voxel feature extraction branch, a graph convolutional network with an attention mechanism is used to extract multi-scale voxel features of local neighborhood and contextual information, and voxelization is performed: the depth, height, and width are... The depth, height, and width of the clicked voxel Divide the data into voxels, ensuring that no more than a certain number of points are present in each voxel. A voxel feature was obtained using a random downsampling method, where T is an algebraic number representing the number of points in each voxel. A spatial graph was constructed based on the sparse voxel features. The set of vertices is represented as The set of edges is represented as ;vertex The neighbors are set as , Input a set of voxel features The voxel map feature filter learns a function Input voxel features The mapping yields a new set of graph voxel features. New image voxel features It maintains the spatial structural relationships between the original voxel features; through attention mechanisms The learned features are filtered and the resulting features will focus on relevant neighbor features; spatial attention is generated by utilizing the spatial relationships of features and connecting them to global features to compensate for global information; Step 2: The image feature extraction branch uses a densely connected 2D convolutional network to aggregate shallower and deeper layers, introducing a pyramid stacking structure to aggregate multi-scale image features; wherein the convolutional blocks are aggregated at multiple depths, the features in each stage of the network are aggregated, the output of an aggregation node is returned to the backbone network, downsampled, and then fed back to the backbone network as input; By merging the blocks of each stage with their subsequent blocks and merging aggregate nodes with the same depth together, a multi-level aggregate pyramid is used to obtain multi-scale features of the image. Step 3: Based on the features extracted by the voxel feature extraction branch and the image feature extraction branch, multi-scale image features and voxel features are fused through multi-modal iterative mutual attention fusion. Features from four voxel convolutional layers of different sizes are extracted and projected onto the corresponding-sized bird's-eye view. Information fusion of different modal features extracted by the two branches is performed through a cross-correlation attention feature fusion module. The point cloud bird's-eye view and image features at the corresponding scales are represented as follows: and The fusion features are obtained through bitwise addition. fusion features Attention weights are calculated by combining point cloud features and image features respectively. , Finally, based on multimodal features, region proposal and classification regression are performed to achieve 3D object detection.

2. The multimodal three-dimensional target detection method according to claim 1, characterized in that, In step one, the attention weights of neighboring vertices , , ,in , Use a multilayer perceptron as the feature mapping function; It helps filter out messy neighbors and obtain meaningful neighborly relationships; The guided filter allocates more attention to the index. i and the neighboring voxels of j; Attention mechanism ,in, It's a connection operation. This represents a multilayer perceptron used in the application.

3. The multimodal three-dimensional target detection method according to claim 1, characterized in that, In step one, for the vertices Attention weights are normalized to handle neighbors at different spatial scales. The feature channels are shown below: Voxel features The final output is expressed as , This represents the Hadamard product. It is a learnable bias; aggregated features on the vertices of the output voxel map are achieved through graph pooling. Obtain new voxel map features ; sparse voxel features Convert to size The conventional 4D dense tensor, in which , , .

4. The multimodal three-dimensional target detection method according to claim 1, characterized in that, In step one, voxels are... A combined average pooling and max pooling operation is used along the channel to obtain an effective feature space descriptor, the spatial response of which is: and ; The two descriptors are concatenated and then convolved through a standard convolutional layer to generate a voxel 3D spatial attention. , ,in This represents the sigmoid function. Indicates the filter size is Convolution operation, M s The importance of each feature channel representing the size voxel of the filter; The output of spatial attention is ,in This represents element-wise multiplication; to supplement global information, a max pooling layer is used to aggregate the entire scene information into a global attention block; then, a global information block is obtained. Features and expand them to Size and The voxel features are then connected to obtain the final voxel map features. Finally, the voxel features are gradually downsampled after 3D sparse convolution. The voxel features are downsampled by 4 times to obtain the 2D bird's-eye view feature map of the region proposal network.

5. The multimodal three-dimensional target detection method according to claim 1, characterized in that, In step two; the depth is Deep aggregation Represented as: ,in This indicates an aggregation operation, and and Defined as C represents the convolution operation; nodes are trained to select important information to maintain the output at the same scale as the input dimension.

6. The multimodal three-dimensional target detection method according to claim 1, characterized in that, In step two, the three-level FPN model and These are the output feature maps of the third, fourth, and fifth stages of ResNet-50, respectively. Based on the structure of ResNet-50, yes 2 times, yes Twice that; the top-level feature map F5 of FPN is composed of exist It comes from kernel convolution operations, and then each high-level feature map of the FPN is upsampled by a factor of 2 and added to it. The feature maps of the lower-level ResNet-50 convolutional kernels are used to construct the lower-level feature maps of FPN.

7. The multimodal three-dimensional target detection method according to claim 1, characterized in that, In step two, the final outputs of F5, F4, and F3 are calculated as follows: ,in For a 1×1 convolutional kernel, To increase the size of the feature map by four times.

8. The multimodal three-dimensional target detection method according to claim 1, characterized in that, Step three includes the following steps: Step B1: Merge features respectively with matrix and Multiply to transform the feature space, then add a bias. and Perform a translation of the feature space, and optimize... , , and The value aligns the fused features with the feature spaces of the point cloud bird's-eye view and the image, and then obtains the result through the hyperbolic tangent function. and With fusion features cross-correlation value Then use The relevant function values ​​are normalized to obtain the attention weights. and ; cross-correlation value after normalization It is converted to a real number between [0,1]; the multimodal attention fusion representation is as follows: ,in It is a feature of fusion. Indicates feature integration; Step B2: Implement 3D object detection based on the CenterPoint-based region proposal network and classification regression learning network.

9. The multimodal three-dimensional target detection method according to claim 8, characterized in that, In step B1, another attention module is used to fuse the input features. This two-stage method is called multimodal iterative attention fusion. Initial integral Rewritten as .

10. A multimodal three-dimensional target detection system for autonomous driving, characterized in that, include: A memory, a processor, and a computer program stored on the memory, the computer program being configured to implement the steps of the multimodal three-dimensional target detection method of any one of claims 1-9 when invoked by the processor.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program configured to implement the steps of the multimodal three-dimensional target detection method according to any one of claims 1-9 when invoked by a processor.

Images