A pure sparse 3D object detection method fusing multi-view information

Abstract

The application discloses a pure sparse 3D target detection method fusing multi-view information, and relates to the fields of artificial intelligence deep learning and intelligent driving perception. The method obtains features in a single view through single direction aggregation of a sparse feature map, enhances the extracted voxel features by fusing features of multiple views, and completes the whole fusion process on the sparse voxel feature map. The application obtains three single-view features through sparse aggregation operation, fuses multi-view features through normalization and cross attention operation, enhances the extracted voxel features, improves the detection precision, and greatly reduces the calculation amount.

Description

Technical Field

[0001] This invention belongs to the fields of artificial intelligence deep learning and intelligent driving perception, and specifically relates to a pure sparse 3D object detection method that integrates multi-view information. Background Technology

[0002] LiDAR 3D point cloud refers to 3D data obtained through LiDAR scanning, used to describe the structure and geometry of objects in space. Compared to two-dimensional image information, 3D point cloud can provide richer structural and geometric information. Therefore, deep learning algorithms based on LiDAR point clouds have wide applications in fields such as medicine, robotics, and autonomous driving. In the field of autonomous driving, vehicle-mounted LiDAR scans and transmits 3D point cloud data of the surrounding environment, and then 3D object detection algorithms detect surrounding vehicles or obstacles so that the vehicle can make reasonable decisions and plan its path.

[0003] LiDAR point clouds are unstructured data, characterized by their disorder and sparsity, making 3D target detection algorithms based on LiDAR point clouds highly challenging. Point cloud-based 3D target detection algorithms involve processing and extracting features from the input point cloud, and then outputting the predicted target category and location information through a detection head. Although some works have focused on fusing point cloud information from multiple perspectives, these methods process dense feature maps, resulting in high computational costs.

[0004] Furthermore, the main tasks based on LiDAR point clouds can be divided into three main tasks: point cloud classification, point cloud segmentation, and 3D object detection. For the 3D object detection task, the 3D detector extracts features from the input point cloud through a 3D backbone network, and then outputs information such as the category, position, and size of the detected object through the detection head. 3D object detection algorithms based on LiDAR point clouds can be divided into three methods: point-based, voxel-based, and pillar-based methods. Specifically, the point-based method directly extracts features from each point in the input point cloud, using point-level feature extraction networks such as PointNet to extract point-level features, and then directly performs regression and classification on each point. The advantage of the point-based method is that it minimizes information loss during feature extraction, but its disadvantage is that the inference speed is relatively low due to the need to process each point. The pillar-based method divides the point cloud space into pillars without z-axis resolution, then encodes the point cloud features within these pillar structures, extracts pillar-level features, and performs classification and regression on the pillars. The advantage of this method is its fast inference speed, but its disadvantage is insufficient accuracy due to low spatial resolution. To balance inference speed and detection accuracy, the widely used LiDAR point cloud 3D target detection algorithm is the voxel-based method. These methods divide the point cloud space into high-resolution squares called voxels, then extract voxel features through 3D sparse convolution, and classify and regress the voxels. This method achieves better detection accuracy than pillar-based methods and faster inference speed than point-based methods, thus achieving a balance between accuracy and speed. However, the challenge of voxel-based LiDAR point cloud 3D target detection algorithms lies in the sparsity of the point cloud; for a given set of voxels, only 10% are non-empty voxels.

[0005] Therefore, research on voxel-based 3D target detection algorithms from multiple perspectives is quite important. Summary of the Invention

[0006] To address the aforementioned technical problems, this invention provides a pure sparse 3D object detection method that fuses information from multiple perspectives. Features from a single perspective are obtained by aggregating sparse feature maps in a single direction. The extracted voxel features are enhanced by fusing features from multiple perspectives. The entire fusion process is completed on the sparse voxel feature map, resulting in considerable computational efficiency.

[0007] To achieve the above objectives, the technical solution adopted by the present invention is as follows: A pure sparse 3D object detection method that integrates multi-view information, the method comprising the following steps: Step 1: Input the raw point cloud data of the lidar. Set the number of points in the point cloud data to N and the feature dimension to D. The raw point cloud data includes the x, y, z three-dimensional coordinate information and reflection intensity information of the point cloud. Step 2: Voxelize the point cloud using point cloud coordinates and the predefined spatial voxel coordinates, and encode voxel features by aggregating the point cloud information within each voxel to obtain voxel input features. Step 3: Pack the spatial voxel coordinates and the corresponding voxel input features into a sparse tensor and input it into the 3D backbone network to extract the deep features of the voxels. Step 4: After obtaining the deep features of the voxels, sum the deep features of the voxels along the same direction using the spatial index of the voxels to obtain the 2D sparse features from that direction's perspective. Repeat step 4 along the x, y, and z directions to obtain the sparse feature maps from a single perspective in each of the x, y, and z directions. , ,

[0008] Step 5: Extract sparse feature maps from a single viewpoint in the x, y, and z directions. , , After normalization, multi-view features are fused through a cross-attention mechanism; Step 6: Input the fused multi-view features into the sparse detection head to predict voxel classification and regression; Step 7: Output the attributes of the detection box, including the category, position, size and orientation of the detection box, to obtain the detection information of the target.

[0009] The beneficial effects of this invention are as follows: This invention obtains three single-view features through sparse aggregation operations, and fuses multi-view features through normalization and cross-attention operations, thereby enhancing the extracted voxel features and improving detection accuracy. It directly fuses multi-view information on the 3D sparse feature map and adopts a pure sparse structure, which greatly reduces the amount of computation. Attached Figure Description

[0010] Figure 1 This is a flowchart of a pure sparse 3D target detection method that integrates multi-view information according to the present invention. Figure 2 This is a network structure diagram of a pure sparse 3D target detection method that integrates multi-view information according to the present invention. Figure 3 A schematic diagram illustrating the principle of multi-view feature cross-fusion; Figure 4 This is a diagram illustrating the implementation effect of the present invention. Detailed Implementation

[0011] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0012] This invention provides a pure sparse 3D target detection method that integrates multi-view information, such as... Figure 1 As shown, the specific steps include the following: Step 1: First, input the raw point cloud data of the lidar. For ease of representation, set the number of points in the input point cloud to N and the feature dimension of the input point cloud to D. Then the data shape of the input point cloud can be represented as (N, D). The input point cloud contains the three-dimensional position information of the point cloud in x, y, z and the reflection intensity information. Step 2: Voxelize the point cloud using point cloud coordinates and the predefined spatial voxel coordinates, and encode voxel features by aggregating the point cloud information within each voxel to obtain voxel input features. Step 3: Pack the spatial voxel coordinates and the corresponding voxel input features into a sparse tensor and input it into the 3D backbone network to extract the deep features of the voxels. The 3D backbone network is composed of conventional 3D sparse convolution and 3D sparse submanifold convolution. The receptive field of a single voxel is expanded by reducing the size of the sparse tensor layer by layer. Step 4: After obtaining the deep features of the voxels, sum the voxel features along the same direction using the spatial location index of the voxels to obtain the 2D sparse features from that viewpoint, such as... Figure 2 As shown, the above operation is repeated along the x, y, and z directions respectively to aggregate the deep features of voxels and obtain single-view features, ultimately resulting in sparse feature maps under the x, y, and z views. , , ; Step 5: Fuse multi-view features through normalization and cross-attention; Simple summation can simplify computation to the greatest extent. However, simple summation leads to feature blurring. For example, along the z-axis, there are at most 6 non-empty voxels, while along the x-axis, there are at most 180 non-empty voxels. Therefore, even if there are many valid targets when summing along the z-axis, the feature summation is still weaker than the feature summation along the x-axis due to the limited number. A simple idea is to align the amplitudes using average pooling, but average pooling weakens high-frequency information, resulting in poor performance. Therefore, this invention proposes to compensate for the feature blurring problem caused by sparse feature maps through normalization operations, that is, for features under a single viewpoint, the values ​​are aligned by subtracting the mean and dividing the variance. Then, since the final required feature is a sparse feature map under BEV (Bird's Eye View) (i.e. Therefore, , and , Pairwise fusion is performed using cross-attention. Since the final output viewpoint is along the z-axis, features along the x-axis and y-axis are not fused. Step 6: Input the fused multi-view features into the sparse detection head to predict voxel classification and regression; Step 7: Output the attributes of the detection box, including its category, position, size, and orientation. This provides the location and category of the detected target, offering the vehicle comprehensive information about its surrounding environment for environmental perception.

[0013] Furthermore, in step 4, the single-view feature is obtained through the following method: For multi-view fusion methods, previous methods first transformed sparse voxel features into dense 2D features so that structured representations could be used to facilitate fusion. However, for unstructured sparse tensors, it is impossible to uniformly represent them as tensors of a single shape because sparse tensors only record information about non-empty voxels. Therefore, how to obtain features from a single viewpoint for sparse tensors is a challenge.

[0014] This invention proposes to directly sum the features of all voxels in the same direction to obtain a sparse feature map under a single view. Specifically, the three-dimensional spatial position index of each non-empty voxel is recorded. Therefore, by representing the three-dimensional position index of the non-empty voxel in a one-dimensional representation according to a certain direction, voxels in the same direction can be found. For example, for the spatial indices [x,y,z]=[179,180,4] and [x,y,z]=[179,180,0], along the z-axis direction, The calculation results are the same, among which, and These represent the resolutions in the x and y directions of the voxel space, respectively. Therefore, two voxels will be grouped into the same group, and features will be aggregated by summation. , , , In this way, we obtain the features from a single viewpoint, such as the single-viewpoint feature map obtained by projecting along x (y or z), denoted as... ( or ). The feature of a voxel at position (x, y, z) is represented by i, j, and k, which represent the specific x, y, and z coordinates during projection, respectively.

[0015] However, such direct summation can lead to feature ambiguity due to uneven spatial distribution. For example, for two different z-coordinates, the number of non-empty voxels differs greatly, resulting in significant differences in the summation results. Therefore, a normalization operation is performed on the features from a single viewpoint: , in, This represents the feature map from a single viewpoint. Indicates along The mean of features from a directional perspective. Indicates along Variance of features from a directional perspective It is a constant to prevent the denominator from being zero. This represents the normalized feature map from a single viewpoint. This avoids the problem of excessively large differences in feature values ​​caused by uneven voxel counts. In ablation experiments, this normalization operation improved mAP by 0.9% and NDS by 0.6%, demonstrating that normalization effectively alleviates the feature ambiguity problem caused by directly summing sparse voxel features. The normalized single-view sparse 2D features are obtained. ( (where C represents the number of sparse features in a single viewpoint and C represents the feature dimension of the deep features of the voxel) Then, these single-viewpoint features are cross-fused.

[0016] Furthermore, the multi-view feature fusion in step 5 is performed as follows: like Figure 3 The diagram shown illustrates the principle of the multi-view feature cross-fusion module. It uses sparse features from the BEV perspective as the primary feature set, and then processes the normalized feature maps from the x and y axis directions. Cross-attention query is performed. Specifically, the features from the three perspectives are normalized, then feature mapping is performed through one-dimensional convolution, and a relation matrix is ​​obtained through matrix multiplication. The relation matrix and the single-view features are multiplied by matrix to obtain the fused features. Finally, the fused features and the BEV single-view features obtained by projection along the z-axis are concatenated together in the channel dimension, and the final sparse features are output through an MLP.

[0017] Since the detection head ultimately needs features under BEV (Browser Active Volume), when fusing multi-view information, it uses... and To enhance Specifically, first calculate and (or The similarity is calculated, and then the weight scores are normalized using Softmax. (or We perform a weighted summation on the result to obtain... (or )for Feature contributions: , in, This represents the attention weight between the normalized sparse feature map viewed along the z-axis and the normalized sparse feature map viewed along the b-axis (b is either x or y). for Features after linear layer mapping for (or The features after linear layer mapping, where the superscript T denotes matrix transpose, and C is the feature dimension of the deep voxel features. For example... Figure 3 As shown, based on respectively and After obtaining multi-view information, the multi-view information is concatenated along the channel dimension, and then the features are filtered through a fully connected layer: , This represents the result of cross-fusion of sparse feature maps from two perspectives along the z-axis and x-axis. This represents a sparse feature map viewed along the z-axis. This represents the result of cross-fusion of sparse feature maps from two perspectives along the z-axis and y-axis; MLP stands for Multilayer Perceptron, which is used for feature dimensionality reduction. This represents the stitched multi-view feature map.

[0018] The advantage of explicitly selecting features through dimensionality reduction in this way is that it prevents overfitting and can effectively select useful information from information from different perspectives.

[0019] For the detection head, this invention uses a sparse detection head, which takes 2D sparse convolution under BEV as input and outputs classification and regression results through 2D submanifold convolution and 2D sparse convolution.

[0020] Table 1 compares the computational cost and performance of different LiDAR point cloud 3D target detection algorithms on the nuscenes dataset. FLOPs represent the computational cost, and mAP and NDS are the evaluation metrics of the nuscenes dataset. It can be seen that the present invention maintains a low computational cost while improving detection accuracy.

[0021] Table 1

[0022] Figure 4The visualization demonstrates the improvement in detection results brought about by this invention. Comparing the results in the first row, it can be seen that for targets falsely detected by the baseline method VoxelNeXt (marked with circles, i.e., only predicted boxes, no ground truth boxes), this invention does not exhibit false detection problems. Comparing the results in the second row, it can be seen that for cases where the baseline method VoxelNeXt misses detection (marked with squares, i.e., only ground truth boxes, no predicted boxes), this invention does not exhibit missed detection problems.

[0023] This invention improves detection accuracy by directly fusing multi-view information on sparse feature maps while maintaining low computational cost.

[0024] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A pure sparse 3D target detection method that integrates multi-view information, characterized in that, The method includes the following steps: Step 1: Input the raw point cloud data of the lidar. Set the number of points in the point cloud data to N and the feature dimension to D. The raw point cloud data includes the x, y, z three-dimensional coordinate information and reflection intensity information of the point cloud. Step 2: Voxelize the point cloud using point cloud coordinates and the predefined spatial voxel coordinates, and encode voxel features by aggregating the point cloud information within each voxel to obtain voxel input features. Step 3: Pack the spatial voxel coordinates and the corresponding voxel input features into a sparse tensor and input it into the 3D backbone network to extract the deep features of the voxels. Step 4: After obtaining the deep features of the voxels, sum the deep features of the voxels along the same direction using the spatial index of the voxels to obtain the 2D sparse features from that direction's perspective. Repeat step 4 along the x, y, and z directions to obtain the sparse feature maps from a single perspective in each of the x, y, and z directions. , , ; Step 5: Extract sparse feature maps from a single viewpoint in the x, y, and z directions. , , After normalization, multi-view features are fused through a cross-attention mechanism; Step 6: Input the fused multi-view features into the sparse detection head to predict voxel classification and regression; Step 7: Output the attributes of the detection box, including the category, position, size and orientation of the detection box, to obtain the detection information of the target; In step 5, the normalization operation for features from a single viewpoint is performed according to the following formula: ， in, This represents the feature map from a single viewpoint, where b can be taken in any direction along the x, y, or z axes. express The mean of features from a given perspective. express Variance of features under different perspectives To represent a constant, This represents the normalized feature map from a single viewpoint. Fusing multi-view features through cross-attention mechanism includes: calculate and similarity, or The similarity is normalized using Softmax to adjust the weight scores, and then... or We perform a weighted summation on the above to obtain or for Feature contributions: ， in, This represents the normalized feature map along the z-axis and along the b-axis. Attention weights between normalized feature maps from different perspectives. for Features after linear layer mapping for or Features after linear layer mapping, where the superscript T indicates matrix transpose, and C is the feature dimension of the deep features of the voxel; Multi-view information is concatenated along the channel dimension, and then features are filtered through a fully connected layer: ， In the formula, This represents the stitched multi-view feature map; MLP stands for Multilayer Perceptron. This represents the result of cross-fusion of feature maps from two perspectives along the z-axis and x-axis. This represents the feature map viewed along the z-axis. This represents the result of cross-fusion of feature maps from two perspectives along the z-axis and y-axis.

Images