A three-dimensional target detection method based on class enhancement and geometry enhancement

Abstract

The application discloses a kind of three-dimensional target detection methods based on class enhancement and geometric enhancement, the method first input laser point cloud, random sampling to 16k points, carries out two furthest point sampling and ball query in succession, obtains 2048 center points.Secondly, to 2048 center points, by class enhancement multiple sampling module carries out two sampling and ball query in succession, obtains two center points, and its corresponding multi-scale aggregation feature 3 and multi-scale aggregation feature 4.Then to multi-scale aggregation feature 4 using MLP predicts centroid shift, so that each point is shifted to centroid, and then using geometric enhancement centroid attention module, obtains the centroid feature of centroid point.Finally, using centroid feature carries out object classification, and predicts object bounding box 3D position, size and direction, and eliminates overlapping bounding box.The application reduces missed detection and improves overall detection performance, and remarkable improvement is achieved in distant object detection.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of automatic driving perception, and particularly relates to a three-dimensional target detection method based on class enhancement and geometric enhancement. BACKGROUND

[0002] Target detection is one of the important tasks in the automatic driving system. Vehicles equipped with technologies such as laser radar and multi-camera systems can capture rich data, enabling the vehicle to accurately perceive the surrounding environment. Laser radar LiDAR point cloud has attracted great attention due to its ability to capture the 3D structure of objects in the scene and its flexible data structure. However, the current LiDAR resolution is limited, resulting in uneven point cloud density at different distances. In addition, the missing points formed by the mutual occlusion between objects limit the complete description of object geometric information. Therefore, target detection based on laser point cloud is still a challenging task.

[0003] The three-dimensional object detection method based on laser radar can be divided into three branches: voxel-based, point-based, and point-voxel-based. The voxel-based method represented by VoxelNet converts point cloud into voxel representation, and then uses 2D or 3D convolution method for feature learning. The point-based method represented by PointNet / PointNet++ directly performs feature learning on point cloud, and then uses specific operators for feature aggregation. The point-voxel-based method combines the advantages of voxel-based and point-based methods to achieve high-precision detection results.

[0004] A typical point-based three-dimensional object detector consists of two main components: point cloud sampling and feature learning. The current most advanced point-based method, such as IA-SSD and SASA, only uses coarse segmentation / classification-based downsampling. This limitation causes the detector to fail to learn enough features for objects with fewer points due to mutual occlusion or large distances. In addition, they also fail to fully exploit the information around key points, resulting in inaccurate detection bounding box regression. SUMMARY

[0005] To solve the challenges of the above-mentioned point-based three-dimensional target detection method, the present application proposes a three-dimensional target detection method based on class enhancement and geometric enhancement. The detection network formed mainly includes two new modules, the purpose of which is to improve the performance of the current point-based three-dimensional target detection.

[0006] 1. To achieve the above-mentioned purpose, the technical solution steps proposed by the present application are as follows:

[0007] Step one, input laser point cloud as the original point cloud, the original point cloud contains 4-dimensional features: center coordinates x, y, z and reflectivity i.

[0008] Step two, pre-processing the original point cloud: randomly sampling the original point cloud to 16k points.

[0009] Step three, for the reserved 16k points, two times of farthest point sampling FPS and ball query are performed in sequence to obtain 4096 and 2048 center points respectively, and their corresponding multi-scale aggregation features 1 and 2, which obtain deeper features while reducing the number of points.

[0010] Step four, for the 2048 center points and multi-scale aggregation feature 2, two times of sampling and ball query are performed through the class-enhanced multi-sampling CEMS module in sequence to obtain 1024 and 512 center points respectively, and their corresponding multi-scale aggregation features 3 and 4.

[0011] Step five, using MLP to predict the centroid offset of the multi-scale aggregation feature 4, so that each point is offset to the centroid, and then using the geometry-enhanced centroid attention GECA module to enhance the geometric information and global information of the centroid point to obtain the centroid feature of the centroid point.

[0012] Step six, using the centroid feature of the centroid point to classify objects and predict the 3D position, size and direction of the object bounding box, and using the non-maximum suppression method to remove overlapping bounding boxes.

[0013] Further, the specific process of step three is as follows:

[0014] For the reserved 16k points, the Euclidean distance between points is used as a measure to perform farthest point sampling FPS to form 4096 center points. In the virtual spherical query range around each center point, 16 and 32 neighbor points are found using ball query with radii of 0.2m and 0.8m. The neighbor points are used to increase the original 4-dimensional feature to 64-dimensional feature using multi-layer perceptron MLP, and the maximum pooling of the feature is used to obtain the feature representing the local neighborhood. The local neighborhood features of two different radii are fused to form multi-scale aggregation MSG feature 1. The above operation is repeated to perform farthest point sampling FPS to form 2048 center points and obtain the corresponding domain features. The neighborhood features of two different radii are fused to form multi-scale aggregation MSG feature 2.

[0015] Further, the specific process of step four is as follows:

[0016] For 2048 center points, sampling is performed by the class-enhanced multi-sampling CEMS module, and then a virtual spherical query range is constructed around each center point using a spherical query with 0.8 m and 1.6 m as the radius to find 16 and 32 neighbor points; the original 64-dimensional feature is upgraded to 128-dimensional using the MLP for the neighbor points, the feature is maximally pooled to obtain a feature representing the local neighborhood, and the neighborhood features of the two different radii are fused to form the multi-scale aggregation MSG feature 3; the above operation is repeated, the class-enhanced multi-sampling CEMS module is sampled, and the corresponding field features are obtained, and the neighborhood features of the two different radii are fused to form the multi-scale aggregation MSG feature 4. This class-enhanced multi-sampling strategy can consider the class information of the object multiple times, and the diffusion-like interpolation operation can solve the missing detection problem caused by mutual occlusion.

[0017] Further, the sampling process of the class-enhanced multi-sampling CEMS module is as follows:

[0018] 4.1, the 2048 center points have 128-dimensional features, then the corresponding class is predicted using the MLP for the feature, the class feature is obtained, and the class score of each point is obtained using the softmax function;

[0019] 4.2, based on the class score, the topk function is used to select the top 1024 points with the highest class score;

[0020] 4.3, for the 2048 center points, the K nearest neighbor KNN algorithm is used to find the nearest 3 points from the 1024 points to re-interpolate the feature;

[0021] 4.4, the interpolated feature is fused with the initial 128-dimensional feature, and 4.1-4.2 is repeated.

[0022] Further, the specific process of step five is: using the MLP to predict an offset relative to the centroid for the 256-dimensional feature of the remaining 512 points; based on the offset, moving the 512 points towards the centroid and obtaining the moved encoding feature; then using the geometry-enhanced centroid attention GECA module to obtain the centroid feature of the centroid point.

[0023] Further, the specific process of the geometry-enhanced centroid attention GECA module is: repeating the use of the graph convolutional network GNN twice to extract the geometric features around the centroid, and then splicing the encoding feature, the class feature and the geometric feature through the self-attention mechanism to obtain the centroid feature.

[0024] Furthermore, the specific operation of using Graph Convolutional Network (GNN) to extract geometric features around the centroid is as follows: 32 points around the centroid are found using the K-Nearest Neighbor (KNN) algorithm, a graph is constructed for each of the 32 points' three neighboring points, and then the features of each neighboring point are increased in dimensionality using MLP and summed as the geometric features of that point.

[0025] Compared with the prior art, the present invention has the following advantages:

[0026] First, this invention proposes a Class Enhanced Multiple Sampling (CEMS) module. This improvement enables the network to learn more information features, which is particularly effective for objects with a limited number of points due to mutual occlusion. By merging object category information and employing advanced sampling techniques, the CEMS module reduces missed detections and improves overall detection performance.

[0027] Secondly, this invention proposes a Geometric Enhanced Centroid Attention (GECA) module. By utilizing the geometric information present in the point cloud, this module helps to perform more accurate object bounding box regression, expand the network's receptive field, and improve the detection of distant objects. By utilizing a Graph Neural Network (GNN) and self-attention, the GECA module achieves significant improvements in distant object detection.

[0028] Third, extensive experiments on large-scale public datasets validate the effectiveness of the proposed method. Experiments demonstrate the excellent detection performance achieved by integrating the CEMS and GECA modules into a point-based 3D object detection framework. These results confirm the practical value and effectiveness of the proposed method in real-world scenarios. Attached Figure Description

[0029] Figure 1 This is a flowchart of the technical solution proposed in this invention;

[0030] Figure 2 This is a diagram of the three-dimensional target detection network structure proposed in this invention. Detailed Implementation

[0031] The following specific embodiments, in conjunction with the appendix, demonstrate this approach. Figure 1 and attached Figure 2 The present invention provides a further detailed description of the technical solution of a three-dimensional target detection method based on class enhancement and geometric enhancement proposed in this invention.

[0032] Dataset processing: The training and testing samples are from public datasets, taking the KITTI dataset as an example. The KITTI dataset contains 7418 training point clouds and 7518 testing point clouds, covering three categories of cars, pedestrians, and bicycles. According to the height of the bounding box and the occlusion level, the stage is divided into three difficulty levels: easy, medium, and difficult. The official evaluation index is the average precision (mAP), and the IoU threshold for cars is 0.7, and the IoU threshold for pedestrians and bicycles is 0.5. The 7418 training samples are divided into 3712 training groups and 3769 validation groups.

[0033] Neural network three-dimensional target detection process: input laser point cloud, the original point cloud contains 4-dimensional features: center coordinates x, y, z, and reflectivity i. The original point cloud is uniformly randomly sampled to 16k points. For the retained 16k points, two times of farthest point sampling (FPS) and ball query are performed to obtain 4096 and 2048 center points, and their corresponding multi-scale aggregation features 1 and multi-scale aggregation features 2. For the 2048 center points and multi-scale aggregation features 2, two times of sampling and ball query are performed through the class-enhanced multi-sampling (CEMS) module to obtain 1024 and 512 center points, and their corresponding multi-scale aggregation features 3 and multi-scale aggregation features 4. The multi-scale aggregation features 4 are used to predict the centroid offset using the MLP, so that each point is offset to the centroid, and then the geometric-enhanced centroid attention (GECA) module is used to obtain the centroid features of the centroid points. The centroid features of the centroid points are used for object classification, and the object bounding box 3D position, size, and direction are predicted. The non-maximum suppression method is used to remove overlapping bounding boxes, and the final detected bounding boxes are output.

[0034] The class-enhanced multi-sampling (CEMS) module and the geometric-enhanced centroid attention (GECA) module in the neural network are introduced in detail below.

[0035] Class-enhanced multi-sampling (CEMS) module: point-based backbone networks usually contain multiple set abstraction (SA) layers, which play a crucial role in the network architecture. The CEMS module is placed between two adjacent SA layers. Given a set of features representing the features of n points in the ith SA layer, first use the MLP layer to classify these points, and then use the softmax function to obtain the class scores of the points Then use the top-K function to select the top n (i+1) key points using these class scores, and finally aggregate the features of the surrounding neighboring points centered on these key points, and the generated domain features are represented as where n (i+1) < n (i) The class scores are represented as follows:

[0036] P (i) = σ(CLS(F (i) ))

[0037] where CLS(·) denotes the point classification layer and σ(·) denotes the softmax function.

[0038] Next, the initial n (i) points are operated using the K-Nearest Neighbor (KNN) algorithm and their nearest 3 neighbors are selected from the set F ′(i) . With the Euclidean distance between points as the weight, the new feature representation is re-interpolated as

[0039] To combine the new feature with the original feature, a feature concatenation operation is performed and a MLP layer is used for feature fusion to obtain the final fused feature The above operations can enable objects with a limited number of points to obtain more accurate information from their surrounding neighbor points through KNN and interpolation operations, improving the accuracy of classification, and the feature fusion operation ensures that the object features are not affected by the background point features to which they are interpolated. The interpolation operation and the feature fusion operation are represented as follows:

[0040] F″ (i) = Interpolate(KNN(F′ (i) ))

[0041]

[0042] where Interpolate(.) is the interpolation operation. denotes feature concatenation.

[0043] To ensure that the neural network can accurately classify point clouds, a cross-entropy (CE) loss function is used to calculate the classification loss L cls , which is represented as follows:

[0044]

[0045] where C denotes the total number of classes in the dataset, y c denotes the one-hot label associated with the true value. p c denotes the class score of the point, and terms (1) and (2) correspond to the first and second sampling operations, respectively. In the experimental setup, the balance coefficients a and β are set to 1.0 and 1.0.

[0046] Geometric-enhanced centroid attention (GECA) module: mainly composed of a graph neural network (GNN) and a self-attention mechanism. First, an MLP is used to predict the coordinates X = {x1, x2, …, x nthe offset of each point from its centroid. Each point is moved closer to the centroid based on the offset, resulting in the updated point coordinates X' = {x'1, x'2,..., x'n}, which represents as follows: n

[0047]

[0048] Secondly, each point is represented as a vertex V, and the connection between points is represented as an edge E, which facilitates the flow of information between adjacent points. The 32 neighbor points around the centroid are found by the K-Nearest Neighbor (KNN) algorithm, and the 3 neighbor points of each of the 32 points are mapped. These neighbor points are organized into a set V , where each element v i = (x i , f i ) represents a point and its 3D coordinates x i and the corresponding point feature f i . Then use the graph neural network (GNN) to update the vertex feature through the features of adjacent edges, and maximum pool the feature to obtain the geometric structure F geo around the centroid, which represents as follows:

[0049]

[0050] F geo = MaxPool(V (t+1) )

[0051] where represents feature concatenation. After concatenating the encoded feature, the category feature and the geometric feature, the self-attention mechanism is used, and then the features are fused through the MLP to obtain the centroid feature F attn , which represents as follows:

[0052]

[0053] F attn = MLP(σ((W1F ctr )·(W2F ctr )^T)·(W3F ctr ))

[0054] where PE(·) represents feature encoding, and W1, W2, W3 represent the corresponding weight matrix.

[0055] ​The neural network is trained for 80 cycles on an NVIDIA RTX3090 graphics card, the BatchSize is 8, the optimizer adopts Adam with a learning rate of 0.001, and the code is implemented by Pytorch. Finally, the experimental results, the method proposed in the application is compared with other published three-dimensional target detection methods, and table 1 shows the average accuracy of the three-dimensional target detection of the car class on the KITTI data set by comparing the existing method.

[0056] Table 1

[0057] Method Type Average accuracy VoxelNet Voxel-based one-stage 65.46 PointPillar Voxel-based one-stage 77.98 SECOND Voxel-based one-stage 76.48 Part-A2 Voxel-based two-stage 79.47 STD Point-voxel two-stage 79.80 PV-RCNN Point-voxel two-stage 83.90 3DSSD Point one-stage 78.86 IA-SSD Point one-stage 79.57 PointRCNN Point two-stage 78.63 Invention Point one-stage 84.24

[0058] The application is not limited to the above-mentioned embodiments. Non-essential modifications and replacements of the embodiments without departing from the principles and structures of the application still belong to the protection scope of the application.

Claims

1. A three-dimensional object detection method based on class enhancement and geometry enhancement, characterized in that, The method comprises the following steps: Step one, input laser point cloud as original point cloud, the original point cloud contains 4-dimensional features: center coordinates x, y, z and reflectivity i; Step two, randomly sample the original point cloud to 16k points; Step three, for the reserved 16k points, perform farthest point sampling FPS and ball query twice to obtain 4096 and 2048 center points and corresponding multi-scale aggregation features 1 and 2; Step four, for the 2048 center points and multi-scale aggregation feature 2, perform sampling and ball query twice through the category-enhanced multi-sampling CEMS module to obtain 1024 and 512 center points and corresponding multi-scale aggregation features 3 and 4; The sampling process of the category-enhanced multi-sampling CEMS module is as follows: 4.1, the reserved 2048 center points have 128-dimensional features, then the features are used to predict the corresponding categories by using an MLP to obtain category features, and then a softmax function is used to obtain the category score of each point; 4.2, based on the category score, the topk function is used to select the top 1024 points with the highest category score; 4.3, for the 2048 center points, the K nearest neighbor KNN algorithm is used to find the nearest 3 points from the 1024 points to re-interpolate the features; 4.4, fuse the interpolated features with the initial 128-dimensional features, and repeat 4.1-4.2; Step five, use an MLP to predict the centroid offset of the multi-scale aggregation feature 4, so that each point is offset to the centroid, and then use a geometry-enhanced centroid attention GECA module to obtain the centroid feature of the centroid point; The specific process of the geometry-enhanced centroid attention GECA module is: repeat twice using a graph convolutional network GNN to extract the geometric features around the centroid, and then concatenate the encoded features, category features and geometric features and pass them through a self-attention mechanism to obtain the centroid feature; Step six, use the centroid feature of the centroid point to classify the object, and predict the 3D position, size and direction of the object bounding box, and use a non-maximum suppression method to remove overlapping bounding boxes.

2. The class and geometry boosting based three-dimensional object detection method of claim 1, wherein, The specific process of step three is as follows: For the reserved 16k points, the Euclidean distance between the points is used as the measurement standard to perform farthest point sampling FPS to form 4096 center points; around each center point, a virtual spherical query range is constructed by using ball query with a radius of 0.2m and 0.8m to find 16 and 32 neighbor points; The neighbor points are used to upgrade the original 4-dimensional features to 64-dimensional features by using a multi-layer perceptron MLP, and the features are maximum-pooled to obtain a feature representing the local neighborhood, the local neighborhood features of the two different radii are fused to form a multi-scale aggregation MSG feature 1; repeat the above operation to perform farthest point sampling FPS to form 2048 center points and obtain the corresponding domain features, and fuse the neighborhood features of the two different radii to form a multi-scale aggregation MSG feature 2.

3. The class and geometry boosting based three-dimensional object detection method of claim 2, wherein, The specific process of step four is as follows: For 2048 center points, sampling is performed by the class-enhanced multi-sampling CEMS module, and then a virtual spherical query range is constructed around each center point using a spherical query with a radius of 0.8 m and 1.6 m to find 16 and 32 neighbor points; the original 64-dimensional feature is upgraded to 128-dimensional using the MLP for the neighbor points, and the feature is maximum-pooled to obtain a feature representing the local neighborhood, the neighborhood features of the two different radii are fused to form the multi-scale aggregation MSG feature 3; the above operation is repeated, the class-enhanced multi-sampling CEMS module is sampled, and the corresponding field features are obtained, the neighborhood features of the two different radii are fused to form the multi-scale aggregation MSG feature 4.

4. The class and geometry boosting based three-dimensional object detection method of claim 3, wherein, The specific process of the fifth step is: using the MLP to predict an offset relative to the centroid for the 256-dimensional feature of the reserved 512 points; moving the 512 points to the centroid based on the offset and obtaining the encoded feature after the movement; Then, the centroid feature of the centroid point is obtained using the geometry-enhanced centroid attention GECA module.

5. The class and geometry boosting based three-dimensional object detection method of claim 4, wherein, The specific operation of using the graph convolution network GNN to extract the geometric feature around the centroid is: 32 points around the centroid are found by the K-nearest neighbor KNN algorithm, graphing is performed on 3 neighbors of each of the 32 points, then the feature of each neighbor point is upgraded using the MLP, and the sum is taken as the geometric feature of the point.

Images