Three-dimensional target detection method based on multi-task supervision and cross attention mechanism

Abstract

The invention provides a three-dimensional target detection method based on multi-task supervision and a cross attention mechanism. The method comprises the following steps: constructing a three-dimensional target detection network; the three-dimensional target detection network comprises an input layer, a category-based down-sampling module, a multi-task-based feature supervision module, a cross attention mechanism-based multi-scale feature extraction module and an output layer; and inputting the laser radar point cloud as an original point cloud, then carrying out random sampling, carrying out feature extraction through a class-based down-sampling module, a multi-task-based feature supervision module and a cross attention mechanism-based multi-scale feature extraction module, obtaining instance features, and finally carrying out to-be-detected target classification. According to the method, the structure expression capability of the network is remarkably improved, and the detection performance and classification precision of small objects are enhanced; the introduced auxiliary task helps the network to better cluster and identify small objects, and the overall detection effect of three-dimensional target detection is improved.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving perception technology, and in particular to a three-dimensional target detection method based on multi-task supervision and cross-attention mechanism, and in particular to a three-dimensional target detection method based on multi-scale feature extraction of multi-task supervision and cross-attention mechanism. Background Art

[0002] The rapid development of autonomous driving technology is reshaping people's transportation methods. Object detection, as an important part of the autonomous driving system, has always faced challenges in detecting small objects (such as pedestrians and cyclists). Autonomous vehicles acquire environmental data by equipping them with perception devices such as monocular cameras and lidars, and process these data through the autonomous driving system to control the vehicle. 3D object detection of lidar point clouds is one of the key technologies. It inputs the point cloud into the 3D object detector and outputs the object detection box in the scene.

[0003] At present, the 3D target detection methods of LiDAR point clouds can be roughly divided into three categories: point-based methods, voxel-based methods, and point-voxel combined methods. The point-based method directly extracts features on the original point cloud, relies on downsampling to save key points, and then extracts instance features for detection. However, due to the large receptive field of the final layer, small object detection is easily interfered by noise. The voxel-based method converts the point cloud into a regular voxel grid and uses 3D sparse convolution to extract features. Although the feature extraction effect is good, quantization errors will be generated in the voxelization process, affecting the detection accuracy. The point-voxel combined method combines the advantages of both. Although the detection accuracy is improved, it brings higher memory and computational consumption. In summary, each method has its own limitations when dealing with small object detection. How to improve detection accuracy while controlling computing resource consumption is an important research direction.

[0004] The most advanced point-based methods, such as 3DSSD and IASSD, only use downsampling and feature learning methods based on distance weight / class weight segmentation. Since these methods do not adequately extract structural features and geometric relationship features in the point cloud during the feature learning stage, the detection of small objects or occluded objects in the scene is relatively inaccurate. This can become a potential safety hazard in complex real-world environments. In addition, they do not fully extract instance features, and when predicting bounding boxes, they often cannot accurately adapt to the complex contours of actual objects. Insufficient feature extraction affects the accuracy of the bounding box when the model deals with small objects, occluded objects, or objects with special shapes in complex backgrounds. This not only affects the accuracy of detection, but also reduces the vehicle's ability to cope with dynamic and complex traffic environments. Summary of the invention

[0005] The first objective of the present invention is to address the deficiencies in the prior art and to provide a three-dimensional target detection method based on multi-task supervision and cross-attention mechanism, which improves the current point-based three-dimensional target detection method for small objects and target detection in complex environments.

[0006] In order to achieve the above object, the technical solution steps proposed by the present invention are as follows: Step 1: construct a three-dimensional object detection network; the three-dimensional object detection network includes an input layer, a category-based downsampling module, a multi-task-based feature supervision module, a multi-scale feature extraction module based on a cross-attention mechanism, and an output layer; Step 2: Input the LiDAR point cloud as the original point cloud into the input layer. The original point cloud consists of 4-dimensional features: center coordinates and reflectivity ; Step 3: Use random sampling algorithm to sample the original point cloud to 10600 points; Step 4: Perform farthest point sampling and ball query on 10,600 points to obtain 4,096 center points and their corresponding first multi-scale aggregation features; perform farthest point sampling and ball query on the 4,096 center points again to obtain 2,048 center points and their corresponding second multi-scale aggregation features; Step 5: For the 2048 center points, downsampling and ball query are performed through the category-based downsampling module to obtain 1024 center points and their corresponding third multi-scale aggregation features; the 1024 center points are again subjected to the farthest point sampling and ball query through the category-based downsampling module to obtain 512 center points and their corresponding fourth multi-scale aggregation features; then the third multi-scale aggregation features and the fourth multi-scale aggregation features are optimized using the multi-task-based feature supervision module; Step 6: Use a multi-layer perceptron to predict the centroid offset and point feature score of the target to be detected for the optimized fourth multi-scale aggregation feature, so that each point is offset to the centroid; define the offset point as an instance point, and then delete the instance point to obtain the downsampled instance point; Step 7: Use a multi-scale feature extraction module based on the cross-attention mechanism to extract features from the downsampled instance points to obtain instance features around the downsampled instance points.

[0007] Step 8: Use instance features to classify the target to be detected, predict the three-dimensional position, size and deflection angle of the target bounding box, and use the non-maximum suppression method to eliminate bounding boxes with too high overlap rates, and output them through the output layer.

[0008] Furthermore, the specific process of step 4 is as follows: For the 10,600 points after random sampling, the Euclidean distance between points is used as the weight, and 4096 center points are formed by sampling the farthest point; a spherical query is used near each center point, with 0.2m and 0.8m as the radius, to construct a spherical query range to find 16 and 32 neighboring points; a multi-layer perceptron is used to upgrade the 4-dimensional features of the neighboring points to 64 dimensions, and the maximum pooling is used to aggregate the features to obtain the point features representing the local neighborhood, and the local neighborhood features of two different radii are fused to form the first multi-scale aggregation feature; the above operation is repeated to perform farthest point sampling to form 2048 center points, and the corresponding domain features are obtained, and the neighborhood features of two different radii are fused to form the second multi-scale aggregation feature.

[0009] Furthermore, the specific process of step five is as follows: For the 2048 center points, sampling is performed through the category-based downsampling module, and then a virtual spherical query range is constructed around each center point with a radius of 0.8m and 1.6m to find 16 and 32 neighbor points; the original 64-dimensional features of the neighbor points are upgraded to 128 dimensions using a multi-layer perceptron, and the maximum pooling is used to aggregate the features to obtain features representing the local neighborhood, and the neighborhood features of two different radii are fused to form a third multi-scale aggregated feature; the above operations are repeated to perform category-based downsampling, and the corresponding domain features are obtained, and the neighborhood features of two different radii are fused to form a fourth multi-scale aggregated feature. The third and fourth multi-scale aggregated features are optimized using a multi-task-based feature supervision module.

[0010] Furthermore, the sampling process of the category-based downsampling module is as follows: 5.1. The 2048 retained center points have 128-dimensional features. Then a multi-layer perceptron is used to predict the corresponding category to obtain the category features and category scores. 5.2. Based on the category scores, retain the top 1024 points with the highest category scores; 5.3. Repeat 5.1-5.2 to finally obtain the third and fourth multi-scale aggregation features.

[0011] Furthermore, the specific process of the multi-task-based feature supervision module is as follows: for each point located near an object in a real scene, feature supervision is performed using a component prediction task, a center prediction task, and a category prediction task. Specifically, for the component prediction task, the multi-scale aggregated features of the point are used to predict the normalized coordinates of the point in the object using a multi-layer perceptron; for the center prediction task, a multi-layer perceptron is used to predict the offset of the point relative to the center of the object; for the category prediction task, a multi-layer perceptron is used to predict the true category of the point.

[0012] Furthermore, the specific process of step six is: using a multilayer perceptron to predict an offset relative to the centroid for the 256-dimensional features of the 512 retained points; moving the 512 points toward the centroid based on the offset, and obtaining the instance point features after the move; and then using the instance point deletion module to downsample the instance points. Specifically, the instance point deletion module uses a virtual spherical non-maximum suppression method for the 512 points to delete multiple points with lower category scores.

[0013] Furthermore, the multi-scale feature extraction module based on the cross-attention mechanism has the following specific process: for the retained instance points, a ball query is used around each instance point, with 0.8m and 1.6m as the radius, to construct a virtual spherical query range to search for points in the optimized third multi-scale aggregation feature, and obtain 16 and 32 neighbor points; for each neighbor point, a 128-dimensional optimized feature is extracted using a converter based on the cross-attention mechanism, and the maximum value pooling is used to aggregate the features to obtain a multi-scale feature representing the local neighborhood, and the neighborhood features of two different radii are fused to form a third multi-scale aggregation feature based on the cross-attention mechanism; the above steps are repeated to obtain a fourth multi-scale aggregation feature based on the cross-attention mechanism, and the two features are spliced ​​to obtain the final instance feature.

[0014] A second object of the present invention is to provide a three-dimensional object detection system for executing the above method, comprising the following modules: Point cloud acquisition module, used to acquire lidar point cloud and perform random sampling; The target detection module processes the randomly sampled point cloud through a three-dimensional target detection network and predicts the target to be detected.

[0015] A third object of the present invention is to provide a computer-readable storage medium having a computer program stored thereon, which implements the above method when the program is executed by a processor.

[0016] A fourth object of the present invention is to provide an electronic device, comprising a processor and a memory, wherein the memory stores a computer program, and the processor implements the above method when executing the computer program.

[0017] Compared with the prior art, the present invention has the following advantages: First, the present invention proposes a multi-task-based feature supervision module. This module introduces component prediction tasks, center prediction tasks, and category prediction tasks, which significantly improves the structural expression ability of the network and enhances the detection performance and classification accuracy of small objects. These auxiliary tasks help the network to better cluster and identify small objects, improving the overall detection effect of three-dimensional target detection.

[0018] Second, the present invention proposes a multi-scale feature extraction module based on the cross-attention mechanism. Through the multi-scale feature sparse extraction combined with the cross-attention mechanism, it can better adapt to targets of different sizes and shapes, especially in the detection of small objects. At the same time, the instance point deletion module reduces the amount of calculation and maintains an efficient reasoning speed.

[0019] Third, the method of the present invention has been comprehensively tested on multiple large-scale public datasets (such as KITTI, ONCE, and Waymo Open). The experimental results show that it has achieved significant improvements in multiple categories, especially in point-based methods, showing the best performance and good robustness. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 A flow chart of the technical solution proposed by the present invention; Figure 2 This is the structural diagram of the three-dimensional target detection network proposed in this invention.

[0021] Figure 3 Schematic diagram of the multi-scale feature extraction module based on the cross-attention mechanism proposed in the present invention. DETAILED DESCRIPTION

[0022] The following is a specific embodiment, and combined with the attached Figure 1 and attached Figure 2 , the technical solution of a three-dimensional target detection method based on class enhancement and geometric enhancement proposed in the present invention is further specifically explained.

[0023] Dataset processing: The training samples and test samples are from public datasets. The present invention has conducted experiments on the popular ONCE dataset to verify the effectiveness of the method of the present invention. For object detection on the ONCE dataset, there are 5 categories (cars, buses, trucks, pedestrians, and cyclists), which are divided into three difficulties according to the distance (less than 30 meters, 30-50 meters, and greater than 50 meters). mAP is used as the main evaluation criterion, and the average accuracy is used to evaluate the model performance (the average of the three difficulty mAPs).

[0024] like Figure 1 As shown, the present invention provides a three-dimensional target detection method based on multi-task supervision and cross attention mechanism, comprising the following steps: Build a 3D object detection network; Input point cloud: First input the laser radar point cloud, the point cloud data contains the center coordinates and reflectivity Four-dimensional features. The point cloud is sampled to 16,000 points using a random sampling algorithm. The sampled point cloud is subjected to two farthest point sampling (FPS) and ball queries to obtain 4096 and 2048 center points and their corresponding first and second multi-scale aggregate features, respectively. Through the category-enhanced downsampling and multi-task feature supervision modules, the 2048 center points are sampled twice and ball queries are performed to obtain 1024 and 512 center points and their corresponding third and fourth multi-scale aggregate features, respectively. For the fourth multi-scale aggregate feature, a multi-layer perceptron (MLP) is used to predict the centroid offset of the target, offset each point to the target centroid, generate instance points, and calculate the point feature score. Then, the instance points are further downsampled through the instance point removal module. The downsampled instance points are feature extracted, and a multi-scale feature extraction module based on the cross-attention mechanism is used to extract rich features from around the instance points. The features of the instance points are used to classify the target and predict the three-dimensional bounding box of the target, including the position, size, and deflection angle. Finally, the bounding boxes with too high overlap are eliminated by non-maximum suppression (NMS).

[0025] like Figure 2 As shown, the three-dimensional object detection network constructed by the present invention includes an input layer, a category-based downsampling module, a multi-task-based feature supervision module, a multi-scale feature extraction module based on a cross-attention mechanism, and an output layer (environment prediction head); the multi-task-based feature supervision module and the multi-scale feature extraction module based on a cross-attention mechanism in the three-dimensional object detection network are introduced in detail below.

[0026] Multi-task based feature supervision module: Multilayer perceptrons are added between each layer of the backbone network, and their output dimensions are different depending on the auxiliary task. These networks are only used during training, so they do not increase the memory and time consumption of the inference phase.

[0027] The present invention represents each layer of point cloud in the backbone network as , where N is the number of layers of the backbone network, and the point-by-point features corresponding to each layer of point cloud are , in general, N=4. The present invention can improve the feature quality by directly constraining the output features of the additional network. This module is designed with three tasks:

[0028] 1. Parts prediction task In this task, the output of the additional branch network is used to predict the position of the foreground point in the object bounding box, so that the feature can perceive the size and shape information of the object, thereby constraining the search range of the feature. For each foreground point, the present invention uses binary cross entropy loss for constraint, the formula is as follows:

[0029] in is the indicator function that makes the constraint in the foreground point. Indicates The component prediction results corresponding to the layer point cloud, Indicates The position of the target component corresponding to the foreground point in the layer point cloud. Specifically, It should be noted that the component positions are constrained to range, this is achieved through Function implementation. The loss of all component prediction branches in the entire network is included in middle.

[0030] 2. Central prediction task In the center prediction task, an additional branch network is used to predict the center point of the object, further improving the positioning accuracy of the network and enhancing the network's clustering ability. Specifically, for each foreground point, the center point of the corresponding bounding box is used as the target point, and The loss is trained, and its formula is as follows:

[0031] in Indicates the offset value of the foreground point corresponding to the center point of the object's bounding box. Represents the network output of the central prediction branch.

[0032] 3. Category Prediction Task This task is used to predict the category scores of point clouds to distinguish foreground points from background points. Therefore, this task has two functions: on the one hand, as an auxiliary task, predicting categories to constrain feature learning; on the other hand, its results can be used to optimize the downsampling process. Its training loss function is as follows:

[0033] Where C represents the number of categories, and different data sets have different numbers of object categories; Indicate point Predicted as category The confidence level, Yes The true category label.

[0034] The structure of the multi-scale feature extraction module based on the cross-attention mechanism is as follows Figure 3 As shown, specifically, in this module: First, a foreground score is predicted for the instance point set, and then the instance point set is sampled by non-maximum suppression according to the foreground score to reduce the number of instance points and weaken the compactness between instance points. The remaining instance point set first uses the original feature extraction method to extract instance features, and then uses the cross-attention mechanism in the features of different levels of the backbone network to extract multi-scale effective information. The specific formula is as follows:

[0035] in, represents the initial instance features, The feature dimension is , As the backbone network The layer is queried features, and the feature dimension is First use (Position encoding layer) performs position encoding of local coordinates to obtain , and then use the fully connected layer to align the query features and key-value features. Then use the multi-head cross attention mechanism for feature extraction, and finally use Align output features and Instance features. Finally, two feature matrices are obtained, and the features are directly added to obtain the final instance features.

[0036] The present invention uses the Pytorch framework to train the neural network on the NVIDIA RTX3090 graphics card for a total of 80 cycles, with a batch size of 8, the optimizer selected as Adam, and the learning rate set to 0.001. The experimental results show that the proposed method performs well when compared with other public 3D object detection methods. Table 1 shows the average accuracy of the proposed method for 3D object detection of automobiles on the ONCE dataset, and a detailed comparison is made with existing methods.

[0037] Table 1 Comparison of the test results of the method of the present invention and the commonly used method method type Average accuracy PointPillars Voxel-based 68.57 CenterPoints Voxel-based 66.79 3D-SSD (Reproduced) Based on point one stage 70.73 IA-SSD Based on point one stage 70.30 IA-SSD (Reproduced) Based on point one stage 70.27 PointRCNN Based on the second stage 52.09 The present invention Based on point one stage 72.60 The present invention is not limited to the above-mentioned embodiments. Without departing from the principle and structure of the present invention, non-substantial modifications and replacements of the implementation methods still fall within the protection scope of the present invention.

Claims

1. A three-dimensional object detection method based on multi-task supervision and cross-attention mechanism, characterized in that: The method comprises the following steps: Step 1: construct a three-dimensional object detection network; the three-dimensional object detection network includes an input layer, a category-based downsampling module, a multi-task-based feature supervision module, a multi-scale feature extraction module based on a cross-attention mechanism, and an output layer; Step 2: Input the LiDAR point cloud as the original point cloud into the input layer. The original point cloud consists of 4-dimensional features: center coordinates and reflectivity ; Step 3: Use random sampling algorithm to sample the original point cloud to 10600 points; Step 4: Perform farthest point sampling and ball query on 10,600 points to obtain 4,096 center points and their corresponding first multi-scale aggregation features; perform farthest point sampling and ball query on the 4,096 center points again to obtain 2,048 center points and their corresponding second multi-scale aggregation features; Step 5: For the 2048 center points, downsampling and ball query are performed through the category-based downsampling module to obtain 1024 center points and their corresponding third multi-scale aggregation features; the 1024 center points are again subjected to the farthest point sampling and ball query through the category-based downsampling module to obtain 512 center points and their corresponding fourth multi-scale aggregation features; then the third multi-scale aggregation features and the fourth multi-scale aggregation features are optimized using the multi-task-based feature supervision module; Step 6: Use a multi-layer perceptron to predict the centroid offset and point feature score of the target to be detected for the optimized fourth multi-scale aggregation feature, so that each point is offset to the centroid; define the offset point as an instance point, and then delete the instance point to obtain the downsampled instance point; Step 7: Use a multi-scale feature extraction module based on a cross-attention mechanism to extract features from the downsampled instance points to obtain instance features around the downsampled instance points; Step 8: Use instance features to classify the target to be detected, predict the three-dimensional position, size and deflection angle of the target bounding box, and use the non-maximum suppression method to eliminate bounding boxes with too high overlap rates, and output them through the output layer.

2. The method according to claim 1, characterized in that The specific process of step 4 is as follows: For the 10,600 randomly sampled points, the Euclidean distance between points is used as the weight, and 4,096 center points are formed by sampling the farthest point. A spherical query is used near each center point, with a radius of 0.2m and 0.8m to construct a spherical query range to find 16 and 32 neighbor points. For neighboring points, a multi-layer perceptron is used to upgrade the 4-dimensional features of the points to 64 dimensions, and the maximum pooling is used to aggregate the features to obtain point features representing the local neighborhood. The local neighborhood features of two different radii are fused to form the first multi-scale aggregated features. Repeat the above operation, perform farthest point sampling to form 2048 center points, obtain the corresponding domain features, and fuse the neighborhood features of two different radii to form the second multi-scale aggregation feature.

3. The method according to claim 1, characterized in that Step 5: The specific process is as follows: For the 2048 center points, downsampling is performed through the category-based downsampling module, and then a virtual spherical query range is constructed around each center point with a radius of 0.8m and 1.6m to find 16 and 32 neighbor points. The original 64-dimensional features of the neighbor points are upgraded to 128 dimensions using a multi-layer perceptron, and the maximum pooling is used to aggregate the features to obtain the features representing the local neighborhood. The neighborhood features of two different radii are fused to form the third multi-scale aggregated features. Repeat the above operations, perform category-based downsampling on the 1024 center points, obtain the corresponding domain features, fuse the neighborhood features of two different radii to form the fourth multi-scale aggregation feature, and use the multi-task-based feature supervision module to optimize the third and fourth multi-scale aggregation features.

4. The method according to claim 3, characterized in that The multi-task-based feature supervision module performs the following steps: for each point located near an object in a real scene, feature supervision is performed using a component prediction task, a center prediction task, and a category prediction task; for the component prediction task, the multi-scale aggregated features of the point are used to predict the normalized coordinates of the point in the object using a multi-layer perceptron; for the center prediction task, a multi-layer perceptron is used to predict the offset of the point relative to the center of the object; for the category prediction task, a multi-layer perceptron is used to predict the true category of the point.

5. The method according to claim 1, characterized in that The specific process of step six is: use a multilayer perceptron to predict an offset relative to the center of mass for the 256-dimensional features of the retained 512 points; move these 512 points toward the center of mass based on the offset, and obtain the instance point features after the move; then use the instance point deletion module to downsample the instance points; specifically, the instance point deletion module uses a virtual spherical non-maximum suppression method for the 512 points to delete multiple points with lower category scores.

6. The method according to claim 1, characterized in that The multi-scale feature extraction module based on the cross attention mechanism performs the following steps: for the retained instance points, a ball query is used around each instance point, with 0.8m and 1.6m as the radius, to construct a virtual spherical query range to search for points in the optimized third multi-scale aggregated features, and obtain 16 and 32 neighbor points; for each neighbor point, a 128-dimensional optimized feature is extracted using a converter based on the cross attention mechanism, and the maximum value pooling is used to aggregate the features to obtain a multi-scale feature representing the local neighborhood, and the neighborhood features of two different radii are fused to form a third multi-scale aggregated feature based on the cross attention mechanism; Repeat the above operations to obtain the fourth multi-scale aggregation feature based on the cross-attention mechanism, and concatenate the two features to obtain the final instance feature.

7. A three-dimensional object detection system for executing the method according to any one of claims 1 to 6, characterized in that: Includes the following modules: Point cloud acquisition module, used to acquire lidar point cloud and perform random sampling; The target detection module processes the randomly sampled point cloud through a three-dimensional target detection network and predicts the target to be detected.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

9. An electronic device comprising a processor and a memory, wherein a computer program is stored in the memory, wherein: When the processor executes the computer program, the method according to any one of claims 1 to 6 is implemented.

Images