Single-Stage 3D Point Cloud Object Detection Method Based on Target Resampling

Abstract

The present invention discloses a single-stage three-dimensional point cloud object detection method based on target resampling, which mainly solves the problem of unbalanced attention between simple point cloud objects and difficult point cloud objects in the prior art. The implementation scheme is as follows: obtain point cloud data and divide the data set; perform data augmentation and target resampling on the training set point cloud data in sequence; perform voxelization preprocessing on the augmented training set and test set; build a lightweight three-dimensional point cloud object detection network and iteratively train it; use the trained network to infer the preprocessed test set point cloud samples to obtain prediction boxes; use non-maximum suppression to filter redundant prediction boxes to obtain the object detection results. By undersampling simple point cloud objects and oversampling difficult point cloud objects for resampling, the present invention balances the attention of the detector to difficult and easy point cloud objects, and uses residual connection and model lightweight detection technology to improve the detection accuracy, which can be used for accurate identification and positioning of objects in three-dimensional space.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision, and particularly relates to a three-dimensional point cloud object detection method, which can be used to identify and locate objects in a three-dimensional space. Background Art

[0002] As a key technology in the perception system of autonomous driving, three-dimensional object detection has received wide attention in recent years. Three-dimensional object detection can locate objects in a three-dimensional space and predict their categories. High-quality perception and positioning results are crucial for subsequent steps such as path planning and motion prediction in autonomous driving.

[0003] Three-dimensional object detection methods can generally be divided into two categories. One is the single-stage detection method, and the other is the two-stage detection method. The network structure of the single-stage detection method is usually relatively simple, and the inference speed is relatively fast. The two-stage detection method adds an additional refinement stage compared to the single-stage detection method, which is used to refine the prediction boxes and confidences generated in the first stage. Therefore, the two-stage detection method usually has higher detection accuracy than the single-stage detection method. However, due to the introduction of the refinement stage, the latency of the two-stage detection method is generally relatively high. Because the single-stage detection method has low latency and a simple network structure, it is often widely used in practical applications.

[0004] Due to the rotational scanning characteristics of lidar, the point cloud density varies with the detection distance. In the area closer to the lidar, the point cloud density is higher, while in the area farther from the lidar, the point cloud density is lower. In addition, there will be occlusion situations for the objects in the lidar point cloud. For example, the lidar rays will be occluded by the object itself and also by other objects. The above situations result in the lidar only being able to detect the local part of the object. For these reasons, the objects in the point cloud can be roughly divided into two categories. One is the simple point cloud object with dense points and low occlusion degree, and the other is the difficult point cloud object with sparse points and serious occlusion degree. Existing three-dimensional object detection algorithms usually have relatively low detection accuracy for difficult point cloud objects, while the accuracy for simple point cloud objects is much higher.

[0005] Yan Yan, Yu-Xing Mao, Bo Li proposed a single-stage three-dimensional object detection method based on sparse convolution in their published paper "Second: Sparsely embedded convolutional detection" (an international peer-reviewed academic journal on sensors and biosensors technology and techniques in 2018). This method uses sparse convolution instead of three-dimensional convolution in the model, thus greatly improving the model inference speed. However, the network structure used in this method is relatively simple, so the model's feature extraction ability is not strong, and it is not easy to extract representative features for difficult point cloud objects.

[0006] Tianjin University discloses a three-dimensional object detection method for point clouds based on deep learning in its patent document with the application number: CN202110334132.8. This method uses hierarchical voxel encoding to obtain voxel features and supplements height information to the point cloud BEV feature map to compensate for the height information lost during the convolutional downsampling process. However, due to the lack of consideration of the balance of the detector's attention to difficult and easy point cloud targets, the detection accuracy of this method is low. Summary of the Invention

[0007] The purpose of the present invention is to propose a single-stage three-dimensional point cloud object detection method based on target resampling in view of the above deficiencies of the prior art, so as to balance the detector's attention to difficult and easy targets and improve the overall detection accuracy of the detector.

[0008] The technical solution of the present invention is as follows: According to the fact that simple point cloud targets account for the majority in the point cloud, such targets are easy to converge during the training process, and the detector usually has a higher detection accuracy for simple point cloud targets and a much lower detection accuracy for difficult point cloud targets. Through the point cloud target resampling method, under-sampling is performed on simple point cloud targets and over-sampling is performed on difficult point cloud targets, so that the detector's attention to difficult and easy point cloud targets is more balanced; and the detection network is improved through residual connection technology and model lightweight technology, so that the detector can more effectively extract the representative features of difficult point cloud targets. The implementation steps are as follows:

[0009] (1) Obtain a point cloud data set through lidar detection and divide it into a training set and a test set at a ratio of 1:1;

[0010] (2) Perform data augmentation preprocessing on the point cloud data of the training set;

[0011] (3) Perform target resampling on the augmented point cloud data of the training set:

[0012] 3a) Divide the point cloud P1 into three intervals according to the detection distance from near to far;

[0013] 3b) Randomly downsample the point clouds in the three intervals at different sampling rates;

[0014] 3c) Divide the targets in the point cloud into simple point cloud targets and difficult point cloud targets according to the number of points contained in the cloud target;

[0015] 3d) Resample the point cloud targets, that is, randomly convert simple point cloud targets into difficult point cloud targets to reduce the number of simple point cloud targets, realize under-sampling of simple point cloud targets, and increase the number and diversity of difficult point cloud targets to realize over-sampling of difficult point cloud targets, so that the detection accuracy of the detector for difficult and easy point cloud targets is more balanced;

[0016] (4) Preprocess the enhanced training set and the original test set by voxelization;

[0017] (5) Build a 3D point cloud object detection network composed of a voxel encoding layer, a sparse residual 3D backbone network, a lightweight multi-scale 2D backbone network, and a multi-task detection head in cascade;

[0018] (6) Set the focal loss as the classification loss function The smooth L1 loss as the bounding box regression loss function The cross-entropy loss as the object orientation prediction loss function The overall loss is the weighted sum of the above three losses. Based on the preprocessed training set data, use the Adam optimization algorithm to train the 3D point cloud object detection network to obtain a trained single-stage 3D point cloud object detection network;

[0019] (7) Input the preprocessed test set into the trained 3D point cloud object detection network, and in the 3D space, locate the objects in the environment and predict the categories to obtain the classification confidence and the corresponding prediction bounding boxes;

[0020] (8) Post-process the prediction results, and use the non-maximum suppression method to filter out redundant prediction bounding boxes to obtain the final detection results.

[0021] Compared with the prior art, the present invention has the following advantages:

[0022] First, due to using the point cloud object resampling method, the present invention undersamples simple point cloud objects and oversamples difficult point cloud objects, so that the detector pays more balanced attention to easy and difficult point cloud objects, effectively improving the detection accuracy of the detector for difficult point cloud objects.

[0023] Second, due to using the residual connection technology and the model lightweight technology to improve the 3D point cloud object detection network, the present invention realizes a lightweight and strong feature extraction ability detection network, enabling the detector to more effectively extract the representative features of difficult point cloud objects, and further improving the detection accuracy while ensuring the efficiency of the detector. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 is the overall implementation flowchart of the present invention;

[0025] Figure 2 is the structure diagram of the 3D point cloud object detection network adopted by the present invention;

[0026] Figure 3 is the structure diagram of the lightweight convolutional block adopted by the present invention to construct the lightweight multi-scale 2D backbone network. DETAILED DESCRIPTION OF THE INVENTION

[0027] The embodiments and effects of the present invention will be further described in detail below with reference to the accompanying drawings.

[0028] Refer to Figure 1 , and the implementation steps of this example are as follows:

[0029] Step 1: Obtain point cloud data and divide it into data sets.

[0030] 1.1) Use a lidar to record point cloud data in multiple scenarios and multiple time periods, and sample the recorded point cloud data at a frequency of 2 Hz to obtain 10,000 frames of point cloud data;

[0031] 1.2) Mark the three-dimensional ground truth boxes for three types of targets, namely motor vehicles, pedestrians, and non-motor vehicles, in the point cloud, and divide the marked point cloud data set into a training set and a test set according to a ratio of 1:1.

[0032] Step 2: Perform data augmentation preprocessing on the point cloud samples in the training set.

[0033] 2.1) Randomly sample n s three-dimensional ground truth boxes and the target point cloud they contain from the training set, and then perform collision detection. If there is an overlapping area between the selected three-dimensional ground truth box and the existing three-dimensional ground truth boxes in the current point cloud sample, it is considered a collision. Finally, paste the target point cloud contained in the three-dimensional ground truth boxes without collision into the current point cloud to obtain point cloud where N0 and N s represent the number of points in the original point cloud and the number of points in the sampled point cloud, respectively;

[0034] 2.2) Randomly flip the point cloud P1 along the x-axis or y-axis to perform random flipping data augmentation;

[0035] 2.3) Randomly generate a rotation angle θ1 in the interval [-π / 4, π / 4], and rotate the point cloud P1 around the z-axis by the angle θ1 to perform random rotation data augmentation;

[0036] 2.4) Randomly generate a scale factor s1 in the interval [0.95, 1.05], and multiply the overall coordinate values of the point cloud P1 by this scale factor s1 to perform random scaling data augmentation;

[0037] 2.5) Shuffle the order of the points in the point cloud P1 after the above data augmentation, and filter out the points and three-dimensional ground truth boxes outside its detection range.

[0038] Step 3: Perform target resampling on the augmented training set point cloud data.

[0039] 3.1) Divide the point cloud P1 into three intervals from near to far according to the detection distance, namely near = [a, b), mid = [b, c), and far = [c, d), where a, b, and c represent three distance values from near to far respectively;

[0040] 3.2) Randomly downsample the point clouds in the three intervals of near, mid, and far at sampling rates of s1, s2, and s3 respectively;

[0041] 3.3) Classify the targets in the point cloud P1 according to the number of points contained in the cloud target:

[0042] If the number of points contained in a point cloud target exceeds the point number threshold n e , then consider this point cloud target as a simple point cloud target;

[0043] Otherwise, determine this point cloud target as a difficult point cloud target;

[0044] 3.4) Process different point cloud targets separately:

[0045] For simple point cloud targets, discard 0, 1, and 2 points on the surface of this point cloud target with probabilities of d1, d2, and d3 respectively, where d1 + d2 + d3 = 1;

[0046] For difficult point cloud targets, keep the original point cloud target unchanged;

[0047] 3.5) Process each target in the point cloud according to 3.4) to achieve resampling of the point cloud targets, that is, randomly convert simple point cloud targets into difficult point cloud targets to reduce the number of simple point cloud targets, achieve undersampling of simple point cloud targets, while increase the number and diversity of difficult point cloud targets, achieve oversampling of difficult point cloud targets, and make the detection accuracy of the detector for the two types of easy and difficult point cloud targets more balanced.

[0048] Step 4, perform voxelization preprocessing on the point cloud samples in the training set after data augmentation and the point cloud samples in the original test set.

[0049] 4.1) Define the voxel size as s x ×s y ×s z , and divide the point cloud according to the defined voxel size within the detection range, with a maximum of n p points assigned to each voxel;

[0050] 4.2) Define voxels without points as empty voxels and voxels with points as non-empty voxels, and judge whether the number of points in the voxel exceeds n p :

[0051] If the number of points in the voxel exceeds n pIf the number is greater than n, discard the extra points;

[0052] If the number is less than n p points, pad with 0s to obtain a non-empty voxel set V = {v1, …, v i , … v N1}, where N1 is the number of non-empty voxels; represents the i-th non-empty voxel, which contains n p points, and the feature dimension of each point is 4.

[0053] Step 5, build a lightweight 3D point cloud object detection network.

[0054] Refer to Figure 2 , and the specific implementation of this detection network is as follows:

[0055] 5.1) Build a voxel encoding layer:

[0056] 5.1.1) This layer is responsible for encoding the point cloud in the voxels. The encoding method is to calculate the feature mean of the points contained in the voxels, that is, encode each non-empty voxel in the input non-empty voxel set as follows:

[0057]

[0058] where p j is the feature of the j-th point in the i-th voxel v i , n is the number of points in each voxel, and f i is the encoded feature of the i-th voxel v i , where N1 is the number of non-empty voxels;

[0059] 5.1.2) Combine all the non-empty voxel encodings to obtain a non-empty voxel feature set:

[0060]

[0061] 5.2) Build a sparse residual 3D backbone network:

[0062] This network is composed of four sparse residual modules, which are used to extract features from 3D voxels to obtain a bird's-eye view feature map. Each module contains a 3D sparse convolutional layer and several residual semi-manifold sparse convolutional blocks. Each residual semi-manifold sparse convolutional block is composed of two semi-manifold sparse convolutional layers. After each semi-manifold sparse convolutional layer, there is a batch normalization layer and a linear rectifier unit ReLU in sequence. There is a residual connection between the input and output of the residual semi-manifold sparse convolutional block;

[0063] The number of residual semi-manifold sparse convolution blocks used in the four sparse residual modules are 1, 1, 2, and 2 respectively, and the number of feature channels are 16, 32, 64, and 64 respectively; each module downsamples and extracts features from the three-dimensional voxel features, and finally splices the three-dimensional voxel features along the height dimension to obtain a bird's-eye view feature map. Where C is the number of feature channels, and L and W are the length and width of the feature map respectively;

[0064] 5.3) Build a lightweight multi-scale two-dimensional backbone network:

[0065] The lightweight multi-scale two-dimensional backbone network consists of two lightweight modules, and each lightweight module is composed of multiple lightweight convolution blocks; each lightweight convolution block uses a depthwise convolution layer and a pointwise convolution layer to replace the traditional convolution layer, thereby effectively reducing the number of network parameters and improving the network calculation efficiency.

[0066] Refer to Figure 3 This lightweight convolution block is sequentially cascaded by a depthwise convolution layer, a batch normalization layer, a pointwise convolution layer, a rectified linear unit ReLU, and a pointwise convolution layer. A residual connection is provided between the input and output of this lightweight convolution block; the number of lightweight convolution blocks in the two lightweight modules are 5 and 5 respectively, and the number of feature channels are 128 and 256 respectively;

[0067] This lightweight multi-scale two-dimensional backbone network is used to extract multi-scale bird's-eye view feature maps, and the specific implementation is as follows:

[0068] 5.3.1) The first lightweight module only extracts features from the input bird's-eye view feature map F BEV without changing the scale, to obtain the first bird's-eye view feature map Where C1 represents the number of feature channels of the feature map F BEV1 and L and W respectively represent the length and width of the feature map F BEV1 ;

[0069] 5.3.2) The second lightweight module downsamples and further extracts features from the first bird's-eye view feature map F BEV1 to obtain the second bird's-eye view feature map Where C2 represents the number of feature channels of the feature map F BEV2 and L' and W' respectively represent the length and width of the feature map F BEV2 ;

[0070] 5.3.3) Upsample the second bird's-eye view feature map F BEV2 back to the original size through transposed convolution to obtain the third bird's-eye view feature map Where C3 represents the number of feature channels of the feature map F BEV3 and L and W respectively represent the length and width of the feature map F BEV3 ;

[0071] 5.3.4) Concatenate the first bird's-eye view feature map F BEV1 and the third bird's-eye view feature map F BEV3 to obtain a multi-scale bird's-eye view feature map

[0072] 5.4) Build a multi-task detection head:

[0073] This multi-task detection head is composed of three convolutional layers with a convolutional kernel size of 1×1 in parallel, responsible for the classification task, box regression task, and target orientation task respectively. These three convolutional layers output classification confidence predictions box regression predictions and target orientation predictions Among them:

[0074] n c represents the number of categories, L and W represent the length and width of the network output respectively. The number 7 represents seven attribute values of box regression predictions, namely the center point position (x, y, z) of the 3D target box, length, width, and height (dx, dy, dz), and the steering angle r. The number 2 represents two directions of target orientation prediction. A prediction of 1 represents the forward direction, and a prediction of 0 represents the reverse direction;

[0075] 5.5) Cascade the voxel encoding layer, sparse residual 3D backbone network, lightweight multi-scale 2D backbone network, and multi-task detection head in sequence to form a 3D point cloud object detection network.

[0076] Step 6, iteratively train the 3D point cloud object detection network.

[0077] 6.1) Set the focal loss as the classification loss function Smooth L1 loss as the box regression loss function Cross-entropy loss as the target orientation prediction loss function Their respective calculations are as follows:

[0078]

[0079]

[0080]

[0081] Among them, α and β are equal to 0.25 and 2 respectively, N c and N p represent the number of classification predictions and the number of positive anchor boxes respectively, c i and represent the classification true value and classification prediction at the i-th position respectively, b ij and respectively represent the j-th attribute value of the predicted box and the ground truth box at the i-th position, d i and respectively represent the ground truth value and the prediction of the target orientation at the i-th position;

[0082] 6.2) Let the overall loss function of the 3D point cloud object detection network be:

[0083]

[0084] where λ1 and λ2 respectively represent the weights of the box regression loss and the target orientation loss;

[0085] 6.3) Batch input the preprocessed training set point cloud data into the 3D point cloud object detection network, and use the attribute values of the ground truth 3D object box and the predicted 3D object box as the input of the loss function where the attribute values include the object category, the position of the center point of the object box, the length, width and height of the object box, and the orientation of the object box;

[0086] 6.4) Set the learning rate to 1e-3, the number of training epochs to 60, the batch size to 4, adopt a one-cycle learning rate adjustment strategy, use the Adam optimization algorithm and the backpropagation algorithm to update the network parameters, and select the network parameters with the minimum loss as the fixed parameters of the network to obtain a trained single-stage 3D point cloud object detection network.

[0087] Step 7, use the trained network to infer the test set point cloud data.

[0088] 7.1) Input the voxelized test set point cloud data into the trained single-stage 3D point cloud object detection network, and obtain the classification confidence prediction, box regression prediction, and target orientation prediction through the inference of this network;

[0089] 7.2) Decode the box regression prediction using the decoding method of the SECOND 3D object detector to obtain the predicted box, and correct the steering angle of the predicted box according to the target orientation prediction:

[0090] If the target orientation prediction is 0, reverse the steering angle of the predicted box;

[0091] If the target orientation prediction is 1, keep the original value;

[0092] Finally, obtain the initial predicted box set and the classification confidence set where n0 represents the number of initial predicted boxes.

[0093] Step 8, use non-maximum suppression to filter out redundant predicted boxes.

[0094] 8.1) Set the classification confidence threshold \(t1\). If the classification confidence of a certain prediction box is lower than \(t1\), then filter out this prediction box; collect the prediction boxes that have not been filtered out and their corresponding classification confidences to obtain a prediction box set and a classification confidence set where \(n1\) represents the number of prediction boxes after filtering out the prediction boxes with low classification confidence;

[0095] 8.2) Input the localization confidence set and the prediction box set Sort the prediction boxes according to the localization confidence, the larger the localization confidence, the higher the ranking, to obtain the sorted localization confidence set and the corresponding prediction box set

[0096] 8.3) Calculate the intersection-over-union ratio between pairwise prediction boxes;

[0097] 8.4) Set the filtering threshold as \(t4\). Taking the \(i\)-th prediction box \(b\) i ′ as a reference, determine whether \(b\) i ′ has a filtering flag:

[0098] If \(b\) i ′ has a filtering flag, then skip the processing of the prediction box \(b\) i ′;

[0099] If \(b\) i ′ does not have a filtering flag, then traverse the prediction box \(b\) i ′ +1 to Determine whether the intersection-over-union ratio of the prediction box \(b\) j ′ ∈[i+1,n1] and the prediction box \(b\) i ′ is greater than the filtering threshold \(t4\):

[0100] If it is greater, then add a filtering flag to the prediction box \(b\) j ′;

[0101] Otherwise, do not add a filtering flag;

[0102] 8.5) Traverse the prediction boxes \(b1'\) to Process each prediction box according to the process in 8.4);

[0103] 8.6) Collect the prediction boxes without filtering flags and their corresponding localization confidences of these boxes as the output of non-maximum suppression processing to complete the filtering of redundant prediction boxes.

[0104] The following combines simulation experiments to further illustrate the effects of the present invention.

[0105] 1. Simulation experiment conditions:

[0106] The simulation experiment hardware platform of the present invention is: AMD Reyzen 5900X CPU processor, 32GB of memory, and Nvidia GeForce RTX 3090 graphics card.

[0107] The simulation experiment software platform of the present invention is: Ubuntu 20.04 operating system and python 3.8.

[0108] The data used in the simulation experiment of the present invention is: KITTI dataset, which contains 7481 frames of publicly available point cloud samples. Among them, 3712 frames of point cloud samples are used as the training set, and another 3769 frames of point cloud samples are used as the validation set. The KITTI dataset contains multiple target categories, namely cars, pedestrians, and non-motor vehicles, and the most widely used one is the car category. Each target category is further divided into three different difficulties, namely easy, medium, and hard.

[0109] The present invention selects the mean average precision AP as the evaluation index, and evaluates the model respectively on the three difficulties of the car category in the KITTI dataset.

[0110] 2. Simulation content and its result analysis:

[0111] Three-dimensional point cloud object detection networks are constructed respectively using the present invention and four existing technologies, Pointpillars, SECOND, SA-SSD, and PV-RCNN;

[0112] These five networks are trained respectively using the same training set data to obtain five trained three-dimensional point cloud object detection networks;

[0113] The same validation set is input into the five trained networks respectively to obtain the corresponding detection results, and the precision of each detection result is evaluated and compared using the existing evaluation index. The comparison results are shown in Table 1:

[0114] Table 1: Comparison table of simulation experiment results between the present invention and existing technologies

[0115]

[0116] The sources of the four existing technologies in Table 1 are as follows:

[0117] PointPillars is a 3D point cloud object detection method proposed by Lang et al. in their published paper "Pointpillars: Fast encoders for object detection from point clouds" (IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2019), abbreviated as PointPillars.

[0118] SECOND is a 3D point cloud object detection method proposed by Yan et al. in their published paper "Second: Sparsely embedded convolutional detection" (Sensors, 2018), abbreviated as SECOND.

[0119] SA-SSD is a 3D point cloud object detection method proposed by He et al. in their published paper "Structure aware single-stage 3d object detection from point cloud" (IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2020), abbreviated as SA-SSD.

[0120] PV-RCNN is a 3D point cloud object detection method proposed by Shi et al. in their published paper "Pv-rcnn: Point-voxel feature set abstraction for 3d object detection" (IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2020), abbreviated as PV-RCNN.

[0121] As can be seen from Table 1, the 3D point cloud object detection method of the present invention is higher than the four existing technical methods in terms of multiple evaluation indicators, indicating that the present invention effectively improves the overall accuracy of the detector by using a new lightweight 3D point cloud object detection network and an object resampling method.

Claims

1. A single-stage 3D point cloud object detection method based on target resampling, characterized in that, It includes the following: (1) Obtain a point cloud data set through lidar detection and divide it into a training set and a test set at a ratio of 1:1; (2) Perform data augmentation on the training set point cloud data; (3) Perform target resampling on the augmented training set point cloud data: 3a) Divide the point cloud P1 into three intervals from near to far according to the detection distance, namely near = [a, b), mid = [b, c), and far = [c, d), where a, b, and c respectively represent three distance values from near to far; 3b) Randomly downsample the point clouds in the three intervals of near, mid, and far at sampling rates of s1, s2, and s3 respectively; 3c) Classify the targets in the point cloud P1 according to the number of points contained in the point cloud target: If the number of points contained in a point cloud target exceeds the point number threshold n e , then this point cloud target is considered a simple point cloud target; Otherwise, identify the point cloud target as a difficult point cloud target; 3d) Process different point cloud targets respectively: For simple point cloud targets, discard 0, 1, and 2 points on the surface of the point cloud target with probabilities of d1, d2, and d3 respectively, where d1 + d2 + d3 = 1; For difficult point cloud targets, keep the original point cloud target unchanged; 3e) Process each target in the point cloud according to 3d) to achieve resampling of the point cloud target, that is, randomly convert simple point cloud targets into difficult point cloud targets to reduce the number of simple point cloud targets, achieve undersampling of simple point cloud targets, and increase the number and diversity of difficult point cloud targets, achieve oversampling of difficult point cloud targets, so that the detection accuracy of the detector for easy and difficult point cloud targets is more balanced; (4) Perform voxelization preprocessing on the augmented training set and the original test set; (5) Build a lightweight 3D point cloud target detection network: 5a) Select a voxel encoding layer for encoding the training set and test set after voxelization to obtain a non-empty voxel feature set F V ; 5b) Build a sparse residual 3D backbone network composed of four sparse residual modules for feature extraction of 3D voxels to obtain a bird's-eye view feature map. Each module contains a 3D sparse convolutional layer and several residual semi-manifold sparse convolutional blocks. Each residual semi-manifold sparse convolutional block is composed of two semi-manifold sparse convolutional layers. After each semi-manifold sparse convolutional layer, a batch normalization layer and a rectified linear unit ReLU are connected in sequence. There is a residual connection between the input and output of the residual semi-manifold sparse convolutional block; 5c) Build a lightweight multi-scale 2D backbone network composed of two lightweight modules for extracting multi-scale bird's-eye view feature maps. Each lightweight module is composed of multiple lightweight convolutional blocks; 5d) Select a multi-task detection head to complete different detection tasks; 5e) Cascade the voxel encoding layer, the sparse residual 3D backbone network, the lightweight multi-scale 2D backbone network, and the multi-task detection head in sequence to form a 3D point cloud target detection network; (6) Train the 3D point cloud target detection network: 6a) Let the focal loss be the classification loss function The smooth L1 loss is the bounding box regression loss function The cross - entropy loss is the object orientation prediction loss function Let the overall loss function of the 3D point cloud object detection network be: where λ1 and λ2 represent the weights of the bounding box regression loss and the object orientation loss respectively; 6b) Based on the preprocessed training set data, use the Adam optimization algorithm to train the 3D point cloud target detection network until the overall loss function converges to obtain a trained single-stage 3D point cloud target detection network; (7) Input the preprocessed test set into the trained 3D point cloud object detection network, and in the 3D space, locate the objects in the environment and predict their categories to obtain the classification confidence and corresponding prediction bounding boxes. (8) Post-process the prediction results, that is, use the non-maximum suppression method to filter out redundant prediction bounding boxes to obtain the final detection results.

2. The method according to claim 1, wherein (2) Data augmentation for the training set is implemented as follows: 2a) Augment the data by pasting the 3D ground truth bounding boxes: Randomly select n from the training set s three-dimensional ground truth boxes and the target point clouds they contain. If there is an overlapping area with the existing three-dimensional ground truth boxes, a collision occurs. The target point clouds contained in the three-dimensional ground truth boxes without collision are pasted into the current point cloud to obtain the point cloud where N0 represents the number of points in the original point cloud, and N s represents the number of points in the target point cloud contained in the sampled three-dimensional ground truth box, and 4 represents the feature dimension of the points in the point cloud; 2b) Randomly flip the point cloud P1 along the x-axis or y-axis for random flipping data augmentation. 2c) Randomly generate a rotation angle θ1 within the interval [-π / 4, π / 4], and rotate the point cloud P1 by θ1 around the z-axis for random rotation data augmentation. 2d) Randomly generate a scale factor s1 within the interval [0.95, 1.05], and multiply the overall coordinate values of the point cloud P1 by this scale factor s1 for random scaling data augmentation of the point cloud. 2e) Shuffle the order of the points in the point cloud P1 after the above data augmentation, filter out the points and 3D ground truth bounding boxes outside its detection range to obtain the data-augmented training set.

3. The method according to claim 1, wherein (4) Voxelization preprocessing for the training set and test set is implemented as follows: 4a) Define the voxel size as s x × s y × s z , and segment the point cloud within the detection range according to the defined voxel size, with a maximum of n points assigned to each voxel; 4b) Define the voxels without points as empty voxels and the voxels with points as non-empty voxels, and determine whether the number of points in the voxel exceeds n: If the number of points in the voxel exceeds n, discard the extra points. If there are less than n, fill with 0 to obtain a non-empty voxel set Where: N1 is the number of non-empty voxels; represents the i-th non-empty voxel, which contains n points, and the feature dimension of each point is 4.

4. The method according to claim 1, wherein The non-empty voxel feature set F obtained by the voxel encoding layer in 5a) above V , is implemented as follows: 5a1) Encode each non-empty voxel according to the following formula: where p j is the feature of the j-th point in the i-th voxel v i , n is the number of points in each voxel, and f i is the encoded feature of the i-th voxel v i ; 5a2) Combine the encodings of all non-empty voxels to obtain the non-empty voxel feature set: where N1 is the number of non-empty voxels.

5. The method according to claim 1, characterized in that In 5b), the numbers of residual semi-manifold sparse convolution blocks used in the four sparse residual modules are 1, 1, 2, and 2 respectively, and the numbers of feature channels are 16, 32, 64, and 64 respectively. Each module downsamples and extracts features from the three-dimensional voxel features, and finally the three-dimensional voxel features are concatenated along the height dimension to obtain the bird's-eye view feature map. Where C is the number of feature channels, and L and W are the length and width of the feature map respectively.

6. The method according to claim 1, characterized in that, In 5c), the lightweight convolutional block is composed of a depthwise convolutional layer, a batch normalization layer, a pointwise convolutional layer, a rectified linear unit ReLU, and a pointwise convolutional layer in sequence. A residual connection is provided between the input and output of this lightweight convolutional block.

7. The method according to claim 1, characterized in that, In 5c), two lightweight modules are used to extract multi-scale bird's-eye view feature maps, which are implemented as follows: The first lightweight module only extracts features from the bird's-eye view feature map F BEV without changing the scale, and obtains the first bird's-eye view feature map where C1 represents the number of feature channels, and L and W respectively represent the length and width of the feature map F BEV1 ; The second lightweight module downsamples and further extracts features from the bird's-eye view feature map F BEV1 to obtain the second bird's-eye view feature map and then upsamples it back to its original size through transposed convolution to obtain the third bird's-eye view feature map where C2 and C3 both represent the number of feature channels, and L' and W' respectively represent the length and width of the second bird's-eye view feature map F BEV2 ; Concatenate the first bird's-eye view feature map F BEV1 and the third bird's-eye view feature map F BEV3 to obtain a multi-scale bird's-eye view feature map 8. The method according to claim 1, wherein The multi-task detection head in 5d) is composed of multiple independent and parallel convolutional layers, which are respectively used to complete the classification task, bounding box regression task, and object orientation task.

9. The method according to claim 1, characterized in that, (6b) Based on the preprocessed training set data, use the Adam optimization algorithm to train the 3D point cloud object detection network, which is implemented as follows: 6b1) Set the learning rate to 1e-3, the number of training epochs to 60, and the batch size to 4. 6b2) Adopt a one-cycle learning rate adjustment strategy, and use the Adam optimization algorithm and backpropagation algorithm to update the parameters of the 3D point cloud object detection network on the preprocessed training set data. 6b3) Select the network parameters with the minimum loss as the fixed parameters of the network to complete the training process.

10. The method according to claim 1, characterized in that, (8) Use non-maximum suppression to remove the redundant prediction bounding boxes in the prediction bounding box set B2, which is implemented as follows: 8a) Input the set of location confidence S = {s1, …, s i , … s n′} and the set of prediction boxes B2 = {b1, …, b i , … b n′}. According to the location confidence, sort the prediction boxes. The larger the location confidence, the higher the ranking. Obtain the sorted set of location confidence S′ and the corresponding set of prediction boxes B′, where n′ is the number of prediction boxes; 8b) Obtain the intersection over union (IoU) between each pair of prediction bounding boxes according to the rotation IoU calculation formula. 8c) Set the filtering threshold as t4, and taking the i-th prediction box b i as a reference, determine whether b i has a filtering flag: If b i has a filtering flag, skip the processing of the prediction box b i ; if b i does not have a filtering flag, traverse the prediction box b i+1 to b n′ , and judge whether the intersection over union of the prediction box b j∈[i+1,n′] and the prediction box b i is greater than the filtering threshold t4: If it is greater than, then for the prediction box b j Add a filtering flag; Otherwise, do not add a filtering flag. 8d) Traverse the prediction boxes b1 to b n′ , and process each prediction box b according to 8c) i∈[1,n′] ; 8e) Collect the prediction bounding boxes without filtering flags and their corresponding localization confidences as the output of the non-maximum suppression to complete the filtering of redundant prediction bounding boxes.

Images