Point cloud target detection method for sparse convolution feature distillation for automatic driving

Abstract

The application discloses a kind of sparse convolution feature distillation's point cloud target detection methods for automatic driving, the method includes: the preprocessing operation is carried out to point cloud data, isolated point and out-of-bound point are filtered out, and the voxelization operation is carried out to point cloud data, obtains training data set;Teacher neural network and student neural network are constructed;The training data set is used to train teacher neural network, and the parameter weight of teacher neural network is obtained;Training data set is used to train student neural network, meanwhile data is input into teacher network and is forward propagated, and distillation Loss is obtained by feature distillation, and the overall loss function is determined by classification Loss, regression Loss and distillation Loss, and the parameters of student network are adjusted using back propagation;The position, size, direction, category information of detection target are obtained by the neural network of the completion training to prediction data set and target prediction.

Description

Technical Field

[0001] This invention belongs to the fields of autonomous driving and deep learning, and specifically relates to a point cloud target detection method based on sparse convolutional feature distillation for autonomous driving. Background Technology

[0002] With the rapid development of various computing hardware and the continuous progress of deep learning algorithms, the ability to process diverse information is becoming stronger and faster, making the real-time application of various sensors such as radar, lidar, and cameras feasible for autonomous driving tasks.

[0003] Object detection provides prerequisite information for autonomous driving's subsequent planning, decision-making, and actions by perceiving the surrounding environment in real time. It is a key technology for autonomous driving, and obtaining an accurate and fast object detection algorithm has become a hot topic in the entire research field.

[0004] LiDAR can acquire point cloud data, boasting extremely high detection accuracy and speed, making it the most suitable detector for autonomous driving. However, in the past, the massive amounts of data generated per second resulted in slow point cloud data processing speeds and low efficiency, making deployment in real-world scenarios difficult. With continuous advancements in hardware technology, point cloud 3D object detection has gradually become a viable solution. In recent years, numerous detection algorithms based on deep learning have been proposed, often exhibiting strong adaptability and extremely high accuracy compared to traditional algorithms. Currently, deep learning-based 3D point cloud object detection algorithms are mainly divided into three types: 1) projection and image-based detection algorithms; 2) point cloud-based detection algorithms; and 3) voxel-based detection algorithms.

[0005] Projection- and image-based detection algorithms primarily map point cloud data onto a two-dimensional plane, using image processing algorithms for feature extraction and target detection. Point cloud-based detection algorithms directly process and detect point cloud data. Voxel-based detection algorithms first distribute the point cloud into a uniformly sized space, then extract and detect features from the voxels. The voxel method preserves a significant portion of the 3D spatial information while avoiding the drawbacks of irregular point cloud data storage, giving it advantages in both accuracy and detection speed.

[0006] Based on the size of the voxels used, the voxel method can be divided into the following two schemes: 1) Divide the point cloud into columnar voxels without dividing the height and encode them to obtain the feature map under the bird's-eye view and then perform further detection; 2) Divide the point cloud into smaller voxels with further division in height, use sparse convolution and other methods to extract features in 3D space, and then perform further detection.

[0007] Both methods have their advantages and disadvantages. Sparse convolution plays a crucial role in feature extraction in 3D space, effectively improving the detection accuracy of 3D objects. However, 3D sparse convolution requires a longer training and inference time, and the operators related to sparse convolution are not yet fully mature, posing a significant obstacle to model deployment and causing inconvenience to industry. Voxel-based methods, by avoiding the use of sparse convolution, can effectively reduce the training and inference time, but this also leads to a slight decrease in detection accuracy compared to networks using 3D sparse convolution. Under these circumstances, a method is needed to improve the detection accuracy of voxel-based methods without increasing the computational cost, bringing its accuracy closer to that of detection networks using 3D sparse convolution.

[0008] Knowledge distillation is a widely used model compression technique. Its basic idea is to transfer the knowledge of a large model (teacher network) to a small model (student network). During the distillation process, the small model learns the generalization ability of the large model and retains performance close to that of the large model.

[0009] Common knowledge distillation techniques include the following: 1) Feature-layer-based distillation, which distills the intermediate layer features output by the network and uses the intermediate layer features output by the teacher network to supervise the intermediate layer features of the student network; 2) Soft-label-based distillation, which distills the classification probabilities output by the network and uses the classification probability soft labels output by the teacher network to supervise the classification probability soft labels of the student network; 3) Relationship-based distillation, which utilizes the relevance of instances appearing in the same scene and uses instance information from the teacher network to help the student model converge faster. Summary of the Invention

[0010] In view of this, the main objective of the present invention is to provide a point cloud target detection method based on sparse convolutional feature distillation for autonomous driving.

[0011] To achieve the above objectives, the technical solution of the present invention is implemented as follows:

[0012] This invention provides a point cloud target detection method based on sparse convolutional feature distillation for autonomous driving, the method comprising:

[0013] Preprocess the point cloud data to filter out isolated and out-of-bounds points, and perform voxelization on the point cloud data to obtain the training dataset.

[0014] Constructing teacher neural networks and student neural networks;

[0015] The teacher's neural network is trained using a training dataset to obtain the weights of each parameter of the teacher's neural network.

[0016] The student neural network is trained using the training dataset, while the data is input into the teacher network for forward propagation. The distilled feature loss is obtained through feature distillation. The overall loss function is determined by the classification loss, regression loss and distilled feature loss. The gradient of the current student network parameters with respect to the overall loss function is determined. The parameters of the student network are adjusted using backpropagation to obtain the trained student neural network.

[0017] The trained student neural network is used to detect the prediction dataset, and the position, size, orientation, and type of the target to be detected are obtained through forward inference by the student network.

[0018] In the above scheme, the construction of the teacher neural network specifically refers to:

[0019] Construct a voxel feature encoding module and obtain voxel features and voxel numbers based on voxel data;

[0020] A 3D sparse convolution module is constructed and used multiple times to extract features from the point cloud at multiple scales. The voxel features at multiple levels are obtained based on the voxel features and voxel indices.

[0021] A bird's-eye view mapping module is constructed, and based on the voxel features of the largest spatial level scale among the voxel features of the multiple level scales, a high compression method is used to transform the 3D voxel feature space into a 2D bird's-eye view.

[0022] A 2D convolution module is constructed, and through a series of convolution and deconvolution operations, downsampling and upsampling are performed in different dimensions to transform the 2D bird's-eye view into a 2D feature map.

[0023] An object detection head is constructed, and based on the 2D feature image, a dense candidate box is obtained using an anchor-size box-based method. Non-maximum suppression is then used for further filtering to obtain the final prediction result.

[0024] In the above scheme, the construction of the student neural network specifically refers to:

[0025] A columnar voxel encoding module is constructed to divide the point cloud into columnar voxels. A multilayer perceptron is used to encode the point cloud within each columnar voxel into a feature vector, thereby obtaining the features of each columnar voxel.

[0026] A bird's-eye view mapping module is constructed, and based on the voxel features of the largest spatial level scale among the voxel features of the multiple level scales, a high compression method is used to transform the 3D voxel feature space into a 2D bird's-eye view.

[0027] Construct a 2D convolution module, and perform downsampling and upsampling in different dimensions through convolution and deconvolution operations to transform the 2D bird's-eye view into a 2D feature map;

[0028] An object detection head is constructed, and based on the 2D feature image, a dense candidate box is obtained using an anchor-size box-based method. Non-maximum suppression is then used for further filtering to obtain the final prediction result.

[0029] In the above scheme, obtaining the distillation loss through characteristic distillation specifically means:

[0030] Step 401: Based on the ground truth of the training set, i.e., the position, size, and object type of the 3D bounding boxes, project the 3D boxes onto the 2D bird's-eye view feature map. Let N be the number of 3D bounding boxes in the current frame. Adjust the size of the 3D bounding boxes in the 2D bird's-eye view feature map according to the downsampling factor of the convolution, and define their position, size, and orientation in the 2D bird's-eye view feature map to obtain the final projection result. Where k = 0, 1, ..., N-1, execute step 402;

[0031] Step 402: Use 3D bounding boxes from the training set k For each bounding box, calculate the integrity score of the point cloud within the box. k This scoring also applies to 3D bounding boxes. k The corresponding 2D bird's-eye view feature map

[0032] Step 403: Based on the projection of each bounding box onto the 2D bird's-eye view feature map Within the box, respectively according to 7*7 sampling points are selected at equal intervals on both the length and width scales to obtain the coordinates of each sampling point in the Grid. k，i , where i = 0, 1, ..., 48;

[0033] Step 404: Based on the coordinates of each sampling point Grid k，i The four points closest to the sampling point in the 2D bird's-eye view feature map are obtained, and they are located in the 2D bird's-eye view feature F of the teacher network. tea and F stu Based on the 2D bird's-eye view features of these four points, bilinear interpolation is used to obtain the features of the teacher network and student network at the sampling points. and Where i = 0, 1, ..., 48;

[0034] Step 405: For the 3D annotation box k The integrity score of the point cloud within the bounding box obtained in step 402 is used.k and the characteristics of teacher and student networks at the sampling points. and Calculate the distillation loss function Loss distill The loss function chosen is the mean squared error loss function (MSELoss):

[0035]

[0036] In the above scheme, step 402, which calculates the point cloud integrity score within each 3D bounding box, specifically involves:

[0037] Step 501: Use 3D bounding boxes from the training set k For all points in the current frame's point cloud, determine whether each point belongs to a bounding box and which bounding box it belongs to. Then, divide the point cloud in the current frame into 3D bounding boxes. Let the total number of bounding boxes in the current frame be _box_. k The number of points contained within is N. k The point cloud formed is denoted as P. k P k The points contained therein are denoted as p. k，i where i = 0, 1, ..., Nk-1;

[0038] Step 502: For the 3D bounding box k The center point has coordinates [x k y k , z k The dimension of the annotation box is [l] k w k h k ], heading angle is yaw k The heading angle is the angle that needs to be rotated counterclockwise from the positive x-axis to the direction of the 3D annotation box. k Inner point cloud P k Around axis [x k y k Rotate the yaw clockwise k The rotated point cloud is obtained. The points contained therein are denoted as Coordinates are Where i = 0, 1, ..., N k -1;

[0039] Step 503: For the new point cloud obtained after rotation as well as The coordinates included are All points Computational point cloud The maximum value of the x-coordinate xmax With minimum value x min The maximum value of the y-coordinate. max With minimum value y min and the maximum value of the z-coordinate z. max With minimum value z min ;

[0040] Step 504: Based on the results calculated in Step 503, calculate the point cloud integrity score within the bounding box. First, calculate the point cloud... The maximum size distributed within space is in The point cloud is obtained by dividing the maximum size of the point cloud distribution in space by the size of the 3D annotation box. Point cloud original integrity score

[0041]

[0042] Step 505: Calculate the balance factor for all 3D bounding boxes within the current frame.

[0043]

[0044] Where k = 0, 1, ..., N, and N is the number of 3D bounding boxes in the current frame. After calculating the balance factor, the balance factor ε and the original integrity score of the point cloud are used as the basis for the calculation. Obtain point cloud integrity score

[0045] In the above scheme, the process of determining the overall loss function through classification loss, regression loss, and distillation feature loss, determining the gradient of the current student network parameters relative to the overall loss function, and using backpropagation to adjust the parameters of the student network to obtain a trained student neural network specifically includes:

[0046] Step 601: For the detection class probability loss, use the Focal Loss function, which can improve the imbalance between classes, as the loss function;

[0047]

[0048] The center position (x, y, z), size (l, w, h), and rotation angle θ of the predicted box are encoded separately, and the regression loss of the detection box is calculated using the L1 loss function (L1Loss).

[0049]

[0050] The orientation of the predicted bounding box is calculated using the L1 loss function (L1Loss);

[0051]

[0052] Step 602: Combine the detection category loss, detection box regression loss, and predicted box orientation loss from step 601. dir Results and distillation loss Loss distill The summation is used as the final loss, which is then used to train the neural network. The weights of each term are β. cls ,β loc ,β dir and β distill ,

[0053] Loss=β cls Loss cls +β loc Loss loc +β dir Loss dir +β distill Loss distill .

[0054] Compared with existing technologies, this invention uses a knowledge distillation method. By using a teacher network with 3D sparse convolution to supervise the columnar voxel student network during the training phase, and then calculating the point cloud integrity score, the interference of low-quality teacher network information on the student network is reduced. This allows the student network to achieve higher detection performance without increasing computational cost. Attached Figure Description

[0055] The accompanying drawings, which are provided to further illustrate the invention and constitute a part of this invention, illustrate exemplary embodiments of the invention and are used to explain the invention, but do not constitute an undue limitation of the invention.

[0056] Figure 1 This is a schematic diagram illustrating the calculation of distillation loss in an embodiment of the present invention; Detailed Implementation

[0057] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0058] This invention provides a point cloud target detection method based on sparse convolutional feature distillation for autonomous driving, which is implemented through the following steps:

[0059] Step 1: Process the training dataset and preprocess the point cloud data in the training dataset to filter out isolated points and outliers. Finally, perform voxelization on the point cloud data to obtain the training dataset S.train ;

[0060] Step 1.1.1: Currently, the KITTI dataset is the most frequently used dataset in point cloud 3D object detection algorithms. Therefore, this invention will implement the specific solution on the KITTI dataset. The Karlsruhe Institute of Technology in Germany and Toyota Research Institute of America jointly collected and labeled the KITTI dataset and made it publicly available. The vehicle is equipped with a Velodyne 64-line 3D LiDAR, two grayscale cameras, two color cameras, and a GPS system, capable of simultaneously recording data in many common formats such as 3D point clouds, 2D black and white and color images, and optical flow. It can be used in multiple research fields such as image segmentation, object detection, and object tracking. The official annotation labels include detection categories such as vans, cars, pickup trucks, pedestrians (standing / sitting), riders, and trams, with truncation and occlusion levels. This invention only uses point cloud data samples collected by the 64-line LiDAR in the KITTI dataset for solution implementation. The total data is 7481, of which 3769 samples are used as a validation dataset for detecting vehicles (cars and vans), pedestrians (standing), and riders.

[0061] Step 1.1.2: Read in the training data, set the batch size to 8, and set the detection space range in the ego coordinate system. The X direction is limited to [0, 69.12) meters, the Y direction to [-39.68, 39.68) meters, and the Z direction to [-3, 1] meters. Set the maximum query radius for each point cloud to R. querry = 0.15 meters, minimum number threshold N th =5, isolating isolated points within the maximum query area that do not meet the minimum number threshold. The point cloud data feature dimension used is C0=5, including the three-dimensional spatial coordinates of the point cloud (XYZ) and signal reflection intensity. There is an additional feature dimension used to record the index of the point cloud in the batch; this dimension only serves to maintain the data calculation process and does not provide spatial information.

[0062] Step 1.2: Set the voxel size of the teacher network [t] x , t y , t z [0.04, 0.04, 0.1] meters, with the maximum number of point clouds in a single voxel set to K. max =5, the maximum number of voxels sampled in a single point cloud image is voxel_num. max =16000, thus obtaining the voxel relation tensor containing the sampled point cloud. Its corresponding voxel coordinate number is The latter is similar to the point cloud data storage structure; the first number is the sequence number of the current voxel data in the batch, and it does not provide spatial information. point This refers to the total number of sampled point cloud data, typically greater than 16,000. To meet computational requirements, we restrict this to 4 ≤ N. point ≤64000. Based on the given initial voxel size and the limited detection space range, the voxel number should be less than 1728, 1984 and 41 in three dimensions, respectively.

[0063] Step 1.3: Set the voxel size of the student network [s] x s y s z [0.16, 0.16, 4] meters, with the maximum number of point clouds in a single voxel set to K. max =32, the maximum number of voxels sampled in a single point cloud image is voxel_num. max =16000, thus obtaining the voxel relation tensor containing the sampled point cloud. Its corresponding voxel coordinate number is The latter is similar to the point cloud data storage structure; the first number is the sequence number of the current voxel data in the batch, and it does not provide spatial information. point This refers to the total number of sampled point cloud data, typically greater than 16,000. To meet computational requirements, we restrict this to 4 ≤ N. point ≤64000. Based on the given initial voxel size and the defined detection space range, the voxel number should be less than 432,496 and 1 in three dimensions, respectively.

[0064] Step 2: Construct the sub-modules of the teacher neural network using 3D sparse convolution:

[0065] Step 2.1: The voxel feature encoding module calculates the average value of VP based on the number of point clouds contained in each voxel to obtain the encoded voxel features.

[0066] Step 2.2: Construct a 3D sparse convolution module. Let M1 = 4, and the final output is the voxel feature xconv. i Let i = 1, 2, 3, 4. Each sparse convolutional layer has a stride of 2 in three dimensions, and all convolutional kernels are set to sizes of 3. The feature output dimensions for the four scales are set to [16, 32, 64, 64], respectively. The final spatial 3D uniform grid numbering for the four scales is [1728, 1984, 41], [864, 992, 21], [432, 496, 11], and [216, 248, 5]. The last submanifold convolutional layer has a stride of 2 in the Z direction and outputs a feature dimension of 96, resulting in the final 3D spatial features.

[0067] xconv1 = Slice1(xconv0)

[0068] xconv k =spconv k-1 (Slice k (xconv k-1 k = 2, 3, 4

[0069] Step 2.3: The bird's-eye view mapping module performs Z-direction feature stacking on the above 3D spatial features to obtain a 3D tensor as a new 2D bird's-eye view feature.

[0070] Step 2.4: In the 2D convolution module, set M2=2. The output dimensions of the convolutional layer modules at each scale are 96 and 192 respectively. The output dimension of the deconvolution is 192 for all layers. The final output bird's-eye view dimension is 384. For the initial input X0 and the final output UP:

[0071] X i =block i (X i-1 )

[0072] UP i =deblock i (X i ), i = 1, 2

[0073] UP = [UP1, UP2]

[0074] Step 2.5: Construct the detection head and set up anchor frames of different shapes for cars, pedestrians and riders. Each anchor has two orientations, 0° and 90°. The initial anchor offset angle is set to 45° and the orientation is set to 2, representing moving closer to and away from the origin of the ego coordinate system.

[0075] Step 3: Construct a student neural network that does not include 3D sparse convolutions

[0076] Step 3.1: The voxel feature encoding module encodes the columnar voxels into a 64-dimensional feature vector using a multilayer perceptron based on the point cloud contained in each voxel sampled, thus obtaining the encoded voxel features.

[0077] Step 3.2: The bird's-eye view mapping module performs feature stacking in the Z direction on the above 3D spatial features to obtain a 3D tensor as a new 2D bird's-eye view feature.

[0078] Step 3.3: In the 2D convolution module, set M2=3. The output dimensions of the convolutional layer modules at each scale are 64, 128, and 256, respectively. The output dimension of the deconvolution is 128 for all modules. The final output bird's-eye view dimension is 384. For the initial input X0 and the final output UP:

[0079] X i =block i (X i-1 )

[0080] UP i =deblock i (X i ), i = 1, 2, 3

[0081] UP = [UP1, UP2, UP3]

[0082] Step 3.4: Construct the detection head and set up anchor frames of different shapes for cars, pedestrians and riders. Each anchor has two orientations, 0° and 90°. The initial anchor offset angle is set to 45° and the orientation is set to 2, representing moving closer to and away from the origin of the ego coordinate system.

[0083] Step 4: Calculate the detection category loss, detection box regression loss, detection box orientation loss, and distillation loss function according to the set parameters, and then perform backpropagation calculation on the neural network model parameters using the Loss function.

[0084] Step 4.1: Calculate the features to be distilled on the bird's-eye view feature map for the teacher neural network and the student neural network. This is done by projecting each 3D ground truth bounding box onto the bounding boxes under the bird's-eye view feature map output by the teacher network and the student network, respectively. 7×7=49 grid points are sampled to obtain the features of the teacher network and the student network under each bounding box. and

[0085] Step 4.2: Calculate the point cloud integrity score within each 3D bounding box. k

[0086] Step 4.3: Calculate the distillation loss function Loss distill The loss function chosen is the mean squared error loss function (MSELoss):

[0087]

[0088] Step 4.4: Set the maximum training period E of the neural network. max =80, batch size set to B n=8, the total number of training samples is set to S=3712, then the number of batches trained in each training cycle is...

[0089] Step 4.5: For the detection class probability loss, use the Focal Loss function, which can improve the imbalance between classes, as the loss function;

[0090]

[0091] The center position (x, y, z), size (l, w, h), and rotation angle θ of the predicted box are encoded separately, and the regression loss of the detection box is calculated using the L1 loss function (L1Loss).

[0092]

[0093] The orientation of the predicted bounding box is calculated using the L1 loss function (L1Loss);

[0094]

[0095] Set the weight values ​​for the loss function in the first-stage detection. Target class probability weight β. cls =1.0, spatial information weight β of the prediction box loc =2.0, the orientation information weight β of the prediction box dir =0.2, distillation loss function β distill =0.2;

[0096] Loss=β cls Loss cls +β loc Loss loc +β dir Loss dir +β distill Loss distill

[0097] Step 4.6: Initialize the trainable parameters of all convolutional and linear connection layers in the neural network using the Xavier uniform distribution function. Initialize all batch normalized layers, setting their mean weights to 0 and their variance weights to 1.

[0098] Step 4.7: Select Adam as the optimizer for the neural network, set the initial learning rate to lr0 = 0.01, select a cosine learning rate curve, and set the initial learning rate to 0.1 of the initial learning rate.

[0099] Step 4.8: Read in one batch of training data, the network performs forward propagation on the data, then calculates the loss function and performs back propagation, and the detection results must be output according to the training mode specifications.

[0100] Step 4.9: Determine if the current cumulative training steps (step) are divisible by the number of batches trained in a single cycle (k = 928). If yes, read the training data from the beginning according to the shuffled results; if no, read the next training data according to the shuffled results.

[0101] Step 4.10: Determine if the current cumulative training steps (step) are equal to B. n If ×k = 74240, end the training and proceed to step five; otherwise, continue training and return to step 4.8.

[0102] Step 5: Evaluate the performance of the invention:

[0103] Step 5.1: Determine the validation dataset and evaluation metrics;

[0104] This invention uses the validation set from the KITTI dataset to validate the performance of the trained model. Using the evaluation criteria provided by KITTI, the average precision (AP) and frame rate (FPS) of the detection algorithm are used to evaluate the detection performance and running speed of the algorithm, respectively. AP is a commonly used metric in object detection algorithms, directly reflecting the detection accuracy of a specific category; the higher the AP value, the higher the accuracy of the algorithm's detection results. FPS is the number of images processed per second, directly reflecting the running speed of the evaluation model. Faced with complex environments and unexpected situations, autonomous driving algorithms need a certain amount of time to react; the higher the FPS, the longer the reaction time.

[0105] The mathematical formula for calculating the AP index is as follows:

[0106]

[0107] Where P is precision, R is recall, TP represents positive samples predicted as positive, FP represents positive samples predicted as negative, and FN represents negative samples predicted as negative, the mathematical formulas for calculating precision and recall are as follows:

[0108]

[0109]

[0110] For each point cloud sample, the Precision and Recall of its detection result can be obtained. By plotting the Precision and Recall of multiple samples as x and y axes to form a Precision-Recall curve, the AP result can be calculated. To reduce computational complexity and implement discrete integration, the interval [0, 1] can be divided into L-1 equal parts using L equal points. The discrete numerical representation of AP is calculated using an approximate infinitesimal element approach; we set L = 11 to calculate the final result.

[0111] During training, prediction results need to be output according to the test mode. To demonstrate the practicality of this invention, three comparative experiments were set up. Experiment A consisted of training and testing the original columnar voxel neural network model without knowledge distillation; Experiment B consisted of training and testing the columnar voxel neural network model described in this invention. During training, all parameters and hyperparameters were set to the same values, and the programs were run on the same machine. For target selection, we only detected vehicles, pedestrians, and riders. Only predictions with a classification threshold higher than 0.7 were considered correct. Based on the official KITTI classification, the difficulty of the detected objects was divided into easy, medium, and hard categories, considering the degree of occlusion and truncation. The final results are shown in Table 1.

[0112] Table 1: Results of the comparative experiment

[0113]

[0114]

[0115] Comparing experimental groups A and B, experimental group B achieved significantly higher AP (Action Points) results for all three target categories compared to experimental group A. Specifically, for vehicle detection, the AP values ​​for the three difficulty levels increased by 0.37, 0.25, and 1.42, respectively; for pedestrian detection, the AP values ​​for the three difficulty levels increased by 0.81, 0.34, and 0.94, respectively; and for rider detection, the AP values ​​for the three difficulty levels increased by 0.91, 0.37, and 1.40, respectively.

[0116] Based on the test results above, experimental group B improved the detection performance compared to experimental group A without increasing any computational cost, proving the effectiveness of the method described in this invention.

[0117] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention.

Claims

1. A point cloud target detection method based on sparse convolutional feature distillation for autonomous driving, the method comprising: Preprocess the point cloud data to filter out isolated and out-of-bounds points, and perform voxelization on the point cloud data to obtain the training dataset. Constructing teacher neural networks and student neural networks; The teacher's neural network is trained using a training dataset to obtain the weights of each parameter of the teacher's neural network. The student neural network is trained using the training dataset, while the data is input into the teacher network for forward propagation. The distilled feature loss is obtained through feature distillation. The overall loss function is determined by the classification loss, regression loss and distilled feature loss. The gradient of the current student network parameters with respect to the overall loss function is determined. The parameters of the student network are adjusted using backpropagation to obtain the trained student neural network. The trained student neural network is used to detect the prediction dataset, and the position, size, orientation and type of the target to be detected are obtained through forward inference of the student network. The process of obtaining distilled loss through characteristic distillation specifically involves: Step 401: Based on the ground truth of the training set, i.e., the position, size, and object type of the 3D bounding boxes, project the 3D boxes onto the 2D bird's-eye view feature map. Let N be the number of 3D bounding boxes in the current frame. Adjust the size of the 3D bounding boxes in the 2D bird's-eye view feature map according to the downsampling factor of the convolution, and define their position, size, and orientation in the 2D bird's-eye view feature map to obtain the final projection result. Where k = 0, 1, ..., N-1, execute step 402; Step 402: Use 3D bounding boxes from the training set k For each bounding box, calculate the integrity score of the point cloud within the box. k This scoring also applies to 3D bounding boxes. k The corresponding 2D bird's-eye view feature map Step 403: Based on the projection of each bounding box onto the 2D bird's-eye view feature map Within the box, respectively according to 7*7 sampling points are selected at equal intervals on both the length and width scales to obtain the coordinates of each sampling point in the Grid. k，i , where i = 0, 1, ..., 48; Step 404: Based on the coordinates of each sampling point Grid k，i The four points closest to the sampling point in the 2D bird's-eye view feature map are obtained, and they are located in the 2D bird's-eye view feature F of the teacher network. tea and F stu Based on the 2D bird's-eye view features of these four points, bilinear interpolation is used to obtain the features of the teacher network and student network at the sampling points. and Where i = 0, 1, ..., 48; Step 405: For the 3D annotation box k The integrity score of the point cloud within the bounding box obtained in step 402 is used. k and the characteristics of teacher and student networks at the sampling points. and Calculate the distillation loss function Loss distill The loss function chosen is the mean squared error loss function (MSELoss).

2. The point cloud target detection method based on sparse convolutional feature distillation for autonomous driving as described in claim 1, characterized in that, The construction of the teacher neural network specifically involves: Construct a voxel feature encoding module and obtain voxel features and voxel numbers based on voxel data; A 3D sparse convolution module is constructed and used multiple times to extract features from the point cloud at multiple scales. The voxel features at multiple levels are obtained based on the voxel features and voxel indices. A bird's-eye view mapping module is constructed, and based on the voxel features of the largest spatial level scale among the voxel features of the multiple level scales, a high compression method is used to transform the 3D voxel feature space into a 2D bird's-eye view. A 2D convolution module is constructed, and through a series of convolution and deconvolution operations, downsampling and upsampling are performed in different dimensions to transform the 2D bird's-eye view into a 2D feature map. An object detection head is constructed, and based on the 2D feature map, a dense candidate box is obtained using an anchor-size box-based method. Non-maximum suppression is then used for further filtering to obtain the final prediction result.

3. The point cloud target detection method for sparse convolutional feature distillation for autonomous driving according to claim 2, characterized in that, The student neural network is constructed as follows: A columnar voxel encoding module is constructed to divide the point cloud into columnar voxels. A multilayer perceptron is used to encode the point cloud within each columnar voxel into a feature vector, thereby obtaining the features of each columnar voxel. A bird's-eye view mapping module is constructed, and based on the voxel features of the largest spatial level scale among the voxel features of the multiple level scales, a high compression method is used to transform the 3D voxel feature space into a 2D bird's-eye view. Construct a 2D convolution module, and perform downsampling and upsampling in different dimensions through convolution and deconvolution operations to transform the 2D bird's-eye view into a 2D feature map; An object detection head is constructed, and based on the 2D feature map, a dense candidate box is obtained using an anchor-size box-based method. Non-maximum suppression is then used for further filtering to obtain the final prediction result.

4. The point cloud target detection method for sparse convolutional feature distillation for autonomous driving according to claim 3, characterized in that, In step 402, the point cloud integrity score within each 3D bounding box is calculated as follows: Step 501: Use 3D bounding boxes from the training set k For all points in the current frame's point cloud, determine whether each point belongs to a bounding box and which bounding box it belongs to. Then, divide the point cloud within the current frame into 3D bounding boxes. Let the total number of bounding boxes in the current frame be _box_. k The number of points contained within is N. k The point cloud formed is denoted as P. k P k The points contained therein are denoted as p. k，i where i = 0, 1, ..., N k -1; Step 502: For the 3D bounding box k The center point has coordinates [x k y k , z k The dimension of the annotation box is [l] k w k h k ], heading angle is yaw k The heading angle is the angle that needs to be rotated counterclockwise from the positive x-axis to the direction of the 3D annotation box. k Inner point cloud P k Around axis [x k y k Rotate the yaw clockwise k The rotated point cloud is obtained. The points contained therein are denoted as Coordinates are Where i = 0, 1, ..., N k -1; Step 503: For the new point cloud obtained after rotation as well as The coordinates included are All points Computational point cloud The maximum value of the x-coordinate x max With minimum value x min The maximum value of the y-coordinate. max With minimum value y min and the maximum value of the z-coordinate z. max With minimum value z min ; Step 504: Based on the results calculated in Step 503, calculate the point cloud integrity score within the bounding box. First, calculate the point cloud... The maximum size distributed within space is in The point cloud is obtained by dividing the maximum size of the point cloud distribution in space by the size of the 3D annotation box. Point cloud original integrity score Step 505: Calculate the balance factor for all 3D bounding boxes within the current frame. Where k = 0, 1, ..., N, and N is the number of 3D bounding boxes in the current frame. After calculating the balance factor, the balance factor ε and the original integrity score of the point cloud are used as the basis for the calculation. Obtain point cloud integrity score 5. The point cloud target detection method for sparse convolutional feature distillation for autonomous driving according to any one of claims 1-4, characterized in that, The process involves determining the overall loss function through classification loss, regression loss, and distillation feature loss; determining the gradient of the current student network parameters relative to the overall loss function; and using backpropagation to adjust the parameters of the student network to obtain a trained student neural network. Specifically, this includes: Step 601: For the detection class probability loss, use the Focal Loss function, which can improve the imbalance between classes, as the loss function; The center position (x, y, z), size (l, w, h), and rotation angle θ of the predicted box are encoded separately, and the regression loss of the detection box is calculated using the L1 loss function. The orientation of the predicted bounding box is calculated using the L1 loss function; Step 602: Combine the detection category loss, detection box regression loss, and predicted box orientation loss from step 601. dir Results and distillation loss Loss distill The summation is used as the final loss to train the neural network, with each weight being β. cls ,β loc ,β dir and β distill , Loss=β cls Loss cls +b loc Loss loc +b dir Loss dir +b distill Loss distill 。

Images