A point cloud 3D object detection method based on foreground enhancement knowledge distillation

Abstract

The present invention discloses a point cloud 3D target detection method based on foreground reinforcement knowledge distillation. First, a network input point cloud corresponding to each frame point cloud in a known data set is constructed; then, a PAD‑Net is constructed. The PAD‑Net is composed of a teacher detection network and a student detection network connected by a feature distillation module. The network input point cloud is input into the teacher detection network and the student detection network of the PAD‑Net to train the PAD‑Net, and the trained student detection network is used as the final foreground point cloud detection network; finally, a laser radar collects point clouds in real time and superimposes the saved historical point clouds to obtain a real-time input point cloud sequence. The real-time input point cloud sequence is input into the final foreground point cloud detection network to predict the categories and positioning information of all targets at the current moment. The present invention can effectively overcome the inherent sparsity defects of single-frame point clouds and improve the detection effect of difficult targets that are severely occluded and far away.

Description

Technical Field

[0001] The present invention relates to a point cloud 3D target detection method in the field of target detection technology, and in particular to a point cloud 3D target detection method based on foreground enhancement knowledge distillation. Background Art

[0002] Object detection in autonomous driving scenarios is a crucial task for unmanned vehicles to perceive their environment. With the development of deep learning technology, object detection methods have significantly improved. The shift from manual feature extraction to automatic information extraction using convolutional neural networks has significantly improved detection quality through an end-to-end approach. LiDAR, due to its ability to provide precise three-dimensional information, has become a widely used sensor for 3D object detection.

[0003] 3D object detection schemes based on single-frame point clouds can be divided into three categories according to the different point cloud encoding methods: spatial voxel-based detection, raw point cloud-based detection, and point cloud-voxel fusion detection. Spatial voxel-based schemes include VoxelNet using 3D convolution, SECOND using sparse 3D convolution, and PointPillars, a height-insensitive voxel column encoding scheme. Spatial voxel-based detection schemes include PointRCNN, a two-stage detection network that integrates point cloud segmentation, 3DSSD, which integrates Euclidean space distance and feature space distance sampling strategies, and IA-SSD, which integrates category-aware sampling and centroid-aware sampling, two task-oriented sampling strategies. Point cloud-voxel fusion detection schemes include PV-RCNN, which obtains high-quality 3D candidate boxes based on voxel encoding and then uses the raw point cloud to obtain more refined local features, SA-SSD, which adds two auxiliary tasks, foreground segmentation and target center point estimation, to enhance the structural awareness of the backbone network, and BADet, which introduces graph neural networks to model the correlation of target boundaries in local neighborhood graphs.

[0004] LiDAR point clouds are sparse, capturing fewer points for distant or heavily occluded targets. This leads to a high rate of missed detections for these difficult targets. Currently, there are two main solutions: multimodal detectors that fuse multiple sensors, such as MV3D, PointPainting, DeepFusion, and TransFusion. The computational complexity of extracting image features leads to poor real-time performance, and the fusion of multimodal features places high demands on sensor calibration. Second, multi-frame solutions based on time-series point cloud input can partially compensate for the sparsity of LiDAR point clouds while being independent of other sensors. Through appropriate network design, multi-frame detectors can also ensure real-time performance, making them promising for autonomous driving scenarios. Furthermore, multi-frame inputs contain information about the scene's motion. Properly extracting motion features allows for predictions of future moments, enabling the cross-fertilization of multiple tasks.

[0005] YOLO4D uses an LSTM architecture based on YOLO3D to extract multi-frame features. FAF aligns the point cloud of the historical frame to the current frame through coordinate transformation. It then voxelizes the sequential point cloud and concatenates it along the time dimension to form a 4D tensor. PointFlowNet concatenates the features of the previous and next frames and feeds them into subsequent convolutional layers. 3D-MAN uses an attention mechanism to fuse candidate boxes of sequential frames. SDP-Net uses pixel-level offsets output by the scene flow estimation task to guide feature transfer and alignment. These methods require the use of sequential structures or transfer and alignment operations, resulting in complex network structures and limited real-time performance. Summary of the Invention

[0006] To address the problems in the background art, the present invention provides a point cloud 3D object detection method based on foreground-enhanced knowledge distillation, suitable for real-time detection systems. This method utilizes a single-frame detection network framework, using the rich features extracted from foreground-aligned point clouds to guide feature extraction from unaligned point clouds. This method effectively overcomes the inherent sparsity of single-frame point clouds, achieving better detection results for difficult targets that are severely occluded or located at a distance, and achieving overall detection performance superior to that of corresponding single-frame detectors. The network requires fewer parameters and exhibits strong real-time performance.

[0007] The technical solution adopted in the present invention is as follows:

[0008] 1) Construct the network input point cloud corresponding to each frame point cloud in the known data set;

[0009] 2) Construct a foreground point cloud alignment distillation network (PAD-Net). The PAD-Net consists of a teacher detection network, a student detection network, and a feature distillation module. The teacher detection network and the student detection network are connected via the feature distillation module. The network input point cloud corresponding to each frame of the point cloud in the known dataset is input into the teacher detection network and the student detection network of the PAD-Net to train the PAD-Net. The trained foreground point cloud alignment distillation network is obtained, and the student detection network in the trained foreground point cloud alignment distillation network is used as the final foreground point cloud detection network.

[0010] 3) The lidar collects point clouds in real time and overlays them with the saved historical point clouds to obtain a real-time input point cloud sequence. The real-time input point cloud sequence is input into the final foreground point cloud detection network to predict the category and positioning information of all targets at the current moment.

[0011] In the above 1), for each frame point cloud of the known data set, each frame point cloud is annotated and associated with the corresponding historical frame point cloud, and the horizontal translation and horizontal rotation of the target in the current frame point cloud are calculated. According to the horizontal translation and horizontal rotation of the target, the target point cloud in the historical frame point cloud is added to the current frame point cloud after rigid body transformation alignment, and the background point cloud in the historical frame point cloud is directly added to the current frame point cloud to obtain the teacher detection network input point cloud sequence corresponding to the current frame point cloud;

[0012] After directly adding the historical frame point cloud to the current frame point cloud, the student detection network input point cloud sequence corresponding to the current frame point cloud is obtained.

[0013] The teacher detection network and the student detection network have the same structure. The student detection network includes a voxelization module, a feature extraction module and a detection module. The input of the student detection network serves as the input of the voxelization module. The voxelization module is connected to the detection module after passing through the feature extraction module. The output of the feature extraction module is input into the feature distillation module, and the output of the detection module serves as the output of the student detection network.

[0014] In the feature distillation module, first, the output of the feature extraction module of the student detection network is passed through the convolution layer to obtain the final distilled feature map, and then the confidence value of the output of the student detection network is generated based on the regression value of the output of the student detection network, which is greater than the preset anchor box confidence threshold th anchor_keep The 3D bounding box corresponding to the anchor box is cascaded with the current frame point cloud detection truth box to obtain the feature distillation box, and then the feature distillation box is projected to the bird's-eye view to obtain the bird's-eye view distillation box. The area where the bird's-eye view distillation box is located is used as the key area, and the feature distillation loss weight of the pixel points in the bird's-eye view distillation box on the final distillation feature map is reset to 1, otherwise it is set to 0. The key area weight is composed of all feature distillation loss weights that are 1, and the key area weight is used to perform feature distillation on the final distillation feature map.

[0015] The total loss function of the student detection network is composed of the target classification loss function, the parameter localization regression loss function and the feature distillation loss function, and the formula is as follows:

[0016]

[0017] Among them, L represents the total loss of the student detection network, M is the number of anchor boxes classified as positive and negative samples, and L cls，m is the classification loss of the mth anchor box, N is the number of anchor boxes classified as positive samples, and L loc，n is the regression loss of the nth anchor box, P is the number of pixels in the key area, and L feat_dist，pis the feature distance loss of the p-th pixel, λ0 is the adjustable weight of the target classification loss of the current frame, λ1 is the adjustable weight of the target regression loss of the current frame, and λ2 is the adjustable weight of the feature distillation loss.

[0018] The calculation formula for the classification loss of the mth anchor box is as follows:

[0019]

[0020] Among them, α and γ are the first and second adjustable parameters; c obj is the anchor box classification confidence, gt anchor is the supervisory signal corresponding to the confidence level; if the anchor box is a positive sample, gt anchor =1, otherwise gt anchor =0.

[0021] The regression loss of the nth anchor frame includes the centroid 3D coordinate regression loss, 3D size regression loss and heading angle regression loss, and the calculation formula is as follows:

[0022] L loc，n =α1L size +α2L pos +α3L rot

[0023] Among them, α1 is the adjustable weight of the target 3D size regression loss, α2 is the adjustable weight of the target center of mass 3D position regression loss, α3 is the adjustable weight of the target heading angle regression loss, L size represents the three-dimensional regression loss, L pos represents the centroid 3D coordinate regression loss, L rot represents the heading angle regression loss;

[0024] 3D regression loss L size The calculation formula is as follows:

[0025]

[0026] Among them, l a Indicates the x-direction size of the positive sample anchor box regression, l gt is the x-direction size l of the positive sample anchor box regression a The corresponding true value, w a is the y-direction size of the positive sample anchor box regression, w gt is the y-direction size w of the positive sample anchor box regression a The corresponding true value, h a Indicates the z-direction size of the positive sample anchor box regression, h gt is the z-direction size h of the positive sample anchor box regression a The corresponding true value, log is the logarithmic function, L reg() represents the Smooth L1 loss function;

[0027] Centroid 3D coordinate regression loss L pos The calculation formula is:

[0028]

[0029] Among them, x a is the x-direction coordinate of the positive sample anchor box regression, x gt is the x-direction coordinate x of the positive sample anchor box regression a The corresponding true value, y a is the y-direction coordinate of the positive sample anchor box regression, y gt is the y-direction coordinate y of the positive sample anchor box regression a The corresponding true value, z a is the z-direction coordinate of the positive sample anchor box regression, z gt is the z-direction coordinate z of the positive sample anchor box regression a The corresponding true value; d a is the diagonal length of the anchor frame bird’s-eye view projection, w a Indicates the y-direction size of the positive sample anchor box regression, l a Indicates the x-direction size of the positive sample anchor box regression;

[0030] Heading angle regression loss L rot The calculation formula is:

[0031] L rot =L reg (sinθ a -sinθ gt )+L reg (cosθ a -cosθ gt )

[0032] +L reg (1-cos 2 θ a -sin 2 θ a )

[0033] Among them, sinθ a Sine of the heading angle predicted by the positive sample anchor frame, sinθ gt The sine value of the heading angle sinθ predicted by the positive sample anchor frame a The corresponding true value, cosθ a is the sine of the heading angle predicted by the positive sample anchor frame, cosθ gt The sine value of the heading angle cosθ predicted by the positive sample anchor frame a The corresponding true value.

[0034] The calculation formula for the feature distillation loss of the p-th pixel is as follows:

[0035]

[0036] Among them, weight feat is the weight of the key area, C is the number of feature map channels, feat s，c The feature value of the cth channel in the final feature map output by the student detection network, feat t，c is the position feature value corresponding to the multi-scale feature output by the feature extraction module in the teacher detection network, L reg () represents the Smooth L1 loss function.

[0037] The present invention constructs a point cloud 3D object detection network PAD-Net based on foreground enhancement knowledge distillation; multi-frame point clouds are aligned foreground by annotating true values, and input into a teacher detection network to complete the object detection task, thereby obtaining a foreground enhanced multi-scale feature map; multi-frame point clouds are directly stacked and input into a student detection network, and the multi-scale features of the student detection network are supervised by the multi-scale features of the foreground enhancement of the corresponding teacher detection network, thereby improving the quality of the multi-scale features of the student detection network.

[0038] Compared with the prior art, the present invention has the following beneficial effects:

[0039] 1) This invention can efficiently fuse multiple frames of sequential point clouds, alleviating the inherent sparsity defect of LiDAR point clouds to a certain extent and improving detection performance, especially for the detection of sparse targets in point clouds;

[0040] 2) The present invention does not rely on camera and GPS data, and does not require the fusion of multi-sensor data, effectively improving algorithm efficiency;

[0041] 3) The network structure of the present invention is simple, the computational efficiency is slightly lower than that of the corresponding single-frame detector, but much higher than that of other multi-frame detectors, and the real-time performance is high;

[0042] 4) The present invention is highly portable and can be used in most single-frame detectors. It can learn to fuse multi-frame point clouds while keeping the network structure unchanged, significantly improving the detection quality.

[0043] In general, the present invention is a single-stage target detector with a streamlined network structure, high detection accuracy, strong real-time performance, and strong portability, and has a good detection effect on difficult targets with sparse point clouds. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 It is a network framework diagram of the present invention.

[0045] Figure 2The input point cloud corresponds to a frame of the KITTI RAW dataset.

[0046] Figure 3 This is a diagram of the student detection network structure during the PAD-Net test.

[0047] Figure 4 This is a schematic diagram of the target detection results using vehicle detection as an example in the KITTI RAW dataset, where the black box represents the true value and the white box represents the detection result of the student detection network. DETAILED DESCRIPTION

[0048] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0049] The present invention comprises the steps of:

[0050] 1) Construct the network input point cloud corresponding to each frame point cloud in the known data set;

[0051] 1), for each frame point cloud of the known data set, the ID number of the true value annotation is used to annotate and associate each frame point cloud with the corresponding historical frame point cloud, and the horizontal translation and horizontal rotation of the target in the current frame point cloud are calculated. According to the horizontal translation and horizontal rotation of the target, the target point cloud in the historical frame point cloud is aligned by rigid body transformation and added to the current frame point cloud, and the background point cloud in the historical frame point cloud is directly added to the current frame point cloud to obtain the teacher detection network input point cloud sequence corresponding to the current frame point cloud; that is, the foreground is multi-frame enhanced by the manually annotated true value frame to obtain a dense, complete-contour target point cloud, and rich foreground feature expressions are extracted through the teacher detection network.

[0052] Specifically:

[0053] For the point clouds of the previous and next frames, the targets are associated by marking the ID number of the true value box. For a certain associated target pair, the 3D bounding box at time t is represented as (x t ,y t , z t ,l,w,h,θ t ), the 3D bounding box at time ti is represented as (x t-i ,y t-i , z t-i ,l,w,h,θ t-i ), where x, y, and z are the coordinates of the center of the labeled object, l, w, and h are the length, width, and height of the labeled object, and θ is the orientation angle of the labeled object. The following formula is used to align the point cloud at time ti to the point cloud at the current time:

[0054]

[0055]

[0056] Δθ=θ t -θ t-i

[0057] in, Represents the three-dimensional coordinates (x, y, z) of each point belonging to the target in the ti frame. The horizontal forward direction is the positive direction of the x-axis, the vertical upward direction is the positive direction of the z-axis, and the horizontal forward right direction is the positive direction of the y-axis. The coordinate origin O = [0, 0, 0] of the point cloud of this frame is the emission point of the lidar. Indicates the three-dimensional coordinates of each point after rigid body transformation alignment to the t frame, R represents the rotation matrix of the point cloud, Δ θ Indicates the heading angle change. The heading angle of the annotation box records the angle between the target direction and the origin of the radar coordinate system. Therefore, each point needs to be translated to the origin of the coordinate system first, rotated, and then translated to the corresponding spatial position in the t frame.

[0058] To distinguish the fused multi-frame point cloud, each point in the point cloud data is represented by a five-dimensional vector (x, y, z, r, t), where (x, y, z) is the point's 3D coordinates in the radar coordinate system, r is the reflection intensity, and t is the timestamp, indicating the point's origin. The foreground is aligned, and the background is overlaid. The resulting point cloud size after fusion is N*5, where N is the total number of point clouds in the multi-frame system.

[0059] After directly adding the historical frame point cloud to the current frame point cloud, the student detection network input point cloud sequence corresponding to the current frame point cloud is obtained, that is, the input point cloud sequence contains the time series dimension.

[0060] 2) Construct the foreground point cloud alignment distillation network PAD-Net, such as Figure 1 As shown, the foreground point cloud alignment distillation network PAD-Net consists of a teacher detection network, a student detection network and a feature distillation module. The teacher detection network and the student detection network are connected through the feature distillation module. The network input point cloud corresponding to each frame point cloud in the known dataset is input into the teacher detection network and the student detection network of the foreground point cloud alignment distillation network PAD-Net to train the foreground point cloud alignment distillation network PAD-Net, and obtain the trained foreground point cloud alignment distillation network. The student detection network in the trained foreground point cloud alignment distillation network is used as the final foreground point cloud detection network;

[0061] The teacher detection network and the student detection network have the same structure. The student detection network includes a voxelization module, a feature extraction module, and a detection module. The input of the student detection network serves as the input of the voxelization module, which is then connected to the detection module after passing through the feature extraction module. The input of the voxelization module of the teacher detection network is the input point cloud sequence of the teacher detection network, and the input of the voxelization module of the student detection network is the input point cloud sequence of the student detection network. The output of the feature extraction module (i.e., multi-scale feature map) is input into the feature distillation module, and the output of the detection module serves as the output of the student detection network (i.e., the final detection feature map). The multi-scale feature map of the teacher detection network is the foreground-enhanced feature of the teacher detection network. Through the distillation of the teacher detection network's features, the student detection network is guided by learning and implicitly aligns the target multi-frame point cloud, enabling the student detection network to generate richer target features and obtain better detection results.

[0062] In the feature distillation module, first, the output of the feature extraction module of the student detection network is passed through the convolution layer to obtain the final distilled feature map. Considering that the spatial area on the final feature map corresponding to the point cloud at the historical moment is prone to generate false positive samples, such samples need to be suppressed. At the same time, in order to avoid the student detection network from missing detection frames, the confidence value in the output of the student detection network (i.e., the final feature map) is generated to be greater than the preset anchor frame confidence threshold th. anchor_keep The 3D bounding box corresponding to the anchor box is cascaded with the current frame point cloud detection truth box to obtain the feature distillation box, and then the feature distillation box is projected to the bird's-eye view to obtain the bird's-eye view distillation box. The area where the bird's-eye view distillation box is located is used as the key area, and the feature distillation loss weight of the pixel points in the bird's-eye view distillation box on the final distillation feature map is reset to 1, otherwise it is set to 0. The key area weight is composed of all feature distillation loss weights of 1. Taking into account the serious imbalance of the foreground and background ratios in the target detection task, the feature similarity of the teacher detection network and the student detection network is used as a means of distillation, and the key area weight is used to perform feature distillation on the final distillation feature map.

[0063] The student detection network loss function is set to supervise the feature distance between the final distillation feature map and the multi-scale feature map of the teacher detection network, and all target categories and positioning information obtained from the final feature map. The parameter values of all parameters of the PAD-Net network structure are obtained. The total loss function of the student detection network is composed of the target classification loss function, the parameter positioning regression loss function, and the feature distillation loss function. The formula is as follows:

[0064]

[0065] Among them, L represents the total loss of the student detection network, M is the number of anchor boxes classified as positive and negative samples, and L cls，mis the classification loss of the mth anchor box, N is the number of anchor boxes classified as positive samples, and L loc，n is the regression loss of the nth anchor box, P is the number of pixels in the key area, and L feat_dist，p is the feature distance loss of the p-th pixel, λ0 is the adjustable weight of the target classification loss of the current frame, λ1 is the adjustable weight of the target regression loss of the current frame, and λ2 is the adjustable weight of the feature distillation loss.

[0066] The classification loss of the mth anchor box is calculated as follows:

[0067]

[0068] Among them, L cls represents the target classification loss, α and γ are the first and second adjustable parameters, which are used to balance positive and negative samples and samples of different difficulty levels respectively; c obj is the anchor box classification confidence, gt αnchor is the supervisory signal corresponding to the confidence level; if the anchor box is a positive sample, gt anchor =1, otherwise gt anchor =0.

[0069] The regression loss function of the nth anchor frame regression loss includes the centroid 3D coordinate regression loss, 3D size regression loss and heading angle regression loss. The calculation formula is as follows:

[0070] L loc，n =α1L size +α2L pos +α3L rot

[0071] Among them, α1 is the adjustable weight of the target 3D size regression loss, α2 is the adjustable weight of the target center of mass 3D position regression loss, α3 is the adjustable weight of the target heading angle regression loss, L size represents the three-dimensional regression loss, L pos represents the centroid 3D coordinate regression loss, L rot represents the heading angle regression loss;

[0072] 3D regression loss L size The calculation formula is as follows:

[0073]

[0074] Among them, l a Indicates the x-direction size of the positive sample anchor box regression, l gt is the x-direction size l of the positive sample anchor box regression a The corresponding true value, w a is the y-direction size of the positive sample anchor box regression, w gt is the y-direction size w of the positive sample anchor box regressiona The corresponding true value, h a Indicates the z-direction size of the positive sample anchor box regression, h gt is the z-direction size h of the positive sample anchor box regression a The corresponding true value, log is the logarithmic function, L reg () represents the Smooth L1 loss function;

[0075] Centroid 3D coordinate regression loss L pos The calculation formula is:

[0076]

[0077] Among them, x a is the x-direction coordinate of the positive sample anchor box regression, x gt is the x-direction coordinate x of the positive sample anchor box regression a The corresponding true value, y a is the y-direction coordinate of the positive sample anchor box regression, y gt is the y-direction coordinate y of the positive sample anchor box regression a The corresponding true value, z a is the z-direction coordinate of the positive sample anchor box regression, z gt is the z-direction coordinate z of the positive sample anchor box regression a The corresponding true value; d a is the diagonal length of the anchor frame bird’s-eye view projection, w a Indicates the y-direction size of the positive sample anchor box regression, l a Indicates the x-direction size of the positive sample anchor box regression;

[0078] Heading angle regression loss L rot The calculation formula is:

[0079] L rot =L reg (sinθ a -sinθ gt )+L reg (cosθ a -cosθ gt )

[0080] +L reg (1-cos 2 θ a -sin 2 θ a )

[0081] Among them, sinθ a Sine of the heading angle predicted by the positive sample anchor frame, sinθ gt The sine value of the heading angle sinθ predicted by the positive sample anchor framea The corresponding true value, cosθ a is the sine of the heading angle predicted by the positive sample anchor frame, cosθ gt The sine value of the heading angle cosθ predicted by the positive sample anchor frame a The corresponding true value.

[0082] The calculation formula for the feature distillation loss of the p-th pixel is as follows:

[0083]

[0084] Among them, weight feat is the weight of the key area, C is the number of feature map channels, feat s，c The feature value of the cth channel in the final feature map output by the student detection network, feat t，c is the position feature value corresponding to the multi-scale feature output by the feature extraction module in the teacher detection network, L reg () represents the Smooth L1 loss function

[0085] The feature distance between the key areas of the multi-scale feature map of the student detection network and the corresponding areas of the multi-scale feature map of the teacher detection network is used as the loss function to supervise the motion information of the learning foreground of the student detection network and improve the detection performance of the student detection network.

[0086] 3) The lidar collects point clouds in real time and overlays them with the saved historical point clouds to obtain a real-time input point cloud sequence. The real-time input point cloud sequence is input into the final foreground point cloud detection network to predict the category and positioning information of all targets at the current moment.

[0087] The embodiment of the method of the present invention and its implementation process are as follows:

[0088] Taking the KITTI RAW public dataset as a known dataset and the detection of vehicle targets as an example, the specific implementation process of the point cloud 3D object detection solution based on foreground enhancement knowledge distillation is described.

[0089] Step 1: Split the KITTI RAW public dataset into a training set and a test set according to the sequence number. The training set contains 20 sequences and the test set contains 17 sequences. For each frame of point cloud in the training set, perform steps 2 to 4.

[0090] Step 2: Using the current frame number as the index, select the point cloud of the index-2 and index-4 frames in the sequence. If index-2 is less than 0, copy the index frame; if index-4 is less than 0, copy the index-2 frame. Associate the two frames using the labeled ID number and index frame, respectively. Calculate the horizontal translation and rotation of the target based on the true value for each associated pair. Align the target point cloud of the historical frame with the rigid body transformation and add it to the current frame point cloud. The background point cloud is directly added to the current frame point cloud. To distinguish point clouds from different frames, add a timestamp to each point. Each point is represented by a five-dimensional vector (x, y, z, r, t), where (x, y, z) is the three-dimensional coordinate of the point in the radar coordinate system, r is the reflection intensity, and t is the timestamp, indicating the time the point originated. For the index frame, t = 0, for the index-2 frame, t = -2, and for the index-4 frame, t = -4. Assume that the number of point clouds in the index frame is N1, the number of point clouds in the index-2 frame is N2, and the number of point clouds in the index-4 frame is N3. Then the point cloud size after foreground alignment and background superposition and fusion is (N1+N2+N3)*5.

[0091] Take the point cloud within the range of 0-51.2m in the X direction, -30.72m-30.72m in the Y direction, and -3m-1m in the Z direction of the laser radar as the detection input. The point cloud after foreground alignment and background superposition fusion is recorded as the teacher detection network input point cloud points tea , as the input of the teacher detection network, the point cloud after the foreground and background are superimposed and fused is recorded as the student detection network input point cloud points stu , as the input of the student detection network. The teacher detection network input point cloud points tea And the student detection network input point cloud points stu Consistent size. Figure 2 (a) Teacher detection network input point cloud points tea , Figure 2 (b) is the input point cloud points for the student detection network stu , the local enlarged image shows the teacher detection network input point cloud points tea The point clouds of the same target in different frames have been aligned, and the student detection network input point cloud points stu The point clouds of the same target in different frames are not aligned and are displayed as a collection of target point clouds at three different spatial positions.

[0092] Step 3: The teacher detection network input point cloud points obtained by fusion of the current frame tea Input the PAD-Net teacher detection network to train the PAD-Net teacher detection network.

[0093] The specific structure of the PAD-Net teacher detection network is shown in Table 1 below:

[0094] Table 1 PAD-Net teacher detection network structure

[0095]

[0096] Take the voxel column size as 0.16*0.16m*4m, num grid = 32. The bird's-eye view features are extracted through the voxelization module, and the feature map size is 320*384*64. That is, H = 320, W = 384, C = 64. The multi-scale feature map output by the teacher detection network has a size of 160*192*384. For the object detection task, the number of categories is 2, including vehicles and background, so num cls = 2. The parameters required for target localization include (x, y, z, l, w, h, sinθ, cosθ), where (x, y, z) are the coordinates of the target's center of mass, (l, w, h) are the length, width, and height of the target, sinθ is the sine of the target's orientation angle, and cosθ is the cosine of the target's orientation angle. Therefore, d = 8. The final feature map sizes of the teacher detection network are 160*192*2 and 160*192*8, respectively.

[0097] points tea Use the following data augmentation:

[0098] (1) Global enhancement: including random translation in the horizontal direction within the range of [-1m, 1m] and along the z-axis [-45°, 45°]

[0099] Random rotation in the range [0.8, 1.2], random scaling in the range [0.8, 1.2].

[0100] (2) Local enhancement: For each target, the point cloud and the ground truth are randomly translated in the horizontal direction within the range of [-0.2m, 0.2m], randomly rotated along the z-axis within the range of [-10°, 10°], and randomly scaled within the range of [0.9, 1.1].

[0101] (3) Randomly insert samples: The point clouds of other sequences in the KITTI RAW dataset are added to

[0102] points tea middle.

[0103] The training parameters are: Adam optimizer is used for parameter update, the initial learning rate is set to 0.001, the weight decay coefficient is 0.01, and the learning rate is updated using periodic learning rate decay, with a maximum learning rate of 0.01 and a minimum of 1x10 -7NVIDIA GTX2080Ti and Intel i7 CPU were used for data processing and model training, and the model converged after 80 rounds.

[0104] Step 4: Teacher detection network input point cloud points obtained by fusion of the current frame tea And the student detection network input point cloud points stu The PAD-Net teacher detection network and the PAD-Net student detection network are input respectively to train the PAD-Net student detection network.

[0105] Specifically: Load the PAD-Net teacher detection network parameter values obtained in step 3 and freeze the parameters. tea Input the PAD-Net teacher detection network to obtain the multi-scale feature map of the teacher detection network, and the student detection network input point cloud points st The input parameters of the PAD-Net student detection network are randomly initialized, and the output of the student detection network feature extraction module is obtained. The structure of the PAD-Net student detection network is consistent with the structure of the PAD-Net teacher detection network shown in Table 1. The output of the PAD-Net student detection network feature extraction module is respectively input into the convolution layer to obtain the distillation final feature map and the detection final feature map. The distillation final feature map and the output of the PAD-Net teacher detection network feature extraction module are used to calculate the feature loss of the key area.

[0106] Detection of the final feature map The student detection network generates all target categories and positioning information, and uses the corresponding supervision signal to update the parameters of the student detection network. The distilled feature map output by the student detection network has a size of 160*192*384. For the target detection task, the number of categories is 2, including vehicles and backgrounds. The parameters required for target positioning include (x, y, z, l, w, h, sinθ, cosθ), where (x, y, z) are the coordinates of the center of mass of the target, (l, w, h) are the length, width, and height of the target, sinθ is the sine of the target orientation angle, and cosθ is the cosine of the target orientation angle. The sizes of the final feature maps detected are 160*192*2 and 160*192*8 respectively;

[0107] For the PAD-Net student detection network student detection network loss function, take:

[0108] λ0=1,λ1=2,λ2=5

[0109] The training parameters are: Adam optimizer is used for parameter update, the initial learning rate is set to 0.0003, the weight decay coefficient is 0.01, and the learning rate is updated using periodic learning rate decay, with a maximum learning rate of 0.003 and a minimum of 3x10 -7 NVIDIA GTX2080Ti and Intel i7 CPU were used for data processing and model training, and the model converged after 80 rounds.

[0110] Step 5: For each frame point cloud in the KITTI RAW test set, load its historical frame point cloud and fuse it to obtain the student detection network input point cloud points stu , input into the final foreground point cloud detection network, predict the category and positioning information of all targets at the current moment. Specifically: remove the PAD-Net teacher detection network and feature distillation structure obtained in step 4, and only save the PAD-Net student detection network as the final foreground point cloud detection network student detection network (the final foreground point cloud detection network student detection network structure diagram is shown in the figure). Figure 3 As shown). Input the student detection network into the point cloud points stu Input the final foreground cloud detection network to obtain the final feature map of the detection. The final feature map of the detection can be divided into the final feature map of classification and the final feature map of regression. Take the target discrimination threshold th Cobj = 0.5, perform the following operations on each pixel on the final classification feature map: If the vehicle category confidence of the pixel is greater than th Cobj , then the anchor box generated by the pixel point is considered to be a positive sample. Output the current frame target positioning information based on the final regression feature map. The final regression feature map contains parameters (x, y, z, l, w, h, sinθ, cosθ), where (x, y, z) represents the three-dimensional coordinates of the target's center of mass, (l, w, h) represents the three-dimensional size of the target, and the target heading angle Where atan2 is the inverse tangent function.

[0111] Typical results of target detection are as follows Figure 4 As shown, the black box represents the true value and the white box represents the detection result of the student detection network.

[0112] The effectiveness of the present invention was evaluated using the KITTI RAW partitioned test set. The average detection accuracy of objects of varying difficulty was compared using a PAD-Net student and teacher detection networks as detectors for a single-frame point cloud, a PAD-Net student and teacher detection networks as detectors for three directly superimposed point clouds, and the final foreground point cloud detection network trained using the present invention for three directly superimposed point clouds.

[0113] Table 2 Comparison of average detection accuracy of different algorithms in the test set

[0114]

[0115] Table 2 shows that directly stacking three point clouds and misaligned targets can cause feature confusion, leading to poor detection. This feature distillation-based approach uses aligned target point clouds as input to the teacher detection network to generate richer feature maps, guiding the learning of misaligned target point clouds. This approach achieves higher detection accuracy than directly stacking point clouds or single-frame point clouds, especially for difficult targets with sparse point clouds.

[0116] The experimental equipment includes an NVIDIA GTX2080Ti GPU and an Intel i7 CPU for model training and testing. The student detection network, which uses a three-frame point cloud overlay, achieves an average detection speed of 43.68ms per frame, with minimal computational effort and high real-time performance.

[0117] As can be seen, the proposed method can effectively overcome the inherent sparsity defect of single-frame point clouds, achieves high detection efficiency for difficult targets that are severely occluded or far away, and outperforms the corresponding single-frame detector in overall detection performance. It also requires fewer network parameters and has strong real-time performance.

Claims

1. A point cloud 3D object detection method based on foreground enhancement knowledge distillation, characterized by: The steps include: 1) Construct the network input point cloud corresponding to each frame point cloud in the known data set; 2) Construct a foreground point cloud alignment distillation network (PAD-Net). The PAD-Net consists of a teacher detection network, a student detection network, and a feature distillation module. The teacher detection network and the student detection network are connected via the feature distillation module. The network input point cloud corresponding to each frame of the point cloud in the known dataset is input into the teacher detection network and the student detection network of the PAD-Net to train the PAD-Net. The trained foreground point cloud alignment distillation network is obtained, and the student detection network in the trained foreground point cloud alignment distillation network is used as the final foreground point cloud detection network. 3) The LiDAR collects point clouds in real time and overlays them with the saved historical point clouds to obtain a real-time input point cloud sequence. This real-time input point cloud sequence is input into the final foreground point cloud detection network to predict the category and location information of all targets at the current moment. The teacher detection network and the student detection network have the same structure. The student detection network includes a voxelization module, a feature extraction module and a detection module. The input of the student detection network serves as the input of the voxelization module. The voxelization module is connected to the detection module after passing through the feature extraction module. The output of the feature extraction module is input into the feature distillation module. The output of the detection module serves as the output of the student detection network. In the feature distillation module, first, the output of the feature extraction module of the student detection network is passed through the convolution layer to obtain the final distilled feature map, and then the confidence value of the output of the student detection network is generated based on the regression value of the output of the student detection network, which is greater than the preset anchor box confidence threshold th anch_keep The 3D bounding box corresponding to the anchor box is cascaded with the current frame point cloud detection truth box to obtain the feature distillation box, and then the feature distillation box is projected to the bird's-eye view to obtain the bird's-eye view distillation box. The area where the bird's-eye view distillation box is located is used as the key area, and the feature distillation loss weight of the pixel points in the bird's-eye view distillation box on the final distillation feature map is reset to 1, otherwise it is set to 0. The key area weight is composed of all feature distillation loss weights that are 1, and the key area weight is used to perform feature distillation on the final distillation feature map.

2. The point cloud 3D object detection method based on foreground enhancement knowledge distillation according to claim 1 is characterized in that: In the above 1), for each frame point cloud of the known data set, each frame point cloud is annotated and associated with the corresponding historical frame point cloud, and the horizontal translation and horizontal rotation of the target in the current frame point cloud are calculated. According to the horizontal translation and horizontal rotation of the target, the target point cloud in the historical frame point cloud is added to the current frame point cloud after rigid body transformation alignment, and the background point cloud in the historical frame point cloud is directly added to the current frame point cloud to obtain the teacher detection network input point cloud sequence corresponding to the current frame point cloud; After directly adding the historical frame point cloud to the current frame point cloud, the student detection network input point cloud sequence corresponding to the current frame point cloud is obtained.

3. The point cloud 3D object detection method based on foreground enhancement knowledge distillation according to claim 1 is characterized in that: The total loss function of the student detection network is composed of the target classification loss function, the parameter localization regression loss function and the feature distillation loss function, and the formula is as follows: Among them, L represents the total loss of the student detection network, M is the number of anchor boxes classified as positive and negative samples, and L cls,m is the classification loss of the mth anchor box, N is the number of anchor boxes classified as positive samples, and L loc,n is the regression loss of the nth anchor box, P is the number of pixels in the key area, and L feat_dist,p is the feature distance loss of the p-th pixel, λ0 is the adjustable weight of the target classification loss of the current frame, λ1 is the adjustable weight of the target regression loss of the current frame, and λ2 is the adjustable weight of the feature distillation loss.

4. The point cloud 3D object detection method based on foreground enhancement knowledge distillation according to claim 3 is characterized in that: The calculation formula for the classification loss of the mth anchor box is as follows: Among them, α and γ are the first and second adjustable parameters; c obj is the anchor box classification confidence, gt anchor is the supervisory signal corresponding to the confidence level; if the anchor box is a positive sample, gt anchor =1, otherwise gt anchor =0.

5. The point cloud 3D object detection method based on foreground enhancement knowledge distillation according to claim 3 is characterized in that: The regression loss of the nth anchor frame includes the centroid 3D coordinate regression loss, 3D size regression loss and heading angle regression loss, and the calculation formula is as follows: L loc,n =α1L size +α2L pos +α3L rot Among them, α1 is the adjustable weight of the target 3D size regression loss, α2 is the adjustable weight of the target center of mass 3D position regression loss, α3 is the adjustable weight of the target heading angle regression loss, L size represents the three-dimensional regression loss, L pos represents the centroid 3D coordinate regression loss, L rot represents the heading angle regression loss; 3D regression loss L size The calculation formula is as follows: Among them, l a Indicates the x-direction size of the positive sample anchor box regression, l gt is the x-direction size l of the positive sample anchor box regression a The corresponding true value, w a is the y-direction size of the positive sample anchor box regression, w gt is the y-direction size w of the positive sample anchor box regression a The corresponding true value, h a Indicates the z-direction size of the positive sample anchor box regression, h gt is the z-direction size h of the positive sample anchor box regression a The corresponding true value, log is the logarithmic function, L reg () represents the Smooth L1 loss function; Centroid 3D coordinate regression loss L pos The calculation formula is: Among them, x a is the x-direction coordinate of the positive sample anchor box regression, x gt is the x-direction coordinate x of the positive sample anchor box regression a The corresponding true value, y a is the y-direction coordinate of the positive sample anchor box regression, y gt is the y-direction coordinate y of the positive sample anchor box regression a The corresponding true value, z a is the z-direction coordinate of the positive sample anchor box regression, z gt is the z-direction coordinate z of the positive sample anchor box regression a The corresponding true value; d a is the diagonal length of the anchor frame bird’s-eye view projection, w a Indicates the y-direction size of the positive sample anchor box regression, L a Indicates the x-direction size of the positive sample anchor box regression; Heading angle regression loss L rot The calculation formula is: L rot =L reg (sinθ a -sinθ gt )+L reg (cosθ a -cosθ gt ) +L reg (1-cos 2 i a -sin 2 i a ) Among them, sinθ a Sine of the heading angle predicted by the positive sample anchor frame, sinθ gt The sine value of the heading angle sinθ predicted by the positive sample anchor frame a The corresponding true value, cosθ a is the sine of the heading angle predicted by the positive sample anchor frame, cosθ gt The sine value of the heading angle cosθ predicted by the positive sample anchor frame a The corresponding true value.

6. The point cloud 3D object detection method based on foreground enhancement knowledge distillation according to claim 3 is characterized in that: The calculation formula for the feature distillation loss of the p-th pixel is as follows: Among them, weight feat is the weight of the key area, C is the number of feature map channels, feat s,c The feature value of the cth channel in the final feature map output by the student detection network, feat t,c is the position feature value corresponding to the multi-scale feature output by the feature extraction module in the teacher detection network, L reg () represents the Smooth L1 loss function.

Images