A 3D point cloud object detection method based on dynamic adaptive data enhancement

Abstract

The present invention belongs to the field of computer vision technology, and specifically relates to a 3D point cloud target detection method based on dynamic adaptive data enhancement; the method comprises the following steps: obtaining an original training set and inputting the set into a pre-trained 3D point cloud target detection model; obtaining a pseudo training set based on real label objects and pseudo label objects; performing data enhancement on the pseudo training set based on a dynamic instance library to obtain an enhanced pseudo training set; inputting the enhanced pseudo training set into the 3D point cloud target detection model to perform pseudo neural network forward propagation, and calculating the classification loss of point cloud objects in the current training point cloud scene without updating the gradient; performing random data enhancement on pseudo label point cloud objects whose classification loss in the enhanced pseudo training set does not reach a threshold to obtain a final training set; using the final training set to train the detection model; and using the trained detection model to obtain target detection results. The present invention increases the diversity of training scenes and objects, and improves the accuracy of 3D point cloud target detection.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision technology, and specifically relates to a 3D point cloud target detection method based on dynamic adaptive data enhancement. Background Art

[0002] 3D point cloud object detection aims to locate and classify objects from 3D sensor data, such as LiDAR point clouds. With the continued decline in the price of LiDAR sensors in recent years, utilizing point cloud data collected by LiDAR sensors for 3D object detection has become a mainstream technology solution for autonomous vehicles, a key technology and research hotspot in this field. However, traditional deep learning-based 3D object detection methods require a large number of 3D annotation boxes for training. Compared to 2D images, the labor cost of annotating objects in 3D point clouds is much higher.

[0003] To reduce the annotation of 3D objects, researchers have proposed a variety of methods; for example, a sparsely supervised 3D point cloud target detection method obtains more high-quality labeled instances through a positive instance mining module and a reliable background mining module. This method generates sufficient reliable supervision information by mining pseudo-labels in the point cloud scene, solving the problem of insufficient 3D annotations and effectively training 3D target detectors. Although sparsely supervised 3D point cloud target detection methods have achieved good results, people have found that even though 3D target detectors are constantly mining high-quality pseudo-labels for training, some instances do not play an effective role during the iterative training process, and there is overfitting on small pedestrian targets.

[0004] Data augmentation is one of the most effective ways to improve model detection performance. Many works in 3D point cloud object detection have applied data augmentation, such as translation, random flipping, shifting, scaling, and rotation. These methods directly extend typical 2D enhancement methods to 3D point clouds, and all perform data augmentation based on the scene level.

[0005] In summary, the existing technologies have the following shortcomings: ① The pseudo labels mined by sparse supervision are used for training, and during the iterative training process, a large number of instances in the scene do not play an effective role; ② Sparsely supervised 3D point cloud object detection suffers from overfitting on small pedestrian targets; ③ Scene-level data augmentation is commonly used in 3D point cloud object detection, rather than object-level data augmentation. Summary of the Invention

[0006] In view of the shortcomings of the existing technology, the present invention proposes a 3D point cloud target detection method based on dynamic adaptive data enhancement, which includes: acquiring point cloud data in real time and inputting it into a trained 3D point cloud target detection model to obtain target detection results;

[0007] The training process of the 3D point cloud object detection model includes:

[0008] S1: Obtain the original point cloud dataset with real labels and preprocess it to obtain the original training set;

[0009] S2: Input the original training set into the pre-trained 3D point cloud object detection model to mine pseudo-label objects; obtain a pseudo training set based on the real label objects and pseudo-label objects;

[0010] S3: Build a dynamic instance library and perform data augmentation on the pseudo training set based on the dynamic instance library to obtain an enhanced pseudo training set;

[0011] S4: Input the enhanced pseudo training set into the 3D point cloud object detection model for pseudo neural network forward propagation, and calculate the classification loss of the point cloud objects in the current training point cloud scene without updating the gradient;

[0012] S5: Randomly perform data augmentation on the pseudo-label point cloud objects whose classification loss in the enhanced pseudo-training set does not reach the threshold to obtain the final training set;

[0013] S6: Input the final training set into the 3D point cloud object detection model for training and output the object detection results; calculate the total model loss and adjust the model parameters according to the total model loss to obtain the trained 3D point cloud object detection model.

[0014] Preferably, the process of preprocessing the original point cloud dataset includes: filtering the original point cloud dataset according to a given spatial range to obtain a filtered point cloud dataset; in the filtered point cloud dataset, with each point cloud data as the center, counting the number of other point clouds within a spherical area with a radius of R; if the number of other point clouds exceeds a threshold T, the point cloud is deleted; otherwise, the point cloud is retained to obtain the original training set.

[0015] Furthermore, the process of constructing a dynamic instance library includes: mining a pseudo-label object once every fixed training round, and adding the currently mined pseudo-label object to the original dynamic instance library to update the dynamic instance library; wherein the dynamic instance library is the original training set in the initial state.

[0016] Preferably, the 3D point cloud target detection model is a PointRCNN model.

[0017] Preferably, the process of performing data augmentation on the pseudo training set includes:

[0018] Perform flip, rotation, and scaling operations on the point cloud scenes in the pseudo training set to obtain a pseudo training set after preliminary data enhancement;

[0019] Select a specified number of bounding boxes from the dynamic instance library, determine whether the selected bounding boxes overlap with existing bounding boxes in the point cloud scene of the pseudo training set, and fill the non-overlapping bounding boxes and the point clouds contained in the boxes into the point cloud scene of the pseudo training set.

[0020] Preferably, step S5 specifically includes:

[0021] Calculate the average classification loss of the point cloud scene where the point cloud object in the enhanced pseudo training set is located and use it as the threshold. Mark the point cloud objects in the enhanced pseudo training set whose classification loss does not reach the threshold.

[0022] Randomly select one of the methods, shape-aware data augmentation based on a single scene or hybrid data augmentation based on paired scenes, to perform data augmentation on the labeled point cloud objects to obtain the final training set.

[0023] Furthermore, the process of performing shape-aware data enhancement on the labeled point cloud objects based on a single scene includes:

[0024] Find the centroid of the bounding box of the marked point cloud object, connect the centroid with the box surface and divide the bounding box and the points within the box into six partitions;

[0025] Each partition chooses not to perform data augmentation or to perform data augmentation in one of five data augmentation methods; the five data augmentation methods are random point deletion, CutMix and CutMixup of corresponding partitions of similar objects, random noise generation, and sparse sampling.

[0026] Furthermore, the process of performing hybrid data enhancement on the marked point cloud object based on the paired scene includes: randomly pairing the marked point cloud object with the remaining point cloud scenes except the point cloud scene where the marked point cloud object is located; and cropping the marked point cloud object and pasting it into the paired point cloud scene.

[0027] Preferably, the formula for calculating the total loss of the model is:

[0028]

[0029] Among them, L represents the total loss of the model, N represents the normalization factor, and L n represents the loss of the point cloud background, P represents the number of point cloud objects that overlap with the final training set and the pseudo training set, represents the classification loss of the p-th coincident point cloud object in the final training set, represents the regression loss of the p-th coincident point cloud object in the final training set, Q represents the number of point cloud objects filled in the dynamic instance library in the final training set, represents the classification loss of the qth filled point cloud object in the final training set, represents the regression loss of the qth filled point cloud object in the final training set.

[0030] The beneficial effects of the present invention are:

[0031] 1. The present invention performs data enhancement by randomly sampling instances from a constructed dynamic instance library and populating them into training scenarios, thereby increasing the diversity of training data.

[0032] 2. The present invention can monitor pseudo labels that do not contribute much to the model during training through the forward propagation of the pseudo neural network.

[0033] 3. The present invention can adaptively use random data enhancement on pseudo-label instances that do not contribute much to training through a carefully designed random data enhancement method, so that such objects are beneficial to model training.

[0034] 4. This invention can alleviate the overfitting of small pedestrian targets in sparsely supervised 3D point cloud target detection.

[0035] 5. The model of the present invention has strong generalization ability and can be used in a variety of deep learning models. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 This is a training flow chart of the 3D point cloud target detection model in the present invention;

[0037] Figure 2 Schematic diagram of the random data enhancement method in the present invention. DETAILED DESCRIPTION

[0038] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0039] This paper proposes a 3D point cloud target detection method based on dynamic adaptive data enhancement, such as Figure 1 As shown, the method includes the following contents: acquiring point cloud data in real time and inputting it into a trained 3D point cloud target detection model to obtain target detection results.

[0040] The training process of the 3D point cloud object detection model includes:

[0041] S1: Obtain the original point cloud dataset with real labels and preprocess it to obtain the original training set.

[0042] Obtain a raw point cloud dataset with real labels. Preferably, data collected by a 64-line lidar in the KITTI dataset is used as the dataset for training a 3D point cloud object detection model to detect cars, pedestrians, and bicycles. First, the point cloud dataset is divided into 7481 training samples and 7518 test samples. The 7481 training samples are further subdivided into 3712 and 3769 samples as the training and test sets, respectively. The training set is the raw point cloud dataset with real labels (the raw point cloud dataset is sparsely annotated, with one point cloud object randomly annotated for each point cloud scene).

[0043] The process of preprocessing the original point cloud dataset includes:

[0044] The original point cloud dataset is filtered according to the given spatial range to obtain the filtered point cloud dataset. Specifically: Given spatial range Range = [X min , Y min , Z min , X max , Y ma , Z max ], keep the range within k i ∈[k min , k max ], k i ∈ point cloud data within the range of x, y, z. Preferably, POINT_CLOUD_RANGE = [0, -40, -3, 70.4, 40, 1], which retains the point cloud data within the range. For each point cloud data, the X-axis is cropped to retain the range of [0, 70.4]m, the Y-axis is cropped to retain the range of [-40, 40]m, and the Z-axis is cropped to retain the range of [-3, 1]m.

[0045] In the filtered point cloud dataset, with each point cloud data as the center, the number of other point clouds within the spherical area with a radius of R is counted. If the number of other point clouds exceeds the threshold T, the point cloud is deleted. Otherwise, the point cloud is retained to obtain the original training set.

[0046] S2: Input the original training set into the pre-trained 3D point cloud object detection model to mine pseudo-label objects; obtain a pseudo training set based on the real label objects and pseudo-label objects.

[0047] For sparsely supervised 3D point cloud object detection, the training scenes are sparsely annotated, with only one bounding box annotated for each point cloud scene. Initially, the present invention trains a detector from scratch on the sparsely annotated dataset and uses the trained 3D point cloud object detection model as a pre-trained model for mining pseudo-labels.

[0048] Preferably, the 3D point cloud target detection model of the present invention adopts the PointRCNN model; the original training set is input into the pre-trained 3D point cloud target detection model for processing, and a pseudo-label point cloud object with high confidence is obtained through the positive instance mining module; a pseudo training set is obtained based on the real label object and the pseudo-label object, that is, the pseudo-label object is added to the real label object to generate a pseudo training set.

[0049] S3: Build a dynamic instance library and perform data augmentation on the pseudo training set based on the dynamic instance library to obtain an enhanced pseudo training set.

[0050] SECOND proposes a mainstream and effective data augmentation method in the existing art. This method generates an instance library that includes all ground truth labels and their associated point cloud data. During training, a specified number of instances are randomly sampled from the instance library and introduced into the current training point cloud. This method significantly increases the amount of ground truth in each point cloud scene and simulates objects in different environments. To avoid results that are impossible in reality, collision testing is performed when sampling ground truth to remove sampled instances that collide with other existing instances in the current point cloud scene.

[0051] Inspired by the GT sampling enhancement proposed by SECOND, this paper further proposes a dynamic data augmentation strategy. During the current epoch of iteratively mining pseudo-labels, the original labeled instances and pseudo-labeled instances in the dataset are aggregated together, i.e., the real labeled point cloud objects and the pseudo-labeled point cloud objects are aggregated. The dynamic instance library is initially composed of the original training set. Pseudo-labeled objects are mined every fixed training round, and the currently mined pseudo-labeled objects are added to the original dynamic instance library to update it. Since the pseudo-label mining process is dynamic, the constructed instance library is also dynamic.

[0052] The process of data augmentation on the pseudo training set includes:

[0053] The point cloud scenes in the pseudo training set are flipped, rotated and scaled to obtain the pseudo training set after preliminary data enhancement. Preferably, each scene is randomly flipped along the X-axis and Y-axis with a probability of 0.5, and then the scene is scaled using a uniform sampling factor of [0.8, 1.2]. The sampled random angle rotates the point cloud along the Z axis.

[0054] Select a specified number of bounding boxes from the dynamic instance library, determine whether the selected bounding boxes overlap with the existing bounding boxes in the point cloud scene of the pseudo training set, and fill the non-overlapping bounding boxes and the point clouds contained in the boxes into the point cloud scene of the pseudo training set, that is, place the bounding boxes and the point clouds contained in the boxes at the corresponding positions in the existing point cloud scene to obtain an enhanced pseudo training set. Whether the bounding boxes overlap is determined by calculating the 3D IoU between the bounding box of the sampled point cloud and the existing bounding box of the scene point cloud. The specific calculation process is as follows:

[0055] 1. Calculate the highest and lowest heights of each box, and calculate the overlap of the vertical heights of the two bounding boxes based on the highest and lowest heights heig ;

[0056] 2. Calculate the overlaps of the two bounding boxes on the BEV view bev ;

[0057] 3. Calculate 3DIoU, multiply the overlapping part on BEV by the overlapping part on vertical height to get the 3D overlapping part, i.e. overlaps 3d =overlaps bev *overlaps height ;

[0058] 4. Calculate the union of the two bounding boxes. The calculation method is: the union of the bounding boxes = the sum of the volumes of the two boxes vol a +vol b )-overlaps 3d );

[0059] 5.IoU 3d =overlaps 3d / (vol a +vol b -overlaps 3d )

[0060] S4: Input the enhanced pseudo training set into the 3D point cloud object detection model for pseudo neural network forward propagation, and calculate the classification loss of the point cloud objects in the current training point cloud scene without updating the gradient.

[0061] Preferably, set batch size = 2; input the enhanced pseudo training set into the 3D point cloud object detection model, set the "with torch.no_grad()" context manager to perform forward propagation to prevent gradient and parameter updates, and obtain the classification loss L of the point cloud object under this condition cls As the contribution of the current scene object.

[0062] Compared with using hand-crafted heuristic methods and pre-trained models to obtain the contribution of the current scene objects, the present invention uses classification loss to simply and effectively indicate the contribution of the current scene objects, which has obvious advantages: (1) As the classification loss is evaluated as the model being trained, these scores are consistent with the current state of model optimization compared with pre-trained models or heuristic methods; (2) The classification score is evaluated as part of the forward loss calculation, which does not incur additional computational cost.

[0063] S5: Perform random data augmentation on the pseudo-label point cloud objects whose classification loss in the enhanced pseudo-training set does not reach the threshold to obtain the final training set.

[0064] Calculate the average classification loss of the point cloud scene where the point cloud object in the enhanced pseudo training set is located and use it as the threshold. Mark the pseudo-labeled point cloud objects whose classification loss in the enhanced pseudo training set does not reach the threshold, that is, mark the pseudo-label of the point cloud object with Aug_Flag = True;

[0065] like Figure 2 As shown, in order to properly utilize the mined pseudo-labels for effective learning and improve the contribution of labeled pseudo-label objects to model training, the present invention adaptively performs random data augmentation on point cloud objects with low classification loss (below the threshold) according to the object augmentation flag Aug_Flag; specifically, one of the following methods is randomly selected: shape-aware data augmentation based on a single scene or hybrid data augmentation based on paired scenes to perform data augmentation on the labeled point cloud objects to obtain the final training set. The definition of random selection is as follows:

[0066] random.choices(Augmentation,Weight)

[0067] Among them, Augmentation and Weight are hyperparameters, Augmentation is a candidate enhancement method, and there is a one-to-one correspondence between Weight and the enhancement method.

[0068] The process of performing shape-aware data enhancement on labeled point cloud objects based on a single scene includes: finding the centroid of the bounding box of the labeled point cloud object, connecting the centroid with the box surface to divide the bounding box and the points within the box into six partitions; each partition chooses not to perform data enhancement or selects one of five data enhancement methods for data enhancement; the five data enhancement methods are random point deletion, CutMix and CutMixup for partitions corresponding to similar objects, random noise generation, and sparse sampling. The process of selecting not to perform data enhancement or selecting one of the five data enhancement methods for data enhancement for each partition includes: setting a probability threshold for the data enhancement method; within each data enhancement method, traversing all labeled point cloud objects, randomly generating a probability value for the labeled point cloud object, and if the probability value is less than the probability threshold of the data enhancement method, selecting a partition from the partitions of the point cloud object that meets the requirements of the enhancement method for data enhancement. Each point cloud object can undergo at most one data enhancement.

[0069] The process of performing hybrid data augmentation based on paired scenes on the marked point cloud objects includes randomly pairing the marked point cloud objects with the remaining point cloud scenes except the point cloud scene where the marked point cloud objects are located; the pairing operation is defined as follows:

[0070] Paris(S batch , random(S batch ))

[0071] Among them, S batch Represents a batch of training data, random(·) represents S batch Random arrangement.

[0072] Cut out the marked point cloud object and paste it into the paired point cloud scene (it must also undergo a collision test before pasting).

[0073] S6: Input the final training set into the 3D point cloud target detection model for processing to obtain the target detection results; calculate the total model loss and adjust the model parameters according to the total model loss to obtain the trained 3D point cloud target detection model.

[0074] The final training set is input into the 3D point cloud object detection model for forward propagation. The total model loss is calculated based on the final 3D detection box and the object category information. The calculation formula is:

[0075]

[0076] Among them, L represents the total loss of the model, N represents the normalization factor, and L n represents the loss of the point cloud background, P represents the number of point cloud objects that overlap with the final training set and the pseudo training set, represents the classification loss of the p-th coincident point cloud object in the final training set, represents the regression loss of the p-th coincident point cloud object in the final training set, Q represents the number of point cloud objects filled in the dynamic instance library in the final training set, represents the classification loss of the qth filled point cloud object in the final training set, represents the regression loss of the qth filled point cloud object in the final training set.

[0077] Adjust the model parameters according to the total loss of the model. When the loss function value is minimized, the model training is completed.

[0078] Preferably, during the training phase, the present invention uses the ADAM optimizer to optimize all parameters of the network, setting the initial learning rate to The batch size is 8; for sparse supervised mining of pseudo labels, set iterative learning M=10, that is, a pseudo label is mined every 10 rounds to generate a corresponding dynamic instance library; read in 1 batch of training data, input the network forward propagation, calculate the loss function according to the model and backpropagate; determine whether the total number of training steps (step) can be divided by the training batch (k) within a cycle. If so, in the next iteration, the training data will be read again. If not, the training data will continue to be read sequentially; determine whether the total number of training steps (step) is equal to the number of cycles (Bn) multiplied by the number of training batches (k). If so, the training ends and the trained neural network is obtained. Otherwise, the training of the neural network will continue.

[0079] The point cloud data is acquired in real time and input into the trained 3D point cloud target detection model for processing, and the classification results of the point cloud objects, i.e. the target detection results, can be output.

[0080] In summary, the present invention has made a targeted design to address the problem that some instances do not play an effective role during the training process of sparsely supervised 3D point cloud target detection methods. By monitoring the classification loss, random data enhancement is adaptively performed on pseudo-label objects that do not contribute much to the training of 3D target detectors, which increases the diversity of training scenes and objects, alleviates the overfitting phenomenon of pedestrians, and achieves higher 3D point cloud target detection accuracy.

[0081] The above embodiments further illustrate the purpose, technical solutions and advantages of the present invention in detail. It should be understood that the above embodiments are only preferred implementation plans of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made to the present invention within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A 3D point cloud target detection method based on dynamic adaptive data enhancement, characterized in that: include: Acquire point cloud data in real time and input it into the trained 3D point cloud target detection model to obtain target detection results; The training process of the 3D point cloud object detection model includes: S1: Obtain the original point cloud dataset with real labels and preprocess it to obtain the original training set; S2: Input the original training set into the pre-trained 3D point cloud object detection model to mine pseudo-label objects; obtain a pseudo training set based on the real label objects and pseudo-label objects; S3: Build a dynamic instance library and perform data augmentation on the pseudo training set based on the dynamic instance library to obtain an enhanced pseudo training set; S4: Input the enhanced pseudo training set into the 3D point cloud object detection model for pseudo neural network forward propagation, and calculate the classification loss of the point cloud objects in the current training point cloud scene without updating the gradient; S5: Randomly perform data augmentation on the pseudo-label point cloud objects whose classification loss in the enhanced pseudo-training set does not reach the threshold to obtain the final training set; specifically, it includes: Calculate the average classification loss of the point cloud scene where the point cloud object in the enhanced pseudo training set is located and use it as the threshold. Mark the point cloud objects in the enhanced pseudo training set whose classification loss does not reach the threshold. Randomly select one of the following methods: shape-aware data augmentation based on a single scene or hybrid data augmentation based on paired scenes to perform data augmentation on the labeled point cloud objects to obtain the final training set; The process of performing single-scene shape-aware data augmentation on labeled point cloud objects includes: Find the centroid of the bounding box of the marked point cloud object, connect the centroid with the box surface and divide the bounding box and the points within the box into six partitions; Each partition chooses not to perform data augmentation or to perform data augmentation in one of five data augmentation methods; the five data augmentation methods are random point deletion, CutMix and CutMixup of corresponding partitions of similar objects, random noise generation, and sparse sampling; The process of performing hybrid data augmentation based on paired scenes on the marked point cloud objects includes: randomly pairing the marked point cloud objects with the remaining point cloud scenes except the point cloud scene where the marked point cloud objects are located; cropping the marked point cloud objects and pasting them into the paired point cloud scenes; S6: Input the final training set into the 3D point cloud object detection model for training and output the object detection results; calculate the total model loss and adjust the model parameters according to the total model loss to obtain the trained 3D point cloud object detection model.

2. The 3D point cloud target detection method based on dynamic adaptive data enhancement according to claim 1, characterized in that: The process of preprocessing the original point cloud dataset includes: filtering the original point cloud dataset according to a given spatial range to obtain a filtered point cloud dataset; in the filtered point cloud dataset, with each point cloud data as the center, counting the number of other point clouds within the spherical area with a radius of R; if the number of other point clouds exceeds the threshold T, the point cloud is deleted; otherwise, the point cloud is retained to obtain the original training set.

3. The 3D point cloud target detection method based on dynamic adaptive data enhancement according to claim 1, characterized in that: The process of building a dynamic instance library includes: mining a pseudo-label object every fixed training round, adding the currently mined pseudo-label object to the original dynamic instance library to update the dynamic instance library; wherein, the dynamic instance library is the original training set in the initial state.

4. The 3D point cloud target detection method based on dynamic adaptive data enhancement according to claim 1, characterized in that: The 3D point cloud target detection model is a PointRCNN model.

5. The 3D point cloud target detection method based on dynamic adaptive data enhancement according to claim 1, characterized in that: The process of data augmentation on the pseudo training set includes: Perform flip, rotation, and scaling operations on the point cloud scenes in the pseudo training set to obtain a pseudo training set after preliminary data enhancement; Select a specified number of bounding boxes from the dynamic instance library, determine whether the selected bounding boxes overlap with existing bounding boxes in the point cloud scene of the pseudo training set, and fill the non-overlapping bounding boxes and the point clouds contained in the boxes into the point cloud scene of the pseudo training set.

6. The 3D point cloud target detection method based on dynamic adaptive data enhancement according to claim 1, characterized in that: The formula for calculating the total loss of the model is: Among them, L represents the total loss of the model, N represents the normalization factor, and L n represents the loss of the point cloud background, P represents the number of point cloud objects that overlap with the final training set and the pseudo training set, represents the classification loss of the p-th coincident point cloud object in the final training set, represents the regression loss of the p-th coincident point cloud object in the final training set, Q represents the number of point cloud objects filled in the dynamic instance library in the final training set, represents the classification loss of the qth filled point cloud object in the final training set, represents the regression loss of the qth filled point cloud object in the final training set.

Images