A data enhancement method for 3D object detection

The laser point cloud is subjected to information loss processing through discrete mesh masks, which solves the problem of deep neural network overfitting in three-dimensional object detection, improves the robustness of the model and reduces time cost.

CN114219748BActive Publication Date: 2024-11-22NINGBO INST OF DALIAN UNIV OF TECH +1
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202111560856.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-12-14
Publication Date
2024-11-22
Estimated Expiration
2041-12-14

AI Technical Summary

Technical Problem

The prior art is difficult to effectively prevent the overfitting of deep neural networks in three-dimensional object detection, especially in the expansion of laser point cloud data sets. Information loss methods are difficult to find a balance between retaining key information and preventing overfitting, resulting in The model is poorly robust.

Method used

Use discrete mesh masks to process information loss on laser point clouds. By randomly generating mesh masks and translating them, it can generate different information loss effects, ensuring the key to retaining the target point cloud while augmenting the dataset. information and reduce time and space complexity.

Benefits of technology

It realizes the preservation of key information on the target point cloud while increasing the size of the data set, significantly improving the robustness of deep neural networks, reducing the risk of overfitting, and reducing time costs.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114219748B_ABST
    Figure CN114219748B_ABST
Patent Text Reader

Abstract

The present invention discloses a data enhancement method for three-dimensional target detection, which performs different information loss on the network by randomly changing the size ratio of a generated grid-shaped mask. When the ratio is small, a denser information loss grid mask is generated, and when the grid ratio is large, a sparser information loss network mask is generated. By reasonably adjusting the grid ratio, the key information of the target point cloud can be retained while increasing the size of the data set, preventing the deep neural network from failing to converge. Compared with the traditional method of dividing the bounding box into several small bounding boxes, the discrete grid mask can retain the key information of the target point cloud even when the grid ratio is large. The present invention can produce different data enhancement effects for the same target box during each information loss operation, significantly increasing the size of the data set, so that the trained deep learning model has better robustness.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention relates to a data preprocessing method for computer vision, and in particular to a data enhancement method applied to three-dimensional target detection. Background Art

[0002] Three-dimensional object detection refers to determining the position of an object in a scene and identifying its category, including the two processes of accurate positioning and classification of the object. Current three-dimensional object detection includes camera-based three-dimensional object detection and lidar-based three-dimensional object detection. Cameras have the advantages of low cost and rich information, but they cannot provide accurate depth information. It is difficult for current vision-based three-dimensional object detection methods to achieve performance similar to that of lidar-based methods. The lidar-based object detection method is: the lidar sensor quickly obtains large-scale, unstructured, and disordered three-dimensional point cloud data by performing three-dimensional scanning and sampling on the surface of the object, where each point contains spatial coordinates (x, y, z) and certain attribute information (such as reflection intensity), which is a simple and real representation of the real world. The lidar-based object detection method can provide accurate information such as target position, size, and driving direction, which plays a vital role in the safety of autonomous driving.

[0003] The rapid success of deep neural networks has greatly promoted the development of autonomous driving. Among them, the laser point cloud detection algorithm based on deep learning has quickly surpassed the traditional point cloud clustering algorithm in detection accuracy. The introduction of deep neural networks allows the model to autonomously learn point cloud features from data, which is much better than the traditional manually designed feature extraction module. However, the deep neural networks currently used for 3D target detection have millions of parameters and require a large amount of training data to achieve good results, otherwise there will be serious overfitting problems. For target detection tasks, data collection and annotation require drawing bounding boxes around all targets in each frame of point cloud. This process requires manual labor and incurs high time and price costs.

[0004] In order to solve the above problems, many scholars and researchers have proposed many methods, among which data enhancement is a very effective method to prevent overfitting. By creating new training examples from existing training examples to expand the size of the training set, it is relatively simple to train a more accurate and stable deep neural network. At present, almost all target detection algorithms use basic data enhancement techniques. Existing computer vision enhancement methods can be roughly divided into three categories: spatial transformation, color distortion, and information loss. Spatial transformation includes random scaling, cropping, flipping, and rotation; color distortion includes changing the brightness and hue of the image. These two methods expand the size of the data set by performing some transformations on the information to simulate real scenes, while information loss deletes part of the information in the data by random erasing, cutting, etc., allowing the deep neural network to learn information that was originally less sensitive or important, increasing the range of perception and thus improving the robustness of the model.

[0005] There are many papers on 2D image enhancement, but there are still few studies on 3D object detection. The complex interactions, mutual occlusions, and changes in perspective and scale between 3D object point clouds make it a challenging problem to obtain accurate 3D world understanding. Although the tasks of 2D and 3D object detection are similar, the laser point cloud has a larger amount of data than 2D images and has the characteristics of sparsity and disorder, so the more mature 2D image enhancement methods cannot be directly used. Therefore, the method of expanding the scale of laser point cloud datasets through data enhancement to prevent overfitting needs further research.

[0006] Since information loss has a low computational cost and good enhancement effect, it has attracted the attention of many researchers in recent years. How to avoid excessive deletion or retention of target point clouds is the core requirement of information loss enhancement methods. A successful information loss method should achieve a reasonable balance between the deletion and retention of data area information. On the one hand, excessive deletion of point cloud information will make it impossible to distinguish the boundary between the background point cloud and the target point cloud, causing the deep neural network to treat the target point cloud with insufficient information as image noise; on the other hand, excessive retention of point cloud information will cause some targets to be completely untouched, resulting in poor robustness of the deep neural network. Summary of the invention

[0007] In order to solve the above problems existing in the prior art, the present invention aims to design a data enhancement method for three-dimensional target detection that can not only distinguish the boundary between the background point cloud and the target point cloud, but also improve the robustness of the deep neural network.

[0008] In order to achieve the above object, the technical solution of the present invention is as follows: A data enhancement method for three-dimensional object detection comprises the following steps:

[0009] A. Read the disordered point cloud P in a frame of laser scanning image, take the disordered point cloud P and its corresponding real bounding box B as input, and divide the disordered point cloud P into background point cloud P according to the geometric position relationship between the disordered point cloud P and the bounding box B. back and the target point cloud P fore , specifically expressed as follows:

[0010] P fore =P∈B

[0011] P back =PP fore

[0012] The parameterization of the bounding box is x, y, z, h, w, l, θ, where x, y, z represent the coordinates of the center of the bounding box in three directions, h, w, l represent the length, width and height of the bounding box, and θ represents the yaw angle of the bounding box;

[0013] B. The target point cloud P fore Divided into [P 1 , P 2 , ..., P n ], divide the bounding box B into [B 1 , B 2 , ..., B n ], and align the target point cloud with the bounding box by translation and rotation to generate the point cloud coordinates P that match the bounding box coord The specific operation is to convert the point cloud in each bounding box into a point cloud with the center of the bounding box B center is the origin, and rotates around the origin in the opposite direction of the bounding box yaw angle θ, which is specifically expressed as follows:

[0014]

[0015] Then, the grid mask unit ratio n is randomly selected according to the predetermined grid ratio range to split the bounding box into n 3 A three-dimensional grid mask M of a small cube is formed, and the grid mask M is compared with the target point cloud P fore The random relative displacement σ produces different information loss effects as follows:

[0016] M=M+σ 0<σ<n

[0017] By combining the mesh mask M with the point cloud coordinates P coord Matching is performed to obtain the point cloud index value P to be retained index as follows:

[0018] P index =P coord ∈M

[0019] The target point cloud is deleted according to the index value, and the processed point cloud P is obtained. mask , specifically expressed as follows:

[0020] P mask =P fore ∈P index

[0021] C. Splice the processed point clouds in all bounding boxes together and merge them with the background point cloud to obtain the data enhanced point cloud P. new , output all point clouds with real labels and bounding boxes, the formula is as follows:

[0022] P new =P back +P mask

[0023] Compared with the prior art, the present invention has the following beneficial effects:

[0024] 1. Since the present invention uses a discrete grid mask to remove point clouds, information loss can be robustly achieved. Different information losses are performed on the network by randomly changing the size ratio of the generated grid-shaped mask. When the ratio is small, a denser information loss grid mask is generated, and when the grid ratio is large, a relatively sparse information loss network mask is generated. By reasonably adjusting the grid ratio, the key information of the target point cloud can be retained while increasing the size of the data set, preventing the deep neural network from failing to converge. Compared with the traditional method of dividing the bounding box into several small bounding boxes, the discrete grid mask can better retain the key information of the target point cloud, and the discrete grid mask of the present invention can retain the key information of the target point cloud even when the grid ratio is large.

[0025] 2. Since the present invention randomly shifts the randomly generated grid mask, different data enhancement effects can be produced for the same target frame during each information loss operation, which significantly increases the size of the data set and makes the trained deep learning model more robust.

[0026] 3. Since the present invention indexes the point cloud according to the relative position of the target point cloud and the center of the bounding box, and enhances the target point cloud by generating an index list and matching it with the grid mask, the time complexity and space complexity generated by matching the relative position of the target point cloud with the grid mask are greatly reduced, which ensures that the preprocessing method does not increase the time cost significantly. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 It is a schematic diagram of the grid mask structure of the present invention.

[0028] Figure 2 It is a flow chart of the method of the present invention. DETAILED DESCRIPTION

[0029] The present invention will be further described below in conjunction with the accompanying drawings.

[0030] In order to correctly understand the current deep learning theory to solve the problem of 3D object detection, we first provide a basic concept about object detection. Traditional 2D object detection takes an image as input and draws a bounding box around the detected object. Then, to promote 2D object detection to 3D, additional estimation of the size, orientation and position of the object in 3D space is required. Specifically, the purpose of 3D object detection is to place an oriented 3D bounding box around the object in 3D space. Similar to the 2D bounding box, the 3D bounding box parameterizes the object as [x, y, z, h, w, l, θ], where [x, y, z] represents the center coordinates of the 3D object bounding box, h, w, l are the length, width and height of the bounding box respectively, and θ is the yaw angle of the bounding box. The bounding box should fit all the detection objects of interest tightly.

[0031] In order to solve the problem of many failure cases of existing three-dimensional target detection information loss strategies, the present invention designs a method GridMask3D that uses a grid mask to perform information loss on the target point cloud. This method reduces the probability of failure cases by expanding the scale of the training data set through a reasonable information loss method. The GridMask3D data enhancement method is characterized by a formula:

[0032]

[0033] Among them, x represents the discrete point cloud in the original bounding box, M represents the grid mask generated according to the three-dimensional bounding box, Represents the discrete point cloud after the grid mask information loss, and the target point cloud is lost by matching the point cloud with the grid mask.

[0034] like Figure 1 As shown, the grid mask M is a three-dimensional grid structure. The present invention generates a grid mask with the same spacing and size with the center of the three-dimensional bounding box as the origin. Different enhancement effects are obtained in each enhancement operation by randomly shifting the grid mask. Finally, information loss is performed by deleting the point cloud in the grid mask. The spacing and size of the cube need to be determined through experimental verification to obtain a better enhancement effect. The specific process is shown in Figure 2 .

[0035] Different from the existing point cloud enhancement algorithms, the information loss enhancement algorithm designed by the present invention neither divides the target point cloud into several continuous regions according to the position relative to the bounding box and randomly removes all the point clouds in some continuous regions, nor directly performs sparse processing on all the target point clouds. By deleting the point clouds in the discontinuous grid mask, the present invention can force the deep neural network to learn the information that is not originally sensitive or important while retaining the necessary target information.

[0036] The present invention is not limited to this embodiment, and any equivalent concepts or changes within the technical scope disclosed by the present invention are included in the protection scope of the present invention.

Claims

1. A data enhancement method for three-dimensional object detection, characterized in that: The following steps are involved: A. Read the disordered point cloud P in a frame of laser scanning image, take the disordered point cloud P and its corresponding real bounding box B as input, and divide the disordered point cloud P into background point cloud P according to the geometric position relationship between the disordered point cloud P and the bounding box B. back and the target point cloud P fore , specifically expressed as follows: P fore =P∈B P back =P-P fore The parameterization of the bounding box is x, y, z, h, w, l, θ, where x, y, z represent the coordinates of the center of the bounding box in three directions, h, w, l represent the length, width and height of the bounding box, and θ represents the yaw angle of the bounding box. B. The target point cloud P fore Divided into [P 1 , P 2 , ..., P n ], divide the bounding box B into [B 1 , B 2 , ..., B n ], and align the target point cloud with the bounding box by translation and rotation to generate the point cloud coordinates P that match the bounding box coord The specific operation is to convert the point cloud in each bounding box into a point cloud with the center of the bounding box B center is the origin, and rotates around the origin in the opposite direction of the bounding box yaw angle θ, which is specifically expressed as follows: Then, the grid mask unit ratio n is randomly selected according to the predetermined grid ratio range to split the bounding box into n 3 A three-dimensional grid mask M of a small cube is formed, and the grid mask M is compared with the target point cloud P fore The random relative displacement σ produces different information loss effects as follows: M=M+σ 0<σ <n By combining the mesh mask M with the point cloud coordinates P coord Matching is performed to obtain the point cloud index value P to be retained index as follows: P index =P coord ∈M The target point cloud is deleted according to the index value, and the processed point cloud P is obtained. mask , specifically expressed as follows: P mask =P fore ∈P index C. Splice the processed point clouds in all bounding boxes together and merge them with the background point cloud to obtain the data enhanced point cloud P. new , output all point clouds with real labels and bounding boxes, the formula is as follows: P new =P back +P mask 。

Citation Information

Patent Citations

  • A neural network coordinate transformation method for a three-dimensional point cloud and a three-dimensional sensor

    CN109685848A

  • Construction method of multi-plane coding point cloud feature deep learning model based on pointpillars

    CN111612059A

  • Three-dimensional target detection method and system based on deep neural network

    CN112258631A

  • Semi-supervised three-dimensional point cloud target detection method

    CN113536920A