A point cloud scene synthesis method based on distance images

Abstract

The application provides a point cloud scene synthesis method based on a distance image, so that the labeled data can be flexibly reused in different laser radar data, different scenes and different positions in the scene, and the synthesized point cloud can be flexibly adapted to the point cloud distribution of any laser radar through point cloud distribution correction. The method can reuse the already labeled high-resolution point cloud, reasonably insert it into the new environment point cloud as the labeling information of the new environment point cloud, and is used for training an algorithm, so that the labeling cost is greatly saved, and the algorithm can be deployed in a new environment more quickly.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of laser radar 3D point cloud object detection, and particularly relates to a point cloud scene synthesis method based on distance images. BACKGROUND

[0002] Laser radar 3D point cloud object detection is a very important part of the automatic driving perception system, which uses the 3D point cloud information measured by the laser radar sensor to accurately perceive the 3D environment around the vehicle body and give accurate prediction of the 3D position information of the surrounding objects. The perception ability of the algorithm to the surrounding environment will directly affect the performance of the automatic driving system in subsequent trajectory prediction, path planning and other links. Therefore, object detection based on laser radar 3D point cloud is crucial to the automatic driving system.

[0003] Unlike traditional 2D image object detection, laser radar 3D point cloud object detection is more complex. First, 2D images are regular pixel data, while 3D point clouds are non-uniformly distributed sparse data in 3D space. The increase in dimension and the change in data structure make 3D point cloud labeling more difficult. Statistical data shows that an experienced labeler needs an average of 0.5-1 seconds to label a 2D bounding box, while an average of 8-10 seconds to label a 3D bounding box. Obviously, this will bring higher cost to 3D point cloud labeling, and how to efficiently reuse existing labeled data is a key problem faced by laser radar 3D object detection.

[0004] One feature of 3D object detection is that the target size in 3D space is absolute, i.e. the size of the same object does not change with the change of position or viewing angle, i.e. there is no influence of projection effect. At the same time, in the automatic driving scene, there is generally no overlap between object target boxes, so the object point clouds of different frames can be extracted according to the target box and inserted into the same frame to realize simple point cloud scene synthesis. The above method is very commonly used in the training of laser radar object detection algorithm and is called Aug-GT method. However, this method has some drawbacks. In order to ensure that the synthesized data still has reasonable data distribution, it produces many restrictions on the insertion conditions: it can only be performed between different frames collected by the same laser radar, and the XY coordinates of the same object point cloud must be consistent in two frames. This leads to strict restrictions on the range and form of labeled data reuse.

[0005] Meanwhile, relevant research shows that the generalization of the laser radar point cloud object detection algorithm between different scenes is poor, that is, even if the objects to be detected have the same characteristics, the algorithm still cannot detect them well when they appear in a new scene (such as different cities in the same country). This means that a laser radar point cloud object detection algorithm applied to an automatic driving system needs to be trained in numerous road environments (different cities, roads, etc.), and it is difficult to ensure that the data collected in these road environments are collected by the same laser radar. In addition, if the objects in each road environment are labeled once, a large amount of manpower and material resources will be consumed. It is noted that although the road conditions are varied, the vehicle models appearing on the road are limited, which means that the same thousands of vehicle models will appear in various scenes. How to efficiently reuse the labeled data so that the algorithm can quickly adapt to a new environment at a low cost is also a key problem faced by laser radar point cloud object detection.

[0006] In summary, the prior art has the following technical defects: (1) the 3D object detection model based on laser radar point cloud trained in a certain scene cannot be well migrated to other scenes, and the flexibility of algorithm deployment is insufficient. (2) The labeling cost of laser radar point cloud is high, but the laser radar point cloud instances labeled in a certain scene can only be migrated in the data collected by the same laser radar, the application range is small, and the reuse rate is low. SUMMARY

[0007] To solve the above technical problems, the present application provides a point cloud scene synthesis method based on distance image, so that the labeled data can be flexibly reused in different laser radar data, different scenes, and different positions in the scene, and the synthesized point cloud can be flexibly adapted to the point cloud distribution of any laser radar through point cloud distribution correction. This method can reuse the already labeled high-resolution point cloud and reasonably insert it into the new environment point cloud as the labeling information of the new environment point cloud for training the algorithm, which greatly saves the labeling cost and enables the algorithm to be deployed in the new environment more quickly.

[0008] To achieve the above purpose, the present application adopts the following technical scheme:

[0009] A point cloud scene synthesis method based on distance image, comprising the following steps:

[0010] Step 1, extracting point cloud object samples of existing labeled data;

[0011] Step 2, screening the point cloud object samples;

[0012] Step 3, constructing a point cloud object sample database;

[0013] Step 4, generating pseudo-labeling boxes in new scene point clouds;

[0014] Step 5, match the pseudo-labeled box with the samples in the point cloud object sample database, and scale and fill several matching samples into the pseudo-labeled box;

[0015] Step 6, convert the point cloud scene after filling the samples from the Cartesian coordinate system to the distance image coordinate system, i.e. the spherical coordinate system;

[0016] Step 7, quantize the continuous coordinates according to the vertical and horizontal viewing angles and angular resolution of the laser radar hardware used in the new scene;

[0017] Step 8, according to the unique correspondence of each pixel on the distance image and the three-dimensional position of the laser radar point cloud, the distribution of the point cloud filled in the pseudo-labeled box in step 5 is corrected by downsampling;

[0018] Step 9, project the point cloud after downsampling in step 8 back to the Cartesian coordinate system.

[0019] Further, the step 1 includes: extracting the point cloud instances P i contained in each object label box B i from the existing high-resolution point cloud data P and the corresponding label box set B j ; wherein the label box B i contains the position information (x, y, z) of the object in the current coordinate system, the size information, length, width and height (l, w, h), and the orientation information of the object, i.e. the rotation angle θ along the Z axis, i.e. B i = {x i , y i , z i , l i , w i , h i , θ i}.

[0020] Further, the step 2 includes: screening the extracted point cloud instances P i of the object according to the quality, removing the samples whose object surface point cloud is relatively sparse, and the point cloud samples whose object surface shape is very incomplete due to serious occlusion; the relatively sparse refers to that the point cloud quantity is lower than a threshold, and the serious occlusion refers to that the volume ratio of the minimum circumscribed rectangle of the point cloud in the target box to the label box is less than a threshold.

[0021] Further, the step 3 includes: centering the coordinates of the screened point cloud instances of the object, i.e. taking the center coordinates (x i , y i , z i) As a new origin, the X-axis is parallel to the target orientation, the Z-axis is parallel to the sensor coordinate system Z-axis, and the Y-axis is determined by the right-hand rule; a sample database is established using the centralized point cloud samples, and classified according to their direction facing the sensor, facilitating subsequent rapid indexing; the angle range of the bounding box facing the sensor α is [0, 2π), for easy retrieval, the angle range is equally divided into 16 intervals, and the continuous angle value is quantized into these 16 intervals, that is, the direction classification result α of each sample is obtained d .

[0022] Further, the step 4 comprises: randomly generating a bounding box B of the point cloud object inserted in the new point cloud r ={x r ,y r ,z r ,l r ,w r ,h r ,θ r};The horizontal position (x r ,y r ) of the object is randomly generated within a given range; The vertical position z r of the object point cloud is determined according to the ground plane in the new scene and the randomly generated horizontal position, the value of the orientation is within [0, 2π) to meet the uniform distribution, and the size of the size is obtained by setting the mean and variance of the three-dimensional Gaussian distribution.

[0023] Further, the step 5 comprises: determining the relative angle α r of the point cloud pseudo bounding box B r relative to the sensor according to the randomly generated or specified point cloud pseudo bounding box B r ; Selecting a point cloud instance P i with similar view angle from the corresponding category object in the point cloud object sample database, and scaling to the size of the point cloud pseudo bounding box B r ; After removing the points in the point cloud at the pseudo bounding box position, fill the scaled sample P ir into the pseudo bounding box.

[0024] Further, the step 6 comprises: mapping the coordinates (x j ,y j ,z j ) of the points p j in the scaled sample P ir to the spherical coordinate system centered on the new laser radar sensor, whose coordinates are (θ j ,φ j ,r j ), respectively, the inclination angle θ j , the azimuth angle φ j and the distance r j , wherein:

[0025]

[0026] Further, step 7 includes: setting the horizontal viewing angle [h] of the new lidar sensor. l ,h h ] and vertical perspective [v l ,v h The horizontal and vertical angular resolutions are δ h ,δ v For the above continuous coordinates (θ) j ,φ j ,r j The tilt angle θ in ) j and azimuth Quantization is performed to obtain the discrete azimuth and tilt coordinates (θ) of each point in the distance image R. jd ,φ jd ,r j ),in:

[0027]

[0028]

[0029] For cases where the vertical resolution of the lidar is uneven, the tilt angle φ will be adjusted. j Quantize to the nearest discrete angle.

[0030] Furthermore, step 8 includes: retaining the point closest to the sensor at each discrete coordinate, i.e.:

[0031]

[0032] Among them, R m For the processed distance image, R(d) θ ,d φ ) represents the distance image at discrete angles (d) θ ,d φ The point p corresponding to ) ; p k ∈R dθ,dφ The quantization angle in step 8 is (d) θ ,d φ ) all points; r refers to the distance from point p to the lidar, Refers to the set R dθ,dφ Select distance coordinate r k The smallest point p k .

[0033] Further, step 9 includes: obtaining the processed distance image R. mAfter that, it is re-projected back to the Cartesian coordinate system, that is, the data distribution characteristics of the laser radar in the approximate new scene are obtained, and the synthesized point cloud P of the object occlusion relationship exists m .

[0034] Advantages:

[0035] (1) The present application can insert instances at any position in any laser radar scene by using existing high-resolution point cloud labeled data, which is used to simulate labeled instances in the target domain.

[0036] (2) The present application can alleviate the distribution difference of data collected by different laser radar sensors to a certain extent through adaptive downsampling based on distance images. By synthesizing high-resolution point cloud labeled instances into other point cloud scenes, zero labeling cost and pre-training-free rapid domain adaptation can be achieved. BRIEF DESCRIPTION OF DRAWINGS

[0037] Figure 1 is a point cloud scene synthesis method flowchart based on distance images of the present application;

[0038] Figure 2a is a point cloud object instance view diagram;

[0039] Figure 2b is a labeled box orientation interval division diagram;

[0040] Figure 3 is a synthesized point cloud scene diagram. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical scheme and advantages of the present application clearer, the present application will be further described in detail below in combination with the drawings and examples. It should be understood that the specific examples described herein are only used to explain the present application and do not limit the present application. In addition, the technical features involved in each embodiment of the present application described below can be combined with each other as long as they do not conflict with each other.

[0042] As shown in Figure 1 , a point cloud scene synthesis method based on distance images of the present application comprises the following steps:

[0043] Step 1, point cloud object sample extraction: extracting the point cloud instances P contained in each object labeled box B from the existing high-resolution point cloud data P and the corresponding labeled box set B i i ={p1,p2,...p j}; wherein the labeled box B i ​The position information (x, y, z) of the object in the current coordinate system, the size information, length, width and height (l, w, h), and the orientation information of the object, i.e. the rotation angle θ along the Z axis, i.e. B i = {x i ,y i ,z i ,l i ,w i ,h i ,θ i}.

[0044] Step 2, point cloud object sample screening: the point cloud instances P i of the objects extracted above are screened by quality, and samples with relatively sparse object surface point clouds (the number of point clouds is lower than a threshold value) and point cloud samples with severe occlusion leading to very incomplete object surface shape (the volume ratio of the minimum circumscribed rectangle of the point cloud in the target box to the labeled box is less than a threshold value) are removed;

[0045] Step 3, point cloud object sample database construction: the coordinates of the screened point cloud instances of the objects are centralized, i.e. taking the center coordinates (x i ,y i ,z i ) of the target box of the point cloud instance as the new origin, the X axis is parallel to the target orientation, the Z axis is parallel to the Z axis of the sensor coordinate system, and the Y axis is determined by the right-hand rule. The centralized point cloud samples are used to establish a sample database, and are classified according to their direction facing the sensor, which facilitates subsequent rapid indexing. The angle range of the labeled box facing the sensor α is [0, 2π), for easy retrieval, this method equally divides the angle range into 16 intervals, and quantizes the continuous angle value into the 16 intervals, i.e. the direction classification result α d of each sample is obtained;

[0046] Step 4, pseudo-labeled box generation: randomly generate a labeled box B r = {x r ,y r ,z r ,l r ,w r ,h r ,θ r} inserted in the new point cloud. The horizontal position (x r ,y r ) of the object is randomly generated within a given range. The vertical position z rThe orientation is determined based on the ground plane in the new scene and the randomly generated horizontal position. The orientation value is uniformly distributed within [0, 2π), while the size is obtained by using a three-dimensional Gaussian distribution with manually set mean and variance (depending on prior knowledge of the new scene). Any position, orientation, and size can be manually set or generated according to a specific rule.

[0047] Step 5: Based on the randomly generated or specified pseudo-label box B of the point cloud r Determine its relative viewing angle α with respect to the sensor. r Select point cloud instances P with similar viewpoints from the corresponding category of objects in the point cloud object sample database. i And scaled to pseudo-label box B r The size of the sample P is determined after clearing the point cloud at the location of the pseudo-labeled box. ir Fill the pseudo-annotation box;

[0048] Step 6, Coordinate Transformation: Transform the scaled sample P ir point p in j coordinates (x) j ,y j ,z j Mapped to a spherical coordinate system centered on the new lidar sensor, its coordinates are (θ). j ,φ j ,r j ),in:

[0049]

[0050]

[0051]

[0052] Step 7, Coordinate Quantization: Set the horizontal viewing angle [h] of the new lidar sensor. l ,h h ] and vertical perspective [v l ,v h The horizontal and vertical angular resolutions are δ h ,δ v For the above continuous coordinates (θ) j ,φ j ,r j The tilt angle θ in ) j and azimuth Quantization is performed to obtain the discrete azimuth and tilt coordinates (θ) of each point in the distance image R. jd ,φ jd ,r j ),in:

[0053]

[0054]

[0055] For cases where the vertical resolution of a lidar is uneven, the tilt angle θ can be adjusted. j Quantize to the nearest discrete angle.

[0056] Step 8, Point Cloud Distribution Correction: Due to the inserted sample P ir The point cloud in the image is relatively dense, so a discrete coordinate corresponds to multiple points, i.e., a set of points. Where R m This is the distance image processed by this method, R(d) θ ,d φ ) represents the distance image at discrete angles (d) θ ,d φ The point corresponding to () . p k =(θ kd ,φ kd ,r k () represents a point and its quantized coordinates in the distance image. Considering occlusion relationships in real-world imaging, the point cloud closest to the sensor is retained at each discrete coordinate, i.e.:

[0057]

[0058] Where, p k ∈R dθ,dφ The quantization angle in step 7 is (d) θ ,d φ ) all points; r refers to the distance from point p to the lidar, Refers to the set R dθ,dφ Select distance coordinate r k The smallest point p k .

[0059] Step 9, Back Projection: Obtain the processed distance image R. m Reprojecting it back into the Cartesian coordinate system yields a synthetic point cloud P that approximates the data distribution characteristics of the new lidar and exhibits object occlusion relationships. m .

[0060] In an ideal scenario of new data acquisition, only empty street data is collected, without any objects to be detected. This point cloud data simulation method allows for the insertion of point cloud object instances in a format consistent with the current LiDAR data distribution, yielding corresponding reliable annotation information. This data is then directly used for supervised training of the object detection algorithm.

[0061] For new data acquisition scenarios that are not ideal, the data includes a new scene point cloud and information about the detected objects within it. Similarly, this method can be used to insert point cloud object instances and obtain corresponding reliable annotation information. At this point, the scene is in an incompletely labeled state, containing both labeled and unlabeled objects, similar to the pseudo-label state in semi-supervised and unsupervised learning. This data is used for pseudo-label-based semi-supervised / unsupervised training, during which object information in the unlabeled data is mined and utilized.

[0062] 2. Example

[0063] In this embodiment, a scene from another publicly available point cloud dataset, KITTI, is synthesized using labeled instance data from the existing publicly available high-resolution point cloud dataset, Waymo Open Dataset, and the synthesized KITTI scene point cloud data is used entirely for training. The process includes the following steps:

[0064] Step 1: Prepare a high-resolution point cloud dataset with annotation information. The annotation information in this dataset will be used for transfer to other scenes. The dataset used in this article is the Waymo Open Dataset;

[0065] Step 2: Constructing a point cloud object sample database. Based on the dataset and its annotation information from Step 1, extract the bounding boxes of objects of interest along with the point clouds they contain. After point cloud quality screening, perform coordinate centering, relative viewpoint calculation, and viewpoint mesh classification, such as... Figure 2a , Figure 2b As shown, a point cloud object sample database is obtained;

[0066] Step 3: Point cloud object sample insertion and scene synthesis. After obtaining the point cloud of the new scene, the pseudo-label box is generated first. The size of the pseudo-label box can be obtained through a three-dimensional Gaussian distribution based on the mean and variance set in the prior. The position and orientation of the pseudo-label box can be randomly set to any position in the new scene.

[0067] Then, the viewing angle of the pseudo-labeled box relative to the sensor is calculated, the corresponding category is found, and samples with the same viewing angle category are searched in the database. After performing corresponding translation, scaling, and rotation transformations according to the position, size, and orientation of the pseudo-labeled box, a new scene point cloud is inserted. Next, the coordinates of the inserted point cloud are projected from the Cartesian coordinate system to the spherical coordinate system. Then, based on the parameter information of the new scene LiDAR, including horizontal viewing angle, vertical viewing angle, horizontal angular resolution, and vertical angular resolution, the continuous azimuth and tilt coordinates of the inserted point cloud are quantized to the laser angle of the nearest LiDAR, i.e., fixed, discrete azimuth and tilt angles. This establishes a one-to-one or many-to-one correspondence between the inserted point cloud and the new scene LiDAR point cloud at the distance to the image pixels. Then, for each pixel in the distance image corresponding to several point clouds, the point closest to the sensor is selected as the value for that pixel in the distance image. Finally, the synthesized distance image is projected back to the Cartesian coordinate system to obtain the scene synthesized point cloud, as shown below. Figure 3 As shown.

[0068] In this embodiment, a 3D object detection algorithm model is trained, assuming that the target of interest—a car—is absent in the new scene. Car samples from another dataset are used to train the 3D object detection algorithm to detect cars in the new scene. After synthesizing the point cloud in step 3, the generated pseudo-labeled boxes are used as annotation information for supervised training of the detection model. Both datasets used here employ 64-line LiDAR sensors, so there is no significant difference in resolution. To achieve the transfer from high-resolution to low-resolution point clouds, multiple car samples can be directly stitched together as inserted objects. This ensures the point density of the inserted point cloud but affects the data distribution. Compared to directly inserting high-resolution point clouds (such as object point clouds collected by a 128-line sensor), the training results of this method will be lower. The training results are evaluated on a validation set of the new scene point cloud (containing car categories and having realistic annotation information), and the results are shown in Table 1. Table 1 also compares the training results of the same detection algorithm model under the supervision of real-labeled data in the new scene and the training results under the supervision of directly inserting car samples from another dataset as annotation data.

[0069]

[0070] Table 1 shows the AP evaluation results on the KITTI validation set using an IoU threshold of 0.7, with higher values ​​being better.

[0071] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A point cloud scene synthesis method based on distance images, characterized in that, Includes the following steps: Step 1: Extract point cloud object samples from existing labeled data; Step 2: Filter point cloud object samples; Step 3: Construct a point cloud object sample database; Step 4: Generate pseudo-annotation boxes in the new scene point cloud; Step 5: Match the pseudo-label box with the samples in the point cloud object sample database, and then scale up several matching samples to fill the pseudo-label box; Step 6: Transform the point cloud scene after filling the samples from the Cartesian coordinate system to the distance image coordinate system, i.e., the spherical coordinate system. This includes: scaling the samples... Points in coordinates Mapped to a spherical coordinate system centered on the new lidar sensor, its coordinates are... ,in: ； Step 7: Quantize the continuous coordinates based on the vertical and horizontal viewing angles and angular resolution of the LiDAR hardware used in the new scene, including: Set the horizontal viewing angle of the new lidar sensor and vertical perspective The horizontal and vertical angular resolutions are respectively For the above continuous coordinates Inclination angle and azimuth Quantization is performed to obtain the distance of each point in the image. Discrete azimuth and inclination coordinates ,in: For cases where the vertical resolution of the lidar is uneven, the tilt angle will be adjusted. Quantize to the nearest discrete angle; Step 8: Based on the unique correspondence between each pixel of the LiDAR point cloud and the 3D position on the distance image, the distribution of the point cloud filled in the pseudo-annotation box in Step 5 is corrected by downsampling. Step 9: Project the downsampled point cloud from Step 8 back into the Cartesian coordinate system.

2. The point cloud scene synthesis method based on distance images according to claim 1, characterized in that, Step 1 includes: processing existing high-resolution point cloud data and the corresponding set of annotation boxes Extract the bounding boxes of each object. Point cloud instances included ; where the annotation box It contains the object's position information in the current coordinate system. Dimensions, length, width, and height And the object's orientation information, i.e., the rotation angle along the Z-axis. ,Right now .

3. The point cloud scene synthesis method based on distance images according to claim 2, characterized in that, Step 2 includes: extracting point cloud instances of the objects. The samples are filtered by quality, removing those with relatively sparse point clouds on the object surface and those with severely occluded point clouds that result in very incomplete object surface shapes. Relatively sparse means that the number of point clouds is below a threshold, and severely occluded means that the ratio of the volume of the minimum bounding rectangle of the point cloud within the bounding box to the volume of the annotation box is less than a threshold.

4. The point cloud scene synthesis method based on distance images according to claim 3, characterized in that, Step 3 includes: centering the coordinates of the point cloud instances of the filtered objects, that is, using the center coordinates of the target bounding box of the point cloud instance. As the new origin, the X-axis is parallel to the target orientation, the Z-axis is parallel to the sensor coordinate system's Z-axis, and the Y-axis is determined using the right-hand rule. A sample database is built using the centered point cloud samples, and they are classified according to their orientation towards the sensor for easy indexing later. The bounding boxes face the sensor. The angle range is To facilitate retrieval, the angle range is divided into 16 equal intervals, and continuous angle values ​​are quantized into these 16 intervals, thus obtaining the orientation classification result for each sample. .

5. The point cloud scene synthesis method based on distance images according to claim 4, characterized in that, Step 4 includes: randomly generating bounding boxes for point cloud objects to be inserted into the new point cloud. ; the horizontal position of the object Randomly generated within a given range; the vertical position of the object's point cloud. The orientation value is determined based on the ground plane in the new scene and a randomly generated horizontal position. The internal structure satisfies a uniform distribution, and the size is obtained by using a three-dimensional Gaussian distribution with manually set mean and variance.

6. The point cloud scene synthesis method based on distance images according to claim 5, characterized in that, Step 5 includes: based on randomly generated or specified point cloud pseudo-annotation boxes. To determine its relative viewing angle with respect to the sensor. Select point cloud instances with similar viewpoints from the corresponding category of objects in the point cloud object sample database. And scaled to the point cloud pseudo-annotation box. Size; after clearing the point cloud at the location of the pseudo-labeled box, the scaled sample Fill the pseudo-annotation box.

7. The point cloud scene synthesis method based on distance images according to claim 1, characterized in that, Step 8 includes: retaining the point closest to the sensor on each discrete coordinate, that is: in, The processed distance image, For distance images at discrete angles The corresponding point ; The quantification perspective in step 8 is... All points; Guidance Distance to lidar, In the set Select distance coordinates The smallest point .

8. The point cloud scene synthesis method based on distance images according to claim 7, characterized in that, Step 9 includes: obtaining the distance image after the above processing. Then, it is reprojected back into the Cartesian coordinate system to obtain the data distribution characteristics of the lidar in the approximate new scene, and the synthetic point cloud with object occlusion relationships. .

Images