3D LiDAR Road Point Cloud Segmentation Method Based on YOLACT

Abstract

A YOLACT-based 3D LiDAR road point cloud segmentation method includes the following steps: transforming the sample data point cloud to spherically project the 3D LiDAR road point cloud data into 2D image data; annotating and segmenting objects to construct an image dataset for training; training based on a convolutional neural network model to obtain a 3D LiDAR road point cloud instance segmentation model; and processing the measured 3D LiDAR road point cloud data to be processed to segment road object instances. The method of the present invention first spherically projects the 3D point cloud data into 2D image data, and then uses the YOLACT end-to-end image instance segmentation algorithm to accurately segment road object instances.

Description

Technical Field

[0001] The present invention relates to the field of point cloud instance segmentation technology, in particular to three-dimensional laser radar road point cloud instance segmentation, and specifically to a method for three-dimensional laser radar road point cloud instance segmentation using the YOLACT image instance segmentation algorithm. Background Art

[0002] Autonomous driving technology uses onboard sensors to perceive the road environment around the vehicle, automatically planning the route and controlling the vehicle to its destination. Among numerous sensors, 3D LiDAR (LiDAR) has been widely used in autonomous driving due to its high ranging accuracy, wide detection range, and unaffected lighting conditions. Segmenting road object instances such as cars and pedestrians from point cloud data acquired by 3D LiDAR is a primary step in autonomous driving, and its accuracy determines its reliability. However, the complexity and diversity of the road environment present numerous challenges for 3D LiDAR road point cloud instance segmentation.

[0003] Existing methods for 3D LiDAR road point cloud instance segmentation are primarily pipeline methods. These methods decompose 3D LiDAR road point cloud instance segmentation into several steps: first, removing the ground point cloud, then clustering the remaining point cloud, and finally classifying the clustered points into road objects. These methods suffer from several issues: ① They rely heavily on manually designed features or decision rules, resulting in poor generalizability; ② Each step is designed independently, resulting in significant error accumulation. Summary of the Invention

[0004] The purpose of this invention is to solve the problem of three-dimensional lidar road point cloud instance segmentation. The present invention proposes a three-dimensional lidar road point cloud instance segmentation method based on YOLACT. First, the three-dimensional point cloud data is spherically projected into two-dimensional image data, and then the YOLACT end-to-end image instance segmentation algorithm is used to achieve accurate segmentation of road target instances.

[0005] The technical solution of the present invention is:

[0006] A YOLACT-based three-dimensional laser radar road point cloud segmentation method, the method comprising the following steps:

[0007] S1. Sample data point cloud transformation: Mount a 3D laser radar on a mobile vehicle to scan the road to obtain sample point cloud data, including the laser reflection intensity I and the 3D rectangular coordinates (x, y, z) of the measurement points; perform point cloud transformation to spherically project the 3D laser radar road point cloud data into 2D image data;

[0008] S2. Construct a training set: Obtain two-dimensional image data of several samples according to step S1, annotate the two-dimensional image data and segment the objects, and construct an image dataset for training;

[0009] S3. Model training: Based on the convolutional neural network model, the image dataset obtained in step S2 is trained to obtain a 3D lidar road point cloud instance segmentation model;

[0010] S4, model prediction: The measured 3D LiDAR road point cloud data to be processed is processed using the steps of S1 to obtain 2D image data. The 2D image data is cut and then input into the 3D LiDAR road point cloud instance segmentation model for processing to segment the road target instances.

[0011] Furthermore, the two-dimensional image data obtained by point cloud transformation in step S1 includes: two-dimensional image pixel coordinates (i, j) and three-channel image data (r, g, b) at the pixel coordinates, and specifically the following steps are adopted:

[0012] S1-1. Let O be the laser beam emission position and P be the measurement point. According to the three-dimensional rectangular coordinates (x, y, z) of the measurement point P, calculate the zenith angle of the laser beam OP. and azimuth angle θ;

[0013]

[0014] S1-2. Obtain the two-dimensional image pixel coordinates (i, j) of the measurement point P according to the following formula;

[0015]

[0016] in: is the zenith angle resolution, is the minimum zenith angle of the three-dimensional lidar, Δθ is the azimuth resolution, θ min is the minimum azimuth angle of the 3D lidar;

[0017] S1-3, construct a three-channel color image with width w and height h, and the default value of the image pixel is 0;

[0018]

[0019] in: is the maximum zenith angle of the 3D lidar, θ max is the maximum azimuth angle of the 3D lidar;

[0020] S1-4. Use the following formula to obtain the three-channel image data (r, g, b) at the pixel coordinate (i, j) of the measurement point P:

[0021]

[0022] S1-5. Use the normalization method to transform the value range of the three-channel image data to 0-255.

[0023] Furthermore, in step S2,

[0024] S2-1. For two-dimensional image data, each image is cut into a uniform size;

[0025] S2-2. Use labelme to mark the road target instances that need to be segmented in each image and obtain the corresponding json file;

[0026] S2-3. Convert the training images and json files into a dataset in coco format.

[0027] Furthermore, in step S2-1, the cropped image is horizontally flipped to expand the data set.

[0028] Furthermore, the model training specifically includes:

[0029] S3-1. Input the image dataset used for training into the convolutional neural network model to generate a feature map;

[0030] S3-2, input the generated feature map into the feature pyramid FPN for fusion;

[0031] S3-3. Perform semantic segmentation on the feature map after feature pyramid FPN fusion and calculate the semantic segmentation loss. At the same time, input the feature map into two parallel branches respectively:

[0032] The first branch uses the FCN-based prototype mask generation network protonet to generate the prototype mask prototypemask;

[0033] The second branch outputs the four coordinates of each target bounding box position, mask coefficient and classification confidence rate through the prediction head network, and performs fast non-maximum suppression FastNMS processing on the results obtained by the Prediction Head network;

[0034] S3-4, combine the processing results of FastNMS and the prototype mask output by Protonet to output the instance information of the final road target, including instance category, instance bounding box and instance mask;

[0035] S3-5. Compare the category, bounding box, and mask information of the road target instance obtained in step S3-4 with the road target instance information marked in step S2, and add the semantic segmentation loss of S3-3. Use the multi-task loss function L as the optimization target, train the model several times, complete the training, and obtain a three-dimensional lidar road point cloud instance segmentation model based on YOLACT.

[0036] Furthermore, the combined operations include superposition, cropping and threshold segmentation.

[0037] Furthermore, the multi-task loss function L is calculated using the following formula:

[0038] L=L cls +L box +L mask +L semantic

[0039] Where: L cls represents the category loss function, L box represents the bounding box loss function, L mask represents the mask loss function, L semantic represents the semantic segmentation loss function.

[0040] Beneficial effects of the present invention:

[0041] The present invention proposes a three-dimensional lidar road point cloud instance segmentation method based on YOLACT. By spherically projecting the three-dimensional lidar road point cloud data into two-dimensional image data, an end-to-end image instance segmentation algorithm with low time / space complexity is adopted to realize three-dimensional lidar road point cloud instance segmentation. It can automatically learn features and decision rules from labeled data, so it has good generalization ability. Moreover, since the three-dimensional lidar road point cloud instance segmentation model is trained as a whole, the problem of error accumulation is avoided.

[0042] Other features and advantages of the present invention will be described in detail in the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The above and other objects, features and advantages of the present invention will become more apparent through a more detailed description of exemplary embodiments of the present invention with reference to the accompanying drawings, wherein like reference numerals generally represent like components throughout the exemplary embodiments of the present invention.

[0044] Figure 1 A schematic diagram of spherical projection in the embodiment is shown.

[0045] Figure 2 A schematic diagram of labelme annotation in the embodiment is shown.

[0046] Figure 3 A schematic diagram of a three-dimensional laser radar road in an embodiment is shown.

[0047] Figure 4 Schematic diagram of the model prediction results in the embodiment. DETAILED DESCRIPTION

[0048] The preferred embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although the preferred embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited to the embodiments set forth herein.

[0049] A YOLACT-based three-dimensional laser radar road point cloud segmentation method, the method comprising the following steps:

[0050] S1. Point cloud transformation: spherical projection of 3D laser radar road point cloud data into 2D image data. Figure 1 As shown, O is the laser beam emission position, P is a measurement point, given the three-dimensional rectangular coordinates (x, y, z) of the measurement point P, the zenith angle of the laser beam OP is calculated. and azimuth angle θ.

[0051]

[0052] in: is the zenith angle resolution, is the minimum zenith angle of the three-dimensional lidar, Δθ is the azimuth resolution, θ min is the minimum azimuth angle of the 3D lidar;

[0053] S1-2, find the pixel coordinates (i, j) of point P, such as Figure 1 As shown, this is a schematic diagram of spherical projection:

[0054]

[0055] S1-3. Construct a three-channel color image with a width of w and a height of h. The default value of the image pixels is 0.

[0056]

[0057] in: and θ max are the maximum zenith angle and maximum azimuth angle of the 3D lidar road point cloud.

[0058] S1-4. Use the following formula to obtain the three-channel image data (r, g, b) at the pixel coordinate (i, j) of the measurement point P:

[0059]

[0060] S1-5. Use maximum and minimum normalization to transform the value range of the three-channel image data to 0-255.

[0061] S2. Training set construction: Collect a certain amount of 3D LiDAR road point cloud data and construct an image dataset for training the 3D LiDAR road point cloud instance segmentation model.

[0062] S2-1. First, use the method described in S1 to transform the 3D LiDAR road point cloud data into image data, cutting each image to a suitable size. Then, horizontally flip each image to expand the dataset to twice its original size.

[0063] S2-3, use labelme to mark the road target instances that need to be segmented in each image, and obtain the corresponding json file, such as Figure 2 The following is a schematic diagram of labelme annotation.

[0064] S2-4 converts the training images and json files into a dataset in coco format.

[0065] S3. Model training. Based on the YOLACT end-to-end image instance segmentation model, the obtained coco format dataset is used to train the 3D lidar road point cloud instance segmentation model. Specifically:

[0066] S3-1. Input the training image into the backbone to generate a feature map.

[0067] S3-2. The generated feature map is input into the feature pyramid FPN to fuse the low-resolution feature map with strong semantic information and the high-resolution feature map with weak semantic information but rich spatial information.

[0068] S3-3. Perform semantic segmentation on the feature map generated by the FPN and calculate the semantic segmentation loss. The feature map is then fed into two parallel branches: the first branch uses the FCN-based protonet network to generate a prototype mask. The second branch uses the Prediction Head network to output the target location, mask coefficient, and classification confidence. The results of the Prediction Head network are then processed using FastNMS.

[0069] S3-4. The processing results of FastNMS and the prototype mask output by Protonet are combined (overlay, cropping, threshold segmentation) to output the instance category, instance bounding box, and instance mask of the final road target.

[0070] S3-5. Compare the road target instance category, instance bounding box, and instance mask obtained in step S3-4 with the marked actual road target instance category, actual bounding box, and actual mask, and add the semantic segmentation loss of S3-3. Take the multi-task loss function L as the optimization target, train the model several times, complete the training, and obtain a three-dimensional lidar road point cloud instance segmentation model based on YOLACT.

[0071] L=L cls +L box +Lmask +L semantic

[0072] Where: L cls represents the category loss function, L box represents the bounding box loss function, L mask represents the mask loss function, L semantic represents the semantic segmentation loss function.

[0073] S4. Model prediction: 3D LiDAR road point cloud data to be processed:

[0074] First, the method described in S1 is used to transform it into image data, each image is cut to a suitable size, and then the model trained in S3 is used to segment the road target instances.

[0075] When implementing:

[0076] The experiment uses the autonomous driving KITTI dataset, which uses a 64-line LiDAR (mounted at the center of the roof) Velodyne HDL-64E to collect road point cloud data. Therefore, the height of the front view is 64. At the same time, due to the influence of the dataset annotation, only the 90° front viewing angle range is considered and divided into 512 cells, so the width of the front view is 512. Maximum zenith angle Minimum zenith angle Zenith angle resolution Maximum azimuth angle θ max =45°, minimum azimuth angle θ min =-45°, azimuth resolution Δθ = 0.17°.

[0077] First, the point cloud is transformed and cut using the steps of S1 to obtain 64×512×3 image data, as shown in Figure 3 The following is a schematic diagram of a three-dimensional laser radar road.

[0078] Next, the training set is constructed using the method described in S2. Then, the 3D LiDAR road point cloud instance segmentation model is trained using the method described in S3. Finally, the model prediction is performed using the method described in S4. Figure 4 The instance segmentation results of the car are given.

[0079] Finally, the mAP values ​​of the two categories under different IoU ratios of the test set are obtained, as shown in Table 1 below, where the value under "all" is the average mAP of all IoU ratios between 50 and 95 for both box and mask.

[0080] all .50 .55 .60 .65 .70 .75 .80 .85 .90 .95 box 54.43 88.28 88.28 88.28 85.68 75.04 70.11 42.03 3.83 2.57 0.21 mask 64.94 95.57 91.46 91.46 86.63 83.42 80.08 80.08 39.78 1.89 0.00

[0081] According to the above table, it can be seen that the prediction accuracy of the YOLACT-based three-dimensional lidar road point cloud instance segmentation method of the present invention is relatively high.

[0082] While various embodiments of the present invention have been described above, the above description is intended to be illustrative, not exhaustive, and not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments.

Claims

1. A 3D LiDAR road point cloud segmentation method based on YOLACT, characterized in that: The method comprises the following steps: S1. Sample data point cloud transformation: Mount a 3D laser radar on a mobile vehicle to scan the road to obtain sample point cloud data, including the laser reflection intensity I and the 3D rectangular coordinates (x, y, z) of the measurement points; perform point cloud transformation to spherically project the 3D laser radar road point cloud data into 2D image data; S2. Construct a training set: Obtain two-dimensional image data of several samples according to step S1, annotate the two-dimensional image data and segment the objects, and construct an image dataset for training; S3. Model training: Based on the convolutional neural network model, the image dataset obtained in step S2 is trained to obtain a 3D lidar road point cloud instance segmentation model; S4, model prediction: The measured 3D LiDAR road point cloud data is processed using the steps of S1 to obtain 2D image data, which is then segmented and input into the 3D LiDAR road point cloud instance segmentation model for processing to segment road target instances. The two-dimensional image data obtained by point cloud transformation in step S1 includes: the two-dimensional image pixel coordinates (i, j) and the three-channel image data (r, g, b) at the pixel coordinates. Specifically, the following steps are used: S1-1. Let O be the laser beam emission position and P be the measurement point. According to the three-dimensional rectangular coordinates (x, y, z) of the measurement point P, calculate the zenith angle of the laser beam OP. and azimuth angle θ; S1-2. Obtain the two-dimensional image pixel coordinates (i, j) of the measurement point P according to the following formula; in: is the zenith angle resolution, is the minimum zenith angle of the three-dimensional lidar, Δθ is the azimuth resolution, θ min is the minimum azimuth angle of the 3D lidar; S1-3, construct a three-channel color image with width w and height h, and the default value of the image pixel is 0; in: is the maximum zenith angle of the 3D lidar, θ max is the maximum azimuth angle of the 3D lidar; S1-4. Use the following formula to obtain the three-channel image data (r, g, b) at the pixel coordinate (i, j) of the measurement point P: S1-5, using the normalization method to transform the value range of the three-channel data of the image to 0-255; The model training specifically includes: S3-1. Input the image dataset used for training into the convolutional neural network model to generate a feature map. S3-2. Input the generated feature map into the feature pyramid FPN for fusion. S3-3. Perform semantic segmentation on the feature map after feature pyramid FPN fusion and calculate the semantic segmentation loss. At the same time, input the feature map into two parallel branches respectively: The first branch uses the FCN-based prototype mask generation network protonet to generate the prototype mask prototypemask; The second branch outputs the four coordinates of each target bounding box position, mask coefficient and classification confidence rate through the prediction head network, and performs fast non-maximum suppression FastNMS processing on the results obtained by the Prediction Head network; S3-4. Combine the processing results of FastNMS and the prototype mask output by Protonet to output the instance information of the final road target, including instance category, instance bounding box and instance mask; S3-5. Compare the category, bounding box and mask information of the road target instance obtained in step S3-4 with the road target instance information marked in step S2, and add the semantic segmentation loss of S3-3. Take the multi-task loss function L as the optimization target, train the model several times, complete the training, and obtain the 3D lidar road point cloud instance segmentation model based on YOLACT.

2. The YOLACT-based 3D laser radar road point cloud segmentation method according to claim 1, characterized in that: In step S2, S2-1. For two-dimensional image data, cut each image into a uniform size; S2-2. Use labelme to mark the road target instances that need to be segmented in each image and obtain the corresponding json file; S2-3. Convert the training images and json files into a dataset in coco format.

3. The YOLACT-based 3D laser radar road point cloud segmentation method according to claim 2, characterized in that: In step S2-1, the cropped image is horizontally flipped to expand the data set.

4. The YOLACT-based 3D laser radar road point cloud segmentation method according to claim 1, characterized in that: Combined operations include overlay, cropping, and threshold segmentation.

5. The YOLACT-based 3D laser radar road point cloud segmentation method according to claim 1, characterized in that: The multi-task loss function L is calculated using the following formula: L=L cls +L box +L mask +L semantic Where: L cls represents the category loss function, L box represents the bounding box loss function, L mask represents the mask loss function, L semantic represents the semantic segmentation loss function.

Images