Street tree target point cloud segmentation method based on YOLACT

Abstract

The present invention provides a street tree target point cloud segmentation method based on YOLACT, which includes the following steps: S1 selects some sample street trees, mounts a two-dimensional radar on a mobile vehicle to scan the street trees, and obtains sample point cloud data; S2 establishes a three-dimensional coordinate system for the scanned point cloud data, and constructs a three-channel color image; S3 crops and annotates the obtained three-channel color image; S4 uses the YOLACT street tree target point cloud segmentation algorithm to train the annotated image until a preset number of iterations is reached, the training is completed, and a model is obtained; S5 uses the trained model to perform target segmentation on the street trees to be tested. The street tree target point cloud segmentation method based on Yolact of the present invention can detect street tree target point clouds in complex urban environments, and quickly and accurately segment street tree targets; it can provide real-time data on target application for the promotion of orchard target application technology.

Description

Technical Field

[0001] The present invention relates to the field of two-dimensional LiDAR point cloud technology, in particular to a roadside tree target point cloud segmentation method based on image segmentation, specifically a method for converting point cloud data obtained by two-dimensional LiDAR scanning into a two-dimensional image and performing roadside tree target point cloud segmentation using Yolact. Background Art

[0002] Due to the influence of factors such as the greenhouse effect and human interference, the number of diseases and pests of street trees is increasing, causing the street trees to wither or die. This not only seriously affects the greening and beautification effects of street trees, but also directly affects the urban ecological environment and residents' lives, becoming a restrictive factor in the refined management of garden greening. Therefore, it is of great significance to quickly and accurately segment the street tree target point cloud and provide real-time data for target application of street trees.

[0003] There are two main methods for street tree point cloud segmentation: one method divides the point cloud data into multiple grids or voxels as the basic data processing unit, then analyzes the morphology of the ground object target, designs features and rules for segmentation, and gradually filters the non-street tree point cloud; the other method directly processes the original point cloud data, extracts multiple features from the neighborhood of each point, and trains a binary classifier to identify the street tree point cloud. Researchers widely use two-dimensional radar for target application in orchards, which has the advantages of flexible and accurate methods.

[0004] The existing methods are based on manually designed segmentation / classification features, and use clustering, regional growth and other methods to extract individual street trees, which are suitable for orchards with relatively simple environments. However, when faced with complex urban environments, there are problems such as weak feature expression ability, difficulty in describing complex changes in objects, and weak modeling capabilities. Summary of the invention

[0005] The purpose of the present invention is to propose a street tree target point cloud segmentation method based on YOLACT in response to the current street tree segmentation method that uses single street tree extraction, which is suitable for orchards with relatively simple environments, but has weak feature expression capabilities, difficulty in describing complex changes in land objects, and weak modeling capabilities when facing complex urban environments. The street tree target point cloud segmentation method based on Yolact of the present invention can detect street tree target point clouds in complex urban environments and quickly and accurately segment street tree targets. It provides real-time data for target application of pesticides for the promotion of targeted application technology in orchards.

[0006] The technical solution of the present invention is:

[0007] The present invention provides a roadside tree target point cloud segmentation method based on YOLACT, the method comprising the following steps:

[0008] S1. Select a part of the roadside tree area to be tested as a sample roadside tree, mount a two-dimensional radar on a moving vehicle to scan the roadside tree, and obtain point cloud data of the sample roadside tree, including the distances r1, r2, r3 and the echo intensities I1, I2, I3 of the three echoes;

[0009] S2. For the scanned point cloud data, a three-dimensional coordinate system is established based on the starting position of the two-dimensional radar for the single echo data; the depth information of the single echo measurement point is selected as the image R channel, the single echo intensity is selected as the image G channel, and the effective echo number of the laser beam is selected as the B channel to construct a three-channel color image;

[0010] S3, cropping and labeling the obtained three-channel color image;

[0011] S4. Use the YOLACT street tree target point cloud segmentation algorithm to train the annotated image until the preset number of iterations is reached, the training is completed, and a street tree target segmentation model is obtained;

[0012] S5. Collect and process the roadside trees to be segmented according to S1-S3, and use the trained model to perform target segmentation on the roadside trees to be segmented.

[0013] Furthermore, in step S1, the measurement distance of the two-dimensional radar is 0.1m-30m, the measurement accuracy is ±30mm, and the scanning range is -135° to 135°.

[0014] Furthermore, step S2 is specifically as follows:

[0015] S2-1. Take the initial position of the two-dimensional radar as the coordinate origin O, establish a rectangular coordinate system O-xyz, the x-axis direction is the movement direction of the two-dimensional radar with the vehicle, the y-axis direction is the scanning depth direction of the two-dimensional radar, and the z-axis direction is the height direction of the scanned target perpendicular to the ground; according to the following formula, transform the echo data in the point cloud data into the three-dimensional coordinate system, and the three-dimensional coordinates of the j-th measurement point in the i-th scanning line are:

[0016]

[0017] Wherein: i represents the number of the scan line, j represents the number of the measurement point, v represents the vehicle speed, r1(i,j) represents the distance of the primary echo of the jth measurement point in the i-th scan line; θ(j) represents the scanning angle of the jth measurement point in the scan line; Δt represents the scanning period of the two-dimensional radar; x(i,j) represents the x-axis coordinate of the primary echo of the jth measurement point in the i-th scan line, y(i,j) represents the y-axis coordinate of the primary echo of the jth measurement point in the i-th scan line, and z(i,j) represents the z-axis coordinate of the primary echo of the jth measurement point in the i-th scan line; S2-2, use the following formula for normalization, and transform the value range of the depth information y of the primary echo measurement point obtained by the two-dimensional laser radar, the primary echo intensity I1 and the number of effective echoes n of the laser beam emitted by the two-dimensional laser radar to a unified range:

[0018]

[0019] Where: k max Indicates the maximum value of the depth / intensity / echo number of the current point cloud data, k min Indicates the minimum value of the current point cloud data depth / intensity / echo number, k represents the actual data of the current point cloud depth / intensity / echo number, and k* represents the normalized point cloud depth / intensity / echo number data;

[0020] S2-3. The depth information y of the first echo measurement point of the two-dimensional laser radar is normalized as the image R channel, the first echo intensity I1 of the two-dimensional laser radar is normalized as the image G channel, and the effective echo number n of the laser beam emitted by the two-dimensional laser radar is normalized as the B channel to construct a three-channel color image.

[0021] Furthermore, step S3 is specifically as follows:

[0022] S3-1, cropping the three-channel color image from left to right with a preset cropping width to obtain several new images;

[0023] S3-2. Use the labelme annotation tool to annotate the image, take points along the outline of the street tree to form a polygon. After the annotation is completed, a mask file is generated, which records the polygon vertex information of each street tree target.

[0024] Furthermore, the cutting width is greater than the width of a tree.

[0025] Furthermore, the trimming width is the width of two trees.

[0026] Beneficial effects of the present invention:

[0027] The Yolac-based street tree target point cloud segmentation method of the present invention can detect street tree target point clouds in complex urban environments and quickly and accurately segment street tree targets; it can provide real-time data on target application of pesticides for the promotion of orchard target application technology.

[0028] The present invention uses mobile laser scanning (MLS) technology to collect three-dimensional point cloud data on one side of the street; then, extracts depth, echo intensity and echo number information to establish a three-channel street point cloud image composed of two-dimensional LiDAR scanning lines; finally, uses the image instance segmentation algorithm YOLACT to establish a street tree target segmentation model, and segments each street tree target from the street point cloud image in a flexible and accurate manner.

[0029] Other features and advantages of the present invention will be described in detail in the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The above and other objects, features and advantages of the present invention will become more apparent through a more detailed description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, wherein like reference numerals generally represent like components throughout the exemplary embodiments of the present invention.

[0031] Figure 1 A schematic diagram of a rectangular coordinate system in an embodiment of the present invention is shown.

[0032] Figure 2 A schematic diagram showing a three-channel color image constructed by depth information, echo intensity and echo number in an embodiment of the present invention.

[0033] Figure 3 A schematic diagram of target marking of roadside trees in an embodiment of the present invention is shown.

[0034] Figure 4 It shows the loss curve of iteration in an embodiment of the present invention. DETAILED DESCRIPTION

[0035] The preferred embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although the preferred embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited to the embodiments set forth herein.

[0036] The present invention provides a roadside tree target point cloud segmentation method based on YOLACT, the method comprising the following steps:

[0037] S1. Select a part of the roadside tree area to be tested as a sample roadside tree, load the two-dimensional radar on a mobile vehicle to scan the roadside tree, and obtain the point cloud data of the sample roadside tree, including the distances r1, r2, r3 and the echo intensities I1, I2, I3 of the three echoes; wherein: the measurement distance of the two-dimensional radar is 0.1m-30m, the measurement accuracy is ±30mm, and the scanning range is -135° to 135°;

[0038] S2. For the scanned point cloud data, a three-dimensional coordinate system is established based on the starting position of the two-dimensional radar for the single echo data; the depth information of the single echo measurement point is selected as the image R channel, the single echo intensity is selected as the image G channel, and the effective echo number of the laser beam is selected as the B channel to construct a three-channel color image, specifically:

[0039] S2-1, take the initial position of the two-dimensional radar as the coordinate origin O, and establish the rectangular coordinate system O-xyz, such as Figure 1 As shown in the figure, it is the coordinate system of the MLS measurement system. The x-axis direction is the movement direction of the two-dimensional radar with the vehicle, the y-axis direction is the scanning depth direction of the two-dimensional radar, and the z-axis direction is the height direction of the scanned target perpendicular to the ground. According to the following formula, the echo data in the point cloud data is converted into the three-dimensional coordinate system. The three-dimensional coordinates of the jth measurement point in the i-th scanning line are:

[0040]

[0041] Wherein: i represents the number of the scan line, j represents the number of the measurement point, v represents the vehicle speed, r1(i,j) represents the distance of the primary echo of the jth measurement point in the i-th scan line; θ(j) represents the scanning angle of the jth measurement point in the scan line; Δt represents the scanning period of the two-dimensional radar; x(i,j) represents the x-axis coordinate of the primary echo of the jth measurement point in the i-th scan line, y(i,j) represents the y-axis coordinate of the primary echo of the jth measurement point in the i-th scan line, and z(i,j) represents the z-axis coordinate of the primary echo of the jth measurement point in the i-th scan line; S2-2, use the following formula for normalization, and transform the value range of the depth information y of the primary echo measurement point obtained by the two-dimensional laser radar, the primary echo intensity I1 and the number of effective echoes n of the laser beam emitted by the two-dimensional laser radar to a unified range:

[0042]

[0043] Where: k max Indicates the maximum value of the depth / intensity / echo number of the current point cloud data, k min Indicates the minimum value of the current point cloud data depth / intensity / echo number, k represents the actual data of the current point cloud depth / intensity / echo number, and k* represents the normalized point cloud depth / intensity / echo number data;

[0044] S2-3, such as Figure 2 As shown, the depth information y of the first echo measurement point of the two-dimensional laser radar is normalized as the image R channel, the first echo intensity I1 of the two-dimensional laser radar is normalized as the image G channel, and the effective echo number n of the laser beam emitted by the two-dimensional laser radar is normalized as the B channel to construct a three-channel color image.

[0045] S3, cropping and labeling the obtained three-channel color image;

[0046] S4. Use the YOLACT street tree target point cloud segmentation algorithm to train the annotated image until the preset number of iterations is reached, complete the training, and obtain the street tree target segmentation model, specifically: S3-1. On a three-channel color image with a specification of 30000×720, crop from left to right with a cropping width of 250 or 450 to obtain 700 new images of 450×720 or 450×720 three-channel color images;

[0047] S3-2. Use the labelme tool to annotate the image and take points along the outline of the roadside tree to form a polygon, such as Figure 3 As shown in the figure, after the labeling is completed, a mask file with the suffix json is generated, which records the polygon vertex information of each street tree target;

[0048] S5, collecting and processing the roadside trees to be segmented according to S1-S3, and using the trained model to perform target segmentation on the roadside trees to be segmented. The clipping width is greater than the width of a tree.

[0049] Furthermore, the trimming width is the width of two trees.

[0050] The Yolac-based street tree target point cloud segmentation method of the present invention can detect street tree target point clouds in complex urban environments and quickly and accurately segment street tree targets, providing real-time data on target application for the promotion of orchard target application technology.

[0051] Test example:

[0052] The annotated images are divided into training set, validation set and test set in a ratio of 5:1:1 to train and test the street tree target segmentation model. The YOLACT street tree target point cloud segmentation algorithm is used to train the street tree target segmentation model on the sliced ​​images.

[0053] The experiment uses the open source deep learning target detection toolbox MMDetection to implement the YOLACT algorithm and establish a street tree target point cloud segmentation model. Table 1 gives the experimental platform configuration parameters.

[0054] Table 1 Experimental platform configuration

[0055] project Configuration operating system Windows 10 CPU AMD Ryzen 5 3600 GPU GeForce RTX 2070 GPU Acceleration CUDA 10.1 + cuDNN 7.6.4 IDE Pycharm Deep Learning Frameworks Pytorch 1.6.0

[0056] Use the ImageNET pre-training parameters, input the images and json mask files of the training set and validation set, and train the street tree segmentation model. Figure 4 The loss curve is given for 55 rounds of iterations (63 times per round) and a total of 3437 iterations. The loss tends to converge after about 3000 iterations.

[0057] The trained model was used to segment the street tree targets on the test set. When the intersection of union (IoU) between the detection mask and the true value mask was greater than 0.5, the street tree was considered to be correctly detected. All 116 street tree targets were correctly detected without false alarms. The average mask IoU was 76.6% and the average detection frame IoU was 84.9%. The average processing time for each scan line was 0.4404ms, which was much shorter than the 25ms scan line cycle, meeting the real-time target detection requirements.

[0058] The embodiments of the present invention have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments.

Claims

1. A roadside tree target point cloud segmentation method based on YOLACT, characterized in that: The method comprises the following steps: S1, select a part of the roadside tree area to be tested as a sample roadside tree, mount the two-dimensional radar on a mobile vehicle to scan the roadside tree, and obtain point cloud data of the sample roadside tree, including the distances r1, r2, r3 and the echo intensities I1, I2, I3 of the three echoes; S2. For the scanned point cloud data, a three-dimensional coordinate system is established based on the starting position of the two-dimensional radar for the single echo data; the depth information of the single echo measurement point is selected as the image R channel, the single echo intensity is selected as the image G channel, and the effective echo number of the laser beam is selected as the B channel to construct a three-channel color image; S3, cropping and labeling the obtained three-channel color image; S4. Use the YOLACT street tree target point cloud segmentation algorithm to train the annotated image until the preset number of iterations is reached, the training is completed, and a street tree target segmentation model is obtained; S5. Collect and process the roadside trees to be segmented according to S1-S3, and use the trained model to perform target segmentation on the roadside trees to be segmented.

2. The YOLACT-based roadside tree target point cloud segmentation method according to claim 1, characterized in that: In step S1, the measurement distance of the two-dimensional radar is 0.1m-30m, the measurement accuracy is ±30mm, and the scanning range is -135° to 135°.

3. The YOLACT-based street tree target point cloud segmentation method according to claim 1, characterized in that: Step S2 is specifically as follows: S2-1. Take the initial position of the two-dimensional radar as the coordinate origin O, establish a rectangular coordinate system O-xyz, the x-axis direction is the movement direction of the two-dimensional radar with the vehicle, the y-axis direction is the scanning depth direction of the two-dimensional radar, and the z-axis direction is the height direction of the scanned target perpendicular to the ground; according to the following formula, transform the echo data in the point cloud data into the three-dimensional coordinate system, and the three-dimensional coordinates of the j-th measurement point in the i-th scanning line are: Wherein: i represents the number of the scan line, j represents the number of the measurement point, v represents the vehicle speed, r1(i, j) represents the distance of the primary echo of the jth measurement point in the i-th scan line; θ(j) represents the scanning angle of the jth measurement point in the scan line; Δt represents the scanning period of the two-dimensional radar; x(i, j) represents the x-axis coordinate of the primary echo of the jth measurement point in the i-th scan line, y(i, j) represents the y-axis coordinate of the primary echo of the jth measurement point in the i-th scan line, and z(i, j) represents the z-axis coordinate of the primary echo of the jth measurement point in the i-th scan line; S2-2, use the following formula for normalization, and transform the value range of the depth information y of the primary echo measurement point obtained by the two-dimensional laser radar, the primary echo intensity I1 and the number of effective echoes n of the laser beam emitted by the two-dimensional laser radar to a unified range: Where: k max Indicates the maximum value of the depth / intensity / echo number of the current point cloud data, k min Indicates the minimum value of the current point cloud data depth / intensity / echo number, k is the actual data of the current point cloud depth / intensity / echo number, k * Represents the normalized point cloud depth / intensity / echo number data; S2-3. The depth information y of the first echo measurement point of the two-dimensional laser radar is normalized as the image R channel, the first echo intensity I1 of the two-dimensional laser radar is normalized as the image G channel, and the effective echo number n of the laser beam emitted by the two-dimensional laser radar is normalized as the B channel to construct a three-channel color image.

4. The YOLACT-based roadside tree target point cloud segmentation method according to claim 1, characterized in that: Step S3 is specifically as follows: S3-1, cropping the three-channel color image from left to right with a preset cropping width to obtain several new images; S3-2. Use the labelme annotation tool to annotate the image, take points along the outline of the street tree to form a polygon. After the annotation is completed, a mask file is generated, which records the polygon vertex information of each street tree target.

5. The YOLACT-based roadside tree target point cloud segmentation method according to claim 4, characterized in that: The cutting width is greater than the width of a tree.

6. The YOLACT-based roadside tree target point cloud segmentation method according to claim 4 or 5, characterized in that: The trimming width is the width of two trees.

Images