Multi-line LiDAR orchard trunk instance segmentation method based on image deep learning

Abstract

The invention provides a multi-line LiDAR orchard trunk instance segmentation method based on image deep learning. The method comprises a point cloud acquisition and preprocessing step, a point cloud spherical projection step, a trunk image semantic segmentation model training step, a trunk image semantic segmentation step, an image spherical inverse projection step and a trunk point cloud instance segmentation step. According to the method, sparse and disordered multi-line LiDAR point cloud is converted into dense and regular image data through a spherical projection technology, and the data processing complexity is remarkably reduced while multi-dimensional information such as three-dimensional coordinates, distance and strength is reserved; a trunk semantic feature is mined from the annotated data in combination with a U-Net architecture network, and a high-precision trunk semantic segmentation model is constructed; a traditional point cloud processing algorithm is introduced for optimization, and the progress from semantic segmentation to instance segmentation is realized. According to the scheme, while the calculation efficiency is considered, the complex interference problems of trunk form diversity, low vertical crowns, ground facilities and the like in the orchard scene can be effectively solved, and accurate real-time segmentation of the orchard trunk instance is realized.

Description

Technical Field

[0001] This invention belongs to the field of intelligent equipment technology for orchards, specifically relating to environmental perception technology in the autonomous navigation of orchard mobile robots. In particular, it relates to a multi-line LiDAR (LiDAR) instance segmentation method for orchard tree trunks based on image deep learning. This method is suitable for using multi-line LiDAR point cloud data to achieve accurate and real-time instance segmentation of orchard tree trunks, providing key environmental information support for the positioning and path planning of orchard mobile robots. Background Technology

[0002] Tree trunk segmentation is the core foundation for orchard mobile robots to achieve autonomous navigation. In orchard environments with distinct tree trunk features, mobile robots need to use sensors to collect environmental data, segment tree trunk instances to obtain their precise locations, and then complete localization and path planning.

[0003] LiDAR (Light Detection and Ranging) acquires high-precision point cloud data by actively emitting laser beams and capturing reflected light. Due to its advantages such as high ranging accuracy, strong anti-interference ability, and long sensing distance, it is widely used in the field of autonomous navigation of mobile robots in orchards.

[0004] (1) Single-line LiDAR: For single-line LiDAR point clouds, jump point recognition or clustering algorithms are usually used to divide the point cloud into clusters, and then geometric features are combined to remove interference clusters to identify tree trunks. However, since single-line LiDAR can only acquire two-dimensional contour data, it is easily affected by ground undulations, making tree trunk identification difficult.

[0005] (2) Multiline LiDAR: For multiline LiDAR point clouds, the conventional processing flow is to first remove the ground and then use cylindrical fitting or clustering algorithms to extract the tree trunk from non-ground points. Although multiline LiDAR can obtain three-dimensional information of fruit trees, existing methods mostly rely on traditional point cloud processing algorithms such as region of interest (ROI) extraction, ground removal, and clustering, combined with simple rules designed by humans to detect the tree trunk. In complex scenes, the generalization ability is insufficient, it is difficult to adapt to the diversity of tree trunk morphology, and the anti-interference ability of low-hanging canopies and ground facilities is weak.

[0006] Deep learning can automatically learn models from labeled data and has good generalization ability. However, its application in point cloud processing is limited by the sparsity, density unevenness, and high computational complexity of LiDAR data, making it difficult to directly apply to tree trunk instance segmentation tasks. Existing solutions mostly adopt a vision-LiDAR fusion strategy, which combines RGB image tree trunk segmentation based on image deep learning with LiDAR point cloud 3D localization through coordinate registration. However, this type of method suffers from problems such as increased hardware costs, high complexity of data spatiotemporal alignment, and lengthy processing procedures.

[0007] To address this, this study proposes a multi-line LiDAR orchard tree trunk instance segmentation method based on image deep learning. The method utilizes spherical projection technology to transform sparse and disordered multi-line LiDAR point clouds into dense and regular image data, significantly reducing data processing complexity while preserving multi-dimensional information such as 3D coordinates, distance, and intensity. A U-Net architecture is used to mine semantic features of the tree trunks from labeled data, constructing a high-precision tree trunk semantic segmentation model. Traditional point cloud processing algorithms are introduced to advance from semantic segmentation to instance segmentation. This approach, while maintaining computational efficiency, effectively addresses the diverse tree trunk morphologies and complex interference issues such as low-hanging canopies and ground facilities in orchard scenarios, achieving accurate and real-time segmentation of orchard tree trunk instances. Summary of the Invention

[0008] The purpose of this invention is to provide a multi-line LiDAR orchard tree trunk instance segmentation method based on image deep learning, which can achieve accurate instance segmentation of orchard tree trunks, taking into account both processing efficiency and anti-interference ability, so as to adapt to complex orchard scenarios and provide reliable environmental perception data for autonomous navigation of orchard mobile robots.

[0009] The technical solution of this invention is:

[0010] This invention provides a multi-line LiDAR method for segmenting orchard tree trunk instances based on image deep learning, the method comprising the following steps:

[0011] S1. Point cloud acquisition and preprocessing steps: Point cloud data of the orchard environment is acquired through a multi-line LiDAR sensor. The sensor is set up according to a specific coordinate system: the x-axis points to the direction of the mobile robot's movement, the y-axis points to the left side of the mobile robot, and the z-axis is perpendicular to the ground and upward. Regions of interest (ROI) are extracted from the point cloud data. The point cloud of the mobile robot itself that is closer and the sparse point cloud that is farther away are removed to obtain the initial point cloud data.

[0012] S2, Point Cloud Spherical Projection Step: Perform spherical projection on the preprocessed point cloud data to convert it into a 5-channel image containing three-dimensional coordinates, distance, and intensity;

[0013] S3. Training steps for the tree trunk image semantic segmentation model: Create a labeled dataset and train the tree trunk image semantic segmentation model;

[0014] S4. Tree trunk image semantic segmentation steps: Perform channel normalization on the 5-channel image generated in S2; input the normalized image into the trained semantic segmentation model to obtain the tree trunk semantic pixels;

[0015] S5. Image spherical inverse projection step: Based on the correspondence between pixel coordinates (u, v) and three-dimensional coordinates (x, y, z) recorded in S2, map the tree trunk semantic pixels back to the point cloud data to obtain the tree trunk semantic point cloud.

[0016] S6. Tree trunk point cloud instance segmentation steps: Using the density clustering with noise (DBSCAN) algorithm, the tree trunk semantic point cloud is divided into multiple tree trunk clusters based on the horizontal coordinates (x, y) to obtain the tree trunk instance point cloud.

[0017] Furthermore, S2 specifically refers to:

[0018] S21. Spherical coordinate transformation: Convert the three-dimensional coordinates (x, y, z) of each point in the point cloud to spherical coordinates. Where r is the distance from the point to the origin, and θ is the azimuth angle. For pitch angle:

[0019]

[0020] S22, Image coordinate mapping: Based on the spherical coordinates of the point cloud Calculate the corresponding image pixel coordinates (u, v), where u is the pixel column number and v is the pixel row number. Use the point's 3D coordinates (x, y, z), distance r, and intensity ρ as pixel values ​​to generate dense, regular 5-channel image data.

[0021]

[0022] In the formula, Δθ, These represent the horizontal and vertical angular resolutions of multi-line LiDAR, respectively; θ min , These are the minimum azimuth angle and the minimum elevation angle, respectively. This indicates a floor operation. For pixels in the image without a corresponding point, all channel values ​​are set to 0.

[0023] Furthermore, S3 specifically refers to:

[0024] S31. Dataset creation: Collect more than 1,000 frames of orchard point cloud data and label the tree trunk regions; based on S2, convert the point cloud data into 5-channel images, convert the point cloud labels into mask images, and construct an orchard tree trunk semantic segmentation dataset.

[0025] S32. Dataset Augmentation: Expand the dataset by using methods such as horizontal flipping and random rotation along the z-axis to improve the model's generalization ability;

[0026] S33. Semantic Segmentation Network Construction: Construct a semantic segmentation network based on the U-Net architecture, including an encoder and a decoder. The encoder extracts deep semantic features of the image through convolutional layers and downsampling operations. The decoder fuses shallow details and deep semantic features through upsampling and skip connections, outputting the trunk semantic segmentation result. Given the high vertical resolution of multi-line LiDAR, to preserve vertical information, the encoder only performs downsampling in the horizontal direction, with L downsampling layers.down The tree structure is adaptively determined based on the characteristics of multi-line LiDAR data and the trunk geometry, avoiding the loss of trunk features. The maximum horizontal resolution of multi-line LiDAR is calculated as follows:

[0027] Δ rhorz，max =r max ·Δθ

[0028] In the formula, r max The furthest distance to the ROI. Calculate the minimum horizontal pixel size of the tree trunk:

[0029]

[0030] In the formula, DBH min The minimum diameter at breast height (DBH) of the tree trunk in the orchard is defined. The horizontal downsampling rate is set to 2. To ensure the tree trunk occupies at least one horizontal pixel in the feature map, the number of downsampling layers is:

[0031]

[0032] S34. Semantic Segmentation Network Training: Calculate the mean and standard deviation of each channel in the dataset, and perform channel normalization on the images (converting them to a standard normal distribution with a mean of 0 and a standard deviation of 1) to accelerate the training process. Train the constructed U-Net network using the dataset, and optimize the network parameters using the weighted cross-entropy loss function until the network converges, obtaining a high-precision trunk image semantic segmentation model. The formula for the weighted cross-entropy loss is:

[0033]

[0034] In the formula, c is the category index (c=1 for the trunk, c=2 for non-trunk); y c For pixel-based true labels; w is the predicted probability that a pixel belongs to class c. c Set the category weights according to the category ratio:

[0035]

[0036] In the formula, N1 and N2 are the number of tree trunk and non-tree trunk pixels in the dataset, respectively.

[0037] The beneficial effects of this invention are:

[0038] The multi-line LiDAR orchard tree trunk instance segmentation method based on image deep learning of the present invention relies only on multi-line LiDAR sensors, without the need for fusion vision devices, thus reducing hardware costs and avoiding the complex process of data spatiotemporal alignment. By converting point clouds into image data through spherical projection technology, multi-dimensional information is preserved, significantly reducing data processing complexity, improving processing efficiency, and meeting real-time requirements. A U-Net architecture network is constructed to mine semantic features of tree trunks, improving the segmentation model's adaptability and generalization ability to complex interferences such as diverse tree trunk morphology and low-hanging canopies and ground facilities.

[0039] Other features and advantages of the present invention will be described in detail in the following detailed description section. Attached Figure Description

[0040] The above and other objects, features and advantages of the present invention will become more apparent from the more detailed description of exemplary embodiments of the invention in conjunction with the accompanying drawings, wherein the same reference numerals generally represent the same components in the exemplary embodiments of the invention.

[0041] Figure 1 The flowchart of the multi-line LiDAR orchard trunk instance segmentation of the present invention is shown.

[0042] Figure 2 A schematic diagram of the installation of a multi-line LiDAR sensor in an embodiment of the present invention is shown, and the sensor coordinate system is marked in the figure.

[0043] Figure 3 The diagram shows an initial point cloud in an embodiment of the present invention, along with artificial semantic annotations, which include seven categories: tree trunk, tree crown, ground, fence, shrubs, and slope.

[0044] Figure 4 The diagram shows a point cloud spherical projection and its result in an embodiment of the present invention. It includes five channels: three-dimensional coordinates (x, y, z), distance r, and intensity ρ. The manually labeled semantic data is projected into an image, and the image is stretched vertically to make it clear.

[0045] Figure 5 A schematic diagram of the semantic segmentation results of a tree trunk image in an embodiment of the present invention is shown.

[0046] Figure 6 A schematic diagram of the image spherical inverse projection result in an embodiment of the present invention is shown.

[0047] Figure 7 A schematic diagram of the tree trunk point cloud instance segmentation result is shown in an embodiment of the present invention. Detailed Implementation

[0048] Preferred embodiments of the invention will now be described in more detail with reference to the accompanying drawings. While preferred embodiments of the invention are shown in the drawings, it should be understood that the invention can be implemented in various forms and should not be limited to the embodiments set forth herein.

[0049] like Figure 1 As shown, a multi-line LiDAR orchard tree trunk instance segmentation method based on image deep learning is proposed, which includes the following steps:

[0050] S1. Point cloud acquisition and preprocessing steps: Point cloud data of the orchard environment is acquired through a multi-line LiDAR sensor. The sensor is set up according to a specific coordinate system: the x-axis points to the direction of the mobile robot's movement, the y-axis points to the left side of the mobile robot, and the z-axis is perpendicular to the ground and upward. Regions of interest (ROI) are extracted from the point cloud data. The point cloud of the mobile robot itself that is closer and the sparse point cloud that is farther away are removed to obtain the initial point cloud data.

[0051] S2. Point Cloud Spherical Projection Step: Perform spherical projection on the preprocessed point cloud data to convert it into a 5-channel image containing 3D coordinates, distance, and intensity. Specifically:

[0052] S21. Spherical coordinate transformation: Convert the three-dimensional coordinates (x, y, z) of each point in the point cloud to spherical coordinates. Where r is the distance from the point to the origin, and θ is the azimuth angle. For pitch angle:

[0053]

[0054] S22, Image coordinate mapping: Based on the spherical coordinates of the point cloud Calculate the corresponding image pixel coordinates (u, v), where u is the pixel column number and v is the pixel row number. Use the point's 3D coordinates (x, y, z), distance r, and intensity ρ as pixel values ​​to generate dense, regular 5-channel image data:

[0055]

[0056] In the formula, Aθ, These represent the horizontal and vertical angular resolutions of multi-line LiDAR, respectively; θ min , These are the minimum azimuth angle and the minimum elevation angle, respectively. This indicates a floor operation. For pixels in the image without a corresponding point, all channel values ​​are set to 0.

[0057] S3. Training steps for the tree trunk image semantic segmentation model: Create a labeled dataset and train the tree trunk image semantic segmentation model, specifically as follows:

[0058] S31. Dataset creation: Collect more than 1,000 frames of orchard point cloud data and label the tree trunk regions; based on S2, convert the point cloud data into 5-channel images, convert the point cloud labels into mask images, and construct an orchard tree trunk semantic segmentation dataset.

[0059] S32. Dataset Augmentation: Expand the dataset by using methods such as horizontal flipping and random rotation along the z-axis to improve the model's generalization ability;

[0060] S33. Semantic Segmentation Network Construction: Construct a semantic segmentation network based on the U-Net architecture, including an encoder and a decoder. The encoder extracts deep semantic features of the image through convolutional layers and downsampling operations. The decoder fuses shallow details and deep semantic features through upsampling and skip connections, outputting the trunk semantic segmentation result. Given the high vertical resolution of multi-line LiDAR, to preserve vertical information, the encoder only performs downsampling in the horizontal direction, with L downsampling layers. down The tree structure is adaptively determined based on the characteristics of multi-line LiDAR data and the trunk geometry, avoiding the loss of trunk features. The maximum horizontal resolution of multi-line LiDAR is calculated as follows:

[0061] Δ rhorz,max =r max ·Δθ

[0062] In the formula, r max The furthest distance to the ROI. Calculate the minimum horizontal pixel size of the tree trunk:

[0063]

[0064] In the formula, DBH min The minimum diameter at breast height (DBH) of tree trunks in the orchard is defined. The horizontal downsampling rate is set to 2. To ensure the tree trunk occupies at least one horizontal pixel in the feature map, the number of downsampling layers is:

[0065]

[0066] S34. Semantic Segmentation Network Training: Calculate the mean and standard deviation of each channel in the dataset, and perform channel normalization on the images (converting them to a standard normal distribution with a mean of 0 and a standard deviation of 1) to accelerate the training process. Train the constructed U-Net network using the dataset, and optimize the network parameters using the weighted cross-entropy loss function until the network converges, obtaining a high-precision trunk image semantic segmentation model. The formula for the weighted cross-entropy loss is:

[0067]

[0068] In the formula, c is the category index (c=1 for the trunk, c=2 for non-trunk); y c For pixel-based true labels; w is the predicted probability that a pixel belongs to class c.c Set the category weights according to the category ratio:

[0069]

[0070] In the formula, N1 and N2 are the number of tree trunk and non-tree trunk pixels in the dataset, respectively.

[0071] S4. Tree trunk image semantic segmentation steps: Perform channel normalization on the 5-channel image generated in S2; input the normalized image into the trained semantic segmentation model to obtain the tree trunk semantic pixels;

[0072] S5. Image spherical inverse projection step: Based on the correspondence between pixel coordinates (u, v) and three-dimensional coordinates (x, y, z) recorded in S2, map the tree trunk semantic pixels back to the point cloud data to obtain the tree trunk semantic point cloud.

[0073] S6. Tree trunk point cloud instance segmentation steps: Using the noisy density clustering (DBSCAN) algorithm, the tree trunk semantic point cloud is divided into multiple tree trunk clusters based on the horizontal coordinates (x, y) to obtain the tree trunk instance point cloud.

[0074] In practice:

[0075] This invention employs a VLP-16 multi-line LiDAR sensor, whose vertical angle resolution is [not specified]. The horizontal angle resolution is 2°, the maximum measurement distance is 100m, and by setting the motor speed to 600rpm, the corresponding horizontal angle resolution Δθ is 0.2° and the frame period is 100ms; Figure 2 As shown, the sensor is mounted on the top of the mobile robot, with its x-axis pointing in the robot's forward direction, its y-axis pointing to the robot's left side, and its z-axis perpendicular to the ground and upward. After collecting point cloud data in the orchard environment, the robot's own point cloud within 1m of the sensor and the sparse point cloud within 10m are removed to obtain the initial point cloud data (e.g., ...). Figure 3 (As shown); Perform spherical projection on the preprocessed point cloud data, where θ min -90° -15°, converted to a 5-channel image of size 900×16 (e.g.) Figure 4 (As shown); 1000 frames of orchard point cloud data were collected, and the tree trunk regions were labeled using the CloudCompare tool. The point cloud data was converted into 5-channel images, and the point cloud labels were converted into mask images to construct a semantic segmentation dataset. The dataset was then augmented to 5000 images and divided into training, validation, and test sets in an 8:1:1 ratio. A semantic segmentation network based on the U-Net architecture was constructed, given r max =10m, DBH min =0.1m, calculate the number of downsampling layers L down=2, the encoder of this network contains 2 downsampling blocks (each block consists of 2 convolutional layers and 1 max pooling layer), and the decoder contains 2 upsampling blocks (each block consists of 1 upsampling layer and 2 convolutional layers). The encoder only downsamples in the horizontal direction to preserve vertical information. An SGD optimizer (momentum coefficient 0.9, initial learning rate 0.01, learning rate decay rate 0.995) is used for training with a batch size of 1 and a maximum number of epochs of 100 to obtain a tree trunk image semantic segmentation model. After channel normalization of the 5-channel image, it is input into the trained semantic segmentation model to obtain the tree trunk semantic pixels (e.g., ...). Figure 5 As shown), the semantic point cloud of the tree trunk is obtained through spherical inverse projection (e.g. Figure 6 As shown); the DBSCAN algorithm is used to segment the semantic point cloud of the trunk into instances, resulting in the trunk instance point cloud (as shown). Figure 7 (As shown); the average frame time of this method is 11ms, which is less than the frame period. The instance segmentation F1 score is 0.94 when the upper IoU threshold is 0.5 on the test set, which can achieve accurate real-time instance segmentation of orchard tree trunks.

[0076] The various embodiments of the present invention have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments.

Claims

1. A multi-line LiDAR orchard tree trunk instance segmentation method based on image deep learning, characterized in that, The method includes the following steps: S1. Point cloud acquisition and preprocessing steps: Point cloud data of the orchard environment is acquired through a multi-line LiDAR sensor. The sensor is set up according to a specific coordinate system: the x-axis points to the direction of the mobile robot's movement, the y-axis points to the left side of the mobile robot, and the z-axis is perpendicular to the ground and upward. Regions of interest (ROI) are extracted from the point cloud data. The point cloud of the mobile robot itself that is closer and the sparse point cloud that is farther away are removed to obtain the initial point cloud data. S2, Point Cloud Spherical Projection Step: Perform spherical projection on the preprocessed point cloud data to convert it into a 5-channel image containing three-dimensional coordinates, distance, and intensity; S3. Training steps for the tree trunk image semantic segmentation model: Create a labeled dataset and train the tree trunk image semantic segmentation model; S4. Tree trunk image semantic segmentation steps: Perform channel normalization on the 5-channel image generated in S2; input the normalized image into the trained semantic segmentation model to obtain the tree trunk semantic pixels; S5. Image spherical inverse projection step: Based on the correspondence between pixel coordinates (u, v) and three-dimensional coordinates (x, y, z) recorded in S2, map the tree trunk semantic pixels back to the point cloud data to obtain the tree trunk semantic point cloud. S6. Tree trunk point cloud instance segmentation steps: Using the noisy density clustering (DBSCAN) algorithm, the tree trunk semantic point cloud is divided into multiple tree trunk clusters based on the horizontal coordinates (x,y) of the tree trunk semantic point cloud to obtain the tree trunk instance point cloud.

2. The multi-line LiDAR orchard tree trunk instance segmentation method based on image deep learning according to claim 1, characterized in that, The s2 mentioned above specifically refers to: S21. Spherical coordinate transformation: Convert the three-dimensional coordinates (x, y, z) of each point in the point cloud to spherical coordinates. Where r is the distance from the point to the origin, and θ is the azimuth angle. For pitch angle: S22, Image coordinate mapping: Based on the spherical coordinates of the point cloud Calculate the corresponding image pixel coordinates (u, v), where u is the pixel column number and v is the pixel row number. Use the point's 3D coordinates (x, y, z), distance r, and intensity ρ as pixel values ​​to generate dense, regular 5-channel image data. In the formula, Δθ, These represent the horizontal and vertical angular resolutions of multi-line LiDAR, respectively; θ min , These are the minimum azimuth angle and the minimum elevation angle, respectively. This indicates a floor operation. For pixels in the image without a corresponding point, all channel values ​​are set to 0.

3. The multi-line LiDAR orchard tree trunk instance segmentation method based on image deep learning according to claim 1, characterized in that, Specifically, S3 refers to: S31. Dataset creation: Collect more than 1,000 frames of orchard point cloud data and label the tree trunk regions; based on S2, convert the point cloud data into 5-channel images, convert the point cloud labels into mask images, and construct an orchard tree trunk semantic segmentation dataset. S32. Dataset Augmentation: Expand the dataset by using methods such as horizontal flipping and random rotation along the z-axis to improve the model's generalization ability; S33. Semantic Segmentation Network Construction: Construct a semantic segmentation network based on the U-Net architecture, including an encoder and a decoder. The encoder extracts deep semantic features of the image through convolutional layers and downsampling operations. The decoder fuses shallow details and deep semantic features through upsampling and skip connections, outputting the trunk semantic segmentation result. Given the high vertical resolution of multi-line LiDAR, to preserve vertical information, the encoder only performs downsampling in the horizontal direction, with L downsampling layers. down The tree structure is adaptively determined based on the characteristics of multi-line LiDAR data and the trunk geometry, avoiding the loss of trunk features. The maximum horizontal resolution of multi-line LiDAR is calculated as follows: Δr horz,max =r max ·Dth In the formula, r max The furthest distance to the ROI. Calculate the minimum horizontal pixel size of the tree trunk: In the formula, DBH min The minimum diameter at breast height (DBH) of tree trunks in the orchard is defined. The horizontal downsampling rate is set to 2. To ensure the tree trunk occupies at least one horizontal pixel in the feature map, the number of downsampling layers is: S34. Semantic Segmentation Network Training: Calculate the mean and standard deviation of each channel in the dataset, and perform channel normalization on the images (converting them to a standard normal distribution with a mean of 0 and a standard deviation of 1) to accelerate the training process. Train the constructed U-Net network using the dataset, and optimize the network parameters using the weighted cross-entropy loss function until the network converges, obtaining a high-precision trunk image semantic segmentation model. The formula for the weighted cross-entropy loss is: In the formula, c is the category index (c=1 for the trunk, c=2 for non-trunk); y c For pixel-based true labels; w is the predicted probability that a pixel belongs to class c. c Set the category weights according to the category ratio: In the formula, N1 and N2 are the number of tree trunk and non-tree trunk pixels in the dataset, respectively.

Images