An Autonomous Navigation Method and System for Mulberry Orchards Based on LiDAR Point Clouds

Abstract

This invention relates to the field of automatic navigation technology, specifically to an autonomous navigation method and system for mulberry orchards based on lidar point clouds. The method includes: preprocessing the point cloud set; performing semantic segmentation using a semantic segmentation network model to obtain point cloud features corresponding to the point cloud set, including point cloud labels; the point cloud labels include a first label associated with a path region and a second label associated with a crop region; acquiring crop regions adjacent to the path region, and determining whether the crop region is continuous or discrete based on the second label; if the crop region is discrete, generating candidate boxes for the point cloud set of the crop region based on the point cloud features, and selecting points of interest from the candidate boxes; if the crop region is continuous, acquiring the boundary point cloud of the crop region as the points of interest; setting a first threshold, and shifting multiple points of interest towards the path region by the first threshold to obtain multiple navigation points; achieving adaptive navigation line extraction for the mulberry orchard scene, improving the reliability of autonomous navigation for agricultural machinery in the mulberry orchard.

Description

Technical Field

[0001] This invention relates to the field of automatic navigation technology, specifically to an autonomous navigation method and system for mulberry orchards based on lidar point clouds. Background Technology

[0002] With the development of agricultural automation and intelligence, lidar technology is increasingly being applied to the autonomous navigation of agricultural machinery. However, the mulberry orchard environment has certain unique characteristics compared to structured traffic scenarios. After summer pruning, mulberry trees are bare, leaving only trunks, at which point the features of individual trunks are clearly defined. In contrast, in the unpruned state of the mulberry orchard, the leaves are lush and dense, resulting in a large amount of point cloud data and indistinct boundaries between trunks. Therefore, selecting an appropriate navigation line extraction method based on the different states of the mulberry orchard can more accurately reflect its actual structure and improve the reliability and efficiency of autonomous navigation.

[0003] Traditional methods of visually extracting feature information such as crop rows and farm roads are inefficient and prone to human error. Existing point cloud semantic segmentation models, such as PointNet++, are mostly used for traffic scenarios and lack applications in agricultural mulberry orchard scenarios. The path navigation method disclosed in application CN201880001094.0 determines first-path obstacle information based on road surface detection from the acquired depth map, combines it with second-path obstacle information determined by image learning from the depth map, and then determines third-path obstacle information that better matches the actual road conditions. Navigation is then performed based on this third-path obstacle information, resulting in more accurate navigation.

[0004] However, depth maps have relatively lower richness and accuracy in three-dimensional spatial information compared to lidar point cloud data. In mulberry orchards, with their complex and dense mulberry trees and irregular terrain, relying solely on depth maps for navigation presents challenges in accurately perceiving and understanding the orchard environment. Therefore, this application provides an autonomous navigation method and system for mulberry orchards based on lidar point clouds, which improves the reliability and efficiency of autonomous navigation for agricultural machinery in mulberry orchards, thereby enhancing the efficiency and safety of mulberry orchard operations. Summary of the Invention

[0005] The purpose of this invention is to address the technical problem that the navigation process in the prior art is difficult to accurately perceive and understand the mulberry orchard environment. Through the embodiments of this application, an autonomous navigation method and system for mulberry orchards based on lidar point clouds is provided. This method enables the identification of mulberry orchard scenes with unique growth cycle changes and obvious environmental characteristics. By using an adaptive navigation line extraction method, the reliability and efficiency of autonomous navigation of agricultural machinery in mulberry orchards are improved, damage to mulberry trees is avoided, and the efficiency and safety of mulberry orchard operations are enhanced.

[0006] To achieve the above-mentioned objectives, the present invention provides the following technical solution:

[0007] An autonomous navigation method for mulberry orchards based on lidar point clouds includes:

[0008] A point cloud is collected, and the preprocessed point cloud is semantically segmented using a semantic segmentation network model to obtain point cloud features corresponding to the point cloud, including point cloud labels; the point cloud labels include a first label for associated path regions and a second label for associated crop regions.

[0009] Obtain the crop region adjacent to the path region, and determine whether the corresponding crop region is a continuous space or a discrete space based on the second label;

[0010] If the crop region is a discrete space, candidate boxes are generated for the point cloud set of the crop region based on the point cloud features, and points of interest are selected from the candidate boxes; if the crop region is a continuous space, the boundary point cloud of the crop region is obtained as the points of interest; a first threshold is preset, and multiple points of interest are offset towards the path region by the first threshold to obtain multiple navigation points.

[0011] As a preferred technical solution of this application, the step of generating candidate boxes from the point cloud set of the crop region based on the point cloud features, and selecting points of interest from the candidate boxes, includes:

[0012] Obtain the points of interest (POIs) in the candidate box that represent the center position; delete the POIs whose distance from the moving carrier is greater than or equal to the first threshold.

[0013] Multiple points of interest (POIs) on both sides of the moving carrier are acquired along the moving direction, and indexes are set for the POIs according to the moving direction; the distance between the POIs on both sides is calculated, and the POIs on both sides with the smallest distance are recorded as adjacent points; the adjacent points with index differences are acquired, and interpolation is performed on the adjacent points along the moving direction; the POIs on both sides and the interpolation points are connected to obtain reference lines.

[0014] Obtain the perpendicular foot of the reference line on the other side from the point of interest on one side, calculate the midpoint between the point of interest and the perpendicular foot, and obtain the navigation line by fitting multiple midpoints.

[0015] As a preferred technical solution of this application, obtaining the boundary point cloud of the crop region as the point of interest includes:

[0016] The point cloud with the smallest distance from the moving carrier that is greater than or equal to the first threshold is obtained as the point of interest; adjacent points of interest are sequentially connected along the moving direction to obtain a reference line;

[0017] The reference lines on both sides of the moving carrier are obtained along the direction of movement. Multiple interval points are set on the reference lines at a preset first interval. The midpoint of the line connecting the two adjacent interval points is calculated. The navigation line is obtained by fitting multiple midpoints.

[0018] As a preferred technical solution of this application, the preprocessing includes: using the statistical filtering algorithm of the PCL point cloud library to remove outliers from the point cloud set, and using the pass-through filter algorithm of the PCL point cloud library to retain the point cloud within a preset area.

[0019] As a preferred technical solution of this application, the semantic segmentation network model includes four SA modules for extracting the point cloud features, four upsampling layers for restoring the number of point cloud features to the size of the point cloud set, and a classification fully connected layer for classifying the point cloud labels.

[0020] The SA module includes a sampling layer, a grouping layer, and a feature layer for randomly sampling point clouds; the grouping layer pre-sets multiple different grouping radii to group the point cloud set; the feature layer uses 3 sets of dynamic edge convolutions to extract the features of the point cloud.

[0021] As a preferred technical solution of this application, the semantic segmentation network model extracts features from the point cloud corresponding to different grouping radii through the SA module to obtain grouping features; and concatenates multiple grouping features to obtain local features;

[0022] The local features are used to extract edge features of adjacent point clouds through the dynamic edge convolution, and the local features and the edge features are fused to obtain point cloud features; the point cloud features are upsampled to restore the number of points in the point cloud to the original number; the point cloud is classified through a fully connected layer.

[0023] As a preferred technical solution of this application, the step of extracting edge features of adjacent point clouds by means of the local features through the dynamic edge convolution includes:

[0024] The feature corresponding to the center point is deleted from the grouped features to obtain new grouped features; multiple new grouped features are concatenated to obtain the local features;

[0025] A weight score matrix is ​​generated based on the coordinates of the point cloud; the local features are multiplied by a preset fully connected layer to obtain a weight parameter matrix; the local features are multiplied by the weight parameter matrix to obtain a feature matrix; the score matrix is ​​multiplied by the feature matrix to associate the coordinates of the point cloud with the weight parameter matrix.

[0026] As a preferred technical solution of this application, restoring the number of point cloud features to the size of the point cloud set includes:

[0027] The point cloud features of the previous upsampling layer are obtained as the interpolation points, and the point cloud features of the next upsampling layer are obtained as the reference points.

[0028] Traverse the points to be interpolated, select multiple reference points with the smallest distance to the points to be interpolated, and calculate the sum of the reciprocals of these multiple distances; use the ratio of the reciprocal of the distance to the sum of the reciprocals as the weight of the reference point; calculate the product of the reference point and the weight, and use the average of the products corresponding to the multiple reference points as the new point cloud feature of the point to be interpolated; concatenate the features before and after interpolation to obtain the point cloud feature with increased dimension.

[0029] As a preferred technical solution of this application, the semantic segmentation network model adopts the transfer learning method. It is first trained in multiple rounds using a public dataset, and then validated using a public dataset. The average intersection-union ratio of multiple categories in the public dataset is calculated, and the semantic segmentation network model corresponding to the largest average intersection-union ratio is selected.

[0030] A first dataset for the mulberry garden scene is established. The semantic segmentation network model is trained multiple times using the training set. The average intersection-union ratio (IUR) of multiple categories in the first dataset is calculated. The semantic segmentation network model with the largest IUR is selected as the final semantic segmentation network model.

[0031] This invention provides an autonomous navigation system for mulberry orchards based on lidar point clouds, comprising:

[0032] The semantic segmentation module is used to collect point clouds and perform semantic segmentation on the preprocessed point clouds using a semantic segmentation network model to obtain point cloud features corresponding to the point clouds, including point cloud labels; the point cloud labels include a first label for associated path regions and a second label for associated crop regions.

[0033] The region identification module is used to obtain the crop region adjacent to the path region, and determine whether the corresponding crop region is a continuous space or a discrete space based on the second label;

[0034] The path generation module, if the crop region is a discrete space, is used to generate candidate boxes for the point cloud set of the crop region based on the point cloud features, and select points of interest from the candidate boxes; if the crop region is a continuous space, it is used to obtain the boundary point cloud of the crop region as the points of interest; a first threshold is preset, and multiple points of interest are offset towards the path region by the first threshold to obtain multiple navigation points, so that the mobile carrier moves along the multiple navigation points.

[0035] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0036] 1. This invention projects point clouds onto a polar coordinate system to form a bird's-eye view. Utilizing an improved semantic segmentation model tailored to mulberry orchard scenarios, semantic segmentation is performed based on the polar coordinate bird's-eye view. The segmentation results are then mapped back to a Cartesian coordinate system, effectively solving the problem of uneven point cloud label distribution caused by uneven point cloud distribution in mulberry orchard scenarios. This improves the accuracy and real-time performance of semantic segmentation. In the special scenario of a mulberry orchard, the point cloud representation in polar coordinates better adapts to the significant differences between dense and sparse mulberry tree areas. This allows the semantic segmentation model provided by this invention to more accurately extract features of semantic categories such as mulberry trees and the ground, providing a more reliable data foundation for subsequent autonomous navigation perception and significantly improving the reliability and efficiency of autonomous navigation.

[0037] 2. The BEV_Mnet backbone network employs four consecutive SA modules to extract point cloud features. A multi-scale grouping method is used; after sampling from the random farthest point, multiple group radii are set, and features are extracted separately for the point cloud sets within each group. Finally, the point cloud features extracted under different group radii are concatenated to obtain the point cloud features output by each SA module. Each SA module contains a fully connected MLP layer composed of three sets of DEdgeConv convolutions. This invention extracts point cloud features through dynamic convolution DEdgeConv, the dynamic process being determined by the position of the point cloud. A weight score matrix is ​​generated to adjust the convolution kernel parameters, and the output result is obtained through weighted operations. This is beneficial for more efficient processing of irregularly distributed and disordered point cloud data.

[0038] 3. The autonomous navigation method provided by this invention can select an appropriate navigation line extraction method based on different states of the mulberry orchard, including summer pruning and no pruning. After summer pruning, the semantic segmentation network model BEV_Mnet can identify the mulberry tree trunks on both sides of the farm road, generate candidate boxes for each trunk, and then extract the navigation line. In the case of no pruning, the navigation line can be extracted based on the boundaries of the left and right tree rows. This adaptive selection of the navigation line extraction method based on different states can more accurately reflect the actual structure of the mulberry orchard, improving the adaptability and reliability of autonomous navigation. This method can perceive the environmental information of the mulberry orchard in real time and update the navigation line in real time according to environmental changes, enabling the autonomous navigation system to better adapt to the complex environmental changes in the mulberry orchard scenario, effectively reducing mulberry tree damage, improving the efficiency and safety of mulberry orchard operations, and facilitating the realization of automated mulberry orchard operations. Attached Figure Description

[0039] Figure 1 This is a schematic diagram of the mulberry orchard autonomous navigation system provided in this application;

[0040] Figure 2A flowchart illustrating the autonomous navigation method for mulberry orchards provided in this application;

[0041] Figure 3 This is a schematic diagram of the structure and semantic segmentation of the BEV_Mnet model;

[0042] Figure 4 This is a schematic diagram of the backbone network structure in the BEV_Mnet model;

[0043] Figure 5 A flowchart illustrating the process of performing dynamic edge convolutions on the BEV_Mnet model;

[0044] Figure 6 A flowchart illustrating the process of generating dynamic convolutional kernels;

[0045] Figure 7 A flowchart illustrating the process of generating candidate bounding boxes for crop regions;

[0046] Figure 8 A flowchart illustrating the training and application process of the BEV_Mnet model;

[0047] Figure 9 A schematic diagram of autonomous navigation in a mulberry orchard scene after summer pruning;

[0048] Figure 10 This is a schematic diagram of autonomous navigation in a mulberry orchard when it is not pruned. Detailed Implementation

[0049] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0050] Therefore, the following detailed description of embodiments of the present invention is not intended to limit the scope of the claimed invention, but merely illustrates some embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0051] It should be noted that, unless otherwise specified, the embodiments and features and technical solutions in the present invention can be combined with each other.

[0052] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0053] In the description of this invention, it should be noted that the terms "upper," "lower," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, or the orientation or positional relationship commonly used when the product of this invention is in use, or the orientation or positional relationship commonly understood by those skilled in the art. These terms are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this invention. In addition, the terms "first," "second," etc., are only used to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0054] For example Figure 1 The image shows an autonomous navigation system for mulberry orchards based on lidar point clouds provided in this embodiment, including:

[0055] The semantic segmentation module is used to collect point clouds and perform semantic segmentation on the preprocessed point clouds using a semantic segmentation network model to obtain point cloud features corresponding to the point clouds, including point cloud labels; the point cloud labels include a first label for associated path regions and a second label for associated crop regions.

[0056] The region identification module is used to obtain the crop region adjacent to the path region, and determine whether the corresponding crop region is a continuous space or a discrete space based on the second label;

[0057] The path generation module, if the crop region is a discrete space, is used to generate candidate boxes for the point cloud set of the crop region based on the point cloud features, and select points of interest from the candidate boxes; if the crop region is a continuous space, it is used to obtain the boundary point cloud of the crop region as the points of interest; a first threshold is preset, and multiple points of interest are offset towards the path region by the first threshold to obtain multiple navigation points, so that the mobile carrier moves along the multiple navigation points.

[0058] The mulberry orchard autonomous navigation system is used to execute the mulberry orchard autonomous navigation method based on lidar point clouds provided in this application, for example... Figure 2 As shown, it includes:

[0059] A mobile vehicle-mounted 3D LiDAR scanner was used to scan the mulberry orchard environment, collecting 3D LiDAR point clouds within the orchard scene to form point cloud sets. Mulberry orchards are mostly distributed in hilly and mountainous areas, with small and densely packed plots and varying elevations. The collected point cloud sets include information on mulberry trees, crop rows, ground, and slopes, resulting in relatively dense point cloud data. Since the point cloud sets are widely distributed throughout the measurement range and contain offset points, preprocessing of the point cloud sets is necessary.

[0060] Because the data packet encapsulated by the LiDAR includes the horizontal rotation angle... Vertical angle of laser beam And distance d. To present the effect of a 3D point cloud map, the angles and distances in 3D space are converted into x, y, z coordinates in a Cartesian coordinate system, centered on the lidar. A Cartesian coordinate system is established with the lidar center as the origin, and a rectangular coordinate system is established with the direction of movement of the lidar's mobile carrier as the positive x-axis, according to the right-hand rule. The lidar rotates 360 degrees clockwise, with the positive x-axis as 0 degrees. The conversion formula is shown below:

[0061] ;

[0062] Where d is the distance. The vertical angle of the laser beam. Let R be the horizontal rotation angle of the laser beam, and R be the radius of the plane from the laser point to the origin. The lidar data packet also provides reflection intensity information to characterize the reflection intensity of the measured object, ranging from [1-255]. In this embodiment, the obtained point cloud is P={x i y i , z i r i}, where (x i y i , z i ) represents the Cartesian coordinates of the point cloud data, r i It is the intensity of reflection.

[0063] For the point cloud set of the 3D LiDAR point cloud in the mulberry garden environment, the statistical filtering algorithm of the PCL point cloud library is used to remove outliers from the point cloud set, and the pass-through filter algorithm of the PCL point cloud library is used to retain the point cloud data within a preset area. The statistical filtering algorithm involves selecting multiple other point clouds that are closest to the reference point cloud in multiple directions around the reference point cloud, calculating the mean, standard deviation, and variance of the first distance between the other point clouds and the reference point cloud, calculating the difference between the first distance and the mean, and deleting outliers whose difference is greater than the product of the standard deviation and variance.

[0064] For agricultural machinery operations in mulberry orchards, the area of ​​greatest interest is usually in front of the machine. The original point cloud contains a 360° range of points, resulting in a large data volume. The direct-pass filtering algorithm sets threshold ranges for the coordinate axes (x, y, z), defining minimum and maximum values ​​for each axis to obtain a preset region. It then iterates through each point cloud in the set, retaining those whose coordinate values ​​fall within the preset region.

[0065] By using the statistical filtering algorithm of the PCL point cloud library to remove outliers and the pass-through filter algorithm of the PCL point cloud library to retain points within a preset area, the quality of point cloud data can be improved before semantic segmentation of the point cloud set.

[0066] Each point in the Cartesian coordinate system point cloud is projected onto a plane containing the x and y axes from a bird's-eye view perspective, forming a two-dimensional planar bird's-eye view. This two-dimensional planar bird's-eye view can be used to extract point cloud features from the point cloud set using a two-dimensional convolutional network, achieving a balance between computational cost and accuracy. Then, a polar coordinate system is established on the two-dimensional planar bird's-eye view, centered on the origin of the LiDAR coordinate system on the x and y axes plane, and offset according to a preset angle. Divide the polar coordinate system into segments with step size 1 and step size 2. Simultaneously, in the radial direction centered on the origin of the coordinate system, according to the preset offset radius... Divide the point cloud into multiple concentric circles outwards. Different angles and different concentric circles divide the point cloud in the polar coordinate system into multiple sector grids. Polar coordinates can balance the distribution of point clouds in different ranges, making semantic segmentation network models for semantic segmentation of point cloud sets more likely to learn and identify features near the LiDAR, thus minimizing information loss caused by feature extraction. The LiDAR point cloud set P = {x} i y i , z i r i} Convert to polar coordinates and distance The calculation formula is as follows:

[0067] ;

[0068] Represent the point cloud as Where K is the number of point cloud coordinate elements, i.e., the dimension of the input feature, set to 4; N is the number of points in the point cloud set. Based on the polar coordinates of the point cloud... The point cloud sets are grouped into different sector grids, and the point cloud set in each sector grid is represented as follows: H and W are the dimensions of the two-dimensional bird's-eye view. It represents the number of point clouds in each sector grid.

[0069] The grouped point cloud sets are input into a semantic segmentation network model for semantic segmentation to obtain the point cloud features corresponding to the point cloud sets; the point cloud features include the point cloud labels and point cloud polar coordinates of the point cloud sets. In this embodiment, the semantic segmentation network model is constructed as a BEV_Mnet model, including: a backbone network formed by 4 SA (set abstraction) modules for feature extraction and 4 upsampling layers for restoring the number of features.

[0070] Based on the semantic segmentation network model and the resulting point cloud features, corresponding point clouds are extracted from the point cloud set using pre-defined point cloud labels and coordinates. The polar coordinates of the point clouds are then mapped back to Cartesian coordinates. For each point cloud... Obtain the corresponding polar coordinates The point cloud of the sector is labeled and its Cartesian coordinates are combined with the label to form a new point cloud feature. This feature is used by the bin candidate box generation method to generate candidate boxes for the crop region based on the second label and its corresponding coordinates. In this embodiment, the preset point cloud labels include a first label representing the path, a second label representing the crop, and a third label (label value 0) representing background information. The point cloud set corresponding to the first label is the path region, including the ground (label value 40), and the point cloud set corresponding to the second label is the crop region, including the left tree row (label value 48) and the right tree row (label value 51). The path region and the crop region are adjacent.

[0071] The invention selects appropriate navigation line extraction methods based on the different states of the mulberry orchard. Modern mulberry orchards, to promote early leaf maturity and increase spring leaf yield, reserve 10%-15% of the mulberry trees for summer pruning during winter management. The invention provided in this application considers the main agricultural machinery operations in mulberry orchards as "management" and "harvesting," at which point the orchard's state is mainly divided into two states: summer pruning and no pruning. After summer pruning, only the trunk of the mulberry tree is retained at a height of no more than 25cm, with no leaves at all. The corresponding extracted features are discrete point clouds, forming multiple discrete crop areas. Without pruning, the mulberry leaves can grow to 50-150cm high, with abundant and dense foliage, almost obscuring the trunk. The corresponding extracted features are continuously distributed point clouds, forming a single continuous crop area. This invention determines whether the point cloud features extracted from the crop area are continuous or discrete based on whether summer pruning has been performed, and selects different navigation line extraction strategies accordingly. Select points of interest from the candidate box; preset a first threshold, and after multiple points of interest are offset towards the path area by the first threshold, multiple navigation points are obtained. The mobile vehicle moves along the multiple navigation points or the navigation line formed by connecting multiple navigation points, and the calculated navigation line is fed back to the autonomous navigation system in real time. Combined with the vehicle's dynamic model and sensor data, autonomous navigation in the mulberry garden scene is realized.

[0072] In one implementation, for example Figure 9As shown, when the mulberry orchard is under summer pruning, the mulberry trees have no leaves, and only the trunks serve as the crop area, resulting in low point cloud density. A semantic segmentation network model is used to identify the mulberry tree trunks on both sides of the farm road, generating candidate boxes for the trunks. Navigation lines are extracted based on the midpoint of the line connecting the center points of the candidate boxes on both sides. It is important to note that the first threshold under summer pruning is used to retain the center position of the mulberry tree trunks on both sides of the current farm road. The first threshold can be the coordinate axis distance between the LiDAR origin and the point cloud corresponding to the left boundary of the left row of trees, or the point cloud corresponding to the right boundary of the right row of trees; it can also be the distance on the coordinate axis corresponding to half the width of the farm road and the sum of the widths of the tree rows. In this embodiment, the first threshold is set to a coordinate axis unit length of 1, that is, by transforming the coordinate axis distance represented by the first threshold into a unit length, the computational load is reduced. Specifically, the geometric center point of the generated candidate box is obtained as the point of interest, the center position of each mulberry tree trunk is extracted, and the center points of mulberry tree trunks that deviate from the current farm road side are removed. The method for removing off-center points includes: using the moving vehicle as the origin, for the center points corresponding to multiple tree trunks on the left, deleting points with a y-axis coordinate greater than 1; for the center points corresponding to the tree trunks on the right, deleting points with a y-axis coordinate less than -1; assigning index values ​​to the center points of the left and right tree trunks sequentially, starting from 0, according to the direction of movement of the moving vehicle. Calculating the distance between the left and right center points, and recording the center points with the smallest distance between them as adjacent points. Determining whether the index values ​​of adjacent points are the same; if there is a difference in the indices, adding compensation center points at 0.1m intervals along the direction of movement for the center points on the left and right, making the line connecting the center points on the left and right sides along the direction of movement smoother; and sequentially connecting the center points and compensation center points to obtain the reference line.

[0073] The number of left and right indices is counted. Since a larger number of navigation points results in a more accurate and smoother navigation line for the mobile vehicle, a perpendicular line is drawn from the center point of the reference line on the side with more indices to the center point of the reference line on the side with fewer indices, obtaining the perpendicular foot point. The midpoint between the center point of the reference line on the side with more indices and the perpendicular foot point is used as the navigation point. For example, if the number of right indices is greater than that on the left, for each center point on the right, its perpendicular foot point on the left reference line is obtained, and the x-axis coordinate of the perpendicular foot point is within the x-coordinate range of the current point set; the midpoint between the right center point and the perpendicular foot point is calculated. Multiple midpoints are used as navigation points and stored. Interpolation is performed between navigation points at intervals of 0.05m to add compensation navigation points. The navigation points and compensation navigation points are then connected sequentially to fit the navigation line.

[0074] In another embodiment, for example Figure 10As shown, when the mulberry orchard is unpruned, the mulberry leaves are numerous and dense, making it difficult to segment individual tree trunks, and the semantic segmentation network model struggles to generate candidate boxes for each tree trunk. In this case, the semantic segmentation network model identifies the tree rows on the left and right sides of the farm road, using the tree rows as crop areas, and extracts navigation lines based on the boundaries of the tree rows. It is important to note that the first threshold in the unpruned state is used to reduce the selection of noisy point clouds near the lidar as boundary point clouds. The first threshold can be the coordinate axis distance between the lidar origin and the point cloud corresponding to the right boundary of the left tree row, or the point cloud corresponding to the left boundary of the right tree row; it can also be the distance corresponding to half the width of the farm road on the coordinate axis. In this embodiment, the first threshold is set to 1 unit length on the coordinate axis, that is, by transforming the coordinate axis distance represented by the first threshold into a unit length, the computational load is reduced. Specifically, the boundary point clouds of the left and right tree rows are extracted from the point cloud features. Since some mulberry trees may have been dug up after disease or death, the point clouds within the same mulberry row may be discontinuous. Therefore, the point clouds on the left and right sides are segmented every 5m along the direction of movement, and index values ​​are labeled sequentially according to the movement direction, starting from 0. After segmenting the left point cloud, the boundary points with the smallest y-axis coordinate and a y-axis coordinate greater than 1 in each segment are extracted as points of interest (POIs). The boundary points of each segment are connected in order of index value to form the left reference line. After segmenting the right point cloud, the boundary points with the largest y-axis coordinate and a y-axis coordinate less than -1 in each segment are extracted as POIs. The boundary points of each segment are connected in order of index value to form the right reference line. Based on the POIs on the left and right sides, fitting is performed along the direction of movement, with interpolation in a step size of 1m. Compensation boundary points are added, and the POIs and compensation boundary points are connected sequentially to obtain the reference lines on the left and right sides. For the reference lines on the left and right sides, interval points are set at 1m intervals. The midpoint of the Euclidean distance between adjacent interval points on the two boundary lines is calculated. Multiple midpoints are used as navigation points, and the navigation lines are fitted sequentially after being connected. By segmenting the tree rows and setting an index value for each segment, the algorithm for calculating navigation points is executed on long-distance mulberry tree rows. The navigation points are then connected in index order, reducing navigation errors caused by discontinuity in mulberry tree rows.

[0075] The semantic segmentation network model performs semantic segmentation on point clouds, extracts point cloud features, and predicts the corresponding point cloud labels, such as "road," "building," "left tree row," and "right tree row," which are represented as numerical values. By transforming the polar coordinate system back to the Cartesian coordinate system, the 2D semantic segmentation prediction result is reshaped back into 3D voxels to separate labels at different heights along the z-axis, resulting in the final semantic segmentation result, where Z represents the number of voxel layers along the z-axis.

[0076] In one embodiment, the semantic segmentation network model is a deep learning-based semantic segmentation model, BEV_Mnet, comprising: a point cloud feature extraction network and a candidate box generation network, wherein the point cloud feature extraction network includes a backbone network and a segmentation network, for example... Figure 3 As shown. Specifically, for example... Figure 4 As shown, the backbone network includes four consecutive Set Abstraction (SA) modules for feature encoding, used to extract multi-scale and high-dimensional point cloud features from the point cloud set. The SA module includes a sampling layer for sampling the point cloud, a grouping layer for grouping the point cloud set according to multiple different preset grouping radii in different SA modules, and a feature layer for extracting point cloud features. Specifically, the SA module uses farthest point sampling to randomly sample n points from the original point cloud, with the input point cloud dimension being... , where C in The number of input channels, i.e., the number of point cloud coordinate elements, is set to 4 in this embodiment; S points are selected within a sphere of a specified radius, centered on the sampling point, for grouping. After grouping, the point cloud dimension is... Because only a single grouping radius scale and a fixed number of sampling points are used, the grouped point clouds are affected by the point cloud density; if the point cloud is too sparse, it is difficult to group with a small radius size; subsequent SA layers are based on the previous layer, and if the first SA layer does not extract features, the subsequent layer cannot extract features from the corresponding sparse point cloud. Therefore, this invention uses the Multi-Scale Grouping (MSG) method for point cloud feature extraction. After sampling the farthest point at random, multiple grouping radii are set, and then feature extraction is performed on the point cloud set within each group. Finally, the point cloud features extracted under different grouping radii are concatenated to obtain the point cloud features output by each SA module.

[0077] To better handle irregular and unordered point cloud data, such as Figure 5 As shown, the BEV_Mnet model of this invention uses Dynamic Edge Convolution (DEdgeConv) to extract point cloud features. It not only considers the features of the points themselves, but also integrates the relative features of the surrounding point clouds, further focusing on the local features of the point clouds, making the point cloud features richer and more effective.

[0078] Specifically, the feature extraction part of each SA module in the BEV_Mnet backbone network includes a fully connected convolutional layer (MLP) consisting of three sets of DEdgeConv convolutions. First, after multi-scale grouping, edge convolutions are used to subtract the group center point (i.e., sampling point) features from the point cloud features within each group, obtaining the relative features of the point cloud centers, with dimensions of [dimensional value missing]. ,in, Let S represent the dimension of the input point cloud and S represent the number of sampling points in the group. Then, feature concatenation is performed by concatenating the original point cloud features from the input SA module with the center-relative features to obtain... Dimensional point cloud features.

[0079] Furthermore, the dynamic convolution process of DEdgeConv for extracting point cloud features includes: generating a weight score matrix that adjusts the convolution kernel parameters based on the position of the point cloud, and obtaining the output result through weighted operations. Specifically, for example... Figure 6 As shown, a weight library is defined. Each B q It is a weight parameter matrix. In the ScoreNet network, the convolutional kernel weight score matrix is ​​generated based on the point cloud coordinate features, and its output dimension is... Q represents the number of convolutional kernel weight parameters. The input to the ScoreNet network is the center point p of the point cloud. i and its neighboring point p j The positional relationships between the inputs are nonlinearly transformed by a multilayer perceptron to obtain an intermediate feature representation. To ensure a reasonable distribution of scores, the ScoreNet network output is normalized using a Softmax function to obtain a normalized weight-score matrix. ,in B represents the q-th weight parameter matrix. q The score in the i-th row and j-th column when constructing the convolution kernel. Weight-score matrix M. ij Each element represents the weight parameter matrix B. q Constructing convolution kernels The weights at different times. The weight score matrix M generated based on the ScoreNet network. ij The convolution kernel is dynamically assembled using the following formula:

[0080] ;

[0081] From the weight parameter matrix B q The desired convolution kernel parameters are selected using weighted selection, and the weight parameter matrix B q The dimension is The stitched point cloud features are then combined with the weight parameter matrix B. q Multiplication yields the dimension as The feature matrix, and the weight score matrix are multiplied by the feature matrix to obtain the weighted result, which has a dimension of . The weighting operation is performed along the Q-dimensional plane, which is equivalent to merging all parameters along the Q-dimensional plane into a single parameter of the convolution kernel. By multiplying the weight score matrix with the feature matrix and then summing the results along the Q-dimensional plane, the convolution kernel parameters are correlated with the point cloud coordinates. Point cloud sets with different group radii thus obtain different feature extraction methods.

[0082] Since max pooling is only related to the maximum value and is independent of the order, it conforms to the unordered characteristics of point clouds. Max pooling is performed after each DEdgeConv dynamic convolution. The point cloud features of dimension 1 are output after max pooling. The backbone network extracts deep features through four consecutive SA (Saint-Signal) modules, outputting a list of point cloud features (sa_features), a list of sampled point cloud coordinates, and a list of sampled point indices.

[0083] Specifically, the segmentation network's task is to classify each point cloud. Since the convolutional operations of consecutive SA modules reduce the number of points in the point cloud, and the segmentation network needs to classify the original point cloud, it needs to upsample the point cloud features extracted by the SA modules to restore the number of points in the feature set to its original value. Loss calculation and model training iterations are then performed. The BEV_Mnet model's segmentation network uses four feature upsampling layers. Each layer uses interpolation to restore the number of points in the point cloud, and the interpolation depends on the features of the preceding and following layers. The interpolation process includes: using M1 points from the previous layer as the points to be interpolated and M2 points from the following layer as reference points, calculating the distances between these M1 points and the M2 reference points. Here, the number of points in the previous layer is M1, and the feature dimension is D1; ​​the number of points in the following layer is M2, and the feature dimension is D2. Then, k=3 of the closest reference points are selected from the M1 points to be interpolated, and the weighted average of the features of these k points is used as the new feature of the point to be interpolated, with a dimension of D2. Each point to be interpolated receives a new feature derived from a weighted average of the features from the next layer. The weighting coefficient is equal to the inverse of the distance to each point divided by the sum of the inverses of the distances to k points; the closer the points are, the larger the weighting coefficient. Interpolation allows the point cloud from the previous layer to acquire features similar to those of the next layer. The features before and after interpolation are then concatenated, and an additional input / output channel is added. The convolutions maintain the feature dimension at D2. The point cloud features output by the BEV_Mnet model feature extraction network ultimately consist of two parts: upsampled point cloud coordinates and upsampled point cloud features.

[0084] Specifically, the point cloud feature extraction network extracts point cloud features through a backbone network, performs semantic segmentation on these features through a segmentation network, and obtains point cloud features including point cloud label classification. Based on the point cloud labels, it classifies the original point cloud set and generates candidate boxes. To generate candidate boxes more efficiently and obtain a small number of high-recall candidate boxes, this paper directly extracts the point cloud labels of each point cloud, centered on the target point. A candidate box is more likely to be a true candidate box if it belongs to the target point. A bin-based candidate box generation method is used, where the original spatial dimensions are re-planned according to the bin size as the unit length to generate candidate boxes. The candidate box generation network process is as follows: Figure 7 As shown.

[0085] In one embodiment, the BEV_Mnet model employs transfer learning, for example... Figure 8 As shown, multiple rounds of training are first performed using multiple public datasets. The public label with the highest intersection-union ratio (IUGR) is selected, and the spatial attributes of the object category corresponding to the public label and the object category to be segmented in the mulberry garden scene are compared, such as the length and / or width and / or height of the object category. The corresponding BEV_Mnet model with the highest spatial attribute similarity is obtained. Then, the BEV_Mnet model is trained using the dataset collected in the mulberry garden scene to improve the classification accuracy of semantic labels in the mulberry garden scene. Finally, the model with the largest average IUGR is selected as the semantic segmentation network model for final application and applied to the actual autonomous navigation scenario in the mulberry garden.

[0086] In one specific implementation, the publicly available dataset uses the SemanticKITTI dataset. This invention adopts the SemanticKITTI dataset's LiDAR point cloud data format, where each point cloud has a label after manual annotation. First, a dictionary named `SemKITTI_label_name` is created, including the names of all categories in the SemanticKITTI dataset and their corresponding label values. Then, all label values ​​are retrieved and sorted, removing label values ​​corresponding to background or invalid categories. In PyTorch, many operations (such as the cross-entropy loss function and label processing in classification tasks) require category labels to start from 0. Therefore, the label value indices are converted to start from 0 to adapt to PyTorch's requirements.

[0087] The training process includes: creating a packaged BEV_Mnet model, selecting polar coordinates as the model type, using a BEV grid size of [480, 360, 32] for the bird's-eye view, a batch size of 2, a feature dimension of 9, a maximum of 256 points per SA module, an output point cloud feature dimension of 512, a convolution kernel size of 1, random sampling of the point cloud, and feature compression to 32. At the beginning of each training cycle, the data is randomly shuffled, and four threads are used to accelerate data loading. An empty list `loss_list` is initialized to store the loss value of each batch in the current cycle. A progress bar object `pbar` is created, using the TQDM library to display the training progress; the total length of the progress bar is set to the length of the training data loader, i.e., the total number of batches of training data. Each batch of data in the training data accelerator is iterated through, using a `try` method to catch possible exceptions. The point cloud of the training data is converted to the format required for training, and the model is used to perform forward propagation on the training data to obtain the prediction results. Calculate the loss values ​​for the prediction results, including Lovász-Softmax loss and cross-entropy loss, setting the ignore index to 255 to exclude invalid classes or background from the calculation. Perform backpropagation on the loss values ​​to calculate the gradient. Update the model parameters using the Adam optimizer and add the current batch's loss value to the `loss_list`. Reset the optimizer's gradients to prepare for the next iteration and update the progress bar to indicate the completion of a batch of training. If an anomaly occurs during training, capture the anomaly information.

[0088] The validation interval is set to 150, and the batch size during validation is 2. The system checks if the current global iteration count has reached the validation interval; if so, model validation is performed. The model is set to evaluation mode, and the data order is maintained during validation without shuffling it. An empty list `hist_list` is initialized to store the confusion matrix for each batch during validation. An empty list `val_loss_list` is initialized to store the loss value for each batch during validation. Gradients are not computed during validation to save resources and improve validation speed. Each batch of data in the validation data loader is traversed, and the validation data is converted to the format required for training. The model is used to predict the validation data to obtain the predicted labels. The loss values ​​of the prediction results are calculated, including the `lovasz_softmax` maximum loss and the cross-entropy loss. To avoid calculating gradients during validation, the tensor is separated from the computation graph using the `detach` method. The logits of the prediction results are converted to class indices by taking the maximum index at each position, the prediction results are moved from the GPU to the CPU, and converted to NumPy arrays. For each validation sample, a confusion matrix is ​​calculated between the predicted results and the true labels. A histogram between predicted values ​​and true labels is quickly calculated. The predicted values ​​and true labels are combined into an integer to count the frequency of each class pair, thus constructing a confusion matrix. After calculating the histogram, only the desired class-related components are retained and added to `hits_list`, while the loss value of the current batch is added to `val_loss_list`. To evaluate model performance, this invention uses the Intersection over Union (IoU) ratio, calculated as follows:

[0089] ;

[0090] Among them, TP i FP i and FN i The predicted number of true positives, false positives, and false negatives for category i are respectively. The mean Intersection over Union (mIoU) is the arithmetic mean of the IoUs, calculated as follows, where N is the total number of semantic categories:

[0091] ;

[0092] After validation, the model is reset to training mode. The Intersection over Union (IoU) for each class is calculated by summing the confusion matrices of all batches. Each class name and its corresponding IoU value is iterated over to calculate the validation mIoU. After validation, the IoU and mIoU for each class are printed. Variables used during validation are deleted, and memory is released. The current validation mIoU is checked against the previously recorded best mIoU. If it is, the best mIoU is updated to the current value, and the current model's weights are saved, ensuring the best-performing model is retained during training.

[0093] During the training process of this invention, the BEV_Mnet model is compared with other models to verify the effectiveness of the invention. The inference time of the BEV_Mnet model provided by the method of this invention and other control models are evaluated. For example, as shown in Table 1, the inference latency of a single frame point cloud in the maximum batch report that can be inferred on a single graphics card is compared. PolarNet can infer a maximum of 5 frames of point clouds at a time on a single graphics card, while the method of this invention can infer a maximum of 10 frames of point clouds at a time.

[0094] Table 1 shows the single-frame point cloud inference time tested on the SemanticKITTI dataset.

[0095] ;

[0096] The results of testing different network models on the SemanticKITTI dataset are shown in Table 2. Comparing the semantic segmentation visualization effects, PolarNet easily confuses fences with vegetation and also makes mistakes in classifying the categories of terrain, road, and parking. The method of this invention shows higher accuracy in identifying the road and fence categories.

[0097] Table 2 Comparison of semantic segmentation results on the SemanticKITTI dataset

[0098] ;

[0099] After training the BEV_Mnet model with the optimal crossover ratio (CROW) based on the publicly available SemanticKITTI dataset, this invention establishes the MulberryPlanting dataset for mulberry orchard scenarios. Data collection periods include winter pruning, summer pruning, and normal spring and autumn growing seasons. Multiple point cloud labels are set, including: pedestrians, buildings, mulberry tree trunks, fences, left-tree and right-tree rows of mulberry trees, roads, other agricultural machinery, and utility poles. The point cloud data is stored in floating-point binary file format, with each data point represented by a four-digit hexadecimal number (floating-point), separated by newlines. A point cloud data point contains coordinates (x, y, z) and r (reflection intensity). The dataset contains 22 sequences of LiDAR point cloud data, with sequences 00-07 and 09-10 used for training, sequence 08 for validation, and sequences 11-21 for testing. The labeled data is stored as txt text files, and the labels contain labeled data for multiple training scenarios. Each labeled file contains multiple attributes. The performance of different network models on the Mulberry Planting dataset is shown in Table 3. This invention achieves the highest IoU in recognizing farm roads, left-tree mulberry trees, and right-tree mulberry trees.

[0100] Table 3 Comparison of semantic segmentation results on the MulberryPlanting dataset

[0101] ;

[0102] Based on the test results in Tables 2 and 3, it can be concluded that, tested on the SemanticKITTI dataset, the deep learning-based semantic segmentation model BEV_Mnet of this invention achieves an mIoU higher than 57.0%, and the IoU for road category recognition reaches as high as 97.5%. In a real mulberry orchard scenario, tested on the MulberryPlanting dataset, this invention can accurately identify farm roads, left-tree rows of mulberry trees, and right-tree rows of mulberry trees, indicating that this method has broad application value. In terms of mulberry orchard navigation construction, this method can accurately identify mulberry trees and road surfaces in real time, providing important data support for the refined management of mulberry orchards; in terms of autonomous navigation, this method can quickly locate the ground and plan paths in real time based on the orientation of the tree rows on both sides, reducing damage to mulberry trees; this method can identify the location of mulberry trees, which is beneficial to the accuracy of automated harvesting and reduces errors in the harvesting process.

[0103] It will be understood by those skilled in the art that all or some of the steps and systems in the methods disclosed above can be implemented as software, firmware, hardware, and suitable combinations thereof. Some or all of the physical components can be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, which can include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, as is known to those skilled in the art, communication media typically contain computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0104] The above embodiments are only used to illustrate the present invention and are not intended to limit the technical solutions described herein. Although the present invention has been described in detail with reference to the above embodiments, the present invention is not limited to the specific embodiments described above. Therefore, any modifications or equivalent substitutions to the present invention, as well as all technical solutions and improvements that do not depart from the spirit and scope of the invention, are covered within the scope of the claims of the present invention.

Claims

1. A method for autonomous navigation in mulberry orchards based on lidar point clouds, characterized in that, include: A point cloud is collected, and the preprocessed point cloud is semantically segmented using a semantic segmentation network model to obtain point cloud features corresponding to the point cloud, including point cloud labels; the point cloud labels include a first label for associated path regions and a second label for associated crop regions. Obtain the crop region adjacent to the path region, and determine whether the corresponding crop region is a continuous space or a discrete space based on the second label; If the crop region is a discrete space, candidate boxes are generated for the point cloud set of the crop region based on the point cloud features. The geometric center point is selected as the point of interest from the candidate boxes, and the center position of each mulberry tree trunk is extracted. Delete points of interest (POIs) whose distance from the moving vehicle is greater than or equal to a first threshold; acquire multiple POIs on both sides of the moving vehicle along the movement direction, and set an index for the POIs according to the movement direction; calculate the distance between the POIs on both sides, and record the POIs on both sides with the smallest distance as adjacent points; acquire the adjacent points with index differences, and interpolate the adjacent points along the movement direction; connect the POIs on both sides and the interpolated points to obtain a reference line; acquire the perpendicular foot of the POI on one side to the reference line on the other side, calculate the midpoint between the POI and the perpendicular foot, and obtain the navigation point; If the crop area is a continuous space, the boundary point cloud of the crop area is obtained as the point of interest, and the point cloud with the smallest distance from the moving vehicle that is greater than or equal to a first threshold is obtained as the point of interest; adjacent points of interest are sequentially connected along the movement direction to obtain a reference line; the reference lines on both sides of the moving vehicle are obtained along the movement direction, and multiple interval points are set on the reference lines at a preset first interval, and the midpoint of the line connecting the adjacent interval points on both sides is used as a navigation point.

2. The method as described in claim 1, characterized in that, The preprocessing includes: using the statistical filtering algorithm of the PCL point cloud library to remove outliers from the point cloud set, and using the pass-through filter algorithm of the PCL point cloud library to retain the point cloud within a preset area.

3. An autonomous navigation system for mulberry orchards based on lidar point clouds, characterized in that, include: The semantic segmentation module is used to collect point clouds and perform semantic segmentation on the preprocessed point clouds using a semantic segmentation network model to obtain point cloud features corresponding to the point clouds, including point cloud labels; the point cloud labels include a first label for associated path regions and a second label for associated crop regions. The region identification module is used to obtain the crop region adjacent to the path region, and determine whether the corresponding crop region is a continuous space or a discrete space based on the second label; The path generation module, if the crop region is a discrete space, is used to generate candidate boxes for the point cloud set of the crop region based on the point cloud features, select the geometric center point as the point of interest from the candidate boxes, and extract the center position of each mulberry tree trunk; Delete points of interest (POIs) whose distance from the moving vehicle is greater than or equal to a first threshold; acquire multiple POIs on both sides of the moving vehicle along the movement direction, and set an index for the POIs according to the movement direction; calculate the distance between the POIs on both sides, and record the POIs on both sides with the smallest distance as adjacent points; acquire adjacent points with index differences, and interpolate the adjacent points along the movement direction; connect the POIs on both sides and the interpolation points to obtain a reference line; acquire the perpendicular foot of the reference line on the other side of the POI on one side, calculate the midpoint between the POI and the perpendicular foot, and obtain a navigation point; if the crop area is a continuous space, acquire the boundary point cloud of the crop area as the POI, acquire the point cloud whose distance from the moving vehicle is greater than or equal to the first threshold and whose distance is the smallest, and use it as the POI; connect adjacent POIs sequentially along the movement direction to obtain a reference line; acquire the reference lines on both sides of the moving vehicle along the movement direction, set multiple interval points on the reference lines at a preset first interval, and use the midpoint of the line connecting the adjacent interval points on both sides as a navigation point.

Images