Lemon fruit tree semantic segmentation method and system based on unmanned aerial vehicle laser radar and transfer learning, storage medium and computer equipment

Abstract

The invention discloses a lemon fruit tree semantic segmentation method and system based on unmanned aerial vehicle laser radar and transfer learning, a storage medium and a computer device, and the method comprises the following steps: carrying out the data collection of a lemon orchard through employing a laser radar scanning device based on an unmanned aerial vehicle; processing the acquired data by adopting an instant positioning and map building algorithm so as to build a real-time three-dimensional point cloud map of the lemon orchard; constructing a neural network model, optimizing the neural model by adopting a deep learning and transfer learning method, and performing real-time semantic segmentation on the three-dimensional point cloud map of the lemon orchard by using the optimized neural network model; based on the semantic segmentation result, lemon fruit tree point cloud and environment point cloud are separated, and corresponding geometric parameters of the lemon fruit tree are calculated and extracted. According to the method, the accurate three-dimensional point cloud map can be reconstructed, the lemon fruit tree point cloud and the three-dimensional geometric parameters thereof in the three-dimensional point cloud map can be extracted, and accurate data support is provided for various subsequent agricultural researches.

Description

Technical Field

[0001] The present invention relates to agricultural monitoring technology, and in particular to a lemon tree semantic segmentation method, system, storage medium and computer equipment based on unmanned aerial vehicle laser radar and transfer learning. Background Art

[0002] In the intelligent management of orchards in hilly and mountainous areas, the reliability and information richness of the environmental perception system directly determine the decision-making ability of the operating drone. Among the current mainstream SLAM technologies, although the visual solution has the advantages of low cost and rich texture information, it is easily affected by sudden changes in illumination, occlusion by branches and leaves, and weak texture environments (such as dense canopies) in orchard scenes, resulting in problems such as feature matching inaccuracy and depth estimation drift, making it difficult to build a stable spatial reference. In comparison, LiDAR, with its active ranging principle and millimeter-level ranging accuracy, can generate high-density, interference-resistant point cloud data in complex terrain, showing significant advantages in tasks such as measuring the gap between fruit trees and terrain elevation modeling. However, existing LiDAR SLAM technology still has two core flaws in hilly orchards. First, due to factors such as terrain undulation and dynamic occlusion of fruit tree canopies, traditional point cloud registration algorithms are prone to feature degradation and error accumulation, resulting in insufficient local accuracy of the reconstructed map and difficulty supporting high-confidence semantic segmentation. Second, existing semantic mapping technology only performs coarse-grained annotation of point clouds through binary classification (fruit tree / non-fruit tree), and is unable to extract agronomic parameters such as canopy volume from the point cloud. The semantic representation of fruit trees should not only include tree type and location, but also be combined with more detailed three-dimensional geometric data (such as tree height, tree center position, canopy width, and other parameters). This multi-level information fusion can provide more comprehensive and accurate data support for orchard management. Summary of the Invention

[0003] The first objective of this invention is to overcome the shortcomings of the above-mentioned existing technologies by providing a method for semantic segmentation of lemon trees based on drone-based lidar and transfer learning. This method can address the problem of semantic segmentation inaccuracy and the lack of functionality of traditional deep semantic segmentation methods in extracting three-dimensional geometric parameters of fruit trees.

[0004] The second object of the present invention is to provide a lemon tree semantic segmentation system based on UAV lidar and transfer learning.

[0005] A third object of the present invention is to provide a storage medium.

[0006] A fourth object of the present invention is to provide a computer device.

[0007] The first object of the present invention is achieved through the following technical solution: This method for semantic segmentation of lemon trees based on drone laser radar and transfer learning includes the following steps:

[0008] S1. Use drone-based LiDAR scanning equipment to collect data from the lemon orchard and use real-time positioning and mapping algorithms to process the collected data to create a real-time 3D point cloud map of the lemon orchard.

[0009] S2. Build a neural network model and optimize it using deep learning and transfer learning methods. Then, use the optimized neural network model to perform real-time semantic segmentation on the 3D point cloud map of the lemon orchard.

[0010] S3. Based on the semantic segmentation results, separate the lemon tree point cloud and the environment point cloud, and calculate and extract the corresponding geometric parameters of the lemon tree.

[0011] Preferably, step S1 includes the following steps:

[0012] S11. Integrate lidar, IMU, GNSS, and an onboard computer into a drone, and deploy a scanning driver and real-time positioning and mapping algorithms on the onboard computer.

[0013] S12. While the drone is flying over the orchard, it uses the ROS robot operating system in the onboard computer to run a scanning driver program. The scanning driver program is used to receive and manage data collected by the lidar and the IMU and GNSS data.

[0014] S13, the real-time positioning and mapping algorithm integrates the data collected by lidar, IMU and GNSS, aligns and corrects the point cloud data, and builds a real-time 3D point cloud map of the lemon orchard through the ikd-tree structure.

[0015] Preferably, in step S13, the process of establishing the three-dimensional point cloud map of the lemon orchard includes the following steps:

[0016] S131: The point cloud data collected by the lidar is accumulated, and the acceleration and angular velocity provided by the IMU are processed through forward propagation to obtain a rough odometry estimated pose.

[0017] S132, in the odometer pose prediction and update stage, the motion-compensated odometer estimated pose and GNSS data are collaboratively integrated and iteratively updated to obtain the latest accurate odometer estimated pose;

[0018] S133, accurate odometry estimated pose is used for incremental point cloud storage of ikd-tree and to complete real-time 3D point cloud map reconstruction of the lemon orchard.

[0019] Preferably, step S132 includes the following specific processes:

[0020] S1321. Using the motion-compensated odometer pose as the basis for state prediction, combined with the absolute position observations provided by GNSS, iteratively calculate the lidar-IMU joint residual and the GNSS observation residual, and simultaneously construct the noise covariance matrix and observation model matrix:

[0021]

[0022] Where z l-i is the residual error calculated jointly by the lidar and IMU, z GNSS is the GNSS residual, is the observation matrix calculated jointly by the lidar and IMU, is the GNSS observation matrix, P is the pose covariance calculated jointly by the lidar and IMU;

[0023] S1322: Predict and update the odometer estimated pose, and obtain the predicted pose for the next frame based on the current frame:

[0024]

[0025] Where, means generalized addition, Θ means generalized subtraction, is the predicted value of the current k-th frame, is the forward propagation value of the current k-th frame, J k is the Jacobian matrix;

[0026] S1323. Dynamically adjust the state update weight through the Kalman gain, successively optimize the covariance ellipsoid of the pose estimation, and finally output a high-precision odometer pose that integrates the characteristics of multiple sensors.

[0027] Preferably, step S2 includes the following steps:

[0028] S21. Build a PointNet++ neural network model and pre-train it using the DALES dataset. Freeze the encoding and decoding layer parameters of the pre-trained PointNet++ neural network model.

[0029] S22. Modify the output layer parameters of the PointNet++ neural network model according to the corresponding category of the semantic segmentation task, and use the lemon orchard dataset composed of the three-dimensional point cloud map of the lemon orchard obtained in step S1 to formally train the PointNet++ neural network model;

[0030] S23. Deploy the formally trained PointNet++ neural network model to the onboard computer to complete the semantic segmentation of the 3D point cloud map of the lemon orchard in real time.

[0031] Preferably, step S3 includes the following specific steps:

[0032] S31, using the Open3D point cloud processing database to separate the lemon tree point cloud from the semantic map obtained by semantic segmentation;

[0033] S32. Use the DBSCAN method to segment the point cloud of individual lemon trees, and calculate geometric parameters based on the point cloud of individual lemon trees.

[0034] The geometric parameters include tree height, tree center, maximum crown width in the east-west direction, and maximum crown width in the north-south direction.

[0035] The second aspect of the present invention is to provide a lemon tree semantic segmentation system based on drone laser radar and transfer learning, comprising:

[0036] The data acquisition module is used to collect data from the lemon orchard and the IMU and GNSS data of the drone during flight;

[0037] The data processing module uses the real-time positioning and mapping algorithm and the ikd-tree structure to build a real-time 3D point cloud map of the lemon orchard based on the data collected by the data acquisition module;

[0038] The semantic segmentation module uses deep learning and transfer learning methods to build a neural network model suitable for the semantic segmentation task of lemon trees. It segments the established 3D point cloud map of the lemon orchard and outputs a 3D semantic map.

[0039] The parameter calculation module, based on a 3D semantic map, accurately separates point cloud clusters of individual lemon trees from complex orchard scenes and quickly calculates the geometric parameters of individual lemon trees.

[0040] A storage medium stores a program, which, when executed by a processor, implements the lemon tree semantic segmentation method based on drone lidar and transfer learning as described in the first purpose.

[0041] A computer device includes a processor and a memory for storing a program executable by the processor. When the processor executes the program stored in the memory, the lemon tree semantic segmentation method based on drone lidar and transfer learning as described in the first purpose is implemented.

[0042] The present invention has the following advantages over the prior art:

[0043] This invention uses a real-time positioning and mapping algorithm to fuse LiDAR, IMU, and GNSS data, significantly improving the accuracy of 3D point cloud map reconstruction for lemon orchards in hilly mountainous areas. Compared with traditional point cloud registration algorithms, this data fusion effectively reduces errors caused by frequent changes in drone altitude, improving mapping accuracy and data reliability.

[0044] 2. This method utilizes a neural network model based on deep learning and transfer learning. Compared to traditional, energy-intensive semantic segmentation methods, it leverages low-frequency lateral vibrations, thereby reducing energy consumption during training and operation. Furthermore, this method can accurately segment and extract the geometric parameters of fruit trees, meeting the high-precision requirements of agricultural environments.

[0045] 3. This method uses semantic segmentation to separate lemon tree point clouds from the surrounding point clouds and segment individual lemon trees using the DBSCAN method, accurately extracting each tree's geometric parameters (such as tree height, tree center position, and crown width). This innovative method improves the data support capabilities for orchard management and provides more accurate information for refined agricultural management. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 This is a flowchart of the lemon tree semantic segmentation method based on drone-mounted lidar and transfer learning of the present invention.

[0047] Figure 2 A flowchart for building a 3D point cloud map of a lemon orchard based on a simultaneous localization and mapping algorithm.

[0048] Figure 3 A flowchart for achieving real-time semantic segmentation.

[0049] Figure 4 This is a structural diagram of the lemon tree semantic segmentation system based on drone-mounted lidar and transfer learning of the present invention.

[0050] Among them, 1 is the drone, 2 is the lidar, 3 is the IMU, 4 is the GNSS, and 5 is the onboard computer. DETAILED DESCRIPTION

[0051] The present invention will be further described below with reference to the accompanying drawings and examples.

[0052] like Figure 1 As shown in the figure, the semantic segmentation method of lemon trees based on UAV lidar and transfer learning includes the following steps:

[0053] S1. Use drone-based LiDAR scanning equipment to collect data from the lemon orchard and use real-time positioning and mapping algorithms to process the collected data to create a real-time 3D point cloud map of the lemon orchard.

[0054] Step S1 includes the following steps:

[0055] S11. Integrate lidar, IMU, GNSS and an onboard computer into a drone, and deploy a scanning driver and real-time positioning and map building algorithms on the onboard computer. Specifically, this implementation integrates three core sensors, LIVOX AVIA lidar, BMI088 IMU and C-RTK2 HP GNSS, on the drone platform, and implements coordinated control of hardware and software through the onboard computer: the lidar acquires three-dimensional point cloud information of the orchard in a high-frequency scanning manner, the IMU outputs the acceleration and angular velocity data of the lidar body in real time to infer the local motion pose, and the GNSS provides the drone's global spatial coordinates as an absolute positioning reference. The three constitute a complementary spatial perception system; the scanning driver deployed on the onboard computer is responsible for synchronously activating the sensors and normalizing the transmitted raw data stream, while the tightly coupled real-time positioning and map building synchronously fuse multi-source information - the short-term pose changes of the IMU are used to compensate for the motion distortion of the lidar point cloud, and the global coordinate constraints of the GNSS are used to suppress the accumulated error in the vertical direction, and finally a three-dimensional point cloud map with centimeter-level accuracy is generated through dynamic alignment.

[0056] S12. When the drone is flying over the orchard, it uses the ROS robot operating system in the onboard computer to run the scanning driver, which is used to receive and manage the data collected by the lidar and the data from the IMU and GNSS. In this implementation, in the task of scanning and acquiring data, a sensor data hub is built based on the ROS robot operating system, and the synchronous collection and centralized management of multi-source heterogeneous data is achieved through a standardized topic mechanism. Specifically, the onboard computer runs a customized scanning driver under the ROS framework: independent topic nodes are created for the three types of sensors, lidar, IMU, and GNSS, and bidirectional control of data flow is achieved through a subscription-publish model. The driver not only issues control instructions such as starting scanning and parameter configuration to the sensor, but also receives and caches the original point cloud, posture, and positioning data returned by the sensor in real time.

[0057] S13. The real-time positioning and mapping algorithm integrates data collected by lidar, IMU, and GNSS, registers and corrects the point cloud data, and creates a real-time 3D point cloud map of the lemon orchard using an ikd-tree structure. Specifically, this implementation achieves multi-sensor data fusion and dynamic environment modeling through a tightly coupled real-time positioning and mapping algorithm: the system uses the lidar point cloud as its core input, integrates the IMU's high-frequency motion parameters (acceleration and angular velocity) to infer local pose changes in real time, and introduces the GNSS's global coordinates as an absolute position constraint. The pose estimate is iteratively optimized using an error Kalman filter to compensate for the point cloud distortion caused by the lidar when the drone is in high-speed motion and complex terrain. The optimized high-precision odometry pose serves as a spatial reference, driving an incremental map-building engine based on an ikd-tree. This structure uses a dynamically balanced binary tree to achieve efficient storage and nearest neighbor retrieval of massive point clouds. The distortion-corrected point cloud segments are inserted into the global map according to their spatial pose using a frame-by-frame registration method. During the insertion process, redundant data is pruned and the spatial index is updated in real time.

[0058] like Figure 2 As shown, in step S13, the process of establishing the three-dimensional point cloud map of the lemon orchard includes the following steps:

[0059] S131. The point cloud data collected by the lidar is accumulated, and the acceleration and angular velocity provided by the IMU (measurements in the x, y, and z directions) are processed through forward propagation to obtain a rough odometer estimated pose. Specifically, the data collected by the lidar and the various data provided by the IMU are jointly used in the calculation of backward motion compensation. The lidar collects and preprocesses point cloud fragments in a 20-millisecond time window: within a single acquisition cycle, the IMU continuously outputs x, y, and z three-axis acceleration and angular velocity data, and the forward propagation algorithm is used to infer the continuous motion trajectory of the drone within this time window, generating a preliminary odometer pose estimate. This pose is then applied to the original point cloud accumulated within the time window by backpropagation - the instantaneous pose of each laser point at the time of acquisition is reversely interpolated based on the kinematic model, and point cloud distortion is eliminated through coordinate transformation.

[0060] S132, in the odometer pose prediction and update stage, the motion-compensated odometer estimated pose and GNSS data are collaboratively integrated and iteratively updated to obtain the latest accurate odometer estimated pose;

[0061] Step S132 includes the following specific processes:

[0062] S1321. Using the motion-compensated odometer pose as the basis for state prediction, combined with the absolute position observations provided by GNSS, iteratively calculate the lidar-IMU joint residual and the GNSS observation residual, and simultaneously construct the noise covariance matrix and observation model matrix:

[0063]

[0064] Where z l-i is the residual error calculated jointly by the lidar and IMU, z GNSS is the GNSS residual, is the observation matrix calculated jointly by the lidar and IMU, is the GNSS observation matrix, P is the pose covariance calculated jointly by the lidar and IMU;

[0065] S1322: Predict and update the odometer estimated pose, and obtain the predicted pose for the next frame based on the current frame:

[0066]

[0067] Where, means generalized addition, Θ means generalized subtraction, is the predicted value of the current k-th frame,

[0068] is the forward propagation value of the current k-th frame, J k is the Jacobian matrix;

[0069] S1323. Dynamically adjust the state update weight through the Kalman gain, successively optimize the covariance ellipsoid of the pose estimation, and finally output a high-precision odometer pose that integrates the characteristics of multiple sensors.

[0070] During this process, GNSS data significantly suppresses the cumulative drift in the Z-axis direction caused by the frequent ascent and descent of the drone in the hilly orchard scene by introducing vertical height constraints in the geodetic coordinate system. The horizontal pose correction relies on the high-frequency local motion observation of the lidar-IMU, forming a complementary mechanism of "global anchoring-local optimization" to ensure the stability and reliability of three-dimensional pose estimation in complex terrain.

[0071] S133, the precise odometry-estimated pose is used for incremental point cloud storage in the ikd-tree and for real-time reconstruction of a 3D point cloud map of the lemon orchard. Specifically, with the current optimal pose as the origin of the global coordinate system, the motion-compensated local point cloud fragments are transformed and mapped to a unified spatial coordinate system. The dynamically balanced binary tree characteristics of the ikd-tree are then leveraged to efficiently organize massive point clouds. This structure updates the spatial index in real time through a frame-by-frame insertion mechanism—it performs voxel-based spatial hashing on the new input point cloud, dynamically adjusts the tree node distribution while maintaining the topological structure, and automatically removes duplicate area point clouds to control data size. Each insertion triggers a nearest neighbor search optimization, and the kd-tree pruning strategy maintains O(logN) query efficiency. Ultimately, a dense 3D point cloud map of the orchard is formed that combines spatial consistency (positioning accuracy better than 5cm) with real-time performance (single-frame processing latency <50ms), adapting to the large-scale scene reconstruction needs of continuous operations on hilly terrain.

[0072] S2. Build a neural network model and optimize it using deep learning and transfer learning methods. Then, use the optimized neural network model to perform real-time semantic segmentation on the 3D point cloud map of the lemon orchard.

[0073] like Figure 3 As shown, step S2 includes the following steps:

[0074] S21. Construct a PointNet++ neural network model and use the DALES dataset to pre-train the PointNet++ neural network model, freezing the encoding layer and decoding layer parameters of the pre-trained PointNet++ neural network model. Specifically, the neural network model used in this embodiment is the PointNet++ neural network model. Aiming at the characteristics of the hilly lemon orchard scene, a transfer learning strategy is used to improve the generalization ability of the model: first, the DALES public point cloud dataset (covering multi-scale complex scenes such as cities, forests, and farmlands) is used to pre-train the PointNet++ network. This dataset is significantly superior to a single orchard scene in terms of spatial structure diversity (including building facades, vegetation levels, terrain undulations, etc.) and data scale (more than 505 million point cloud annotated samples), forcing the model encoder to learn more discriminative geometric feature extraction capabilities (such as branch and leaf gap recognition and irregular terrain modeling), and the decoder establishes a robust mapping relationship from abstract features to semantic segmentation, thereby avoiding overfitting problems in subsequent small sample training.

[0075] After pre-training, all network parameters of PointNet++'s encoding layer (responsible for hierarchical local feature aggregation) and decoding layer (implementing feature upsampling and semantic prediction) are locked, preserving its ability to express general features learned in complex scenarios. This freezing operation not only maintains the network's deep understanding of point cloud geometry and topology (key features such as trunk curvature and fruit clustering), but also limits subsequent model optimization to the fully connected classification layer. Through targeted fine-tuning to adapt to the unique semantic labeling system of lemon orchards, this achieves low-cost knowledge transfer from general scenarios to specific tasks.

[0076] S22. Modify the output layer parameters of the PointNet++ neural network model based on the corresponding categories of the semantic segmentation task. The PointNet++ neural network model is formally trained using the lemon orchard dataset, consisting of the 3D point cloud map of the lemon orchard obtained in step S1. Specifically, the pre-trained PointNet++ model is adapted to the semantic segmentation requirements of the lemon orchard scene. First, the fully connected structure of the network output layer is reconstructed based on the orchard target category system (including lemon trees, ground and noise points, man-made objects, and other plants). The number of neurons and activation function are adjusted to match the actual classification dimensions. The 3D point cloud data collected in step S1 is then manually annotated to construct a lemon orchard dataset containing spatial coordinates and semantic labels. Data augmentation (random rotation and scaling) is used to increase sample diversity. During the formal training phase, while keeping the encoder-decoder layer parameters frozen, only the output layer is fine-tuned in a supervised manner. Backpropagation using the cross-entropy loss function optimizes the classification weights, gradually strengthening the model's ability to identify specific orchard targets (lemon trees), ultimately achieving centimeter-level point cloud semantic segmentation accuracy.

[0077] S23. Deploy the formally trained PointNet++ neural network model to the onboard computer to complete the semantic segmentation of the lemon orchard's 3D point cloud map in real time. The optimized PointNet++ semantic segmentation model is deployed to the drone's onboard computer through the TensorRT acceleration engine to build an end-to-end real-time point cloud processing pipeline: During the flight mission, the raw point cloud stream collected by the lidar is input into the deployed neural network model at a frequency of 20Hz, and millisecond-level semantic reasoning is achieved through GPU acceleration, outputting a semantic point cloud with category labels. The segmentation results are published in real time through the ROS node, and after being fused with the synchronously generated positioning pose data, the OctoMap voxelization method is used to construct a dynamically updated semantic map - this map not only retains the geometric details of the original point cloud (resolution 5cm) but also embeds precise category semantic information.

[0078] S3. Based on the semantic segmentation results, separate the lemon tree point cloud and the environment point cloud, and calculate and extract the corresponding geometric parameters of the lemon tree.

[0079] Step S3 includes the following specific steps:

[0080] S31. Using the Open3D point cloud processing database, the lemon tree point cloud is separated from the semantic map generated by semantic segmentation. The labeled point cloud data generated by semantic segmentation is read and a point cloud attribute filter is used to quickly extract the subset of point clouds labeled "lemon tree." The processed individualized fruit tree point cloud is rendered in real time through the Open3D visualization interface and simultaneously exported as a standardized LAS file containing 3D spatial coordinates, tree height eigenvalues, and point cloud density parameters. This provides high-quality input data for subsequent precise agricultural analysis, such as individual fruit tree modeling and canopy volume calculation.

[0081] S32. Use the DBSCAN method to segment individual lemon tree point clouds and calculate geometric parameters based on the individual lemon tree point cloud. Based on the subset of lemon tree point clouds processed by Open3D, a density clustering algorithm (DBSCAN) is used to separate individual trees: the neighborhood radius (eps = 0.2m) and the minimum number of neighborhood points (min_pts = 5) are set as clustering thresholds, and spatially continuous point cloud clusters are identified as independent fruit tree individuals through density reachability analysis. This algorithm effectively handles complex situations such as intertwined branches and leaves between fruit trees and adhesion of point clouds of adjacent plants, outputting topologically complete individual fruit tree point cloud clusters.

[0082] Perform multi-level 3D geometric parameter extraction on a single point cloud cluster: First, calculate the vertical extreme value (z max With z min Difference) is used as the tree height parameter, and the expression is as follows:

[0083] H=z max -z min

[0084] Secondly, the center position of the plant is defined by the centroid coordinates of the point cloud (the average of the three-axis coordinates of x, y, and z) to solve the problem of center positioning under irregular crown shapes. The formula is as follows:

[0085]

[0086] Among them, x c ,y c ,z c is the coordinate of the centroid, n is the number of points of a single fruit tree, x i ,y i ,z i is the coordinate of the i-th point. Finally, the point cloud is projected along the horizontal plane, and the maximum extension distance in the east-west direction (x-axis range) and the north-south direction (y-axis range) are extracted as the crown width parameter. The projected area is simultaneously calculated to represent the crown coverage.

[0087] This implementation case breaks through the limitation of traditional binary classification labeling, which only distinguishes fruit trees from non-fruit trees. It achieves fine segmentation of lemon tree point clouds and integrates detailed three-dimensional geometric information (such as tree height, tree core and crown width) to provide comprehensive and accurate data support for orchard management.

[0088] like Figure 4 As shown, the second aspect of the present invention is to provide a lemon tree semantic segmentation system based on drone laser radar and transfer learning, comprising:

[0089] The data acquisition module is used to collect data from the lemon orchard and the IMU and GNSS data of the drone during flight. Specifically, this implementation uses a drone as a vehicle to carry the data acquisition module. This data acquisition module is mainly composed of an integrated lidar (LIVOX AVIA), an IMU (BMI088), and a GNSS (C-RTK2 HP). The lidar collects dense point clouds of the orchard environment at a frequency of 10Hz, the IMU provides 200Hz high-frequency motion parameters to compensate for point cloud distortion, and the GNSS outputs centimeter-level global coordinates through dual-frequency RTK positioning.

[0090] The data processing module uses the real-time positioning and map construction algorithm and the ikd-tree structure to establish a real-time three-dimensional point cloud map of the lemon orchard based on the data collected by the data acquisition module; the data processing module of this embodiment is mainly composed of the real-time positioning and map construction algorithm and the ikd-tree structure, and through the tight coupling of the real-time positioning and map construction algorithm, multi-source data is integrated in real time, and then the ikd-tree structure is used to generate a high-resolution global three-dimensional point cloud map, providing a spatial benchmark for subsequent analysis.

[0091] The semantic segmentation module, based on deep learning and transfer learning methods, constructs a neural network model suitable for the semantic segmentation task of lemon trees, which segments the established 3D point cloud map of the lemon orchard to output a 3D semantic map. This semantic segmentation module uses a dedicated neural network model that can accurately distinguish lemon trees from the background and other objects, achieving fine semantic segmentation of point cloud data, thereby providing detailed semantic information for fruit tree identification and positioning.

[0092] The parameter calculation module, based on a three-dimensional semantic map, accurately separates point cloud clusters of individual lemon trees from complex orchard scenes and quickly calculates individual geometric parameters of lemon trees, such as tree height, tree center position, and crown width, thereby providing reliable data support for precise orchard management and agricultural decision-making.

[0093] The above-mentioned lemon tree semantic segmentation system based on drone lidar and transfer learning, the system's hardware subsystems (including lidar, IMU, GNSS, and airborne computer) are integrated into the drone platform, and the construction of a three-dimensional environment dynamic reference framework is realized through a tightly coupled SLAM algorithm; the software architecture is deployed on the airborne computing unit, forming a complete processing chain of multimodal data acquisition and fusion (lidar scanning module), point cloud semantic analysis based on transfer learning (semantic segmentation module), and quantification of single-tree agricultural parameters (parameter calculation module), realizing full-process embedded real-time computing from raw point cloud acquisition to plant-level growth parameter output.

[0094] A storage medium stores a program, which, when executed by a processor, implements the lemon tree semantic segmentation method based on drone lidar and transfer learning as described in the first purpose.

[0095] A computer device includes a processor and a memory for storing a program executable by the processor. When the processor executes the program stored in the memory, the lemon tree semantic segmentation method based on drone lidar and transfer learning as described in the first purpose is implemented.

[0096] The above specific implementation manner is a preferred embodiment of the present invention and does not limit the present invention. Any other changes or other equivalent replacement methods that do not deviate from the technical solution of the present invention are included in the protection scope of the present invention.

Claims

1. A lemon tree semantic segmentation method based on UAV lidar and transfer learning, characterized by: The following steps are involved: S1. Use drone-based LiDAR scanning equipment to collect data from the lemon orchard and use real-time positioning and mapping algorithms to process the collected data to create a real-time 3D point cloud map of the lemon orchard. S2. Build a neural network model and optimize it using deep learning and transfer learning methods. Then, use the optimized neural network model to perform real-time semantic segmentation on the 3D point cloud map of the lemon orchard. S3. Based on the semantic segmentation results, separate the lemon tree point cloud and the environment point cloud, and calculate and extract the corresponding geometric parameters of the lemon tree.

2. The lemon tree semantic segmentation method based on drone laser radar and transfer learning according to claim 1 is characterized in that, Step S1 includes the following steps: S11. Integrate lidar, IMU, GNSS, and an onboard computer into a drone, and deploy a scanning driver and real-time positioning and mapping algorithms on the onboard computer. S12. While the drone is flying over the orchard, it uses the ROS robot operating system in the onboard computer to run a scanning driver program. The scanning driver program is used to receive and manage data collected by the lidar and the IMU and GNSS data. S13, the real-time positioning and mapping algorithm integrates the data collected by lidar, IMU and GNSS, aligns and corrects the point cloud data, and builds a real-time 3D point cloud map of the lemon orchard through the ikd-tree structure.

3. The lemon tree semantic segmentation method based on drone laser radar and transfer learning according to claim 1 is characterized in that, In step S13, the process of establishing the three-dimensional point cloud map of the lemon orchard includes the following steps: S131: The point cloud data collected by the lidar is accumulated, and the acceleration and angular velocity provided by the IMU are processed through forward propagation to obtain a rough odometry estimated pose. S132, in the odometer pose prediction and update stage, the motion-compensated odometer estimated pose and GNSS data are collaboratively integrated and iteratively updated to obtain the latest accurate odometer estimated pose; S133, accurate odometry estimated pose is used for incremental point cloud storage of ikd-tree and to complete real-time 3D point cloud map reconstruction of the lemon orchard.

4. The lemon tree semantic segmentation method based on drone laser radar and transfer learning according to claim 1 is characterized in that, Step S132 includes the following specific processes: S1321. Using the motion-compensated odometer pose as the basis for state prediction, combined with the absolute position observations provided by GNSS, iteratively calculate the lidar-IMU joint residual and the GNSS observation residual, and simultaneously construct the noise covariance matrix and observation model matrix: Where z l-i is the residual error calculated jointly by the lidar and IMU, z GNSS is the GNSS residual, is the observation matrix calculated jointly by the lidar and IMU, is the GNSS observation matrix, P is the pose covariance calculated jointly by the lidar and IMU; S1322: Predict and update the odometer estimated pose, and obtain the predicted pose for the next frame based on the current frame: Where, means generalized addition, Θ means generalized subtraction, is the predicted value of the current k-th frame, is the forward propagation value of the current k-th frame, J k is the Jacobian matrix; S1323. Dynamically adjust the state update weight through the Kalman gain, successively optimize the covariance ellipsoid of the pose estimation, and finally output a high-precision odometer pose that integrates the characteristics of multiple sensors.

5. The lemon tree semantic segmentation method based on drone laser radar and transfer learning according to claim 1 is characterized in that Step S2 includes the following steps: S21. Build a PointNet++ neural network model and pre-train it using the DALES dataset. Freeze the encoding and decoding layer parameters of the pre-trained PointNet++ neural network model. S22. Modify the output layer parameters of the PointNet++ neural network model according to the corresponding category of the semantic segmentation task, and use the lemon orchard dataset composed of the three-dimensional point cloud map of the lemon orchard obtained in step S1 to formally train the PointNet++ neural network model; S23. Deploy the formally trained PointNet++ neural network model to the onboard computer to complete the semantic segmentation of the 3D point cloud map of the lemon orchard in real time.

6. The lemon tree semantic segmentation method based on drone laser radar and transfer learning according to claim 1 is characterized in that Step S3 includes the following specific steps: S31, using the Open3D point cloud processing database to separate the lemon tree point cloud from the semantic map obtained by semantic segmentation; S32. Use the DBSCAN method to segment the point cloud of individual lemon trees, and calculate geometric parameters based on the point cloud of individual lemon trees.

7. The lemon tree semantic segmentation method based on drone laser radar and transfer learning according to claim 6 is characterized in that: The geometric parameters include tree height, tree center, maximum crown width in the east-west direction, and maximum crown width in the north-south direction.

8. A lemon tree semantic segmentation system based on UAV lidar and transfer learning, characterized by: include: The data acquisition module is used to collect data from the lemon orchard and the IMU and GNSS data of the drone during flight; The data processing module uses the real-time positioning and mapping algorithm and the ikd-tree structure to build a real-time 3D point cloud map of the lemon orchard based on the data collected by the data acquisition module; The semantic segmentation module uses deep learning and transfer learning methods to build a neural network model suitable for the semantic segmentation task of lemon trees. It segments the established 3D point cloud map of the lemon orchard and outputs a 3D semantic map. The parameter calculation module, based on a 3D semantic map, accurately separates point cloud clusters of individual lemon trees from complex orchard scenes and quickly calculates the geometric parameters of individual lemon trees.

9. A storage medium storing a program, characterized in that: When the program is executed by a processor, the lemon tree semantic segmentation method based on drone lidar and transfer learning as described in any one of claims 1 to 7 is implemented.

10. A computer device comprising a processor and a memory for storing a program executable by the processor, characterized in that When the processor executes the program stored in the memory, the lemon tree semantic segmentation method based on UAV lidar and transfer learning as described in any one of claims 1 to 7 is implemented.

Images