A Global Localization Method and System for Orchard Operation Robots Based on Large Language Model-Multi-Metric ICP

Abstract

This application discloses a global localization method and system for orchard robots based on a large language model and multi-metric ICP. The method first constructs a global semantic prior map containing structured text descriptions and macroscopic geometric feature vectors; then, it collects real-time point cloud data, performs semantic segmentation, and generates text descriptions and geometric feature vectors for the current scene; a coarse-to-fine two-stage registration strategy is adopted. In the coarse registration stage, features are filtered through information entropy, and the optimal candidate region is selected by an adaptive fusion score combining semantic matching degree and geometric feature similarity, and the coarse registration pose is obtained through the NDT algorithm. In the fine registration stage, a multi-metric ICP optimization objective function is constructed, fusing distance, feature, intensity, and semantic residual terms. Using the coarse registration pose as the initial value, the weights of each metric term are dynamically adjusted based on the large language model to solve for the optimal pose transformation; the global localization result of the robot is obtained based on the optimal pose transformation. This application improves the accuracy and robustness of localization.

Description

Technical Field

[0001] This application belongs to the field of autonomous navigation and precise positioning technology for orchard robots, and in particular relates to a global positioning technology based on cross-modal fusion under a large language model, which is applicable to unstructured cross-temporal dynamic orchard environments with satellite signal rejection. Background Technology

[0002] Existing global localization methods for mobile robots primarily rely on Global Positioning System (GPS), visual sensors, or LiDAR point cloud registration algorithms. In orchard environments, these methods have significant drawbacks:

[0003] 1. Insufficient robustness of single-sensor positioning:

[0004] In orchard environments, GPS signals are easily obstructed by dense fruit tree canopies and greenhouse steel frames, leading to decreased accuracy or even failure. Meanwhile, visual sensors exhibit poor positioning stability under drastic changes in lighting conditions. Both of these problems stem from over-reliance on information from a single sensor, failing to provide consistently reliable positioning information in complex and dynamically changing orchard environments. Therefore, multimodal fusion methods are needed to improve the robustness and accuracy of overall positioning.

[0005] 2. Challenges of cross-temporal and spatial robot operations:

[0006] Orchard environments exhibit significant spatiotemporal dynamics. In the temporal dimension, robots need to perform tasks across different seasons, years, and even different times of day. Factors such as tree growth, fruit harvesting, and seasonal leaf fall can cause significant differences in sensor data (point clouds, images) from the same location at different times. Furthermore, changes in lighting and weather exacerbate the difficulties of feature extraction and matching.

[0007] In terms of spatial dimension, the orchard exhibits significant spatial heterogeneity. Different plots vary in fruit tree varieties, planting row spacing, tree age, and canopy morphology. Furthermore, the undulating terrain, furrow structure, and non-standardized distribution of obstacles contribute to a complex and non-uniform spatial structure. This spatial heterogeneity makes it difficult to directly generalize models trained on features from a single region or to extract key points for localization to other areas of the orchard.

[0008] Existing methods are mostly based on static prior maps for positioning, which lacks the ability to adapt to the joint changes in the environment across time and space. Therefore, in long-term operations across seasons and regions, problems such as matching failure and decreased positioning accuracy are very likely to occur, making it difficult to guarantee the continuous reliability of global positioning.

[0009] 3. Traditional point cloud registration algorithms are ineffective and lack semantic association capabilities:

[0010] Classical ICP algorithms rely on initial values, are prone to getting trapped in local optima, and suffer from low positioning accuracy. While the Normal Distribution Transform (NDT) algorithm is relatively efficient, its accuracy is greatly affected by grid size, making it difficult to balance accuracy and efficiency in large orchard scenarios, and it is typically only suitable for coarse registration. Point cloud processing methods based on semantic information require extensive data annotation and model training, resulting in poor real-time performance and weak generalization ability, making them difficult to deploy in vast and variable orchard environments. Existing methods lack mechanisms to utilize cross-spatial semantic association knowledge, failing to effectively combine prior knowledge of the orchard scenario (such as fruit tree arrangement structure and seasonal variation patterns) to assist registration decisions. This leads to insufficient robustness in registration during complex, dynamic, and long-term operations. Summary of the Invention

[0011] This application provides a global localization method and system for orchard operation robots based on a large language model-multi-metric ICP, which can improve the robustness and accuracy of localization.

[0012] The technical solution provided in this application is as follows:

[0013] Firstly, this application provides a global localization method for orchard operation robots based on a large language model-multi-metric ICP, comprising the following steps:

[0014] S1. Constructing a Prior Map: Before the localization task begins, a 3D point cloud map of the orchard environment is generated based on data obtained by a robot equipped with LiDAR and IMU sensors surveying the orchard environment. The 3D point cloud map is semantically segmented using the RandLA-Net model, which combines random sampling and local feature aggregation, and each point is assigned a semantic label. Based on the semantic labels, the map is divided into several candidate regions, and a structured text description and macroscopic geometric feature vector are generated and stored for each candidate region, ultimately forming the prior map used for the localization task.

[0015] S2. Real-time data acquisition, point cloud semantic segmentation and textual description: After the localization task begins, the robot equipped with LiDAR and IMU sensors conducts real-time exploration of the orchard environment to obtain source point clouds; the RandLA-Net model is used to perform semantic segmentation on the source point clouds to obtain the segmented semantic-geometric features;

[0016] S3, Coarse Registration Stage:

[0017] S3.1 Feature Filtering: Filtering a subset of features with information entropy lower than a threshold from the semantic-geometric features;

[0018] S3.2, Parallel processing of two branches: Based on the feature subset, the semantic branch uses a large language model to calculate the semantic matching degree between the structured text description of the current scene and the structured text description of the candidate region, and the geometric branch calculates the geometric feature similarity between the macroscopic geometric feature vector of the current scene and the macroscopic geometric feature vector of the candidate region through cosine similarity.

[0019] S3.3 Semantic-Geometric Fusion Scoring: Semantic uncertainty is calculated based on the category probability of RandLA-Net semantic segmentation in step S2, and geometric uncertainty is calculated based on the standard deviation of the matching residuals of macro-geometric feature vectors. Adaptive semantic scoring weight coefficients and adaptive geometric scoring weight coefficients are calculated based on semantic uncertainty and geometric uncertainty. The semantic matching degree and geometric feature similarity are fused by weighted summation to obtain a comprehensive score. The candidate region with the highest comprehensive score is selected as the optimal one, and the sub-map point cloud corresponding to the optimal candidate region in the prior map is extracted.

[0020] S3.4 Normal Distribution Transform (NDT) Registration: Based on the sub-map point cloud corresponding to the optimal candidate region and the initial pose provided by the tight coupling of the lidar-IMU, the optimal pose transformation between the source point cloud and the sub-map point cloud is solved by the NDT algorithm to obtain the coarse registration pose.

[0021] S4. Fine registration stage: Using the coarse registration pose as the initial value, a multi-metric optimization objective function containing four types of residual terms (distance, feature, intensity, and semantics) is constructed based on the iterative nearest point ICP algorithm. The optimal pose transformation is obtained by solving the objective function. The global localization result of the robot is obtained based on the optimal pose transformation.

[0022] In one possible implementation, step S1 generates a structured text description for each candidate region, including: generating standardized text based on the proportion of various semantic tags and geometric parameter features within the region; generating a macroscopic geometric feature vector, including: extracting the mean, covariance matrix eigenvalues, surface curvature statistics, and semantic tag spatial distribution entropy of the point cloud 3D coordinates within the region, and combining them with the straight-line fitting parameters of the fruit tree rows to form a standardized macroscopic geometric feature vector.

[0023] In one possible implementation, in step S3.1, the information entropy filtering formula is:

[0024] ;

[0025] in, For each type of orchard, specific features are extracted from the original point cloud. Features Information entropy For the first Candidate regions, For the set of all candidate regions, For feature-based Determine the candidate region for point cloud allocation The conditional probability;

[0026] The threshold formula is: ;in, For the threshold, The mean of the entropy of all feature information. is the standard deviation of the entropy of all feature information.

[0027] In one possible implementation, in step S3.3, the semantic uncertainty of a point P... The calculation formula is:

[0028] ;

[0029] in, The total number of semantic categories. For category index size; The point P obtained from RandLA-Net semantic segmentation belongs to the category The probability of;

[0030] For candidate regions The statistical value of the semantic uncertainty of all points within it is taken as Overall semantic uncertainty .

[0031] Geometric uncertainty The calculation formula is:

[0032] ;

[0033] ;

[0034] ;

[0035] in, Standard deviation, For geometric matching residuals; Candidate region The geometric score, This is the macroscopic geometric feature vector of the current frame. Candidate region ; macroscopic geometric eigenvectors;

[0036] Lower bound constraints are set for semantic and geometric uncertainties. , ,in This is the corrected semantic uncertainty. This is the corrected geometric uncertainty. For small positive numbers (such as ).

[0037] In one possible implementation, in step S3.3, the adaptive semantic scoring weight coefficients... Adaptive geometric scoring weighting coefficients ;

[0038] The formula for calculating the overall score is: + .

[0039] In one possible implementation, in step S3.4, the objective function for NDT registration is:

[0040] ;

[0041] ;

[0042] in, For coarse alignment of pose, This represents the total probability response of the transformed point cloud within the submap grid. The first point in the source cloud Points Points in the submap coordinate system after rotation and translation pose transformation and These are the mean and covariance matrices of the three-dimensional coordinates of all points within the grid, respectively.

[0043] In one possible implementation, in step S4, the multi-metric optimization objective function is:

[0044] ;

[0045] in, For optimal pose transformation, It is the optimal rotation matrix. It is the optimal translation vector; , , and These are the first points in the source point cloud. Points With the first in the target point cloud Corresponding points Distance, features, strength, and semantic metrics; , , and These are the weighting coefficients for distance, feature, strength, and semantic metrics, respectively.

[0046] Secondly, this application provides a global localization system for orchard operation robots based on large language model-multi-metric ICP, including: a memory and a processor;

[0047] The memory is used to store computer programs;

[0048] The processor is used to invoke the computer program to execute the method described above.

[0049] Thirdly, this application provides a computer-readable storage medium storing a computer program that, when run on an electronic device, causes the electronic device to perform the method described above.

[0050] Fourthly, this application provides a computer program product, including a computer program that, when run on an electronic device, causes the electronic device to perform the method described above.

[0051] The specific implementation methods of the second to fourth aspects of this application can refer to the implementation methods of the first aspect, and will not be elaborated here.

[0052] This application has the following beneficial effects:

[0053] 1. Significantly Improved Positioning Accuracy: This application innovatively adopts an LLM-enhanced multi-metric ICP (LLM-MDFI-ICP) algorithm, fusing four types of residual terms—distance, feature, intensity, and semantics—to construct an optimization objective function. Furthermore, it dynamically adjusts the weights of each metric based on point cloud semantic labels using LLM. Compared to traditional algorithms such as classic ICP and NDT, this scheme effectively overcomes the shortcomings of insufficient adaptability of a single metric and the tendency to get trapped in local optima, significantly improving positioning accuracy in complex orchard scenarios, especially in densely fruit-bearing areas and uneven terrain.

[0054] 2. Enhanced Environmental Adaptability and Robustness: A two-stage registration framework, from coarse to fine, is adopted. In the coarse registration stage, the optimal candidate region is selected through LLM semantic-geometric dual-branch fusion scoring, providing precise constraints for NDT registration and reducing sensitivity to initial pose. In the fine registration stage, semantic consistency measurement is incorporated to achieve adaptive registration for different terrain features such as ground, fruit trees, and obstacles. Simultaneously, a multimodal fusion mechanism relying on LiDAR and semantic context eliminates dependence on GPS signals and lighting conditions, effectively maintaining stable positioning performance in complex situations such as orchard canopy occlusion and seasonal environmental changes in unstructured orchards.

[0055] 3. Optimization of Computational Efficiency and Practicality: In the point cloud semantic segmentation stage, a lightweight solution combining RandLA-Net with seed point growth and BFS-RANSAC algorithms is adopted. This eliminates the need for large-scale manual annotation and complex model training, significantly reducing the system's computational resource requirements. Simultaneously, a high-discriminative-power features are extracted through an information entropy filtering mechanism, reducing redundant computation and ensuring the algorithm's real-time deployment feasibility on embedded agricultural robot platforms, thus better meeting the engineering needs of actual orchard operations.

[0056] 4. Cross-scene semantic understanding capability: This application innovatively introduces a large language model to endow the system with deep semantic understanding capability: (1) The LLM-driven semantic branch can understand the context information of the orchard scene and provide semantic prior knowledge for the registration process; (2) The adaptive weight allocation mechanism dynamically adjusts the contribution of each metric based on semantic labels to achieve semantic-guided intelligent registration; (3) Compared with the traditional pure geometric method, it has stronger cross-seasonal and cross-scene adaptability and can effectively cope with long-term environmental changes such as fruit tree growth and seasonal changes. Attached Figure Description

[0057] Figure 1 This application presents an embodiment of the LLM-MDFI-ICP global localization algorithm framework diagram (showing modules such as semantic segmentation, coarse registration (NDT), and fine registration (LLM-MDFI-ICP) and data flow, presenting the complete technical link of global localization).

[0058] Figure 2 : Schematic diagram of point cloud downsampling process (showing how RandLA-Net uses "random sampling + local feature aggregation" to downsample the point cloud from its initial size (N~10) 5 (The process of gradually downsampling to N~10² is presented to intuitively show the core logic of point cloud preprocessing before semantic segmentation.)

[0059] Figure 3 : Flowchart of fruit tree target point cloud segmentation (including steps such as BFS clustering and RANSAC line fitting, presenting the technical process of accurately separating structured fruit tree point clouds from the original vegetation point cloud).

[0060] Figure 4 Fine-grained registration flowchart (including four types of residual metrics: distance, feature, intensity, and semantics, showing the specific execution steps of the fine-grained registration stage of the LLM-MDFI-ICP algorithm);

[0061] Figure 5 A schematic diagram illustrating four residual metrics for optimizing the objective function in LLM-MDFI-ICP. Figure 5 (a), (b), (c), and (d) respectively demonstrate the visualization logic of the four types of residual metrics: distance, feature, intensity, and semantics, which correspond to the core indicators of multi-dimensional fusion registration. Detailed Implementation

[0062] To enable those skilled in the art to better understand the present application, the technical solution of the present application will be further described in detail below with reference to the embodiments and accompanying drawings.

[0063] It should be noted that the terms "comprising" and "having" and any variations thereof in this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units that are explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to such process, method, product, or device.

[0064] Furthermore, the terms "installation," "setup," "equipped with," "connection," "linked," and "socketing" should be interpreted broadly. For example, "connection" can be a fixed connection, a detachable connection, or an integral structure; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, or an internal connection between two devices, components, or parts. Those skilled in the art can understand the specific meaning of the above terms in this application according to the specific circumstances.

[0065] Due to the complexity of orchard environments (e.g., uneven terrain, dense fruit trees), the difficulty of sensor data processing, and the failure of existing algorithms to fully utilize multi-dimensional environmental information (e.g., point cloud intensity, geometric features), high-precision environmental modeling in dynamic open agricultural scenarios is challenging. Orchards, as typical unstructured, open, and dynamically changing agricultural scenarios, exhibit irregularities in terrain undulations, vegetation distribution, and artificial facilities, increasing the difficulty of high-precision environmental modeling. Furthermore, factors such as fruit tree growth, pruning, and seasonal changes lead to significant differences in the scene's appearance at different times, further exacerbating the deviation between the model and the actual environment. Regarding global localization with cross-seasonal map constraints, existing methods largely rely on static prior maps constructed in a single period, lacking effective mechanisms for fusing and updating map information under different seasons and environmental conditions, making it difficult to provide stable and reliable localization constraints in long-term operational tasks. In addition, fusing multiple types of information for high-precision registration faces difficulties such as complex data fusion strategies and excessive computational load, restricting the real-time performance and deployment feasibility of algorithms in practical orchard robot systems. Large Language Models (LLMs) have powerful semantic reasoning and zero-shot generalization capabilities, which can inject prior knowledge into point cloud registration. However, how to efficiently combine LLMs with point cloud registration remains a challenge.

[0066] In view of the above problems, this application provides a global localization method and system for orchard robots based on Large Language Model-Multi-Metric Iterative Closest Point (LLM-MDFI-ICP). Addressing the issues of severe GPS signal attenuation due to vegetation obstruction and inaccurate visual sensing features due to sudden changes in illumination and shadow interference in orchard environments, a two-stage mechanism guided by a large language model is constructed. This improves the robot's global localization accuracy, system fault tolerance, and long-term adaptability to dynamic environments in unstructured agricultural and forestry scenarios, and enhances localization robustness in degraded environments with multi-source perception. Considering that traditional iterative closest point and its variant algorithms are prone to getting trapped in local extrema in the structurally repetitive rows of fruit trees when an accurate initial pose is lacking, leading to registration failure, this application introduces semantic association constraints to reduce the sensitivity of point cloud registration to the initial pose and enhance the discriminativeness of feature matching, thereby improving the algorithm's global convergence ability and cross-scene robustness under inaccurate initial pose conditions. To avoid reliance on large-scale manual annotation and computationally intensive deep learning models, this application employs a lightweight, adaptive scene semantic understanding method that integrates geometric structure analysis and contextual reasoning using a large language model. While maintaining high semantic parsing efficiency, this significantly reduces the system's dependence on labeled data and computational resources, improving its deployment feasibility and practicality on embedded platforms. Addressing the differences in geometric structure and reflectivity among various features such as ground, fruit trees, and obstacles in an orchard scene, this application introduces a large language model as a semantic prior knowledge source, constructing a semantically aware multi-metric ICP registration optimization model to achieve semantically guided adaptive registration of multiple feature categories. By reasoning the functional importance of different features in the localization task using the large language model, the weight allocation of these features in four metrics—distance, feature, intensity, and semantic consistency—is dynamically adjusted. This achieves semantic-level adaptive optimization of the registration process, significantly improving the system's localization accuracy and stability in heterogeneous environments.

[0067] Specific embodiments according to this application will now be described with reference to the accompanying drawings.

[0068] Example 1:

[0069] The technical solution provided in this application is as follows:

[0070] Firstly, this application provides a global localization method for orchard operation robots based on a large language model-multi-metric ICP, comprising the following steps:

[0071] S1. Constructing a Prior Map: Before the localization task begins, a 3D point cloud map of the orchard environment is generated based on data obtained by a robot equipped with LiDAR and IMU sensors surveying the orchard environment. The 3D point cloud map is semantically segmented using a Random Sampling and Local Feature Aggregation Network (RandLA-Net) model, and point-by-point semantic labels are assigned. Based on the semantic labels, the map is divided into several candidate regions, and a structured text description and macroscopic geometric feature vector are generated and stored for each candidate region, ultimately forming the prior map used for the localization task.

[0072] In some embodiments, generating a three-dimensional point cloud map of the orchard environment includes: generating a globally consistent three-dimensional point cloud map of the orchard environment through synchronous positioning and mapping technology.

[0073] Among them, the Each candidate region can be denoted as , , The number of candidate regions; The corresponding stored structured text description can be denoted as: The macroscopic geometric eigenvector can be denoted as Structured text descriptions and macroscopic geometric feature vectors together constitute two types of descriptors used for localization.

[0074] In this step, considering the uneven point cloud density and complex vegetation structure in the orchard environment, the efficient large-scale point cloud semantic segmentation network RandLA-Net is used as the core. This network, through its innovative local feature aggregation module and random sampling strategy, can efficiently process dense point clouds in orchard scenes while maintaining low computational overhead. The RandLA-Net model has been pre-trained on a large-scale outdoor scene dataset and specifically fine-tuned with orchard data to ensure consistent semantic segmentation standards. This network can accurately predict point-by-point semantic labels for categories such as ground, fruit trees (including trunks and branches), and obstacles.

[0075] In some embodiments, candidate regions Generate structured text descriptions This includes: generating standardized text based on the proportion of various semantic tags within the region (e.g., "fruit trees account for 80%, ground accounts for 20%, obstacles account for 0%) and geometric parameter features (e.g., if tree row spacing, ground slope, canopy height) as (e.g., if trees account for 80%, ground slope is 7°, there are no obstacles, fruit tree row spacing is 2.5m, canopy height is 3.2m) as... This achieves text standardization, facilitating subsequent semantic matching using LLM (Large Language Model). [Candidate regions are defined here.] Generate macroscopic geometric feature vectors This includes: employing a global feature aggregation strategy to extract the mean (spatial center position), covariance (dispersion degree) matrix eigenvalues ​​of the 3D coordinates of point clouds within the region, surface curvature statistics (reflecting surface smoothness, its statistical value is such as standard deviation), and semantic label spatial distribution entropy, etc., combined with the straight-line fitting parameters of the fruit tree rows (slope or direction vector, intercept, reflecting spatial arrangement rules) to construct a standardized macroscopic geometric feature vector (numerical array, such as a 12-dimensional macroscopic set of this vector) as... Macro-geometric feature vectors focus on the macro-geometric properties of the point cloud in the region, avoiding interference from local details, ensuring the stability of the features, and facilitating the rapid calculation of geometric matching degree through "cosine similarity".

[0076] S2. Real-time data acquisition, point cloud semantic segmentation and textual description: After the localization task begins, the robot equipped with LiDAR and IMU sensors conducts real-time exploration of the orchard environment to acquire source point cloud (raw point cloud data); the RandLA-Net model is used to perform semantic segmentation on the source point cloud to obtain segmented semantic-geometric features; based on the segmented semantic-geometric features, a structured textual description and macroscopic geometric feature vector of the current scene (current frame) are generated;

[0077] In some embodiments, to enhance the structural consistency of the segmentation results in orchard scenarios, during the semantic segmentation of the source point cloud using the RandLA-Net model, a seed point ground growth method based on adaptive principal component analysis (PCA) for non-flat terrain is adopted to effectively handle the unique ridge and furrow terrain and undulating slopes of orchards, ensuring the accuracy of the ground model. Furthermore, the vegetation category point cloud output by RandLA-Net is processed a second time by combining breadth-first search (BFS) and random sampling consistency (RANSAC) algorithms. Based on the spatial distribution characteristics of row planting of fruit trees, the structured fruit tree point cloud that conforms to the agricultural planting pattern is accurately separated, resulting in the segmented semantic-geometric features.

[0078] S3, Coarse Registration Stage:

[0079] S3.1 Feature Filtering: Filtering a subset of features with information entropy lower than a threshold from the semantic-geometric features;

[0080] To improve the efficiency and accuracy of semantic matching, this step introduces a feature selection mechanism based on information entropy before generating the text description. This mechanism uses information entropy to select a subset of features with high information value for region localization from the semantic-geometric features obtained through segmentation. .

[0081] In some embodiments, the information entropy filtering formula is:

[0082] ;

[0083] in, For each type of orchard, specific features are extracted from the original point cloud. Features Information entropy For the first Candidate regions, For the set of all candidate regions, For feature-based Determine the candidate region for point cloud allocation In some embodiments, the conditional probability is calculated using the following formula:

[0084] ;

[0085] in, Representation of features With candidate regions The number of times they appear at the same time Features Total number of occurrences;

[0086] Retaining information entropy below a threshold The features constitute the optimized feature subset. This effectively eliminates ambiguous features that are common and do not help distinguish regions.

[0087] The threshold formula is:

[0088] ;

[0089] in, The mean of the entropy of all feature information. is the standard deviation of the entropy of all feature information.

[0090] Information entropy essentially measures the uncertainty of a feature in distinguishing scene categories—the lower the entropy value, the less uncertainty the feature has in determining which orchard candidate region a point cloud belongs to, and the stronger the feature's discriminative ability (higher discriminative power). Therefore, the above methods can be used to filter out a subset of features with high informational value for region localization.

[0091] S3.2, Parallel processing of two branches: Based on the feature subset, the semantic branch uses a large language model to calculate the semantic matching degree between the structured text description of the current scene and the structured text description of the candidate region, and the geometric branch calculates the geometric feature similarity between the macroscopic geometric feature vector of the current scene and the macroscopic geometric feature vector of the candidate region through cosine similarity.

[0092] This step is based on As input, semantic branching and geometric branching are processed in parallel, and the dual-branch score is output as a fusion decision.

[0093] In some embodiments, the process includes:

[0094] (1) Semantic branching: based on Generate a structured text description of the current scene. For each candidate region in the map, a pre-trained Large Language Model (LLM) is used to compute... Semantic match degree (semantic score) between it and its pre-stored structured text description:

[0095] ;

[0096] in Candidate areas in the map Pre-stored structured text description, for and Semantic association weights between them (·) represents the probability calculation function for the pre-trained large language model;

[0097] in, The calculation logic is as follows: and Concatenate into prompt text LLM processing Output the original score distribution of all candidate words (tokens). (logits); The original score distribution is obtained through the Softmax function. Convert to a probability distribution; extract the probability values ​​of specific candidate words (tokens) representing "semantic matching" from the distribution as the final output, where... It is the position index of a specific candidate word representing "semantic matching" within the probability distribution.

[0098] (2) Geometric branching: From Macroscopic geometric features are extracted, and the macroscopic geometric feature vectors of the current frame and the candidate regions are constructed. The geometric feature similarity between the current frame and each candidate region is quantified by cosine similarity.

[0099] ;

[0100] in, Candidate region The geometric score, This is the macroscopic geometric feature vector of the current frame. Candidate region The macroscopic geometric eigenvectors, The dot product of two macroscopic geometric eigenvectors. and These are the magnitudes of two macroscopic geometric eigenvectors, respectively, where the magnitudes are calculated using the L2 norm.

[0101] S3.3 Semantic-Geometric Fusion Scoring: Semantic uncertainty is calculated based on the category probability of RandLA-Net semantic segmentation in step S2, and geometric uncertainty is calculated based on the standard deviation of the matching residuals of macro-geometric feature vectors. Adaptive semantic scoring weight coefficients and adaptive geometric scoring weight coefficients are calculated based on semantic uncertainty and geometric uncertainty. The semantic matching degree and geometric feature similarity are fused by weighted summation to obtain a comprehensive score. The candidate region with the highest comprehensive score is selected as the optimal one, and the sub-map point cloud corresponding to the optimal candidate region in the prior map is extracted.

[0102] In some embodiments, semantic uncertainty Class probability calculation based on RandLA-Net semantic segmentation; semantic uncertainty of a point P. The calculation formula is:

[0103] ;

[0104] in, The total number of semantic categories. For the category index, traverse from 1 to... Consider each category one by one; The point P obtained from RandLA-Net semantic segmentation belongs to the category The probability of;

[0105] For candidate regions Take the statistical value (such as mean, median, or standard deviation) of the semantic uncertainty of all points within it as... Overall semantic uncertainty ;

[0106] Geometric uncertainty The standard deviation of the residuals based on macroscopic geometric feature vector matching is calculated using the following formula:

[0107] ;

[0108] in, Standard deviation, For geometric matching residuals, ;

[0109] To ensure stable computation even under extreme conditions such as semantic and geometric feature degradation, lower bound constraints are set for semantic and geometric uncertainties: , ,in This is the corrected semantic uncertainty. This is the corrected geometric uncertainty. For small positive numbers (such as ), to ensure and All are greater than zero.

[0110] In some embodiments, an adaptive semantic scoring weight coefficient is established. :

[0111] ;

[0112] Adaptive geometric scoring weight coefficient :

[0113] ;

[0114] The combined score is output by merging the two-branch scores using a weighted summation method.

[0115] + .

[0116] S3.4 Normal Distributions Transform (NDT) Registration: Using the sub-map point cloud corresponding to the optimal candidate region as the reference and the initial pose provided by the tight coupling of the lidar-IMU as the basis, the optimal pose transformation between the source point cloud and the sub-map point cloud is solved by the NDT algorithm to obtain the coarse registration pose.

[0117] In this step, the point cloud of the sub-map corresponding to the region with the highest comprehensive score (the optimal candidate region) in the global map is segmented (divided into multiple grids according to a fixed spatial size); the point cloud within each grid is solved. The normal distribution parameters include the mean of the three-dimensional coordinates of all points within the grid. Covariance Matrix To quantify the spatial distribution pattern of the point cloud within the grid; where It is the total number of points within the grid. It is the first to the second in the grid. The coordinates of a 3D point represent the spatial position of the first point within the grid.

[0118] According to the mean Covariance Matrix The spatial probability of any point within the grid is described by a normal distribution, and the formula is:

[0119] ;

[0120] In the formula, Show the first grid within the grid The three-dimensional coordinates of each point.

[0121] The core objective of NDT-based point cloud computing is to calculate the source point cloud acquired in real time by lidar. (Dynamic data), and the optimal pose transformation (rotation matrix) between the corresponding points of the sub-map with the highest comprehensive score selected in the coarse registration stage (static prior data). +Translation vector This approach maximizes the probability response function of the transformed real-time feature point cloud within the sub-map grid, ensuring the spatial distribution of the real-time point cloud is as consistent as possible with that of the sub-map point cloud. This provides precise initial pose constraints for subsequent fine registration. This represents the number of points within the source point cloud.

[0122] Among them, source cloud The initial pose is provided by a tightly coupled lidar-IMU, that is, the approximate pose of the robot relative to the working area is first obtained through sensor fusion, and then the initial pose is optimized by NDT to avoid registration failure due to excessive initial value deviation of NDT.

[0123] Solving the rotation matrix that aligns the source point cloud with the submap point cloud based on the NDT point cloud registration algorithm. Translation vector As coarse registration pose .

[0124] In some embodiments, the objective function for NDT registration is:

[0125] ;

[0126] ;

[0127] In the formula, This represents the total probability response of the transformed point cloud within the submap grid. The first point in the source cloud Points The objective function for the point in the sub-map coordinate system after rotation and translation pose transformation can be solved based on the Gauss-Newton method.

[0128] S4. Fine registration stage: Using the coarse registration pose as the initial value, a multi-metric optimization objective function containing four types of residual terms (distance, feature, intensity, and semantics) is constructed based on the ICP algorithm to obtain the optimal pose transformation; the global localization result of the robot is obtained based on the optimal pose transformation.

[0129] The robot's global localization result, including its 6-DOF global pose, can be calculated through the optimal pose transformation (rotation matrix R + translation vector t); where the translation vector... Corresponding to X-axis coordinates, Y-axis coordinates, and Z-axis coordinates; rotation matrix The three attitude angles, roll, pitch, and yaw, are obtained by Euler angle transformation (or quaternion calculation).

[0130] This step uses the coarse registration pose as the initial value and performs fine-grained pose optimization. A multi-metric ICP algorithm for LLM semantic enhancement is proposed, and a multi-metric optimization objective function is constructed:

[0131] ;

[0132] in, The final optimized pose transformation is obtained. It is the optimal rotation matrix. It is the optimal translation vector, used to describe the spatial pose relationship between the source point cloud and the target point cloud. , , and These are the weighting coefficients for distance, feature, intensity, and semantic metric, respectively. The four weights are adaptively adjusted based on the semantic labels of the orchard scene point cloud.

[0133] In some embodiments, the weighting coefficients satisfy normalization constraints: .

[0134] Residual term classification: The first three terms are geometric branch measures (distance, feature, intensity), and the last term is semantic branch measure (semantic consistency).

[0135] The residuals of each metric are defined as follows:

[0136] (1) Distance metric:

[0137] ;

[0138] In the formula, Represents the first point in the source point cloud. Points With the first in the target point cloud Corresponding points The distance residuals reflect the correspondence between point clouds; superscript and These represent the source point cloud and the target point cloud, respectively.

[0139] Measure the consistency of Euclidean distance between the transformed point cloud and the target point cloud.

[0140] (2) Feature measurement terms:

[0141] ;

[0142] In the formula, Represents the first point in the source point cloud. Points With the first in the target point cloud Corresponding points Characteristic residuals; Let R be the pose transformation operator (consisting of R and t), representing the rotation and translation operations on the point cloud; this formula calculates the matching error based on the feature descriptor, where This is a feature extraction function used to extract geometric feature descriptors from point clouds.

[0143] (3) Intensity measurement item:

[0144] ;

[0145] In the formula, Represents the first point in the source point cloud. Points With the first in the target point cloud Corresponding points The intensity residual; this formula utilizes the matching error of point cloud reflection intensity information, where This is the intensity feature function, used to extract the reflection intensity features of the point cloud.

[0146] (4) Semantic consistency measure (LLM semantic residual):

[0147] ;

[0148] In the formula, Represents the first point in the source point cloud. Points With the first in the target point cloud Corresponding points The semantic residual; this formula, based on the semantic association similarity index, quantifies the semantic matching error difference between real-time points and map points, where This is a semantic similarity evaluation function. This represents the maximum semantic similarity (used to normalize semantic residuals).

[0149] The Multi-Dimension Fusion Index (MDFI) is the core metric system designed in this application to optimize point cloud registration accuracy. It encompasses four key residual terms (dimensional metrics) that together constitute the optimization objective function of the LLM-MDFI-ICP algorithm: Distance residual measures the consistency of the Euclidean distance between the transformed source and target point clouds; Feature residual reflects the matching error based on point cloud geometric feature descriptors; Intensity residual reflects the matching error using lidar point cloud reflection intensity information; and Semantic residual evaluates the semantic similarity between real-time points and map points through LLM, quantifying semantic consistency. This multi-metric optimization objective function considers the semantic residual term, i.e., semantic enhancement is performed based on LLM. The four metrics are dynamically weighted through LLM to adapt to the unstructured and diverse terrain features (ground, fruit trees, obstacles) of the orchard environment.

[0150] In some embodiments, an adaptive weight allocation mechanism is employed:

[0151] The weights of the four types of residual terms are adaptively adjusted based on the semantic labels of the orchard scene point cloud. The weight calculation function is as follows:

[0152] = ;

[0153] = ;

[0154] = ;

[0155] = ;

[0156] In the formula, For weighted functions, It is any non-zero positive number; For the first The weight of the semantic label of each point on the corresponding metric. , , , These represent different semantic labels, namely the weights of obstacles, fruit trees, ground, and branches / leaves on their respective metrics (distance, feature, intensity, semantics), which can be determined manually based on experience. The branch / leaves term is implemented by LLM through semantic reasoning to determine the weights of fruit trees (…). The tree trunk and branches are subdivided into subdivisions to differentiate the weights of each metric.

[0157] The above weight calculation function adaptively adjusts the weights based on the point cloud semantic labels {obstacles, fruit trees, ground, branches and leaves}.

[0158] In some embodiments, the weight coefficients of semantic metrics are calculated using a weight calculation function, and then LLM performs inference optimization based on current scene features (semantic distribution features, such as tree density, ground flatness, and obstacle distribution) to make semantic matching more adaptable to the real-time environment. Based on scene characteristics, LLM outputs the weights of each semantic label on the corresponding metric, ultimately outputting the robot's precise 6-DOF global pose, namely X-axis coordinates, Y-axis coordinates, Z-axis coordinates, roll angle, pitch angle, and yaw angle.

[0159] Application Example 1: Global Localization Experiment of Unstructured Orchard Robot in the Field

[0160] The experimental platform is a mobile robot equipped with a 16-line LiDAR, IMU, and integrated navigation system. The prior offline map is constructed offline using SLAM technology. It is a point cloud map containing the 3D structure of the orchard scene and annotated with semantic labels such as obstacles, fruit trees, and ground.

[0161] In this application example, data acquisition and point cloud semantic segmentation are performed as follows:

[0162] The robot navigates through an unstructured orchard, with LiDAR acquiring 3D point cloud data in real time. For each frame of the point cloud, the RandLA-Net model is used for initial semantic segmentation, obtaining point-by-point semantic labels for categories such as ground, fruit trees, shrubs, and obstacles. To enhance segmentation consistency, an adaptive PCA-based seed point ground growing method is further employed to handle uneven terrain, and BFS and RANSAC algorithms are combined for secondary processing of the vegetation point cloud to accurately separate the structured fruit tree point cloud that conforms to the agricultural planting pattern.

[0163] Experimental results: In unstructured orchard environments, compared with classic methods such as ICP, GICP, NDT and PointNetLK, the two-stage registration enhanced by LLM in this application shows significant advantages.

[0164] In the coarse registration stage, LLM semantic matching effectively improves the accuracy of initial pose estimation and overcomes the matching ambiguity of traditional methods in repetitive structural environments.

[0165] Fine-grained registration stage: The LLM-driven semantic consistency metric and adaptive weighting mechanism significantly improve the registration accuracy and stability in heterogeneous environments.

[0166] Overall results: Compared with the comparison method, this application shows better positioning success rate and environmental adaptability in the complex scene of orchard.

[0167] Application Example 2: Global Positioning Experiment of Autonomous Spraying Robot in Orchard

[0168] The autonomous spraying robot platform is equipped with a 16-line LiDAR, an IMU inertial measurement unit, and a combined navigation system; it also integrates a spraying system. The prior map is a semantic point cloud map collected over multiple seasons, containing detailed semantic information such as the fruit tree canopy, trunk location, and row passages.

[0169] In this application example, the process of data acquisition and real-time analysis includes: the spraying robot autonomously moves through the orchard, the LiDAR collects environmental point clouds in real time, and IMU data is recorded simultaneously. A lightweight RandLA-Net model is used to perform initial semantic segmentation on each frame of point cloud. Basic categories such as ground, fruit tree canopy, and trunk are labeled. The adaptive PCA seed point growth method is used to correct the ground point labels in slope and furrow areas. Through BFS clustering and PANSAC linear fitting, the target fruit tree point cloud conforming to the row planting pattern is accurately separated. Interference points such as weeds and fallen leaves are removed, and finally, point cloud data with clear semantic labels is output, providing a foundation for locating and distinguishing spraying targets (leaf clusters, fruit areas, and interstitial areas).

[0170] In this application example, the adaptive weight allocation mechanism is as follows:

[0171] The weighting function is based on the semantic labels of the point cloud. , , , The weights are dynamically adjusted, with the semantic metric weights being optimized by LLM based on the semantic distribution features of the current scene. LLM outputs the importance weights of each semantic category in the registration based on scene characteristics such as tree density, ground flatness, and obstacle distribution, and finally outputs a high-precision pose relative to the spraying target in real time.

[0172] Experimental Results: In unstructured orchard environments, when applied to the global localization of autonomous spraying robots, this application addresses the shortcomings of the ICP algorithm, such as sensitivity to initial pose and tendency to get trapped in local optima between orchard rows. It provides prior knowledge of orchard row matching through LLM semantic branches, reducing dependence on initial pose and improving the coarse registration accuracy of the spraying target area. To address the GICP algorithm's reliance on geometric characteristics and insufficient robustness under obstacle interference, it incorporates LLM semantic consistency metrics to accurately distinguish target trees from obstacles, ensuring registration stability during spraying. Furthermore, to address the limitations of the NDT algorithm, such as sensitivity to grid parameters and insufficient accuracy in densely populated orchard areas, it employs a "LLM optimal region selection + multi-metric fine registration" strategy, significantly improving target area localization accuracy and optimizing spray coverage and precision. In addition, the localization results can directly guide the spraying module to dynamically adjust operational parameters, avoiding spraying in non-target areas and solving the problems of "overspraying" and "missed spraying" in traditional spraying robots. This simultaneously improves operational efficiency and agricultural utilization, fully meeting the precision operation technology requirements of autonomous spraying robots in orchards.

[0173] Example 2:

[0174] This embodiment provides a global localization system for orchard operation robots based on large language model-multi-metric ICP, including: a memory and a processor;

[0175] The memory is used to store computer programs;

[0176] The processor is configured to invoke the computer program to execute the method as described in Embodiment 1.

[0177] Example 3:

[0178] This embodiment provides a computer-readable storage medium storing a computer program. When the computer program is run on an electronic device, it causes the electronic device to perform the method described in Embodiment 1.

[0179] Example 4:

[0180] This embodiment provides a computer program product, including a computer program that, when run on an electronic device, causes the electronic device to perform the method described in Embodiment 1.

[0181] The specific implementation of the system, electronic device, computer-readable storage medium, and computer program product provided in this application can be referred to the specific embodiments of the above methods, and will not be repeated here.

[0182] Obviously, those skilled in the art should understand that the various units or steps of this application described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using computer-executable program code, thereby storing them in a storage device for execution by a computing device, or fabricating them separately as individual integrated circuit modules, or fabricating multiple modules or steps into a single integrated circuit module. Thus, this application is not limited to any particular combination of hardware and software.

[0183] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A global positioning method for orchard working robots based on large language model-multiple metric ICP, characterized in that, The method comprises the following steps: S1, constructing a priori map: before the positioning task starts, based on the data obtained by the robot carrying a laser radar and an IMU sensor to explore the orchard environment, a three-dimensional point cloud map of the orchard environment is generated; the three-dimensional point cloud map is segmented by using a random sampling and local feature aggregation network RandLA-Net model, and a point-by-point semantic label is given; based on the semantic label, the map is divided into a plurality of candidate regions, and a structured text description and a macro-geometric feature vector are generated and stored for each candidate region, and finally a priori map for the positioning task is formed; S2, real-time data acquisition, point cloud semantic segmentation and text description: after the positioning task starts, the robot carrying the laser radar and the IMU sensor is used to explore the orchard environment in real time to obtain the source point cloud; the source point cloud is segmented by using the RandLA-Net model to obtain the segmented semantic-geometric features; Based on the segmented semantic-geometric features, a structured text description and a macro-geometric feature vector of the current scene are generated; S3, coarse registration stage: S3.1, feature selection: a feature subset with an information entropy lower than a threshold is selected from the semantic-geometric features; S3.2, double-branch parallel processing: based on the feature subset, the semantic branch calculates the semantic matching degree of the structured text description of the current scene and the structured text description of the candidate region, and the geometric branch calculates the geometric feature similarity between the macro-geometric feature vector of the current scene and the macro-geometric feature vector of the candidate region by using the cosine similarity; S3.3, semantic-geometric fusion scoring: based on the class probability of the RandLA-Net semantic segmentation in step S2, the semantic uncertainty is calculated, and based on the standard deviation of the matching residual of the macro-geometric feature vector, the geometric uncertainty is calculated; based on the semantic uncertainty and the geometric uncertainty, the adaptive semantic scoring weight coefficient and the adaptive geometric scoring weight coefficient are calculated, the semantic matching degree and the geometric feature similarity are fused by weighted summation to obtain a comprehensive score, and the candidate region with the highest comprehensive score is selected as the optimal one, and the sub-map point cloud corresponding to the optimal candidate region in the prior map is extracted; S3.4, normal distribution transformation NDT registration: taking the sub-map point cloud corresponding to the optimal candidate region as the reference and taking the initial pose provided by the laser radar-IMU tight coupling as the basis, the optimal pose transformation of the source point cloud and the sub-map point cloud is solved by the NDT algorithm to obtain the coarse registration pose; S4, fine registration stage: taking the coarse registration pose as the initial value, a multi-metric optimization objective function containing distance, feature, intensity and semantic residual terms is constructed based on the iterative closest point ICP algorithm, and the optimal pose transformation is solved; based on the optimal pose transformation, the global positioning result of the robot is obtained.

2. The method of claim 1, wherein, In the step S1, the structured text description is generated for each candidate region, including: generating a standardized text based on the proportion of each type of semantic label and the geometric parameter characteristics in the region; and generating a macroscopic geometric feature vector, including: extracting the mean value of the point cloud three-dimensional coordinates, the eigenvalue of the covariance matrix, the surface curvature statistical value, and the semantic label space distribution entropy in the region, and combining the straight line fitting parameters of the fruit tree row to form a standardized macroscopic geometric feature vector.

3. The method of claim 1, wherein, In the step S3.1, the information entropy screening formula is: ； wherein, is the information entropy of the feature , is the information entropy of the feature , is the i-th candidate region, is the set of all candidate regions, is the conditional probability of determining the point cloud belonging to the candidate region based on the feature , . The threshold formula is: ; wherein, is a threshold value, is a mean value of all feature information entropy, is a standard deviation of all feature information entropy.

4. The method of claim 1, wherein, In step S3.3, the semantic uncertainty of a point P is calculated The formula for calculating the semantic uncertainty of a point P is: ； wherein, is the total number of semantic classes, is the number of class indices; is the probability that a point P obtained by RandLA-Net semantic segmentation belongs to class . for the candidate region , taking the statistical value of the semantic uncertainty of all points inside it as the semantic uncertainty of the whole ; Geometric uncertainty The formula for calculating the geometric uncertainty is: ； ； ； wherein, is the standard deviation, is the geometric matching residual; is the macro-geometric feature vector of the candidate region is the geometric score of the candidate region, is the macro-geometric feature vector of the current frame, is the macro-geometric feature vector of the candidate region is the macro-geometric feature vector of the candidate region Setting lower bounds on semantic and geometric uncertainty , where is the revised semantic uncertainty, is the revised geometric uncertainty, is a small positive number.

5. The method of claim 4, wherein, In step S3.3, the adaptive semantic score weight coefficient ; Adaptive geometry scoring weight coefficients ; The comprehensive score calculation formula is: + .

6. The method of claim 1, wherein, In the step S3.4, the objective function of the NDT registration is: ； ； wherein, is the coarse registration pose, denotes the total probability response of the transformed point cloud in the submap grid, is the i-th point in the source point cloud, is the i-th point in the source point cloud, is the i-th point in the source point cloud after rotation and translation pose transformation in the submap coordinate system, and are the mean and covariance matrix of the three-dimensional coordinates of all points in the grid, respectively.

7. The method of claim 1, wherein, In the step S4, the multi-measure optimization objective function is: ； wherein, is the optimal pose transformation, is the optimal rotation matrix, is the optimal translation vector; , , and are the distance, feature, intensity and semantic measure terms for the i-th point in the source point cloud and the j-th corresponding point in the target point cloud, respectively; , , and are the distance, feature, intensity and semantic measure terms for the i-th point in the source point cloud and the j-th corresponding point in the target point cloud, respectively; , , and are the weighting coefficients for the distance, feature, intensity and semantic measure terms, respectively.

8. A global positioning system for orchard working robots based on large language model-multiple metric ICP, characterized in that, including: a memory and a processor; the memory is configured to store a computer program; the processor is configured to call the computer program to execute the method in any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that, The computer program is stored in the computer readable storage medium, and when the computer program runs on the electronic device, the electronic device implements the method in any one of claims 1 to 7.

10. A computer program product comprising a computer program, characterized in that, The computer program runs on the electronic device, and the electronic device implements the method in any one of claims 1 to 7.

Images