Pesticide spraying robot positioning method and system based on graph neural network multi-source fusion

Abstract

The invention relates to the technical field of path planning, in particular to a pesticide spraying robot positioning method and system based on graph neural network multi-source fusion. Performing data preprocessing based on the acquired multi-source sensor information; graph node multi-modal data embedding is carried out on the preprocessed information based on perception information mapping; a multi-modal space-time diagram structure is built based on graph node multi-modal data embedding; performing graph feature propagation and fusion based on space-time attention weighting; and carrying out positioning calculation and path decision on the fused features based on spatial-temporal feature constraints. According to the method, a sensor observation graph structure is constructed, multi-source information such as GPS, IMU, vision, radar and the like is modeled in a node form, and nonlinear feature fusion is realized through graph convolution and an attention mechanism.

Description

Technical Field

[0001] This invention relates to the field of path planning technology, and in particular to a method and system for locating a spraying robot based on multi-source fusion of graph neural networks. Background Technology

[0002] With the development of smart agriculture, orchard spraying robots are widely used in tasks such as fruit tree spraying, pest and disease control, and plant protection inspection.

[0003] High-precision positioning is one of the key technologies for achieving accurate spraying and path reuse. However, the orchard environment has obvious unstructured and dynamically changing characteristics: dense tree distribution, severe foliage obstruction, uneven ground surface, and GPS signals are often reflected or interrupted; at the same time, uneven tree arrangement and variable lighting conditions lead to unstable visual SLAM positioning features. Traditional multi-source fusion positioning methods (such as VINS systems based on extended Kalman filter (EKF) or graph optimization) can fuse GPS, IMU, visual, and wheel speed information, but their feature fusion strategies are mostly linear or statistical weight allocations, making it difficult to capture the nonlinear correlations and spatiotemporal dependencies between complex sensors.

[0004] Existing orchard spraying robots face technical challenges such as low accuracy of multi-source positioning data fusion, poor system robustness, and insufficient real-time performance when performing autonomous spraying operations in complex orchard environments. These challenges make it difficult to meet the requirements of precision agriculture for centimeter-level positioning accuracy and continuous operation stability.

[0005] First, the unreliability of a single positioning source is a significant issue. In orchards, GPS / RTK signals are easily affected by canopy obstruction and multipath reflection, resulting in a significant decrease in positioning accuracy. Visual SLAM systems are prone to failure under conditions of changing lighting, repetitive textures, or highly reflective leaf surfaces. While IMUs and wheel speed odometry can compensate for positioning drift in the short term, errors accumulate over time. Furthermore, most existing multi-source fusion algorithms employ linear weighted fusion methods such as Extended Kalman Filter (EKF) or graph optimization. These methods assume that the noise from each sensor follows a Gaussian distribution and has fixed weights, failing to accurately model the nonlinear correlations and dynamic confidence changes between different modalities. This leads to poor fusion results under complex terrain or sudden interference.

[0006] Secondly, the spatiotemporal asynchrony and feature mismatch of sensor data are serious problems. In the orchard operating environment, the sampling frequencies of various sensors (GPS, IMU, vision, radar) differ significantly, and are affected by network latency and mechanical vibration, resulting in temporal misalignment and spatial drift of data. Traditional synchronization methods are unable to guarantee temporal consistency. In addition, the complex terrain in orchards, with undulating ground, mud, or slope changes causing wheel slippage and attitude deviations, distorts the features of IMU and odometer data, thus affecting the overall fusion accuracy.

[0007] Furthermore, existing fusion methods lack environmental adaptability and semantic understanding capabilities. In unstructured scenarios like orchards, the distribution of environmental features changes with the seasons and the growth status of fruit trees, making it difficult for traditional localization algorithms to operate stably over the long term. Simultaneously, existing algorithms do not fully utilize stable semantic features such as tree trunks, tree rows, and ground markers, failing to maintain globally consistent localization in repetitive or similar scenarios.

[0008] To address the above problems, there is an urgent need to propose a multi-source fusion localization method based on graph neural networks (GNNs). Summary of the Invention

[0009] To address the aforementioned problems, this invention provides a method and system for locating spraying robots based on multi-source fusion of graph neural networks.

[0010] In a first aspect, the present invention provides a method for locating a spraying robot based on multi-source fusion of graph neural networks, which adopts the following technical solution: A method for localizing a pesticide spraying robot based on multi-source fusion using graph neural networks, comprising: Acquiring information from multiple sensor sources based on a distributed time synchronization architecture; Data preprocessing is performed based on the acquired multi-source sensor information; Multimodal data embedding of graph nodes is performed on the preprocessed information based on perceptual information mapping; Constructing a multimodal spatiotemporal graph structure based on multimodal data embedding of graph nodes; Graph feature propagation and fusion based on spatiotemporal attention weighting; Based on spatiotemporal feature constraints, the fused features are used for localization calculation and path decision-making.

[0011] Secondly, a positioning system for a spraying robot based on graph neural network multi-source fusion includes... The data acquisition module is configured to acquire information from multiple sensor sources based on a distributed time synchronization architecture. The preprocessing module is configured to perform data preprocessing based on the acquired multi-source sensor information; The embedding module is configured to embed graph node multimodal data into the preprocessed information based on the perceptual information mapping. The spatiotemporal graph module is configured to construct a multimodal spatiotemporal graph structure based on the embedding of multimodal data from graph nodes; The fusion module is configured to perform graph feature propagation and fusion based on spatiotemporal attention weighting; The localization decision module is configured to perform localization calculations and path decisions based on the fused features according to spatiotemporal feature constraints. Thirdly, the present invention provides a computer-readable storage medium storing a plurality of instructions adapted for loading and execution by a processor of a terminal device of the aforementioned method for locating a spraying robot based on graph neural network multi-source fusion.

[0012] Fourthly, the present invention provides a terminal device, including a processor and a computer-readable storage medium, wherein the processor is used to implement various instructions; the computer-readable storage medium is used to store multiple instructions, the instructions being adapted to be loaded and executed by the processor to provide the described method for locating a spraying robot based on graph neural network multi-source fusion.

[0013] In summary, the present invention has the following beneficial technical effects: This invention constructs a sensor observation graph structure, modeling multi-source information such as GPS, IMU, vision, and radar as nodes, and achieves nonlinear feature fusion through graph convolution and attention mechanisms. This method can adaptively learn the correlation weights between sensors in an orchard environment, dynamically correct abnormal observation data, and improve overall localization robustness and accuracy. Simultaneously, it introduces a spatiotemporal synchronization mechanism and semantic-assisted node modeling to solve asynchronous fusion and feature drift problems, thereby achieving high-precision, adaptive localization of orchard spraying robots in complex environments.

[0014] This invention achieves centimeter-level positioning accuracy and highly robust navigation capabilities in complex orchard environments. The method combines multi-source sensor information with the nonlinear feature modeling capabilities of graph neural networks. Through spatiotemporal correlation modeling and adaptive fusion, it enables high-precision positioning and navigation of the robot. Simultaneously, the model can robustly handle potential GPS signal loss, visual feature drift, and terrain changes. Attached Figure Description

[0015] Figure 1 This is a schematic diagram of a pesticide spraying robot localization method based on graph neural network multi-source fusion according to Embodiment 1 of the present invention; Figure 2 This is a spatiotemporal diagram of the structure of Embodiment 1 of the present invention; Figure 3 This is a schematic diagram comparing the trajectories of the paths between rows of the same fruit tree in Embodiment 1 of the present invention. Detailed Implementation

[0016] The present invention will be further described in detail below with reference to the accompanying drawings.

[0017] Example 1 Reference Figure 1 This embodiment of a method for locating a spraying robot based on graph neural network multi-source fusion includes: (1) Data acquisition and correction The data acquisition module is a fundamental component of this method. Its main function is to synchronously acquire and preprocess multi-source sensor information on the robot platform, providing unified and time-consistent multimodal data input for the subsequent graph neural network fusion module. This module mainly includes four types of sensors: Global Positioning System (GPS / RTK), Inertial Measurement Unit (IMU), wheel speed encoder, and vision camera. The data acquisition module adopts a distributed time synchronization architecture, with each sensor node aligned using the ROS Time unified clock. The acquired data is then timestamped. Organized into multimodal data frames, each frame of data Represented as: in, It is the global coordinate position provided by GPS. These are the triaxial acceleration and triaxial angular velocity of the inertial measurement unit, respectively. It is the linear velocity measured by the wheel speed encoder. It is an RGB or grayscale image captured by the camera.

[0018] 1) GPS data representation GPS provides global position observation in unobstructed environments. : , in, For the actual location, This is a Gaussian noise term. It is a real-time calculated GPS confidence level used to address signal drift caused by fruit tree shading.

[0019] IMU outputs triaxial acceleration and angular velocity Its observation model is: , in, Let be the rotation matrix from the machine body to the world coordinate system. The gravity vector For bias terms, It is random noise.

[0020] 2) Wheel speed gauge data representation Wheel speed gauges provide the average linear velocity of the wheel system. Its model is: , in For the wheel radius, , These represent the angular velocities of the left and right wheels, respectively. The displacement increment can be estimated by combining this with the robot's kinematic model. , 3) Image data representation A visual camera provides two-dimensional image observations, and relative pose estimation is obtained through feature point extraction and matching. , in, This represents the pose estimation function based on feature matching.

[0021] (2) Data preprocessing supplementary data preprocessing process After completing multi-source synchronization, the data acquisition module unifies the data from each sensor into the robot base coordinate system through linear interpolation and extrinsic parameter calibration. The unified multi-source fusion observation can be expressed as: , superscript This indicates a transformation to the robot's base coordinate system.

[0022] In the complex natural environment of orchards, robots often face problems such as incomplete or temporary sensor data failures when performing spraying operations. These issues include GPS signal interruptions due to tree canopy obstruction, image degradation of visual sensors under strong light, water mist, or nighttime conditions, and wheel speedometer drift on slippery surfaces, all leading to abnormal sensor data. To alleviate the problem of missing or low-quality modalities in certain situations, this invention employs a modal validity assessment strategy, detecting the confidence level of all data modalities over a period of time. Specifically, for each sensor modality... The system periodically calculates its confidence level. This confidence level comprehensively considers data stability, noise variance, and time continuity. The specific calculation method is as follows: , in, For current observations, This is the predicted value from the previous moment. To observe the noise variance, This is the time continuity factor, indicating whether there is a sampling discontinuity in the current mode. It is 0 if sampling is lost. At that time, the mode is considered a missing mode or an untrusted mode. A gated mask matrix is ​​then used. This represents the available state of each modality at the current moment, specifically as follows: , In the input feature matrix During construction, features corresponding to the missing modes are masked: , Among them, This represents the Hadamard element-wise product operation. The formula means that if the confidence value of the current mode is less than the threshold, it is set to 0; if it is greater than the threshold, the mode is retained.

[0023] (3) Multimodal node embedding method In the multi-source fusion localization system based on graph neural networks of this invention, nodes are the basic units of the spatiotemporal graph structure, and their definition and feature representation directly determine the accuracy and expressive power of subsequent feature propagation and fusion. Addressing the characteristics of heterogeneous data from multiple sensors in orchard spraying robots, this invention proposes a unified node representation model that maps perceptual information from different modalities and time frames into high-dimensional embedding vectors of a unified dimension, thereby achieving cross-modal feature alignment and unified encoding.

[0024] 1) Time-dimensional embedding method In the time dimension, each moment The multi-source sensing state is abstracted into a set of nodes. ,in This represents the modes of GPS / RTK, IMU, camera, and wheel speedometer sensors. Each node... Corresponding to time The Each sensor's observations are processed by a multi-source fusion and time-series correction module, and the output is a vector. To facilitate cross-modal information fusion, this invention structures data from different modalities into multi-dimensional vectors containing common semantics such as position, velocity, attitude, acceleration, and confidence. This unified feature form is as follows: , in, Indicates location information, Indicates the increment of velocity or displacement. For acceleration, Indicates a gesture, Modal confidence reflects the reliability of the sensor data at the current moment. For the GPS mode, position and velocity can be directly obtained from GNSS calculations; the IMU mode provides angular velocity and acceleration information, which can be used to calculate short-time pose through integration; the wheel speed sensor mode provides wheel travel and angle, which are used to correct planar displacement; and although the visual mode cannot directly measure absolute position, relative displacement and direction can be calculated through optical flow tracking, and further combined with image semantic features to form a structured high-dimensional description.

[0025] 2) Intermodal embedding method To achieve unified processing of cross-modal features, this invention performs linear embedding and normalization on the original features of each node, mapping the feature spaces of different modalities to an embedding representation space of the same dimension. The specific transformation formula is as follows: , in, A mode-specific linear mapping matrix. For bias terms, For a unified embedding dimension, through training, the system can automatically learn the optimal mapping relationship for each modality feature, making the embedded node features comparable and complementary in a unified space.

[0026] In the complex operating environment of orchard spraying robots, relying solely on physical quantities such as geometric position, speed, or mileage is insufficient to fully reflect the dynamic changes and scene differences of the environment. Therefore, this invention introduces a semantic-assisted feature encoding mechanism in the node feature embedding stage. By performing high-level semantic parsing on visual and LiDAR data, the environmental semantic structure and geometric motion information are jointly modeled, thereby significantly improving the representation ability and task adaptability of spatiotemporal graph nodes. Specifically, the original images of the visual modality are first processed by the lightweight semantic segmentation network SegNet to extract pixel distribution probability maps of key categories such as fruit trees, branches and leaves, ground, nozzles, obstacles, and path boundaries. Subsequently, through region statistics and spatial projection, the segmentation results are transformed into semantic vectors: , in Indicates the first The semantic vector represents the proportion or confidence level of a region within the current field of view. This semantic vector reflects the type and structural features of the robot's current scene, such as dense tree canopies, open spaces, and passageways. Furthermore, to unify the representation of semantic and geometric features, the semantic vector of the visual modality is combined with the displacement estimated by visual odometry. ,speed Together they form an extended feature: , And it is mapped to a unified embedding space through a modality-specific linear transformation layer: , To suppress short-term noise and enhance the temporal smoothness of nodes, this invention introduces a sliding window averaging strategy after node feature encoding, utilizing multi-frame information within the temporal neighborhood for smoothing: , in The time window length is typically 3 to 5 frames, which can effectively smooth out the instantaneous instability caused by sudden disturbances or data jitter.

[0027] Through the above embedding process, all modal nodes form a unified node feature matrix in the input layer of the graph neural network: , The matrix is ​​stacked sequentially in the time dimension to form a sequence of node features. This provides an input foundation for subsequent spatiotemporal relationship modeling and graph feature propagation. The node definition and feature embedding method of this invention integrates the geometric, dynamic, and semantic features of multi-source heterogeneous sensors. Its key advantages are: achieving cross-modal feature alignment using a modality-specific mapping network; enhancing feature robustness by introducing confidence weights and temporal smoothing mechanisms; and improving environmental understanding through semantic enhancement mechanisms.

[0028] (4) Edge definition and spatiotemporal graph construction method After completing node definition and feature embedding, this invention further constructs a spatiotemporal graph structure model capable of characterizing the spatial dependence and temporal evolution relationships among multimodal sensors. This graph structure uses nodes as basic information carriers and edges as relational constraints to achieve joint modeling and feature propagation of the multi-source perception states of an orchard spraying robot in both temporal and modal dimensions. By introducing dynamic topology and weighting mechanisms in the spatiotemporal domain, the model can effectively describe the robot's temporal consistency and modal complementarity under different operating environments, thereby achieving highly robust and accurate localization estimation.

[0029] 1) Definition of graph structure In a graph structure, a set of nodes is defined as: , in Indicates a time step. Represents the number of sensor modes. The set of edges in the graph. It consists of two parts: temporal edges and modal edges, namely: , Temporal edges are used to describe the dynamic evolution of the same mode across consecutive time frames. For any mode If node Compared to the previous time node If temporal continuity exists, then temporal edges are established. Edge weights The degree of stability and reliability reflecting state changes is defined as: , in, For this mode at time The position vector, For smoothing coefficients, Modal confidence is used. This design allows moments with smooth motion and reliable data to receive higher weights during graph propagation, thereby suppressing the effects of anomalous jumps or noise. Furthermore, to enhance temporal modeling capabilities, this invention introduces a sliding time window mechanism during temporal edge construction, i.e., each node... It is not only connected to the previous frame node, but also to the past. The nodes of the frame establish weak connections: , This allows for explicit modeling of short-term historical information within a graph structure, capturing nonlinear motion trends, and improving the dynamic prediction stability of the robot on rugged orchard terrain.

[0030] 2) Establishment of intermodal edges Modal edges are used to describe the spatial correlation and information complementarity between different sensors within the same time frame. For any given time... If mode With mode If spatial correlation exists (e.g., overlapping relative observation areas in the same coordinate system), then modal edges are established. Modal edge weights reflect the degree of consistency between observations from two modes, and are defined as follows: , in, The spatial smoothing coefficient is... This represents the correlation coefficient between two modes, which can be obtained from sensor calibration parameters or through online learning.

[0031] To simultaneously consider temporal and modal interaction relationships, this invention jointly constructs the above two types of edges into a dynamically weighted graph: , in This represents the matrix of all edge weights. The graph is continuously updated in time, forming a spatiotemporal sequence graph structure. , Each Corresponding robot at time The full-modal perception graph has edge weights that are dynamically adjusted based on modal confidence, spatial distance, and temporal stationarity, enabling real-time structural adaptation.

[0032] Through the above design, the spatiotemporal graph constructed by this invention achieves dynamic coupling between modes in the spatial dimension and captures the evolutionary laws of motion states in the temporal dimension, forming a comprehensive perception structure with geometric, semantic, and temporal features. This structure can adaptively adjust topological weights in the complex environment of an orchard, achieving robust handling of occlusion, signal drift, and local mode loss, thereby providing a reliable data foundation and structural support for high-precision robot positioning, path planning, and autonomous control.

[0033] (5) Spatiotemporal graph feature propagation and fusion mechanism After constructing the spatiotemporal graph, this invention further proposes a spatiotemporal attention-weighted graph feature propagation and fusion mechanism to achieve efficient and robust information interaction and feature updates among multi-source sensors. The core idea of ​​this mechanism is to introduce dynamic attention weights into the graph neural network structure, enabling nodes to adaptively select the most reliable and relevant neighbor information during feature aggregation, thereby maintaining localization and perception accuracy even in modal instability, complex environments, or data gaps. During the graph propagation phase, the model performs graph feature propagation on each layer... The node state is updated, and the update process includes three parts: temporal propagation, modality fusion, and attention-weighted aggregation.

[0034] 1) Time-based graph information dissemination First, consider the continuous change of the same mode over time. For nodes... Its temporal neighborhood is ,in This represents the length of the time window. Time-series propagation is achieved through edge weights. Complete the weighted accumulation of historical information: , in, This is the temporal attention coefficient, used to measure the contribution of historical frames to the current frame. This coefficient is dynamically calculated using the temporal attention function, specifically as follows: , Where is the temporal similarity function, defined as: , This design enables the model to automatically focus on historical frames with consistent motion trends based on state similarity, thereby achieving adaptive modeling and anomaly filtering of continuous motion patterns of orchard robots.

[0035] 2) Information dissemination in the spatial dimension In the spatial dimension, nodes of different modalities exhibit complementary information and spatial correlation at the same time. This invention achieves cross-modal feature aggregation through a modal attention fusion mechanism. For nodes... Its modal neighborhood is The modal aggregation formula is: , in, These are the inter-modal attention weights, used to reflect the mutual trust between different modalities at the current time. The modal attention weights are jointly determined by feature correlation and modal confidence. in, The feature similarity function between modalities can be represented by dot product or cosine similarity. This mechanism enables the model to automatically suppress low-confidence modalities (such as when GPS signals are blocked or visually blurred) while enhancing the influence of high-confidence modalities, thereby achieving dynamic weighted fusion when perceptual quality is unbalanced.

[0036] 3) Modal data fusion After completing time-series propagation and modality fusion, the node state is jointly updated using a residual fusion strategy: , in, This represents a vector concatenation operation. This is the weight matrix of the fusion layer. This is a bias term. Residual connections ensure gradient stability during deep propagation while preserving original node information and preventing excessive feature smoothing. To further enhance the modeling capability of spatiotemporal dependencies, this invention integrates a temporal recursive unit after the graph convolutional propagation layer to achieve dynamic state transfer across layers: , This hybrid structure combines the spatial topological advantages of graph modeling with the temporal memory capabilities of recursive networks, enabling the robot to accurately capture pose changes, motion inertia, and environmental evolution characteristics during continuous operations. After multi-layer spatiotemporal propagation and attention-weighted updates, the final feature set of all nodes forms a globally fused representation: , (6) Location calculation and path decision After completing the spatiotemporal graph feature propagation and fusion, the system has obtained a unified representation feature set of the multimodal sensors at each time step. This invention further proposes a localization and path decision-making module based on spatiotemporal feature constraints, enabling orchard robots to achieve high-precision localization, continuous trajectory estimation, and autonomous path optimization in unstructured environments. This module consists of two core parts: a localization subsystem responsible for mapping fused features to the pose state in the global coordinate system; and a path decision-making subsystem that performs global and local path optimization based on predicted state and environmental semantic information, providing safe, smooth, and energy-efficient trajectories for tasks such as spraying or fertilizing.

[0037] 1) A hybrid solution framework based on graph optimization and filtering fusion The goal of the localization and solution phase is to base the solution on multi-source fusion features. To obtain the temporal pose estimation of the robot in the world coordinate system To improve accuracy and robustness, this invention employs a hybrid solution framework based on graph optimization and filtering fusion. First, the motion state vector is defined: , in , These are linear velocity and angular velocity, respectively. The system makes predictions based on the estimated state and control input from the previous step: , in For motion model, For control quantities (such as rotation angle and speed). This is process noise.

[0038] Multimodal observations are fused using a graph neural network to form spatiotemporal feature vectors. This feature encompasses multi-dimensional information such as location, orientation, and scene semantics. The system establishes an observation model through a nonlinear observation function: , in To address observation noise, and to obtain the optimal state estimate, this invention employs an improved extended Kalman filter (EKF), combined with spatiotemporal attention weights to dynamically adjust the covariance matrix: , , in The weighting factor is based on modal confidence and can adaptively adjust the observation update intensity according to the current sensor reliability, thus maintaining positioning stability even when GPS is obstructed or visually degraded. Furthermore, to reduce long-term drift errors, the system introduces a pose graph reconstruction mechanism based on graph optimization. This mechanism is constructed from historical keyframes. and constraint edges Composition factor diagram: , By minimizing the residual energy function Obtain a globally consistent pose estimate. Optimization employs the Levenberg-Marquardt iteration: , in, For Jacobian matrices, For the residual vector, denoted as the damping coefficient. This mechanism achieves joint estimation of short-time filtering and global graph optimization, balancing real-time performance and accuracy.

[0039] 2) A hybrid solution framework based on graph optimization and filtering fusion Obtaining continuous pose trajectory Subsequently, the system needs to generate the optimal path plan based on environmental constraints and task requirements. Due to the characteristics of orchard environments such as undulating terrain, random obstacles, and unstructured paths, traditional geometry-inspired algorithms (such as A* and Dijkstra's algorithm) are prone to getting trapped in local optima or becoming inefficient. Therefore, this invention proposes a path decision-making mechanism based on graph feature prediction and dynamic constraint optimization, combined with deep reinforcement learning, to achieve hierarchical path generation at both the global and local levels.

[0040] First, based on spatiotemporal graph fusion features Construct an environment accessibility estimation function: , in For the location of environmental grid nodes, For feature sampling function, The sigmoid function outputs a position. The probability of passage. A cost map is generated based on this probability field: , in Indicates the distance to the target point. As a cost of local curvature, These are the weight parameters. The path planning problem is transformed into cost minimization optimization: , The objective function simultaneously constrains path smoothness and cost minimization, achieving initial path generation through gradient optimization or sampling-based search (such as RRT*). To further enhance decision-making intelligence in dynamic environments, this invention introduces a reinforcement learning policy network into local path optimization. ,in This indicates the current state of the robot and the characteristics of its surrounding environment. This represents the control actions (speed and angle). Policy updates are based on the reward function. , Minimize negative expected reward: , in This serves as a discount factor. During training, the system generates multi-scene samples in a simulation environment using fused features, learning safe path strategies under conditions of dense obstacles, varying slopes, or fruit tree occlusion. Finally, the path decision module outputs a global reference trajectory. With local control sequence Combining the real-time pose estimation output from the positioning module, the system executes nonlinear model predictive control (MPC) at the control layer: , This system enables the robot to continuously track its planned trajectory within a limited predicted step length, while adaptively adjusting its speed and turning angle to cope with complex terrain and dynamic obstacles. The innovation of this localization and path decision-making module lies in the organic integration of graph neural network feature output, filtering and fusion calculation, and reinforcement learning path optimization. The spatiotemporal graph provides high-dimensional feature semantic support, making pose estimation and environmental modeling more accurate; filtering and graph optimization ensure the temporal continuity and consistency of the estimation results; and the policy optimization module achieves closed-loop fusion of perception and decision-making, enhancing the autonomy and safety of the orchard robot in real-world operating environments.

[0041] (6) Network training parameter settings and dataset 1) Introduction to the dataset To verify the effectiveness and robustness of the multimodal fusion localization and path planning system based on spatiotemporal graph neural networks, the experiment combined a self-collected orchard environment dataset with a publicly available multimodal mobile robot dataset for training and testing. The self-collected dataset covers complex natural environments such as dense fruit trees, undulating terrain, varying light levels, and partial shading. The data acquisition platform was a self-developed four-wheel-drive spraying robot equipped with an RGB-D camera, RTK-GPS, IMU, and wheel odometer module. Each data sequence includes image frames, inertial navigation acceleration and angular velocity, GPS coordinates, and control command signals. The sampling frequency was 10 Hz, totaling approximately 8 hours of continuous operation data, with 6 hours used for training and 2 hours for verification and testing. The publicly available dataset used was the Multimodal Mobile Robotic Dataset for a Typical Greenhouse, which is specifically designed for agricultural environments (such as greenhouses) and is characterized by collecting mobile robot sensor data in scenarios with plant shading, weak GNSS signals, and complex environments.

[0042] 2) Experimental Environment Setup Regarding system hyperparameter settings, the node embedding dimension of the spatiotemporal graph neural network is set to 128, and the time window length is [not specified]. Number of convolutional layers in the graph Each layer uses the LeakyReLU activation function and BatchNorm normalization; the temporal attention module employs a 4-head multi-head attention mechanism; modal attention confidence decay factor. Set to 0.85. The Adam optimizer is used during the training phase, with an initial learning rate of... Weight decay coefficient Batch size 16. A discount factor is used in the reinforcement learning path decision part. The initial exploration rate was 0.2, which decreased linearly to 0.05 with each iteration. All experiments were conducted on a computer equipped with an Intel i9-14900K and an NVIDIA RTX 4090 GPU.

[0043] 3) Performance Analysis To comprehensively verify the performance of the multimodal fusion localization and path planning system based on spatiotemporal graph neural network (GNN-Fusion) of this invention, system experiments were conducted on aspects such as localization accuracy, trajectory smoothness, robustness, real-time performance, and path optimization capability. All experiments were carried out on a self-built orchard dataset and the Citrus-Farm public dataset, and the comparison methods included the following three: (1) ORB-SLAM2; (2) EKF-Fusion (3) VINS-Fusion All algorithms were run with the same sensor input and a sampling frequency of 10 Hz. Evaluation metrics included positioning error RMSE, trajectory smoothness, attitude stability, and real-time processing speed (FPS).

[0044] Table 1 Experimental Verification and Performance Analysis method Modal type Average positioning error (m) trajectory smoothness <![CDATA[Attitude stability (rad 2 )]]> FPS ORB-SLAM2 Visual 0.26 0.024 0.081 48 EKF-Fusion IMU+GPS+Odom 0.18 0.014 0.073 62 VINS-Fusion Visual + IMU 0.12 0.009 0.047 39 This article's method Visual + IMU + GPS + Odom 0.045 0.005 0.022 55 As shown in Table 1, the single-modal ORB-SLAM2 achieves moderate accuracy in unobstructed rows of fruit trees, but the error increases significantly under varying lighting conditions or in areas with strong shading. EKF-Fusion suppresses some drift through linear fusion, but the error remains high in nonlinear motion. VINS-Fusion maintains a relatively smooth trajectory, but scale drift exists in unstructured orchard scenarios. The GNN-Fusion model proposed in this invention utilizes a spatiotemporal graph structure to characterize the nonlinear relationships between modalities, reducing the localization error to 0.045 m and improving trajectory smoothness by more than 44%, significantly outperforming other methods.

[0045] Figure 3 The paper presents a comparison of the trajectories of four methods along the same fruit tree row path. It can be observed that the ORB-SLAM2 trajectory exhibits significant drift in shaded areas; EKF-Fusion shows slight drift accumulation; VINS-Fusion is generally smooth but exhibits angle jumps at sharp turns; while the trajectory of the method presented in this paper almost coincides with the ground truth, and the curve shows good continuity.

[0046] To verify the stability of the algorithm under sensor failure conditions, a portion of the modal data was randomly masked during testing. The results are as follows: Table 2 Modal Missing Performance Analysis Missing mode EKF-Fusion error (m) VINS-Fusion error (m) Method error (m) in this paper none 0.18 0.12 0.045 GPS 0.34 — 0.07 Visual modality — 0.26 0.09 IMU 0.28 — 0.08 Experimental results show that the GNN-Fusion of this invention can still maintain centimeter-level positioning accuracy even in the case of modality loss. This capability is particularly important in orchard environments where signal obstruction, dust interference, or water mist imaging failure occur.

[0047] The path planning section compares the performance differences between traditional graph search algorithms (Dijkstra, A*) and the reinforcement learning control strategy of this invention. The experimental scenarios included two types of tasks: static tree row navigation and dynamic obstacle avoidance. The results are shown in Table 3 below: Table 3. Path Planning Performance and Energy Consumption Analysis method Average path length (m) Average energy consumption (J / m³) Success rate (%) Dijkstra 67.8 13.2 82.4 A* 65.3 12.7 85.6 This article's method 59.9 11.5 95.2 As shown in Table 3, the spatiotemporal feature-driven reinforcement learning strategy of this invention can dynamically adjust the trajectory direction based on environmental semantic information and local obstacles, reducing redundant turning and energy consumption while ensuring a safe distance. Compared with the classic search algorithm, the average path is shortened by about 11.3%, energy consumption is reduced by 9.7%, and the task success rate is improved by more than 10%.

[0048] Example 2 This embodiment provides a pesticide spraying robot localization system based on graph neural network multi-source fusion, including: The data acquisition module is configured to acquire information from multiple sensor sources based on a distributed time synchronization architecture. The preprocessing module is configured to perform data preprocessing based on the acquired multi-source sensor information; The embedding module is configured to embed graph node multimodal data into the preprocessed information based on the perceptual information mapping. The spatiotemporal graph module is configured to construct a multimodal spatiotemporal graph structure based on the embedding of multimodal data from graph nodes; The fusion module is configured to perform graph feature propagation and fusion based on spatiotemporal attention weighting; The localization decision module is configured to perform localization calculation and path decision based on spatiotemporal feature constraints on the fused features.

[0049] A computer-readable storage medium storing a plurality of instructions adapted for loading and execution by a processor of a terminal device of the aforementioned method for locating a spraying robot based on graph neural network multi-source fusion.

[0050] A terminal device includes a processor and a computer-readable storage medium, the processor being used to implement various instructions; the computer-readable storage medium being used to store multiple instructions, the instructions being adapted to be loaded and executed by the processor to provide a method for locating a spraying robot based on graph neural network multi-source fusion.

[0051] The above are all preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Therefore, all equivalent changes made in accordance with the structure, shape and principle of the present invention should be covered within the scope of protection of the present invention.

Claims

1. A method for localizing a pesticide spraying robot based on multi-source fusion of graph neural networks, characterized in that, include: Acquiring information from multiple sensor sources based on a distributed time synchronization architecture; Data preprocessing is performed based on the acquired multi-source sensor information; Multimodal data embedding of graph nodes is performed on the preprocessed information based on perceptual information mapping; Constructing a multimodal spatiotemporal graph structure based on multimodal data embedding of graph nodes; Graph feature propagation and fusion based on spatiotemporal attention weighting; Based on spatiotemporal feature constraints, the fused features are used for localization calculation and path decision-making.

2. The method for localizing a spraying robot based on multi-source fusion of graph neural networks according to claim 1, characterized in that, The multi-source sensor information acquisition based on the distributed time synchronization architecture includes data acquisition through the Global Positioning System (GPS), Inertial Measurement Unit (INS), Wheel Speed ​​Encoder, and Visual Camera. Each sensor node is aligned using the ROS Time unified clock, and the acquired data is timestamped. Organized into multimodal data frames, each frame of data Represented as: ， in, It is the global coordinate position provided by GPS. These are the triaxial acceleration and triaxial angular velocity of the inertial measurement unit, respectively. It is the linear velocity measured by the wheel speed encoder. These are RGB or grayscale images captured by a camera; GPS data is also represented. for : , in, For the actual location, This is a Gaussian noise term. It calculates GPS confidence in real time to compensate for signal drift caused by fruit tree shading; the IMU outputs three-axis acceleration. and angular velocity The observation model is: ， in, Let be the rotation matrix from the machine body to the world coordinate system. The gravity vector For bias terms, This is random noise; wheel speed gauge data provides the average linear velocity of the wheel train. The model is: in For the wheel radius, , These represent the angular velocities of the left and right wheels, respectively; image data is obtained through two-dimensional image observation provided by a vision camera, and relative pose estimation is obtained through feature point extraction and matching. ， in, This represents the pose estimation function based on feature matching.

3. The method for localizing a spraying robot based on multi-source fusion of graph neural networks according to claim 2, characterized in that, The data preprocessing based on the acquired multi-source sensor information includes unifying the sensor data to the robot base coordinate system through linear interpolation and extrinsic parameter calibration after multi-source synchronization is completed. The unified multi-source fusion observation is represented as: ， superscript This indicates a transformation to the robot's base coordinate system; then, a modal effectiveness evaluation strategy is employed for each sensor mode. Calculate its confidence level periodically. Taking into account data stability, noise variance, and time continuity using confidence levels, it can be expressed as: ， in, For current observations, This is the predicted value from the previous moment. To observe the noise variance, This is the time continuity factor, indicating whether there is a sampling discontinuity in the current mode. It is 0 if sampling is lost. At that time, the mode is considered a missing mode or an untrusted mode; subsequently, a gated mask matrix is ​​used. The available states of each modality at the current moment are represented as: In the input feature matrix During construction, features corresponding to the missing modes are masked: , in, This indicates the Hadamard element-wise product operation.

4. The method for localizing a spraying robot based on multi-source fusion of graph neural networks according to claim 3, characterized in that, The method of embedding graph nodes into multimodal data based on perceptual information mapping of preprocessed information includes, in the time dimension, embedding each time point... The multi-source sensing state is abstracted as a set of nodes. ,in Represents the GPS / RTK, IMU, camera, and wheel speed sensor modes, for each node. Corresponding to time The Each sensor's observations are processed by a multi-source fusion and time-series correction module, and the output is a vector. Then, the data from different modalities are structured into a unified feature form of multi-dimensional vectors with general semantics; then, the original features of each node are linearly embedded and normalized to map the feature spaces of different modalities to the same-dimensional embedding representation space. The specific transformation formula is as follows: in, A mode-specific linear mapping matrix. For bias terms, To achieve a unified embedding dimension, a semantic-assisted feature encoding mechanism is introduced. This mechanism performs high-level semantic parsing on visual and LiDAR data, jointly modeling the environmental semantic structure and geometric motion information. Specifically, the original visual images are processed by the lightweight semantic segmentation network SegNet to extract pixel distribution probability maps. Then, through region statistics and spatial projection, the segmentation results are transformed into semantic vectors. ， in Indicates the first The proportion or confidence level of the class region in the current field of view; and to unify the representation of semantic and geometric features, the semantic vector of the visual modality and the displacement estimated by visual odometry. ,speed Together they form an extended feature: And mapped to a unified embedding space through a modality-specific linear transformation layer: To suppress short-term noise and enhance the temporal smoothness of nodes, a sliding window averaging strategy is introduced after node feature encoding, utilizing multi-frame information within the temporal neighborhood for smoothing: in Given the time window length, all modal nodes eventually form a unified node feature matrix in the input layer of the graph neural network: The matrix, when stacked consecutively in the time dimension, forms a sequence of node features. .

5. The method for localizing a spraying robot based on multi-source fusion of graph neural networks according to claim 4, characterized in that, The construction of a multimodal spatiotemporal graph structure based on graph node multimodal data embedding includes defining the node set in the graph structure as follows: in Indicates a time step. Represents the number of sensor modes, the set of edges of the graph. It consists of two parts: temporal edges and modal edges, namely: Temporal edges are used to describe the dynamic evolution relationship of the same mode across consecutive time frames. For any mode... If node Compared to the previous moment node If temporal continuity exists, then temporal edges are established. Edge weight The degree of stability and reliability reflecting state changes is defined as: ,in, For this mode at time The position vector, For smoothing coefficients, This represents the modal confidence level.

6. The method for localizing a spraying robot based on multi-source fusion of graph neural networks according to claim 5, characterized in that, The multimodal spatiotemporal graph structure constructed by embedding multimodal data based on graph nodes also includes the establishment of intermodal edges. These intermodal edges are used to describe the spatial correlation and information complementarity between different sensors within the same time frame, for any given moment. If mode With mode If spatial correlation exists, then modal edges are established. Modal edge weights reflect the degree of consistency between the observations of two modes, and are defined as follows: ， in, The spatial smoothing coefficient is... The correlation coefficient between the two modes is represented by the edge coefficient. To simultaneously consider temporal and modal interactions, the two types of edges are jointly constructed into a dynamically weighted graph. in Representing all edge weight matrices, this graph is continuously updated in time, forming a spatiotemporal sequence graph structure: Each Corresponding robot at time The full-modal perception map.

7. The method for localizing a spraying robot based on multi-source fusion of graph neural networks according to claim 6, characterized in that, The method of graph feature propagation and fusion based on spatiotemporal attention weighting includes introducing dynamic attention weights into the graph neural network structure, and applying them to each layer during the graph propagation phase. The node state is updated, where the propagation of graph information in the time dimension includes updating the node's state. Its temporal neighborhood is ,in Given the time window length, time-series propagation is achieved through edge weights. Complete the weighted accumulation of historical information: in, The temporal attention coefficient measures the contribution of historical frames to the current frame. Spatial information propagation includes the complementary information and spatial correlation between different modal nodes at the same time. Cross-modal feature aggregation is achieved through a modal attention fusion mechanism for nodes. Its modal neighborhood is The modal aggregation formula is: in, The intermodal attention weights are jointly determined by feature correlation and modal confidence. in, The inter-modal feature similarity function is used; finally, modal data fusion includes, after completing temporal propagation and modal fusion, the node states are jointly updated through a residual fusion strategy: in, This represents a vector concatenation operation. This is the weight matrix of the fusion layer. This is a bias term.

8. The method for localizing a spraying robot based on multi-source fusion of graph neural networks according to claim 7, characterized in that, The method of performing localization calculation and path decision-making on the fused features based on spatiotemporal feature constraints includes obtaining a unified representation feature set of multimodal sensors at each time step after completing spatiotemporal map feature propagation and fusion. Then, location calculation and path decision-making based on spatiotemporal feature constraints are performed. The goal of the location calculation stage is to determine the path based on multi-source fusion features. To obtain the temporal pose estimation of the robot in the world coordinate system A hybrid solution framework based on graph optimization and filtering fusion is adopted, and the motion state vector is defined as follows: in , These are linear velocity and angular velocity, respectively, which are predicted based on the estimated state and control input: ,in For motion model, To control the quantity, The process noise is represented by multimodal observations, which are then fused using a graph neural network to form a spatiotemporal feature vector. And an observation model is established through a nonlinear observation function: in To observe noise and obtain the optimal state estimate, an improved extended Kalman filter is used, combined with a spatiotemporal attention weight to dynamically adjust the covariance matrix, expressed as: ， in The weighting factors are based on modal confidence. To reduce long-term drift errors, a graph optimization-based pose graph reconstruction mechanism is introduced, constructing a pose graph reconstruction based on historical keyframes. and constraint edges Composition factor diagram: ， By minimizing the residual energy function Obtain a globally consistent pose estimate.

9. The method for localizing a spraying robot based on multi-source fusion of graph neural networks according to claim 8, characterized in that, The method of performing localization calculation and path decision on the fused features based on spatiotemporal feature constraints also includes obtaining continuous pose trajectories. Then, based on environmental constraints and task requirements, an optimal path plan is generated, firstly based on spatiotemporal graph fusion features. Construct an environment accessibility estimation function: ， in For the location of environmental grid nodes, For feature sampling function, The sigmoid function outputs a position. The passability probability is used to generate a cost map based on the probability field: in Indicates the distance to the target point. As a cost of local curvature, The weight parameters are then used to transform the path planning problem into cost minimization optimization, with the objective function being: ， Finally, a reinforcement learning policy network is introduced into the local path optimization. ,in This indicates the current state of the robot and the characteristics of its surrounding environment. This indicates the control action, and the policy update is based on the reward function: Minimize negative expected reward: , in This is the discount factor.

10. A positioning system for a spraying robot based on multi-source fusion of graph neural networks, characterized in that, include: The data acquisition module is configured to acquire information from multiple sensor sources based on a distributed time synchronization architecture. The preprocessing module is configured to perform data preprocessing based on the acquired multi-source sensor information; The embedding module is configured to embed graph node multimodal data into the preprocessed information based on the perceptual information mapping. The spatiotemporal graph module is configured to construct a multimodal spatiotemporal graph structure based on the embedding of multimodal data from graph nodes; The fusion module is configured to perform graph feature propagation and fusion based on spatiotemporal attention weighting; The localization decision module is configured to perform localization calculation and path decision based on spatiotemporal feature constraints on the fused features.

Images