A pesticide spraying robot positioning method and system based on a graph neural network multi-source fusion

Abstract

The application relates to the technical field of path planning, in particular to a pesticide spraying robot positioning method and system based on a graph neural network multi-source fusion. Data preprocessing is carried out based on acquired multi-source sensor information; graph node multi-modal data embedding is carried out on the preprocessed information based on perception information mapping; a multi-modal space-time graph structure is constructed based on the graph node multi-modal data embedding; graph feature propagation and fusion are carried out based on space-time attention weighting; and positioning calculation and path decision are carried out on the fused features based on space-time feature constraints. The application constructs a sensor observation graph structure, models multi-source information such as GPS, IMU, vision and radar in the form of nodes, and realizes nonlinear feature fusion through graph convolution and attention mechanism.

Description

TECHNICAL FIELD

[0001] The application relates to the technical field of path planning, and in particular to a multi-source fusion positioning method and system for a pesticide spraying robot based on a graph neural network. BACKGROUND

[0002] With the development of intelligent agriculture, orchard pesticide spraying robots are widely used in tasks such as fruit tree spraying, pest control and plant protection inspection.

[0003] High-precision positioning is one of the key technologies for realizing precise spraying and path reuse. However, the orchard environment has obvious characteristics of unstructured and dynamic change: the distribution of trees is dense, the branches and leaves are severely blocked, the ground is uneven, and the GPS signal is often reflected or blocked; at the same time, the arrangement of fruit trees is uneven, and the lighting conditions are variable, which leads to unstable visual SLAM positioning features. Although traditional multi-source fusion positioning methods (such as EKF or VINS system based on graph optimization) can fuse GPS, IMU, vision and wheel speed information, the feature fusion strategy is mostly linear or statistical weight distribution, which is difficult to capture the nonlinear correlation and spatiotemporal dependence between complex sensors.

[0004] Existing orchard pesticide spraying robots face technical problems such as low accuracy of multi-source positioning data fusion, poor system robustness and insufficient real-time performance when performing autonomous spraying operations in complex orchard environments, making it difficult to meet the requirements of fine agriculture for centimeter-level positioning accuracy and continuous operation stability.

[0005] Firstly, the unreliability of a single positioning source is prominent. In the orchard, GPS / RTK signals are easily affected by tree canopy blocking and multipath reflection, resulting in a significant decrease in positioning accuracy; visual SLAM systems are prone to failure under conditions of changing light, repeated texture or strong reflective leaf surfaces; IMU and wheel odometry can compensate for positioning drift in the short term, but the error will accumulate over time. In addition, existing multi-source fusion algorithms mostly use linear weighted fusion methods such as extended Kalman filter (EKF) or graph optimization, which are based on the assumption that the noise of each sensor follows a Gaussian distribution and has a fixed weight. These methods cannot accurately model the nonlinear correlation and dynamic confidence changes between different modal data, resulting in poor fusion results under complex terrain or sudden disturbances.

[0006] Secondly, the problem of temporal and spatial asynchrony and feature mismatch of sensor data is serious. In the orchard operation environment, the sampling frequencies of various sensors (GPS, IMU, vision, radar) differ greatly, and are affected by network delay and mechanical vibration, resulting in temporal misalignment and spatial drift of data, making it difficult for traditional synchronization methods to ensure temporal consistency. In addition, the complex terrain in the orchard, such as ground undulation, muddy or slope changes, leads to wheel slip and attitude deviation, causing feature distortion of IMU and odometry data, and thus affecting the overall fusion accuracy.

[0007] Thirdly, the existing fusion method lacks environmental adaptability and semantic understanding ability. In the orchard, the distribution of environmental features changes with the season and the growth state of the fruit trees, and the traditional positioning algorithm is difficult to work stably for a long time. At the same time, the existing algorithm does not fully utilize the stable semantic features such as fruit tree trunk, tree row and ground sign, and cannot maintain consistent positioning in repeated or similar scenes.

[0008] To solve the above problems, a multi-source fusion positioning method based on graph neural network (GNN) is needed. SUMMARY

[0009] To solve the above problems, the application provides a multi-source fusion positioning method for a pesticide spraying robot based on a graph neural network.

[0010] In a first aspect, the application provides a multi-source fusion positioning method for a pesticide spraying robot based on a graph neural network, which adopts the following technical solution:

[0011] A multi-source fusion positioning method for a pesticide spraying robot based on a graph neural network, comprising:

[0012] Obtaining multi-source sensor information based on a distributed time synchronization architecture;

[0013] Data preprocessing based on the obtained multi-source sensor information;

[0014] Graph node multi-modal data embedding of the preprocessed information based on perception information mapping;

[0015] Constructing a multi-modal spatio-temporal graph structure based on graph node multi-modal data embedding;

[0016] Graph feature propagation and fusion based on spatio-temporal attention weighting;

[0017] Positioning calculation and path decision based on the fused features based on spatio-temporal feature constraints.

[0018] In a second aspect, a multi-source fusion positioning system for a pesticide spraying robot based on a graph neural network comprises

[0019] The data acquisition module is configured to obtain multi-source sensor information based on a distributed time synchronization architecture;

[0020] The preprocessing module is configured to perform data preprocessing based on the obtained multi-source sensor information;

[0021] The embedding module is configured to perform graph node multi-modal data embedding of the preprocessed information based on perception information mapping;

[0022] The spatio-temporal graph module is configured to construct a multi-modal spatio-temporal graph structure based on the graph node multi-modal data embedding;

[0023] The fusion module is configured to perform graph feature propagation and fusion based on the spatio-temporal attention weighting;

[0024] The positioning decision module is configured to perform positioning calculation and path decision on the fused features based on the spatio-temporal feature constraint

[0025] In a third aspect, the present application provides a computer readable storage medium, wherein a plurality of instructions are stored, the instructions being adapted to be loaded and executed by a processor of a terminal device to implement the positioning method of the pesticide spraying robot based on multi-source fusion of a graph neural network.

[0026] In a fourth aspect, the present application provides a terminal device, comprising a processor and a computer readable storage medium, the processor being used to implement the instructions; and the computer readable storage medium being used to store a plurality of instructions, the instructions being adapted to be loaded and executed by the processor to implement the positioning method of the pesticide spraying robot based on multi-source fusion of a graph neural network.

[0027] To sum up, the present application has the following beneficial technical effects:

[0028] The present application constructs a sensor observation graph structure, models multi-source information such as GPS, IMU, vision, radar, etc. in the form of nodes, and realizes nonlinear feature fusion through graph convolution and attention mechanism. This method can adaptively learn the correlation weight between sensors in the orchard environment, dynamically correct abnormal observation data, and improve the overall positioning robustness and accuracy. At the same time, the spatio-temporal synchronization mechanism and semantic auxiliary node modeling are introduced to solve the problems of asynchronous fusion and feature drift, so as to realize high-precision and adaptive positioning of the pesticide spraying robot in the complex environment.

[0029] The present application realizes centimeter-level positioning accuracy and high-robustness navigation ability in a complex orchard environment. This method combines multi-source sensor information and the nonlinear feature modeling capability of graph neural networks, realizes high-precision positioning and navigation of the robot through spatio-temporal correlation modeling and adaptive fusion, and the model can robustly handle possible GPS signal loss, visual feature drift and terrain changes. BRIEF DESCRIPTION OF DRAWINGS

[0030] Figure 1 is a schematic diagram of a positioning method of a pesticide spraying robot based on multi-source fusion of a graph neural network according to an embodiment of the present application;

[0031] Figure 2 is a spatio-temporal graph structure model diagram according to an embodiment of the present application;

[0032] Figure 3Fig. 1 is a schematic diagram of the trajectory of the same inter-row path of fruit trees of Example 1 of the present application. DETAILED DESCRIPTION

[0033] The present application will be further described in detail below with reference to the accompanying drawings.

[0034] Example 1

[0035] Referring to Figure 1 , the positioning method of the spraying robot based on the graph neural network multi-source fusion of the present embodiment comprises:

[0036] (1) Data acquisition and correction

[0037] The data acquisition module is the basic component of the present method, and the main function is to synchronously acquire and pre-process the multi-source sensor information on the robot platform, to provide unified and time-consistent multi-modal data input for the subsequent graph neural network fusion module. This module mainly includes four types of sensors: global positioning system (GPS / RTK), inertial measurement unit (IMU), wheel speed encoder, and vision camera. The data acquisition module adopts a distributed time synchronization architecture, and each sensor node is aligned through the ROS Time unified clock. The collected data is organized into multi-modal data frames according to the time stamp , and each frame of data is represented as:

[0038]

[0039] wherein, is the global coordinate position provided by the GPS, are the three-axis acceleration and three-axis angular velocity of the inertial measurement unit, respectively, is the linear speed measured by the wheel speed encoder, is the RGB or grayscale image collected by the camera.

[0040] 1) GPS data representation

[0041] The GPS provides global position observation in an unobstructed environment:

[0042] ,

[0043] wherein, is the true position, is the Gaussian noise term, is the real-time calculated GPS confidence to deal with signal drift caused by fruit tree obstruction.

[0044] The IMU outputs three-axis acceleration and angular velocity , and the observation model is:

[0045] ,

[0046] where, is the rotation matrix from the body to the world coordinate system, is the gravity vector, is the bias term, is the random noise.

[0047] 2) Wheel encoder data representation

[0048] Wheel encoders provide wheel train average linear velocity , which is modeled as:

[0049] ,

[0050] where is the wheel radius, , are the left and right wheel angular velocities, respectively. Combining with the robot kinematics model, the displacement increment can be estimated as:

[0051] ,

[0052] 3) Image data representation

[0053] Vision cameras provide two-dimensional image observations, and the relative pose estimation is obtained through feature point extraction and matching:

[0054] ,

[0055] where, denotes the pose estimation function based on feature matching.

[0056] (2) Data preprocessing supplement data preprocessing process

[0057] After the multi-source synchronization is completed, the data acquisition module unifies each sensor data to the robot base coordinate system through linear interpolation and external parameter calibration , and the unified multi-source fusion observation can be represented as:

[0058] ,

[0059] where the superscript denotes the conversion to the robot base coordinate system.

[0060] In the complex natural environment of orchards, robots often face problems such as incomplete or temporary sensor data failures when performing spraying operations. These issues include GPS signal interruptions due to tree canopy obstruction, image degradation of visual sensors under strong light, water mist, or nighttime conditions, and wheel speedometer drift on slippery surfaces, all leading to abnormal sensor data. To alleviate the problem of missing or low-quality modalities in certain situations, this invention employs a modal validity assessment strategy, detecting the confidence level of all data modalities over a period of time. Specifically, for each sensor modality... The system periodically calculates its confidence level. This confidence level comprehensively considers data stability, noise variance, and time continuity. The specific calculation method is as follows:

[0061] ,

[0062] in, For current observations, This is the predicted value from the previous moment. To observe the noise variance, This is the time continuity factor, indicating whether there is a sampling discontinuity in the current mode. It is 0 if sampling is lost. At that time, the mode is considered a missing mode or an untrusted mode. A gated mask matrix is ​​then used. This represents the available state of each modality at the current moment, specifically as follows:

[0063] ,

[0064] In the input feature matrix During construction, features corresponding to the missing modes are masked:

[0065] ,

[0066] Among them, This represents the Hadamard element-wise product operation. The formula means that if the confidence value of the current mode is less than the threshold, it is set to 0; if it is greater than the threshold, the mode is retained.

[0067] (3) Multimodal node embedding method

[0068] In the multi-source fusion localization system based on graph neural networks of this invention, nodes are the basic units of the spatiotemporal graph structure, and their definition and feature representation directly determine the accuracy and expressive power of subsequent feature propagation and fusion. Addressing the characteristics of heterogeneous data from multiple sensors in orchard spraying robots, this invention proposes a unified node representation model that maps perceptual information from different modalities and time frames into high-dimensional embedding vectors of a unified dimension, thereby achieving cross-modal feature alignment and unified encoding.

[0069] 1) Time dimension embedding method

[0070] At the time dimension, the multi-source perception state at each time is abstracted as a set of nodes , where represents GPS / RTK, IMU, camera and wheel speed sensor modalities. Each node corresponds to the sensor at time , whose observation is output as a vector after multi-source fusion and timing correction module. In order to facilitate cross-modal information fusion, the data of different modalities are structured into multi-dimensional vectors containing common semantics such as position, velocity, attitude, acceleration and confidence. The unified feature form is:

[0071] ,

[0072] wherein represents position information, represents velocity or displacement increment, is acceleration, represents attitude, is modal confidence, which is used to reflect the credibility of the sensor data at the current time. For GPS modality, position and velocity can be directly obtained by GNSS solution; IMU modality provides angular velocity and acceleration information, which can be calculated by integration to calculate short-time attitude; wheel speedometer modality provides wheel travel and angle, which is used to correct the plane displacement; while the visual modality cannot directly measure the absolute position, but can calculate the relative displacement and direction through optical flow tracking, and further combine image semantic features to form a structured high-dimensional description.

[0073] 2) Inter-modal embedding method

[0074] In order to realize the unified processing of cross-modal features, the original features of each node are linearly embedded and normalized, and the feature spaces of different modalities are mapped to the same dimension embedding representation space. The specific transformation formula is:

[0075] ,

[0076] wherein is a linear mapping matrix specific to the modality, is a bias term, is the unified embedding dimension. Through training, the system can automatically learn the optimal mapping relationship of each modality feature, so that the embedded node features have comparability and complementarity in the unified space.

[0077] In the complex working environment of orchard spraying robot, it is difficult to fully reflect the dynamic changes and scene differences of the environment by relying solely on physical quantities such as geometric position, speed or mileage. Therefore, the invention introduces a semantic auxiliary feature coding mechanism in the node feature embedding stage, and through high-level semantic analysis of visual and lidar data, the environmental semantic structure and geometric motion information are jointly modeled, thereby significantly improving the representation ability and task adaptability of the spatio-temporal graph nodes. Specifically, the original image of the visual modality is first processed by the lightweight semantic segmentation network SegNet, and the pixel distribution probability map of key categories such as fruit trees, branches and leaves, ground, spray head, obstacles and path boundary is extracted. Subsequently, through regional statistics and spatial projection, the segmentation result is converted into a semantic vector:

[0078] ,

[0079] wherein represents the proportion or confidence of the th region in the current field of view. The semantic vector can reflect the type and structural features of the scene where the robot is currently located, such as dense tree canopy area, open space, inter-row channel, etc. At the same time, in order to unify the semantic features and geometric features, the semantic vector of the visual modality and the displacement and velocity estimated by the visual odometry are concatenated to form an extended feature:

[0080] ,

[0081] and mapped to a unified embedding space through a modal-specific linear transformation layer:

[0082] ,

[0083] To suppress short-term noise and enhance the temporal smoothness of the nodes, the invention introduces a sliding window average strategy after the node feature coding, which uses the multi-frame information in the time neighborhood for smoothing processing:

[0084] ,

[0085] wherein is the length of the time window, usually taking 3 to 5 frames, which can effectively smooth the instantaneous unstable features caused by sudden disturbances or data jitter.

[0086] Through the above embedding processing, all modal nodes form a unified node feature matrix in the input layer of the graph neural network:

[0087] ,

[0088] After stacking the matrix in the time dimension, a node feature sequence , which provides the input basis for subsequent spatio-temporal relationship modeling and graph feature propagation. The node definition and feature embedding method of the invention has the key advantages of: using modal-specific mapping network to realize cross-modal feature alignment; introducing confidence weight and time smoothing mechanism to enhance feature robustness; and improving environmental understanding ability through semantic enhancement mechanism.

[0089] (4) Edge definition and spatio-temporal graph construction method

[0090] After completing the node definition and feature embedding, the invention further constructs a spatio-temporal graph structure model that can depict the spatial dependence and temporal evolution relationship between multi-modal sensors. This graph structure takes nodes as the basic information carrier and edges as the relationship constraints, realizing the joint modeling and feature propagation of the multi-source perception state of the orchard spraying robot in the time sequence and modal dimensions. By introducing dynamic topology and weighting mechanism in the spatio-temporal domain, the model can effectively describe the temporal consistency and modal complementarity of the robot in different working environments, thereby realizing high robustness and high precision positioning estimation.

[0091] 1) Graph structure definition

[0092] In the graph structure, the node set is defined as:

[0093] ,

[0094] wherein represents the time step, represents the number of sensor modalities. The edge set of the graph is composed of two parts: time edges and modal edges, that is:

[0095] ,

[0096] The time edge is used to describe the dynamic evolution relationship between the same modality in consecutive time frames. For any modality , if the node has temporal continuity with the previous time node , the time edge is established. The weight of the edge reflects the smoothness and credibility of the state change, and is defined as:

[0097] ,

[0098] wherein is the position vector of this modality at time , is the smoothing coefficient, is the modal confidence. This design makes the moment of smooth motion and reliable data get higher weight in graph propagation, so as to suppress the influence of abnormal jump or noise. In addition, to enhance the time modeling ability, the invention introduces a sliding time window mechanism when building the timing edge, that is, each node is not only connected with the previous frame node, but also with the nodes of the past frames:

[0099]

[0100] In this way, short-term historical information can be explicitly modeled in the graph structure, nonlinear motion trends can be captured, and the dynamic prediction stability of the robot on the rugged ground of the orchard can be improved.

[0101] 2) Inter-modal edge construction

[0102] The modal edge is used to describe the spatial correlation and information complementarity of different sensors in the same time frame. For any time , if the modal and the modal have spatial correlation (for example, the relative observation region in the same coordinate system has overlap), the modal edge is established. The weight of the modal edge reflects the consistency degree of the observation results of the two modalities, and is defined as:

[0103]

[0104] where is the spatial smoothing coefficient, is the correlation coefficient between the two modalities, which can be obtained according to the sensor calibration parameters or online learning.

[0105] In order to consider the temporal and modal interaction relationship at the same time, the invention jointly constructs the above two types of edges into a dynamic weighted graph:

[0106]

[0107] where is the weight matrix of all edges. This graph is continuously updated in time, forming a spatiotemporal sequence graph structure:

[0108]

[0109] Each corresponds to the full-modal perception graph of the robot at time . The weight of the edge is dynamically adjusted according to the modal confidence, spatial distance and temporal smoothness, realizing real-time structure adaptation.

[0110] ​​​​Through the above design, the spatiotemporal graph constructed by the application realizes dynamic coupling between modes in the spatial dimension and captures the evolution law of the motion state in the time dimension, forming a comprehensive perception structure with geometric, semantic and timing characteristics. The structure can adaptively adjust the topological weight in the complex environment of the orchard, realize robust processing of occlusion, signal drift and local mode loss, and thus provide a reliable data foundation and structural support for high-precision positioning, path planning and autonomous control of the robot.

[0111] (5) Spatiotemporal graph feature propagation and fusion mechanism

[0112] After the construction of the spatiotemporal graph, the application further proposes a graph feature propagation and fusion mechanism based on spatiotemporal attention weighting, which is used to realize efficient and robust information interaction and feature updating between multiple source sensors. The core idea of the mechanism is: by introducing dynamic attention weights in the graph neural network structure, the node can adaptively select the most credible and most relevant neighbor information when aggregating features, so as to maintain the positioning and perception accuracy in the case of unstable modes, complex environment or data loss. In the graph propagation stage, the model updates the node state of each layer , and the update process includes three parts: temporal propagation, mode fusion and attention weighted aggregation.

[0113] 1) Temporal propagation of graph information

[0114] First, consider the continuous change of the same mode in the time dimension. For a node , its temporal neighborhood is , where is the length of the time window. Temporal propagation completes the weighted accumulation of historical information through the edge weight :

[0115] ,

[0116] where is the temporal attention coefficient, which is used to measure the contribution of historical frames to the current frame. The coefficient is dynamically calculated by the temporal attention function, and the specific calculation method is:

[0117] ,

[0118] where is the temporal similarity function, which is defined as:

[0119] ,

[0120] This design enables the model to automatically focus on the historical frames with consistent motion trends according to the state similarity, thereby realizing adaptive modeling and abnormal filtering of the continuous motion pattern of the orchard robot.

[0121] 2)Spatial dimension information propagation

[0122] In the spatial dimension, different modal nodes have information complementarity and spatial correlation at the same time. The application realizes cross-modal feature aggregation through a modal attention fusion mechanism. For a node , its modal neighborhood is . The modal aggregation formula is:

[0123] ,

[0124] Where, is the inter-modal attention weight, which reflects the mutual trust degree between different modalities at the current time. The modal attention weight is determined by the feature correlation and the modal confidence:

[0125] Where, is the inter-modal feature similarity function, which can use dot product or cosine similarity. This mechanism enables the model to automatically suppress low-confidence modalities (such as GPS signals blocked or visual blur) while enhancing the influence of high-confidence modalities, thereby achieving dynamic weighted fusion when the perception quality is uneven.

[0126] 3)Modality data fusion

[0127] After completing the temporal propagation and modal fusion, the node state is updated through a residual fusion strategy:

[0128] ,

[0129] Where, represents the vector concatenation operation, is the fusion layer weight matrix, is the bias term. The residual connection ensures the stability of the gradient in the deep propagation process, while preserving the original node information to prevent feature over-smoothing. To further improve the modeling ability of spatial and temporal dependence, the application connects a temporal recurrent unit after the graph convolution propagation layer to realize dynamic state transfer across layers:

[0130] ,

[0131] This hybrid structure combines the spatial topological advantage of graph structure modeling and the time memory ability of recurrent network, enabling the robot to accurately capture the features of pose changes, motion inertia and environmental evolution in continuous operation. After multiple layers of spatio-temporal propagation and attention weighted update, the final feature set of all nodes forms a global fusion representation:

[0132] ,

[0133] (6) Positioning solution and path decision

[0134] After the completion of spatio-temporal graph feature propagation and fusion, the system has obtained a unified representation feature set of multi-modal sensors at each time step The application further proposes a positioning solution and path decision module based on spatio-temporal feature constraints, which realizes high-precision positioning, continuous trajectory estimation and autonomous path optimization decision of orchard operation robots in unstructured environments. The module consists of two core parts: one is the positioning solution subsystem, which is responsible for mapping the fused features to the pose state in the global coordinate system; the other is the path decision subsystem, which performs global and local path optimization based on the predicted state and environmental semantic information, and provides safe, smooth and energy-optimal trajectories for tasks such as pesticide spraying or fertilization.

[0135] 1) Hybrid solution framework based on graph optimization and filter fusion

[0136] The goal of the positioning solution stage is to obtain the time-series pose estimation of the robot in the world coordinate system based on multi-source fused features . To improve accuracy and robustness, the application adopts a hybrid solution framework based on graph optimization and filter fusion. First, define the motion state vector:

[0137] ,

[0138] where , are linear velocity and angular velocity, respectively. The system predicts based on the estimated state of the previous step and the control input:

[0139] ,

[0140] where is the motion model, is the control quantity (such as turning angle, speed), is the process noise.

[0141] Multi-modal observations are fused through a graph neural network to form a spatio-temporal feature vector . This feature contains multi-dimensional information such as position, direction, scene semantics, etc. The system establishes an observation model through a nonlinear observation function:

[0142] ,

[0143] where is the observation noise. To obtain the optimal state estimate, the application uses an improved extended Kalman filter (EKF) that dynamically adjusts the covariance matrix in combination with spatio-temporal attention weights:

[0144] ,

[0145] ,

[0146] where is the weight factor based on modal confidence, which can adaptively adjust the observation update strength according to the current sensor reliability, so as to maintain the positioning stability when the GPS is blocked or the vision is degraded. In addition, in order to reduce the long-term drift error, the system introduces a pose graph reconstruction mechanism based on graph optimization. A factor graph composed of historical key frames and constraint edges is constructed:

[0147] ,

[0148] The global consistent pose estimation is obtained by minimizing the residual energy function . The optimization adopts Levenberg-Marquardt iteration:

[0149] ,

[0150] where, is the Jacobian matrix, is the residual vector, is the damping coefficient. This mechanism realizes the joint estimation of short-term filtering and global graph optimization, and takes into account the real-time performance and accuracy.

[0151] 2) Hybrid solution framework based on fusion of graph optimization and filtering

[0152] After obtaining the continuous pose trajectory , the system needs to generate the optimal path planning according to the environmental constraints and task requirements. Due to the characteristics of the orchard environment such as terrain undulation, random obstacles, and unstructured path, traditional algorithms based on geometric heuristics (such as A* and Dijkstra) are prone to local optimization or low efficiency. Therefore, the present invention proposes a path decision mechanism based on graph feature prediction and dynamic constraint optimization, which combines the idea of deep reinforcement learning to realize hierarchical path generation globally and locally.

[0153] Firstly, based on the spatio-temporal graph fusion feature , an environmental passability estimation function is constructed:

[0154] ,

[0155] where is the position of the environmental grid node, is the feature sampling function, is the Sigmoid function, and the output represents the passability probability of the position . According to this probability field, a cost map is generated:

[0156] ,

[0157] wherein denotes the distance to the goal point, is the local curvature cost, is the weight parameter. The path planning problem is converted to a cost minimization optimization:

[0158] ,

[0159] The objective function simultaneously constrains the path smoothness and cost minimization, and the preliminary path generation is achieved by gradient optimization or sampling-based search (such as RRT*). To further improve the decision-making intelligence in dynamic environments, the invention introduces a reinforcement learning policy network in local path optimization wherein denotes the current robot state and the surrounding environment features, denotes the control action (speed and turning angle). The policy update is based on the reward function:

[0160] ,

[0161] Minimize the negative expected reward:

[0162] ,

[0163] wherein is the discount factor. During the training process, the system generates multi-scenario samples in the simulation environment using fused features, and learns the safe path strategy under the conditions of dense obstacles, slope changes, or fruit tree obstruction. Finally, the path decision module outputs the global reference trajectory and the local control sequence . Combined with the real-time pose estimation output by the positioning module, the system executes through nonlinear model predictive control (MPC) at the control layer:

[0164] ,

[0165] so that the robot continuously tracks the planned trajectory within a limited prediction step, while adaptively adjusting the speed and turning angle to respond to complex terrain and dynamic obstacles. The innovation of the positioning solution and path decision module lies in the organic combination of graph neural network feature output, filtering fusion solution, and reinforcement learning path optimization. The spatio-temporal graph provides high-dimensional feature semantic support, making the pose estimation and environment modeling more accurate; filtering and graph optimization ensure the continuity and consistency of the estimation results in time; and the strategy optimization module realizes the closed-loop fusion of perception-decision, enhancing the autonomy and safety of the orchard robot in real working environment.

[0166] (6) Network training parameter settings and data sets

[0167] 1) Introduction of data set

[0168] To verify the effectiveness and robustness of the multi-modal fusion positioning and path planning system based on the spatio-temporal graph neural network of the present application, the experiment adopts a combination of self-collected orchard environment data sets and public multi-modal mobile robot data sets for training and testing. The self-collected data set covers complex natural environments such as dense fruit trees, terrain undulations, light changes, and partial occlusions. The data collection platform is a self-developed four-wheel drive spraying robot equipped with an RGB-D camera, an RTK-GPS, an IMU, and a wheel odometer module. Each data sequence contains image frames, inertial navigation acceleration and angular velocity, GPS coordinates, and control command signals. The sampling frequency is 10 Hz, and the total continuous operation data is about 8 hours, of which 6 hours are used for training and 2 hours for verification and testing. The public data set uses the Multimodal Mobile Robotic Dataset for a Typical Greenhouse data set, which is designed for agricultural environments such as greenhouses, and is characterized by collecting mobile robot sensor data in scenarios with plant occlusions, weak GNSS signals, and complex environments.

[0169] 2) Experimental environment setup

[0170] In terms of system hyperparameter settings, the node embedding dimension of the spatio-temporal graph neural network is set to 128, the time window length , the number of graph convolution layers , the LeakyReLU activation function and BatchNorm normalization are used in each layer; the time attention module uses a 4-head multi-head attention mechanism; the modal attention confidence decay factor is set to 0.85. The Adam optimizer is used in the training stage, with an initial learning rate , a weight decay coefficient , and a batch size of 16. The reinforcement learning path decision-making part uses a discount factor , an initial exploration rate of 0.2, and a linear decay to 0.05 with iterations. All experiments are conducted on a computer equipped with an Intel I9-14900K and an NVIDIA RTX 4090 GPU.

[0171] 3) Performance analysis

[0172] To comprehensively verify the performance of the multi-modal fusion positioning and path planning system based on the spatio-temporal graph neural network (GNN-Fusion) of the present application, experiments were conducted from the aspects of positioning accuracy, trajectory smoothness, robustness, real-time performance, and path optimization ability. All experiments were conducted on self-built orchard data sets and Citrus-Farm public data sets, and the comparison methods included the following three:

[0173] (1) ORB-SLAM2; (2) EKF-Fusion (3) VINS-Fusion

[0174] All algorithms run at the same sensor input with a sampling frequency of 10 Hz. Evaluation metrics include localization error RMSE, trajectory smoothness, pose stability, and real-time processing rate FPS.

[0175] Table 1 Experimental verification and performance analysis

[0176] Method Modality type Mean positioning error (m) Trajectory smoothness radial stability 2 )]]> FPS ORB-SLAM2 Vision 0.26 0.024 0.081 48 EKF-Fusion IMU+GPS+Odom 0.18 0.014 0.073 62 VINS-Fusion Vision+IMU 0.12 0.009 0.047 39 The method herein Vision+IMU+GPS+Odom 0.045 0.005 0.022 55

[0177] As can be seen from Table 1, the single-mode ORB-SLAM2 can obtain moderate accuracy in the unobstructed fruit tree row, but the error significantly increases in the light change or strong obstruction area. EKF-Fusion suppresses part of the drift through linear fusion, but the error is still high in nonlinear motion. VINS-Fusion can maintain a relatively smooth trajectory, but there is a scale drift in the unstructured scene of the orchard. The GNN-Fusion model proposed in the application uses a spatio-temporal graph structure to depict the nonlinear relationship between modalities, and the localization error is reduced to 0.045 m, and the trajectory smoothness is improved by more than 44%, which is significantly better than other methods.

[0178] Figure 3 Trajectory comparison of four methods in the same fruit tree row path is shown. It can be observed that the ORB-SLAM2 trajectory has obvious drift in the obstruction area; EKF-Fusion has slight drift accumulation, VINS-Fusion is smooth overall but has angle jumps at sharp turns; and the trajectory of the method of the present application is almost coincident with the true value, and the curve continuity is good.

[0179] To verify the stability of the algorithm under sensor failure conditions, part of the modal data is randomly shielded for testing. The results are as follows:

[0180] Table 2 Performance analysis of modal absence

[0181] Missing modality EKF-Fusion error (m) VINS-Fusion error (m) The method herein error (m) None 0.18 0.12 0.045 GPS 0.34 — 0.07 Vision modality — 0.26 0.09 IMU 0.28 — 0.08

[0182] The experimental results show that the GNN-Fusion of the application can still maintain centimeter-level localization accuracy under modal absence. This ability is particularly important for signal obstruction, dust interference or water mist imaging failure in the orchard environment.

[0183] The path planning part compares the performance difference between the traditional graph search algorithm (Dijkstra, A*) and the reinforcement learning control strategy of the application. The experimental scene includes static fruit tree row navigation and dynamic obstacle avoidance tasks. The results are shown in Table 3 as follows:

[0184] Table 3 Path planning performance and energy consumption analysis

[0185] Method Mean path length (m) Mean energy consumption (J / m) Success rate (%) Dijkstra 67.8 13.2 82.4 A* 65.3 12.7 85.6 The method herein 59.9 11.5 95.2

[0186] As shown in Table 3, the spatio-temporal feature driven reinforcement learning strategy can dynamically adjust the trajectory direction according to the environmental semantic information and local obstacles, reduce redundant steering and energy consumption under the premise of ensuring safety distance. Compared with the classic search algorithm, the average path is shortened by about 11.3%, the energy consumption is reduced by 9.7%, and the task success rate is increased by more than 10%.

[0187] Embodiment 2

[0188] The embodiment provides a pesticide spraying robot positioning system based on a graph neural network multi-source fusion, comprising:

[0189] A data acquisition module configured to acquire multi-source sensor information based on a distributed time synchronization architecture;

[0190] A preprocessing module configured to perform data preprocessing based on the acquired multi-source sensor information;

[0191] An embedding module configured to perform graph node multi-modal data embedding on the preprocessed information based on the perception information mapping;

[0192] A spatio-temporal graph module configured to construct a multi-modal spatio-temporal graph structure based on the graph node multi-modal data embedding;

[0193] A fusion module configured to perform graph feature propagation and fusion based on spatio-temporal attention weighting;

[0194] A positioning decision module configured to perform positioning calculation and path decision based on the fused features under the constraint of spatio-temporal features.

[0195] A computer-readable storage medium having a plurality of instructions stored therein, the instructions being adapted to be loaded and executed by a processor of a terminal device, and implementing a pesticide spraying robot positioning method based on a graph neural network multi-source fusion.

[0196] A terminal device comprising a processor and a computer-readable storage medium, the processor being configured to implement instructions, and the computer-readable storage medium being configured to store a plurality of instructions, the instructions being adapted to be loaded and executed by the processor, and implementing a pesticide spraying robot positioning method based on a graph neural network multi-source fusion.

[0197] The above are preferred embodiments of the present application, which do not limit the protection scope of the present application, therefore: any equivalent changes made on the structure, shape, principle of the present application should be covered within the protection scope of the present application.

Claims

1. A pesticide spraying robot positioning method based on a graph neural network multi-source fusion, characterized in that, include: Acquiring information from multiple sensor sources based on a distributed time synchronization architecture; Data preprocessing is performed based on the acquired multi-source sensor information; Multimodal data embedding of graph nodes is performed on the preprocessed information based on perceptual information mapping; Constructing a multimodal spatiotemporal graph structure based on multimodal data embedding of graph nodes; Graph feature propagation and fusion based on spatiotemporal attention weighting; Based on spatiotemporal feature constraints, the fused features are used for localization calculation and path decision-making; The method of graph feature propagation and fusion based on spatiotemporal attention weighting includes introducing dynamic attention weights into the graph neural network structure, and applying them to each layer during the graph propagation phase. The node state is updated, where the propagation of graph information over time includes node... Its temporal neighborhood is: ,in Given the time window length, time-series propagation is achieved through edge weights. Complete the weighted accumulation of historical information: ,in, The temporal attention coefficient measures the contribution of historical frames to the current frame. Spatial information propagation includes the complementary information and spatial correlation between different modal nodes at the same time. Cross-modal feature aggregation is achieved through a modal attention fusion mechanism for nodes. Its modal neighborhood is: The modal aggregation formula is: ,in, The intermodal attention weights are jointly determined by feature correlation and modal confidence. ,in, The inter-modal feature similarity function is used; finally, modal data fusion includes, after completing temporal propagation and modal fusion, the node states are jointly updated through a residual fusion strategy: ,in, This represents a vector concatenation operation. This is the weight matrix of the fusion layer. For bias terms; The method of performing localization calculation and path decision-making on the fused features based on spatiotemporal feature constraints includes obtaining a unified representation feature set of the multimodal sensors at each time step after completing the spatiotemporal map feature propagation and fusion. Then, location calculation and path decision-making based on spatiotemporal feature constraints are performed. The goal of the location calculation stage is to determine the path based on multi-source fusion features. To obtain the temporal pose estimation of the robot in the world coordinate system A hybrid solution framework based on graph optimization and filtering fusion is adopted, and the motion state vector is defined as follows: in , These are linear velocity and angular velocity, respectively, which are predicted based on the estimated state and control input: ,in For motion model, To control the quantity, The process noise is represented by multimodal observations, which are then fused using a graph neural network to form a spatiotemporal feature vector. And an observation model is established through a nonlinear observation function: in To observe the noise and obtain the optimal state estimate, an improved extended Kalman filter is used, combined with a spatiotemporal attention weight to dynamically adjust the covariance matrix, expressed as: ， ,in The weighting factors are based on modal confidence. To reduce long-term drift errors, a graph optimization-based pose graph reconstruction mechanism is introduced, constructing a pose graph reconstruction based on historical keyframes. and constraint edges Composition factor diagram: ， By minimizing the residual energy function Obtain globally consistent pose estimates.

2. The method for localizing a spraying robot based on multi-source fusion of graph neural networks according to claim 1, characterized in that, The multi-source sensor information acquisition based on the distributed time synchronization architecture includes data acquisition through the Global Positioning System (GPS), Inertial Measurement Unit (INS), Wheel Speed ​​Encoder, and Visual Camera. Each sensor node is aligned using the ROS Time unified clock, and the acquired data is timestamped. Organized into multimodal data frames, each frame of data Represented as: ， in, It is the global coordinate position provided by GPS. These are the triaxial acceleration and triaxial angular velocity of the inertial measurement unit, respectively. It is the linear velocity measured by the wheel speed encoder. These are RGB or grayscale images captured by a camera; GPS data is also represented. for : , in, For the actual location, This is a Gaussian noise term. It calculates GPS confidence in real time to compensate for signal drift caused by fruit tree shading; the IMU outputs three-axis acceleration. and angular velocity The observation model is: ， in, Let be the rotation matrix from the machine body to the world coordinate system. The gravity vector For bias terms, This is random noise; wheel speed gauge data provides the average linear velocity of the wheel train. The model is: in For the wheel radius, , These represent the angular velocities of the left and right wheels, respectively; image data is obtained through two-dimensional image observation provided by a vision camera, and relative pose estimation is obtained through feature point extraction and matching. ， in, This represents the pose estimation function based on feature matching.

3. The method for localizing a spraying robot based on multi-source fusion of graph neural networks according to claim 2, characterized in that, The data preprocessing based on the acquired multi-source sensor information includes unifying the sensor data to the robot base coordinate system through linear interpolation and extrinsic parameter calibration after multi-source synchronization is completed. The unified multi-source fusion observation is represented as: ， superscript This indicates a transformation to the robot's base coordinate system; then, a modal effectiveness evaluation strategy is employed for each sensor mode. Calculate its confidence level periodically. Taking into account data stability, noise variance, and time continuity using confidence levels, it can be expressed as: ， in, For current observations, This is the predicted value from the previous moment. To observe the noise variance, This is the time continuity factor, indicating whether there is a sampling discontinuity in the current mode. It is 0 if sampling is lost. At that time, the mode is considered a missing mode or an untrusted mode; subsequently, a gated mask matrix is ​​used. The available states of each modality at the current moment are represented as: In the input feature matrix During construction, features corresponding to the missing modes are masked: ， in, This indicates the Hadamard element-wise product operation.

4. The method for localizing a spraying robot based on multi-source fusion of graph neural networks according to claim 3, characterized in that, The method of embedding graph node multimodal data into the preprocessed information based on perceptual information mapping includes, in the time dimension, embedding each time step... The multi-source sensing state is abstracted as a set of nodes. ,in Represents the GPS / RTK, IMU, camera, and wheel speed sensor modes, for each node. Corresponding to time The Each sensor's observations are processed by a multi-source fusion and time-series correction module, and the output is a vector. Then, the data from different modalities are structured into a unified feature form of multi-dimensional vectors with general semantics; then, the original features of each node are linearly embedded and normalized to map the feature spaces of different modalities to the same-dimensional embedding representation space. The specific transformation formula is as follows: in, A mode-specific linear mapping matrix. For bias terms, To achieve a unified embedding dimension, a semantic-assisted feature encoding mechanism is introduced. This mechanism performs high-level semantic parsing on visual and LiDAR data, jointly modeling the environmental semantic structure and geometric motion information. Specifically, the original visual images are processed by the lightweight semantic segmentation network SegNet to extract pixel distribution probability maps. Then, through region statistics and spatial projection, the segmentation results are transformed into semantic vectors. ， in Indicates the first The proportion or confidence level of the class region in the current field of view; and to unify the representation of semantic and geometric features, the semantic vector of the visual modality and the displacement estimated by visual odometry. ,speed Together they form an extended feature: And mapped to a unified embedding space through a modality-specific linear transformation layer: To suppress short-term noise and enhance the temporal smoothness of nodes, a sliding window averaging strategy is introduced after node feature encoding, utilizing multi-frame information within the temporal neighborhood for smoothing: ,in Given the time window length, all modal nodes eventually form a unified node feature matrix in the input layer of the graph neural network: The matrix, when stacked consecutively in the time dimension, forms a sequence of node features. .

5. The method for localizing a spraying robot based on multi-source fusion of graph neural networks according to claim 4, characterized in that, The construction of a multimodal spatiotemporal graph structure based on graph node multimodal data embedding includes defining the node set in the graph structure as follows: ,in Indicates a time step. Represents the number of sensor modes, the set of edges of the graph. It consists of two parts: temporal edges and modal edges, namely: Temporal edges are used to describe the dynamic evolution relationship of the same mode across consecutive time frames. For any mode... If node Compared to the previous moment node If temporal continuity exists, then temporal edges are established. Edge weight The degree of stability and reliability reflecting state changes is defined as: ,in, For this mode at time The position vector, For smoothing coefficients, This represents the modal confidence level.

6. The method for localizing a spraying robot based on multi-source fusion of graph neural networks according to claim 5, characterized in that, The multimodal spatiotemporal graph structure constructed by embedding multimodal data based on graph nodes also includes the establishment of intermodal edges. These intermodal edges are used to describe the spatial correlation and information complementarity between different sensors within the same time frame, for any given moment. If mode With mode If spatial correlation exists, then modal edges are established. Modal edge weights reflect the degree of consistency between the observations of two modes, and are defined as follows: ， in, The spatial smoothing coefficient is... The correlation coefficient between the two modes is represented by the edge coefficient. To simultaneously consider temporal and modal interactions, the two types of edges are jointly constructed into a dynamically weighted graph. in Representing all edge weight matrices, this graph is continuously updated in time, forming a spatiotemporal sequence graph structure: Each Corresponding robot at time The full-modal perception map.

7. The method for localizing a spraying robot based on multi-source fusion of graph neural networks according to claim 6, characterized in that, The method of performing localization calculation and path decision on the fused features based on spatiotemporal feature constraints also includes obtaining continuous pose trajectories. Then, based on environmental constraints and task requirements, an optimal path plan is generated, firstly based on spatiotemporal graph fusion features. Construct an environment accessibility estimation function: ， in For the location of environmental grid nodes, For feature sampling function, The sigmoid function outputs a position. The passability probability is used to generate a cost map based on the probability field: ,in Indicates the distance to the target point. As a cost of local curvature, The weight parameters are then used to transform the path planning problem into cost minimization optimization, with the objective function being: ， Finally, a reinforcement learning policy network is introduced into the local path optimization. ,in This indicates the current state of the robot and the characteristics of its surrounding environment. This indicates the control action, and the policy update is based on the reward function: Minimize negative expected reward: ， in This is the discount factor.

8. A pesticide spraying robot localization system based on graph neural network multi-source fusion, executing the pesticide spraying robot localization method based on graph neural network multi-source fusion as described in claim 1, characterized in that, include: The data acquisition module is configured to acquire information from multiple sensor sources based on a distributed time synchronization architecture. The preprocessing module is configured to perform data preprocessing based on the acquired multi-source sensor information; The embedding module is configured to embed graph node multimodal data into the preprocessed information based on the perceptual information mapping. The spatiotemporal graph module is configured to construct a multimodal spatiotemporal graph structure based on the embedding of multimodal data from graph nodes; The fusion module is configured to perform graph feature propagation and fusion based on spatiotemporal attention weighting; The localization decision module is configured to perform localization calculation and path decision based on spatiotemporal feature constraints on the fused features.

Images