Agricultural spraying robot system with visual navigation function

Abstract

The invention relates to the field of agricultural robots, and discloses an agricultural spraying robot system with visual navigation, and the system comprises a multi-mode sensing module, a dynamic environment modeling module, a time sequence prediction module, a multi-target path planning module, a spraying control module, and a hardware acceleration unit. The method comprises the steps that a dynamic semantic map is constructed through multi-modal perception, agricultural semantics and an attention mechanism are fused to predict an obstacle trajectory, a safe path is generated in combination with multi-objective optimization, the spraying amount is adjusted based on crop density, and real-time operation is ensured by means of hardware acceleration. According to the method, a dynamic semantic map is constructed through multi-modal perception, agricultural semantics is combined to enhance trajectory prediction, a safe path is generated by utilizing multi-objective optimization, the spraying amount is adaptively adjusted according to the crop density, and real-time control is realized by adopting heterogeneous hardware acceleration; the defects of a traditional agricultural robot in the aspects of navigation precision, obstacle avoidance safety, resource utilization and response speed are overcome, and the intelligent level of complex farmland operation is improved.

Description

Technical Field

[0001] The present invention relates to the field of agricultural robots, and in particular to an agricultural spray robot system with visual navigation. Background Art

[0002] With the development of precision agriculture technologies, agricultural robots are increasingly being used in plant protection operations. However, existing technologies face significant challenges in the complex real-world farmland. Traditional navigation systems often rely on a single sensor for environmental perception, struggling to cope with varying light levels, crop occlusion, and dynamic obstacles in the field. This results in inaccurate environmental modeling and delayed obstacle avoidance decisions. Especially in the middle and late stages of crop growth, visual blind spots caused by dense vegetation and the sparsity of point cloud data further reduce the reliability of navigation systems.

[0003] Existing solutions for handling dynamic obstacles are typically based on static environment assumptions or short-term trajectory predictions. They lack the integration and utilization of prior knowledge about farmland operation scenarios, resulting in significant deviations in obstacle behavior predictions and an inability to effectively support robots in planning safe paths in advance. Furthermore, traditional path planning algorithms often employ single-objective optimization, making it difficult to balance the conflicting demands of operational efficiency, energy economy, and obstacle avoidance safety, easily leading to circuitous paths or increased risk.

[0004] During pesticide spraying, existing spray control systems generally adopt a uniform spraying strategy, ignoring the spatial heterogeneity of crop distribution. This results in both insufficient coverage in high-density crop areas and excessive spray application in low-density areas. Furthermore, computationally intensive tasks such as environmental perception, decision-making planning, and control execution are often limited by the computing power of general-purpose processors. This can easily lead to system response delays in complex scenarios, impacting operational continuity and real-time obstacle avoidance capabilities.

[0005] Therefore, the present invention proposes an agricultural spray robot system with visual navigation to solve the shortcomings of the existing technology. Summary of the Invention

[0006] In response to the shortcomings of the existing technology, the present invention provides an agricultural spray robot system with visual navigation, which solves the problems of traditional agricultural robots such as incomplete environmental perception, low dynamic obstacle prediction accuracy, insufficient single-objective optimization of path planning, extensive pesticide spraying and poor system real-time performance.

[0007] To achieve the above objectives, the present invention is implemented through the following technical solutions: an agricultural spray robot system with visual navigation, the system comprising:

[0008] A multimodal perception module, which is used to simultaneously collect visual data, lidar point cloud data, and inertial measurement data of the farmland environment to generate multimodal perception data;

[0009] A dynamic environment modeling module generates a dynamic semantic map based on the multimodal perception data through spatiotemporal heterogeneous graph modeling, wherein the dynamic semantic map includes vector annotations of crop areas, obstacles, and traversable areas;

[0010] A time series prediction module receives historical sequence data of the dynamic semantic map and predicts the future trajectory of the obstacle and the corresponding confidence level;

[0011] A multi-objective path planning module, which combines the real-time annotation information of the dynamic semantic map with the future trajectory and confidence of obstacles output by the time series prediction module, and generates an optimal path that balances energy consumption, mission time, and safety risks through a multi-objective optimization algorithm;

[0012] The spray control module dynamically adjusts the spray amount according to the node density distribution of the crop area in the dynamic semantic map, wherein the node density distribution is obtained by counting the number of nodes in the crop area in the dynamic semantic map.

[0013] Preferably, the multimodal perception module includes:

[0014] RGB-D camera, used to obtain RGB images and depth information of farmland in real time;

[0015] Multi-line LiDAR, used for high-frequency scanning to generate sparse point cloud data of farmland environments;

[0016] Inertial measurement unit, used to measure the acceleration and angular velocity of the robot with high precision.

[0017] Preferably, the dynamic environment modeling module constructs a spatiotemporal heterogeneous graph in the following manner:

[0018] The pixel blocks in the visual data, the LiDAR point cloud clusters, and the inertial measurement data are mapped into nodes in the graph structure. The node features include the pixel RGB-D value, the point cloud 3D coordinates and reflection intensity, and the combined vector of acceleration and angular velocity.

[0019] Spatial edges are constructed based on the spatial proximity between nodes. If the Euclidean distance between two nodes in three-dimensional space is less than a preset distance threshold, a connection is established.

[0020] Based on the temporal continuity of sensor data, the nodes corresponding to the same physical entity at adjacent timestamps are connected as temporal edges;

[0021] Node features are updated through dynamic graph convolution, and its aggregation function is:

[0022] ;

[0023] in, Indicates the Layer Node The eigenvector of For nodes The set of neighbors of is a learnable weight matrix; is a nonlinear activation function; For nodes and The attention weight between .

[0024] Preferably, the attention weight in the dynamic graph convolution Calculated as follows:

[0025] ;

[0026] in, and Node and The eigenvector of is the learnable parameter matrix; Represents vector concatenation operation; is a rectified linear unit with leakage parameters.

[0027] Preferably, the time series prediction module includes the following steps:

[0028] Set the preset row spacing of the farmland and agricultural machinery operation direction Encoded into agricultural semantic vectors ,in, Obtained through multi-layer perceptron mapping;

[0029] Sequence the historical dynamic semantic map The obstacle position coordinates in are converted into position encoding vectors , and with Splicing to form enhanced feature vector;

[0030] The spatiotemporal correlation of obstacle trajectories is calculated through a multi-head self-attention mechanism, and its attention weight is:

[0031] ;

[0032] in, 、 、 are query, key, and value matrices respectively; is the key vector dimension;

[0033] Predicting the future based on attention output The position offset of the obstacle in the frame , and calculate the corresponding confidence ,in, A preset positive integer.

[0034] Preferably, the cost function of the multi-objective optimization algorithm includes the following three optimization objectives:

[0035] Energy consumption cost: related to the path curvature and moving speed, the calculation formula is:

[0036] ;

[0037] in, Indicates the path at time curvature; is the movement speed; and are the power system parameters; Indicates the total task duration from the starting point to the end point of the path;

[0038] Task time cost: related to the total path duration, calculated as:

[0039] ;

[0040] Dynamic risk cost: It is related to the confidence level and distance of the predicted obstacle trajectory and is calculated as follows:

[0041] ;

[0042] in, The output of the time series prediction module is Confidence of frame obstacle trajectory; is the Euclidean distance between the robot and the obstacle at the prediction moment; is the risk attenuation coefficient; is the total number of predicted frames.

[0043] Preferably, the confidence level in the dynamic risk cost Calculated as follows:

[0044] Obstacle position offset based on the output of the timing prediction module , calculate the predicted position Matching error with historical trajectory;

[0045] The Sigmoid function is used to map the matching error to the interval [0, 1], and we get :

[0046] ;

[0047] in, For the The error norm between the frame prediction position and the historical trajectory; is the normalization coefficient.

[0048] Preferably, the dynamic adjustment process of the spray control module includes:

[0049] Based on the node distribution of the crop area in the dynamic semantic map, the number of crop nodes per unit area is counted to generate a crop density distribution function. ,in, is the plane coordinate of the farmland;

[0050] According to the preset maximum spray flow With the maximum density threshold , calculate the current position Spray volume:

[0051] ;

[0052] The spray volume Converted into a pulse width modulation signal to control the opening and closing frequency and duration of the spray valve.

[0053] Preferably, the system further includes a hardware acceleration unit for accelerating the calculation process of the dynamic environment modeling module and the time series prediction module, specifically including:

[0054] The dynamic graph convolution calculation in the dynamic environment modeling module is parallelized and pipelined, and its speedup ratio is satisfy:

[0055] ;

[0056] in, is the total number of nodes; Serial processing time for a single node; is the number of pipeline stages; is the single-stage pipeline clock cycle;

[0057] The multi-head self-attention calculation of the time series prediction module is optimized by matrix block, and its computational complexity is reduced to:

[0058] ;

[0059] in, is the key vector dimension; is the dimension of the value vector; is the block size;

[0060] The weight allocation formula of the dynamic priority scheduling policy is:

[0061] ;

[0062] in, For the task The computing resource allocation weight of For the task The real-time requirement level; For the task Resource utilization rate; and is a configurable coefficient; The total number of tasks currently to be scheduled;

[0063] The density-flow conversion of the spray control module is mapped to a hardware logic circuit, and its output response delay satisfy:

[0064] ;

[0065] in, is the hardware clock frequency; It is a single-cycle operation.

[0066] The present invention also provides a method for an agricultural spray robot with visual navigation, the method comprising the following steps:

[0067] S1. Synchronously collect visual data, lidar point cloud data, and inertial measurement data of the farmland environment through a multimodal perception module to generate a multimodal perception data stream.

[0068] S2. Build a spatiotemporal heterogeneous graph based on multimodal perception data streams. Update node features through dynamic graph convolution to generate a dynamic semantic map containing vector annotations of crop areas, obstacles, and traversable areas. Spatial edges are constructed by nodes with a Euclidean distance less than a preset threshold, and temporal edges connect nodes with the same entity at adjacent timestamps.

[0069] S3. Based on the historical sequence data of the dynamic semantic map, the future position offset and confidence of the obstacle are predicted using the agricultural semantic embedding vector and the multi-head self-attention mechanism. The agricultural semantic embedding vector is generated by encoding the farmland row spacing and operation direction;

[0070] S4. Integrate the real-time annotation of the dynamic semantic map with the obstacle prediction results, and calculate the weighted sum of energy consumption cost, task time cost, and dynamic risk cost through a multi-objective optimization algorithm to generate the optimal path. The dynamic risk cost is calculated by the exponential decay of obstacle confidence and distance.

[0071] S5. Dynamically adjust the spray volume based on the node density distribution of the crop area in the dynamic semantic map. The spray volume is proportional to the node density, and the valve opening and closing is controlled by the pulse width modulation signal.

[0072] S6. Through the hardware acceleration unit, dynamic graph convolution, multi-head self-attention and multi-objective optimization calculations are accelerated in parallel. The dynamic priority scheduling strategy allocates computing resources according to the real-time performance of the task to ensure the real-time performance of path planning and spray control.

[0073] The present invention provides an agricultural spray robot system with visual navigation, which has the following beneficial effects:

[0074] This invention achieves dynamic semantic understanding of farmland environments by fusing visual, lidar, and inertial measurement data to construct a spatiotemporal heterogeneous graph. Dynamic graph convolution combined with an attention mechanism accurately labels crop areas, obstacles, and traversable areas. This addresses the incomplete environmental modeling issues of traditional methods in complex farmland scenarios, providing high-precision, real-time map support for autonomous navigation.

[0075] 2. By encoding agricultural prior knowledge, such as row spacing and working direction, into semantic vectors and integrating it with a multi-head self-attention mechanism, this invention achieves dual optimization of obstacle trajectory prediction and confidence assessment. This design significantly improves the ability to predict the motion trends of dynamic obstacles, avoiding the risk of misjudgment caused by traditional algorithms that ignore the laws of agricultural scenarios, and ensuring the rationality and reliability of the robot's obstacle avoidance decisions.

[0076] 3. This invention proposes a multi-objective cost function that collaboratively optimizes energy consumption, time, and dynamic risk. Combined with a Pareto-optimal solution screening mechanism, it generates a navigation path that balances obstacle avoidance safety, operational efficiency, and energy conservation. Compared to single-objective planning methods, this solution can adaptively adjust path strategies in complex farmland scenarios, avoiding global performance losses caused by local optimization.

[0077] 4. This invention adjusts spray volume in real time based on the node density distribution of a dynamic semantic map, achieving precise matching of pesticide spray volume with crop distribution density. By controlling valve opening and closing through pulse modulation signals, this approach not only ensures effective pest control in high-density areas but also reduces overuse of pesticide in low-density areas, effectively addressing the resource waste and environmental pollution associated with traditional uniform spraying.

[0078] 5. This invention's hardware acceleration unit, designed for computationally intensive tasks, significantly improves the response speed of dynamic environment modeling, trajectory prediction, and path planning through pipeline parallelism, matrix partitioning, and priority scheduling strategies. This design overcomes the decision-making delay bottleneck caused by insufficient computing power in traditional agricultural robots, ensuring the system's full-link, real-time, closed-loop control capabilities in complex farmland scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0079] Figure 1 This is a system architecture diagram of the present invention;

[0080] Figure 2Flow chart of the method of the present invention. DETAILED DESCRIPTION

[0081] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the present specification. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0082] See also Figure 1 An embodiment of the present invention provides an agricultural spray robot system with visual navigation, the system comprising:

[0083] A multimodal perception module, which is used to simultaneously collect visual data, lidar point cloud data, and inertial measurement data of the farmland environment to generate multimodal perception data;

[0084] In this embodiment, the multimodal perception module uses heterogeneous sensor fusion technology to synchronize the collection and preprocessing of farmland environmental data. This module integrates a visual perception unit, a 3D scanning unit, and a motion measurement unit to construct a spatiotemporally aligned multimodal data stream, providing high-precision, low-noise perception input for subsequent environmental modeling.

[0085] The visual perception unit uses an RGB-D camera as a core component, which is configured to simultaneously output RGB color images and depth information of the farmland scene. During implementation, the optical lens of the RGB-D camera is mounted on the robot body at a preset pitch angle to ensure that the field of view covers the front working area. Preferably, the depth information is generated by active infrared structured light or time-of-flight method, which can overcome the interference of natural lighting changes on depth estimation. For areas where depth data is missing due to leaf occlusion or reflection, a cross-frame dynamic completion algorithm is used to perform interpolation repair based on the local continuity of the depth maps of adjacent frames, which is specifically expressed as:

[0086] ;

[0087] in, is the depth value after repair; For the front Depth map of the frame; is the pixel offset estimated based on the robot motion; is the time decay weight coefficient.

[0088] The three-dimensional scanning unit uses a multi-line laser radar to achieve three-dimensional point cloud acquisition of the farmland environment. Preferably, the laser radar operates in a horizontal rotation scanning mode, and generates sparse point cloud data through the collaboration of a multi-channel laser transmitter and a receiver. During implementation, the point cloud data is processed through motion distortion correction to eliminate deformation errors caused by the robot's own motion. The specific method is: based on the angular velocity measurement value of the inertial measurement unit, the acquisition time of each laser point in the single-frame point cloud is compensated for inverse motion. The coordinate transformation formula is:

[0089] ;

[0090] in, is the coordinate of the original point; is the rotation matrix; is the angular velocity; is the linear velocity; It is the time difference between the current point and the start time of the frame.

[0091] The motion measurement unit includes a six-axis inertial measurement device for real-time acquisition of the robot's three-axis acceleration and three-axis angular velocity. During implementation, the inertial measurement unit outputs raw data at a fixed frequency and performs noise suppression and zero bias compensation through a Kalman filter algorithm. Preferably, the state equation of the filter is defined as:

[0092] ;

[0093] in, is the state vector, which contains the true value estimates of acceleration and angular velocity; is the control input; is the process noise.

[0094] The multimodal perception module further includes a multi-source data synchronization mechanism for eliminating the acquisition delay between heterogeneous sensors. During implementation, data synchronization is achieved by combining hardware trigger signals with software timestamp alignment. Specifically, the RGB-D camera and lidar receive the same external trigger pulse to start single-frame data acquisition, while the data from the inertial measurement unit is aligned to the same time base through high-precision clock interpolation. Preferably, the timestamp alignment error is constrained by the following formula:

[0095] ;

[0096] in, and are the acquisition frequencies of RGB-D camera and lidar respectively.

[0097] A dynamic environment modeling module generates a dynamic semantic map based on the multimodal perception data through spatiotemporal heterogeneous graph modeling, wherein the dynamic semantic map includes vector annotations of crop areas, obstacles, and traversable areas;

[0098] In this embodiment, the dynamic environment modeling module uses spatiotemporal heterogeneous graph modeling technology to transform multimodal sensory data into a semantic map with spatiotemporal correlation. This module uses a graph structure to represent dynamic entities and their relationships within the farmland environment, generating a vector-annotated map containing crop areas, obstacles, and traversable areas. This provides a structured environmental understanding foundation for subsequent path planning and spray control.

[0099] The construction of the spatiotemporal heterogeneous graph first involves a multimodal node mapping process. Specifically, the superpixel blocks in the visual data are extracted as visual nodes, and the feature vector of each node is composed of the RGB color mean, depth mean and texture descriptor of the corresponding pixel area. Preferably, the superpixel blocks are generated by a simple linear iterative clustering algorithm to ensure the continuity of adjacent pixels in color and depth space. For lidar point cloud data, a density-based clustering algorithm is used to generate three-dimensional point cloud clusters, and each point cloud cluster is mapped to a geometric node, whose feature vector includes the three-dimensional coordinates of the cluster center, the mean reflection intensity and the principal component of the point cloud distribution covariance matrix. In addition, the acceleration and angular velocity data output by the inertial measurement unit are calculated by the mean and variance within the sliding time window to form motion feature nodes.

[0100] The spatial edge connection of the spatiotemporal heterogeneous graph is realized based on the three-dimensional spatial proximity between nodes. During the implementation process, if the Euclidean distance between two nodes in the three-dimensional coordinate system is less than a preset spatial threshold, a spatial edge is established to characterize their spatial correlation. Preferably, the spatial threshold is dynamically adjusted according to the row spacing of farmland crops and the operating width of the robot to adapt to the topological structure requirements of different operating scenarios. For the time continuity constraint, the module connects the nodes representing the same physical entity in adjacent timestamps as time edges. Specifically, the entity consistency is judged by a cross-frame node feature matching algorithm, and the matching criterion is defined as:

[0101] ;

[0102] in, and are nodes of adjacent frames respectively; is the node feature vector; is the similarity threshold.

[0103] The dynamic graph convolution layer implements iterative updates of node features through the attention mechanism. In each layer of graph convolution, the node feature aggregation function is defined as:

[0104] ;

[0105] in, Indicates the Layer Node The eigenvector of For nodes The set of neighbors of is the trainable weight matrix; is a nonlinear activation function; For nodes and The attention weight Calculated as follows:

[0106] ;

[0107] in, and Node and The eigenvector of is the learnable parameter matrix; Represents vector concatenation operation; is a rectified linear unit with leakage parameters. Preferably, the attention mechanism enables the model to adaptively focus on key information in spatially adjacent nodes and temporally continuous nodes.

[0108] The generation process of the dynamic semantic map further includes a semantic annotation submodule. During implementation, a fully connected classification network is used to predict semantic categories of node features output by graph convolution. Category labels include crops, obstacles, traversable areas, and other dynamic entities. Preferably, the loss function of the classification network is a weighted sum of cross entropy loss and graph structure consistency loss, and its formula is:

[0109] ;

[0110] in, is the cross entropy loss; For nodes The predicted category probability vector of ; is the set of all edges in the graph; and is the balance coefficient.

[0111] A time series prediction module receives historical sequence data of the dynamic semantic map and predicts the future trajectory of the obstacle and the corresponding confidence level;

[0112] In this embodiment, the time series prediction module integrates prior knowledge of agricultural scenarios with a spatiotemporal attention mechanism to accurately predict and assess the confidence of future obstacle trajectories. Based on historical sequence data from a dynamic semantic map, this module extracts correlation features between obstacle movement patterns and farmland operation scenarios, constructs a spatiotemporal correlation model, and provides a dynamic risk prediction basis for path planning.

[0113] The module first encodes the inherent attributes of the farmland operation scene. Specifically, the farmland preset row spacing and agricultural machinery operation direction Input multi-layer perceptron to generate agricultural semantic prior vector , and its calculation process is defined as:

[0114] ;

[0115] in, It is a mapping network composed of a fully connected layer and a nonlinear activation function; and The agricultural semantic vector encodes the spatial constraint characteristics of crop layout rules on obstacle movement.

[0116] During implementation, the module converts the historical dynamic semantic map sequence The obstacle position coordinates in Convert to positional encoding vector Preferably, a sinusoidal position encoding method is used to map the two-dimensional coordinates to a high-dimensional space:

[0117] ;

[0118] in, is the coding dimension; is the dimension index. and agricultural semantic vectors Splicing to form enhanced features , thereby combining the obstacle location information with the farmland scene structure constraints.

[0119] The module models the spatiotemporal correlation of obstacle trajectories through a multi-head self-attention mechanism. Specifically, the enhanced features Linear projection is the query matrix , key matrix and value matrix , whose dimensions are The agricultural semantic prior correction term is introduced into the attention weight calculation process, and the formula is defined as:

[0120] ;

[0121] in, The agricultural semantic prior correlation matrix is ​​used to enhance the attention weights of trajectory patterns that conform to farmland operation patterns. The multi-head mechanism captures the spatiotemporal dependencies between different subspaces by performing multiple sets of attention calculations in parallel and concatenating the results.

[0122] Based on the attention output, the module predicts the future The position offset of the obstacle in the frame ,in, is a preset positive integer. Preferably, a temporal convolutional network is used to model the multi-frame offset sequence, and its convolution kernel slides along the time dimension to extract motion trend features. At the same time, the trajectory prediction confidence is calculated through the error matching mechanism , the specific method is: compare the predicted position Sliding average position with historical trajectory The deviation is normalized using the Sigmoid function:

[0123] ;

[0124] in, is the normalization coefficient; The confidence metric quantifies the degree of consistency between the predicted trajectory and the historical motion pattern.

[0125] A multi-objective path planning module, which combines the real-time annotation information of the dynamic semantic map with the future trajectory and confidence of obstacles output by the time series prediction module, and generates an optimal path that balances energy consumption, mission time, and safety risks through a multi-objective optimization algorithm;

[0126] In this embodiment, the multi-objective path planning module integrates environmental semantic information with dynamic risk prediction to construct a multi-constraint optimization model to generate a navigation path. Based on the real-time annotation information of the dynamic semantic map and the obstacle trajectory prediction results, this module quantifies the energy consumption, time, and risk costs of path planning. It then uses a multi-objective optimization algorithm to solve for the Pareto optimal solution set, ultimately outputting a navigation path that balances operational efficiency and safety.

[0127] The module first defines a multi-objective cost function, which includes the following three optimization objectives:

[0128] Energy consumption cost: It is related to the path curvature and movement speed of the robot during movement, and its integral form is expressed as:

[0129] ;

[0130] in, Indicates the path at time curvature; is the movement speed; and are the power system parameters; represents the total task duration from the starting point to the end point of the path; the squared curvature term reflects the steering energy consumption, and the speed linear term represents the uniform motion energy consumption. The total energy consumption of the entire path is accumulated by integration.

[0131] Task time cost: directly related to the total path duration, and its expression is:

[0132] ;

[0133] This cost item is used to constrain the robot to complete the task as quickly as possible to avoid reduced efficiency due to detours.

[0134] Dynamic risk cost: obstacle trajectory confidence based on the output of the time series prediction module and the distance between the robot and the obstacle Calculation, its formula is defined as:

[0135] ;

[0136] in, The output of the time series prediction module is Confidence of frame obstacle trajectory; is the Euclidean distance between the robot and the obstacle at the prediction moment; is the risk attenuation coefficient; is the total number of predicted frames. Make close, high-confidence obstacles incur a higher risk penalty, forcing paths away from potential collision areas.

[0137] The confidence level in the dynamic risk cost Calculated by: Obstacle position offset based on the output of the time series prediction module , calculate the predicted position Sliding average position with historical trajectory The matching error , and use the Sigmoid function to map it to the [0, 1] interval:

[0138] ;

[0139] in, For the The error norm between the frame prediction position and the historical trajectory; The confidence level quantifies the consistency between the predicted trajectory and the historical motion pattern. The larger the error, the lower the confidence level, thereby reducing the impact of abnormal predictions on risk costs.

[0140] The multi-objective optimization algorithm uses the improved NSGA-II framework to implement Pareto frontier search. During the implementation process, the initial population is generated through random sampling and heuristic rules, where the heuristic rules include:

[0141] Prioritize crop rows: Generate candidate path points in the areas between crop rows marked on the dynamic semantic map to avoid crushing crops;

[0142] Risk area avoidance: based on dynamic risk cost Heat map, excluding path points in high-risk areas;

[0143] Motion smoothness constraint: Bezier curve interpolation is used to ensure the continuity of path curvature and reduce the energy consumption surge caused by sharp turns.

[0144] Preferably, the crossover and mutation operators are adaptively adjusted according to the characteristics of agricultural scenarios:

[0145] Directional crossing: Increase the probability of path point crossing in the inter-row area of ​​crops to strengthen the navigation trend between rows;

[0146] Risk-aware mutation: Directed perturbations are introduced in the neighborhood of high-risk areas to drive the path toward low-risk directions.

[0147] The module ultimately selects the optimal path from the Pareto solution set using a preset weight vector. Preferably, the weight allocation strategy dynamically adjusts based on the operation phase: prioritizing time reduction at the start of the operation and optimizing risk when approaching obstacles. Before outputting the path, the selected path is further locally optimized using gradient descent to eliminate minor jitter and ensure motion control feasibility.

[0148] a spray control module that dynamically adjusts the spray volume according to a node density distribution of the crop area in the dynamic semantic map, wherein the node density distribution is obtained by counting the number of nodes in the crop area in the dynamic semantic map;

[0149] In this embodiment, the spray control module implements precise variable-rate spray control in farmland scenarios by analyzing the density of crop area nodes within a dynamic semantic map. Based on the semantic map information output by the environmental modeling module, this module calculates crop distribution density in real time and maps this density feature into a spray flow control signal. This ensures that the pesticide spray rate matches the actual crop distribution requirements, minimizing pesticide waste and environmental pollution.

[0150] The module first extracts the node distribution data of the crop area from the dynamic semantic map. Specifically, the nodes marked as crop categories in the dynamic semantic map form the density statistics basis, and the spatial coordinates of each node are and its coverage Used to calculate the plane density distribution. Preferably, the coverage It is determined based on the spatial resolution of the sensor from which the node originates. For example, the visual node corresponds to the actual physical size of the superpixel block, and the lidar node corresponds to the three-dimensional projection area of ​​the point cloud cluster.

[0151] Based on the node distribution, the module generates the crop density distribution function through the kernel density estimation algorithm. In the implementation process, the Gaussian kernel function is used to perform smooth interpolation on discrete nodes, and its calculation formula is:

[0152] ;

[0153] in, is the total number of crop nodes in the current field of view; is a bandwidth parameter that controls the spatial smoothness of the density estimation. Preferably, the bandwidth parameter Adaptive adjustment based on robot movement speed: increase when speed is high To improve computing efficiency, reduce the To improve spatial resolution.

[0154] The module is based on the preset maximum spray flow With the maximum density threshold , calculate the current position Target spray volume , and its mapping relationship is:

[0155] ;

[0156] in, is a calibration parameter obtained through historical data statistics, which represents the maximum reasonable density of crop nodes per unit area. When the spray volume is limited to , avoid local overspray.

[0157] The spray volume control signal is realized by pulse width modulation (PWM) technology. Specifically, the module continuously controls the spray volume. Converted into a PWM signal with adjustable duty cycle, its pulse width Proportional to the amount of spray:

[0158] ;

[0159] in, is the reference spray volume corresponding to unit pulse width; Preferably, the reference spray volume is determined by a flow calibration experiment and stored in a system parameter table.

[0160] During implementation, the module further integrates a delay compensation mechanism to eliminate the position deviation caused by the robot movement. When moving, the actual spray position Relative to the calculated position The correction formula is:

[0161] ;

[0162] in, To control the delay of the signal from calculation to execution; and Preferably, the delay parameter is updated in real time through system response testing.

[0163] A hardware acceleration unit, used to accelerate the calculation process of the dynamic environment modeling module and the time series prediction module;

[0164] In this embodiment, the hardware acceleration unit optimizes the real-time performance of the dynamic environment modeling module and the time series prediction module through a parallel computing architecture and dynamic resource scheduling strategy. This unit features dedicated hardware logic for computationally intensive tasks such as graph convolution, attention mechanisms, and control signal generation, ensuring the system meets the low-latency, high-throughput requirements of complex agricultural scenarios.

[0165] The hardware acceleration unit first performs pipeline parallel decomposition on the dynamic graph convolution calculation. During the implementation process, node feature aggregation and attention weight calculation are divided into multiple independent pipeline stages, and each stage achieves the overlap of calculation and data transmission through a double buffering mechanism. Characterized by the following formula:

[0166] ;

[0167] in, is the total number of nodes; Serial processing time for a single node; is the number of pipeline stages; is a single-stage pipeline clock cycle; preferably, the number of pipeline stages Dynamically adjust according to hardware resource capacity. When the size is large, a multi-stage fine-grained pipeline is used to improve parallel efficiency.

[0168] For the multi-head self-attention calculation of the time series prediction module, the hardware acceleration unit adopts a matrix block optimization strategy. , key matrix and value matrix By block size Divide into sub-matrix blocks and reduce memory access overhead by reusing on-chip cache. The optimized computational complexity is reduced to:

[0169] ;

[0170] in, and are the dimensions of the key vector and value vector respectively; is the block size. Preferably, the block size Adaptive configuration based on hardware cache capacity and When the dimension is high, a small block strategy is adopted to avoid cache overflow.

[0171] The hardware acceleration unit integrates a dynamic priority scheduler to allocate computing resources based on the real-time requirements of the task and resource occupancy rate. The distribution formula is defined as:

[0172] ;

[0173] in, For the task The real-time requirement level (e.g. path planning task is set to the highest level); For the task Resource utilization rate; and Preferably, the scheduler adopts a preemptive scheduling strategy, which pauses the computation pipeline of low-priority tasks when a high-priority task arrives.

[0174] In response to the real-time requirements of the spray control module, the hardware acceleration unit solidifies the density-flow conversion mapping into a hardware logic circuit. This circuit realizes density distribution through a lookup table and a linear interpolator. To spray volume Real-time mapping with delayed response satisfy:

[0175] ;

[0176] in, is the hardware clock frequency; Preferably, the lookup table content is dynamically updated according to calibration experimental data to adapt to the nonlinear characteristics of different pesticide flow rates.

[0177] See also Figure 2 The present invention also provides a method for an agricultural spray robot with visual navigation, the method comprising the following steps:

[0178] S1. The multimodal perception module uses an RGB-D camera, multi-line LiDAR, and an inertial measurement unit to synchronously collect farmland environmental information. The RGB-D camera captures color images and depth information of the crop area, the LiDAR generates 3D point cloud data to represent terrain and obstacle contours, and the IMU measures the robot's acceleration and angular velocity in real time for motion compensation. Data from each sensor is synchronized at the microsecond level through hardware trigger signals and a timestamp interpolation algorithm, forming a spatiotemporally aligned multimodal data stream, providing highly consistent input for subsequent processing.

[0179] S2. Construct a spatiotemporal heterogeneous graph model based on multimodal data streams. The superpixel blocks, lidar point cloud clusters and IMU motion features in the visual data are mapped to graph nodes respectively. The node features include color, depth, three-dimensional coordinates, reflection intensity and motion parameters. The spatial edge connection rule is: if the Euclidean distance between two nodes in three-dimensional space is less than the preset threshold, a spatial edge is established to represent the proximity relationship; the temporal edge is generated by a cross-frame node matching algorithm to connect the nodes representing the same physical entity in adjacent timestamps. The node features are iteratively updated through the dynamic graph convolution layer, and the dynamic semantic map is output in combination with the fully connected classification network, which contains vectorized annotation information of crop areas, obstacles and passable areas;

[0180] S3, the time series prediction module, receives historical sequence data from the dynamic semantic map, extracts obstacle location coordinates, and integrates prior knowledge of the farmland scene. The spacing between farmland rows and the direction of operation are encoded into agricultural semantic vectors using a multi-layer perceptron, which are used to correct the physical rationality of the trajectory prediction. A multi-head self-attention mechanism is used to analyze the spatiotemporal correlation of obstacle trajectories, capturing the underlying patterns of historical motion patterns and future displacements. The prediction results include the position offset of the obstacle within several future time frames and its confidence. The confidence is calculated by the matching error between the predicted position and the historical trajectory. The smaller the error, the higher the confidence.

[0181] The S4 multi-objective path planning module integrates real-time semantic maps with obstacle prediction results to construct an optimization objective function that includes energy consumption, task time, and dynamic risk. The energy cost is linearly related to the path curvature and movement speed, the task time cost is directly related to the total path duration, and the dynamic risk cost is calculated using an exponential decay model based on the obstacle confidence and its real-time distance from the robot. A multi-objective optimization algorithm is used to generate a Pareto optimal path solution set, and a preset weight strategy is used to select the optimal path, ensuring both operational efficiency and energy economy while ensuring obstacle avoidance safety.

[0182] S5, the spray control module, adjusts the pesticide spray rate based on the node density distribution of the crop area in the dynamic semantic map. Node density is calculated using a kernel density estimation algorithm. High-density areas correspond to high spray flow rates, while low-density areas reduce flow rates to conserve pesticide. The spray rate command is converted into a pulse-width modulated signal, and precise flow control is achieved by adjusting the valve opening and closing frequency and duty cycle. A motion delay compensation mechanism is also introduced to correct spray position deviations based on the robot's real-time speed, ensuring accurate coverage of the target crop area.

[0183] The S6 hardware acceleration unit parallelizes and accelerates computationally intensive tasks such as dynamic graph convolution, multi-head self-attention, and multi-objective optimization. Dynamic graph convolution improves throughput through pipeline decomposition and sparse adjacency matrix compression; the self-attention mechanism uses matrix block optimization to reduce memory access overhead; and multi-objective optimization reduces computational latency through heuristic rule initialization and rapid convergence strategies. A dynamic priority scheduler dynamically allocates computing resources based on the real-time nature of tasks. Combined with a preemptive scheduling mechanism, it ensures low-latency response for critical tasks, guaranteeing the system's real-time operational capabilities in all scenarios.

[0184] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. An agricultural spray robot system with visual navigation, characterized in that: The system comprises: A multimodal perception module, which is used to simultaneously collect visual data, lidar point cloud data, and inertial measurement data of the farmland environment to generate multimodal perception data; A dynamic environment modeling module generates a dynamic semantic map based on the multimodal perception data through spatiotemporal heterogeneous graph modeling, wherein the dynamic semantic map includes vector annotations of crop areas, obstacles, and traversable areas; A time series prediction module receives historical sequence data of the dynamic semantic map and predicts the future trajectory of the obstacle and the corresponding confidence level; A multi-objective path planning module, which combines the real-time annotation information of the dynamic semantic map with the future trajectory and confidence of obstacles output by the time series prediction module, and generates an optimal path that balances energy consumption, mission time, and safety risks through a multi-objective optimization algorithm; The spray control module dynamically adjusts the spray amount according to the node density distribution of the crop area in the dynamic semantic map, wherein the node density distribution is obtained by counting the number of nodes in the crop area in the dynamic semantic map.

2. The agricultural spray robot system with visual navigation according to claim 1, characterized in that: The multimodal perception module includes: RGB-D camera, used to obtain RGB images and depth information of farmland in real time; Multi-line LiDAR, used for high-frequency scanning to generate sparse point cloud data of farmland environments; Inertial measurement unit, used to measure the acceleration and angular velocity of the robot with high precision.

3. The agricultural spray robot system with visual navigation according to claim 1, characterized in that: The dynamic environment modeling module constructs a spatiotemporal heterogeneous graph in the following way: The pixel blocks in the visual data, the LiDAR point cloud clusters, and the inertial measurement data are mapped into nodes in the graph structure. The node features include the pixel RGB-D value, the point cloud 3D coordinates and reflection intensity, and the combined vector of acceleration and angular velocity. Spatial edges are constructed based on the spatial proximity between nodes. If the Euclidean distance between two nodes in three-dimensional space is less than a preset distance threshold, a connection is established. Based on the temporal continuity of sensor data, the nodes corresponding to the same physical entity at adjacent timestamps are connected as temporal edges; Node features are updated through dynamic graph convolution, and its aggregation function is: ； in, Indicates the Layer Node The eigenvector of For nodes The set of neighbors of is a learnable weight matrix; is a nonlinear activation function; For nodes and The attention weight between .

4. The agricultural spray robot system with visual navigation according to claim 3, characterized in that: Attention weights in the dynamic graph convolution Calculated as follows: ； in, and Node and The eigenvector of is the learnable parameter matrix; Represents vector concatenation operation; is a rectified linear unit with leakage parameters.

5. The agricultural spray robot system with visual navigation according to claim 1, characterized in that: The time series prediction module includes the following steps: Set the preset row spacing of the farmland and agricultural machinery operation direction Encoded into agricultural semantic vectors ,in, Obtained through multi-layer perceptron mapping; Sequence the historical dynamic semantic map The obstacle position coordinates in are converted into position encoding vectors , and with Splicing to form enhanced feature vector; The spatiotemporal correlation of obstacle trajectories is calculated through a multi-head self-attention mechanism, and its attention weight is: ； in, 、 、 are query, key, and value matrices respectively; is the key vector dimension; Predicting the future based on attention output The position offset of the obstacle in the frame , and calculate the corresponding confidence ,in, A preset positive integer.

6. The agricultural spray robot system with visual navigation according to claim 1, characterized in that: The cost function of the multi-objective optimization algorithm includes the following three optimization objectives: Energy consumption cost: related to the path curvature and moving speed, the calculation formula is: ； in, Indicates the path at time curvature; is the movement speed; and are the power system parameters; Indicates the total task duration from the starting point to the end point of the path; Task time cost: related to the total path duration, calculated as: ； Dynamic risk cost: It is related to the confidence level and distance of the predicted obstacle trajectory and is calculated as follows: ； in, The output of the time series prediction module is Confidence of frame obstacle trajectory; is the Euclidean distance between the robot and the obstacle at the prediction moment; is the risk attenuation coefficient; is the total number of predicted frames.

7. The agricultural spray robot system with visual navigation according to claim 6, characterized in that: The confidence level in the dynamic risk cost Calculated as follows: Obstacle position offset based on the output of the timing prediction module , calculate the predicted position Matching error with historical trajectory; The Sigmoid function is used to map the matching error to the interval [0, 1], and we get : ； in, For the The error norm between the frame prediction position and the historical trajectory; is the normalization coefficient.

8. The agricultural spray robot system with visual navigation according to claim 1, characterized in that: The dynamic adjustment process of the spray control module includes: Based on the node distribution of the crop area in the dynamic semantic map, the number of crop nodes per unit area is counted to generate a crop density distribution function. ,in, is the plane coordinate of the farmland; According to the preset maximum spray flow With the maximum density threshold , calculate the current position Spray volume: ； The spray volume Converted into a pulse width modulation signal to control the opening and closing frequency and duration of the spray valve.

9. The agricultural spray robot system with visual navigation according to claim 1, characterized in that: The system further includes a hardware acceleration unit for accelerating the calculation process of the dynamic environment modeling module and the time series prediction module, specifically including: The dynamic graph convolution calculation in the dynamic environment modeling module is parallelized and pipelined, and its speedup ratio is satisfy: ； in, is the total number of nodes; Serial processing time for a single node; is the number of pipeline stages; is the single-stage pipeline clock cycle; The multi-head self-attention calculation of the time series prediction module is optimized by matrix block, and its computational complexity is reduced to: ； in, is the key vector dimension; is the dimension of the value vector; is the block size; The weight allocation formula of the dynamic priority scheduling policy is: ； in, For the task The computing resource allocation weight of For the task The real-time requirement level; For the task Resource utilization rate; and is a configurable coefficient; The total number of tasks currently to be scheduled; The density-flow conversion of the spray control module is mapped to a hardware logic circuit, and its output response delay satisfy: ； in, is the hardware clock frequency; It is a single-cycle operation.

10. A method for an agricultural spray robot with visual navigation, applied to the system according to any one of claims 1 to 9, characterized in that: The method comprises the following steps: S1. Synchronously collect visual data, lidar point cloud data, and inertial measurement data of the farmland environment through a multimodal perception module to generate a multimodal perception data stream. S2. Build a spatiotemporal heterogeneous graph based on multimodal perception data streams. Update node features through dynamic graph convolution to generate a dynamic semantic map containing vector annotations of crop areas, obstacles, and traversable areas. Spatial edges are constructed by nodes with a Euclidean distance less than a preset threshold, and temporal edges connect nodes with the same entity at adjacent timestamps. S3. Based on the historical sequence data of the dynamic semantic map, the future position offset and confidence of the obstacle are predicted using the agricultural semantic embedding vector and the multi-head self-attention mechanism. The agricultural semantic embedding vector is generated by encoding the farmland row spacing and operation direction; S4. Integrate the real-time annotation of the dynamic semantic map with the obstacle prediction results, and calculate the weighted sum of energy consumption cost, task time cost, and dynamic risk cost through a multi-objective optimization algorithm to generate the optimal path. The dynamic risk cost is calculated by the exponential decay of obstacle confidence and distance. S5. Dynamically adjust the spray volume based on the node density distribution of the crop area in the dynamic semantic map. The spray volume is proportional to the node density, and the valve opening and closing is controlled by the pulse width modulation signal. S6. Through the hardware acceleration unit, dynamic graph convolution, multi-head self-attention and multi-objective optimization calculations are accelerated in parallel. The dynamic priority scheduling strategy allocates computing resources according to the real-time performance of the task to ensure the real-time performance of path planning and spray control.

Images