Agricultural robot group intelligent scheduling method

Abstract

The invention provides an agricultural robot group intelligent scheduling method, which comprises the following steps: performing visual semantic conversion on an operation behavior video acquired by each agricultural robot to obtain a corresponding semantic feature vector; according to the similarity between the plurality of semantic feature vectors corresponding to each agricultural robot and the semantic feature center vector corresponding to each agricultural robot, determining an operation state response value of each agricultural robot at the moment corresponding to each frame of image; according to the plurality of operation state response values corresponding to any two agricultural robots, a cooperative pressure difference value corresponding to any two agricultural robots is determined; and according to a plurality of operation state response values corresponding to any agricultural robot and the cooperative pressure difference values between the agricultural robot and other agricultural robots except the agricultural robot, determining a system overall scheduling loss value, and performing global scheduling on the agricultural robots according to the system overall scheduling loss value. Therefore, the better system cooperation capability is realized.

Description

Technical Field

[0001] This invention relates to the field of intelligent agriculture, and in particular to a method for intelligent scheduling of agricultural robot swarms. Background Technology

[0002] With the continuous improvement of agricultural intelligence and automation, agricultural robots have been widely used in various stages such as sowing, fertilizing, spraying, weeding, harvesting, and agricultural product transportation. Especially in large-scale planting bases, orchards, and greenhouses, deploying multiple collaborative robot systems has become an important means to improve production efficiency, reduce labor costs, and enhance the quality of agricultural products. However, how to achieve collaborative scheduling and obstacle avoidance of multiple robots in open, dynamic, and complex agricultural environments still faces significant technical challenges.

[0003] Most existing multi-robot scheduling systems employ rule-based centralized task allocation methods, resulting in fixed scheduling logic and delayed responses. These methods are ill-suited to the realities of agriculture, characterized by high task density, limited spatial resources, and overlapping task windows. While some research has introduced heuristic optimization strategies such as graph-based path planning or ant colony optimization to mitigate task conflicts, these methods typically rely on static environment modeling and lack the ability to model dynamic spatial interactions between robots. Furthermore, traditional scheduling systems offer weak support for user interaction, have complex control interfaces, and lack the ability to understand and respond to fuzzy natural language commands, limiting system usability and field deployment efficiency. Moreover, existing multi-robot scheduling systems utilize scheduling optimization algorithms combined with global path planning techniques to achieve task allocation and obstacle avoidance control within a framework. However, these solutions generally lack modeling and utilization of temporal information about task behavior, failing to achieve frame-level trajectory estimation and conflict probability expression, and lacking the ability to provide dynamic scheduling feedback using natural language. Consequently, they struggle to support the flexible collaboration and human-robot co-location scheduling requirements of large-scale multi-robot systems. Summary of the Invention

[0004] To address the aforementioned technical problems, the technical solution adopted by this invention is as follows: According to one aspect of this application, a method for intelligent scheduling of agricultural robots swarms is provided, which is applied to a scheduling system connected to a number of agricultural robots; Among them, the intelligent scheduling method for agricultural robot swarms includes: Step S100: Perform visual semantic transformation on the visual features of the video of the operation behavior of each agricultural robot in the same target operation area within the target time period to obtain the semantic feature vector of each frame of the video of the operation behavior of each agricultural robot within the target time period; the length of the target time period is a preset duration, and the end time of the target time period is the current time. Step S200: Based on the similarity between the semantic feature vector of each frame image corresponding to each agricultural robot and the preset semantic feature center vector corresponding to each agricultural robot, determine the operation status response value of each agricultural robot at the time corresponding to each frame image; the operation status response value represents the operation status of the agricultural robot at the corresponding time. Step S300: Based on several operational status response values ​​of any two agricultural robots within the target time period, determine the collaborative pressure difference value of any two agricultural robots within the target time period; the collaborative pressure difference value characterizes the overall conflict degree of overlapping operational spaces when two agricultural robots perform operational tasks within the target time period; the collaborative pressure difference value of two agricultural robots is directly proportional to the overall conflict degree of the two agricultural robots. Step S400: Determine the overall system scheduling loss value based on several operational status response values ​​of any agricultural robot within the target time period, and the collaborative pressure difference value between the agricultural robot and other agricultural robots besides the agricultural robot. Step S500: Perform global scheduling of several agricultural robots based on the overall system scheduling loss value and the preset scheduling strategy.

[0005] In one exemplary embodiment of this application, step S100 includes: Step S110: Obtain the operation behavior video of each agricultural robot within the target time period to obtain the operation behavior video list A=(A1,A2,...,A1) i ,...,A m ); where i = 1, 2, ..., m; m is the number of agricultural robots; A i The video of the i-th agricultural robot's operational behavior collected within the target time period; Step S120: Perform uniform frame extraction on each operation video to obtain n frames of images corresponding to each operation video. Step S130: Perform visual semantic transformation on each frame image corresponding to each work behavior video to obtain the semantic feature vector of each frame image corresponding to each work behavior video. Wherein, the semantic feature vector U of the t-th frame image of the video of the i-th agricultural robot's work behavior is... it =tanh(W2×ReLU(W1×Ev(I it )+b1)+b2); t=0,1,...,n; In the formula, tanh() is the preset hyperbolic tangent function; W2 is the preset weight matrix of the second fully connected neural network in the multilayer perceptron; ReLU() is the preset modified linear unit activation function; W1 is the preset weight matrix of the first fully connected neural network in the multilayer perceptron; I it Let be the t-th frame of the video representing the operational behavior of the i-th agricultural robot; Ev(I it ) for I it The input is fed into a preset visual sensor, and the I signal output by the visual sensor is... it The corresponding visual feature vector; b1 is the bias vector of the first fully connected neural network in the preset multilayer perceptron; b2 is the bias vector of the second fully connected neural network in the preset multilayer perceptron.

[0006] In one exemplary embodiment of this application, step S200 includes: Step S210: The operational status response value T of the i-th agricultural robot at the time corresponding to the t-th frame image. it =softmax_over(a1×exp(-a2×||U it -B i || 2 )); In the formula, softmax_over() is the preset normalization function; a1 is the preset temperature adjustment factor; a1=1 or a1=5; exp() is the exponential function with the natural constant as the base; a2 is the preset time tightness control factor; 0.01≤a2≤10; B i Let ||U be the preset semantic feature center vector corresponding to the i-th agricultural robot; it -B i || 2 For U it With B i Semantic similarity between them; Among them, several agricultural robots have the same time corresponding to the t-th frame image.

[0007] In one exemplary embodiment of this application, the semantic feature center vector B corresponding to the i-th agricultural robot i =∑ ei d=1 C id ×U id ; Where ei is the number of image frames in the video of the i-th agricultural robot performing its work during the target time period; C id The preset time weight is the d-th frame image corresponding to the i-th agricultural robot's operation in the video of its operation behavior within the target time period; U idLet d be the semantic feature vector of the d-th frame image of the i-th agricultural robot performing its work in the video of its work behavior during the target time period.

[0008] In one exemplary embodiment of this application, step S300 includes: Step S310: From the m agricultural robots, randomly select the i-th and j-th agricultural robots as the agricultural robots for determining the collaborative pressure difference value; j=1,2,...,m; and j≠i; Step S320: Determine the cooperative pressure difference value P corresponding to the i-th agricultural robot and the j-th agricultural robot within the target time period. ij =∑ n-1 t=0 (1 / (1+exp(-T it ×T jt )))×(1-S ijt ); In the formula, S ijt The spatial isolation coefficient is determined based on the overlapping state of the working spaces of the i-th and j-th agricultural robots at the time corresponding to the t-th frame image.

[0009] In one exemplary embodiment of this application, the spatial isolation coefficient S between the i-th agricultural robot and the j-th agricultural robot at the time corresponding to the t-th frame image is... ijt Determined through the following steps: Step S321: Rasterize the target work area to obtain several grid areas corresponding to the target work area; the area of ​​each grid area is equal. Step S322: If the grid region where the i-th agricultural robot is located at the time corresponding to the t-th frame image is different from the grid region where the j-th agricultural robot is located at the time corresponding to the t-th frame image, and the distance between the position of the i-th agricultural robot at the time corresponding to the t-th frame image and the position of the j-th agricultural robot at the time corresponding to the t-th frame image is greater than a preset interval distance, then let S... ijt =1; otherwise, S ijt =0.

[0010] In one exemplary embodiment of this application, step S400 includes: Step S410: Based on the response values ​​of several operational states of the i-th agricultural robot within the target time period, and the collaborative pressure difference between the i-th agricultural robot and other agricultural robots besides the i-th agricultural robot, determine the overall system scheduling loss value L. opt =∑ m i=1 ∑ n-1 t=0 (Di ×T it ×(E it +∑ m j=1 (F ij ×P ij ))); In the formula, D i E is the preset task priority factor corresponding to the i-th agricultural robot; it F represents the unit energy consumption cost of the i-th agricultural robot at the time corresponding to the t-th frame of the image; ij The preset weighting coefficients are used to determine the interference of the i-th agricultural robot with the j-th agricultural robot.

[0011] In one exemplary embodiment of this application, the scheduling system is further configured to perform the following steps: Step S010: In response to receiving a natural language query statement input by the user, obtain a behavior description statement of the operation behavior of each agricultural robot at the time corresponding to each frame of the image within the target time period. Step S020: Based on the semantic similarity between the behavior description statement of each agricultural robot at the time corresponding to each frame of the image within the target time period and the natural language query statement, determine the target response statement corresponding to the natural language query statement.

[0012] In one exemplary embodiment of this application, step S020 includes: Step S021: Determine the semantic query value corresponding to each agricultural robot based on the semantic similarity between the behavior description statement and the natural language query statement of each frame image within the target time period. Step S022: Among several agricultural robots, the agricultural robot with the largest semantic query value is identified as the target robot; Step S023: Determine the target robot's response to the natural language query as the target response statement.

[0013] In one exemplary embodiment of this application, the semantic query value G corresponding to the i-th agricultural robot i =∑ n-1 t=0 (H i ×sim(Q text J it )×T it ); In the formula, H i Q represents the preset importance weight corresponding to the i-th agricultural robot; text For natural language query statements; J itA behavioral description statement for the i-th agricultural robot's operation at the time corresponding to the t-th frame image; sim(Q text J it ) is a function for determining the semantic similarity between a natural language query and the behavior description of the i-th agricultural robot at the time corresponding to the t-th frame image.

[0014] The present invention has at least the following beneficial effects: The intelligent scheduling method for agricultural robots of the present invention first performs visual semantic transformation on the visual features of the operation behavior videos of each agricultural robot in the same target operation area within a target time period, obtaining the semantic feature vector of each frame of the operation behavior video corresponding to each agricultural robot within the target time period. Then, based on the similarity between the semantic feature vector of each frame of the video corresponding to each agricultural robot and the preset semantic feature center vector corresponding to each agricultural robot, the operation state response value, which represents the operation state of each agricultural robot at the time corresponding to each frame of the video, is determined. Finally, based on several operation state response values ​​corresponding to any two agricultural robots within the target time period, the operation state response value of any two agricultural robots within the target time period is determined. The system uses a collaborative pressure difference value within a specified time period to characterize the overall conflict level of overlapping work spaces when performing tasks. Then, based on several work status response values ​​of any agricultural robot within the target time period, and the collaborative pressure difference value between that agricultural robot and other agricultural robots, the overall system scheduling loss value is determined. Finally, based on the overall system scheduling loss value and a preset scheduling strategy, several agricultural robots are globally scheduled. This forms a closed-loop structure integrating multimodal perception, temporal modeling, conflict perception, task optimization, and natural language interaction, breaking through the limitations of existing modular and fragmented scheduling schemes. It demonstrates stronger system coordination capabilities and adaptability to field deployment, exhibiting significant practical value and promising prospects for wider application. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 A flowchart of an intelligent scheduling method for agricultural robot swarms provided in an embodiment of the present invention. Detailed Implementation

[0017] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] In collaborative operations of multiple agricultural robots, a key challenge is to develop an intelligent scheduling mechanism that can express the spatiotemporal continuity and conflict probability characteristics of robot behavior, while also enabling global scheduling optimization and human-machine semantic interaction feedback. Existing technologies generally struggle to simultaneously achieve the four key capabilities of "high-precision trajectory modeling, dynamic conflict decoupling, multi-factor scheduling optimization, and natural language control," leading to problems such as frequent task conflicts, high energy consumption, and high costs of manual intervention in actual operation. Especially in agricultural scenarios with dense tasks, high overlap in robot paths, and frequent environmental changes, a lack of accurate modeling and dynamic control of task time boundaries can easily cause resource congestion and execution failures. Furthermore, traditional scheduling systems cannot understand ambiguous, unstructured human language commands, nor can they achieve precise scheduling responses based on semantic intent, severely limiting the intelligent interaction capabilities and on-site deployment efficiency of the scheduling system. Therefore, there is an urgent need for a scheduling method for multiple agricultural robots with semantic perception, trajectory modeling, conflict resolution, and instruction alignment capabilities to support a more complex, human-centered, and intelligent agricultural robot swarm collaborative task scheduling system. Therefore, in view of the technical problems existing in the above-mentioned prior art, this application proposes an intelligent scheduling method for agricultural robot swarms.

[0019] The agricultural robot swarm intelligent scheduling method of this application is applied to a scheduling system. The scheduling system is connected to several agricultural robots and is used to coordinate the operation of the agricultural robots based on the collected operation data information of the several agricultural robots.

[0020] Among them, such as Figure 1 As shown, the intelligent scheduling method for agricultural robot swarms includes: Step S100: Perform visual semantic transformation on the visual features of the operation behavior videos of each agricultural robot in the same target operation area within the target time period to obtain the semantic feature vector of each frame of the operation behavior video of each agricultural robot within the target time period. The target time period has a preset length and the target time period ends at the current time.

[0021] In agricultural production environments, agricultural robots typically rely on multiple visual sources (such as forward-facing cameras, side-facing cameras, and drone overhead views) to acquire visual information about the target work area. However, raw image data is difficult for scheduling algorithms to understand directly. Therefore, step S100 proposes a "Multi-Perceptual Transformation Mechanism" (MPTM). The core idea of ​​this mechanism is to use a pre-trained visual encoder to extract features from the input video frame sequence and map these features to a semantic space that can be recognized by a scheduling language model through a set of nonlinear transformations. This mechanism encodes unstructured video frame information into a unified semantic embedding representation, thereby providing consistent and processable feature input for the subsequent scheduling inference module.

[0022] Specifically, step S100 includes steps S110-S130: Step S110: Obtain the operation behavior video of each agricultural robot within the target time period to obtain the operation behavior video list A=(A1,A2,...,A1) i ,...,A m ); where i = 1, 2, ..., m; m is the number of agricultural robots; A i The video of the i-th agricultural robot's operational behavior collected within the target time period; Operational behavior videos are typically video clips that continuously record the operational behavior of an agricultural robot, captured by the robot's visual source.

[0023] Step S120: Perform uniform frame extraction on each operation video to obtain n frames of images corresponding to each operation video. To facilitate subsequent processing, each job action video needs to be preprocessed, that is, each input job action video is sampled into n frames at equal intervals to ensure that different video lengths have consistency in the time dimension, and to ensure a balance between the sparsity of the time series and computational efficiency. n can be 100.

[0024] Step S130: Perform visual semantic transformation on each frame image corresponding to each work behavior video to obtain the semantic feature vector of each frame image corresponding to each work behavior video. Wherein, the semantic feature vector U of the t-th frame image of the video of the i-th agricultural robot's work behavior is... it =tanh(W2×ReLU(W1×Ev(I it )+b1)+b2); t=0,1,...,n; In the formula, tanh() is a preset hyperbolic tangent function that compresses the output range to [-1,1] to enhance semantic boundary control; W2 is a preset weight matrix of the second fully connected neural network in the multilayer perceptron, with dimensions f×f2; ReLU(*) is a preset modified linear unit activation function with an output of max(0,*), used to increase nonlinear expressive power; W1 is a preset weight matrix of the first fully connected neural network in the multilayer perceptron, with dimensions f2×f1; I it Let be the t-th frame of the video representing the operational behavior of the i-th agricultural robot; Ev(I it ) for I it The input is fed into a pre-defined visual perceptron (the visual perceptron is typically an image coding model based on Transformer (a sequence-to-sequence model based on attention mechanism) such as CLIP ViT-L / 14, which has been pre-trained on a large amount of image and text data and has excellent visual semantic abstraction capabilities to ensure that the image coding model maintains consistency in image semantic recognition in agricultural environments), and the output I from the visual perceptron is... it The corresponding visual feature vectors; b1 is the bias vector of the first fully connected neural network in the preset multilayer perceptron, with a dimension of f2×1; b2 is the bias vector of the second fully connected neural network in the preset multilayer perceptron, with a dimension of f×1.

[0025] Where f1 can be 768 or 1024; f2 is set as the intermediate dimension, which can be 512; and f is set as the final semantic space dimension, which can be 256.

[0026] Each frame of the video representing the work activity is input into a visual perceptron, which converts each frame into an f1-dimensional visual feature vector (i.e., Ev(I)). it Next, to align with the semantic space of the text language model, these visual feature vectors need further transformation. This transformation is performed by a Multilayer Perceptron (MLP, fully connected neural network) network A_mlp. This MLP network consists of two fully connected neural network layers, applying ReLU activation function and tanh activation function respectively, ultimately outputting an f-dimensional semantic feature vector U. it This serves as the representation of the image frame in the unified scheduling semantic space. The semantic feature vectors of all image frames form the feature matrix U, with dimensions n×f.

[0027] Through the MPTM mechanism, the scheduling system can uniformly map video information acquired by various visual sensors of agricultural robots into the semantic space of the scheduling language model, enabling downstream tasks (such as temporal localization, scheduling decision-making, and human-machine question answering) to process multi-source data with a unified structure, thereby improving the intelligence, generalization ability, and response speed of the overall scheduling system.

[0028] Step S200: Based on the similarity between the semantic feature vector of each frame image corresponding to each agricultural robot and the preset semantic feature center vector corresponding to each agricultural robot, determine the operation status response value of each agricultural robot at the time corresponding to each frame image. The operation status response value is a real number between 0 and 1, representing the operation status of the agricultural robot at the corresponding moment, and is used to indicate whether the agricultural robot is in an active or critical operation state at that moment.

[0029] Several agricultural robots occur at the same time corresponding to the t-th frame of the image.

[0030] In agricultural operations, multiple agricultural robots (such as harvesting robots, seeding robots, and spraying robots) operate simultaneously in different areas. The execution time and temporal status of these robot tasks are key input information for the scheduling system. However, traditional scheduling methods often treat the operation time as a coarse interval, ignoring the evolution trend, continuity, and boundary ambiguity of the tasks over time. Therefore, step S200 proposes a "Temporal Trajectory Index Encoding" (TTIE) model. By constructing a refined operation time distribution function, it characterizes the dynamic evolution of each agricultural robot's operation behavior over the entire time series. The TTIE model constructs a time density function for each agricultural robot, giving it higher "credibility" of operation activities at certain times, thus achieving finer-grained temporal scheduling support. This function is not a simple interval label but a continuous, weighted probability distribution that dynamically responds to the operation behavior of agricultural robots across the entire time dimension.

[0031] Specifically, the operational status response value T of the i-th agricultural robot at the time corresponding to the t-th frame image. it =softmax_over(a1×exp(-a2×||U it -B i || 2 )); In the formula, softmax_over() is a preset normalization function; a1 is a preset temperature adjustment factor used to adjust the "sharpness" or "smoothness" of the normalized distribution. The larger a1 is, the more the output tends to concentrate on a certain peak; a1=1 or a1=5; a1 is used to balance the smoothness of the softmax_over function result; exp() is an exponential function with the natural constant as the base; a2 is a preset time tightness control factor used to adjust the sensitivity of the exponential distribution to distance. The larger a2 is, the more sensitive the agricultural robot's behavior is to differences in time characteristics; 0.01≤a2≤10; a2 can be fine-tuned through the validation set; B i Let U be the preset semantic feature center vector corresponding to the i-th agricultural robot, representing the "temporal feature centroid" of the i-th agricultural robot's operation behavior in this video segment. It is also an f-dimensional vector, which can be obtained by weighted averaging of the features of the agricultural robot at historical operation moments; ||U it -B i || 2 For U it With B i The semantic similarity between them can be calculated by U it With B i The square of the Euclidean distance between them is determined.

[0032] The operational status response values ​​corresponding to several agricultural robots can be constructed into an operational status response matrix T with dimensions m×n. Each row of the operational status response matrix T represents the complete operational time trajectory distribution of an agricultural robot.

[0033] Specifically, the semantic feature center vector of each agricultural robot in the semantic space can be obtained through prior operation labels or historical trajectories. Specifically, the semantic feature center vector B corresponding to the i-th agricultural robot... i =∑ ei d=1 C id ×U id ; In the formula, ei represents the number of image frames in the video of the i-th agricultural robot performing its work during the target time period; C id The preset time weight (such as mean or Gaussian weight) is given to the d-th frame image of the i-th agricultural robot performing its work in the video of its work behavior within the target time period; U id Let d be the semantic feature vector of the d-th frame image of the i-th agricultural robot performing its work in the video of its work behavior during the target time period.

[0034] For the i-th agricultural robot, the semantic feature vector U of a certain frame image is only valid if... it Approaching its semantic feature center vector B iAt that time, its job status response value T in that frame it Only those values ​​closer to B will be assigned higher values. This method uses an exponential decay mechanism to assign higher values ​​to values ​​closer to B. i Frames with higher weights are assigned to create a centrally symmetric temporal activity probability distribution, thus naturally constructing a trajectory response with clear temporal boundaries but also capable of ambiguous transitions. Applying the softmax_over function to the entire time axis t ensures that for the i-th agricultural robot, it has a unique task distribution throughout the entire time period, and all T... it The sum is 1. This design facilitates frame-level task allocation, boundary detection, conflict analysis, and other operations in subsequent scheduling inference.

[0035] By employing the Time-Track Exponential Encoding (TTIE) model, the scheduling system moves beyond binary "whether a task is being performed" labels. Instead, it characterizes the behavioral trends of agricultural robots over time using a continuous probability distribution. This modeling approach significantly enhances the scheduling system's understanding of task temporal boundaries, making it particularly suitable for optimizing strategies such as conflict detection, priority allocation, and delay coordination in high-density collaborative work scenarios. Furthermore, the exponentially decaying time encoding format offers excellent interpretability and visualization, facilitating understanding of human-machine interaction and scheduling verification processes.

[0036] Step S300: Determine the collaborative pressure difference value of any two agricultural robots within the target time period based on several operational status response values ​​of any two agricultural robots within the target time period. The collaborative pressure difference value represents the overall degree of conflict when two agricultural robots perform tasks within a target time period and their work spaces overlap. The collaborative pressure difference value of the two agricultural robots is directly proportional to the overall degree of conflict between them.

[0037] In agricultural robot systems, multiple agricultural robots typically need to enter adjacent or intersecting areas simultaneously to perform their respective tasks, such as spraying, sowing, or harvesting. When these agricultural robots fail to effectively stagger their movements spatially or temporally, it can lead to problems such as path congestion, resource contention, and task conflicts, thereby affecting overall operational efficiency and safety. Therefore, step S300 proposes a "Collaborative Decoupling via Differential Compression" (CDDC) controller to quantitatively assess the degree of conflict among agricultural robots and guide the scheduling system to reconstruct and optimize task time and paths. The key to this mechanism is the construction of a spatiotemporal overlap index matrix, which measures the degree of behavioral coupling between each pair of robots in the temporal dimension, and introduces a spatial isolation mechanism for differential pressure adjustment. By globally minimizing this matrix, decoupling, conflict avoidance, and path relaxation in group scheduling can be achieved.

[0038] Furthermore, step S300 includes steps S310-S320: Step S310: From the m agricultural robots, randomly select the i-th and j-th agricultural robots as the agricultural robots for determining the collaborative pressure difference value; j=1,2,...,m; and j≠i; Step S320: Determine the cooperative pressure difference value P corresponding to the i-th agricultural robot and the j-th agricultural robot within the target time period. ij =∑ n-1 t=0 (1 / (1+exp(-T it ×T jt )))×(1-S ijt ); The collaborative pressure difference between the i-th and j-th agricultural robots within the target time period is a quantitative indicator that measures whether they frequently perform tasks and have spatial overlap within the same time period.

[0039] In the formula, T it ×T jt This represents the joint activity level of the i-th and j-th agricultural robots at the time corresponding to the t-th frame of the image. A larger value indicates that they are more likely to operate simultaneously within the same frame; S ijt The spatial isolation coefficient is determined based on the overlapping state of the working spaces of the i-th and j-th agricultural robots at the time corresponding to the t-th frame of the image. It is used to indicate whether the i-th and j-th agricultural robots are physically isolated at the time corresponding to the t-th frame of the image; (1-S ijt ) represents the non-isolation strength and its weight in the differential pressure calculation.

[0040] Specifically, the spatial isolation coefficient S between the i-th agricultural robot and the j-th agricultural robot at the time corresponding to the t-th frame image. ijt Determined through steps S321-S322: Step S321: Rasterize the target work area to obtain several grid areas corresponding to the target work area; the area of ​​each grid area is equal. Step S322: If the grid region where the i-th agricultural robot is located at the time corresponding to the t-th frame image is different from the grid region where the j-th agricultural robot is located at the time corresponding to the t-th frame image, and the distance between the position of the i-th agricultural robot at the time corresponding to the t-th frame image and the position of the j-th agricultural robot at the time corresponding to the t-th frame image is greater than a preset interval distance, then let S... ijt =1 (indicating that the two are in different sub-regions or height layers, with no actual interference); otherwise, S ijt =0 (indicating that there is significant spatial overlap between the two).

[0041] P ij This can be accomplished through matrix parallel multiplication, supports GPU acceleration, and is especially suitable for large-scale robotic systems. Furthermore, the scheduling system can set conflict warning thresholds; when P... ij When the conflict threshold is exceeded, rescheduling logic is triggered. Furthermore, P can also be... ij Converting the matrix into a heatmap can visually show which agricultural robots are densely coupled, which helps schedulers make informed decisions.

[0042] The Collaborative Differential Pressure Decoupling Controller (CDDC) combines temporal and spatial factors to characterize the comprehensive conflict degree between different agricultural robots in two dimensions: "whether they operate simultaneously at a certain moment" and "whether the operation occurs in an overlapping area." Among these, T... it ×T jt This reflects the overlap in the timing of the tasks, (1-S) ijt The product of 1 / (1+exp(-T)) indicates the degree of spatial overlap. The larger the product, the more significant the spatiotemporal coupling between the i-th and j-th agricultural robots at that moment. it ×T jt The sigmoid function is used to further nonlinearly compress the value to reduce the impact of extreme values ​​on the total pressure difference and enhance the model's sensitivity to edge conflicts.

[0043] Step S400: Determine the overall system scheduling loss value based on several operational status response values ​​of any agricultural robot within the target time period, and the collaborative pressure difference value between the agricultural robot and other agricultural robots besides the agricultural robot. In intelligent scheduling systems for multi-robot collaborative operations, simply modeling spatiotemporal conflicts (such as pressure difference index P) is insufficient. ij This is still insufficient to support efficient allocation of the entire task. Therefore, the scheduling system needs to comprehensively consider multiple factors such as job energy consumption, robot task importance, time distribution, and mutual influence to generate a set of job scheduling schemes that are "highly executable, low in energy consumption, and have minimal conflicts." Therefore, step S400 proposes the "Task Compatibility Optimization Engine" (TCOE), which is the core of the scheduling optimization built based on this requirement. This engine uses the overall scheduling loss function L... opt With the core as the basis, through the operation trajectory T it By weighting and integrating the collaborative pressure difference and the operation cost, a differentiable optimization objective is constructed, which is applicable to various optimization algorithms (such as evolutionary strategies, reinforcement learning schedulers, integer linear programming, etc.). The final output is the start and end times of the agricultural robot's task, the specific operation path, and the collaborative avoidance strategy between robots on the entire time axis.

[0044] Furthermore, step S400 includes step S410: Step S410: Based on the response values ​​of several operational states of the i-th agricultural robot within the target time period, and the collaborative pressure difference between the i-th agricultural robot and other agricultural robots besides the i-th agricultural robot, determine the overall system scheduling loss value L. opt =∑ m i=1 ∑ n-1 t=0 (D i ×T it ×(E it +∑ m j=1 (F ij ×P ij ))); In the formula, D i This is a preset task priority factor for the i-th agricultural robot. A larger value indicates a more critical task for that robot (e.g., main task robot, time-limited task). It is configured by the user or output by the task parser and has dynamic adjustability. it The energy consumption value per unit for the i-th agricultural robot at the time corresponding to the t-th frame of the image is usually estimated from path planning results or device power consumption model. It can be estimated by mapping the travel path and action commands, for example, in spraying tasks, using unit liquid consumption and path length; F ijThe weighting coefficient is a preset value for the interference of the i-th agricultural robot by the j-th agricultural robot. It can be dynamically adjusted by combining the historical interaction intensity or importance, and can be obtained through simulation scenarios or historical conflict samples.

[0045] Through the overall scheduling loss function L opt After processing, the fields returned by each agricultural robot include: [start time, end time, task path, task action sequence], which can be used for modules such as control deployment, resource pre-allocation, and path obstacle avoidance.

[0046] Overall scheduling loss function L opt The basic idea is to extend the scheduling task from a single objective (such as minimizing energy consumption) to a multi-factor weighted summation model. For each agricultural robot, its scheduling cost is determined not only by its operating energy consumption at each moment (E it It is also affected by the degree of conflict between other agricultural robot tasks (P) ij To enhance the model's practical scheduling flexibility, D i The parameters provide a means of task priority control, which can be used to prioritize high-urgency robots (such as pilot robots and unmanned transport vehicles) and reduce the frequency of scheduling interventions for low-priority equipment. And F ij This reflects the structural cooperative constraints among agricultural robots. For example, some robots are inherently large or highly disruptive, and should be given a higher disruption weight. The entire function ultimately forms a multi-objective weighted cumulative function, which, by minimizing it, generates the optimal scheduling path and time configuration.

[0047] Step S500: Perform global scheduling of several agricultural robots based on the overall system scheduling loss value and the preset scheduling strategy.

[0048] After receiving the field information returned by each agricultural robot, the scheduling system performs global scheduling of several agricultural robots according to the preset scheduling strategy. The scheduling strategy is a user-preset scheduling scheme, such as scheduling an agricultural robot with a preset trajectory and preset task when a certain scheduling condition is met.

[0049] In addition, the scheduling system is also used to execute steps S010-S020: Step S010: In response to receiving a natural language query statement input by the user, obtain a behavior description statement of the operation behavior of each agricultural robot at the time corresponding to each frame of the image within the target time period. Step S020: Based on the semantic similarity between the behavior description statement and the natural language query statement of each agricultural robot at the time corresponding to each frame of the image within the target time period, determine the target response statement corresponding to the natural language query statement. In multi-robot intelligent scheduling systems, traditional control methods typically rely on parameter settings or structured operation commands, which pose a high barrier to entry for agricultural managers and hinders real-time control and flexible deployment. To address this issue, step S020 introduces a "Fuzzy Semantic-Driven Interaction (FSDI) module," enabling the scheduling system to support control interaction at the natural language level. Users can use near-conversational commands, such as "Which robot was working in the south area just now?" or "Is there a robot spraying now?", to perform scheduling queries or interventions without inputting precise coordinates or numbers. The core idea of ​​the fuzzy semantic-driven interaction module is to establish a semantic similarity mapping between the user's input natural language query and the robot behavior records in the scheduling system. By combining trajectory probability distribution and robot importance weighting, the most matching target robot and its behavioral state are predicted. The final result not only supports scheduling queries but can also be used for reverse reasoning of target tasks, scheduling adjustments, and decision support.

[0050] Furthermore, step S020 includes steps S021-S023: Step S021: Determine the semantic query value corresponding to each agricultural robot based on the semantic similarity between the behavior description statement and the natural language query statement of each frame image within the target time period. Wherein, the semantic query value G corresponds to the i-th agricultural robot. i =∑ n-1 t=0 (H i ×sim(Q text J it )×T it ); In the formula, H i Q represents the importance weight of the i-th agricultural robot, indicating its priority in the scheduling process. This importance weight can be dynamically set based on factors such as task level, historical feedback, and criticality, or it can be statically set (i.e., assigning a uniform value to all agricultural robots). text For natural language queries, such as "Which robot was just in the southeast corner?"; J it This is a behavioral description of the i-th agricultural robot's operation at the time corresponding to the t-th frame of the image. It is a natural language description automatically constructed based on the robot's identifier, location, and operational actions, such as "Robot 1 is spraying, location: South area," or "Robot 2 is sowing in the North area," etc.; sim(Q text J it) is a function for determining the semantic similarity between a natural language query and the behavior description of the i-th agricultural robot at the time corresponding to the t-th frame image. It is usually evaluated using sentence vector cosine similarity or a large language model. A bidirectional encoder (such as BERT or RoBERTa) is used to extract sentence vectors and calculate cosine similarity.

[0051] Step S022: Among several agricultural robots, the agricultural robot with the largest semantic query value is identified as the target robot; The target robot is the robot entity that best matches the semantics of the instructions and the behavior of the time trajectory.

[0052] Step S023: Determine the target robot's response to the natural language query as the target response statement.

[0053] After identifying the target robot, the corresponding response information (such as agricultural robot identifier, current task, start and end time, and image snapshot) is obtained based on the natural language query statement. The response information is then integrated into the target response statement in natural language.

[0054] The fuzzy semantic-driven interaction module uses the user's input natural language query statement Q text At its core, the system uses a semantic matching mechanism to determine which agricultural robot performed the task that best fits the given semantics at what time. For each agricultural robot, the scheduling system iterates through the behavior description statements J of all frames in its entire time series. it and using sim(Q) text J it Calculate the semantic proximity between the description and the user's intent; then multiply the semantic proximity by the intensity T of the agricultural robot's operational state at that moment. it Finally, a weighted summation is performed on all time frames. i This is used to adjust the priority of different agricultural robots, preventing all robots from responding to ambiguous commands with the same probability. Through the above weighted sum, the scheduling system can automatically determine the agricultural robot and its behavior segment most likely to respond to the natural language command. The final target response statement can be used for: outputting a description of the current job status, extracting key image frames for visual feedback, intervening in the scheduling system (such as "pause the robot's current job"), and guiding subsequent command chains (such as asking "When did this job start?").

[0055] Therefore, to address the challenges in modeling trajectory conflicts, lack of global optimization capabilities in scheduling strategies, and insufficient intelligence in user command interaction during multi-robot collaborative agricultural operations, this invention proposes a scheduling method with closed-loop intelligent control capabilities, which has the following key innovations: First, an innovative Time Trajectory Exponential Encoding (TTIE) model was constructed. Based on frame-level semantic features, the working state of each agricultural robot was continuously probabilistically modeled. For the first time, the dynamic response expression of agricultural robot behavior in the time dimension was realized. It can accurately capture the start and end boundaries and intensity changes of tasks, providing refined data support for conflict judgment and scheduling optimization, as well as frame-level probabilistic support for path conflict control and natural language retrieval. TTIE constructs the continuous working state distribution in the time domain, providing accurate time overlap weights for the Cooperative Differential Pressure Decoupling Controller (CDDC). At the same time, it provides a timing pointer for the fuzzy language localization of the fuzzy semantic-driven interaction module FSDI, enabling it to parse uncertain time expressions such as "just now" and "a while later".

[0056] Secondly, a Collaborative Differential Pressure Decoupling Controller (CDDC) is proposed. This controller integrates the distribution of operational trajectories with spatial location relationships to construct a dynamic conflict matrix between robots. This matrix forms a crucial constraint and evaluation basis for scheduling optimization. Furthermore, a nonlinear compression function is used to quantify and adjust conflict risk, thereby achieving proactive intervention and path separation for high-density operational behaviors. The conflict matrix P output by the CDDC is... ij It not only reflects the behavioral interference between agricultural robots in the spatial and temporal dimensions, but also provides the most core conflict cost item for the Task Compatibility Optimization Engine (TCOE), achieving true "cooperative decoupling".

[0057] Furthermore, a Task Compatibility Optimization Engine (TCOE) was developed. TCOE is the central computing unit of the multi-objective fusion scheduling system. It dynamically combines the trajectory distribution of TTIE, the conflict intensity of CDDC, the job energy consumption model, and the task priority to output the optimal scheduling structure. It can be regarded as the "convergence point" and "decision center" of all the previous steps. It integrates job energy consumption, task priority, time distribution, and conflict cost into a multi-objective loss function and generates the optimal task arrangement through a global optimization method, which effectively reduces the overall system energy consumption and improves resource utilization efficiency.

[0058] Finally, a fuzzy semantic-driven interaction module (FSDI) was introduced, which supports dynamic matching of natural language instruction parsing with job trajectories and semantic descriptions. This allows users to query scheduling status or intervene in task execution in a fuzzy or colloquial manner. FSDI establishes a bridge from natural language to trajectory distribution, enabling the scheduling system to "understand people" and connect user intent with system status in reverse, thus opening up a human-machine closed loop. Its input processing relies on the embedding space of MPTM, time encoding consistent with TTIE, and scheduling status interface with TCOE, ultimately closing the "input-decision-feedback" interaction loop and greatly improving the system's human-machine collaboration capabilities.

[0059] Through the combined application of the above-mentioned innovative modules, this invention constructs a closed-loop intelligent scheduling system covering the entire process from perception, modeling, optimization to interaction. The various technical features are highly complementary, progressively supportive, and information-advancing in a closed-loop structure. Each module not only completes its own independent task but also provides key inputs or constraints for downstream steps, forming a collaborative structure covering the entire process from perception to decision-making to interaction. This effectively solves the key technical problems of traditional methods in spatiotemporal modeling, conflict prediction, and human-computer interaction, and significantly improves the group collaboration efficiency and deployment intelligence level of agricultural robots in real and complex environments.

[0060] To address the core technical problem of "intelligent collaborative scheduling of multiple agricultural robots in complex agricultural scenarios" solved by this invention, several alternative solutions exist, such as task priority tables based on static rules, job graph modeling methods based on graph neural networks, or scheduling strategy generation using multi-agent reinforcement learning. While these solutions can achieve preliminary scheduling goals in specific task types or scenarios, they still have significant shortcomings in terms of systematicity, real-time performance, and interactivity. For example, graph-based scheduling schemes typically establish a connected graph between task points and paths, achieving resource scheduling by solving for the shortest path or maximum flow. However, these methods struggle to accurately model the continuity of robot behavior over time and cannot handle issues such as ambiguous job time boundaries and overlapping operations by multiple robots. While multi-agent reinforcement learning methods possess adaptive capabilities, they face bottlenecks in agricultural scenarios, such as scarce training samples and unstable environmental modeling. Furthermore, they are costly to train and complex to deploy, making them difficult to implement in actual farms. In addition, some traditional systems attempt to control conflicts by setting up task queues or deploying virtual traffic lights, but these methods lack an understanding of the semantics of the task content and cannot provide high-level user interaction support. Their ability to respond to and interpret natural language commands is extremely limited.

[0061] Therefore, compared with existing multi-robot scheduling systems based on graph theory, rule tables, or traditional path planning algorithms, this invention achieves significant improvements in scheduling expression capabilities, conflict perception accuracy, human-robot interaction intelligence, and global optimization capabilities, and has the following key advantages: First, existing technologies typically employ discrete task labeling methods (such as task block numbering or fixed time period allocation), which cannot accurately capture the continuous evolution of robot behavior over time. This invention, however, constructs a Time Trajectory Exponential Encoding (TTIE) model, achieving frame-level continuous modeling of robot task states. This model can express fuzzy behavioral boundaries and changes in task intensity, significantly improving the scheduling system's adaptability to dynamic behaviors.

[0062] Secondly, traditional scheduling algorithms struggle to effectively identify implicit conflicts between robots caused by spatial proximity or task overlap, typically relying on preset avoidance rules or static safety distances, resulting in conservative scheduling outcomes and low resource utilization. The Cooperative Differential Pressure Decoupling Controller (CDDC) proposed in this invention overcomes this bottleneck. By combining temporal distribution and spatial isolation characteristics to construct a conflict matrix, it accurately quantifies the risk of task overlap and supports dynamic adjustment and real-time avoidance, thereby improving the system's scheduling flexibility and safety.

[0063] Furthermore, compared to existing scheduling systems that lack user-friendly interaction mechanisms and cannot understand unstructured input, this invention introduces a fuzzy semantic-driven interaction module (FSDI), which combines semantic matching of a large language model with joint reasoning of trajectory behavior. This allows users to directly complete status queries and scheduling control using natural language such as "Who was working in the southeast corner just now?", significantly improving operational convenience and system intelligence.

[0064] Finally, in terms of overall systemic approach, this invention forms a closed-loop structure integrating multimodal perception, temporal modeling, conflict perception, task optimization, and natural language interaction. These five technical features exhibit strong coupling and high support, and their combination not only creates a synergistic mechanism in terms of functionality but also demonstrates overall practical benefits far exceeding the sum of the capabilities of individual modules. This combination approach demonstrates outstanding substantive characteristics in structure, algorithm, and application path, achieving significant technological advancements in intelligent scheduling, human-computer interaction, robustness, and generalization. It breaks through the limitations of existing modular and fragmented scheduling schemes, exhibiting stronger system synergy and adaptability to real-world deployments, and possesses significant practical value and promising prospects for widespread application.

[0065] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for intelligent scheduling of agricultural robot swarms, characterized in that, It is applied to a scheduling system, which is connected to several agricultural robots; The method for intelligent scheduling of agricultural robot swarms includes: Step S100: Perform visual semantic transformation on the visual features of the operation behavior videos of each agricultural robot in the same target operation area within the target time period to obtain the semantic feature vector of each frame of the operation behavior video of each agricultural robot within the target time period; the length of the target time period is a preset duration, and the end time of the target time period is the current time. Step S200: Based on the similarity between the semantic feature vector of each frame image corresponding to each agricultural robot and the preset semantic feature center vector corresponding to each agricultural robot, determine the operation status response value of each agricultural robot at the time corresponding to each frame image; the operation status response value represents the operation status of the agricultural robot at the corresponding time. Step S300: Based on several operational status response values ​​corresponding to any two agricultural robots within the target time period, determine the collaborative pressure difference value corresponding to any two agricultural robots within the target time period; the collaborative pressure difference value characterizes the overall conflict degree of overlapping operational spaces when the two agricultural robots perform operational tasks within the target time period; the collaborative pressure difference value corresponding to the two agricultural robots is directly proportional to the overall conflict degree corresponding to the two agricultural robots; Step S400: Determine the overall system scheduling loss value based on several operational status response values ​​of any of the agricultural robots within the target time period, and the collaborative pressure difference value between the agricultural robot and other agricultural robots besides the agricultural robot. Step S500: Based on the overall system scheduling loss value and the preset scheduling strategy, perform global scheduling on several of the agricultural robots.

2. The method according to claim 1, characterized in that, Step S100 includes: Step S110: Obtain the operation behavior video of each agricultural robot within the target time period to obtain an operation behavior video list A=(A1,A2,...,A1). i ,...,A m ); where i = 1, 2, ..., m; m is the number of agricultural robots; A i The video of the i-th agricultural robot's operational behavior collected within the target time period; Step S120: Perform uniform frame extraction on each of the aforementioned work behavior videos to obtain n frames of images corresponding to each of the aforementioned work behavior videos; Step S130: Perform visual semantic transformation on each frame image corresponding to each of the aforementioned work behavior videos to obtain the semantic feature vector of each frame image corresponding to each of the aforementioned work behavior videos. Wherein, the semantic feature vector U of the t-th frame image of the video of the i-th agricultural robot's work behavior is... it =tanh(W2×ReLU(W1×Ev(I it )+b1)+b2); t=0,1,...,n; In the formula, tanh() is the preset hyperbolic tangent function; W2 is the preset weight matrix of the second fully connected neural network in the multilayer perceptron; ReLU() is the preset modified linear unit activation function; W1 is the preset weight matrix of the first fully connected neural network in the multilayer perceptron; I it Let be the t-th frame of the video representing the operational behavior of the i-th agricultural robot; Ev(I it ) for I it The input is fed into a preset visual sensor, and the I signal output by the visual sensor is... it The corresponding visual feature vector; b1 is the bias vector of the first fully connected neural network in the preset multilayer perceptron; b2 is the bias vector of the second fully connected neural network in the preset multilayer perceptron.

3. The method according to claim 2, characterized in that, Step S200 includes: Step S210: The operational status response value T of the i-th agricultural robot at the time corresponding to the t-th frame image. it =softmax_over(a1×exp(-a2×||U it -B i || 2 )); In the formula, softmax_over() is the preset normalization function; a1 is the preset temperature adjustment factor; a1=1 or a1=5; exp() is the exponential function with the natural constant as the base; a2 is the preset time tightness control factor; 0.01≤a2≤10; B i Let ||U be the preset semantic feature center vector corresponding to the i-th agricultural robot; it -B i || 2 For U it With B i Semantic similarity between them; Among them, several of the agricultural robots have the same time corresponding to the t-th frame image.

4. The method according to claim 3, characterized in that, The semantic feature center vector B corresponding to the i-th agricultural robot i =∑ ei d=1 C id ×U id ; Where ei is the number of image frames in the video of the i-th agricultural robot performing its work during the target time period; C id The preset time weight is the d-th frame image corresponding to the i-th agricultural robot's operation in the video of its operation behavior within the target time period; U id The semantic feature vector of the d-th frame image of the i-th agricultural robot performing its work in the video of its work behavior in the target time period.

5. The method according to claim 4, characterized in that, Step S300 includes: Step S310: From the m agricultural robots, select the i-th agricultural robot and the j-th agricultural robot as the agricultural robots for determining the cooperative pressure difference value; j=1,2,...,m; and j≠i; Step S320: Determine the cooperative pressure difference value P corresponding to the i-th and j-th agricultural robots within the target time period. ij =∑ n-1 t=0 (1 / (1+exp(-T it ×T jt )))×(1-S ijt ); In the formula, S ijt The spatial isolation coefficient is determined based on the overlapping state of the working spaces of the i-th and j-th agricultural robots at the time corresponding to the t-th frame image.

6. The method according to claim 5, characterized in that, The spatial isolation coefficient S between the i-th and j-th agricultural robots at the time corresponding to the t-th frame image. ijt Determined through the following steps: Step S321: The target work area is rasterized to obtain several raster areas corresponding to the target work area; the area of ​​each raster area is equal. Step S322: If the grid region where the i-th agricultural robot is located at the time corresponding to the t-th frame image is different from the grid region where the j-th agricultural robot is located at the time corresponding to the t-th frame image, and the distance between the position of the i-th agricultural robot at the time corresponding to the t-th frame image and the position of the j-th agricultural robot at the time corresponding to the t-th frame image is greater than a preset interval distance, then let S... ijt =1; otherwise, S ijt =0.

7. The method according to claim 6, characterized in that, Step S400 includes: Step S410: Based on the several operational status response values ​​of the i-th agricultural robot within the target time period, and the cooperative pressure difference value between the i-th agricultural robot and the other agricultural robots besides the i-th agricultural robot, determine the overall system scheduling loss value L. opt =∑ m i=1 ∑ n-1 t=0 (D i ×T it ×(E it +∑ m j=1 (F ij ×P ij ))); In the formula, D i E is the preset task priority factor corresponding to the i-th agricultural robot; it F is the unit energy consumption cost of the i-th agricultural robot at the time corresponding to the t-th frame of the image; ij The preset weighting coefficient is used to determine the interference of the i-th agricultural robot with the j-th agricultural robot.

8. The method according to claim 7, characterized in that, The scheduling system is also used to perform the following steps: Step S010: In response to receiving a natural language query statement input by the user, obtain a behavior description statement of the operation behavior of each agricultural robot at the time corresponding to each frame of the image within the target time period; Step S020: Based on the semantic similarity between the behavioral description statement of the operation behavior of each agricultural robot at the time corresponding to each frame of the image within the target time period and the natural language query statement, determine the target response statement corresponding to the natural language query statement.

9. The method according to claim 8, characterized in that, Step S020 includes: Step S021: Determine the semantic query value corresponding to each agricultural robot based on the semantic similarity between the behavioral description statement of the operation behavior of each frame image in the target time period and the natural language query statement. Step S022: Among the various agricultural robots, the agricultural robot with the largest semantic query value is identified as the target robot; Step S023: Determine the response statement of the target robot to the natural language query statement as the target response statement.

10. The method according to claim 9, characterized in that, The semantic query value G corresponding to the i-th agricultural robot i =∑ n-1 t=0 (H i ×sim(Q text J it )×T it ); In the formula, H i Q is the preset importance weight corresponding to the i-th agricultural robot; text The natural language query statement; J it The statement describing the behavior of the i-th agricultural robot at the time corresponding to the t-th frame of the image; sim(Q text J it ) is a function for determining the semantic similarity between the natural language query statement and the behavior description statement of the i-th agricultural robot at the time corresponding to the t-th frame image.

Images