An agent closed-loop decision method of a robot and the robot

Abstract

The application provides an intelligent body closed-loop decision method of a robot and the robot, and relates to the technical field of robot decision, and the method comprises the following steps: acquiring picking data; multiple intelligent bodies make collaborative decisions according to the picking data, which comprises the following steps: inputting multi-modal input data into a perception intelligent body to obtain a fruit candidate set; inputting the fruit candidate set into an agronomy model intelligent body and verifying the ripening mechanism to obtain ripening picking data; inputting the ripening picking data and picking task data into a task planning intelligent body to obtain a task-level picking sequence; inputting the task-level picking sequence into a picking control intelligent body to generate a motion candidate sequence and verify the dynamic reachability to obtain an execution trajectory; monitoring the safety state through a safety management intelligent body, and triggering a safety degradation action when an anomaly is detected; and collecting the working data of the robot and screening sample data. According to the application, the action illusion, fruit damage risk and safety hazards are reduced, and the application is convenient for tracing and optimizing.

Description

Technical Field

[0001] This invention relates to the field of robot decision-making technology, and more specifically, to a closed-loop decision-making method for a robot agent and a robot. Background Technology

[0002] Harvesting robots are typically composed of target detection, 3D localization, path planning, and robotic arm control modules connected in series. The development of large-scale models and visual-language-action models has provided robots with the possibility of unifying the modeling of vision, language, task knowledge, and action output.

[0003] However, in related technologies, in scenarios such as jujube and blueberry harvesting where there is dense shading, significant differences in ripeness, fragile fruit surfaces, and rapidly changing on-site environments, the harvesting robots are mainly composed of the aforementioned modules connected in series. There is a lack of information sharing and collaborative optimization between the modules, making it difficult to achieve cross-module reasoning and task-level adaptation. Furthermore, the general robot VLA model (Vision-Language-Action Model) cannot directly meet the stringent requirements of agricultural harvesting.

[0004] When a general-purpose robot VLA model directly controls a harvesting robot, there are risks such as model illusion, unreachable actions, neglecting fruit damage, ignoring the safety boundaries of the robotic arm, and being unable to explain agronomic reasons. At the same time, the large amount of first-person perspective videos and fruit damage records generated over a long period of operation are only passively archived and cannot form a continuous improvement capability. Summary of the Invention

[0005] The problem this invention aims to solve is how to overcome the shortcomings of directly controlling a harvesting robot with a single large model, such as illusion, unreachable movements, high risk of fruit damage, and inability to continuously evolve.

[0006] To address the aforementioned problems, in a first aspect, the present invention provides a closed-loop decision-making method for a robot's intelligent agent, comprising: Acquire harvesting data, which includes multimodal input data and harvesting task data; Multiple pre-defined agents make collaborative decisions based on the harvesting data, including: inputting the multimodal input data into a perception agent to obtain a fruit candidate set; inputting the fruit candidate set into an agronomic model agent and verifying the ripening mechanism to obtain mature harvesting data; inputting the mature harvesting data and the harvesting task data into a task planning agent to obtain a task-level harvesting sequence; inputting the task-level harvesting sequence into a harvesting control agent to generate an action candidate sequence and verify its power reachability to obtain the robot's execution trajectory; and having a safety management agent monitor the safety status in real time during the robot's harvesting process and trigger a safety degradation action when an anomaly is detected. The robot's operational data is collected and sample data is filtered. The sample data is used to update the decision parameters of the harvesting control agent.

[0007] This invention utilizes the cooperation of multiple intelligent agents. The perception agent generates a candidate set of fruits, providing a data foundation for subsequent decision-making. The agronomic model agent obtains agronomic constraint states through an external mechanistic model and cross-validates them with visual perception results to obtain mature harvesting data, avoiding the illusions caused by large models relying solely on visual judgment of maturity. The task planning agent generates a task-level harvesting sequence based on mature harvesting data and harvesting task data. The harvesting control agent generates a candidate sequence of actions and verifies their reachability to obtain the execution trajectory, eliminating the risks of unreachable actions and collisions. The safety management agent monitors in real time. The system establishes a safe state and triggers safety degradation actions to achieve full-process safety protection. The data governance agent collects operational data and filters sample data to update decision parameters, enabling continuous model evolution. In summary, the multi-agent architecture divides and coordinates perception, agronomic deduction, action planning, and safety verification. The large model is only responsible for generating action candidates, and its output must be cross-validated by agronomy, power accessibility, and safety before it can be executed. This fundamentally reduces the risk of action illusion, fruit damage, and safety hazards. At the same time, it provides a clear explanation chain so that the causes of fruit damage and misharvesting can be accurately located to specific links, facilitating traceability and optimization.

[0008] In a second aspect, the present invention provides a robot, including a memory and a processor; The memory is used to store computer programs; The processor is configured to implement the robot's intelligent agent closed-loop decision-making method as described in the first aspect when executing the computer program.

[0009] Thirdly, the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the robot's intelligent agent closed-loop decision-making method as described in the first aspect.

[0010] The robot's intelligent agent closed-loop decision-making method, robot, and computer-readable storage medium provided by this invention have the same beneficial effects as the robot's intelligent agent closed-loop decision-making method compared to the prior art, and will not be repeated here. Attached Figure Description

[0011] Figure 1 A flowchart illustrating a closed-loop decision-making method for a robot agent according to an embodiment of the present invention is shown. Figure 2 This diagram illustrates the process of agent cooperation in an embodiment of the present invention. Figure 3A schematic diagram of the robot according to an embodiment of the present invention is shown. Detailed Implementation

[0012] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0013] It should be noted that relational terms such as "first" and "second" in this invention are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0014] In the description of this specification, references to terms such as "embodiment," "one embodiment," and "one implementation" indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or implementation is included in at least one embodiment or illustrative implementation of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or implementation. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or implementations.

[0015] Reference Figure 1 As shown in the figure, this embodiment of the invention proposes a closed-loop decision-making method for a robot's intelligent agent; The robot's agent closed-loop decision-making method includes: S100: Acquire harvesting data, which includes multimodal input data and harvesting task data.

[0016] Specifically, the robot collects harvesting data in real time through multimodal sensor components deployed on it (such as RGB cameras, depth cameras, LiDAR, force sensors, GPS, and inertial measurement units). Multimodal input data includes image or video data, 3D point clouds, robot pose status, force feedback, and environmental time-series variables (such as temperature, humidity, and light intensity). Harvesting task data consists of user-defined operational goals given in natural language, such as "prioritize harvesting winter jujubes that are 80% ripe" or "keep the fruit bloom intact," including but not limited to task-level instructions such as crop type, plot number, target quality, harvesting time limit, and allowable fruit loss rate.

[0017] S200: Multiple pre-defined intelligent agents make collaborative decisions based on the harvesting data, including: inputting the multimodal input data into a perception intelligent agent to obtain a fruit candidate set; inputting the fruit candidate set into an agronomic model intelligent agent and verifying the ripening mechanism to obtain mature harvesting data; inputting the mature harvesting data and the harvesting task data into a task planning intelligent agent to obtain a task-level harvesting sequence; inputting the task-level harvesting sequence into a harvesting control intelligent agent to generate an action candidate sequence and verify the power reachability to obtain the robot's execution trajectory; and having a safety management intelligent agent monitor the safety status in real time during the robot's harvesting process and trigger a safety degradation action when an anomaly is detected.

[0018] Specifically, the perception agent receives multimodal input data and invokes at least one of the following models: target detection, semantic segmentation, 3D reconstruction, maturity recognition, and anomaly detection. It identifies jujube or blueberry fruits, leaves, branches, obstacles, lesions, picking containers, personnel, and restricted areas, outputting a candidate fruit set. The agronomic model agent, based on the candidate fruit set, invokes an external mechanistic model (e.g., a growth-maturity digital twin system based on the LINTUL-3 heuristic model) to obtain the agronomic constraints of each candidate fruit (e.g., thermal time, light accumulation, dry matter, maturity index, picking window). It then cross-validates the agronomic constraints with the visual perception results in the candidate fruit set, obtaining maturity picking data based on the cross-validation results. Finally, the task planning agent, based on the maturity picking data and picking task data, generates a task-level picking sequence by considering maturity benefits, over-ripening risk, chassis movement costs, and robotic arm movement costs. The harvesting control agent generates action candidate sequences by calling the visual language action model based on the task-level harvesting sequence. These action candidate sequences are then verified through collision detection, kinematic reachability verification, and dynamic constraint verification. The verified action candidate sequences are output as execution trajectories. The safety management agent performs safety status monitoring before, during, and after harvesting. If any of the following conditions—collision probability, end-effector force, personnel distance, or fruit damage probability—does not meet the corresponding threshold, a safety degradation action is triggered (e.g., refusal to execute, reduction of speed, withdrawal of the robotic arm, skipping the target, or requesting manual review).

[0019] S300: Collect the robot's operation data and filter sample data, which is used to update the decision parameters of the harvesting control agent.

[0020] Specifically, operational data generated during robot operations is collected through data governance agents and other means. This includes records of successful or failed harvesting, fruit damage detection results, safety degradation event records, and conflicting samples where visual perception results are inconsistent with agronomic constraints. The data governance agent selects high-value samples from the operational data based on a value scoring formula. The value scoring formula includes, but is not limited to, maturity confidence, occlusion coefficient, successful or failed harvesting records, fruit damage detection results, and the degree of inconsistency between visual perception results and agronomic constraints. Samples with value scores exceeding a preset threshold are labeled and stored in a training repository for training the visual language action model invoked by the harvesting control agent. After offline evaluation, the trained model is deployed to the edge. If the operational metrics of the new model are lower than those of the current model, a rollback is triggered, meaning the new model is abandoned and the current model is reverted to use.

[0021] It should be noted that the above steps S100 to S300 constitute the complete process of the intelligent agent closed-loop decision-making method of the robot of the present invention. In this invention, step S100 is the data acquisition stage, step S200 is the multi-agent collaborative decision-making stage, and step S300 is the data flywheel update stage. Through the division of labor and collaboration among multiple specialized agents, such as the perception agent, this invention achieves a closed-loop decision-making process for the harvesting robot, encompassing "perception → decision-making → execution → evaluation → update." The perception agent generates a fruit candidate set based on multimodal input data. The agronomic model agent uses an external mechanism model to obtain agronomic constraint states based on the fruit candidate set and cross-validates these states with the visual perception results in the fruit candidate set to obtain mature harvesting data. The task planning agent generates a task-level harvesting sequence based on the mature harvesting data and harvesting task data. The harvesting control agent uses a visual language action model to generate action candidate sequences and performs collision detection verification, kinematic reachability verification, and dynamic constraint verification, using the verified action candidate sequences as execution trajectories. The safety management agent monitors the safety status in real time and triggers safety degradation actions. The data management agent collects operational data and filters sample data to update the decision parameters of the harvesting control agent.

[0022] The reason why this invention uses multiple dedicated intelligent agents instead of a single large model for direct control is that a single large model is a black box. Visual input directly outputs joint torque, which can easily produce fatal motion illusions under unfamiliar orchard lighting conditions. For example, it may mistakenly identify fruit that has turned red due to disease as ripe and grab it forcefully, or plan the robotic arm to an unreachable position outside the joint limits. The multi-agent architecture decouples perception, agronomic deduction, action planning, and safety verification. The large model is only responsible for generating action candidates. Its output must undergo mechanistic cross-verification by the agronomic model agent, three-layer execution verification by the harvesting control agent, and a veto safety verification by the safety governance agent before it can be executed. The "absolute control" of the large model is gradually stripped away, fundamentally eliminating action illusions, fruit damage risks (by cutting off the sources of fruit damage risks at three levels: the agronomic model agent prevents the accidental harvesting of unripe fruit through mechanistic cross-verification; the harvesting control agent prevents collision damage and clamping force damage through dynamic accessibility verification; and the safety governance agent intervenes promptly through real-time monitoring of anomalies, with these three mechanisms jointly ensuring harvest quality) and safety hazards. At the same time, the multi-agent architecture provides a clear chain of explanations. For example, if vision indicates ripeness but the mechanism shows insufficient accumulated temperature, the planner's inverse kinematics has no solution, or safety detection triggers a degradation, the cause of fruit damage and accidental harvesting can be accurately located to perception errors, decision-making mistakes, or execution deviations, facilitating tracing and optimization.

[0023] In practical application, this embodiment involves the following: the perception agent generates a candidate set of fruits, providing a data foundation for subsequent decision-making; the agronomic model agent obtains agronomic constraint states through an external mechanism model and cross-validates them with visual perception results to obtain mature harvesting data, avoiding the illusion caused by large models relying solely on visual judgment of maturity; the task planning agent generates a task-level harvesting sequence based on mature harvesting data and harvesting task data; the harvesting control agent generates a candidate sequence of actions and verifies their reachability to obtain the execution trajectory, eliminating the risks of unreachable actions and collisions; and the safety management agent monitors safety in real time. The system operates in all states and triggers safety degradation actions to achieve full-process safety protection. The data governance agent collects operational data and filters sample data to update decision parameters, enabling continuous model evolution. In summary, the multi-agent architecture divides and coordinates perception, agronomic deduction, action planning, and safety verification. The large model is only responsible for generating action candidates, and its output must be cross-validated by agronomy, power accessibility, and safety before it can be executed. This fundamentally reduces the risk of action illusion, fruit damage, and safety hazards. At the same time, it provides a clear explanation chain so that the cause of fruit damage and misharvesting can be accurately located to specific links, facilitating traceability and optimization.

[0024] This invention is mainly applicable to the intelligent agent closed-loop decision-making scenario of small fruit picking robots such as jujube, blueberry, goji berry, and grape, and is especially suitable for unstructured orchard environments with dense shading, large differences in ripeness, easily damaged fruit surfaces, and rapidly changing on-site environments.

[0025] As an optional embodiment of the present invention, the collaborative decision-making by multiple preset intelligent agents based on the harvesting data further includes: The harvesting data is encapsulated into structured messages, and the agents communicate with each other through these structured messages. The structured messages include the following fields: multimodal visual input, spatial and pose states, force feedback, environmental time series variables, orchard global map, crop mechanism states, harvesting task instructions, and compliance and safety rules.

[0026] Specifically, instead of communicating via free text in natural language, the agents encapsulate harvesting data into structured messages according to predefined field structures. These structured messages are standardized data structures in key-value pairs or JSON format, containing eight predefined fields: a multimodal visual input field for image or video data; a spatial and pose state field for 3D point cloud and robot pose data; a force feedback field for force and motion feedback data; an environmental time-series variable field for environmental time-series data; an orchard global map field for orchard global map data; a crop mechanism state field for crop state data output from an external mechanism model; a harvesting task instruction field for task objective data; and a compliance and safety rule field for compliance and safety rule data. After receiving a structured message, each agent reads the required data from the corresponding fields and, after completing its respective responsibilities, encapsulates the processing results into a new structured message in the same format and passes it to the next agent. The data structure of the structured message is as follows: (1) in, Structured messages transmitted between intelligent agents; For multimodal visual input fields, carrying image or video data; This is a spatial and pose state field, carrying 3D point cloud and robot pose data; This is the force feedback field, which carries force and motion feedback data; This is an environmental time-series variable field that carries environmental time-series data. This is a global map field for the orchard, containing global map data for the orchard. This is the crop mechanism state field, which carries the crop state data output by the external mechanism model; The task instruction field is used to carry the task target data; This is the compliance and security rules field, which contains compliance and security rule data.

[0027] In practical applications, this embodiment replaces free text communication with natural language by encapsulating the collected data into structured messages containing predefined fields for inter-agent communication. Structured messages force each agent to read and write information within defined field boundaries, avoiding illusions and comprehension biases that large models may experience when processing free text, thus improving the system's robustness and parsability. Simultaneously, the standardized message format unifies the input and output structures of each agent, facilitating system expansion and module replacement.

[0028] like Figure 2 As shown, in an optional embodiment of the present invention, the multimodal input data is input into the perceptual agent to obtain a candidate fruit set, including: The multimodal input data is input into the perceptual agent, which identifies jujube or blueberry fruits, leaves, branches, obstacles, lesions, picking containers, personnel, and restricted areas, and outputs the candidate set of fruits.

[0029] Specifically, the perceptual agent receives multimodal input data and processes it by invoking at least one of the following models: object detection, semantic segmentation, 3D reconstruction, maturity recognition, and anomaly detection. The perceptual agent identifies jujube or blueberry fruits, leaves, branches, obstacles, lesions, picking containers, personnel, and restricted areas, and outputs a candidate fruit set. Each fruit in the candidate set includes bounding box or segmentation mask, 3D coordinates, category, maturity confidence score, occlusion coefficient, and anomaly risk.

[0030] The data structure of the fruit candidate set output by the perceptual agent is as follows: (2) in, The set of candidate fruits output by the perceiving agent; For sensing intelligent agents; For multimodal visual input (image or video data); For spatial and pose states (point cloud and robot pose data); These are environmental time-series variables (temperature, light intensity, humidity, etc.). Let i be the bounding box or segmentation mask for the i-th fruit; Let be the three-dimensional coordinates of the i-th fruit; Let i be the category or variety of the i-th fruit; Let be the confidence level of the maturity or quality of the i-th fruit; Let be the occlusion coefficient of the i-th fruit; denoted as the risk of disease, damage, or misdiagnosis for the i-th fruit; n represents the total number of candidate fruits.

[0031] In practical application, this embodiment utilizes a perceptual agent that invokes multiple visual models to output a structured set of candidate fruits, providing a data foundation for subsequent decision-making by agronomic model agents and task planning agents. The candidate fruit set includes maturity confidence, occlusion coefficient, and anomaly risk, while also providing key indicators (uncertainty indicators and fruit damage indicators) for data value scoring. This allows the perceptual agent to not only serve current harvesting decisions but also the sample selection for the data flywheel.

[0032] like Figure 2 As shown, in an optional embodiment of the present invention, the step of inputting the candidate fruit set into an agronomic model agent and verifying the ripening mechanism to obtain ripe harvesting data includes: Based on the fruit candidate set, an external mechanism model is invoked to obtain the agronomic constraint state output by the external mechanism model; Specifically, the agronomic model agent, acting as a dedicated tool agent within the large model system, invokes an external mechanistic model based on the spatial coordinate information of each target in the fruit candidate set. This external mechanistic model is a growth-maturity digital twin system based on the LINTUL-3 heuristic mechanistic model. This system extracts corresponding variety parameters, meteorological data, soil data, and other input variables based on the spatial coordinates of each target, drives the model's operation, and outputs the agronomic constraint states of each candidate fruit. These agronomic constraint states include leaf area index, heat duration, light accumulation, dry matter, fruit load, maturity index, water stress, nitrogen stress, harvest window, quality risk, and overripe risk. This step corresponds to the process in formula (3) where the agronomic model agent receives the fruit candidate set and invokes the external mechanistic model.

[0033] The agronomic constraints are cross-validated with the fruit candidate set, and the ripening and harvesting data are obtained based on the cross-validation results.

[0034] Specifically, the agronomic model agent maps the agronomic constraint states output by the external mechanism model one-to-one with the visual perception results of each target in the fruit candidate set, performing cross-validation target by target. Visual perception results include the apparent features of the target fruit such as color, texture, and shape, while agronomic constraint states include physiological indicators such as the target fruit's maturity index and harvesting window. The agronomic model agent integrates the visual perception results and agronomic constraint states to determine the actual maturity state of each target. If the visual perception results are consistent with the agronomic constraint states, the maturity harvesting data for that target is obtained based on the combined judgment. The maturity harvesting data includes the predicted fruit maturity index, the optimal fruit harvesting window, the predicted fruit quality assessment, the risk of overripeness, and explanatory information regarding the target state and decision, i.e., the formula: (3) in, A set of mature harvest data output by the agronomic model agent; For agricultural model intelligent agents; The set of candidate fruits output by the perceiving agent; The crop mechanism state output by the external mechanism model; Let be the predicted value of the fruit ripening index for the i-th target; The optimal picking window for the i-th target; For the quality prediction and evaluation of the i-th objective; The risk of over-maturity for the i-th objective; Provide information on the state and decision interpretation for the i-th objective.

[0035] The agronomic model agent must not rely solely on the subjective judgment of maturity based on the language model; it must output a fusion result of the mechanistic model state and visual evidence. If the visual perception result is inconsistent with the agronomic constraint state, a conflict resolution process will be initiated.

[0036] In practical application, this embodiment utilizes external mechanistic models such as LINTUL-3 to obtain agronomic constraints, enabling maturity assessment to be based on both visual appearance characteristics and physiological growth indicators, thus avoiding the illusion caused by relying solely on visual judgment. The agronomic model agent acts as a tool invocation agent; the larger model must not subjectively judge maturity but can only make decisions by comprehensively considering the hard physiological data provided by the mechanistic tool and visual appearance. This cross-validation mechanism of visual and mechanistic methods ensures the objectivity and accuracy of maturity assessment, reducing the risk of misjudgment due to abnormal appearances such as diseases and sunscald.

[0037] like Figure 2 As shown, in an optional embodiment of the present invention, the mature harvesting data and the harvesting task data are input into the task planning agent to obtain a task-level harvesting sequence, including: Based on the mature harvesting data and the harvesting task data, a task-level harvesting sequence is generated by comprehensively considering the mature harvesting benefits, overripe risk, chassis movement costs, and robotic arm movement costs.

[0038] Specifically, the task planning agent generates a picking sequence based on the ripeness and picking data of candidate fruits, picking task data, the robot's current position, the reachability of the robotic arm, and the inter-row passage status. This sequence is not simply sorted by distance, but rather the optimal picking order is calculated by comprehensively considering ripeness benefits, overripeness risk, re-inspection value, robotic arm movement costs, and chassis movement costs through an optimization formula.

[0039] The task planning agent solves for the optimal harvesting sequence using an optimization formula: (4) in, For task planning level harvesting sequence, it represents the optimal harvesting order for all candidate targets; This means selecting the order from all possible harvesting orders that maximizes the total revenue minus the total cost; Let be the comprehensive profit function for harvesting candidate fruits, representing the profit value for harvesting the i-th target fruit, where Let be the maturity index for the i-th objective. Let be the quality assessment value of the i-th objective. Let be the over-maturity risk value of the i-th target; The cost of chassis movement when harvesting the i-th target; The cost of the robotic arm's movement when picking the i-th target; This formula represents the safety and failure risk cost of harvesting the i-th target. The task planning agent uses this formula to comprehensively consider the maturity benefits, over-maturity risks, chassis movement costs, and robotic arm movement costs to select the harvesting order with the greatest overall benefit, avoiding inefficient harvesting caused by sorting by distance or subjective sorting by a large model.

[0040] In practical application, this embodiment optimizes both the profit function and the cost function to ensure that the harvesting sequence maximizes both fruit value and operational efficiency, avoiding inefficient harvesting sequences and repetitive path movements caused by distance-based sorting or subjective sorting in large models. In formula (4) By taking into account the ripeness, quality, and risk of overripe fruit, the cost function considers the energy consumption of chassis movement and the movement time of the robotic arm, thus maximizing the harvesting efficiency.

[0041] like Figure 2 As shown, in an optional embodiment of the present invention, the step of cross-validating the agronomic constraint state with the fruit candidate set and obtaining the ripening and harvesting data based on the cross-validation results includes: The visual perception results of each target in the fruit candidate set are cross-validated with the corresponding agronomic constraint state; Specifically, the agronomic model agent compares the visual perception result of each target fruit in the fruit candidate set with the agronomic constraint state corresponding to that target output by the external mechanism model. The visual perception result includes apparent features such as color, texture, and maturity confidence level obtained by the perception agent from the target fruit. The agronomic constraint state includes physiological indicators such as maturity index and harvesting window output by the external mechanism model. The agronomic model agent then compares the visual perception result with the agronomic constraint state target by target to determine whether they are consistent in maturity assessment.

[0042] If the visual perception result of the target is inconsistent with the corresponding agronomic constraint state in terms of maturity determination, the target is marked as a conflict sample; Specifically, when a target is visually perceived as mature but its corresponding agronomic constraint is deemed immature, or vice versa, the agronomic model agent determines that the target's visual perception and agronomic constraint are inconsistent in their maturity assessment. The agronomic model agent then marks this target as a conflicting sample. Both the visual perception and agronomic constraint data of the conflicting sample are retained as the basis for subsequent cross-validation and model optimization.

[0043] The conflicting samples are marked as to be picked later or transferred to the review queue, and the conflicting samples are stored in the operation data.

[0044] Specifically, the task planning agent receives conflict samples marked by the agronomic model agent and marks them as either delayed harvesting or moved to a review queue based on the conflict type and task objective. When the conflict is due to visual signs of maturity but insufficient physiological accumulation as indicated by mechanistic state, it is marked as delayed harvesting, to be carried out only after the mechanistic state meets the standard. When the conflict is due to visual abnormalities (such as reddening of disease) but normal mechanistic state, it is marked as moved to a review queue for manual verification of the actual maturity status. The data governance agent stores the marked conflict samples in the operational data, serving as a high-value sample source for the data flywheel. This step corresponds to the data source for the model deviation index (i.e., maturity deviation), which is the degree of inconsistency between the visual perception result and the corresponding agronomic constraint state in maturity determination.

[0045] In practical application, this embodiment accurately identifies samples where visual perception results conflict with agronomic constraints by cross-validating the consistency between the results and the underlying mechanisms. Conflicting samples are marked for delayed harvesting or transferred to a review queue to avoid misharvesting due to visual illusions. Simultaneously, conflicting samples are stored in the operational data and fed into a data flywheel, providing high-value, challenging examples for subsequent model training and continuously improving the visual model's ability to recognize abnormal appearances.

[0046] like Figure 2 As shown, in an optional embodiment of the present invention, the step of inputting the task-level picking sequence into the picking control agent to generate a candidate sequence of actions and perform dynamic reachability verification to obtain the robot's execution trajectory includes: Based on the task-level picking sequence, the visual language action model is invoked to obtain the action candidate sequence output by the visual language action model; Specifically, the picking control agent transforms the task-level picking objective into actionable actions for the robot. The picking control agent invokes a visual-language-action model, which understands the natural language task instructions based on the task-level picking sequence, generates action intentions by combining visual input, and outputs a candidate sequence of actions. This candidate sequence includes the chassis micro-movement trajectory, the lifting and lowering adjustment trajectory, the robotic arm's pre-grasping pose, the end effector approach direction, the gripper opening and closing strategy, the force control threshold, the picking trajectory, and the placement path.

[0047] The candidate action sequences are subjected to dynamic constraint verification, collision detection verification, and kinematic reachability verification respectively. If the candidate action sequence passes all verifications, the candidate action sequence that passes all verifications is output as the execution trajectory of the robot.

[0048] Specifically, the picking control agent sends the action candidate sequence output by the visual language action model to the collision detection verification module, kinematic reachability verification module, and dynamic constraint verification module for parallel verification. Each of the three verification modules makes an independent judgment. If all three verification modules determine that the action candidate sequence passes, it is output as the robot's execution trajectory to the underlying motion controller for execution, i.e., through the following formula: (5) in, Perform actions for the robot; For visual language action models; For visual input; For task instructions; For kinematic constraints; For collision constraints; Force control constraint; For safety constraints. If any verification module determines that a step has failed, the following steps are performed to regenerate the candidate sequence of alternative actions.

[0049] In practical application, this embodiment generates candidate action sequences by invoking a visual language action model, enabling the robot to understand natural language task instructions and adapt to the unstructured orchard environment. Through independent verification of the candidate action sequences at three levels—dynamics, collision, and kinematics—the final executed trajectory simultaneously meets the requirements of reachability, safety, and force control accuracy. The large model is only responsible for generating candidate solutions; the final execution must undergo rigorous verification by a traditional planner. This approach leverages the generalization and reasoning capabilities of the large model while ensuring execution safety through the constraints of the traditional planner.

[0050] As an optional embodiment of the present invention, the step of performing dynamic constraint verification, collision detection verification, and kinematic reachability verification on the candidate action sequences includes: The collision interference verification between the robot's first robotic arm and the picking environment is performed based on the three-dimensional point cloud in the multimodal input data. If the sweeping volume of the first robotic arm interferes with the obstacle and the intrusion depth of the interference area exceeds the preset collision threshold, the collision detection verification is deemed to have failed. Specifically, the collision detection verification module constructs a picking environment map based on the 3D point cloud generated by the perceptual agent, and performs collision interference calculations between the geometric model of the robot arm and the picking environment map. It calculates the minimum distance and intrusion depth between the robot arm's sweeping volume and obstacles corresponding to the candidate action sequence. If there is no interference between the sweeping volume and the obstacle, or if there is interference but the intrusion depth does not exceed the preset collision threshold, the collision detection verification is deemed successful. Conversely, if the sweeping volume interferes with the obstacle and the intrusion depth exceeds the preset collision threshold, the collision detection verification is deemed unsuccessful. This step corresponds to the collision constraint verification process in formula (5).

[0051] The inverse kinematics solution is performed on the candidate action sequence. If the activity angle of each joint of the robot exceeds the mechanical limit or there is no solution for inverse kinematics, the kinematic reachability verification is deemed to have failed. Specifically, the kinematic reachability verification module performs inverse kinematics calculation based on the target pose coordinates output by the visual language action model from the picking control agent. It calculates the joint angle values ​​of each joint in the pose of each keyframe in the action candidate sequence and compares these joint angle values ​​with their corresponding mechanical limits. If the inverse kinematics has a solution and all joint angle values ​​are within the corresponding mechanical limit range, the kinematic reachability verification is passed. If the inverse kinematics has no solution or any joint angle value exceeds the corresponding mechanical limit range, the kinematic reachability verification is failed. This step corresponds to the kinematic constraint verification process in formula (5).

[0052] Determine whether the expected torque of each joint in the candidate action sequence exceeds the maximum motor output torque, and whether the expected closing force of the robot's end gripper is greater than the upper limit of the damage force allowed for the fruit variety. If either one exceeds, it is determined that the dynamic constraint verification has not been passed. Specifically, the dynamic constraint verification module uses the robot dynamics model to evaluate the expected velocity and acceleration at each keyframe corresponding to the action candidate sequence. It calculates the expected torque value required by each joint to track the action candidate sequence and compares this value with the maximum output torque of the corresponding motor. Simultaneously, it calculates the expected closing force value required by the end effector during the gripping phase corresponding to the action candidate sequence and compares this value with the upper limit of the allowable breakage force for the current fruit variety. If the expected torque of all joints does not exceed the maximum output torque of the corresponding motor and the expected closing force of the end effector does not exceed the upper limit of the allowable breakage force for the fruit variety, the dynamic constraint verification is deemed successful. If the expected torque of any joint exceeds the maximum output torque of the corresponding motor or the expected closing force of the end effector exceeds the upper limit of the allowable breakage force for the fruit variety, the dynamic constraint verification is deemed unsuccessful. This step corresponds to the force control constraint verification process in formula (5).

[0053] If the action candidate sequence fails any of the verifications of the dynamic constraint, the collision detection, and the kinematic reachability, the action candidate sequence is regenerated by the picking control agent.

[0054] Specifically, if any of the collision detection verification, kinematic reachability verification, or dynamic constraint verification fails, the picking control agent rejects the current action candidate sequence. The picking control agent records the verification type and reason for the failure of the action candidate sequence, and feeds the failure log back to the visual language action model. The visual language action model then regenerates an alternative action candidate sequence based on the reason for the failure.

[0055] Below is an example of a jujube picking scene.

[0056] Winter jujube scene 1 (standard picking): The task is to "pick winter jujubes at 80% maturity". The perceptual agent integrates visual information (red circle area accounts for more than 80%) and mechanistic data (accumulated temperature meets the standard) provided by the agronomic model agent. The task planning agent generates a bottom-up picking sequence through formula (4). The picking control agent calls the visual language action model to generate a smooth detachment trajectory of "approach-clamp-rotate-pull down". After collision detection verification, kinematic reachability verification and dynamic constraint verification, it is executed.

[0057] Scene 2 of winter jujubes (handling the conflict between visual and mechanistic elements): A certain winter jujube had large areas of red spots on its skin, which the visual model judged as "ripe." However, the agronomic model agent retrieved mechanistic data and found that the area had recently experienced extremely poor sunlight, indicating that the reddening was actually caused by disease. The visual perception result and the agronomic constraints did not match in their maturity determination. The agronomic model agent marked this target as a conflict sample, and the system automatically refused to pick it, recording it as a "high-risk sample for accidental picking" and storing it in the operation data, which was then transmitted back to the data flywheel.

[0058] Scene 3 of winter jujube (peeling dense fruit clusters): The target jujubes are obscured by a large number of leaves and unripe green fruits. The picking control agent calls the visual language action model to generate a candidate sequence of "slightly nudge" actions. After collision detection verifies that the collision margin is sufficient, the deep grasping is performed to prevent accidental damage to the green fruits.

[0059] Below is an example from a blueberry picking scenario.

[0060] Blueberry Scene 1 (Harvesting Requirements for Preservatives): The user issues a "fresh, high-quality picking" task. The picking task data is interpreted as having increased quality weight and heavier penalties for fruit damage. In the dynamic constraint verification, the picking control agent actively limits the expected closing force of the end gripper to less than the upper limit of the damage force allowed by the blueberry variety. The visual language action model generates an enveloping approach trajectory rather than a fingertip pinching approach.

[0061] Blueberry Scenario 2 (Internal Fruit Cluster Delay Strategy): The perception agent identifies two green berries on the outside and one ripe purple berry inside a blueberry bunch. The task planning agent assesses that forcibly picking the inner berry would cause the outer green berry to fall, resulting in excessive risk and cost of fruit damage. Therefore, it outputs a "delay picking" strategy, recording the coordinates and waiting several days for the outer berry to ripen before picking.

[0062] Blueberry Scenario 3 (Harvesting under Environmental Stress): The agronomic model agent, upon receiving a weather forecast, detected an impending heavy rainstorm, triggering a high humidity risk warning. The system automatically adjusted the weights of each indicator in the multi-objective reward function, increasing the weight coefficient of the successful harvest item and appropriately relaxing the weight coefficient of the fruit damage penalty item. This enabled the robot to enter a high-frequency, continuous, and rapid harvesting mode to reduce economic losses due to rain-induced fruit cracking.

[0063] In practical applications, this embodiment employs three independent layers of verification to comprehensively constrain the candidate action sequences output by the visual language action model. Collision detection verification ensures operational safety, kinematic reachability verification ensures action feasibility, and dynamic constraint verification ensures that forces and moments do not exceed physical limits. The large model "suggests," and the traditional planner "approves." If any layer of verification fails, the model is regenerated, fundamentally eliminating the safety hazards of a single large model directly outputting unreachable or collision-risk actions.

[0064] As an optional embodiment of the present invention, the step of using a safety management intelligent agent to monitor the safety status in real time during the robot's harvesting process and triggering a safety degradation action when an anomaly is detected includes: The safety status monitored in real time by the safety management intelligent agent includes collision probability, end force, distance between the robot and the personnel, and probability of fruit damage. Specifically, the safety management intelligent agent performs safety status monitoring before, during, and after harvesting. The collision probability is calculated in real-time by the safety management intelligent agent based on the 3D point cloud generated by the perception intelligent agent, assessing the collision risk value between the robotic arm and the harvesting environment. The end effector force is collected in real-time by force sensors deployed at the end of the robotic arm. The distance between the robot and personnel is detected by LiDAR or vision sensors, determining the shortest physical distance between the robot and personnel at the work site. The probability of fruit damage is assessed in real-time by the safety management intelligent agent based on a comparison of fruit images before and after harvesting.

[0065] The safety degradation action is triggered by a safety determination formula, which is as follows: when the collision probability is less than a first preset threshold, the end force is less than a second preset threshold, the interval distance is greater than a third preset threshold, and the fruit damage probability is less than a fourth preset threshold, the state is determined to be safe. When any one of the collision probability, end force, personnel distance, and fruit damage probability does not meet the corresponding condition, the safety degradation action is triggered.

[0066] Specifically, the safety determination formula is as follows: (6) in, For safety determination variables, 0 indicates that the operation is allowed, and 0 indicates that a security degradation is triggered; This represents the collision probability. Force is applied at the end; The distance between the robot and the person; This represents the probability of fruit breakage. The first preset threshold; The second preset threshold; The third preset threshold; 1 is the fourth preset threshold; 1[·] is an indicator function, which takes the value of 1 only when all conditions within the parentheses are met simultaneously, and takes the value of 0 when any condition is not met.

[0067] In practical application, this embodiment implements a safety monitoring mechanism through a safety judgment formula. The four dimensions—collision probability, end force, personnel distance, and fruit damage probability—constitute a complete safety monitoring indicator system. An anomaly in any dimension triggers a safety downgrade. The four preset thresholds can be configured independently according to different crop varieties and operational scenarios. Real-time data from each dimension used for safety judgment are recorded, providing a data foundation for subsequent safety strategy optimization.

[0068] As an optional embodiment of the present invention, the triggering of the security degradation action includes: Perform at least one of the following degradation actions: refuse to perform the current action, reduce the movement speed of the second robotic arm, withdraw the second robotic arm, skip the current picking scenario, or request manual review; Specifically, the safety management agent triggers a safety downgrade when any condition in the safety determination formula is not met. Refusing to execute the current action terminates the current action when any safety condition is not met. Reducing the robotic arm's movement speed applies when the collision probability is close to the first preset threshold or the distance to the person is close to the third preset threshold. Retracting the robotic arm applies when the end-effector force exceeds the second preset threshold or when a collision interference between the robotic arm and an obstacle is detected (the second robotic arm and the first robotic arm can be the same arm or different arms). Skipping the current picking scenario applies when the current target fruit is determined to be unreachable or has a high risk of damage. Requesting manual review applies when the visual perception result is inconsistent with the agronomic constraint state and both confidence levels are higher than the corresponding thresholds, or when repeated attempts to reach the same target have failed a preset number of times.

[0069] The safety degradation action is stored in the operation data.

[0070] Specifically, the security governance agent stores the event records of the security degradation actions it performs in the operational data. The event records include the degradation action type, triggering conditions, triggering time, the values ​​of each security monitoring parameter at the triggering time, and corresponding action candidate information. The event records stored in the operational data enter the data flywheel, serving as a candidate source of high-value samples for model training and security policy optimization.

[0071] The following exception handling scenarios can trigger security degradation actions.

[0072] Scenario 1 for handling exceptions (emergency stop due to personnel intrusion): The safety management intelligent agent detects that the distance of a moving person is less than the third preset threshold using lidar. The interval distance condition in the safety judgment formula is not met, triggering a safety downgrade. The robotic arm stops abruptly and locks in place until the person leaves the warning area.

[0073] Scenario 2 (Dynamics Unreachable): The picking control agent calls the action candidate sequence output by the visual language action model. After kinematic reachability verification, the inverse kinematics solution finds that there is a singularity and exceeds the joint limit. It is determined that the kinematic reachability verification fails. The picking control agent rejects the action candidate sequence and regenerates an alternative action candidate sequence.

[0074] Scenario 3 for handling abnormal situations (force control abnormal escape): The end effector force sensor detected a surge in resistance torque. During the dynamic constraint verification, the force on the end effector exceeded the second preset threshold, triggering a safety degradation. The controller instantly released the gripper and retreated along the original trajectory to prevent the branch from breaking.

[0075] In practical application, this embodiment utilizes a coordinated approach of various graded degradation actions, such as refusal to execute, speed reduction, robotic arm retraction, target skipping, and requesting manual review, to enable the system to adopt differentiated safety response strategies based on different anomaly types and severity. Refusal to execute and robotic arm retraction provide rapid safety responses, speed reduction provides progressive protection, skipping the current target ensures operational continuity, and requesting manual review handles complex anomalies that the system cannot autonomously determine. Compared to a single hard stop, this invention can maximize operational efficiency while ensuring safety. Simultaneously, the event records of safety degradation actions are stored in the operational data, providing high-value anomaly samples for the data flywheel. This allows the system to continuously optimize its safety judgment strategy through data accumulation and model training, reducing the recurrence probability of similar anomalies.

[0076] As an optional embodiment of the present invention, the step of collecting the robot's operational data and filtering sample data, wherein the sample data is used to update the decision parameters of the harvesting control agent, includes: Sample data is selected from the operational data according to the value scoring formula, which is determined based on at least one of the following indicators: the maturity confidence and occlusion coefficient of each target in the fruit candidate set, the records of successful or failed harvesting, the fruit damage detection results, and the maturity deviation degree corresponding to the conflicting samples. Specifically, the data governance agent is responsible for classifying, cleaning, desensitizing, labeling, retrieving, and versioning the field operation data. The data governance agent obtains conflict samples from the operation data. These conflict samples originate from samples marked as inconsistent between visual perception results and agronomic constraint states during cross-validation of the agronomic model agent. The maturity confidence score is determined by the confidence score field in the fruit candidate set output by the perception agent, and the occlusion coefficient is determined by the occlusion coefficient field in the fruit candidate set output by the perception agent. Successful or failed harvesting records are determined by the execution results output by the harvesting control agent after each harvesting operation. Fruit damage detection results are determined by the safety governance agent after harvesting, checking for fruit damage. The maturity deviation is the degree of inconsistency between the visual perception results and the corresponding agronomic constraint states in maturity determination, i.e., the model deviation index in the value scoring formula below.

[0077] The sample data whose value scores exceed the fifth preset threshold are labeled, and the labeled sample data is stored in the training warehouse to train the visual language action model. Specifically, the data governance agent calculates the value score for each conflict sample based on the value scoring formula. The value scoring formula is: (7) in, For the sample Value rating This is an uncertainty indicator, determined based on maturity confidence level and shading coefficient; The failure indicator is determined based on successful or failed harvesting records. This is a fruit damage indicator, determined based on the fruit damage detection results; This is a model deviation index, namely maturity deviation. It is a novelty index, representing the rarity of a sample in the feature space; This is a privacy risk indicator, representing whether the sample contains personal privacy information that requires desensitization as a penalty. - The corresponding weight coefficients are used; the data governance agent identifies conflict samples with value scores exceeding the fifth preset threshold as high-value samples and actively labels them. Labeling methods include automatic pseudo-labeling, manual sampling, and expert verification. Labeled samples are stored in the training repository for subsequent training of the visual language action model invoked by the harvesting control agent.

[0078] The trained visual language action model is evaluated offline, and the new model that passes the evaluation is deployed to the edge. If the performance metrics of the new model are lower than those of the current visual language action model, a rollback is triggered.

[0079] Specifically, the data governance agent trains the visual language action model using labeled samples from the training repository. After training, the trained visual language action model is evaluated offline in a non-task environment using historical data or test datasets to verify its performance metrics. The performance metrics are determined by a multi-objective reward function, which is: (8) in, The overall reward value is used to measure the performance of the new model. For successful harvesting, this is a measure of harvesting success or operational efficiency; This is a quality item, representing a measure of the integrity of the harvested fruit. This is a penalty for fruit damage, representing a measure of fruit damage caused by harvesting; This is a penalty for missed sampling, representing a measure of missed and erroneous sampling; This is an energy consumption penalty term, representing a measure of the energy consumption of system operation and actions; For safety penalties, this indicates a measure of violating safety regulations or causing danger; For data value items, it represents a measure of the positive incentive for acquiring high-value data; to These are the corresponding hyperparameter weights.

[0080] The new model, which has passed the evaluation, will be deployed to edge devices in a canary release manner for small-scale practical testing. If the new model's performance metrics in actual operations are lower than those of the current visual language action model, an automatic rollback will be triggered, restoring the current model version. The rollback condition is that any one of the following: fruit loss rate, false picking rate, or number of safety events of the new model is higher than that of the current model, or the picking success rate is lower than that of the current model.

[0081] Regarding the relationship between the cloud and the edge in this invention, the cloud deploys large-scale model inference services, model training services, knowledge base services, and data flywheel services, while the edge deploys lightweight perception models, local planners, force control controllers, security daemons, and model version managers. The cloud is used for large-scale model training, knowledge base updates, cross-plot experience summarization, and strategy version management, while the edge is used for real-time perception, local planning, security monitoring, low-latency control, and anomaly degradation. When the network is stable, the edge uploads action slices and fruit damage labels to the cloud in real time to trigger fine-tuning of the large-scale model. After evaluation, incremental weights are issued, supporting canary releases and A / B testing (two model versions, one old and one new, are deployed on different robots and run in parallel to compare actual operation indicators, such as fruit damage rate and harvesting success rate, to verify whether the new version is superior to the current version. After verification, the deployment scope is expanded). When the network is unavailable, the edge continues to perform harvesting based on local models and rules, encrypting and caching high-value abnormal samples and sending them back after the network is restored, thus balancing intelligent decision-making capabilities and operational reliability.

[0082] The following are several examples of edge-cloud collaboration scenarios.

[0083] Edge-cloud collaboration scenario 1 (network stability online optimization): Action slices and damage labels are uploaded to the cloud in real time from the edge device. The cloud triggers fine-tuning of the large model. After the evaluation is passed, the incremental weights are sent to the edge device using the 5G network to achieve daily model updates.

[0084] Edge-cloud collaboration scenario 2 (offline and network disconnection operation): When there is no network signal deep in the orchard, the edge device continues to perform harvesting based on the locally cached lightweight perception model and local rules, encrypts high-value abnormal samples and writes them into the local queue, and then transmits them back in batches when the chassis moves to an area with network access.

[0085] Edge-cloud collaboration scenario 3 (canary release and A / B testing): A strategy version V2.0 for the new season of winter jujubes was trained in the cloud. V2.0 was only issued to the two robots in plot A, and the fruit loss rate was compared with other robots running V1.0 in the background. If V2.0 triggered the safety threshold, the system automatically issued an instruction to roll back to V1.0.

[0086] In practical application, this embodiment automatically filters high-value samples from a large amount of operational data using a value scoring formula, avoiding the resource waste caused by full annotation of all data. Maturity confidence, occlusion coefficient, successful or failed harvesting records, fruit damage detection results, and maturity deviation are used as screening dimensions to ensure that the selected samples can specifically improve the visual-language-action model's processing capabilities in challenging scenarios such as occlusion, low confidence, abnormal fruit damage, and visual-mechanical conflicts. Annotated samples are stored in a training repository to drive model training. After offline evaluation and canary release, they are deployed to the edge, forming a complete data flywheel closed loop of "collection-screening-annotation-training-evaluation-deployment-rollback," enabling the model to continuously evolve in ongoing operations and achieve the technical effect of "becoming more accurate with each collection." Simultaneously, the automatic rollback mechanism ensures rapid recovery when a new model is inferior to the current model in operational metrics, guaranteeing operational safety and stability and reducing production risks associated with model updates.

[0087] It should be noted that the following explains the construction method and relationships of the multi-agent system in this application: In this invention, multiple specialized intelligent agents are not composed of a single large model. The core innovation of this invention lies precisely in opposing the use of a single large model to directly control the robot, because a single large model is a "black box," which is prone to causing fatal motion illusions under complex orchard lighting conditions, ignoring safety boundaries, and failing to explain agronomic reasons. The large model only serves as an internal component of the harvesting control intelligent agent, used to understand task instructions and generate action candidates. Its output must be verified step by step by other intelligent agents before it can be executed.

[0088] The specific construction methods of each intelligent agent are as follows: The perception intelligent agent is encapsulated and constructed from traditional visual models (such as object detection, semantic segmentation, 3D reconstruction, etc.), and is responsible for outputting a structured set of fruit candidates; the agronomic model intelligent agent is constructed from an externally available crop mechanism model (such as the LINTUL-3 growth-maturity digital mechanism model) as a tool for mandatory invocation, providing hard physiological data constraints; the task planning intelligent agent is constructed from a multi-objective optimization function, and is responsible for solving the optimal harvesting order by comprehensively considering maturity benefits, movement costs, and risks; the harvesting control intelligent agent internally combines a visual language action model with a traditional edge motion planner, with the large model responsible for generating action intentions and the planner responsible for performing three-layer physical verification of dynamics, collision, etc.; the safety governance intelligent agent is constructed from a hard safety judgment formula, including threshold judgments such as collision probability, end force, and safe distance, and has veto power; the data governance intelligent agent is constructed from a multi-dimensional data value scoring formula, used to select high-value samples to drive model evolution.

[0089] The agents establish collaborative and constraint relationships through a "structured message pipeline" and a "cross-validation interpretation chain." To prevent comprehension biases arising from the large model, the independently constructed agents do not communicate using free text in natural language. Instead, they communicate through standardized data structures (such as key-value pairs or structured messages containing specific fields like multimodal visual input, spatial pose, and mechanistic state). Information is transmitted through a standardized data pipeline. Each agent reads upstream information only within specified field boundaries, processes it, and then encapsulates the results in the same format before transmitting them to downstream agents. In terms of control structure, agents maintain a "checks and balances" relationship (the "absolute control" of the large model is progressively stripped away by other agents): the perception agent provides visual results but must undergo cross-validation of the mechanistic state by the agronomic agent to prevent misjudging disease spots as ripe; the action candidates generated by the large model must undergo kinematic, collision, and dynamic verification by the edge motion planner, as well as a veto by the safety agent, to prevent actions from being unreachable or causing injury to people or trees. This mechanism provides the system with a clear "task interpretation chain" (e.g., visual perception indicates ripeness → mechanistic perception indicates immaturity → planner considers unreachable), allowing for precise identification of fruit damage and misharvesting causes as perceptual errors, decision-making mistakes, or execution deviations, thus achieving anti-illusion, strong controllability, and accurate accountability.

[0090] like Figure 3 As shown, the present invention also provides a robot, including a memory 310 and a processor 320; The memory 310 is used to store computer programs; The processor 320 is configured to implement the robot's intelligent agent closed-loop decision-making method as described in the first aspect when executing the computer program.

[0091] The specific implementation method of this embodiment can be referred to the corresponding implementation method described above, and will not be described again here.

[0092] This invention provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor 320, it implements the robot's intelligent agent closed-loop decision-making method as described above.

[0093] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments described above. Any references to storage, databases, or other media used in the embodiments provided by this invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0094] The above description is merely a specific embodiment of the present invention, enabling those skilled in the art to understand or implement the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the present invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features of the invention herein.

[0095] While the present invention has been disclosed above, its scope of protection is not limited thereto. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the present invention, and all such changes and modifications will fall within the scope of protection of the present invention.

Claims

1. A closed-loop decision-making method for a robot's intelligent agent, characterized in that, include: Acquire harvesting data, which includes multimodal input data and harvesting task data; Multiple pre-defined agents make collaborative decisions based on the harvesting data, including: inputting the multimodal input data into a perception agent to obtain a fruit candidate set; inputting the fruit candidate set into an agronomic model agent and verifying the ripening mechanism to obtain mature harvesting data; inputting the mature harvesting data and the harvesting task data into a task planning agent to obtain a task-level harvesting sequence; inputting the task-level harvesting sequence into a harvesting control agent to generate an action candidate sequence and verify its power reachability to obtain the robot's execution trajectory; and having a safety management agent monitor the safety status in real time during the robot's harvesting process and trigger a safety degradation action when an anomaly is detected. The robot's operational data is collected and sample data is filtered. The sample data is used to update the decision parameters of the harvesting control agent.

2. The robot agent closed-loop decision-making method according to claim 1, characterized in that, The collaborative decision-making process by multiple pre-set intelligent agents based on the harvesting data also includes: The harvesting data is encapsulated into structured messages, and the agents communicate with each other through these structured messages. The structured messages include the following fields: multimodal visual input, spatial and pose states, force feedback, environmental time series variables, orchard global map, crop mechanism states, harvesting task instructions, and compliance and safety rules.

3. The robot agent closed-loop decision-making method according to claim 1, characterized in that, The step of inputting the candidate fruit set into the agronomic model agent and verifying the ripening mechanism to obtain ripe harvesting data includes: Based on the fruit candidate set, an external mechanism model is invoked to obtain the agronomic constraint state output by the external mechanism model; The agronomic constraints are cross-validated with the fruit candidate set, and the ripening and harvesting data are obtained based on the cross-validation results.

4. The robot agent closed-loop decision-making method according to claim 3, characterized in that, The step of cross-validating the agronomic constraints with the fruit candidate set, and obtaining the ripe harvest data based on the cross-validation results, includes: The visual perception results of each target in the fruit candidate set are cross-validated with the corresponding agronomic constraint state; If the visual perception result of the target is inconsistent with the corresponding agronomic constraint state in terms of maturity determination, the target is marked as a conflict sample; The conflicting samples are marked as to be picked later or transferred to the review queue, and the conflicting samples are stored in the operation data.

5. The robot agent closed-loop decision-making method according to claim 1, characterized in that, The step of inputting the task-level picking sequence into the picking control agent to generate action candidate sequences and perform dynamic reachability verification to obtain the robot's execution trajectory includes: Based on the task-level picking sequence, the visual language action model is invoked to obtain the action candidate sequence output by the visual language action model; The candidate action sequences are subjected to dynamic constraint verification, collision detection verification, and kinematic reachability verification respectively. If the candidate action sequence passes all verifications, the candidate action sequence that passes all verifications is output as the execution trajectory of the robot.

6. The robot agent closed-loop decision-making method according to claim 5, characterized in that, The process of performing dynamic constraint verification, collision detection verification, and kinematic reachability verification on the candidate action sequences includes: The collision interference verification between the robot's first robotic arm and the picking environment is performed based on the three-dimensional point cloud in the multimodal input data. If the sweeping volume of the first robotic arm interferes with the obstacle and the intrusion depth of the interference area exceeds the preset collision threshold, the collision detection verification is deemed to have failed. The inverse kinematics solution is performed on the candidate action sequence. If the activity angle of each joint of the robot exceeds the mechanical limit or there is no solution for inverse kinematics, the kinematic reachability verification is deemed to have failed. Determine whether the expected torque of each joint in the candidate action sequence exceeds the maximum motor output torque, and whether the expected closing force of the robot's end gripper is greater than the upper limit of the damage force allowed for the fruit variety. If either one exceeds, it is determined that the dynamic constraint verification has not been passed. If the action candidate sequence fails any of the verifications of the dynamic constraint, the collision detection, and the kinematic reachability, the action candidate sequence is regenerated by the picking control agent.

7. The robot agent closed-loop decision-making method according to claim 1, characterized in that, The provision that the safety management intelligent agent monitors the safety status in real time during the robot's harvesting process and triggers a safety degradation action when an anomaly is detected includes: The safety status monitored in real time by the safety management intelligent agent includes collision probability, end force, distance between the robot and the personnel, and probability of fruit damage. The safety degradation action is triggered by a safety determination formula, which is as follows: when the collision probability is less than a first preset threshold, the end force is less than a second preset threshold, the interval distance is greater than a third preset threshold, and the fruit damage probability is less than a fourth preset threshold, the state is determined to be safe. When any one of the collision probability, end force, personnel distance, and fruit damage probability does not meet the corresponding condition, the safety degradation action is triggered.

8. The robot agent closed-loop decision-making method according to claim 7, characterized in that, The triggering of the security degradation action includes: Perform at least one of the following degradation actions: refuse to perform the current action, reduce the movement speed of the second robotic arm, withdraw the second robotic arm, skip the current picking scenario, or request manual review; The safety degradation action is stored in the operation data.

9. The robot agent closed-loop decision-making method according to claim 5, characterized in that, The process of collecting the robot's operational data and filtering sample data, wherein the sample data is used to update the decision parameters of the harvesting control agent, includes: Sample data is selected from the operational data according to the value scoring formula, which is determined based on at least one of the following indicators: the maturity confidence and occlusion coefficient of each target in the fruit candidate set, the records of successful or failed harvesting, and the results of fruit damage detection. The sample data whose value scores exceed the fifth preset threshold are labeled, and the labeled sample data is stored in the training warehouse to train the visual language action model. The trained visual language action model is evaluated offline, and the new model that passes the evaluation is deployed to the edge. If the performance metrics of the new model are lower than those of the current visual language action model, a rollback is triggered.

10. A robot, characterized in that, It includes a processor and a memory, wherein the memory stores a program that, when executed by the processor, implements the intelligent agent closed-loop decision-making method for the robot according to any one of claims 1-9.

Images