Robot operation instruction control method and system

Abstract

The application relates to a robot operation instruction control method and system, wherein the method comprises the following steps: obtaining a natural language operation instruction input by a user to a robot; calling a large language model under a preset output format and field value constraint, analyzing a task representation, and performing mode checking; writing environment state data into an asynchronous cache; based on a corresponding operation target and operation constraint and a timestamp of the task representation, performing matching calculation in the asynchronous cache to generate an executable task tuple; completing the operation target under the condition of meeting the operation constraint; monitoring through a double-channel holding mechanism, and processing through a fault-tolerant strategy when an exception occurs; and publishing the state and result events in the process from the natural language operation instruction to the execution completion in a unified format, realizing a stable closed loop of the whole process of the robot from a natural language-analyzing-perceiving-executing, controlling the output of the large language model, and improving the task start success rate and the task completion rate.

Description

Technical Field

[0001] This invention relates to the field of remote sensing image analysis technology, and in particular to a robot operation command control method and system. Background Technology

[0002] In recent years, with the rapid development of Large Language Model (LLM) and robotics, using natural language to guide robots to complete complex tasks has become an important research direction in human-computer interaction.

[0003] However, a series of key technical challenges remain in translating natural language instructions into robot-executable tasks, hindering the reliability, safety, and practical application of such systems. First, traditional mobile robots often rely on fixed instruction sets, graphical processes, or manual configuration of task parameters, making it difficult to cover the frequent omissions, referentials, aliases, and complex constraints in natural language. Directly parsing free text with simple rules or scripts can easily lead to missing fields, semantic ambiguity, and unreproducible execution results. Second, LLM (Limited Language Modeling) is prone to "illusions" during open generation, easily outputting unconstrained text, spelling variations, or out-of-bounds values, introducing uncontrollable risks to the robot system. Furthermore, if the parsing results are not synchronized with the perception timeline and object list, stuttering often occurs where there is a task but no object or an object but no task. Third, the perception module typically publishes the attribute and pose information of objects in the environment at fixed intervals, while task parsing is triggered during user interaction; these two are inherently asynchronous on the timeline, affecting the overall success rate of the operation. Finally, the lack of a unified lifecycle event output makes fault location, comparative experiments, and performance evaluation difficult. Existing solutions often lack a unified lifecycle event model and standardized log output, making fault location, comparative experiments, and performance evaluation difficult.

[0004] In summary, there is an urgent need in this field for a robot operation command control scheme that can controllably constrain LLM outputs, improve task initiation success rate and task completion rate, and enhance the observability and evaluability of robot control. Summary of the Invention

[0005] In view of this, it is necessary to provide a robot operation instruction control method and system to solve the technical problems that existing robot operation instruction control methods cannot controllably constrain LLM output, which reduces the robot's task start success rate, task completion rate, and the low safety and reliability of robot control.

[0006] To address the aforementioned problems, in a first aspect, the present invention provides a robot operation instruction control method, comprising: The robot receives natural language task instructions input by the user, which include the task objectives and constraints to be completed. Under the preset output format and field value constraints, the large language model is invoked to parse the natural language job instructions into a structured task representation; The task representation is subjected to pattern validation, and if the validation fails, the large language model is invoked under the preset output format and field constraints to obtain a new task representation. Receive environmental status data, which includes the type, attributes, and pose information of the object being operated on, and write the environmental status data into an asynchronous cache in chronological order; Based on the task representation, the corresponding job target, job constraints, and timestamp, a matching calculation is performed in the asynchronous cache, and an executable task tuple containing at least the object identifier, object pose, and target region of the operated object is generated according to the matching result. The robot is controlled to perform navigation and grasping operations based on the object pose in the executable task tuple, and to perform placement operations based on the target area, thereby completing the task objective under the condition of satisfying the task constraints. The start, execution and completion status of each stage of the navigation and grasping operation and the placement operation are monitored through a dual-channel handshake mechanism, and a fault tolerance strategy is invoked to handle abnormalities. The status and result events of the natural language job instruction from parsing to execution completion are published in a unified format.

[0007] In one possible implementation, the structured task is represented as a fielded data representation, which is in JSON or Protobuf format. The fields in the fieldized data representation include at least one of the following: target category, color, quantity, and target region.

[0008] In one possible implementation, the step of calling the large language model under preset output format and field value constraints includes: The output format of the large language model is limited to a key-value pair structure by using templated prompt words; The allowed field names in the key-value pair structure are limited by a field whitelist, and the allowed value range of each field name is declared in the templated prompt.

[0009] In one possible implementation, writing the environment state data into the asynchronous cache in chronological order includes: The asynchronous cache is divided into multiple time windows; The received environmental status data is stored in the corresponding time window according to the timestamp. The time windows are managed using the most recently valid and expired elimination rules.

[0010] In one possible implementation, the matching calculation based on the job target, job constraints, and timestamp corresponding to the task representation in the asynchronous cache includes: Based on the timestamp, select the environmental state data corresponding to the target time window from the asynchronous cache; The structured task representation is matched with the selected environmental state data based on object attribute similarity, spatial proximity, and temporal proximity. The object attribute similarity, spatial proximity, and temporal proximity are weighted and summed according to preset weights to obtain the matching result.

[0011] In one possible implementation, the dual-channel handshake mechanism includes a control channel and a status channel; the monitoring of the start, execution, and completion status of each stage of the navigation and grasping operation and the placement operation through the dual-channel handshake mechanism includes: Control commands and synchronization events for each stage are issued through the control channel; The state channel receives state enumerations representing each stage reported by the robot. In this process, a task identifier and an object identifier are written into the messages transmitted in the control channel and the status channel, so as to associate the messages transmitted in different channels according to the identifier.

[0012] In one possible implementation, the fault-tolerance strategy includes at least one of the following: Maximum retry limit, timeout rollback to the previous stable state, request to re-aware and update the executable task tuple, skip the current object and continue with subsequent tasks.

[0013] In one possible implementation, controlling the robot to perform navigation and grasping operations based on the object pose in the executable task tuple, and controlling the robot to perform placement operations based on the target area, to complete the task objective under the condition of satisfying the task constraints, includes: The navigation, grasping, and placement operations are programmed using a preset execution state machine. During the grasping and placement phases of the execution state machine, the robot is controlled to sequentially perform pre-pose adjustment, target pose execution, and pose confirmation, respectively. During the grasping or placement phase, when a pose error exceeding a preset threshold is detected, a fine-tuning action is triggered and executed.

[0014] In one possible implementation, publishing state and result events in a uniform format includes: Standardized message mapping is performed on the natural language job instruction parsing success event, the executable task tuple generation event, the state change events of each stage of the navigation, grasping and placement operation, and the job goal achievement event; Output the mapped events to the external interface.

[0015] Secondly, the present invention also provides a robot operation instruction control system, comprising: The acquisition unit is used to acquire natural language task instructions input by the user to the robot, the natural language task instructions including the task objectives and task constraints to be completed; The parsing unit is used to call the large language model under the preset output format and field value constraints to parse the natural language job instructions into a structured task representation; The verification unit is used to perform pattern verification on the task representation, and when the verification fails, it triggers the invocation of the large language model under the preset output format and field constraints to obtain a new task representation. A receiving unit is used to receive environmental state data, which includes the type, attributes and pose information of the operated object, and write the environmental state data into an asynchronous cache in chronological order. The matching unit is used to perform matching calculations in the asynchronous cache based on the job target, job constraints and timestamp corresponding to the task representation, and generate an executable task tuple that contains at least the object identifier, object pose and target region of the operated object based on the matching result. An execution unit is used to control the robot to perform navigation and grasping operations based on the object pose in the executable task tuple, and to control the robot to perform placement operations based on the target area, so as to complete the task objective under the condition of satisfying the task constraints; An exception handling unit is used to monitor the start, execution and completion status of each stage of the navigation and grasping operation and the placement operation through a dual-channel handshake mechanism, and to call a fault tolerance strategy to handle exceptions. The publishing unit is used to publish the status and result events of the natural language job instruction from parsing to execution completion in a unified format.

[0016] Thirdly, the present invention also provides a computer-readable storage medium for storing a computer-readable program or instruction, which, when executed by a processor, can implement the steps of the robot operation instruction control method described in any of the above implementations.

[0017] The beneficial effects of this invention are: The robot operation instruction control method provided by this invention acquires natural language operation instructions input by the user to the robot. These instructions include the task objective and constraints to be completed. Under preset output formats and field value constraints, a large language model is invoked to parse the natural language operation instructions into a structured task representation, achieving controlled parsing of the instructions and thus a stable and controllable conversion from natural language to structured tasks. The task representation undergoes pattern verification. If verification fails, the large language model is invoked again under preset output formats and field constraints to obtain a new task representation. Through multiple constraint mechanisms, the free generation of the large language model is restricted to predefined fields and their value ranges, reducing the probability of missing fields, out-of-bounds values, and semantic "illusions," resulting in a parsed task representation with good machine readability, verifiability, and reproducibility. Environmental state data, including the type, attributes, and pose information of the operated object, is received and written to an asynchronous cache in chronological order, achieving highly robust asynchronous alignment between parsing and perception. Based on the task objective, constraints, and timestamp corresponding to the task representation, matching calculations are performed in the asynchronous cache, and the results are used to determine the appropriate method. This paper generates executable task tuples containing at least the object identifier, object pose, and target region of the operated object. By asynchronously caching and pairing these tuples, tasks and perception data can be reliably paired without mandatory synchronization, improving task initiation success rate and reducing average initiation latency. Based on the object pose in the executable task tuple, the robot is controlled to perform navigation and grasping operations, and based on the target region, it is controlled to perform placement operations. Under the condition of satisfying the operational constraints, the robot completes the operational objective, ensuring collaborative work within a unified execution state machine. A dual-channel handshake mechanism monitors the start, execution, and completion states of each stage of navigation, grasping, and placement operations, and invokes fault-tolerance strategies to handle anomalies, achieving closed-loop control based on "natural language operation instructions—perception—navigation—robotic arm operation," thus improving the safety and reliability of robot control. The paper publishes the state and result events of the natural language operation instructions from parsing to execution completion in a unified format, enhancing the observability and evaluability of robot control. This application achieves controllable constraints on LLM output, improving the robot's task initiation success rate, task completion rate, and the safety and reliability of robot control. Attached Figure Description

[0018] Figure 1 A schematic flowchart of an embodiment of the robot operation instruction control method provided by the present invention; Figure 2 A schematic diagram of the fields for a structured task representation provided by the present invention; Figure 3 A schematic diagram illustrating the time window and pairing strategy provided by this invention; Figure 4 This is a timing diagram of the dual-channel handshake and message association provided by the present invention; Figure 5 This is a schematic diagram of the execution state machine provided by the present invention; Figure 6 This is another flowchart illustrating the robot operation instruction control method provided by the present invention; Figure 7 This is a structural block diagram of the robot operation instruction control system provided by the present invention; Figure 8 A schematic diagram of the robot operation instruction control system provided by the present invention. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0020] In the description of the embodiments of the present invention, unless otherwise stated, "a plurality of" means two or more.

[0021] The terms "first," "second," etc., used in the embodiments of this invention are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a technical feature defined with "first" or "second" may explicitly or implicitly include at least one of that feature.

[0022] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0023] This invention provides a robot operation instruction control method and system, which will be described below.

[0024] The executing entity of the robot operation instruction control method in this application embodiment can be the robot operation instruction control system provided in this application embodiment, or different types of electronic devices such as server equipment, physical host, or user equipment (UE) that integrate the robot operation instruction control system. The robot operation instruction control system can be implemented in hardware or software. The UE can be a terminal device such as a smartphone, tablet computer, laptop computer, handheld computer, desktop computer, or personal digital assistant (PDA).

[0025] Figure 1 A schematic flowchart of an embodiment of the robot operation instruction control method provided by the present invention is shown below. Figure 1 As shown, the robot operation command control method includes: S101. Obtain the natural language task instructions input by the user to the robot, wherein the natural language task instructions include the task objectives and task constraints to be completed.

[0026] In this embodiment, the robot can be configured with a perception module and an execution module. It can receive natural language task instructions input by the user through the instruction interface module. The instructions include task objectives (such as color, quantity, target area, etc.) and task constraints, where the task constraints are the necessary constraints.

[0027] The task objective to be completed refers to the specific task to be accomplished in the natural language task instruction, i.e., the object that the robot needs to operate on. Task constraints refer to the conditions or limitations imposed upon completing the task objective, or the state the robot should be in after completing the task objective. Taking a robot used in a smart warehousing scenario as an example, the user can be a warehouse manager who, through the robot's human-machine interface, inputs the natural language instruction: "Please deliver 3 red packages to area A and 2 blue packages to area B." Here, "red packages," "blue packages," and their corresponding quantities constitute the task objective, while "areas A" and "areas B" are the task constraints.

[0028] S102. Under the preset output format and field value constraints, the large language model is invoked to parse the natural language job instructions into a structured task representation.

[0029] Here, LLM can be a GPT model. The default output format is a JSON array.

[0030] Specifically, by calling the large language model under preset output format and field value constraints, strong constraints are applied to the output of the large language model, and natural language job instructions are parsed into structured task representations. This achieves controlled parsing of natural language job instructions, and thus realizes stable and controllable conversion from natural language to structured tasks.

[0031] S103. Perform pattern verification on the task representation, and if the verification fails, trigger the invocation of the large language model under the preset output format and field constraints to obtain a new task representation.

[0032] Specifically, pattern validation can be performed on the syntactic correctness, field completeness, and value validity of the task representation. If the validation fails (e.g., due to the presence of unauthorized fields or values), a new task representation is generated again using LLM under the same output format and field value constraints. Understandably, in this embodiment, through a multi-constraint mechanism of controlled LLM parsing, pattern validation of the parsed output, and constrained regeneration, the free generation of the large language model is limited to predefined fields and their value ranges. This reduces the probability of missing fields, out-of-bounds values, and semantic "illusions," resulting in a parsed task representation with good machine readability, verifiability, and reproducibility.

[0033] It is worth noting that when the verification fails, the output is still subject to the constraints of the output mode and fields, and a regeneration is triggered at least once until the verification requirements are met or the preset maximum number of retries is reached.

[0034] S104. Receive environmental state data, which includes the type, attributes, and pose information of the object being operated on, and write the environmental state data into an asynchronous cache in chronological order.

[0035] The robot is equipped with a perception module for continuous environmental awareness. This module can consist of vision and pose sensors to acquire environmental state data, such as the type, attributes, pose information, and timestamp of the object being manipulated. In a smart warehousing scenario, the object being manipulated can be any package within the field of view, and the corresponding environmental state data can include the package's ID, color, 3D pose, and timestamp.

[0036] Specifically, environmental status data is written to an asynchronous cache in chronological order, divided into time windows. This asynchronous cache can maintain data up to a preset time (e.g., the last 5 seconds), divided into multiple time windows, with expired data automatically evicted.

[0037] Understandably, in this embodiment, the environmental state data corresponding to historical sensing frames is continuously cached asynchronously, thereby achieving highly robust asynchronous alignment between parsing and sensing.

[0038] S105. Based on the job target, job constraints and timestamp corresponding to the task representation, perform matching calculation in the asynchronous cache, and generate an executable task tuple that contains at least the object identifier, object pose and target region of the operated object according to the matching result.

[0039] The executable task tuple is a package of instructions that carries the robot's execution intent, minimizes the scope, and can directly drive the robot's actions, corresponding to the task instructions. For example, the executable task tuple, <obj_id, pose, zone>, can be obtained by pairing the task JSON with the corresponding awareness frame of the environmental state data in the asynchronous cache.

[0040] Specifically, based on the timestamp carried by the structured task representation and preset matching rules, environmental state data within the target time window is selected from the asynchronous cache. Matching calculations are performed between the structured task representation and each environmental state data point. When the integrity condition is met and the matching score is greater than or equal to a preset confidence threshold, an executable task tuple is generated, which includes at least the object identifier, object pose, and target region. Understandably, by generating executable task tuples through asynchronous caching and pairing, tasks can be reliably paired with sensing data without forced synchronization, improving the task startup success rate and shortening the average startup latency.

[0041] In one specific implementation, using a structured task representation and its timestamp, at least one frame of environmental state data closest to the timestamp of the structured task representation is retrieved from the asynchronous cache. Based on the task objective and constraints, candidate objects are filtered from the environmental state data, and a corresponding comprehensive matching score is calculated for each candidate object. From all candidate objects, the candidate object with the highest matching score exceeding a preset confidence threshold is selected. Its identifier (ID) and pose information are then bound to the target region specified in the task, and packaged to generate an executable task tuple that can directly drive the robot to execute. In simulation comparisons, enabling asynchronous pairing significantly improves the task startup success rate and completion rate.

[0042] S106. Based on the object pose in the executable task tuple, control the robot to perform navigation and grasping operations, and based on the target area, control the robot to perform placement operations, and complete the task objective under the condition of satisfying the task constraints.

[0043] Specifically, each executable task tuple can be processed sequentially according to a preset unified execution state machine. The executable task tuple is then published to the message bus of the execution module. The navigation submodule and operation (such as the robotic arm) submodule in the execution module complete the navigation phase and operation phase sequentially according to the unified execution state machine. The execution state machine includes at least navigation to the pre-grasp position, execution of the grasp operation, navigation to the placement position, and execution of the placement operation, ensuring that the robot works collaboratively under the unified execution state machine.

[0044] S107. The start, execution and completion status of each stage of the navigation and grasping operation and the placement operation are monitored through a dual-channel handshake mechanism, and a fault tolerance strategy is invoked to handle abnormalities.

[0045] Specifically, during the execution of the state machine, a dual-channel handshake mechanism consisting of a control channel and a state channel monitors and confirms the start, execution, completion, and failure states of each stage. For example, the monitoring module listens to the state channel in real time. If a stage times out, such as navigation timeout of 45 seconds, a fault-tolerance strategy is triggered: for example, retrying the current stage, replanning the path, or triggering re-sensing to update the target pose. All anomalies and processing actions are recorded. Understandably, this embodiment clearly defines stage divisions and state transitions, achieving reliable state synchronization and confirmation through dual control and state channels. This enhances the coordination and fault tolerance capabilities of the navigation and robotic arm subsystems, enabling rapid detection of anomalies (timeouts, failures) and automatic handling through strategies (retry, replanning, skipping), reducing the risk of task "freezing," improving the overall task completion rate, and realizing closed-loop control based on "natural language operation instructions—sensing—navigation—robotic arm operation," thus improving the safety and reliability of robot control.

[0046] S108. Publish the status and result events of the natural language job instruction from parsing to execution completion in a unified format.

[0047] Specifically, the event publishing module can be used to standardize and encode the state and result events of natural language task instructions from parsing to execution completion, such as task reception, successful parsing, asynchronous pairing completion, start / end of each stage, anomaly detection and handling, and task closed-loop completion. These lifecycle events are then published to the visualization and scoring interface, displaying the task flow, timing curves, and key indicators on the host computer or visualization platform. This standardized publishing of the entire lifecycle events facilitates full-process observability and evaluability, greatly benefiting teaching demonstrations, competition scoring, engineering debugging, and system performance optimization. It also provides a complete data foundation for fault location and effect evaluation. Used for teaching demonstrations, competition scoring, and engineering debugging, it enhances the observability and evaluability of robot control.

[0048] Understandably, this embodiment enables controllable constraints on LLM output, thereby improving the robot's task initiation success rate, task completion rate, and the safety and reliability of robot control.

[0049] In summary, the robot operation instruction control method provided by this invention obtains natural language operation instructions input by the user to the robot. These instructions include the task objective and constraints to be completed. Under preset output format and field value constraints, a large language model is invoked to parse the natural language operation instructions into a structured task representation, achieving controlled parsing of the natural language operation instructions and thus a stable and controllable conversion from natural language to structured tasks. The task representation undergoes pattern verification, and upon verification failure, a large language model is invoked under preset output format and field constraints to obtain a new task representation. Through multiple constraint mechanisms, the free generation of the large language model is restricted to predefined fields and their value ranges, reducing the probability of missing fields, out-of-bounds values, and semantic "illusions," resulting in a parsed task representation with good machine readability, verifiability, and reproducibility. Environmental state data, including the type, attributes, and pose information of the operated object, is received and written to an asynchronous cache in chronological order, achieving highly robust asynchronous alignment between parsing and perception. Based on the task objective, task constraints, and timestamp corresponding to the task representation, matching calculations are performed in the asynchronous cache, and... The matching results generate executable task tuples containing at least the object identifier, object pose, and target region of the operated object. Executable task tuples are generated asynchronously and through pairing, reliably pairing tasks with perception data without forced synchronization, improving task initiation success rate and shortening average initiation latency. Based on the object pose in the executable task tuple, the robot is controlled to perform navigation and grasping operations, and based on the target region, it is controlled to perform placement operations, completing the task objective under the condition of satisfying the task constraints, ensuring collaborative work of the robot under a unified execution state machine. A dual-channel handshake mechanism monitors the start, execution, and completion states of each stage of navigation and grasping operations and placement operations, and invokes fault-tolerance strategies to handle anomalies, realizing closed-loop control based on "natural language task instructions—perception—navigation—robotic arm operation," improving the safety and reliability of robot control. The state and result events of the natural language task instructions from parsing to execution completion are published in a unified format, enhancing the observability and evaluability of robot control. This application achieves controllable constraints on LLM output, improving the robot's task initiation success rate, task completion rate, and the safety and reliability of robot control.

[0050] In some embodiments of the present invention, the structured task is represented as a fielded data representation, which is in JSON or Protobuf format; the fields in the fielded data representation include at least one of target category, color, quantity, and target region.

[0051] Specifically, the structured task is represented as a field-based data representation, which is in JSON or Protobuf format. The fields in the field-based data representation include at least one of the following: target category, color, quantity, and target region. Natural language job instructions are parsed into controlled field-based data representations, such as JSON or Protobuf structures. This achieves the control of the LLM's restricted output as a field-based data representation, avoiding the problems of uncertain and difficult-to-constrain LLM outputs, and preventing unstructured text, illegal fields, or out-of-bounds values ​​from directly entering the control link and causing security risks.

[0052] In one specific implementation, a large language model is invoked under preset output patterns and field constraints. The output is limited to a collection of JSON key-value pairs using templated prompts. A field whitelist constrains the allowed field names, declaring the set or range of values ​​for each field in the prompts, ensuring the model only outputs predefined fields and their values. The parsed structured task representation input pattern validator checks syntax correctness, field integrity, and value validity according to preset JSON Schema or regular expression rules. If validation fails, a limited number of regenerations are triggered under the same output pattern and constraints until validation passes or the maximum number of retries is reached.

[0053] It should be noted that the field-based data representation can also include priority and confidence. Priority is used to provide a scheduling basis for the task queue. When the system processes multiple task tuples simultaneously, the execution module can determine the execution order based on the priority field, thereby optimizing the overall job efficiency or meeting specific job process requirements. Confidence is generated by the large language model at output and is used to represent its certainty about the parsing result (e.g., mapping "red" to the field "color": "red"), providing a quality signal for downstream decision-making. This field can be used as an input parameter for execution or fault tolerance strategies.

[0054] In some embodiments of the present invention, the step of calling the large language model under preset output format and field value constraints includes: limiting the output format of the large language model to a key-value pair structure through templated prompt words; limiting the allowed field names in the key-value pair structure through a field whitelist, and declaring the allowed value range of each field name in the templated prompt words.

[0055] Specifically, the output format of LLM is limited to a key-value pair structure by using templated prompts; the allowed field names in the key-value pair structure are limited by a field whitelist, and the allowed value range of each field name is declared in the templated prompts. This restricts the free generation of large language models to predefined fields and their value ranges, significantly reducing the probability of missing fields, out-of-bounds values, and semantic "illusions," and making the parsing results have good machine readability, verifiability, and reproducibility.

[0056] In one specific approach, the pre-defined constraints include: Output format: Must be a JSON array.

[0057] Field whitelist: Only allow output of the color, num, and to fields.

[0058] Value constraints: color is limited to ["red", "blue"]; to is limited to ["A", "B", "C"]; num is an integer greater than 0.

[0059] Upon receiving a system prompt containing the above constraints, LLM parses the natural language instruction into a structured task representation: json. [ {"color":"red","num":1,"to":"A"}, {"color":"blue","num":1,"to":"B"} ] In another specific implementation, such as Figure 2 The diagram shows the fields of a structured task representation. The structured task representation uses a JSON-style key-value pair collection. The corresponding fields, types, value ranges, and whether they are required are all given in the table. The `color` field is an enumeration type, limited to {red, blue}, used to specify the target packaging color; the `num` field is an integer type, with a value range of 1..N, representing the number of targets to be processed; the `to` field is an enumeration type, limited to {A, B, C}, corresponding to three placement areas. All three fields are required. The `priority` field is an optional integer used to represent the task priority, with a recommended value of 1..5; the `confidence` field is an optional floating-point number, with a value range of 0.0..1.0, used to record the confidence level of large model parsing or asynchronous pairing. This field table allows for pattern constraints and integrity checks on the output results of large models, providing a unified task description format for downstream modules.

[0060] In some embodiments of the present invention, writing the environmental state data into the asynchronous cache in chronological order includes: dividing the asynchronous cache into multiple time windows; storing the received environmental state data into the corresponding time window according to the timestamp; and managing the time windows using the most recently valid and expired elimination rules.

[0061] Specifically, the perception module continuously receives multiple frames of environmental state data. Each frame includes information such as the type, color, number, and pose of the object being manipulated, and is written to the asynchronous cache in timestamp order. The asynchronous cache is divided into multiple time windows, and each time window stores the perception data within its corresponding time range. Expired time windows are cleared using an eviction policy to ensure cache capacity and timeliness. Furthermore, environmental state data is saved by time window so that it can be matched with the most recent valid data later by combining the task timestamp. Thus, during matching, environmental state data whose timestamp matches the timestamp corresponding to the natural language job instruction and is not expired is prioritized.

[0062] In some embodiments of the present invention, the matching calculation based on the job target, job constraints and timestamp corresponding to the task representation in the asynchronous cache includes: selecting environmental state data corresponding to the target time window from the asynchronous cache according to the timestamp; performing matching calculations on the structured task representation and the selected environmental state data based on object attribute similarity, spatial proximity and temporal proximity, and performing weighted summation calculations on the object attribute similarity, spatial proximity and temporal proximity according to preset weights to obtain the matching result.

[0063] The target time window is the time window in which the time difference between the timestamp corresponding to the task representation is less than a preset time threshold Δt.

[0064] Specifically, based on the timestamps corresponding to the task representations and preset matching rules, candidate environmental state data can be selected from the asynchronous cache to generate executable task tuples. The preset matching rules can be a multi-dimensional matching score model, where the dimensions include attribute similarity, spatial proximity, and temporal proximity. Attribute similarity, spatial proximity, and temporal proximity can be calculated for the task representation's corresponding job objectives and constraints with the environmental state data corresponding to multiple target time windows, and then weighted and combined. When the matching score meets the confidence threshold and the corresponding environmental state data has not expired, an executable task tuple is generated, which includes at least the object identifier, object pose, and target region. In the multi-dimensional matching score model, various weights and thresholds can be optimized according to the scenario, or machine learning methods can be introduced to automatically learn the parameters. Understandably, through asynchronous caching of multiple time windows and a matching score model that simultaneously considers object attributes, spatial proximity, and temporal proximity, task parsing results and perceived data can be robustly paired on the timeline without forced synchronization, significantly improving task startup success rate and shortening average startup latency.

[0065] In one specific implementation, the structured task representation carries fields such as task generation timestamp, target category, color, quantity, and target region. The matching module selects one or more candidate time windows from the asynchronous cache based on the task timestamp and calculates a multi-dimensional matching score for each candidate object. Attribute similarity: The degree to which attributes such as color and category in the task match the attributes of the object; Spatial proximity: The spatial positional relationship of an object relative to a preset work area or the robot's current pose; Time proximity: The difference between the object observation time and the task timestamp.

[0066] The three types of indicators are weighted according to preset weights to obtain a comprehensive matching score. When the score is not lower than the confidence threshold and the candidate frame has not expired, an executable task tuple containing at least object identifier, object pose, target region and other information is generated for the task, and the generation of multiple object sequential lists is supported.

[0067] In one specific implementation, such as Figure 3 The diagram illustrates the time window and pairing strategy: The environment aggregation module maintains an asynchronous cache window in chronological order, distributing continuously sensed environmental state data across time windows such as t-2, t-1, t, and t+1. When the task JSON arrives, its timestamp is used as a reference to select candidate frames within a preset time window, and a matching score is calculated based on the rules of "identical attributes + spatial proximity + time difference not exceeding Δt". The frame with the highest score and not expired is selected from the remaining candidate frames as the target frame, thereby generating an executable task tuple containing object identifier, object pose, and target region.

[0068] Table 2 shows the main thresholds. Table 2 summarizes the key threshold parameters used in the pairing process, such as the pairing score lower limit score_min, navigation timeout T_nav, robotic arm waiting timeout T_arm, navigation reserved offset d_nav, and feedback sampling period Δt_fb. These parameters can be configured according to the actual scenario to achieve a balance between matching accuracy and startup latency.

[0069] Table 2. Main Threshold Table

[0070] This embodiment provides recommended configurations for several key threshold parameters. The pairing score threshold `score_min` is set to 0.7, limiting the overall score of "attribute consistency + spatial proximity" to at least this value for a successful match, thus suppressing incorrect pairings. The navigation timeout `T_nav` is set to 45.0 s; if the chassis fails to reach the predetermined pre-grab / placement position within this time, the task controller triggers cancellation and retry logic. The robotic arm waiting timeout `T_arm` is set to 20.0 s, constraining the waiting time for the robotic arm's execution results; exceeding this threshold is considered a failure of the current action and enters a fault-tolerant branch. The navigation reserved offset `d_nav` is set to 0.4 m, providing a safe distance when generating the pre-grab pose to avoid interference between the chassis, worktable, and robotic arm. The feedback sampling period `Δt_fb` is set to 0.3 s, corresponding to the state sampling and retry determination period of the task controller's main loop, achieving a balance between response speed and computational load.

[0071] Table 3 shows the comparison and effect data: comparing the effects of different strategy configurations under the same simulation scenario, it can be seen that the method of the present invention has significant advantages in terms of startup success rate and task completion rate. The baseline scheme is "without asynchronous pairing", with a startup success rate of only 78.2% and an average startup latency of 1.62 s. Some tasks failed due to changes in target state or information asynchrony. After enabling asynchronous pairing, the startup success rate increased to 96.4%, and the average startup latency decreased to 0.74 s, indicating that the time window and pairing strategy can fully reuse the environment cache and reduce invalid restarts. On this basis, after adding handshake and fault tolerance mechanisms, the task completion rate further increased from 93.8% to 98.1%. While basically maintaining the startup latency (slightly increased by 0.78 s), it effectively reduced the impact of anomalies such as capture failure and navigation timeout on the overall task, verifying the comprehensive optimization effect of the method of the present invention in terms of reliability and real-time performance.

[0072] Table 3. Comparison and Results Data Table

[0073] In some embodiments of the present invention, the dual-channel handshake mechanism includes a control channel and a status channel; the monitoring of the start, execution, and completion status of each stage of the navigation and grasping operation and the placement operation through the dual-channel handshake mechanism includes: issuing control instructions and synchronization events for each stage through the control channel; and receiving state enumerations representing each stage reported by the robot through the status channel; wherein, a task identifier and an object identifier are written into the messages transmitted in the control channel and the status channel, so as to associate the messages transmitted in different channels according to the identifiers.

[0074] During the execution of the state machine, a two-channel handshake mechanism consisting of a control channel and a state channel is used to monitor and confirm the start, execution, completion, and failure states of each stage.

[0075] The control channel is used to publish phase control instructions and key synchronization events; the status channel is used to publish the status enumeration of the execution module, including at least one or more of the following: {READY (ready to capture), START (start capturing), DOING (capturing in progress), DONE (capturing successfully completed), FAIL (capturing failed)}. The status is reported in the form of enumeration values. Messages in both the control and status channels carry task identifiers and object identifiers to establish correlation bindings between different messages.

[0076] Specifically, control commands and synchronization events for each stage are issued through the control channel; state enumerations representing each stage are received from the robot via the state channel, enabling the definition of navigation and operation completion times and eliminating collaboration errors caused by polling between modules through event broadcasting. Simultaneously, failures (FAIL) at each stage can be captured in real time by the task controller, and combined with identifiers, the specific task and object causing the problem can be immediately located, providing accurate input for quickly triggering fault-tolerant strategies. Furthermore, all critical interaction messages are standardized and carry identifiers, providing a complete and structured data chain for system debugging, log analysis, and performance evaluation, ensuring reliable synchronization and state traceability between the navigation and operation subsystems.

[0077] In one specific implementation, such as Figure 4 The diagram shown illustrates the timing of the two-channel handshake and message association. The specific process is as follows: Navigation Completion and Event Reporting (Control Channel): Once the robot chassis navigation subsystem (Nav2) completes its movement towards the target point (e.g., the pre-grasp point), it immediately publishes a critical synchronization event via the control channel topic ` / task_sync_nav2arm`. For example, upon reaching the pre-grasp pose, the event `nav_done_grasp` is published; upon reaching the pre-placement point in the placement area, the event `nav_done_place` is published. This event message carries a task identifier (e.g., `mission_id: 1001`) and an object identifier (e.g., `obj_id: red_box_001`), used to establish a unique association in all subsequent messages.

[0078] Task Controller Response and Command Distribution: The task controller continuously listens to the control channel. Upon receiving the `nav_done_grasp` event, it parses the object identifier `red_box_001`. The controller then publishes this object identifier to the robotic arm subsystem via a separate topic ` / pick_target_id`, informing it of the specific target to be manipulated. Simultaneously, the controller sends specific motion commands to the robotic arm (e.g., calling the grasping plan via the MoveIt motion interface).

[0079] Operation Execution and Status Feedback (Status Channel): After receiving the instruction, the robotic arm subsystem begins to execute the grasping action and continuously reports its execution status through the status channel topic / task_sync_arm2nav. The status message also carries the same task identifier and object identifier as the event message (mission_id: 1001, obj_id: red_box_001).

[0080] Status monitoring and next stage triggering: The task controller simultaneously listens to the status channel. When it receives the ARM_STATUS_DONE status and the identifier matches, it determines that the current capture stage has been successfully completed. The controller then triggers the next stage (placement stage) by sending a placement point command to the target area "A" through the navigation interface, thereby initiating a new round of "navigation-operation" handshake loop.

[0081] As shown in Table 1, this is a QoS and message association table. To ensure the reliability of the above process, this embodiment configures a Quality of Service (QoS) policy for key topics. The control channel topic / task_sync_nav2arm and the status channel topic / task_sync_arm2nav are both configured as RELIABLE (reliable transmission) and TRANSIENT_LOCAL (local persistence).

[0082] This configuration ensures that critical synchronization events and status messages are not lost even in the event of brief network fluctuations or node restarts, and that nodes that come online later can read the missed messages.

[0083] The topic / pick_target_id uses the same configuration to ensure reliable transmission of the target identifier.

[0084] Consistent task and object identifiers across all messages enable the system to unambiguously associate and bind control events, status feedback, and specific targets in complex multi-task, multi-object concurrent scenarios, achieving precise cross-module collaboration and status tracking.

[0085] Table 1. QoS and Message Association Table

[0086] This embodiment configures a highly reliable QoS policy separately for critical control and status channels. The topics / task_sync_nav2arm, / task_sync_arm2nav, and / pick_target_id all use RELIABLE +TRANSIENT_LOCAL to ensure that capture / placement start signals, completion statuses, and target object IDs are not lost during network jitter or brief node restarts, and can be reread by later-coming nodes. The environment awareness topics / object_detection / object_poses and the visualization topics / mission_state, / nav_status, and / arm_status use the default QoS with a depth of 10, allowing for a certain degree of latency and packet loss to reduce system bandwidth overhead and meet the design requirement of "strong consistency for control channels and weak consistency for awareness and visualization channels."

[0087] In another specific implementation, to illustrate the implementation process of this embodiment more specifically, the following uses a warehouse sorting task to demonstrate a standardized event log segment generated by the robot during operation.

[0088] Example event log (fragment): [2025-11-13T19:03:04] / mission_state: Received a list of 2 tasks. [2025-11-13T19:03:04] / nav_status: Navigating to pre-fetch point red_cube_1 (x=2.13,y=-1.05) [2025-11-13T19:03:04] / task_sync_nav2arm: nav_done_grasp [2025-11-13T19:03:04] / arm_status: Scraping in progress [2025-11-13T19:03:04] / task_sync_arm2nav: arm_done [2025-11-13T19:03:04] / nav_status: Navigate to placement area A (1.00,2.00) [2025-11-13T19:03:04] / task_sync_nav2arm: nav_done_place [2025-11-13T19:03:04] / task_sync_arm2nav: arm_done [2025-11-13T19:03:04] / mission_state: Completed moving red_cube_1 The above logs demonstrate the complete closed-loop process from task parsing, asynchronous pairing, state machine execution to two-channel handshake.

[0089] In some embodiments of the present invention, the fault tolerance strategy includes at least one of the following: a maximum number of retries limit, timeout rollback to the previous stable state, requesting re-sensing and updating the executable task tuple, and skipping the current object and continuing with subsequent tasks.

[0090] Specifically, within the limit of attempts, the current stage can be retried, the previous stable state can be rolled back and the path can be replanned, the re-sensing can be triggered and the executable task tuple can be updated, the current object can be skipped and the subsequent task can continue.

[0091] In one specific implementation, the monitoring and fault tolerance module checks in real time for navigation timeouts, abnormal robotic arm movements, and missing perception data. Upon detecting a timeout, unmet preconditions, or execution failure, the module executes the following strategies based on a preset protocol: retry the current stage (limited number of times); roll back to the previous stable state and replan the path; trigger re-perception, updating the object pose or target region in the executable task tuple; skip the current object and continue with subsequent tasks. Simultaneously, it records traceability information such as the exception type, triggering conditions, and processing results.

[0092] In some embodiments of the present invention, step S106 includes: arranging the navigation, grasping and placement operations through a preset execution state machine; controlling the robot to sequentially perform pre-pose adjustment, target pose execution and pose confirmation during the grasping and placement phases of the execution state machine; and triggering and executing fine-tuning actions when the pose error exceeds a preset threshold during the grasping or placement phase.

[0093] The preset execution state machine includes at least the following states: standby, navigation to the pre-grab position, grab, navigation to the placement position, placement, and completion / failure. The grab and placement phases can be further subdivided into pre-pose, target pose, and confirmed pose to improve pose accuracy.

[0094] Specifically, the execution state machine sets up steps for pre-pose adjustment, target pose execution, and pose confirmation in the grasping and placement phases, respectively. When the pose error is detected to exceed the preset threshold, a fine-tuning action is triggered to ensure that the robotic arm approaches the target with the optimal path and posture, thereby ensuring operational safety and avoiding equipment damage.

[0095] In one specific implementation, such as Figure 5 The diagram shows a state machine for execution. This state machine processes executable task tuples sequentially, specifically as follows: State Transitions and Events: After receiving the executable task tuple, the state machine triggers state transitions sequentially, starting from the standby state. The transitions between the stages of navigation to the pre-grasp pose, grasp, navigation to the place pose, and placement are event-driven and monitored by the dual-channel handshake mechanism. Typical events include NAV_DONE_GRASP and NAV_DONE_PLACE reported by the navigation module, and ARM_DONE or ARM_FAIL reported by the robotic arm module.

[0096] Fine-grasp / placement phase control: In the grasp and place phases, pre-pose adjustment, target pose execution, and pose confirmation steps are performed respectively. If the error between the current pose and the target pose exceeds a preset threshold in any step, a pose fine-tuning action is triggered.

[0097] Exception handling: If any stage is not completed within the preset time (such as navigation timeout or capture failure), the state machine will trigger the built-in fault tolerance strategy (such as retry, replanning or skipping the current object), and can enter the next stage or jump to the Fail state according to the strategy execution result.

[0098] In some embodiments of the present invention, the step of publishing status and result events in a unified format includes: performing standardized message mapping on the natural language job instruction parsing success event, the executable task tuple generation event, the status change events of each stage of the navigation, grasping and placing operation, and the job goal achievement event; and outputting the mapped events to an external interface.

[0099] Specifically, by outputting analysis results, pairing processes, execution status, and exception information through a standardized lifecycle event model, it facilitates visualization and quantitative evaluation in teaching and competition scenarios, and also provides a unified data foundation for troubleshooting and performance tuning in engineering deployments.

[0100] In one specific implementation, such as Figure 6 The diagram shown is another flowchart illustrating the robot's operation command control method. Command Acquisition: Receives natural language commands containing job objectives and constraints.

[0101] Controlled parsing and validation: Under preset format and value constraints, the Large Language Model (LLM) is invoked for parsing, and the output structured task representation is validated for pattern. If the validation fails, the constrained regeneration is triggered.

[0102] Environment aggregation and asynchronous caching: Continuously receive environment-aware data and write it to the asynchronous cache according to time windows.

[0103] Asynchronous pairing: Based on the timestamp of the task representation and the cached data, a matching calculation is performed to generate an executable task tuple containing the object identifier, pose, and target region.

[0104] State machine execution and dual-channel handshake: Based on task tuples, the robot is controlled to complete navigation, grasping and placement operations in sequence through a unified execution state machine, and the state at each stage is monitored and confirmed by a dual-channel handshake mechanism.

[0105] Event publishing: Publish the entire process of a task from parsing to execution completion in a unified format.

[0106] In one specific implementation, such as Figure 7 The diagram shown is a structural block diagram of the robot operation command control system. Each module is decoupled from the service interface via a message bus, forming a complete closed-loop system from command input to task execution and feedback. Specifically: Command Interface Module: Receives natural language job commands input by the user.

[0107] Parsing and Verification Module: Calls the large language model to parse natural language instructions into structured task representations and performs pattern verification.

[0108] Environment aggregation and asynchronous caching module: Receives and manages environmental state data from the perception module.

[0109] Matching module: Matches the structured task representation with the environment data in the cache to generate executable task tuples.

[0110] The execution module (including the navigation submodule and the operation submodule) controls the robot to perform navigation and robotic arm operations based on the executable task tuple.

[0111] Monitoring and fault tolerance module: Monitors the execution status through a dual-channel handshake mechanism and triggers fault tolerance strategies in case of anomalies.

[0112] Event publishing module: Standardizes and publishes events throughout the entire lifecycle.

[0113] To better implement the robot operation instruction control method in the embodiments of the present invention, based on the robot operation instruction control method, correspondingly, as follows: Figure 8 As shown, this embodiment of the invention also provides a robot operation instruction control system, the robot operation instruction control system 800 including: The acquisition unit 801 is used to acquire natural language task instructions input by the user to the robot, the natural language task instructions including the task objectives and task constraints to be completed; The parsing unit 802 is used to call the large language model under the preset output format and field value constraints to parse the natural language job instruction into a structured task representation; The verification unit 803 is used to perform pattern verification on the task representation, and when the verification fails, it triggers the invocation of the large language model under the preset output format and field constraints to obtain a new task representation. The receiving unit 804 is used to receive environmental state data, which includes the type, attributes and pose information of the operated object, and writes the environmental state data into an asynchronous buffer in chronological order. The matching unit 805 is used to perform matching calculations in the asynchronous cache based on the job target, job constraints and timestamp corresponding to the task representation, and generate an executable task tuple that contains at least the object identifier, object pose and target region of the operated object based on the matching result. The execution unit 806 is used to control the robot to perform navigation and grasping operations based on the object pose in the executable task tuple, and to control the robot to perform placement operations based on the target area, so as to complete the task objective under the condition of satisfying the task constraints; The exception handling unit 807 is used to monitor the start, execution and completion status of each stage of the navigation and grasping operation and the placement operation through a dual-channel handshake mechanism, and to call the fault tolerance strategy for handling when an exception occurs. The publishing unit 808 is used to publish the status and result events of the natural language job instruction from parsing to execution completion in a unified format.

[0114] The robot operation instruction control system 800 provided in the above embodiments can realize the technical solutions described in the above robot operation instruction control method embodiments. The specific implementation principles of each module or unit can be found in the corresponding content in the above robot operation instruction control method embodiments, and will not be repeated here.

[0115] Accordingly, this application also provides a computer-readable storage medium for storing computer-readable programs or instructions. When the programs or instructions are executed by a processor, they can implement the steps or functions of the robot operation instruction control methods provided in the above-described method embodiments.

[0116] Those skilled in the art will understand that all or part of the processes of the methods described in the above embodiments can be implemented by a computer program instructing related hardware (such as a processor, controller, etc.), and the computer program can be stored in a computer-readable storage medium. The computer-readable storage medium may be a disk, optical disk, read-only memory, or random access memory, etc.

[0117] The robot operation instruction control method and system provided by the present invention have been described in detail above. Specific examples have been used to illustrate the principle and implementation of the present invention. The description of the above embodiments is only for the purpose of helping to understand the method and core idea of ​​the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation and application scope based on the idea of ​​the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A robot operation command control method, characterized in that, include: The robot receives natural language task instructions input by the user, which include the task objectives and constraints to be completed. Under the preset output format and field value constraints, the large language model is invoked to parse the natural language job instructions into a structured task representation; The task representation is subjected to pattern validation, and if the validation fails, the large language model is invoked under the preset output format and field constraints to obtain a new task representation. Receive environmental status data, which includes the type, attributes, and pose information of the object being operated on, and write the environmental status data into an asynchronous cache in chronological order; Based on the task representation, the corresponding job target, job constraints, and timestamp, a matching calculation is performed in the asynchronous cache, and an executable task tuple containing at least the object identifier, object pose, and target region of the operated object is generated according to the matching result. The robot is controlled to perform navigation and grasping operations based on the object pose in the executable task tuple, and to perform placement operations based on the target area, thereby completing the task objective under the condition of satisfying the task constraints. The start, execution and completion status of each stage of the navigation and grasping operation and the placement operation are monitored through a dual-channel handshake mechanism, and a fault tolerance strategy is invoked to handle abnormalities. The status and result events of the natural language job instruction from parsing to execution completion are published in a unified format.

2. The robot operation instruction control method according to claim 1, characterized in that, The structured task is represented as a fielded data representation, which is in JSON or Protobuf format. The fields in the fieldized data representation include at least one of the following: target category, color, quantity, and target region.

3. The robot operation instruction control method according to claim 1, characterized in that, The process of calling the large language model under preset output format and field value constraints includes: The output format of the large language model is limited to a key-value pair structure by using templated prompt words; The allowed field names in the key-value pair structure are limited by a field whitelist, and the allowed value range of each field name is declared in the templated prompt.

4. The robot operation instruction control method according to claim 1, characterized in that, The step of writing the environmental state data into the asynchronous cache in chronological order includes: The asynchronous cache is divided into multiple time windows; The received environmental status data is stored in the corresponding time window according to the timestamp. The time windows are managed using the most recently valid and expired elimination rules.

5. The robot operation instruction control method according to claim 1, characterized in that, The matching calculation based on the job target, job constraints, and timestamp corresponding to the task representation in the asynchronous cache includes: Based on the timestamp, select the environmental state data corresponding to the target time window from the asynchronous cache; The structured task representation is matched with the selected environmental state data based on object attribute similarity, spatial proximity, and temporal proximity. The object attribute similarity, spatial proximity, and temporal proximity are weighted and summed according to preset weights to obtain the matching result.

6. The robot operation instruction control method according to claim 1, characterized in that, The monitoring of the start, execution, and completion status of each stage of the navigation, grasping, and placement operations via a dual-channel handshake mechanism includes: Control commands and synchronization events for each stage are issued through the control channel; The state channel receives state enumerations representing each stage reported by the robot. In this process, a task identifier and an object identifier are written into the messages transmitted in the control channel and the status channel, so as to associate the messages transmitted in different channels according to the identifier.

7. The robot operation instruction control method according to claim 1, characterized in that, The fault tolerance strategy includes at least one of the following: Maximum retry limit, timeout rollback to the previous stable state, request to re-aware and update the executable task tuple, skip the current object and continue with subsequent tasks.

8. The robot operation instruction control method according to claim 1, characterized in that, The process of controlling the robot to perform navigation and grasping operations based on the object pose in the executable task tuple, and controlling the robot to perform placement operations based on the target area, to complete the task objective under the condition of satisfying the task constraints, includes: The navigation, grasping, and placement operations are programmed using a preset execution state machine. During the grasping and placement phases of the execution state machine, the robot is controlled to sequentially perform pre-pose adjustment, target pose execution, and pose confirmation, respectively. During the grasping or placement phase, when a pose error exceeding a preset threshold is detected, a fine-tuning action is triggered and executed.

9. The robot operation instruction control method according to claim 1, characterized in that, The publication of status and result events in a unified format includes: Standardized message mapping is performed on the natural language job instruction parsing success event, the executable task tuple generation event, the state change events of each stage of the navigation, grasping and placement operation, and the job goal achievement event; Output the mapped events to the external interface.

10. A robot operation instruction control system, characterized in that, include: The acquisition unit is used to acquire natural language task instructions input by the user to the robot, the natural language task instructions including the task objectives and task constraints to be completed; The parsing unit is used to call the large language model under the preset output format and field value constraints to parse the natural language job instructions into a structured task representation; The verification unit is used to perform pattern verification on the task representation, and when the verification fails, it triggers the invocation of the large language model under the preset output format and field constraints to obtain a new task representation. A receiving unit is used to receive environmental state data, which includes the type, attributes and pose information of the operated object, and write the environmental state data into an asynchronous cache in chronological order. The matching unit is used to perform matching calculations in the asynchronous cache based on the job target, job constraints and timestamp corresponding to the task representation, and generate an executable task tuple that contains at least the object identifier, object pose and target region of the operated object based on the matching result. An execution unit is used to control the robot to perform navigation and grasping operations based on the object pose in the executable task tuple, and to control the robot to perform placement operations based on the target area, so as to complete the task objective under the condition of satisfying the task constraints; An exception handling unit is used to monitor the start, execution and completion status of each stage of the navigation and grasping operation and the placement operation through a dual-channel handshake mechanism, and to call a fault tolerance strategy to handle exceptions. The publishing unit is used to publish the status and result events of the natural language job instruction from parsing to execution completion in a unified format.

Images