A somatically intelligent instruction execution method, system, device, and medium

Abstract

The application provides a body intelligent instruction execution method, system, device and medium, comprising obtaining current environment state information of a robot and a natural language task instruction input by a user; at least based on the environment state information, the task instruction, a pre-defined standardized function interface document and a thinking model example, a prompt word is generated, and the generated prompt word is input into a large language model; the task instruction is decomposed into a sub-task sequence by using the large language model, and the decomposed sub-task sequence is organized into a complete executable code block; static logic verification is performed on the code block before code execution; if the verification fails, an error description is generated to feed back to the large language model for code correction; in the code execution stage, if abnormal information is returned after code execution, the abnormal information is returned to the large language model, so that the large language model generates corrected code according to the abnormal reason. The application improves the autonomous decision level and task execution success rate of the robot.

Description

Technical Field

[0001] This invention relates to the field of embodied intelligence technology, and in particular to an embodied intelligence instruction execution method, apparatus, device, and medium. Background Technology

[0002] With the rapid development of robotics technology, quadrupedal robots (commonly known as "robot dogs") have been widely used in fields such as power line inspection, industrial park security, and disaster relief due to their superior mobility and adaptability to complex unstructured terrain (such as stairs, ruins, and narrow passages). Traditional quadrupedal robot control systems typically rely on predefined behavior trees or finite state machines (FSMs). While this deterministic control method offers high stability, it lacks the ability to understand unstructured instructions and generalize decision-making capabilities. Operators must send specific waypoints or control commands through specialized ground station software, as the robot struggles to understand high-level natural language tasks involving complex semantics and logical reasoning, such as "check if the pressure of that red fire extinguisher over there is normal."

[0003] In recent years, large language models (LLMs), represented by GPT-4 and Llama, have demonstrated powerful natural language understanding and code generation capabilities, providing a new path for robots to achieve "embodied intelligence." Current mainstream research attempts to directly use LLMs as the "brain" of robots, translating user commands directly into Python control code or ROS (Robot Operating System) instructions through prompt engineering.

[0004] However, in practical applications, this robot decision-making scheme directly based on LLM has the following significant technical defects and pain points:

[0005] Logical Illusion and Code Inexecutability: Large language models are essentially generative models based on probabilistic prediction, lacking a rigorous understanding of the logic of the physical world. When generating robot control code, the model is highly susceptible to "hallucinations," such as calling functions that do not exist in the robot's API library, generating Python code snippets with syntax errors, or omitting critical parameters (e.g., failing to specify target coordinates for grasping actions). Such inexecutable code can lead to program crashes and even cause safety accidents such as robot malfunction.

[0006] Lack of awareness of physical constraints (Grounding Problem): General-purpose LLMs are unaware of the specific structural limitations of a robot. For example, the model might command the robot's robotic arm to grasp an object beyond its maximum workspac, or perform an operation directly without any prerequisites (such as moving closer to the target first). This "mind-body separation" decision-making approach makes the generated task plans physically unfeasible.

[0007] Poor open-loop decision-making and fault tolerance: Existing LLM-based robot systems mostly adopt an "open-loop" control mode, that is, generating complete task code once and issuing it for execution. However, the real environment is dynamically changing (e.g., the path is suddenly blocked by an obstacle, or the target object's position changes). Once an anomaly occurs during execution, the system cannot automatically capture the error and feed it back to the LLM for re-planning, causing the task to fail directly. It lacks the closed-loop adjustment capability of humans, which allows for "doing and seeing, and dynamically adjusting."

[0008] Insufficient planning capabilities for long-sequence tasks: When faced with complex long-sequence tasks (such as "inspecting the entire floor and sorting out all abnormal items"), the directly generated code is often logically chaotic. The model struggles to maintain long-distance contextual consistency and is prone to losing sight of the objective or confusing the order of steps in the middle or later stages of the task.

[0009] Therefore, there is an urgent need for a technical solution that can effectively combine the semantic understanding capabilities of large language models with the rigor of robot control systems. This solution not only needs to understand abstract natural language instructions, but also needs to establish a complete "generation-verification-feedback" closed-loop mechanism. This mechanism should reduce logical complexity through hierarchical task decomposition, ensure code compliance through static syntax analysis, and achieve dynamic fault tolerance through runtime feedback, thereby truly enabling four-wheeled robots to make intelligent autonomous decisions and perform operations in complex environments. Summary of the Invention

[0010] This invention provides a method, system, device, and medium for executing instructions based on embodied intelligence, which addresses the problems of code logic illusion, non-physical closed loops, and low success rate of long sequence task execution when using large language models to control robots in the prior art.

[0011] According to one aspect of the present invention, a method for executing instructions based on embodied intelligence is provided, comprising:

[0012] Obtain the robot's current environmental status information and the natural language task instructions input by the user;

[0013] Based at least on the environmental state information, the task instructions, the predefined standardized function interface document, and the mind chain example, prompt words are generated and then input into the large language model;

[0014] The task instructions are decomposed into a sequence of subtasks using a large language model, and the decomposed subtask sequences are organized into complete executable code blocks.

[0015] Based on the abstract syntax tree parser, static logic verification is performed on the code block before code execution; if the verification passes, the execution phase begins; if the verification fails, an error description is generated and fed back to the large language model for code correction.

[0016] During the code execution phase, the code is executed line by line. If an exception is returned after the code execution, the exception information is sent back to the large language model, so that the large language model can regenerate and correct the code according to the reason for the exception.

[0017] Optionally, the step of generating prompt words based at least on the environmental state information, the task instructions, predefined standardized function interface documents, and thought chain examples, and inputting the generated prompt words into the large language model, further includes:

[0018] Construct an atomic motion primitive library for the robot; the atomic motion primitive library encapsulates the robot's low-level hardware control interface into a standardized function interface that can be called by a large language model, the standardized function interface including navigation primitives, operation primitives and perception primitives.

[0019] Optionally, the thought chain example includes natural language instructions, the thought chain derivation process, and the corresponding correct code; the correct code is code containing standardized functions.

[0020] The thought chain derivation process includes, in sequence, a perception step, a reasoning step, and an action step;

[0021] The perception step requires the large language model to first call perception primitives to confirm environmental conditions;

[0022] The reasoning step requires the large language model to analyze whether the current conditions meet the task prerequisites.

[0023] The action steps require the model to call navigation or operation primitives based on the reasoning results.

[0024] Optionally, the step of using a large language model to decompose the task instructions into a sequence of subtasks, and organizing the decomposed subtask sequences into complete executable code blocks, includes:

[0025] Using a large language model based on hierarchical task network logic, the task instructions are recursively decomposed into a sequence of subtasks, and the decomposed subtask sequences are organized into complete executable code blocks.

[0026] Optionally, the generation logic of the hierarchical task network specifically includes:

[0027] The large language model first identifies the core verbs and target nouns in the task instructions, and then determines the task as a compound task or an atomic task.

[0028] For complex tasks, the large language model decomposes the complex task into several ordered sub-tasks by retrieving its internal knowledge base.

[0029] For each subtask, repeat the subtask decomposition process until all subtasks can be directly mapped to a single function call in the atomic action primitive library;

[0030] During code generation, the decomposed subtask sequence is organized into complete logical code blocks containing conditional judgments, loop traversal, and exception handling.

[0031] Optionally, the abstract syntax tree-based parser performs static logic verification on the code block before code execution; if the verification passes, the execution phase begins; if the verification fails, an error description is generated and fed back to the large language model for code correction, including:

[0032] Convert the code block into an abstract syntax tree structure;

[0033] Traverse all Call nodes in the syntax tree and extract the identifier of the called function;

[0034] The extracted function identifiers are compared with the whitelist of the atomic action primitive library. If an undefined function call is found, it is determined to be a phantom error, execution is terminated and an illegal function call exception is thrown. The parameter passing relationship in the syntax tree is analyzed to check whether the number and type of parameters during function calls match the definitions in the atomic action primitive library.

[0035] Perform logical dependency checks; the logical dependency checks include: traversing the code path to detect whether there are cases where variables are not assigned values ​​before use or necessary pre-checks are not performed before execution of operations;

[0036] If it is detected that the code logic path lacks a call to the awareness primitive function or fails to check the return value of the awareness primitive function for null before calling the operation primitive function, it is determined to be a logic violation, the code execution is terminated, and feedback information indicating missing preconditions is generated and sent to the large language model.

[0037] Optionally, during the code execution phase, the code is executed line by line. If an exception is returned after code execution, the exception information is sent back to the large language model, so that the large language model can regenerate corrected code based on the cause of the exception. This includes:

[0038] The current task is immediately suspended when a primitive function returns False or the system catches a runtime exception thrown by the underlying hardware.

[0039] Capture the current scene snapshot; the scene snapshot includes: the name of the function that reported the error, the error code, the current position coordinates of the robot, the joint angles of the robotic arm, and the latest visual perception image description;

[0040] The on-site snapshot is encapsulated into a failure report in natural language format, and the failure report is appended to the historical dialogue context with the large language model;

[0041] Send a replanning instruction to the large language model, requiring the model to analyze the cause of failure based on the failure report, and generate local correction code to recover from the current state and continue to complete the task without resetting the entire task.

[0042] According to another aspect of the present invention, an instruction execution system with embodied intelligence is provided, comprising:

[0043] The instruction acquisition module is used to acquire the robot's current environmental status information and the natural language task instructions input by the user;

[0044] The prompt word generation module is used to generate prompt words based at least on the environmental state information, the task instructions, the predefined standardized function interface document, and the mind chain example, and input the generated prompt words into the large language model;

[0045] The task decomposition module is used to decompose the task instructions into a sequence of subtasks using a large language model, and to organize the decomposed subtask sequences into complete executable code blocks.

[0046] The verification module is used to perform static logic verification on the code block before code execution, based on the abstract syntax tree parser. If the verification passes, the execution phase begins; if the verification fails, an error description is generated and fed back to the large language model for code correction.

[0047] The replanning module is used to execute code line by line during the code execution phase. If an exception is returned after code execution, the exception information is sent back to the large language model so that the large language model can regenerate corrected code based on the reason for the exception.

[0048] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0049] At least one processor; and

[0050] A memory communicatively connected to the at least one processor; wherein,

[0051] The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to execute the embodied intelligence instruction execution method according to any embodiment of the present invention.

[0052] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions, the computer instructions being configured to cause a processor to execute, when executed, implement the instruction execution method of embodied intelligence as described in any embodiment of the present invention.

[0053] The technical solution of this invention obtains the robot's current environmental state information and the natural language task instructions input by the user; generates prompt words based on the environmental state information, task instructions, predefined standardized function interface documents, and thought chain examples, and inputs the generated prompt words into a large language model, enabling the large model to generate corresponding task analysis according to preset thought logic, and convert the analyzed task sequence into code for driving the robot; in addition, the large language model is used to decompose the task instructions into sub-task sequences, and the decomposed sub-task sequences are organized into complete executable code blocks, thereby decomposing abstract natural language instructions into logically rigorous long... The code sequence involves a series of actions. Before execution, static logic checks are performed on the code blocks. If a check fails, an error description is generated and fed back to the large language model for code correction. During execution, the code is executed line by line. If an exception is returned after execution, it is sent back to the large language model, which then regenerates corrected code based on the cause of the exception. This dual closed-loop system of generation, verification, execution, and feedback effectively mitigates the risk of generating illusory code and dangerous actions. It also endows the robot with the resilience to autonomously correct its strategies when encountering execution failures in dynamic environments, significantly improving the autonomous decision-making level and task execution success rate of the four-wheeled orthotic intelligent robot.

[0054] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0055] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0056] Figure 1 This is a flowchart of an instruction execution method for embodied intelligence provided in Embodiment 1 of the present invention;

[0057] Figure 2 This is a flowchart of an instruction execution method for embodied intelligence provided according to Embodiment 2 of the present invention;

[0058] Figure 3 This is an architecture diagram of an instruction execution system with embodied intelligence provided according to Embodiment 3 of the present invention;

[0059] Figure 4 This is a schematic diagram of the structure of an electronic device that implements the instruction execution method of embodied intelligence according to embodiments of the present invention. Detailed Implementation

[0060] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0061] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0062] Example 1

[0063] Figure 1 This is a flowchart illustrating an instruction execution method for embodied intelligence, as provided in Embodiment 1 of the present invention. Figure 1 As shown, the method includes:

[0064] S101. Obtain the robot's current environmental status information and the natural language task instructions input by the user.

[0065] The robot's current environmental state information refers to the semantic and positional information of objects around the robot at its current location, including the semantic labels and positional information of each object in the current scene. User-inputted natural language commands are task instructions in natural language, such as "go get a bottle of water".

[0066] S102. Generate prompt words based at least on the environmental state information, the task instructions, the predefined standardized function interface document, and the thought chain example, and input the generated prompt words into the large language model.

[0067] Among them, environmental state information is used to enable the large language model to understand the current scene information and the current state of the robot; task instructions are used to enable the large language model to understand the task that needs to be processed; thought chain examples are used to enable the large language model to construct the execution steps corresponding to the task according to the thought logic in the examples. For example, first, let the robot perceive the surrounding environment, then reason whether the current environmental information meets the preconditions of the task, and finally execute the task; predefined standardized function interface documents are used to enable the large language model to convert the task execution actions into code that can drive the robot to run.

[0068] In this embodiment, environmental state information, the task instructions, predefined standardized function interface documents, and thought chain examples can be concatenated into the final prompt words and input into the large language model. After receiving the prompt words, the large language model can generate code to drive the robot to run.

[0069] S103. The task instructions are decomposed into a sequence of subtasks using a large language model, and the decomposed subtask sequences are organized into a complete executable code block.

[0070] Specifically, the large language model decomposes task instructions into several subtasks. For example, the task instruction "fetch a water cup" can be broken down into three subtask steps: "navigate to the table," "find the water cup," and "perform the grasping action." The large language model can then convert each subtask into code to drive the robot, and organize the code corresponding to each subtask into a complete executable code block to drive the robot to complete a long sequence of actions.

[0071] S104. Based on the abstract syntax tree parser, static logic verification is performed on the code block before code execution; if the verification passes, the execution phase begins; if the verification fails, an error description is generated and fed back to the large language model for code correction.

[0072] The abstract syntax tree parser is used to convert code blocks into an abstract syntax tree. In this embodiment, the abstract syntax tree parser can convert the code into an abstract syntax tree before code execution. Static logic verification of the code block is performed by traversing the abstract syntax tree. If the verification passes, the code execution phase proceeds; if the verification fails, an error description is generated and fed back to the large language model for code correction.

[0073] S105. During the code execution phase, the code is executed line by line. If an exception is returned after the code execution, the exception is sent back to the large language model so that the large language model can regenerate the corrected code based on the cause of the exception.

[0074] It should be noted that for standardized function interfaces, each primitive function returns corresponding information after execution, and the underlying driver can also throw hardware exceptions during execution. That is, exception information can include function return information, information about the function that encountered the error, and hardware exception information. The large model can analyze the exception information returned after code execution to determine the cause of the exception and then correct the code accordingly.

[0075] Embodiments of this invention acquire the robot's current environmental state information and the natural language task instructions input by the user; based on the environmental state information, task instructions, predefined standardized function interface documents, and thought chain examples, prompt words are generated and input into a large language model. This allows the large model to generate corresponding task analysis according to preset thought logic and convert the resulting task sequence into code to drive the robot. Furthermore, the large language model decomposes the task instructions into sub-task sequences and organizes these sub-task sequences into complete executable code blocks, thereby breaking down abstract natural language instructions into logically rigorous long sequences. The code block undergoes static logic verification before execution. If the verification fails, an error description is generated and fed back to the large language model for code correction. During code execution, the code is executed line by line. If an exception is returned after execution, the exception information is sent back to the large language model, which then regenerates the corrected code based on the cause of the exception. This dual closed-loop system of generation, verification, execution, and feedback effectively suppresses the risk of generating illusory code and dangerous actions. It also gives the robot the resilience to autonomously correct its strategies when encountering execution failures in dynamic environments, significantly improving the autonomous decision-making level and task execution success rate of the four-wheeled orthotic intelligent robot.

[0076] Example 2

[0077] Figure 2 This is a flowchart illustrating an embodied intelligence instruction execution method provided in Embodiment 2 of the present invention. Figure 2 As shown, the method includes:

[0078] S201. Construct an atomic motion primitive library for the robot; the atomic motion primitive library encapsulates the robot's low-level hardware control interface into a standardized function interface that can be called by a large language model, the standardized function interface including navigation primitives, operation primitives and perception primitives.

[0079] The robot's atomic motion primitive library includes functions for driving the robot's operation; by writing the encapsulation code of the low-level driver, the hardware capabilities of the four-wheeled robot can be abstracted into a set of standard Python function interfaces, thus forming a standardized function interface.

[0080] In this embodiment, the standardized function interface may include navigation primitives, operation primitives, and perception primitives.

[0081] The navigation primitives encapsulate the chassis motion controller and SLAM (Simultaneous Localization and Mapping) algorithm, defining functions such as `Navigation(target_coords)`. The `Navigation` function receives 3D coordinate parameters and internally implements path planning and obstacle avoidance logic. Specifically, it can receive a string variable `target_semantic_name` (e.g., "kitchen", the target semantic name). Internally, the function first reads pre-built semantic map data and converts the input semantic name into a 3D coordinate point `goal_pose` in the global coordinate system through a lookup table. Then, the program instantiates a ROS2 `ActionClient` (in visual grasping scenarios, it is the key bridge connecting the "visual algorithm result (target semantics + 3D coordinates)" and the "robot execution module (navigation / grasping actuator)"), sending the 3D coordinate point `goal_pose` to the chassis navigation server. The navigation server then controls the mobile chassis to move to the target based on the 3D coordinate point. The function has a timeout threshold timeout_sec (which can be set to 60 seconds). If the chassis reports arrival status within the threshold, the function returns a boolean value True; if the timeout occurs or the path planner returns an unreachable status, the function returns a boolean value False.

[0082] Perception primitives are used to encapsulate visual detection algorithms. One such function is `Find(object_description)`, which receives a string variable `object_description` describing the target object (e.g., "redwater bottle"). The function calls a camera to acquire semantic information of the current scene or coordinate data of a specific object. Specifically, the function internally calls the `realsense_camera` driver to acquire the RGB image data `img_rgb` and depth image data `img_depth` of the current frame. The program calls an open-vocabulary object detection model (such as YOLO-World), taking `img_rgb` and `object_description` as input. If the confidence score of the detection model is greater than a set threshold (e.g., 0.6), the program converts the pixel coordinates of the detection box center to 3D coordinates in the camera coordinate system using the intrinsic parameter matrix of the depth camera. Combined with the robot's current pose transformation matrix, it calculates the absolute coordinates `target_coords` of the target object in the world coordinate system. The function ultimately returns a list of coordinates containing (x, y, z); if no target is detected, it returns a null value `None`.

[0083] The operation primitives are used to encapsulate the chassis motion controller and SLAM (Simultaneous Localization and Mapping) algorithm. They can be defined as the function `Grasp(target_coords)`. The `Grasp` function receives a list of the target's 3D coordinates as input and is used for force and position control logic of the robotic arm's end effector. Specifically, the `Grasp` function internally calls the inverse kinematics solver to calculate the target list of joint angles for the six-axis robotic arm, `joint_angles`. If the solution is successful, the program sends position control commands to the motor driver via the CAN bus. After the action is completed, the program reads the force sensor value `force_val` at the gripper's end. If `force_val` is greater than a preset gripping threshold (indicating that an object has been gripped), the function returns `True`; otherwise, it returns `False`.

[0084] S202. Obtain the robot's current environmental status information and the natural language task instructions input by the user.

[0085] The robot's current environmental state information includes the semantic and positional information of surrounding objects at the robot's current location, as well as information such as the robot's current position, battery level, and robotic arm posture. Task instructions are used to help the large language model understand the task that needs to be processed.

[0086] In one embodiment, the system can perform spatiotemporal registration of point cloud data collected by the LiDAR on the robot and visual image data collected by the camera; use the Open-Vocabulary Object Detection algorithm to process the visual images and extract semantic labels and location information of objects in the scene; and map the extracted semantic labels to the coordinate system of the point cloud map to construct a semantic 3D environment description text as the environment input context of the large language model.

[0087] S203. Generate prompt words based at least on the environmental state information, the task instructions, the predefined standardized function interface document, and the thought chain example, and input the generated prompt words into the large language model.

[0088] In this embodiment, prompt words can be generated based on environmental status information, the task instructions, predefined standardized function interface documents, and mind chain examples.

[0089] Specifically, the program initializes a `PromptManager` class to manage the interaction context with the large language model. First, it constructs system prompts, which declare the signatures of all defined primitive functions, their parameter descriptions, and usage limitations, injecting them into the context in the form of API documentation. Then, task instructions, environmental state information (including the current robot position, battery level, and robotic arm posture), and a thought chain example are concatenated after the system prompts to generate the final prompts.

[0090] In one embodiment, the thought chain derivation process sequentially includes a perception step, a reasoning step, and an action step; the perception step requires the large language model to first call perception primitives to confirm environmental conditions; the reasoning step requires the large language model to analyze whether the current conditions meet the task prerequisites; the action step requires the model to call navigation or operation primitives based on the reasoning results.

[0091] The thought chain example provides several sets of "instruction-code" comparison samples. The samples enforce the tripartite logical structure of perception-reasoning-action, which requires the model to generate comments before generating code, describing the observation of the environment (Perception) (perception step) and the logical judgment of the task premise (Reasoning step), and finally generate the specific function call (Action) (action step).

[0092] It should be noted that in this embodiment, the prompt word also needs to specify that the code output by the model must be wrapped in a specific Markdown tag (such as ```python) so that it can be extracted by regular expressions later.

[0093] S204. Using a large language model based on hierarchical task network logic, the task instructions are recursively decomposed into a sequence of subtasks, and the decomposed subtask sequences are organized into complete executable code blocks.

[0094] In one embodiment, the generation logic of the hierarchical task network specifically includes: the large language model first identifies the core verbs and target nouns in the task instructions, and determines the task as a composite task or an atomic task; for composite tasks, the model decomposes them into several ordered subtasks by searching its internal knowledge base; for each subtask, the decomposition process is repeated until all subtasks can be directly mapped to a single function call in the atomic action primitive library; during code generation, the decomposed subtask sequence is organized into a complete logical code block containing condition judgment, loop traversal, and exception handling.

[0095] Specifically, the large language model recursively decomposes the input task instructions into several sub-task sequences based on the task instructions and the robot's current environmental state information. For example, for the task instruction "throw away the empty bottle," it can be decomposed into multiple ordered sub-task sequences based on the robot's current position and the target object's position: "navigate to the kitchen" -> "find the table" -> "scan the bottle" -> "grab the bottle" -> "navigate to the trash can" -> "place the bottle." Each sub-task is then further refined until it can be directly mapped to a single function call in the atomic action primitive library. Finally, the large language model generates a complete Python script code containing control flow (such as if conditional statements and for loops).

[0096] S205. Based on the abstract syntax tree parser, static logic verification is performed on the code block before code execution; if the verification passes, the execution phase begins; if the verification fails, an error description is generated and fed back to the large language model for code correction.

[0097] In one embodiment, the specific steps include: converting the code block into an abstract syntax tree structure;

[0098] Traverse all Call nodes in the syntax tree and extract the identifier of the called function;

[0099] The extracted function identifiers are compared with the whitelist of the atomic action primitive library. If an undefined function call is found, it is determined to be a phantom error, execution is terminated and an illegal function call exception is thrown. The parameter passing relationship in the syntax tree is analyzed to check whether the number and type of parameters during function calls match the definitions in the atomic action primitive library.

[0100] Perform logical dependency checks; the logical dependency checks include: traversing the code path to detect whether there are cases where variables are not assigned values ​​before use or necessary pre-checks are not performed before execution of operations;

[0101] If it is detected that the code logic path lacks a call to the awareness primitive function or fails to check the return value of the awareness primitive function for null before calling the operation primitive function, it is determined to be a logic violation, the code execution is terminated, and feedback information indicating missing preconditions is generated and sent to the large language model.

[0102] Specifically, a code validator can be used to perform security checks on the generated code string before code execution. First, the Python built-in library `ast.parse(generated_code_str)` can be called to convert the code string into an abstract syntax tree (AST) object. Then, static validation can be performed on the AST, which may include intercepting illegal functions and checking logical dependencies.

[0103] The process of intercepting illegal functions includes: traversing all function call nodes in the abstract syntax tree, checking whether the function name of each node is included in the whitelist of the robot primitive library (such as navigation primitives, operation primitives and perception primitives), and if an undefined function name is found (such as Jump() generated by model illusion), a custom exception SecurityError is immediately thrown.

[0104] The logical dependency check process includes: analyzing the code's control flow; when traversing to a Grasp operation function call node, the program checks whether the Find-aware function call node has already been traversed upstream of the current code path, and whether the parameter variables of the Grasp operation function originate from the return value of the Find-aware function. If a direct call to a Grasp operation function is detected without a preceding Find-aware function, or if a string constant is directly passed to the Grasp operation function, the validator returns a validation failure flag `validation_success = False` and the error message `error_msg`.

[0105] S206. During the code execution phase, the code is executed line by line. If an exception is returned after the code execution, the exception is sent back to the large language model so that the large language model can regenerate the corrected code based on the cause of the exception.

[0106] It should be noted that for standardized function interfaces, each primitive function returns corresponding information after execution, and the underlying driver can also throw hardware exceptions during execution. That is, exception information can include function return information, information about the function that encountered the error, and hardware exception information. The large model can analyze the exception information returned after code execution to determine the cause of the exception and then correct the code accordingly.

[0107] In one embodiment, it specifically includes:

[0108] The current task is immediately suspended when a primitive function returns False or the system catches a runtime exception thrown by the underlying hardware.

[0109] Capture the current scene snapshot, which includes: the name of the function that reported the error, the error code, the current position coordinates of the robot, the joint angles of the robotic arm, and the latest visual perception image description;

[0110] The on-site snapshot is encapsulated into a failure report in natural language format, and the failure report is appended to the historical dialogue context with the large language model;

[0111] Send a replanning instruction to the large language model, requiring the model to analyze the cause of failure based on the failure report, and generate local correction code to recover from the current state and continue to complete the task without resetting the entire task.

[0112] Specifically, when a code block passes static logic verification and enters the Python interpreter for execution, the system initiates a runtime monitoring process. During execution, if an atomic primitive returns False (e.g., the robotic arm planning has no solution) or throws a hardware exception, the program immediately interrupts the current execution flow. The system automatically captures a snapshot of the current situation, including the name of the function that caused the error; the return value or exception information at the time of the error; the robot's current coordinates; and the latest visual observation results. Subsequently, the system converts the snapshot into a failure report described in natural language and appends the failure report to the historical dialogue context with the large language model, constructing a new prompt, such as "The previous step failed because..., please generate remedial code without resetting the task." The program calls the large language model again to obtain a new code block (e.g., the model may generate code to make the robot "take a step back" and then "search again"), and repeats the verification process of step S205 until the task succeeds or the maximum number of retries is reached.

[0113] Example 3

[0114] Figure 3 This is a schematic diagram of the architecture of an embodied intelligence instruction execution system provided in Embodiment 3 of the present invention. Figure 3As shown, the system includes:

[0115] The instruction acquisition module 301 is used to acquire the robot's current environmental status information and the natural language task instructions input by the user;

[0116] The prompt word generation module 302 is used to generate prompt words based at least on the environmental state information, the task instructions, the predefined standardized function interface document, and the mind chain example, and input the generated prompt words into the large language model;

[0117] The task decomposition module 303 is used to decompose the task instruction into a sequence of subtasks using a large language model, and to organize the decomposed sequence of subtasks into a complete executable code block.

[0118] The verification module 304 is used to perform static logic verification on the code block before code execution based on the abstract syntax tree parser; if the verification passes, the execution phase begins; if the verification fails, an error description is generated and fed back to the large language model for code correction.

[0119] The replanning module 305 is used to execute the code line by line during the code execution phase. If an exception is returned after the code execution, the exception information is sent back to the large language model so that the large language model can regenerate the corrected code based on the reason for the exception.

[0120] The embodied intelligence instruction execution system provided in the embodiments of the present invention can execute the embodied intelligence instruction execution device method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0121] Example 4

[0122] Figure 4 A schematic diagram of an electronic device 10, which can be used to implement embodiments of the present invention, is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0123] like Figure 4As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0124] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0125] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as an embodied intelligence instruction execution method.

[0126] In some embodiments, an embodied intelligence instruction execution method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the embodied intelligence instruction execution method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to execute an embodied intelligence instruction execution method by any other suitable means (e.g., by means of firmware).

[0127] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0128] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0129] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0130] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0131] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0132] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0133] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0134] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for executing instructions with embodied intelligence, characterized in that, include: Obtain the robot's current environmental status information and the natural language task instructions input by the user; Construct a library of atomic action primitives for robots; The atomic action primitive library encapsulates the robot's low-level hardware control interface into a standardized function interface that can be called by a large language model. The standardized function interface includes at least perception primitives, navigation primitives, and operation primitives. Among them, perception primitives are used to return the three-dimensional coordinates of the target, and operation primitives are used to perform operation actions with the three-dimensional coordinates of the target as target parameters. Based at least on the environmental state information, the task instructions, the predefined standardized function interface document, and the mind chain example, prompt words are generated and then input into the large language model; The thought chain example includes natural language instructions, the thought chain derivation process, and the corresponding correct code; the correct code is code containing standardized functions; the thought chain derivation process sequentially includes a perception step, a reasoning step, and an action step; the perception step requires the large language model to first call perception primitives to confirm environmental conditions; the reasoning step requires the large language model to analyze whether the current conditions meet the task prerequisites; the action step requires the model to call navigation or operation primitives based on the reasoning results; The task instructions are decomposed into a sequence of subtasks using a large language model, and the decomposed subtask sequences are organized into complete executable code blocks; each subtask can be directly mapped to a single function call in the atomic action primitive library; Based on the abstract syntax tree parser, static logic verification is performed on the code block before code execution; the static logic verification includes: The code block is converted into an abstract syntax tree structure, and the function call node, assignment node, variable reference node and condition judgment node in the abstract syntax tree are traversed. When an operation class primitive call is detected, it is determined whether a perception class primitive call exists upstream of the code path where the operation class primitive call is located; Determine whether the return value of the perception primitive has been checked for non-emptiness; Based on the assignment node, variable reference node, and function call node, trace the source of the target parameter of the operation class primitive and determine whether the target parameter of the operation class primitive is obtained by the target three-dimensional coordinates returned by the perception class primitive. If there is no call to a perception primitive, no non-empty check is performed on the return value of the perception primitive, or the target parameter of the operation primitive is not obtained by passing the target three-dimensional coordinates returned by the perception primitive, then it is determined that the physical precondition is violated, the code execution is terminated, and an error description is generated and fed back to the large language model for code correction; if it is not determined that the physical precondition is violated, then the static logic check is determined to be passed, and the code execution stage is entered. During the code execution phase, the code is executed line by line. If an exception is returned after the code execution, the exception information is sent back to the large language model, so that the large language model can regenerate and correct the code according to the reason for the exception.

2. The method for executing instructions based on embodied intelligence according to claim 1, characterized in that, The perception primitives include the Find function, which receives a description of the target object, determines the target's three-dimensional coordinates in the world coordinate system based on the current visual image and depth information, and returns a null value if no target is detected. The operation primitives include the Grasp function, which performs a robotic arm grasping action with the target's three-dimensional coordinates as input parameters.

3. The method for executing instructions based on embodied intelligence according to claim 2, characterized in that, If the target parameter of the operation primitive is a string constant, a string constant corresponding to the target noun in the user's natural language task instruction, an unassigned variable, or a variable not obtained by passing the return value of the perception primitive through an assignment statement, variable reference, or function parameter, then it is determined that the target parameter of the operation primitive is not obtained by passing the target three-dimensional coordinates returned by the perception primitive.

4. The method for executing instructions based on embodied intelligence according to claim 1, characterized in that, During the code execution phase, the code is executed line by line. If an exception is returned after code execution, the exception information is sent back to the large language model, which then regenerates corrected code based on the cause of the exception. This includes: During code execution, when a primitive function returns False or the system catches a runtime exception thrown by the underlying hardware, the current task is immediately suspended. Capture the current scene snapshot; the scene snapshot includes: the name of the function that reported the error, the error code, the current position coordinates of the robot, the joint angles of the robotic arm, and the latest visual perception image description; The on-site snapshot is encapsulated into a failure report in natural language format, and the failure report is appended to the historical dialogue context with the large language model; A replanning instruction is sent to the large language model, requiring the model to analyze the cause of failure based on the failure report, and generate local correction code to recover from the current state and continue to complete the task without resetting the entire task. The local correction code re-enters the static logic verification process described in claim 1 before execution.

5. An embodied intelligence instruction execution system, characterized in that, include: The instruction acquisition module is used to acquire the robot's current environmental status information and the natural language task instructions input by the user; The primitive library construction module is used to construct the atomic action primitive library of the robot. The atomic action primitive library encapsulates the robot's low-level hardware control interface into a standardized function interface that can be called by a large language model. The standardized function interface includes at least perception primitives, navigation primitives, and operation primitives. Among them, the perception primitives are used to return the three-dimensional coordinates of the target, and the operation primitives are used to perform operation actions with the three-dimensional coordinates of the target as target parameters. The prompt word generation module is used to generate prompt words based at least on the environmental state information, the task instructions, the predefined standardized function interface document, and the thought chain example, and input the generated prompt words into the large language model. The thought chain example includes natural language instructions, the thought chain derivation process, and the corresponding correct code. The correct code is code containing standardized functions. The thought chain derivation process includes a perception step, a reasoning step, and an action step in sequence. The perception step requires the large language model to first call a perception primitive to confirm the environmental conditions. The reasoning step requires the large language model to analyze whether the current conditions meet the task prerequisites. The action step requires the model to call a navigation primitive or an operation primitive based on the reasoning result. The task decomposition module is used to decompose the task instructions into a sequence of subtasks using a large language model, and organize the decomposed subtask sequences into complete executable code blocks; each subtask can be directly mapped to a single function call in the atomic action primitive library; The verification module is used to perform static logic verification on the code block before code execution based on the abstract syntax tree parser. If the verification passes, the execution phase begins; if the verification fails, an error description is generated and fed back to the large language model for code correction. Specifically, the verification module is used to convert the code block into an abstract syntax tree structure and traverse the function call nodes, assignment nodes, variable reference nodes, and condition judgment nodes in the abstract syntax tree. When an operation class primitive call is detected, it is determined whether a perception class primitive call exists upstream of the code path where the operation class primitive call is located; Determine whether the return value of the perception primitive has been checked for non-emptiness; Based on the assignment node, variable reference node, and function call node, trace the source of the target parameter of the operation class primitive and determine whether the target parameter of the operation class primitive is obtained by the target three-dimensional coordinates returned by the perception class primitive. If there is no call to a perception primitive, no non-empty check is performed on the return value of the perception primitive, or the target parameter of the operation primitive is not obtained by passing the target three-dimensional coordinates returned by the perception primitive, then it is determined that the physical precondition is violated, the code execution is terminated, and an error description is generated and fed back to the large language model for code correction. An execution module is used to execute the executable code block when the verification module does not determine that the physical preconditions are violated. The replanning module is used to execute the code line by line during the code execution phase. If an exception is returned after the code execution, the exception information is sent back to the large language model, so that the large language model can regenerate the local correction code according to the reason for the exception. The local correction code is then sent to the verification module so that it can perform static logic verification again before execution.

6. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the embodied intelligence instruction execution method according to any one of claims 1-4.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that, when executed by a processor, implement the instruction execution method of embodied intelligence as described in any one of claims 1-4.

Images