Interactive imitation learning system and method for embodied robots based on large language models

Abstract

The application discloses a somatotopic robot interactive imitation learning system and method based on a large language model, introduces the large language model into robot learning and training, guides the large language model to automatically generate executable code strategies in stages through a standardized prompt template, uses the executable code strategies as a virtual training benchmark for imitation learning, and only needs to input a natural language task description to complete the construction of the task benchmark, without manual demonstration and underlying code writing throughout the process, so that the task deployment period is greatly shortened, a neural network intelligent agent is trained by taking the code strategy as a supervision source, and finally, the output strategy has stronger environmental adaptability compared with pure code execution, and low deployment cost and execution robustness are considered.

Description

Technical Field

[0001] This invention relates to the field of technology, and more specifically, to an embodied robot interactive imitation learning system and method based on a large language model. Background Technology

[0002] In long-sequence tasks involving embodied robots, such as industrial sorting and assembly, imitation learning is the mainstream technical path for robots to acquire task execution capabilities. Traditional robot imitation learning technology relies on professionals to remotely collect teaching trajectories and then fit robot control strategies through behavior cloning. For long-sequence complex tasks containing multiple consecutive sub-steps, manual teaching is labor-intensive and costly, and the debugging and deployment cycle for a single task is long, making it difficult to adapt to the rapid changeover requirements of multi-category, small-batch production models. At the training feedback level, this type of technology often directly compares the differences in action values ​​to determine the quality, without binding action judgments to the phased goals of task execution. It cannot identify the semantic differences of the same robot state at different task stages, which easily leads to semantic misalignment in action evaluation, resulting in deviations in the training direction. The cumulative effect of errors in long-sequence tasks is particularly prominent, and the success rate of the ultimately trained strategy is difficult to guarantee. At the sample utilization level, this type of technology generally adopts a uniform sampling training method without distinguishing the training value of samples. High-value samples with incorrect decisions cannot be prioritized for learning, resulting in slow overall training convergence speed and low efficiency in the utilization of sample resources.

[0003] In recent years, robot task solutions combining large language models have gradually emerged. These solutions leverage the semantic understanding and logic generation capabilities of large language models to convert natural language task descriptions into robot control logic, thus lowering the threshold for task definition to some extent. However, most existing solutions directly use the control logic generated by the large language model as the final execution strategy. Therefore, in long-sequence tasks, a deviation in a single step can easily lead to the failure of subsequent steps, resulting in poor fault tolerance in task execution. At the same time, the control logic generated by these solutions is mostly based on fixed rules, lacking the ability to adapt to non-ideal working conditions such as workpiece pose deviation, changes in ambient lighting, and material dimensional tolerances in real-world scenarios, making it difficult to guarantee task stability during actual implementation. Summary of the Invention

[0004] To overcome the shortcomings of existing robot task design based on large language models, this invention provides an embodied robot interactive imitation learning system and method based on large language models.

[0005] The technical solution of this invention is as follows:

[0006] An embodied robot interactive imitation learning system based on a large language model, including

[0007] The large language model teacher module has built-in standardized prompt templates, including planner prompt templates, action function prompt templates, and check function prompt templates. It receives the natural language task description input by the user, breaks down the complete task into an ordered sequence of steps that integrate multiple sub-tasks, generates the corresponding action function and check function for each sub-step, and outputs the executable code corresponding to the training task.

[0008] The host computer includes a training scheduling unit for training process control and scheduling, a code execution unit for loading executable code and generating reference actions and stage flags, a stage state management unit for maintaining task stages and stage switching, a deviation calculation unit for calculating two types of deviations: action direction and geometry, a feedback unit for deviation judgment and feedback instruction output, a dynamic weighting unit for recording training samples and dynamic weight allocation, a computing storage unit for storage, and a deployment control unit for exporting the agent module strategy network and deploying the robot end.

[0009] The robot is equipped with a perception and acquisition unit and an intelligent agent module. The perception and acquisition unit collects and transmits current environmental images and robot body state data, while the intelligent agent module outputs the robot's execution and strategy parameters for iterative optimization based on the data from the perception and acquisition unit.

[0010] The interactive imitation learning method for embodied robots based on a large language model, as described above, is as follows:

[0011] Before training begins, the user inputs a natural language task description into the large language model teacher module, guiding the module to codify the training task.

[0012] The user calls the planner prompt template to guide the large language model to break down the complete task into an ordered sequence of steps that integrate multiple sub-tasks. Then, for each sub-step, the user calls two types of prompt templates: action function and check function, to guide the large language model to generate the corresponding action function and check function. This allows the teacher module of the large language model to output the executable code corresponding to the entire training task, i.e., the code strategy.

[0013] The training scheduling unit of the host computer calls the code strategy to execute several rounds of training tasks in the simulation environment or real machine environment, collects the state and action corresponding to all successful trajectories, and uses the state and action to form a pre-training dataset. Then, the pre-training dataset is input into the intelligent agent module, and the neural network parameters are initially fitted through supervised learning, so that the intelligent agent module has basic task execution capabilities.

[0014] During training, the robot executes the following within each control cycle:

[0015] The perception and acquisition unit collects images of the current environment and the robot's own state, and synchronously sends unified state data to the host computer and the intelligent agent module;

[0016] The robot's intelligent agent module receives state data, extracts features, and performs temporal reasoning to output the predicted action.

[0017] The host computer's code execution unit receives status data, combines it with the current stage identifier output by the host computer's stage status management unit, takes the received status data and the current stage identifier as input, calculates and outputs the corresponding reference action according to the code strategy, and calls the check function to output the current stage completion flag.

[0018] The host computer's stage status management unit verifies the current stage completion flag. If it determines that the current subtask has been completed, it triggers a stage switch, updates the current stage flag, and then triggers the code execution unit to calculate the reference action of the subtask corresponding to the new stage after the switch. If it determines that the subtask has not been completed, it sends a signal to trigger the host computer's deviation calculation unit to calculate the action direction deviation and geometric deviation.

[0019] The feedback unit of the host computer makes a judgment based on the preset threshold ranges of two items: motion direction deviation and geometric deviation. If both deviations are lower than the lower limit of the threshold range, it outputs evaluative feedback and controls the robot to execute the predicted action of the intelligent agent module. If either deviation exceeds the upper limit of the threshold range, it controls the robot to execute the reference action.

[0020] The host computer's dynamic weighting unit records the robot's state, actual actions, current stage identifier, and feedback type for each step. After each control cycle ends, it calculates the total number of steps and corrections for that control cycle, assigns dynamic high weights to the correction samples according to the weighting formula, and stores all the data in the structured experience pool of the computing storage unit.

[0021] The training scheduling unit of the host computer triggers parameter updates. The agent module reads samples from the weighted structured experience pool and updates the neural network parameters within the agent module using the stochastic gradient descent algorithm with the goal of minimizing the negative log-likelihood loss.

[0022] When the number of training control cycles reaches the preset value, or the success rate of the task reaches the target for multiple consecutive control cycles, the deployment control unit only exports the policy network within the trained agent module and deploys the policy network to the robot controller.

[0023] The current stage identifier is a sequence-type state variable used to mark which sub-step the robot is currently in within the entire long sequence task; the stage completion flag is a Boolean judgment result output by the check function corresponding to the current stage, used to determine whether the current sub-step has met the completion conditions.

[0024] The aforementioned embodied robot interactive imitation learning method based on a large language model models the training task as a Markov decision process with stage states, denoted as a tuple. Where S is the state space containing robot joint angles, end effector pose, gripper state, and environmental image features; A is the action space composed of robot motion control commands; T is the state transition function; and R is the implicit reward. Discount factor, where K is the set of stages obtained from task decomposition;

[0025] Explicitly introduce the current task stage identifier in the state. This ensures that all subsequent action judgments are bound to the current subtask context, avoiding pure action comparisons that are detached from task semantics;

[0026] During operation, the stage status management unit maintains the task stage sequence and outputs the current stage identifier;

[0027] The code execution unit loads executable code as a code strategy, takes the synchronized state data and current stage identifier from the sensing and acquisition unit as input, calculates and outputs the corresponding reference action, which satisfies the formula: ,in, The mapping logic corresponding to the code strategy, The state data at time t, This serves as an identifier for the current task phase.

[0028] At the same time, the code execution unit calls the check function corresponding to the current subtask and outputs the stage completion flag: , ;

[0029] The stage status management unit verifies the current stage completion flag: if it determines that the current subtask has been completed, it triggers a stage switch and updates the stage flag. It triggers the code execution unit to calculate the reference action of the subtask corresponding to the new stage after the switch; if it is determined that the subtask is not completed, the current stage identifier is kept unchanged, and a signal is sent to trigger the deviation calculation unit of the host computer to calculate the action direction deviation and geometric deviation.

[0030] The aforementioned embodied robot interactive imitation learning method based on a large language model involves a deviation calculation unit receiving the predicted and reference actions from the agent module, and calculating two types of quantification results: action direction deviation and task-related geometric deviation.

[0031] (1) Action direction deviation: Calculate the vector angle between the predicted action and the reference action of the agent module to measure the consistency of the action direction. The formula is: ,in, To prevent extremely small positive numbers with a denominator of zero, Predict actions for the agent module. Reference actions generated for code strategies;

[0032] (2) Task-related geometric deviation: Extract the physical geometric quantities that are strongly related to the current subtask, calculate the Euclidean distance between the corresponding geometric quantities of the two types of actions, and measure the actual physical effect of the actions. The formula is: ,in, A mapping function for extracting geometric quantities related to the current subtask from the state and action, including the relative pose of the end effector to the target, the gripper opening and closing state, and the distance to the target region;

[0033] The feedback unit makes a joint judgment based on preset threshold ranges for both motion direction deviation and geometric deviation.

[0034] If both deviations are below the lower limit of the corresponding threshold range, an evaluative feedback is output to control the robot to execute the predicted action of the agent module;

[0035] If any deviation exceeds the upper limit of the corresponding threshold range, corrective feedback is output to control the robot to perform the reference action.

[0036] The aforementioned embodied robot interactive imitation learning method based on a large language model,

[0037] The dynamic weighting unit records the robot's state, actual actions performed, current stage identifier, and feedback type for each step.

[0038] After each training session, the total number of time steps and the number of corrective feedback events for that session are counted. Dynamic high weights are assigned to the corrective samples according to the weighting formula, and all weighted sample data are stored in the structured experience pool of the computing storage unit.

[0039] Sample dynamic weights The calculation rule is as follows: for the sample corresponding to the evaluative feedback, let ;

[0040] For the sample corresponding to corrective feedback, let Where N is the total number of time steps in the current round. This represents the number of corrective feedback responses that occurred in that round.

[0041] The agent module employs a parameterized Gaussian random strategy. The module reads samples from the weighted structured experience pool and updates the neural network parameters using the stochastic gradient descent algorithm with the goal of minimizing the negative log-likelihood loss.

[0042] The loss function is: ,in, Let t be the action actually performed by the robot. These are the network parameters for the agent module's strategy.

[0043] The aforementioned embodied robot interactive imitation learning method based on a large language model,

[0044] Before training begins, the teacher module of the large language model is scheduled to generate and output the code strategy;

[0045] During the warm start phase, the code strategy is invoked to execute several rounds of training tasks in the simulation environment or real machine environment, collect the state and action corresponding to all successful trajectories, form a pre-training dataset from the state and action, and then input the pre-training dataset into the agent module. Through supervised learning, the preliminary fitting of the neural network parameters is completed, so that the agent module has basic task execution capabilities.

[0046] During formal training, each unit is scheduled to synchronously complete state acquisition, action generation, deviation calculation, feedback output and sample recording according to the control cycle, and the parameter update operation of the agent module is triggered in rounds.

[0047] When the number of training rounds reaches a preset value, or the success rate of the task reaches the target for multiple consecutive rounds, the deployment control unit is called to export only the policy network within the trained agent module and deploy the policy network to the robot controller, realizing unmanned closed-loop training from task initialization to final deployment.

[0048] The aforementioned embodied robot interactive imitation learning method based on a large language model constructs a code strategy that adopts a structure of one-to-one pairing of ordered subtask steps and functions;

[0049] Each subtask step corresponds to a set of independent action functions and check functions. The action functions are used to receive status data and output the reference action corresponding to the current subtask, and the check functions are used to receive status data and output the stage completion flag of the current subtask.

[0050] The subtasks are linked together in the order of decomposition. After the completion flag of the previous subtask is triggered, the system automatically switches to the next subtask and calls the corresponding function, forming a phased, executable, and verifiable control logic. This decomposes the semantics of high-level tasks into low-level execution rules that can be implemented step by step.

[0051] The aforementioned embodied robot interactive imitation learning method based on a large language model includes an intelligent agent module comprising a three-level architecture: convolutional feature extraction, temporal contextual reasoning, and Gaussian distribution output.

[0052] After receiving the environmental images and robot body state data transmitted by the perception and acquisition unit, the intelligent agent module first extracts the visual features of the environmental images through a convolutional layer, then splices and fuses them with the normalized body state features, and then processes the context information of the sequence decision through a temporal network. Finally, it outputs a parameterized Gaussian action distribution and samples it to obtain the predicted action of the intelligent agent module.

[0053] The agent module policy is modeled as a parameterized stochastic policy. ,in, The mean of the actions predicted by the neural network based on the state. is the covariance parameter.

[0054] The aforementioned embodied robot interactive imitation learning method based on a large language model,

[0055] Before training starts, the training scheduling unit calls the code strategy to automatically execute several rounds of training tasks in the simulation environment or real machine environment, collect the state and action corresponding to all successful trajectories, and form a pre-training dataset from the state and action.

[0056] The training scheduling unit then inputs the pre-trained dataset into the agent module, and completes the initial fitting of the neural network parameters through supervised learning.

[0057] The above-mentioned interactive imitation learning method for embodied robots based on large language models, when the number of training control cycles reaches a preset value, or the success rate of tasks in multiple consecutive control cycles meets the standard, removes the feedback arbitration and process scheduling related logic between the large language model teacher module and the host computer from the deployment control unit, and only exports the policy network in the trained agent module, and deploys the policy network to the robot controller. After deployment, the robot only relies on the perception acquisition unit and the agent module to perceive the environment and output action commands.

[0058] According to the above-described solution, the beneficial effects of this invention are as follows:

[0059] This invention introduces a large language model into robot learning and training. By using standardized prompt templates to guide the large language model to automatically generate staged executable code strategies, these strategies serve as a virtual training benchmark for imitation learning. Users only need to input a natural language task description to complete the construction of the task benchmark. The entire process requires no manual instruction or low-level code writing, significantly shortening the task deployment cycle. At the same time, the neural network agent is trained using the code strategy as a supervision source. The final output strategy has stronger environmental adaptability compared to pure code execution, balancing low deployment cost and execution robustness.

[0060] In long-sequence industrial operations involving multiple steps such as grasping, alignment, assembly, and placement, the correctness of the same robot action varies significantly across different task stages. For example, gripper closure is a correct action in the grasping stage but an incorrect action in the placement stage. Existing imitation learning schemes often rely on direct comparison of global action values ​​without considering the semantics of the task stage, making them prone to feedback misjudgments, leading to training direction deviations and amplifying accumulated errors over long tasks. This invention explicitly introduces a current stage identifier into the state, binding all deviation calculations and feedback judgments to the current sub-task objective. This ensures that the evaluation of each action step matches the operational requirements of the current stage, eliminating training bias caused by stage confusion at its source and improving the stability and final execution accuracy of long-sequence task training.

[0061] Real industrial production lines commonly encounter non-ideal conditions such as workpiece placement deviations, incoming material posture tolerances, and changes in ambient lighting. Traditional behavior cloning methods completely replicate the taught trajectory, exhibiting extremely low tolerance for fluctuations in operating conditions; even slight deviations can lead to task failure. Pure reinforcement learning schemes rely solely on sparse success / failure rewards, resulting in slow training convergence and high sample costs, making it difficult to meet the requirements for rapid deployment on production lines. This invention calculates deviations from two dimensions: action direction and task-related geometric quantities. Combined with threshold intervals, it achieves hierarchical feedback: when the deviation is small, the agent's autonomous decision-making is preserved, allowing the robot to autonomously adapt to fluctuations in operating conditions; when the deviation exceeds the limit, reference actions are triggered for correction, preventing the task from deviating completely from the target. This ensures both the convergence direction of training and endows the policy with sufficient environmental adaptability to adapt to the fluctuating conditions of real production lines.

[0062] In long-sequence task training, robots are typically prone to decision-making errors only in a few critical steps (such as precise alignment and snap-fit ​​assembly), while the remaining steps exhibit high stability. Existing imitation learning methods generally employ uniform sampling, assigning equal weights to high-value erroneous decision samples and ordinary samples. This results in low efficiency in correcting weak points and a long overall training cycle. This invention dynamically assigns high weights to erroneous samples that trigger corrective feedback, guiding the model to focus on optimizing error-prone critical task steps. This improves sample utilization efficiency, shortens the training convergence cycle, and helps production lines complete the debugging and deployment of new tasks more quickly, reducing downtime for debugging.

[0063] Most existing robot solutions powered by large language models still require real-time invocation of the large language model or reliance on a host computer to execute control logic during operation. This results in issues such as network latency, network outage risks, and computational instability, failing to meet the low-latency, high-reliability safety control requirements of industrial environments. This invention employs a completely decoupled training and deployment architecture: during training, the large language model and host computer complete benchmark generation, feedback correction, and full-process scheduling; during deployment, the large language model and all training and control logic are completely removed, with only a lightweight policy network deployed to the robot's local controller. After deployment, the robot can independently execute tasks relying solely on its own sensing unit and edge-side inference, exhibiting low control latency, high operational stability, and direct compatibility with industrial safety regulations and real-time control requirements.

[0064] Traditional end-to-end robot learning solutions often employ random initialization strategies to train from scratch. This results in a large amount of meaningless random exploration in the early stages of training, wasting computing power and time, and increasing the risk of device collisions during real-device training. This invention, before formal interactive training, uses automatically generated code strategies to collect successful trajectories for supervised pre-training of the agent. This enables the agent to possess basic task execution capabilities before formal training, significantly reducing blind exploration in the early stages of training, shortening the overall debugging cycle, and reducing device safety risks during the initial stages of real-device training. This makes it more suitable for real-device training scenarios in industrial settings. Attached Figure Description

[0065] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0066] Figure 1 This is a flowchart of the interactive imitation learning method of the present invention.

[0067] Figure 2 The flowchart shows the feedback and weighting process for a single training control cycle. Detailed Implementation

[0068] To make the technical problems to be solved, the technical solutions, and the beneficial effects of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0069] An embodied robot interactive imitation learning system based on a large language model, including

[0070] The large language model teacher module has built-in standardized prompt templates, including planner prompt templates, action function prompt templates, and check function prompt templates. It receives the natural language task description input by the user, breaks down the complete task into an ordered sequence of steps that integrate multiple sub-tasks, generates the corresponding action function and check function for each sub-step, and outputs the executable code corresponding to the training task.

[0071] The host computer includes a training scheduling unit for training process control and scheduling, a code execution unit for loading executable code and generating reference actions and stage flags, a stage state management unit for maintaining task stages and stage switching, a deviation calculation unit for calculating two types of deviations: action direction and geometry, a feedback unit for deviation judgment and feedback instruction output, a dynamic weighting unit for recording training samples and dynamic weight allocation, a computing storage unit for storage, and a deployment control unit for exporting the agent module strategy network and deploying the robot end.

[0072] The robot is equipped with a perception and acquisition unit and an intelligent agent module. The perception and acquisition unit collects and transmits current environmental images and robot body state data, while the intelligent agent module outputs the robot's execution and strategy parameters for iterative optimization based on the data from the perception and acquisition unit.

[0073] like Figure 1 As shown, the learning method formed by the above-mentioned embodied robot interactive imitation learning system based on a large language model is as follows:

[0074] Before training begins, the user inputs a natural language task description into the large language model teacher module, guiding the module to codify the training task.

[0075] The user calls the planner prompt template to guide the large language model to break down the complete task into an ordered sequence of steps that integrate multiple sub-tasks. Then, for each sub-step, the user calls two types of prompt templates: action function and check function, to guide the large language model to generate the corresponding action function and check function. This allows the teacher module of the large language model to output the executable code corresponding to the entire training task, i.e., the code strategy.

[0076] The training scheduling unit of the host computer calls the code strategy to execute several rounds of training tasks in the simulation environment or real machine environment, collects the state and action corresponding to all successful trajectories, and uses the state and action to form a pre-training dataset. Then, the pre-training dataset is input into the intelligent agent module, and the neural network parameters are initially fitted through supervised learning, so that the intelligent agent module has basic task execution capabilities.

[0077] like Figure 2 As shown, during the training process, the robot executes the following within each control cycle:

[0078] The perception and acquisition unit collects images of the current environment and the robot's own state, and synchronously sends unified state data to the host computer and the intelligent agent module;

[0079] The robot's intelligent agent module receives state data, extracts features, and performs temporal reasoning to output the predicted action.

[0080] The host computer's code execution unit receives status data, combines it with the current stage identifier output by the host computer's stage status management unit, takes the received status data and the current stage identifier as input, calculates and outputs the corresponding reference action according to the code strategy, and calls the check function to output the current stage completion flag.

[0081] The host computer's stage status management unit verifies the current stage completion flag. If it determines that the current subtask has been completed, it triggers a stage switch, updates the current stage flag, and then triggers the code execution unit to calculate the reference action of the subtask corresponding to the new stage after the switch. If it determines that the subtask has not been completed, it sends a signal to trigger the host computer's deviation calculation unit to calculate the action direction deviation and geometric deviation.

[0082] The feedback unit of the host computer makes a judgment based on the preset threshold ranges of two items: motion direction deviation and geometric deviation. If both deviations are lower than the lower limit of the threshold range, it outputs evaluative feedback and controls the robot to execute the predicted action of the intelligent agent module. If either deviation exceeds the upper limit of the threshold range, it controls the robot to execute the reference action.

[0083] The host computer's dynamic weighting unit records the robot's state, actual actions, current stage identifier, and feedback type for each step. After each control cycle ends, it calculates the total number of steps and corrections for that control cycle, assigns dynamic high weights to the correction samples according to the weighting formula, and stores all the data in the structured experience pool of the computing storage unit.

[0084] The training scheduling unit of the host computer triggers parameter updates. The agent module reads samples from the weighted structured experience pool and updates the neural network parameters within the agent module using the stochastic gradient descent algorithm with the goal of minimizing the negative log-likelihood loss.

[0085] When the number of training control cycles reaches the preset value, or the success rate of the task reaches the target for multiple consecutive control cycles, the deployment control unit only exports the policy network within the trained agent module and deploys the policy network to the robot controller.

[0086] The current stage identifier is a sequence-type state variable used to mark which sub-step the robot is currently in within the entire long sequence task; the stage completion flag is a Boolean judgment result output by the check function corresponding to the current stage, used to determine whether the current sub-step has met the completion conditions.

[0087] The training task is modeled as a Markov decision process with phased states, denoted as a tuple. Where S is the state space containing robot joint angles, end effector pose, gripper state, and environmental image features; A is the action space composed of robot motion control commands; T is the state transition function; and R is the implicit reward. Discount factor, where K is the set of stages obtained from task decomposition;

[0088] Explicitly introduce the current task stage identifier in the state. This ensures that all subsequent action judgments are bound to the current subtask context, avoiding pure action comparisons that are detached from task semantics;

[0089] During operation, the stage status management unit maintains the task stage sequence and outputs the current stage identifier;

[0090] The code execution unit loads executable code as a code strategy, takes the synchronized state data and current stage identifier from the sensing and acquisition unit as input, calculates and outputs the corresponding reference action, which satisfies the formula: ,in, The mapping logic corresponding to the code strategy, The state data at time t, This serves as an identifier for the current task phase.

[0091] At the same time, the code execution unit calls the check function corresponding to the current subtask and outputs the stage completion flag: , ;

[0092] The stage status management unit verifies the current stage completion flag: if it determines that the current subtask has been completed, it triggers a stage switch and updates the stage flag. It triggers the code execution unit to calculate the reference action of the subtask corresponding to the new stage after the switch; if it is determined that the subtask is not completed, the current stage identifier is kept unchanged, and a signal is sent to trigger the deviation calculation unit of the host computer to calculate the action direction deviation and geometric deviation.

[0093] The deviation calculation unit receives the predicted action and reference action from the agent module, and calculates two types of quantification results: action direction deviation and task-related geometric deviation.

[0094] (1) Action direction deviation: Calculate the vector angle between the predicted action and the reference action of the agent module to measure the consistency of the action direction. The formula is: ,in, To prevent extremely small positive numbers with a denominator of zero, Predict actions for the agent module. Reference actions generated for code strategies;

[0095] (2) Task-related geometric deviation: Extract the physical geometric quantities that are strongly related to the current subtask, calculate the Euclidean distance between the corresponding geometric quantities of the two types of actions, and measure the actual physical effect of the actions. The formula is: ,in, A mapping function for extracting geometric quantities related to the current subtask from the state and action, including the relative pose of the end effector to the target, the gripper opening and closing state, and the distance to the target region;

[0096] The feedback unit makes a joint judgment based on preset threshold ranges for both motion direction deviation and geometric deviation.

[0097] If both deviations are below the lower limit of the corresponding threshold range, an evaluative feedback is output to control the robot to execute the predicted action of the agent module;

[0098] If any deviation exceeds the upper limit of the corresponding threshold range, corrective feedback is output to control the robot to perform the reference action.

[0099] The dynamic weighting unit records the robot's state, actual actions performed, current stage identifier, and feedback type for each step.

[0100] After each training session, the total number of time steps and the number of corrective feedback events for that session are counted. Dynamic high weights are assigned to the corrective samples according to the weighting formula, and all weighted sample data are stored in the structured experience pool of the computing storage unit.

[0101] Sample dynamic weights The calculation rule is as follows: for the sample corresponding to the evaluative feedback, let ;

[0102] For the sample corresponding to corrective feedback, let Where N is the total number of time steps in the current round. This represents the number of corrective feedback responses that occurred in that round.

[0103] The agent module employs a parameterized Gaussian random strategy. The module reads samples from the weighted structured experience pool and updates the neural network parameters using the stochastic gradient descent algorithm with the goal of minimizing the negative log-likelihood loss.

[0104] The loss function is: ,in, Let t be the action actually performed by the robot. These are the network parameters for the agent module's strategy.

[0105] Before training begins, the teacher module of the large language model is scheduled to generate and output the code strategy;

[0106] During the warm start phase, the code strategy is invoked to execute several rounds of training tasks in the simulation environment or real machine environment, collect the state and action corresponding to all successful trajectories, form a pre-training dataset from the state and action, and then input the pre-training dataset into the agent module. Through supervised learning, the preliminary fitting of the neural network parameters is completed, so that the agent module has basic task execution capabilities.

[0107] During formal training, each unit is scheduled to synchronously complete state acquisition, action generation, deviation calculation, feedback output and sample recording according to the control cycle, and the parameter update operation of the agent module is triggered in rounds.

[0108] When the number of training rounds reaches a preset value, or the success rate of the task reaches the target for multiple consecutive rounds, the deployment control unit is called to export only the policy network within the trained agent module and deploy the policy network to the robot controller, realizing unmanned closed-loop training from task initialization to final deployment.

[0109] The resulting code strategy employs a structure that pairs ordered subtask steps with functions one-to-one.

[0110] Each subtask step corresponds to a set of independent action functions and check functions. The action functions are used to receive status data and output the reference action corresponding to the current subtask, and the check functions are used to receive status data and output the stage completion flag of the current subtask.

[0111] The subtasks are linked together in the order of decomposition. After the completion flag of the previous subtask is triggered, the system automatically switches to the next subtask and calls the corresponding function, forming a phased, executable, and verifiable control logic. This decomposes the semantics of high-level tasks into low-level execution rules that can be implemented step by step.

[0112] The intelligent agent module comprises a three-level architecture: convolutional feature extraction, temporal context reasoning, and Gaussian distribution output.

[0113] After receiving the environmental images and robot body state data transmitted by the perception and acquisition unit, the intelligent agent module first extracts the visual features of the environmental images through a convolutional layer, then splices and fuses them with the normalized body state features, and then processes the context information of the sequence decision through a temporal network. Finally, it outputs a parameterized Gaussian action distribution and samples it to obtain the predicted action of the intelligent agent module.

[0114] The agent module policy is modeled as a parameterized stochastic policy. ,in, The mean of the actions predicted by the neural network based on the state. is the covariance parameter.

[0115] Before training starts, the training scheduling unit calls the code strategy to automatically execute several rounds of training tasks in the simulation environment or real machine environment, collect the state and action corresponding to all successful trajectories, and form a pre-training dataset from the state and action.

[0116] The training scheduling unit then inputs the pre-trained dataset into the agent module, and completes the initial fitting of the neural network parameters through supervised learning.

[0117] When the number of control cycles during training reaches the preset value, or the success rate of the task in multiple consecutive control cycles meets the target, the deployment control unit removes the feedback arbitration and process scheduling logic between the large language model teacher module and the host computer, and only exports the policy network in the trained agent module. The policy network is then deployed to the robot controller. After deployment, the robot relies solely on the perception acquisition unit and the agent module to perceive the environment and output action commands.

[0118] The system's large language model teacher module incorporates three types of standardized prompt templates, corresponding to task planning, action function generation, and check function generation, respectively. During use, the user inputs a task description in natural language. The module first calls the planner prompt template to break down the complete task into multiple subtasks to be executed sequentially. Then, for each subtask, it calls the action function and check function prompt templates to generate the corresponding execution logic and completion judgment logic, ultimately outputting a complete set of directly executable training task code. The standardized prompt template design constrains the output format and content boundaries of the large language model, preventing freely generated content from being unusable for robot control. The technical benefit of this design is a significant reduction in the threshold for defining robot tasks, enabling the rapid generation of structured task control logic without requiring professionals to write low-level control code.

[0119] Before formal training begins, the system performs a warm-start pre-training process. The training scheduling unit on the host computer calls the generated executable code and runs multiple rounds of tasks in a simulation or real machine environment, collecting trajectory data from all successfully completed tasks, including environmental images, robot body states, and corresponding action commands for each step, thus forming a pre-training dataset. The training scheduling unit inputs this dataset into the agent module on the robot, and completes the initial fitting of neural network parameters through supervised learning. The reason for setting up the warm-start stage is to avoid the agent module starting from scratch and exploring aimlessly, reducing ineffective attempts in the early stages of training. The technical effect of this stage is that the agent module has basic task execution capabilities before formal interactive training, effectively shortening the overall training cycle and reducing resource consumption during the training process.

[0120] During training, the system employs a phased state management mechanism. The system models the entire task as a decision-making process with phased states, explicitly adding a current phase identifier to the state information to indicate which subtask has been executed. Each subtask corresponds to an independent action function and a check function. The action function outputs a reference action for that subtask based on the current state, and the check function outputs a judgment result indicating whether the subtask is completed based on the current state. The host computer's phase state management unit is responsible for maintaining the task's phase order, verifying the completion status of subtasks, and automatically switching to the next phase and calling the corresponding function upon completion of a subtask. The introduction of phase identifiers in the states aims to solve the semantic misalignment problem in action judgments in long-sequence tasks: the correct actions corresponding to the same robot state are not the same in different subtasks, and comparing action values ​​without considering the phase context can lead to numerous misjudgments. The technical effect of this design is that it binds all action judgments to the target of the current subtask, improving the accuracy of deviation judgments and avoiding deviations in training direction due to task phase confusion.

[0121] Within each control cycle, the host computer performs deviation verification and feedback control on the output actions of the agent module. The deviation calculation unit simultaneously calculates two types of deviation indicators: one is the action direction deviation, which measures the consistency of the motion directions by calculating the vector angle between the predicted action and the reference action; the other is the task-related geometric deviation, which extracts the physical geometric quantities related to the current subtask and calculates the Euclidean distance between the corresponding geometric quantities of the two types of actions to measure the difference in the actual physical effect of the actions. The feedback unit makes a joint judgment based on the preset threshold range of the two types of deviations. When both types of deviations are below the lower threshold, the robot is allowed to execute the predicted action of the agent module; when either type of deviation exceeds the upper threshold, the robot is controlled to execute the reference action for correction. The design of dual-dimensional deviation and interval threshold is adopted because the numerical deviation of a single action cannot accurately reflect the actual physical effect of the action, while a fixed threshold would excessively restrict the exploration space of the agent. The technical effect of this mechanism is to reserve a reasonable exploration space for the agent while ensuring that the training does not deviate from the correct task objective, so that the final trained strategy has stronger environmental adaptability and generalization ability.

[0122] During training, the system dynamically assigns weights to training samples to improve training efficiency. The dynamic weighting unit records the state, actual action, current stage identifier, and feedback type for each step. After each training round, it calculates the total number of steps and the number of corrective feedback occurrences, assigning higher dynamic weights to samples corresponding to corrective feedback. All weighted samples are stored in a structured experience pool. The agent module reads the weighted sample data and updates the neural network parameters using stochastic gradient descent, with the optimization objective of minimizing the negative log-likelihood loss. The design of assigning higher weights to corrective samples is because these samples correspond to decision errors by the agent module and have higher training value than normally executed samples. The technical effect of this design is that it allows the agent module to focus on correcting its weaknesses, improves the efficiency of sample data utilization, accelerates policy convergence, and simultaneously improves the control accuracy of the final policy.

[0123] The robot's intelligent agent module employs a three-level network architecture, sequentially performing visual feature extraction, multimodal temporal fusion, and action distribution output. After receiving environmental images and robot body state data, the module first extracts visual features through convolutional layers, then concatenates and fuses them with normalized body state features. Next, a temporal network processes the contextual information for sequential decision-making, ultimately outputting a parameterized Gaussian action distribution, from which predicted actions are obtained. Once the number of training rounds reaches a preset value, or the success rate of multiple consecutive task rounds meets requirements, the deployment control unit removes the large language model module and feedback scheduling logic required during the training phase, deploying only the trained policy network to the robot's local controller. This lightweight edge-side deployment design is adopted because the training phase requires complex control and correction logic, while actual operating scenarios demand low latency and high reliability in robot control, eliminating reliance on external devices. The technical effect of this design is that the deployed robot can operate independently using only its own perception and control unit, meeting the real-time control requirements of industrial scenarios and achieving a fully unmanned closed-loop process from task definition to final deployment.

[0124] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. An embodied robot interactive imitation learning system based on a large language model, characterized in that, include The large language model teacher module has built-in standardized prompt templates, including planner prompt templates, action function prompt templates, and check function prompt templates. It receives the natural language task description input by the user, breaks down the complete task into an ordered sequence of steps that integrate multiple sub-tasks, generates the corresponding action function and check function for each sub-step, and outputs the executable code corresponding to the training task. The host computer includes a training scheduling unit for training process control and scheduling, a code execution unit for loading executable code and generating reference actions and stage flags, a stage state management unit for maintaining task stages and stage switching, a deviation calculation unit for calculating two types of deviations: action direction and geometry, a feedback unit for deviation judgment and feedback instruction output, a dynamic weighting unit for recording training samples and dynamic weight allocation, a computing storage unit for storage, and a deployment control unit for exporting the agent module strategy network and deploying the robot end. The robot is equipped with a perception and acquisition unit and an intelligent agent module. The perception and acquisition unit collects and transmits the current environmental image and the robot's own state data. The intelligent agent module outputs the robot's execution and strategy parameters for iterative optimization based on the data from the perception and acquisition unit. The deviation calculation unit receives the predicted action and reference action from the agent module, and calculates two types of quantification results: action direction deviation and task-related geometric deviation. (1) Action direction deviation: Calculate the vector angle between the predicted action and the reference action of the agent module to measure the consistency of the action direction. The formula is: ,in, To prevent extremely small positive numbers with a denominator of zero, Predict actions for the agent module. Reference actions generated for code strategies; (2) Task-related geometric deviation: Extract the physical geometric quantities that are strongly related to the current subtask, calculate the Euclidean distance between the corresponding geometric quantities of the two types of actions, and measure the actual physical effect of the actions. The formula is: ,in, This is a mapping function for extracting geometric quantities relevant to the current subtask from the state and action, including the relative pose of the end effector to the target, the gripper's opening and closing state, and the distance to the target region. This represents the state data at time t. The feedback unit makes a joint judgment based on preset threshold ranges for both motion direction deviation and geometric deviation. If both deviations are below the lower limit of the corresponding threshold range, an evaluative feedback is output to control the robot to execute the predicted action of the agent module; If any deviation exceeds the upper limit of the corresponding threshold range, corrective feedback is output to control the robot to perform the reference action; The dynamic weighting unit records the robot's state, actual actions performed, current stage identifier, and feedback type for each step. After each training session, the total number of time steps and the number of corrective feedback events for that session are counted. Dynamic high weights are assigned to the corrective samples according to the weighting formula, and all weighted sample data are stored in the structured experience pool of the computing storage unit. Sample dynamic weights The calculation rule is as follows: for the sample corresponding to the evaluative feedback, let ; For the sample corresponding to corrective feedback, let Where N is the total number of time steps in the current round. This represents the number of corrective feedback responses that occurred in that round. The agent module employs a parameterized Gaussian random strategy. The module reads samples from the weighted structured experience pool and updates the neural network parameters using the stochastic gradient descent algorithm with the goal of minimizing the negative log-likelihood loss. The loss function is: ,in, Let t be the action actually performed by the robot. These are the network parameters for the agent module's strategy.

2. An embodied robot interactive imitation learning method based on a large language model, characterized in that, The learning method of the embodied robot interactive imitation learning system based on a large language model as described in claim 1 is as follows: Before training begins, the user inputs a natural language task description into the large language model teacher module, guiding the module to codify the training task. The user calls the planner prompt template to guide the large language model to break down the complete task into an ordered sequence of steps that integrate multiple sub-tasks. Then, for each sub-step, the user calls two types of prompt templates: action function and check function, to guide the large language model to generate the corresponding action function and check function. This allows the teacher module of the large language model to output the executable code corresponding to the entire training task, i.e., the code strategy. The training scheduling unit of the host computer calls the code strategy to execute several rounds of training tasks in the simulation environment or real machine environment, collects the state and action corresponding to all successful trajectories, and uses the state and action to form a pre-training dataset. Then, the pre-training dataset is input into the intelligent agent module, and the neural network parameters are initially fitted through supervised learning, so that the intelligent agent module has basic task execution capabilities. During training, the robot executes the following within each control cycle: The perception and acquisition unit collects images of the current environment and the robot's own state, and synchronously sends unified state data to the host computer and the intelligent agent module; The robot's intelligent agent module receives state data, extracts features, and performs temporal reasoning to output the predicted action. The host computer's code execution unit receives status data, combines it with the current stage identifier output by the host computer's stage status management unit, takes the received status data and the current stage identifier as input, calculates and outputs the corresponding reference action according to the code strategy, and calls the check function to output the current stage completion flag. The host computer's stage status management unit verifies the current stage completion flag. If it determines that the current subtask has been completed, it triggers a stage switch, updates the current stage flag, and then triggers the code execution unit to calculate the reference action of the subtask corresponding to the new stage after the switch. If it determines that the subtask has not been completed, it sends a signal to trigger the host computer's deviation calculation unit to calculate the action direction deviation and geometric deviation. The feedback unit of the host computer makes a judgment based on the preset threshold ranges of two items: motion direction deviation and geometric deviation. If both deviations are lower than the lower limit of the threshold range, it outputs evaluative feedback and controls the robot to execute the predicted action of the intelligent agent module. If either deviation exceeds the upper limit of the threshold range, it controls the robot to execute the reference action. The host computer's dynamic weighting unit records the robot's state, actual actions, current stage identifier, and feedback type for each step. After each control cycle ends, it calculates the total number of steps and corrections for that control cycle, assigns dynamic high weights to the correction samples according to the weighting formula, and stores all the data in the structured experience pool of the computing storage unit. The training scheduling unit of the host computer triggers parameter updates. The agent module reads samples from the weighted structured experience pool and updates the neural network parameters within the agent module with the goal of minimizing the negative log-likelihood loss through the stochastic gradient descent algorithm. When the number of training control cycles reaches the preset value, or the success rate of the task reaches the target for multiple consecutive control cycles, the deployment control unit only exports the policy network in the trained agent module and deploys the policy network to the robot controller. The current stage identifier is a sequence-type state variable used to mark which sub-step the robot is currently in within the entire long sequence task; the stage completion flag is a Boolean judgment result output by the check function corresponding to the current stage, used to determine whether the current sub-step has met the completion conditions.

3. The interactive imitation learning method for embodied robots based on a large language model as described in claim 2, characterized in that, The training task is modeled as a Markov decision process with phased states, denoted as a tuple. Where S is the state space containing robot joint angles, end effector pose, gripper state, and environmental image features; A is the action space composed of robot motion control commands; T is the state transition function; and R is the implicit reward. Discount factor, where K is the set of stages obtained from task decomposition; Explicitly introduce the current task stage identifier in the state. This ensures that all subsequent action decisions are bound to the current subtask context; During operation, the stage status management unit maintains the task stage sequence and outputs the current stage identifier; The code execution unit loads executable code as a code strategy, takes the synchronized state data and current stage identifier from the sensing and acquisition unit as input, calculates and outputs the corresponding reference action, which satisfies the formula: ,in, The mapping logic corresponding to the code strategy, This serves as an identifier for the current task phase. At the same time, the code execution unit calls the check function corresponding to the current subtask and outputs the stage completion flag: , ; The stage status management unit verifies the current stage completion flag: if it determines that the current subtask has been completed, it triggers a stage switch and updates the stage flag. It triggers the code execution unit to calculate the reference action of the subtask corresponding to the new stage after the switch; if it is determined that the subtask is not completed, the current stage identifier is kept unchanged, and a signal is sent to trigger the deviation calculation unit of the host computer to calculate the action direction deviation and geometric deviation.

4. The embodied robot interactive imitation learning method based on a large language model as described in claim 2, characterized in that, Before training begins, the teacher module of the large language model is scheduled to generate and output the code strategy; During the warm start phase, the code strategy is invoked to execute several rounds of training tasks in the simulation environment or real machine environment, collect the state and action corresponding to all successful trajectories, form a pre-training dataset from the state and action, and then input the pre-training dataset into the agent module. Through supervised learning, the preliminary fitting of the neural network parameters is completed, enabling the agent module to have basic task execution capabilities. During formal training, each unit is scheduled to synchronously complete state acquisition, action generation, deviation calculation, feedback output and sample recording according to the control cycle, and the parameter update operation of the agent module is triggered in rounds. When the number of training rounds reaches a preset value, or the success rate of the task reaches the target for multiple consecutive rounds, the deployment control unit is called to export only the policy network within the trained agent module and deploy the policy network to the robot controller, realizing unmanned closed-loop training from task initialization to final deployment.

5. The embodied robot interactive imitation learning method based on a large language model as described in claim 2, characterized in that, The resulting code strategy employs a structure that pairs ordered subtask steps with functions one-to-one; Each subtask step corresponds to a set of independent action functions and check functions. The action functions are used to receive status data and output the reference action corresponding to the current subtask, and the check functions are used to receive status data and output the stage completion flag of the current subtask. The subtasks are linked together in the order of decomposition. After the completion flag of the previous subtask is triggered, the system automatically switches to the next subtask and calls the corresponding function, forming a phased, executable, and verifiable control logic. This decomposes the semantics of high-level tasks into low-level execution rules that can be implemented step by step.

6. The interactive imitation learning method for embodied robots based on a large language model as described in claim 2, characterized in that, The intelligent agent module comprises a three-level architecture: convolutional feature extraction, temporal context reasoning, and Gaussian distribution output. After receiving the environmental images and robot body state data transmitted by the perception and acquisition unit, the intelligent agent module first extracts the visual features of the environmental images through a convolutional layer, then splices and fuses them with the normalized body state features, and then processes the context information of the sequence decision through a temporal network. Finally, it outputs a parameterized Gaussian action distribution and samples it to obtain the predicted action of the intelligent agent module. The agent module policy is modeled as a parameterized stochastic policy. ,in, The mean of the actions predicted by the neural network based on the state. is the covariance parameter.

7. The embodied robot interactive imitation learning method based on a large language model as described in claim 2, characterized in that, Before training starts, the training scheduling unit calls the code strategy to automatically execute several rounds of training tasks in the simulation environment or real machine environment, collect the state and action corresponding to all successful trajectories, and form a pre-training dataset from the state and action. The training scheduling unit then inputs the pre-trained dataset into the agent module, and completes the initial fitting of the neural network parameters through supervised learning.

8. The interactive imitation learning method for embodied robots based on a large language model as described in claim 2, characterized in that, When the number of control cycles during training reaches the preset value, or the success rate of the task in multiple consecutive control cycles meets the target, the deployment control unit removes the feedback arbitration and process scheduling logic between the large language model teacher module and the host computer, and only exports the policy network in the trained agent module. The policy network is then deployed to the robot controller. After deployment, the robot relies solely on the perception acquisition unit and the agent module to perceive the environment and output action commands.

Images