Robot control method based on model-based and rl

Abstract

The application provides a robot control method based on Model-Based and RL, and relates to the technical field of automatic control.The method maps a target task into a state node by constructing a high-level task graph model and a predefined task action set, performs graph search in combination with a current world state, plans an optimal task action path, and outputs a first action instruction.In a simulation environment, a baseline strategy is called to generate a control instruction, a feasibility check is performed, a simulation and real environment deviation is compensated through an error prediction model, a real control instruction is corrected, and a robot is driven to execute.The success probability weight of state transition in the graph is dynamically updated based on the execution result, and the state after execution is fed back for subsequent decision-making.The application improves the reliability, adaptability and execution safety of simulation strategy migration to a real robot, and realizes closed-loop optimization and continuous self-adaptation of a complex task.

Description

Technical Field

[0001] This invention relates to the field of automation control technology, and more specifically to a robot control method based on Model-Based and Reinforcement (RL). Background Technology

[0002] Traditional robot control methods are mainly divided into two categories: model-based control and model-free reinforcement learning control. Model-based control methods rely on accurate physical dynamic models for planning and control. While they are stable in structured environments, their performance is heavily dependent on the accuracy of the model. They exhibit poor generalization ability when faced with model mismatch, environmental disturbances, or unknown loads, and cannot adapt to complex nonlinear tasks. On the other hand, model-free reinforcement learning techniques learn strategies autonomously through trial and error and interaction with the environment, avoiding the difficulties of accurate modeling and showing great potential in complex tasks. However, these methods often suffer from low data efficiency and high training costs, and the extensive random exploration during training is neither safe nor practical to perform directly on a physical robot. In recent years, although a technical approach combining simulation and reinforcement learning has emerged—training policies in a simulation environment and then transferring them to physical robots—it is still hampered by the "simulation to reality" gap. Because simulation models cannot fully reproduce real physical characteristics, policies that perform well in simulation experience a sharp decline in performance in the real world. Existing solutions primarily focus on improving the fidelity of simulation models or leveraging domain randomization to enhance the robustness of strategies. However, the former incurs extremely high modeling costs, while the latter lacks abstraction and guidance of task logic, making it unsuitable for complex sequential tasks requiring multi-step planning and high-level reasoning. Therefore, there is an urgent need in this field for a robot control method that can balance the safety and data efficiency of simulation training, effectively bridge the gap between simulation and reality, and possess both high-level task understanding and low-level fine-grained control capabilities. Summary of the Invention

[0003] This application provides a model- and reinforcement learning-based robot control method and system, aiming to solve the core technical problems of poor reliability, insufficient adaptability, and lack of safety in the transfer of simulation training strategies to real physical robots in the prior art.

[0004] The first aspect disclosed in this application provides a robot control method based on Model-Based and RL, including:

[0005] S1: Construct a two-layer world model of the target robot to form a Model-Based model for task execution and optimization. The two-layer world model includes a low-level dynamics model and a high-level task graph model. The low-level dynamics model is used to generate a virtual robot that can simulate the physical behavior of the target robot. The high-level task graph model is used to characterize the transition relationship between the various task states of the target robot during task execution. The transition relationship is driven by the actions in the predefined set of task actions.

[0006] S2: In the simulation environment of the virtual robot, for each task action in the task action set, a neural network policy is trained through a reinforcement learning algorithm to learn the mapping relationship from the current state to the control action command corresponding to the task action. The trained neural network policy is saved as the baseline policy corresponding to the task action for subsequent use in the simulation environment and the real environment to generate the corresponding control action command.

[0007] S3: Receive the target task, obtain the target robot's body state and surrounding environment state, map the target task to the target state node in the high-level task graph model, determine the above two states as the current world state, run the graph search algorithm to plan the best task action path, and output the first task action in the path in sequence.

[0008] S4: In the simulation environment of the virtual robot, based on the current world state, the baseline strategy corresponding to the first task action is invoked to generate a simulation control action command. The command is executed in the simulation environment, and the feasibility of its execution process is verified. If the verification passes, the post-execution state predicted by the simulation is input into the pre-trained error prediction model to obtain the predicted error amount. Based on the error amount, the control action command generated by the same baseline strategy in the real environment is compensated and corrected to obtain the corrected real control command, which drives the target robot to execute. The real-world state after execution is returned to step S3 as a feedback signal, and the execution result is monitored.

[0009] S5: Determine the execution result.

[0010] If the action is executed successfully, the success probability weight of the corresponding state transition edge in the graph is increased.

[0011] If an action fails to execute, the success probability weight of the corresponding state transition edge in the graph is reduced.

[0012] One or more technical solutions provided in this application have at least the following technical effects or advantages:

[0013] 1) By constructing a high-level task graph model, abstract tasks are decomposed into semantic state nodes and task action transition relationships, giving the task planning process a clear logical structure and improving the interpretability and structure of task decisions; 2) Graph search is performed based on the current world state and the target state, and edge weights are dynamically adjusted based on the success rate prediction of the baseline strategy and execution priority, enabling path planning to adaptively adjust according to historical execution feedback, avoiding repeated failed paths, improving the long-term task success rate, and enhancing the interpretability and structure of task decisions; 3) Feasibility verification is performed in a simulation environment, and a pre-trained error prediction model is used to feedforward correct the real control commands, compensating for simulation modeling errors and environmental uncertainties in advance, significantly improving the baseline strategy... 4) Dynamically update the success probability weights of state transition edges in the task graph based on the actual execution results, enabling the system to learn from practical experience, gradually optimize the task planning strategy, and form a closed-loop learning mechanism of "execution → feedback → optimization", realizing the continuous optimization and adaptive evolution of task decision logic; 5) By using the feedback information of the current world state after execution as the input of the next decision cycle, the continuous transmission of task state is realized, supporting the robot to complete complex tasks that require multi-step action coordination, and supporting the coherent execution of multi-step complex tasks; 6) By using reinforcement learning to train the baseline strategy and error prediction model together to compensate for modeling bias, the dependence on high-precision dynamic modeling is reduced, and the applicability of the method in low-cost or complex environments is improved.

[0014] In summary, this invention effectively solves the key technical problems of poor reliability, insufficient adaptability, and lack of safety in the process of migrating simulation-based robot control strategies to real physical robots, and effectively improves the migration performance from simulation to reality. Attached Figure Description

[0015] Figure 1 This is a schematic diagram of the robot control method based on Model-Based and RL provided in the embodiments of this application. Detailed Implementation

[0016] This application provides a robot control method based on Model-Based and Reinforcement (RL). The technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, not all of them. It should be understood that this application is not limited to the exemplary embodiments described herein. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application. It should also be noted that, for ease of description, only the parts relevant to this application are shown in the accompanying drawings, not all of them.

[0017] like Figure 1 As shown, this application provides a robot control method based on Model-Based and RL, which specifically includes the following steps:

[0018] S1: Construct a two-layer world model of the target robot to form a Model-Based model for task execution and optimization. The two-layer world model includes a low-level dynamics model and a high-level task graph model. The low-level dynamics model is used to generate a virtual robot that can simulate the physical behavior of the target robot. The high-level task graph model is used to characterize the transition relationship between the various task states of the target robot during task execution. The transition relationship is driven by the actions in the predefined set of task actions.

[0019] Specifically, the two-layer world model refers to a composite cognitive architecture that simultaneously comprises a low-level physical dynamics model and a high-level task logic model. The former is used to simulate the robot's physical behavior with high fidelity, providing a foundation for control; the latter is used to abstractly represent task semantics and state transition relationships, providing a basis for planning. Together, they form a complete Model-Based framework, supporting a range of functions such as task planning, simulation verification, error compensation, and online learning, ultimately significantly improving the reliability, safety, and adaptability of robot task execution. This two-layer architecture ensures both the physical feasibility of action execution and endows the robot with high-level task understanding and autonomous learning capabilities, providing a unified cognitive foundation for subsequent simulation optimization, error compensation, and online adaptation. In the fields of artificial intelligence and robotics, a world model refers to an agent's internal representation of the environment, used for prediction, planning, and decision-making. It is not the real world, but rather the world "in the robot's mind," allowing the robot to "simulate" the consequences of actions "in its mind" without having to resort to trial and error each time.

[0020] Furthermore, a two-layer world model of the target robot is constructed to form a model-based model for task execution and optimization, including:

[0021] The two-layer world model is a composite structure that simultaneously includes a low-level physical dynamics model and a high-level task graph model. The former is used to simulate the real physical behavior of the target robot, while the latter is used to represent task semantics and state transition relationships. This two-layer structure together constitutes a complete Model-Based framework.

[0022] Constructing the underlying physical dynamics model includes:

[0023] Obtain the 3D CAD model and bill of materials of the target robot, construct an initial parameter file containing complete physical parameters and topology information, input the initial parameter file into the robot simulation environment, and automatically generate a virtual robot that can simulate the real physical behavior of the target robot;

[0024] Constructing a high-level task graph model includes:

[0025] Based on the functional requirements of the target robot, a set of preset task actions are defined to form a task action set. Each task action includes the action name, execution conditions, required input state information and expected output result.

[0026] Based on the set of task actions, the key stages in the task execution process are abstracted into multiple task state nodes, and each state node represents a discrete semantic stage in the process of the target robot performing the task.

[0027] For any two task state nodes, if there exists a task action whose execution prerequisite is consistent with the conditions described by the first state node, and the system can reach the state described by the second state node after executing the action, then a directed state transition edge is established between the two state nodes.

[0028] Each state transition edge is assigned an initial success probability weight and execution priority. The success probability weight is set based on historical execution data or expert experience and is used to characterize the probability of the action being successfully executed in the current world state. All state nodes and established state transition edges are integrated into a directed graph structure to form a high-level task graph model.

[0029] Specifically, a high-fidelity underlying physical dynamics model is constructed to describe the mathematical relationship between the robot's joint torques and its motion (position, velocity, acceleration). The 3D CAD model file of the target robot (e.g., STEP, SolidWorks format) and the bill of materials (BOM) are obtained. The BOM contains all the raw materials, parts, subassemblies, and components needed to manufacture the robot, along with information on the quantity, specifications, and model of each material. Key parameters are defined to obtain the complete physical parameters of the target robot and the initial parameter file of its topology. Here, the complete physical parameters of the target robot refer to a set of physical parameters describing how each rigid body link of the robot moves and is subjected to forces in the physical world. This includes at least: the number of all rigid body links, and the mass, center of mass position, and inertia tensor of each link. The complete physical parameters are calculated as: mass (m) of each link + center of mass position [cx, cy, cz] + inertia tensor [Ixx, Iyy, Izz, Ixy, Ixz, Iyz]. The initial parameter file for the topology describes the connection relationships and spatial order between the links of the target robot, i.e., "who connects to whom" and "how they are connected." This includes: link structure, defining an ordered list of all rigid body links and specifying their connection order; joint type, specifying the type of joint connecting two links (rotational or translational); and joint geometric parameters, parameters describing the relative position and orientation between the coordinate systems of two adjacent links. In this embodiment, the most commonly used parameter is the DH parameter, used to define the relative position and orientation between links. The initial parameter file for the robot's complete physical parameters and topology organizes all the above information in a structured way for subsequent modeling software or algorithms to use. Import the initial parameter file into a robot simulation platform (such as PyBullet, MuJoCo, Gazebo, or ROS+Gazebo). The simulation engine automatically parses and constructs the multibody dynamics system of the virtual robot based on the file content. For example, the simulation engine creates a corresponding rigid body object for each link and loads its mass, center of mass, and inertia tensor. Based on DH parameters or joint definitions, it establishes correct joint constraints (such as rotation axis direction, range of motion, damping, and friction coefficient) between adjacent links. The 3D CAD model (in STL, OBJ, or COLLADA format) is bound to the corresponding link as the visual and collision geometry for visualization and collision detection. The simulation engine automatically constructs the forward / inverse dynamic equations of the system based on the Newton-Euler or Lagrange method, enabling it to respond to external control commands (such as joint torques or target positions) and simulate real physical effects such as gravity, inertia, contact forces, and friction. The final generated virtual robot can perform motion behaviors consistent with real robots in the simulation environment, including but not limited to: joint drive response, center of gravity balance control, interaction with environmental objects (such as grasping and pushing), and dynamic response under external force disturbances.This virtual robot, acting as a "digital twin," provides a high-fidelity simulation environment for subsequent reinforcement learning training, task simulation verification, and error prediction.

[0030] The process of constructing a high-level task graph model is as follows: (1): Define a set of task actions. Based on the functional requirements of the target robot, a set of executable task actions is predefined to form a set of task actions. Each task action corresponds to an operation unit with clear semantics, which is used to complete a specific sub-task. For example, for a service robot, task actions may include: "move to a specified location", "grab a target object", "avoid obstacles", "place an object", "open a door", etc. Each task action contains the following information: Action name: used to uniquely identify the action (e.g., "grab object A"); Execution conditions: the preconditions that need to be met to execute the action (e.g., "the target object is within the field of vision" and "there are no obstacles in front of the robotic arm"); Required input state information: the current world state on which the action depends (e.g., the pose of the target object and the pose of the robot body); Expected output result: the state that should be achieved after the action is successfully executed (e.g., "the end effector contacts and clamps the target object"). This set of task actions serves as the basic input for high-level task logic modeling. (2): Abstract task state nodes. The set of task actions defined in (1) abstracts the key progress stages that the robot may experience during the execution of the task into multiple task state nodes. Each task state node represents a discrete, high-level semantic stage in the task execution process, reflecting the overall progress of the task rather than the underlying continuous physical variables. For example, for the task of "grabbing an object from point A and transporting it to point B", the following state nodes can be abstracted: S0: Initial standby state (the robot is in the starting position, and the end effector is empty); S1: The grasping area has been reached (the robot moves to the vicinity of the target object); S2: The target object has been grasped (the grasping action is completed, and the sensor feedback is successful); S3: The placement area has been reached, such as when the robot moves to the target placement point; S4: The object has been placed, and the task is completed. The state node is the semantic refinement result of the underlying information such as "joint angle", "end effector position coordinates" or "obstacle distribution data". For example, "there is an obstacle ahead" is the perception result, and "the obstacle has been avoided" is the state node. (3): Establishing state transition edges For any two task state nodes in (2), if there is a task action that can make the system transition from the first state node to the second state node, then a state transition edge is established between the two. Each state transition edge represents a feasible path to jump from one task stage to the next task stage. Each state transition edge is associated with a task action from (1) and contains the following attributes: associated action: the specific task action that triggers the transition (e.g., "grab an object"); action description: the function description of the action; execution conditions: the environmental and ontological state conditions that must be met before the action is executed (e.g., "the target object is visible and can be grabbed"); transition direction: a directed connection that indicates the direction of the state transition.For example, between state node S1: The grabbing area has been reached and S2: The target object has been grabbed, a state transition edge is established, which is associated with the task action "grab the object". The execution condition is "the target object is within the grabbing range and there are no obstacles". (4): The initial state transition edge parameters are assigned to each state transition edge in S1.2.3 with an initial success probability weight and execution priority: Success probability weight: represents the probability of the task action being successfully executed in the current world state. The initial value is set based on historical execution data statistics or domain expert experience (e.g., "grab action" is set to 0.95 when the lighting is good and 0.7 when the lighting is poor); Execution priority: is used to select the optimal path among multiple feasible paths. High reliability or low energy consumption actions are given higher priority. These parameters will serve as the basis for task planning and path search. (5): Construct a high-level task graph model. Integrate all task state nodes in (2) and the state transition edges established in (3) into a directed graph structure to form a high-level task graph model. This model organizes task logic in the form of a graph, supporting task path planning based on graph search algorithms (such as A* and Dijkstra's algorithm). This task graph model is deployed in the robot's host control system.

[0031] S2: In the simulation environment of the virtual robot, for each task action in the task action set, a neural network policy is trained through a reinforcement learning algorithm to learn the mapping relationship from the current state to the control action command corresponding to the task action. The trained neural network policy is saved as the baseline policy corresponding to the task action for subsequent use in the simulation environment and the real environment to generate the corresponding control action command.

[0032] Specifically, without consuming real robot resources, the system pre-learns and solidifies the control capabilities of each task action, forming a baseline policy library. This policy library not only supports efficient task execution but also provides a reliable foundation for subsequent simulation verification, error prediction, and instruction correction, significantly improving the autonomy and robustness of the robot system. In this embodiment, reinforcement learning (RL) is used to train a neural network policy in a high-fidelity simulation environment, enabling it to master the underlying control capabilities of each predefined task action (such as "grasping," "moving," and "placing"), forming a reusable baseline policy library. The essence of the baseline policy is a general policy function π(a|s), which is a neural network model. The input is the current state s (such as robot pose, target position, etc.), and the output is the control action a (such as joint target position or torque). It does not care whether the environment is simulation or reality, but only cares about "what the input state is." As long as the input state representation is consistent (i.e., "state space alignment"), the same policy can run in both simulation and reality. The "state" in both simulation and real-world environments uses a unified semantic representation: "robotic arm end-effector position = (x, y, z)", "target object pose = T", "obstacle ahead = yes / no". This standardization ensures the policy's portability across environments. Analogously, the action logic of "reaching for a cup" is the same in both reality and VR, as long as the "cup's position" is expressed consistently. In the real-world environment, the baseline policy is not directly used to control the real robot but serves as the basis for subsequent "simulation prediction" and "command correction". Real-world control commands are generated based on the baseline policy output, combined with an error prediction model for compensation.

[0033] Furthermore, the system learns the mapping relationship from the current state to the control action command corresponding to the task action, and the trained neural network policy is saved as the baseline policy for the task action, including:

[0034] (1) In the simulation environment, for each task action in the task action set, according to the pre-set initial state range corresponding to the task action, at least one of the following is randomly generated: joint angle of the virtual robot, end effector pose, target object pose, obstacle distribution information and relative distance between the robot and the target, in order to initialize the simulation environment and obtain the current state. The initial state range is determined by expert experience based on the typical execution scenario of the task action.

[0035] (2) Input the current state into the neural network strategy to generate control action instructions for completing the target task. Execute the instructions in the simulation environment to drive the virtual robot, obtain the next state after execution, and calculate the corresponding reward value according to the preset reward function. The preset reward function includes task completion reward, distance reward, action smoothness reward, collision penalty and energy consumption penalty.

[0036] (3) Store the current state, control action instructions, reward value and next state as experience data in the experience playback buffer, and use the next state as the new current state. Return to step (2) to continue execution until the current training round ends.

[0037] (4) During the training process, a batch of empirical data is periodically sampled from the experience replay buffer, the policy gradient is calculated using a deep reinforcement learning algorithm, and the parameters of the neural network policy are updated.

[0038] (5) Test the execution success rate of the updated neural network strategy in an independent verification environment. When the execution success rate reaches the preset threshold continuously and stably for several consecutive times, stop training and use the current neural network strategy as the baseline strategy output corresponding to the task action, which is used to generate control action instructions based on the input state in subsequent task execution.

[0039] Specifically, in the simulation environment, a neural network policy capable of reliably executing various macroscopic task actions is trained; this is called the baseline policy. The specific steps include: for each task action in the defined set of task actions (such as "move to a specified location," "grasp a target object," "avoid obstacles," and "place an object"), constructing a corresponding reinforcement learning training task in the simulation environment. Each training task includes: a state space: including the virtual robot's ontological state (e.g., joint angles, end effector pose) and environmental state (e.g., target object position, obstacle distribution); an action space: for the virtual robot's control inputs (e.g., joint torques or target position); and a reward function: designed with sparse or dense rewards based on the task objective. For example, in the "grasp" task, a positive reward is given for successfully contacting and clamping an object, while a negative reward is given for colliding with an obstacle. The system independently trains a high-performance, highly robust neural network policy for each preset task action. This policy, through multi-reward guidance and verification mechanisms in the simulation, possesses good generalization ability, providing a reliable baseline control capability for safe and efficient execution in the real environment. Specific steps: (1): Construct a reinforcement learning training task corresponding to each task action. For each task action defined in S1 (such as "move to a specified position", "grab the target object", "avoid obstacles", "place object"), construct an independent reinforcement learning training task in the simulation environment. Each training task includes the following elements: State space: composed of the target robot's body state and the surrounding environment state, specifically including: robot joint angles; position and orientation of the end effector in the world coordinate system; pose of the target object; distribution information of obstacles; relative distance between the robot and the target; Action space: the control input for the robot at the next moment, specifically: target torque or target position of each joint; or movement speed of the end effector; Initial state distribution: at the beginning of each training round, the relative position of the robot and the target object is randomly initialized to enhance the generalization ability of the policy; Termination condition: the round ends when the maximum number of steps is reached, or the task objective is completed (such as successful grabbing), or a collision occurs. Each task action corresponds to an independent training task to ensure the modularity and reusability of policy training. (2): Define a preset reward function. For each training task, a preset reward function is designed to guide the neural network policy to converge toward the target task. This reward function is a weighted sum of multiple reward items, which specifically include: task completion reward, triggered only when the task is accomplished or a serious error occurs; distance reward, encouraging the simulated robot's end effector to approach the target; motion smoothness reward, penalizing drastic changes in the simulated robot's motion to improve control stability; collision penalty, avoiding dangerous actions; and energy consumption penalty, penalizing the simultaneous occurrence of high torque and high speed to improve energy efficiency. This reward function provides dense feedback (such as distance reward) in the early stages of training and relies on sparse rewards (task completion) in the later stages, effectively solving the sparse reward problem.(3): Execute the training process and perform multiple training rounds in the simulation environment. Each round includes the following steps: Initialize the environment: Set the initial pose of the robot and the target object according to the initial state distribution; State observation: Obtain the current state S. t (Including joint angles, end-effector positions, target pose, etc.); Motion generation: S t Input neural network policy π(a|s), output control action command a t ;Execute this action instruction: Execute a in the simulation environment t Drive the virtual robot; State transition: The simulation engine updates the system state to obtain the next state S. t+1 Calculate the reward: Calculate the reward R for the current step according to the preset reward function defined in (2). t Storage experience: (S) t a t R t S t+1 ) Store in the experience replay buffer; determine if the round has ended: if the termination condition is met (such as task completion, timeout, collision), then reset the environment and start a new round; otherwise t←t+1, return to step (2). (4): Update the neural network policy parameters. After each training round or fixed number of steps, sample a batch of state transition sequences from the experience replay buffer and perform the following operations: use a deep reinforcement learning algorithm (such as PPO or SAC) to calculate the policy gradient; update the parameters of the neural network policy to maximize the long-term reward expectation; (5): determine convergence and output the baseline policy. After every N training rounds, test the performance of the current policy in an independent verification environment; the verification environment includes the initial state and obstacle configuration that did not appear in the training; calculate the execution success rate: number of rounds in which the task was successfully completed / total number of test rounds; when the execution success rate reaches the preset threshold (such as ≥90%) for M consecutive tests and the fluctuation is less than 5%, the policy is determined to be converged; save the parameters of the current neural network policy as the baseline policy for subsequent task execution.

[0040] S3: Receive the target task, obtain the target robot's body state and surrounding environment state, map the target task to the target state node in the high-level task graph model, determine the above two states as the current world state, run the graph search algorithm to plan the best task action path, and output the first task action in the path in sequence.

[0041] Furthermore, starting from the current state node and ending at the target state node, the graph search algorithm is used to plan the optimal task action path from the start point to the end point, and the current action of the first task action in this path is output, including:

[0042] (1) Based on the current world state, determine the corresponding task state node in the high-level task graph model as the starting point, and take the target state node mapped by the target task as the ending point.

[0043] (2) For each state transition edge in the task graph, construct the comprehensive cost of the state transition edge based on its associated execution priority and initial success probability weight; wherein, the execution priority is a static scheduling preference pre-set according to task logic and expert experience when constructing the task graph, and is mapped to the corresponding scheduling cost; the initial success probability weight is a priori value set according to simulation verification results or expert experience when training the baseline policy, and is converted into the corresponding risk cost; wherein, the comprehensive cost is calculated by weighted summation.

[0044] The overall cost = α·scheduling cost + β·risk cost, where α and β are preset weighting coefficients;

[0045] (3) Given a defined starting point and ending point, and combining the calculated comprehensive cost of each side, run a graph search algorithm on the high-level task graph model to find the path with the minimum total cost from the starting point to the ending point, which is the optimal task action path.

[0046] (4) Extract the first task action from the best task action path and output it as the task action to be executed to trigger the subsequent control execution process based on the baseline strategy; the remaining task actions are kept in the path for subsequent cycles to decide whether to continue execution or replan based on the actual execution situation.

[0047] Specifically, the system transforms the abstract target task into an executable action path based on a high-level task graph and outputs the first action command to initiate the underlying control process. This process fully utilizes the two-layer world model constructed in S1 and the baseline strategy trained in S2 to achieve seamless integration of task-level decision-making and action-level control, laying the foundation for subsequent feasibility verification in simulation and execution of corrective actions in a real environment. In one embodiment, the steps of receiving the target task and planning the optimal task action path include: S3.1: Receiving the target task and parsing its semantics. Receiving the target task from the upper-level instruction system or human-machine interface, such as "move object A from region R1 to region R2" or "open valve V"; performing semantic parsing on the target task to extract key task elements: target object (e.g., "object A", "valve V"); starting position and target position (e.g., "R1 → R2"); mapping the task to a target state node N in the high-level task graph model. goalFor example, "Object A is located in R2 and is held" corresponds to an end state node in the graph; this mapping is based on the predefined state semantic encoding rules of the task graph ("ObjectInRegion(object=A, region=R2) ∧ GripperClosed"). S3.2: Perceive and construct the current world state, which is obtained through the target robot's sensor system (such as vision camera, force sensor, IMU): Body state: current joint angle, end effector pose, gripper opening and closing state; Environmental state: current pose of target object A, obstacle distribution, accessibility of region R1 / R2; The perceived data is fused into the current world state Ncurrent, for example, "Object A is located in R1 and is not held" corresponds to an initial state node in the graph, and the mapping is based on the state recognition rules of the task graph to ensure alignment with the graph semantic space. S3.3: Run a graph search algorithm in the task graph model to plan the optimal task action path. Starting from Ncurrent and ending at Ngoal, run a graph search algorithm (such as A*, Dijkstra's algorithm, or heuristic search) in the high-level task graph model. During the search, each edge represents an executable task action (such as "grab object A", "move above R2", "release object"). The weight of the edge is determined by the following factors: execution priority, action complexity, the success rate prediction of the baseline strategy, and environmental dynamics. The search outputs an optimal task action path from the starting point to the ending point. =[a1, a2, ..., an], where each ai is an action in the task action set. S3.4: Output the current action of the first task action in the path, starting from the planned optimal path. Extract the first task action a1.

[0048] S4: In the simulation environment of the virtual robot, based on the current world state, the baseline strategy corresponding to the first task action is invoked to generate a simulation control action command. The command is executed in the simulation environment, and the feasibility of its execution process is verified. If the verification passes, the post-execution state predicted by the simulation is input into the pre-trained error prediction model to obtain the predicted error amount. Based on the error amount, the control action command generated by the same baseline strategy in the real environment is compensated and corrected to obtain the corrected real control command, which drives the target robot to execute and monitors the execution result.

[0049] Furthermore, the corrected actual control commands are obtained to drive the target robot to execute, including:

[0050] The simulation control action command is executed in the simulation environment to drive the virtual robot to simulate the execution process of the command, and the state sequence during the execution process is recorded, including joint angle changes, end-effector trajectory, distance to obstacles, and dynamic parameters.

[0051] Based on the recorded state sequence and the predicted state after execution, determine whether the preset feasibility conditions are met; if they are met, the simulation control action command is deemed feasible, and proceed to the next process.

[0052] The predicted state after the simulation is completed is input into the pre-trained error prediction model, and the predicted error corresponding to the execution of the action is output.

[0053] Based on the prediction error, the real control commands generated by the same baseline strategy in the real environment are fed forward to obtain the corrected control commands, which drive the target robot to perform the corrected actions.

[0054] Specifically, the feasibility of the current action command is pre-verified using a virtual simulation environment, and the performance deviation between the simulation and the real environment is estimated through a pre-trained error prediction model. This allows for feedforward correction of the real control commands generated by the baseline strategy, improving the success rate and stability of the real robot's actions. This step achieves a closed-loop enhancement mechanism of "simulation verification → error prediction → command correction," effectively mitigating the Sim-to-Real gap problem. In one specific embodiment, after receiving the current action command (e.g., "grasp object A") output by S3, the system enters the pre-execution verification and command enhancement stage. First, based on the command, the corresponding neural network strategy is called from the baseline strategy library trained by S2. The current world state obtained by S3 (including robot joint angles, end-effector pose, object A pose, and obstacle distribution) is used as input, and a continuous sequence of simulation control commands (e.g., outputting joint target torque every 50ms) is generated through strategy inference. Subsequently, this command sequence is executed in the virtual robot simulation environment, driving the virtual robot to simulate the action process and monitoring dynamic behavior in real time, including whether a collision occurs, whether the joint torque exceeds the limit, and whether the trajectory is stable. After the action is executed, the predicted post-execution state in the simulation (such as whether the object is clamped and the final pose of the end effector) is obtained, and it is determined whether the preset feasibility conditions are met (such as no collision, task success, and execution time less than 3 seconds). If not met, it is determined to be infeasible and fed back to S3 to replan the path; if met, the predicted state is input into the pre-trained error prediction model. This model is trained based on previously collected "simulation-real" paired data and can predict pose, force control, or timing deviations, outputting the corresponding prediction error (such as the end effector Z-axis being 1.8cm higher and the clamping force being delayed by 200ms). The system performs feedforward correction on the real control commands generated in the real environment using the same baseline strategy. The predicted error includes at least one of the following: end effector pose deviation, joint angle deviation, or target object final position deviation. Based on this error, the adjustment amount to be compensated in the current control action command is derived in reverse. For example, if it is predicted that the end effector will deviate from the target by 2cm, an offset compensation of +2cm is added to the original target pose; if it is predicted that the gripper will close too early during grasping, the action timing is adjusted, delaying the closing command by 0.1s. The adjustment amount is fused with the original simulated control action command to generate a corrected control command containing compensation information. The corrected control command is output to the real environment to drive the target robot to execute. Finally, the command is sent to the target robot for execution, and the actual result is monitored in real time by sensors.

[0055] Furthermore, based on the recorded state sequence and the predicted state after execution, it is determined whether the preset feasibility conditions are met, including:

[0056] (1) Obtain the predicted post-execution state after the simulation is completed, as well as the dynamic behavior data recorded during the execution process. The dynamic behavior data includes: joint torque, motion speed, minimum distance to obstacles, and end trajectory stability index.

[0057] (2) Determine whether the state after the prediction execution meets the task success conditions. If not, it is determined to be infeasible.

[0058] (3) Determine whether there are situations in the dynamic behavior data that exceed the preset safety threshold, including: collision, joint torque exceeding the limit, speed exceeding the limit, or excessive end trajectory oscillation amplitude. If any of these situations exist, it is determined to be infeasible.

[0059] (4) Determine whether the execution time of the action exceeds the preset time threshold. If it does, it is determined to be infeasible.

[0060] (5) If all the judgments in (2), (3) and (4) are satisfied, it is determined to be feasible; otherwise, it is determined to be infeasible, and the corresponding failure type label is generated and fed back to S3 to adjust the task path or action parameters.

[0061] Specifically, in one embodiment, after the system executes the control sequence corresponding to the current action command in the simulation environment, it enters the feasibility verification stage. First, the system acquires the predicted post-execution state after the action is completed, including whether the target object is successfully gripped, the final pose of the end effector, and changes in the object's pose. Simultaneously, it extracts dynamic monitoring data during execution, including joint torques, velocities, accelerations, minimum distances to obstacles, and robot center-of-gravity stability. Based on this information, the system checks each preset feasibility condition: if the predicted state indicates that the task objective has not been achieved (e.g., gripping failure, target not moved into position), it is deemed infeasible; if a collision with the environment or the robot itself occurs during execution (minimum distance less than a safety threshold), or any joint torque / velocity exceeds physical limits, it is deemed infeasible; if the action execution time exceeds a preset threshold (e.g., more than 5 seconds), or the end effector trajectory oscillation amplitude exceeds the allowable range (reflecting control instability), it is also deemed infeasible. Only when all conditions are met is the action deemed "feasible." This process ensures that only actions that demonstrate safety and effectiveness in the simulation will proceed to the actual execution stage. If the system determines that the task is not feasible, it generates a failure type label (such as "collision" or "clamping failure") and sends it back to the S3 module, triggering task path replanning or parameter adjustment. This avoids performing high-risk actions in real-world environments and improves the overall robustness and safety of the system.

[0062] Furthermore, the pre-trained error prediction model includes:

[0063] By synchronously executing the same task actions in a simulation environment and on a real robot, multiple sets of states after simulation execution and corresponding states after real execution are collected to form a simulation-real paired dataset.

[0064] Based on the paired dataset, the difference between the simulated state and the real state for each execution is calculated and used as the real error label;

[0065] Construct a neural network model, setting its input to the state and task action type after simulation execution, and setting its output to at least one of position deviation, attitude deviation, and timing offset.

[0066] The neural network model is trained using paired datasets and true error labels to obtain a pre-trained error prediction model.

[0067] During the runtime phase, the predicted state obtained after the simulation execution is input into the pre-trained error prediction model, and the predicted error corresponding to the current action execution is output.

[0068] Specifically, a pre-trained error prediction model is used to estimate the execution deviation between the simulation and the real environment. This model, trained before system deployment, is structured as a multilayer perceptron (MLP) or graph neural network (GNN). Its input consists of spatially and task-related state features, and its output is a quantifiable prediction error. Training data is obtained by simultaneously collecting a large number of "simulation-real" paired samples: under the same initial state, the same task actions (such as grasping and placing) are performed on both the simulation environment and the real robot. The states after simulation execution (such as end effector pose, gripping force changes, and action completion time) and the corresponding states after real execution are recorded, and the difference between the two is calculated as the real error label. Model input features include: end effector pose, target object pose, contact force estimates, environmental friction parameters, and action type encoding in the simulation; the output consists of three-dimensional position deviations (Δx, Δy, Δz), attitude angle deviations (Δroll, Δpitch, Δyaw), and temporal offsets (such as action trigger delay). During training, the mean squared error (MSE) loss function is used to optimize the model parameters to ensure generalization to unseen states. During the operational phase, the system inputs the simulated predicted state obtained in S4.2 into the model. The model outputs the prediction error of the corresponding action in the real environment, such as "the Z-axis of the end effector is 2.1 cm high, and the clamping action is delayed by 180 ms". This error is then used to feedforward correct the actual control commands, significantly reducing the risk of execution failure due to inaccurate modeling or environmental uncertainties. Through this mechanism, the system achieves a control upgrade from "passive trial and error" to "active compensation".

[0069] S5: Determine the execution result.

[0070] If the action is executed successfully, the success probability weight of the corresponding state transition change in the graph is increased.

[0071] If an action fails, the success probability weight of the corresponding state transition in the graph is reduced.

[0072] The actual current world state after execution is returned as a feedback signal to step S3 for planning subsequent action instructions, thereby achieving continuous optimization and adaptation of the task decision-making logic.

[0073] Specifically, based on the execution results in the real environment, the state transition relationships in the high-level task graph model are dynamically updated to achieve continuous optimization and adaptation of the task decision logic. In one specific embodiment, after the target robot completes the action execution in S4, the system enters the execution result evaluation and model update stage. First, the system collects the real-world state after execution through the real robot's sensors (such as vision cameras, force sensors, encoders) and compares it with the expected target to determine the action execution result. If the task objective is achieved (e.g., the object is successfully grasped and moved to the target area without collision), it is determined as "execution successful"; if the task is not completed, a collision occurs, the grasping fails, or the safety threshold is exceeded, it is determined as "execution failed". Subsequently, the system updates the state transition edge weights in the high-level task graph model according to the execution result: if the execution is successful, the success probability weight of the corresponding state transition edge in the graph is increased; if the execution fails, its success probability weight is decreased. The specific update method is as follows: locate the state transition edge corresponding to the action in the high-level task graph model, and obtain the state transition edge e. i Success probability weight P i The initial value is set by the simulation success rate of the baseline strategy in S2. After each execution, P is adjusted according to the results using a moving average or exponentially weighted update rule. i For example, upon success: P i ←α·P i +(1-α)·1; In case of failure: P i ←α·P i +(1-α)·0, where α is the forgetting factor, writes the updated weights back to the graph model. The updated success probability weights will be used for path planning in subsequent S3, serving as the basis for edge weights in the graph search algorithm. Simultaneously, the system returns the current real-world state after execution as a feedback signal to the S3 module, serving as the initial state input for the next action decision, thereby achieving continuous optimization and adaptive evolution of the task decision logic. This mechanism enables the system to learn from real-world experience, gradually improving the task success rate in complex environments.

[0074] Furthermore, the criteria for judging the execution result include:

[0075] At least one of the following: mission objective achievement, whether a collision occurred, whether joint torque exceeded limits, and end-effector trajectory stability.

[0076] Specifically, these four items are a minimum complete set of judgments designed systematically based on the three core dimensions of robot task execution: reliability, safety, and performance integrity. Together, they constitute a four-dimensional evaluation system of "function-safety-hardware-quality," none of which can be omitted. Regarding task objective achievement, the specific judgment methods are: for grasping tasks: determining whether the gripping force is established using force sensors and whether the object moves with the end effector using vision; for placement tasks: determining whether the object is located in the target area and in the correct posture; for movement tasks: determining whether the end effector has reached the target pose and remains stable. This is directly related to the "target state node" in S3 and is the semantic basis of closed-loop feedback. Regarding collision occurrence, the specific judgment methods are: detecting contact based on sudden changes in force / torque sensors; detecting the distance to obstacles based on vision or depth cameras; inferring external impact based on abnormal increases in joint current. In simulations, this can be directly obtained through the collision detection API, ensuring the safe operating boundary of the system in a real environment. This is the real-world verification of the "feasibility check" in S4. Whether the joint torque exceeds the limit is determined by real-time reading of the torque values ​​(or current values ​​converted) fed back from each joint encoder, and comparing them with the continuous operating torque and peak torque in the motor specifications. If the torque continuously exceeds the threshold (e.g., >90% of rated torque) for a preset time, it is considered to be out of limit, preventing hardware damage and ensuring long-term operational reliability. It also reflects whether the control strategy is "too aggressive," providing feedback for S2 strategy optimization. End-effector trajectory stability is determined by calculating the positional jitter energy of the end-effector near the target pose.

[0077] , t=1, 2, ..., T,

[0078] Where E represents the position jitter energy of the end effector relative to the target pose within the last K+1 time steps, t is the time step index, representing a discrete time point in the control system, T is the current time, i.e., the time when the action is completed, usually the last control cycle after the action ends; K is the backtracking time window length, representing the k+1 most recent time steps before the end of the action, t=TK, representing the time point k steps backward from the current time T, i.e., the time period from the kth control cycle before the end of the action until the current time T, p t p represents the actual position vector of the end effector at time step t, derived from the robot's forward kinematics or sensor feedback; target The target position vector is the target point (such as the placement point or grab point) that the action is expected to reach. E is the square of the position deviation at time step t, representing the degree of deviation of the end from the target at the current moment. The squared term amplifies the impact of larger deviations. If E exceeds a threshold, it is considered unstable; or the maximum deviation amplitude or frequency component of the trajectory is detected (e.g., FFT analysis to check for high-frequency oscillations) to determine whether it converges to the target tolerance range within a specified time. The dynamic performance of the control strategy is evaluated, providing real-world validation for the "action smoothness reward" in S2 and driving the strategy towards a more robust evolution.

[0079] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A robot control method based on Model-Based and RL, characterized in that, The method includes: S1: Construct a two-layer world model of the target robot to form a Model-Based model for task execution and optimization. The two-layer world model includes a low-level dynamics model and a high-level task graph model. The low-level dynamics model is used to generate a virtual robot that can simulate the physical behavior of the target robot. The high-level task graph model is used to characterize the transition relationship between the various task states of the target robot during task execution. The transition relationship is driven by the actions in the predefined set of task actions. S2: In the simulation environment of the virtual robot, for each task action in the task action set, a neural network policy is trained through a reinforcement learning algorithm to learn the mapping relationship from the current state to the control action command corresponding to the task action. The trained neural network policy is saved as the baseline policy corresponding to the task action for subsequent use in the simulation environment and the real environment to generate the corresponding control action command. S3: Receive the target task, obtain the target robot's body state and surrounding environment state, map the target task to the target state node in the high-level task graph model, determine the above two states as the current world state, run the graph search algorithm to plan the best task action path, and output the first task action in the path in sequence. S4: In the simulation environment of the virtual robot, based on the current world state, the baseline strategy corresponding to the first task action is invoked to generate a simulation control action command. The command is executed in the simulation environment, and the feasibility of its execution process is verified. If the verification passes, the post-execution state predicted by the simulation is input into the pre-trained error prediction model to obtain the predicted error amount. Based on the error amount, the control action command generated by the same baseline strategy in the real environment is compensated and corrected to obtain the corrected real control command, which drives the target robot to execute. The real-world state after execution is returned to step S3 as a feedback signal, and the execution result is monitored. S5: Determine the execution result. If the action is executed successfully, the success probability weight of the corresponding state transition edge in the graph is increased. If an action fails to execute, the success probability weight of the corresponding state transition edge in the graph is reduced. This includes constructing a two-layer world model of the target robot to form a Model-Based model for task execution and optimization, including: The two-layer world model is a composite structure that simultaneously includes a low-level physical dynamics model and a high-level task graph model. The former is used to simulate the real physical behavior of the target robot, while the latter is used to represent task semantics and state transition relationships. Together, the low-level physical dynamics model and the high-level task graph model constitute a complete model-based framework. Constructing the underlying physical dynamics model includes: Obtain the 3D CAD model and bill of materials of the target robot, construct an initial parameter file containing complete physical parameters and topology information, input the initial parameter file into the robot simulation environment, and automatically generate a virtual robot that can simulate the real physical behavior of the target robot; Constructing a high-level task graph model includes: Based on the functional requirements of the target robot, a set of preset task actions are defined to form a task action set. Each task action includes the action name, execution conditions, required input state information and expected output result. Based on the set of task actions, the key stages in the task execution process are abstracted into multiple task state nodes, and each state node represents a discrete semantic stage in the process of the target robot performing the task. For any two task state nodes, if there exists a task action whose execution prerequisite is consistent with the conditions described by the first state node, and the system can reach the state described by the second state node after executing the action, then a directed state transition edge is established between the two state nodes. Each state transition edge is assigned an initial success probability weight and execution priority. The success probability weight is set based on historical execution data or expert experience and is used to characterize the probability of the action being successfully executed in the current world state. All state nodes and established state transition edges are integrated into a directed graph structure to form a high-level task graph model.

2. The robot control method based on Model-Based and RL as described in claim 1, characterized in that, This allows the system to learn the mapping relationship from the current state to the control action command corresponding to the task action, and the trained neural network policy is saved as the baseline policy for the task action, including: (1) In the simulation environment, for each task action in the task action set, according to the pre-set initial state range corresponding to the task action, at least one of the following is randomly generated: joint angle of virtual robot, end effector pose, target object pose, obstacle distribution information and relative distance between robot and target, in order to initialize the simulation environment and obtain the current state. The initial state range is determined by expert experience based on the typical execution scenario of the task action. (2) Input the current state into the neural network strategy to generate control action instructions for completing the target task. Execute the instructions in the simulation environment to drive the virtual robot, obtain the next state after execution, and calculate the corresponding reward value according to the preset reward function. The preset reward function includes task completion reward, distance reward, action smoothness reward, collision penalty and energy consumption penalty. (3) Store the current state, control action instructions, reward value and next state as experience data in the experience replay buffer, and set the next state as the new current state. Return to step (2) and continue execution until the current training round ends. (4) During the training process, a batch of experience data is periodically sampled from the experience replay buffer, the policy gradient is calculated using a deep reinforcement learning algorithm, and the parameters of the neural network policy are updated. (5) Test the execution success rate of the updated neural network strategy in an independent verification environment. When the execution success rate reaches the preset threshold multiple times in a row, stop training and use the current neural network strategy as the baseline strategy output corresponding to the task action, which is used to generate control action instructions based on the input state in subsequent task execution.

3. The robot control method based on Model-Based and RL as described in claim 1, characterized in that, Starting from the current state node and ending at the target state node, the graph search algorithm plans the optimal task action path from the start to the end, and outputs the current action of the first task action on this path, including: (1) Based on the current world state, determine the corresponding task state node in the high-level task graph model as the starting point, and take the target state node mapped by the target task as the ending point. (2) For each state transition edge in the task graph, construct the comprehensive cost of the state transition edge based on its associated execution priority and initial success probability weight; wherein, the execution priority is a static scheduling preference pre-set according to task logic and expert experience when constructing the task graph, and mapped to the corresponding scheduling cost; the initial success probability weight is a priori value set according to simulation verification results or expert experience when training the baseline policy, and converted into the corresponding risk cost; wherein, the comprehensive cost is calculated by weighted summation. The overall cost = α·scheduling cost + β·risk cost, where α and β are preset weighting coefficients; (3) Given a defined starting point and ending point, and combining the calculated comprehensive cost of each side, run a graph search algorithm on the high-level task graph model to find the path with the minimum total cost from the starting point to the ending point, which is the optimal task action path. (4) Extract the first task action from the best task action path and output it as the task action to be executed to trigger the subsequent control execution process based on the baseline strategy; the remaining task actions are kept in the path for subsequent cycles to decide whether to continue execution or replan based on the actual execution situation.

4. The robot control method based on Model-Based and RL as described in claim 1, characterized in that, The corrected control commands are obtained to drive the target robot to execute, including: The simulation control action command is executed in the simulation environment to drive the virtual robot to simulate the execution process of the command, and the state sequence during the execution process is recorded, including joint angle changes, end-effector trajectory, distance to obstacles, and dynamic parameters. Based on the recorded state sequence and the predicted state after execution, determine whether the preset feasibility conditions are met; if they are met, the simulation control action command is deemed feasible, and proceed to the next process. The predicted state after the simulation is completed is input into the pre-trained error prediction model, and the predicted error corresponding to the execution of the action is output. Based on the prediction error, the real control commands generated by the same baseline strategy in the real environment are fed forward to obtain the corrected control commands, which drive the target robot to perform the corrected actions.

5. The robot control method based on Model-Based and RL as described in claim 4, characterized in that, Based on the recorded state sequence and the predicted state after execution, determine whether the preset feasibility conditions are met, including: (1) Obtain the predicted post-execution state after the simulation is completed, as well as the dynamic behavior data recorded during the execution process. The dynamic behavior data includes: joint torque, motion speed, minimum distance to obstacles, and end trajectory stability index. (2) Determine whether the state after the prediction execution meets the task success conditions. If not, it is determined to be infeasible. (3) Determine whether there are situations in the dynamic behavior data that exceed the preset safety threshold, including: collision, joint torque exceeding the limit, speed exceeding the limit, or excessive end trajectory oscillation amplitude. If any of these situations exist, it is determined to be infeasible. (4) Determine whether the execution time of the action exceeds the preset time threshold. If it does, it is determined to be infeasible. (5) If all the judgments in (2), (3) and (4) are satisfied, it is determined to be feasible; otherwise, it is determined to be infeasible, and the corresponding failure type label is generated and fed back to S3 to adjust the task path or action parameters.

6. The robot control method based on Model-Based and RL as described in claim 4, characterized in that, Pre-trained error prediction models include: Pre-trained error prediction models include: By synchronously executing the same task actions in a simulation environment and on a target robot, multiple sets of simulated execution states and corresponding real execution states are collected to form a simulation-real paired dataset. Based on the paired dataset, the difference between the simulated state and the real state for each execution is calculated and used as the real error label; Construct a neural network model, setting its input to the state and task action type after simulation execution, and setting its output to at least one of position deviation, attitude deviation, and timing offset. The neural network model is trained using paired datasets and true error labels to obtain a pre-trained error prediction model. During the runtime phase, the predicted state obtained after the simulation execution is input into the pre-trained error prediction model, and the predicted error corresponding to the current action execution is output.

7. The robot control method based on Model-Based and RL as described in claim 1, characterized in that, The criteria for judging the execution result include: At least one of the following: mission objective achievement, whether a collision occurred, whether joint torque exceeded limits, and end-effector trajectory stability.

Images