An intelligent robot full-body motion training method based on residual motion learning

Abstract

This invention discloses a method for training the whole-body movements of an intelligent robot based on residual action learning. The method includes: acquiring the robot's state and task objective; solving for basic action commands based on a dynamic model; extracting context vectors representing hidden dynamic characteristics from historical sequences using a recurrent neural network; combining these with the current state input to a residual policy network to generate residual action commands; superimposing these two to form candidate actions; performing flexible safety filtering by solving a quadratic programming problem involving control obstacle functions and relaxation variables; outputting the final action command; calculating a composite reward including safety correction penalties after execution; and finally, using the reward and action differences to construct a regularization term to update the network. This invention effectively overcomes the Sim2Real generalization gap, enabling the reinforcement learning model to natively internalize physical boundaries, thus improving the robot's action safety, consistency, and robustness in real physical environments.

Description

Technical Field

[0001] This invention relates to the field of intelligent robot control technology, and in particular to a method for training the whole-body movements of an intelligent robot based on residual motion learning. Background Technology

[0002] In recent years, with the rapid development of embodied intelligence and artificial intelligence technologies, whole-body motion control technology for intelligent robots has become a research hotspot in the field. Existing technologies are mainly divided into two paradigms: traditional control based on dynamic models and deep reinforcement learning based on data-driven approaches. How to integrate the advantages of both to achieve whole-body motion control for robots that combines physical stability and environmental adaptability has become a core development trend in the industry.

[0003] CN120335354A discloses a fault-tolerant control method and system for quadruped robots based on residual learning. This scheme is mainly aimed at fault-tolerant scenarios for quadruped robots. It generates reference joint positions through a body mechanism model, and uses data to drive the model output residual correction to couple and generate control commands. It also designs a reward function and damage parameter identification network adapted to fault-tolerant scenarios. However, this scheme is only applicable to specific fault-tolerant scenarios for quadruped robots and cannot adapt to the general whole-body motion control requirements of humanoid and other redundant degree-of-freedom robots. It does not introduce a temporal history information extraction mechanism, and its adaptive identification capability for environmental and body dynamic deviations is insufficient. At the same time, it lacks explicit motion safety constraints and flexible filtering mechanisms, and does not design a corresponding strategy optimization mechanism for motion correction, resulting in high safety risks and poor stability of reinforcement learning training.

[0004] CN115808931B discloses a motion control method, device, system, equipment, and storage medium for underwater robots. This solution mainly targets the heading control scenario of underwater robots. It outputs control actions through a feedback controller and a reinforcement learning residual controller, which are then weighted and superimposed to achieve motion control. The simulation parameters are randomized to improve the anti-disturbance capability. However, this solution is only for low-dimensional planar control scenarios of underwater robots and does not have the ability to control the whole-body motion of high-dimensional redundant degree-of-freedom robots. The residual controller adopts a full-action output mode, which cannot reduce the ineffective exploration space of reinforcement learning, resulting in low training convergence efficiency. It lacks an adaptive course learning mechanism, resulting in insufficient transfer and generalization ability from simulation to reality. At the same time, it lacks systematic motion safety constraints and supporting training optimization mechanisms, making it difficult to meet the requirements for safe control and stable training of the whole-body motion of robots in high-dimensional continuous motion spaces.

[0005] In addition, existing technologies still have many shortcomings in practical deployment and application. First, purely data-driven reinforcement learning strategies lack explicit physical law constraints, making them prone to generating commands that violate robot physical boundaries or disrupt dynamic equilibrium during the action exploration and execution phases, leading to severe hardware damage and irreversible fall instability. Second, due to the many nonlinear dynamic properties of real physical systems that are difficult to model, existing methods face a significant generalization gap from simulation to reality. Furthermore, single control strategies, lacking deep feature extraction of historical dynamic information, struggle to adaptively perceive implicit dynamic deviations in the current operating environment, resulting in significant performance degradation when transferred to real robots. Moreover, existing fusion control schemes often employ a rigid, one-size-fits-all approach when optimizing action commands, lacking flexible solution mechanisms based on control barrier functions and relaxation variables. This not only disrupts the continuity and smoothness of actions but also makes reinforcement learning inefficient in sample utilization when facing complex safety constraints, failing to meet the stable training requirements in high-dimensional continuous action spaces. Summary of the Invention

[0006] The purpose of this section is to outline some aspects of embodiments of the present invention and to briefly describe some preferred embodiments. Simplifications or omissions may be made in this section, as well as in the abstract and title of this application, to avoid obscuring the purpose of these documents; however, such simplifications or omissions should not be construed as limiting the scope of the invention.

[0007] In view of the aforementioned existing problems, this invention is proposed. Therefore, this invention provides a method for training the whole-body movements of an intelligent robot based on residual motion learning, to solve the problems mentioned in the background art.

[0008] To solve the above-mentioned technical problems, the present invention provides the following technical solution: a method for training the whole-body movements of an intelligent robot based on residual motion learning, comprising:

[0009] Obtain the current state information and task objective of the intelligent robot, and calculate a basic action command based on the dynamic model of the intelligent robot and the task objective;

[0010] The current state information and a context vector generated based on the historical information sequence are input into a residual policy network to generate a residual action instruction;

[0011] The basic action instruction is combined with the residual action instruction to form a candidate action instruction. For the candidate action instruction, an optimization problem with preset safety constraints is solved to generate a final action instruction that satisfies the safety constraints.

[0012] The final action command is applied to the intelligent robot, and a reward signal is obtained based on the state changes of the intelligent robot and the completion of the task;

[0013] The parameters of the residual policy network are updated using the reward signal, state information, and the difference between the final action instruction and the candidate action instruction.

[0014] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0015] 1. This invention extracts historical state, action, and reward sequences from a recurrent neural network encoder to generate a context vector containing hidden dynamic features, guiding the residual policy network to generate action compensation. This mechanism enables the model to implicitly identify and compensate for unmodeled nonlinear dynamic errors in the real world, such as friction and wear, and load variations, in a data-driven manner. Furthermore, a learning mechanism that progresses from easy to difficult establishes a smooth generalization path from simulation training to real-world physical deployment, avoiding the significant performance degradation that occurs during traditional model transfer.

[0016] 2. This invention combines the basic movements generated by traditional whole-body control with the residual movements generated by reinforcement learning, thus reducing the ineffective exploration space of the agent in reinforcement learning. Based on this, a quadratic programming safety filter layer based on control obstacle functions and relaxation variables is introduced. Complex physical constraints are transformed into linear inequalities, which not only avoids robot self-collision, exceeding limits, and instability / falling, but also achieves flexible solutions through relaxation variables when facing extreme constraint deadlocks. This abandons the traditional one-size-fits-all hard truncation method, ensuring the robustness of the control program and the smooth continuity of robot movements.

[0017] 3. Furthermore, this invention addresses the gradient divergence problem in traditional reinforcement learning caused by external security controllers tampering with actions by reconstructing the training mechanism. By introducing a composite penalty reward proportional to the underlying correction amount and relaxation variables, and adding a behavior cloning regularization term to the loss function of the Actor network, the control system can utilize the differences in security instructions as expert demonstrations to guide network updates. Moreover, combined with a priority experience replay mechanism, the black-box model is forced to learn and converge within the physical safety set, ultimately achieving the effect that the network's native output is safe, thus improving the sample utilization and convergence stability of reinforcement learning. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein:

[0019] Figure 1 This is a flowchart illustrating the overall process of a method for training the whole-body motion of an intelligent robot based on residual motion learning, as described in one embodiment of the present invention. Detailed Implementation

[0020] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of the present invention.

[0021] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0022] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0023] Furthermore, in the description of this invention, it should be noted that the terms "upper," "lower," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. These terms are used solely for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. In addition, the terms "first," "second," or "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0024] Example 1

[0025] Reference Figure 1 This is the first embodiment of the present invention, which provides a method for training the whole-body movements of an intelligent robot based on residual motion learning, including:

[0026] S1. Obtain the current state information and task objective of the intelligent robot, and calculate a basic action instruction based on the dynamic model of the intelligent robot and the task objective.

[0027] It should be noted that, since the motion of intelligent robots in the real physical world is a typical high-dimensional nonlinear control problem, relying entirely on model-free reinforcement learning to directly output actions can easily lead to unexpected action mutations or dangerous postures that violate physical laws. Therefore, in this invention, we adopt the traditional Whole-Body Control (WBC) theory as a foundation to construct a basic action baseline that conforms to physical priors.

[0028] Furthermore, the current state information of the intelligent robot is obtained (in this embodiment, this current state information includes the joint angles of each joint of the robot). Joint angular velocity The system includes the global pose of the base and the task objectives issued by higher levels (which can be the target centroid trajectory or the end effector trajectory). The whole-body control task of the intelligent robot is decomposed into multiple sub-tasks with different priorities. Since humanoid or legged robots are typical redundant degree-of-freedom systems, in this embodiment, the priorities are set from high to low as follows: the first priority is the contact stability task (to ensure that the foot or support surface maintains stable contact with the ground and does not slip or lift off the ground), the second priority is the centroid motion task (to track the target trajectory and maintain macroscopic balance), and the third priority is the joint posture task (to make the robot tend to a natural and relaxed default posture and avoid approaching the physical limits of the joints).

[0029] Furthermore, for the aforementioned multi-priority tasks, the null space projection method is employed to perform a rigorous hierarchical solution, ensuring that the execution of low-priority tasks will never interfere with the achievement of high-priority tasks. For any given... The subtask of the level, with its expected joint space acceleration. The iterative calculation formula is as follows:

[0030]

[0031] in, This indicates the priority level of the task (in this embodiment, ). Indicates before satisfaction Under high-priority task constraints, the cumulative joint acceleration commands have already been calculated; in particular, when hour, . Indicates the first The Jacobian matrix of each subtask has the physical meaning of establishing a velocity mapping relationship from the robot joint space to the specific task space (such as the centroid coordinate space). Indicates the preceding The null projection matrix of a combination of high-priority tasks has the effect of filtering, that is, restricting the action instructions of the current task to a subspace that will not affect the high-priority tasks, thereby achieving safe decoupling. The Moore-Ponous pseudoinverse of the matrix is ​​used to find the least squares norm solution under redundant degrees of freedom, ensuring the energy optimality of the action. Indicates the first The expected acceleration of each subtask in the task space can typically be generated by combining a typical proportional-derivative (PD) control law with the task objective, i.e. , For the mission objective, This is the gain coefficient. The product of the derivative of the Jacobian matrix and the joint velocity represents the compensation for the Coriolis / eccentric acceleration term caused by the robot's motion.

[0032] It should be noted that by iteratively solving the above formula, the final result can be obtained. This refers to the globally desired joint acceleration that satisfies all levels of constraints.

[0033] Simultaneously, after obtaining the global desired joint acceleration, based on the rigid body inverse dynamics model of the intelligent robot, the joint driving torque required to achieve this acceleration is calculated, which is the basic motion command (denoted as ). Its inverse dynamic equation is expressed as follows:

[0034]

[0035] in, This is the generalized inertial matrix of the robot, reflecting the robot's mass distribution. The nonlinear mechanical behavior of the robot during high-speed coupled motion of its joints is demonstrated by the Coriolis force and centrifugal force matrices. This is the gravity vector. The selection matrix is ​​used to distinguish between the robot's passive degrees of freedom (e.g., floating base) and actively driven joints. It is the product of the Jacobian transpose of the contact force and the actual ground reaction force, used to characterize the supporting effect of the external environment on the robot.

[0036] It should be noted that by solving for the aforementioned basic motion commands, the robot can be endowed with basic balance and motion capabilities, thereby narrowing the exploration space of artificial intelligence algorithms. However, in practical applications, due to factors such as manufacturing and assembly errors, frictional wear, and elastic deformation of linkages, parameters such as the generalized inertia matrix, Coriolis force, and centrifugal force matrix in the simulation environment often exhibit dynamic deviations from real physical laws that are difficult to analytically model (Sim2Real). Therefore, based on this deficiency, it is necessary to introduce a residual policy network based on context vectors in the subsequent steps to specifically learn and compensate for the unmodeled dynamic errors in the above equations in a data-driven manner.

[0037] S2. Input the current state information and a context vector generated based on the historical information sequence into a residual policy network to generate a residual action instruction.

[0038] Specifically, as mentioned above, because real physical environments contain unmodeled dynamic characteristics such as joint friction and wear, changes in external load, motor response delays, and changes in ground material, the basic motion commands calculated under the ideal model will deviate during actual execution. For intelligent robots, these dynamic physical parameters are often not directly observable (i.e., the robot control system exhibits a partially observable Markov decision process, POMDP). Furthermore, since the current state at a single time step cannot reflect these hidden dynamic characteristics, this invention introduces a temporal representation learning mechanism based on historical information.

[0039] Furthermore, during the robot's operation, the control system collects interaction data from the robot over a past time window (e.g., data from the past 50 control cycles) at a preset fixed frequency (usually a control cycle). This collected interaction data constructs a historical information sequence. This historical information sequence contains the observed state at each time step. Actual actions performed and reward signals obtained from the environment .

[0040] It should be noted that by introducing historical information sequences, the physical resistance of the current environment can be indirectly inferred. For example, if a large driving torque (action) was given in the past, but the joint angular velocity (state) changed slowly, then the historical information sequence contains implicit information that the current environment has high friction or additional load.

[0041] Furthermore, the aforementioned historical information sequence is used as input and fed into a Recurrent Neural Network Encoder (RNN). In this embodiment, to address the vanishing gradient problem that may occur during training of the historical information sequence, the RNN encoder preferably employs a Gated Recurrent Unit (GRU) or a Long Short-Term Memory (LSTM) network. At each time step... (in, For the current time step, Within a window of time spanning the past period, the update formula for the hidden state inside the RNN encoder can be expressed as follows:

[0042]

[0043] in, Represented as a first A concatenated vector of the state, action, and reward of a step. This is the hidden state from the previous time step. These are the learnable weight parameters in the RNN encoder. It is a non-linearly activated cyclic unit mapping function.

[0044] It should be noted that after the entire historical window of data has been traversed, the hidden state output by the RNN encoder at the last moment is a context vector, denoted as... The context vector is a low-dimensional, continuous vector in a latent feature space that represents the implicit contextual information of the intelligent robot's current physical environment or its own dynamic characteristics.

[0045] Furthermore, the current state information of the robot will be obtained. The extracted context vector is concatenated with its features and used as input to a residual policy network (a multilayer perceptron (MLP) structure). This residual policy network then outputs residual action instructions. The output of these instructions is a mapping operation, expressed as:

[0046]

[0047] in, For the network parameters of the residual policy network, This is the output of the compensated joint torque or the desired compensated position.

[0048] It is important to note that at this point, the compensation joint torque or compensation desired position is no longer a blind full control command, but a precise fine-tuning for dynamic errors (such as friction compensation torque) that cannot be covered in S1.

[0049] Specifically, in a particular application scenario of this embodiment, the model's input, output, and hierarchical structure are specifically set as follows:

[0050] The RNN encoder employs a two-layer GRU network with 128 and 64 hidden layer nodes, respectively. The residual policy network uses a three-layer fully connected multilayer perceptron (MLP) structure. Its input layer dimension is the sum of the concatenation vector dimension and the context vector dimension at each time step input to the RNN encoder. The hidden layer contains three fully connected layers with non-linear activation functions (either ELU or ReLU), with the number of nodes configured sequentially as [256, 128, 64]. The output layer uses the Tanh activation function, and the output dimension strictly corresponds to the number of degrees of freedom of the robot, i.e., outputting a residual action command vector with the same dimension as the number of degrees of freedom.

[0051] Meanwhile, in order to enable the RNN encoder to truly learn to identify physical properties from historical data and to enable the residual policy network to generalize in the real world, the present invention jointly trains the recurrent neural network encoder and the residual policy network in a simulation environment and designs a learning mechanism from easy to difficult.

[0052] Specifically, firstly, a set of physical parameters that need to be randomized is defined in the simulation environment. (For example, the robot's link mass, joint friction coefficient, sensor noise, and system latency). Before each training iteration begins, the data is processed from a uniformly distributed... A set of parameters is sampled to configure the simulator, where, , indicating uniform distribution These are the default standard parameters. The randomization range is then defined. Subsequently, the control performance of the residual policy network within the current parameter range is monitored in real time by the control system (in this embodiment, the control performance is measured by the average cumulative reward value of recent iterations). The randomization range of the parameters is automatically adjusted based on the performance. The formula for adjusting the strategy is as follows:

[0053]

[0054] in, This is the adjusted randomization range. To expand the range step size. This represents the average cumulative reward value. This is a preset performance threshold.

[0055] It should be noted that in the early stages of training, the randomization range is small, the simulation environment is close to the standard model, and the task difficulty is low, enabling the robot to converge quickly and learn basic balance. However, as the network's capabilities improve (i.e., the average cumulative reward value exceeds the preset performance threshold), the control system automatically and gradually expands the random range of physical parameters and external disturbances, thereby achieving automatic course learning from easy to difficult. The advantage of this is that it avoids the problem of reinforcement learning non-convergence caused by an overly complex initial environment, while forcing the RNN encoder to keenly capture subtle differences in historical information sequences, thereby enhancing the ability of the aforementioned context vector to recognize changes in the surrounding environment, thus achieving generalization from simulation training to real deployment.

[0056] S3. Combine the basic action instruction with the residual action instruction to form a candidate action instruction. For the candidate action instruction, generate a final action instruction that satisfies the safety constraints by solving an optimization problem with preset safety constraints.

[0057] It should be noted that although the generated residual motion commands can effectively compensate for the unmodeled dynamic errors caused by Sim2Real, the residual policy network is essentially a data-driven black-box model in this field, and it still risks outputting uncontrollable actions in complex generalization environments. If the basic motion commands and residual motion commands are simply superimposed and directly output to the robot's underlying actuator, sudden changes in motion that violate physical limits can easily occur (e.g., a sudden output of extremely large torque causing the motor to burn out, or the robot deviating from the support polygon and falling). Traditional methods usually use hard truncation, but this not only disrupts the continuity of the motion but also causes gradient breakage during reinforcement learning training. To address this problem, the present invention introduces a safety filtering layer based on Control Barrier Functions (CBF) between the candidate motion commands and the physical actuator. This filtering layer mainly performs a flexible safety projection of the motion by solving a Quadratic Programming (QP) optimization problem.

[0058] Furthermore, the basic motion instructions obtained in S1 are linearly superimposed with the residual motion instructions obtained in S2 to form candidate motion instructions that have not undergone safety verification. :

[0059]

[0060] in, is the scaling factor for the residual action, used to adjust the degree of reinforcement learning's intervention on the traditional control baseline.

[0061] Furthermore, based on the electromechanical characteristics and physical laws of the intelligent robot, several physical safety boundary conditions (i.e., safety sets) are preset. In this embodiment, these safety boundary conditions include at least the following three items:

[0062] Hardware limits: The position, speed, and torque of each joint of the intelligent robot must be strictly limited to the maximum / minimum values ​​specified in its motor datasheet.

[0063] Self-collision avoidance constraint: The shortest spatial distance between the robot's body parts (e.g., the left arm and the torso) must be greater than or equal to a preset safety margin, that is, self-collision will not occur.

[0064] Dynamic balance constraint: The predicted zero-moment point coordinates of the robot during movement must always be located within the reserved safety area inside the support polygon formed by the contact between its feet and the ground, in order to ensure the robot's overall dynamic anti-tipping capability.

[0065] Furthermore, since most of the aforementioned safety boundary conditions concern the robot's state space... Nonlinear geometric constraints (denoted as) To achieve rapid solution, the present invention is based on the fundamental principle of control obstacle function, transforming these state constraints into linear inequality constraints regarding the final action command.

[0066] Specifically, in this embodiment, taking any safety boundary condition as an example, in order to ensure that the state of the control system never leaves the safety set, the following control obstacle inequality must be satisfied:

[0067]

[0068] Furthermore, since the dynamics in a control system are often a control affine system, by calculating the Lie derivative, the above inequality can be expanded and rewritten as relating to the final action command. The standard form of the linear inequality:

[0069]

[0070] in, The real-valued, continuously differentiable function defining the safety boundary (in this embodiment, the function can be defined as a self-collision distance) .in, This represents the shortest spatial distance between the various components of the robot's body. (This is a preset safety margin). It is a monotonically increasing extended class function (in this embodiment, the function is a simplified positive constant gain used to characterize the buffer speed of the control system as it approaches the safety boundary). This is the constraint Jacobian matrix associated with the current state of the control system. This is the boundary vector that includes the safety margin and the nonlinear drift term.

[0071] It should be noted that, through the above transformation, the present invention can map complex physical security issues into a single set of purely linear mathematical inequalities.

[0072] Meanwhile, in reality, some surrounding environments may experience extreme disturbances, causing conflicts between the aforementioned safety boundary conditions at a specific time step (for example, to maintain the robot's predicted zero-moment point during movement, instantaneous acceleration is necessary, violating the speed limit constraint). In such cases, directly applying a rigid solution will likely result in no solution for the optimizer, leading to control program crashes. To address this issue, this invention introduces a relaxation variable into the safety constraints of the control system when constructing the quadratic programming problem.

[0073] Specifically, the objective function and constraints of this quadratic programming optimization problem are constructed as follows:

[0074] The objective function of the quadratic programming optimization problem is:

[0075]

[0076] Constraints of the quadratic programming optimization problem:

[0077]

[0078]

[0079] in, The final action command that needs to be solved and satisfies the preset safety constraints. The diagonal weight matrix is ​​positive definite and is used to penalize the deviation between the joint action command and the candidate command. The first term of the objective function aims to minimize the difference between the final action command and the output candidate action command, that is, to preserve the action intention explored by reinforcement learning while ensuring absolute safety. It is an introduced slack variable, which is represented as a scalar or a vector with the same dimension as the constraint. It is a vector consisting entirely of 1s. The penalty coefficient for the slack variables (in this embodiment, this coefficient is set to be higher than the positive definite diagonal weight matrix). (on an order of magnitude). The second term of the objective function is the added penalty term. These represent the upper and lower limits of the motor's absolute physical output.

[0080] It is important to emphasize that, under most normal circumstances, the optimizer will force the slack variables to equal zero in order to minimize the total cost. In this case, the final action command obtained satisfies all physical boundary conditions. However, when multiple safety boundary conditions lead to irreconcilable deadlock conflicts, the extremely large penalty coefficient will force the slack variables to take a very small non-zero positive value. In other words, the control system can be allowed to flexibly break through some soft constraints at a cost that minimizes the degree of violation, thus ensuring that the quadratic programming solver always has a solution. This not only avoids the control jitter caused by traditional hard truncation methods but also ensures the robustness of the control system program under extreme physical environments. Furthermore, the final action command obtained is the final action command.

[0081] S4. Apply the final action command to the intelligent robot and obtain reward signals based on the changes in the intelligent robot's state and the completion of the task.

[0082] Furthermore, in traditional reinforcement learning, reward signals typically focus only on the task objective (e.g., the robot's movement speed). However, in this invention, because we introduce a safety filtering layer based on a control obstacle function, without targeted guidance for reinforcement learning, the residual policy network is prone to inertia. That is, the residual policy network will arbitrarily output extremely large or dangerous candidate action commands, relying entirely on the underlying quadratic programming optimizer for hard correction. This not only greatly increases the computational burden on the underlying controller but also causes the robot to move stiffly and frequently trigger motor current limiting during actual physical execution. To solve this problem, this invention designs a composite reward function that includes a safety intervention penalty term.

[0083] Furthermore, the final motion commands that satisfy the absolute safety constraints (which can be the target driving torque or position expectation of each joint) are obtained from the solution and sent to the joint motor drivers of the intelligent robot for execution via the underlying communication bus (e.g., EtherCAT or CAN bus). After the current control cycle ends, the robot's current state is collected and updated by the robot's onboard sensors (e.g., joint encoders, IMU inertial measurement units, etc.), completing the Markov state transition from the current state to the next state.

[0084] Furthermore, positive base rewards are calculated based on the changes in the state of the intelligent robot and the completion of high-level tasks.

[0085] Specifically, in this embodiment, taking a centroid velocity tracking task as an example, the mathematical expression of its basic reward can be set as follows:

[0086]

[0087] in, This represents the robot's current actual center-of-mass velocity vector. The expected centroid velocity vector issued by the high-level task. This is the sensitivity scaling factor for velocity tracking error. This is the survival reward constant (equivalent to giving a fixed reward as long as the robot does not fall). This constant is mainly used to encourage the robot's movements to conform to the set motion target as much as possible.

[0088] In addition, to force the residual policy network to internally learn the physical security boundary, this invention also introduces a negative penalty term proportional to the instruction difference. Its formula is defined as:

[0089]

[0090] in, The correction penalty weight coefficient (usually a large positive number). Let L2 norm be the square of a vector. This represents the amount of correction made by the underlying quadratic programming optimizer to the candidate action instructions output by the residual policy network.

[0091] It should be noted that when the candidate action instructions output by the residual policy network are already within the safe set, the quadratic programming optimizer does not need to intervene. In this case, the final action instruction equals the candidate action instruction, and the penalty term is 0. However, when the residual policy network outputs a dangerous action, the quadratic programming optimizer is forced to project it back onto the safe boundary, resulting in a large difference in the correction amount. By feeding this difference back to the network as a negative reward, the action distribution of the residual policy network can be implicitly forced to gradually converge towards the safe set, ultimately achieving the effect that the native output of the residual policy network is safe, thereby reducing the intervention frequency of the underlying controller.

[0092] Furthermore, when faced with extreme physical constraint conflicts, the quadratic programming optimizer will enable slack variables to ensure a solution. This indicates that the robot's current physical state is extremely dangerous, and even the safety filter layer cannot cover all safety boundary conditions. Therefore, the most severe penalty must be imposed on this situation. In this example, we also define a slack variable penalty term for this penalty. :

[0093]

[0094] in, The maximum relaxation penalty weight coefficient ( ). For the first The value of the slack variable introduced by the safety constraint.

[0095] It should be noted that the slack variable penalty term is proportional to the magnitude of the non-zero value of the slack variable. The purpose of this setting is to warn the agent in reinforcement learning that it must plan its action trajectory in advance in order to avoid entering states that cannot be avoided even if all efforts are made (i.e., unrecoverable sets).

[0096] Furthermore, the control system can obtain a global reward signal for updating the residual policy network parameters by linearly adding the above items.

[0097] It should be noted that by using a composite reward mechanism that controls the degree of intervention of the obstacle function, the present invention can retain the exploration ability of reinforcement learning while using the underlying mechanical constraints to reverse shape the gradient of the residual policy network, thereby solving the problem of abrupt changes in robot actions when pure data-driven generalization is applied to the real physical world.

[0098] S5. Update the parameters of the residual policy network by utilizing the reward signal, state information, and the difference between the final action instruction and the candidate action instructions.

[0099] Furthermore, traditional reinforcement learning faces the problem of action mapping mismatch when dealing with actions tampered with by external security controllers (especially the quadratic programming optimizer in this invention). That is, the residual policy network believes it has output action A and received a reward, but the environment actually executes the filtered action B. This causes a significant deviation in the direction of residual policy gradient calculation, leading to training divergence. To address this issue and accelerate the internalization of security boundary conditions by the residual policy network, this invention combines the fundamental principles of maximum entropy reinforcement learning and Prioritized Experience Replay (PER) to design a network update method that includes behavior cloning regularization.

[0100] Furthermore, since the parameters of the residual policy network are randomly distributed in the initial state, directly allowing it to participate in control would result in highly destructive residual motion commands. Therefore, in the early stages of training (i.e., the warm-up phase), the control system temporarily disables the residual policy network (forcing the output of compensated joint torques or compensated desired positions to be equal to 0). At this time, the intelligent robot relies solely on the basic motion commands solved based on the dynamic model in S1 for execution. Simultaneously, the control system records the interaction sequence during this phase, constructing an initial set of experience data. And store it in the experience replay pool In the middle. Among them, This is an enhanced observation feature that incorporates robot state information and context vectors.

[0101] It should be noted that by laying a batch of safe underlying baseline data in the experience replay pool, the risk of hardware damage caused by blind exploration during the cold start phase of reinforcement learning can be avoided.

[0102] Furthermore, after the warm-up period, the residual strategy network will be formally integrated into the control loop. In each control cycle, the control system will process the candidate action commands output by the residual strategy network, the final action command filtered by the control obstacle function and quadratic programming, the global reward signal from environmental feedback, and the state transition results. Packaged into a complete experience tuple Stored in the experience replay pool.

[0103] Meanwhile, during this period, in order to improve sample utilization, the control system needs to calculate the time difference error for each empirical data point, denoted as... The formula for calculating this time difference error is as follows:

[0104]

[0105] in, A dual Critic network is used to evaluate the value of the current action (to prevent overestimation of Q-value). These are the parameters for the Critic network. This is the corresponding target Critic network, used to stabilize the training objective. The reward discount factor (in this embodiment, the value is 0.99) is used to characterize the degree of importance attached to future long-term returns. The action predicted by the target network based on the observed state transition results at the next time step.

[0106] It should be noted that the absolute value of the time difference error directly reflects the degree of surprise between the actual situation and the Critic network's expectations.

[0107] Furthermore, based on the calculated time difference error, the control system sets a sampling priority for each experience in the experience playback pool. (in, To prevent the use of tiny positive numbers with a priority of 0, the control system performs non-uniform sampling (drawing a batch of data) from the empirical replay pool according to the following probability distribution each time the Critic network parameters are updated:

[0108]

[0109] in, It is the first in the experience replay pool A number of empirical data points were extracted for non-uniform sampling probabilities used in updating the Critic network parameters. The priority index determines the degree of skewness in priority sampling.

[0110] It should be noted that when the underlying quadratic programming optimizer significantly intervenes in the neural network's actions to ensure safety, it often generates extremely high penalty rewards, leading to very high temporal difference errors. Through the aforementioned non-uniform sampling operation, the Critic network can be forced to frequently recall these dangerous edge states that trigger safety interventions, thereby improving the learning efficiency of dangerous regions and reducing repeated errors.

[0111] Furthermore, for the sampled batch data, this embodiment uses the classic gradient descent method to update the neural network encoder.

[0112] Specifically, firstly, the parameters of the dual Critic network are updated to minimize the value prediction error. The loss function used is the standard mean squared error (MSE).

[0113]

[0114] in, This represents the mathematical expectation of tuples that follow an experience replay pool distribution. It is the second dual Critic network. The loss function for each parameter of the Critic network.

[0115] Next, update the residual policy network (i.e., the Actor network, whose parameters are denoted as...). To address the aforementioned action mapping mismatch problem, this invention utilizes the difference between the retained candidate action instructions and the final action instructions to explicitly add a behavior cloning regularization during policy gradient updates. The final loss function of the residual policy network... The structure is as follows:

[0116]

[0117] in, This represents the mathematical expectation of observed features that follow an empirical replay pool distribution. It is a candidate action instruction generated in real time under the latest residual strategy network parameters. is the entropy temperature coefficient, used to control the randomness of reinforcement learning exploration. The first term of the formula is the policy improvement objective of standard reinforcement learning, which aims to maximize the expected gains of the environment (i.e., make the robot's actions more efficient and intelligent). The second term of the formula is the action difference constraint term. For safety regularization weight coefficients. The purpose of this action difference constraint is to treat the final action instruction obtained by the underlying quadratic programming optimizer as an expert, and to force the residual policy network to mimic the decision logic of the safety filter by minimizing the Euclidean distance between the candidate action instructions generated in real time and the final action instruction.

[0118] Furthermore, after calculating the final loss function, the Adam optimizer is used to calculate the gradient of the loss function for the residual policy network. gradient of the loss function of the dual Critic network The parameters of the Critic network and the residual policy network are updated along the gradient descent direction. Then, a soft update mechanism is used to slowly synchronize the current network parameters to the target network, ensuring that the dynamic characteristics of the entire training loop tend to stabilize and converge. Finally, through environmental interaction and reverse correction of the underlying physical boundaries, the dynamic laws are internalized, enabling the original candidate action commands, which have not undergone secondary planning and correction, to perfectly match the final executed action commands. This achieves the effect of shifting from relying on external intervention to ensure safety to network-native decision-making that ensures safety.

[0119] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for training the whole-body movements of an intelligent robot based on residual motion learning, characterized in that, include: Obtain the current state information and task objective of the intelligent robot, and calculate a basic action command based on the dynamic model of the intelligent robot and the task objective; The current state information and a context vector generated based on the historical information sequence are input into a residual policy network to generate a residual action instruction; The process of generating the context vector includes: Collect historical information sequences of the robot's state, actions, and rewards over a past period of time; The historical information sequence is input into a recurrent neural network encoder, which extracts and outputs a situation vector representing the current physical environment or dynamic characteristics of the intelligent robot. The training process for the residual policy network and the recurrent neural network encoder includes: In the simulation environment, the physical parameters and external disturbances of the intelligent robot are randomized. Based on the performance of the residual policy network under different randomization parameters, the range of parameters in the randomization process is automatically adjusted to form a training course from easy to difficult, which is used to enhance the recurrent neural network encoder's ability to recognize environmental changes and the generalization ability of the residual policy network. The basic action instruction is combined with the residual action instruction to form a candidate action instruction. For the candidate action instruction, an optimization problem with preset safety constraints is solved to generate a final action instruction that satisfies the safety constraints. The optimization problem to be solved is a quadratic programming problem, and the objective function of the problem is to minimize the difference between the final action command and the candidate action commands. When constructing the quadratic programming problem, a slack variable is introduced for at least one security constraint, and a penalty term for the slack variable is added to the objective function to ensure that a solution that minimizes the degree of constraint violation can still be obtained when the security constraints cannot be satisfied simultaneously. The final action command is applied to the intelligent robot, and a reward signal is obtained based on the state changes of the intelligent robot and the completion of the task; The parameters of the residual policy network are updated using the reward signal, state information, and the difference between the final action instruction and the candidate action instruction.

2. The method for training the whole-body movements of an intelligent robot based on residual motion learning as described in claim 1, characterized in that, Calculating the basic action command includes: The whole-body control task of the intelligent robot is decomposed into multiple sub-tasks with different priorities. The sub-tasks include contact stability task, center of mass movement task, and joint posture task. The null-space projection method is used to solve each subtask in descending order of priority, calculate the desired joint acceleration that satisfies all task constraints, and calculate the basic motion commands required to achieve the desired joint acceleration based on the inverse dynamics model.

3. The method for training the whole-body movements of an intelligent robot based on residual motion learning as described in claim 1, characterized in that, The pre-defined safety constraints in the quadratic programming problem include: It is formed by transforming the physical safety boundary conditions of the intelligent robot into a set of linear inequalities with respect to the final action command, and the transformation process is based on the principle of control barrier function.

4. The method for training the whole-body motion of an intelligent robot based on residual motion learning as described in claim 3, characterized in that, The physical security boundary conditions include at least one or more of the following combinations: The position, speed, or torque of each joint of the intelligent robot does not exceed its limit value; The various parts of the intelligent robot's body do not collide with each other. The zero-moment point predicted by the intelligent robot during movement is always located within the safe area reserved inside the polygon supporting its feet.

5. The method for training the whole-body movements of an intelligent robot based on residual motion learning as described in claim 1, characterized in that, The reward signal includes: A negative penalty item; The magnitude of the penalty term is proportional to the amount of correction between the final action instruction and the candidate action instruction, and is also proportional to the magnitude of the non-zero value of the slack variable.

6. The method for training the whole-body movements of an intelligent robot based on residual motion learning as described in claim 1, characterized in that, The method further includes an initialization and sampling step, which includes: In the early stages of training, initial experience data is collected and stored in the experience replay pool by only executing actions driven by the basic motion instructions. During training, the sampling priority of each empirical data point is determined based on its temporal difference error, and the data in the empirical replay pool is non-uniformly sampled based on the sampling priority for use in updating the parameters of the residual policy network.

Images