Robust motion tracking training method for humanoid robot

Abstract

This invention relates to the field of robotics, and specifically provides a robust motion tracking training method for humanoid robots, comprising the following steps: S1, acquiring a reference motion trajectory; S2, acquiring the robot's current state in the training environment; S3, calculating the relative observables of the reference posture; S4, constructing input data for a policy network; S5, setting a recovery state; S6, applying a recovery assistance force; S7, calculating a recovery success index; S8, updating the assistance force magnitude; S9, performing reinforcement learning optimization; and S10, deploying the trained policy. This invention reduces heading drift and posture tracking errors, dynamically adjusts the assistance force magnitude based on actual recovery success during the robot's recovery training phase, avoids training instability or assistance force dependence caused by fixed annealing, improves the motion recovery capability of humanoid robots under complex movements, disturbed states, and conditions with increased tracking errors, and enhances the motion tracking stability of the trained policy under conditions of no or low assistance force.

Description

Technical Field

[0001] This invention relates to the field of robotics, and in particular to a robust motion tracking training method for humanoid robots. Background Technology

[0002] When humanoid robots perform complex full-body movements such as dancing, walking, turning, squatting, and arm swinging, they typically need to undergo imitation learning or reinforcement learning training based on pre-given reference motion trajectories to enable the robot to reproduce the reference actions in simulated or real-world environments. Existing humanoid robot motion tracking methods usually use the robot's own joint states, angular velocities, gravity direction projections, historical actions, and joint positions in the reference trajectory as inputs to a policy network, which then outputs control commands for each joint of the robot.

[0003] Current humanoid robot motion tracking methods still have the following problems in actual training and deployment due to technological limitations. (1): When relying solely on local observations such as the robot's own gravity direction and joint state, the robot's perception of the overall posture and heading posture in the reference trajectory is insufficient, which easily leads to accumulated errors in the heading angle, resulting in problems such as yaw drift, inconsistent orientation, or overall rotational deviation of the robot during execution. (2): For humanoid robots that have fallen, partially fallen, or whose initial state deviates from the reference trajectory or has a large tracking error, external auxiliary forces or restoring forces are usually used to help the robot reapproach the reference trajectory. However, the magnitude of the auxiliary force is often determined by annealing with a fixed number of training iterations or annealing with a preset timetable, which fails to fully reflect the robot's actual recovery ability in different training stages and different environmental instances. (3) Fixed annealing may lead to two types of problems: first, the auxiliary force is reduced too early before the strategy has the ability to recover, making the training unstable; second, the auxiliary force is still too large after the strategy has the ability to recover, making the strategy dependent on external assistance for a long time and reducing the deployment stability under unassisted conditions. (4) It is not adaptable to complex reference actions, external disturbances, initial state deviations and simulation-to-simulation or simulation-to-real migration scenarios, which can easily lead to problems such as increased tracking error, slow recovery, discontinuous actions or stagnation of performance improvement in the later stage of training. Considering the above factors, it is necessary to provide a new training method so that humanoid robots can perceive the reference motion posture more accurately and dynamically adjust the auxiliary force according to the recovery effect, thereby improving the stability and anti-disturbance recovery ability of complex motion tracking. Summary of the Invention

[0004] To overcome the shortcomings of existing humanoid robot motion tracking methods, which suffer from insufficient heading and attitude perception, easy accumulation of reference motion tracking errors, reliance on fixed auxiliary force annealing for recovery training, and insufficient disturbance recovery capability due to technological limitations, this invention provides a robust motion tracking training method for humanoid robots. This method can directly perceive the relative relationship between the robot's current posture and the reference trajectory posture through a policy network, thereby reducing heading drift and attitude tracking errors. It can also dynamically adjust the auxiliary force based on the actual recovery success during the robot's recovery training phase, avoiding training instability or auxiliary force dependence caused by fixed annealing. This improves the motion recovery capability of humanoid robots under complex movements, disturbance states, and increased tracking errors, and enhances the motion tracking stability of the post-trained policy under conditions of no or low auxiliary force.

[0005] The technical solution adopted by this invention to solve its technical problem is: A robust motion tracking training method for a humanoid robot includes the following steps: S1, acquiring a reference motion trajectory; S2, acquiring the robot's current state in the training environment; S3, calculating the relative attitude observables of the reference posture, which are obtained by calculating the relative attitude relationship between the reference anchor point posture in the reference trajectory and the robot's current anchor point posture; S4, constructing input data for a policy network, which is formed by combining the relative attitude observables of the reference posture with the robot's current state, historical states, and reference motion commands. S5. Set Recovery State: During training, determine whether the robot is in a recovery state based on the current tracking error or initial state sampling results. S6. Apply Recovery Assist Force: When the robot is in a recovery state, calculate the direction of the assist force based on the deviation between the robot's current anchor point position and the reference anchor point position. S7. Calculate Recovery Success Index: Calculate the recovery success index at the end of each training round, or when the robot switches from the recovery state to the normal tracking state. S8. Update Assist Force Magnitude: When the robot successfully recovers, reduce the magnitude of the assist force corresponding to that training environment. S9. Reinforcement Learning Optimization: Input the policy network into the reinforcement learning policy network, which outputs the motion commands for each joint of the robot. S10. Deploy the Trained Policy: After training, deploy the policy network to a simulation environment or a real humanoid robot.

[0006] Furthermore, in step S1, the reference motion trajectory is obtained from motion capture data, manually edited actions, simulated generated actions, and existing robot motion data, including multiple time frames. Each time frame includes reference root node position, reference root node posture, reference joint position, reference joint velocity, and reference end effector position information.

[0007] Furthermore, in step S2, the state data includes the current joint position, current joint velocity, current base attitude, current angular velocity, current gravity direction projection, previous action, current root node or anchor point position, current root node or anchor point attitude, and historical state sequence.

[0008] Furthermore, in step S3, the attitude representation is as follows: the first two columns of the rotation matrix are represented in six dimensions, quaternion, axis-angle, Euler angle, and continuous rotation encoding.

[0009] Further, in step S4, the policy network input includes one or more of the following: current joint position, current joint velocity, current angular velocity, current gravity direction projection, previous action, reference joint position, reference linear velocity, reference angular velocity, reference attitude relative observation, historical state sequence, and reference motion commands for several future frames.

[0010] Furthermore, in step S5, when the robot's anchor point height is lower than a preset threshold, the anchor point posture error is greater than a preset threshold, the end effector height error is greater than a preset threshold, the overall tracking error is greater than a preset threshold, it is randomly initialized to a state deviating from the reference trajectory, or it deviates from the reference trajectory after being subjected to external disturbances, it is determined that the robot is in a recovery state.

[0011] Furthermore, in step S6, the auxiliary force is applied to one or more parts of the robot's head, torso, pelvis, shoulders, rigid body near the center of mass, and other parts used to assist the robot in restoring its overall rigid body structure.

[0012] Furthermore, in step S7, the successful recovery indicators include one or more of the following: robot anchor point height error, anchor point position error, anchor point posture error, head height error, end effector height error, overall tracking error, whether the robot remains standing, and whether it returns to the vicinity of the reference trajectory within a preset time.

[0013] Furthermore, in step S8, each parallel training environment maintains an independent auxiliary force magnitude, enabling robots in different environments to anneal independently based on their respective recovery capabilities, and not all environments sharing the same global auxiliary force value.

[0014] Furthermore, in step S9, during robot training, the reward function includes one or more of the following: reference joint position tracking, reference joint velocity tracking, reference root node position tracking, reference anchor point position tracking, reference anchor point posture tracking, end effector position or height tracking, motion smoothing, posture stabilization, successful recovery reward and energy consumption penalty, and continuous increase in tracking error penalty.

[0015] Compared with the prior art, the beneficial effects of this invention are: (1) It adopts a general motion tracking strategy as the basic model for cleaning and correcting motion data, rather than relying solely on manual rules, inverse kinematics, or single post-processing; (2) It performs continuous training on a single or group of target actions on the general strategy, so that the strategy can be quickly adapted from "generally trackable" to "target action expert trackable"; (3) It generates new motion files through the rollout of the expert strategy in the physical simulation environment, so that the output data is naturally filtered by robot dynamics, kinematics, and contact constraints; (4) It adjusts the motion accuracy and physical repair effect through continuous training rounds; when higher motion accuracy is required, the number of continuous training rounds can be increased to reduce the over-smoothing of small jumps, small translations, or subtle posture changes; (5) It uses a symmetric network with privileged information to train the general strategy, so that the training phase can make full use of the global state and contact information in the simulation, thereby improving the robust tracking and correction capability of the general strategy for redirecting abnormal actions. Attached Figure Description

[0016] Figure 1 This is a schematic flowchart of a robust motion tracking training method for a humanoid robot according to the present invention. Detailed Implementation

[0017] Figure 1 As shown, a robust motion tracking training method for a humanoid robot includes the following steps: (1) Obtaining a reference motion trajectory: First, the reference motion trajectory of the humanoid robot is obtained. The reference motion trajectory can be obtained from motion capture data, manually edited actions, simulated actions, or existing robot motion data. The reference motion trajectory includes multiple time frames. Each time frame includes information such as the reference root node position, reference root node posture, reference anchor point position, reference anchor point posture, reference joint position, reference joint velocity, reference end effector position, reference linear velocity, and reference angular velocity. Among them, the anchor point can be a rigid body near the robot's pelvis, torso, head, center of mass, or any key rigid body used to characterize the overall motion state of the robot. In this embodiment, the robot's pelvis or torso is selected as the anchor point to represent the overall motion state of the robot. For the t-th frame, the reference anchor point posture is denoted as R_ref_t_w, and the reference anchor point position is denoted as p_ref_t_w.

[0018] Figure 1 As shown, (2) Obtain the current state of the robot: In the training environment, obtain the current state information of the robot at the current moment, including the current joint position q_t, the current joint velocity q_dot_t, the robot anchor point posture R_robot_t_w, the robot anchor point position p_robot_t_w, the current angular velocity, the current gravity direction projection, and the previous moment action a_t_minus_1.

[0019] Figure 1 As shown, (3) Calculate the relative observation of the reference posture: Calculate the relative posture relationship between the reference anchor point posture and the robot's current anchor point posture based on the reference anchor point posture in the reference trajectory and the robot's current anchor point posture. Specifically, let the posture of the reference anchor point in the world coordinate system be R_ref_w, and the posture of the robot's current anchor point in the world coordinate system be R_robot_w. Then, the relative posture R_rel of the reference posture relative to the robot's current posture is expressed as: R_rel = transpose(R_robot_w) × R_ref_w, where transpose(R_robot_w) represents the transpose of the robot's current anchor point posture matrix, and R_rel represents the relative rotation relationship of the reference anchor point posture in the robot's current anchor point coordinate system. Further, the relative posture is converted into a posture representation form suitable for neural network input. The posture representation forms include, but are not limited to: the six-dimensional representation of the first two columns of the rotation matrix, quaternion representation, axis-angle representation, Euler angle representation, and continuous rotation encoding representation. Preferably, the six-dimensional posture representation composed of the first two columns of the rotation matrix is ​​used to avoid Euler angle singularity and quaternion sign discontinuity problems. Specifically, by taking the first two columns of the R_rel rotation matrix and flattening them, we obtain the six-dimensional attitude observation as follows: `ori_rel_6d`: `ori_rel_6d = flatten(first_two_columns(R_rel))`, where `ori_rel_6d` represents the relative rotation relationship between the robot's current pose and the reference pose. Through this step, the policy network can directly obtain the relative error information between the robot's current pose and the reference pose, thereby improving heading and attitude tracking capabilities and reducing the unobservable heading problem that arises when relying solely on gravity direction projection.

[0020] Figure 1 As shown, (4) Constructing the policy network input: The current robot state, historical state, reference motion command, and reference attitude relative observation are combined into the policy network input. In the embodiment, the policy network input includes: current gravity direction projection, current angular velocity, current joint position, current joint velocity, previous action, reference linear velocity, reference angular velocity, reference joint position, reference attitude relative observation, several historical frames of state, and several future frames of reference motion command. The policy network outputs the robot's joint control command for the next moment based on the above input. By introducing the reference attitude relative observation in this step, the policy network can directly obtain the relative rotation relationship between the robot's current posture and the reference posture, thereby improving the robot's ability to perceive changes in the overall direction and heading of the reference trajectory.

[0021] Figure 1As shown, (5) Set recovery state: During the training process, determine whether the robot is in recovery state based on the current tracking error or the initial state sampling result. When any of the following conditions are met, the robot can be determined to be in recovery state: the robot anchor point height is lower than the preset threshold, the robot anchor point posture error is greater than the preset threshold, the robot end effector height error is greater than the preset threshold, the robot overall tracking error is greater than the preset threshold, the robot is randomly initialized to a state that deviates from the reference trajectory, or the robot deviates from the reference trajectory after being disturbed by external forces.

[0022] Figure 1 As shown, (6) Apply recovery assistance force: During the training process, when the robot is in the recovery state, initial deviation state, near fall state, or large tracking error state, an assistance force is applied to the robot to help the robot return to the vicinity of the reference motion trajectory. The assistance force can be applied to one or more of the following parts: head, torso, pelvis, shoulder, rigid body near the center of mass, and other rigid body parts used to assist the overall recovery of the robot. Specifically, the direction of the assistance force can be determined according to the deviation between the robot's current position and the reference position, or according to the deviation between the robot's anchor point position and the reference anchor point position. For example, if the reference anchor point position is p_ref_w and the robot's current anchor point position is p_robot_w, then the position deviation between the two is as follows: pos_error = p_ref_w - p_robot_w; the direction of the assistance force dir_force can be obtained by normalizing pos_error, such as: dir_force = pos_error / (norm(pos_error) + epsilon), where norm(pos_error) represents the length of the position deviation vector and epsilon is a small constant to prevent division by zero. If the current auxiliary force is F, then the auxiliary force force applied to the robot target rigid body can be expressed as: force = F × dir_force. The auxiliary force is applied to the robot's head, torso, pelvis, shoulder or other target rigid body to make the robot recover to the vicinity of the reference trajectory (unlike the method of reducing the auxiliary force according to the number of training iterations, this invention dynamically adjusts the auxiliary force according to the actual recovery effect of the robot).

[0023] Figure 1As shown, (7) Calculate the recovery success index: At the end of each training round, or when the robot switches from the recovery state to the normal tracking state, calculate the recovery success index. The recovery success index includes one or more of the following: robot anchor height error, robot anchor position error, robot anchor posture error, robot head height error, robot end effector height error, robot overall tracking error, whether the robot remains standing, and whether the robot returns to the vicinity of the reference trajectory within a preset time. In the embodiment, the relative error between the robot head height and the reference head height is used as the recovery success index. Let the current head height of the robot be h_robot and the reference head height be h_ref, then the relative head height error height_error can be expressed as follows: `height_error = abs(h_robot - h_ref) / h_ref`. When `height_error` is less than a preset threshold `delta`, the robot is considered to have successfully recovered. In other embodiments, height error, position error, and attitude error can be used simultaneously for a comprehensive judgment. Let the height error be `e_h`, the position error be `e_p`, the attitude error be `e_R`, and the comprehensive error be `e_total`, then it can be expressed as the following formula: e_total = alpha × e_h + beta × e_p + gamma × e_R, where alpha, beta, and gamma are weighting coefficients; When the total error e_total is less than a preset threshold, the robot is considered to have recovered successfully. For example, when delta is set to 0.05, it means that the relative error between the robot's current head height and the reference head height is less than 5%, and the robot is considered to have recovered successfully.

[0024] Figure 1As shown, (8) Update the auxiliary force size: When the robot recovers successfully, reduce the auxiliary force size corresponding to the training environment. Let the auxiliary force size before the update be F_old, the auxiliary force size after the update be F_new, the auxiliary force annealing step size be Delta_F, and the minimum auxiliary force be F_min. Then the auxiliary force update method is: F_new = max(F_old - Delta_F, F_min); where max means taking the larger of the two values. That is, whenever the robot meets the recovery success condition, the auxiliary force size decreases Delta_F, but will not be lower than the preset minimum auxiliary force F_min. When the robot fails to recover, the auxiliary force size can be kept unchanged: F_new = F_old; or the auxiliary force can be appropriately increased according to the degree of failure by the following formula: F_new = min(F_old + Delta_F_fail, F_max), where F_min is the minimum auxiliary force, F_max is the maximum auxiliary force, Delta_F is the successful annealing step size, and Delta_F_fail is the failure compensation step size. In this embodiment, each parallel training environment maintains an independent assist force magnitude, allowing robots in different environments to anneal independently based on their respective recovery capabilities, rather than all environments sharing the same global assist force value. In this way, the assist force annealing process no longer depends solely on the number of training iterations, but is correlated with the robot's actual recovery capability, thereby preventing the strategy from losing assist force prematurely or becoming overly reliant on it.

[0025] Figure 1 As shown, (9) Reinforcement learning optimization: The policy network is trained using a reinforcement learning algorithm. The reinforcement learning algorithm can be PPO, SAC, TD3, or other algorithms suitable for continuous control tasks. The reward function includes motion tracking reward and control regularization term. In the embodiment, the reward function includes: reference joint position tracking reward, reference joint velocity tracking reward, reference root node position tracking reward, reference anchor point position tracking reward, reference anchor point posture tracking reward, end effector position or height tracking reward, motion smoothing reward, energy consumption penalty, posture stability reward, successful recovery reward, and tracking error continuously increasing penalty. Among them, the anchor point posture tracking reward can be calculated based on the posture error between the robot's current anchor point posture and the reference anchor point posture.

[0026] Figure 1As shown, (10) Deploy the trained strategy: After training, the strategy network is deployed to a simulation environment or a real humanoid robot. During the robot deployment phase, the recovery assist force can be removed, allowing the robot to complete motion tracking entirely on its own strategy; alternatively, a low-intensity assist force can be retained in the safety protection mode for protective recovery when the robot deviates significantly from the reference action. Since the strategy network has learned the relationship between the reference posture and the observation and recovery control during the training phase, it can track the reference motion more stably during the deployment phase, reduce yaw drift, and improve the stability of complex action execution.

[0027] Figure 1 As shown, through all the above technical solutions, after obtaining the reference motion trajectory and current motion state of the humanoid robot, this invention calculates the relative attitude observation of the reference attitude with respect to the robot's current attitude based on the reference anchor point attitude and the robot's current anchor point attitude. This relative attitude observation, along with the joint state, historical state, and reference motion commands, is input into the policy network to generate robot joint control commands. When the robot is in a recovery state, a recovery assistance force is applied to the robot's target rigid body, and the magnitude of the assistance force is dynamically adjusted according to the robot's recovery success index, realizing adaptive annealing based on the recovery effect. Compared with existing methods that rely solely on gravity direction projection and fixed assistance force annealing, this invention can enhance the policy network's perception of the reference heading attitude, reduce yaw drift during motion tracking, reduce the policy's dependence on external assistance forces, and improve the motion tracking stability of the humanoid robot in complex actions, disturbed states, and recovery scenarios, providing favorable technical support for the stable and reliable operation of the robot.

[0028] It should be understood that the specific embodiments described above are merely illustrative or explanatory of the principles of the invention and do not constitute a limitation thereof. Therefore, any modifications, equivalent substitutions, improvements, etc., made without departing from the spirit and scope of the invention should be included within the protection scope of the invention. Furthermore, the appended claims are intended to cover all variations and modifications falling within the scope and boundaries of the appended claims, or equivalent forms of such scope and boundaries.

Claims

1. A method for training robust motion tracking in a humanoid robot, characterized in that, Includes the following steps, S1. Obtain the reference motion trajectory; S2. Obtain the current state of the robot in the training environment; S3. Calculate the relative observable of the reference attitude. The observable is obtained by calculating the relative attitude relationship between the reference anchor point attitude in the reference trajectory and the robot's current anchor point attitude. S4. Construct the policy network input data, which is a combination of reference attitude relative observables, robot current state, historical state, and reference motion commands. S5. Set recovery state: During training, determine whether the robot is in a recovery state based on the robot's current tracking error or the initial state sampling results. S6. Apply recovery assist force: When the robot is in recovery state, calculate the direction of the assist force based on the deviation between the robot's current anchor point position and the reference anchor point position; S7. Calculate the recovery success index: At the end of each training round, or when the robot switches from the recovery state to the normal tracking state, calculate the recovery success index. S8. Update Assist Force Size: When the robot recovers successfully, reduce the assist force size corresponding to this training environment; S9. Reinforcement Learning Optimization: Input the policy network into the reinforcement learning policy network, and the policy network outputs the motion commands for each joint of the robot. S10. Deploy the trained strategy: After training, deploy the strategy network to a simulation environment or a real humanoid robot.

2. The method for robust motion tracking training of a humanoid robot according to claim 1, characterized in that, In step S1, the reference motion trajectory is obtained from motion capture data, manually edited actions, simulation-generated actions, and existing robot motion data. It includes multiple time frames, and each time frame includes reference root node position, reference root node pose, reference joint position, reference joint velocity, and reference end effector position information.

3. The method for robust motion tracking training of a humanoid robot according to claim 1, characterized in that, In step S2, the state data includes the current joint position, current joint velocity, current base attitude, current angular velocity, current gravity direction projection, previous action, current root node or anchor point position, current root node or anchor point attitude, and historical state sequence.

4. The method for robust motion tracking training of a humanoid robot according to claim 1, characterized in that, In step S3, the attitude representation is as follows: the first two columns of the rotation matrix are represented in six dimensions, quaternion, axis-angle, Euler angle, and continuous rotation encoding.

5. The method for robust motion tracking training of a humanoid robot according to claim 1, characterized in that, In step S4, the policy network input includes one or more of the following: current joint position, current joint velocity, current angular velocity, current gravity direction projection, previous action, reference joint position, reference linear velocity, reference angular velocity, reference attitude relative observation, historical state sequence, and reference motion commands for several future frames.

6. The method for robust motion tracking training of a humanoid robot according to claim 1, characterized in that, In step S5, the robot is determined to be in a recovery state when the robot's anchor point height is lower than a preset threshold, the anchor point posture error is greater than a preset threshold, the end effector height error is greater than a preset threshold, the overall tracking error is greater than a preset threshold, the robot is randomly initialized to a state deviating from the reference trajectory, or the robot deviates from the reference trajectory after being subjected to external disturbances.

7. The method for robust motion tracking training of a humanoid robot according to claim 1, characterized in that, In step S6, an auxiliary force is applied to one or more parts of the robot, including the head, torso, pelvis, shoulders, rigid body near the center of mass, and other parts used to assist the robot in restoring its overall rigid body structure.

8. The method for robust motion tracking training of a humanoid robot according to claim 1, characterized in that, In step S7, the indicators for successful recovery include one or more of the following: robot anchor height error, anchor position error, anchor posture error, head height error, end effector height error, overall tracking error, whether the robot remains standing, and whether it returns to the vicinity of the reference trajectory within a preset time.

9. The method for robust motion tracking training of a humanoid robot according to claim 1, characterized in that, In step S8, each parallel training environment maintains an independent assist force magnitude, allowing robots in different environments to anneal independently based on their respective recovery capabilities, and not all environments share the same global assist force value.

10. The robust motion tracking training method for a humanoid robot according to claim 1, characterized in that, In step S9, during robot training, the reward function includes one or more of the following: reference joint position tracking, reference joint velocity tracking, reference root node position tracking, reference anchor point position tracking, reference anchor point posture tracking, end effector position or height tracking, motion smoothing, posture stabilization, successful recovery reward and energy consumption penalty, and continuous increase in tracking error penalty.

Images