Multi-stage course assisted robot action learning method

Abstract

The invention discloses a multi-stage course assisted robot action learning method, and belongs to the technical field of robot motion control. The method solves the problems that an existing method is still difficult to achieve accurate control over the jumping distance and the landing point, the real-time performance is poor, and the generalization ability among various types of jumping targets is poor. And in combination with a staged reward shaping and whole body coordination guiding mechanism, the humanoid robot forms a stable angular momentum management and body coordination mode in each key stage, so that not only are the jumping performance and the drop point precision improved, but also the posture recovery capability and the anti-disturbance performance in a high-impact scene are remarkably enhanced. And two core technologies of torque control domain optimization and friction identification compensation are provided, so that the training strategy is subjected to dynamic constraint consistent with that of real hardware in a simulation stage. The real machine feasibility of the strategy is greatly improved by constructing a continuous derivable torque-angular velocity feasible region model and friction parameter identification based on an improved Stribeck model. The method can be applied to robot motion control.

Description

Technical Field

[0001] This invention belongs to the field of robot motion control technology, specifically relating to a multi-stage curriculum-assisted robot motion learning method. Background Technology

[0002] Enabling humanoid robots to traverse complex terrain with agility and stability is a highly challenging research direction. Current research primarily focuses on enhancing their walking and running abilities on unstructured terrain, while jumping plays an irreplaceable role in traversing large vertical or horizontal obstacles (such as ravines, steps, or rocks). However, research on jumping remains relatively limited and faces several challenges: humanoid robots need to generate sufficient ground reaction force during takeoff, handle underactuated dynamics during airborne phases, and quickly recover stability after experiencing high impact forces upon landing. Furthermore, precise control of angular momentum is required to achieve high-precision landing point control if a specific landing location is desired.

[0003] Model-based control and optimization methods have yielded significant results in the field of humanoid robot jumping. For example, Qi et al. enabled a humanoid robot to complete a 50cm vertical jump by combining center-of-mass trajectory optimization with angular momentum control; Boston Dynamics' Atlas humanoid robot can even perform a somersault. Such jumps typically employ a hierarchical optimization framework: first, the jump trajectory is planned offline based on whole-body dynamics and a contact model, and then simplified system dynamics are used for online control. However, this method has inherent limitations: the trajectory needs to be recalculated for each new jump objective; furthermore, due to the high computational complexity of optimization, trajectory planning often can only be performed offline, making it difficult to achieve real-time adaptability.

[0004] In recent years, data-driven reinforcement learning methods have made significant progress in controlling complex humanoid robots. For short-distance, continuous hops, one approach utilizes simple reward designs to demonstrate strong dynamic performance, but struggles to achieve precise control over jump distance and landing point. Another paradigm employs high-fidelity reference actions for imitation, enabling humanoid robots to reproduce long-distance, large-amplitude jumps. However, these methods face difficulties in autonomous task execution because they heavily rely on target-specific reference actions, which not only reduces task generalization ability but also requires a large amount of expensive motion capture data to obtain diverse sets of reference jumps.

[0005] In motion generalization schemes based on a single reference action, course learning has shown good results on bipedal humanoid robots (such as Cassie). However, in humanoid robot scenarios, due to the large mass and high moment of inertia of the upper body, the angular momentum effect is significant, requiring stronger whole-body coordination to achieve stable and powerful jumps. When transferring the jumping strategy to real humanoid robots, the joint actuators need to complete high-speed movements and output large torques in a very short time. This further amplifies the dynamic differences between simulation and reality, making it difficult for existing methods to achieve stable results in sim-to-real (sim-to-real) simulations.

[0006] In summary, existing methods still struggle to achieve precise control of jump distance and landing point, suffer from poor real-time performance, and exhibit poor generalization ability across multiple types of jumping targets. Therefore, a control strategy that achieves precise control of jump distance and landing point for humanoid robots, generalizes across multiple types of jumping targets, and maintains real-time control and overall coordination of the humanoid robot is a pressing issue that needs to be addressed. Summary of the Invention

[0007] The purpose of this invention is to address the problems of existing methods still being unable to achieve precise control of jump distance and landing point, poor real-time performance, and poor generalization ability among multiple types of jump targets. Therefore, a multi-stage course-assisted robot motion learning method is proposed.

[0008] The technical solution adopted by the present invention to solve the above-mentioned technical problems is: a multi-stage course-assisted robot motion learning method, the method specifically including the following steps:

[0009] Step 1: Construct the input state vectors of the Actor network and the Critic network in the Actor-Critic architecture respectively. The input state vector of the Actor network is the observation obtained through ontology perception, and the input state vector of the Critic network is privileged information.

[0010] Step 2: Train the Actor network and Critic network in the Actor-Critic architecture through three stages: mastery, expansion, and deployment.

[0011] First, training is conducted in the mastery phase. The switching condition from the mastery phase to the extension phase is: in the mastery phase, the humanoid robot's average survival time in the environment reaches a set threshold.

[0012] The switching condition from the expansion phase to the deployment phase is that both the terrain course and the command course in the expansion phase have reached stable convergence.

[0013] After the deployment phase is completed, the training returns to the expansion phase. Once the training converges, the trained Actor network is obtained.

[0014] Step 3: Use the actual state vector of the humanoid robot as the input of the trained Actor network, output the action through the trained Actor network, and then convert the output action into control commands for the humanoid robot.

[0015] Furthermore, the input vector of the Actor network ;

[0016] in, Indicates the control command for the current step. , Indicates the desired position of the left foot. Indicates the desired position of the right foot. Indicates the desired implementation direction;

[0017] Indicates the future reference motion of the current step. , This indicates a reference action for the third step in the future. This indicates a reference action for the 12th step in the future. This indicates the reference movement for the 21st step in the future. Each reference movement includes the reference position and reference speed of each joint.

[0018] This indicates the historical actions and feedback information of the current step. , This represents the observations for the current and 14 historical steps. Each step's observations include the Euler angles and angular velocity of the floating base, and the actual positions and velocities of each joint. This represents the historical 15-step action, where each action is the expected joint position corresponding to each step.

[0019] The input vector of the Critic network includes privileged information for the current step. Step privilege information and Privileged information for each step.

[0020] Furthermore, the total loss function used during the training of the Actor-Critic architecture is:

[0021]

[0022] in, Indicates strategy loss. Represents the loss of the value function. Indicates symmetry loss, and All are hyperparameters;

[0023]

[0024] in, This represents a function that maps actions to corresponding mirrored items. This indicates that the input vector Functions that map to the corresponding mirror item. This represents the policy of the Actor network. Represents the L2 norm. It indicates a desire for the expected value.

[0025] Furthermore, the environmental termination conditions for the mastery phase and the expansion phase are as follows:

[0026] Condition (1) The time step limit set for the environment is reached;

[0027] Condition (2), actual height of the base ;

[0028] Condition (3), Foot trajectory tracking error Exceeding the threshold, i.e. ,in, Indicates the current foot height. Indicates the height of the foot during the reference movement. This represents the actual elevation difference between the ground and the horizon. Indicates taking the absolute value;

[0029] Condition (4) After landing, the deviation between the actual position and the expected position of the humanoid robot base or the actual orientation of the humanoid robot. Oriented towards The deviation exceeds the threshold, that is or ;

[0030] in, Indicates the humanoid robot base in Actual position in the axial direction and A vector consisting of the actual positions along the axial direction. Indicates the base is Desired position in axial direction and A vector consisting of the desired positions along the axial direction. This indicates the actual yaw angle of the humanoid robot's base. This represents the desired yaw angle of the base;

[0031] The current environment is terminated when conditions (1), (2), (3), or (4) are met.

[0032] Furthermore, the training terrain for the mastery phase is flat ground;

[0033] The training terrain in the expansion phase consists of random uniform terrain and discrete obstacle terrain, and upgrade and downgrade conditions for the training terrain in the expansion phase are set.

[0034] The condition for terrain upgrade is that the humanoid robot survives for more than 15 seconds on the current terrain.

[0035] The terrain degradation condition is that the humanoid robot survives for less than 4 seconds on the current terrain, condition (2) is met, or condition (3) is met.

[0036] Furthermore, the reward function used during the training process of the Actor-Critic architecture... for:

[0037]

[0038] in, This indicates a style imitation award. This indicates the reward for completing the task. Indicates a guiding reward. Indicates the reward for the smoothed item;

[0039]

[0040] in, A vector representing the actual positions of each joint. A vector representing the reference positions of each joint. This indicates the actual position of the humanoid robot's hand. Indicates the reference position of the humanoid robot's hand. This represents the vector consisting of the reference positions of the humanoid robot's two feet along the z-axis. This represents the vector consisting of the actual positions of the humanoid robot's two legs along the z-axis. This indicates the reference position of the humanoid robot's base in the z-axis direction. This indicates the actual position of the humanoid robot's base along the z-axis. This indicates a reward for foot contact. Indicates a foot speed bonus;

[0041]

[0042] in, , , and Indicates the stage indicator quantity;

[0043] This represents the normalized ground reaction force of the left foot. This represents the normalized ground reaction force of the right foot. Indicates the speed of the left foot. This indicates the speed of the right foot.

[0044] Furthermore, the task completion reward for:

[0045]

[0046] in, This represents the vector consisting of the actual position of the base in the x-axis direction and the actual position of the base in the y-axis direction. This represents the vector consisting of the reference position of the base in the x-axis direction and the desired position of the base in the y-axis direction. This represents a vector composed of the actual yaw angle, pitch angle, and roll angle of the base. This represents a vector composed of the base's desired yaw angle, pitch angle, and roll angle. Indicates footstep tracking reward, This indicates a reward for maintaining an upright posture;

[0047] Footstep tracking reward The calculation method is as follows:

[0048]

[0049] in, This indicates the distance from the left foot to the target landing point. This indicates the distance from the right foot to the target landing point. Scaling factor and All are indicator values;

[0050] Upright posture reward The calculation method is as follows:

[0051]

[0052] in, This indicates the actual position of the humanoid robot's base along the x-axis. This represents the actual position of the humanoid robot's head along the x-axis, an intermediate variable. , This represents the actual position vector of the humanoid robot's head. This represents the actual position vector of the humanoid robot's base. This represents the vertically upward unit vector in the world coordinate system.

[0053] Furthermore, the aforementioned guiding item rewards for:

[0054]

[0055] in, This indicates the actual velocity of the base in the z-axis direction. This indicates a reward guided by the torso. This indicates a reward system that guides the recipient's center of gravity.

[0056]

[0057] in, Indicates control input, Indicates a neutral torso angle. Input scaling factor, For the real-time measurement of the torso's tilt angle, Indicates the lower bound of the trunk tilt angle. Indicates the upper limit of the trunk tilt angle. For adaptive parameters, and Indicates intermediate variables. The threshold representing the torso tilt angle. Represents the inverse hyperbolic tangent function. Represents a function;

[0058] The centroid guidance reward for:

[0059]

[0060] in, For indicator functions, and As an intermediate variable;

[0061] Indicates when hour, The value is 1, when hour, The value is 0;

[0062] Indicates when hour, The value is 1, when hour, The value is 0;

[0063]

[0064] in, This indicates that the centroid is relative to the center of the supporting polygon. Displacement on the axis, , Indicates the base is Desired position along the axis, Indicates that the center of the supporting polygon is at Position on the axis Indicates that the center of mass is at Position on the axis; It is a symbolic function; For activation functions; These are the direction-dependent normalized coefficients. The value can be:

[0065]

[0066]

[0067]

[0068] in, Indicates the center of the supporting polygon to The distance along the axial direction supporting the polygon boundary.

[0069] Furthermore, the smoothing item reward for:

[0070]

[0071] in, This represents a vector composed of the movements of each joint in the current step. Indicates the first The vector composed of the movements of each joint in the step. Indicates the first The vector composed of the movements of each joint in the step. Indicates the first The torque of each joint, Indicates the first The actual speed of each joint Indicates the first The actual acceleration of each joint.

[0072] Furthermore, during the actual machine testing, the friction parameters of each joint of the humanoid robot were identified, and the dynamic equations of the humanoid robot were compensated based on the identified friction parameters, specifically as follows:

[0073] Step 1: Construct an objective function to identify friction parameters using the error minimization criterion. :

[0074]

[0075] in, The friction parameters to be identified are in vector form. , This is the error in frictional torque. , For the first The actual frictional torque of each joint To identify frictional torque, This is the estimated result of the Coulomb friction coefficient; This is the estimated result of the maximum static friction coefficient. The estimated results for the Stribeck velocity threshold, This is the estimated result of the viscous friction coefficient;

[0076] That is, frictional torque The expression is:

[0077]

[0078]

[0079] in, This refers to the frictional torque in the classic Stribeck friction model. Coulomb friction coefficient; The maximum static friction coefficient, It is the base of the natural logarithm. For the first The actual angular velocity of each joint express The modulus, The Stribeck velocity threshold, The coefficient of viscous friction is... For symbolic functions, For the set number angular velocity threshold of each joint This is the empirical constant for the attenuation of the Stribeck curve;

[0080] The frictional torque of the humanoid robot's joints can then be calculated based on the above formula.

[0081] Collect joint angular velocity and torque data, and establish the dynamic equations of the humanoid robot after friction parameter compensation:

[0082]

[0083] In the formula, The output joint torque, To compensate for the frictional torque, This is the actual position vector of the humanoid robot's joints. The velocity vector of a humanoid robot's joint. The acceleration vector of a humanoid robot's joint. The inertia matrix for a humanoid robot, The velocity term matrix is ​​related to centrifugal force and Coriolis force. This is the gravity term.

[0084] Based on the final torque feasible domain model of the actuator and the compensated humanoid robot dynamics model, the model returns to the extension stage for retraining until the training process of the extension stage converges, and the final trained model is obtained.

[0085] The method for obtaining the final feasible region model of the actuator's torque is as follows:

[0086] Step S1: Construct a motor torque-angular velocity capability model for each joint actuator of the humanoid robot:

[0087] That is, by combining the peak torque and angular velocity boundary of the motor, an initial feasible region reflecting the actual hardware performance is formed;

[0088] Step S2: Correct based on actual machine test data

[0089] By collecting torque data under different angular velocities and loads, the torque capacity attenuation caused by magnetic saturation, current limiting and friction is fitted, and the initial feasible region is smoothly degenerated into a motor torque-angular velocity boundary curve that conforms to the actual actuator capability, thus obtaining the final torque feasible region model.

[0090] The beneficial effects of this invention are:

[0091] 1. This invention proposes a course-driven, multi-stage reinforcement learning framework that enables humanoid robots to extend from single jumping movements to diverse, fully coordinated jumping tasks. It achieves autonomous adaptation of a unified strategy to multiple jumping targets, solving the problems of strong reliance on reference motions and weak task generalization ability in existing methods. Simultaneously, by combining staged reward shaping and full-body coordination guidance mechanisms, the humanoid robot develops stable angular momentum management and body coordination patterns in key stages such as takeoff, flight, and landing. This not only improves jumping performance and landing accuracy but also significantly enhances posture recovery and anti-disturbance performance in high-impact scenarios. The framework of this invention provides a generalizable paradigm for parameterized reference motions, allowing for systematic expansion of motion capabilities and robustly enabling diversified learning of highly dynamic and complex movements. Furthermore, it eliminates the need to recalculate trajectories for new jumping movements, effectively improving real-time performance.

[0092] 2. This invention identifies key dynamic discrepancies between simulation and reality in high-dynamic actions and proposes two core technologies: torque control domain optimization and friction identification compensation. This ensures that the training strategy is subject to dynamic constraints consistent with real hardware during the simulation phase. By constructing a continuously differentiable torque-angular velocity feasible domain model and identifying friction parameters based on an improved Stribeck model, this invention significantly improves the practical feasibility of the strategy and substantially enhances the ability of humanoid robots to learn and implement complex actions in high-dynamic task scenarios. Attached Figure Description

[0093] Figure 1 This is a flowchart of a reinforcement learning phase of the present invention;

[0094] Figure 2 It is a multi-stage training framework diagram;

[0095] Figure 3 It is a feasible domain model of the actual torque in the pitch direction of the hip joint obtained by fitting actual machine data;

[0096] In the diagram, Joint Velocity represents angular velocity, and Torque represents torque.

[0097] Figure 4 It is a feasible domain model of the true torque in the lateral roll direction of the hip joint obtained by fitting actual machine data;

[0098] Figure 5 It is a feasible domain model of the true torque in the yaw direction of the hip joint obtained by fitting actual machine data;

[0099] Figure 6 It is a feasible domain model of the true torque in the pitch direction of the ankle joint, obtained by fitting actual machine data;

[0100] Figure 7 It is a feasible domain model of the true torque in the lateral roll direction of the ankle joint, obtained by fitting actual machine data;

[0101] Figure 8 This is a comparison diagram of the improved Stribeck friction model and the classic Stribeck friction model;

[0102] Figure 9 This is a flowchart of the genetic annealing optimization algorithm;

[0103] Figure 10 It is a simulation of a humanoid robot performing a vertical jump under a pushing force;

[0104] Figure 11 This is a comparison of the actual and expected torque of the joints in actual machine deployment. Figure 1 ;

[0105] Figure 12 This is a comparison of the actual and expected torque of the joints in actual machine deployment. Figure 2 . Detailed Implementation

[0106] This invention models the environment as a discrete-time Markov Decision Process (MDP) with a continuous state space. Action space State transition function and reward function State transition function Defined in a given state and actions When, transition to state The probability density, and the reward function Provide feedback for each state-action transition. (Policy) Reinforcement learning maps states to actions, and its goal is to find a policy. This maximizes the expected discount return:

[0107]

[0108] in, This is the discount factor. For large-scale continuous state spaces, a parameterized strategy is used. (For example, a multilayer perceptron) is used to represent the policy. Policy parameters Update via gradient ascent:

[0109]

[0110] in, By strategy The sampled trajectory is estimated. This invention employs Proximal Policy Optimization (PPO) to solve this policy gradient problem. PPO ensures the stability and efficiency of updates by limiting the step size, significantly improving algorithm performance compared to other policy optimization methods. Furthermore, reasonable hyperparameter configurations are set based on the dynamic characteristics of the humanoid robot in complex terrain.

[0111] Algorithm initialization includes key parts such as policy network structure setting, advantage estimation method, loss function construction, and optimizer configuration, with entropy weights set to 0.001 and learning rate set to [missing information]. Discount Factor Set to 0.994, GAE coefficient The value is set to 0.9. The total number of training iterations in the training process is set to 6000 to ensure that the policy is fully learned and converged.

[0112] The method of the present invention will be described in detail below with reference to the environmental modeling results.

[0113] Specific Implementation Method 1: Combination Figure 1 This embodiment describes a multi-stage curriculum-assisted robot motion learning method, which specifically includes the following steps:

[0114] Step 1: Construct the input state vectors of the Actor network and the Critic network in the Actor-Critic architecture respectively. The input state vector of the Actor network is the observation obtained through ontology perception, and the input state vector of the Critic network is privileged information.

[0115] Step 2: Train the Actor network and Critic network in the Actor-Critic architecture in three stages: Mastery, Extended, and Deployment, to give the humanoid robot the ability to perform diverse jumps in real-world scenarios;

[0116] In the Mastery phase, the humanoid robot learns to perform stable in-place jumps based on reference actions. In the Extended phase, the humanoid robot undergoes multi-objective training in simulation to acquire the ability to jump to predefined positions. In the Deployment phase, the humanoid robot evaluates the trained strategy in a real-world environment, updates the physical parameters of the simulation controller using real-world data, and then returns to the Extended phase for training. The model trained again in the Extended phase is the final trained model.

[0117] This invention designs environmental termination conditions, phased tasks, and scenario settings for different training stages to better achieve the training objectives at each stage. In each training stage, the humanoid robot starts from an initial state, is loaded into a specific terrain environment, performs designated tasks, and interacts with the environment until the termination condition for each training stage is triggered.

[0118] First, training is conducted in the mastery phase. The switching condition from the mastery phase to the extension phase is: in the mastery phase, the average survival time of the humanoid robot in the environment reaches a set threshold (the threshold set in this invention is 23 seconds).

[0119] The transition from the expansion phase to the deployment phase is conditional upon the following: both the terrain and command courses in the expansion phase have reached stable convergence (during training, the difficulty of the terrain course is adjusted through environment reset; after the terrain course adjustment, the command course gradually increases in difficulty from the level already reached in the previous terrain course), and the humanoid robot can achieve the correct landing posture at the target location. and .

[0120] It should be noted that: In order to explore the humanoid robot's ability to perform jumps over a wider range, the difficulty adjustment rules for the instruction course are as follows:

[0121] In the Extended phase, instruction courses for the humanoid robot are randomly sampled from the following range: rice, Rice, and Standing time is Seconds. Each episode switches between "jump" and "stand" phases, and a random standing interval is added after each jump. This design enhances the policy's robustness to different initial states.

[0122] Introduced course-based instruction offsets The instruction offset is then added to the sampled instructions from the previous difficulty level instruction course. The sampling instruction is defined as follows:

[0123]

[0124] in, The basic sampling instruction, i.e. rice, rice, , For level The offset, after adding the level 1 offset to the basic sampling command, rice, rice, That is, for For each increase in difficulty level, the upper limit of sampling increases by 0.1, and the lower limit of sampling decreases by 0.1. The same applies to other parameters, the only difference being the degree of adjustment.

[0125] The upgrade condition for the instruction course is that the normalized reward for the humanoid robot's landing position and orientation is satisfied: ,in, , ;

[0126] Humanoid robots advance to higher levels after successfully completing instructional tasks at the current difficulty level.

[0127] After the deployment phase is completed, the training returns to the expansion phase. Once the training converges, the trained Actor network is obtained.

[0128] Step 3: Use the actual state vector of the humanoid robot as the input of the trained Actor network, output the action through the trained Actor network, and then convert the output action into control commands for the humanoid robot.

[0129] The training of the model in this invention is performed in parallel on an NVIDIA GPU, sampling data from 4096 environment instances per round. In each training iteration, trajectory information such as state-action-reward-value estimation is first collected according to the current policy, and the advantage function is calculated based on the discounted cumulative reward and the value function. Subsequently, the sampled data is cached in a unified experience pool and randomly rearranged and partitioned into mini-batches to improve the stability of gradient estimation and avoid correlation between samples.

[0130] On mini-batch data, policy parameters are updated through multiple epochs. During policy updates, the policy network and value network are simultaneously optimized based on gradient backpropagation. Each parameter update is constrained by a shaving ratio to ensure stable policy update step size. As training progresses, the policy gradually improves in the Mastery → Extended curriculum structure, evolving from imitating reference actions to generalizing to diverse jump instructions. Throughout the training process, domain randomization is always enabled to prevent the policy from overfitting to the simulation environment and to improve its transfer success rate in the real-world deployment phase.

[0131] This invention generalizes the learning of high-dynamic actions (e.g., jumping) in humanoid robots from a single reference motion to diversified implementations, establishing a task-parameterized motion generalization paradigm. A staged reward shaping mechanism enhances the jumping performance of the humanoid robot, promoting coordinated whole-body movement and achieving greater jump distance, higher jump accuracy, and stronger disturbance resistance. Key discrepancies in the migration from simulation to reality during high-dynamic execution are identified, and targeted optimization of the torque control domain and friction identification compensation are implemented, enabling the successful deployment of the trained diverse jumping skills in real humanoid robot systems. Furthermore, to improve the robustness of the strategy and ensure its successful transfer to a real humanoid robot platform, domain randomization is consistently employed during training to enhance the strategy's robustness, thereby achieving successful transfer to the real humanoid robot platform.

[0132] Specific Implementation Method Two: This implementation method differs from Specific Implementation Method One in that the input vector of the Actor network... ;

[0133] in, Indicates the control command for the current step. , Indicates the desired position of the left foot. Indicates the desired position of the right foot. Indicates the desired implementation direction;

[0134] Define the desired foot position and orientation, and extend the basic control commands to support flexible gait constraints;

[0135] Indicates the future reference motion of the current step. , This indicates a reference action for the third step in the future. This indicates a reference action for the 12th step in the future. This indicates the reference action for the 21st step in the future. Each reference action includes the reference position of each joint (in this invention, the reference angle of the joint is used as the reference position of the joint) and the reference speed.

[0136] This indicates the historical actions and feedback information of the current step. , This represents the observations for the current and 14 historical steps. Each step's observations include the Euler angles and angular velocity of the floating base, and the actual positions and velocities of each joint. This represents the historical 15-step action, where each action is the expected joint position corresponding to each step.

[0137] The input vector of the Critic network includes privileged information for the current step. Step privilege information and Privileged information for a step; taking the current step as an example, the privileged information within the current step includes... , Current step observations The values ​​of the reference position and actual position of each joint in the current step, the base linear velocity of the current step (3D), the random linear velocity perturbation in the planar direction of the current step (2D), the random angular velocity perturbation of the current step (3D), the ground friction coefficient of the current step (1D), the overall mass of the humanoid robot (1D), the expected foot contact state of the current step (2D), and the actual foot contact state of the current step (2D). Step privilege information and The privilege information of a step contains the same content as the privilege information of the current step, the only difference being that it contains information for the corresponding step.

[0138] The other steps and parameters are the same as in Specific Implementation Method 1.

[0139] In this invention, both the Actor and Critic networks are constructed using MLPs. The Actor network has a dimension of 1057→512→256→128→20, while the Critic network has a dimension of 501→768→256→128→1. In simulations, certain task-related attributes (such as global position and base linear velocity) are readily available and can significantly improve policy training performance. However, these attributes are difficult to observe in real-world environments. Therefore, this invention employs an asymmetric Actor-Critic framework to address the challenges of humanoid robot control in the real world, whereby the Critic network utilizes privileged information as input. The Actor network is embedded within a hierarchical control structure, with a high-level reinforcement learning policy network... Generate desired joint position at a frequency of 100Hz The joint-level PD controller will operate at a frequency of 500Hz. Convert to desired joint torque To drive joint movement, desired joint torque The dimension is related to the number of joints of the humanoid robot.

[0140] For the Unitree G1 humanoid robot, twenty degrees of freedom (DoFs) distributed across the leg, shoulder, and elbow joints are used for jump control, so the observations for each step are: ,in, This represents the vector composed of the Euler angles of the floating base. This represents the vector consisting of the angular velocities of the floating base, while This represents a vector composed of joint positions derived from the airborne encoder. This represents a vector composed of joint velocities derived from the airborne encoder.

[0141] For the electrically driven Dexbot humanoid robot, twelve dedicated leg degrees of freedom are used to perform jumping, while the upper body posture is mechanically constrained. Its simplified observation space is... .

[0142] Specific Implementation Method Three: This implementation method differs from Specific Implementation Method One or Two in that the total loss function used during the Actor-Critic architecture training process is:

[0143]

[0144] in, Indicates strategy loss. Represents the loss of the value function. Indicates symmetry loss, and All are hyperparameters;

[0145]

[0146] in, This represents a function that maps actions to corresponding mirrored items. This indicates that the input vector Functions that map to the corresponding mirror item. This represents the strategy of the Actor network, which is to output an action based on the mapped input vector. Represents the L2 norm. It indicates a desire for the expected value.

[0147] Other steps and parameters are the same as in specific implementation method one or two.

[0148] Symmetry is a key characteristic for evaluating the quality of humanoid robot movements. This invention performs symmetry by minimizing the difference between the policy output and its mirror image movement. Using a movement mirroring optimization strategy improves energy efficiency and accelerates training. In this invention, hyperparameters... and The values ​​are 1 and 0.4 respectively.

[0149] Specific Implementation Method Four: This implementation method differs from Specific Implementation Methods One to Three in that the environmental termination condition for the mastery phase and the expansion phase is:

[0150] Condition (1) The upper limit of the time step set for the environment is reached. In this invention, the duration of the environment for the jump task is set to 2400 time steps (24 seconds).

[0151] Condition (2), actual height of the base For example, a humanoid robot might fall and become unable to continue performing its task;

[0152] Condition (3), Foot trajectory tracking error Exceeding the threshold, i.e. ,in, This indicates the current foot height (i.e., the average height of the left and right feet). This indicates the reference foot height for the action (i.e., the average reference height of the left and right feet). This represents the actual elevation difference between the ground and the horizon. Indicates taking the absolute value;

[0153] Condition (4), after landing (i.e. , (Indicates the landing time), the deviation between the actual and expected positions of the humanoid robot's base, or the actual orientation of the humanoid robot. Oriented towards The deviation exceeds the threshold, that is or ;

[0154] in, Indicates the humanoid robot base in Actual position in the axial direction and A vector consisting of the actual positions along the axial direction. Indicates the base is Desired position in axial direction and A vector consisting of the desired positions along the axial direction. This indicates the actual yaw angle of the humanoid robot's base. This represents the desired yaw angle of the base;

[0155] The current environment is terminated when conditions (1), (2), (3), or (4) are met.

[0156] The other steps and parameters are the same as those in one of the specific implementation methods one to three.

[0157] The termination mechanism set in this implementation method can avoid interference from invalid data and focus on effective learning situations, thereby ensuring the efficiency of the training process.

[0158] Specific Implementation Method Five: This implementation method differs from one of the specific implementation methods one to four in that the training terrain during the mastery stage is a flat surface, in order to improve the humanoid robot's proficiency in jumping skills.

[0159] The training terrain in the expansion phase consists of random uniform terrain (which is terrain without obstacles, but includes some small potholes and bumps; the difficulty variation under this terrain condition is mainly reflected in the depth of the potholes, the height of the bumps, and the density of the potholes and bumps) and discrete obstacle terrain, to simulate the scenarios that humanoid robots may encounter in real environments. The difficulty of the random uniform terrain is less than that of the discrete obstacle terrain. A terrain course learning mechanism is used to gradually increase the difficulty, that is, first learn under random uniform terrain, and then learn under discrete obstacle terrain. Upgrade and downgrade conditions for the training terrain in the expansion phase are set.

[0160] The terrain upgrade condition is that the humanoid robot survives for more than 15 seconds on the current terrain.

[0161] The terrain degradation condition is that the humanoid robot survives for less than 4 seconds on the current terrain, or condition (2) or condition (3) is met, that is, after resetting the environment, the terrain is degraded and trained.

[0162] The other steps and parameters are the same as those in one of the specific implementation methods one to four.

[0163] This implementation ensures that the humanoid robot can maintain stability even when it cannot perfectly execute the new jumping maneuver under moderately challenging terrain conditions.

[0164] Specific Implementation Method Six: This implementation method differs from Specific Implementation Methods One through Five in that reward shaping is crucial for achieving the humanoid robot's flexible and agile jumping ability. The reward function used during the Actor-Critic architecture training process... for:

[0165]

[0166] in, This indicates a style imitation award. This indicates the reward for completing the task. Indicates a guiding reward. Indicates the reward for the smoothed item;

[0167] Style imitation is used to guide the agent to reproduce reference actions. The humanoid robot is guided to imitate the reference actions in terms of joint position, base height, hand position, foot height, and foot contact state. To cope with changes in terrain height, this invention introduces a style imitation term into the vertical displacement of the reference. This allows the humanoid robot to compensate for changes in landing height by adjusting the tracking of the base and foot height. The style imitation reward is specifically as follows:

[0168]

[0169] in, A vector representing the actual positions of each joint. A vector representing the reference positions of each joint. This indicates the actual position of the humanoid robot's hand. Indicates the reference position of the humanoid robot's hand. This represents the vector consisting of the reference positions of the humanoid robot's two feet along the z-axis. This represents the vector consisting of the actual positions of the humanoid robot's two legs along the z-axis. This indicates the reference position of the humanoid robot's base in the z-axis direction. This indicates the actual position of the humanoid robot's base along the z-axis. This indicates a reward for foot contact. Indicates a foot speed bonus;

[0170] by For example, tracking error Weight , The calculation method is as follows:

[0171]

[0172] To capture the changes in contact state during the "takeoff-air-landing" phase (modeled as DS-AIR-DS phase), we designed phase-related reward items. and The dual support (DS) and airborne (AIR) phases are manually timed based on the contact state in the reference motion and use phase indicator quantities. , , and To regulate limb dynamics, gait period parameters The system dynamically determines the positive or negative nature of rewards, thereby adjusting the reward structure through stage-sensitive coefficients.

[0173] These indicators (all taking values ​​of -1 or 1) adjust ground reaction force (GRF) and foot speed in opposite directions at different stages: in the DS stage, GRF is rewarded and swing leg speed is penalized; in the AIR stage, GRF is penalized and swing leg acceleration is encouraged. The reward format is as follows:

[0174]

[0175] in, , , and This indicates the phase indicator, specifically during the two-footed support phase. and The value is -1. and The value is 1 during the takeoff phase. and The value is 1. and The value is -1;

[0176] This represents the normalized ground reaction force of the left foot. This represents the normalized ground reaction force of the right foot. Indicates the speed of the left foot. This indicates the speed of the right foot.

[0177] The other steps and parameters are the same as those in one of the specific implementation methods one to five.

[0178] Specific Implementation Method Seven: This implementation method differs from Specific Implementation Methods One through Six in that the task aims to guide the humanoid robot's base to the desired position while maintaining the correct posture. During this process, the desired pitch and roll angles of the base are set to 0 to ensure system stability. The task completion reward... for:

[0179]

[0180] in, This represents the vector consisting of the actual position of the base in the x-axis direction and the actual position of the base in the y-axis direction. This represents the vector consisting of the reference position of the base in the x-axis direction and the desired position of the base in the y-axis direction. This represents a vector composed of the actual yaw angle, pitch angle, and roll angle of the base. This represents a vector consisting of the desired yaw angle, pitch angle, and roll angle of the base. The desired pitch angle and desired roll angle are both set to 0 by default. Indicates footstep tracking reward, This indicates a reward for maintaining an upright posture;

[0181] Landing location bonus and footstep tracking bonus The drive base gradually approaches the target position, while ensuring that both feet land precisely within the predefined support area. This foot tracking reward... The calculation method is as follows:

[0182]

[0183] in, This indicates the distance from the left foot to the target landing point. This indicates the distance from the right foot to the target landing point. The scaling factor (in this invention, ), and All are indicator values;

[0184] like If the value exceeds 0.2, it is directly considered... If it is 0, If the value exceeds 0.2, it is directly considered... The value is 0. When the left foot lands, The value is 1 when the left foot is in the air. The value is 0; when the right foot lands, The value is 1 when the right foot is in the air. The value is 0.

[0185] To enhance stability during flight and landing, this invention introduces... To maintain the humanoid robot's upper body in an upright or slightly forward-leaning posture, the upright posture is rewarded. The calculation method is as follows:

[0186]

[0187] in, This indicates the actual position of the humanoid robot's base along the x-axis. This represents the actual position of the humanoid robot's head along the x-axis, an intermediate variable. , This represents the actual position vector of the humanoid robot's head. This represents the actual position vector of the humanoid robot's base. This represents the vertically upward unit vector in the world coordinate system.

[0188] Upright posture reward Using different attenuation coefficients for forward and backward leaning postures helps to establish preferences in upper body postures.

[0189] The other steps and parameters are the same as those in one of the specific implementation methods one to six.

[0190] Specific Implementation Method Eight: This implementation method differs from Specific Implementation Methods One through Seven in that it introduces a stage-specific reward mechanism to guide stage-specific actions. During the air phase, base speed guidance is implemented. In the vertical jump task, the z-axis speed reward is enhanced to increase jump height; and to achieve a precise landing, a formula is used during the take-off phase. A rough estimate of the desired linear velocity and angular velocity is made to optimize the motion planning during the takeoff phase. The guiding item reward... for:

[0191]

[0192] in, This indicates the actual velocity of the base in the z-axis direction. This indicates a reward guided by the torso. This indicates a reward system that guides the recipient's center of gravity.

[0193] Unlike bipedal robots like Cassie, which lack an upper body, the large mass of a humanoid robot's torso significantly impacts its jumping performance. Inspired by observations of human kinematics, this invention applies torso tilting and twisting guidance during the takeoff phase to support directional jumping. (Bonus) Control input Mapped to desired tilt angle ,in, These are pitch angle, roll angle, and yaw angle, respectively. The specific nonlinear mapping method is as follows:

[0194]

[0195] in, Indicates control input, Indicates a neutral torso angle. Input scaling factor, For the real-time measurement of the torso's tilt angle, Indicates the lower bound of the trunk tilt angle. Indicates the upper limit of the trunk tilt angle. For adaptive parameters, and Indicates intermediate variables. The threshold representing the torso tilt angle. Represents the inverse hyperbolic tangent function. Represents a function;

[0196]

[0197] Using hyperbolic tangent function Adjusting trunk angle tracking: When the tilt angle is small ( When the slope is steep, it allows for rapid correction; at the same time, its saturation characteristics prevent excessive increase in the inclination angle. Excessive changes in parameters ensure jump accuracy. Dynamic adjustments are made based on the desired tilt angle to maintain a balance between stability and adaptability. This formula achieves three core capabilities: forward tilt ( Used to increase jump distance, lateral tilt ( Used for directional control, yaw and twist ( Used for multi-axis motion. The clip operation ensures that trunk movements are physiologically reasonable.

[0198] Humans can generalize diverse jumping movements from similar take-off postures, primarily due to precise control of the center of mass (COM) trajectory. This invention incorporates a reward system. The humanoid robot is guided through a dual-operation mode. The centroid-guided reward... for:

[0199]

[0200] Among them, when At that time, the reward actively prompts the center of mass to deviate from the center of mass of the polygonal support along the command axis during the take-off phase; For indicator functions, and As an intermediate variable;

[0201] Indicates when hour, The value is 1, when hour, The value is 0;

[0202] Indicates when hour, The value is 1, when hour, The value is 0;

[0203]

[0204] in, This indicates that the centroid is relative to the center of the supporting polygon. Displacement on the axis (the supporting polygon is formed by the convex hull of the current contact points of the soles of both feet with the ground on the horizontal plane), , Indicates the base is Desired position along the axis, Indicates that the center of the supporting polygon is at Position on the axis Indicates that the center of mass is at Position on the axis; This is a sign function used to ensure that the reward is positive only when the centroid is offset in the direction of the instruction; For activation functions; These are the direction-dependent normalized coefficients. The value can be:

[0205]

[0206]

[0207] This design allows for the rewarding of centroid shifts in different directions at an appropriate scale, thereby improving directional control during the takeoff phase. In stable scenarios ( During the landing and stationary phases, the reward encourages the center of mass to remain within the polygon supported by both feet:

[0208]

[0209] in, Indicates the center of the supporting polygon to The distance along the axial direction supporting the polygon boundary;

[0210] when When the centroid is inside the supporting polygon, no penalty is incurred; when the centroid crosses the supporting boundary, the penalty term increases linearly with the degree of boundary crossing.

[0211] The other steps and parameters are the same as those in any of the specific implementation methods one to seven.

[0212] Specific Implementation Method Nine: This implementation method differs from Specific Implementation Methods One through Eight in that the smoothing item reward ensures smoother and safer task execution by penalizing sudden changes in continuous movements, joint torques, and speeds. The smoothing item reward... for:

[0213]

[0214] in, This represents a vector composed of the movements of each joint in the current step. Indicates the first The vector composed of the movements of each joint in the step. Indicates the first The vector composed of the movements of each joint in the step. Indicates the first The torque of each joint, Indicates the first The actual speed of each joint Indicates the first The actual acceleration of each joint.

[0215] The other steps and parameters are the same as those in one of the specific implementation methods one to eight.

[0216] It should be noted that in the calculation , , and During the process, only the reward for the current step is calculated; that is, all parameters involved in the reward calculation are those for the current step. This invention omits subscripts at the levels of some parameters. .

[0217] This invention adjusts the weights of each term in the reward function at different training stages. In the Mastery phase, larger weights are allocated to the style term to ensure the humanoid robot's movements align with the reference trajectory. In the Extended phase, weights are biased towards the task term to enhance the humanoid robot's ability to perform tasks beyond the reference movements, such as forward jumps and jump turns. Each training phase has a specific guiding term. The smoothing term maintains a consistent weight throughout training and is initially set small to avoid overly conservative behavior.

[0218] The complete reward function format and weight settings are shown in Table 1:

[0219] Table 1

[0220]

[0221]

[0222] Specific Implementation Method Ten: This implementation method differs from Specific Implementation Methods One through Nine in that, in the simulation environment, reinforcement learning strategies often directly output idealized joint drive commands, only "hard-truncating" the torque through constant upper and lower limits. When a real humanoid robot performs a high-dynamic jumping task, the output torque of the joint actuator is affected by nonlinear factors such as actuator current limiting, back electromotive force, and temperature rise, causing the torque capability to decrease significantly with speed and load, which differs significantly from the idealized constant torque model in the simulation. To reduce this impact, actuator torque feasible region optimization is introduced in the Deployment stage, allowing the strategy to be updated within the real feasible region during the training phase, ensuring the reliability of the strategy on the real machine.

[0223] The specific method for optimizing the feasible region of actuator torque during the deployment phase is as follows:

[0224] Step S1: Construct a motor torque-angular velocity capability model for each joint actuator of the humanoid robot:

[0225] That is, by combining the peak torque and angular velocity boundary of the motor, an initial feasible region reflecting the actual hardware performance is formed;

[0226] Step S2: Correct based on actual machine test data

[0227] By collecting torque data under different angular velocities and loads, the torque capacity attenuation caused by magnetic saturation, current limiting, and friction is fitted, and the initial feasible region is smoothly degenerated into a motor torque-angular velocity boundary curve that conforms to the actual actuator capability, thus obtaining the final torque feasible region model, such as... Figure 3 ( Set to 5) Figure 4 ( Set to 0) Figure 5 ( Set to 0) Figure 6 ( (Set to 12) and Figure 7 ( As shown in 4);

[0228] The final feasible torque domain model is embedded in the deployed controller to clamp the torque output of the strategy in real time, ensuring that the output of each joint is always within the actual torque control domain. This optimization effectively reduces joint saturation and reverse torque accumulation during the take-off phase and improves the torque buffering capacity during the landing phase, guaranteeing the consistency and execution stability of the trained strategy on a real humanoid robot.

[0229] In high-dynamic jumping tasks, joint friction characteristics significantly affect the low-speed response, take-off force, and landing cushioning performance of humanoid robots. To reduce sim-real bias, this invention introduces a torque-level friction identification and compensation mechanism during the Deployment phase. Specifically, during actual testing, the friction parameters of each joint of the humanoid robot are identified, and the robot's dynamic equations are compensated based on these identified friction parameters.

[0230] Step 1: Select the joints of the humanoid robot whose friction parameters are to be identified, and lock the remaining joints. Make the joints move along a set cosine trajectory, and collect and filter data during the movement. Based on the experimental data, construct the Stribeck model parameters as the variables to be identified.

[0231] Construct an objective function to identify friction parameters using the error minimization criterion. :

[0232]

[0233] in, The friction parameters to be identified are in vector form. , This is the error in frictional torque. , For the first The actual frictional torque of each joint To identify frictional torque, This is the estimated result of the Coulomb friction coefficient; This is the estimated result of the maximum static friction coefficient. The estimated results for the Stribeck velocity threshold, This is the estimated result of the viscous friction coefficient;

[0234] To more accurately describe the frictional behavior of joints during low-speed and reverse switching processes, this invention employs an improved Stribeck friction model. This model unifies static friction, Coulomb friction, and velocity-dependent viscous friction into a continuously differentiable form, avoiding the discontinuities that occur near zero velocity in traditional piecewise friction models. That is, the frictional torque... The expression is:

[0235]

[0236]

[0237] in, This refers to the frictional torque in the classic Stribeck friction model. Coulomb friction coefficient; The maximum static friction coefficient, It is the base of the natural logarithm. For the first The actual angular velocity of each joint express The modulus, The Stribeck velocity threshold, The coefficient of viscous friction is... For symbolic functions, For the set number angular velocity threshold of each joint This is the empirical constant for the attenuation of the Stribeck curve;

[0238] The frictional torque of the humanoid robot's joints can then be calculated based on the above formula. During the calculation, a genetic annealing optimization algorithm is used to search for the optimal parameter set. The specific process is as follows: Figure 9 As shown, to ensure the model matches the actual friction curve across the entire velocity domain, the final friction model is as follows. Figure 8 As shown in the figure, this model can naturally depict the smooth transition of friction force from the static friction zone to the dynamic friction zone with joint speed, making the simulated friction behavior closer to the actual drive system.

[0239] After identifying the friction model parameters, the model was integrated into the actual controller to perform real-time friction torque compensation on the strategy output torque. The compensated torque command can effectively eliminate joint lag and jitter in the low-speed range, improve the consistency of force application during the take-off phase, and reduce attitude oscillations caused by friction nonlinearity during the landing phase, thereby significantly enhancing the execution stability of the training strategy in a real humanoid robot.

[0240] Establish the mapping relationship between the joint friction torque and the corresponding joint velocity of the humanoid robot, collect joint angular velocity and torque data, and establish the dynamic equations of the humanoid robot after friction parameter compensation:

[0241]

[0242] In the formula, This is the output joint torque (this torque needs to be limited based on the final torque feasible domain model). To compensate for the frictional torque, This is the actual position vector of the humanoid robot's joints. The velocity vector of a humanoid robot's joint. The acceleration vector of a humanoid robot's joint. The inertia matrix for a humanoid robot, The velocity term matrix is ​​related to centrifugal force and Coriolis force. This is the gravity term.

[0243] Since the humanoid robot model is known, therefore, , and All of these are known quantities.

[0244] Based on the final feasible torque domain model and the compensated humanoid robot dynamics model, the training returns to the extension stage for retraining until the training process in the extension stage converges, resulting in the final trained model.

[0245] The other steps and parameters are the same as those in any of the specific implementation methods one to nine.

[0246] In summary, this invention proposes a three-stage curriculum-driven reinforcement learning framework that enables humanoid robots to learn highly dynamic jumping skills progressively through a "mastery stage → extension stage → deployment stage." We train the initial imitation strategy using human reference jumping movements that include contact states, enabling the humanoid robot to reproduce a set of predefined jumps. Subsequently, through phased guided rewards, the humanoid robot is encouraged to develop coordinated whole-body movement patterns, covering multiple jump types. Finally, to address the actuator tracking degradation problem under high torque during the take-off phase, we optimize the physical parameters of the simulation controller through comparison with real machine data, narrowing the difference between the simulation and the real actuator, and effectively transferring the learned jumping strategy to a real humanoid robot platform. This framework not only enables diverse jumps but also provides a generalizable paradigm for parameterized reference motion, allowing for the expansion of systems with highly dynamic and complex motion capabilities.

[0247] Experimental Section

[0248] A trunk center of mass-guided reward is introduced during the takeoff phase to shape the pre-jump posture. This densifies the previously sparse target rewards, allowing the guidance strategy to learn generalizable multi-target jump skills. For example... Figure 10As shown, the diversity of learned strategies enables the humanoid robot to resist perturbations from multiple directions through compensatory small jumps. Furthermore, arm coordination plays a crucial role in enhancing robustness during the whole-body jump. In the Extended phase, the humanoid robot learns through diverse jumping tasks, gradually developing a set of general strategies: increasing jump height through arm movements, assisting in body reorientation, and maintaining balance in the air. In quantitative evaluation, we compared the performance of the proposed method, the baseline method, and the armless configuration under applied random perturbations in various directions using two dimensions: jump distance (maximum successful distance, success rate greater than 80%) and perturbation resistance (maximum thrust that allows more than 80% of the humanoid robot to remain standing under perturbation). The baseline method relies solely on velocity for guidance, while the proposed method introduces additional centroid-trunk stage guidance. The results show that the proposed method exhibits stronger adaptability to dynamic changes and external perturbations, resulting in significant improvements in jump accuracy and distance.

[0249] Furthermore, the performance of the strategy was validated through deployment experiments with a real humanoid robot. The strategy successfully generalizes a single reference action to diverse jumping tasks, including jumping forward to a target point of 1.5m, jumping backward 0.75m, and lateral jumps spanning 0.7m. By simply adjusting the target command, the humanoid robot can adaptively adjust its whole-body coordination behavior, such as adjusting the torso lean angle and varying the arm swing amplitude, to adapt to the requirements of different jump directions and distances. The framework also demonstrates the ability to combine jumping actions; the humanoid robot can complete a composite jump task integrating a forward jump of 0.4m, a lateral jump of 0.2m, and a 45° yaw turn, indicating that the strategy has the ability to decompose and reassemble actions for complex tasks. Actual-desired torque comparison of joints during real-world deployment is also provided. Figure 11 and Figure 12 As shown, after optimizing the simulator parameters during the deployment phase, the actual torque of each joint of the humanoid robot can accurately track the expected torque and has a good coordinated force effect.

Claims

1. A multi-stage curriculum-assisted robot motion learning method, characterized in that, The method specifically includes the following steps: Step 1: Construct the input state vectors of the Actor network and the Critic network in the Actor-Critic architecture respectively. The input state vector of the Actor network is the observation obtained through ontology perception, and the input state vector of the Critic network is privileged information. Step 2: Train the Actor network and Critic network in the Actor-Critic architecture through three stages: mastery, expansion, and deployment. First, training is conducted in the mastery phase. The switching condition from the mastery phase to the extension phase is: in the mastery phase, the humanoid robot's average survival time in the environment reaches a set threshold. The switching condition from the expansion phase to the deployment phase is that both the terrain course and the command course in the expansion phase have reached stable convergence. After the deployment phase is completed, the training returns to the expansion phase. Once the training converges, the trained Actor network is obtained. Step 3: Use the actual state vector of the humanoid robot as the input of the trained Actor network, output the action through the trained Actor network, and then convert the output action into control commands for the humanoid robot.

2. The multi-stage curriculum-assisted robot motion learning method according to claim 1, characterized in that, The input vector of the Actor network ; in, Indicates the control command for the current step. , Indicates the desired position of the left foot. Indicates the desired position of the right foot. Indicates the desired implementation direction; Indicates the future reference motion of the current step. , This indicates a reference action for the third step in the future. This indicates a reference action for the 12th step in the future. This indicates the reference movement for the 21st step in the future. Each reference movement includes the reference position and reference speed of each joint. This indicates the historical actions and feedback information of the current step. , This represents the observations for the current and 14 historical steps. Each step's observations include the Euler angles and angular velocity of the floating base, and the actual positions and velocities of each joint. This represents the historical 15-step action, where each action is the expected joint position corresponding to each step. The input vector of the Critic network includes privileged information for the current step. Step privilege information and Privileged information for each step.

3. The multi-stage curriculum-assisted robot motion learning method according to claim 2, characterized in that, The total loss function used during the training of the Actor-Critic architecture is: in, Indicates the strategy loss. Represents the loss of the value function. Indicates symmetry loss, and All are hyperparameters; in, This represents a function that maps actions to corresponding mirrored items. This indicates that the input vector Functions that map to the corresponding mirror item. This represents the policy of the Actor network. Describing the L2 norm, It indicates a desire for the expected value.

4. The multi-stage curriculum-assisted robot motion learning method according to claim 3, characterized in that, The environmental termination conditions for the mastery and expansion phases are as follows: Condition (1) The time step limit set for the environment is reached; Condition (2), actual height of the base ; Condition (3), Foot trajectory tracking error Exceeding the threshold, i.e. ,in, Indicates the current foot height. Indicates the height of the foot during the reference movement. This represents the actual elevation difference between the ground and the horizon. Indicates taking the absolute value; Condition (4) After landing, the deviation between the actual position and the expected position of the humanoid robot base or the actual orientation of the humanoid robot. Oriented towards The deviation exceeds the threshold, that is or ; in, Indicates the humanoid robot base in Actual position in the axial direction and A vector consisting of the actual positions along the axial direction. Indicates the base is Desired position in axial direction and A vector consisting of the desired positions along the axial direction. This indicates the actual yaw angle of the humanoid robot's base. This represents the desired yaw angle of the base; The current environment is terminated when conditions (1), (2), (3), or (4) are met.

5. The multi-stage curriculum-assisted robot motion learning method according to claim 4, characterized in that, The training terrain for the mastery phase is flat ground; The training terrain in the expansion phase consists of random uniform terrain and discrete obstacle terrain, and upgrade and downgrade conditions for the training terrain in the expansion phase are set. The condition for terrain upgrade is that the humanoid robot survives for more than 15 seconds on the current terrain. The terrain degradation condition is that the humanoid robot survives for less than 4 seconds on the current terrain, condition (2) is met, or condition (3) is met.

6. The multi-stage curriculum-assisted robot motion learning method according to claim 5, characterized in that, The reward function used during the training of the Actor-Critic architecture for: in, This indicates a style imitation award. This indicates the reward for completing the task. Indicates a guiding reward. Indicates the reward for the smoothed item; in, A vector representing the actual positions of each joint. A vector representing the reference positions of each joint. This indicates the actual position of the humanoid robot's hand. Indicates the reference position of the humanoid robot's hand. This represents the vector consisting of the reference positions of the humanoid robot's two feet along the z-axis. This represents the vector consisting of the actual positions of the humanoid robot's two legs along the z-axis. This indicates the reference position of the humanoid robot's base in the z-axis direction. This indicates the actual position of the humanoid robot's base along the z-axis. This indicates a reward for foot contact. Indicates a foot speed bonus; in, , , and Indicates the stage indicator quantity; This represents the normalized ground reaction force of the left foot. This represents the normalized ground reaction force of the right foot. Indicates the speed of the left foot. This indicates the speed of the right foot.

7. The multi-stage curriculum-assisted robot motion learning method according to claim 6, characterized in that, The task completion reward for: in, This represents the vector consisting of the actual position of the base in the x-axis direction and the actual position of the base in the y-axis direction. This represents the vector consisting of the reference position of the base in the x-axis direction and the desired position of the base in the y-axis direction. This represents a vector composed of the actual yaw angle, pitch angle, and roll angle of the base. This represents a vector composed of the base's desired yaw angle, pitch angle, and roll angle. Indicates footstep tracking reward, This indicates a reward for maintaining an upright posture; Footstep tracking reward The calculation method is as follows: in, This indicates the distance from the left foot to the target landing point. This indicates the distance from the right foot to the target landing point. Scaling factor and All are indicator values; Upright posture reward The calculation method is as follows: in, This indicates the actual position of the humanoid robot's base along the x-axis. This represents the actual position of the humanoid robot's head along the x-axis, an intermediate variable. , This represents the actual position vector of the humanoid robot's head. This represents the actual position vector of the humanoid robot's base. This represents the vertically upward unit vector in the world coordinate system.

8. The multi-stage curriculum-assisted robot motion learning method according to claim 7, characterized in that, The guiding item reward for: in, This indicates the actual velocity of the base in the z-axis direction. This indicates a reward guided by the torso. This indicates a reward system that guides the recipient's center of gravity. in, Indicates control input, Indicates a neutral torso angle. Input scaling factor, For real-time measurement of the torso's tilt angle, Indicates the lower bound of the trunk tilt angle. Indicates the upper limit of the trunk tilt angle. For adaptive parameters, and Indicates intermediate variables. The threshold representing the torso tilt angle. Represents the inverse hyperbolic tangent function. Represents a function; The centroid-guided reward for: in, For indicator functions, and As an intermediate variable; Indicates when hour, The value is 1, when hour, The value is 0; Indicates when hour, The value is 1, when hour, The value is 0; in, This indicates that the centroid is relative to the center of the supporting polygon. Displacement on the axis, , Indicates the base is Desired position along the axis, Indicates that the center of the supporting polygon is at Position on the axis Indicates that the center of mass is at Position on the axis; It is a symbolic function; For activation functions; These are the direction-dependent normalized coefficients. The value can be: in, Indicates the center of the supporting polygon to The distance along the axial direction supporting the polygon boundary.

9. A multi-stage curriculum-assisted robot motion learning method according to claim 8, characterized in that, The smoothing item reward for: in, This represents a vector composed of the movements of each joint in the current step. Indicates the first The vector composed of the movements of each joint in the step. Indicates the first The vector composed of the movements of each joint in the step. Indicates the first The torque of each joint, Indicates the first The actual speed of each joint Indicates the first The actual acceleration of each joint.

10. A multi-stage curriculum-assisted robot motion learning method according to claim 9, characterized in that, During the actual machine testing, the friction parameters of each joint of the humanoid robot were identified, and the dynamic equations of the humanoid robot were compensated based on the identified friction parameters. Specifically: Step 1: Construct an objective function to identify friction parameters using the error minimization criterion. : in, The friction parameters to be identified are in vector form. , This is the error in frictional torque. , For the first The actual frictional torque of each joint To identify frictional torque, This is the estimated result of the Coulomb friction coefficient; This is the estimated result of the maximum static friction coefficient. The estimated results for the Stribeck velocity threshold, This is the estimated result of the viscous friction coefficient; Friction torque The expression is: in, This refers to the frictional torque in the classic Stribeck friction model. Coulomb friction coefficient; The maximum static friction coefficient, It is the base of the natural logarithm. For the first The actual angular velocity of each joint express The modulus, The Stribeck velocity threshold, The coefficient of viscous friction is... For symbolic functions, For the set number angular velocity threshold of each joint This is the empirical constant for the attenuation of the Stribeck curve; The frictional torque of the humanoid robot's joints can then be calculated based on the above formula. Collect joint angular velocity and torque data, and establish the dynamic equations of the humanoid robot after friction parameter compensation: In the formula, The output joint torque, To compensate for the frictional torque, This is the actual position vector of the humanoid robot's joints. The velocity vector of a humanoid robot's joint. The acceleration vector of a humanoid robot's joint. The inertia matrix for a humanoid robot, The velocity term matrix is ​​related to centrifugal force and Coriolis force. This is the term related to gravity. Based on the final torque feasible domain model of the actuator and the compensated humanoid robot dynamics model, the model returns to the extension stage for retraining until the training process of the extension stage converges, and the final trained model is obtained. The method for obtaining the final feasible region model of the actuator's torque is as follows: Step S1: Construct a motor torque-angular velocity capability model for each joint actuator of the humanoid robot: That is, by combining the peak torque and angular velocity boundary of the motor, an initial feasible region reflecting the actual hardware performance is formed; Step S2: Correct based on actual machine test data By collecting torque data under different angular velocities and loads, the torque capacity attenuation caused by magnetic saturation, current limiting and friction is fitted, and the initial feasible region is smoothly degenerated into a motor torque-angular velocity boundary curve that conforms to the actual actuator capability, thus obtaining the final torque feasible region model.

Images