A gait imitation learning method for humanoid robots combined with periodic rewards

Abstract

A method for learning the gait imitation of a humanoid robot combined with periodic rewards relates to the field of robot motion control technology. In view of the problem of unstable posture of humanoid robots when walking in a humanoid posture in a plane in the prior art, this application constructs a reference action library that integrates contact information as a reference for imitation reward items and periodic contact reward items. This application creates a comprehensive reference action library for basic actions and their corresponding periodic contact information. This strategy introduces periodic reward items by imitating the style of the reference action and its contact information, which not only improves the realism and style consistency of the robot's actions, but also enhances the attention to the details of the interaction between the feet and the ground during the execution of the action, thereby ensuring the stability of the posture of the humanoid robot when walking in a humanoid posture in a plane.

Description

Technical Field

[0001] The present invention relates to the technical field of robot motion control, and in particular to a gait imitation learning method for a humanoid robot combined with periodic rewards. Background Art

[0002] In recent years, thanks to advances in artificial intelligence and robotics, the motion control of humanoid robots has once again attracted widespread attention. While traditional model-based motion control methods have achieved some success, their reliance on precise modeling and inability to adapt to complex tasks still limit their application. Humanoid robots exhibit complex dynamics, with motion involving multiple degrees of freedom, connections, and uncertainties. Traditional algorithms often simplify and abstract models when modeling and controlling these complexities. For example, the linear inverted pendulum (LIP) model reduces the robot's legs to a point mass and retractable, massless legs, making it impossible to precisely control each joint. In contrast, reinforcement learning algorithms can infer the dynamic characteristics of humanoid robots from their interactions with the environment, eliminating the need for explicit modeling and thus enabling more precise motion control.

[0003] Reinforcement learning has proven to be an effective approach for designing bipedal robot controllers. In his research on the Cassie robot, Jonah Siekmann implemented a periodic bipedal gait by incorporating a clock signal into the reward function. Furthermore, Zhongyu Li used a library of actions generated using hybrid zero dynamics (HZD) as a reference and introduced an imitation learning reward term to learn the style of the reference actions. However, applying reinforcement learning to humanoid robots presents more challenges than applying it to the lighter Cassie, primarily due to the impact of the heavy upper body on stability and safety. In his research on the HRP robot, Rohan demonstrated that, through periodic rewards and foothold planning, the robot was able to achieve programmed footsteps and turn maneuvers. However, the walking posture of the humanoid robot implemented using this approach was not truly human-like, often remaining in a semi-squat position, which placed a greater load on the knee joint motors. This ultimately led to instability when the humanoid robot walked in a humanoid-like manner on a flat surface. Summary of the Invention

[0004] The purpose of the present invention is to propose a humanoid robot gait imitation learning method combined with periodic rewards to address the problem of unstable posture when a humanoid robot walks in a plane in a human-like posture in the prior art.

[0005] The technical solution adopted by the present invention to solve the above technical problems is:

[0006] A humanoid robot gait imitation learning method combined with periodic rewards, comprising the following steps:

[0007] Step 1: Construct a reference action library, which contains basic actions and periodic contact information corresponding to each basic action. The periodic contact information includes continuous DS, alternating DS and SS, and alternating SS, where DS is the double-foot contact phase and SS is the single-foot support phase.

[0008] Step 2: Design a reward function based on the reference action library;

[0009] Step 3: Train the deep reinforcement learning model based on the reward function designed in step 2;

[0010] Step 4: Use the trained deep reinforcement learning model to imitate the gait of the humanoid robot.

[0011] Furthermore, the specific steps of constructing the reference action library are:

[0012] Step 11: Acquire an original data set, wherein the original data set includes each basic action and a sequence of human lower limb joint position information corresponding to each basic action;

[0013] Step 1 and 2: Use motion redirection to map the human lower limb joint position information sequence to the humanoid robot to obtain the robot lower limb joint position information sequence;

[0014] Step 13: Process the robot's lower limb joint position information sequence using the motion constraint and control frequency alignment algorithm to obtain a trimmed robot lower limb joint position information sequence, and use the trimmed robot lower limb joint position information sequence to construct a reference action library.

[0015] Furthermore, the motion constraints include position limits and speed limits, specifically:

[0016] joint Location restrictions Speed limit Hip pitch -10～80 15 Hip pitch -15～30 8 Hip pitch -15～15 2 Knee pitch 0～120 22 Ankle pitch -30～70 10 Ankle lateral swing -15～15 5

[0017] The unit of position limit is degree, and the unit of speed limit is rad / s.

[0018] Furthermore, the original dataset in step 1 is obtained through the CMU motion capture dataset or the open source human motion library Rofunc.

[0019] Furthermore, the reward function includes an imitation learning term, a task term, and a regularization term;

[0020] The imitation learning items include and

[0021] The task items are

[0022] The regularization term includes and

[0023]

[0024]

[0025]

[0026]

[0027]

[0028]

[0029]

[0030]

[0031] in, Indicates the base imitation reward, Indicates joint imitation reward, Shows foot contact force imitation reward, Express speed imitation reward, Indicates the reward for the step task. represents the action change regularization reward, represents the moment regularization reward, represents the torque change regularization reward, and q base,t They are respectively represented as the base orientation of the reference action at time t and the actual base orientation, and q t They are respectively represented as the joint position of the reference action at time t and the actual joint position, Indicates the left foot ground reaction force phase indicator, Indicates the right foot ground reaction force phase indicator, Indicates the left foot speed phase indicator, Indicates the right foot speed phase indicator, F left and F right are the normalized ground reaction forces of the left and right feet, respectively, S left and S right Represent the normalized body speed of the left and right feet respectively, a represents the current action, a prev represents the action at the previous moment, τ represents the joint torque, τ prev represents the joint torque at the previous moment, d foot Represents the distance between the foot and the target point, droot Indicates the distance between the base and the target point.

[0032] Furthermore, the specific steps of step three are:

[0033] Step 31: Import the reinforcement learning environment and encapsulate the environment;

[0034] Step 32: Initialize the Actor-Critic algorithm, which includes a policy network, a value network, and a PPO algorithm;

[0035] Step 33: setting the training stages to stage 1 and stage 2, wherein the first stage is a single task, and the second stage is a composite task, wherein the single task performs a basic movement task, and the composite task performs a combined gait task;

[0036] Step 3 and 4: Set the training round T itr , and take the training round T itr Half of the phase change round T J , then based on the initialized Actor-Critic algorithm, training is performed, with iteration rounds from 0 to T J Between, the first stage reward function weight is used for training, and the iteration round is T J to T itr Between , the second stage reward function weights are used for training. The first stage reward function weights and the second stage reward function weights are as follows:

[0037]

[0038] Step 35: Initialize the reinforcement learning environment, including setting the robot's initial state and task objectives, and introducing random perturbations;

[0039] Step 36: Use Ray to sample the reinforcement learning environment in parallel and concatenate the sampling results into a buffer;

[0040] Step 37: Randomly sample the indexes in the buffer and set the batch size. Then, split the buffer data into multiple small batches based on the selected indexes. Each batch contains a complete data sample. Any remaining samples that are not enough to form a complete small batch are discarded.

[0041] Step 38: Start a training cycle epoch. The total number of cycles is epochs. The process of training the model using the entire dataset is a training cycle. Each epoch uses a random small batch of data to update the strategy.

[0042] Step 39: Calculate the loss function and back-propagate to update the network weights. The loss function includes PPO loss, mirror symmetry loss, and entropy loss.

[0043] Furthermore, the policy network and the value network are fully connected networks, i.e., multi-layer perceptrons.

[0044] The value network consists of two hidden layers, each containing 256 neurons and using the ReLU activation function;

[0045] The policy network consists of two hidden layers, each containing 256 neurons, and uses the ReLU activation function. The output of the network passes through a TanH layer.

[0046] Furthermore, the basic movements include squatting, moving forward, and running;

[0047] The single tasks include walking, standing, backing up, and moving sideways;

[0048] The complex task includes turning, changing-speed walking, and changing-speed running.

[0049] Furthermore, the steps 3 and 5 also include the step of dynamically adjusting the training hyperparameters, specifically:

[0050] Adjust the annealing rate by monitoring the current highest reward and dynamically adjust the termination threshold based on specific operating conditions;

[0051] The specific method of adjusting the annealing rate by monitoring the current highest reward is:

[0052] When the reward exceeds two-thirds of the maximum trajectory length and the annealing rate is greater than 0.5, reduce the annealing rate;

[0053] The dynamic adjustment of the termination threshold based on specific operating conditions is specifically:

[0054] If the termination condition is enabled and the termination threshold is less than 0.35, the termination threshold is increased according to the number of iterations.

[0055] Furthermore, the weights of the PPO loss, mirror symmetry loss, and entropy loss are 1, 0.4, and 0.1, respectively.

[0056] The beneficial effects of the present invention are:

[0057] This application constructs a reference action library that integrates contact information as a reference for imitation bonus items and periodic contact bonus items. This application creates a comprehensive reference action library for basic actions and their corresponding periodic contact information. This strategy introduces periodic bonus items by imitating the style of reference actions and their contact information. This not only improves the realism and style consistency of the robot's actions, but also enhances the attention to the details of the interaction between the feet and the ground during the execution of the action, thereby ensuring the stability of the humanoid robot's posture when walking in a human-like posture on a plane.

[0058] This application addresses the walking problem of humanoid robots by adopting an end-to-end control method based on model-free deep reinforcement learning. The control strategy is directly generated through the system state quantity without the need to model the complex humanoid robot system. It is a general method for such tasks.

[0059] This application constructs a multi-stage curriculum learning process to effectively develop control strategies that can flexibly move on a plane and perform multiple tasks. By dividing the training into three phases: single-task training, composite-task training, and domain randomization training, this application gradually increases task complexity and environmental uncertainty, thereby constructing a control system with high adaptability, flexibility, and robustness. This phased training approach ensures that the robot mastered basic skills while enhancing its generalization and stability in diverse real-world environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 Components of the reference action library for fusion of contact information constructed for this application;

[0061] Figure 2 A flowchart for building a reference action library of fused contact information for this application;

[0062] Figure 3 The multi-stage course training process for this application;

[0063] Figure 4 Specific training process for this application;

[0064] Figure 5 This is an ablation experiment diagram for this application that introduces the imitation reward term. It includes two parts: the humanoid robot's walking gait display and the knee joint torque change;

[0065] Figure 6 This is an ablation experiment diagram for multi-stage curriculum learning in this application. It contains two parts: training curve and training screenshots. The blue line of the training curve represents the survival time of each round, and the green line shows the average sum of rewards. DETAILED DESCRIPTION

[0066] It should be noted that, unless there is any conflict, the various embodiments disclosed in this application can be combined with each other.

[0067] Specific implementation method 1: refer to Figure 1 Specifically describing this embodiment, a humanoid robot gait imitation learning method combined with periodic rewards described in this embodiment is characterized by comprising the following steps:

[0068] Step 1: Construct a reference action library, which contains basic actions and periodic contact information corresponding to each basic action. The periodic contact information includes continuous DS, alternating DS and SS, and alternating SS, where DS is the double-foot contact phase and SS is the single-foot support phase.

[0069] Step 2: Design a reward function based on the reference action library;

[0070] Step 3: Train the deep reinforcement learning model based on the reward function designed in step 2;

[0071] Step 4: Use the trained deep reinforcement learning model to imitate the gait of the humanoid robot.

[0072] Step 1: Build a reference action library that integrates contact information. Figure 1 As shown in , this action library consists of a series of predetermined basic actions and their corresponding periodic contact information. The basic actions selected in the library include squatting, moving forward, and running. There are two types of contact between the robot and the ground: the double-foot contact phase (DS), that is, the state where both feet are in contact with the ground at the same time; and the single-foot support phase (SS), in which one foot is in contact with the ground to provide support and the other foot is suspended in the air. The periodic contact information is in Figure 1 The basic movements and periodic contact information are systematically combined to construct a comprehensive reference movement library covering a variety of sports content.

[0073] Construct a reference action library that integrates contact information. The implementation process is as follows: Figure 2 shown.

[0074] The construction of the reference motion library consists of two steps. First, the lower limb joint position information sequence and the corresponding motion sequence are obtained as the original dataset. Then, the lower limb joint position information sequence is mapped to the humanoid robot through motion redirection, resulting in the robot's lower limb joint position information sequence, which is also used as the reference dataset. Furthermore, after applying motion constraints and control frequency alignment algorithms to this sequence, the trimmed robot lower limb joint position information sequence is obtained and used as the reference motion library.

[0075] Raw datasets can be obtained from a variety of sources, including traditional control solutions, motion capture datasets, open-source human motion libraries, and manual creation using software like Blender. After obtaining the raw reference motions, they need to be mapped to the humanoid robot through motion retargeting.

[0076] The motion redirection method is implemented as follows. First, a set of source key points on the humanoid model are determined and mapped to target key points on the humanoid robot, such as the foot and hip positions. At each time step, the 3D position of each key point i in the motion of the humanoid model is recorded. These positions are then used to define the target keypoint positions x of the robot. i (q t ), where q t Represents the robot's posture, expressed in generalized coordinates. By applying inverse kinematics (IK), a series of postures q are constructed based on the 3D positions of the source keypoints and the positions of the robot's target keypoints. 0:T , the goal is to minimize the following objective function:

[0077]

[0078] Among them, the first term is the sum of squares of position errors, which is used to ensure that the robot's key points are as close as possible to the key point positions of the original humanoid model. The second term is the regularization term, which represents the default posture. W is a diagonal matrix of regularization coefficients. This term encourages the robot's posture to remain close to the default posture to maintain the naturalness of the action and avoid unrealistic postures. Using the obtained posture sequence q 0:T , that is, the humanoid robot can perform the redirected movement.

[0079] A reference motion library is generated by applying motion constraints and aligning the control frequencies to the reference dataset. Redirecting the original motions may exceed the limits of the robot's joints. Furthermore, the robot's soles may not be flat when contacting the ground, leading to inaccurate contact information. Therefore, humanoid joint position constraints and foot contact constraints are added to the trajectories in the reference dataset to ensure compliance with the reference motions. To address the issue of frame rate inconsistencies between the original dataset and the required control frequency, a frame insertion and extraction method is used for frame rate alignment.

[0080] Contact parameters include the four types described above. The durations of the bipedal contact phase (DS) and the unipedal contact phase (SS) can be specified manually or directly obtained from a reference action set. Specifically, the robot is instructed to perform a predetermined action sequence in a simulation environment, such as walking, while adjusting the relative height of the robot's feet to the virtual ground to achieve a slight touchdown effect. This operation aims to accurately simulate the moment when the foot contacts the ground without affecting the robot's normal walking or movement patterns, thereby capturing key contact parameters.

[0081] Step 2: Reward function design. The reward function consists of the following three parts: (1) imitation learning term (2) task term (3) regularization term. The use of the reference action library that integrates contact information is mainly reflected in the imitation learning term. The composition of the reward function is mainly shown in the following formula.

[0082] r=ω1·r imitate +ω2·r task +ω3·r reg

[0083] Imitation learning items

[0084] In the imitation process, in addition to accurately mimicking the style of the reference action, special emphasis is placed on contact information. To this end, this invention effectively incorporates contact information into the guidance of the robot's movements by introducing a periodic reward term. This approach ensures that the imitation process not only remains faithful to the appearance and style of the reference action, but also fully considers the details of the interaction with the environment during execution, such as the contact pattern between the foot and the ground during gait.

[0085] The imitation reward guides the robot to maintain the same base orientation, joint angles, and contact states as the reference action. The robot is encouraged to maintain consistency with the base orientation of the reference action at each time step, represented by a quaternion. and q base,t It is represented by the reference action orientation and the actual joint orientation at time t. In actual use, this part of the reward will gradually be replaced by the orientation of the desired movement direction.

[0086] The robot is encouraged to match the joint positions in the reference motion at each time step, with the error measured by the 2-norm. and q t They are respectively represented as the joint position of the reference action at time t and the actual joint position.

[0087] Base error

[0088]

[0089] Joint error

[0090]

[0091] The imitation of the contact state is described by the periodic reward term. The symmetrical bipedal gait is defined by the periodic motion of the legs, and the reference motion library contains four types of contact information. Taking the (2) gait in the reference library as an example, in a complete gait cycle, the body goes through the double-leg support phase (DS) and the single-leg support phase (SS) twice. In the double-leg support phase, both feet are in contact with the ground to provide stability; in the single-leg support phase, one foot swings while the other foot bears the weight of the body and performs the support function. For gaits (1) and (3), there are only the double-leg support phase (DS) and the single-leg support phase (SS), respectively.

[0092] In order to precisely control the gait cycle, phase indicators are set, namely ground reaction force phase indicator and speed phase indicator These phase indicators are responsible for adjusting the ground reaction force of the support leg and the speed of the swing leg. The value of the phase indicator is related to the phase. In the DS phase,

[0093] In the single-leg support phase (taking the right foot as the supporting foot SS (right) as an example), and In this framework, the rewards of ground reaction force and velocity can be quantified as follows:

[0094]

[0095]

[0096] Among them, F left and F right is the normalized ground reaction force, S left and S right is the normalized body velocity. The values of the four phase indicators mentioned above can be obtained based on the value of φ in the gait cycle, and then the value of the reward function can be calculated.

[0097] Task Item

[0098] In the task, you need to guide the robot to walk at the target speed and set footprint. The goal is to accurately guide the robot's foot to the next predetermined target point. The reward is calculated based on . The specific reward formula is:

[0099]

[0100] Among them, k hitIt is a hyperparameter that balances the influence of foot accuracy and root position, so as to achieve accurate docking between the foot and the target point while ensuring the overall direction of the robot body moving towards the target point.

[0101] The progress reward incentivizes the policy to move the robot's main body (root) forward to reach the next target position. Compared to the step reward, the progress reward decays more slowly and focuses on the accuracy of the robot's overall forward direction rather than the positioning of specific steps. This allows the robot to adjust its pace to adapt to environmental changes or task requirements while maintaining its forward direction.

[0102] Regularization term

[0103] The introduction of regularization terms is to ensure the smoothness of the action, enhance energy efficiency, and provide guarantees for the migration of the algorithm from simulation to actual systems.

[0104] Action regularization term This term ensures the consistency and smoothness of actions by penalizing the differences between consecutive actions. prev Represents the action at the previous moment, ω a is a regularization coefficient. This form of reward encourages small changes in action, reduces sudden changes in the robot's movement, and helps improve the robot's action consistency. It is defined as:

[0105]

[0106] Moment regularization term To improve energy efficiency, this term penalizes large joint torque values. τ is a regularization coefficient used to adjust the effect of torque. This reward mechanism tends to favor less strenuous actions, reducing unnecessary energy consumption while avoiding excessive stress on the robot's joints. It is defined as follows:

[0107]

[0108] Moment change regularization term This item focuses on minimizing the rate of change of joint torque, thus making the robot move more smoothly. prev Refers to the joint torque at the previous moment, ω Δτ is the regularization coefficient. This regularization term helps prevent damage to the robot or the environment due to sudden torque changes by reducing the sharp changes in torque. It is defined as:

[0109]

[0110] Step three, multi-stage course learning design. In order to develop a set of control strategies that can move flexibly on a plane and perform multiple tasks, direct training on difficult tasks is often difficult to achieve the goal directly. The challenges in the initial learning stage may cause the robot to exhibit overly conservative behavior patterns, thereby hindering the process of learning new tasks. To address this problem, the present invention proposes a course learning program consisting of three stages: (1) single task training, (2) composite task training, and (3) domain randomization training. The transition criteria of the entire training process are based on the preset target of the number of iterations. The contents of each stage are as follows: Figure 3 shown.

[0111] In the first phase, the robot primarily learns to perform basic motion tasks, such as walking, standing, backing up, and lateral movement. The task setup and periodic reward parameters are designed to maintain consistency with the reference motion library, or with only minor adjustments. This phase provides the robot with a solid foundation of skills.

[0112] In the second phase, the robot's tasks shifted to more complex gait combinations, such as turning, variable-speed walking, and running. During this phase, the weight of the reference action was reduced, while the weight of the task-specific reward was increased, encouraging the robot to maintain flexibility while completing more complex tasks. Specifically, the metrics for variable-speed walking and running tasks were adjusted from a fixed step length to a target speed, while a certain range was set for the step length and gait period to introduce randomness. Furthermore, the generation of turning gaits incorporated gait patterns trained in the first phase. The goal of this phase was to improve the robot's ability to walk flexibly on a flat surface.

[0113] Step 4: Import the reinforcement learning environment and encapsulate it. Using the Ray library, a high-performance framework for distributed computing, it is possible to build and manage parallel computing. This not only improves the efficiency of sample collection, but also significantly shortens the training cycle because it is performed simultaneously in parallel instances. On this basis, the environment is encapsulated with mirror symmetry to achieve the estimation of behavioral symmetry observations. The overall process of the training phase is as follows: Figure 4 shown.

[0114] Step 5: Initialize the policy network and the value network. Both are fully connected networks, namely multi-layer perceptrons (MLPs). Determine the control policy network π based on the robot's action space dimensions. θ and value network Q φ The network structure of . θ is the network parameter that controls the policy function, and φ is the network parameter that controls the value function. The policy network is responsible for generating actions that are appropriate for a given state, while the value network is used to estimate the expected reward for a state or state-action pair.

[0115] Step 6. Initialize the PPO algorithm instance and set the relevant hyperparameters. The main parameters are as follows. Discount factor γ: Calculates the current value of future rewards. Learning rate η: Controls the step size of weight updates. Clipping coefficient ∈: Used to limit the amplitude of policy updates to avoid excessive update steps. Coefficient c1 of the value function: In the total loss, the coefficient of the value function loss term. Entropy coefficient c2: In the total loss, the coefficient that increases the policy entropy to encourage exploration. GAE (Generalized Advantage Estimation) parameter λ: Used to balance TD error and Monte Carlo estimation. The total number of training iterations T itr : The number of complete passes through the training dataset. The length of each episode of the task: An episode is the process from the beginning to the end of the task. The maximum number of steps each episode contains is called its length. Batch size: Defines the number of samples used in each network update. Buffer size. Epochs: The number of times the model will repeatedly learn using the entire collected training set.

[0116] Step 7: Start an iteration, if the training round T is reached itr , then it ends. If the iteration round reaches the stage change round T J , then adjust the reward function weight according to step 3.

[0117] Step 8: Initialize the environment. This involves setting the robot's initial state and mission objectives, and introducing random perturbations to enhance the robustness of the policy during training. This step also involves dynamically adjusting hyperparameters, specifically by monitoring the current maximum reward to adjust the annealing rate appropriately. The termination threshold is also dynamically adjusted under specific operating conditions, achieving a more effective balance between exploration and exploitation, improving training flexibility and efficiency.

[0118] Step 9: Parallel data sampling. Parallel sampling is implemented across multiple environments using Ray, and the collected data from multiple environments is concatenated into a buffer. The goal is to collect sufficient interaction data for subsequent policy updates.

[0119] Step 10: Advantage function estimation. The advantage function is calculated by subtracting the estimated value function from the cumulative reward for each state step. It intuitively represents the additional reward obtained by taking a specific action compared to the evaluator's expected average performance. The purpose of calculating the advantage is to reduce variance while maintaining unbiasedness to accelerate training and improve stability. The advantage function is then normalized to reduce the differences between different advantage values and bring them to the same magnitude.

[0120] Step 11: Randomly sample the indices in the buffer and set the batch size. Based on the selected indices, split the buffer data into multiple mini-batches, each containing a complete data sample. Any remaining samples that are insufficient to form a complete mini-batch are discarded to ensure a consistent data size within each batch. The goal is to optimize data processing efficiency during training and, therefore, improve the performance of the learning algorithm.

[0121] Step 12: Start a training cycle (epoch). The total number of epochs is epochs. A training cycle refers to the process of training the model using the entire dataset. Each epoch gradually updates the policy using random mini-batches of data. Mini-batch updates reduce variance and improve learning stability. Epochs is the preset total number of training cycles, which determines the number of times the model repeatedly learns the entire training set, ensuring that the model has ample learning opportunities to absorb the information in the dataset.

[0122] Step 13: Calculate the loss function and back-propagate to update the network weights. The loss function of the policy network combines the following three parts: PPO loss (actor_loss), mirror symmetry loss (mirror_loss) and entropy loss (entropypenalty)

[0123] The calculation of the PPO loss (actor_loss) first involves evaluating the ratio of the action probability distribution between the current policy and the previous policy, based on which the gradient clipped surrogate loss is calculated. This loss is adjusted through the clipping operation to moderate the amplitude of the policy update, thereby determining the final actor_loss. The mirror symmetry loss (mirror_loss) is calculated by comparing the difference between the original action (generated directly by the policy network) and the action mapped back after mirror observation processing. It aims to minimize the deviation between the two, thereby forcing the policy to learn symmetric action patterns. The entropy penalty aims to encourage exploration by increasing the uncertainty of the policy.

[0124] The loss function of the value network is based on the mean squared error between the temporal difference (TD) objective and the value function. This loss aims to optimize the value network so that its predicted state values are closer to the actual rewards.

[0125] Example:

[0126] Step 1: Build a reference action library that integrates contact information. As shown in the figure, this action library consists of a series of predetermined basic actions and their corresponding periodic contact information. The basic actions selected in the library include squatting, moving forward, and running. There are two types of contact between the robot and the ground: the double-foot contact phase (DS), that is, the state where both feet are in contact with the ground at the same time; and the single-foot support phase (SS), in which one foot is in contact with the ground to provide support and the other foot is suspended in the air. Periodic contact information is generated in Figure 1 The basic movements and periodic contact information are systematically combined to construct a comprehensive reference movement library covering a variety of sports content.

[0127] The construction of the reference motion library consists of two steps. First, the lower limb joint position information sequence and the corresponding motion sequence are obtained as the original dataset. Then, the lower limb joint position information sequence is mapped to the humanoid robot through motion redirection, resulting in the robot's lower limb joint position information sequence, which is also used as the reference dataset. Furthermore, after applying motion constraints and control frequency alignment algorithms to this sequence, the trimmed robot lower limb joint position information sequence is obtained and used as the reference motion library.

[0128] The original datasets were obtained from various sources. The squatting posture was acquired using a ZMP-based method. The walking reference motions were derived from the CMU motion capture dataset. The running and jumping original datasets were derived from the open-source human motion library Rofunc. After acquiring the original reference motions, they were mapped to the humanoid robot using motion redirection.

[0129] After applying motion constraints and aligning the control frequencies to the reference dataset, a reference motion library is generated. The motion constraints for humanoid robots include position and speed limits, as shown in the table below. For data points outside the motion constraint range, a clipping function is enabled to scale the out-of-range points to the maximum or minimum allowed value.

[0130] joint Position limit (°) Speed limit (rad / s) Hip pitch -10～80 15 Hip pitch -15～30 8 Hip pitch -15～15 2 Knee pitch 0～120 22 Ankle pitch -30～70 10 Ankle lateral swing -15～15 5

[0131] Contact parameters are directly derived from a reference action set. The robot executes a predetermined action sequence in the simulation environment while adjusting the relative height of the robot's foot to the virtual ground to achieve a slight touchdown effect. This operation aims to accurately simulate the moment of foot-ground contact without affecting the robot's normal walking or movement patterns, thereby capturing key contact parameters.

[0132] The contact information during the reference motions is as follows: (1) Squatting: DS phase lasts 1 second. (2) Contact information: DS phase 0.15 seconds, SS phase 0.85 seconds. (3) Running: DS phase 0 seconds, SS phase 0.4 seconds.

[0133] Step 2: Reward function design. The parameters of each reward function are shown in the following table.

[0134]

[0135] Step 3: Multi-stage curriculum design. In order to develop a control strategy that can flexibly move on a plane and perform multiple tasks, direct training on difficult tasks is often difficult to achieve the goal. The challenges in the initial learning stage may cause the robot to exhibit overly conservative behavior patterns, thereby hindering the process of learning new tasks. To address this problem, the present invention proposes a curriculum learning scheme consisting of two stages: (1) single task training, and (2) composite task training. The transition criteria of the entire training process are based on the preset target number of iterations.

[0136]

[0137] In the first stage, the robot mainly learns to perform basic motion tasks, such as walking, standing, backward and sideways.

[0138] In the second phase, the robot's tasks shifted to more complex gait combinations, such as turning, variable-speed walking, and variable-speed running. During this phase, the weight of the reference action was reduced, while the weight of the task-specific reward was increased, encouraging the robot to maintain flexibility while completing more complex tasks. Specifically, the metrics for variable-speed walking and running tasks were adjusted from a fixed step length to a target speed, while also introducing randomness by setting certain ranges for step length and gait period. Furthermore, the generation of turning gaits incorporated gait patterns trained in the first phase. The goal of this phase was to improve the robot's ability to walk flexibly on a flat surface.

[0139] In this implementation, the weights of each part of the reward function are shown in the table above. The basic idea is that the early stage is more inclined to imitate the action, while the later stage hopes to achieve better generalization performance while maintaining the action style. J Set to 5000, that is, every 5000 rounds, the weight of each part of the reward function is adjusted. The total length of the first phase is 10,000 rounds, and the total length of the second and third phases is 10,000 rounds.

[0140] Step 4: Import the reinforcement learning environment and encapsulate the environment. Encapsulating the parallel environment improves the efficiency of sample collection and significantly shortens the training cycle. On this basis, the environment is encapsulated with mirror symmetry to achieve the estimation of the observation of behavioral symmetry. In this solution, the Ray distributed computing framework is used to implement parallel computing. The training platform CPU model is Intel Xeon Gold 5218R@2.1GHz, which has 16 cores and supports hyperthreading technology, providing up to 32 threads. In actual use, Ray uses 30 logical cores (threads) to execute tasks or schedule work.

[0141] Step 5: Initialize the policy network and value network in the Actor-Critic method. The policy network directly determines the robot's behavior. Its goal is to learn a policy π(a|s), which is the probability distribution of choosing action a given state s. The policy network outputs a probability assessment of each possible action, thereby guiding the robot's actions in the environment. The policy network receives the current state of the environment as input and outputs a set of action probabilities, which the robot then uses to make decisions. The value network evaluates the effectiveness of the robot's actions. Specifically, it estimates the expected value of the cumulative rewards the robot can expect to receive in the future under the current policy, starting from a certain state s. This evaluation helps the policy network better adjust its parameters to select the most optimal actions.

[0142] In this case, both are fully connected networks, namely multi-layer perceptrons (MLPs). The control strategy function π is determined according to the dimension of the robot's action space. θ and the value function Q φ The network structure of . θ is the network parameter that controls the policy function, and φ is the network parameter that controls the value function. The policy network is responsible for generating actions that are appropriate for a given state, while the value network is used to estimate the expected reward for a state or state-action pair.

[0143] Both MLP networks have two hidden layers, each containing 256 neurons and using the ReLU activation function. To limit the range of the actuator output, the output of the control policy network passes through a TanH layer.

[0144] The inputs to the policy and value networks (41 dimensions) consist of the robot state (31 dimensions), the external state (8 dimensions), and the clock signal (2 dimensions). The robot state includes the joint positions (12 dimensions) and velocities (12 dimensions) of each actuator, the root joint orientation (4 dimensions), and the angular velocity of the root (pelvis) (3 dimensions). The external state is an 8-dimensional vector. It is described by the 3-dimensional position and 1-dimensional heading of the two upcoming steps T1 = [x1, y1, z1, θ1] and T2 = [x2, y2, z2, θ2]. Because of the periodic reward term, a clock period term is required. This is a 2-dimensional clock signal to prevent learning instability caused by 0-1 jumps using only a 1-dimensional signal. The clock signal is represented as follows, where φ represents the phase of the current motion cycle and L represents the total length of the current motion cycle.

[0145]

[0146] The policy network outputs the torques for each motor. The RL policy outputs the desired joint positions (lower body joints) for the robot's 12 actuators. These are then offset by the motor positions of the half-sitting (HS) and fed into the PD controller to generate the torques for each joint.

[0147] Step 6: Initialize the PPO algorithm instance and set the relevant hyperparameters. The main parameters are as follows.

[0148] Discount factor γ (used to calculate the current value of future rewards), learning rate η (control the step size of weight update), clipping coefficient ∈ (used to limit the amplitude of policy update to avoid excessive update steps), coefficient c1 of the value function (in the total loss, the coefficient of the value function loss term), entropy coefficient c2 (in the total loss, the coefficient that increases the policy entropy to encourage exploration), GAE (Generalized Advantage Estimation) parameter λ (used to balance TD error and Monte Carlo estimation), total number of training iterations T itr (the number of times the training dataset is fully passed), the length of each episode of the task (an episode refers to the process from the beginning to the end of the task, and the maximum number of steps contained in each episode is called its length), the batch size (which defines the number of samples used each time the network is updated), the buffer size, and the number of epochs (the number of times the model will repeatedly learn using the entire collected training set).

[0149] In this embodiment, the discount factor γ = 0.99, the learning rate η = 0.0001, the clipping coefficient ∈ = 0.2, the coefficient of the value function c1 = 0.5, the entropy coefficient c2 = 0.05, the GAE parameter λ = 0.95, and the total number of training iterations T itr=20000, the length of each task episode is 400, the batch size is 64, the buffer size is 12000, and the epochs is 3.

[0150] Step 7: Start an iteration, if the training round T is reached itr , then it ends. If the iteration round reaches the stage change round T J , then adjust the reward function weight according to step 3.

[0151] Step 8: Initialize the environment. This includes setting the robot's initial state and mission objectives, and introducing random perturbations. Initial position and velocity are randomized. By adding a small random perturbation (between -0.02 and 0.02) to the initial position and velocity vectors, randomness is injected into the model's startup state.

[0152] Reset the task. According to the stage in step 3, extract and execute the tasks in the corresponding stage.

[0153] Step 9: Parallel data sampling. Parallel sampling of multiple environments is achieved through Ray, and the collected data from multiple environments is concatenated into a buffer. The buffer size is 12,000.

[0154] Step 10: Advantage function estimation. The advantage function is calculated by subtracting the estimated value function from the cumulative reward at each state step. It intuitively represents the additional reward that can be obtained by taking a specific action compared to the average performance expected by the evaluator.

[0155] Step 11: Batch process the sampled data buffer. Randomly sample the index of the sampled data buffer and then divide the data into multiple batches based on the randomly selected index and the set batch size.

[0156] In this embodiment, the batch size is set to 64 and the buffer size is set to 12000. Therefore, the data is divided into 187 batches.

[0157] Step 12: Start a training epoch. The total number of epochs is epochs. A training epoch is the process of training the model using the entire dataset. Each epoch gradually updates the policy using random mini-batches of data. Mini-batch updates reduce variance and improve learning stability. Epochs is the preset total number of training epochs, which determines the number of times the model repeatedly learns the entire training set, ensuring that the model has ample learning opportunities to absorb the information in the dataset.

[0158] In this embodiment, the total number of epochs is set to 3. If the total number of epochs is exceeded, step 7 is executed.

[0159] Step 13: Calculate the loss function and back-propagate to update the network weights. The loss function of the policy network combines the following three parts: PPO loss (actor_loss), mirror symmetry loss (mirror_loss) and entropy loss (entropypenalty)

[0160] The calculation of the PPO loss (actor_loss) first involves evaluating the ratio of the action probability distribution between the current policy and the previous policy, based on which the gradient clipped surrogate loss is calculated. This loss is adjusted by clipping to moderate the magnitude of the policy update, thereby determining the final actor_loss. The mirror symmetry loss (mirror_loss) is calculated by comparing the difference between the original action (generated directly by the policy network) and the action after mirroring the observation and then mapping it back. The entropy penalty is implemented by calculating the weighted average of the entropy of the policy probability distribution.

[0161] In this implementation, the weights of actor_loss, mirror_loss, and entropy penalty are 1, 0.4, and 0.1, respectively.

[0162] The loss function of the value network is based on the mean squared error between the temporal difference (TD) objective and the value function. This loss aims to optimize the value network so that its predicted state values are closer to the actual rewards.

[0163] Experimental part

[0164] In the embodiment of the present invention, we use the MuJoCo simulation environment to train and test the proposed method. In order to verify the importance of the reference action library for fusion contact parameters, we conducted an ablation experiment. Figure 5 The results presented in show that when the imitation reward term is combined with the cycle reward term ( Figure 5 The obtained walking gait is closer to human walking posture, and the knee joint torque is significantly lower than that of the case where only the cycle reward item is used. In addition, in order to verify the necessity of multi-stage curriculum learning, we tried to remove the multi-stage training and directly conduct more complex task training such as variable speed walking. Figure 6 As shown, the blue line represents the survival time of each round, and the green line represents the sum of the average rewards. Figure 6As shown in the first half of the figure, the humanoid robot's repeated failures in attempting complex maneuvers led to an overly conservative strategy. This caused the reward function to converge prematurely after achieving stable standing, resulting in a lower final reward than the multi-stage curriculum learning approach. This method not only achieved a more human-like gait while ensuring the stability of the training process, but also demonstrated the effectiveness and necessity of multi-stage curriculum learning in training complex humanoid robot walking strategies.

[0165] It should be noted that the specific embodiments are merely explanations and illustrations of the technical solutions of the present invention and cannot be used to limit the scope of protection. Any minor changes made based on the claims and description of the present invention shall still fall within the scope of protection of the present invention.

Claims

1. A gait imitation learning method for a humanoid robot combined with periodic rewards, characterized in that The following steps are involved: Step 1: Construct a reference action library, which contains basic actions and periodic contact information corresponding to each basic action. The periodic contact information includes continuous DS, alternating DS and SS, and alternating SS, where DS is the double-foot contact phase and SS is the single-foot support phase. Step 2: Design a reward function based on the reference action library; Step 3: Train the deep reinforcement learning model based on the reward function designed in step 2; Step 4: Use the trained deep reinforcement learning model to imitate the gait of the humanoid robot; The reward function includes an imitation learning term, a task term, and a regularization term; The imitation learning items include and The task items are The regularization term includes and in, Indicates the base imitation reward, Indicates joint imitation reward, Shows foot contact force imitation reward, Express speed imitation reward, Indicates the reward for the step task. represents the action change regularization reward, represents the moment regularization reward, represents the torque change regularization reward, and q base,t They are respectively represented as the base orientation of the reference action at time t and the actual base orientation, and q t They are respectively represented as the joint position of the reference action at time t and the actual joint position, Indicates the left foot ground reaction force phase indicator, Indicates the right foot ground reaction force phase indicator, Indicates the left foot speed phase indicator, Indicates the right foot speed phase indicator, F left and F right are the normalized ground reaction forces of the left and right feet, respectively, S left and S right Represent the normalized body speed of the left and right feet respectively, a represents the current action, a prev represents the action at the previous moment, τ represents the joint torque, τ prev represents the joint torque at the previous moment, d foot Represents the distance between the foot and the target point, d root Indicates the distance between the base and the target point; The specific steps of step three are: Step 31: Import the reinforcement learning environment and encapsulate the environment; Step 32: Initialize the Actor-Critic algorithm, which includes a policy network, a value network, and a PPO algorithm; Step 33: setting the training stages to stage 1 and stage 2, wherein the first stage is a single task, and the second stage is a composite task, wherein the single task performs a basic movement task, and the composite task performs a combined gait task; Step 3 and 4: Set the training round T itr , and take the training round T itr Half of the phase change round T J , then based on the initialized Actor-Critic algorithm, training is performed, with iteration rounds from 0 to T J Between, the first stage reward function weight is used for training, and the iteration round is T J to T itr Between , the second stage reward function weights are used for training. The first stage reward function weights and the second stage reward function weights are as follows: In the first stage, if t≤T J , then the weight ω is: If t>T J , then the weight ω is: In the second stage, if t≤T J , then the weight ω is: If t>T J , then the weight ω is: Step 35: Initialize the reinforcement learning environment, including setting the robot's initial state and task objectives, and introducing random perturbations; Step 36: Use Ray to sample the reinforcement learning environment in parallel and concatenate the sampling results into a buffer; Step 37: Randomly sample the indexes in the buffer and set the batch size. Then, split the buffer data into multiple small batches based on the selected indexes. Each batch contains a complete data sample. Any remaining samples that are not enough to form a complete small batch are discarded. Step 38: Start a training cycle epoch. The total number of cycles is epochs. The process of training the model using the entire dataset is a training cycle. Each epoch uses a random small batch of data to update the strategy. Step 39: Calculate the loss function and back-propagate to update the network weights. The loss function includes PPO loss, mirror symmetry loss, and entropy loss.

2. The humanoid robot gait imitation learning method combined with periodic rewards according to claim 1 is characterized in that The specific steps of constructing the reference action library are: Step 11: Acquire an original data set, wherein the original data set includes each basic action and a sequence of human lower limb joint position information corresponding to each basic action; Step 1 and 2: Use motion redirection to map the human lower limb joint position information sequence to the humanoid robot to obtain the robot lower limb joint position information sequence; Step 13: Process the robot's lower limb joint position information sequence using the motion constraint and control frequency alignment algorithm to obtain a trimmed robot lower limb joint position information sequence, and use the trimmed robot lower limb joint position information sequence to construct a reference action library.

3. The humanoid robot gait imitation learning method combined with periodic rewards according to claim 2 is characterized in that The motion constraints include position limits and speed limits, specifically: Hip joint pitch, position limit is -10~80, speed limit is 15, Knee joint pitch, position limit is 0~120, speed limit is 22, Ankle joint pitch, position limit is -30~70, speed limit is 10, Ankle side swing, position limit is -15~15, speed limit is 5, The unit of position limit is degree, and the unit of speed limit is rad / s.

4. The method for learning gait imitation of a humanoid robot combined with periodic rewards according to claim 2 is characterized in that The original dataset in step 1 is obtained through the CMU motion capture dataset or the open source human motion library Rofunc.

5. The humanoid robot gait imitation learning method combined with periodic rewards according to claim 1 is characterized in that The policy network and value network are fully connected networks, i.e., multi-layer perceptrons. The value network consists of two hidden layers, each containing 256 neurons and using the ReLU activation function; The policy network consists of two hidden layers, each containing 256 neurons, and uses the ReLU activation function. The output of the network passes through a TanH layer.

6. The method for learning gait imitation of a humanoid robot combined with periodic rewards according to claim 1 is characterized in that The basic movements include squatting, moving forward, and running; The single tasks include walking, standing, backing up, and moving sideways; The complex task includes turning, changing-speed walking, and changing-speed running.

7. The method for learning gait imitation of a humanoid robot combined with periodic rewards according to claim 6 is characterized in that Steps 3 and 5 also include the step of dynamically adjusting training hyperparameters, specifically: Adjust the annealing rate by monitoring the current highest reward and dynamically adjust the termination threshold based on specific operating conditions; The specific method of adjusting the annealing rate by monitoring the current highest reward is: When the reward exceeds two-thirds of the maximum trajectory length and the annealing rate is greater than 0.5, reduce the annealing rate; The dynamic adjustment of the termination threshold based on specific operating conditions is specifically: If the termination condition is enabled and the termination threshold is less than 0.35, the termination threshold is increased according to the number of iterations.

8. The method for learning gait imitation of a humanoid robot combined with periodic rewards according to claim 7 is characterized in that The weights of the PPO loss, mirror symmetry loss, and entropy loss are 1, 0.4, and 0.1, respectively.

Images