A gait control method for humanoid robot based on two-stage training strategy

Abstract

The present invention proposes a gait control method for a humanoid robot based on a two-stage training strategy. First, a reinforcement learning stable walking motion learning framework is established, and two-stage training is performed based on the established reinforcement learning stable walking motion learning framework, and finally the gait control of the humanoid robot is realized. The present invention effectively promotes the development of humanoid robot technology and provides new possibilities for future research and innovation. Through these technical effects, the present invention not only promotes the development of humanoid robot technology, but also provides new possibilities for the application of robots in a wider range of fields, especially in scenarios that require close interaction between robots and humans.

Description

Technical Field

[0001] The present invention relates to the field of robotics technology, and in particular to a gait control method for a humanoid robot based on a double-stage training strategy. Background Art

[0002] As an important branch of robotics, humanoid robots are inspired by human body structure and movement ability, which makes them more flexible and adaptable when performing tasks. With the continuous advancement of artificial intelligence and machine learning technology, the application scope of humanoid robots has expanded from simple industrial automation to more complex environments, such as disaster relief, complex terrain exploration, home services, and as assistants for humans in space exploration.

[0003] However, although humanoid robots have made remarkable achievements in simulating human walking and performing tasks, they still face many challenges when faced with the ever-changing and challenging terrains in the real world.

[0004] To overcome these challenges, researchers have been exploring new algorithms and techniques to improve the autonomy and robustness of humanoid robots. Among them, reinforcement learning, as an effective strategy, has demonstrated its potential in solving complex control problems in multiple fields. Through reinforcement learning, robots can learn the optimal gait control strategy in their interaction with the environment.

[0005] The prior art has the following technical problems:

[0006] 1. Adaptability problem: Existing gait control systems are often optimized for specific types of terrain. When encountering terrain that has not been trained, the robot's walking performance will drop significantly.

[0007] 2. Joint oscillation: During the actual walking process, the robot's joints may experience unexpected oscillations, which not only affects the robot's movement efficiency but may also cause damage to the robot's hardware.

[0008] 3. Sim-to-Real Conversion Problem: Control strategies that perform well in simulation environments are often difficult to apply directly to real robots because simulation environments cannot fully simulate the complexity of the real world.

[0009] 4. Inaccurate speed control: When walking at high speed or on changing terrain, the robot's gait control accuracy will be affected, resulting in unstable walking or falling. Summary of the invention

[0010] To solve the above technical problems, the present invention proposes a gait control method for a humanoid robot based on a two-stage training strategy, which effectively improves the applicability of the humanoid robot in a variety of practical application scenarios, promotes the development of humanoid robot technology, and provides new possibilities for future research and innovation.

[0011] To achieve the above object, the technical solution adopted by the present invention is:

[0012] A gait control method for a humanoid robot based on a two-stage training strategy, characterized in that it comprises the following steps:

[0013] S1, build a reinforcement learning stable walking motion learning framework:

[0014] S11, humanoid motion control is modeled as a partially observable Markov decision process, expressed as

[0015] The complete state, partial observed state and action are represented as

[0016] The state transition probability is defined as p(s t+1 |s t , a t ), the strategy π is based on the current observation results

[0017] Select action: a t ~π(·|o t ),

[0018] The discount factor γ is used to weigh future rewards,

[0019] The reward function r is defined as r t = r(s t ,a t ), the goal is to maximize the cumulative discounted reward, expressed as

[0020] S12, build an asymmetric action-comment architecture, as follows:

[0021] Build the actor network, critic network, and estimator network,

[0022] An estimator for enhancing the stability of the transition phase and directly learning human-like motion using imitation learning and a PD controller for controlling the humanoid robot and receiving feedback signals therefrom are provided.

[0023] The estimator is connected to the estimator network, and the estimator network takes o t:t-h =[o t ,o t-1 ,…o t-h ]T As input, it can output the context state vector z t and the estimated line speed

[0024] The estimator includes a decoder, an encoder, and a simulator. The speed component is trained under the supervision of the actual line speed in the simulator, and the potential state z t The training is performed using the variational autoencoder paradigm.

[0025] Potential state z t is fed into the decoder, which generates the observations for the next frame,

[0026] The total loss of the estimator is defined as: in represents the regression loss for predicting line speed, represents the VAE loss of reconstructing observations,

[0027] To estimate the linear velocity, and the true speed v t The mean square error loss is applied between

[0028] For the variational autoencoder loss, by using:

[0029]

[0030] The first term represents the predicted next observation value The actual observed value o t+1 The reconstruction loss between them, the second term quantifies the latent variable z t The Kullback-Leibler divergence between the prior and posterior distributions of ;

[0031] S13, formulate a reward system as follows:

[0032] The reward system consists of several components:

[0033] r=r gait +r command +r root -r renergy +p·r reference

[0034] where r gait Adjust gait pattern, r command Facilitates speed command tracking, r root Ensure that the base of the humanoid robot maintains an upright posture and correct orientation, renergy Penalize excessive energy consumption, r referenceTo encourage following the reference motion, p belongs to {0,1} only during the training phase, once it is equal to 1, the reward is cancelled during the training phase.

[0035] The detailed reward function and weights are as follows:

[0036]

[0037] S2, two-stage training based on the reinforcement learning stable walking motion learning framework:

[0038] S21, Phase 1 - Basic gait training:

[0039] First, the humanoid robot is trained on a preset simple terrain, using a human-designed reference motion that is designed based on the structure and kinematics of the humanoid robot.

[0040] Then, multiple reward functions are introduced to encourage the humanoid robot to quickly learn stable walking, including reference motion gait patterns, velocity command tracking, upright posture of the robot base, and correct orientation;

[0041] S22, Phase 2 - Adaptive Gait Training:

[0042] First, the reward for the reference motion is removed, allowing the humanoid robot to learn an adaptive gait on a preset difficult terrain.

[0043] Then, the proportion of difficult terrain and the height of the climb were increased to train the adaptability of the humanoid robot;

[0044] S23, imperative course learning:

[0045] A curriculum learning strategy based on speed commands is adopted to gradually improve the speed tracking ability of the humanoid robot, thereby achieving a smooth transition from low speed to high speed during the training process;

[0046] S24, state estimation method:

[0047] Design an estimator network to estimate the velocity and latent variables of the humanoid robot using historical observation data;

[0048] S25, Reinforcement Learning:

[0049] First, the reinforcement learning algorithm is used to optimize the gait control strategy of the humanoid robot.

[0050] Then, through interaction with the environment, the humanoid robot learns how to walk stably on terrains of varying difficulty;

[0051] S26, Simulation to Reality Conversion:

[0052] A variety of randomization and delay strategies are introduced. The randomization strategies include observation randomization and robot property randomization, and the delay strategies include random delays of actions and torques.

[0053] As a preferred technical solution of the present invention: the operating frequency of the PD controller is 1000 Hz.

[0054] As a preferred technical solution of the present invention: in step S22, the preset difficult terrain includes a plane, steps, a slope and stairs.

[0055] As a preferred technical solution of the present invention: in step S25, the reinforcement learning algorithm is the PPO algorithm.

[0056] Compared with the prior art, the present invention has the following beneficial effects:

[0057] The present invention achieves remarkable technical effects of a humanoid robot in imitating human walking and running, as follows:

[0058] 1. Enhanced adaptability:

[0059] The robot is able to adapt to changing and complex terrain conditions, including but not limited to flat surfaces, steps, slopes and stairs.

[0060] 2. Improved speed control accuracy:

[0061] The robot exhibits improved accuracy when executing velocity command tracking, reducing velocity control errors caused by terrain changes or speed increases.

[0062] 3. Reduce joint oscillation:

[0063] Through the improved gait control strategy, the oscillation of the robot joints is effectively controlled, thereby improving the walking stability and the durability of the robot.

[0064] 4. Successful simulation to reality conversion:

[0065] The robot is able to reproduce in the real world the gaits learned in simulation, significantly improving the success rate of deployment from simulation to reality.

[0066] 5. Improved energy efficiency:

[0067] Optimized gait and motion control reduce unnecessary energy consumption and improve the energy efficiency of the robot when performing tasks.

[0068] 6. Enhanced robustness:

[0069] The robot can maintain stable walking when facing external disturbances and non-ideal conditions, showing greater robustness.

[0070] 7. Improved mission success rate:

[0071] When performing specific tasks such as climbing stairs and crossing slopes, the robot showed a high success rate, proving the effectiveness of the control strategy.

[0072] 8. Improved walking stability:

[0073] The robot can maintain a more stable posture when walking on different terrains, reducing the risk of falling and losing balance.

[0074] 9. Expanded application scope:

[0075] The technical solution of the present invention improves the applicability of the robot in a variety of practical application scenarios, such as search and rescue, industrial automation, and home services.

[0076] The technical solution of the present invention can effectively promote the development of humanoid robot technology and provide new possibilities for future research and innovation. Through these technical effects, the present invention not only promotes the development of humanoid robot technology, but also provides new possibilities for the application of robots in a wider range of fields, especially in scenarios that require close interaction between robots and humans. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] Figure 1 It is the overall framework diagram of the present invention. DETAILED DESCRIPTION

[0078] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments:

[0079] like Figure 1 As shown, the present invention proposes a gait control method for a humanoid robot based on a two-stage training strategy, comprising the following steps:

[0080] S1, build a reinforcement learning stable walking motion learning framework:

[0081] S11, humanoid motion control is modeled as a partially observable Markov decision process, expressed as

[0082] The complete state, partial observed state and action are represented as The state transition probability is defined as p(s t+1 |s t , a t ), strategy π selects actions based on the current observations: a t ~π(·|o t ),

[0083] The discount factor γ is used to weigh future rewards,

[0084] The reward function r is defined as r t = r(s t ,a t ), the goal is to maximize the cumulative discounted reward, expressed as

[0085] S12, build an asymmetric action-comment architecture, as follows:

[0086] Build the actor network, critic network, and estimator network,

[0087] An estimator for enhancing the stability of the transition phase and directly learning motions similar to humans by imitation learning and a PD controller for controlling the humanoid robot and receiving feedback signals therefrom are set. The operating frequency of the PD controller is 1000 Hz.

[0088] The estimator is connected to the estimator network, and the estimator network takes o t:t-h =[o t ,o t-1 ,…o t-h ] T As input, it can output the context state vector z t and the estimated line speed

[0089] The estimator network is designed to process multiple frames of observations to infer features of the environment and robot state. By combining current and historical observations, our estimator network can provide more accurate and robust latent state estimates, ultimately improving the performance of the policy.

[0090] The estimator includes a decoder, an encoder, and a simulator. The speed component is trained under the supervision of the actual line speed in the simulator, and the potential state z t The training is performed using the variational autoencoder paradigm.

[0091] Potential state z t is fed into the decoder, which generates the observations for the next frame,

[0092] The total loss of the estimator is defined as: in represents the regression loss for predicting line speed, represents the VAE loss of reconstructing observations,

[0093] To estimate the linear velocity, and the true speed v t The mean square error loss is applied between

[0094] For the variational autoencoder loss, by using:

[0095]

[0096] The first term represents the predicted next observation value The actual observed value o t+1 The reconstruction loss between them, the second term quantifies the latent variable z t The Kullback-Leibler divergence between the prior and posterior distributions of ;

[0097] S13, formulate a reward system as follows:

[0098] Reward systems are used to drive humanoid robots to achieve agile, stable, and energy-efficient locomotion in diverse environments and tasks.

[0099] The reward system consists of several components:

[0100] r=r gait +r command +r root -r renergy +p·r reference

[0101] where r gait Adjust gait pattern, r command Facilitates speed command tracking, r root Ensure that the base of the humanoid robot maintains an upright posture and correct orientation, renergy Penalize excessive energy consumption, r reference To encourage following the reference motion, p belongs to {0,1} only during the training phase, once it is equal to 1, the reward is cancelled during the training phase.

[0102] The detailed reward function and weights are as follows:

[0103]

[0104] S2, two-stage training based on the reinforcement learning stable walking motion learning framework:

[0105] S21, Phase 1 - Basic gait training:

[0106] First, the humanoid robot is trained on a preset simple terrain, using a human-designed reference motion that is designed based on the structure and kinematics of the humanoid robot.

[0107] Then, multiple reward functions are introduced to encourage the humanoid robot to quickly learn stable walking, including reference motion gait patterns, velocity command tracking, upright posture of the robot base, and correct orientation;

[0108] S22, Phase 2 - Adaptive Gait Training:

[0109] First, the reward for the reference motion is removed, allowing the humanoid robot to learn an adaptive gait on a preset difficult terrain.

[0110] Then, the proportion of difficult terrain and the height of the climb were increased to train the adaptability of the humanoid robot;

[0111] Through the above training, the humanoid robot of the present invention has strong adaptability and can adapt to changeable and complex terrain conditions, including but not limited to planes, steps, slopes and stairs.

[0112] When performing specific tasks such as climbing stairs and crossing slopes, the humanoid robot showed a high success rate, proving the effectiveness of the control strategy.

[0113] When walking on terrains of varying difficulty, the humanoid robot can maintain a more stable posture, reducing the risk of falling and losing balance. Compared with existing technologies, walking stability is greatly improved.

[0114] S23, imperative course learning:

[0115] A curriculum learning strategy based on speed commands is adopted to gradually improve the speed tracking ability of the humanoid robot, thereby achieving a smooth transition from low speed to high speed during the training process;

[0116] The humanoid robot can achieve a smooth transition from low speed to high speed, effectively improving the efficiency of energy utilization. It reduces unnecessary energy consumption through optimized gait and motion control, and improves the energy efficiency of the robot when performing tasks.

[0117] S24, state estimation method:

[0118] Design an estimator network to estimate the velocity and latent variables of the humanoid robot using historical observation data;

[0119] Through imperative curriculum learning and state estimation methods, the speed control accuracy of the humanoid robot is effectively improved, so that the humanoid robot can show higher accuracy when executing speed command tracking, and reduce the speed control error caused by terrain changes or speed increase.

[0120] The design of the estimator network can provide more accurate and robust latent state estimates and ultimately improve the performance of the strategy, allowing the humanoid robot to maintain stable walking and exhibit greater robustness in the face of external disturbances and non-ideal conditions.

[0121] S25, Reinforcement Learning:

[0122] First, the reinforcement learning algorithm is used to optimize the gait control strategy of the humanoid robot.

[0123] Then, through interaction with the environment, the humanoid robot learns how to walk stably on terrains of varying difficulty;

[0124] At the core of reinforcement learning is the signal of the reward system, which is a key component guiding the learning process. The design of the reward function is crucial to shaping the strategy of the intelligent agent because it directly affects the actions chosen and the movement patterns exhibited by the humanoid robot. In the movement of the humanoid robot, the signal of the reward system reflects the efficiency and effectiveness of the movement, and embodies the safety and adaptability required to navigate in unpredictable terrains and situations.

[0125] Through reinforcement learning, the joint oscillation of the humanoid robot is effectively reduced, and through the improved gait control strategy, the oscillation of the robot's joints is effectively controlled, thereby improving the walking stability and durability of the robot.

[0126] S26, Simulation to Reality Conversion:

[0127] A variety of randomization and delay strategies are introduced. The randomization strategies include observation randomization and robot property randomization, and the delay strategies include random delays of actions and torques.

[0128] The successful conversion from simulation to reality enables the humanoid robot to reproduce the gaits learned in the simulation environment in the real world, significantly improving the success rate of deployment from simulation to reality.

[0129] In step S22, the preset difficult terrain includes a plane, steps, a slope and stairs.

[0130] In step S25, the reinforcement learning algorithm is the PPO algorithm.

[0131] The technical solution of the present invention improves the applicability of the robot in a variety of practical application scenarios, such as search and rescue, industrial automation, and home services.

[0132] The technical solution of the present invention can effectively promote the development of humanoid robot technology and provide new possibilities for future research and innovation. Through these technical effects, the present invention not only promotes the development of humanoid robot technology, but also provides new possibilities for the application of robots in a wider range of fields, especially in scenarios that require close interaction between robots and humans.

[0133] The above description is only a preferred embodiment of the present invention and does not constitute any other form of limitation to the present invention. Any modification or equivalent change made based on the technical essence of the present invention still falls within the scope of protection required by the present invention.

Claims

1. A gait control method for a humanoid robot based on a two-stage training strategy, characterized in that: The steps include: S1, build a reinforcement learning stable walking motion learning framework: S11, humanoid motion control is modeled as a partially observable Markov decision process, expressed as The complete state, partial observed state and action are represented as p is the state transition probability and is defined as p(s t+1 |s t , a t ), the strategy π selects an action based on the current observation: a t ~π(·|o t ), the discount factor γ is used to measure future rewards, and the reward function r is defined as r t = r(s t ,a t ), the goal is to maximize the cumulative discounted reward, expressed as J(π) = E τ～p(·|π) [Σ t γ t r(s t ,a t )]; S12, build an asymmetric action-comment architecture, as follows: Build the actor network, critic network, and estimator network, An estimator for enhancing the stability of the transition phase and directly learning human-like motion using imitation learning and a PD controller for controlling the humanoid robot and receiving feedback signals therefrom are provided. The estimator is connected to the estimator network, and the estimator network takes o t:t-h =[o t ,o t-1 ,…o t-h ] T As input, it can output the context state vector z t and the estimated line speed The estimator includes a decoder, an encoder, and a simulator. The speed component is trained under the supervision of the actual line speed in the simulator, and the potential state z t The training is performed using the variational autoencoder paradigm. Potential state z t is fed into the decoder, which generates the observations for the next frame, The total loss of the estimator is defined as: in represents the regression loss for predicting line speed, represents the VAE loss of reconstructing observations, To estimate the linear velocity, and the true speed v t The mean square error loss is applied between For the variational autoencoder loss, by using: The first term represents the predicted next observation value The actual observed value o t+1 The reconstruction loss between them, the second term quantifies the latent variable z t The Kullback-Leibler divergence between the prior and posterior distributions of ; S13, formulate a reward system as follows: The reward system consists of several components: r＝r gait +r command +r root -r renergy +p·r reference where r gait Adjust gait pattern, r command Facilitates speed command tracking, r root Ensure that the base of the humanoid robot maintains an upright posture and correct orientation, renergy Penalize excessive energy consumption, r reference To encourage following the reference motion, p belongs to {0,1} only during the training phase, once it is equal to 1, the reward is cancelled during the training phase. S2, two-stage training based on the reinforcement learning stable walking motion learning framework: S21, Phase 1 - Basic gait training: First, the humanoid robot is trained on a preset simple terrain, using a human-designed reference motion that is designed based on the structure and kinematics of the humanoid robot. Then, multiple reward functions are introduced to encourage the humanoid robot to quickly learn stable walking, including reference motion gait patterns, velocity command tracking, upright posture of the robot base, and correct orientation; S22, Phase 2 - Adaptive Gait Training: First, the reward for the reference motion is removed, allowing the humanoid robot to learn an adaptive gait on a preset difficult terrain. Then, the proportion of difficult terrain and the height of the climb were increased to train the adaptability of the humanoid robot; S23, imperative course learning: A curriculum learning strategy based on speed commands is adopted to gradually improve the speed tracking ability of the humanoid robot, thereby achieving a smooth transition from low speed to high speed during the training process; S24, state estimation method: Design an estimator network to estimate the velocity and latent variables of the humanoid robot using historical observation data; S25, Reinforcement Learning: First, the reinforcement learning algorithm is used to optimize the gait control strategy of the humanoid robot. Then, through interaction with the environment, the humanoid robot learns how to walk stably on terrains of varying difficulty; S26, Simulation to Reality Conversion: A variety of randomization and delay strategies are introduced. The randomization strategies include observation randomization and robot property randomization, and the delay strategies include random delays of actions and torques.

2. The method for controlling gait of a humanoid robot based on a two-stage training strategy according to claim 1, characterized in that: In step S12, the operating frequency of the PD controller is 1000 Hz.

3. The method for controlling gait of a humanoid robot based on a two-stage training strategy according to claim 1, characterized in that: In step S22, the preset difficult terrain includes a plane, steps, a slope and stairs.

4. The method for controlling gait of a humanoid robot based on a two-stage training strategy according to claim 1, characterized in that: In step S25, the reinforcement learning algorithm is the PPO algorithm.

Images