An adaptive gait conversion method for humanoid robots

Abstract

The present invention proposes an adaptive gait transition method for a humanoid robot. First, inverse kinematics is used to calculate the joint positions of a target robot. After scaling, the reference position and orientation of the target robot in each frame are set as the reference position and orientation of the source motion data. Then, a humanoid motion learning framework is built to provide more accurate and robust potential state estimation, thereby ultimately improving the performance of the strategy. Finally, a gait transition algorithm is generated based on the humanoid motion learning framework. The present invention adopts unified motion redirection technology and an asymmetric action-comment architecture in combination with a WGAN-div discriminator to achieve a smooth transition from walking to running without interruption, thereby improving the naturalness and coherence of the motion. The present invention not only promotes the development of humanoid robot technology, but also provides new possibilities for the application of robots in a wider range of fields, especially in scenarios that require close interaction between robots and humans.

Description

Technical Field

[0001] The present invention relates to the field of robot technology, and in particular to an adaptive gait conversion method for a humanoid robot. Background Art

[0002] There are significant technical challenges in the current field of robotics, especially in the walking and running motion control of humanoid robots. Traditional robotic motion control methods rely on accurate joint trajectory tracking and gait feature replication to achieve motion patterns closer to humans. However, these methods face many limitations when dealing with complex movements and designing natural human movements, mainly in the following aspects:

[0003] Movement Coordination: The fluidity and adaptability of human movement require precise coordination among multiple joints, which is a complex problem for robotic control systems.

[0004] Motion Imitation Learning: Although imitation learning provides a way for robots to learn tasks from expert demonstrations, it is highly dependent on high-quality expert demonstrations, which are often difficult to obtain and have poor generalization ability to new scenarios.

[0005] Generative Adversarial Imitation Learning and Adversarial Motion Prior Methods: These methods have made significant progress on quadruped robots, but their application to humanoid robots still needs further development to achieve optimal motion performance.

[0006] The comparison with the existing technology is as follows:

[0007] Technical comparison with patent CN202311631726.0 "Robot control method, device, robot and computer-readable storage medium"

[0008] Patent CN202311631726.0 uses the least squares generative adversarial network (Least-Squares GAN, LSGAN) loss to replace reward function learning, which helps alleviate the gradient vanishing problem of the cross entropy loss function used in traditional GAN. In addition, the discriminator output is between -1 and 1, which provides a more consistent scale for the generative model and makes the training process more stable. The present invention uses Wasserstein loss, which can directly use the output of the discriminator as a reward to learn a more characteristic reward function in the state space, which is more helpful for learning highly dynamic actions.

[0009] Patent CN202311631726.0 integrates adversarial learning components and reinforcement learning algorithms to guide the control strategy to produce behaviors consistent with the data distribution of the hybrid reference motion. The present invention also uses an asymmetric Actor-Critic architecture and an estimator to optimize the learning of the entire strategy, enhance the stability and generalization of the robot's different gaits, and achieve similar effects on the real machine as in the simulation environment.

[0010] However, there are the following problems in the prior art:

[0011] 1. Movement coordination and fluency: Existing robot motion control systems are deficient in simulating the fluidity and adaptability of human motion, especially in the transition between different motion modes (such as walking and running).

[0012] 2. Dependence on expert demonstrations: Traditional imitation learning methods rely on high-quality expert demonstrations, which are costly and time-consuming to obtain, limiting the robot’s ability to learn and adapt in new environments.

[0013] 3. Generalization ability: Existing methods have limited generalization ability in new scenarios, which may lead to performance degradation or error accumulation in practical applications.

[0014] 4. Stability of motion mode transition: During the transition between different motion modes (such as from walking to running), the robot’s stability and adaptability are insufficient, affecting its performance in dynamic environments.

[0015] 5. Energy efficiency: Existing robot motion patterns may not be optimized in terms of energy consumption, especially in applications that require long-term operation, which limits the practicality of the robot.

[0016] 6. Accuracy of motion imitation: Although imitation learning can replicate certain gait features, it still has certain limitations when imitating complex human movements, especially in dynamically changing environments.

[0017] 7. Conversion from simulation to reality: During the conversion from the simulation environment to the real world, the robot's motion control strategy may not be directly applicable and requires additional adjustments and optimization. Summary of the invention

[0018] To solve the above technical problems, the present invention proposes an adaptive gait transition method for a humanoid robot, which significantly improves the generalization ability of the humanoid robot in new scenarios through imitation learning, reduces the need for adaptability to specific environments, and effectively solves the problem of transition from a simulated environment to the real world, so that the motion control strategies learned in a simulated environment can be directly applied to robots in the real world.

[0019] To achieve the above object, the technical solution adopted by the present invention is:

[0020] A method for adaptive gait conversion of a humanoid robot, characterized by comprising the following steps:

[0021] S1, action remapping:

[0022] Collect real human motion data and reposition the motion data to fit the humanoid robot as follows:

[0023] First, the real human motion is defined as the source motion, and the source motion data is collected through a motion capture device.

[0024] Then, a set of keypoints are selected in the source motion and the target humanoid robot respectively, and the 3D position of each keypoint in the source motion is given at each time step t by scaling the source motion data to fit the size of the target humanoid robot.

[0025] Then, the joint positions of the target humanoid robot are calculated by using inverse kinematics, and after scaling, the reference position and orientation of the target humanoid robot in each frame are set as the reference position and orientation of the source motion data;

[0026] S2. Build a humanoid motion learning framework:

[0027] S21. Model humanoid motion control as a partially observable Markov decision process, expressed as

[0028] Among them, the complete state, partial observation state and action are represented as and

[0029] The state transition probability is defined as p(s t+1 |s t , a t ),

[0030] The strategy π selects an action based on the current observation: a t ~π(·|o t ),

[0031] The discount factor γ, used to weigh future rewards,

[0032] The reward function r is defined as r t = r(s t , a t ), the goal is to maximize the cumulative discounted reward, expressed as

[0033] S22. Build an asymmetric action-comment architecture, as follows:

[0034] Build the estimator network, evaluation network, and action network,

[0035] The action-critic architecture consists of an estimator module and a controller module. The estimator module contains the environment state s t Data and evaluation network for evaluating the state of the environment, estimator and context state vector z t and estimated line speed The controller module includes a PD controller for controlling the humanoid robot.

[0036] The estimator is connected to the estimator network, which is denoted as ε and takes o t：t-h =[o t , o t-1 ,…o t-h ] T As input, the output is connected to the action network, which can output the context state vector z t and the estimated line speed To the action network,

[0037] After being parsed and planned by the action network, it is sent to the PD controller, which controls the humanoid robot to generate actions.

[0038] S23. Construct a reward system, as follows:

[0039] First, the WGAN-div discriminator is connected to the asymmetric action-comment architecture. The PD controller transmits the generated action to the WGAN-div discriminator for discrimination.

[0040] The reward system includes style rewards and task rewards. The style rewards are determined by the WGAN-div discriminator, and the task rewards are determined by the PD controller.

[0041] The WGAN-div discriminator is a discriminator network D parameterized by θ. θ is integrated into the action-critic framework to evaluate the Wasserstein-1 distance between the generated data distribution and the reference data distribution.

[0042] Reward system In addition to the task rewards from the environment t In addition, the WGAN-div discriminator also provides style rewards r for policy training s , so that the actions output by the strategy maintain the same style as the reference actions,

[0043] Then, by adopting Wasserstein GAN with divergence penalty for reward function learning, the Wasserstein-1 distance is defined as:

[0044]

[0045] in Indicates that with boundary value P r and P g Take the maximum lower bound on the set of all joint distributions λ(x, y),

[0046] WGAN-div Discriminator D θ The Lipschitz constraint Maximize the following objectives:

[0047]

[0048] This constraint is enforced using a divergence penalty,

[0049] D is optimized by maximizing the expected difference between real samples and generated samples. θ , and use a divergence penalty to enforce a 1-Lipschitz constraint, where C 1 Represents the set of 1-Lipschitz-continuous functions:

[0050]

[0051] Discriminator D θ A behavior that uses multiple state transitions as input to facilitate transitions between actions.

[0052] As a preferred technical solution of the present invention: in step S1, toes, ankles, knees, elbows and hands of the source motion and target humanoid robots are selected as key points.

[0053] As a preferred technical solution of the present invention: in step S22, the estimator includes a simulator, an encoder and a decoder, the speed component is trained under the supervision of the actual linear speed in the simulator, and the potential state z t The training is performed using the variational autoencoder paradigm.

[0054] When the potential state z t When fed into the decoder, the decoder generates observations for the next frame and the total loss of the estimator is defined as: in represents the regression loss for predicting line speed, represents the VAE loss of reconstructing observations,

[0055] Estimated line speed and the true speed v t The mean square error loss is applied between

[0056] For the variational autoencoder loss, use:

[0057]

[0058] The first term represents the predicted next observation value The actual observed value o t+1 The reconstruction loss between them, the second term quantifies the latent variable z t The Kullback-Leibler divergence between the prior and posterior distributions of .

[0059] Compared with the prior art, the present invention has the following beneficial effects:

[0060] The present invention achieves remarkable technical effects of a humanoid robot in imitating human walking and running, as follows:

[0061] 1. Seamless motion transition:

[0062] The unified motion redirection technology and asymmetric action-comment architecture adopted in the present invention, combined with the WGAN-div discriminator, achieve a smooth transition from walking to running of the humanoid robot without interruption, effectively improving the naturalness and coherence of the motion.

[0063] 2. Improve the accuracy of motion imitation:

[0064] Through the imitation learning component and the style reward provided by the WGAN-div discriminator, the present invention significantly improves the accuracy of the humanoid robot in imitating human motion, making the motion of the humanoid robot closer to natural human behavior.

[0065] 3. Enhance the stability and adaptability of exercise:

[0066] The integrated estimator module enhances stability during the transition between different gait patterns, allowing the humanoid robot to move more stably and adaptively when facing different environments and task requirements.

[0067] 4. Optimize energy efficiency:

[0068] The algorithm of the present invention can achieve higher energy efficiency by imitating human movement, which is particularly important for designing battery-driven or long-term operating robotic systems.

[0069] 5. Improve the compliance of sports:

[0070] The humanoid robot of the present invention can exhibit lower contact force and more symmetrical gait by imitating human movement, thereby achieving more natural and smoother gait execution and improving movement compliance.

[0071] 6. Reduce the workload of development and deployment:

[0072] By automating the motion learning and conversion process, the present invention reduces the workload in the development and deployment of humanoid robot motion control, saving time and cost.

[0073] 7. Improve generalization ability:

[0074] Compared with traditional methods that rely on high-quality expert demonstrations, the method of the present invention significantly improves the generalization ability of humanoid robots in new scenarios through imitation learning, reducing the need for adaptability to specific environments.

[0075] The method of the present invention effectively solves the problem of conversion from a simulated environment to the real world, so that the motion control strategy learned in the simulated environment can be directly applied to robots in the real world.

[0076] Through the above technical effects, the present invention not only promotes the development of humanoid robot technology, but also provides new possibilities for the application of robots in a wider range of fields, especially in scenarios that require close interaction between robots and humans. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] Figure 1 It is the selection point of the key point on the humanoid robot;

[0078] Figure 2 It is the overall framework diagram of the present invention;

[0079] Figure 3 It is the estimator training framework diagram. DETAILED DESCRIPTION

[0080] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments:

[0081] like Figure 1-3 As shown, the present invention proposes a humanoid robot adaptive gait conversion method, comprising the following steps:

[0082] S1, action remapping:

[0083] Collect real human motion data and reposition the motion data to fit the humanoid robot as follows:

[0084] First, the real human motion is defined as the source motion, and the source motion data is collected through a motion capture device.

[0085] Then, a set of keypoints are selected in the source motion and the target humanoid robot respectively, and the 3D position of each keypoint in the source motion is given at each time step t by scaling the source motion data to fit the size of the target humanoid robot.

[0086] Then, the joint positions of the target humanoid robot are calculated by using inverse kinematics, and after scaling, the reference position and orientation of the target humanoid robot in each frame are set as the reference position and orientation of the source motion data;

[0087] The above steps achieve accurate redirection of motion data by precisely selecting key points of real humans and humanoid robots, namely toes, ankles, knees, elbows and hands, and by utilizing inverse kinematics calculations.

[0088] Data scaling and basic position adjustment are performed to ensure that the basic position and orientation of the humanoid robot remain consistent with the human motion data when simulating human motion, thereby improving the naturalness and consistency of the motion.

[0089] By imitating human movements, humanoid robots can exhibit lower contact forces and more symmetrical gaits, achieving more natural and smoother gait execution and improving movement compliance.

[0090] S2. Build a humanoid motion learning framework:

[0091] S21. Model humanoid motion control as a partially observable Markov decision process, expressed as

[0092] Among them, the complete state, partial observation state and action are represented as and

[0093] The state transition probability is defined as p(s t+1 |s t , a t ),

[0094] The strategy π selects an action based on the current observation: a t ~π(·|o t ),

[0095] The discount factor γ, used to weigh future rewards,

[0096] The reward function r is defined as r t = r(s t , a t ), the goal is to maximize the cumulative discounted reward, expressed as

[0097] S22. Build an asymmetric action-comment architecture, as follows:

[0098] Build the estimator network, evaluation network, and action network,

[0099] The action-critic architecture consists of an estimator module and a controller module. The estimator module contains the environment state st Data and evaluation network for evaluating the state of the environment, estimator and context state vector z t and estimated line speed The controller module includes a PD controller for controlling the humanoid robot.

[0100] The estimator is connected to the estimator network, which is denoted as ε and takes o t：t-h =[o t , o t-1 ,…o t-h ] T As input, the output is connected to the action network, which can output the context state vector z t and the estimated line speed To the action network,

[0101] After being parsed and planned by the action network, it is transmitted to the PD controller, which controls the humanoid robot to generate actions;

[0102] The robot’s adaptability and stability when switching between different gait modes (including walking and running) are improved by building an asymmetric action-critic architecture, which includes an estimator to enhance the stability of the transition phase and uses imitation learning to directly learn human-like movements, such as Figure 2 shown.

[0103] The humanoid robot directly absorbs human movements through imitation learning, while using an estimator to enhance stability when transitioning between different gait patterns, improving the robot's adaptability and stability.

[0104] S23. Construct a reward system, as follows:

[0105] First, the WGAN-div discriminator is connected to the asymmetric action-comment architecture. The PD controller transmits the generated action to the WGAN-div discriminator for discrimination.

[0106] The reward system includes style rewards and task rewards. The style rewards are determined by the WGAN-div discriminator, and the task rewards are determined by the PD controller.

[0107] The WGAN-div discriminator is a discriminator network D parameterized by θ. θ is integrated into the action-critic framework to evaluate the Wasserstein-1 distance between the generated data distribution and the reference data distribution.

[0108] Reward system In addition to the task rewards from the environment t In addition, the WGAN-div discriminator also provides style rewards r for policy training s, so that the actions output by the strategy maintain the same style as the reference actions,

[0109] Then, by adopting Wasserstein GAN with divergence penalty for reward function learning, the Wasserstein-1 distance is defined as:

[0110]

[0111] in Indicates that with boundary value P r and P g Take the maximum lower bound on the set of all joint distributions λ(x, y),

[0112] WGAN-div Discriminator D θ The Lipschitz constraint Maximize the following objectives:

[0113]

[0114] This constraint is enforced using a divergence penalty,

[0115] D is optimized by maximizing the expected difference between real samples and generated samples. θ , and use a divergence penalty to enforce a 1-Lipschitz constraint, where C 1 Represents the set of 1-Lipschitz-continuous functions:

[0116]

[0117] Discriminator D θ A behavior that uses multiple state transitions as input to facilitate transitions between actions.

[0118] The reward system is a combination of style rewards and task rewards. By utilizing the style rewards provided by the WGAN-div discriminator and the task rewards provided by the environment, the strategy training is optimized to improve the accuracy of imitating human movements.

[0119] Enforcement of 1-Lipschitz constraints: Through the divergence penalty mechanism, the discriminator is ensured to satisfy the 1-Lipschitz continuity constraint, thereby providing a stable training signal and improving learning efficiency.

[0120] Through the humanoid motion learning framework and the style reward provided by the WGAN-div discriminator, the present invention significantly improves the accuracy of humanoid robots in imitating human motion, making the robot's motion closer to natural human behavior.

[0121] The integrated estimator component enhances stability during transitions between different gait patterns, allowing the robot to move more stably and adaptively when facing different environments and task demands.

[0122] In step S22, the estimator includes a simulator, an encoder and a decoder, the speed component is trained under the supervision of the actual linear speed in the simulator, and the potential state z t The training is performed using the variational autoencoder paradigm.

[0123] When the potential state z t When fed into the decoder, the decoder generates observations for the next frame and the total loss of the estimator is defined as: in represents the regression loss for predicting line speed, represents the VAE loss of reconstructing observations,

[0124] To estimate the line speed, and the true speed v t The mean square error loss is applied between

[0125] For the variational autoencoder loss, use:

[0126]

[0127] The first term represents the predicted next observation value The actual observed value o t+1 The reconstruction loss between them, the second term quantifies the latent variable z t The Kullback-Leibler divergence between the prior and posterior distributions of .

[0128] The unified motion redirection technology and asymmetric action-criticism architecture adopted in this paper, combined with the WGAN-div discriminator, achieve a smooth transition of the robot from walking to running without interruption, improving the naturalness and coherence of the movement.

[0129] Compared with traditional methods that rely on high-quality expert demonstrations, the method of the present invention significantly improves the generalization ability of humanoid robots in new scenarios through imitation learning, reducing the need for adaptability to specific environments.

[0130] The method of the present invention effectively solves the problem of conversion from a simulated environment to the real world, so that the motion control strategy learned in the simulated environment can be directly applied to robots in the real world.

[0131] Through the above technical effects, the present invention not only promotes the development of humanoid robot technology, but also provides new possibilities for the application of robots in a wider range of fields, especially in scenarios that require close interaction between robots and humans.

[0132] The above description is only a preferred embodiment of the present invention and does not constitute any other form of limitation to the present invention. Any modification or equivalent change made based on the technical essence of the present invention still falls within the scope of protection required by the present invention.

Claims

1. A method for adaptive gait conversion of a humanoid robot, characterized in that: The steps include: S1, action remapping: Collect real human motion data and reposition the motion data to fit the humanoid robot as follows: First, the real human motion is defined as the source motion, and the source motion data is collected through a motion capture device. Then, a set of keypoints are selected in the source motion and the target humanoid robot respectively, and the 3D position of each keypoint in the source motion is given at each time step t by scaling the source motion data to fit the size of the target humanoid robot. Then, the joint positions of the target humanoid robot are calculated by using inverse kinematics, and after scaling, the reference position and orientation of the target humanoid robot in each frame are set as the reference position and orientation of the source motion data; S2. Build a humanoid motion learning framework: S21. Model humanoid motion control as a partially observable Markov decision process, expressed as Among them, the complete state, partial observation state and action are represented as and The state transition probability is defined as p(s t+1 |s t , a t ), The strategy π selects an action based on the current observation: a t ~π(·|o t ), The discount factor γ, used to weigh future rewards, The reward function r is defined as r t = r(s t , a t ), the goal is to maximize the cumulative discounted reward, expressed as S22. Build an asymmetric action-comment architecture, as follows: Build the estimator network, evaluation network, and action network, The action-critic architecture consists of an estimator module and a controller module. The estimator module contains the environment state s t Data and evaluation network for evaluating the state of the environment, estimator and context state vector z t and estimated line speed The controller module includes a PD controller for controlling the humanoid robot. The estimator is connected to the estimator network, which is denoted as ε and takes o t：t-h =[o t , o t-1 ,…o t-h ] T As input, the output is connected to the action network, which can output the context state vector z t and the estimated line speed To the action network, After being parsed and planned by the action network, it is sent to the PD controller, which controls the humanoid robot to generate actions. S23. Construct a reward system, as follows: First, the WGAN-div discriminator is connected to the asymmetric action-comment architecture. The PD controller transmits the generated action to the WGAN-div discriminator for discrimination. The reward system includes style rewards and task rewards. The style rewards are determined by the WGAN-div discriminator, and the task rewards are determined by the PD controller. The WGAN-div discriminator is a discriminator network D parameterized by θ. θ is integrated into the action-critic framework to evaluate the Wasserstein-1 distance between the generated data distribution and the reference data distribution. Reward system In addition to the task rewards from the environment t In addition, the WGAN-div discriminator also provides style rewards r for policy training s , so that the actions output by the strategy maintain the same style as the reference actions, Then, by adopting Wasserstein GAN with divergence penalty for reward function learning, the Wasserstein-1 distance is defined as: in Indicates that with boundary value P r and P g The maximum lower bound of the set of all joint distributions λ(x, y) of WGAN-div discriminator D θ The Lipschitz constraint Maximize the following objectives: This constraint is enforced using a divergence penalty, D is optimized by maximizing the expected difference between real samples and generated samples. θ , and use a divergence penalty to enforce a 1-Lipschitz constraint, where C 1 Represents the set of 1-Lipschitz-continuous functions: Discriminator D θ A behavior that uses multiple state transitions as input to facilitate transitions between actions.

2. The method for adaptive gait conversion of a humanoid robot according to claim 1, characterized in that: In step S1, the toes, ankles, knees, elbows and hands of the source motion and target humanoid robots are selected as key points.

3. The method for adaptive gait conversion of a humanoid robot according to claim 1, characterized in that: In step S22, the estimator includes a simulator, an encoder and a decoder, the speed component is trained under the supervision of the actual linear speed in the simulator, and the potential state z t The training is performed using the variational autoencoder paradigm. When the potential state z t When fed into the decoder, the decoder generates observations for the next frame and the total loss of the estimator is defined as: in represents the regression loss for predicting line speed, represents the VAE loss of reconstructed observations, the estimated linear velocity and the true speed v t The mean square error loss is applied between For the variational autoencoder loss, use: The first term represents the predicted next observation value The actual observed value o t+1 The reconstruction loss between them, the second term quantifies the latent variable z t The Kullback-Leibler divergence between the prior and posterior distributions of .

Images