Humanoid robot fall recovery control method based on multi-stage curriculum learning

Abstract

The human-like robot fall recovery control method based on multi-stage course learning provided in the application divides the robot fall and getting-up action into multiple key frames, and learns frame by frame in stages, simultaneously introduces a hybrid internal model and a reinforcement learning framework, the hybrid internal model obtains the current observation value of the robot in real time, obtains feature information through velocity estimation, enhances the robustness of the strategy, reduces the difference between simulation and reality, combines the hybrid internal optimization and the proximal policy optimization of the reinforcement learning framework, interacts with the simulation environment, makes the robot learn to quickly and stably transition from one key frame to the next key frame, realizes the learning of complex human motion behaviors, optimizes the safety performance of human-computer interaction, enables the robot to reliably operate in medical care, home service and other scenes requiring close human-machine cooperation, and lays a technical foundation for constructing a more intelligent and safer next-generation service robot system.

Description

Technical Field

[0001] This invention relates to the field of robotics, specifically to a fall recovery control method for humanoid robots based on multi-stage curriculum learning. Background Technology

[0002] Humanoid robots, as an important branch of robotics, aim to mimic human motor abilities, possessing complex movements such as walking, standing, and recovery from falls. Although significant progress has been made in motion control in humanoid robots in recent years, their autonomous recovery capabilities after falls still face major challenges. Traditional fall recovery methods mostly rely on preset motion trajectories or rule-based control strategies. These methods struggle to adapt to diverse fall postures and complex environmental disturbances, and exhibit low robustness and adaptability in practical applications.

[0003] Therefore, the existing technology presents the following technical challenges:

[0004] Adaptability issues: Existing multi-stage course learning methods typically optimize the action strategies for each stage independently, but discontinuity and instability can occur during stage transitions, leading to recovery failures.

[0005] Long recovery time and instability: Existing methods are difficult to satisfy both recovery time and stability at the same time, resulting in a failure rate in multiple recovery attempts.

[0006] The challenge of transitioning from simulation to reality: strategies that perform well in simulation often fail in real-world environments, mainly due to environmental differences and unmodeled dynamic disturbances.

[0007] Furthermore, existing humanoid robots generally suffer from the following problems:

[0008] High requirements for motor performance: Existing recovery strategies place high demands on the precision, response speed, and shock resistance of the joint motors, leading to increased hardware costs and making them difficult to implement on ordinary platforms. Summary of the Invention

[0009] To address the aforementioned technical problems, this invention proposes a fall recovery control method for humanoid robots based on multi-stage learning. Through a multi-stage training strategy, the method progressively learns recovery paths from different fall postures to a stable standing position. Furthermore, domain randomization and reward constraints are introduced at each stage to improve the strategy's environmental adaptability and recovery efficiency. Experiments demonstrate that this method can achieve efficient and stable fall recovery under various complex ground conditions.

[0010] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0011] A fall recovery control method for humanoid robots based on multi-stage curriculum learning is characterized by the following steps:

[0012] S1. Construct a hybrid internal model that integrates regular observations and privileged observations, and use historical observation data to estimate the robot's velocity and latent variables;

[0013] S2. Decompose the robot's falling and getting up action into continuous keyframes, and learn them frame by frame in stages;

[0014] S3. Introduce a reinforcement learning framework that combines hybrid internal optimization and proximal policy optimization. The reinforcement learning framework utilizes regular observations and privileged observations stored in the hybrid internal model. The source encoder extracts feature representations from regular observations, and the target encoder generates contrastive targets to guide representation learning. These latent representations are used by the actuator network to generate actions and are valued by the evaluator. The actions are then executed in the simulator, and rewards are calculated based on similarity to predefined keyframe targets.

[0015] S4. Add a mirror loss function to measure the policy symmetry of the robot before and after mirror mapping, and ensure that the robot performs consistently in left-right or up-down symmetrical movements.

[0016] S5, design a reward function to guide the robot to learn fall recovery actions from different aspects.

[0017] As a preferred technical solution of the present invention: In step S1, the network of the hybrid internal model is taken As input, and output the context state vector and estimated linear velocity ,

[0018] The hybrid internal model network is a feature that processes multiple frames of observations to infer the environment and robot state. By combining current and historical observations, the estimator network can provide latent state estimates.

[0019] As a preferred technical solution of the present invention: In step S2, the phased learning is specifically as follows:

[0020] The first stage requires the robot to learn from the first frame to the second frame. After successfully learning the second frame, it will move on to the next learning stage, which requires it to learn from the third frame, and so on.

[0021] In the second stage, after the robot can successfully stand up in the simulation, various randomization methods are added. When the robot can still stand up after adding randomization, restricted training is added again. After completing the real-world verification, the third stage begins.

[0022] In the third stage, all fourteen-dimensional joint positions output by the hybrid internal model network are used to enable the robot to recover from any supine posture to a flat position and finally stand up.

[0023] As a preferred technical solution of the present invention: In step S3, the proximal policy optimization adopts Proximal Policy Optimization as the optimization algorithm. By continuously interacting with the simulation environment, the robot can gradually master the ability to quickly and stably transition from one keyframe to the next keyframe.

[0024] As a preferred technical solution of the present invention, step S4 is specifically as follows:

[0025] When the robot executes policy π(s) in state s, it transforms the state to a symmetric state M[s] through mirror mapping M. Then, the policy network calculates the new action π(M[s]). If the policy is perfectly symmetric, this action, after another mirror mapping M[π(M[s])], should be completely consistent with the policy output in the original state. The mirror loss is defined as the difference in the L2 norm between the two, and the formula is:

[0026]

[0027] By minimizing this loss, we can ensure that the robot performs consistently in left-right or up-down symmetrical movements.

[0028] As a preferred technical solution of the present invention: in step S5, the reward function is specifically as follows:

[0029] The reference joint position is used to encourage the robot to learn the current action in the next keyframe.

[0030] Base height and height increase are used to encourage the robot to gradually increase its height.

[0031] Orientation and body up are used to help the robot learn to a semi-standing position more quickly by adjusting its hand support posture.

[0032] The hand force increase is used in the first two frames of training to encourage the robot to coordinate hand and leg movements.

[0033] Feet force increase is used to encourage robots to land on both feet.

[0034] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0035] This invention achieves remarkable technical results in humanoid robots mimicking human walking and running through a series of innovative technical means, as detailed below:

[0036] 1. Improve the speed of recovery after a fall:

[0037] In real-world robot testing, the average recovery time from lying supine to standing is only 1.2 seconds.

[0038] 2. Improve speed control precision:

[0039] The robot exhibits higher accuracy in executing speed commands and tracking, reducing speed control errors caused by terrain changes or speed increases.

[0040] The introduction of internal models effectively predicts future states, reduces control delays, and enables smooth acceleration and deceleration.

[0041] 3. Reduce joint vibration:

[0042] Smooth joint transitions reduce reliance on high-torque motors, decrease wear, and extend system lifespan.

[0043] 4. Successful simulation-to-reality conversion:

[0044] The robot can reproduce fall recovery actions learned in the simulation environment in the real world, which significantly improves the success rate of deployment from simulation to reality.

[0045] 5. Optimized energy consumption and motor requirements:

[0046] Optimized motion planning and motion control reduce unnecessary energy consumption, improve the robot's energy efficiency when performing tasks, and reduce motor torque requirements.

[0047] 6. Enhanced robustness:

[0048] When faced with external disturbances and non-ideal conditions, the robot can quickly recover from a fall, stand up, and remain stable, demonstrating stronger robustness.

[0049] 7. Expanded application scope:

[0050] The technical solution of this invention improves the applicability of robots in various practical application scenarios, such as disaster scenarios with many interferences, robots are prone to falling over, and factory floors are uneven. This method can quickly restore the robot to a standing state after falling, avoiding secondary damage to the robot.

[0051] 8. Promote technological progress:

[0052] The implementation of this invention will promote the development of humanoid robot technology, and the innovative fall recovery safety mechanism will provide new possibilities for future research and innovation.

[0053] Through the aforementioned technical effects, this invention optimizes the safety performance of human-computer interaction, enabling robots to operate reliably in scenarios requiring close human-machine collaboration, such as medical care and home services. This lays the technical foundation for building a more intelligent and safer next-generation service robot system and is expected to drive the entire robot industry towards higher safety standards. Attached Figure Description

[0054] Figure 1 This is an overall framework diagram of the present invention;

[0055] Figure 2 This is a keyframe for planning the fall recovery state;

[0056] Figure 3 This is a detailed table diagram of the reward function. Detailed Implementation

[0057] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments:

[0058] like Figure 1 As shown, the fall recovery control method for humanoid robots based on multi-stage course learning proposed in this invention includes the following steps:

[0059] S1. Construct a hybrid internal model that integrates regular observations and privileged observations, and use historical observation data to estimate the robot's velocity and latent variables;

[0060] S2. Decompose the robot's falling and getting up action into continuous keyframes, and learn them frame by frame in stages;

[0061] S3. Introduce a reinforcement learning framework that combines hybrid internal optimization and proximal policy optimization. The reinforcement learning framework utilizes regular observations and privileged observations stored in the hybrid internal model. The source encoder extracts feature representations from regular observations, and the target encoder generates contrastive targets to guide representation learning. These latent representations are used by the actuator network to generate actions and are valued by the evaluator. The actions are then executed in the simulator, and rewards are calculated based on similarity to predefined keyframe targets.

[0062] S4. Add a mirror loss function to measure the policy symmetry of the robot before and after mirror mapping, and ensure that the robot performs consistently in left-right or up-down symmetrical movements.

[0063] S5, design a reward function to guide the robot to learn fall recovery actions from different aspects.

[0064] In step S1, the network of the hybrid internal model is taken As input, and output the context state vector and estimated linear velocity ,

[0065] The hybrid internal model network is designed to process multi-frame observations to infer the features of the environment and robot state. By combining current and historical observations, the estimator network can provide latent state estimates.

[0066] A reinforcement learning framework combining Hybrid Intrinsic Optimization (HIM) and Proximal Policy Optimization (PPO) is proposed for motion learning in humanoid robots. This framework utilizes regular and privileged observations stored in a buffer. A source encoder extracts feature representations from the regular observations, while a target encoder generates contrastive targets to guide representation learning. These latent representations are used by an actuator network to generate actions, which are then valued by an evaluator. The actions are executed in a simulator, and rewards are calculated based on similarity to predefined keyframe targets.

[0067] Updates are performed using policy gradients, thereby facilitating the learning of complex humanoid motion behaviors. The network portion of the hybrid internal model is... As input, and output the context state vector and estimated linear velocity .

[0068] This network is designed to process features from multiple frames of observations to infer the environment and robot state. By combining current and historical observations, our estimator network can provide more accurate and robust latent state estimates, ultimately improving the performance of the policy.

[0069] In step S2, the phased learning is as follows:

[0070] The first stage requires the robot to learn from the first frame to the second frame. After successfully learning the second frame, it will move on to the next learning stage, which requires it to learn from the third frame, and so on.

[0071] In the second stage, after the robot can successfully stand up in the simulation, various randomization methods are added. When the robot can still stand up after adding randomization, restricted training is added again. After completing the real-world verification, the third stage begins.

[0072] In the third stage, all fourteen-dimensional joint positions output by the hybrid internal model network are used to enable the robot to recover from any supine posture to a flat position and finally stand up.

[0073] In the initial training, only ten of the fourteen joints were used (four arm joints and six leg joints, excluding the yaw and roll angle joints of the legs). Keyframe training was conducted as follows: Figure 2As shown, the first stage requires the robot to learn from the first frame to the second frame. After successfully learning the second frame, it moves on to the next stage, learning from the third frame, and so on. This training method significantly reduces the difficulty of parameter tuning and accelerates the training speed. Once the robot can successfully stand in the simulation, various randomizations are added, such as randomization of joint positions, velocities, and robot orientation angles. If the robot can still stand after adding randomizations, further training with torque and velocity constraints is required. After completing real-world verification, the final stage is entered, expanding the joints and using all 14-dimensional joint positions output by the network, enabling the robot to recover from any supine posture to a flat position and ultimately stand. This progressive learning method significantly accelerates the robot's mastery of complex recovery movements.

[0074] In step S3, the proximal policy optimization uses Proximal Policy Optimization as the optimization algorithm. By continuously interacting with the simulation environment, the robot can gradually master the ability to quickly and stably transition from one keyframe to the next.

[0075] Step S4 is as follows:

[0076] When the robot executes policy π(s) in state s, it transforms the state to a symmetric state M[s] through mirror mapping M. Then, the policy network calculates the new action π(M[s]). If the policy is perfectly symmetric, this action, after another mirror mapping M[π(M[s])], should be completely consistent with the policy output in the original state. The mirror loss is defined as the difference in the L2 norm between the two, and the formula is:

[0077]

[0078] By minimizing this loss, the robot can be made to perform consistently in left-right or up-down symmetrical movements, thereby improving the coordination and stability of recovery actions.

[0079] In step S5, the reward function is as follows:

[0080] The reference joint position is used to encourage the robot to learn the current action in the next keyframe.

[0081] Base height and height increase are used to encourage the robot to gradually increase its height.

[0082] Orientation and body up are used to help the robot learn to a semi-standing position more quickly by adjusting its hand support posture.

[0083] The hand force increase is used in the first two frames of training to encourage the robot to coordinate hand and leg movements.

[0084] Feet force increase is used to encourage robots to land on both feet.

[0085] like Figure 3 The reward function and its corresponding formula are shown. The reference joint pos encourages the current action to learn the next keyframe and is the most important of all rewards. Since the height of the keyframe gradually increases, the base height and height increase are applicable in all keyframe learning processes, encouraging the robot to gradually increase its height. Orientation and body up help the hand support state learn to the semi-standing state more quickly. Hand force increase is mainly used for the first two frames of training, encouraging the hands to cooperate with the legs to achieve the effect of quickly getting up.

[0086] Due to design differences, some robots cannot land on their feet while lying flat. Feet force increase during training encourages foot landing, helping robots with this limitation learn the target movement more quickly. The parameters used include: feet orientation, stand on feet, standstill, stand still vel (stationary standing position), ankle vel limit, dof vel limit, dof acc (joint acceleration), base acc (center of gravity acceleration), action smoothness, and foot contact forces. These parameters help the robot stand stably, preventing jumping due to excessive speed in the x and z directions, penalizing feet leaving the ground, and ensuring a stable standing process. Torque limit and dof pos limits are used to minimize damage to the actual robot's motors. Hip roll and yaw are used in the final stage of joint dimension expansion to prevent excessive leg spread angles caused by hip roll and yaw.

[0087] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any other way. Any modifications or equivalent changes made based on the technical essence of the present invention shall still fall within the scope of protection claimed by the present invention.

Claims

1. A fall recovery control method for humanoid robots based on multi-stage curriculum learning, characterized by: Includes the following steps: S1. Construct a hybrid internal model that integrates regular observations and privileged observations, and use historical observation data to estimate the robot's velocity and latent variables; In step S1, the network of the hybrid internal model is taken As input, and output the context state vector and estimated linear velocity , The hybrid internal model network is a feature that processes multi-frame observations to infer the environment and robot state. By combining current and historical observations, the estimator network can provide latent state estimates. S2. Decompose the robot's falling and getting up action into continuous keyframes, and learn them frame by frame in stages; S3. Introduce a reinforcement learning framework that combines hybrid internal optimization and proximal policy optimization. The reinforcement learning framework utilizes regular observations and privileged observations stored in the hybrid internal model. The source encoder extracts feature representations from regular observations, and the target encoder generates contrastive targets to guide representation learning. These latent representations are used by the actuator network to generate actions and are valued by the evaluator. The actions are then executed in the simulator, and rewards are calculated based on similarity to predefined keyframe targets. S4. Add a mirror loss function to measure the policy symmetry of the robot before and after mirror mapping, and ensure that the robot performs consistently in left-right or up-down symmetrical movements. Step S4 is as follows: When the robot executes policy π(s) in state s, it transforms the state to a symmetric state M[s] through mirror mapping M. Then, the policy network calculates the new action π(M[s]). If the policy is perfectly symmetric, this action, after another mirror mapping M[π(M[s])], should be completely consistent with the policy output in the original state. The mirror loss is defined as the difference in the L2 norm between the two, and the formula is: ； By minimizing this loss, we can ensure that the robot performs consistently in left-right or up-down symmetrical movements. S5, design a reward function to guide the robot to learn fall recovery actions from different aspects.

2. The fall recovery control method for humanoid robots based on multi-stage course learning according to claim 1, characterized in that: In step S2, the phased learning is as follows: The first stage requires the robot to learn from the first frame to the second frame. After successfully learning the second frame, it will move on to the next learning stage, which requires it to learn from the third frame, and so on. In the second stage, after the robot can successfully stand up in the simulation, various randomization methods are added. When the robot can still stand up after adding randomization, restricted training is added again. After completing the real-world verification, the third stage begins. In the third stage, all fourteen-dimensional joint positions output by the hybrid internal model network are used to enable the robot to recover from any supine posture to a flat position and finally stand up.

3. The fall recovery control method for humanoid robots based on multi-stage course learning according to claim 1, characterized in that: In step S3, the proximal policy optimization uses Proximal Policy Optimization as the optimization algorithm. By continuously interacting with the simulation environment, the robot can gradually master the ability to quickly and stably transition from one keyframe to the next.

4. The fall recovery control method for humanoid robots based on multi-stage course learning according to claim 1, characterized in that: In step S5, the reward function is as follows: The reference joint position is used to encourage the robot to learn the current action in the next keyframe. Base height and height increase are used to encourage the robot to gradually increase its height. Orientation and body up are used to help the robot learn to a semi-standing position more quickly by adjusting its hand support posture. The hand force increase is used in the first two frames of training to encourage the robot to coordinate hand and leg movements. Feet force increase is used to encourage robots to land on both feet.

Images