Biped humanoid robot robust motion control method based on state reconstruction

Abstract

The invention discloses a biped humanoid robot robust motion control method based on state reconstruction. The biped humanoid robot robust motion control method comprises the following steps of S1, collecting robot sensor data; step S2, state reconstruction; s3, generating a robust control strategy; s4, closed-loop motion control is conducted, specifically, the motion instruction generated in the S3 is combined with real-time feedback of the robot to conduct closed-loop control, and posture adjustment and gait balance of the robot are ensured; according to the method, sensor data are collected in real time, a state reconstruction module based on an encoder-decoder network and a gating cycle unit (GRU) is used for conducting state reconstruction on original data with noise and partial observation missing, and therefore accurate robot motion state estimation is obtained. An asymmetric actor-commentator reinforcement learning strategy is combined with closed-loop feedback to realize robust gait control of the robot on complex terrains (such as stairs, uneven road surfaces and concave-convex terrains).

Description

Technical Field

[0001] The present invention belongs to the field of robot control, and in particular relates to a robust motion control method of a bipedal humanoid robot based on state reconstruction. Background Art

[0002] Conventional bipedal robot motion control methods typically rely on schemes such as zero-moment point (ZMP) control, model predictive control (MPC), and whole-body control (WBC). However, these methods often lack stability and robustness when dealing with high noise levels, partial observation data, and dynamic environmental changes. In recent years, reinforcement learning methods have begun to be applied to robotic control. However, direct application of reinforcement learning is challenging due to sensor noise, incomplete information, and significant discrepancies between simulation and reality. Therefore, there is an urgent need to develop a control method that can combine state reconstruction techniques and utilize online data for real-time optimization to achieve highly robust motion control for bipedal humanoid robots on complex terrain. Summary of the Invention

[0003] The purpose of the present invention is to provide a robust motion control method for a bipedal humanoid robot based on state reconstruction, and a control method that uses online data for real-time optimization to achieve highly robust motion control of a bipedal humanoid robot on complex terrain.

[0004] To solve the above technical problems, the technical solution adopted by the present invention is: a robust motion control method for a bipedal humanoid robot based on state reconstruction, comprising the following steps:

[0005] Step S1: Collect robot sensor data: Use IMU, joint encoders and foot contact sensors to collect dynamic data.

[0006] Step S2: State reconstruction, including the following steps:

[0007] Step S2.1: Input the data output from step S1 above into the encoder module, which uses a one-layer GRU structure to embed historical observation data into the latent space;

[0008] Step S2.2: The decoder generates a complete state estimate based on the encoded output and partial completion information. The state includes the robot's position, velocity, acceleration, and environmental information.

[0009] Step S3: Robust control strategy generation, including the following steps:

[0010] Step S3.1: Using the state estimates generated by the state reconstruction module, we pass them through an asymmetric actor-critic reinforcement learning architecture, where the actor network generates action commands and the critic network provides policy evaluation based on future reward estimates;

[0011] Step S3.2: A composite loss function based on periodic rewards (including state reconstruction error, policy gradient loss, and value estimation loss) is used in the policy optimization process to achieve end-to-end policy learning.

[0012] Step S4: Closed-loop motion control, combining the motion instructions generated in step S3 with the real-time feedback of the robot to perform closed-loop control to ensure the robot's posture adjustment and gait balance.

[0013] Step S5: Online strategy adjustment, including the following steps:

[0014] Step S5.1: Based on real-time sensor feedback (such as joint position error, foot contact status, and external disturbance information), the strategy parameters are adjusted using an adaptive online optimization algorithm.

[0015] Step S5.2: Achieve immediate response to dynamic environmental changes to further improve robustness and adaptability.

[0016] Furthermore, the training of the asymmetric actor-critic architecture in step S3 adopts a policy gradient algorithm, and the future reward discount factor γ and experience replay technology are combined in the optimization process to ensure that the strategy has high robustness in both simulation and actual scenarios.

[0017] Furthermore, the closed-loop motion control system in step S4 includes both a low-level PD controller and a high-frequency closed-loop update mechanism. The operating frequency of the low-level controller reaches 500 Hz, and the high-level strategy update frequency is 100 Hz to meet the requirements of the robot's rapid dynamic response in complex terrain.

[0018] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are:

[0019] 1. The present invention collects sensor data in real time and uses a state reconstruction module based on an encoder-decoder network and a gated recurrent unit (GRU) to reconstruct the state of raw data with noise and partial missing observations, thereby obtaining accurate robot motion state estimation.

[0020] 2. The present invention adopts an asymmetric actor-critic reinforcement learning strategy combined with closed-loop feedback to achieve robust gait control of the robot on complex terrain (such as stairs, uneven roads, and bumpy terrain).

[0021] 3. The online strategy adjustment module of the present invention uses real-time feedback data to update network parameters to adapt to environmental changes. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] To facilitate understanding by those skilled in the art, the present invention is further described below with reference to the accompanying drawings.

[0023] Figure 1 This is the overall system framework of RCSR-Net of the present invention.

[0024] Figure 2 This is the appearance diagram of the YMBot robot of the present invention. DETAILED DESCRIPTION

[0025] The following will clearly and completely describe the technical solutions of the present invention with reference to the embodiments. Obviously, the embodiments described are only some of the embodiments of the present invention, rather than all of them. The embodiments of the present invention and all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0026] The present invention provides a robust motion control method for a bipedal humanoid robot based on state reconstruction, which includes the following steps:

[0027] Step S1: Collect robot sensor data: Use multiple sensors including IMU, joint encoders, and foot contact sensors to normalize the data of each sensor to compensate for the information deviation caused by sensor sampling delay and different sampling rates.

[0028] Step S2: State reconstruction, including the following steps:

[0029] Step S2.1: Input the data output from step S1 above into the encoder module, which uses a one-layer GRU structure to embed historical observation data into the latent space;

[0030] Step S2.2: The encoder receives the preprocessed data and uses GRU to capture time series related information. The decoder generates a complete state estimate based on the encoded output and some completed information. The state includes the robot's posture, velocity, acceleration, and environmental information. Through supervised learning, the state reconstruction network is trained using known complete state data to ensure good prediction accuracy in noisy environments.

[0031] Step S3: Robust control strategy generation, including the following steps:

[0032] Step S3.1: Use the state data output by the state reconstruction module as the input of the reinforcement learning policy network through an asymmetric actor-critic reinforcement learning architecture, where the actor network generates action instructions and the critic network estimates the value of the state and provides feedback on the policy gradient;

[0033] Step S3.2: A composite loss function based on periodic rewards (including state reconstruction error, policy gradient loss, and value estimation loss) is used in the policy optimization process to achieve end-to-end policy learning.

[0034] Step S4: Closed-loop motion control, combining the motion instructions generated in step S3 with the real-time feedback of the robot to perform closed-loop control to ensure the robot's posture adjustment and gait balance.

[0035] Step S5: Online strategy adjustment, including the following steps:

[0036] Step S5.1: Based on real-time sensor feedback (such as joint position error, foot contact status, and external disturbance information), the strategy parameters are adjusted using an adaptive online optimization algorithm.

[0037] Step S5.2: Achieve immediate response to dynamic environmental changes to further improve robustness and adaptability.

[0038] The training of the asymmetric actor-critic architecture in step S3 adopts a policy gradient algorithm. The future reward discount factor γ and the experience replay technique are combined in the optimization process to ensure that the strategy has high robustness in both simulation and actual scenarios.

[0039] Among them, the closed-loop motion control system in step S4 includes both a low-level PD controller and a high-frequency closed-loop update mechanism. The operating frequency of the low-level controller reaches 500Hz, and the high-level strategy update frequency is 100Hz to meet the robot.

[0040] In such Figure 1 As shown, in this embodiment, the robot is equipped with an IMU, joint encoders, and foot contact sensors to collect sensor data. After preprocessing, the data is input into the state reconstruction module, which uses an architecture based on an encoder-decoder network and a gated recurrent unit (GRU) to reconstruct part of the observation data, thereby outputting a complete state vector including the robot's posture, velocity, and local environment elevation information. The state data is then used as input to the reinforcement learning policy network (using an asymmetric actor-critic architecture) to generate robust control instructions, which are finally converted into power commands for each actuator through closed-loop feedback to achieve the purpose of stable walking in complex terrain. At the same time, the online policy adjustment module uses real-time feedback data to update the network parameters to adapt to the requirements of fast dynamic response in complex terrain with changing environments.

[0041] 1. Problem Background

[0042] Our approach adopts the reinforcement learning problem setting, formally represented as a five-tuple M =<S,A,T,O,R,γ> Where S and A represent the state space and action space respectively, T(s'|s,a) represents the state transition dynamics, R(s,a) is the reward function, and γ∈[0,1] is the discount factor. O represents the observation space.

[0043] In our framework, reinforcement learning is implemented through adaptation to simulated environments and complex terrain. In simulated environments, the agent can fully observe the state s∈S, meaning it has full visibility of the current state. However, in real-world tasks on complex terrain, the real environment suffers from partial observability. The agent only has partial observations o∈O, which cannot provide complete state information due to sensor limitations and environmental noise.

[0044] Therefore, the agent's decision-making process is carried out in a discrete-time partially observable Markov decision process (POMDP). The agent needs to make decisions based on incomplete and intermittent observations. In this setting, the agent's goal is to optimize the policy π(a|o≤t) to maximize the expected cumulative reward where r t is the immediate reward obtained at time step t, and γ is the discount factor.

[0045]

[0046] For bipedal humanoid robots, in complex and changing terrain, the strategy must not only consider the current state, but also effectively use partial observation data to make decisions to ensure that the robot can walk steadily and adapt to various terrain changes.

[0047] 2. RCSR-Net: Robust Control and State Reconstruction Network

[0048] In this paper, we propose a framework called RCSR-Net (Robust Control with State Reconstruction Network) to train robust motion control of a bipedal humanoid robot on complex terrain using reinforcement learning. Compared to traditional reinforcement learning methods, RCSR-Net bridges the gap between simulation and complex terrain control through an effective state reconstruction mechanism in a simulated environment, improving the robot's adaptability in dynamic environments.

[0049] The RCSR-Net framework has the following two characteristics:

[0050] ① Encoder-decoder architecture: This architecture can embed partially observed historical sensor data into a latent space and reconstruct the complete state of the robot from it, thereby providing accurate state estimation for the control strategy.

[0051] ② Policy gradient method: Through environmental interaction, the policy gradient method is used to continuously optimize the controller, which can improve the robot's motion performance by optimizing complex goals in a dynamic environment.

[0052] 1) RCSR-Net encoder-decoder architecture

[0053] In an ideal world, if the robot could obtain complete state information, accurate sensor data, and a perfect simulation model, the gap between simulation and reality would be eliminated. However, real-world environments often suffer from sensor data noise, incomplete observations, and model simplifications, leading to gaps between simulation and reality. We view this gap as noise introduced into the real state:

[0054] ① Environmental noise: The real-world environment is complex and unpredictable, especially in complex terrain, where robots face challenges such as rugged terrain and external disturbances.

[0055] ② Dynamic noise: Due to the limitations of dynamic modeling in the physical world, dynamics such as ground friction and object deformation are often simplified in simulations, resulting in inaccuracies.

[0056] ③Sensor noise: The sensor itself inevitably has measurement noise, such as inertial measurement unit (IMU) drift and inaccurate joint position readings.

[0057] ④ Masking noise: Some information cannot be obtained due to the lack of specific sensors on the robot, such as linear velocity and contact force measurement, which is not available in reality.

[0058] To deal with these noises, our framework first simulates noisy observations in a simulation environment and uses an encoder-decoder architecture to denoise these observations, thereby accurately recovering the true state and dynamics, such as Figure 1 As shown, the formula is as follows:

[0059] P(s t )=E o≤t [∫ z P Decoder (s t |z t )·P Encoder (z t |o≤t)];

[0060] 2) Strategy Learning

[0061] In the RCSR-Net framework, we adopt an asymmetric actor-critic architecture that enhances data utilization efficiency during training through optimization strategies and is particularly effective in complex dynamic environments.

[0062]

[0063] The critic part calculates the value function loss:

[0064] L v =‖R t -V(st )‖ 2 ;

[0065] 3. Reward Function

[0066] Our reward function guides the robot to follow velocity commands, maintain a stable gait, and ensure soft contact, thus enabling the robot to perform robust locomotion over complex terrains and obstacles.

[0067] The reward function consists of four key components: (1) velocity tracking, (2) cycle reward, (3) foot trajectory tracking, and (4) regularization term. In addition, we introduce a tracking loss defined as:

[0068] φ(e,w)=exp(-w·||e|| 2 );

[0069] Where e represents the tracking error and w is the strength of the error tolerance.

[0070] A novel aspect of our reward design is that it addresses the sparsity of contact force feedback. Rather than relying solely on contact force, our system focuses on tracking foot velocity. This is achieved by designing foot trajectories that incorporate a predetermined velocity upon ground contact, ensuring a consistent and robust reward signal for each step. This strategy promotes softer ground contact, reduces impact forces, and enhances simulation-to-reality transfer.

[0071] 4. Quintic polynomial foot trajectory interpolation

[0072] In our invention, we improve the locomotion of bipedal humanoid robots by strategically designing foot trajectories. These trajectories are determined using quintic polynomial interpolation, a method that is particularly effective at meeting the precise kinematic requirements of humanoid robots during their gait cycles. This technique not only facilitates smooth motion but also ensures precise foot positioning, which is crucial for maintaining stability and efficiency in humanoid robots.

[0073] Quintic polynomial interpolation has advantages in robotic motion planning because it can provide smooth trajectories and accurately control velocity and acceleration. The general form of the quintic polynomial is:

[0074]

[0075] Here, t represents the time variable, and a0, a1, ..., a5 are coefficients that need to be determined. We use T to denote the duration of the swing phase. In our cycle reward design, when one leg is in the swing phase, the other leg is in the stance phase. A swing phase and a stance phase together complete a complete gait cycle. The trajectory of the robot foot during the swing phase is defined by f(t), which is shaped by kinematic constraints at key moments in the gait cycle. The specific constraints are:

[0076] 1. Initial foot height at t=0: f(0)=h0, where h0 is the initial height.

[0077] 2. Initial foot velocity at t=0: f′(0)=v0, where v0 is the initial velocity.

[0078] 3. Initial foot acceleration at t=0: f″(0)=acc0, where acc0 is the initial acceleration.

[0079] 4. Maximum foot height is reached at the midpoint of the swing phase: f(T / 2) = h max , where h max is the target foot height.

[0080] 5. Foot height at the end of the swing phase: f(T) = h swing , where h swing is the final height.

[0081] 6. Foot velocity at the end of the swing phase: f′(T) = v swing , where v swing is the terminal velocity.

[0082] By adjusting the coefficients of the quintic polynomial, we can not only improve the foot's ability to navigate obstacles, but also maintain smooth motion and reduce shock upon contact. These smooth transitions result in gentler landings, enhancing the robot's stability and promoting the development of efficient and adaptable robots that can safely traverse a variety of terrains.

[0083] 5. Loss Function Design

[0084] The RCSR-Net framework integrates the goals of state reconstruction and policy optimization through a composite loss function. The total loss function is the weighted sum of denoising loss, policy loss, and value loss, formally expressed as:

[0085] L RCSR-Net =L denoise +λ π L π +λ v L v ;

[0086] Among them, λ π and λ v are the weight coefficients of policy loss and value loss, respectively. This approach ensures that the policy can be effectively optimized during state estimation, thereby achieving robustness and adaptability of robot motion control on complex terrains.

[0087] L=L policy +λ1L critic +λ2L rec ;

[0088] Among them, λ1 and λ2 are hyperparameters for adjusting the weights.

[0089] As described above, this embodiment describes in detail a robust control method for a bipedal humanoid robot based on state reconstruction. The overall process includes data acquisition, state reconstruction, strategy generation, closed-loop control, and online optimization. By introducing an encoder-decoder state reconstruction network, an asymmetric actor-critic architecture, quintic polynomial foot trajectory planning, and domain randomization techniques, this invention not only achieves smooth robot motion in complex terrain, but also exhibits high environmental adaptability and robustness.

[0090] The preferred embodiments of the present invention disclosed above are intended only to help illustrate the present invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the present invention to specific embodiments. Obviously, many modifications and variations are possible based on the contents of this specification. These embodiments are selected and described in detail in this specification to better explain the principles and practical applications of the present invention, thereby enabling those skilled in the art to better understand and utilize the present invention. The present invention is limited only by the claims and their full scope and equivalents.

Claims

1. A robust motion control method for a bipedal humanoid robot based on state reconstruction, characterized in that: The following steps are involved: Step S1: Collect robot sensor data and use IMU, joint encoders and foot contact sensors to collect dynamic data. Step S2: State reconstruction, including the following steps: Step S2.1: Input the data output from step S1 above into the encoder module, which uses a one-layer GRU structure to embed historical observation data into the latent space; Step S2.2: The decoder generates a complete state estimate based on the encoded output and partial completion information. The state includes the robot's position, velocity, acceleration, and environmental information. Step S3: Robust control strategy generation, including the following steps: Step S3.1: Using the state estimates generated by the state reconstruction module, we pass them through an asymmetric actor-critic reinforcement learning architecture, where the actor network generates action commands and the critic network provides policy evaluation based on future reward estimates; Step S3.2: A composite loss function based on periodic rewards (including state reconstruction error, policy gradient loss, and value estimation loss) is used in the policy optimization process to achieve end-to-end policy learning. Step S4: Closed-loop motion control, combining the motion instructions generated in step S3 with the real-time feedback of the robot to perform closed-loop control to ensure the robot's posture adjustment and gait balance. Step S5: Online strategy adjustment, including the following steps: Step S5.1: Based on real-time sensor feedback (such as joint position error, foot contact status, and external disturbance information), the strategy parameters are adjusted using an adaptive online optimization algorithm. Step S5.2: Achieve immediate response to dynamic environmental changes to further improve robustness and adaptability.

2. The robust motion control method for a bipedal humanoid robot based on state reconstruction according to claim 1, characterized in that: The training of the asymmetric actor-critic architecture in step S3 adopts a policy gradient algorithm. The future reward discount factor γ and the experience replay technique are combined in the optimization process to ensure that the strategy has high robustness in both simulation and actual scenarios.

3. The method for robust motion control of a bipedal humanoid robot based on state reconstruction according to claim 1, characterized in that: The closed-loop motion control system in step S4 includes both a low-level PD controller and a high-frequency closed-loop update mechanism. The operating frequency of the low-level controller reaches 500 Hz, and the high-level strategy update frequency is 100 Hz to meet the robot's requirement for fast dynamic response in complex terrain.

Images