Robust walking control method and system for high-gear-ratio humanoid robot based on potential dynamics self-adaption

Abstract

The invention belongs to the technical field of robot control, and discloses a high-gear-ratio humanoid robot robust walking control method and system based on potential dynamics self-adaption, and the method comprises the steps: designing a control strategy training frame based on deep reinforcement learning, and carrying out the optimization through an Actor-Critic structure and a PPO algorithm; constructing a potential dynamic adaptive network LDAN, and extracting environment and ontology dynamic parameters through a variational auto-encoder; designing a multi-dimensional reward function; constructing a periodic gait library by using von Mises distribution and motion capture data; gradually introducing terrain disturbance and dynamic change through curriculum type simulation training; the trained strategy and the LDAN module are deployed on the high-gear-ratio driven humanoid robot, the problems that the high-gear-ratio humanoid robot is poor in motion stability, poor in adaptability and insufficient in action expression in a variable environment are solved, and the high-gear-ratio driven humanoid robot has the advantages of being high in robustness, high in natural expressivity and high in migration ability.

Description

Technical Field

[0001] The present invention belongs to the technical field of robot control, and in particular relates to a robust walking control method and system for a high gear ratio humanoid robot based on potential dynamics adaptation. Background Art

[0002] Humanoid robots, due to their human-like morphology and potential for walking, manipulating, and interacting in unstructured environments, have garnered widespread attention in recent years in industrial, medical, and home service scenarios. To achieve efficient, stable, and expressive humanoid motion control, current technologies primarily rely on methods such as motion planning, motion capture and imitation, system identification, and reinforcement learning. However, in real-world applications, migrating strategies from simulation to reality remains challenging due to limitations such as sensor accuracy, dynamic modeling errors, motor nonlinearity, and contact uncertainty.

[0003] Especially in the context of humanoid robots widely adopting high-gear-ratio servo motors to achieve high output torque, their inherent structural backlash, friction nonlinearity, and low back-drivability cause significant variations in their dynamic behavior in actual operation, making it difficult to accurately model using traditional system identification methods. Furthermore, motors can overheat under high loads or for extended periods, further degrading servo performance and limiting the stability and expressiveness of control strategies in real-world environments.

[0004] Existing reinforcement learning methods offer excellent policy expression capabilities by training policy networks in simulation environments to generate control actions. However, due to significant dynamic differences between simulation and reality, directly deploying conventional policy networks often leads to control failures. Some research has employed domain randomization or residual models for policy transfer, which, while improving robustness to a certain extent, still suffers from issues such as slow convergence, poor generalization, and unnatural gait expression.

[0005] Furthermore, traditional gait control often relies on fixed cycles, predefined trajectories, or regular functions to generate lower-limb movements. Upper-limb movements have limited expressive power, making it difficult to capture the body language characteristics of natural human walking. While recent research has incorporated motion capture data for upper-limb movement generation, the lack of a unified generation mechanism and phase coordination framework has resulted in insufficient full-body coordination and unnatural performance.

[0006] Therefore, how to integrate dynamic features such as external disturbances and motor changes under the reinforcement learning architecture, and combine upper limb expression with lower limb support to generate coordinated gait, has become an important research direction for achieving robust and expressive humanoid motion control. Summary of the Invention

[0007] The purpose of the present invention is to provide a robust walking control method and system for a high gear ratio humanoid robot based on potential dynamics adaptation, so as to solve the technical problems of poor motion stability, weak control adaptability and insufficient motion expression of high gear ratio humanoid robots in changing environments.

[0008] To solve the above technical problems, the specific technical solutions of the present invention, a method and system for robust walking control of a high gear ratio humanoid robot based on potential dynamics adaptation, are as follows:

[0009] A robust walking control method for a high gear ratio humanoid robot based on latent dynamics adaptation comprises the following steps:

[0010] Step 1: Design a deep reinforcement learning training framework for the control strategy: Construct a control strategy based on deep reinforcement learning, adopt an actor-critic structure, and use the proximal policy optimization (PPO) algorithm for training optimization;

[0011] Step 2: Latent Dynamics Adaptive Network (LDAN) Design: Build an encoder-decoder network with a variational autoencoder (VAE) structure. This network takes in historical observation data from the robot and outputs a set of latent vectors representing the environment and the robot's dynamic parameters. This latent vector serves as an additional input to the control strategy to achieve implicit dynamic compensation.

[0012] Step 3: Reward function design: Construct a multi-dimensional reward function to ensure that the strategy is stable and has good expressiveness;

[0013] Step 4: Periodic gait library construction: Use the von Mises distribution to construct periodic guidance signals, combine them with motion capture data to generate natural upper limb movements, use an inverse kinematics solver to synchronously generate foot trajectories, and ultimately generate a full-body gait as a reference to guide policy learning;

[0014] Step 5: Course-based simulation training: Gradually introduce terrain disturbances and dynamic parameter changes in the simulation platform to improve the policy generalization ability; after training, the policy and LDAN module are directly deployed on the real robot system;

[0015] Step 6: Hardware Deployment: Suitable for Humanoid Robots with High Gear Ratio Drive, 30Hz Control Frequency, and Real-time Inference

[0016] The latent vector of the LDAN module is used to dynamically adjust the control strategy.

[0017] Furthermore, the step 1 includes the following steps:

[0018] Deep reinforcement learning training framework for control policies:

[0019] The policy network Actor and the value network Critic in the Actor-Critic structure respectively undertake the functions of action decision-making and value evaluation. During the training process, the proximal policy optimization algorithm PPO is introduced to iteratively update the strategy. The Actor network generates the next action instruction based on the robot state observation, and the Critic network estimates the value function of the current state to guide the strategy update. The PPO algorithm constrains the change range of the new and old strategies through the clipped policy gradient loss function, thereby stabilizing the training process while ensuring learning efficiency. First, the robot's state is initialized in the simulation environment, and the Actor network is based on the current observation s t Output action a t , the action acts on the simulation environment to obtain a new state and reward value r t ; At the same time, the Critic network is based on s t Calculate the state value V(s t ) is used to evaluate strategy performance;

[0020] LDAN adopts the structure of a variational autoencoder (VAE), consisting of an encoder network and a decoder network. The encoder network takes the sensor information of the robot during operation as input and extracts hidden features that characterize the dynamic characteristics of the environment. The encoder maps these features into a low-dimensional latent vector z, namely the dynamic encoding vector. During training, LDAN introduces the Kullback-Leibler divergence regularization term to bring the encoded posterior distribution q(z|x) closer to the prior distribution p(z). The decoder network takes the latent vector z as input and attempts to reconstruct the output related to the environmental dynamics. The encoder and decoder are trained together so that the latent vector z fully contains information about the differences in environmental dynamics. The training loss consists of the reconstruction error and the KL divergence term. The reconstruction error measures the gap between the decoder output and the actual observation, and the KL divergence term restricts the distribution of z to avoid overfitting to a specific environment.

[0021] The reasoning process of LDAN is as follows: during policy execution, every several control steps, the robot state observation sequence within the latest time window is input into the LDAN encoder. The encoder outputs the dynamic latent vector z of the current environment, which characterizes the dynamic deviation between the current environment and the standard model. This latent vector is then input into the high-level layer of the behavior policy network Actor, enabling the policy to make action decisions based on the current estimated dynamic characteristics. In addition, LDAN is adaptively updated every time new environmental conditions change.

[0022] Furthermore, step 1 includes the composition of the dynamic encoding vector and the inference input and output: the dynamic encoding vector d is the potential dynamic feature vector extracted by LDAN, which contains dynamic parameter information of multiple dimensions. The composition of the dynamic encoding vector includes contact friction coefficient, joint and connecting rod mass deviation, joint friction and back driving force, and unmodeled dynamic factors;

[0023] During the training phase, the input of LDAN is a series of state and action sequences sampled in the simulation environment, as well as known environment parameters;

[0024] In the LDAN inference phase, the input comes from real robot sensor data.

[0025] Furthermore, step 1 includes the input and action space of the behavior strategy network: the behavior strategy network is responsible for mapping from state observation to robot action output. Its input information format and action output space are designed as follows:

[0026] 1. State input format: The input of the policy network consists of a multimodal observation vector, including:

[0027] f) Robot state: current angle and angular velocity of each joint;

[0028] g) Body posture and velocity: Body posture, angular velocity, and linear acceleration obtained through the torso IMU;

[0029] h) Environmental contact information: The contact status between the robot foot and the ground, which is input into the strategy network by the ground contact sensor equipped on each leg or the support phase information inferred from the joint angle;

[0030] i) Dynamic latent encoding vector: The current environment dynamics code d output by LDAN is spliced ​​into the policy network

[0031] Network input enables the strategy to be aware of different environmental dynamic conditions and to adjust output actions in a targeted manner; j) Task instructions: high-level task commands are attached as input as needed.

[0032] Summarize the above information to form the input vector s of the policy network t , which is updated at each control moment;

[0033] Action space design: The policy network outputs the robot's next control action instructions. The action space of the actor network is defined as the target angle increment instructions for each major joint;

[0034] The behavioral policy network perceives comprehensive state inputs and outputs a series of desired joint angle adjustments as actions. Under deep reinforcement learning training, the network learns to map inputs to appropriate actions, thereby enabling the robot to achieve balanced walking and coordinated movements under various dynamic conditions.

[0035] Furthermore, the step 3 includes the following steps:

[0036] Design ideas for key reward items:

[0037] Forward Speed ​​Tracking Reward R v :The robot needs to walk at a predetermined speed, and the reward function encourages the actual speed to approach the expected speed v des , R v Take the negative absolute value of the difference between the actual speed and the target speed, so that the smaller the speed difference, the higher the reward. The reward reaches its maximum when the speed error is zero.

[0038] Posture Balance Reward R p :R p According to the tilt angle deviation of the robot's torso posture, Δφ and Δθ represent the deviation angles of pitch and roll from vertical, respectively. Penalizing their squares can provide smooth correction at small angles and give greater penalty strength at large angles. α and β are scale coefficients. The importance of pitch and roll is adjusted according to actual needs. The upper limb motion error penalty R arm : By using a pre-built upper limb reference motion and encouraging the robot arm joint angles to follow this reference during training, and are the actual angle and reference angle of the j-th upper limb joint of the robot, respectively. Penalizing their squared errors makes the strategy tend to imitate the reference action;

[0039] Gait cycle constraint reward R T :T actual The time difference between two consecutive landings of the same foot is measured, that is, the length of the robot's current gait cycle; T ref is the expected gait period, R T Penalize the square of the cycle deviation so that the robot tends to maintain a constant walking rhythm;

[0040] Energy consumption smoothing reward R e :R e Output torque τ for each joint j The square of is penalized;

[0041] Foot landing bonus R f : Through R f Measure the position difference between the actual landing point of the robot foot and the reference landing point, guide the strategy to learn the reasonable stride length and landing position, and refer to the landing point Given by the periodic gait generator; the fall penalty R fall : Once the robot's body is detected to have dropped suddenly to the ground and is judged to have fallen, a large negative reward is immediately given and the current training episode is terminated.

[0042] Furthermore, step 4 includes the following steps:

[0043] Periodic gait generator: First, determine the parameterized description of gait. Humanoid walking is a periodic motion that can be characterized by the gait period T and the time phase φ∈[0,2π) within the period. A periodic gait generator G(φ;η) is designed. It outputs the target reference posture or foot trajectory of each joint of the robot based on the phase φ, where η is the gait style parameter. G uses a simple combination of sinusoidal functions or stored key posture interpolation and uses the von Mises distribution to process the phase. The von Mises distribution is a probability distribution defined on a circle. Its shape is similar to the normal distribution but is applicable to angular variables. This embodiment uses the von Mises distribution to sample the gait phase to generate adjacent gait key points to ensure a smooth transition between 0 and 2π.

[0044] Human motion data retargeting: A general human motion retargeting method is used. First, a rich set of walking clips with varying speeds, strides, and styles are selected from a motion capture database. For each clip, the real person's skeletal joint angles or end-point trajectory are extracted at the keyframes. Then, based on the size and joint range of the GTX-III robot, inverse kinematics is used to map these key poses to robot-implemented joint angle combinations. During the IK solution, the goal is to maintain global pose similarity. An optimization algorithm is used to minimize the error caused by the proportional difference between the GTX-III and the real person. Unachievable poses are fine-tuned or discarded.

[0045] Gait data storage and classification: The robot joint trajectories obtained in the previous step are stored in the gait library and labeled and classified according to the gait attributes. For each category, its average gait cycle and stride parameters are calculated. Each gait is stored as multiple cycles of data and can be played back in a loop. The gait library data includes not only the changes in joint angles over time, but also some features calculated from these trajectories. These features will be used to calculate the reward function.

[0046] Methods of application in training: In reinforcement learning training, the reference data of the gait library can be used in many ways:

[0047] Reference reward calculation*: The reference gait corresponding to the current desired speed in the gait library is used to calculate R arm 、R T 、R fDuring training, the current simulated gait phase φ is inferred by accumulating time or steps. The reference posture of the corresponding phase is then retrieved from the gait generator or library, compared with the actual posture of the robot, and reward feedback is given. Initial strategy guidance: In the early stages of training, to accelerate convergence, a reference gait is used as the initial demonstration. Specifically, in the first batch of training episodes, the robot executes with a reference gait, thereby collecting some trajectory data close to the ideal gait for the reinforcement learning algorithm to learn.

[0048] Diversified gait training: The gait library contains multiple styles. During training, different reference styles are randomly extracted for different episodes, allowing the policy to see a wider range of walking styles.

[0049] Furthermore, the step 5 includes the following steps:

[0050] Simulation course training: Using the concept of course learning, the difficulty of the training environment is gradually increased. In the initial stage, the robot is trained to walk on a flat and hard surface. The environment parameters are set close to the ideal model. This stage enables the policy to master basic walking balance skills and imitate simple reference gaits. As training progresses, more challenging situations are gradually introduced, including:

[0051] e) Terrain complexity improvement: In subsequent simulation stages, different terrains are introduced, starting with small variations and then gradually increasing the unevenness and slope;

[0052] f) Dynamic randomization: Using domain randomization technology, a series of dynamic parameters are randomly perturbed in the simulation;

[0053] g) Gradual reward weighting: During course training, the emphasis of the reward function is adjusted at different stages;

[0054] h) Curriculum learning allows the strategy to converge in an easier environment by gradually increasing the challenge, and then uses the existing strategy as the initial

[0055] The strategy continues to train in a more difficult environment;

[0056] Dealing with and migrating virtual and real differences:

[0057] a) Offline Verification and Simulation Comparison: Before conducting on-device experiments, the policy is replicated and verified using a high-fidelity simulator. This includes testing policy behavior using a more sophisticated robot model. If anomalies are detected, training is rolled back or the network is adjusted until the high-fidelity simulation test passes.

[0058] b) Gradual acceleration / amplitude limiting: When first running the policy on a real robot, conservative scaling is applied to the actions output by the policy, and then these limits are gradually lifted to allow the policy to fully function.

[0059] c) LDAN online adjustment: Let the robot stand still or move in a small range, and continuously update the LDAN encoding output through sensor data until it stabilizes. That is, give LDAN an online "calibration" time to ensure that its output correctly reflects the real dynamics. During this period, the strategy mainly maintains balance and does not make large steps. Once

[0060] When the output converges, it means that LDAN has a good understanding of the real environment and can start walking normally.

[0061] Training and adjustment in a real-world environment: The GTX-III robot is allowed to walk several steps, and sensor data and policy outputs are recorded. This data is then fed into the training framework for a small amount of policy fine-tuning. After one or two rounds of fine-tuning, the final policy is determined for subsequent long-term operation.

[0062] The present invention also discloses a robot control system for the control method, wherein the control host is configured with a CPU or a GPU and can run a reinforcement learning inference model; the robot is equipped with a high reduction ratio servo joint and an IMU posture measurement module;

[0063] The control frequency is not less than 30Hz, and the actuator response time is less than 10ms; the system does not rely on force sensors when running.

[0064] Online dynamic adaptation can be achieved.

[0065] The robust walking control method and system for a high gear ratio humanoid robot based on potential dynamics adaptation of the present invention have the following advantages:

[0066] Strong robustness: By introducing potential dynamic coding, the method of the present invention can adapt to dynamic changes such as friction, mass, and center of gravity offset, thereby improving control stability and anti-interference ability.

[0067] Highly natural and expressive: Combining a periodic gait library with upper limb movement redirection, the guidance strategy generates full-body movements that conform to natural human movements, improving expressiveness and humanoid simulation.

[0068] Strong transferability: After training in simulation, this method can be directly deployed on a real robot without relying on additional dynamic identification or fine-tuning, reducing deployment costs.

[0069] Wide range of applicable platforms: This method can be applied to various types of humanoid robots that use high-gear ratio servo drive systems and is universally applicable in industrial, service and research platforms.

[0070] High system integrity: The present invention includes a complete description of both the software control algorithm and the hardware system deployment, and has the possibility of direct engineering implementation, meeting patent disclosure and reproduction requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] Figure 1 Figure 2: The overall framework of the control system described in the present invention. It shows the complete system components, including the perception input module, latent dynamics adaptive network (LDAN), behavior strategy network, reward function module, simulation training environment, and real robot platform.

[0072] Figure 2 Schematic diagram of the Latent Dynamics Adaptive Network (LDAN) architecture. This diagram shows how historical observations are encoded into latent vectors d and z via an encoder, then jointly decoded to predict the next observation, forming a VAE structure. This is then jointly optimized via KL divergence and reconstruction error.

[0073] Figure 3 Schematic diagram of the periodic gait generation process. This diagram shows how the periodic phase variable φ is generated from a von Mises probability distribution, combined with trajectory parameters to generate the foot and arm trajectories, and then the complete body motion is derived through inverse kinematics.

[0074] Figure 4 Schematic diagram of the LDAN module's response to dynamic parameter estimation under different gait and load change conditions;

[0075] Figure 5 This is a schematic diagram of the action sequence of realizing multiple gaits (such as standing stepping, forward movement, and rotation) and natural upper limb expression by the control strategy of the present invention during the actual deployment of a humanoid robot.

[0076] Figure 6 This is a schematic diagram of the action sequence implemented in different environments during the actual deployment of the humanoid robot using the control strategy of the present invention. DETAILED DESCRIPTION

[0077] In order to better understand the purpose, structure and function of the present invention, the following is a further detailed description of a robust walking control method and system for a high gear ratio humanoid robot based on potential dynamics adaptation in conjunction with the accompanying drawings.

[0078] The present invention provides a robust walking control method for a high gear ratio humanoid robot based on potential dynamics adaptation, comprising the following steps:

[0079] Step 1: Design a deep reinforcement learning training framework for the control strategy: Construct a control strategy based on deep reinforcement learning, adopt an actor-critic structure, and use the proximal policy optimization (PPO) algorithm for training optimization.

[0080] Deep reinforcement learning training framework for control policies:

[0081] The policy network (Actor) and value network (Critic) in the Actor-Critic structure are used to take on the functions of action decision-making and value evaluation respectively. During the training process, the Proximal Policy Optimization (PPO) algorithm is introduced to iteratively update the policy. The Actor network generates the next action instruction based on the robot state observation, and the Critic network estimates the value function of the current state to guide the policy update. The PPO algorithm constrains the change range of the new and old policies through the clipped policy gradient loss function, thereby stabilizing the training process while ensuring learning efficiency. The training process is as follows Figure 1 As shown: First, the state of the robot is initialized in the simulation environment, and the Actor network is based on the current observation s t Output action a t , the action acts on the simulation environment to obtain a new state and reward value r t ; At the same time, the Critic network is based on s t Calculate the state value V(s t ) is used to evaluate strategy performance. The PPO algorithm collects trajectory data from multiple parallel simulation environments, uses the advantage function estimated by the value network and the clipping strategy target to update the parameters of the Actor network, and synchronously updates the parameters of the Critic network. This training framework makes full use of parallel simulation sampling to improve data efficiency, and ensures the stability of strategy evaluation and update through the Actor-Critic method. In this embodiment, since the humanoid robot driven by a high reduction ratio has complex nonlinear behavior in joint dynamics, the PPO algorithm can better adapt to the continuous action space and handle the gradient variance during strategy update, thereby achieving robust gait learning.

[0082] Figure 1 The diagram shows the control strategy training framework of this embodiment. Figure 1 As can be seen in the figure, the training system includes a parallel simulation environment, an actor-critic neural network, and a policy update module. The actor (behavior policy network) takes the robot state and environmental observations as input and outputs action instructions; the critic (evaluation network) estimates the value of the state; together, they form the main body of reinforcement learning. By interacting with the environment to collect state-action-reward sequences, the parameters of the actor and critic networks are iteratively updated using PPO. This framework continuously optimizes the policy during training, enabling the robot to learn robust walking movements. In the early stages of training, the policy gradually accumulates experience through extensive trial and error simulations. As training progresses, the critic network's assessment of state value becomes increasingly accurate, guiding the actor network to output more optimal actions. Leveraging the actor-critic structure, training can efficiently converge to a robust control policy, laying the foundation for subsequent deployment across virtual and real environments.

[0083] Step 2: Latent Dynamics Adaptive Network (LDAN) Design: Build an encoder-decoder network with a variational autoencoder (VAE) structure. This network takes the robot's observation history as input and outputs a set of latent vectors representing the environment and the robot's dynamic parameters, including mass, friction coefficient, center of mass offset, and servo gains. This latent vector serves as an additional input to the control strategy, enabling implicit dynamic compensation.

[0084] Structure and Inference of Latent Dynamics Adaptive Network LDAN:

[0085] like Figure 2 As shown in the figure, LDAN adopts the structure of Variational Autoencoder (VAE), which consists of two parts: encoder network and decoder network. The encoder network takes the sensor information of the robot during operation as input, such as joint status, end effector force, IMU measurement, etc., to extract hidden features that characterize the dynamic characteristics of the environment. The encoder maps these features into a low-dimensional latent vector z, that is, the dynamic encoding vector. To ensure the continuity and universality of the distribution of z, LDAN introduces the Kullback-Leibler divergence (KL divergence) regularization term during training to bring the encoded posterior distribution q(z|x) closer to the prior distribution p(z) (generally the standard normal distribution is selected). The decoder network takes the latent vector z as input and attempts to reconstruct outputs related to the dynamics of the environment, such as the state change of the robot at the next moment. or expected dynamical parameters. The encoder-decoder is trained jointly so that the latent vector z fully captures information about the differences in environment dynamics. The training loss consists of a reconstruction error that measures the difference between the decoder output and the true observations, and a KL divergence term that constrains the distribution of z to avoid overfitting to a specific environment.

[0086] The LDAN's inference process is as follows: During policy execution, the latest sequence of robot state observations within a time window is fed into the LDAN's encoder every several control steps. The encoder outputs a latent vector z of the current environment's dynamics, which captures the deviation between the dynamics of the current environment and the standard model. This latent vector is then fed into the higher layers of the behavioral policy network (Actor), enabling the policy to make action decisions based on the currently estimated dynamics. For example, if the friction torque of the real robot's joints is greater than that of the simulated model, the LDAN reflects this in z, prompting the policy to output a greater driving force to overcome the friction. Furthermore, the LDAN can adaptively update with each new environmental change, such as a change in floor material or load. The encoder can extract the corresponding change and update z. Because the LDAN learns from a large number of simulated environments with different random dynamic parameters during training, its encoder has strong generalization capabilities. During real-world deployment, it can infer unmodeled dynamic factors based solely on sensor data, thereby adjusting the control policy in real time. In this way, the LDAN achieves online adaptation to the difference between virtual and real dynamics, narrowing the "reality gap" between the simulator and the real world, and ensuring the robustness of the policy on the real robot.

[0087] Composition of dynamic encoding vector and inference input and output:

[0088] The dynamic encoding vector d in this embodiment is the potential dynamic feature vector extracted by LDAN, which contains dynamic parameter information of multiple dimensions. The composition of the dynamic encoding vector includes but is not limited to the following aspects:

[0089] 1. Contact friction coefficient: reflects the difference in friction between the robot's sole and the ground. For example, different ground materials (carpet, smooth floor) or shoe sole materials will affect the friction coefficient. The relevant dimension in d will encode the ground friction

[0090] Mechanical properties: when the friction is low, prompts the strategy to prevent slipping; when the friction is high, prompts the strategy to avoid excessive force.

[0091] 2. Joint and Link Mass Deviation: This reflects the difference in mass or center of mass distribution of each robot link relative to the ideal model. Mass distribution errors in robot components with high-reduction ratio drive joints can affect kinematic inertia and force response. The corresponding dimensions in the dynamics encoding vector record information about overall mass increases or decreases or center of mass position shifts, enabling the strategy to output greater torque when the mass is heavier and adjust the posture to maintain balance when the center of mass shifts.

[0092] 3. Joint Friction and Back-Drive Force: High-ratio harmonic reducer joints have internal friction and non-back-drive characteristics. Actual joint damping, dead zone, and other characteristics may differ from the simulation model. The d-vector can include encoding for joint friction coefficients or viscous damping, allowing the strategy to perceive the actual joint flexibility. If the encoding indicates high joint friction, the strategy will increase joint drive force output or slow down the movement rhythm to ensure proper movement.

[0093] 4. Other unmodeled dynamic factors: These include motor output torque errors, control delays, and other factors. Idealization is often assumed in simulations, and these factors can introduce discrepancies in real systems. LDAN represents these factors through additional dimensions in the latent vector. For example, when the actual motor output is insufficient, the corresponding encoded dimension will deviate from zero, prompting the policy to compensate. When control lag exists, the encoded representation may allow the policy to preemptively adjust the gait cycle.

[0094] During the training phase, the input to LDAN is a series of state and action sequences sampled from the simulation environment, as well as known environmental parameters (as teacher signals). For example, for a robot's motion trajectory, the friction coefficient used, the connecting rod mass calibration value, etc. are known. During training, the encoder output d is linked to these known parameters. The specific training method includes: on the one hand, reconstructing the robot's next state through the decoder to allow d to capture the impact of dynamics on state evolution; on the other hand, when a supervised signal is available, allowing d to approximate the dynamic parameters set by the simulation (this can be achieved by adding a penalty for the difference between d and the true parameters in the loss function). After training, the encoder learns to map the sensor input to a latent vector that reflects the changes in physical parameters.

[0095] During the LDAN inference (i.e., deployment) phase, input primarily comes from real-world robot sensor data. A time window of sensory information is typically selected as encoder input to enhance robustness against noise and transient changes. These inputs can include joint angles and angular velocities over several steps, trunk linear acceleration and angular velocity measured by the IMU, and contact switch signals indicating foot contact. The encoder processes this time window of data and outputs a dynamic encoding d of the current environment. This d is fed into the behavioral policy network in real time as an additional observation. The behavioral policy network therefore makes decisions based not only on the current robot pose but also on the "context" information about the environmental dynamics provided by d. For example, when d indicates low ground friction, the policy might increase the foot thrust contact time during a stride to prevent slippage; when d indicates a heavier load, the policy might slightly reduce walking speed to maintain stability. The LDAN decoder can often be omitted during deployment (only the encoder is used for inference), or the encoder output can be fed into the decoder to monitor the deviation between prediction and reality and continuously adjust d. In summary, the dynamic encoding vector plays a bridging role in operation, condensing complex physical differences into low-dimensional vectors for strategy use, achieving efficient online adaptation.

[0096] Input and action space of behavioral policy network:

[0097] The behavior strategy network (i.e., Actor network) of this embodiment is responsible for mapping from state observation to robot action output. Its input information format and action output space are designed as follows:

[0098] 2. State input format: The behavior policy network needs to perceive the robot's state and environmental information in order to determine the appropriate action instructions. In this embodiment, the input of the policy network consists of a multimodal observation vector, including:

[0099] a) Robot state: The current angle and angular velocity of each joint. Because the GTX-III humanoid robot has multiple degrees of freedom (for example, six joints per leg, for a total of approximately 21 degrees of freedom, encompassing the legs, arms, torso, and head), the input vector contains the angles and angular velocities of all active joints. These states describe the robot's current posture and motion.

[0100] b) Body posture and velocity: The torso IMU obtains the body posture (such as pitch and roll angles), angular velocity, and linear acceleration. This information provides feedback on the robot's overall balance and movement speed. For example, the IMU's pitch angle can reflect the robot's forward or backward tilt, while the angular velocity can indicate whether the robot is swaying and require corresponding gait adjustments.

[0101] c) Environmental contact information: The contact status between the robot's foot and the ground. Ground contact sensors installed on each leg, or support phase information inferred from joint angles, are input into the policy network. For example, binary information indicates whether the left or right foot is currently in contact with the ground, as well as the duration of the previous contact phase. This helps the policy determine the gait phase and support status, thereby coordinating the movements of both legs.

[0102] d) Dynamic Latent Encoding Vector: The dynamic encoding d of the current environment, output by the LDAN. This vector contains implicit information such as friction and mass deviation, as described in the previous section. Adding d to the policy network input enables the policy to be aware of different environmental dynamic conditions and adjust its output actions accordingly.

[0103] e) Task Instructions: As needed, high-level task commands can be added as input, such as the desired walking speed, direction, or gait style parameters. In the basic walking task of this embodiment, it is assumed that the robot needs to walk in a straight line at a predetermined speed, so the input may include the desired forward speed v des For reference.

[0104] Summarize the above information to form the input vector s of the policy network t , updated at every control moment. Since the control of high-reduction-ratio humanoid robots typically requires high frequency (see below for the control frequency of the GTX-III), the input observations change smoothly between consecutive time steps. The policy network can also include recurrent structures (such as LSTM units) to utilize historical information. However, in this embodiment, it mainly relies on windowed observations and latent codes to capture dynamic features.

[0105] Action Space Design: The policy network outputs the robot's next control action command. For humanoid robots with high reduction ratios and no force sensing, a robust approach is to have the policy output the desired joint target position (angle) or target velocity, which is then tracked by the underlying joint controller. This is because directly outputting force / torque is difficult to achieve accurately without force sensing, and high reduction ratio joints are insensitive to force control but respond well to position control. Therefore, in this embodiment, the actor network's action space is defined as the target angle increment command for each major joint. For example, each joint corresponds to an action output, indicating the angle (radians) to increase or decrease from the current joint angle. The robot's underlying control uses this target angle as input and moves the joint to the target position via a PD controller or servo drive. For the GTX-III platform with a position control interface, this output method balances control accuracy and ease of implementation. The dimensionality of the action space is consistent with the robot's degrees of freedom (e.g., a total of 21 dimensions, each corresponding to the target angle adjustment value for a joint).

[0106] In some implementations, it is also possible to choose to let the strategy output the target angular velocity of the joint or directly output the PWM duty cycle of the control motor. However, given the characteristics of the high reduction ratio joint system, the target angle method is easier to achieve a stable humanoid gait. When outputting the angle, the range of motion and safety limits of each joint must also be considered. The policy network will automatically avoid outputting actions that exceed the physical limits after training. In addition, in order to achieve the periodicity of the walking gait, the output of the policy network in this embodiment is usually continuous and updated in each control cycle (for example, once every 0.03 seconds) to make the robot joint move smoothly towards the target angle. Through appropriate PD gain configuration, the actual action can be guaranteed to be close to the intention of the policy output. In short, the behavioral policy network outputs a series of desired joint angle adjustments as actions by sensing comprehensive state inputs (including its own posture, environmental contact, dynamic encoding, etc.). The network learns to map the input to the appropriate action under deep reinforcement learning training, thereby achieving balanced walking and coordinated movements of the robot under various dynamic conditions.

[0107] Step 3: Reward function design: Construct a multi-dimensional reward function, including: target speed tracking, posture maintenance, cycle consistency, foot landing accuracy, upper limb expression error, energy consumption and movement smoothness, etc., to ensure that the strategy has good expressiveness while being stable.

[0108] In reinforcement learning training, the design of the reward function (reward function) directly influences the behavior pattern to which the strategy converges. This example carefully designs a reward function with multiple sub-items to encourage the robot to achieve robust and expressive walking. The rewards and design principles for each sub-item are shown in Table 1.

[0109] Table 1 lists the reward function sub-items used for walking strategy training in this embodiment, their meanings, and weight parameters.

[0110]

[0111] In the above table, each reward sub-item is weighted and constitutes the total instantaneous reward R = ∑ i w i R i (Except for the fall penalty, in which case the simulation is terminated immediately and a large one-time penalty is imposed.) The following describes the design philosophy of the key reward items:

[0112] 1. Forward speed tracking reward R v :The robot needs to walk at a predetermined speed. The reward function encourages the actual speed to approach the expected speed v des , which can prevent the strategy from stagnating in pursuit of stability, and also avoid the imbalance caused by running too fast. vThe negative absolute value of the difference between the actual speed and the target speed is taken, so that the smaller the speed difference, the higher the reward. The reward is maximized when the speed error is zero. Appropriate weights ensure that speed tracking is one of the main optimization goals.

[0113] 2. Posture balance reward R p : In order to ensure the robot's upper body is upright and balanced, R p Constructed based on the tilt angle deviation of the robot's torso posture. Δφ and Δθ represent the deviations of pitch and roll from vertical, respectively. Penalizing their squares provides smooth corrections at small angles and imposes a stronger penalty at large tilt angles. α and β are scaling factors that adjust the importance of pitch and roll according to actual needs. This reward encourages the policy to control the robot's torso in a near-vertical position, thereby reducing the risk of falls.

[0114] 3. Upper limb movement error penalty R arm : To achieve "expressive" walking, we expect the robot's upper limbs to swing according to the natural gait of humans. By using pre-built upper limb reference motion (such as the arm swing trajectory when humans walk),

[0115] The robot arm joint angles are encouraged to follow this reference during training. and where _{\theta} ...

[0116] 4. Gait cycle constraint reward R T :Human walking cadence is usually stable and rhythmic. To improve the naturalness and consistency of robot gait, we introduce constraints on the gait cycle. actual It can be measured based on the time difference between two consecutive landings of the same foot, that is, the length of the robot's current gait cycle; T ref is the expected gait period (derived from the gait library or human gait statistics, such as 0.8 seconds / step). T Penalizes the square of the cycle deviation, so that the robot tends to maintain a constant walking rhythm. Adding this can prevent the strategy from drastically changing the step length or frequency between steps, thus producing a symmetrical and smooth gait. In actual implementation, if the strategy needs to adapt to different speeds, T can be dynamically adjusted. ref With v des In this embodiment, T ref The human cadence at the corresponding speed is taken from the gait library. The appropriate coefficient ensures that the cyclic stability accounts for a certain proportion of the policy objectives.

[0117] 5. Energy Consumption Smoothing Reward R e : To prevent the robot joints from moving too violently, resulting in high energy consumption or uneven movement, R e Output torque τ for each joint j The square of the weight is penalized (in the absence of sensing, a proxy value for the motor control input can be used). This acts as L2 norm regularization, encouraging the policy to achieve actions with small and smooth control signals. This reward helps extend motor life, reduce battery consumption, and make the robot's movements smoother and less jerky. The weight is typically kept small as a secondary optimization objective to prevent the policy from sacrificing too much other performance for energy conservation.

[0118] 6. Foot landing point bonus R f :Combined with the gait library reference, we hope that the robot will step in the right position each step. For example, in the human gait reference, the next step should be placed at a suitable distance in front of the body's center of gravity to maintain balance. f Measuring the difference between the actual landing point of the robot foot and the reference landing point (e.g. using the square of the Euclidean distance) can guide the strategy to learn a reasonable stride length and landing position. It can be given by a periodic gait generator (see the gait library section below), which takes into account factors such as the current gait phase and center of gravity position. Penalizing deviations allows the robot to gradually learn to follow the reference plan and improve the stability and regularity of the gait. The weight can be set according to the importance. If the main focus is on avoiding falls and speed, the weight can be moderate or slightly low, but in order to achieve more coordinated upper limb movement, the weight can be set to the upper limb movement.

[0119] For humanoid movements, this item cannot be too low.

[0120] 7. Fall Penalty R fall :Any falling behavior is strictly avoided in this embodiment. Set a rule for this: once the

[0121] If the robot's body suddenly drops to the ground (or its posture deviates significantly from its equilibrium angle), it is detected as falling. A large negative reward (such as -100 or -500) is immediately applied, and the current training episode is terminated. This measure acts as a hard constraint, ensuring that the policy quickly recognizes the significant cost of falling during learning, thereby favoring exploration of safe action spaces. Furthermore, there is no mechanism for continuous negative rewards during policy deployment, but this penalty during training is sufficient to ensure that the final policy prioritizes avoiding falls.

[0122] The above sub-rewards are weighted and summed to form the final reward function R total , guiding reinforcement learning optimization. By reasonably adjusting each w iWeights can be used to strike a balance between the stability of the robot's walking and the naturalness of its movements. For example, by assigning a high w1 and appropriate w2, w fall , ensuring the robot first learns to walk stably and maintain balance. Subsequently, w3 and w4 are appropriately increased, prompting the strategy to incorporate expressive factors such as arm swing and gait rhythm while maintaining stability. Experiments have shown that this reward function design can effectively train a gait control strategy that balances robustness and naturalness.

[0123] Step 4: Construction of a periodic gait library: Use the von Mises distribution to construct a periodic guidance signal, combine it with motion capture data to generate natural upper limb movements, use the inverse kinematics solver to synchronously generate foot trajectories, and finally generate a full-body gait as a reference to guide policy learning.

[0124] Method for constructing gait library

[0125] To provide a reference for ideal gait trajectories and enhance the expressiveness of the adjustment strategy, this embodiment constructs a humanoid gait library. The gait library contains a variety of periodic walking motion data, which can be obtained from human motion capture data or generated by algorithms. This data is used to provide reference trajectories during training (for reward calculation or initialization of the strategy) and can be used to guide the strategy to achieve gaits of different styles or speeds. The construction of the gait library includes the following steps:

[0126] 1. Periodic gait generator: First, determine the parameterized description method of the gait. Humanoid walking is a periodic motion, which can be characterized by the gait period T and the time phase φ∈[0,2π) within the period. In this embodiment, a periodic gait generator G(φ;η) is designed, which outputs the target reference posture or foot trajectory of each joint of the robot based on the phase φ. Where η is a gait style parameter, such as walking speed, stride size, etc. G can use a simple combination of sinusoidal functions, or store key posture interpolation. In order to ensure the continuity and periodicity of the generated gait at different phases, we use the von Mises distribution to process the phase. The von Mises distribution is a probability distribution defined on a circle. Its shape is similar to the normal distribution but is applicable to angular variables. This embodiment uses the sampling of the von Mises distribution on the gait phase to generate adjacent gait key points to ensure a smooth transition at 0 and 2π (i.e., phase cycle). Specifically, several key phase points φ can be selected first. k , specify the reference robot pose at these points (which can come from human data or be manually set). Then smooth the von Mises distribution around these phases so that the reference trajectory throughout the cycle is continuous and close to the natural human motion form.

[0127] 2. Human motion data redirection: Many reference trajectories in the gait library can be directly derived from motion capture data of real people walking. Mapping human motion data to the GTX-III robot requires solving the problems of scale and joint configuration differences. This embodiment adopts a general human motion redirection method: first, a rich walking clip is selected from the motion capture database. These clips contain different speeds, strides and styles (such as large or small arm swing amplitude, marching, trotting, etc.). For each clip, the real person's skeletal joint angles or end (hand and foot) trajectories are extracted in the key frame. Then, based on the size and joint range of the GTX-III robot, the inverse kinematics (IK) method is applied to map these key postures into joint angle combinations that can be achieved by the robot. When solving IK, the goal is to maintain global posture similarity, for example: make the robot's foot end position match the real person's foot end position scaling value as much as possible, align the torso orientation, and make the position angle of the upper limb end relative to the torso as close as possible. Since there are proportional differences between GTX-III and real people in terms of leg length, arm length, etc., the errors caused by these differences are minimized through optimization algorithms. For unrealizable postures (outside the range of motion of the robot joints), fine-tuning or discarding is performed. After this redirection process, each segment of real-life gait data is converted into a joint trajectory suitable for the robot.

[0128] 3. Gait data storage and classification: The robot joint trajectory obtained in the previous step is stored in the gait library. And the gait is labeled and classified according to the attributes of the gait, such as: classification by walking speed (slow walking, medium speed, fast walking), classification by style (normal walking, striding, etc.). For each category, calculate its average gait cycle, stride and other parameters. In our implementation, the gait library covers natural walking at different speeds from 0.2 meters to 0.5 meters per second, as well as variant gaits with different swing amplitudes of both hands. Each gait is stored in multiple cycles of data and can be played in a loop. The data in the gait library includes not only the changes in joint angles over time, but also some features calculated from these trajectories, such as the phase relationship between the upper limb swing and the lower limb step, the projection trajectory of the center of gravity over time, etc. These features will be used in the reward function (such as the aforementioned R arm and R T ,R f ) calculation.

[0129] 4. Application in training: In reinforcement learning training, the reference data of the gait library can be used in a variety of ways:

[0130] 1. Reference reward calculation*: As mentioned above, the reference gait corresponding to the current desired speed in the gait library is used to calculate R arm 、R T 、R fDuring training, the current simulated gait phase φ is inferred by accumulating time or steps. A reference pose corresponding to this phase is then retrieved from a gait generator or library. This pose is compared with the robot's actual pose and reward feedback is provided. This allows the policy to mimic key features of the reference gait when driven by rewards.

[0131] 2. Initial strategy guidance: In the early stages of training, in order to accelerate convergence, a reference gait can be used as an initial demonstration. Specifically, in the first batch of training episodes, the robot is allowed to execute with a reference gait (with a small random perturbation added), thereby collecting some trajectory data close to the ideal gait for the reinforcement learning algorithm to learn. This is similar to the hot start of imitation learning. In this embodiment, the humanoid reference gait guidance can be used in the first 10,000 simulation steps, and then the strategy can be gradually allowed to learn autonomously, thereby achieving a balance between stability and exploration.

[0132] 3. Diverse gait training: The gait library contains a variety of styles. During training, different reference styles can be randomly sampled for different episodes, allowing the policy to experience a wider range of walking styles. This improves the policy's adaptability to various gaits and prevents overfitting to a single sample, resulting in a more human-like style in real-world deployment. The von Mises distribution plays a key role in this. By sampling gait phase and style, the reference signal is randomized while retaining periodic characteristics, thereby improving the policy's robustness to gait variations.

[0133] By constructing and utilizing the above-mentioned gait library, the robot of this embodiment not only learns to walk, but also learns to walk in a natural human way. Figure 3 The effect of the gait library is shown as follows: Figure 3 As shown, the robot's upper limb swing is coordinated with its lower limb stride, mirroring the anti-synchronous pattern of the left and right hands and feet during human walking. The robot's gait cycle is essentially constant, and even under slight perturbations, it can quickly return to the reference rhythm. This is all thanks to the prior reference provided by the gait library, enabling reinforcement learning to optimize not only "steady walking" but also "human-like walking."

[0134] Step 5: Course-based simulation training: Gradually introduce terrain perturbations and dynamic parameter changes in a simulation platform (such as Isaac Gym) to improve the policy generalization ability; after training, the policy and LDAN module are directly deployed on the real robot system.

[0135] In this embodiment, a phased training strategy is adopted to ensure that the control strategy can be smoothly transferred from simulation to the real robot, including course training in the simulation environment and adaptation process in the real environment.

[0136] 2. Simulation Curriculum Training: Considering that direct training in a complex environment may lead to convergence difficulties, we adopt the concept of curriculum learning to gradually increase the difficulty of the training environment. In the initial stage, the robot is trained to walk on a flat, hard surface, with environmental parameters set close to the ideal model (such as standard friction coefficient and no external disturbance). This stage focuses on enabling the policy to master basic walking balance skills and imitate simple reference gaits. As training progresses, more challenging situations are gradually introduced, including:

[0137] a) Increasing Terrain Complexity: In subsequent simulation stages, introduce varying terrain, such as gently undulating roads, small obstacles (e.g., height fields formed by randomly spaced blocks), and inclined ramps. Initially, the variations are small, then the degree of unevenness and slopes are gradually increased. This teaches the system strategies for dealing with various ground conditions and improves its adaptability to unknown, real-world terrain.

[0138] b) Dynamic Randomization: Using domain randomization, a series of dynamic parameters are randomly perturbed during simulation. This includes random variations in the ground friction coefficient within a certain range, random variations in the mass of each robot link by 5% to 10% above or below the nominal value, random application of response delays to joint motors, and even external thrust perturbations (simulating sudden impacts). These random variations cause the policy network and LDAN to experience various "reality gaps" during training, thereby learning to maintain stability under uncertain parameters. Importantly, the model uncertainty of high-reduction ratio joints is reflected through random joint friction and efficiency, and the policy and LDAN learn robust responses through a large number of samples.

[0139] c) Gradual reward weight change: During the course training, the emphasis of the reward function is adjusted at different times. For example, first use stability reward (speed tracking R v , Balance R p , fall penalties, etc.), and gradually increase the performance-related rewards (upper limb R arm , gait cycle R T This ensures that the strategy first “knows how to move” and then “moves well”.

[0140] Curriculum learning gradually increases the challenge, allowing the policy to converge in easier environments. This approach then uses the existing policy as the initial policy for further training in more challenging environments. This approach avoids non-convergence caused by initial disruptions caused by difficult scenarios and is equivalent to progressive domain randomization of the policy across multiple environments. Figure 4This diagram illustrates the simulation training process, with the horizontal axis representing training time and the vertical axis representing environmental difficulty, such as terrain height or disturbance force. As can be seen, the environmental difficulty increases in steps with each training iteration, and the strategy's performance undergoes repeated optimization at each new stage, ultimately maintaining good performance even in the most challenging environments.

[0141] 3. Addressing and Migrating Virtual and Real Differences: After the above-mentioned extensive simulation training, the resulting strategy, along with the LDAN, has proven robust under various random conditions. To further ensure safety and performance when deploying it on a real GTX-III robot, this example employs the following measures:

[0142] a) Offline Verification and Simulation Comparison: Before conducting real-world experiments, the policy is replicated and verified using a high-fidelity simulator. This includes testing the policy's behavior using a more refined robot model (accounting for joint flexibility, actuator hysteresis, and other factors). If anomalies are detected, such as when certain actions output by the policy cause robot instability in simulation, training is rolled back or the network is adjusted until the high-fidelity simulation test passes. This step acts like a dry run before a real-world experiment, reducing the risk of direct real-world debugging.

[0143] b) Gradual Acceleration / Amplitude Limiting: When first running the policy on a real robot, apply conservative scaling to the policy's output. For example, constrain joint target angles to a narrow range and start with a low walking speed. Then, gradually remove these constraints to allow the policy to fully function. This prevents the policy from making overly aggressive movements early on due to factors not considered in the real world.

[0144] c) Online adjustment of LDAN: Since LDAN has learned dynamic adaptation in simulation, there may still be uncovered factors in the real environment, such as changes in motor performance under the influence of temperature. Therefore, in the initial stage of actual machine operation, the robot can be allowed to stand in place or move in a small range, and the encoding output of LDAN can be continuously updated through sensor data until it stabilizes. In other words, give LDAN an online "calibration" time to ensure that its output correctly reflects the real dynamics. During this period, the strategy mainly maintains balance and does not make large steps. Once the output converges, it means that LDAN has a full understanding of the real environment and can start walking normally. Experiments have shown that even if the real environment is different from that during training (such as slightly lower friction), LDAN can adjust the output within seconds to make the strategy compensate for this.

[0145] These differences can be eliminated to smoothly achieve strategy migration from simulation to reality.

[0146] 4. Training and adjustment in actual environment: Most of the strategy optimization work in this embodiment is completed in simulation, and does not require a large amount of real training data. However, in order to ensure the completeness of the patent embodiment, fine-tuning schemes in real environments are also considered. If conditions permit, the GTX-III robot can be allowed to actually walk a few steps under safe traction protection, record sensor data and strategy outputs, and input these data into the training framework for a small amount of strategy fine-tuning (using real data to fine-tune the strategy or continue training LDAN). Due to the use of a model-free RL method, it is not advisable to adjust parameters drastically when real data is limited to prevent overfitting; but some small corrections, such as slightly increasing / decreasing certain reward weights or network output biases, can be made through field test observations. After one or two rounds of fine-tuning, the final strategy is determined for subsequent long-term operation.

[0147] Through the above strategies, this embodiment successfully achieved the strategy migration from simulation to real high-reduction ratio humanoid robot. Actual tests show that the robot can still walk stably and resist external interference (such as sudden push) in the absence of force sensors, and the upper limbs maintain regular swinging during walking, and the gait rhythm is very close to the design reference. In particular, it shows good adaptability on different floors (wooden floors, carpets), verifying the online dynamic compensation provided by LDAN. The robot can also pass smoothly on non-flat floors (such as 5cm high undulating roads) that have been trained in the course. These results confirm the effectiveness of the method of this embodiment in improving the robustness and expressiveness of humanoid robot walking.

[0148] Step 6: Hardware deployment: This method is applicable to a humanoid robot with a high gear ratio drive and a control frequency of 30 Hz. The latent vector of the LDAN module is inferred in real time to dynamically adjust the control strategy.

[0149] GTX-III hardware platform configuration and operating frequency

[0150] The method of this embodiment is implemented on the GTX-III humanoid robot platform. The GTX-III is a small humanoid robot with a high reduction ratio drive. Its hardware configuration and control system features are as follows:

[0151] Robot Structure: The GTX-III robot is approximately 50 cm tall (with a trunk and limbs similar in proportion to an adult's upper body) and weighs approximately 5 kg. It has a total of 21 degrees of freedom: 6 degrees of freedom in each leg (3 at the hip, 1 at the knee, and 2 at the ankle), 3 degrees of freedom in each arm (2 at the shoulder and 1 at the elbow), 1 degree of freedom at the waist (waist rotation / sideways swing), and 1 to 2 degrees of freedom in the head. All major joints utilize high-reduction-ratio rotary actuators with an integrated harmonic reducer, achieving a reduction ratio of approximately 100:1 or greater. This high reduction ratio results in high torque output, enabling the robot to support its own weight while walking, but it also results in small joint return clearance and poor back drive (i.e., joints that are not easily driven by external forces).

[0152] Sensing and Actuation: The GTX-III is equipped with a high-resolution encoder at each joint for position feedback, but does not have a dedicated joint force / torque sensor (this is a force-free sensing design). Simple touch pads or pressure sensors are installed on the soles of the feet to detect whether the soles of the feet are touching the ground. An inertial measurement unit (IMU) is built into the body, providing three-axis acceleration and three-axis angular velocity for posture estimation. The robot is driven by multiple motor drive modules, each of which performs low-level servo control (typically PID or control with model compensation) based on the target position / velocity given by the upper controller. The controller can read information such as joint angles and motor currents in real time. The brain of the GTX-III is an embedded computer equipped with a multi-core CPU and the necessary communication interfaces, communicating with each joint drive module via Ethernet or CAN bus. Because the reinforcement learning inference network needs to be run, the embedded PC can optionally be equipped with a small GPU for acceleration. However, in this embodiment, the CPU can meet the real-time computing requirements by optimizing the network structure.

[0153] Control Frequency: The GTX-III control system supports a high closed-loop refresh rate. In this embodiment, the low-level joint control loop (within the motor drive) runs at approximately 1kHz for motor current control and position maintenance. The high-level policy control frequency, i.e., the update frequency of the reinforcement learning policy network (Actor network) and LDAN inference, is selected to be 50Hz. In other words, every approximately 20 milliseconds, sensor data is processed and fed into the Actor and LDAN encoders, which calculate new joint target angle commands and then issue them to each joint for execution. Between high-level commands, the low-level joint control maintains servo control according to the last received target position. A 50Hz decision frequency meets the real-time requirements of humanoid walking without overloading computing resources. Testing has shown that a single forward inference of the LDAN encoder and Actor network takes less than 5 milliseconds on the GTX-III CPU, which is well within the 20 millisecond budget.

[0154] Communication and Latency: The GTX-III's joint drivers are connected via a high-speed bus, resulting in negligible communication latency (less than 1ms) at a 50Hz command frequency. Data from the IMU and plantar sensors is acquired via interfaces such as the serial port and CAN, with the attitude solution updated once per cycle for the strategy to use. The entire sensing-decision-execution loop has a latency of approximately 10ms, which is considered near real-time for control. High-reduction-ratio joints have inherent delays due to their mechanical characteristics (phase lag caused by elastic deformation of the gear transmission), but the strategy partially learns to compensate for these effects through LDAN during training, and no significant lag accumulation occurs during actual movement.

[0155] Computing Resources: An embedded computer running Ubuntu and real-time processes was used to execute neural network inference and high-level control logic. PPO training was performed on an offline workstation, and only the trained neural network model parameters (file size approximately several megabytes) were loaded during deployment. Due to its limited size, the GTX-III lacked a high-capacity battery. While the system ran continuously for approximately 30 minutes, it was sufficient for multiple rounds of experimental verification. The control software architecture was implemented in a hybrid C++ / Python framework, with C++ responsible for real-time loops and low-level communication. The Python module invoked the trained policy model through bindings, facilitating the use of readily available deep learning inference libraries.

[0156] In summary, the GTX-III hardware platform, with its small mechanical structure with high torque output, verifies the effectiveness of the method described in this embodiment. Although its joints have no torque feedback and there are dynamic errors caused by high reduction ratios, the robot still achieves stable walking and adaptive control through deep reinforcement learning strategies combined with compensation of the potential dynamic adaptive network. The detailed description of the hardware configuration and frequency setting ensures that the content of the implementation method of this patent is sufficient and feasible, and provides a reference for those skilled in the art to implement similar functions.

[0157] Figure 4 Schematic diagram of the LDAN module's response to dynamic parameter estimation under different gait and load change conditions;

[0158] Figure 5 ,6 is a schematic diagram of the action sequence of realizing multiple gaits (such as standing stepping, forward movement, and rotation) and natural upper limb expression during the actual deployment of the humanoid robot using the control strategy of the present invention.

[0159] Through experimental evaluation, it can be determined that the "potential dynamic adaptive robust and expressive walking control method for a high-reduction-ratio humanoid robot in a virtual-real environment" described in this embodiment has achieved the intended purpose. Without relying on joint force sensing, zero-sample walking directly transferred from simulation to real humanoid robots is achieved, and it remains stable under complex terrain and interference. At the same time, the robot's gait is rich in humanoid features, such as coordinated arm swings and a stable walking rhythm, which enhances the naturalness and practicality of the movements. Various technical details and experimental results have been fully explained in the above specific implementation methods, which are sufficient to support the realization of the claims of this patent.

[0160] It will be understood that the present invention is described by way of some embodiments, and it will be appreciated by those skilled in the art that various changes or equivalent substitutions may be made to these features and embodiments without departing from the spirit and scope of the present invention. In addition, under the teachings of the present invention, these features and embodiments may be modified to adapt to specific circumstances and materials without departing from the spirit and scope of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are intended to be protected by the present invention.

Claims

1. A robust walking control method for a high gear ratio humanoid robot based on latent dynamics adaptation, characterized by: The steps include: Step 1: Design a deep reinforcement learning training framework for the control strategy: Construct a control strategy based on deep reinforcement learning, adopt an actor-critic structure, and use the proximal policy optimization (PPO) algorithm for training optimization; Step 2: Latent Dynamics Adaptive Network (LDAN) Design: Build an encoder-decoder network with a variational autoencoder (VAE) structure. This network takes in historical observation data from the robot and outputs a set of latent vectors representing the environment and the robot's dynamic parameters. This latent vector serves as an additional input to the control strategy to achieve implicit dynamic compensation. Step 3: Reward function design: Construct a multi-dimensional reward function to ensure that the strategy is stable and has good expressiveness; Step 4: Periodic gait library construction: Use the von Mises distribution to construct periodic guidance signals, combine them with motion capture data to generate natural upper limb movements, use an inverse kinematics solver to synchronously generate foot trajectories, and ultimately generate a full-body gait as a reference to guide policy learning; Step 5: Course-based simulation training: Gradually introduce terrain disturbances and dynamic parameter changes into the simulation platform to improve strategy generalization capabilities; After training, the strategy and LDAN module are directly deployed on the real robot system; Step 6: Hardware deployment: Applicable to a humanoid robot with a high gear ratio drive, with a control frequency of 30 Hz, and real-time inference of the latent vector of the LDAN module for dynamic adjustment of the control strategy.

2. The robust walking control method for a high gear ratio humanoid robot based on potential dynamics adaptation according to claim 1 is characterized in that: The step 1 comprises the following steps: Deep reinforcement learning training framework for control policies: The policy network Actor and the value network Critic in the Actor-Critic structure respectively undertake the functions of action decision-making and value evaluation. During the training process, the proximal policy optimization algorithm PPO is introduced to iteratively update the strategy. The Actor network generates the next action instruction based on the robot state observation, and the Critic network estimates the value function of the current state to guide the strategy update. The PPO algorithm constrains the change range of the new and old strategies through the clipped policy gradient loss function, thereby stabilizing the training process while ensuring learning efficiency. First, the robot's state is initialized in the simulation environment, and the Actor network is based on the current observation s t Output action a t , the action acts on the simulation environment to obtain a new state and reward value r t ; At the same time, the Critic network is based on s t Calculate the state value V(s t ) is used to evaluate strategy performance; LDAN adopts the structure of a variational autoencoder (VAE), consisting of an encoder network and a decoder network. The encoder network takes the sensor information of the robot during operation as input and extracts hidden features that characterize the dynamic characteristics of the environment. The encoder maps these features into a low-dimensional latent vector z, namely the dynamic encoding vector. During training, LDAN introduces the Kullback-Leibler divergence regularization term to bring the encoded posterior distribution q(z|x) closer to the prior distribution p(z). The decoder network takes the latent vector z as input and attempts to reconstruct the output related to the environmental dynamics. The encoder and decoder are trained together so that the latent vector z fully contains information about the differences in environmental dynamics. The training loss consists of the reconstruction error and the KL divergence term. The reconstruction error measures the gap between the decoder output and the actual observation, and the KL divergence term restricts the distribution of z to avoid overfitting to a specific environment. The reasoning process of LDAN is as follows: during policy execution, every several control steps, the robot state observation sequence within the latest time window is input into the LDAN encoder. The encoder outputs the dynamic latent vector z of the current environment, which characterizes the dynamic deviation between the current environment and the standard model. This latent vector is then input into the high-level layer of the behavior policy network Actor, enabling the policy to make action decisions based on the current estimated dynamic characteristics. In addition, LDAN is adaptively updated every time new environmental conditions change.

3. The robust walking control method for a high gear ratio humanoid robot based on potential dynamics adaptation according to claim 1 is characterized in that: Step 1 includes the composition of the dynamic encoding vector and the inference input and output: the dynamic encoding vector d is the potential dynamic feature vector extracted by LDAN, which contains dynamic parameter information of multiple dimensions. The composition of the dynamic encoding vector includes contact friction coefficient, joint and connecting rod mass deviation, joint friction and back driving force, and unmodeled dynamic factors; During the training phase, the input of LDAN is a series of state and action sequences sampled in the simulation environment, as well as known environment parameters; In the LDAN inference phase, the input comes from real robot sensor data.

4. The robust walking control method for a high gear ratio humanoid robot based on potential dynamics adaptation according to claim 1 is characterized in that: Step 1 includes the input and action space of the behavior strategy network: the behavior strategy network is responsible for mapping from state observation to robot action output. Its input information format and action output space are designed as follows:

1. State input format: The input of the policy network consists of a multimodal observation vector, including: a) Robot state: current angle and angular velocity of each joint; b) Body posture and velocity: Body posture, angular velocity, and linear acceleration obtained through the torso IMU; c) Environmental contact information: The contact status between the robot's foot and the ground, which is input into the strategy network using the ground contact sensors on each leg or the support phase information inferred from the joint angles; d) Dynamic latent encoding vector: The current environment dynamics code d output by the LDAN is spliced ​​into the policy network input, making the policy aware of different environmental dynamic conditions and able to adjust the output action in a targeted manner; e) Task instructions: Attach high-level task commands as input as needed. Summarize the above information to form the input vector s of the policy network t , which is updated at each control moment; Action space design: The policy network outputs the robot's next control action instructions. The action space of the actor network is defined as the target angle increment instructions for each major joint; The behavioral policy network perceives comprehensive state inputs and outputs a series of desired joint angle adjustments as actions. Under deep reinforcement learning training, the network learns to map inputs to appropriate actions, thereby enabling the robot to achieve balanced walking and coordinated movements under various dynamic conditions.

5. The robust walking control method for a high gear ratio humanoid robot based on potential dynamics adaptation according to claim 1 is characterized in that: The step 3 comprises the following steps: Design ideas for key reward items: Forward Speed ​​Tracking Reward R v :The robot needs to walk at a predetermined speed, and the reward function encourages the actual speed to approach the expected speed v des , R v Take the negative absolute value of the difference between the actual speed and the target speed, so that the smaller the speed difference, the higher the reward. The reward reaches its maximum when the speed error is zero. Posture Balance Reward R p :R p According to the tilt angle deviation of the robot's torso posture, Δφ and Δθ represent the deviation angles of pitch and roll from vertical, respectively. Penalizing their squares can provide smooth correction at small angles and give greater penalty strength at large angles. α and β are scale coefficients. The importance of pitch and roll is adjusted according to actual needs. The upper limb motion error penalty R arm : By using a pre-built upper limb reference motion and encouraging the robot arm joint angles to follow this reference during training, and are the actual angle and reference angle of the j-th upper limb joint of the robot, respectively. Penalizing their squared errors makes the strategy tend to imitate the reference action; Gait cycle constraint reward R T :T actual The time difference between two consecutive landings of the same foot is measured, that is, the length of the robot's current gait cycle; T ref is the expected gait period, R T Penalize the square of the cycle deviation so that the robot tends to maintain a constant walking rhythm; Energy consumption smoothing reward R e :R e Output torque τ for each joint j The square of is penalized; Foot landing bonus R f : Through R f Measure the position difference between the actual landing point of the robot foot and the reference landing point, guide the strategy to learn the reasonable stride length and landing position, and refer to the landing point Given by the periodic gait generator; the fall penalty R fall : Once the robot's body is detected to have dropped suddenly to the ground and is judged to have fallen, a large negative reward is immediately given and the current training episode is terminated.

6. The robust walking control method for a high gear ratio humanoid robot based on potential dynamics adaptation according to claim 1 is characterized in that: The step 4 comprises the following steps: Periodic gait generator: First, determine the parameterized description of gait. Human walking is a periodic motion, which can be represented by the gait period T and the time phase within the period φ∈[0,2π). Design a periodic gait generator G(φ; η), which outputs the target reference posture or foot trajectory of each joint of the robot based on the phase φ, where η is a gait style parameter, G uses a simple sine function combination or stored key posture interpolation, and uses the von Mises distribution to process the phase. The von Mises distribution is a probability distribution defined on the circumference. Its shape is similar to the normal distribution but is applicable to angular variables. This embodiment uses the von Mises distribution to generate adjacent gait key points on the gait phase to ensure a smooth transition at 0 and 2π; Human motion data retargeting: A general human motion retargeting method is used. First, a rich set of walking clips with varying speeds, strides, and styles are selected from a motion capture database. For each clip, the real person's skeletal joint angles or end-point trajectory are extracted at the keyframes. Then, based on the size and joint range of the GTX-III robot, inverse kinematics is used to map these key poses to robot-implemented joint angle combinations. During the IK solution, the goal is to maintain global pose similarity. An optimization algorithm is used to minimize the error caused by the proportional difference between the GTX-III and the real person. Unachievable poses are fine-tuned or discarded. Gait data storage and classification: The robot joint trajectories obtained in the previous step are stored in the gait library and labeled and classified according to the gait attributes. For each category, its average gait cycle and stride parameters are calculated. Each gait is stored as multiple cycles of data and can be played back in a loop. The gait library data includes not only the changes in joint angles over time, but also some features calculated from these trajectories. These features will be used to calculate the reward function. Methods of application in training: In reinforcement learning training, the reference data of the gait library can be used in many ways: Reference reward calculation*: The reference gait corresponding to the current desired speed in the gait library is used to calculate R arm 、R T 、R f Reward term, during training, the current simulated gait phase φ is inferred by accumulating time or steps, and then the reference posture of the corresponding phase is queried from the gait generator or library, compared with the actual posture of the robot, and reward feedback is given; Initial strategy guidance: In the early stages of training, to accelerate convergence, a reference gait is used as the initial demonstration. Specifically, in the first batch of training episodes, the robot executes a reference gait, thereby collecting some trajectory data close to the ideal gait for the reinforcement learning algorithm to learn. Diversified gait training: The gait library contains multiple styles. During training, different reference styles are randomly extracted for different episodes, allowing the policy to see a wider range of walking styles.

7. The robust walking control method for a high gear ratio humanoid robot based on potential dynamics adaptation according to claim 1 is characterized in that: The step 5 comprises the following steps: Simulation course training: Using the concept of course learning, the difficulty of the training environment is gradually increased. In the initial stage, the robot is trained to walk on a flat and hard surface. The environment parameters are set close to the ideal model. This stage enables the policy to master basic walking balance skills and imitate simple reference gaits. As training progresses, more challenging situations are gradually introduced, including: a) Terrain complexity improvement: In the subsequent stages of simulation, different terrains are introduced, with small changes at the beginning. Then gradually increase the unevenness and slope; b) Dynamic randomization: Using domain randomization technology, a series of dynamic parameters are randomly perturbed in the simulation; c) Gradual reward weighting: During course training, the emphasis of the reward function is adjusted at different stages; d) Curriculum learning allows the strategy to converge in easier environments by gradually increasing the challenge, and then continues training in more difficult environments using the existing strategy as the initial strategy; Dealing with and migrating virtual and real differences: a) Offline Verification and Simulation Comparison: Before conducting on-device experiments, the policy is replicated and verified using a high-fidelity simulator. This includes testing policy behavior using a more sophisticated robot model. If anomalies are detected, training is rolled back or the network is adjusted until the high-fidelity simulation test passes. b) Gradual acceleration / amplitude limiting: When first running the policy on a real robot, conservative scaling is applied to the actions output by the policy, and then these limits are gradually lifted to allow the policy to fully function. c) Online LDAN Adjustment: The robot is allowed to stand still or move in small movements. Sensor data is used to continuously update the LDAN's encoded output until it stabilizes. This gives the LDAN an online "calibration" period to ensure that its output accurately reflects real-world dynamics. During this period, the strategy focuses on maintaining balance and avoids large strides. Once the output converges, it indicates that the LDAN has a good understanding of the real environment and can begin walking normally. Training and adjustment in a real-world environment: The GTX-III robot is allowed to walk several steps, and sensor data and policy outputs are recorded. This data is then fed into the training framework for a small amount of policy fine-tuning. After one or two rounds of fine-tuning, the final policy is determined for subsequent long-term operation.

8. A robot control system applicable to the control method according to any one of claims 1 to 7, characterized in that: The control host is configured with a CPU or GPU and can run a reinforcement learning inference model; the robot has a high-reduction ratio servo joint and an IMU posture measurement module; the control frequency is not less than 30 Hz, and the actuator response time is less than 10 ms; the system does not rely on force sensors during operation and can achieve online dynamic adaptation.

Images