Humanoid robot complex terrain motion control method and system based on DAMP

Abstract

The invention provides a DAMP-based humanoid robot complex terrain motion control method and system, and belongs to the field of robot motion control, and the method comprises the steps: constructing a DAMP reinforcement learning control frame fusing a denoising world model module, an adversarial motion prior module, a dynamic reward interpolation mechanism and a Lipschitz continuity punishment module; firstly, a denoising world model is utilized to extract robust potential state representation and reconstruct a real state; secondly, diversified human-like actions are generated based on human body motion capture data; then adaptively adjusting the weights of simulation rewards and task rewards through a dynamic reward interpolation mechanism, optimizing a learning strategy, applying Lipschitz continuity to punish the smoothness of a control strategy, and reducing jitter of control output; and finally, overall training is carried out based on the unified training target function, so that the autonomous motion ability of the humanoid robot in the complex terrain is improved, and the motion stability and naturalness of the robot in the complex terrain are improved.

Description

Technical Field

[0001] This invention relates to the field of robot motion control technology, and in particular to a method and system for controlling the motion of a humanoid robot in complex terrain based on DAMP. Background Technology

[0002] Research and development in the field of motion control for humanoid robots in complex terrain has attracted widespread attention. These robots need to cope with varied environments such as stairs, slopes, and obstacles. Compared to quadruped robots, humanoid robots typically face a higher center of gravity and more complex degrees of freedom, making them more prone to instability in unstable terrain. Currently, many traditional control methods, such as model-based control (e.g., zero-moment point (ZMP) control, model predictive control (MPC), and whole-body control (WBC)), perform well in structured environments, but their adaptability in complex terrain is poor due to their reliance on accurate models. Therefore, improving the stable and natural mobility of humanoid robots in diverse and complex terrains has become a core research objective.

[0003] In recent years, with the rapid development of artificial intelligence technology, especially in the fields of reinforcement learning (RL) and imitation learning (IL), the motion control technology of humanoid robots has begun to develop towards greater intelligence and adaptability. Researchers have attempted to improve the terrain adaptability of robots by using reinforcement learning algorithms such as Hybrid Internal Model (HIM) and Dream WaQ. These methods solve control problems through simulation training, but still face challenges such as low learning efficiency due to reward sparsity and insufficient naturalness of generated actions. Meanwhile, imitation learning methods attempt to generate human-like actions through human demonstrations, but usually require high-quality demonstration data and have limitations in generalization ability. In addition, the accuracy and robustness of state estimation have also become important factors affecting the control performance of robots.

[0004] Despite the existence of various control methods, these techniques still have significant shortcomings. First, the reward function design in reinforcement learning is complex, easily leading to unnatural gait and low energy efficiency. Imitation learning generally relies on expensive human demonstration data, resulting in insufficient generalization ability and low success rate in complex terrains such as high steps and steep slopes, often exhibiting unsmooth control processes. Second, existing methods are susceptible to sensor noise, and the sensitivity of state estimation affects the stability of the control system. Finally, the lack of a unified control framework prevents the effective integration of denoising modeling, adversarial imitation, and policy smoothness control, thus hindering a comprehensive improvement in the robot's adaptability to complex environments. Summary of the Invention

[0005] To overcome the shortcomings of existing technologies, the purpose of this invention is to provide a method and system for controlling the motion of humanoid robots in complex terrain based on DAMP. By constructing a control framework that integrates a denoised world model, adversarial motion priors, dynamic reward interpolation mechanisms, and Lipschitz continuity penalty, the method improves the stability and naturalness of the robot's motion in complex terrain, achieves autonomous adaptive motion control, and is applicable to a variety of application scenarios.

[0006] To achieve the above objectives, the present invention provides the following solution: On the one hand, the present invention provides a method for controlling the motion of a humanoid robot in complex terrain based on DAMP, comprising the following steps: A DAMP reinforcement learning control framework is constructed, which integrates a denoised world model module, an adversarial motion prior module, a dynamic reward interpolation mechanism, a Lipschitz continuous penalty module, and a unified training objective function module. Based on the DAMP reinforcement learning control framework, historical and real-time observation data of the humanoid robot are obtained, and robust latent state representations are extracted and real states are reconstructed through the denoised world model module. By combining the adversarial motion prior module with human motion capture data, diverse and natural human-like movements are generated. By using a dynamic reward interpolation mechanism, the weight distribution of imitation rewards and task rewards is adaptively adjusted according to the task rewards; The Lipschitz continuity penalty module is used to impose a smoothness constraint on the policy gradient, thereby reducing control output jitter. The DAMP reinforcement learning control framework is trained by fusing denoising loss, policy gradient loss, value function error, entropy regularization and LCP gradient penalty based on a unified training objective function module. The trained DAMP reinforcement learning control framework is deployed to the humanoid robot to output joint torque control signals, enabling the humanoid robot to achieve autonomous motion control in complex terrain.

[0007] On the other hand, the present invention also provides a system for implementing the above-mentioned DAMP-based humanoid robot motion control method in complex terrain, comprising: The data acquisition module is used to acquire historical and real-time observation data of the humanoid robot. The observation data includes joint angles, joint velocities, base posture, and base angular velocity. The framework building module is used to build the DAMP reinforcement learning control framework, which integrates a denoised world model module, an adversarial motion prior module, a dynamic reward interpolation mechanism, a Lipschitz continuity penalty module, and a unified training objective function module. The state processing module is used to extract robust latent state representations and reconstruct the real state through the denoised world model module. The motion generation module is used to generate diverse and natural human-like movements by combining the adversarial motion prior module with human motion capture data. The reward adjustment module is used to adaptively adjust the weight distribution of imitation rewards and task rewards based on task rewards using a dynamic reward interpolation mechanism. The smoothing constraint module is used to impose smoothing constraints on the policy gradient through the Lipschitz continuity penalty module, thereby reducing control output jitter. The model training module is used to train the DAMP reinforcement learning control framework and introduce a domain randomization policy based on the unified training objective function module, which integrates denoising loss, policy gradient loss, value function error, entropy regularization and LCP gradient penalty. The control output module is used to deploy the trained DAMP reinforcement learning control framework to the humanoid robot, outputting a 500Hz joint torque control signal to drive the robot to move autonomously in complex terrain without relying on external sensing devices such as depth cameras and LiDAR.

[0008] According to specific embodiments provided by the present invention, the present invention discloses the following technical effects: This invention, based on a denoised world model and adversarial motion prior (DAMP) reinforcement learning control framework, significantly improves the motion stability and adaptability of humanoid robots in complex terrains. By constructing a denoised world model module, the robot can extract robust latent state representations from historical and real-time observation data, enhancing its perception of dynamic environments. This improvement ensures smoother robot motion under various complex terrain conditions, reducing the likelihood of imbalance and thus improving its safety and reliability in practical applications. Secondly, this invention generates humanoid movements through the adversarial motion prior module, enhancing the naturalness and diversity of robot actions. Combined with human motion capture data, the robot can generate motion trajectories that better conform to human behavioral patterns, making it more flexible and efficient in task execution. This generation of humanoid movements not only enhances the robot's interactive capabilities but also broadens its applications in service, healthcare, and entertainment fields, aligning with human intuitive expectations. Finally, this invention utilizes a dynamic reward interpolation mechanism and a Lipschitz continuity penalty module to optimize reward allocation and policy smoothness during reinforcement learning, thereby improving learning efficiency. Adaptively adjusting the weights of imitation rewards and task rewards enables the robot to quickly adapt to complex environments and improve learning efficiency, reducing the training difficulties caused by reward sparsity in traditional methods. Furthermore, applying smoothness constraints effectively controls motion output jitter, ensuring more stable control performance in dynamic environments, thereby improving the overall performance and practical application potential of the humanoid robot. Attached Figure Description

[0009] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0010] Figure 1 This is a flowchart of a DAMP-based humanoid robot motion control method for complex terrain according to the present invention. Figure 2 This is a diagram illustrating the overall architecture of the DAMP system provided in an embodiment of the present invention. Detailed Implementation

[0011] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0012] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0013] like Figure 1 and Figure 2 As shown, this embodiment provides a method for controlling the motion of a humanoid robot in complex terrain based on DAMP, including the following steps: Step 100: Construct the DAMP reinforcement learning control framework, which integrates a denoised world model module, an adversarial motion prior module, a dynamic reward interpolation mechanism, a Lipschitz continuity penalty module, and a unified training objective function module.

[0014] The implementation process of constructing the DAMP reinforcement learning control framework described above includes: A denoised world model module was built, using an LSTM encoder-decoder structure. The input dimension of historical observation data and the output dimension of the latent state were configured, and the simulation parameter range of sensor noise was set.

[0015] Furthermore, when constructing the denoising world model module, a bidirectional LSTM encoder-decoder structure was adopted. The encoder consisted of three LSTM layers with 64 hidden neurons per layer, and a dropout layer (with a dropout ratio of 0.1) was added after each layer's output to prevent overfitting. The encoder input dimension was set to 18 dimensions to match the observed data dimension (2 x 5 joints in both legs + 2 x 4 joints in both arms + 3 x 3 dimensions of base posture + 3 x 3 x 3 dimensions of base angular velocity). The latent state output dimension was optimized to 32 dimensions using principal component analysis (PCA). The decoder input dimension was consistent with the latent state dimension, and the output dimension was consistent with the encoder input dimension. The decoder activation function was the sigmoid function. The sensor noise simulation parameters were based on a normal distribution with a mean of 0 and a variance ranging from 0.01 to 0.05, determined based on the statistical results of measured noise variance from three sets of the same type of robot sensors.

[0016] An adversarial motion prior module was built, and the network structure of the discriminator and generator was configured based on the WGAN-div loss function to determine the input format of human motion capture data and the output dimension of human-like actions.

[0017] Furthermore, when constructing the Adversarial Motion Prior Module (AMPModule), the discriminator and generator are configured based on the WGAN-div loss function. The discriminator uses a 3-layer MLP network with a 24-dimensional input dimension, 128 and 64 hidden layer neurons, and a 1-dimensional output layer. The activation functions are ReLU, LeakyReLU, and Linear, respectively. The generator uses a 4-layer MLP network with a 32-dimensional input dimension, 256, 128, and 64 hidden layer neurons, and a 24-dimensional output dimension. The activation functions are ReLU, ReLU, LeakyReLU, and Tanh, respectively. The generator is initialized using the Xavier method.

[0018] Establish a dynamic reward interpolation mechanism, set the initial weight ratio between imitation rewards and task rewards, and configure the trigger conditions for adaptive weight adjustment.

[0019] Furthermore, when building the Dynamic Reward Interpolation Mechanism, the initial weight ratio of imitation reward to task reward is set to 0.6:0.4, the weight adjustment increment is fixed at 0.1 / time, the adjustment trigger condition is that the task reward for 3 consecutive training steps exceeds the mean value of the most recent 100 training steps μ±2σ, and the upper and lower limits of the weight are constrained to 0.1~0.9.

[0020] Build the Lipschitz continuity penalty module, define the smoothness constraint threshold of the policy gradient, and set the calculation method of LCP gradient penalty.

[0021] Furthermore, when constructing the Lipschitz Continuity Penalty Module (LCP Module), the policy gradient smoothness constraint threshold is 0.5 rad / s. 2 Based on 50% of the robot's maximum permissible angular velocity (1 rad / s), the LCP gradient penalty calculation formula is as follows: when At that time, penalty value Otherwise, the penalty value is equal to 0, where, for Actions at any given moment; for Actions at any given moment; For gradient operators; It is an L2 norm.

[0022] A unified training objective function module was built, clarifying the fusion logic of denoising loss, policy gradient loss, value function error, entropy regularization and LCP gradient penalty, and determining the linkage calculation rules for each loss term.

[0023] Furthermore, when constructing the Unified Training Objective Function Module, the weighted summation formula for each loss term is: Total Loss = 0.3 × Denoising Loss + 0.4 × Policy Gradient Loss + 0.1 × Value Function Error + 0.05 × Entropy Regularization + 0.15 × LCP Gradient Penalty. Here, the denoising loss is the error loss between reconstructing the true state and the robot's actual state; the policy gradient loss is the loss for guiding policy optimization; the value function error is the error between the value network prediction and the target value; the entropy regularization is a regularization term to improve the policy's exploratory nature; and the LCP gradient penalty is a penalty term to constrain the smoothness of the policy gradient. The policy gradient loss adopts the clip loss of the PPO algorithm (the clip range is set to [0.9, 1.1]).

[0024] Step 200: Based on the DAMP reinforcement learning control framework, acquire the historical and real-time observation data of the humanoid robot, extract robust latent state representations and reconstruct the real state through the denoising world model module.

[0025] Specifically, the process of acquiring observation data and extracting the reconstructed state includes: Collect historical and real-time observation data of the humanoid robot, including joint angles, joint velocities, base posture, and base angular velocity; Historical observation data is input into the LSTM encoder of the denoised world model module, and encoding operations are performed in combination with simulated sensor noise to output a latent state representation. The latent state representation is input into the LSTM decoder, and the robot's true state is reconstructed through decoding operations. By adjusting the network parameters of the encoder and decoder in reverse based on the state reconstruction error, the robustness of latent state extraction and the accuracy of real state reconstruction are improved.

[0026] Furthermore, the preprocessed historical observation data is concatenated with the real-time observation data and input into the encoder (LSTM) of the Denoising World Model Module. Simulated sensor noise is incorporated, and encoding operations are performed to output a 32-dimensional Latent State Representation (z). t The LSTM decoder (MLP) outputs an 18-dimensional reconstructed true state based on a 32-dimensional latent state representation, and the latent state extraction and true state reconstruction processes satisfy the following formula: ; ; In the formula, To reconstruct the true state, This represents the actual state of the robot. For LSTM encoders, For LSTM decoder, for The historical observation sequence prior to the time point, To simulate the normal distribution of sensor noise, The state robustness coefficient is calculated based on the variance of the observed sequence. For noise reduction loss, This represents the mathematical expectation, used to statistically average observed data with random fluctuations, simulated noise, or state variables, ensuring the robustness of loss calculation. Finally, the network parameters of the encoder and decoder are adjusted inversely based on the denoising loss to improve the robustness of latent state extraction and the accuracy of real state reconstruction.

[0027] Step 300: By combining the adversarial motion prior module with human motion capture data, generate diverse and natural human-like movements.

[0028] Specifically, the process of generating human-like movements through the adversarial motion prior module includes: Collect joint motion data of the human body in scenarios such as climbing stairs, walking on slopes, and crossing obstacles in complex terrain, and construct a human-like motion dataset; The human-like motion dataset is input into the discriminator of the adversarial motion prior module to train the discriminator's ability to distinguish between real human motion and generated motion. The generator receives the latent state output by the denoised world model module and combines it with the feedback signal from the discriminator to generate the initial humanoid action. The generator parameters are iteratively optimized based on the WGAN-div loss function to calculate the difference between the generated actions and real human actions, and output diverse and natural human-like actions.

[0029] Furthermore, human motion data is collected to construct a reference dataset, which, after preprocessing, is input into the discriminator of the Adversarial Motion Prior Module (AMP Module). Adversarial training is conducted in two phases: the first phase trains the discriminator independently, with the optimization objective being to minimize the classification error between real human actions and randomly generated actions; the second phase employs an alternating training mode for the discriminator and generator (i.e., actor, MLP). The generator receives the latent state representation (zt) output from the denoised world model module, outputs initial human-like actions to form a generated dataset, and inputs it along with the reference dataset into the discriminator. The discriminator outputs the differentiation results and feeds them back to the generator to optimize the action generation effect, while simultaneously outputting an imitation reward (r). s The PPO module is then used. Based on the WGAN-div loss function, the discriminator loss and generator loss are calculated, and after iterative optimization, diverse and natural human-like actions are output.

[0030] Step 400: Utilize a dynamic reward interpolation mechanism to adaptively adjust the weight distribution of imitation rewards and task rewards based on task rewards.

[0031] Specifically, the implementation process of adjusting weight allocation using a dynamic reward interpolation mechanism includes: The task reward of the humanoid robot is calculated in real time, and the task reward includes indicators such as speed tracking accuracy, motion stability, energy consumption, and foot contact naturalness. Analyze the mean and variance of historical task rewards to determine the threshold range for weight adjustment; When the current task reward is higher than the threshold, increase the weight of the task reward and decrease the weight of the imitation reward. When the current task reward is below the threshold, the weight of the imitation reward is increased and the weight of the task reward is decreased, thus achieving adaptive dynamic adjustment of the weight.

[0032] Furthermore, the input of the Dynamic Reward Interpolation Mechanism receives task rewards (r... t ) and Imitation Reward (r s The task reward is generated based on joint state data fed back by the PD controller, while the mimicry reward is output by the discriminator of the adversarial motion prior module. The weight adjustment logic is as follows: A threshold range is determined by statistically analyzing the mean and standard deviation of the most recent historical task rewards. When the task rewards for multiple consecutive training steps exceed the threshold, the weight ratio of the two types of rewards is adaptively adjusted. The adjusted weights are constrained within a preset range to avoid frequent fluctuations. The adjusted fused reward is input into the PPO (Proximal Policy Optimization) module to guide policy optimization.

[0033] Step 500: Apply smoothness constraints to the policy gradient using the Lipschitz continuity penalty module to reduce control output jitter.

[0034] Specifically, the implementation process of applying smoothness constraints through the Lipschitz continuity penalty module includes: Obtain the output action gradient of the policy network in adjacent time states; Calculate the magnitude of change in the motion gradient and determine whether it exceeds the preset smoothness constraint threshold. When the magnitude of the action gradient change exceeds the threshold, the LCP gradient penalty is calculated through the Lipschitz continuity penalty module and backpropagated to the policy network; The network parameters are adjusted based on the LCP gradient penalty strategy to constrain the magnitude of action gradient changes and reduce control output jitter.

[0035] Furthermore, the input of the nalty Module (LCP Module) receives the action gradients of adjacent time steps output by the policy network (i.e., the generator of the adversarial motion prior module), calculates the magnitude of the action gradient change, and compares it with a preset smoothness constraint threshold. When the magnitude of the action gradient change exceeds the threshold, the module generates an LCP gradient penalty and backpropagates it to the policy network; when it does not exceed the threshold, the penalty value is 0. This penalty mechanism constrains the magnitude of the policy gradient change, reduces control output jitter, and ensures the smoothness of the action output. Its gradient penalty calculation logic is directly related to the adjustment of the policy network parameters.

[0036] Step 600: Train the DAMP reinforcement learning control framework by fusing denoising loss, policy gradient loss, value function error, entropy regularization and LCP gradient penalty based on the unified training objective function module.

[0037] Specifically, the implementation process of training the DAMP reinforcement learning control framework includes: The denoising loss output by the denoising world model module, the policy gradient loss output by the policy network, the value function error calculated by the value function, the policy entropy regularization term, and the LCP gradient penalty output by the Lipschitz continuity penalty module are input into the unified training objective function. The gradients of each network parameter are calculated using the backpropagation algorithm, and the parameters of the denoised world model module, the adversarial motion prior module, and the policy network are iteratively updated using the gradient descent method. During training, a domain randomization strategy is introduced to randomize the physical parameters of environmental friction coefficient, robot component mass, and motor output strength. The terrain passability, motion smoothness, and energy efficiency indicators are monitored in real time. Training stops when the indicators reach the preset standards, resulting in a trained DAMP reinforcement learning control framework.

[0038] Furthermore, the Unified Training Objective Function Module receives the loss terms output by each core module as input: the denoising loss from the denoising world model module, the policy gradient loss from the policy network, the value function error calculated by the Critic (MLP), the entropy regularization term, and the LCP gradient penalty from the Lipschitz continuity penalty module. The total loss is obtained by weighting and summing the loss terms according to preset weights, and the parameters of the denoising world model module, the adversarial motion prior module, and the policy network are iteratively updated using the backpropagation algorithm. During training, each loss term is linked to the PPO module; training stops when the monitoring metric reaches a preset standard, resulting in the trained DAMP reinforcement learning control framework.

[0039] Step 700: Deploy the trained DAMP reinforcement learning control framework to the humanoid robot, output joint torque control signals, and realize the autonomous motion control of the humanoid robot in complex terrain.

[0040] Specifically, the process of deploying the framework and implementing autonomous motion control includes: The trained DAMP reinforcement learning control framework is embedded into the control unit of the humanoid robot, and the output interface for joint torque control signals is configured. The DAMP reinforcement learning control framework receives observation data from the robot in real time and outputs joint torque control signals through the linkage calculation of various modules. The joint torque control signal output frequency is 500Hz, which drives the robot's joint actuators to move, thereby enabling autonomous adaptive control in complex terrain using only observation data collected by the robot's own sensors.

[0041] Furthermore, the trained DAMP reinforcement learning control framework is embedded into the humanoid robot's embedded control unit. Parameter files for core modules such as the encoder, decoder, generator, and discriminator are loaded, and the observation data receiving interface and joint torque control signal output interface are configured. During runtime, the framework receives observation data collected by the robot's own sensors in real time. After the denoising world model module extracts latent states, the adversarial motion prior module generates humanoid movements, and the PPO module optimizes the strategy, it outputs joint torque control signals. The joint state is fed back to the framework in real time, forming a closed-loop control system. This eliminates the need for external sensing devices, enabling autonomous motion control in complex terrain.

[0042] This embodiment also provides a system for implementing the above-described DAMP-based humanoid robot motion control method in complex terrain, including: The data acquisition module is used to acquire historical and real-time observation data of the humanoid robot. The observation data includes joint angles, joint velocities, base posture, and base angular velocity. The framework building module is used to build the DAMP reinforcement learning control framework, which integrates a denoised world model module, an adversarial motion prior module, a dynamic reward interpolation mechanism, a Lipschitz continuity penalty module, and a unified training objective function module. The state processing module is used to extract robust latent state representations and reconstruct the real state through the denoised world model module. The motion generation module is used to generate diverse and natural human-like movements by combining the adversarial motion prior module with human motion capture data. The reward adjustment module is used to adaptively adjust the weight distribution of imitation rewards and task rewards based on task rewards using a dynamic reward interpolation mechanism. The smoothing constraint module is used to impose smoothing constraints on the policy gradient through the Lipschitz continuity penalty module, thereby reducing control output jitter. The model training module is used to train the DAMP reinforcement learning control framework and introduce a domain randomization policy based on the unified training objective function module, which integrates denoising loss, policy gradient loss, value function error, entropy regularization and LCP gradient penalty. The control output module is used to deploy the trained DAMP reinforcement learning control framework to the humanoid robot, outputting a 500Hz joint torque control signal to drive the robot to move autonomously in complex terrain without relying on external sensing devices such as depth cameras and LiDAR.

[0043] Specifically, the system provided in this embodiment is characterized by: an observation data acquisition module as the input layer, responsible for acquiring historical and real-time observation data of the humanoid robot, covering core data types such as joint angles, joint velocities, base posture, and base angular velocity; and a denoising world model module undertaking the core functions of data preprocessing and state extraction, employing an LSTM encoder-decoder structure with a built-in sensor noise simulation mechanism, responsible for data preprocessing, encoding operations, and outputting a latent state representation (z). t The system decodes and reconstructs the real state; the Adversarial Motion Prior Module (AMP Module), as the core action generation layer, consists of a generator (Generator, Actor, MLP) and a discriminator. It relies on adversarial training with a reference dataset and a generated dataset to generate diverse natural human-like actions; the dynamic reward interpolation mechanism and the Lipschitz continuity penalty module constitute the policy optimization auxiliary layer, the former responsible for adaptively adjusting the imitation reward (r... s ) and Task Rewards tThe weight allocation, which imposes a smoothness constraint on the policy gradient to reduce control jitter, is implemented. The unified training objective function module serves as the core optimization layer, integrating denoising loss, policy gradient loss, value function error, entropy regularization term, and LCP gradient penalty. It completes the iterative update of the entire framework parameters through backpropagation. The embedded deployment and control output module serves as the output layer, solidifying the trained framework into the embedded control unit. It configures the joint torque control signal output interface to drive the robot joint actuators and forms a closed-loop control based on joint state feedback.

[0044] Furthermore, this embodiment also provides specific experiments to further verify the claims.

[0045] This embodiment is based on the N2 humanoid robot platform. The robot weighs 33 kg, is 1.18 m tall, and has 5 degrees of freedom for each leg and 4 degrees of freedom for each arm. The experiment is divided into two phases: simulation training and real-world scenario verification. The simulation training phase is primarily conducted using the Isaac Gym simulation environment. During training, the observation space covers core state data such as robot joint angles, joint velocities, base posture, and base angular velocity. The reward function comprehensively incorporates multiple dimensions of indicators, including speed tracking accuracy, motion stability, energy consumption, motion smoothness, and naturalness of foot contact. Simultaneously, a domain randomization strategy is employed to randomize physical parameters such as environmental friction coefficient, robot component mass, and motor output intensity, effectively improving the model's Sim2Real transferability. The controller output is a joint torque signal, with the output frequency set to 500 Hz to ensure real-time motion response. After the simulation training was completed, the DAMP reinforcement learning control framework was deployed to the control unit of the real N2 humanoid robot. The entire process did not rely on external perception devices such as depth cameras and LiDAR, but only used the robot's own sensors to collect observation data to complete control decisions. Subsequently, the robot's motion performance was verified under typical complex terrains such as slopes, steps, and obstacles. The test results showed that the robot's motion success rate under various complex terrains was ≥90%, and it had the advantages of smooth control and low energy consumption.

[0046] Therefore, the above-mentioned humanoid robot motion control method and system based on DAMP in complex terrain improves the robot's motion stability and naturalness in complex terrain by constructing a control framework that integrates a denoised world model, adversarial motion priors, dynamic reward interpolation mechanism, and Lipschitz continuity penalty, and achieves autonomous adaptive motion control, which is suitable for a variety of application scenarios.

[0047] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. Furthermore, those skilled in the art will recognize that, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for controlling the motion of a humanoid robot in complex terrain based on DAMP, characterized in that, Includes the following steps: A DAMP reinforcement learning control framework is constructed, which integrates a denoised world model module, an adversarial motion prior module, a dynamic reward interpolation mechanism, a Lipschitz continuous penalty module, and a unified training objective function module. Based on the DAMP reinforcement learning control framework, historical and real-time observation data of the humanoid robot are obtained, and robust latent state representations are extracted and real states are reconstructed through the denoised world model module. By combining the adversarial motion prior module with human motion capture data, diverse and natural human-like movements are generated. By using a dynamic reward interpolation mechanism, the weight distribution of imitation rewards and task rewards is adaptively adjusted according to the task rewards; The Lipschitz continuity penalty module is used to impose a smoothness constraint on the policy gradient, thereby reducing control output jitter. The DAMP reinforcement learning control framework is trained by fusing denoising loss, policy gradient loss, value function error, entropy regularization and LCP gradient penalty based on a unified training objective function module. The trained DAMP reinforcement learning control framework is deployed to the humanoid robot to output joint torque control signals, enabling the humanoid robot to achieve autonomous motion control in complex terrain.

2. The method for controlling the motion of a humanoid robot in complex terrain based on DAMP according to claim 1, characterized in that, The implementation process of constructing the DAMP reinforcement learning control framework includes: A denoised world model module was built, using an LSTM encoder-decoder structure. The input dimension of historical observation data and the output dimension of the latent state were configured, and the simulation parameter range of sensor noise was set. An adversarial motion prior module was built, and the network structure of the discriminator and generator was configured based on the WGAN-div loss function to determine the input format of human motion capture data and the output dimension of human-like actions. Establish a dynamic reward interpolation mechanism, set the initial weight ratio between imitation rewards and task rewards, and configure the trigger conditions for adaptive weight adjustment; Build the Lipschitz continuity penalty module, define the smoothness constraint threshold of the policy gradient, and set the calculation method of LCP gradient penalty; A unified training objective function module was built, clarifying the fusion logic of denoising loss, policy gradient loss, value function error, entropy regularization and LCP gradient penalty, and determining the linkage calculation rules for each loss term.

3. The method for controlling the motion of a humanoid robot in complex terrain based on DAMP according to claim 1, characterized in that, The process of acquiring observation data and extracting the reconstructed state includes: Collect historical and real-time observation data of the humanoid robot, including joint angles, joint velocities, base posture, and base angular velocity; Historical observation data is input into the LSTM encoder of the denoised world model module, and encoding operations are performed in combination with simulated sensor noise to output a latent state representation. The latent state representation is input into the LSTM decoder, and the robot's true state is reconstructed through decoding operations. By adjusting the network parameters of the encoder and decoder in reverse based on the state reconstruction error, the robustness of latent state extraction and the accuracy of real state reconstruction are improved.

4. The method for controlling the motion of a humanoid robot in complex terrain based on DAMP according to claim 3, characterized in that, The latent state extraction and real state reconstruction process of the denoised world model module satisfies the following formula: ； ； In the formula, To reconstruct the true state, This represents the actual state of the robot. For LSTM encoders, For LSTM decoder, for The historical observation sequence prior to the time point, To simulate the normal distribution of sensor noise, The state robustness coefficient is calculated based on the variance of the observed sequence. For noise reduction loss, Representing mathematical expectation, it is used to statistically average observed data, simulated noise, or state variables with random fluctuations, ensuring the robustness of loss calculation.

5. The method for controlling the motion of a humanoid robot in complex terrain based on DAMP according to claim 1, characterized in that, The process of generating human-like movements through the adversarial motion prior module includes: Collect joint motion data of the human body in scenarios such as climbing stairs, walking on slopes, and crossing obstacles in complex terrain, and construct a human-like motion dataset; The human-like motion dataset is input into the discriminator of the adversarial motion prior module to train the discriminator's ability to distinguish between real human motion and generated motion. The generator receives the latent state output by the denoised world model module and combines it with the feedback signal from the discriminator to generate the initial humanoid action. The generator parameters are iteratively optimized based on the WGAN-div loss function to calculate the difference between the generated actions and real human actions, and output diverse and natural human-like actions.

6. The method for controlling the motion of a humanoid robot in complex terrain based on DAMP according to claim 1, characterized in that, The implementation process of adjusting weight allocation using a dynamic reward interpolation mechanism includes: The task reward of the humanoid robot is calculated in real time, and the task reward includes indicators such as speed tracking accuracy, motion stability, energy consumption, and foot contact naturalness. Analyze the mean and variance of historical task rewards to determine the threshold range for weight adjustment; When the current task reward is higher than the threshold, increase the weight of the task reward and decrease the weight of the imitation reward. When the current task reward is below the threshold, the weight of the imitation reward is increased and the weight of the task reward is decreased, thus achieving adaptive dynamic adjustment of the weight.

7. The method for controlling the motion of a humanoid robot in complex terrain based on DAMP according to claim 1, characterized in that, The implementation process of applying smoothness constraints through the Lipschitz continuity penalty module includes: Obtain the output action gradient of the policy network in adjacent time states; Calculate the magnitude of change in the motion gradient and determine whether it exceeds the preset smoothness constraint threshold. When the magnitude of the action gradient change exceeds the threshold, the LCP gradient penalty is calculated through the Lipschitz continuity penalty module and backpropagated to the policy network; The network parameters are adjusted based on the LCP gradient penalty strategy to constrain the magnitude of action gradient changes and reduce control output jitter.

8. The method for controlling the motion of a humanoid robot in complex terrain based on DAMP according to claim 1, characterized in that, The implementation process of training the DAMP reinforcement learning control framework includes: The denoising loss output by the denoising world model module, the policy gradient loss output by the policy network, the value function error calculated by the value function, the policy entropy regularization term, and the LCP gradient penalty output by the Lipschitz continuity penalty module are input into the unified training objective function. The gradients of each network parameter are calculated using the backpropagation algorithm, and the parameters of the denoised world model module, the adversarial motion prior module, and the policy network are iteratively updated using the gradient descent method. During training, a domain randomization strategy is introduced to randomize the physical parameters of environmental friction coefficient, robot component mass, and motor output strength. The terrain passability, motion smoothness, and energy efficiency indicators are monitored in real time. Training stops when the indicators reach the preset standards, resulting in a trained DAMP reinforcement learning control framework.

9. A method for controlling the motion of a humanoid robot in complex terrain based on DAMP according to claim 1, characterized in that, The process of deploying the framework and implementing autonomous motion control includes: The trained DAMP reinforcement learning control framework is embedded into the control unit of the humanoid robot, and the output interface for joint torque control signals is configured. The DAMP reinforcement learning control framework receives observation data from the robot in real time and outputs joint torque control signals through the linkage calculation of various modules. The joint torque control signal output frequency is 500Hz, which drives the robot's joint actuators to move, thereby enabling autonomous adaptive control in complex terrain using only observation data collected by the robot's own sensors.

10. A system for implementing the DAMP-based humanoid robot motion control method for complex terrain as described in any one of claims 1 to 9, characterized in that, include: The data acquisition module is used to acquire historical and real-time observation data of the humanoid robot. The observation data includes joint angles, joint velocities, base posture, and base angular velocity. The framework building module is used to build the DAMP reinforcement learning control framework, which integrates a denoised world model module, an adversarial motion prior module, a dynamic reward interpolation mechanism, a Lipschitz continuity penalty module, and a unified training objective function module. The state processing module is used to extract robust latent state representations and reconstruct the real state through the denoised world model module. The motion generation module is used to generate diverse and natural human-like movements by combining the adversarial motion prior module with human motion capture data. The reward adjustment module is used to adaptively adjust the weight distribution of imitation rewards and task rewards based on task rewards using a dynamic reward interpolation mechanism. The smoothing constraint module is used to impose smoothing constraints on the policy gradient through the Lipschitz continuity penalty module, thereby reducing control output jitter. The model training module is used to train the DAMP reinforcement learning control framework and introduce a domain randomization policy based on the unified training objective function module, which integrates denoising loss, policy gradient loss, value function error, entropy regularization and LCP gradient penalty. The control output module is used to deploy the trained DAMP reinforcement learning control framework to the humanoid robot, outputting a 500Hz joint torque control signal to drive the robot to move autonomously in complex terrain without relying on external sensing devices such as depth cameras and LiDAR.

Images