A training method and a control method of an actor-critic network for humanoid robot humanoid walk and run learning control
By designing the Actor-critic network structure and reward function, and combining the correspondence between the human body and the robot model, the problems of anthropomorphism, dynamic rationality, and gait continuity in the motion control of humanoid robots were solved, and stable and coordinated anthropomorphic walking and running control was achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- HUAZHONG UNIV OF SCI & TECH
- Filing Date
- 2026-03-20
- Publication Date
- 2026-06-05
Smart Images
Figure CN122143010A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of robotics technology, and more specifically, relates to a training method and control method for an Actor-critic network for anthropomorphic walking and running learning control of humanoid robots. Background Technology
[0002] Humanoid robots, due to their structural morphology and movement similarities to humans, can adapt to complex and ever-changing human production and living environments, and have broad application prospects in fields such as industrial manufacturing, logistics and distribution, and public services. During human upright walking, the lower limbs complete the stepping and support, while the upper limbs and torso compensate for the disturbances caused by the swinging of the lower limbs through coordinated movements, thereby achieving stable and efficient whole-body coordinated walking. Simulating this whole-body coordination mechanism to generate humanoid robot motion with anthropomorphic characteristics is one of the important research directions in the field of robot motion control.
[0003] Existing methods for achieving walking and running in humanoid robots can be broadly categorized into two types: model-based methods and learning-based methods. Model-based methods typically combine Model Predictive Control (MPC) and Whole-Body Control (WBC) frameworks, achieving gait planning and stability control through explicit dynamic modeling. Although these methods offer good interpretability and stability, the high degree of freedom and complex dynamic models of humanoid robots often lead to the use of simplified models such as inverted pendulums or single rigid bodies in practical applications. This makes it difficult to fully characterize the dynamic coupling between the upper limbs, torso, and lower limbs, resulting in stiff gaits and a lack of human-like coordinated whole-body behavior.
[0004] Learning-based methods utilize reinforcement learning or imitation learning to train control strategies in simulated environments with complete dynamics, and then deploy the models onto real robots. With the development of deep reinforcement learning technology, these methods have shown strong adaptability in complex terrains and high-degree-of-freedom control tasks. However, traditional reinforcement learning methods typically rely on a large number of manually designed reward functions, which are difficult to tune and highly sensitive to weight settings, easily leading to unnatural joint movements or abnormal gaits.
[0005] To improve the anthropomorphism of movement, the Adversarial Motion Priors (AMP) method has been introduced into the field of humanoid robot control. By learning the distribution characteristics of human motion data through a discriminator, the strategy is guided to generate movements that are close to human style. However, when this type of method is directly applied to the walking and running control of humanoid robots, the following shortcomings still exist: (1) In methods that only consider the movement style, the differences between human and robot dynamics are not fully considered. Directly imitating human movement can easily lead to excessive angular momentum of the center of mass, affecting walking stability and increasing the demand for foot friction torque; (2) There is an uneven distribution of human motion data during the acquisition and redirection process, which can easily cause inconsistent responses of the strategy on the left and right limbs, resulting in gait asymmetry; (3) Existing methods are difficult to uniformly describe the continuous change process between walking and running, and are difficult to simultaneously cover the double support phase of low-speed walking and the flight phase of high-speed running, resulting in unsmooth gait switching.
[0006] In summary, existing technologies still struggle to simultaneously achieve anthropomorphic walking and running movements in humanoid robots, while maintaining the anthropomorphism of the movement morphology, the rationality of the whole-body dynamics, and the continuous evolution of gait across different speed ranges. Therefore, it is necessary to propose a learning control method that integrates prior knowledge of human motion and dynamics with continuous gait modeling to realize anthropomorphic walking and running movements in humanoid robots. Summary of the Invention
[0007] In view of the above-mentioned defects or improvement needs of the existing technology, the present invention provides a training method and control method for an Actor-critic network for humanoid robot anthropomorphic walking and running learning control, which solves the problem that existing control methods are difficult to simultaneously take into account the anthropomorphism of the motion form, the rationality of the whole body dynamics, and the continuous evolution characteristics of the gait in different speed ranges.
[0008] To achieve the above objectives, according to one aspect of the present invention, a method for training an Actor-critic network for anthropomorphic walking and running learning control of a humanoid robot is provided, the method comprising the following steps: Establish a correspondence between human body models and robot models, and use this correspondence to convert the motion vectors of the human body models into robot joint motion vectors, thereby establishing a motion dataset of robot joint motion vectors. The preset expected walking and running speeds of the robot are preprocessed, and the preprocessed data is input into the policy network of the Actor-critic network as observation data. The output is the joint motion that controls the robot's movement. The controller converts the joint motion into the joint driving torque that controls the robot, and uses the joint driving torque to simulate and control the robot. The robot's real-time motion state, joint position, and velocity during simulation control are obtained, and the total reward function of the Actor-critic network is calculated accordingly. The linear velocity of the robot body and the force at the foot end are obtained during the robot simulation control process. The linear velocity and the force at the foot end are input into the value network of the Actor-critic network along with the observed data to obtain the state value of the robot's current state. The discriminator determines whether the joint positions and velocities obtained during robot simulation control come from the motion dataset, and calculates the discriminator loss function accordingly. The total loss function of the Actor-critic network is calculated using the total reward function, state value, and discriminator loss function. The policy network is then adjusted to minimize the total loss function, thereby enabling the training of the Actor-critic network.
[0009] More preferably, the total reward function includes kinematic anthropomorphic reward, dynamic anthropomorphic reward, and task reward, and its formula is as follows:
[0010]
[0011] in, Weighting of kinematic anthropomorphic rewards From state to state Total reward at the time, , As a reward for the task, For the anthropomorphic reward of dynamics, , These are the center-of-mass angular momentum rewards in the anthropomorphic reward system of dynamics. Z-direction center of mass angular momentum smoothing reward XY direction centroid angular momentum smoothing reward The weighting coefficients.
[0012] More preferably, the kinematic anthropomorphic reward The formula for task rewards is as follows: :
[0013]
[0014] in, For each sub-reward in the task rewards The set, index For set The indices of the neutron reward items correspond to the sets in sequence. Each sub-reward, For the first Item Reward The corresponding weighting coefficients, Indicates in and Take the larger value. Discriminator pair The judgment result.
[0015] More preferably, the formula for the total loss function is as follows:
[0016]
[0017]
[0018] in, Represents the PPO loss function. The weighting coefficients are the mirror loss coefficients of the strategy. Let be the mirror consistency loss function of the policy network. The overall loss function of the discriminator network, For the discriminative loss term based on adversarial motion priors, The coefficients used to balance the weights of mirror consistency constraints. For the discriminator's mirror consistency loss, Discriminator pair The judgment result, For the discriminator pair The judgment result, and They are respectively The mirror state.
[0019] More preferably, the pre-built motion dataset is constructed by: establishing a correspondence between a human model and a robot model, and using this correspondence to convert the motion vectors of the human model into robot joint motion vectors, thereby obtaining a motion dataset of robot joint motion vectors.
[0020] More preferably, the correspondence between the human body model and the robot model is established by minimizing the geometric error between the human body model and the robot model at the matching joints, as shown in the following formula:
[0021] in, It refers to the geometric error at the matching joints between the human body model and the robot model. For the first generation generated by the SMPL-X model The position of each matching joint in the world coordinate system. These are the morphological parameters of the SMPL-X model, controlling the shape of various human body parts within the SMPL-X model. To match the position of joints relative to the pelvic joints in the robot model. The location of the human pelvic joint in the world coordinate system. The number of joints to be matched. These are regularization weights for morphological parameters, used to constrain the amplitude of morphological parameters and prevent unreasonable human body shapes. This is a global scale parameter that controls the overall scaling ratio of the SMPL-X human body model.
[0022] More preferably, the formula for converting the motion vectors of the human body model into robot joint motion vectors is as follows:
[0023]
[0024] in, For the robot's generalized coordinates, including joint angles and the position of the root node and posture , A set of joints for human-machine matching. and and represent the in the SMPL-X human body model, respectively. The position and pose of each matching joint. and The weights represent the robot joint poses and positions obtained through robot forward kinematics. For the first For the pose error weights of the matched joints, For the first Weights for the positional errors of the matched joints. and These are the upper and lower limits of the robot's joint angles, respectively.
[0025] More preferably, the controller is a PD controller, and the control formula of the PD controller is as follows:
[0026] in, and These are the proportional coefficient vector and differential coefficient vector of each joint, respectively. This indicates the robot's default joint angle configuration. This refers to the joint position. For joint velocity, It's a joint movement. for The joint driving torque at any given moment.
[0027] According to another aspect of the present invention, a control method for humanoid robot walking and running is provided. This control method uses an Actor-critic network trained by the training method described above to control the humanoid robot. The method includes the following steps: The desired speed of the robot is preprocessed, and the preprocessed data is input into the policy network in the Actor-critic network to output the joint movements of the robot. The controller converts the joint movements into joint driving torques to control the robot, and uses these joint driving torques to control the humanoid robot.
[0028] According to another aspect of the present invention, a humanoid robot is provided, which is controlled using the control method described above.
[0029] In summary, the technical solutions conceived by this invention have the following beneficial effects compared with the prior art: 1. This invention constructs a control model for the walking and running motion of a humanoid robot using an Actor-critic network structure. In the early stage of control, the Actor-critic network structure is trained with the desired speed as input. During the training process, the total reward function is calculated by combining real-time motion state. The state value of the body and feet during the control process is evaluated, and the loss function of the discriminator is calculated using real-time joint positions and velocities during simulation. Finally, the total loss function is calculated using the above three methods, and the Actor-critic network structure is adjusted from multiple dimensions to improve the control effect of the Actor-critic network structure. This achieves stable walking and running control of the humanoid robot under different speed conditions in actual control, which combines anthropomorphism, dynamic rationality, continuous transition, and symmetry.
[0030] 2. The total reward function of this invention, based on the task reward, introduces kinematic anthropomorphic reward and dynamic anthropomorphic reward. The kinematic anthropomorphic reward is constructed through the discriminator's results, aiming to make the robot's movement style similar to human movement style, achieving an anthropomorphic effect in kinematics. The dynamic anthropomorphic reward is constructed through the center of mass angular momentum, aiming to suppress unreasonable dynamic phenomena caused by directly imitating human data at the kinematic level due to differences in human-robot dynamics, such as excessive arm swing, thereby improving movement stability and energy consumption performance.
[0031] 3. The total loss function of this invention is extended based on the PPO loss function. First, a discriminator loss function is introduced, enabling the discriminator to train collaboratively with the policy network during adversarial learning, thereby guiding the policy to generate actions that are closer to the distribution of human movement. Second, a mirror consistency loss between the discriminator and the policy network is added during training, enabling the model to learn overall symmetrical and stable human gait behavior even when there is asymmetry in the distribution of the original human movement data, thereby improving gait symmetry.
[0032] 4. This invention introduces adversarial motion priors at the kinematic level and a constraint mechanism based on the center of mass angular momentum at the dynamic level. Combined with a unified gait phase based on velocity regulation and a network bisymmetric mechanism, it enables a humanoid robot to have anthropomorphic motion patterns, continuous and smooth walking and running states, and left-right symmetrical whole-body coordinated movements under different speed conditions. At the same time, it ensures reasonable overall dynamic characteristics and has good versatility and engineering application value. Attached Figure Description
[0033] Figure 1 This is a schematic diagram of a training framework for anthropomorphic walking and running strategies of a humanoid robot, provided according to a preferred embodiment of the present invention.
[0034] Figure 2 This is a schematic diagram of the degrees of freedom of a humanoid robot provided according to a preferred embodiment of the present invention.
[0035] Figure 3 This is a flowchart of human-machine data redirection provided according to a preferred embodiment of the present invention.
[0036] Figure 4 This is a schematic diagram of variable gait phase based on speed regulation provided according to a preferred embodiment of the present invention.
[0037] Figure 5 This is a schematic diagram of mirror data provided according to a preferred embodiment of the present invention.
[0038] Figure 6 This is a snapshot of the robot's walking and running motion provided according to a preferred embodiment of the present invention.
[0039] Figure 7 This is a walking and running speed tracking comparison curve provided according to a preferred embodiment of the present invention.
[0040] Figure 8 This is a comparison curve of the center of mass angular momentum of walking and running motion according to a preferred embodiment of the present invention.
[0041] Figure 9 This is a comparison curve of the z-axis torque at the foot end provided according to a preferred embodiment of the present invention, wherein (a) is the torque curve of the foot end at low speed. The curve comparing the z-axis torque at the foot end is shown in (b), which represents the torque at medium speed. The curve comparing the z-axis torque at the foot end is shown in (c), which represents high-speed running. Comparison curve of z-axis torque at the foot end.
[0042] Figure 10 These are energy consumption comparison curves at different speeds provided according to a preferred embodiment of the present invention, wherein (a) is low-speed walking. Energy consumption comparison curves, (b) is for medium-speed walking. Energy consumption comparison curves, (c) is for high-speed running Energy consumption comparison curve.
[0043] Figure 11 The curves provided in the preferred embodiment of the present invention are comparison curves of whether or not a symmetry mechanism is added, wherein (a) is the pitch angle of the left and right hip joints when a symmetry mechanism is added, and (b) is the pitch angle of the left and right hip joints when a symmetry mechanism is not added. Detailed Implementation
[0044] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.
[0045] This invention proposes a humanoid robot walking and running learning control method that incorporates kinematic and dynamic priors. Its control strategy framework is as follows: Figure 1 As shown in the figure, the proposed control method consists of two stages: training and inference. Dashed lines in the figure represent processes only included in the training stage, while solid lines represent processes common to both the training and application stages. The training process for the Actor-critic network is as follows: S1 Transformation of human-machine motion data to construct a motion dataset.
[0046] To effectively apply prior human motion data to the anthropomorphic walking and running learning control of humanoid robots, this invention first performs human-robot motion data redirection processing on the human motion data. The human motion data is selected from the AMASS motion capture dataset, which describes the joint movements and overall morphological features of the human body based on the SMPL-X human model. Figure 2 The humanoid robot serves as the target platform for human-machine motion data redirection.
[0047] like Figure 3As shown, the human-machine motion data redirection process in this embodiment includes: matching joint selection, human-machine shape alignment, and inverse kinematics solution steps, so as to convert the high-degree-of-freedom, shape-variable motion representation in the human model into a joint motion sequence that satisfies the kinematic constraints of the humanoid robot.
[0048] In the motion retargeting process, the joint matching relationship between the human model and the humanoid robot model is first established. Considering that the number of joints in the humanoid robot is usually less than that in the human model, and that some human joints do not have a one-to-one correspondence in the robot structure, this embodiment selects joints with consistent kinematic meaning in both the robot and the human model as matching joints. Specifically, in the upper limbs, the robot's shoulder pitch joint and elbow pitch joint are selected, corresponding to the human model's shoulder joint and elbow joint, respectively; in the lower limbs, the robot's hip pitch joint, knee pitch joint, and ankle pitch joint are selected, corresponding to the human model's hip, knee, and ankle joints, respectively. In addition, a correspondence is established between the robot's pelvis and the human model's Pelvis joint, and the robot's pelvis is defined as the root node of the entire robot joint system. Through the above matching method, a total of 11 joints are finally selected between the human model and the humanoid robot as matching joints for subsequent human motion to robot motion retargeting calculations.
[0049] SMPL-X morphological parameters provided in the AMASS motion capture dataset The human body shape parameters are generated based on the specific data collected. The proportions of the human body model often differ from those of the target robot, and directly using them for robot inverse kinematics solving can easily introduce significant geometric errors. Therefore, before optimizing the robot's joint angles, this embodiment first adjusts the overall shape of the human body model to make it as consistent as possible with the robot model in terms of spatial scale and joint distribution.
[0050] In the SMPL-X model, morphological parameters This is a low-dimensional vector used to control the overall body shape features. This embodiment selects a 16-dimensional morphological parameter vector and introduces a global scale parameter. The human body model is scaled up as a whole. This is done by adjusting the morphological parameters. With scale parameters Joint optimization is performed to match the human model with the robot model in terms of height, limb length, and torso proportions while keeping the topology unchanged.
[0051] The human-machine morphological alignment problem is solved using an unconstrained optimization method based on gradient descent, and the Adam optimizer is used to optimize the morphological parameters. and scale parameters Joint optimization is performed. The objective function is defined as the geometric error between the human model and the robot model at the matching joints, and its form is:
[0052] in, This indicates that the SMPL-X model has different morphological parameters. The next generated The position of each matching joint in the world coordinate system. This indicates the position of the matched joint relative to the pelvic joint in the robot model. The location of the human pelvic joint in the world coordinate system. The number of joints to be matched. =0.0001 is the regularization weight for the morphological parameters, used to constrain the amplitude of the morphological parameters and prevent unreasonable human body shapes.
[0053] After completing the joint selection and human-robot morphological alignment, it is necessary to further convert the human motion data into a joint motion sequence that satisfies the robot's kinematic constraints. This embodiment employs an inverse kinematics optimization framework to solve for the robot's generalized coordinates, considering both the spatial pose consistency of the human and robot at key joints and joint position constraints during the optimization process. For a given human target pose, the following inverse kinematics optimization problem is constructed:
[0054]
[0055] in, Represents the robot's generalized coordinates, including joint angles. and the position of the root node and posture . This represents the set of joints used in human-machine matching. and and represent the in the SMPL-X human body model, respectively. The position and pose of each matching joint. and This represents the robot joint pose and position obtained through the robot's forward kinematics. The rotation error operator is defined as:
[0056] Weight For the first For the pose error weights of the matched joints, For the first The position error weights for the matched joints are used to balance the importance of the pose and position errors of each matched joint, and their specific values are shown in Table 1. and These are the upper and lower limit constraints for the robot joints.
[0057]
[0058] The inverse kinematics optimization problem is solved using the Mink solver. After solving, a kinematically feasible sequence of robot joint angles with anthropomorphic characteristics is obtained. and the position of the root node and posture .
[0059] In this embodiment, multiple sets of typical human motion data were selected for redirection processing, including forward walking and turning, backward walking, standing to forward walking, and forward walking to running. The redirected robot motion data serves as reference motion data in subsequent anthropomorphic walking and running learning control.
[0060] S2 Actor-critic network structure (1) The Actor-critic network includes the policy network, the value network, and the discriminator network.
[0061] The control policy training is implemented based on an actor-critic reinforcement learning architecture. The policy network outputs joint motion distribution parameters, including the motion mean and learnable motion standard deviation, based on the robot's current observation data, thereby constructing a probability distribution for motion sampling. By sampling specific joint control actions from this distribution, the policy possesses both exploratory and convergent capabilities during training.
[0062] The policy network takes as input preprocessed data the desired robot walking / running speed and outputs joint motions. The desired walking / running speed includes the robot's linear velocity in the x and y directions and its angular velocity around the z-axis. The preprocessed data includes gait phase, robot-feedbacked body angular velocity, Euler angles, joint positions, and joint velocities. The desired walking / running speed and the preprocessed data are used together as observation data. The policy network outputs joint motions. This refers to the desired relative positions of the robot's various joints. Number of active joint degrees of freedom.
[0063] The value network takes as input the robot's linear velocity, foot forces, and observation data during robot simulation control, and outputs the state value of the robot's current state. The value network estimates the value function of the current state, providing benchmark information for policy updates. Its network structure is similar to the policy network, but its output is a single scalar value representing the expected cumulative reward in the current state. By estimating the state value, the variance in the policy gradient estimation process can be effectively reduced, improving the stability of the training process.
[0064] The discriminator is used to determine whether the joint positions and velocities obtained during robot simulation control come from the motion dataset, and to calculate the discriminator loss function accordingly.
[0065] (2) Reward function In this invention, the total reward function is divided into three main categories: task reward, kinematic anthropomorphic reward, and dynamic anthropomorphic reward. Task reward refers to the reward used to ensure the robot's basic control performance during walking and running, including but not limited to linear velocity tracking rewards and angular velocity tracking rewards, calculated based on the robot's feedback motion state. Dynamic anthropomorphic reward refers to constructing anthropomorphic rewards at the center-of-mass dynamics level, making human-robot walking behavior closer to human behavior from a dynamic perspective, reducing the adverse effects caused by differences in human-robot dynamics. Kinematic anthropomorphic reward evaluates the kinematic similarity to human data, constructed through the discriminator's output results. During the data collection phase, the discriminator's input consists of joint positions and joint velocities from the robot's feedback data.
[0066] (21) Anthropomorphic reward function of kinematics The anthropomorphic kinematic reward function is mainly composed of a discriminator network and is used to evaluate the robot's motion style.
[0067] The discriminator network is implemented using a multi-layer feedforward neural network structure. Its input is a concatenated vector of robot joint states from two adjacent frames, where the joint states include the robot's joint positions at the current and next time steps. and joint velocity Information. Using the input format described above, the discriminator network can perceive changes in the robot's joints, thereby discriminating the style of motion segments. The discriminator network consists of several fully connected layers and nonlinear activation functions, and outputs a scalar discriminant value through a linear mapping, which characterizes the similarity between the current motion segment and prior human motion.
[0068] During training, the discriminator network aims to distinguish between data from expert data distributions and data distributions generated by the current policy. Expert data refers to prior human motion data obtained after redirecting human-machine motion data. To improve the stability of adversarial training, this embodiment optimizes the discriminator using a least-squares-based adversarial loss function, defined as:
[0069] in, This indicates the joint position and velocity information for the current and next states. Indicates the distribution of expert data. This represents the distribution of joint data generated by the current strategy. Represents the parameters of the discriminator Find the gradient. This is the gradient regularization constant. The gradient regularization term is used to constrain the gradient norm of the discriminator output with respect to the input, and is generally taken as... This is to prevent the discriminator from overfitting the training data and to improve the stability of adversarial training.
[0070] This invention uses the discriminator output to construct a kinematic anthropomorphic reward, which guides the policy to generate motion that more closely approximates the expert distribution. Specifically, a kinematic anthropomorphic reward function is defined. for:
[0071] in, Indicates in and The purpose of taking the larger value in this formula is to limit the reward to... Within the range.
[0072] When the discriminator classifies the motion segments generated by the policy as closely resembling expert data, the policy receives a significant kinematic anthropomorphic reward; when the discriminator can accurately distinguish the policy-generated data, the kinematic anthropomorphic reward approaches zero. Through this method, the policy update direction is guided towards the prior distribution of human motion.
[0073] (22) Dynamic anthropomorphic reward Maintaining overall motion stability and body coordination is a crucial prerequisite for achieving an anthropomorphic gait in humanoid robots' walking and running movements. Although reference motion data obtained through human motion redirection can recreate the kinematic features of human walking and running quite well, direct imitation of human motion may still produce unreasonable phenomena at the dynamic level due to the fundamental differences between humanoid robots and humans in terms of mass distribution, inertia characteristics, and actuation methods. For example, in the absence of overall dynamic constraints, the robot may maintain its motion by excessively swinging its arms, resulting in a significant increase in the angular momentum of the center of mass during walking or running, thereby affecting the overall motion stability.
[0074] To address this, this invention, based on an adversarial learning control framework, introduces a dynamic anthropomorphic reward function based on the center of mass angular momentum to constrain the robot's walking and running motion from a holistic dynamics perspective. Unlike local methods that only constrain foot contact forces or individual joint movements, center of mass dynamics characterizes the coupling relationships between the robot's rigid bodies at a whole-body level, comprehensively reflecting the influence of upper limb, lower limb, and trunk movements on the overall dynamic behavior.
[0075] Centroidal dynamics describes the linear and angular momentum of a robot system relative to its center of mass (CoM). Its dynamic equations can be expressed as follows:
[0076] in, Let ω be the angular momentum of the center of mass. For the mass center line momentum, Let's say it's the center-of-mass momentum matrix. For the robot's generalized coordinates, Indicates the position of the root node. Indicates the orientation of the root node. Let be the derivative of the generalized coordinates of the robot system with respect to time.
[0077] The rate of change of the center of mass momentum is determined only by the net external force and net external torque acting on the system, and its expression is:
[0078] in, Indicates the first The position vector of each contact point relative to the system's centroid. Indicates the environment in the first The contact force acting on the robot at each contact point The vector of gravitational acceleration. is the gravitational acceleration constant.
[0079] During natural human walking, although the lower limbs continuously generate angular momentum around the center of mass during swinging and support, humans can effectively counteract this excess angular momentum through the coordinated action of trunk posture adjustment and upper limb arm swing, thus maintaining overall stability. Especially in the vertical direction (usually defined as the z-axis), the angular momentum of the center of mass remains within a small range during human walking, thereby avoiding unnecessary twisting or swaying of the body.
[0080] Based on the above analysis, this embodiment introduces a constraint on the vertical component of the center-of-mass angular momentum in the reward function design. Let: To encourage robots to learn coordinated full-body movements similar to humans, this paper designs the following center-of-mass angular momentum reward item.
[0081] in, This is a constant used to adjust the sensitivity of the reward to the magnitude of the angular momentum. =5, The nominal angular momentum is calculated using the following formula:
[0082] in , This represents a 12-dimensional zero row vector.
[0083] Furthermore, to further suppress drastic changes in the center-of-mass angular momentum over time, this embodiment introduces a smoothing reward for the center-of-mass angular momentum in the Z direction, which takes the form of:
[0084] Furthermore, a smoothing bonus for the center-of-mass angular momentum in the XY direction is introduced for the horizontal angular momentum component:
[0085] The anthropomorphic reward of the overall dynamics can be expressed as:
[0086] in, The weight of this reward can be adjusted freely according to the actual situation; in this embodiment, it is taken as... , , .
[0087] During training, the CusADi tool can be used to calculate... Center of mass momentum matrix.
[0088] Through the above-mentioned anthropomorphic reward design of dynamics, when the vertical component of the center of mass angular momentum of the robot approaches the target value during the walking and running process, a large reward can be obtained; when there is a large tendency to rotate around the vertical axis, the reward decreases rapidly, thereby effectively suppressing unnecessary torso twisting and swaying, and prompting the robot to learn a whole-body coordinated walking and running movement pattern similar to that of humans.
[0089] (23) Task Rewards In this embodiment, the anthropomorphic walking and running learning control method for the humanoid robot proposed by the present invention is trained and verified based on the Isaac Gym simulation platform. The training environment is configured on a computing device equipped with an NVIDIA 4090D graphics card. 4096 robots are created in the simulation environment, and parallel simulation is used for reinforcement learning training. The training parameters and task reward functions are shown in Tables 2 and 3.
[0090]
[0091] In Table 3, to unify the expression forms of different physical quantities, an exponential error mapping function is introduced
[0092] where represents the error term, is the corresponding weight coefficient. This function is used to map the error to a bounded reward value to improve the numerical stability of the training process.
[0093] In Table 3, represents the actual linear velocities of the robot body in the x and y directions, represents the desired horizontal linear velocity command; represents the actual angular velocity of the robot around the vertical axis, is the corresponding desired angular velocity command. represents the Euler angles of the robot body in the roll and pitch directions; represents the height of the body in the vertical direction, is the reference body height. and respectively represent the actual foot-end contact state and the desired foot-end contact state at time , and their values are 0 or 1. represents the linear velocity of the foot-end in the horizontal plane and is used to characterize the degree of slip during the support phase. and respectively represent the joint angle and joint angular velocity of the joint, represents the reference joint angle, which is consistent with the default joint angle in this embodiment; distinguishes between the standing and walking states. When , it represents standing, , it represents walking. represents the joint output torque. , and respectively represent the policy output actions of the current and the previous two control cycles and are used to construct an action smoothness penalty term. respectively represent the contact forces in the z direction of the left and right foot-ends.
[0094] The formula for calculating task rewards is as follows:
[0095] In the formula, The set of all rewards in Table 3 For the first Item Reward The corresponding weighting coefficients.
[0096] Figure 1 The motion state used to calculate the reward is specifically the robot's actual linear velocity. angular velocity about the z-axis Euler angles of the robot body in the roll and pitch directions The height of the fuselage in the vertical direction Actual foot contact state Joint position Joint speed Joint torque , .
[0097] (24) Total reward function The kinematic anthropomorphic reward, task reward, and dynamic anthropomorphic reward are weighted and fused to form the total reward function for strategy optimization, which has the following form.
[0098]
[0099] in, To determine the weight of the kinematic anthropomorphic reward, the proportion of the kinematic anthropomorphic reward is controlled. In this embodiment, the weight is set as follows: . Anthropomorphic reward based on the dynamics of the center of mass angular momentum. The task rewards are used to ensure the robot's basic control performance during walking and running, including but not limited to linear velocity tracking rewards, angular velocity tracking rewards, posture stability rewards, body height maintenance rewards, foot contact matching rewards, foot slippage penalties, standing maintenance rewards, and motion smoothness penalties.
[0100] (3) Loss function Because the motion samples obtained by retargeting human-machine motion data may have an uneven distribution in the left and right directions, such as more samples supporting the left foot than the right foot, or samples biased towards starting or turning on one side in some speed ranges, if the discriminator network and policy network are directly used to train the discriminator network and policy network, it is easy to cause inconsistent response weights of the policy on the left and right legs or left and right arms, which can lead to problems such as left and right gait asymmetry, body rotation bias, or unilateral force habit.
[0101] To address this, this embodiment proposes a data-network dual-symmetry mechanism to suppress left-right asymmetry at both the data layer and network layer levels, thereby improving the gait symmetry and overall motion stability learned by the policy.
[0102] At the data layer, based on the left-right symmetry between the human body model and the humanoid robot structure, a mirror dataset is constructed from the original human motion data. Specifically, by replacing the corresponding joints on the left and right sides of the human body or robot and performing sign transformations on the relevant posture parameters, joint angles, and velocity parameters, the original motion samples are mapped to their left-right mirror forms, thus obtaining mirror samples that are equivalent in a dynamic sense but in opposite left-right directions, such as... Figure 5 As shown, the original motion samples and their corresponding mirror images together constitute the expanded expert dataset, which is used to train the discriminator network so that the discriminator is not biased towards the left and right directions when learning human motion styles.
[0103] At the network layer, mirror consistency constraints are applied to both the policy network output and the discriminator network to further suppress learning bias in the left and right directions.
[0104] Specifically, in the policy network, the input observations Perform a left-right mirror transformation to obtain a mirrored observation. and will and The corresponding joint motion outputs are obtained by inputting the inputs into the policy network. and Subsequently, the mirror joint was moved. Apply an inverse mirror transformation to map it back to the original coordinate system to obtain the inverse mirror joint motion. By comparing the original joint movements Reverse mirror joint movement Construct the mirror consistency loss function:
[0105] The loss term is added as a regularization term to the optimization objective of the policy network to constrain the policy to output consistent control behavior under left-right symmetrical input conditions.
[0106] Simultaneously, during the adversarial prior learning process, a mirror consistency constraint is introduced into the discriminator network. Specifically, the original state transition pair... and their corresponding mirror state transition pairs Simultaneously inputting into the discriminator network, respectively, we obtain discriminative outputs. and And construct the symmetric loss function of the discriminator:
[0107] By minimizing the aforementioned loss, the discriminator can provide consistent discrimination results on left-right symmetrical motion segments, thereby avoiding bias in the left-right direction during the training process.
[0108] During training, the overall loss function of the discriminator network consists of the adversarial discriminative loss and the mirror consistency loss, which can be expressed as:
[0109] in, For the discriminative loss term based on adversarial motion priors, The coefficient used to balance the weights of mirror consistency constraints is generally taken as... .
[0110] Accordingly, the loss function of the network training module introduces a mirror consistency regularization term in addition to the original task reward, dynamic anthropomorphic reward, and adversarial style reward. Its loss form can be expressed as:
[0111] in, Represents the PPO loss function. The weighting coefficients for the policy mirror loss are generally taken as... .
[0112] By combining the data layer and network layer into a dual-symmetric mechanism, the distribution balance of training data in the left and right directions is enhanced on the one hand, and the output consistency of the policy network and discriminator network is constrained under the condition of left and right symmetry on the other hand. This effectively improves the problem of left and right gait asymmetry in the walking and running process of humanoid robots, reduces unilateral bias behavior, and improves the stability and anthropomorphic coordination of the overall movement.
[0113] Training the S3Actor-critic network (1) The preset expected walking speed of the robot is preprocessed, and the preprocessed data is input into the policy network of the Actor-critic network as observation data. The joint motions that control the robot's movement are output. The joint motions are converted into joint driving torques that control the robot using the controller. The robot is then simulated and controlled using these joint driving torques.
[0114] (11) In human walking and running, the gait structure exhibits continuous evolution characteristics with movement speed, mainly reflected in the smooth adjustment of the gait cycle length and the proportion of the support phase with speed: in low-speed walking, the gait cycle is relatively long and there is a clear double support phase; as the movement speed increases, the gait cycle gradually shortens and the duration of the double support phase gradually decreases; in running, a flight phase occurs during the switching between the left and right feet. In order to characterize the above continuous change law in the reinforcement learning control framework, this invention constructs a unified gait phase model based on speed regulation to generate a continuous anthropomorphic gait sequence suitable for walking and running.
[0115] In this embodiment, let the robot's desired speed be... The gait cycle is adjusted according to changes in speed. Defined as:
[0116] in, This indicates the duration of a single gait cycle. , indicating that Variables are restricted to a range According to the above definition, the gait period can be continuously shortened as the desired speed increases.
[0117] Based on the gait cycle, a normalized gait phase variable is introduced. This is used to describe the robot's relative position within a complete gait cycle at the current moment. The gait phase is updated incrementally over time, and its update rule is as follows:
[0118] in, This definition indicates the policy update time, and guarantees that the phase variable updates periodically. It changes continuously and automatically returns to the starting position after completing one cycle.
[0119] To avoid non-physically meaningful gait changes when the robot is stationary or nearly stationary, this embodiment introduces a stationary determination condition. This condition is determined when the robot's desired velocity L2 norm... hour, Given a threshold, it is generally taken as... The phase remains unchanged, that is
[0120] To map gait phase to the alternating support relationship of the left and right feet, a phase signal is defined:
[0121] Without considering gait structure adjustments, the basic support functions of the left and right feet are defined as follows:
[0122] in, This is an indicator function; it returns 1 if the condition within the parentheses is true, and 0 otherwise. =1 indicates a supporting phase. =0 indicates a oscillating phase, when the phase signal... During the change, the supporting foot alternates between left and right, thus forming a periodic single-support gait.
[0123] To characterize the differences in support structures between walking and running, this embodiment further introduces a method based on gait cycles. The support phase adjustment mechanism continuously corrects the support relationship during the left and right foot switching phase. First, the width function of the switching interval is defined:
[0124] And based on this, the switching interval is defined:
[0125] When the gait cycle is long (low-speed walking), switch zones. Introducing simultaneous support from both feet to simulate the dual support phase in human walking, when... hour:
[0126] When the gait cycle is short (fast walking or running), switch zones. The internal support is removed, allowing both feet to leave the ground simultaneously during the transition phase, creating a flight phase. hour:
[0127] like Figure 4 As shown, the above-mentioned gait phase model based on speed regulation can realize the continuous change of gait phase and duty cycle of walking and running under a unified phase description framework. The gait phase, as part of the observation data, and the support state can be used to construct the foot-lifting reward in the task reward function, enabling humanoid robots to achieve smooth, stable and anthropomorphic walking and running movements under different speed conditions.
[0128] (12) Controller time joint movements Converted into joint drive torque via a low-level proportional-derivative (PD) controller The humanoid robot is driven to complete walking and running movements in a physical simulation environment. The controller uses a PD controller, and the formula for the PD controller is:
[0129] in, and These are the proportional coefficient vector and differential coefficient vector of each joint, respectively. This indicates the robot's default joint angle configuration. This refers to the joint position. This refers to the joint velocity.
[0130] (2) Obtain the real-time motion state, joint position, and velocity during the robot simulation control process and calculate the total reward function of the Actor-critic network accordingly. Specifically, the motion state includes the robot's actual linear velocity, angular velocity around the z-axis, Euler angles of the robot's body in the roll and pitch directions, height of the body in the vertical direction, actual foot contact state, and joint torque. The linear velocity of the robot body and the force at the feet are acquired during the robot simulation control process. This linear velocity and foot force, along with the observed data, are input into the value network of the Actor-critic network to obtain the state value of the robot's current state. The discriminator determines whether the joint positions and velocities obtained during the robot simulation control process come from the motion dataset and calculates the discriminator loss function accordingly.
[0131] The total loss function of the Actor-critic network is calculated using the total reward function, state value, and discriminator loss function. The policy network is then adjusted to minimize the total loss function, thereby enabling the training of the Actor-critic network.
[0132] Reinforcement learning training is conducted in a physical simulation environment. The simulation system supports the parallel operation of multiple robot instances, each corresponding to a complete humanoid robot model to improve sample collection efficiency. The robot model is described based on the URDF format and includes the geometry, mass distribution, joint configuration, and collision detection information of each rigid body of the robot. Simultaneously, reasonable constraints are set on joint positions, velocities, and output torques to ensure stability and safety during the simulation training process.
[0133] At the start of each training round, the robot's initial state is randomized. This includes applying small-scale random perturbations to the robot's initial position, posture, and velocity, allowing the robot to train under different initial conditions, thereby improving the robustness and generalization ability of the policy.
[0134] The training process alternates between a data collection phase and a network update phase. In the data collection phase, the current policy interacts with the simulation environment, collecting trajectory information such as the robot's state, actions, and rewards during its movement. The generalized advantage estimation (PPO) method is used to calculate the advantage function at each time step to balance the relationship between bias and variance. In the network update phase, the policy network is updated based on the PPO, incorporating the discriminator loss term. and policy-symmetric loss term The relevant network parameters are jointly optimized. Specifically, a discriminator loss term is used to train the discriminator network to distinguish between robot-generated motion data and human reference motion data; a policy symmetry loss term is used to constrain the policy network to output consistent control actions under left-right mirror input conditions, thereby reducing left-right asymmetry during motion. Simultaneously, the value network is trained by minimizing the state value estimation error to improve the accuracy of predicting future rewards.
[0135] During the training phase, the parameters of the policy network, value network, and discriminator network are continuously optimized to gradually reduce the overall loss function, thereby obtaining a control policy with better performance. The training phase is further divided into a data collection phase and a network update phase.
[0136] During the data collection phase, velocity commands are randomly sampled and input into the strategy network based on the current observation data to generate joint movements. These movements are then driven by the PD controller to propel the robot through the simulation environment. Simultaneously, after each joint movement, the values of each reward function are calculated based on data feedback from the simulation environment. The observation data, privileged observations, and reward values for each control step are stored as training data for the network training module in subsequent network update phases. Privileged observations refer to environmental information that can be directly obtained from the simulation environment during training but cannot be directly measured or is difficult to obtain accurately during actual robot operation, such as contact information and terrain information.
[0137] After the data collection phase is completed, the network update phase begins. The network training module utilizes the observation data, privileged observations, joint movements, and reward samples stored during the data collection phase, combined with the state value output by the value network, to update the parameters of the policy network, value network, and discriminator network, thereby gradually reducing the loss function. In this invention, the policy network and value network are updated using the Proximal Policy Optimization (PPO) algorithm commonly used in legged robots, continuously optimizing the policy towards obtaining higher rewards. The value network is trained by minimizing the state value estimation error to improve the accuracy of predicting future rewards. The discriminator network's training data comes from two parts: one part is the motion data generated by the robot during the data collection phase, and the other part is reference motion data from real human motion datasets. The discriminator learns to distinguish between these two types of data using their true labels.
[0138] Furthermore, a symmetry loss term is introduced during the network update phase to constrain the consistency of the policy network and discriminator network's responses to left and right movements. This suppresses potential left-right asymmetry issues during robot movement, improves the gait symmetry learned by the policy, and achieves joint optimization of the policy network, value network, and discriminator by minimizing the total loss function.
[0139] After updating the parameters of the policy network, value network, and discriminator network, the new network parameters are used in the next data collection phase. By repeatedly performing the "data collection - network update" training cycle, the policy network gradually learns to stably complete walking and running tasks under different speed commands, while exhibiting movement characteristics similar to humans at the kinematic and dynamic levels. When the training process converges, the final control policy is obtained.
[0140] S4 Validation To verify the effectiveness of each technical module in this invention, multiple comparative implementation schemes were constructed under the same simulation environment and basic training configuration for verification. Specifically, these include: In the first comparative scheme, only the basic reinforcement learning reward function is used to train the humanoid robot. It does not introduce adversarial motion priors, does not use symmetric mechanisms, and does not include dynamic anthropomorphic rewards (Baseline). The second contrast scheme introduces adversarial motion priors based on the first contrast scheme, and constrains the motion style generated by the strategy (G) through a discriminator. The third comparison scheme further introduces a data-network dual-symmetry mechanism based on the second comparison scheme to alleviate the problem of uneven distribution of expert motion data in the left and right directions (GS). The solution in this embodiment, based on the third comparative solution, introduces a dynamic anthropomorphic reward function based on the center of mass angular momentum to constrain the robot's walking and running motion from a global dynamics perspective (GSM). All the above solutions were uniformly evaluated under the same test conditions after training convergence.
[0141] After training, the trained control strategy is deployed to a simulation environment or a real robot to achieve humanoid walking and running control. The deployment and training inference phases are similar. First, the user inputs a speed command, and the gait planning module generates gait phases based on this command. The generated gait phases, speed commands, and the robot's angular velocity, Euler angles, joint positions, and joint velocities fed back from the simulation environment are used as observation data and input into the policy network. The policy network outputs joint movements, which are converted into torques for each joint by the PD controller and input to the simulation environment or the real robot to enable robot movement.
[0142] In this embodiment, the trained strategy is tested in the Mujoco simulation environment. First, continuous speed command testing is performed on the strategy trained using the method of this invention to analyze its motion morphology changes at different speed stages. During the test, the robot starts from a standstill, gradually accelerates to 2 m / s, and then decelerates to a stop. The test results show that during this continuous speed change, the robot can smoothly switch between standing, walking, and running states within a single strategy framework, and the motion morphology exhibits an evolutionary pattern similar to that of human walking and running, such as… Figure 6 and 7 As shown, in the low-speed phase, the robot maintains a stable standing or low-speed walking posture with its arms swinging naturally. As the speed command increases, the robot gradually increases its step frequency and adjusts its upper body posture. In the high-speed phase, the robot enters a stable running state, exhibiting obvious leg lifting movements and coordinated arm swinging behavior. Upon receiving a stop command, the robot can smoothly decelerate through body posture adjustment and coordinated arm swinging, eventually returning to a standing state. Furthermore, the speed tracking performance of different implementation schemes under 0–2 m / sx directional speed commands is compared. The results show that all schemes can complete the speed tracking task well, indicating that the introduction of adversarial motion priors, symmetry mechanisms, and anthropomorphic dynamic rewards does not weaken the robot's basic response capability to desired speed commands.
[0143] During the steady-state walking phase, the changes in the robot's center of mass angular momentum around the vertical axis are compared and analyzed. For example... Figure 8As shown, the scheme without the introduction of a dynamic anthropomorphic reward generates significant fluctuations in the vertical axis centroid angular momentum during walking. Introducing an adversarial motion prior reduces the amplitude of the centroid angular momentum. Furthermore, by introducing a dynamic reward based on the centroid angular momentum, the peak value and fluctuation amplitude of the centroid angular momentum are further reduced. These results demonstrate that the dynamic anthropomorphic reward proposed in this invention can effectively suppress unnecessary rotational tendencies from an overall dynamic perspective, based on the generation of anthropomorphic motion patterns.
[0144] According to the principles of center-of-mass dynamics, a larger angular momentum at the center of mass usually means that the foot needs to provide a greater frictional torque to maintain motion stability, thus increasing the risk of foot slippage. Under low-speed walking conditions, a comparative analysis of the vertical axis frictional torque of the foot in various scenarios is conducted, such as... Figure 9 The figure shows the foot Z-axis torque under the x-direction velocity command of 0–2 m / s, where (a) is the low-speed walking torque. The curve comparing the z-axis torque at the foot end is shown in (b), which represents the torque at medium speed. The curve comparing the z-axis torque at the foot end is shown in (c), which represents high-speed running. Comparison curve of z-axis torque at the foot end. Figure 9 The results show that the foot friction torque required by the method of the present invention is significantly reduced at the corresponding speed stages, indicating that by reducing the vertical axis center of mass angular momentum, the robot's dependence on ground friction can be effectively reduced, thereby improving motion stability.
[0145] also, Figure 10 These are energy consumption comparison curves at different speeds in a specific example provided by the present invention, wherein (a) is the energy consumption curve at low speed. Energy consumption comparison curves, (b) is for medium-speed walking. Energy consumption comparison curves, (c) is for high-speed running Energy consumption comparison curve. For example... Figure 10 As shown in Table 4, the analysis of energy consumption at different speed stages reveals that, in the low, medium, and high speed stages, the scheme using the method of this invention reduces both peak power and average positive power compared to the GS group while maintaining stable motion, demonstrating more reasonable dynamic behavior while maintaining anthropomorphic motion style.
[0146] Table 4. Average positive power (in W) at different speeds
[0147] To verify the effectiveness of the data-network bisymmetric mechanism, a comparative analysis was conducted on schemes that introduced and did not introduce the mechanism. Figure 11These are comparison curves showing whether or not a symmetry mechanism is incorporated in a specific example provided by this invention. (a) represents the hip joint pitch angles of the left and right legs when a symmetry mechanism is incorporated, and (b) represents the hip joint pitch angles of the left and right legs when a symmetry mechanism is not incorporated. Figure 11 As shown, observe the movement trajectory of the hip joint pitch angle of the left and right legs. The results show that in Figure 11 In (a) without the introduction of a symmetry mechanism, the robot is prone to asymmetric movements of its left and right limbs during walking; while Figure 11 In (b), after introducing the data-network bisymmetric mechanism, the movement trajectories of the left and right leg joints remained basically symmetrical, and gait stability was significantly improved. This result shows that the bisymmetric mechanism proposed in this invention can effectively alleviate the strategy bias problem caused by the uneven distribution of expert data.
[0148] This invention employs a combination of human motion data redirection, reinforcement learning based on adversarial motion priors, anthropomorphic rewards based on center of mass angular momentum, unified gait phase based on velocity regulation, and a data-network structure dual-symmetric mechanism to address the technical problem in existing technologies where humanoid robot walking and running control struggles to simultaneously achieve anthropomorphic motion patterns, overall dynamic rationality, gait continuity, and left-right symmetry. Specifically, firstly, by establishing joint correspondences between the human model and the robot model, and using human motion capture data for human-robot motion redirection, a robot reference motion dataset with anthropomorphic characteristics is constructed, providing human motion priors for control policy learning. Secondly, a discriminator network is introduced during reinforcement learning training, guiding the policy to generate control actions that gradually approximate the distribution of human motion data through adversarial motion prior learning, thereby improving the anthropomorphism of the robot's motion morphology. Thirdly, a dynamic anthropomorphic reward based on the center of mass angular momentum is introduced into the reward function to constrain the robot's walking and running processes from the overall dynamics level, reducing unreasonable motion phenomena caused by differences in human-robot dynamics, improving motion stability and energy efficiency, and reducing the risk of slippage. In addition, by constructing a unified gait phase model based on speed regulation, the robot can achieve a continuous and smooth transition from standing, walking to running under the same control framework. At the same time, by constructing a mirror dataset and introducing a data-network bisymmetric mechanism in the policy network and discriminator network, the policy bias problem caused by uneven left-right distribution of training data is alleviated, improving the left-right symmetry of gait and overall coordination.
[0149] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.
Claims
1. A training method for an Actor-critic network for anthropomorphic walking and running learning control of humanoid robots, characterized in that, The method includes the following steps: The preset expected walking and running speeds of the robot are preprocessed, and the preprocessed data is input into the policy network of the Actor-critic network as observation data. The output is the joint motion that controls the robot's movement. The controller converts the joint motion into the joint driving torque that controls the robot, and uses the joint driving torque to simulate and control the robot. The robot's real-time motion state, joint position, and velocity during simulation control are obtained, and the total reward function of the Actor-critic network is calculated accordingly. The linear velocity of the robot body and the force at the foot end are obtained during the robot simulation control process. The linear velocity and the force at the foot end are input into the value network of the Actor-critic network along with the observed data to obtain the state value of the robot's current state. The discriminator determines whether the joint positions and velocities obtained during robot simulation control come from the motion dataset of a pre-built robot model, and calculates the discriminator loss function accordingly. The total loss function of the Actor-critic network is calculated using the total reward function, state value, and discriminator loss function. The policy network is then adjusted to minimize the total loss function, thereby enabling the training of the Actor-critic network.
2. The training method of the Actor-critic network for anthropomorphic walking and running learning control of a humanoid robot as described in claim 1, characterized in that, The total reward function includes kinematic anthropomorphic reward, dynamic anthropomorphic reward, and task reward, and its formula is as follows: in, Weighting for anthropomorphic kinematic rewards. From state to state Total reward at the time, , As a reward for the task, For the anthropomorphic reward of dynamics, , These are the center-of-mass angular momentum rewards in the anthropomorphic reward system of dynamics. Z-direction center of mass angular momentum smoothing reward XY direction centroid angular momentum smoothing reward The weighting coefficients.
3. The training method of the Actor-critic network for anthropomorphic walking and running learning control of a humanoid robot as described in claim 2, characterized in that, The kinematic anthropomorphic reward The formula for task rewards is as follows: : in, For each sub-reward in the task rewards The set, index For set The indices of the neutron reward items correspond to the sets in sequence. Each sub-reward, For the first Item Reward The corresponding weighting coefficients, Indicates in and Take the larger value. Discriminator pair The judgment result.
4. The training method of the Actor-critic network for anthropomorphic walking and running learning control of a humanoid robot as described in claim 1, characterized in that, The formula for the total loss function is as follows: in, Represents the PPO loss function. The weighting coefficients are the mirror loss coefficients of the strategy. Let be the mirror consistency loss function of the policy network. The overall loss function of the discriminator network, For the discriminative loss term based on adversarial motion priors, The coefficients used to balance the weights of mirror consistency constraints. For the discriminator's mirror consistency loss, Discriminator pair The judgment result, For the discriminator pair The judgment result, and They are respectively The mirror state.
5. The training method of the Actor-critic network for anthropomorphic walking and running learning control of a humanoid robot as described in claim 1, characterized in that, The pre-built motion dataset of the robot model is constructed as follows: a correspondence is established between the human model and the robot model, and the motion vectors of the human model are converted into robot joint motion vectors using this correspondence, thereby obtaining the motion dataset of robot joint motion vectors.
6. The training method of the Actor-critic network for anthropomorphic walking and running learning control of a humanoid robot as described in claim 5, characterized in that, The correspondence between the human body model and the robot model is established by minimizing the geometric error between the human body model and the robot model at the matching joints, as shown in the following formula: in, It refers to the geometric error at the matching joints between the human body model and the robot model. For the first generation generated by the SMPL-X model The position of each matching joint in the world coordinate system. These are the morphological parameters of the SMPL-X model, controlling the shape of various parts of the human body within the SMPL-X model. To match the position of joints relative to the pelvic joints in the robot model. The location of the human pelvic joint in the world coordinate system. The number of joints to be matched. These are regularization weights for morphological parameters, used to constrain the amplitude of morphological parameters and prevent unreasonable human body shapes. This is a global scale parameter that controls the overall scaling ratio of the SMPL-X human body model.
7. A training method for an Actor-critic network for anthropomorphic walking and running learning control of a humanoid robot as described in claim 5 or 6, characterized in that, The formula for converting the motion vectors of the human body model into the motion vectors of the robot joints is as follows: in, For the robot's generalized coordinates, including joint angles and the position of the root node and posture , A set of joints for human-machine matching. It is a three-dimensional real vector space. and and represent the in the SMPL-X human body model, respectively. The position and pose of each matching joint. and The weights represent the robot joint positions and orientations obtained through the robot's forward kinematics. For the first For the pose error weights of the matched joints, For the first Weights for the positional errors of the matched joints, and These are the upper and lower limits of the robot's joint angles, respectively.
8. The training method of the Actor-critic network for anthropomorphic walking and running learning control of a humanoid robot as described in claim 1, characterized in that, The controller is a PD controller, and the control formula of the PD controller is as follows: in, and These are the proportional coefficient vector and differential coefficient vector of each joint, respectively. This indicates the robot's default joint angle configuration. This refers to the joint position. For joint velocity, It is an n-dimensional real vector space. It's a joint movement. for The joint driving torque at any given moment.
9. A control method for humanoid robot walking and running, characterized in that, The control method uses an Actor-critic network trained by the training method described in any one of claims 1-8 to control a humanoid robot, and the method includes the following steps: The desired speed of the robot is preprocessed, and the preprocessed data is input into the policy network in the Actor-critic network to output the joint movements of the robot. The controller converts the joint movements into joint driving torques to control the robot, and uses these joint driving torques to control the humanoid robot.
10. A humanoid robot, characterized in that, The humanoid robot is controlled using the control method described in claim 9.