A circadian rhythm-based humanoid robot walk-run unified control method and system thereof

Abstract

The application discloses a kind of based on biological rhythm humanoid robot walk run unified control method and system, belong to humanoid robot technical field.Capture human walking and running action data, extract rhythm characteristics from frequency and phase time domain;Based on the rhythm characteristics extracted in step 1, the construction of rhythm generator is carried out;Based on the strategy gradient method of constraint reinforcement learning algorithm, walk run unified control strategy is constructed, walk run unified control strategy uses the rhythm information generated based on rhythm generator, and then is trained using actor-critic, which jointly constructs the bionic motion control system Walk2Run under the drive of speed, realizes natural gait transition and frequency regulation, while meeting the physical ability limit of robot, improves motion fluency and energy efficiency.The existing humanoid robot walking of the present application has not reached the problem of "energy optimization" of human walking, and the migration of existing data set requires higher robot motion ability, and the generalization ability is weak.

Description

Technical Field

[0001] This invention belongs to the field of humanoid robot technology, specifically relating to a unified control method and system for walking and running of humanoid robots based on biological rhythms. Background Technology

[0002] Humanoid robots, possessing human-like structure and mobility, have shown great potential in service, healthcare, and disaster response scenarios. Achieving stable and efficient dynamic motion control, especially in rhythmic movements such as walking and running, remains a key challenge.

[0003] Unlike humans who rely on a Central Pattern Generator (CPG) to achieve natural gait transitions, current mainstream control methods (such as Model Predictive Control (MPC) and Reinforcement Learning (RL) policies) focus primarily on trajectory tracking or motion imitation, lacking the modeling and utilization of rhythmic structures. This deficiency makes it difficult for robots to achieve natural gait transitions during speed changes or task switching, resulting in low motion efficiency, heavy parameter tuning burden, and poor adaptability.

[0004] While recent adversarial imitation methods (such as Adversarial Motion Prior, AMP) have made progress in improving the naturalness of movements, their learning objectives primarily focus on "postural similarity," neglecting the structural role of rhythmic signals in regulating cyclical motion. Therefore, these methods still have significant limitations in achieving speed adaptation, motion mode switching, and long-term energy-efficient control. Furthermore, although humanoid robots have become morphologically similar to humans, structural limitations such as joint degrees of freedom and range of motion often make it difficult to achieve optimal control strategies by directly imitating human trajectories. This results in weak transferability of humanoid motion and a high dependence on hyperparameter tuning. This further amplifies the limitations of current walking and running control methods in achieving natural, rhythmic control.

[0005] Current humanoid robot motion control methods can be mainly divided into two categories: dynamic model-based control methods and reinforcement learning-based methods. The former, represented by classic ZMP, MPC, and whole-body control, achieves analytical control of the robot's state through physical modeling and optimization calculations. These methods are effective in improving the robot's dynamic stability and trajectory controllability. However, due to the dual requirements of model accuracy and real-time performance, research necessitates the design of simplified models to achieve control of the humanoid robot. The linear inverted pendulum (LIPM) technique can be used to calculate foot position. A fusion optimization approach, utilizing spring inverted pendulum models, single-rigid-body dynamics, and center-of-mass dynamics, calculates foot position and contact forces, achieving better walking control. However, these model-based control strategies, which rely purely on simplified dynamics, are inherently limited by their simplification and model mismatch, leading to conservative motion control and failing to fully utilize the robot's capabilities.

[0006] In contrast, reinforcement learning offers a new paradigm for humanoid robot control policies. In recent years, deep reinforcement learning has made significant progress in controlling the motion of quadrupedal robots, bipedal robots, and humanoid robots. These methods, through an end-to-end training process, eliminate the heavy reliance on models. However, the transfer from simulation to reality remains a major challenge in reinforcement learning control. Mainstream solutions include using high-fidelity physical simulators, introducing domain randomization techniques, and employing teacher-student policy distillation frameworks to improve the generalization ability of policies. Furthermore, some research attempts to utilize observation history to construct latent dynamic representations to enhance the state representation capabilities of policies. Currently, most reinforcement learning-based humanoid robot control methods employ reward-driven PPO algorithms. Although these methods have demonstrated effectiveness on various platforms, two main problems remain: first, most reinforcement learning methods use reward-based optimization frameworks, neglecting safety and physical constraints in the control process; second, during policy transfer, conflicts may arise between the robot's actual motion capabilities and training priors, thus affecting policy stability.

[0007] Constraint-based reinforcement learning methods offer potential solutions to the aforementioned problems. These methods are based on the Restricted Markov Decision Process (CMDP) framework, and researchers have proposed various policy optimization methods, which can be categorized into Lagrange multipliers and trust domain methods. Lagrange multipliers transform the constrained problem into an unconstrained one by introducing Lagrange multipliers and then solving it using a primordial-dual approach. Some methods update the policy by linearly approximating the objective function within the trust domain, thus resolving the constraints. This subfield encompasses various algorithms, which can be further subdivided into first-order and higher-order methods. Liu et al. proposed Intrinsic Point Policy Optimization (IPO), which utilizes a logarithmic barrier function to optimize the constraint function, thereby optimizing the policy within the constrained region. IPO-based methods have enabled constraint-based motion control of quadruped robots, significantly reducing the workload of parameter tuning and addressing the problem of unreasonable policy output, thus improving control safety and transferability. However, in the field of humanoid robot control with limited cognitive capacity, constraint-based reinforcement learning methods have not yet been systematically introduced. In biomimetic motion tasks, the inconsistency between prior objectives derived from human motion data and the robot's motion capabilities is particularly prominent. It is believed that integrating biomimetic motion priors with constraint-based strategy optimization methods will bring more stable, efficient and natural control strategies to humanoid robots.

[0008] Biomimetic control methods attempt to draw inspiration from biological movement to improve the naturalness of legged robots in gait rhythm and movement style. These methods can be broadly divided into two categories: control methods based on bio-inspired models and policy training based on imitation learning.

[0009] With the development of motor neuroscience, the motor control mechanisms of biological nervous systems have gradually been introduced into the field of robotics. Research has found that the Central Pattern Generator (CPG), present in the animal nervous system, is considered a key control structure for generating repetitive movements (such as walking and swimming), driving rhythmic movements through internal neural oscillators with imperceptible input or simplified sensory feedback. In legged robots, early research primarily borrowed CPG structures to regulate gait rhythms and improve walking stability and movement continuity. However, the CPG is essentially an abstraction of a complex biological system, and its structural modeling, regulatory mechanisms, and coupling methods with sensory feedback still lack unified standards, limiting its widespread application in robotic systems.

[0010] Another approach relies on large-scale human motion capture datasets, extracting human-like movement patterns through imitation learning or adversarial learning. Imitation learning methods are generally divided into operational imitation and performance imitation. Operational imitation focuses on the precise control of robotic arms and wheeled platforms, often based on teleoperation or demonstration trajectory training; while performance imitation focuses on reproducing human-like behavior, requiring highly coordinated dynamic control capabilities. The latter has achieved some success in simulation environments, but transferring it to real humanoid robots remains challenging. Some work has indeed introduced imitation learning into legged robots, but the imitation area is only the upper body, while the lower limbs rely on reinforcement learning to maintain stability.

[0011] Among adversarial simulation methods, the Adversarial Motion Prior (AMP) approach has attracted the most attention. Based on the GAIL framework, this method learns complex motion styles by determining whether state transitions originate from real data. AMP does not rely on explicit reference trajectories or reward functions, allowing the policy to autonomously extract high-level motion features from unstructured motion data. The Adversarial Motion Prior (AMP) method uses the GAIL framework to learn implicit motion styles from state transition distributions. Subsequent work extends AMP to the latent space to learn large-scale, reusable adversarial skill embeddings, making it suitable for physically simulated characters.

[0012] This technical approach has demonstrated excellent performance in the field of quadruped robots. Existing technologies use AMP to learn quadruped robot motion on flat ground without requiring complex reward functions. This approach can not only transfer human styles to the policy network, but also transfer model-controlled trajectory styles to the policy network, thus validating the effectiveness of AMP in learning robust motion on complex terrain.

[0013] While AMP can effectively capture human-like movement styles, its core focus remains primarily on the distributional similarity of state transitions, lacking explicit modeling of the rhythmic structure of movement. Rhythm, as a key characteristic of periodic movements (such as walking and running), is crucial for enhancing the naturalness and stability of control strategies. Summary of the Invention

[0014] This invention provides a unified control method and system for walking and running of humanoid robots based on biological rhythms. It addresses the problems of existing humanoid robot walking not achieving the "energy optimization" of human walking, as well as the high requirements for robot motion capabilities and weak generalization ability of existing datasets.

[0015] This invention is achieved through the following technical solution: A unified control method for walking and running of a humanoid robot based on biological rhythms, the method comprising the following steps: Step 1: Capture human walking and running motion data, and extract rhythmic features from the frequency and phase time domains; Step 2: Based on the rhythm features extracted in Step 1, construct a rhythm generator; Step 3: Construct a unified control policy for walking and running based on the policy gradient method of constrained reinforcement learning algorithm; Step 4: The unified walking and running control strategy in Step 3 uses rhythm information generated by the rhythm generator in Step 2, and then trains it with actor-critic to jointly construct the speed-driven bionic motion control system Walk2Run, which realizes natural gait transition and frequency adjustment, while meeting the physical limitations of the robot and improving motion smoothness and energy efficiency.

[0016] Furthermore, step 1 specifically involves extracting rhythmic features from both frequency and phase time domain perspectives based on human walking and running motion capture data, and mapping them to the robot joint space to provide structured priors for subsequent rhythmic perception and control. The motion trajectory of key joints and pelvis of the human lower limbs is obtained by a high-precision motion capture system, and the rhythmic signals are extracted by Fourier analysis. Then, these rhythmic features are mapped to the robot skeleton space to construct a "bionic rhythmic mapping".

[0017] Furthermore, step 2 specifically involves dynamically generating a rhythmic signal with a matching frequency based on the robot's current motion state to drive the generation of periodic gait. Using the center-of-mass velocity as input and human-like rhythms as labels, a rhythm generator was trained, enabling the robot to autonomously generate gait patterns that match those of humans in various motion states.

[0018] Furthermore, the constraint reinforcement learning algorithm in step 3 specifically involves introducing a smoothing barrier function to improve the constraint reinforcement learning algorithm to meet the value range restrictions and enhance the stability and reliability of the policy training process.

[0019] Furthermore, in step 4, the actor-critic specifically refers to a training framework based on the Actor-Critic architecture, where the Critic network is further divided into two sub-modules: one sub-module is used to evaluate the overall reward, and the other sub-module is used to model multiple cost terms to describe the degree of violation of robot position, velocity, and torque limits. Walk2Run in step 4 is used to achieve natural gait transition and frequency adjustment, while meeting the physical limitations of the robot and improving motion smoothness and energy efficiency.

[0020] Furthermore, the improved constrained reinforcement learning algorithm specifically involves formalizing the control problem as a constrained Markov decision process and combining it with an improved barrier function structure to achieve smooth control of the constraint boundary during the training process.

[0021] Furthermore, to guide the unified control strategy of walking and running to generate human-style motion trajectories, a GAN-based discriminator network is introduced. Using human joint trajectories as a reference, the network evaluates the degree of human-likeness of the robot's current movement and uses the discrimination result as an additional reward signal to encourage the strategy to generate more natural motion behavior.

[0022] Furthermore, the unified walking and running control strategy uses rhythm information as an auxiliary input and dynamically adjusts the movement frequency in combination with the current center of mass motion state, enabling the robot to adaptively switch between walking and running modes according to the task objective and body state.

[0023] A unified control system for walking and running of a humanoid robot based on biorhythms, the system using the aforementioned unified control method for walking and running of a humanoid robot based on biorhythms, the system comprising: Rhythm feature extraction module: Captures human walking and running motion data, and extracts rhythm features from the frequency and phase time domains; Rhythm Generator: Based on the rhythm features extracted in step 1, a rhythm generator is constructed; Unified control policy module for walking and running: Based on the policy gradient method of constrained reinforcement learning algorithm, a unified control policy for walking and running is constructed. Walk2Run control module: The unified walking and running control strategy adopts rhythm information generated by a rhythm generator and then trained with actor-critic to jointly construct the speed-driven bionic motion control system Walk2Run, realizing natural gait transition and frequency adjustment, while meeting the physical limitations of the robot and improving motion smoothness and energy efficiency.

[0024] An application of the above-mentioned unified control method for walking and running of humanoid robots based on biological rhythms is characterized by its application in the field of humanoid robot movement, enabling humanoid robots to achieve a walking mode with the lowest energy consumption, and has good engineering adaptability and expansion value.

[0025] The beneficial effects of this invention are: This invention enables humanoid robots to achieve efficient and natural gait adjustment capabilities during variable-speed tasks.

[0026] This invention extracts rhythmic frequency and phase features from human motion data, trains a rhythm generator to provide an adjustable cadence reference, and integrates it as a guiding signal into policy learning. Simultaneously, an improved constraint reinforcement learning algorithm based on a barrier function is introduced to ensure the training stability and behavioral feasibility of the policy while improving motor ability.

[0027] Experimental results show that this invention enables robots to achieve adaptive rhythmic adjustment in the low-to-high speed range, improving gait stability and behavioral naturalness. Introducing rhythmic rewards enhances the policy's sensitivity and response speed to speed changes, enabling smooth switching between walking and running behaviors. Furthermore, the safe reinforcement learning mechanism effectively constrains joint overstepping behavior during the training phase, improving the policy's safety and robustness. In terms of energy efficiency, the rhythmic control strategy exhibits lower transport energy consumption in medium-to-high speed walking and running tasks, demonstrating significant energy efficiency advantages. Attached Figure Description

[0028] Figure 1 This is a schematic diagram of the actual application deployment of the Walk2Run invention on the Unitree G1 platform.

[0029] Figure 2 This is a schematic diagram of the Walk2Run control framework of the present invention.

[0030] Figure 3 This is a schematic diagram of the human joint mapping relationship of the present invention.

[0031] Figure 4 This is a schematic diagram of the centroid linear velocity estimator of the present invention.

[0032] Figure 5 This is a schematic diagram illustrating the step frequency effect of the present invention.

[0033] Figure 6 This is a schematic diagram of a single speed command tracking comparison experiment according to the present invention.

[0034] Figure 7 This is a schematic diagram of the speed change command tracking and comparison experiment of the present invention.

[0035] Figure 8 This is a schematic diagram comparing the energy transmission costs of the present invention.

[0036] Figure 9 This is a schematic diagram comparing the energy transmission consumption of the present invention.

[0037] Figure 10 This is a schematic diagram illustrating the landing effect of the present invention.

[0038] Figure 11 This is a schematic diagram of the Walk2Run walking-running switching experiment effect of the present invention, wherein (a) walking effect in the training environment, (b) walking effect in Sim2Sim, (c) walking effect in a real scene, (d) natural switching effect between walking and running, (e) running effect in the training environment, (f) running effect in Sim2Sim, and (g) running effect in a real scene.

[0039] Figure 12This is a comparative schematic diagram of the centroid state estimation method of the present invention.

[0040] Figure 13 This is a schematic diagram of the training data distribution and output frequency of the rhythm pattern generator of the present invention.

[0041] Figure 14 This is a schematic diagram comparing the step size characteristics under fixed-frequency control and rhythmic modulation of the present invention across speed.

[0042] Figure 15 This is a schematic diagram comparing the motion performance of fixed frequency and rhythm control in high-speed scenarios according to the present invention, wherein (a) is a schematic diagram of the motion performance of fixed frequency in high-speed scenarios, and (b) is a schematic diagram of the motion performance of rhythm control in high-speed scenarios.

[0043] Figure 16 This is a schematic diagram of constraint effectiveness analysis under the constraint reinforcement learning framework of the present invention, wherein (a) is a schematic diagram of position constraint effect analysis, (b) is a schematic diagram of velocity constraint effect analysis, and (c) is a schematic diagram of torque constraint effect analysis.

[0044] Figure 17 This is a schematic diagram comparing speed tracking under the speed change command of the present invention.

[0045] Figure 18 This is a schematic diagram comparing transportation costs under variable speed operating conditions according to the present invention. Detailed Implementation

[0046] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods are omitted so as not to obscure the description of this application with unnecessary detail.

[0047] It should be understood that, when used in this specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0048] It should also be understood that the terminology used in this application specification is for the purpose of describing particular embodiments only and is not intended to limit the application. As used in this application specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0049] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0050] Many specific details are set forth in the following description in order to provide a full understanding of this application. However, this application may also be implemented in other ways different from those described herein. Those skilled in the art can make similar extensions without departing from the spirit of this application. Therefore, this application is not limited to the specific embodiments disclosed below.

[0051] Implementation Method 1 This embodiment provides a unified control method for walking and running of a humanoid robot based on biological rhythms, the method including the following steps: Step 1: Capture human walking and running motion data, and extract rhythmic features from the frequency and phase time domains; Step 2: Based on the rhythm features extracted in Step 1, construct a rhythm generator; Step 3: Construct a unified control policy for walking and running based on the policy gradient method of constrained reinforcement learning algorithm. Step 4: The unified walking and running control strategy in Step 3 uses rhythm information generated by the rhythm generator in Step 2, and then trains it with actor-critic to jointly construct the speed-driven bionic motion control system Walk2Run, which realizes natural gait transition and frequency adjustment, while meeting the physical limitations of the robot and improving motion smoothness and energy efficiency.

[0052] Furthermore, step 1 specifically involves extracting rhythmic features from both frequency and phase time domain perspectives based on human walking and running motion capture data, and mapping them to the robot joint space to provide structured priors for subsequent rhythmic perception and control. The motion trajectory of key joints and pelvis of the human lower limbs is obtained by a high-precision motion capture system, and Fourier analysis is used to extract rhythmic signals such as step frequency and phase. Then, these rhythmic features are mapped to the robot skeleton space to construct a "bionic rhythm mapping".

[0053] To avoid reliance on real motion capture equipment, this invention uses the CMU Mocap dataset, which contains various rhythmic motion samples. After screening, six sequences (total duration approximately 72 seconds) related to walking and running were selected to model the rhythmic features of the human lower limbs.

[0054] This dataset uses a skeletal model containing 31 joints to represent the human body, where each joint is modeled as a 3-DOF spherical joint, and its rotational posture and relative position are recorded in a local coordinate system. This approach can characterize the posture changes and dynamic features of the human body during movement.

[0055] These joint data were transformed into the robot's lower limb joint space through skeleton mapping. The main features extracted included joint pose (jointpos∈R12), root linear velocity (rootvel∈R3), and angular velocity (rootang∈R3), providing a reference for rhythm modeling and humanoid strategy learning.

[0056] 1) Joint trajectory information acquisition: Retargeting yields the joint angles, mapping the local rotation (local_rot) of the human motion capture data to the robot's joint angles (dof_pos). In the motion capture data, the pose information of each joint is represented as quaternions. The motion of a robot joint is a rotation about a single axis. The local rotation information represented by quaternions in the motion capture data needs to be mapped to the rotation angle of the robot joint. Since the joint motion of a robot can usually be regarded as rotation about a single axis, the quaternion representation needs to be converted into an axis-angle representation (see Equation 1 for details).

[0057] (1) Where θ is the rotation angle and v is the rotation axis, according to the above formula, the quaternion Convertable to and :

[0058]

[0059] In general, different joints of a humanoid robot correspond to different degrees of freedom. For a humanoid robot, the degrees of freedom for each joint are different. The hip, like a human, has 3 degrees of freedom, corresponding to 3 joints; the knee joint has 1 degree of freedom, corresponding to 1 joint; and the ankle joint has 2 degrees of freedom, corresponding to 2 joints. Based on this relationship, the joint trajectories can be mapped according to equation (4):

[0060] 2) Obtaining rhythm information: Using Fast Fourier Transform (FFT), as shown in Equation (5), the relationship between the human center of mass motion state and motion frequency is analyzed to reveal human motion rhythm information. FFT can effectively convert complex time-domain signals into frequency-domain representations, thereby revealing the frequency components, which is crucial for the extraction of human gait cycles.

[0061]

[0062] in, It is frequency The spectrum on It is the original time-domain signal. It refers to the duration. This is determined by the main peak of the spectrum. The extraction of , as shown in Equation (6), can obtain the most significant gait rhythm frequency, which can serve as a key reference for subsequent rhythm modeling.

[0063]

[0064] 3) Center of mass motion estimator: Since the linear velocity of the center of mass cannot be directly obtained from sensors such as IMUs, this experiment constructs a linear velocity estimator based on a multilayer perceptron (MLP) to complete the center of mass state information and serve as a key input to the rhythm generator to characterize the robot's real motion state. The estimator takes 15 consecutive frames of historical observations as input, including angular velocity and linear acceleration recorded by the IMU, as well as the joint states at the corresponding moments. During the training phase, noisy observations are used as input, and the ground truth linear velocity in the simulation is used as the supervision signal. Optimization is performed using mean squared error (MSE) loss. Combined with the angular velocity measured by the IMU, a complete estimate of the robot's center of mass motion state can finally be achieved.

[0065] Furthermore, step 2 specifically involves designing a RhythmGenerator inspired by a Central Pattern Generator (CPG), which can dynamically generate rhythmic signals with matching frequencies based on the robot's current motion state, driving the generation of periodic gait. A center-of-mass state estimator was trained using multi-frame joint angles and IMU data to achieve accurate perception of the robot's center-of-mass linear velocity, providing dynamic context for the rhythm generator. Based on this, a Central Pattern Generator (CPG) style rhythm generator was trained using the center-of-mass velocity as input and human-like rhythms as labels, enabling the robot to autonomously generate gait patterns matching those of humans in various motion states.

[0066] Based on the center of mass motion state and joint frequencies obtained in the previous step, a rhythm generator is established to generate corresponding motion rhythms according to the robot's motion. In this invention, the rhythm generator is based on a multilayer perceptron, as shown in the following equation:

[0067] in Indicates the current moment. Indicates frequency, This indicates the current state of motion of the center of mass, including its linear velocity and angular velocity. It is a fully connected neural network. As shown in Equation (8), the rhythm generator is trained using the mean squared error (MSE) loss function:

[0068] in It is the predicted frequency. This is the true frequency. The network is trained using a backpropagation algorithm to reduce the difference between the predicted and true frequencies, thereby generating rhythmic cycles.

[0069] After inferring frequency information from the rhythm generator, phase information is further calculated. To ensure a smooth phase transition under frequency changes, the following formula is used for phase calculation:

[0070] This formula ensures a smooth phase transition when the frequency changes, thus avoiding discontinuous gait switching.

[0071] To verify the effectiveness of the method of this invention, a robot variable speed walking experiment was designed. In the experiment, the robot moved at speeds of 0.5 m / s, 1.0 m / s, and 1.5 m / s respectively. The experimental results are as follows: Figure 5 As shown, whenever the robot changes speed, the rhythm generator automatically adjusts the rhythm based on the motion of the center of mass, so that the motion frequency gradually increases with the increase of speed.

[0072] By comparing it with the Humanoid-Gym method based on a fixed-frequency sinusoidal clock signal, it was found that the motion method based on a rhythm generator proposed in this invention shows a significant improvement in tracking speed. Figure 6 As shown, the control method based on the rhythm generator can respond to and achieve the desired motion speed command more quickly.

[0073] At a high-speed walking speed of 1.5 m / s, the method of this invention is 1.98 seconds faster than Humanoid-Gym, representing a 40% improvement. This difference indicates that the control strategy proposed in this invention can provide more precise and efficient motion execution at high speeds.

[0074] like Figure 7As shown, during variable-speed walking, the robot's response speed was further tested when tracking signals of 0.5 m / s, 1.0 m / s, and 1.5 m / s. Experimental results show that the method of this invention is 0.4 seconds, 0.3 seconds, and 0.6 seconds faster than Humanoid-Gym, respectively, representing improvements of 40%, 30%, and 45%. This indicates that the rhythm generator can better adapt to changing speed commands, improving the robot's response speed while optimizing the stability of motion control.

[0075] Furthermore, walking based on natural rhythms also demonstrates superior energy consumption. This is evidenced by tests on transportation costs, such as... Figure 8 and 9 As shown, it was found that the motion based on natural rhythms reduced energy consumption by approximately 10% compared to the Humanoid-Gym method based on a fixed frequency. This result demonstrates that the rhythm generation method proposed in this invention not only has significant advantages in speed and efficiency but also effectively reduces the energy consumption of robots.

[0076] Finally, the experimental results clearly demonstrate that the natural rhythm-based method can generate more stable gait at different movement speeds. At different speeds, the fixed-frequency Humanoid-Gym method may result in one foot always leading and an irregular gait, such as... Figure 10 As shown. However, the gait control method based on rhythm generation effectively avoids this problem, ensuring gait balance and stability.

[0077] Furthermore, the constraint reinforcement learning algorithm in step 3 specifically involves introducing a smoothing barrier function to improve the constraint reinforcement learning algorithm to meet the value range restrictions and enhance the stability and reliability of the policy training process.

[0078] The unified walking-running control strategy in step 3 specifically involves using a constraint-based policy gradient method to construct a unified walking-running control strategy.

[0079] 1) Restricted Markov Decision Process: Reinforcement learning problems are based on Markov Decision Processes (MDPs), where an MDP is a tuple. .in, It is a finite sequence of states. It is a finite sequence of actions. It is a reward function. It is the state transition probability function. Here, λ is the initial state probability, and λ is the initial time discount. Typically, the goal of MDP is to maximize the cumulative reward. As shown in equation (10).

[0080]

[0081] Generally speaking, From the value function Approximately, as shown in equation (11):

[0082] The action value function is represented as shown in equation (12):

[0083] The degree of advantage of the actions that can be taken in the current state can be characterized by the advantage cost function, as shown in equation (13):

[0084] CMDP is an extension of MDP by introducing constraint C, and its goal is to maximize the cumulative reward. At the same time, ensure all constraints The condition is satisfied, as shown in equation (14):

[0085]

[0086] 2) Optimization of constrained strategies: When solving the CMDP problem, the strategy is... It is the result of optimization. Although it is possible to guarantee during the optimization process... Satisfy constraints However, it is difficult to directly guarantee the updated strategy. This also satisfies the constraints. Therefore, the CPO (Constrained Policy Optimization) literature demonstrates through inference that there is an upper and lower bound on the performance difference between the two policies, as shown in equation (15):

[0087] The specific forms of the upper and lower bounds are shown in equation (16):

[0088] in, It is the maximum deviation from the constraint on returns. It is used to limit the magnitude of policy updates. Divergence, its function is to ensure and It won't deviate too much during updates. Based on The dominance function. Based on the above inference, the CMDP problem can be constructed in the following form:

[0089] (17) The above construction will Alternative Thus, indirectly controlling the optimization objective. The constraints must be satisfied. Most modern policy optimization methods (such as PPO and TRPO) impose constraints using CLIP or confidence region methods. and The KL divergence between them ensures that the distance between them is small. Therefore, the constraint conditions can be further simplified, as shown in equation (18):

[0090] However, hard constraints are not easy to compute. To further simplify the optimization, the traditional method uses the Lagrange multiplier method to combine the benefits and constraints into an optimization objective, as shown in equation (19):

[0091] in: .

[0092] However, the traditional Lagrange multiplier method is not suitable for solving application scenarios requiring multiple constraints, such as humanoid robots. Referring to the IPO method, soft constraints are introduced through a barrier function. The barrier function has the property shown in equation (20):

[0093] To ensure computational feasibility, the barrier function is often approximated by a smoothing function (log). This is then used to establish a policy optimization problem based on the PPO method. L CLIP Consider confidence region pairs for the PPO method The approximation is shown in equation (21):

[0094] in It is a hyperparameter that controls the degree of smoothness. In the optimal state, taking the derivative of both sides of the objective function above, we obtain the following optimization conditions, as shown in equation (22):

[0095] From the perspective of the Lagrange multiplier method, the multiplier corresponding to each constraint The optimal solution using the barrier function is obtained. As shown in equation (23):

[0096] Further analysis of the dual gap shows that the optimal solution of the soft constraint... The performance value is usually less than the optimal solution π under hard constraints. h ,Right now:

[0097] Therefore, it can be seen that the accuracy and optimization results of the barrier function are related to the hyperparameters. and number of constraints Closely related. Through reasonable adjustments and It can make soft constraint solutions As close as possible to the hard constraint solution This effectively balances computational cost with constraint satisfaction. Furthermore, during the calculation process, there exists a cost term... There is a possibility that the value is greater than 0, which does not conform to the definition of log and will lead to the failure of the solution. Therefore, the barrier function is modified as shown in equation (25):

[0098] This renovation ensures The time barrier function is still defined and smoothly transitions to the negative value region.

[0099] 3) Constraint design: To ensure that the strategy output satisfies the physical motion capabilities of the robot, this invention explicitly introduces three types of constraints during training: joint position, joint velocity, and joint torque limits. Therefore, the hyperparameter m is 3, and the hyperparameter t is defined as 2000. Each constraint is constructed in the form of a violation ratio, i.e., the frequency of exceeding the physical limit in all joints is counted as the loss signal for that constraint. Each constraint has an equal weight in the total loss; specific definitions are shown in Table 1.

[0100] Table 1. Constraint Function Design

[0101] Furthermore, in step 4, the actor-critic specifically refers to a training framework based on the Actor-Critic architecture, where the Critic network is further divided into two sub-modules: one sub-module is used to evaluate the overall reward, and the other sub-module is used to model multiple cost terms to describe the degree of violation of robot position, velocity, and torque limits. Walk2Run in step 4 is used to achieve natural gait transition and frequency adjustment, while meeting the physical limitations of the robot and improving motion smoothness and energy efficiency.

[0102] Furthermore, the improved constrained reinforcement learning algorithm specifically involves formalizing the control problem as a constrained Markov decision process and combining it with an improved barrier function structure to achieve smooth control of the constraint boundary during the training process.

[0103] Furthermore, to guide the unified walking and running control strategy to generate human-style motion trajectories, a GAN-based discriminator network is introduced. Using human joint trajectories as a reference, the network evaluates the degree of human-likeness of the robot's current movement and uses the discrimination result as an additional reward signal to encourage the strategy to generate more natural motion behaviors.

[0104] Furthermore, the unified walking and running control strategy uses rhythm information as an auxiliary input and dynamically adjusts the movement frequency in combination with the current center of mass motion state, enabling the robot to adaptively switch between walking and running modes according to the task objective and body state.

[0105] The unified walking-running motion control strategy includes: 1) Design of reward function based on rhythm information To effectively integrate rhythmic information into robot motion, a landing frequency planning method based on phase information was designed. This planning is first defined based on a gait mask matrix, which represents the desired motion state of each joint of the robot within a specific time step. This is illustrated in equations (26), (27), and (28).

[0106]

[0107]

[0108] The rhythmic reward function is designed based on this mask matrix, as shown in equation (29):

[0109] in, This indicates the robot's foot contact status. This represents the desired state of the current gait mask. A positive reward is given when the robot's foot contact matches the rhythmic gait mask; otherwise, a penalty is applied. This ensures the robot moves according to the generated rhythm.

[0110] 2) Trajectory feature learning from prior knowledge of motion-versus-motion confrontation To guide the policy in generating more human-like natural actions, a style reward mechanism based on adversarial learning is introduced. Specifically, a discriminator network is constructed. Used to determine a given state transition pair Whether it comes from human motion data. The discriminator is trained using least squares loss, with the goal of classifying real samples as 1 and policy-generated samples as -1. Gradient penalty is introduced to improve training stability, as shown in equation (30):

[0111] The objective of the last term in equation (30) is the gradient penalty, with a coefficient of . This reduces the discriminator's tendency to assign non-zero gradients on the manifold of real data samples, thereby reducing oscillations during GAN training and improving training stability. The humanoid style reward is defined as shown in equation (31), where...

[0112]

[0113] Additional offsets and scaling were applied to limit the reward to the range [0, 1].

[0114] 3) Reward Design This invention designs a composite reward function consisting of three types of reward items, namely: humanoid style reward. Rhythmic rewards With basic rewards .in, and As mentioned earlier, the basic reward This includes several sub-items used to constrain the physical rationality and control stability of the robot, as shown in Table 2.

[0115] Finally, by combining all the rewards and their weights ω, the total reward function is obtained as shown in equation (32):

[0116] Table 2 Other Reward Settings

[0117] While reinforcement learning excels in high-dimensional control tasks, traditional methods often neglect modeling the robot's dynamic constraints, potentially leading to unreasonable or even dangerous action outputs. To improve the physical feasibility and training stability of the strategy, this invention introduces a constraint optimization-based reinforcement learning method. This method effectively controls the risk of constraint violations, such as joint position, velocity, and torque, while ensuring policy performance, avoiding reliance on complex manual reward designs. To this end, the control problem is formalized as a constrained Markov decision process, and combined with an improved barrier function structure, smooth control of the constraint boundaries is achieved during training.

[0118] The experiment based on Implementation Method 1 is as follows: 1) Experimental setup In this invention, the humanoid robot platform G1 (29-DOF version) launched by Unitree Robotics was used as the experimental platform. The deployment system was implemented based on Python, and the core modules, including state estimation, rhythm generator, and policy network, were all executed by Libtorch inference. It was also integrated with the robot's underlying control interface through Unitree's official SDK to realize the online deployment and operation of the algorithm on the actual platform.

[0119] Given the limitations of real robot platforms, such as the difficulty in obtaining true speed values ​​and the uncontrollability of experimental environments, the experimental design is divided into two parts: simulation experiments and real machine deployment experiments. In order to make the simulation results as close as possible to the actual deployment situation, the evaluation was not performed directly in the training environment Isaac Gym, but a high-fidelity Sim2Sim simulation environment based on MuJoCo was constructed. Existing research has shown that MuJoCo is closer to the actual robot system in terms of dynamic modeling accuracy and physical consistency

[41] . Therefore, the robot model and task configuration consistent with the training environment were reconstructed in it to obtain evaluation results with more realistic reference value.

[0120] The specific experimental items include the following: Design comparative experiments to evaluate the performance of the state centroid estimation module and the rhythm generator. The comparative experiments will compare the accuracy of centroid estimation based on MLP and LSTM with that of velocity estimation using Kalman filtering. The output frequency of the rhythm generator under different operating conditions and its ability to generate natural rhythmic motion under its guidance strategy will be analyzed, followed by a comparative analysis of the motion effects of natural rhythms and fixed rhythms.

[0121] Design a comparative experiment to evaluate the effectiveness of the constraint reinforcement learning strategy. By comparing the difference in the motor limit violation rate between the improved strategy proposed in this paper and the traditional PPO, we can verify its ability to improve motor ability while maintaining feasibility.

[0122] By setting different target speed sequences, the experiment tested whether the robot could achieve stable walking-running switching based on a rhythmic adjustment mechanism, and compared the adaptability differences of non-rhythmic guidance strategies under speed changes. This experiment was conducted simultaneously on the MuJoCo simulation and real robot platforms to evaluate the transferability and robustness of the algorithm.

[0123] By measuring energy consumption performance under different constant speed and variable speed tasks, energy efficiency analysis experiments were conducted to evaluate the impact of introducing a rhythm mechanism on motion efficiency and further verify the energy efficiency advantages of the system.

[0124] 2) Centroid State Estimator and Rhythm Generation Effect This part of the experiment mainly verifies two components: firstly, the effectiveness of the center-of-mass state estimator and the rhythm generator. To verify the effectiveness of the center-of-mass state estimator, estimation models were trained based on two different neural network architectures: one using a multilayer perceptron (MLP) and the other based on a long short-term memory (LSTM) network. The training objective was to regress the robot's current center-of-mass linear velocity. The experimental scenario was set in the MuJoCo environment, controlling a humanoid robot to perform a variable-speed motion task, and the performance of the trained estimator was evaluated. Simultaneously, for comparison, an extended Kalman filter (EKF) based on IMU data was introduced as a representative of traditional methods. Figure 12 As shown, the MLP-trained estimator exhibits the best estimation accuracy and stability across all speed ranges, effectively capturing the robot's motion state. The LSTM-trained estimator performs well in the low-speed range (approximately 0.5 m / s), but its accuracy slightly decreases in high-speed scenarios. In contrast, the EKF method shows significant drift over long periods of operation. Furthermore, the performance of the three methods in YZ axis velocity estimation is relatively similar, with relatively small differences.

[0125] Secondly, regarding the rhythm generation module, the training data covers three typical motion states: slow walking (0.0-0.3m / s, 0.3-0.32Hz), fast walking (0.6-0.9m / s, 0.80-0.95Hz), and fast running (1.6-2.4m / s, 0.95-1.40Hz), such as... Figure 13 As shown. The input speed was limited to the range of 0-2 m / s in the evaluation. Experimental results show that in the speed range below 1 m / s, the generated rhythmic frequency increases positively. However, after the speed exceeds 1 m / s, the frequency increase tends to level off due to the significant increase in stride length as the gait transitions from walking to running; the frequency accelerates again after the speed exceeds 1.5 m / s. It should be noted that the generation of the robot's frequency depends on the robot's three-dimensional velocity and angular velocity at the center of mass, but for the purpose of demonstrating the effect, [the remaining text is incomplete and requires further context]. Figure 13 This demonstrates the relationship between the generated rhythm and the velocity along the x-axis.

[0126] To further evaluate the effect of rhythmic signals on motion behavior, the Humanoid-Gym control framework was modified based on the method described in

[41] to support rhythmic rewards. In the original Humanoid-Gym, the frequency is a fixed constant, but with the addition of rhythmic rewards, the control strategy can adaptively adjust the rhythmic output according to the center of mass state for motion. Experiments showed that as speed increases, the introduction of rhythmic signals can significantly improve stride consistency and gait stability. The generated stride length is generally smaller than that of the fixed frequency strategy, avoiding the limit activation of joint space, thus providing greater redundancy space and energy efficiency potential for high-speed motion.

[0127] 3) Constraint Effect Analysis To demonstrate the effectiveness of the constrained reinforcement learning framework, the cost of the PPO algorithm and the algorithm regarding the constraint terms during training were compared, such as... Figure 16 As shown. To ensure fairness, all strategies use the same network structure, training hyperparameters, and task settings; the only difference lies in whether or not a constraint term is introduced into the loss function. Figure 16 As shown in (a), during training, the algorithm effectively suppressed the robot's joint movement positions from exceeding safe limits compared to the PPO algorithm, demonstrating stronger convergence and boundary awareness. However, as... Figure 16 As shown in (b) and (c), the probability of violating the limit behavior in terms of velocity and torque is much lower than the probability of position occurrence in the PPO-based method. Therefore, the algorithm is lenient in these two aspects and its effect is similar to that of the PPO method.

[0128] To further quantify the degree to which the strategy adheres to physical constraints at the execution level, the violation rate of different strategies in the later stages of training was statistically analyzed, as shown in Table 3. The violation rate is defined as the proportion of actions that violate various constraints in the strategy output within a fixed movement time period, covering performance under different movement types (walking and running). The results show that in the running scenario, the unconstrained PPO strategy has a violation rate of 8% for positional constraints, while this rate drops to 3.7% after introducing constraints, effectively avoiding potential movement anomalies and hardware damage risks. Under speed and torque constraints, both methods maintain low violation rates, indicating that the constraint mechanism improves the physical feasibility of the strategy without compromising the original control performance. It should be noted that because the true reference values ​​of joint torques cannot be extracted from the motion-assisted data, the Dataset column for the tau item in the table is left empty.

[0129] Table 3. Strategy Output Violation Rate under Different Motor Behaviors

[0130] 4) Speed ​​change experiment To verify the effectiveness of the proposed motion control framework in actual deployment and the contribution of the rhythm module, comparative and ablation experiments were designed. The system's response capability and natural walking-running switching behavior were tested by speed change commands.

[0131] First, such as Figure 11 As shown in (d), the algorithm was successfully deployed in a real-world environment, enabling the humanoid robot to switch naturally between walking and running during speed changes. To demonstrate the effect more systematically, Figure 11 (a)-(c) demonstrate the robot's posture when walking at 1.0 m / s in the training environment, Sim2Sim environment, and real-world environment. During this process, the robot maintains a straight-knee gait, with stable movements that closely resemble the natural walking style of humans. Figure 11(e)-(g) demonstrate the robot's posture when running at 1.5 m / s. The robot exhibits a larger stride and a more pronounced leg lift, consistent with human running characteristics. The behavior remains consistent across the three environments, validating the policy's transferability on a real platform and the generalization ability of the rhythm generator.

[0132] To further quantify the role of the rhythm module in velocity response, velocity variation experiments were conducted in the Sim2Sim environment to test the system's response at three target velocities (0.5 m / s, 1.0 m / s, and 1.5 m / s). Figure 17 As shown, three algorithm configurations are compared: (1) the original Humanoid-Gym controller (fixed frequency), (2) the complete Walk2Run controller, and (3) the Walk2Run controller without the rhythm module. The results show that the Humanoid-Gym controller has a slightly better response in low-speed scenarios (0.5 m / s). In medium- and high-speed scenarios (1.0 m / s and 1.5 m / s), the Walk2Run controller exhibits faster speed tracking capabilities. The Walk2Run controller without the rhythm module has the worst response across all speed ranges, indicating that the rhythm regulation mechanism plays an important role in improving the system's speed adaptability. Overall, the rhythm-aware control framework proposed in this paper not only achieves smooth and natural walking-running switching in simulation and reality, but also demonstrates superior response capabilities and stability under dynamic speed commands.

[0133] 5) Energy efficiency analysis To further analyze the effectiveness of this motion framework, the robot's Co-operational Time (COT) is analyzed based on the variable speed experiment. COT is a common evaluation system for the energy consumption of robots during transportation, as shown in equation (33).

[0134]

[0135] in, and They represent the first The torque and angular velocity of each joint at time i. For robot quality, This represents the displacement of the robot's center of mass along the forward direction during the mission.

[0136] In the speed change experiment, the COT curve is as follows: Figure 18 As shown in the figure. The results show that in the initial acceleration stage, both Walk2Run and Walk2Run without a rhythm module exhibit low energy consumption; however, in the low-speed steady-state stage, the Humanoid-Gym method has slightly better energy efficiency. As the speed increases to the medium-high speed range, the Humanoid-Gym method and the method without a rhythm module gradually surpass Humanoid-Gym in energy efficiency, showing better energy utilization efficiency.

[0137] It is worth noting that during speed switching, the rhythmic algorithm performs more stably in terms of frequency regulation and gait stability, and therefore outperforms the non-rhythmic version in overall energy efficiency.

[0138] In addition, COT measurements were performed on each method under different constant speed conditions, and the specific results are shown in Table 4. It can be seen that Humanoid-gym performs better than the Walk2Run controller at low speeds. However, Walk2Run performs better at medium and high speeds.

[0139] Table 4 Comparison of COT for the three methods at different speeds

[0140] Implementation Method 2 This embodiment provides a unified control system for humanoid robot walking and running based on biorhythms. The system uses a unified control method for humanoid robot walking and running based on biorhythms as described in Embodiment 1. The system includes: Rhythm feature extraction module: Captures human walking and running motion data, and extracts rhythm features from the frequency and phase time domains; Rhythm Generator: Based on the rhythm features extracted in step 1, a rhythm generator is constructed; Unified control policy module for walking and running: Based on the policy gradient method of constrained reinforcement learning algorithm, a unified control policy for walking and running is constructed. Walk2Run control module: The unified walking and running control strategy adopts rhythm information generated by a rhythm generator and then trained with actor-critic to jointly construct the speed-driven bionic motion control system Walk2Run, realizing natural gait transition and frequency adjustment, while meeting the physical limitations of the robot and improving motion smoothness and energy efficiency.

[0141] Implementation Method 3 This embodiment describes an application of the unified control method for walking and running of humanoid robots based on biological rhythms as described in Embodiment 1. Its characteristic is that it is applied to the field of humanoid robot movement, enabling humanoid robots to achieve a walking mode with the lowest energy consumption, and has good engineering adaptability and expansion value.

Claims

1. A unified control method for walking and running of a humanoid robot based on biological rhythms, characterized in that, The method includes the following steps: Step 1: Capture human walking and running motion data, and extract rhythmic features from the frequency and phase time domains; Step 2: Based on the rhythm features extracted in Step 1, construct a rhythm generator; Step 3: Construct a unified control policy for walking and running based on the policy gradient method of constrained reinforcement learning algorithm; Step 4: The unified walking and running control strategy in Step 3 adopts the rhythm information generated by the rhythm generator in Step 2, and then uses actor-critic for training. Together, they construct the speed-driven bionic motion control system Walk2Run, which realizes natural gait transition and frequency adjustment, while meeting the physical limitations of the robot and improving motion smoothness and energy efficiency. Step 2 specifically involves dynamically generating a rhythmic signal with a matching frequency based on the robot's current motion state to drive the generation of periodic gait. Using the center-of-mass velocity as input and human-like rhythms as labels, a rhythm generator was trained, enabling the robot to autonomously generate gait patterns that match those of humans in various motion states. The constraint reinforcement learning algorithm in step 3 specifically involves introducing a smoothing barrier function to improve the constraint reinforcement learning algorithm to meet the value range limit and enhance the stability and reliability of the policy training process. To guide the unified control strategy of walking and running to generate human-style motion trajectories, a GAN-based discriminator network is introduced. Using human joint trajectories as a reference, the network evaluates the degree of human-likeness of the robot's current movement and uses the discrimination result as an additional reward signal to encourage the strategy to generate more natural motion behavior. Specifically, this includes designing reward functions based on rhythm information; Learning trajectory features from prior knowledge of motion-based adversarial activities; The design of the rewards.

2. The method according to claim 1, characterized in that, Specifically, step 1 involves extracting rhythmic features from both frequency and phase time domain perspectives based on human walking and running motion capture data, and mapping them to the robot joint space to provide structured priors for subsequent rhythmic perception and control. The motion trajectories of key joints and pelvis of the human lower limbs are obtained by a high-precision motion capture system, and rhythmic signals are extracted by Fourier analysis. These rhythmic features are then mapped to the robot skeleton space to construct a "bionic rhythmic mapping".

3. The method according to claim 1, characterized in that, In step 4, the actor-critic specifically refers to a training framework based on the Actor-Critic architecture, where the Critic network is further divided into two sub-modules: one sub-module is used to evaluate the overall reward, and the other sub-module is used to model multiple cost terms to describe the degree of violation of robot position, velocity, and torque limits. Walk2Run in step 4 is used to achieve natural gait transition and frequency adjustment, while meeting the physical limitations of the robot and improving motion smoothness and energy efficiency.

4. The method according to claim 1, characterized in that, Specifically, the improved constrained reinforcement learning algorithm formalizes the control problem into a constrained Markov decision process and combines it with an improved barrier function structure to achieve smooth control of the constraint boundary during training. The improved barrier function for, （25） in, Hyperparameters for controlling the smoothness , As a cost item.

5. The method according to claim 1, characterized in that, The unified walking and running control strategy uses rhythm information as an auxiliary input and dynamically adjusts the movement frequency in combination with the current center of mass motion state, enabling the robot to adaptively switch between walking and running modes according to the task objective and body state.

6. A unified control system for walking and running of a humanoid robot based on biological rhythms, characterized in that, The system uses a unified control method for walking and running of a humanoid robot based on biological rhythms as described in any one of claims 1-5, and the system includes: Rhythm feature extraction module: Captures human walking and running motion data, and extracts rhythm features from the frequency and phase time domains; Rhythm Generator: Based on the rhythm features extracted in step 1, a rhythm generator is constructed; Unified control policy module for walking and running: Based on the policy gradient method of constrained reinforcement learning algorithm, a unified control policy for walking and running is constructed. Walk2Run control module: The unified walking and running control strategy adopts rhythm information generated by a rhythm generator and then trained with actor-critic to jointly construct the speed-driven bionic motion control system Walk2Run, realizing natural gait transition and frequency adjustment, while meeting the physical limitations of the robot and improving motion smoothness and energy efficiency.

7. An application of the unified control method for walking and running of a humanoid robot based on biological rhythms as described in any one of claims 1-5, characterized in that, It is applied to the field of humanoid locomotion in robotics, enabling humanoid robots to walk with the lowest energy consumption, and has good engineering adaptability and expansion value.

Images