Bio-rhythm-based humanoid robot walking and running unified control method and system

Abstract

The invention discloses a humanoid robot walking and running unified control method and system based on a biological rhythm, and belongs to the technical field of humanoid robots. Capturing human walking and running action data, and extracting rhythm features from frequency and phase time domains; constructing a rhythm generator based on the rhythm features extracted in the step 1; according to a strategy gradient method based on a constraint reinforcement learning algorithm, a walking and running unified control strategy is constructed, rhythm information generated based on a rhythm generator is adopted in the walking and running unified control strategy, then action-critic is used for training, a bionic motion control system Walk2Run under speed driving is jointly constructed, natural gait transition and frequency adjustment are achieved, and the walking and running unified control strategy is obtained. And meanwhile, the physical capacity limitation of the robot is met, and motion fluency and energy efficiency are improved. The problems that an existing humanoid robot does not achieve energy optimization of human walking during walking, and mobility of an existing data set has high requirements for the motion ability of the robot and is poor in generalization ability are solved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of humanoid robots, and particularly relates to a walking and running unified control method and system for a humanoid robot based on biological rhythms. BACKGROUND

[0002] Humanoid robots have human-like structure and movement ability, and show great potential in scenarios such as service, medical treatment and disaster response. It is still a key challenge to achieve stable and efficient dynamic motion control, especially in rhythmic movements such as walking and running.

[0003] Unlike the natural gait switching of humans relying on the central pattern generator (CPG), the existing mainstream control methods (such as model predictive control MPC and reinforcement learning RL strategy) focus more on trajectory tracking or action imitation, and lack modeling and utilization of rhythmic structure. This defect makes it difficult for robots to achieve natural gait transition in speed change or task switching, resulting in low motion energy efficiency, heavy parameter adjustment burden and poor adaptability.

[0004] Although recent adversarial imitation methods (such as Adversarial Motion Prior, AMP) have made progress in improving the naturalness of motion, their learning goals mainly focus on "pose similarity", ignoring the structural role of rhythmic signals in regulating periodic motion. Therefore, such methods still have obvious deficiencies in achieving speed adaptation, motion mode switching and long-time energy efficiency control. In addition, although humanoid robots have approached humans in form, due to structural limitations such as joint degrees of freedom and movement range, direct imitation of human trajectories often fails to achieve optimal control strategies, resulting in weak transferability of humanoid motion and high dependence of strategies on hyperparameter adjustment. This further magnifies the limitations of current walking and running control methods in achieving natural and rhythmic control.

[0005] Current humanoid robot motion control methods can be divided into two categories: control methods based on dynamic models and methods based on reinforcement learning. The former, represented by classic ZMP, MPC, and whole-body control, achieves analytical control of the robot state through physical modeling and optimization. This method has good results in improving the dynamic stability and controllability of the robot trajectory, but due to the dual requirements of model accuracy and real-time performance, a simplified model needs to be designed to control the humanoid robot. Linear inverted pendulum (LIPM) technology can be used to calculate foot position. The fusion optimization method uses spring inverted pendulum model, single rigid body dynamics, and center of mass dynamics to calculate foot position and contact force, achieving better walking control effect. However, these model-based control strategies that rely solely on simplified dynamics are inherently limited by their simplification and model mismatch, resulting in conservative motion control and failing to fully exploit the capabilities of the robot.

[0006] In contrast, reinforcement learning methods provide a new paradigm for humanoid robot control strategies. In recent years, deep reinforcement learning has made significant progress in quadruped robot motion, biped robot motion, and humanoid robot motion control tasks. This method eliminates the need for high dependence on models through an end-to-end training process. However, the simulation-to-reality transfer problem remains a major challenge for reinforcement learning control. The mainstream solutions include using high-fidelity physical simulators, introducing domain randomization techniques, and adopting teacher-student policy distillation frameworks to improve the generalization ability of the strategy. In addition, some research attempts to use observation history to construct potential dynamic representation to enhance the state expression ability of the strategy. Currently, most humanoid robot control methods based on reinforcement learning use reward-driven PPO algorithms. Although these methods have shown effectiveness on multiple platforms, there are still two major problems: first, most reinforcement learning methods use reward-based optimization frameworks, ignoring safety and physical constraints during control; second, during policy transfer, there may be conflicts between the actual motion capabilities of the robot and the training priors, affecting the stability of the policy.

[0007] Constraint-based reinforcement learning methods provide a potential solution to the above problems. Based on the constrained Markov decision process (CMDP) framework, researchers have proposed various policy optimization methods, which can be divided into Lagrangian methods and trust region methods. Based on the Lagrangian method, the constrained problem is transformed into an unconstrained problem by introducing a Lagrange multiplier, and the "primal-dual" method is used for solution. Some methods update the policy by linearly approximating the objective function within the trust region to solve the constraints. This subfield contains a variety of algorithms that can be further divided into first-order methods and higher-order methods. Liu et al. proposed the interior point policy optimization (IPO), which uses a logarithmic barrier function to optimize the constraint function in order to optimize the policy within the region that satisfies the constraints. Based on the IPO method, constraint-based quadruped robot motion control was implemented, greatly reducing the parameter tuning workload and policy unreasonable output problems, and improving the safety and mobility of control. However, constraint-based reinforcement learning methods have not been systematically introduced into the field of humanoid robot control within limited cognitive. In bionic motion tasks, there is a problem of inconsistency between the prior goals from human motion data and the robot's motion capabilities. It is believed that the integration of bionic motion priors and constraint-based policy optimization methods will bring more stable, efficient, and natural control strategies to humanoid robots.

[0008] Bionic control methods attempt to draw inspiration from biological motion to improve the naturalness of leg-foot robots in gait rhythm and motion style. This type of method can be roughly divided into two categories: control methods based on biological-inspired models and policy training based on imitation learning.

[0009] With the development of motor neuroscience, the motion control mechanism of the biological nervous system has been gradually introduced into the field of robotics. Research has found that the central pattern generator (CPG) exists in the nervous system of animals, which is considered to be the key control structure for generating repetitive motion (such as walking and swimming) in animals. Through internal neural oscillators, it drives rhythmic motion with no sensory input or simplified sensory feedback. In legged robots, early research mainly borrowed from CPG structures to regulate gait rhythm, improving walking stability and motion continuity. However, CPG is essentially an abstraction of complex biological systems, and its structure modeling form, regulation mechanism, and coupling method with sensory feedback still lack a unified standard, which restricts its widespread application in robotic systems.

[0010] Another class of methods relies on large-scale human motion capture datasets to extract human-like motion patterns through imitation learning or adversarial learning. In imitation learning methods, it can be divided into operational imitation and performance imitation. Operational imitation focuses on the precise control of manipulator and wheeled platform, usually based on teleoperation or demonstration trajectory training. Performance imitation focuses on the reproduction of human-like behavior, which requires highly coordinated dynamic control capabilities. The latter has achieved certain results in simulation environments, but it is still challenging to migrate to real humanoid robots. Some work indeed introduces imitation learning into leg-foot robots, but the imitation area is only placed on the upper body, and the lower limbs are maintained stable based on reinforcement learning.

[0011] The adversarial motion prior method (AMP) in adversarial imitation is the most concerned. This method is based on the GAIL framework, which realizes the learning of complex motion style by judging whether the state transition comes from real data. AMP does not rely on explicit reference trajectory or reward function, so that the strategy can autonomously extract high-level motion features from unstructured motion data. The adversarial motion prior (AMP) method uses the GAIL framework to learn the implicit motion style from the state transition distribution, and the subsequent work extends AMP to the latent space to learn a large-scale reusable adversarial skill embedding suitable for physical simulation characters.

[0012] This technical route has shown excellent performance in the field of quadruped robots. The existing technology uses AMP to learn the motion of a quadruped robot on a flat ground without the need for a complex reward function. This technical route not only can migrate human style into the policy network, but also can migrate trajectory style based on model control into the policy network, so that the effectiveness of AMP in learning robust motion on complex terrain is also verified.

[0013] Although AMP can effectively capture human-like motion style, its core still mainly focuses on the similarity of state transition distribution, and lacks explicit modeling of motion rhythm structure. As a key feature of periodic motion (such as walking and running), rhythm is crucial to improve the naturalness and stability of the control strategy. SUMMARY

[0014] The present application provides a humanoid robot walking and running unified control method based on biological rhythm and a system thereof. The existing humanoid robot walking does not achieve the problem of "energy optimization" of human walking, and the migration of the existing data set requires higher robot motion ability and weaker generalization ability.

[0015] The present application is realized by the following technical solutions: A humanoid robot walking and running unified control method based on biological rhythm, the method comprising the following steps: Step 1: Capture human walking and running action data, and extract rhythm features from frequency and phase time domain; Step 2: Based on the rhythm features extracted in Step 1, construct a rhythm generator; Step 3: Based on the policy gradient method of constraint reinforcement learning algorithm, construct a walk-run unified control strategy; Step 4: The walk-run unified control strategy in Step 3 uses the rhythm information generated by the rhythm generator in Step 2, and is trained using actor-critic, together constructing a biomimetic motion control system Walk2Run under speed driving, achieving natural gait transition and frequency regulation, while meeting the physical ability limit of the robot, improving motion fluency and energy efficiency.

[0016] Further, the Step 1 is specifically based on human walking and running motion capture data to extract rhythm features from both frequency and phase time domain perspectives, and map them to robot joint space, providing structured prior for subsequent rhythm perception control; The motion trajectories of human lower limb key joints and pelvis are obtained through a high-precision motion capture system, and the rhythm signals are extracted using Fourier analysis, and then these rhythm features are mapped to the robot skeleton space to construct a "bionic rhythm mapping".

[0017] Further, the Step 2 is specifically to dynamically generate rhythm signals matching the frequency according to the current motion state of the robot to drive periodic gait generation; The rhythm generator is trained with centroid velocity as input and human-like rhythm as label, so that the robot can autonomously generate a step frequency pattern matching human in each motion state.

[0018] Further, the constraint reinforcement learning algorithm in Step 3 is specifically to introduce a smoothing barrier function to improve the constraint reinforcement learning algorithm to meet the value range limit and improve the stability and reliability of the policy training process.

[0019] Further, the actor-critic in Step 4 is specifically that the training framework is based on the Actor-Critic architecture, where the Critic network is further divided into two sub-modules: one sub-module is used to evaluate the overall reward, and the other sub-module is used to model multiple cost items respectively, to express the degree of violation of robot position, velocity and torque limits; The Walk2Run in Step 4 is used to achieve natural gait transition and frequency regulation, while meeting the physical ability limit of the robot, improving motion fluency and energy efficiency.

[0020] Further, the improved constraint reinforcement learning algorithm is specifically that the control problem is formalized as a constrained Markov decision process, and combined with the improved barrier function structure, the smooth control of the constraint boundary in the training process is realized.

[0021] Further, to guide the generation of a human-style motion trajectory for the walk-run unified control strategy, a discriminator network based on GAN is introduced to evaluate the human-like degree of the current motion of the robot with human joint trajectory as the reference, and the discrimination result is taken as an additional reward signal to encourage the strategy to generate more natural motion behavior.

[0022] Further, the walk-run unified control strategy takes rhythm information as auxiliary input and dynamically adjusts the motion frequency in combination with the current center of mass motion state, so that the robot can adaptively switch between walking and running modes according to the task target and the body state.

[0023] A humanoid robot walk-run unified control system based on biological rhythm, the system uses a humanoid robot walk-run unified control method based on biological rhythm as described above, the system comprises: Rhythm feature extraction module: capturing human walking and running action data, extracting rhythm features from frequency and phase time domain; Rhythm generator: based on the rhythm features extracted in step 1, the construction of the rhythm generator is carried out; Walk-run unified control strategy module: based on the policy gradient method of constraint reinforcement learning algorithm, the walk-run unified control strategy is constructed, Walk2Run control module: the walk-run unified control strategy uses the rhythm information generated by the rhythm generator, and then uses actor-critic for training, which together constructs a bionic motion control system Walk2Run under the driving of speed, realizes natural gait transition and frequency adjustment, and at the same time meets the physical ability limit of the robot, improves the motion fluency and energy efficiency.

[0024] An application of the above-mentioned humanoid robot walk-run unified control method based on biological rhythm, characterized by being applied to the field of human-like motion of robots, realizing the walking mode of the human-like robot with the lowest energy consumption, and having good engineering adaptability and expansion value.

[0025] The beneficial effects of the present application are: The present application is used to realize the efficient and natural gait adjustment ability of humanoid robots in variable speed tasks.

[0026] The present application extracts rhythm frequency and phase features from human action data, trains a rhythm generator to provide adjustable step frequency reference, and integrates it into strategy learning as a guide signal. At the same time, an improved constraint reinforcement learning algorithm based on barrier function is introduced to improve the motion ability while ensuring the training stability and behavior feasibility of the strategy.

[0027] The experimental results show that the robot can realize adaptive rhythm adjustment in the low-speed to high-speed interval, and the stability of the gait and the naturalness of the behavior are improved. After introducing the rhythm reward, the strategy has higher sensitivity and response speed to speed changes, thereby realizing smooth switching between walking and running behaviors. In addition, the safety reinforcement learning mechanism effectively restricts the joint out-of-limit behavior in the training stage, and improves the safety and robustness of the strategy; in terms of energy efficiency, the rhythm control strategy exhibits lower transportation energy consumption in the medium and high speed walking and running tasks, and has good energy efficiency advantage. BRIEF DESCRIPTION OF DRAWINGS

[0028] Figure 1 is a practical application deployment schematic diagram of the Walk2Run of the present application on a Unitree G1 platform.

[0029] Figure 2 is a control framework schematic diagram of the Walk2Run of the present application.

[0030] Figure 3 is a human joint mapping relationship schematic diagram of the present application.

[0031] Figure 4 is a center of mass linear velocity estimator schematic diagram of the present application.

[0032] Figure 5 is a step frequency effect schematic diagram of the present application.

[0033] Figure 6 is a single speed instruction tracking comparison experiment schematic diagram of the present application.

[0034] Figure 7 is a variable speed instruction tracking comparison experiment schematic diagram of the present application.

[0035] Figure 8 is an energy transmission cost comparison schematic diagram of the present application.

[0036] Figure 9 is an energy transmission consumption comparison schematic diagram of the present application.

[0037] Figure 10 is a foot landing effect schematic diagram of the present application.

[0038] Figure 11 is a Walk2Run walking and running switching experiment effect schematic diagram of the present application, wherein (a) is a walking effect in a training environment, (b) is a walking effect in Sim2Sim, (c) is a walking effect in a real scene, (d) is a natural walking and running switching effect, (e) is a running effect in a training environment, (f) is a running effect in Sim2Sim, and (g) is a running effect in a real scene.

[0039] Figure 12is a comparison diagram of the centroid state estimation method of the present application.

[0040] Figure 13 is a diagram of the training data distribution and output frequency of the rhythm pattern generator of the present application.

[0041] Figure 14 is a diagram of the cross-speed comparison of the step size characteristics under the constant frequency control and rhythm modulation of the present application.

[0042] Figure 15 is a diagram of the motion performance comparison between the constant frequency and rhythm control in the high-speed scenario of the present application, where (a) is a diagram of the motion performance of the constant frequency in the high-speed scenario, and (b) is a diagram of the motion performance of the rhythm control in the high-speed scenario.

[0043] Figure 16 is a diagram of the constraint performance analysis under the constraint reinforcement learning framework of the present application, where (a) is a diagram of the position constraint effect analysis, (b) is a diagram of the speed constraint effect analysis, and (c) is a diagram of the torque constraint effect analysis.

[0044] Figure 17 is a diagram of the speed tracking comparison under the variable speed instruction of the present application.

[0045] Figure 18 is a diagram of the transportation cost comparison under the variable speed working condition of the present application. DETAILED DESCRIPTION

[0046] In the following description, for purposes of explanation and not limitation, specific details are set forth such as particular architectures, technologies, techniques, etc. in order to provide a thorough understanding of the embodiments of the present application. However, it will be apparent to those skilled in the art that the present application can be practiced in other embodiments that depart from these specific details. In other instances, detailed descriptions of well-known systems, devices, circuits, and methods are omitted so as not to obscure the description of the present application with unnecessary detail.

[0047] It is to be understood that the terminology "including," "includes," "has," and / or "have," when used in this specification, specifies the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0048] It is also to be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting. As used in this specification and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.

[0049] With reference to the drawings accompanying the present application specification, the technical solutions in the embodiments of the present application are described clearly and completely, obviously, the described embodiments are only some of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without making creative efforts are within the scope of protection of the present application.

[0050] In the following description, many specific details are set forth in order to provide a thorough understanding of the present application, but the present application can also be practiced without other different ways from those described herein, and those skilled in the art can make similar extensions without departing from the connotation of the present application, therefore the present application is not limited by the specific embodiments disclosed below.

[0051] Embodiment one The embodiment provides a humanoid robot walk-run unified control method based on biological rhythm, and the method comprises the following steps: Step 1: capturing human walking and running action data, extracting rhythm characteristics from frequency and phase time domain; Step 2: based on the rhythm characteristics extracted in step 1, constructing a rhythm generator; Step 3: based on the policy gradient method of constraint reinforcement learning algorithm, constructing a walk-run unified control strategy, Step 4: the walk-run unified control strategy of step 3 adopts the rhythm information generated based on the rhythm generator of step 2, and then is trained by actor-critic, and a bionic motion control system Walk2Run under the driving of speed is constructed, natural gait transition and frequency adjustment are realized, the physical ability limit of the robot is met, and the motion fluency and energy efficiency are improved.

[0052] Further, the step 1 is specifically based on human walking and running action capture data, rhythm characteristics are extracted from frequency and phase time domain, and are mapped to robot joint space, to provide structured prior for subsequent rhythm perception control; The motion trajectories of human lower limbs key joints and pelvis are obtained through a high-precision motion capture system, and rhythm signals such as step frequency and phase are extracted by using Fourier analysis, and then these rhythm characteristics are mapped to robot skeleton space, and a “bionic rhythm mapping” is constructed.

[0053] In order to avoid the dependence on real action capture equipment, the CMU Mocap dataset is selected in the present application, the dataset contains a plurality of rhythm motion samples, after screening, 6 sequences (total length about 72 seconds) related to walking and running are selected, and are used to model the rhythm characteristics of human lower limbs.

[0054] The dataset represents human body using a skeleton model containing 31 joints, where each joint is modeled as a 3-DOF spherical joint and its rotation pose and relative position are recorded in local coordinate system. This way can depict the pose change and dynamic characteristics of human body in the process of motion.

[0055] The joint data is converted to the robot lower limb joint space through skeleton mapping. The main features extracted include joint pose jointpos∈R12, root linear velocity rootvel∈R3, and angular velocity rootang∈R3, which provide reference for rhythm modeling and human-like strategy learning.

[0056] 1) Joint trajectory information acquisition: Retargeting to obtain joint angles, mapping the local rotation (local_rot) of human motion capture data to the joint angle (dof_pos) of the robot. In the motion capture data, the pose information of each joint is a quaternion form, while the joint movement of the robot is a rotation around an axis. The local rotation information in the form of quaternion in the motion capture data needs to be mapped to the rotation angle of the robot joint. Since the joint movement of the robot can be generally regarded as rotation around a single axis, the quaternion representation needs to be converted to axis-angle representation (see formula 1 for details).

[0057] (1) Where θ is the rotation angle and v is the rotation axis. According to the above formula, the quaternion can be converted to and :

[0058]

[0059] Different parts of a general humanoid robot correspond to different degrees of freedom. For humanoid robots, the degrees of freedom corresponding to each joint are different. The hip is the same as human beings, with 3 degrees of freedom corresponding to 3 joints; the knee joint has 1 degree of freedom corresponding to 1 joint; the ankle joint has 2 degrees of freedom corresponding to 2 joints. According to this relationship, the joint trajectory can be mapped according to formula (4):

[0060] 2) Rhythm information acquisition: The relationship between the human center of mass motion state and the motion frequency is analyzed by using fast Fourier transform (FFT) as shown in equation (5) to reveal the human motion rhythm information. FFT can effectively convert a complex time domain signal into a frequency domain representation, thereby revealing the frequency components therein, which is crucial for the extraction of human gait cycles.

[0061]

[0062] wherein, is the frequency spectrum on the frequency is the original time domain signal, is the time length. By extracting the main peak of the spectrum, as shown in equation (6), the most significant gait rhythm frequency can be obtained as a key reference for subsequent rhythm modeling.

[0063]

[0064] 3) Center of mass motion estimator: Since the center of mass linear velocity cannot be directly obtained by sensors such as IMU, a center of mass linear velocity estimator based on multi-layer perception (MLP) is constructed in this experiment to complete the center of mass state information and serve as a key input for the rhythm generator to depict the real motion state of the robot. The estimator takes the continuous 15-frame historical observation as input, including the angular velocity and linear acceleration recorded by the IMU, as well as the joint state at the corresponding time. In the training stage, the observation with added noise is used as input, and the linear velocity true value in the simulation is used as the supervision signal, and the mean square error (MSE) loss is used for optimization. Combined with the angular velocity measured by the IMU, the complete estimation of the robot center of mass motion state can be finally realized.

[0065] Further, the step 2 is specifically to design a RhythmGenerator inspired by the central pattern generator (CPG), which can dynamically generate a rhythm signal matching the frequency according to the current motion state of the robot to drive the generation of periodic gait; The center of mass state estimator is trained using multiple frames of joint angles and IMU data to achieve accurate perception of the robot center of mass linear velocity and provide dynamic context for the rhythm generator. Based on this, a central pattern generator (CPG) style rhythm generator is trained with the center of mass velocity as input and the human-like rhythm as label, so that the robot can autonomously generate a step frequency pattern matching the human in each motion state.

[0066] According to the center of mass motion state and joint frequency obtained through the above step, a rhythm generator is established to generate corresponding motion rhythm according to the motion of the robot. In the present application, the rhythm generator is based on multi-layer perception, as shown in equation:

[0067] wherein denotes the current time, denotes the frequency, denotes the current center of mass motion state, including the center of mass linear velocity and angular velocity, is a fully connected neural network. As shown in equation (8), the rhythm generator is trained using a mean square error (MSE) loss function:

[0068] wherein is the predicted frequency, is the true frequency. By using the backpropagation algorithm, the network is trained to reduce the difference between the predicted frequency and the true frequency, thereby realizing rhythm cycle generation.

[0069] After inferring the frequency information according to the rhythm generator, the phase information is further calculated. In order to ensure smooth transition of the phase in the case of frequency change, the following formula is used for phase calculation:

[0070] This formula can ensure smooth transition of the phase when the frequency changes, thereby avoiding discontinuous gait switching.

[0071] To verify the effectiveness of the method of the present application, a robot variable speed walking experiment is designed. In the experiment, the robot moves at a speed of 0.5 m / s, 1.0 m / s and 1.5 m / s, respectively, and the experimental results are shown in Figure 5 Whenever the robot changes speed, the rhythm generator will automatically adjust the rhythm according to the motion state of the center of mass, so that the motion frequency gradually increases with the increase of the speed.

[0072] By comparing with the Humanoid-Gym method based on a fixed frequency sinusoidal clock signal, it is found that the motion method based on the rhythm generator proposed in the present application shows a significant efficiency improvement in tracking speed. As shown in Figure 6 The control method based on the rhythm generator can respond more quickly and reach the expected motion speed instruction.

[0073] Under the high-speed walking working condition of 1.5 m / s, the method of the present application is 1.98 seconds faster than Humanoid-Gym, with an improvement of 40%. This difference shows that the control strategy proposed in the present application can provide more accurate and efficient motion execution under high-speed motion.

[0074] As Figure 7As shown, the response speed of the robot in tracking signals of 0.5 m / s, 1.0 m / s and 1.5 m / s was further tested during the variable-speed walking process. The experimental results show that the method of the application is 0.4 seconds, 0.3 seconds and 0.6 seconds faster than Humanoid-Gym respectively, and is increased by 40%, 30% and 45% respectively. This shows that the rhythm generator can better adapt to the change of speed instruction, improve the response speed of the robot, and optimize the stability of motion control.

[0075] In addition, the walking method based on natural rhythm also shows superiority in energy consumption. Through the test of transportation cost, as shown in Figure 8 and 9 , it is found that the motion based on natural rhythm reduces the energy consumption by about 10% compared with the Humanoid-Gym method based on fixed frequency. This result proves that the rhythm generation method proposed in the application not only has significant advantages in speed and efficiency, but also can effectively reduce the energy consumption of the robot.

[0076] Finally, the experimental results clearly show that the method based on natural rhythm can generate a more stable gait at different motion speeds. At different speeds, the Humanoid-Gym method based on fixed frequency may have the problem that one foot always leads and the gait is irregular, as shown in Figure 10 . The gait control method based on rhythm generation effectively avoids this problem and ensures the balance and stability of the gait.

[0077] Further, the constraint reinforcement learning algorithm of step 3 is specifically to introduce a smooth barrier function, improve the constraint reinforcement learning algorithm to meet the range limit, and improve the stability and reliability of the strategy training process.

[0078] The walk-run unified control strategy of step 3 is specifically to use a constraint-based policy gradient method to construct a walk-run unified control strategy.

[0079] 1) Constrained Markov Decision Process: Reinforcement learning problems are built on Markov Decision Processes (MDP), which is a tuple . Wherein, is a finite state sequence, is a finite action sequence, is a reward function, is a state transition probability function, is the initial state probability, and λ is the initial time discount. The goal of MDP is usually to maximize the cumulative reward , as shown in equation (10).

[0080]

[0081] Generally speaking, From the value function Approximately, as shown in equation (11):

[0082] The action value function is represented as shown in equation (12):

[0083] The degree of advantage of the actions that can be taken in the current state can be characterized by the advantage cost function, as shown in equation (13):

[0084] CMDP is an extension of MDP by introducing constraint C, and its goal is to maximize the cumulative reward. At the same time, ensure all constraints The condition is satisfied, as shown in equation (14):

[0085]

[0086] 2) Optimization of constrained strategies: When solving the CMDP problem, the strategy is... It is the result of optimization. Although it is possible to guarantee during the optimization process... Satisfy constraints However, it is difficult to directly guarantee the updated strategy. This also satisfies the constraints. Therefore, the CPO (Constrained Policy Optimization) literature demonstrates through inference that there is an upper and lower bound on the performance difference between the two policies, as shown in equation (15):

[0087] The specific forms of the upper and lower bounds are shown in equation (16):

[0088] in, It is the maximum deviation from the constraint on returns. It is used to limit the magnitude of policy updates. Divergence, its function is to ensure and It won't deviate too much during updates. Based on The dominance function. Based on the above inference, the CMDP problem can be constructed in the following form:

[0089] (17) The above configuration will Instead , so as to indirectly control Satisfy the constraints. Most modern policy optimization methods (such as PPO and TRPO) can ensure that the distance between the two is small by limiting the And KL divergence between them by using CLIP or confidence domain method. Therefore, the constraint condition can be further simplified, as shown in equation (18):

[0090] However, the hard constraint is not convenient for calculation. In order to further simplify the optimization, the traditional method uses the Lagrange multiplier method to combine the reward and the constraint into an optimization target, as shown in equation (19):

[0091] Where: .

[0092] But the traditional Lagrange multiplier method is not conducive to solving the application scene of humanoid robots that need multiple constraints. Referring to the IPO method, soft constraints are introduced through barrier functions. The barrier function has the property as shown in equation (20):

[0093] To ensure the feasibility of calculation, the barrier function is often approximated by a smoothing function (log). Using it, a policy optimization problem is established based on the PPO method. Where L CLIP The confidence domain is considered for the approximation of In the PPO method, as shown in equation (21):

[0094] Where is a hyperparameter that controls the smoothing degree. At Optimal, take the derivative of both sides of the above objective function to get the following optimization condition, as shown in equation (22):

[0095] From the perspective of the Lagrange multiplier method, the multiplier Corresponding to each constraint, the optimal solution using the barrier function is , as shown in equation (23):

[0096] Further analysis of the dual gap shows that the optimal solution of the soft constraint The performance value of the soft constraint is usually less than the optimal solution of the hard constraint h i.e.

[0097] It can be seen that the accuracy of the barrier function and the optimization result are closely related to the hyperparameters and the number of constraints By reasonably adjusting and , the soft constraint solution can be made as close as possible to the hard constraint solution , so as to effectively trade off the calculation cost and the constraint satisfaction degree. In addition, there is a cost term in the calculation process, which has a possibility of being greater than 0, which does not conform to the log definition and will cause the solution to fail, so the barrier function is modified as shown in equation (25):

[0098] This modification ensures that the barrier function is still defined when , and smoothly transitions to the negative value region.

[0099] 3) Constraint term design: To ensure that the policy output meets the physical motion ability of the robot body, the present application explicitly introduces three types of constraint terms in the training process: joint position, joint velocity and joint torque limit. Therefore, the hyperparameter m is 3, and the hyperparameter t is defined as 2000. Each constraint is constructed in the form of violation ratio, that is, the frequency of all joints exceeding the physical limit is counted as the loss signal of the constraint. Each constraint has the same weight in the total loss, and the specific definition is shown in Table 1.

[0100] Table 1 Constraint function design

[0101] Further, the actor-critic in step 4 is specifically, the training framework is based on the Actor-Critic architecture, wherein the Critic network is further divided into two sub-modules: one sub-module is used to evaluate the overall reward, and the other sub-module is used to model multiple cost terms respectively, to express the violation degree of the robot position, velocity and torque limit; The Walk2Run in step 4 is used to realize natural gait transition and frequency adjustment, while meeting the physical ability limit of the robot, improving the motion fluency and energy efficiency.

[0102] Further, the improved constraint reinforcement learning algorithm is specifically, the control problem is formalized as a constrained Markov decision process, and the improved barrier function structure is combined to realize smooth control of the constraint boundary in the training process.

[0103] Further, to guide the generation of human-like motion trajectories for the unified walk-run control strategy, a discriminator network based on GAN is introduced to evaluate the human-likeness of the robot's current motion with human joint trajectories as reference, and the discrimination results are used as additional reward signals to encourage the strategy to generate more natural motion behaviors.

[0104] Further, the walk-run unified control strategy takes rhythm information as auxiliary input and dynamically adjusts the motion frequency based on the current center of mass motion state, enabling the robot to adaptively switch between walk and run modes according to task goals and body states.

[0105] The walk-run unified motion control strategy includes: 1) Reward function design based on rhythm information To effectively integrate rhythm information into robot motion, a footfall frequency planning based on phase information is designed. This planning is first defined based on a gait mask matrix, which represents the expected motion state of each joint of the robot within a specific time step. As shown in equations (26), (27), and (28):

[0106]

[0107]

[0108] According to this mask matrix, a rhythm reward function is designed as shown in equation (29):

[0109] where, represents the robot's foot contact situation, represents the expected state of the current gait mask. When the robot's foot contact situation is consistent with the rhythm gait mask, a positive reward is given, otherwise a penalty is given. This ensures that the robot moves according to the generated rhythm.

[0110] 2) Trajectory feature learning of motion adversarial prior To guide the strategy to generate more natural actions with human characteristics, a style reward mechanism based on adversarial learning is introduced. Specifically, a discriminator network is constructed to determine whether a given state transition pair is from human motion data. The discriminator is trained by least squares loss, aiming to judge real samples as 1 and strategy-generated samples as -1, while introducing gradient penalty to improve training stability, as shown in equation (30):

[0111] The last term of formula (30) aims to penalize the gradient, with a coefficient , to reduce the tendency of the discriminator to assign non-zero gradients on the manifold of real data samples, to reduce oscillations in GAN training and improve training stability. The human-like style reward is defined as shown in formula (31), where

[0112]

[0113] where an additional offset and scaling are applied to limit the reward to the range [0, 1].

[0114] 3) Reward design The present application designs a composite reward function composed of three types of reward terms, namely: human-like style reward , rhythm reward and basic reward . Among them, and have been introduced in the foregoing, and the basic reward includes multiple sub-terms for constraining the physical rationality of the robot and the control smoothness, as shown in Table 2.

[0115] Finally, the total reward function is obtained by integrating each reward and its weight ω, as shown in formula (32):

[0116] Table 2 Other reward settings

[0117] Although reinforcement learning performs well in high-dimensional control tasks, traditional methods often ignore the modeling of dynamic constraints of the robot, which easily produces unreasonable or even dangerous action outputs. To improve the physical feasibility of the policy and the stability of the training, the present application introduces a reinforcement learning method based on constraint optimization, which effectively controls the risk of violation of joint position, velocity and torque constraints while ensuring the performance of the policy, avoiding the reliance on complex manual reward design. Therefore, the control problem is formalized as a constrained Markov decision process, and combined with an improved barrier function structure, smooth control of the constraint boundary during the training process is achieved.

[0118] Based on the first embodiment, the experiment is specifically, 1) Experimental setup In the present application, the humanoid robot platform G1 (29 degrees of freedom version) developed by Yushu Technology is used as the experimental platform. The deployment system is implemented based on Python, and the core modules including state estimation, rhythm generator and policy network are executed by Libtorch inference, and integrated with the robot bottom control interface through the official SDK of Yushu, to realize online deployment and operation of the algorithm on the actual platform.

[0119] In view of the limitations of real robot platform, such as difficulty in obtaining true speed value and uncontrollable experimental environment, the experimental design is divided into two parts: simulation experiment and real robot deployment experiment. In order to make the simulation experiment results as close as possible to the real deployment situation, the simulation experiment is not directly evaluated in the training environment Isaac Gym, but a high-fidelity simulation environment based on MuJoCo is constructed. Existing research shows that MuJoCo is closer to the actual robot system in terms of dynamics modeling accuracy and physical consistency

[41] , so the robot model and task configuration consistent with the training environment are reconstructed in it to obtain more realistic evaluation results.

[0120] Specific experimental projects include the following: Design comparative test to evaluate the effect of state centroid state estimator and rhythm generator. Design comparative test to compare the accuracy of centroid estimation based on MLP and LSTM with the speed estimation performance of Kalman filter. Analyze the output frequency of rhythm generator under different working conditions and its ability to guide the generation of natural rhythm movement, and then compare the movement effect analysis of natural rhythm and fixed rhythm.

[0121] Design comparative test to evaluate the effect of constraint reinforcement learning strategy. By comparing the difference in motion limit violation rate between the improved strategy proposed in this paper and the traditional PPO, the ability to improve the movement ability while maintaining feasibility is verified.

[0122] By setting different target speed sequences, test whether the robot can realize stable walk-run switching according to the rhythm adjustment mechanism, and compare the adaptability difference of non-rhythm guide strategy under speed change. This experiment is carried out simultaneously on MuJoCo simulation and real robot platform to evaluate the transferability and robustness of the algorithm.

[0123] By measuring energy consumption performance in different constant speed and variable speed tasks, energy efficiency analysis experiment is carried out to evaluate the influence of introducing rhythm mechanism on movement efficiency, and further verify the energy efficiency advantage of the system.

[0124] 2) Centroid state estimator and rhythm generation effect This part of the experiment mainly verifies two parts, first is the effect of the centroid state estimator and the rhythm generator. First, to verify the effectiveness of the centroid state estimator, based on two kinds of neural network structure respectively trains the estimation model: one adopts multilayer perceptron (MLP), the other is based on long short-term memory network (LSTM). The training target is to regress the current centroid linear velocity of the robot. The experimental scene is set in the MuJoCo environment to control the humanoid robot to perform variable speed motion tasks, and the performance of the trained estimator is evaluated. At the same time, in order to make a comparison, the extended Kalman filter (EKF) based on IMU data is introduced as a representative of the traditional method. As shown in Figure 12 , the estimator trained based on MLP shows the best estimation accuracy and stability in each speed interval, and can better capture the motion state of the robot. The estimator trained by LSTM performs well in the low speed (about 0.5m / s) interval, but the accuracy decreases slightly in the high speed scene. In contrast, the EKF has obvious drift in long time running. In addition, the performance of the three methods in YZ axis speed estimation is relatively close, and the difference is relatively small.

[0125] Secondly, in the rhythm generation module, the training data covers three typical motion states: slow walking area (0.0-0.3m / s, 0.3-0.32Hz), fast walking area (0.6-0.9m / s, 0.80-0.95Hz) and fast running area (1.6-2.4m / s, 0.95-1.40Hz), as shown in Figure 13 . In the evaluation, the input speed is limited to the range of 0-2m / s. The experimental results show that in the interval where the speed is less than 1m / s, the generated rhythm frequency increases positively. While the speed exceeds 1m / s, the step length increases significantly due to the transition of gait from walking to running, resulting in a flat growth of frequency; when the speed exceeds 1.5m / s, the frequency rises again. It should be pointed out that the generation of robot frequency depends on the three-dimensional speed and angular velocity of the robot centroid, but in order to show the effect, Figure 13 , the relationship between the generated rhythm and the x-axis speed is shown.

[0126] In order to further evaluate the regulation effect of rhythm signal on motion behavior, based on the method described in

[41] , the Humanoid-Gym control framework is modified to support rhythm reward. In the original Humanoid-Gym, the frequency is a fixed constant, while after adding the rhythm reward, the control strategy can adaptively adjust the rhythm output according to the centroid state to perform motion. It is found that with the increase of speed, the introduction of rhythm signal can significantly improve the step length consistency and gait stability, and the generated step length is smaller than that of the fixed frequency strategy, avoiding the limit activation of joint space, thereby providing greater redundancy space and energy efficiency potential for high speed motion.

[0127] 3) Constraint effect analysis To demonstrate the effectiveness of the constrained reinforcement learning framework, the value of the loss function with respect to the constraint term during training is compared between PPO and the proposed algorithm, as shown in Figure 16 To ensure fairness, all policies use the same network structure, training hyperparameters, and task settings, with the only difference being whether the constraint term is introduced in the loss function. Figure 16 As shown in (a), during training, the algorithm effectively suppresses the behavior of the robot joint motion position exceeding the safety limit compared to the PPO algorithm, demonstrating stronger convergence and boundary awareness. However, as shown in Figure 16 (b) and (c), in terms of speed and torque, the PPO-based method has a much lower probability of violating the limit than the position, and the algorithm is tolerant to both aspects, with similar results to the PPO method.

[0128] To further quantify the degree to which the policy adheres to physical constraints at the execution level, the violation rate of different policy outputs in the later training period is calculated, as shown in Table 3. The violation rate is defined as the proportion of actions that violate various constraints in the policy output within a fixed motion period, covering performance under different motion types (walking and running). The results show that in the running scenario, the violation rate of the PPO strategy without constraints is 8%, while the violation rate after introducing constraints is reduced to 3.7%, effectively avoiding potential motion abnormalities and hardware damage risks. In terms of speed and torque constraints, both methods can maintain a low violation rate, indicating that the constraint mechanism improves the physical feasibility of the policy without compromising the original control performance. It is worth noting that due to the inability to extract the true reference value of joint torque in the dynamic compensation data, the Dataset column for the tau item is left blank.

[0129] Table 3 Violation rate of policy output under different motion behaviors

[0130] 4) Variable speed experiment To verify the effectiveness of the proposed motion control framework in actual deployment and the contribution of the rhythm module, comparative and ablation experiments are designed to test the system's response ability and walk-run natural switching behavior through speed change commands.

[0131] First, as shown in Figure 11 (d), the algorithm is successfully deployed in a real environment, achieving natural walk-run switching of the humanoid robot during variable speed. To more systematically demonstrate the effect, Figure 11 (a)-(c) shows the robot's posture when walking at 1.0 m / s in the training environment, Sim2Sim environment, and real environment. During this process, the robot maintains a straight-knee gait, with stable movements close to the natural walking style of humans. Figure 11(e)-(g) show the posture when running at 1.5 m / s, the robot shows a larger stride and obvious leg lifting action, which is consistent with the characteristics of human running, and the behavior in the three environments remains consistent, verifying the migration ability of the strategy on the real platform and the generalization ability of the rhythm generator.

[0132] To further quantify the role of the rhythm module in speed response, a speed change experiment was conducted in the Sim2Sim environment to test the response of the system at three target speeds (0.5 m / s, 1.0 m / s, 1.5 m / s). As shown in Figure 17 , three algorithm configurations were compared: (1) the original Humanoid-Gym controller (fixed frequency), (2) the complete Walk2Run controller, and (3) the Walk2Run controller without the rhythm module. The results show that in the low-speed scenario (0.5 m / s), the response of Humanoid-Gym is slightly better. In the medium and high-speed scenarios (1.0 m / s and 1.5 m / s), the Walk2Run controller shows faster speed tracking ability. The Walk2Run without the rhythm module performs the worst at all speed segments, indicating that the rhythm regulation mechanism plays an important role in improving the speed adaptability of the system. Overall, the rhythm-aware control framework proposed in this paper not only realizes smooth and natural walk-run transition in simulation and reality, but also exhibits superior response ability and stability under dynamic speed instructions.

[0133] 5) Energy efficiency analysis To further analyze the effectiveness of the motion framework, the COT of the robot was analyzed based on the variable speed experiment. COT is a common evaluation system for transportation energy consumption of robots as shown in equation (33).

[0134]

[0135] where, and represent the torque and angular velocity of the jth joint at the ith time, is the mass of the robot, is the displacement of the robot's center of mass along the forward direction during the task.

[0136] In the variable speed experiment, the COT curve is shown in Figure 18 . The results show that in the initial stage of acceleration, both Walk2Run and Walk2Run without the rhythm module show lower energy consumption; but in the low-speed steady state stage, the energy efficiency of the Humanoid-Gym method is slightly better. As the speed rises to the medium and high-speed range, the methods of and without the rhythm gradually exceed Humanoid-Gym in energy efficiency, showing better energy utilization efficiency. ​

[0137] It is worth noting that the rhythmic algorithm performs more stably in terms of frequency adjustment and gait stability during speed switching, and thus is superior to the version without rhythm in terms of overall energy efficiency.

[0138] In addition, COT measurements were also conducted under different constant speed conditions, and the specific results are shown in Table 4. It can be seen that the performance of Humanoid-gym is superior to that of Walk2Run controller at low speed. However, Walk2Run performs better at medium and high speeds.

[0139] Table 4 Comparison of COT of three methods at different speeds

[0140] Embodiment Two The embodiment provides a humanoid robot walk-run unified control system based on biological rhythm, which uses a humanoid robot walk-run unified control method based on biological rhythm as described in embodiment one, and the system comprises: a rhythm feature extraction module: capturing human walking and running action data, and extracting rhythm features from the frequency and phase time domain; a rhythm generator: based on the rhythm features extracted in step 1, the construction of the rhythm generator is carried out; a walk-run unified control strategy module: based on the policy gradient method of constraint reinforcement learning algorithm, the walk-run unified control strategy is constructed, a Walk2Run control module: the walk-run unified control strategy uses the rhythm information generated by the rhythm generator, and then uses actor-critic for training, which jointly constructs the bionic motion control system Walk2Run under speed driving, realizes natural gait transition and frequency adjustment, meets the physical ability limit of the robot, and improves the motion fluency and energy efficiency.

[0141] Embodiment Three The embodiment is an application of the humanoid robot walk-run unified control method based on biological rhythm as described in embodiment one, and is characterized in that it is applied to the field of humanoid motion of robots, realizes the walking mode of the humanoid robot with the lowest energy consumption, and has good engineering adaptability and expansion value.

Claims

1. A circadian rhythm-based humanoid robot walk-run unified control method, characterized by, The method comprises the following steps: Step 1: capture human walking and running action data, extract rhythm features from frequency and phase time domain; Step 2: based on the rhythm features extracted in step 1, construct a rhythm generator; Step 3: based on the policy gradient method of constraint reinforcement learning algorithm, construct a walk-run unified control strategy; Step 4: the walk-run unified control strategy of step 3 adopts the rhythm information generated based on the rhythm generator of step 2, and is trained by actor-critic, which jointly constructs a biomimetic motion control system Walk2Run under speed driving, realizes natural gait transition and frequency regulation, and meets the physical ability limit of the robot, and improves the motion fluency and energy efficiency.

2. The method of claim 1, wherein, The step 1 is specifically based on human walking and running action capture data, extracts rhythm features from frequency and phase time domain, and maps to robot joint space, providing structured prior for subsequent rhythm perception control; The motion trajectories of human lower limbs and pelvis are obtained by a high-precision motion capture system, and the rhythm signals are extracted by Fourier analysis, and then these rhythm features are mapped to the robot skeleton space to construct "bionic rhythm mapping".

3. The method of claim 1, wherein, The step 2 is specifically to dynamically generate rhythm signals matching the frequency according to the current motion state of the robot to drive periodic gait generation; The rhythm generator is trained with center of mass velocity as input and human rhythm as label, so that the robot can autonomously generate a step frequency pattern matching human in each motion state.

4. The method of claim 1, wherein, The constraint reinforcement learning algorithm of step 3 is specifically to introduce a smooth barrier function to improve the constraint reinforcement learning algorithm to meet the value range limit and improve the stability and reliability of the policy training process.

5. The method of claim 4, wherein, The actor-critic in step 4 is specifically that the training framework is based on the Actor-Critic architecture, and the Critic network is further divided into two sub-modules: one sub-module is used to evaluate the overall reward, and the other sub-module is used to model multiple cost items respectively to express the violation degree of robot position, velocity and torque limit; The Walk2Run in step 4 is used to realize natural gait transition and frequency regulation, while meeting the physical ability limit of the robot, improving the motion fluency and energy efficiency.

6. The method of claim 4, wherein, The improved constraint reinforcement learning algorithm is specifically that the control problem is formalized as a constrained Markov decision process, and combined with the improved barrier function structure, the smooth control of the constraint boundary in the training process is realized; The improved barrier function is, （25） wherein, is a hyperparameter controlling the degree of smoothing, , is a cost term.

7. The method of claim 1, wherein, To guide the walk-run unified control strategy to generate motion trajectories with human style, a discriminator network based on GAN is introduced to evaluate the human degree of the current motion of the robot with human joint trajectory as reference, and the discrimination result is used as an additional reward signal to encourage the strategy to generate more natural motion behavior; Specifically, it includes designing a reward function based on rhythm information; Trajectory feature learning of motion confrontation prior; Design of reward.

8. The method of claim 1, wherein, The walk-run unified control strategy takes rhythm information as auxiliary input, and dynamically adjusts the motion frequency combined with the current center of mass motion state, so that the robot can adaptively switch between walking and running modes according to the task target and body state.

9. A biorhythm-based walking and running unification control system for a humanoid robot, characterized by, The system uses the walk-run unified control method of the humanoid robot based on biological rhythm according to any one of claims 1-8, and the system comprises: a rhythm feature extraction module: capturing human walking and running action data, extracting rhythm features from frequency and phase time domain; a rhythm generator: based on the rhythm features extracted in step 1, the construction of the rhythm generator; a walk-run unified control strategy module: based on the policy gradient method of the constraint reinforcement learning algorithm, the walk-run unified control strategy is constructed, a Walk2Run control module: the walk-run unified control strategy uses the rhythm information generated by the rhythm generator, and then uses actor-critic for training, which jointly constructs the bionic motion control system Walk2Run under the speed drive, realizes natural gait transition and frequency regulation, meets the physical ability limit of the robot, and improves the motion fluency and energy efficiency.

10. Use of a method for unified control of walking and running of a biorhythm-based humanoid robot according to any one of claims 1 to 8, characterized in that, Applied to the field of humanoid motion of robots, the humanoid robot realizes the walking mode with the lowest energy consumption, and has good engineering adaptability and expansion value.

Images