Quadruped robot gait reinforcement learning training method fusing bionic walking characteristics

Abstract

The invention discloses a bionic walking feature fused quadruped robot gait reinforcement learning training method, which comprises the steps of S1, bionic gait feature modeling for extracting key features from a motion mode of a natural quadruped animal and constructing a bionic template capable of directly guiding robot gait control; and S2, constructing a reinforcement learning training environment, and simulating diversified actual scenes by constructing a high-fidelity simulation platform. According to the method, key features are extracted from natural four-footed animal gaits, a bionic template library is constructed, and natural features such as nonlinear rhythm and dynamic symmetry of animal movement are fused into robot gait control, so that the problem of action mechanical stiffness caused by dependence on manual design of a track in a traditional method is effectively solved; movement of the robot is closer to natural biological gaits, impact generated when the robot interacts with the environment is reduced while movement energy consumption is reduced, and movement smoothness is improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of robot learning training, and particularly relates to a quadruped robot gait reinforcement learning training method fusing bionic walking characteristics. BACKGROUND

[0002] In recent years, with the rapid development of robot technology, quadruped robots have been widely used in practical scenarios such as security patrol, post-disaster search and rescue, and field material transportation due to their adaptability to complex terrains. Such applications have put forward strict requirements for the gait control of quadruped robots: not only the stability of the movement process needs to be ensured to avoid tilting or slipping, but also natural bionic movement postures need to be realized to reduce energy consumption and improve movement efficiency; at the same time, strong task adaptability is required, which can dynamically adjust the gait according to different terrains and task requirements. To meet the above requirements, researchers generally adopt a joint control method combining a trajectory generator (TG) and a reinforcement learning policy network. The core logic of this method is to provide a basic movement trajectory template through the trajectory generator, and then dynamically adjust the trajectory parameters by using the reinforcement learning policy network, so as to balance the structure and flexibility of movement. Among them, PMTG (Policies Modulating Trajectory Generators, policy modulating trajectory generators) is a classic framework in this field, which is widely used in the gait control research of quadruped robots. The core design idea of PMTG is based on “parameterized trajectory modulation”, which constrains the movement space of the robot through the preset trajectory template, avoids the exploration confusion caused by the too large action space in reinforcement learning, and to some extent, ensures the diversity and basic stability of the gait.

[0003] Although PMTG provides an effective solution for quadruped robot gait control, it still has significant defects in practical applications, mainly in the following three aspects: first, the action is mechanical, and lacks natural bionic characteristics. PMTG relies on a simplified trajectory function designed by humans as the foundation. This type of trajectory can only simulate the simple periodicity of limb movement, and cannot cover the complex nonlinear rhythm, dynamic symmetry, and landing rhythm in natural animal gait. Therefore, the robot movement is prone to show a mechanical posture, not only the energy consumption is high, but also the stability may be reduced due to the "unnaturalness" of the interaction with the environment. Second, the model training efficiency is low, and the convergence speed is slow. In PMTG, the core task of the reinforcement learning strategy network is to learn the adjustment rules for the trajectory generator parameters from the randomly initialized state. Since the adjustment space of the trajectory parameters (amplitude, frequency, phase, etc.) is large, and there is a complex coupling relationship between the parameters, the strategy network needs to explore a large amount of space in a high-dimensional space, resulting in an unstable learning process and slow convergence speed, which requires a large amount of computing resources and training time, making it difficult to meet the needs of rapid deployment in practical applications. Third, the generalization ability to different terrains is weak. The terrain differences faced by quadruped robots in actual scenes are significant (such as the friction difference between flat ground and sandy ground, and the support stability difference between steep slopes and gravel roads), and there are essential differences in the gait requirements of robots in different terrains (such as the need to reduce the foot end sinking depth in sandy ground and adjust the center of gravity inclination angle on steep slopes).

[0004] In summary, although the PMTG framework realizes the basic adjustment of the quadruped robot gait through the structure of "trajectory generator + strategy network", it still has obvious shortcomings in the three core dimensions of bionic constraints (naturalness), training efficiency (convergence speed), and terrain adaptation (generalization ability). These defects limit the application effect of quadruped robots in complex actual scenes, therefore, optimization and improvement of PMTG has become an important research direction in the field of quadruped robot gait control, and needs to break through the existing technical bottlenecks from the aspects of biological inspiration of trajectory design, efficiency improvement of training mechanism, and dynamic modeling of terrain-gait mapping. SUMMARY

[0005] The purpose of the present application is to overcome the technical defects of the current quadruped robot, such as unnatural generated action, slow model convergence speed, and poor generalization ability in different terrains, and to provide a quadruped robot gait reinforcement learning training method fused with bionic walking characteristics.

[0006] The purpose of the present application is achieved by the following technical scheme: a quadruped robot gait reinforcement learning training method fused with bionic walking characteristics, comprising the following steps: S1, bionic gait feature modeling, for extracting key features from the motion patterns of natural quadruped animals, and constructing a bionic template that can directly guide robot gait control; S2, build a reinforcement learning training environment, simulate diversified actual scenarios by building a high-fidelity simulation platform, and design a reward mechanism integrating bionic features to provide reasonable training scenarios and feedback signals for policy network learning; S3, policy training and optimization, train the policy network through the reinforcement learning algorithm, combine the dynamically adjusted bionic constraints, realize the autonomous optimization of the robot gait, and finally output the deployable control strategy.

[0007] Further, the "bionic gait feature modeling" in step S1 specifically includes the following steps: S11, collect and pre-process bionic gait data, according to the motion video of quadruped animals and the published animal gait data set, use the key point recognition algorithm OpenPose to automatically extract the main joint trajectory in the animal motion process, perform degree of freedom normalization and conversion, and convert the natural animal gait data into 12-degree-of-freedom control parameters consistent with the structure of the quadruped robot; S12, extract gait rhythm parameters, use pose estimation and trajectory analysis methods to extract key point time series data, obtain time series of gait cycle, joint angle, and landing sequence information; S13, build a bionic gait template library, use the extracted parameters to build a standard cycle trajectory function as a template, and fit the trajectory parameters into a periodic function containing amplitude, frequency, phase, and bias.

[0008] The step S2 specifically includes the following steps: S21, build a simulation environment based on a physics engine, and the terrain can be parameterized and dynamically generated during training; S22, deploy a quadruped robot model with 12 degrees of freedom, and install a state observation module; S23, set a basic reward function and introduce a bionic gait similarity index to generate a new reward function.

[0009] The calculation formula of the basic reward function in step S23 is wherein is the expected speed and angle matching term, is the body posture stability, is the energy consumption penalty term, , , is the respective weight term; the calculation formula of the bionic gait similarity index is wherein represents the phase state obtained by aligning the trajectories of the i-th leg of the robot in time, represents the corresponding phase state of the target bionic trajectory; the calculation formula of the new reward function is wherein The gait similarity index is weighted by a coefficient.

[0010] The step S3 specifically comprises the following steps: S31, a proximal strategy optimization algorithm is used to optimize the strategy network, and a pruning objective function is used to improve training stability; S32, a regular term weight is dynamically adjusted, and the formula is , wherein represents the weight of the gait similarity at the current time t, the initial time point of the bionic weight, an exponential decay function; S33, when the reward function converges or reaches a set condition, the trained strategy network weight and parameter setting are derived for deployment.

[0011] The 12-DOF control parameters in the step S11 are matched with the structure of the quadruped robot, and can accurately correspond to the motion control requirements of each joint of the robot.

[0012] The state observation module in the step S22 at least includes a joint encoder, an IMU, an RGB-D camera and a foot sensor, which are used to receive speed, angle, contact force, RGB image and depth information feedback.

[0013] The exponential decay function in the step S32 can reasonably control the weakening speed of the bionic constraint weight, and balance the relationship between stability and exploration in model training.

[0014] The periodic function in the step S13 has a function formula , wherein A is the amplitude, is the frequency, is the phase, and B is the bias.

[0015] Compared with the prior art, the present application has the following advantages and beneficial effects: (1) The present application extracts key features from the gait of natural quadruped animals and constructs a bionic template library, and integrates the nonlinear rhythm, dynamic symmetry and other natural characteristics of animal motion into robot gait control, effectively solving the problem of mechanical stiffness caused by relying on artificial design trajectory in traditional methods, making the robot motion closer to natural biological gait, reducing the impact when interacting with the environment while reducing the motion energy consumption, and improving the motion fluency.

[0016] (2) The application provides an initial motion framework for reinforcement learning by means of a bionic gait template, reduces the learning space of the policy network, and avoids the blindness of exploration starting from random initialization; at the same time, the pruning objective function of the proximal policy optimization algorithm and the dynamically adjusted bionic constraint weight are combined, the unstable factors in the training process are reduced, the model convergence speed is significantly accelerated, the calculation resource consumption is reduced, and the training efficiency is improved.

[0017] (3) The application builds a parameterized dynamic simulation environment containing flat ground, steep slope, sand, gravel and other terrains, so that the policy network can contact diversified scenes in training; and the bionic gait similarity index and the dynamically adjusted constraint mechanism are integrated in the reward function, so that the robot can adapt to the basic terrain relying on the bionic characteristics, and can also independently explore the optimal gait for special terrain, greatly improving the adaptation and generalization ability to different terrains.

[0018] (4) The body posture stability term and the expected speed matching term in the basic reward function ensure the basic stability and task target achievement of the robot motion; and the dynamically adjusted bionic constraint weight ensures stability through strong bionic guidance in the early training, and relaxes the constraint to allow the policy to explore better behaviors in the later period, so that the robot can maintain stability and flexibly adjust the gait to adapt to different task requirements (such as load adaptation during material transportation and rapid movement during search and rescue).

[0019] (5) The application converts the collected animal gait data into 12-degree-of-freedom control parameters consistent with the structure of the quadruped robot, and the amplitude, frequency and other parameters of the periodic trajectory function are matched with the joint motion characteristics of the robot, so that the bionic characteristics can directly act on the joint control of the robot, avoiding control deviation caused by parameter mismatch, and improving the accuracy and effectiveness of gait control. BRIEF DESCRIPTION OF DRAWINGS

[0020] Figure 1 It is a whole process schematic diagram of the application.

[0021] Figure 2 It is a process schematic diagram of bionic gait feature modeling of the application.

[0022] Figure 3 It is a process schematic diagram of constructing a reinforcement learning training environment of the application.

[0023] Figure 4 It is a process schematic diagram of policy training and optimization of the application. DETAILED DESCRIPTION

[0024] The application will be further described in detail below in conjunction with the embodiments, but the embodiments of the application are not limited thereto.

[0025] EMBODIMENT

[0026] As Figure 1 shown, the gait reinforcement learning training method of the quadruped robot with bionic walking characteristics described in this embodiment includes the following three steps in the whole.

[0027] S1, bionic gait feature modeling, is used to extract key features from the motion patterns of natural quadruped animals, and to construct a bionic template that can directly guide the gait control of the robot. The specific process of this step is shown in Figure 2 , which specifically includes the following steps: S11, collect and preprocess bionic gait data, according to the motion video of quadruped animals and the published animal gait data set, use the key point recognition algorithm OpenPose to automatically extract the main joint trajectory in the animal motion process, and perform degree of freedom normalization and conversion, convert the natural animal gait data into 12 degree of freedom control parameters consistent with the structure of the quadruped robot. This step can make the robot directly use the gait of natural animals as a template for motion, effectively solving the problem of "mechanical stiffness and lack of bionic characteristics" in traditional methods.

[0028] The preferred collection object of quadruped animals in this embodiment is a horse or a dog. OpenPose is a key point recognition algorithm based on deep learning, which is applied to the motion analysis of quadruped animals (such as horses and dogs) in this embodiment, which is a key node from video to quantitative data. It has the following two core functions, one is to identify key joints, by analyzing animal motion video, automatically positioning and marking the core joint nodes on the limbs (such as hip joint, knee joint, ankle joint, shoulder joint, etc.), outputting the two-dimensional / three-dimensional coordinates of these joints in each frame of image; the second is to generate trajectory data, which sorts the joint coordinates of consecutive frames by time, and obtains the motion path of each joint with time, i.e. "joint trajectory".

[0029] The degree of freedom normalization is used to eliminate the physical differences between animals and robots. The "degree of freedom" referred to here refers to the independent dimension of joint activity. Because of the natural differences in limb structure and joint activity range between animals and robots (for example: the activity angle of a dog's hip joint may be -40°-60°, while the corresponding joint of a robot can only move within the range of -30°-50° due to mechanical limitations), it is necessary to eliminate this difference through "normalization". This elimination includes two aspects of range mapping and scale unification, wherein the range mapping refers to scaling the motion angle (or displacement) range of the animal joint to the physical feasible range of the corresponding joint of the robot in proportion; the scale unification refers to eliminating the trajectory scale difference caused by the size of the animal individual (such as a big dog and a small dog) or the shooting distance, standardizing the joint motion amplitude to a relative value (such as a proportion relative to the length of the limb), ensuring that animal data from different sources can be processed uniformly.

[0030] The degree of freedom conversion refers to the mapping from the animal structure to the 12 degrees of freedom of the robot 12. Since the limb structure of the quadruped robot is usually "3 degrees of freedom per leg", it has 4 legs and thus has 12 degrees of freedom, which specifically correspond to the 3 core joints of each leg, i.e. the hip joint (front and rear swing, 1 degree of freedom), the hip joint (left and right swing, 1 degree of freedom), and the knee joint (flexion and extension, 1 degree of freedom). Specifically, it is to establish a one-to-one correspondence between the animal joints and the 12 degrees of freedom of the robot. When analyzing the movement of the left front leg of the dog, the "left front leg hip joint front and rear swing angle" extracted by OpenPose corresponds to the "left front leg hip joint front and rear swing degree of freedom" of the robot, the "left front leg knee joint flexion angle" corresponds to the "left front leg knee joint degree of freedom" of the robot, and so on. Finally, the 3 key joint movement data of each leg of the animal are converted into 3 degree of freedom control parameters of the corresponding leg of the robot, and 12 parameters of 4 legs form a control signal that can be directly read by the robot.

[0031] The natural animal gait data is effectively converted into 12 degree of freedom control parameters consistent with the structure of the quadruped robot, and the structure is . Wherein, represents the set of 12 joint angle control parameters corresponding to the quadruped robot bionic gait at time t, then respectively correspond to the target angles of the 12 joints of the quadruped robot at time t.

[0032] S12, extract gait rhythm parameters, use pose estimation and trajectory analysis method to extract key point time sequence data, obtain time sequence of gait cycle, joint angle, landing sequence and other information.

[0033] S13, construct a bionic gait template library, use the extracted parameters to construct a standard cycle trajectory function as a template, and fit the trajectory parameters into a periodic function containing amplitude, frequency, phase and bias.

[0034] This step is to convert the features extracted from the animal gait into a key step of reusable and standardized motion pattern of the robot, and the construction of the standard cycle trajectory function is to fit the scattered parameters into a periodic mathematical function, so that the joint motion law is calculable and reproducible.

[0035] Wherein, the periodic function formula is wherein A is an amplitude, is a frequency, is a phase, and B is a bias.

[0036] After the above step S1 is completed, step S2 of building a reinforcement learning training environment is performed, that is, a high-fidelity simulation platform is built to simulate diversified actual scenes, and a reward mechanism integrating bionic features is designed to provide reasonable training scenes and feedback signals for the learning of the policy network.

[0037] The specific process of step S2 is shown in Figure 3 and includes the following steps: S21, a simulation environment based on a physical engine is built, and the terrain in the training can be parameterized and dynamically generated. The essence of this step is to build a simulation platform. In this embodiment, a simulation environment based on a physical engine (Isaac Gym) is built, which contains typical terrains such as flat ground, steep slope, steps, sand, and gravel, to ensure that the terrain can be parameterized and dynamically generated in each training.

[0038] S22, a quadruped robot model with 12 degrees of freedom is deployed, and a state observation module is installed. The state observation module is a core component that allows the robot to "perceive the environment and its own state", which is equivalent to the sensory system of the robot. This state observation module includes at least four types of sensors: first, joint encoders are used to collect the angles and angular velocities of the 12 joints of the robot in real time; second, IMUs (Inertial Measurement Units) are used to detect the pitch angle, roll angle, and acceleration of the robot's body; third, RGB-D cameras are used to identify the environment "what" through RGB images and measure "how far" through depth information; and fourth, foot sensors are used to detect the contact force and contact state between the foot and the ground.

[0039] That is, the joint encoders, IMUs, RGB-D cameras, and foot sensors in the state observation module are used to receive feedback of speed, angle, contact force, RGB images, and depth information for input to the policy network.

[0040] S23, a basic reward function is set and a bionic gait similarity index is introduced to generate a new reward function. The purpose of this step is to guide the policy network of reinforcement learning to learn to adjust the gait through the reward function, and ultimately train a stable, efficient, natural, and bionic quadruped robot motion pattern.

[0041] This step is the core design for the robot to learn to walk, which includes three logical contents, namely, basic reward, introduction of bionic and new reward function. The "basic reward function" focuses on the "basic ability of the robot to complete the task", and the purpose of introducing the bionical gait similarity index is to give the robot a "goal of imitating the natural gait of animals" to make the robot "walk naturally like a living being"; finally, the new reward function is generated by weighting the "basic reward function" and "bionical gait similarity index" to obtain the final "new reward function".

[0042] The calculation formula of the basic reward function is , wherein is the expected speed and angle matching item, is the body posture stability, is the energy consumption penalty item, , , is the respective weight item. The calculation formula of the bionical gait similarity index is , wherein represents the phase state obtained after the end trajectory of the i-th leg of the robot is aligned in time, represents the corresponding phase state of the target bionical trajectory; and the calculation formula of the new reward function is , wherein is the gait similarity index weighting coefficient.

[0043] After completing step S2, step S3, i.e., policy training and optimization, is continued, the policy network is trained through the reinforcement learning algorithm, the dynamic adjustment of the bionic constraint is combined, the autonomous optimization of the robot gait is realized, and finally the deployable control strategy is output.

[0044] The specific process of this step S3 is shown in Figure 4 , which includes the following steps: S31, the policy network is optimized by using the proximal policy optimization algorithm, and the training stability is improved by using the clipping objective function.

[0045] This step is the core content of the reinforcement learning training of the quadruped robot gait, which makes the "policy network (the 'brain' of the robot) stable to learn to walk".

[0046] The so-called policy network (actor network) is the decision center of the robot, which receives the information (joint angle, terrain, body posture, etc.) of the state observation module and outputs the control instructions of 12 joints. The proximal policy optimization algorithm (PPO) is one of the reinforcement learning algorithms, which limits the update amplitude of the policy network by using the clipping objective function to achieve the purpose of improving the training stability.

[0047] S32, the weight of the regularization term is dynamically adjusted, and the formula is wherein represents the weight of gait similarity at the current time t, the initial time point of the bionic weight, an exponential decay function.

[0048] The purpose of this step is to make the model more intelligent to adapt to the needs of different stages during training, so as to ultimately improve the performance stability and practical application effect.

[0049] Among them, the regular term (such as L1 regularization, L2 regularization, weight decay, etc.) is a "constraint tool" in model training, which prevents the model from "overfitting" the training data by imposing a penalty (for example, limiting the absolute value of the parameter) on the model parameters.

[0050] The exponential decay function can reasonably control the weakening speed of the bionic constraint weight and balance the relationship between stability and exploration in model training.

[0051] S33, when the reward function converges or reaches the set condition, the trained strategy network weight and parameter setting are derived for deployment.

[0052] The convergence of the reward function in this step means that the average reward obtained by the robot in the training no longer improves significantly and tends to be stable, which means that the strategy is close to the optimal. The set condition is reached, which means that the target set by the human is reached. The trained strategy network weight and parameter setting are derived, which means that the optimal strategy network weight and parameter after training are derived, so as to transfer the trained model from the "training environment" to the "actual application scene" for running.

[0053] As described above, the present application can be well implemented.

Claims

1. A quadruped robot gait reinforcement learning training method fusing bionic walking features, characterized in that, The method comprises the following steps: S1, bionic gait feature modeling, for extracting key features from the motion patterns of natural quadruped animals, and constructing a bionic template that can directly guide the gait control of robots; S2, constructing a reinforcement learning training environment, by building a high-fidelity simulation platform, simulating diversified actual scenes, and designing a reward mechanism integrating bionic features, to provide reasonable training scenarios and feedback signals for the learning of the strategy network; S3, strategy training and optimization, training the strategy network through a reinforcement learning algorithm, combining dynamically adjusted bionic constraints to realize the autonomous optimization of the gait of the robot, and finally outputting a deployable control strategy.

2. The method of claim 1, wherein the method is characterized by: The "bionic gait feature modeling" in step S1 specifically comprises the following steps: S11, collecting and preprocessing bionic gait data, according to the motion videos of quadruped animals and the published animal gait data set, using the key point recognition algorithm OpenPose to automatically extract the main joint trajectories in the animal motion process, performing degree of freedom normalization and conversion, and converting the natural animal gait data into 12-degree-of-freedom control parameters consistent with the structure of the quadruped robot; S12, extracting gait rhythm parameters, using posture estimation and trajectory analysis methods to extract key point time series data, obtaining time series of gait cycle, joint angle, and landing sequence information; S13, constructing a bionic gait template library, using the extracted parameters to construct a standard cycle trajectory function as a template, and fitting the trajectory parameters into a periodic function containing amplitude, frequency, phase, and bias.

3. The method of claim 2, wherein the method is characterized by: Step S2 specifically comprises the following steps: S21, building a simulation environment based on a physics engine, the terrain in the training can be parameterized and dynamically generated; S22, deploying a quadruped robot model with 12 degrees of freedom, and installing a state observation module; S23, setting a basic reward function and introducing a bionic gait similarity index to generate a new reward function.

4. The method of claim 3, wherein the method is characterized by, The calculation formula of the basic reward function in step S23 is wherein is the expected speed and angle matching term, is the body posture stability, is the energy consumption penalty term, , , is the respective weight term; the calculation formula of the bionic gait similarity index is wherein represents the phase state of the end trajectory of the i-th leg of the robot after time alignment, represents the corresponding phase state of the target bionic trajectory; the calculation formula of the new reward function is wherein is the gait similarity index weighting coefficient.

5. The gait reinforcement learning training method of a quadruped robot with bionic walking characteristics according to any one of claims 1-4, characterized in that, Step S3 specifically comprises the following steps: S31, using the proximal policy optimization algorithm to optimize the strategy network, and using the clipping objective function to improve the stability of the training; S32, dynamically adjusting the regular term weight, the formula is wherein indicates the weight of gait similarity at the current time t, the initial time point of the bionic weight, exponential decay function; S33, when the reward function converges or reaches the set condition, the strategy network weight and parameter setting obtained by training are exported for deployment.

6. The method of claim 5, wherein the method is characterized by: The 12-degree-of-freedom control parameters in step S11 are adapted to the structure of the quadruped robot, and can accurately correspond to the motion control requirements of each joint of the robot.

7. The method of claim 6, wherein the method is characterized by: The "state observation module" in step S22 at least includes joint encoders, IMUs, RGB-D cameras, and foot sensors, for receiving speed, angle, contact force, RGB image, and depth information feedback.

8. The method of claim 7, wherein the method is characterized by: The exponential decay function in step S32 can reasonably control the weakening speed of the bionic constraint weight, balancing the stability and exploration in model training.

9. The method of claim 2-4, wherein the method is characterized by, The periodic function described in step S13 has a function formula of where A is an amplitude, is a frequency, is a phase, and B is a bias.

Images