A quadruped robot motion control method based on deep reinforcement learning
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-05-27
- Publication Date
- 2026-08-11
AI Technical Summary
但由于大多数仿真器无法完全模拟现实环境的复杂性，这些控制器在从仿真到现实的迁移(sim-to-real transfer)过程中往往会有比较大的性能损失
[0035] 1. It can automatically learn motion strategies in simulation, reduce the difference between simulation and reality, and achieve robust motion of quadruped robots.
Smart Images
Figure CN118818968B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of robot control technology and relates to a motion control method for quadruped robots based on deep reinforcement learning. Background Technology
[0002] With the continuous development of robot control technology, the application of quadruped robots is becoming increasingly widespread. Compared with wheeled and tracked robots, quadruped robots have higher degrees of freedom and discrete footholds, which gives them a significant advantage in complex terrain operations. They can be widely used in search and rescue, reconnaissance, industrial inspection, and exploration of unknown environments.
[0003] However, the high degrees of freedom of quadruped robots also present significant challenges to motion control. In recent years, many model-based methods have been applied to the motion control of quadruped robots, but these methods often require careful design for various scenarios and are still prone to corner cases. In contrast, reinforcement learning methods can autonomously learn a motion controller through trial and error, achieving better control performance in a variety of scenarios. This approach often requires training in a simulator before deployment on a real quadruped robot. However, since most simulators cannot fully simulate the complexity of real-world environments, these controllers often suffer significant performance losses during the simulation-to-real transfer process. Summary of the Invention
[0004] This invention provides a motion control algorithm for quadruped robots based on deep reinforcement learning, which can automatically learn motion strategies in simulation, reduce the difference between simulation and reality, and achieve robust motion of quadruped robots.
[0005] The technical solution adopted in this invention is:
[0006] A motion control algorithm for a quadruped robot based on deep reinforcement learning, the specific steps of which are as follows:
[0007] S1. Establish a model of the quadruped robot, including a dynamic model for simulating the quadruped robot and a driver model for identifying and simulating the motor driver of the quadruped robot. The driver model adopts the Empirical Actuator Model (EAM).
[0008] S2. Describe the motion process of the quadruped robot as a Markov process, design a reward function, and use a deep reinforcement learning algorithm in the simulation environment established in S1 to optimize the motion policy of the quadruped robot using the Multi-Loss Proximal Policy Optimization (MLPPO) algorithm to train and obtain the motion controller.
[0009] S3. Deploy the trained motion controller onto the quadruped robot.
[0010] Furthermore, step S1 specifically includes the following steps:
[0011] S11. Establish a dynamic model of the quadruped robot, including the base mass and inertia tensor of the quadruped robot, the mass and inertia tensor of each joint link, the installation position and limit of each joint, and the collision model of each joint.
[0012] S12. Establish the actuator model for the quadruped robot. The mathematical expression for the empirical actuator model is as follows:
[0013]
[0014]
[0015] Where, q t and Let t be the position and velocity of the joint at time t. in For the input delay of the joint, For tt in Given the desired joint position at any given time, and For proportional-differential gain, t represents the desired output torque of the joint. out τ is the output delay of the driver torque. m This is the external characteristic curve of the motor; the external characteristic curve of the motor is the curve showing how the maximum output torque of the motor changes with the motor speed.
[0016] Furthermore, the model generated in step S11 is described by a Unified Robotics Description Format (URDF) file, and the robot model is simulated using Multi-Joint Dynamics with Contact (MuJoCo).
[0017] Furthermore, step S2 specifically includes the following steps:
[0018] S21. Describe the motion process of the quadruped robot as a Markov Decision Process (MDP), including the state space. Action space State transition function and reward function At time t, the parameterized policy π θ Actions are generated based on historical states. The environment updates its state based on the state transition function. And calculate the reward The goal of MDP is to maximize discount rewards and in Let γ be the expected value, and γ be the discount factor for the reward.
[0019] S22. Collect training data in the simulation environment. In each environment step, collect the current environment state, give the action of the current frame by the policy, convert the action into joint torque through the experience-driven model, run the simulation to obtain the state of the next frame, calculate the reward value based on the two frame states and actions, and save each state in the cache.
[0020] S23. After collecting a certain number of states, update the policy using Multi-Loss Proximal Policy Optimization (MLPPO); a parameterized policy can be represented as the conditional probability p of an action with respect to a state. θ (a t |s t ), where θ is the parameter of the strategy; the optimization objective of MLPPO is
[0021] minL ppo +w symmetry L symmetry +w smooth L smooth
[0022] Among them, L ppo The loss function for standard PPO;
[0023]
[0024] Among them, A t Let L be the advantage at time t, θ′ be the parameters of the strategy used to collect data, and ε be the pruning ratio; symmetry and L smooth The specific objective functions designed for quadruped robots are symmetric loss and smoothing loss, respectively. symmetry and w smooth These are the weights of the two objective functions. The two objective functions can be expressed as...
[0025]
[0026]
[0027] in, and These represent symmetrical mappings of states and actions, respectively.
[0028] Furthermore, the state space in step S21 Including robot linear velocity command c x ,c y Angular velocity command c r 3D base linear velocity v, 3D base angular velocity ω, 12D joint angle q, 12D joint angular velocity and the base roll angle ψ x and pitch angle ψ y .
[0029] Furthermore, the action space in step S21 The desired angles for the 12 joint angles.
[0030] Furthermore, the reward function in step S21 It is a weighted sum of a series of rewards These include rewards for tracking a given speed command, penalties for power and joint motion, and rewards for base posture and stable motion.
[0031] Furthermore, step S3 specifically includes the following steps:
[0032] S31. Obtain various state variables required by the policy network from the onboard sensors of the quadruped robot, where the robot's angular velocity and attitude are obtained from the robot's inertial measurement unit; the joint angles and joint angular velocities are obtained from the joint encoders; and the robot's linear velocity is obtained from the state estimator.
[0033] S32. The motion controller of the quadruped robot generates the desired joint position in real time through the inference strategy network at a fixed frequency, and then sends it to the joint motor to achieve whole-body control.
[0034] The beneficial effects of this invention are:
[0035] 1. It can automatically learn motion strategies in simulation, reduce the difference between simulation and reality, and achieve robust motion of quadruped robots.
[0036] 2. An Experience-Driven Model (EAM) is used to identify the actuators and motion strategies of the actual robot during training, thereby reducing the gap between simulation and reality.
[0037] 3. A reinforcement learning approach was used to achieve accurate tracking of a quadruped robot to given speed commands. Through a reinforcement learning framework with multiple loss functions, the symmetry and smoothness of the policy were optimized while maximizing the reward function. This policy enabled the quadruped robot to run at a high speed of 4.2 m / s, achieving a speed tracking error of less than 0.07 m / s over a wide command range. Furthermore, the policy exhibited excellent symmetry, smoothness, and aesthetics, and outperformed model-based controllers in terms of energy efficiency. Attached Figure Description
[0038] Figure 1 This is a schematic diagram of the structure of the present invention. Detailed Implementation
[0039] The present invention will be further described below with reference to specific embodiments, but the invention is not limited to these specific embodiments. Those skilled in the art should recognize that the present invention covers all alternatives, improvements, and equivalents that may be included within the scope of the claims.
[0040] In the description of this invention, it should be understood that the terms "center," "longitudinal," "lateral," "length," "width," "thickness," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," "clockwise," and "counterclockwise," etc., indicating orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, are only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the invention. Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined with "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, unless otherwise stated, "a plurality of" means two or more, unless otherwise expressly defined.
[0041] In this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.
[0042] In this invention, unless otherwise explicitly specified and limited, "above" or "below" the second feature can include direct contact between the first and second features, or contact between the first and second features through another feature between them. Furthermore, "above," "over," and "on top" of the second feature includes the first feature directly above or diagonally above the second feature, or simply indicates that the first feature is at a higher horizontal level than the second feature. "Below," "below," and "under" the second feature includes the first feature directly below or diagonally below the second feature, or simply indicates that the first feature is at a lower horizontal level than the second feature.
[0043] See Figure 1 This invention provides a motion control algorithm for quadruped robots based on deep reinforcement learning, the specific steps of which are as follows:
[0044] S1. Establish a model of the quadruped robot, including a dynamic model for simulating the quadruped robot and a driver model for identifying and simulating the motor driver of the quadruped robot. The driver model adopts the Empirical Actuator Model (EAM).
[0045] Specifically, the following steps are included:
[0046] S11. Establish a dynamic model of the quadruped robot, including the base mass and inertia tensor of the quadruped robot, the mass and inertia tensor of each joint link, the installation position and limit of each joint, and the collision model of each joint; the generated model is described by a unified robot description format URDF file, and the robot model is simulated using contact-based multi-joint dynamics MujoCo.
[0047] S12. Establish the actuator model for the quadruped robot. The mathematical expression for the empirical actuator model is as follows:
[0048]
[0049]
[0050] Where, q t and Let t be the position and velocity of the joint at time t. in For the input delay of the joint, For tt in Given the desired joint position at any given time, and For proportional-differential gain, t represents the desired output torque of the joint. out τ is the output delay of the driver torque. mThis is the external characteristic curve of the motor. The external characteristic curve of the motor is the curve showing how the maximum output torque of the motor changes with the motor speed.
[0051] Data sets were obtained by collecting the joint motor operation data of a quadruped robot at a frequency of 1000 Hz, and then iterated through t. in ∈[0,t max ], t out ∈[0,t max [The result is obtained by minimizing the mean square error through linear regression.] and The set of parameters with the smallest mean square error is used as the identification value of the joint motor.
[0052] S2. Describe the motion process of the quadruped robot as a Markov process, design a reward function, and use a deep reinforcement learning algorithm in the simulation environment established in S1 to optimize the motion strategy of the quadruped robot using a proximal policy optimization algorithm with multiple loss functions, and train to obtain a motion controller.
[0053] Specifically, the following steps are included:
[0054] S21. Describe the motion process of the quadruped robot as a Markov Decision Process (MDP), including the state space. Action space State transition function and reward function At time t, the parameterized policy π θ Actions are generated based on historical states. The environment updates its state based on the state transition function. And calculate the reward The goal of MDP is to maximize discount rewards and in Let be the mathematical expectation, and γ be the discount factor for the reward. The state space is... Including robot linear velocity command c x ,c y Angular velocity command c r 3D base linear velocity v, 3D base angular velocity ω, 12D joint angle q, 12D joint angular velocity and the base roll angle ψ x and pitch angle ψ y Action space The desired angles for the 12 joint angles.
[0055] The reward function in this embodiment It is a weighted sum of a series of rewards This includes rewards for tracking a given velocity command, penalties for power and joint motion, and rewards for base attitude and stable motion. The reward functions are shown in Table 1 below, where t... i Let be the swing time of the i-th leg.
[0056] Table 1 Reward Function
[0057]
[0058]
[0059] S22. Collect training data in the simulation environment. In each environment step, collect the current environment state, give the action of the current frame by the policy, convert the action into joint torque through the experience-driven model, run the simulation to obtain the state of the next frame, calculate the reward value based on the two frame states and actions, and save each state in the cache.
[0060] S23. After collecting a certain number of states, update the policy using Multi-Loss Proximal Policy Optimization (MLPPO). A parameterized policy can be represented as the conditional probability p of an action with respect to a state. θ (a t |s t ), where θ is the parameter of the policy. The optimization objective of MLPPO is
[0061] minL ppo +w symmetry L symmetry +w smooth L smooth
[0062] Among them, L ppo The loss function for standard PPO;
[0063]
[0064] Among them, A t Let L be the advantage at time t, θ′ be the parameters of the strategy used to collect data, and ε be the pruning ratio; symmetry and L smooth The specific objective functions designed for quadruped robots are symmetric loss and smoothing loss, respectively. symmetry and w smooth These are the weights of the two objective functions. The two objective functions can be expressed as...
[0065]
[0066]
[0067] in, and These represent symmetric mappings of states and actions, respectively. The symmetric mapping relationships are shown in Table 2.
[0068] Table 2 Symmetrical mapping of states and actions
[0069]
[0070] S3. Deploy the trained motion controller onto the quadruped robot.
[0071] Specifically, the steps include the following:
[0072] S31. Obtain various state variables required by the policy network from the onboard sensors of the quadruped robot, where the robot's angular velocity and attitude are obtained from the robot's inertial measurement unit; the joint angles and joint angular velocities are obtained from the joint encoders; and the robot's linear velocity is obtained from the state estimator.
[0073] S32. The motion controller of the quadruped robot generates the desired joint position in real time through the inference strategy network at a fixed frequency of 100Hz, and then sends it to the joint motor to achieve whole-body control.
Claims
1. A motion control algorithm for a quadruped robot based on deep reinforcement learning, the specific steps of which are as follows: S1. Establish a model of the quadruped robot, including a dynamic model for simulating the quadruped robot and a driver model for identifying and simulating the motor actuators of the quadruped robot. The driver model adopts an empirical driver model; the mathematical expression of the empirical driver model is as follows: ； ； in, and They are respectively The position and speed of the joint at all times, For the input delay of the joint, for Given the desired joint position at any given time, and For proportional-differential gain, The desired output torque of the joint; For the output delay of the driver torque, This is the external characteristic curve of the motor, which is the curve showing how the maximum output torque of the motor changes with the motor speed. S2. Describe the motion process of the quadruped robot as a Markov process, design a reward function, and use a deep reinforcement learning algorithm in the simulation environment established in S1 to optimize the motion strategy of the quadruped robot using a proximal policy optimization algorithm with multiple loss functions, and train to obtain a motion controller. Specifically, the following steps are included: S21. Describe the motion process of the quadruped robot as a Markov process, including the state space. Action space State transition function and reward function ;exist Timing, Parameterization Strategy Actions are generated based on historical states. The environment updates its state based on the state transition function. And calculate the reward The goal of MDP is to maximize discount rewards and ,in For mathematical expectation, This is the discount factor for the reward; S22. Collect training data in the simulation environment. In each environment step, collect the current environment state, give the action of the current frame by the policy, convert the action into joint torque through the experience-driven model, run the simulation to obtain the state of the next frame, calculate the reward value based on the two frame states and actions, and save each state in the cache. S23. After collecting a certain number of states, optimize the update policy using a proximal policy with multiple loss functions; a parameterized policy can be represented as the conditional probability of an action with respect to a state. ,in The parameters of the strategy are: MLPPO's optimization objective is... ； in, The loss function for standard PPO; ； in, for The advantage at this moment Parameters for the strategy when collecting data. For cutting ratio; and The specific objective functions designed for quadruped robots are symmetric loss and smoothing loss, respectively. and These are the weights of the two objective functions; the two objective functions can be expressed as... ； ； in, and These represent symmetrical mappings of states and actions, respectively. S3. Deploy the trained motion controller onto the quadruped robot.
2. The motion control algorithm for a quadruped robot based on deep reinforcement learning according to claim 1, characterized in that: Step S1 specifically includes the following steps: S11. Establish a dynamic model of the quadruped robot, including the base mass and inertia tensor of the quadruped robot, the mass and inertia tensor of each joint link, the installation position and limit of each joint, and the collision model of each joint. S12. Establish an experience-driven model for the quadruped robot.
3. The motion control algorithm for a quadruped robot based on deep reinforcement learning according to claim 2, characterized in that: The model generated in step S11 is described by a unified robot description format file, and the robot model is simulated using contact-based multi-joint dynamics.
4. The motion control algorithm for a quadruped robot based on deep reinforcement learning according to claim 1, characterized in that: State space in step S21 Including robot linear velocity commands Angular velocity command 3D base linear velocity 3D base angular velocity 12-dimensional joint angles 12-dimensional joint angular velocity and base roll angle and pitch angle .
5. The motion control algorithm for a quadruped robot based on deep reinforcement learning according to claim 1, characterized in that: Action space in step S21 The desired angles for the 12 joint angles.
6. The motion control algorithm for a quadruped robot based on deep reinforcement learning according to claim 1, characterized in that: Reward function in step S21 It is a weighted sum of a series of rewards This includes rewards for tracking a given speed command, penalties for power and joint motion, and rewards for base posture and stable motion.
7. The motion control algorithm for a quadruped robot based on deep reinforcement learning according to claim 1, characterized in that: Step S3 specifically includes the following steps: S31. Obtain various state variables required by the policy network from the onboard sensors of the quadruped robot, where the robot's angular velocity and attitude are obtained from the robot's inertial measurement unit; the joint angles and joint angular velocities are obtained from the joint encoders; and the robot's linear velocity is obtained from the state estimator. S32. The motion controller of the quadruped robot generates the desired joint position in real time through the inference strategy network at a fixed frequency, and then sends it to the joint motor to achieve whole-body control.