Reinforced learning quadruped robot inverted pendulum stabilization method based on mixed state estimation

Abstract

The invention relates to the technical field of robots and automation, and discloses a quadruped robot inverted pendulum stability control method based on deep reinforcement learning, and the method specifically comprises the steps: fixing a first-order inverted pendulum on a body of a quadruped robot; motion of the quadruped robot is modeled as a partially observable Markov decision process, and a strategy output by the partially observable Markov decision process is trained through a proximity strategy optimization algorithm based on an actor-commentator model; the actor-commentator model comprises a strategy network and a value network; in the strategy training process, a domain randomization technology is adopted to randomize parameters of the environment; a reward function is designed to train the value network and provide a supervisory signal by comprehensively considering speed tracking reward, stability penalty, and inverted pendulum penalty terms. The end-to-end four-foot robot inverted pendulum stabilizing method based on mixed state estimation is designed, and the balance capacity and the stability performance of the robot are improved.

Description

Technical Field

[0001] The invention relates to the field of robots and automation technology, and in particular to a reinforcement learning quadruped robot inverted pendulum stabilization method based on mixed state estimation. Background Art

[0002] Existing research on quadruped robots is mostly focused on motion control, with less attention paid to the comprehensive optimization of the stability and control performance of the robot body, which limits its adaptability and application potential in higher-level tasks. As a classic nonlinear dynamic system, the inverted pendulum has always been regarded as a standard test platform for verifying the effectiveness and robustness of control methods. By fixing a first-order inverted pendulum on the body of a quadruped robot, the present invention intuitively demonstrates the robot's balance ability and stability performance in a dynamic environment, providing an innovative research perspective and technical means for improving the comprehensive performance of the robot.

[0003] The motion control methods of quadruped robots are generally divided into traditional model-based control methods and learning-based control methods. Traditional model-based methods rely on accurate system modeling, usually including multiple complex modules such as state estimation, terrain reconstruction, and whole-body controllers. These methods are usually based on strict assumptions, such as no collision and no sliding. However, these assumptions are often difficult to meet in practical applications, which limits the applicability of traditional control methods. In addition, for the disturbances that the inverted pendulum may face in practical applications on quadruped robots, such as changes in the mass of the inverted pendulum in simulation and in reality, the offset of the center of mass position, the fluctuation of the damping coefficient and the friction force, etc., traditional algorithms lack sufficient adaptive capabilities and are therefore difficult to effectively cope with these complex dynamic changes.

[0004] In recent years, the motion control method of quadruped robots based on reinforcement learning has made significant progress, especially in complex scenarios such as outdoor parkour, and has a certain ability to manipulate robotic arms. The method based on deep reinforcement learning transforms complex optimization problems into optimization in the offline training stage by learning decision-making strategies, thereby significantly reducing the dependence on precise models and showing stronger robustness and adaptability. However, most of the existing reinforcement learning research focuses on motion control, and pays less attention to the comprehensive optimization of the stability and control performance of the robot body, which limits its adaptability and application potential in higher-level tasks.

[0005] To address the problem of balancing an inverted pendulum on a quadruped robot, some current methods have improved the actor-critic network based on Deep Deterministic Policy Gradient (DDPG), designed a hierarchical reinforcement reward function, and obtained control strategies that improve the robot's balance and stability performance through interactive training with a quadruped robot balancing inverted pendulum model. However, these methods have some shortcomings. For example, the DDPG strategy is tightly coupled with the value function and is easily affected by overestimation bias, resulting in an unstable training process. In addition, DDPG relies on Gaussian noise or noise processes to guide policy exploration, which is inefficient in high-dimensional action spaces and can easily cause the strategy to fall into local optimality, limiting its performance. Summary of the invention

[0006] In order to solve the above technical problems, the present invention provides a reinforcement learning quadruped robot inverted pendulum stabilization method based on hybrid state estimation, which enables the quadruped robot to rely solely on its own perception to achieve robust motion control while stably maintaining the inverted pendulum system carried by the fuselage. By designing a parameter estimator based on hybrid state information, the key parameters of the inverted pendulum are accurately estimated in real time, and the posture of the robot is dynamically adjusted, thereby achieving a deep integration of motion control and system stability.

[0007] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0008] A quadruped robot inverted pendulum stability control method based on deep reinforcement learning is designed to estimate the key parameters of the inverted pendulum in real time and dynamically adjust the posture of the quadruped robot, thereby achieving a deep integration of motion control and system stability; specifically, it includes:

[0009] A first-order inverted pendulum is fixed to the body of the quadruped robot;

[0010] The motion of the quadruped robot is modeled as a partially observable Markov decision process, and the strategy output by the partially observable Markov decision process is trained by a neighboring strategy optimization algorithm based on the actor-critic model.

[0011] The actor-critic model includes a policy network and a value network; wherein the policy network outputs a policy according to the input state, samples or directly selects a specific action from the probability distribution corresponding to the policy, and the input state includes the observation quantity and state estimation quantity of the quadruped robot itself; the value network is used to evaluate the performance of the current policy network;

[0012] Use domain randomization techniques to randomize the parameters of the environment during policy training;

[0013] The reward function is designed by comprehensively considering the speed tracking reward, stability penalty, and inverted pendulum penalty to train the value network and provide supervision signals.

[0014] Furthermore, the strategy network outputs a strategy based on the input state, samples or directly selects a specific action from the probability distribution corresponding to the strategy, and the input state includes the observation quantity of the quadruped robot itself. t And state estimators, including:

[0015] The state estimator includes the implicit state z t , the linear velocity v of the fuselage t and the estimated parameters p of the inverted pendulum t ;

[0016] Strategy π φ (a t |o t ,v t ,z t ,p t ) According to the observation quantity of the quadruped robot itself t , implicit state z t , the linear velocity v of the fuselage t and the estimated parameters p of the inverted pendulum t , infer action a t ;

[0017] Among them, t is the index of the current time step, and the parameter p of the inverted pendulum is t is a vector:

[0018]

[0019] in, is the friction coefficient of the inverted pendulum, is the mass of the inverted pendulum, is the displacement of the center of mass of the inverted pendulum on the z-axis; the observation value of the quadruped robot itself o t is a vector containing self-perception information:

[0020]

[0021] Among them, c t ,ω t , g t 、f t ,θ t , a t-l They are the linear velocity command of the quadruped robot, the angular velocity of the body, the gravity unit vector of the body coordinate system, the foot end contact Boolean quantity, the joint angle, the inverted pendulum angle, the joint angular velocity and the inverted pendulum angular velocity and the action of the previous time step;

[0022] The position offset of the quadruped robot joint output by the strategy is used as action a t , a t is the offset of the initial standing posture of the quadruped robot θ def , so the desired joint angle θ of the robot des is defined as:

[0023] θ des =θ def +a t ;

[0024] The desired angle of each joint is tracked by a proportional-derivative controller.

[0025] Furthermore, the domain randomization technology is used in the strategy training process to randomize the parameters of the environment, specifically including:

[0026] The parameters of the environment include the fuselage load weight, PD controller parameters, center of mass offset, and system delay;

[0027] Different degrees of random noise are added to the observations of the quadruped robot itself that are input into the policy network, and domain randomization is performed on the body load weight, PD controller parameters, center of mass offset and system delay.

[0028] Furthermore, the value network is used to evaluate the performance of the current strategy network, specifically including:

[0029] Inputs to the value network t Including the observations of the quadruped robot itself t , the linear velocity v of the fuselage t , the parameter p of the inverted pendulum t :

[0030] s t =[o t v t p t ] T .

[0031] Furthermore, the reward function is designed by comprehensively considering the speed tracking reward, stability penalty and inverted pendulum penalty term to train the value network and provide a supervision signal, specifically including:

[0032] The speed tracking bonus includes the tracking of linear and angular velocity;

[0033] The stability penalty includes limiting the quadruped robot's body speed in the z-axis direction, angular velocity in the x-axis and y-axis directions, orientation, joint acceleration, joint force, body height, action frequency, and smoothness performance;

[0034] The inverted pendulum penalty term includes penalties for the inverted pendulum angle and speed.

[0035] Further, the state estimation is estimated by a state estimator;

[0036] The state estimator is composed of a memory encoder and a source encoder. The memory encoder adopts a long short-term memory network; the source encoder adopts a multi-layer perceptron;

[0037] The quadruped robot's own observations t After being passed into the long short-term memory network, the obtained output is fed to the source encoder to obtain the state estimate; the state estimate includes the explicit linear velocity v of the fuselage t , the parameter p of the inverted pendulum t and the implicit state z t ;

[0038] Based on the explicit linear velocity v of the fuselage t , the parameter p of the inverted pendulum t , the actual value of the fuselage linear speed and the actual inverted pendulum parameters Calculate the error using the root mean square error

[0039]

[0040] MSE(·) denotes root mean square error;

[0041] For the next time step, the observation Through the observations over the past period of time Make predictions, H represents the time step; As the target vector, As the source vector, input the target encoder and source encoder respectively to get the target vector and the source vector The target vector and the source vector Take L2 normalization to get the normalized matrix E, and then normalize the target vector and the source vector Perform normalized exponential function operation on the dot product of the normalized matrix to obtain the target prediction probability and source prediction probability

[0042]

[0043] Where τ is the temperature parameter, e krepresents the kth element in the normalized matrix E; based on the prediction results and goals of cluster assignment, the only goal of representation learning is defined as maximizing the prediction accuracy by calculating the cross information entropy J:

[0044]

[0045] H represents the time domain length of the observation, They represent the expected values ​​of the source vector and the target vector respectively, which are calculated by the Sinkhorn-Knopp algorithm. The cross entropy J is used as the gradient of the target encoder, and the cross entropy J and the root mean square error are calculated. The sum is used as the gradient for the LSTM network and source encoder for training.

[0046] Compared with the prior art, the beneficial technical effects of the present invention are:

[0047] The present invention designs an end-to-end hybrid state estimation-based inverted pendulum stabilization method for a quadruped robot, which improves the robot's balance ability and stability performance. The state estimator of hybrid implicit and explicit information can not only accurately estimate the linear velocity of the quadruped robot, but also estimate in real time the key parameters of the inverted pendulum system carried by the fuselage, including the mass, center of mass position, and friction of the inverted pendulum. Through this design, the quadruped robot can still maintain efficient and stable balance control capabilities when equipped with different types of inverted pendulum systems or when there are differences between the simulation and the actual system, thereby significantly enhancing the adaptability and robustness of the system.

[0048] The conventional DDPG algorithm is unstable during the training process and is prone to fall into local optimality. To this end, the present invention adopts the PPO algorithm for improvement, which effectively limits the amplitude of each strategy update and avoids large fluctuations in the strategy during the optimization process, so that the action space can be covered more smoothly. Especially in high-dimensional environments, PPO shows more efficient exploration capabilities and robustness. In addition, LSTM is introduced for mixed state estimation, which can better process sequence data and capture time dependencies. By extracting mixed state information, the strategy can capture more key features from the environment, significantly improving its stability and adaptability, thereby showing more outstanding performance in complex dynamic tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 It is the overall system block diagram of the present invention. DETAILED DESCRIPTION

[0050] A preferred embodiment of the present invention is described in detail below with reference to the accompanying drawings.

[0051] This paper proposes a method for stabilizing the inverted pendulum of a quadruped robot based on deep reinforcement learning. By fixing a first-order inverted pendulum on the body of the quadruped robot, the robot's balance ability and stability performance in a dynamic environment are intuitively demonstrated. Based on this experimental platform, this paper combines hybrid state estimation and designs an end-to-end motion learning framework based on the neighboring strategy optimization actor-critic model, which realizes the synchronous training of motion strategy and state estimator, effectively improving the control effect and robustness of the system.

[0052] 1. Reinforcement learning problem description:

[0053] Due to the lack of external perception sensors, the terrain information cannot be fully observed. Therefore, the present invention models the motion problem as a partially observable Markov decision process (POMDP). The state at time step t is defined as x t , the agent's policy execution action a t , the environment passes through the transition probability P(x t+1 |x t ,a t ) transfers to the state x of the next time step t+1 , and returns the reward value r t and a partial state observable The goal of reinforcement learning is to find a strategy π that maximizes the expected future trajectory (also called the expected cumulative reward):

[0054]

[0055] where γ t ∈[0,1) is the discount factor.

[0056] In order to learn robust blind motion capabilities in a single training phase, the present invention uses the Proximal Policy Optimization (PPO) algorithm to train the strategy and combines it with an asymmetric actor-critic framework to improve the learning efficiency and performance of the model. The PPO algorithm introduces a truncated probability ratio function to limit the difference between the new and old strategies when the strategy is updated, thereby effectively controlling the fluctuations in the optimization process. Compared with other policy gradient methods, the PPO algorithm achieves a good balance between computational complexity and performance, providing a reliable guarantee for efficient and stable strategy learning.

[0057] 2. Policy Network:

[0058] The policy network outputs a policy based on the input state, usually a probability distribution, from which specific actions are sampled or directly selected. Policy πφ (a t |o t ,v t ,z t ,p t ) needs to give parameter φ and infer action a t . You need to input your own observations o t , the linear velocity v of the fuselage t , implicit state z t , and the parameter p of the inverted pendulum t , where p t is a vector of dimension n×1:

[0059]

[0060] in, is the friction coefficient of the inverted pendulum, is the mass of the inverted pendulum, is the offset of the center of mass of the inverted pendulum on the z-axis. t is an n×1 dimensional vector containing self-perception information:

[0061]

[0062] Among them, c t ,ω t , g t 、f t ,θ t , a t-1 They are the linear velocity command of the quadruped robot, the angular velocity of the body, the gravity unit vector of the body coordinate system, the foot end contact Boolean quantity, the joint angle, the inverted pendulum angle, the joint angular velocity and the inverted pendulum angular velocity and the action of the previous time step;

[0063] The strategy output is the position offset of the 12 joints of the quadruped robot as action a t ∈R 12 , a t is the offset of the initial standing posture of the quadruped robot θ def , so the desired joint angle θ of the robot des is defined as:

[0064] θ des =θ def +a t ;

[0065] The desired angle of each joint is tracked by a proportional-derivative (PD) controller, setting P=40.0 and D=1.0 as physical parameters.

[0066] 3. Domain Randomization:

[0067] Domain randomization is a technology widely used in the field of reinforcement learning and robot control, which aims to improve the generalization ability of the model in real-world environments. The core idea is to enable the model to learn under diverse conditions by randomizing environmental parameters during training, thereby showing stronger adaptability and robustness in unknown scenarios. The present invention adds different degrees of random noise to the observations of the policy network, and at the same time performs domain randomization on key factors such as fuselage load weight, PD controller parameters, center of mass offset, and system delay.

[0068] 4. Value Network:

[0069] The value network is used to evaluate the performance of the current strategy and help the strategy network learn a better strategy. In order to obtain more accurate fuselage linear velocity and inverted pendulum parameters, the input of the value network includes not only its own observations, but also t , and also added privileged observations including the linear velocity v of the fuselage t , the parameter p of the inverted pendulum t :

[0070] s t =[o t v t p t ] T .

[0071] 5. Reward function design:

[0072] The reward function provides a supervisory signal for training the value network. Its design comprehensively considers speed tracking rewards and stability penalties to achieve stable and natural motion behavior. Speed ​​tracking rewards include accurate tracking of linear and angular velocities, while stability penalties cover multiple dimensions, including limiting the speed of the fuselage in the z-axis direction, the angular velocities in the roll and pitch directions, the offset of the gravity component, and the joint acceleration. In addition, for the task of stabilizing the inverted pendulum, the reward function also introduces penalties for the angle and speed of the inverted pendulum to further improve the execution stability of the task, as shown in Table 1. The total reward for the strategy that takes action in each state is as follows:

[0073] r t (s t ,a t )=∑r i w i ;

[0074] Here, i is the index of each reward.

[0075] Table 1 Reward function terms

[0076]

[0077]

[0078] 6. State Estimator:

[0079] The state estimator, as an important input of the policy network, consists of a memory encoder (using LSTM) and a source encoder. The memory encoder is usually implemented in two ways: one is to stack a series of historical observations as the input of the MLP; the other is to use a model architecture that can capture past information, such as a recursive neural network (RNN, Recurrent Neural Network) or a temporal convolutional neural network (TCN, Temporal Convolutional Network). However, architectures like MLP and TCN require a certain amount of memory space to be reserved for storing historical observation data, which puts a lot of pressure on the use of onboard resources. In contrast, RNN can embed historical information through hidden states, thereby reducing the reliance on directly storing total historical observations. Based on this, the present invention selects a long short-term memory network (LSTM, Long Short-Term Memory) as the implementation architecture of RNN.

[0080] Self-observable t After entering the LSTM, the output is fed to the source encoder, using a multilayer perceptron (MLP) to obtain an implicit and explicit mixed state estimator. The state estimator includes the explicit linear velocity v of the fuselage t , the parameter p of the inverted pendulum t , and the implicit state z t Among them, for the explicit state quantity, the root mean square error (MSE, Mean Squared Error) and the true value of the fuselage linear velocity are used and the actual inverted pendulum parameters To calculate the error

[0081]

[0082] For the next moment, the observed value Can pass through the past The observations over a period of time are predicted, and H represents the time step, which is set to H = 5. As the target vector, As the source vector, input the target encoder and source encoder respectively to get the target vector and the source vector Take L2 normalization for both to get the normalized matrix, and then normalize the target vector and the source vector The predicted probability is obtained by performing a normalized exponential function operation on the dot product of the normalized matrix and

[0083]

[0084] Where τ is the temperature parameter.

[0085] Now that the prediction results and goals of cluster assignment have been obtained, the only goal of representation learning is defined as maximizing the prediction accuracy by calculating the cross information entropy J:

[0086]

[0087] The cross information entropy J is used as the gradient of the target encoder, and the cross information entropy J and the root mean square error The sum is used as the gradient of LSTM and source encoder for training.

[0088] It is obvious to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential features of the present invention. Therefore, from any point of view, the embodiments should be regarded as exemplary and non-limiting, and the scope of the present invention is defined by the appended claims rather than the above description, and it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims are included in the present invention, and any reference numerals in the claims should not be regarded as limiting the claims involved.

[0089] In addition, it should be understood that although this specification is described in terms of implementation methods, not every implementation method contains only one independent technical solution. This narrative method of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation methods that can be understood by those skilled in the art.

Claims

1. A quadruped robot inverted pendulum stabilization control method based on deep reinforcement learning, characterized in that: By designing a real-time estimation of the key parameters of the inverted pendulum and dynamically adjusting the posture of the quadruped robot, a deep integration of motion control and system stability is achieved; specifically, the following features are included: A first-order inverted pendulum is fixed to the body of the quadruped robot; The motion of the quadruped robot is modeled as a partially observable Markov decision process, and the strategy output by the partially observable Markov decision process is trained by a neighboring strategy optimization algorithm based on the actor-critic model. The actor-critic model includes a policy network and a value network; wherein the policy network outputs a policy according to the input state, samples or directly selects a specific action from the probability distribution corresponding to the policy, and the input state includes the observation quantity and state estimation quantity of the quadruped robot itself; the value network is used to evaluate the performance of the current policy network; Use domain randomization techniques to randomize the parameters of the environment during policy training; The reward function is designed by comprehensively considering the speed tracking reward, stability penalty, and inverted pendulum penalty to train the value network and provide supervision signals.

2. According to the method of claim 1, the inverted pendulum stability control method of a quadruped robot based on deep reinforcement learning is characterized in that: The strategy network outputs a strategy based on the input state, samples or directly selects specific actions from the probability distribution corresponding to the strategy, and the input state includes the observation value of the quadruped robot itself. t And state estimators, including: The state estimator consists of the implicit state z t , the linear velocity v of the fuselage t and the estimated parameters p of the inverted pendulum t ; Strategy π φ (a t |o t ,v t ,z t ,p t ) According to the observation quantity of the quadruped robot itself t , implicit state z t , the linear velocity v of the fuselage t and the estimated parameters p of the inverted pendulum t , infer action a t ; Among them, t is the index of the current time step, and the parameter p of the inverted pendulum is t is a vector: in, is the friction coefficient of the inverted pendulum, is the mass of the inverted pendulum, is the displacement of the center of mass of the inverted pendulum on the z-axis; the observation value of the quadruped robot itself o t is a vector containing self-perception information: Among them, c t ,ω t , g t 、f t ,θ t , a t-1 They are the linear velocity command of the quadruped robot, the angular velocity of the body, the gravity unit vector of the body coordinate system, the foot end contact Boolean quantity, the joint angle, the inverted pendulum angle, the joint angular velocity and the inverted pendulum angular velocity and the action of the previous time step; The position offset of the quadruped robot joint output by the strategy is used as action a t , a t is the offset of the initial standing posture of the quadruped robot θ def , so the robot's desired joint angle θ des is defined as: i des =θ def +a t ; The desired angle of each joint is tracked by a proportional-derivative controller.

3. According to the method of claim 1, the inverted pendulum stability control method of a quadruped robot based on deep reinforcement learning is characterized in that: The domain randomization technology is used in the strategy training process to randomize the parameters of the environment, specifically including: The parameters of the environment include the fuselage load weight, PD controller parameters, center of mass offset, and system delay; Different degrees of random noise are added to the observations of the quadruped robot itself that are input into the policy network, and domain randomization is performed on the body load weight, PD controller parameters, center of mass offset and system delay.

4. According to the method of claim 1, the inverted pendulum stability control method of a quadruped robot based on deep reinforcement learning is characterized in that: The value network is used to evaluate the performance of the current policy network, including: Inputs to the value network t Including the observations of the quadruped robot itself t , the linear velocity v of the fuselage t , the parameter p of the inverted pendulum t : with t =[o t in t p t ] T 。 5. The method for controlling the inverted pendulum stability of a quadruped robot based on deep reinforcement learning according to claim 1, characterized in that: The reward function is designed by comprehensively considering the speed tracking reward, stability penalty and inverted pendulum penalty to train the value network and provide supervision signals, including: The speed tracking bonus includes the tracking of linear and angular velocity; The stability penalty includes limiting the quadruped robot's body speed in the z-axis direction, angular velocity in the x-axis and y-axis directions, orientation, joint acceleration, joint force, body height, action frequency, and smoothness performance; The inverted pendulum penalty term includes penalties for the inverted pendulum angle and speed.

6. The method for controlling the inverted pendulum stability of a quadruped robot based on deep reinforcement learning according to claim 1, characterized in that: The state estimation is obtained by estimating the state estimator; The state estimator is composed of a memory encoder and a source encoder. The memory encoder adopts a long short-term memory network; the source encoder adopts a multi-layer perceptron; The quadruped robot's own observation quantity t After being passed into the long short-term memory network, the obtained output is fed to the source encoder to obtain the state estimate; the state estimate includes the explicit linear velocity v of the fuselage t , the parameter p of the inverted pendulum t and the implicit state z t ; Based on the explicit linear velocity v of the fuselage t , the parameter p of the inverted pendulum t , the actual value of the fuselage linear speed and the actual inverted pendulum parameters Calculate the error using the root mean square error MSE(·) denotes root mean square error; For the next time step, the observation Through the observations over the past period of time Make predictions, H represents the time step; As the target vector, As the source vector, input the target encoder and source encoder respectively to get the target vector and the source vector The target vector and the source vector Take L2 normalization to get the normalized matrix E, and then normalize the target vector and the source vector Perform normalized exponential function operation on the dot product of the normalized matrix to obtain the target prediction probability and source prediction probability Where τ is the temperature parameter, e k represents the kth element in the normalized matrix E; based on the prediction results and goals of cluster assignment, the only goal of representation learning is defined as maximizing the prediction accuracy by calculating the cross information entropy J: H represents the time domain length of the observation, They represent the expected values ​​of the source vector and the target vector respectively, which are calculated by the Sinkhorn-Knopp algorithm. The cross entropy J is used as the gradient of the target encoder, and the cross entropy J and the root mean square error are calculated. The sum is used as the gradient for the LSTM network and source encoder for training.

Images