A reinforcement learning-based method for stabilizing an inverted pendulum in a quadruped robot using mixed state estimation.

Abstract

This invention relates to the field of robotics and automation technology, and discloses a method for stabilizing an inverted pendulum in a quadruped robot based on deep reinforcement learning. The method specifically comprises: attaching a first-order inverted pendulum to the quadruped robot's body; modeling the quadruped robot's motion as a partially observable Markov decision process, and training the policy output by the partially observable Markov decision process using a neighboring policy optimization algorithm based on an actor-critic model; the actor-critic model includes a policy network and a value network; domain randomization is used during policy training to randomize environment parameters; and a reward function is designed by comprehensively considering speed tracking rewards, stability penalties, and inverted pendulum penalties to train the value network and provide a supervisory signal. This invention designs an end-to-end hybrid state estimation-based method for stabilizing an inverted pendulum in a quadruped robot, improving the robot's balance and stability performance.

Description

Technical Field

[0001] This invention relates to the field of robotics and automation technology, specifically to a reinforcement learning-based method for stabilizing an inverted pendulum in a quadruped robot, based on hybrid state estimation. Background Technology

[0002] Existing research on quadruped robots largely focuses on motion control, with less attention paid to the comprehensive optimization of robot stability and maneuverability, thus limiting their adaptability and application potential in higher-level tasks. The inverted pendulum, as a classic nonlinear dynamic system, has long been considered a standard testing platform for verifying the effectiveness and robustness of control methods. This invention, by fixing a first-order inverted pendulum to the quadruped robot's body, visually demonstrates the robot's balance and stability in dynamic environments, providing an innovative research perspective and technical means to improve the robot's overall performance.

[0003] Motion control methods for quadruped robots are generally divided into traditional model-based control methods and learning-based control methods. Traditional model-based methods rely on accurate system modeling, typically including multiple complex modules such as state estimation, terrain reconstruction, and whole-body controllers. These methods are usually based on strict assumptions, such as no collisions and no slippage. However, these assumptions are often difficult to meet in practical applications, limiting the applicability of traditional control methods. Furthermore, traditional algorithms lack sufficient adaptive capabilities to handle disturbances that quadruped robots with inverted pendulums may encounter in practical applications, such as changes in the mass of the inverted pendulum, shifts in the center of mass, fluctuations in damping coefficients and friction forces between simulations and the actual robot. Therefore, they are unable to effectively cope with these complex dynamic changes.

[0004] Reinforcement learning-based motion control methods for quadruped robots have made significant progress in recent years, demonstrating excellent performance, especially in complex scenarios such as parkour, and exhibiting some robotic arm manipulation capabilities. Deep reinforcement learning-based methods learn decision-making strategies, transforming complex optimization problems into optimizations during offline training, thus significantly reducing reliance on precise models and exhibiting stronger robustness and adaptability. However, existing reinforcement learning research largely focuses on motion control, with less attention paid to the comprehensive optimization of robot stability and maneuverability, which limits its adaptability and application potential in higher-level tasks.

[0005] To address the problem of balancing an inverted pendulum on a quadruped robot, some current methods have improved actor-commentator networks based on Deep Deterministic Policy Gradient (DDPG), designed hierarchical reinforcement reward functions, and obtained control policies that enhance the robot's balance ability and stability through interactive training with a quadruped robot model balancing an inverted pendulum. However, these methods have some drawbacks. For example, the policy and value function of DDPG are tightly coupled, making them susceptible to overestimation bias, which leads to instability in the training process. Furthermore, DDPG relies on Gaussian noise or noisy processes to guide policy exploration, which is inefficient in high-dimensional action spaces and easily leads to the policy getting trapped in local optima, limiting its performance. Summary of the Invention

[0006] To address the aforementioned technical problems, this invention provides a reinforcement learning-based method for stabilizing an inverted pendulum in a quadruped robot, based on hybrid state estimation. This method enables the quadruped robot to maintain a stable inverted pendulum system mounted on its body, relying solely on its own perception to achieve robust motion control. By designing a parameter estimator based on hybrid state information, the key parameters of the inverted pendulum are estimated accurately in real time, and the robot's posture is dynamically adjusted, thereby achieving a deep integration of motion control and system stability.

[0007] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0008] A deep reinforcement learning-based method for stabilizing a quadruped robot's inverted pendulum is proposed. This method involves designing a system to estimate the key parameters of the inverted pendulum in real time and dynamically adjusting the quadruped robot's posture, thereby achieving a deep integration of motion control and system stability. Specifically, it includes:

[0009] The first-order inverted pendulum is fixed to the body of the quadruped robot;

[0010] The motion of the quadruped robot is modeled as a partially observable Markov decision process, and the policy output by the partially observable Markov decision process is trained by a neighbor policy optimization algorithm based on the actor-critic model.

[0011] The actor-critic model includes a policy network and a value network. The policy network outputs a policy based on the input state, sampling from the probability distribution corresponding to the policy or directly selecting a specific action. The input state includes the quadruped robot's own observations and state estimates. The value network is used to evaluate the performance of the current policy network.

[0012] Domain randomization is used during policy training to randomize the parameters of the environment;

[0013] A reward function is designed by comprehensively considering the speed tracking reward, stability penalty, and inverted pendulum penalty term to train the value network and provide supervision signals.

[0014] Furthermore, the policy network outputs a policy based on the input state, sampling from the probability distribution corresponding to the policy or directly selecting a specific action. The input state includes observations of the quadruped robot itself. t And state estimators, specifically including:

[0015] State estimators include implicit state z t The linear velocity v of the fuselage t And the estimated parameters p of the inverted pendulum. t ;

[0016] Strategy π φ (a t |o t ,v t ,z t ,p t Based on the quadruped robot's own observations t Implicit state z t The linear velocity v of the fuselage t And the estimated parameters p of the inverted pendulum. t Infer action a t ;

[0017] Where t is the index of the current time step, and p is the parameter of the inverted pendulum. t It is a vector:

[0018]

[0019] in, Let be the coefficient of friction of the inverted pendulum. For the mass of the inverted pendulum, The offset of the inverted pendulum's center of mass along the z-axis; the observables of the quadruped robot itself. t It is a vector containing its own perceptual information:

[0020]

[0021] Among them, c t ω t g t f t θ t , a t-l These are the quadruped robot's body linear velocity command, body angular velocity, gravity unit vector in the body coordinate system, foot contact Boolean value, joint angle, inverted pendulum angle, joint angular velocity, inverted pendulum angular velocity, and the action of the previous time step;

[0022] The position offset of the quadruped robot joints output by the strategy is used as action 'a'. t a t The offset θ of the initial standing posture of the quadruped robot def Therefore, the robot's expected joint angle θ des Defined as:

[0023] θ des =θ def +a t ;

[0024] The desired angle of each joint is tracked by a proportional-derivative controller.

[0025] Furthermore, the use of domain randomization techniques during policy training to randomize the parameters of the environment specifically includes:

[0026] Environmental parameters include fuselage load weight, PD controller parameters, center of gravity offset, and system delay;

[0027] Different levels of random noise are added to the quadruped robot's own observations input to the policy network, while the robot's load weight, PD controller parameters, centroid offset, and system delay are randomized in the domain.

[0028] Furthermore, the value network is used to evaluate the performance of the current policy network, specifically including:

[0029] Inputs to the value network t Including the observations of the quadruped robot itself. t The linear velocity v of the fuselage t The parameter p of the inverted pendulum t :

[0030] s t =[o t v t p t ] T .

[0031] Furthermore, the reward function designed by comprehensively considering velocity tracking reward, stability penalty, and inverted pendulum penalty term to train the value network and provide supervision signals specifically includes:

[0032] Velocity tracking rewards include tracking of linear velocity and angular velocity;

[0033] Stability penalties include limiting the quadruped robot's body velocity in the z-axis direction, angular velocity in the x and y-axis directions, orientation, joint acceleration, joint force, body height, motion frequency, and smoothness performance.

[0034] The pendulum penalty includes penalties for the pendulum's angle and speed.

[0035] Furthermore, the state estimate is obtained by a state estimator;

[0036] The state estimator consists of a memory encoder and a source encoder. The memory encoder uses a long short-term memory network, and the source encoder uses a multilayer perceptron.

[0037] The quadruped robot's own observations t After being fed into the Long Short-Term Memory network, the output is fed to the source encoder to obtain the state estimate; the state estimate includes the explicit linear velocity v of the fuselage. t The parameter p of the inverted pendulum t and implicit state z t ;

[0038] Based on the explicit linear velocity v of the fuselage t The parameter p of the inverted pendulum t Actual fuselage linear velocity And the parameters of the actual inverted pendulum Error is calculated using root mean square error.

[0039]

[0040] MSE(·) represents the root mean square error;

[0041] For the observations at the next time step Based on observations over a period of time To make a prediction, H represents the time step; As the target vector As the source vector, it is input into the target encoder and the source encoder respectively to obtain the target vector. and source vector For the target vector and source vector Take L2 normalization to obtain the normalized matrix E, and then apply it to the target vector. and source vector The target prediction probability is obtained by performing a normalized exponential function operation on the dot product with the normalized matrix. Source prediction probability

[0042]

[0043] Where τ is the temperature parameter, e kLet J represent the k-th element in the normalized matrix E; based on the prediction results and objectives of clustering assignment, the unique objective of representation learning is defined as maximizing prediction accuracy by calculating the cross-information entropy J.

[0044]

[0045] H represents the time domain length of the observation. Let J represent the expected values ​​of the source vector and the target vector, respectively, calculated using the Sinkhorn-Knopp algorithm; let the cross-information entropy J be used as the gradient of the target encoder, and let the cross-information entropy J and the root mean square error be used as the gradients. The sum of these values ​​is used as the gradients for training the Long Short-Term Memory network and the source encoder.

[0046] Compared with the prior art, the beneficial technical effects of the present invention are:

[0047] This invention presents an end-to-end, hybrid state estimation-based method for stabilizing an inverted pendulum in a quadruped robot, improving the robot's balance and stability. The state estimator, which combines explicit and implicit information, not only accurately estimates the linear velocity of the quadruped robot but also estimates key parameters of the inverted pendulum system mounted on the robot in real time, including the pendulum's mass, center of mass position, and friction. This design enables the quadruped robot to maintain efficient and stable balance control even when equipped with different types of inverted pendulum systems or when there are differences between the simulated and actual systems, thus significantly enhancing the system's adaptability and robustness.

[0048] Conventional DDPG algorithms suffer from instability during training and are prone to getting trapped in local optima. To address this, this invention employs an improved PPO algorithm, effectively limiting the magnitude of each policy update and avoiding large fluctuations in the policy during optimization, thus enabling smoother coverage of the action space. Particularly in high-dimensional environments, PPO demonstrates more efficient exploration capabilities and robustness. Furthermore, the introduction of LSTM for mixed-state estimation better handles sequential data and captures temporal dependencies. By extracting mixed-state information, the policy can capture more key features from the environment, significantly improving its stability and adaptability, thereby exhibiting superior performance in complex dynamic tasks. Attached Figure Description

[0049] Figure 1 This is an overall system block diagram of the present invention. Detailed Implementation

[0050] A preferred embodiment of the present invention will now be described in detail with reference to the accompanying drawings.

[0051] This invention proposes a method for stabilizing a quadruped robot using an inverted pendulum based on deep reinforcement learning. By fixing a first-order inverted pendulum to the quadruped robot's body, the method visually demonstrates the robot's balance and stability in dynamic environments. Based on this experimental platform, this invention combines hybrid state estimation and designs an end-to-end motion learning framework based on a neighbor policy optimization actor-critic model. This enables synchronous training of the motion policy and the state estimator, effectively improving the system's control performance and robustness.

[0052] 1. Reinforcement Learning Problem Description:

[0053] Due to the lack of external sensing sensors, terrain information cannot be fully observed. Therefore, this invention models the motion problem as a partially observable Markov Decision Process (POMDP). The state at time step t is defined as x. t The agent's strategy execution action a t The environment is determined by the transition probability P(x). t+1 |x t ,a t Transition to the state x at the next time step t+1 and return the reward value r. t and a partial state observation The goal of reinforcement learning is to find a policy π that maximizes the expected value of future trajectories (also known as the expected value of cumulative rewards).

[0054]

[0055] Where γ t ∈[0,1) is the discount factor.

[0056] To learn robust blind motion capabilities within a single training phase, this invention employs the Proximal Policy Optimization (PPO) algorithm for policy training, combined with an asymmetric actor-critic framework to enhance the model's learning efficiency and performance. The PPO algorithm, by introducing a truncated probability ratio function, limits the range of differences between the old and new policies during policy updates, effectively controlling fluctuations in the optimization process. Compared to other policy gradient methods, the PPO algorithm achieves a good balance between computational complexity and performance, providing a reliable guarantee for efficient and stable policy learning.

[0057] 2. Policy Network:

[0058] The policy network outputs a policy based on the input state, typically a probability distribution from which specific actions are sampled or directly selected. (Policy π)φ (a t |o t ,v t ,z t ,p t Given a parameter φ, we need to infer the action a. t You need to input your own observations. t The linear velocity v of the fuselage t Implicit state z t And the parameter p of the inverted pendulum. t , where p t It is an n×1 dimensional vector:

[0059]

[0060] in, Let be the coefficient of friction of the inverted pendulum. For the mass of the inverted pendulum, This represents the offset of the inverted pendulum's center of mass along the z-axis. The self-observation is o. t It is an n×1 dimensional vector containing its own perceptual information:

[0061]

[0062] Among them, c t ω t g t f t θ t , a t-1 These are the quadruped robot's body linear velocity command, body angular velocity, gravity unit vector in the body coordinate system, foot contact Boolean value, joint angle, inverted pendulum angle, joint angular velocity, inverted pendulum angular velocity, and the action of the previous time step;

[0063] The strategy output is the position offset of the 12 joints of the quadruped robot as action 'a'. t ∈R 12 a t The offset θ of the initial standing posture of the quadruped robot def Therefore, the robot's expected joint angle θ des Defined as:

[0064] θ des =θ def +a t ;

[0065] The desired angle of each joint is tracked by a proportional-derivative (PD) controller, with P = 40.0 and D = 1.0 set as physical parameters.

[0066] 3. Domain randomization:

[0067] Domain randomization is a technique widely used in reinforcement learning and robot control to improve the generalization ability of models in real-world environments. Its core idea is to randomize environmental parameters during training, enabling the model to learn under diverse conditions and thus exhibit stronger adaptability and robustness in unknown scenarios. This invention incorporates varying degrees of random noise into the observations of the policy network and performs domain randomization on key factors such as fuselage load weight, PD controller parameters, centroid shift, and system delay.

[0068] 4. Value Network:

[0069] Value networks are used to evaluate the performance of the current policy, helping the policy network learn better policies. To obtain more accurate parameters for the fuselage linear velocity and the inverted pendulum, the input to the value network includes not only its own observations. t It also added privileged observations, including the linear velocity v of the fuselage. t The parameter p of the inverted pendulum t :

[0070] s t =[o t v t p t ] T .

[0071] 5. Reward function design:

[0072] The reward function provides supervision signals for training the value network. Its design integrates velocity tracking rewards and stability penalties to achieve stable and natural motion behavior. Velocity tracking rewards include accurate tracking of linear and angular velocities, while stability penalties cover multiple dimensions, including limiting the fuselage velocity along the z-axis, angular velocities in roll and pitch, offsets of the gravitational component, and joint accelerations. Furthermore, for the stable inverted pendulum task, the reward function also introduces penalties for the pendulum angle and velocity to further improve task performance stability, as shown in Table 1. The total reward for the policy taking action in each state is as follows:

[0073] r t (s t ,a t )=∑r i w i ;

[0074] Where i is the index of each reward.

[0075] Table 1 Reward Function Items

[0076]

[0077]

[0078] 6. State Estimator:

[0079] The state estimator, as a crucial input to the policy network, consists of a memory encoder (using LSTM) and a source encoder. Memory encoders are typically implemented in two ways: one is by stacking a series of historical observations as input to the MLP; the other is by utilizing model architectures capable of capturing past information, such as recurrent neural networks (RNNs) or temporal convolutional neural networks (TCNs). However, architectures like MLPs and TCNs require reserved memory space to store historical observation data, placing significant pressure on onboard resources. In contrast, RNNs can embed historical information through hidden states, thereby reducing the reliance on directly storing all historical observations. Therefore, this invention chooses a Long Short-Term Memory (LSTM) network as the implementation architecture for the RNN.

[0080] Self-observation o t After being fed into the LSTM, the output is fed to the source encoder, which employs a multilayer perceptron (MLP) to obtain implicit and explicit hybrid state estimates. The state estimates include the explicit linear velocity v of the machine body. t The parameter p of the inverted pendulum t and implicit state variable z t For explicit state variables, the root mean square error (MSE) and the true value of the fuselage linear velocity are used. And the parameters of the actual inverted pendulum Used to calculate error

[0081]

[0082] For the observation at the next moment You can go through the past Prediction is made based on observations over a period of time, where H represents the time step, and is set to H=5. As the target vector As the source vector, it is input into the target encoder and the source encoder respectively to obtain the target vector. and source vector Apply L2 normalization to both to obtain a normalized matrix, and then apply this normalization to the target vector. and source vector The predicted probability is obtained by performing a normalized exponential function operation on the dot product with the normalized matrix. and

[0083]

[0084] Where τ is the temperature parameter.

[0085] Now, this invention has obtained the prediction results and objectives of cluster assignment. By calculating the cross-information entropy J, the unique objective of representation learning is defined as maximizing prediction accuracy:

[0086]

[0087] Using the cross-information entropy J as the gradient of the target encoder, and combining the cross-information entropy J with the root mean square error... The sum of these values ​​is used as the gradients for training the LSTM and the source encoder.

[0088] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention, and no reference numerals in the claims should be construed as limiting the scope of the claims.

[0089] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. A reinforcement learning-based method for stabilizing an inverted pendulum in a quadruped robot using hybrid state estimation, characterized in that, By designing a system that estimates the key parameters of the inverted pendulum in real time and dynamically adjusts the posture of the quadruped robot, a deep integration of motion control and system stability is achieved; specifically including: The first-order inverted pendulum is fixed to the body of the quadruped robot; The motion of the quadruped robot is modeled as a partially observable Markov decision process, and the policy output by the partially observable Markov decision process is trained by a neighbor policy optimization algorithm based on the actor-critic model. The actor-critic model includes a policy network and a value network. The policy network outputs a policy based on the input state, sampling from the probability distribution corresponding to the policy or directly selecting a specific action. The input state includes the quadruped robot's own observations and state estimates. The value network is used to evaluate the performance of the current policy network. Domain randomization is used during policy training to randomize the parameters of the environment; A reward function is designed by comprehensively considering velocity tracking reward, stability penalty, and inverted pendulum penalty term to train the value network and provide supervision signals; The state estimate is obtained by a state estimator; The state estimator consists of a memory encoder and a source encoder. The memory encoder uses a long short-term memory network, and the source encoder uses a multilayer perceptron. The quadruped robot's own observations After being fed into the Long Short-Term Memory network, the output is fed to the source encoder to obtain the state estimate; the state estimate includes the explicit linear velocity of the fuselage. Parameters of an inverted pendulum and implicit state ; Linear velocity based on explicit chassis Parameters of an inverted pendulum Actual fuselage linear velocity And the parameters of the actual inverted pendulum The error is calculated using the root mean square error. : ； Indicates the root mean square error; For the observations at the next time step Based on observations over a period of time To make a prediction, H represents the time step; As the target vector As the source vector, it is input into the target encoder and the source encoder respectively to obtain the target vector. and source vector For the target vector and source vector L2 normalization yields the normalized matrix. Then, for the target vector and source vector The target prediction probability is obtained by performing a normalized exponential function operation on the dot product with the normalized matrix. Source prediction probability : ； ； in For temperature parameters, Represents the normalized matrix The k-th element in the cluster; based on the prediction results and target of the clustering assignment, the cross-information entropy is calculated. The sole objective of representation learning is defined as maximizing prediction accuracy: ； The time domain length of the observed quantity Let represent the expected values ​​of the source vector and the target vector, respectively, calculated using the Sinkhorn-Knopp algorithm; and let the cross-information entropy be... As the gradient of the target encoder, the cross-information entropy and root mean square error The sum of these values ​​is used as the gradients for training the Long Short-Term Memory network and the source encoder.

2. The reinforcement learning-based method for stabilizing an inverted pendulum in a quadruped robot based on hybrid state estimation as described in claim 1, characterized in that, The policy network outputs a policy based on the input state, sampling from the probability distribution corresponding to the policy or directly selecting a specific action. The input state includes observations of the quadruped robot itself. And state estimators, specifically including: State estimators include implicit states. linear velocity of the fuselage And the estimated parameters of the inverted pendulum. ; Strategy Based on the quadruped robot's own observations Implicit state linear velocity of the fuselage And the estimated parameters of the inverted pendulum. Inferring the action ; in, The index of the current time step, the parameters of the inverted pendulum. It is a vector: ； in, Let be the coefficient of friction of the inverted pendulum. For the mass of the inverted pendulum, The offset of the center of mass of the inverted pendulum along the z-axis; the observables of the quadruped robot itself. It is a vector containing its own perceptual information: ； in, , , , , , , , , These are the quadruped robot's body linear velocity command, body angular velocity, gravity unit vector in the body coordinate system, foot contact Boolean value, joint angle, inverted pendulum angle, joint angular velocity, inverted pendulum angular velocity, and the action of the previous time step; The position offset of the quadruped robot joints output by the strategy is used as the action. , The offset of the initial standing posture of the quadruped robot Therefore, the robot's expected joint angle Defined as: ； The desired angle of each joint is tracked by a proportional-derivative controller.

3. The reinforcement learning-based method for stabilizing an inverted pendulum in a quadruped robot based on hybrid state estimation as described in claim 1, characterized in that, The use of domain randomization techniques during policy training to randomize environment parameters specifically includes: Environmental parameters include fuselage load weight, PD controller parameters, center of gravity offset, and system delay; Different levels of random noise are added to the quadruped robot's own observations input to the policy network, while the robot's load weight, PD controller parameters, centroid offset, and system delay are randomized in the domain.

4. The reinforcement learning-based method for stabilizing an inverted pendulum in a quadruped robot based on hybrid state estimation as described in claim 1, characterized in that, The value network is used to evaluate the performance of the current policy network, specifically including: Input to the value network Including observations of the quadruped robot itself linear velocity of the fuselage Parameters of an inverted pendulum : 。 5. The reinforcement learning-based method for stabilizing an inverted pendulum in a quadruped robot based on hybrid state estimation as described in claim 1, characterized in that, The reward function, which comprehensively considers velocity tracking reward, stability penalty, and inverted pendulum penalty term, is designed to train the value network and provide supervision signals. Specifically, it includes: Velocity tracking rewards include tracking of linear velocity and angular velocity; Stability penalties include limiting the quadruped robot's body velocity in the z-axis direction, angular velocity in the x and y-axis directions, orientation, joint acceleration, joint force, body height, motion frequency, and smoothness performance. The pendulum penalty includes penalties for the pendulum's angle and speed.

Images