Optimization method of bipedal robot walking stability based on improved PPO algorithm

Abstract

This invention discloses a method for optimizing bipedal robot walking stability based on an improved PPO algorithm. Specifically, by introducing long-short-term memory (LSTM) into a neural network, the method selectively retains information from previous samples to fit the predictions of subsequent samples. Furthermore, an attention mechanism is added to the action network to dynamically and adaptively weight input features, fitting the variable weight dependence of state information at different time steps on subsequent actions. This results in an improved PPO algorithm that can address bipedal robot walking control issues in various environments, including parameter adjustment for simulation training and stability of walking motion. The simulated robot walking environment employed is the Bipedalwalker‑v3 environment within Gym developed by Open AI; pre-trained control parameters are trained and adjusted using the improved PPO algorithm; and stability of the bipedal robot's walking motion is achieved through a trained controller.

Description

Technical Field

[0001] The present invention relates to a method for improving the walking stability of a biped robot, and in particular to a method for optimizing the walking stability of a biped robot based on an improved PPO algorithm. Background Art

[0002] Bipedal robots, with their humanoid structure, can adapt to complex terrains such as stairs, streets, and uneven surfaces, and possess flexible locomotion. Bipedal robots possess numerous degrees of freedom, and controlling their adaptive walking in complex environments requires a flexible and diverse gait. This requires control strategies that can search for optimal solutions across a vast parameter space while simultaneously processing high-dimensional sensor information. If the robot encounters uneven surfaces with bumps or depressions while walking, the end of its swinging leg can touch the ground prematurely or later, causing the robot to distort its posture and fall. Enabling bipedal robots to adapt to various surfaces and achieve fast and stable walking is a key research issue.

[0003] In recent years, many researchers at home and abroad have attempted to combine offline gait planning with various online gait correction algorithms to achieve stable and continuous walking in actual walking environments. Some researchers use the traditional inverted pendulum model and the zero moment point (ZMP) to obtain angle sensor information on the joints of a bipedal robot after the planned gait. Based on this information, an inverse kinematics algorithm is used to calculate real-time data for adjusting the body posture. This data is compared with the pre-planned expected gait data, and the resulting error is processed and sent back to the control system for online gait adjustment control. This offline gait planning method is suitable for walking with a specific gait in a specific environment. When the environment changes, the robot will still walk according to the programmed gait planning method, which can cause the robot to fall because the gait is not adapted to the environment, and it is not adaptable.

[0004] DRL, a machine learning algorithm framework proposed by Google in 2015, has been widely applied in recent years to the problem of adaptive walking in bipedal robots. Using reinforcement learning algorithms, the robots use high-dimensional visual information as input to learn to adopt good behaviors such as avoiding walls and moving along a centerline. Adaptive walking in unknown environments is a partially observable Markov decision process (POMDP) ​​problem, where perception and decision-making are combined. The robot determines its current state based solely on proprioceptive sensor information and external radar data, independent of trajectory planning and gait design. Traditional Markov decision process (MDP)-based reinforcement learning strategies are unable to accurately measure the agent's current state in partially observable environments. Summary of the Invention

[0005] Purpose of the invention: In response to the above problems, the purpose of the present invention is to provide a bipedal robot walking stability optimization method based on an improved PPO algorithm, which can ensure the stability of the bipedal robot walking in a variety of environments.

[0006] Technical solution: A biped robot walking stability optimization method based on an improved PPO algorithm, comprising the following steps:

[0007] Step 1: Construction of improved PPO algorithm neural network model;

[0008] Establish an action network model and an evaluation network model. The action network uses one fully connected layer, one LSTM layer, and one MHSA layer. The evaluation network uses two fully connected layers and one LSTM layer. Step 2: Establish a bipedal robot motion model and select a simulation experimental environment.

[0009] The overall structure of the bipedal robot was simplified into a simple structure consisting of a waist, two thighs, and two calves. The simple structure includes two hip joints and two knee joints, all of which are rotational joints. A four-degree-of-freedom bipedal robot motion model was established, and Bipedalwalker-v3 in OpenAI Gym was selected as the simulation experimental environment.

[0010] Step 3: Pre-training of biped robot control parameters;

[0011] The constructed neural network model is used to process the environmental information transmitted by the bipedal robot in the simulation experimental environment and provide feedback to verify the feasibility of the neural network construction; the hyperparameters of the reinforcement learning model of the improved PPO algorithm include the learning rate η that affects the convergence speed and convergence value, the discount factor γ used to calculate the impact of past action rewards on current action rewards, λ that helps to speed up the learning speed, Epsilon (ε) that affects the range of gradient clipping, the number of samples taken in one training batch_size, and the maximum number of steps t;

[0012] Step 4: Stability analysis of bipedal robot walking motion;

[0013] Through training in a simulation environment and adjusting hyperparameters, the sliding average reward value of the bipedal robot's movements is used to prove its motion stability and verify the feasibility of the improved network.

[0014] Pre-training adjustment of control parameters uses deep reinforcement learning of the network structure of the improved PPO algorithm to train the controller; the walking motion stability of the bipedal robot is manifested by the reward value after training.

[0015] The present invention uses a deep network to handle the problem of observable Markov decision process in continuous space, and uses an algorithm based on policy gradient to ensure that actions are randomly selected in continuous space according to the learned good strategy.

[0016] Furthermore, the construction and training of the action network model includes the following steps:

[0017] S1: sequentially establish input layer 1, fully connected layer 1, long short-term memory layer 1, multi-head attention mechanism layer, and output layer 1;

[0018] S2: The environmental information s is input into the fully connected layer 1 through the input layer 1, and then enters the long short-term memory layer 1 after the output of the fully connected layer 1. The value after the output of the long short-term memory layer 1 is then input into the multi-head attention mechanism layer to obtain a high-dimensional representation of the state information with weight differences. The state information at different time steps is fitted to the variable weight dependency of subsequent actions, and finally output through an output layer 1;

[0019] S3: The output layer outputs two values: the mean and the variance. These two values ​​are then used to construct a normal distribution. This action probability distribution is then used to randomly sample a robot action. The randomly sampled action is then executed by the bipedal robot motion model.

[0020] S4: Combine all stored states s and input them into the action network under the new and old strategies θ, and obtain the normal distribution 1 and normal distribution 2 of the robot action probabilities under different strategies. Combine all stored actions into actions and input them into normal distribution 1 and normal distribution 2 to obtain prob1 and prob2 corresponding to each action. Then divide prob1 by prob2 to obtain the importance weight, i.e., ratio. Use importance sampling to correct the difference between the two action distributions of different strategies θ and θ, and calculate the loss function of the action network:

[0021] J θ， (θ)=E[min(r,clip(r,1-ε,1+ε))A θ， (s t ,a t )],

[0022] Using the gradient update formula To update the action network.

[0023] Furthermore, the bipedal robot executes multiple rounds of random sampling actions output by the action network model, interacting with the environment from the initial state s0 of the environment information to the final state s t The sample is regarded as a trajectory, represented by τ, τ: (s0, a0, s1, a1...s t-1 , a t-1 , st ), in a trajectory training, the robot judges the state of the current environment s t ∈S, according to the strategy π:S→A, take action a t ∈π(s t ), get reward r and get the next state s t+1 ; The goal of strategy optimization is to be in state s t , take action a t After that, until the expected cumulative return value G t The round ends when it is maximized, where:

[0024] Furthermore, the construction and training of the evaluation network model includes the following steps:

[0025] Step 1: Build the input layer 2, fully connected layer 2, fully connected layer 3, long short-term memory layer 2, and output layer 2 in sequence;

[0026] Step 2: Input all states s in the trajectory τ into the value network, and pass through the input layer 2, fully connected layer 2, fully connected layer 3, long short-term memory layer 2, and output layer 2 of the value network in sequence to obtain the state value V(s) corresponding to all states of the biped robot in a trajectory. t ):

[0027]

[0028] Step 3: G t As a robot performs a t Arrival state s t+1 After taking different actions, the average value of the expected cumulative rewards is G t =r t +γV(s t+1 ), we get the advantage function A π (s t ,a t )=G t -V(s t ), the advantage function is estimated by first-order time difference, and we get Where: t =r t +γV(s t+1 )-V(s t ), and thus find the loss function of the evaluation network Then backpropagate to update the value network, where N is the size of batch_size and T is the number of steps.

[0029] The best simulation environment in step 2 is the Bipedalwalker-v3 environment in OpenAI Gym.

[0030] Optimally, the environmental information s includes the biped robot’s posture, velocity, joint angles, and external radar information.

[0031] By using the improved PPO algorithm, long short-term memory is introduced into the network, and the temporal relationship between samples with the same distribution is utilized to improve the degree of fit between the incomplete observation state and the target output in the POMDP problem. In dealing with the temporal problem, the verification and filtering of state information are realized, thereby alleviating the problem of irrelevant information accumulated during the long-term trial and error learning process interfering with the robot's decision-making. LSTM (Long Short-Term Memory) is an improved RNN. It consists of an input gate t , an output gate output t and a forget gate t composition. forget t The previous unit information and the current state are used as input; the information ratio is controlled by the input gate, input t The proportion of information retained; C t The retained information and new information are weighted as the state of the current unit; output t Determine how much information is output and pass part of the information of the current unit to the next unit. t And the calculated output h at time t-1 t-1 As input, output y t and h t , pass in the next time t+1 for the next calculation, and use a state parameter c to realize the function of the forget gate.

[0032] In order to obtain a high-dimensional representation of state information with weight differences and fit the variable weight dependency of state information at different time steps on subsequent actions, the present invention adds a multi-head attention mechanism after long-short term memory. Each head focuses on different state information, performs parallel calculations, and finally splices them together. The attention mechanism can be divided into three steps. The first step is to obtain the state information of the robot after each action as input X = [x1L x n ], obtain the query vector sequence Q, key vector sequence K and value vector sequence V through linear transformation; the second is to calculate the attention distribution α i =softmax(s(k i ,q i )), where s(k i ,q i ) is the attention scoring mechanism. This paper adopts the dot product model, s(k i ,q i )=k i T q i; Third, calculate the weighted average of the state input information based on the attention distribution α: The multi-head attention mechanism is a variant of the attention mechanism, which uses multiple queries Q = [q1L q m ], state input information X=[x1L x n After the LSTM output, each attention unit focuses on different parts of the LSTM output information, calculates in parallel, and finally outputs different weights of the state information, which are then spliced:

[0033] Beneficial effects: Compared with the prior art, the advantages of the present invention are:

[0034] 1. By introducing long-short-term memory into the neural network, information from previous samples is selectively retained to fit the predictions of subsequent samples. This enables verification and filtering of state information when dealing with time series problems, thereby alleviating the problem of irrelevant information accumulated during long-term trial-and-error learning interfering with the robot's decision-making.

[0035] 2. After the input state passes through the LSTM output, an attention mechanism is introduced into the action network to fit the variable weight dependency of state information at different time steps on subsequent actions and value functions. This avoids the situation where, after the introduction of long short-term memory, the robot takes inappropriate actions and falls because the current information is not utilized by the strategy at later times.

[0036] 3. The improved PPO algorithm can make the trained network converge;

[0037] 4. The control stability and robustness of the bipedal robot controller in complex terrain environments are enhanced, enabling the robot to adapt to differentiated environmental changes. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 is a flow chart of the present invention;

[0039] Figure 2 Action network model for the improved PPO algorithm

[0040] Figure 3 Evaluation network model for the improved PPO algorithm;

[0041] Figure 4 This is the key frame diagram of the biped robot;

[0042] Figure 5 are the output values ​​of the four joint actions of the biped robot;

[0043] Figure 6 is the reward curve graph;

[0044] Figure 7 is the loss function graph. DETAILED DESCRIPTION

[0045] The present invention will be further described below with reference to the accompanying drawings and specific examples. It should be understood that these examples are only used to illustrate the present invention and are not used to limit the scope of the present invention.

[0046] Controlling the gait of bipedal robots in complex environments is difficult, and walking stability is crucial. Drawing on the research achievements and existing challenges in the field of humanoid bipedal robots, this paper proposes using an improved PPO algorithm to address the problem of bipedal robot walking control in complex environments. Without requiring a complex dynamics model for the bipedal robot, this paper applies a reinforced deep learning approach to achieve long-distance stable gait control for bipedal robots under fast walking conditions.

[0047] A biped robot walking stability optimization method based on improved PPO algorithm, such as Figure 1 As shown, it includes the establishment of a humanoid bipedal robot model and the selection of the environment, pre-training control parameters and the walking motion of the bipedal robot.

[0048] The humanoid bipedal robot model uses a simplified 4-DOF linkage model. Pre-trained control parameters are trained using a deep reinforcement learning controller with an improved PPO network structure. The stability of the bipedal robot's walking motion is primarily ensured by the trained controller. First, the simulated robot model and environment use the Bipedalwalker-v3 environment in Gym, developed by Open AI. During pre-training, importance sampling and gradient clipping are typically used to ensure better convergence of the improved PPO deep reinforcement learning network. A policy gradient algorithm is then used to ensure random selection of actions in a continuous action space based on the learned policy (action distribution). Finally, for the policy gradient algorithm, the action output of the action network is normally distributed with respect to mean and variance, ensuring that each action is selected, executed, rewarded, and then transitioned to the next state.

[0049] The PPO algorithm updates the strategy by collecting multiple trajectories containing n steps, which requires a large amount of sampling data and will also lead to the accumulation of irrelevant information in the long-term trial and error process. In order to selectively retain the sample information of the previous stage to fit the prediction of subsequent samples, the verification and filtering of the state information is implemented in the processing of timing issues, which alleviates the problem of irrelevant information accumulated in the long-term trial and error learning process interfering with the robot's decision-making; at the same time, in order to fit the variable weight dependence of the state information of different time steps on the subsequent actions, it avoids the situation where the robot takes inappropriate actions and falls because the information at the current moment is not utilized by the strategy at the later moment. The present invention provides a new correction algorithm based on the proximal policy optimization algorithm. This method can optimize the action part in the action-review algorithm, combined with the overall network structure after the improvement of the PPO network structure, such as Figure 2 、 3 The biped robot walking stability optimization method based on the improved PPO algorithm proposed in the present invention and applicable to the biped robot comprises the following steps:

[0050] Step 1: Construction of improved PPO algorithm neural network model;

[0051] Build action network models and evaluation network models. The action network uses one fully connected layer, one LSTM layer, and one MHSA layer. The evaluation network uses two fully connected layers and one LSTM layer.

[0052] Step 2: Establish a bipedal robot motion model and select a simulation experimental environment;

[0053] The overall structure of the bipedal robot was simplified into a simple structure consisting of a waist, two thighs, and two calves. The simple structure includes two hip joints and two knee joints, all of which are rotational joints. A four-degree-of-freedom bipedal robot motion model was established, and the toolkit in OpenAI Gym was selected as the simulation experimental environment.

[0054] Step 3: Pre-training of biped robot control parameters;

[0055] The constructed neural network model is used to process the environmental information transmitted by the bipedal robot in the simulation experimental environment and provide feedback to verify the feasibility of the neural network construction; the hyperparameters of the reinforcement learning model of the improved PPO algorithm include the learning rate η that affects the convergence speed and convergence value, the discount factor γ used to calculate the impact of past action rewards on current action rewards, λ that helps to speed up the learning speed, Epsilon (ε) that affects the range of gradient clipping, the number of samples taken in one training batch_size, and the maximum number of steps t;

[0056] Step 4: Stability analysis of bipedal robot walking motion;

[0057] Through training in a simulation environment and adjusting hyperparameters, the sliding average reward value of the bipedal robot's movements is used to prove its motion stability and verify the feasibility of the improved network.

[0058] An improved PPO algorithm is used to adjust hyperparameters to achieve stable gait control for a bipedal robot. This improved algorithm is a model-free, offline reinforcement learning method. It is also an actor-critic reinforcement learning agent that computes the optimal policy that maximizes long-term reward. Key parameters of this algorithm are listed in Table 1.

[0059] Table 1 Key parameters of the improved PPO algorithm

[0060]

[0061] The construction and training of the action network model includes the following steps:

[0062] S1: sequentially establish input layer 1, fully connected layer 2, long short-term memory layer 3, multi-head attention mechanism layer 4, and fully connected layer 2 5;

[0063] S2: The environmental information s is input into the fully connected layer 2 through the input layer 1. After the output of the fully connected layer 2, it enters the long short-term memory layer 3. The value after the output of the long short-term memory layer 3 is then input into the multi-head attention mechanism layer 4 to obtain a high-dimensional representation of the state information with weight differences. The state information at different time steps is fitted to the variable weight dependency of subsequent actions. Finally, it is output through an output layer 5. The environmental information s includes the bipedal robot's position, velocity, joint angles, and external radar information.

[0064] S3: Output layer 15 outputs two values: the mean and the variance. These two values ​​are then used to construct a normal distribution, which represents the probability distribution of the robot taking an action in its current state. This action probability distribution is then used to randomly sample an action for the robot. The randomly sampled action is then executed by the bipedal robot motion model.

[0065] S4: Combine all stored states s and input them into the action network under the new and old strategies θ, and obtain the normal distribution 1 and normal distribution 2 of the robot action probabilities under different strategies. Combine all stored actions into actions and input them into normal distribution 1 and normal distribution 2 to obtain prob1 and prob2 corresponding to each action. Then divide prob1 by prob2 to obtain the importance weight, i.e., ratio. Use importance sampling to correct the difference between the two action distributions of different strategies θ and θ, and calculate the loss function of the action network:

[0066] J θ，(θ)=E[min(r,clip(r,1-ε,1+ε))A θ， (s t ,a t )],

[0067] Using the gradient update formula To update the action network.

[0068] The bipedal robot executes multiple rounds of random sampling actions output by the action network model, interacting with the environment from the initial state s0 of the environment information to the final state s t The sample is regarded as a trajectory, represented by τ, τ: (s0, a0, s1, a1...s t-1 , a t-1 , s t ), in a trajectory training, the robot judges the state of the current environment s t ∈S, according to the strategy π:S→A, take action a t ∈π(s t ), get reward r and get the next state s t+1 ; The goal of strategy optimization is to be in state s t , take action a t After that, until the expected cumulative return value G t The round ends when it is maximized, where: In the design of the reward function, moving forward will receive a positive reward signal, falling will receive a reward signal of -100, and at the same time, driving each joint to rotate will receive a certain negative reward signal.

[0069] Long short-term memory utilizes the temporal relationship between samples of the same distribution, improves the degree of fit of the incomplete observation state to the target output in the POMDP problem, and realizes the verification and filtering of state information in processing temporal problems, thereby alleviating the problem of irrelevant information accumulated in the long-term trial-and-error learning process interfering with the robot's decision-making.

[0070] The construction and training of the evaluation network model includes the following steps:

[0071] Step 1: Build the input layer 2 (6), the fully connected layer 2 (7), the fully connected layer 3 (8), the long short-term memory layer 2 (9), and the output layer 2 (10) in sequence;

[0072] Step 2: Input all states s in the trajectory τ into the value network, and pass through the input layer 2 6, the fully connected layer 2 7, the fully connected layer 3 8, the long short-term memory layer 2 9, and the output layer 2 10 of the value network in sequence to obtain the state value V(s) corresponding to all states of the biped robot in a trajectory. t ):

[0073]

[0074] Step 3: In order to reduce the error, G t As a robot performs a t Arrival state s t+1 After taking different actions, the average value of the expected cumulative rewards is G t =r t +γV(s t+1 ), we get the advantage function: A π (s t ,a t )=G t -V(s t ), the advantage function is estimated by first-order time difference, and we get Where: t =r t +γV(s t+1 )-V(s t ), and thus find the loss function of the evaluation network Then backpropagate to update the value network, where N is the size of batch_size and T is the number of steps.

[0075] The control strategy of the robot is determined based on the control parameters to make the robot walk, and the feasibility of this method is verified using the environment in OpenAI Gym.

[0076] Select one of the stable walking control experiments, and the key frames of the robot's motion on uneven ground are as follows: Figure 4 As shown, the figure shows a bipedal robot from Figure 4 The continuous forward walking process from a to 4d is very stable, and falls rarely occur. The hip and knee joints can alternate and periodically move, and the hip joint swings back and forth to control the uniform step length. The knee joint swing allows the robot to step steadily on the ground, thus showing the normal walking of the robot. Figure 4 It can be concluded that the bipedal robot can always walk normally on the rugged terrain during the entire walking task, adjust its gait to adapt to the complex terrain, and never fall, and always maintain a stable posture; this stability can also be verified from the subsequent joint output values ​​and reward image curves. After the robot walks stably, the output values ​​of the actions made by the four joints during its movement are as follows Figure 5 As shown by Figure 5 It can be seen that the four joints of the robot fluctuate periodically around the 0 axis, indicating that the robot's joints are performing stable periodic motion; the reward value image is as follows Figure 6 As shown, the loss function diagram is as follows Figure 7 As shown by Figure 6 、 7From the curve in , we can see that after continuous trial and error, the rewards gradually converged steadily after 1,000 rounds. This means that after multiple rounds of exploration, the robot can walk stably on the ground with actions that have relatively high rewards, and the stability of the bipedal robot during walking is also guaranteed.

Claims

1. A biped robot walking stability optimization method based on an improved PPO algorithm, characterized by The following steps are involved: Step 1: Construction of improved PPO algorithm neural network model; Build action network models and evaluation network models. The action network uses one fully connected layer, one LSTM layer, and one MHSA layer. The evaluation network uses two fully connected layers and one LSTM layer. Step 2: Establish a bipedal robot motion model and select a simulation experimental environment; The overall structure of the bipedal robot was simplified into a simple structure consisting of a waist, two thighs, and two calves. The simple structure includes two hip joints and two knee joints, all of which are rotational joints. A four-degree-of-freedom bipedal robot motion model was established, and the toolkit in OpenAI Gym was selected as the simulation experimental environment. Step 3: Pre-training of biped robot control parameters; The constructed neural network model is used to process the environmental information transmitted by the bipedal robot in the simulation experimental environment and provide feedback to verify the feasibility of the neural network construction; the hyperparameters of the reinforcement learning model of the improved PPO algorithm include the learning rate η that affects the convergence speed and convergence value, the discount factor γ used to calculate the impact of past action rewards on current action rewards, λ that helps to speed up the learning speed, Epsilon (ε) that affects the range of gradient clipping, the number of samples taken in one training batch_size, and the maximum number of steps t; Step 4: Stability analysis of bipedal robot walking motion; Through training in a simulation environment and adjusting hyperparameters, the sliding average reward value of the bipedal robot's movements is used to prove its motion stability and verify the feasibility of the improved network. The construction and training of the action network model includes the following steps: S1: sequentially establish input layer 1 (1), fully connected layer 1 (2), long short-term memory layer 1 (3), multi-head attention mechanism layer (4), and output layer 1 (5); S2: The environmental information s is inputted into the fully connected layer 1 (2) through the input layer 1 (1), and then enters the long short-term memory layer 1 (3) after being outputted by the fully connected layer 1 (2). The value outputted by the long short-term memory layer 1 (3) is then inputted into the multi-head attention mechanism layer (4) to obtain a high-dimensional representation of the state information with weight differences, and the state information of different time steps is fitted to the variable weight dependency of the subsequent actions, and finally outputted through an output layer 1 (5); S3: Output layer 1 (5) outputs two values, one is the mean and the other is the variance. Then, a normal distribution is constructed using these two values. Then, a robot action is randomly sampled through this action probability distribution. The randomly sampled action is executed by the bipedal robot motion model. S4: Combine all stored states s and input them into the action network under the new and old strategies θ' and θ to obtain the normal distribution 1 and normal distribution 2 of the robot action probabilities under different strategies. Combine all stored actions into actions and input them into normal distribution 1 and normal distribution 2 to obtain prob1 and prob2 corresponding to each action. Then divide prob1 by prob2 to obtain the importance weight, i.e., ratio. Use importance sampling to correct the difference between the two action distributions of different strategies θ and θ', and calculate the loss function of the action network: J θ’ (θ)=E[min(r,clip(r,1-ε,1+ε))A θ’ (s t ,a t )], Using the gradient update formula To update the action network; The bipedal robot executes multiple rounds of random sampling actions output by the action network model, interacting with the environment from the initial state s0 of the environment information to the final state s t The sample is regarded as a trajectory, represented by τ, τ: (s0, a0, s1, a1...s t-1 , a t-1 , s t ), in a trajectory training, the robot judges the state of the current environment s t ∈S, according to the strategy π:S→A, take action a t ∈π(s t ), get reward r and get the next state s t+1 ; The goal of strategy optimization is to be in state s t , take action a t After that, until the expected cumulative return value G t The round ends when it is maximized, where: The construction and training of the evaluation network model includes the following steps: Step 1: Build the input layer 2 (6), fully connected layer 2 (7), fully connected layer 3 (8), long short-term memory layer 2 (9), and output layer 2 (10) in sequence; Step 2: Input all states s in the trajectory τ into the value network, and pass through the input layer 2 (6), fully connected layer 2 (7), fully connected layer 3 (8), long short-term memory layer 2 (9), and output layer 2 (10) of the value network in sequence to obtain the state value V(s) corresponding to all states of the biped robot in a trajectory. t ): Step 3: G t As a robot performs a t Arrival state s t+1 After taking different actions, the average value of the expected cumulative rewards is G t =r t +γV(s t+1 ), we get the advantage function A π (s t ,a t )=G t -V(s t ), the advantage function is estimated by first-order time difference, and we get in: δ t =r t +γV(s t+1 )-V(s t ), and thus find the loss function of the evaluation network Then backpropagate to update the value network, where N is the size of batch_size and T is the number of steps.

2. The method for optimizing bipedal robot walking stability based on the improved PPO algorithm according to claim 1, characterized in that: In step 2, the simulation experiment environment selected is the Bipedalwalker-v3 environment in OpenAI Gym.

3. The method for optimizing bipedal robot walking stability based on the improved PPO algorithm according to claim 1, characterized in that: The environmental information s includes the biped robot's position, velocity, joint angles and external radar information.

Images