A reinforcement learning method and program product for controlling the motion of a biped robot based on trajectory self-correction

Abstract

The present invention discloses a reinforcement learning method for controlling the motion of a bipedal robot based on trajectory self-correction, which uses a strategy model to output decision actions and interact with the bipedal robot online to obtain real environment samples; uses an environment dynamics model to simulate the motion state of the bipedal robot, and uses the obtained real environment samples and a supervised learning method to train the environment dynamics model; then interacts with the strategy model to obtain virtual samples, aligns the real samples with the virtual samples for trajectory self-correction, uses error control to filter the virtual samples, and after the environment dynamics model is updated, uses the strategy to interact with the environment dynamics model to generate new and more accurate training virtual samples, and uses the real environment samples and the new training virtual samples to jointly train the strategy model. The present invention only requires a small amount of interaction between the robot and the environment to achieve a cumulative reward similar to that of model-free reinforcement learning, thereby improving the learning efficiency and accuracy of the bipedal robot motion strategy.

Description

Technical Field

[0001] The invention belongs to the field of robot motion control and relates to a reinforcement learning method for controlling the motion of a biped robot based on trajectory self-correction. Background Art

[0002] How to learn the optimal strategy to control the motion of a bipedal robot in a shorter time is a challenging control problem. The motion of the bipedal robot can be modeled as a Markov decision process, and then a reinforcement learning strategy optimization algorithm can be used to solve it.

[0003] Model-based reinforcement learning is a method of reinforcement learning in which the agent attempts to learn a model of the environment and then uses the model to make decisions and learn. The agent usually interacts directly with the environment and learns the optimal strategy through trial and error. Model-based reinforcement learning introduces an environmental model, which can be used to generate simulated samples, reducing the algorithm's demand for samples. The advantage of model-based reinforcement learning is that it can more effectively use environmental information to make decisions, thereby improving learning efficiency.

[0004] Although the existing model-based reinforcement learning method can solve the motion strategy of the biped robot, this method is sensitive to the accuracy of the model. When the model is inaccurate, the samples generated by the model will contain large errors. Using the simulated samples with errors to learn the strategy will slow down the learning speed of the strategy, and in severe cases, cause the strategy to diverge. On the other hand, in the real environment, the acquisition of samples often has a large acquisition cost. Therefore, how to learn an accurate model with only a small number of real samples and use the model to promote strategy learning is an urgent problem to be solved. Summary of the invention

[0005] Purpose of the invention: In order to overcome the deficiencies in the prior art, a reinforcement learning method for controlling the motion of a bipedal robot based on trajectory self-correction is provided to solve the problem of promoting rapid strategy learning when the number of real samples is small.

[0006] To achieve the above object, the technical solution of the present invention is as follows: A reinforcement learning method for controlling the motion of a biped robot based on trajectory self-correction comprises the following steps:

[0007] Step 1: Initialize the parameters of the robot action strategy model and the environment dynamics model, the input of the environment dynamics model is the state dimension and action dimension of the robot, and the output of the environment dynamics model is the state dimension and reward dimension of the next state of the robot;

[0008] Step 2: Set the current time step t = 0 and sample the initial state s0 from the initial state distribution P(s0);

[0009] Step 3: Online sampling of real samples: In state s t Output action a according to the robot action strategy model t , the robot performs action a t Then get the state s t+1 and reward r t+1 , get the real sample (s t ,a t ,s t+1 ,r t+1 );

[0010] Step 4: The real sample (s t ,a t ,s t+1 ,r t+1 ) Insert the real sample experience replay pool to update the real sample experience replay pool and update the current state s t =s t+1 ;

[0011] Step 5: Use samples in the real sample experience replay pool to update the parameters of the environment dynamics model;

[0012] Step 6: Randomly select a sample’s state and trajectory action from the real sample pool and input it into the environment dynamics model to obtain the virtual state and virtual reward, and construct a virtual sample set;

[0013] Step 7: Align the virtual samples in the virtual sample set with the real samples in the corresponding real sample pool to construct an aligned virtual sample set;

[0014] Step 8: Use the real samples in the real sample pool and the virtual samples in the aligned virtual sample set to update the parameters of the environment dynamics model;

[0015] Step 9: Starting from the initial state, generate the current action according to the robot action strategy model, input the environment dynamics model to obtain the next state and reward, and construct a training virtual sample set;

[0016] Step 10: Take out the real samples in the real sample pool and the training virtual samples in the training virtual sample set to update the robot action strategy model parameters;

[0017] Step 11: Repeat steps 3 to 10 until the current time step reaches the maximum value and then proceed to step 12;

[0018] Step 12: Repeat steps 2 to 11 until the maximum number of iterations is reached and then proceed to step 13;

[0019] Step 13: Based on the state s t The updated robot action strategy model outputs action at .

[0020] In the learning stage of the environment dynamics model, the scheme of the present invention first uses it to construct a multi-step virtual sample trajectory. The virtual sample set and the real samples in the real sample experience replay pool are aligned to further train the environment dynamics model, thereby achieving efficient learning. In the strategy training stage, the real samples in the real sample experience replay pool and the virtual samples in the training virtual sample set generated by the environment dynamics model are directly used to jointly train the strategy, thereby achieving rapid convergence of the strategy with only a small number of real samples.

[0021] Furthermore, when constructing the virtual sample set in step 6, a sample s is randomly selected from the real sample pool. k As the initial state and from s k The corresponding action a k The initial H actions {a k ,a k+1 ,a k+2 ,…,a k+H-1}, H is the length of the sampling trajectory, starting from the initial state s t ′=s k Start, according to the current action a′ t =a k , the state s t ′ and action a′ t Input environment dynamics model T ω Get virtual state s t ' +1 and virtual rewards t ' +1 =T ω (s t ′,a′ t ), let a′ t+1 =a k+1 , the virtual state s t ' +1 and action a′ t+1 Input environment dynamics model T ω Get the next virtual state s t ' +2 and virtual rewards t ' +2 =T ω (s t ' +1 ,a′ t+1 ), repeat until the virtual state s is obtained t ' +H and virtual rewards t ' +H , the virtual sample set is {(s t ′,a′t ,r t ' +1 ,s t ' +1 ),(s t ' +1 ,a′ t+1 ,r t ' +2 ,s t ' +2 ),…,(s t ' +H-1 ,a′ t+H-1 ,r t ' +H ,s t ' +H )}.

[0022] Furthermore, when aligning the virtual samples in the virtual sample set with the real samples in the corresponding real sample pool in step 7, the real sample collection corresponding to the virtual sample set is {(s k ,a k ,r k+1 ,s k+1 ),(s k+1 ,a k+1 ,r k+2 ,s k+2 ),…,(s k+H-1 ,a k+H-1 ,r k+H ,s k+H )}, the aligned virtual sample set is {(s′ k ,a′ k ,r k+1 ,s k+1 ),(s′ k+1 ,a′ k+1 ,r k+2 ,s k+2 ),…,(s′ k+H-1 ,a′ k+H-1 ,r k+H ,s k+H )}, calculate the Euclidean distance d between the states of each paired real sample and the aligned virtual sample. When the Euclidean distance is greater than a preset threshold d0, the aligned virtual sample is discarded. Otherwise, the aligned virtual sample is added to the aligned virtual sample set. The value of d0 is the mean of the errors of all paired real samples and aligned virtual samples in the previous iteration.

[0023] To avoid compound errors, the virtual sample trajectory is aligned with the real sample trajectory, and the Euclidean distance is used to judge the error of the aligned samples. When the threshold is less than the preset value, the virtual sample is added to the aligned virtual sample set.

[0024] Furthermore, when constructing the training virtual sample set in step 9, from the initial state s′ t =s0, the robot action strategy model π θ Generate current action a′ t =π θ (s′ t ), the state s′ t and action a′ t Input environment dynamics model T ω Get state s′ t+1 and reward r′ t+1 =T ω (s′ t ,a′ t ), let a′ t+1 =π θ (s′ t ), the state s′ t+1 and action a′ t+1 Input environment dynamics model T ω Get the next state s′ t+2 and reward r′ t+2 =T ω (s′ t+1 ,a′ t+1 ), repeat until the next state s′ is obtained t+H and reward r′ t+H , the training virtual sample set is {(s′ t ,a′ t ,r′ t+1 ,s′ t+1 ),(s′ t+1 ,a′ t+1 ,r′ t+2 ,s′ t+2 ),…,(s′ t+H-1 ,a′ t+H-1 ,r′ t+H ,s′ t+H )}.

[0025] Furthermore, the parameters of the environment dynamics model are updated in step 5 and step 8 by using a gradient descent algorithm. is the gradient of the loss function L(ω) of the environment dynamics model with respect to the parameter ω, and α2 is the learning rate.

[0026] Furthermore, the environmental dynamics model is a probabilistic neural network consisting of three fully connected layers, and the loss function

[0027]

[0028] st is the state at time t, a t is the action at time t, s t+1 is the state at the next moment, μ ω (s t ,a t ) is the mean of the state at the next moment, is the variance of the state at the next moment, and the output of the environmental dynamics model is

[0029] Furthermore, the step 10 updates the robot action strategy model parameters by using a gradient descent algorithm, and the parameters of the robot action strategy model are updated. is the gradient of the loss function L(θ) of the robot action strategy model with respect to the parameter θ, and α1 is the learning rate.

[0030] Furthermore, the loss function of the robot action strategy model is

[0031]

[0032] D KL represents KL divergence calculation, Q represents action value function, and V represents state index function.

[0033] Furthermore, in step 4, the real sample (s t ,a t ,s t+1 ,r t+1 ) is inserted into the real sample experience replay pool to update the real sample experience replay pool. When the number of samples in the real sample experience replay pool is less than the maximum number M allowed, it is directly added. Otherwise, the real sample (s t ,a t ,s t+1 ,r t+1 ) replaces the sample that was first added to the real sample experience replay pool.

[0034] The present invention also provides a computer program product, including a computer program or a computer executable instruction, which, when executed by a processor, implements the aforementioned reinforcement learning method for controlling the motion of a bipedal robot based on trajectory self-correction.

[0035] Compared with the prior art, the advantages of the present invention are: (1) a more accurate environmental dynamics model can be learned by aligning only a small number of real samples with simulated samples generated by the environmental dynamics model, which reduces the demand for real samples and reduces the learning cost of the environmental dynamics model of the bipedal robot; (2) the trajectory self-correction method can be used to align and correct the simulated trajectory with errors generated by the model with real trajectory samples, and the corrected samples are used to correct the environmental dynamics model, thereby improving the accuracy of the environmental dynamics model, making the simulated samples of strategy learning closer to the real samples, alleviating the cumulative error caused by the environmental dynamics model, avoiding misleading strategy learning due to inaccurate simulated samples, accelerating strategy learning, and improving the sample efficiency of bipedal robot strategy learning. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 The invention discloses a flow chart of a reinforcement learning method for controlling the motion of a biped robot based on trajectory self-correction.

[0037] Figure 2 Schematic diagram of a trajectory self-correction method in an embodiment of the present invention.

[0038] Figure 3 Schematic diagram of the comparison of average cumulative rewards using the self-correction method and other methods in an embodiment of the present invention. DETAILED DESCRIPTION

[0039] The present invention is further explained below in conjunction with the accompanying drawings and specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention. After reading the present invention, various equivalent forms of modifications to the present invention by those skilled in the art all fall within the scope defined by the claims attached to this application.

[0040] The present invention combines Figure 1 and Figure 2 As shown, this embodiment provides an optimal strategy learning method for biped robot motion control based on trajectory self-correction model-based reinforcement learning, which applies torque to the 9 connecting rods and 8 connecting heads (including two legs) of the biped robot through the learned optimal strategy, coordinates the two legs to move forward, so that its final behavior strategy can obtain more rewards, that is, continue to move forward without falling. The specific optimal strategy learning method includes the following steps:

[0041] Step 1. Initialize hyperparameters: construct robot action strategy models π represented by fully connected neural networks θ and environmental dynamics model T ω ,The input of the environment dynamics model is 23 dimensions, which includes 17 state dimensions and 6 action dimensions of the biped robot, as shown in Table 1 and Table 2 respectively.

[0042] Table 1 State space of bipedal robot

[0043]

[0044]

[0045] Table 2 Action space of bipedal robot

[0046] serial number action Control minimum Control Maximum unit 0 Torque applied to the posterior thigh trochanter -1 1 Torque(N m) 1 Torque applied to the hind leg trochanter -1 1 Torque(N m) 2 Torque applied to the rear trochanter -1 1 Torque(N m) 3 Torque applied to the anterior thigh trochanter -1 1 Torque(N m) 4 Torque applied to the anterior trochanter -1 1 Torque(N m) 5 Torque applied to the front trochanter -1 1 Torque(N m)

[0047] The dimension of the next state output by the environment dynamics model is 18, which includes 17 state dimensions and 1 reward dimension. Robot action strategy model π θ and environmental dynamics model T ω The corresponding network parameters are θ and ω respectively; these parameters are sampled using a uniform distribution between [0, 0.1] to initialize these parameters and implement the robot action strategy model π θ and environmental dynamics model T ω Initialization of the robot action strategy model π θ and environmental dynamics model T ω The learning rates α1 and α2 are both set to 0.001, the length of the sampling trajectory is H = 2, the maximum number of iterations is E = 200000, the initial state distribution is P(s0), the maximum number of samples in the real sample experience replay pool is M = 150000, the current number of iterations is e, and the current number of iterations is initialized to e = 0.

[0048] Step 2: Initialize the environment: Set the current time step t = 0, and start from the initial state distribution P(s0) = {s1, s2, ..., s E} and sample the initial state s0.

[0049] Step 3: Online sampling of real samples: In state s t Output action a according to the robot action strategy model t , the robot performs action a t Then get the state s t+1 and reward r t+1 , get the real sample (s t ,a t ,s t+1 ,r t+1 ).

[0050] Step 4: Update the real sample experience replay pool: Update the real sample (s t ,a t ,s t+1 ,r t+1) is inserted into the real sample experience replay pool. When the number of samples in the real sample experience replay pool is less than M, it is directly added; otherwise, (s t ,a t ,s t+1 ,r t+1 ) replaces the first sample added to the real sample experience replay pool and updates the current state to s t =s t+1 .

[0051] Step 5: Update the parameters of the environment dynamics model ω: Use the samples in the real sample experience playback pool to calculate the environment dynamics model T ω The loss function L(ω) is then calculated to obtain the gradient of the loss function L(ω) with respect to the parameter ω. Update the environment dynamics model parameters through the gradient descent algorithm

[0052] Environmental Dynamics Model T ω Using deep probabilistic neural network To achieve this, the input of the environment dynamics model network is the state action (s t ,a t ) has 23 dimensions, and the output is the state s at the next time t+1 t+1 The mean μ ω (s t ,a t ) and variance

[0053] The environment dynamics model is a probabilistic neural network consisting of three fully connected layers, with 200 network nodes in each layer. The samples learned by the environment dynamics model come from real samples generated by the actual interaction between the intelligent agent and the bipedal robot (s t ,a t ,s t+1 ), 1≤t≤T, T represents the length of the sampled trajectory, that is, the number of real samples in the trajectory. The policy network is trained using supervised learning, and the environment dynamics model T is calculated using the negative maximum likelihood estimation method. ω The loss function L(ω):

[0054]

[0055] Among them, (μ ω (s t ,a t )+s t ) is the next state s t+1 estimate.

[0056] Step 6: Construct a virtual sample collection: randomly select a sample s from the real sample poolk As the initial state and from s k The corresponding action a k The initial H actions {a k ,a k+1 ,a k+2 ,…,a k+H-1}, H is the length of the sampling trajectory, starting from the initial state s t ′=s k Start, according to the current action a′ t =a k , the state s t ′ and action a′ t Input environment dynamics model T ω Get virtual state s t ' +1 and virtual rewards t ' +1 =T ω (s t ′,a′ t ), let a′ t+1 =a k+1 , the virtual state s t ' +1 and action a′ t+1 Input environment dynamics model T ω Get the next virtual state s t ' +2 and virtual rewards t ' +2 =T ω (s t ' +1 ,a′ t+1 ), repeat H times until the virtual state s is obtained t ' +H and virtual rewards t ' +H , a total of H virtual samples are generated. The virtual sample set is {(s t ′,a′ t ,r t ' +1 ,s t ' +1 ),(s t ' +1 ,a′ t+1 ,r t ' +2 ,s t ' +2 ),…,(s′ k+H-1 ,a′ k+H-1 ,r t ' +H ,s t ' +H)}.

[0057] Step 7: Use the real sample and the virtual sample to align and correct the trajectory: The trajectory of the H samples selected from the real sample pool is {(s k ,a k ,r k+1 ,s k+1 ),(s k+1 ,a k+1 ,r k+2 ,s k+2 ),…,(s k+H-1 ,a k+H-1 ,r k+H ,s k+H )}, the corresponding virtual trajectory is

[0058] {(s′ k ,a′ k ,r k ' +1 ,s′ k+1 ),(s′ k+1 ,a′ k+1 ,r k ' +2 ,s′ k+2 ),…,(s′ k+H-1 ,a′ k+H-1 ,r k ' +H ,s′ k+H )}, and then align the trajectory: let the next state and reward of the virtual sample be the next state and reward of the real reward, such as the virtual sample after alignment at time step k is (s′ k ,a′ k ,r k+1 ,s k+1 ), then starting from time step k, H time steps are aligned in sequence, and the aligned virtual sample set can be obtained as {(s′ k ,a′ k ,r k+1 ,s k+1 ),(s′ k+1 ,a′ k+1 ,r k+2 ,s k+2 ),…,(s′ k+H-1 ,a′ k+H-1 ,r k+H ,s k+H )}, and then calculate the Euclidean distance d between the states of each real sample and the aligned virtual sample, such as the real sample is (s k ,a k ,r k+1 ,s k+1 ), the aligned virtual sample is (s′ k,a′ k ,r k+1 ,s k+1 ), then the Euclidean distance When d is greater than the threshold d0, it is considered that the aligned virtual sample still has a large error, and the sample is discarded. Otherwise, the aligned simulated sample is added to the aligned virtual sample set. The value of d0 is the mean of the errors of all paired real samples and aligned virtual samples in the previous iteration.

[0059] Step 8: Update the environmental dynamics model parameter ω: Similar to step 5, use the aligned virtual sample set {(s′ k ,a′ k ,r k+1 ,s k+1 ),(s′ k+1 ,a′ k+1 ,r k+2 ,s k+2 ),…,(s′ k+H-1 ,a′ k+H-1 ,r k+H ,s k+H )} to calculate the environmental dynamics model T ω The loss function L(ω) is then calculated to obtain the gradient of the loss function L(ω) with respect to the parameter ω. Update the environment dynamics model parameters through the gradient descent algorithm

[0060] Step 9: Get virtual training samples for training strategy: From the initial state s′ t =s0, the robot action strategy model π θ Generate current action a′ t =π θ (s′ t ), the state s′ t and action a′ t Input environment dynamics model T ω Get state s′ t+1 and reward r′ t+1 =T ω (s′ t ,a′ t ), let a′ t+1 =π θ (s′ t ), the state s′ t+1 and action a′ t+1 Input environment dynamics model T ω Get the next state s′ t+2 and reward r′ t+2 =T ω (s′ t+1 ,a′t+1 ), repeat until the next state s′ is obtained t+H and reward r′ t+H , the training virtual sample set is

[0061] {(s′ t ,a′ t ,r′ t+1 ,s′ t+1 ),(s′ t+1 ,a′ t+1 ,r′ t+2 ,s′ t+2 ),…,(s′ t+H-1 ,a′ t+H-1 ,r′ t+H ,s′ t+H )}.

[0062] Step 10: Update the strategy parameters θ: Take out the actual samples in the real sample experience playback pool and the new training virtual samples, and update the parameters of the robot action strategy model is the gradient of the loss function L(θ) of the robot action strategy model with respect to the parameter θ, and α1 is the learning rate.

[0063] Use the SAC strategy optimization algorithm to optimize the strategy and the loss function of the robot action strategy model

[0064] L(θ)=E st {D KL (π θ ||exp(QV))}.

[0065] Step 11: Determine whether the maximum time step of the plot has been reached: if reached, proceed to step 12; otherwise, update the current time step t=t+1 and proceed to step 3 to continue execution.

[0066] Step 12: Determine whether the maximum number of episodes has been reached: if reached, proceed to step 13; otherwise, update the current episode number e=e+1, and proceed to step 2 to continue execution.

[0067] Step 13: Based on state s t The updated robot action strategy model outputs action a t , that is, the optimal parameter θ of the learned robot action strategy model * , generating the optimal strategy π θ* (s t ,a t ).

[0068] Figure 3This is a comparison chart of the cumulative rewards of the embodiment of the present invention and other methods in the biped robot simulation experiment, where the patent is the top line, that is, during the entire simulation, the reward of the biped robot is higher than that of other cases. Init-200 means that the initial state of the real trajectory is used as the initial state of the trajectory sampling, and the sampling length H = 200. Ran-100 means randomly selecting a state as the initial state, and the sampling length H = 100. Figure 3 It can be seen that the reinforcement learning method of the present invention for controlling the motion of a biped robot based on trajectory self-correction can learn a better strategy using only relatively fewer real samples, and the demand for real samples is small, which reduces the learning cost of the optimal strategy of the intelligent agent; the corrected simulated trajectory samples and the real trajectory samples are used to jointly train the environmental dynamics model, and finally the intelligent agent strategy and the environmental dynamics model interact to generate new and more accurate simulated trajectory samples, and the real environment samples and the new simulated samples are used to jointly train the strategy. The sample efficiency of the intelligent agent strategy learning is improved.

Claims

1. A reinforcement learning method for controlling the motion of a bipedal robot based on trajectory self-correction, It is characterized in that The steps include: Step 1: Initialize the parameters of the robot action strategy model and the environment dynamics model, the input of the environment dynamics model is the state dimension and action dimension of the robot, and the output of the environment dynamics model is the state dimension and reward dimension of the next state of the robot; Step 2: Set the current time step , from the initial state distribution Initial state of sampling ; Step 3: Online sampling of real samples: In state Output actions according to the robot action strategy model , the robot performs actions After getting the status and rewards , get the real sample ; Step 4: Take the real sample Insert into the real sample experience replay pool, update the real sample experience replay pool, and update the current status ; Step 5: Use samples in the real sample experience replay pool to update the parameters of the environment dynamics model; Step 6: Randomly select a sample from the real sample pool, input the state and trajectory action of the environment dynamics model to obtain the virtual state and virtual reward, and construct a virtual sample set; when constructing the virtual sample set, randomly select a sample from the real sample pool As the initial state and from Corresponding actions Initial H actions , H is the length of the sampling trajectory, from the initial state Departure, according to the current action , the state and actions Input Environment Dynamics Model Get virtual status and virtual rewards ,make , the virtual state and actions Input Environment Dynamics Model Get the next virtual state and virtual rewards , repeat until you get the virtual state and virtual rewards , the virtual sample set is ; Step 7: Align the virtual samples in the virtual sample set with the real samples in the corresponding real sample pool to construct an aligned virtual sample set; the real sample set corresponding to the virtual sample set is , the aligned virtual sample set is , calculate the Euclidean distance d between the states of each paired real sample and the aligned virtual sample, and when the Euclidean distance is greater than a preset threshold , then discard the aligned virtual sample; otherwise, add the aligned virtual sample to the aligned virtual sample set. The value of is the mean error of all paired real samples and aligned virtual samples in the previous iteration; Step 8: Use the real samples in the real sample pool and the virtual samples in the aligned virtual sample set to update the parameters of the environment dynamics model; Step 9: Starting from the initial state, generate the current action according to the robot action strategy model, and input it into the environment dynamics model to obtain the next state and reward, and construct a training virtual sample set; Step 10: Take out the real samples in the real sample pool and the training virtual samples in the training virtual sample set to update the robot action strategy model parameters; Step 11: Repeat steps 3 to 10 until the current time step reaches the maximum and then proceed to step 12; Step 12: Repeat steps 2 to 11 until the maximum number of iterations is reached and then proceed to step 13; Step 13: Based on the state The action is output by the updated robot action strategy model .

2. The reinforcement learning method for controlling the motion of a biped robot based on trajectory self-correction according to claim 1, It is characterized in that When constructing the training virtual sample set in step 9, from the initial state Starting from the robot action strategy model Generate current action , the state and actions Input Environment Dynamics Model Get Status and rewards ,make , the state and actions Input Environment Dynamics Model Get the next state and rewards , repeat until the next state is obtained and rewards , the training virtual sample set is .

3. The reinforcement learning method for controlling the motion of a biped robot based on trajectory self-correction according to claim 1, It is characterized in that The parameters of the environment dynamics model are updated in step 5 and step 8 by using a gradient descent algorithm. is the loss function of the environment dynamics model Parameters The gradient of is the learning rate.

4. The reinforcement learning method for controlling the motion of a biped robot based on trajectory self-correction according to claim 3, It is characterized in that The environmental dynamics model is a probabilistic neural network consisting of three fully connected layers, and the loss function , is the state at time t, is the action at time t, is the state at the next moment, is the mean of the state at the next moment, is the variance of the state at the next moment, and the output of the environmental dynamics model is .

5. The reinforcement learning method for controlling the motion of a biped robot based on trajectory self-correction according to claim 1, It is characterized in that The step 10 updates the robot action strategy model parameters by using a gradient descent algorithm. The parameters of the robot action strategy model are is the loss function of the robot action strategy model Parameters The gradient of is the learning rate.

6. The reinforcement learning method for controlling the motion of a biped robot based on trajectory self-correction according to claim 5, It is characterized in that The loss function of the robot action strategy model , represents KL divergence calculation, Q represents action value function, V represents state index function, It is the robot action strategy model.

7. The reinforcement learning method for controlling the motion of a biped robot based on trajectory self-correction according to claim 1, It is characterized in that In step 4, the real sample When inserting into the real sample experience replay pool to update the real sample experience replay pool, when the number of samples in the real sample experience replay pool is less than the maximum number allowed M When adding directly, otherwise the real sample Replace the sample that was first added to the real sample experience replay pool.

8. A computer program product comprising a computer program or computer executable instructions, It is characterized in that When the computer program or computer executable instructions are executed by a processor, the reinforcement learning method for controlling the motion of a bipedal robot based on trajectory self-correction as described in any one of claims 1 to 7 is implemented.

Images