A robot optimization control method based on reinforcement learning and evolutionary algorithm

By combining reinforcement learning and evolutionary algorithms, the robot motion strategy is optimized using Gaussian noise and fitness functions, local optimal problems in the existing technology are solved, and the robot control effect and optimization speed are improved.

CN114879486BActive Publication Date: 2024-11-08FUDAN UNIVERSITY
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
CN202210186763.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-02-28
Publication Date
2024-11-08
Estimated Expiration
2042-02-28

AI Technical Summary

Technical Problem

The strategy search algorithm based on reinforcement learning in existing robot control is prone to fall into local optimization, resulting in poor control effect.

Method used

Combining reinforcement learning and evolutionary algorithms, through the complementarity of policy networks and genetic algorithms, Gaussian noise and fitness functions are used to optimize action strategies, avoid local optimization, and improve strategy search capabilities.

Benefits of technology

It effectively improves the training and optimization speed of the robot's action strategy network, improves the control effect, and enables the robot to learn gait control strategies faster and avoids local optimization.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114879486B_ABST
    Figure CN114879486B_ABST
Patent Text Reader

Abstract

The present invention relates to a robot optimization control method based on reinforcement learning and evolutionary algorithm, comprising the following steps: S1: obtaining the state space and action space of the robot to be controlled; initializing the environment, obtaining the initial state and corresponding samples of the robot, and storing them in an experience pool; S3: adding Gaussian noise to the initial action, obtaining its state evolution action pair and storing them in an archive; S4: obtaining samples and state evolution action pairs of each time step, and storing them accordingly; S5: updating the parameters of the policy network using policy gradients, and updating the parameters of the policy network using evolutionary action gradients; S6: gradually executing steps S4-S5 until the robot falls over or accumulates preset trajectory time steps, and the robot completes a trajectory; S7: reinitializing the environment, repeating steps S3-S6 until the preset maximum time step is reached. Compared with the prior art, the present invention has the advantages of fast optimization speed and good effect.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of robot control, and in particular to a robot optimization control method based on reinforcement learning and evolutionary algorithms. Background Art

[0002] One of the goals of artificial intelligence (AI) is to simulate the interaction between an intelligent agent and the environment. This process can be modeled as a Markov decision process (MDP). As a mathematical model of sequential decision, MDP describes the process in which an agent performs action a according to the current observed state s in an environment with Markov properties, obtains a reward r, and the environment transfers to a new state s′ according to the state transition probability p(s′|s,a) until termination. The strategy π is responsible for selecting action a according to the current environmental state s to complete the entire sequential decision process.

[0003] We generally use a neural network (NN) θ to parameterize the policy π, which can achieve a mapping from state s to action a. The purpose of the policy search algorithm is to search for an optimal or approximately optimal policy π * To maximize the objective function, or to maximize the cumulative reward of the entire decision-making process. Efficient policy search algorithms can be applied to a variety of practical tasks, such as robot control tasks in continuous action spaces.

[0004] In existing robot control, policy search algorithms are generally designed based on reinforcement learning. However, policy search algorithms usually solve non-convex problems under extremely complex sequential decision-making processes. Policy search based solely on reinforcement learning can easily fall into local optimality, resulting in poor robot control effect. Summary of the invention

[0005] The purpose of the present invention is to provide a robot optimization control method based on reinforcement learning and evolutionary algorithm in order to overcome the defects of the above-mentioned prior art.

[0006] The purpose of the present invention can be achieved by the following technical solutions:

[0007] A robot optimization control method based on reinforcement learning and evolutionary algorithm comprises the following steps:

[0008] S1: Obtain the state space and action space of the robot to be controlled;

[0009] S2: Initialize the environment, obtain the initial state s0 of the robot, build and initialize the strategy network π θ (s), input the initial state into the policy network to obtain the initial action a0, the robot executes the initial action and obtains the reward r0 according to the reward function, and at the same time the robot transfers to the next state s1, and stores the sample (s0, a0, r0, s1) into the experience pool R;

[0010] S3: Add Gaussian noise to the initial action a0 to transform it into an action set G0 containing N noise actions. The action set G0 will be used as the initial population of the genetic algorithm, and the evolved actions will be obtained through iterative evolution of the population. Will Together with the corresponding state s0, it forms a state evolution action pair Deposited in Archives A;

[0011] S4: Each time step obtains and executes the corresponding action according to the corresponding state, and obtains the sample corresponding to each time step (s t , a t , r t ,s t+1 ) is stored in the experience pool, and the state evolution action pair is obtained for each time step deposited in archives;

[0012] S5: After each time step of the preset step length, multiple samples are sampled from the experience pool, and the policy network parameters are updated using the policy gradient. Then, multiple evolutionary actions are randomly sampled from the archive, and the parameters of the policy network are updated using the evolutionary action gradient.

[0013] S6: Steps S4 to S5 are executed step by step until the robot falls over or the preset trajectory time steps are accumulated and the robot completes a trajectory;

[0014] S7: Reinitialize the environment and repeat steps S3 to S6 until the preset maximum time step is reached to complete the optimization of the policy network.

[0015] Preferably, the step of obtaining the state evolution action pair in steps S3 and S4 specifically includes:

[0016] Step 1: Action a for the current time step t Add Gaussian noise to turn it into an action set G containing N noise actions t , as the initial population of the genetic algorithm;

[0017] Step 2: Get the fitness of each action in the initial population, select multiple actions with the highest fitness to form an elite action population E, and other actions in the initial population to form a common population P;

[0018] Step 3: Randomly select an action from the elite action population Randomly select an action from the normal population right and Perform crossover to produce two offspring actions;

[0019] Step 4: Repeat step 3 multiple times to obtain multiple offspring actions to form a offspring population O, and mutate the offspring population O. The mutated offspring population O and the elite population E form a new generation of population G, completing a population iteration;

[0020] Step 5: Use the new generation population as the initial population, repeat steps 2 to 4 multiple times, and obtain the final generation population G * , G * The action with the highest fitness is a * As the evolution action at the current time step and the corresponding state s t Composition state evolution action pair

[0021] Preferably, the crossover in the third step is a single-point crossover.

[0022] Preferably, the mutation in the fourth step is: mutating each action in the offspring population O according to a preset probability, and the mutation method is: sampling noise from a Gaussian distribution with a mean of 0 and a variance of 1, and adding the noise to the value of a random dimension of the action for mutation disturbance.

[0023] Preferably, the fitness is obtained according to the fitness function:

[0024]

[0025] in, is the action to obtain fitness, For Action The fitness of Execute for the current time step Rewards received, The action a selected by the current policy network starting from time t+1 t 'Continue to move forward until the cumulative reward obtained at the termination time T.

[0026] Preferably, the reward function is:

[0027] r(s,a)=r alive +r velocity

[0028] Among them, r aliveis the survival reward. When the robot does not fall over at the current time step, r alive =1, otherwise, r alive =0, the r velocity Bonus for speed:

[0029]

[0030] Among them, x is the current position of the robot, is the position of the robot in the previous time step, and l is the length of the time step.

[0031] Preferably, the step S4 specifically includes:

[0032] S41: The state s of the tth time step t Send it to the policy network to get the corresponding action a t ;

[0033] S42: The robot performs action a t , and obtain the reward r according to the reward function t , and the robot moves to the next state s t+1 , the sample (s t , a t , r t ,s t+1 ) is stored in the experience pool;

[0034] S43: Initial action a t Add Gaussian noise to turn it into an action set G containing N noise actions t , the action set G t Will be used as the initial population of the genetic algorithm to obtain the evolutionary action Will The corresponding state s t Composition state evolution action pair Deposited in archives.

[0035] Preferably, the formula for the evolution action gradient in step S5 is:

[0036]

[0037] Among them, θ is the policy network parameter, L evo is the evolution action gradient, is the action retrieved from archive A, π θ (s i ) is the output action a of the policy network i =π θ (s i ), Action a is the output of the policy network i and evolutionary actions The mathematical expectation of the mean square error between them.

[0038] Preferably, the state space is the sensor data of the robot, and the action space is the control signal of the robot.

[0039] Preferably, the robot is a four-legged ant robot, including a body and four robotic legs, each robotic leg includes two motors, the sensor data in the state space include the position, direction, speed, angular velocity of the body and the speed and angular velocity of the leg joints, the action space is the torque signal output by the eight motors of the four robotic legs, and the action space has a total of eight dimensions.

[0040] Compared with the prior art, the present invention has the following advantages:

[0041] The robot control optimization method of the present invention takes into account the complementarity between reinforcement learning and evolutionary algorithms, and can combine the two to learn from each other's strengths and weaknesses, thereby improving the algorithm's strategy search ability. On the basis of the strategy search algorithm based on reinforcement learning, an evolutionary algorithm is introduced to optimize the actions output by the strategy network to improve the algorithm's strategy search ability, combining the advantages of both reinforcement learning and evolutionary algorithms. Reinforcement learning has gradient information and is faster in strategy search, while evolutionary algorithms have global optimization capabilities and can avoid local optimality. The method proposed by the present invention cleverly combines the two for better strategy search, can effectively realize the training and optimization of the robot action strategy network, improve the optimization speed and control effect, and enable the robot to learn the gait control strategy faster. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 is a flow chart of the present invention;

[0043] Figure 2 is a schematic diagram of an ant robot of the present invention;

[0044] Figure 3 is a schematic diagram of the mutation;

[0045] Figure 4 It is a learning curve diagram of the present invention. DETAILED DESCRIPTION

[0046] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. Note that the following embodiments are merely illustrative in nature, and the present invention is not intended to limit its applicable objects or uses, and the present invention is not limited to the following embodiments.

[0047] Example

[0048] A robot optimization control method based on reinforcement learning and evolutionary algorithm, such as Figure 1As shown, the following steps are included:

[0049] S1: Get the state space State and action space Action of the robot to be controlled.

[0050] In this embodiment, Figure 2 As shown, the robot is a four-legged ant robot, including a body and four robotic legs. Each robotic leg includes two motors, and each motor serves as a joint to form a robotic leg structure.

[0051] Specifically, in this embodiment, the sensor

[0052] The state space is the robot's sensor data, and the action space is the robot's control signal.

[0053] The sensor data in the state space include the position, direction, speed, angular velocity of the body and the speed and angular velocity of the leg joints, totaling 111 dimensions, that is, each state s in the state space State is a 111-dimensional vector. It represents the value of the current sensor data of the robot's legs

[0054] The action space Action is the torque signal output by the 8 motors of the 4 legs, with a total of 8 dimensions. That is, each action a in the action space Action is an 8-dimensional vector.

[0055] In addition, in this embodiment, the reinforcement learning algorithm adopts Soft Actor Critic (SAC), and the evolutionary algorithm adopts GA algorithm. The parameter settings of SAC and GA are shown in Table 1. The maximum time step is set to 1,000,000, that is, when the maximum time step is reached, the experiment automatically stops.

[0056] Table 1 SAC and GA parameter settings

[0057]

[0058] S2: Initialize the environment, obtain the robot's initial state s0, initialize the policy network parameters θ, input the initial state into the policy network to obtain the initial action a0, the robot executes the initial action and obtains the reward r0 according to the reward function, and at the same time the robot transfers to the next state s1, and stores the sample (s0, a0, r0, s1) into the experience pool R.

[0059] The reward function r(s,a) in the present invention represents the reward value that the robot can obtain by performing action a in the state s of the current time step, which can be written as:

[0060] r(s,a)=r alive +r velocity

[0061] The reward function consists of two parts. The first part is the survival reward of the robot. When the robot does not fall over at the current time step, r alive =1, otherwise, r alive = 0. The second part is the robot's speed bonus

[0062]

[0063] That is, the robot's current position x and the previous time step The difference in position The length of the time step is l, usually l = 0.1s), which is used as the speed reward of the robot. The purpose of the task is to maximize the cumulative reward value during the entire gait walking process, that is, the ant robot must learn to walk and walk as fast as possible.

[0064] S3: Add Gaussian noise to the initial action a0 to transform it into an action set G0 containing N noise actions. The action set G0 will be used as the initial population of the genetic algorithm, and the evolved actions will be obtained through iterative evolution of the population. Will Together with the corresponding state s0, it forms a state evolution action pair Deposited in Archives A;

[0065] The archive A and experience pool are storage spaces that can be read and written by R.

[0066] S4: Each time step obtains and executes the corresponding action according to the corresponding state, and obtains the sample corresponding to each time step (s t , a t , r t ,s t+1 ) is stored in the experience pool, and the state evolution action pair is obtained for each time step Deposited in archives.

[0067] Among them, S4 is specifically:

[0068] S41: The state s of the tth time step t Send it to the policy network to get the corresponding action a t ;

[0069] S42: The robot performs action a t , and obtain the reward r according to the reward function t , and the robot moves to the next state s t+1 , the sample (s t , a t , r t ,s t+1 ) is stored in the experience pool;

[0070] S43: Initial action at Add Gaussian noise to turn it into an action set G containing N noise actions t , the action set G t Will be used as the initial population of the genetic algorithm to obtain the evolutionary action Will The corresponding state s t Composition state evolution action pair Deposited in archives.

[0071] In this embodiment, the method of obtaining the state evolution action pair in S3 and the method of obtaining the state evolution action pair in S4 are the same. In this embodiment, N is 10:

[0072] Step 1: Action a for the current time step t Add Gaussian noise to turn it into an action set G containing 10 noise actions t , as the initial population of the genetic algorithm;

[0073] Step 2: Get the fitness of each action in the initial population, select the two actions with the highest fitness to form the elite action population E, and the other eight actions in the initial population to form the ordinary population P. The fitness function is:

[0074]

[0075] in, is the action to obtain fitness, For Action The fitness of Execute for the current time step Rewards received, The action a selected by the current policy network starting from time t+1 t′ The cumulative reward obtained by continuing to move forward until the termination time T. Fitness is the state s at the current time step t t Execute the action to be evaluated Rewards And from time t+1, follow the action a selected by the current policy network t′ The cumulative reward obtained by continuing to move forward until the end time T The sum of these two items is the fitness of the action to be evaluated. The fitness value represents the fitness of the action to be evaluated. The higher the fitness, the better the action and the higher the expected reward.

[0076] Step 3: Randomly select an action from the elite action population E Randomly select an action from the common population P like Figure 3 As shown, and Perform a single-point crossover to produce two offspring actions.

[0077] Step 4: Perform step 3 four times in total to obtain 8 offspring actions to form the offspring population O, and mutate each action of the offspring population O with a probability of 10%. The mutated offspring population O and the elite population E form a new generation of population G, completing a population iteration. Specifically, noise ∈ is sampled from a Gaussian distribution with a mean of 0 and a variance of 1, and the value of a certain dimension of the action is randomly added with the noise for mutation perturbation. The mutated offspring population O and the elite population E form a new generation of population G.

[0078] Step 5: Use the new generation population as the initial population, repeat steps 2 to 4 10 times, and obtain the final generation population G * , G * The action with the highest fitness is a * As the evolution action at the current time step and the corresponding state s t Composition state evolution action pair

[0079] S5: After each time step, 100 samples are sampled from the experience pool, and the parameters θ of the policy network are updated using the SAC policy gradient. Then 100 evolutionary actions are randomly sampled from the archives, and the parameters of the policy network are updated using the evolutionary action gradient:

[0080]

[0081] Among them, θ is the policy network parameter, L evo is the evolution action gradient, is the action retrieved from archive A, π θ (s i ) is the output action a of the policy network i =π θ (s i ), Action a is the output of the policy network i and evolutionary actions The mathematical expectation of the mean square error between them.

[0082] S6: Steps S4 to S5 are executed step by step until the robot falls over or stops automatically after 1000 time steps, and the robot completes a trajectory. Here, the robot falls over in various ways, including when the value of the fall sensor set on the robot is 1Ture, the robot is judged to have fallen over.

[0083] S7: Reinitialize the environment and repeat steps S3 to S6 until the preset maximum time step of 1,000,000 is reached. The optimization of the policy network is completed, and the robot's motion control can be performed according to the optimized policy network.

[0084] In this embodiment, in order to reflect the performance of the policy network, the performance of the current policy network, that is, the cumulative reward it receives in the gait control task, is tested every 5,000 time steps. Specifically, the ant robot is allowed to go through 10 trajectories, and the final cumulative reward is averaged as the performance of the policy network at the current time, and the data is recorded and saved. The entire training process is repeated 10 times, and the performance of the policy network saved during the experiment is plotted as a learning curve during the learning process to visualize the policy search capability of the method of the present invention, such as Figure 4 As shown, Figure 4 The horizontal axis represents the number of time steps, and the vertical axis represents the cumulative reward of the policy network in the gait control task at the corresponding time step. The larger the reward, the faster the ant robot can walk, which means the strategy search of the method is stronger and the better ant robot gait control strategy can be searched faster. Figure 4 It can be seen that the strategy search method proposed in the present invention has a more powerful strategy search capability. This is due to the introduction of the genetic algorithm, which can optimize the actions output by the strategy network to obtain evolutionary actions. The evolutionary actions can be used to guide the parameter update of the strategy network, thereby enhancing the strategy search capability of the method and enabling the ant robot to learn the gait control strategy faster.

[0085] The above embodiments are merely examples and do not limit the scope of the present invention. These embodiments can also be implemented in various other ways, and various omissions, substitutions, and changes can be made without departing from the technical concept of the present invention.

Claims

1. A robot optimization control method based on reinforcement learning and evolutionary algorithm, characterized in that: The following steps are involved: S1: Obtain the state space and action space of the robot to be controlled; S2: Initialize the environment, obtain the initial state s0 of the robot, build and initialize the strategy network π θ (s), input the initial state into the policy network to obtain the initial action a0, the robot executes the initial action and obtains the reward r0 according to the reward function, and at the same time the robot transfers to the next state s1, and stores the sample (s0, a0, r0, s1) into the experience pool R; S3: Add Gaussian noise to the initial action a0 to transform it into an action set G0 containing N noise actions. Use the action set G0 as the initial population of the genetic algorithm and obtain the evolved action through iterative evolution of the population. Will Together with the corresponding state s0, it forms a state evolution action pair Deposited in Archives A; S4: Each time step obtains and executes the corresponding action according to the corresponding state, and obtains the sample corresponding to each time step (s t ,a t ,r t ,s t+1 ) is stored in the experience pool, and the state evolution action pair is obtained for each time step deposited in archives; S5: After each time step of the preset step length, multiple samples are sampled from the experience pool, and the policy network parameters are updated using the policy gradient. Then, multiple evolutionary actions are randomly sampled from the archive, and the parameters of the policy network are updated using the evolutionary action gradient. S6: Steps S4 to S5 are executed step by step until the robot falls over or the preset trajectory time steps are accumulated and the robot completes a trajectory; S7: Reinitialize the environment and repeat steps S3 to S6 until the preset maximum time step is reached to complete the optimization of the policy network; The steps of obtaining the state evolution action pair in steps S3 and S4 specifically include: Step 1: Action a for the current time step t Add Gaussian noise to turn it into an action set G containing N noise actions t , as the initial population of the genetic algorithm; Step 2: Get the fitness of each action in the initial population, select multiple actions with the highest fitness to form an elite action population E, and other actions in the initial population to form a common population P; Step 3: Randomly select an action from the elite action population Randomly select an action from the normal population right and Perform crossover to produce two offspring actions; Step 4: Repeat step 3 multiple times to obtain multiple offspring actions to form a offspring population O, and mutate the offspring population O. The mutated offspring population O and the elite population E form a new generation of population G, completing a population iteration; Step 5: Use the new generation population as the initial population, repeat steps 2 to 4 multiple times, and obtain the final generation population G * , G * The action with the highest fitness is a * As the evolution action at the current time step and the corresponding state s t Composition state evolution action pair 2. The robot optimization control method based on reinforcement learning and evolutionary algorithm according to claim 1, characterized in that: The crossover in the third step is a single-point crossover.

3. The robot optimization control method based on reinforcement learning and evolutionary algorithm according to claim 1, characterized in that: The mutation in the fourth step is: mutating each action in the offspring population O according to a preset probability, and the mutation method is: sampling noise from a Gaussian distribution with a mean of 0 and a variance of 1, and adding the noise to the value of a random dimension of the action for mutation disturbance.

4. The robot optimization control method based on reinforcement learning and evolutionary algorithm according to claim 1, characterized in that: The fitness is obtained according to the fitness function: in, is the action to obtain fitness, For Action The fitness of Execute for the current time step Rewards received, The action a selected by the current policy network starting from time t+1 t′ The cumulative reward obtained by continuing to move forward until the end time T.

5. The robot optimization control method based on reinforcement learning and evolutionary algorithm according to claim 1, characterized in that: The reward function is: r(s,a)=r alive +r velocity Among them, r alive is the survival reward. When the robot does not fall over at the current time step, r alive =1, otherwise, r alive =0, the r velocity Bonus for speed: Among them, x is the current position of the robot, is the position of the robot in the previous time step, and l is the length of the time step.

6. The robot optimization control method based on reinforcement learning and evolutionary algorithm according to claim 1, characterized in that: The step S4 specifically includes: S41: The state s of the tth time step t Send it to the policy network to get the corresponding action a t ; S42: The robot performs action a t , and obtain the reward r according to the reward function t , and the robot moves to the next state s t+1 , the sample (s t ,a t ,r t ,s t+1 ) is stored in the experience pool; S43: Action a t Add Gaussian noise to turn it into an action set G containing N noise actions t , the action set G t Will be used as the initial population of the genetic algorithm to obtain the evolutionary action Will The corresponding state s t Composition state evolution action pair Deposited in archives.

7. The robot optimization control method based on reinforcement learning and evolutionary algorithm according to claim 1, characterized in that: The formula for the evolution action gradient in step S5 is: Among them, θ is the policy network parameter, L evo is the evolution action gradient, is the action retrieved from archive A, π θ (s i ) is the output action a of the policy network i =π θ (s i ), Action a is the output of the policy network i and evolutionary actions The mathematical expectation of the mean square error between them.

8. The robot optimization control method based on reinforcement learning and evolutionary algorithm according to claim 1, characterized in that: The state space is the sensor data of the robot, and the action space is the control signal of the robot.

9. The robot optimization control method based on reinforcement learning and evolutionary algorithm according to claim 8, characterized in that: The robot is a four-legged ant robot, including a body and four machine legs, each machine leg includes two motors, the sensor data in the state space includes the position, direction, speed, angular velocity of the body and the speed and angular velocity of the leg joints, the action space is the torque signal output by the eight motors of the four machine legs, and the action space has a total of eight dimensions.

Citation Information

Patent Citations

  • Vertical layered space-time signal detection method based on M-elite evolution algorithm

    CN101431358A

  • Inverse kinematics solving method and device for service robot in intelligent space

    CN108255058A

  • Unmanned ship autonomous navigation method based on deep reinforcement learning and genetic algorithm

    CN110362089A

  • Training method and device for walking controller of quadruped robot, terminal and storage medium

    CN111638646A

  • Robot path planning method and planning system

    CN112987742A