A deep reinforcement learning control method for mobile robots

Abstract

The present invention relates to a deep reinforcement learning control method for a mobile robot, and belongs to the technical field of robot intelligent control. In the robot learning behavior strategy stage, the present invention proposes an off-policy likelihood ratio policy gradient model learning algorithm with historical behavior reuse under the maximum entropy reinforcement learning framework, which is used for end-to-end behavior control of the mobile robot. The method improves the utilization rate of data in the robot behavior learning process, thereby improving the learning speed and sampling efficiency, and saving the interaction cost between the robot and the environment. At the same time, in the evaluation stage of the robot's behavior strategy model, the present invention proposes a unified evaluation network model to simultaneously approximate the state value function and the action value function, and the evaluation network can directly give an advantage function of a state and an action. Compared with using two networks to approximate the state value function and the action value function respectively, the evaluation network has the advantages of fast learning speed and the ability to reduce parameter cumulative errors.

Description

Technical Field

[0001] The present invention relates to a robot autonomous behavior control method based on deep reinforcement learning, and belongs to the technical field of robot intelligent control. Background Art

[0002] Mobile robots refer to a type of integrated system that integrates multiple functions such as environmental perception and recognition, planning and decision-making, behavior and control. It can receive control signals from the designer, run according to pre-programmed instructions, and act according to rules and strategies formulated by artificial intelligence technology. At present, an important trend in the development of mobile robots is intelligence, that is, mobile robots are required to achieve autonomous control rather than passively accepting instructions. Therefore, mobile robots are required to have more powerful and flexible perception, decision-making, and control capabilities.

[0003] With the widespread application of mobile robots, their operation application scenarios have become diverse and complex. How to make mobile robots operate stably in complex scenarios with dynamic and static obstacles, and how to improve their efficiency and intelligence in production operations have become the research focus in this field.

[0004] Mobile robots are required to have stable and good operating capabilities, and the robustness and reliability of their control methods are key. Therefore, various artificial intelligence methods such as deep reinforcement learning and imitation learning have been continuously proposed and gradually applied to practical engineering fields. Among them, deep reinforcement learning is a general learning framework that combines the perception ability of deep learning with the decision-making ability of reinforcement learning. It can achieve direct control from raw perception input to decision-making action output through end-to-end learning. This end-to-end approach is naturally applicable to solving many technical problems such as automatic control systems, mobile robot control, autonomous driving, and game theory.

[0005] However, the existing model-free deep reinforcement learning methods have a very high cost of generating data by interacting with the environment during the learning process, and robots cannot expand multiple threads to interact with the environment in the real world like distributed reinforcement learning methods. Therefore, how to improve the utilization of interaction data and stably learn various behavioral skills in complex environments has become a key technical problem that needs to be solved in the field of robot reinforcement learning control. Summary of the invention

[0006] The purpose of the present invention is to solve the technical problems of low data utilization and poor learning stability faced in deep reinforcement learning control of mobile robots, and creatively propose a deep reinforcement learning control method for mobile robots.

[0007] The innovation of the present invention is that in the robot learning behavior strategy stage, an off-policy likelihood ratio policy gradient model learning algorithm (AC-HBR) with historical behavior reuse is proposed under the maximum entropy reinforcement learning framework for end-to-end behavior control of mobile robots. This method improves the utilization rate of data in the robot behavior learning process, thereby improving the learning speed and sampling efficiency, and saving the interaction cost between the robot and the environment.

[0008] At the same time, in order to effectively evaluate the robot's control strategy, a unified evaluation network model is proposed to simultaneously approximate the state value function and the action value function during the evaluation of the robot's strategy. The advantage function of the state and action can be directly given by the evaluation network model. Compared with using two networks to approximate the state value function and the action value function respectively, the evaluation network model can give the advantage function of the robot's action with higher accuracy, and has the advantages of fast speed and ability to reduce parameter accumulation errors.

[0009] The present invention is implemented by adopting the following technical solutions.

[0010] A deep reinforcement learning control method for a mobile robot comprises the following steps:

[0011] First, define the infinite time Markov process as Among them, S represents a finite set of continuous state spaces, A represents a finite set of continuous action spaces, In a given state s t ∈S, take action a t ∈A and then transfer to state s t+1 The probability of ∈S.

[0012] Define π to represent the strategy of the agent, and define π(a t |s t ) indicates that the agent is in a given state s t Take action a t probability.

[0013] In deep reinforcement learning, φ is defined as the parameter of the policy model, then π φ represents the Gaussian strategy model of the agent's neural network, and the mean and standard deviation of the strategy output by the neural network; r represents the action a taken by the agent t Then transfer to state s t+1 , define r t Indicates that the agent is in state s t Take action a t The reward signal returned by the environment to the agent, γ represents the discount factor of the infinite time Markov process, γ∈[0,1].

[0014] In the maximum entropy reinforcement learning framework, the soft action value function is: The soft state value function is: Where α is the temperature entropy coefficient, Express expectations.

[0015] Step 1: Construct the robot's state space, action space, and reward function.

[0016] Design the state vector s based on the information that the robot can perceive. Design the action space a based on the actuators that the robot has. Design the reward function r(s) in the robot behavior learning process based on the corresponding goals. t ,a t ), s t 、a t They represent the state and action of the robot at time t. The state space s and action space a of the robot are defined by the developer according to the characteristics of the robot and engineering practice.

[0017] Step 2: The robot learns the behavior strategy model through interaction data.

[0018] The off-policy likelihood ratio policy gradient robot behavior learning method is adopted. According to the sampling state, actions are sampled from the current policy model and the experience replay pool at the same time to optimize the robot's behavior strategy.

[0019] This method can make full use of the robot's historical behavior and improve data utilization in the process of learning the robot's behavior strategy. In this method, the policy gradient loss function for the policy model parameters is As shown in formula 1:

[0020]

[0021] Among them, E b represents the experience replay pool; BC(·) represents the Box-Cox data distribution transformation of the input variables, BC(logπ φ (a t |s t )) represents the Box-Cox data distribution transformation of the logarithmic probability of the actions collected from the experience replay pool together with the state under the current policy model.

[0022] Step 3: Evaluate the robot behavior policy model and predict the advantage function of the sampled actions.

[0023] The unified evaluation network model structure is as follows Figure 2 As shown, it includes a state input layer, an action input layer, an action expansion layer, a hidden layer, and a value function output layer.

[0024] In the predicted state value function V π (s t ), define an action a t A vector a with all zero elements of the same shape e When the input of the action input layer is detected to be zero, the output of the action expansion layer is set to zero, and the state value function is obtained by the unified evaluation network: V π (s t )=Q(s t ,a e ).

[0025] Define the variable θ as the model parameter of the unified evaluation network. When this unified evaluation network model is used to simultaneously approximate the state value function and the action value function, the policy gradient loss function of the model is shown in Equation 3:

[0026]

[0027] in, Represented by having policy parameters The state value function predicted by the target evaluation network, Represents the state value function V π The gradient loss function is represents the gradient loss function of the action value function Q; Q θ (s t ,a t ), They respectively represent the action-value function and state-value function represented by the unified evaluation network represented by the parameter θ.

[0028] In the learning process of the robot behavior strategy model, when calculating the advantage function of the sampled action, the advantage function A(s) of the sampled action is directly predicted by the unified evaluation network model according to the following formula: t ,a t ):

[0029]

[0030] Step 4: Conduct robot training.

[0031] The robot interacts with the environment and saves the interaction data in the experience revisit pool. Every N steps, repeat steps 2 and 3, and then the robot interacts with the environment again to collect data, thus completing the autonomous learning control of the mobile robot.

[0032] The number of steps N of the robot's interaction with the environment is set independently according to the actual situation. For example, the number of steps N can be set to 300.

[0033] Beneficial Effects

[0034] Compared with the prior art, the method of the present invention has the following advantages:

[0035] 1. In the learning process of the robot behavior strategy model, the present invention proposes a policy gradient algorithm with historical behavior reuse, which improves the learning speed and sampling efficiency, and accelerates the learning speed and learning stability of the robot's behavior skills.

[0036] 2. In the evaluation stage of the robot's behavior strategy model, the present invention proposes a unified evaluation network model to simultaneously approximate the state value function and the action value function. The evaluation network can directly give an advantage function of the state and action. Compared with using two networks to approximate the state value function and the action value function respectively, the evaluation network has the advantages of fast learning speed and the ability to reduce parameter cumulative errors. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 This is a schematic diagram of the "BipedalWalkerHardcore" robot training environment;

[0038] Figure 2 It is a schematic diagram of a unified evaluation network model structure proposed by the present invention;

[0039] Figure 3 This is a comparison chart of the reward curves of the robot behavior deep reinforcement learning algorithm proposed in the present invention and SAC (soft actor-critic) in the "BipedalWalkerHardcore" robot training environment.

[0040] Figure 4 It is a comparison chart of the cumulative reward curves of the robot behavior deep reinforcement learning algorithm proposed in this invention and SAC in other series of robot control benchmark environments in Gym and PyBullet.

[0041] Figure 5 The present invention Figure 3 and Figure 4 All hyperparameter settings used in the experiments. DETAILED DESCRIPTION

[0042] The present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0043] Example

[0044] like Figure 1 As shown, the behavior control of the "BipedalWalkerHardcore" robot in the Gym environment is taken as an example for explanation.

[0045] Step 1: Construct the robot's state space, action space, and reward function.

[0046] Specifically, the "BipedalWalkerHardcore" robot learns a series of skills such as running, crossing pits, crossing obstacles, and going down stairs in complex road conditions through interaction with the environment. The robot perceives environmental information through the sensors it carries, and the policy model gives the actions taken by the robot in the current state based on the state quantity constructed by these perception information. The perception information of the "BipedalWalkerHardcore" robot includes the angular velocity of each component, horizontal velocity, vertical velocity, joint position, contact between the leg and the ground, and the measurement values ​​of 10 laser radar rangefinders. These perception information are spliced ​​into a 24-dimensional state vector s. And the 4-dimensional action space a of the robot is constructed by the torque of the two joints under the robot's hips and the torque of the knee joint. Since the action space of this environment is a 4-dimensional continuous action space. The goal of the "BipedalWalkerHardcore" robot is to reach the end point quickly, and to control the torque of the four joints to be the smallest. The closer to the target, the greater the reward function returned by the environment. When the robot falls, the reward returned by the environment is -100. The environment is designed according to this rule and directly returns the corresponding reward function r(s,a) to the robot.

[0047] Step 2: The robot learns the behavior strategy model through interaction data.

[0048] Specifically, the robot updates the parameters of the strategy model according to the following formula during the learning process of the behavior strategy:

[0049]

[0050] Among them, clip(logπ φ (a t |s t )) represents logπ φ (a t |s t ) is sheared, and in the present invention, logπ is set φ (a t |s t ) is less than -30, let it be equal to -30. Also, since clip(logπ φ (a t |s t )) may be negative. When performing the Box-Cox transformation on it, it is necessary to clip the logarithmic probability of all actions sampled from the experience replay pool (logπ φ (a t |s t )) minus the smallest negative value corresponding to these logarithmic probabilities. And for all clips (logπ φ (at |s t ))After the Box-Cox transformation is completed, the transformed data are added with the corresponding smallest negative value of these log-probabilities.

[0051] In the process of interaction with the environment, in order to ensure the range of motion that the robot joints can withstand, the technology of the SAC (softactor-critic) algorithm is borrowed to apply the reversible compression function tanh(·) to limit the motion collected from the Gaussian strategy, and the logarithmic probability density function of the compressed motion is calculated by the following formula:

[0052]

[0053] in, is a random variable, D represents the dimension of the action vector; μ(u|s) is the probability density function corresponding to the Gaussian strategy, u i is the i-th element in u.

[0054] When using actions collected from the experience replay pool to calculate the policy gradient of the current policy, the collected tanh(·) compressed actions are converted to the corresponding actions a sampled from the Gaussian policy by the following formula: G :

[0055]

[0056] Among them, a tanh represents an action that has been compressed by tanh(·).

[0057] The automatic entropy coefficient adjustment method in the SAC algorithm is used to adjust the temperature entropy coefficient α:

[0058]

[0059] in, represents a minimum expected entropy.

[0060] Step 3: Evaluate the robot behavior policy model and predict the advantage function of the sampled actions.

[0061] When the unified evaluation network model proposed in the present invention is used to simultaneously approximate the state value function and the action value function, the gradient loss function of the model is as follows:

[0062]

[0063]

[0064] in, Represented by having policy parameters The target evaluation network predicts the state value function. In addition, to address the overestimation problem, the "double q-learning" technique is used. Therefore, two unified evaluation networks are used to simultaneously learn the state value function and the action value function, and in the calculation When , the relatively small action-value function of the two networks is adopted.

[0065] In the learning process of the robot behavior strategy model, when calculating the advantage function of the sampled action, the advantage function of the sampled action is directly predicted by the unified evaluation network model according to the following formula:

[0066]

[0067] Step 4: Robot training.

[0068] Specifically, the robot interacts with the environment and stores the interaction data in the experience revisit pool. Steps 2 and 3 are repeated every certain number of steps (set to 300 steps in the experiment of the present invention), and then the robot interacts with the environment again to collect data.

[0069] During the training process, the relevant hyperparameters are set first.

[0070] Specifically, all common hyperparameters used by the deep reinforcement learning algorithm AC-HBR for end-to-end behavior control of robots proposed in the present invention and the SAC deep reinforcement learning algorithm used for comparative experiments are listed in Figure 5 In. Figure 5 As shown in the figure, among the common hyperparameters, reward scaling refers to directly multiplying the environment reward by a constant, adjusting the reward value without destroying the reward function, and thus indirectly adjusting the Q value to an appropriate size. For convenience and to improve comparability, the reward scaling values ​​of the AC-HBR and SAC algorithms in all task tests are set to 0.2. The number of neural network units in the unified evaluation network extension layer is set to 84, and the hyperparameter β is given as 0.003. In the test of robot behavior learning, β has a wide adjustment range and has almost no effect on the performance of the algorithm within the adjustment range. In the performance test of the algorithm, multiple random seeds are used to test the algorithm.

[0071] Figure 3 The performance comparison curve of the algorithm proposed in the present invention and SAC in the robot behavior control in the "BipedalWalkerHardcore" environment is shown. Figure 4 The performance comparison curves of the algorithm proposed in the present invention and SAC in robot behavior control in Gym and PyBullet environments are shown.

Claims

1. A deep reinforcement learning control method for a mobile robot, characterized in that: The following steps are involved: Define the infinite time Markov process as Among them, S represents a finite set of continuous state spaces, A represents a finite set of continuous action spaces, Indicates that in a given state s t ∈S, take action a t ∈A and then transfer to state s t+1 ∈S probability; Define π to represent the strategy of the agent, and define π(a t |s t ) indicates that the agent is in a given state s t Take action a t probability; In deep reinforcement learning, φ is defined as the parameter of the policy model, then π φ represents the Gaussian strategy model of the agent's neural network, and the mean and standard deviation of the strategy are output by the neural network; r represents the action a taken by the agent t Then transfer to state s t+1 , define r t Indicates that the agent is in state s t Take action a t The reward signal returned by the environment to the agent, γ represents the discount factor of the infinite-time Markov process, γ∈[0,1]; In the maximum entropy reinforcement learning framework, the soft action value function is: The soft state value function is: Where α is the temperature entropy coefficient, express expectations; Step 1: Construct the robot's state space, action space and reward function; Design the state vector s according to the information that the robot can perceive; design the action space a according to the actuators of the robot; design the reward function r(s) in the robot behavior learning process t ,a t ), s t 、a t They represent the state and action of the robot at time t respectively; the state space s and action space a of the robot are defined by the developer according to the characteristics of the robot and engineering practice; Step 2: The robot learns the behavior strategy model through interaction data; Adopting the off-policy likelihood ratio policy gradient robot behavior learning method, according to the sampling state, actions are sampled from the current policy model and the experience replay pool at the same time to optimize the robot's behavior strategy; Among them, the policy gradient loss function for the policy model parameters is As shown in formula 1: Among them, E b represents the experience replay pool; BC(·) represents the Box-Cox data distribution transformation of the input variables, BC(logπ φ (a t |s t )) represents the Box-Cox data distribution transformation of the logarithmic probability of the actions collected from the experience replay pool together with the state under the current policy model; Step 3: Evaluate the robot behavior policy model and predict the advantage function of the sampled actions; Unified evaluation network model structure, including state input layer, action input layer, action expansion layer, hidden layer and value function output layer; In the predicted state value function V π (s t ), define an action a t A vector a with all zero elements of the same shape e ; When the input of the action input layer is detected to be zero, the output of the action expansion layer is set to zero, and the state value function is obtained by the unified evaluation network: V π (s t )=Q(s t ,a e ); Define the variable θ as the model parameter of the unified evaluation network; when this unified evaluation network model is used to simultaneously approximate the state value function and the action value function, the policy gradient loss function of the model is shown in Formula 3: in, Represents a policy parameter The state value function predicted by the target evaluation network, Represents the state value function V π The gradient loss function is represents the gradient loss function of the action value function Q; Q θ (s t ,a t ), They represent the action-value function and state-value function represented by the unified evaluation network represented by the parameter θ; In the learning process of the robot behavior strategy model, when calculating the advantage function of the sampled action, the advantage function A(s) of the sampled action is directly predicted by the unified evaluation network model according to the following formula: t ,a t ): Step 4: Conduct robot training; The robot interacts with the environment and saves the interaction data in the experience revisit pool; every N steps, repeat steps 2 and 3, and then the robot interacts with the environment again to collect data, thereby completing the autonomous learning control of the mobile robot; Among them, the interval number N of the interaction between the robot and the environment is set autonomously according to the actual situation.

2. A mobile robot deep reinforcement learning control method as claimed in claim 1, characterized in that: In step 2, during the interaction between the robot and the environment, a reversible compression function tanh(·) is applied to limit the actions collected from the Gaussian strategy, and the logarithmic probability density function of the compressed action is calculated by the following formula: in, is a random variable, D represents the dimension of the action vector; μ(u|s) is the probability density function corresponding to the Gaussian strategy, u i is the i-th element in u; When using actions collected from the experience replay pool to calculate the policy gradient of the current policy, the collected tanh(·) compressed actions are converted to the corresponding actions a sampled from the Gaussian policy by the following formula: G : Among them, a tanh represents an action that has been compressed by tanh(·); The temperature entropy coefficient α is adjusted by using the automatic entropy coefficient adjustment method: in, represents a minimum expected entropy.

Images