Deep reinforcement learning method for fast control strategy convergence

Abstract

The invention belongs to the technical field of deep learning, and particularly provides a novel deep reinforcement learning method which is named as DCCP (Dynamic Clipping Proximal Policy) and is used for solving the problem that a current deep reinforcement learning algorithm cannot effectively control strategy convergence for some strategy control tasks. According to the method, the strategy convergence effect is achieved mainly by using the self-adaptive dynamic cutting rate and the optimized loss function expression. The method can prevent the deep reinforcement learning neural network from converging to a local minimum value in advance, encourages the strategy to perform positive exploration, and can well converge in a scene with sparse data distribution.

Description

Technical Field

[0001] The present invention belongs to the technical field of deep learning, and specifically relates to a deep reinforcement learning method. Background Art

[0002] With the continuous breakthroughs in hardware technology for efficient computing in recent years, the field of artificial intelligence (AI) has developed rapidly. At present, artificial intelligence has a large number of important applications in many fields of social production and life, profoundly affecting the political, economic and other system structures of human beings, and is an important field leading the new generation of scientific and technological revolution. As artificial intelligence becomes increasingly important, it is necessary to accelerate the research speed of AI algorithms and design more powerful AI algorithms to meet the ever-evolving application needs. In recent years, the most popular AI technology is the machine learning method using neurons and artificial neural networks. Reinforcement learning in machine learning is widely regarded by the academic community as the artificial intelligence algorithm that is closest to human behavior patterns and has attracted people's attention. Deep reinforcement learning (DRL) is a learning method based on reinforcement learning using deep neural networks. After rapid development in recent years, many related algorithms have been proposed and have achieved considerable results. The study of DRL algorithms is the core research content of DRL and the key to supporting the development of reinforcement learning. The study of DRL algorithms is of great significance. With the in-depth study of various learning methods, the theory of deep reinforcement learning has been fully formed in modern times, mainly divided into algorithms based on value functions and algorithms based on policy functions. In 2013, the DeepMind research team of Google in the United States proposed the algorithm DQN (Deep Q-Network) based on the value function. However, the algorithm DQN based on the value function has the disadvantage of weak ability to handle continuous actions, so people have successively proposed many excellent algorithms based on policy functions. Among them, PPO (Proximal Policy Optimization) has stable performance and powerful capabilities. It is the most widely used deep reinforcement learning algorithm. However, for some policy control tasks, PPO cannot effectively control policy convergence, so the PPO algorithm needs to be further improved. For example, in 2020, Chloe et al. revealed three failure modes of PPO, pointed out that there are problems with the PPO algorithm and gave relevant improvements (HSUC-, MENDLER-DUNNERC, HARDTM. Revisiting design choices in proximal policy optimization [S]. arXiv: 2009.10897 [cs. LG]. 2020.2220: 29pp.). In the same year, Wangshu et al. published PPOS and PPORB algorithms that improved PPO (ZHU Wang-shu, ROSENDOA. Proximal policy optimization smoothed algorithm [S]. arXiv: 2012.02439 [cs. LG]. 2020.). There are endless discussions in the industry about the improvement of the PPO algorithm, with the goal of obtaining a deep reinforcement learning algorithm with better performance and the same high versatility than PPO, which is the focus of current deep reinforcement learning research.

[0003] To address the problem that the PPO algorithm cannot effectively control strategy convergence, the present invention proposes a new deep reinforcement learning method named DCCP (Dynamic Clipping Proximal Policy). Summary of the invention

[0004] Symbol table To facilitate the following description, some symbols used in this article and their meanings are listed here:

[0005] The purpose of this invention is to propose a new deep reinforcement learning method. In order to solve the problem that the PPO algorithm cannot effectively control the strategy convergence problem, the strategy convergence effect is achieved by optimizing the clipping rate in PPO and changing the loss function expression.

[0006] The objective of the present invention is achieved through the following technical solutions:

[0007] (I) Methodology and process, including:

[0008] (1) Initialization parameter settings

[0009] (1.1) Define the environment object Env, which can generate the status information of a certain scene, and can also return the reward value of the action in the previous state, as well as the current state and additional information after receiving the action.

[0010] (1.2) Then, the action network and value evaluation network are designed in the intelligent agent, as well as the Trainer that can train the network, which can output the current action and state value function values ​​at the input state, as well as the parameters after training the network.

[0011] (2) The main body of the method is divided into two parts: reasoning and training:

[0012] (2.1) Reasoning part:

[0013] (2.1.1) Let the return state of the environment Env to the agent Agent at a certain time t be s t , the agent will change the state s t Input into the action network and value evaluation network, and obtain the strategy π according to the network parameters θt and the value function V θt , after data processing, we get the action and state value function pair {a t ,v t}, and returns the action to the environment Env.

[0014] (2.1.2) After receiving the action, the environment returns the reward value, state, and information data set {r t+1 ,s t+1 ,info}. Info is the additional information of the current environment, which generally reflects the loop termination of the environment.

[0015] (2.1.3) After receiving the return data set from the environment, the agent stores the sampled data tuples generated by this reasoning process into the experience pool for learning. When the reasoning part reaches the set parameters, the training part begins.

[0016] (2.2) Training part:

[0017] (2.2.1) Set up the training network Trainer, which uses the same network as the agent Agent. Specifically, after each training, the parameters θ of Trainer are updated to the action network and value evaluation network in the agent. Suppose the total size of the data stored in the experience pool is Data_size, and the training network Trainer takes a data size of Batch_size from the experience pool for training each time.

[0018] (2.2.2) The training method adopts the mini-batch gradient descent method. According to the settings of the hyperparameters epoch and mini_batch_size, a mini-batch of data of size mini_batch_size is sampled from the data of size Batch_size each time for training. A total of epoch*Batch_size / mini_batch_size training times are performed. After each training with Batch_size samples, the number of iterations of this training is increased by 1.

[0019] (2.2.3) Perform gradient updates according to the Adam optimizer rules. For details, see the description in part (II). After this small batch training is completed, continue to iterate according to the hyperparameter settings and the current situation until the training is completed.

[0020] (2.2.4) After training is completed, the new network parameters θnew are obtained, and θnew is set as the network parameters of the agent Agent and the training network Trainer, and the reasoning part is re-entered.

[0021] (II) To explain the gradient update in (I), first define the loss function in DCCP, including:

[0022] Assume that the environmental sampling data tuple corresponding to a certain time t is {t,s t ,a t ,v t ,R t+1 ,π θold ,info}.

[0023] where s t is the sample state at time t, a t is the action at time t, v t is the value function, R t+1is the return value at time t+1, π θold is the strategy, and info is the additional information of the current environment.

[0024] Calculation of the policy loss function: In order to ensure a better strategy, we use the logarithmic strategy method to calculate the action network of the training network Trainer at time t given the sample state s t The state value v ω (s t ) for the current return value R t The logarithmic ratio function is as follows: Secondly, define a dynamic clipping value with hyperparameters and adaptability to ensure that the overall learning algorithm gradually tends to an excellent strategy over time: Where t is the latest time step in the inference process, T is the total time step, and one time step is defined as the process of sampling the environmental parameter value once; ε0 is a hyperparameter, and its value is generally defined as 0.2. At the same time, the limiting function is defined as: In the above formula and They represent the logarithmic strategy ratio and the dynamic clipping value respectively. μ is the introduced hyperparameter that determines the scale of functional clipping. Then find the advantage function A t Such as: δ t =R t+1 +γ*v t+1 -v t A t =δ t +γ*λ*self The reward attenuation factor γ takes a value between 0 and 1 and reflects the attitude towards future rewards; λ is a hyperparameter. Combining the above formula, we can get the policy loss function for this batch of training as follows: Where Et represents the expected value. If there are T groups of sampled data, the average value is calculated by weighted summing of all possible values ​​in t∈[1,T].

[0025] Calculation of state value loss function: First, calculate the evaluation network of the training network Trainer at a given sample state st The state value v ψ (s t ) for the current return value R t The mean square error formula is: S t (ψ)1=(v ψ (s t )-R t ) 2 Then use the old state value function v saved in the sample t After trimming, we get the second mean square error formula: in for: The state value loss function is defined as the maximum value of the two mean square error formulas, and the expression is as follows:

[0026] Calculation of the loss function of the entropy function: Suppose the action space corresponding to the current environment Env is A Env , the actions you can take are: Strategy π θ represents a probability vector space, that is, The strategy π of the action network in the training network θ (a t |s t ) means in state s t Take action a t The probability of t ∈A Env ; then strategy π θ The entropy function Ent(π θ ) is the formula: The loss function of the entropy function can be obtained as follows:

[0027] The total loss function and gradient update: After obtaining the above three loss functions, the total loss function of the training network in this batch can be obtained as follows: make Represents the overall parameters of the training network Trainer. The total loss function of DCCP can be used to obtain the loss function of the network parameters. Calculate the derivative to obtain the parameter The loss function gradient grad is: Then use the Adam optimizer to update and optimize the neural network parameters. The optimizer update rule is as follows: in Momentum estimation is used to advance the gradient, prevent premature convergence to the local minimum, and encourage the strategy to actively explore; It is the exponential moving average of the squared gradient. In scenarios where data distribution is sparse, it can better utilize the information of sparse gradients. Their expressions are: Among them, β1 and β2 are exponential decay rates, which control the influence of momentum and control the square of gradient respectively, and are generally taken as constant values ​​of 0.9 and 0.999 respectively. Beneficial Effects of the Invention

[0028] 1. Prevent deep neural networks from converging to local minima prematurely and encourage strategies to actively explore.

[0029] 2. In scenarios where data distribution is sparse, the sparse gradient information can be better utilized to adjust the changes in the learning rate, and it can converge well under unstable objective function conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 It is the DCCP reasoning process in the example; AC_NN is the action network, VF_NN is the evaluation network, and ST_NN is the policy network. The policy network and the value network share a part of the network, which is called the latent network Latent. Its input is the environment state, and its output is latent_state, which represents the intermediate state latent vector.

[0031] Figure 2 This is the DCCP training process in the example

[0032] Figure 3 It is the overall flow chart of DCCP in the example; DETAILED DESCRIPTION

[0033] In order to further understand the content of the present invention, the present invention is described in detail in conjunction with the accompanying drawings. It should be understood that the embodiments are only for explaining the present invention and are not intended to limit it.

[0034] 1. Initialize the action network parameters θ in the agent agent , evaluate the network parameters ψ agentInitialize the training network action network parameters θ train , evaluate the network parameters ψ train .

[0035] 2. Initialize the environment object Env and collect the initial environment state in the environment object Env.

[0036] 3. Initialize an experience pool of size Batch_size.

[0037] 4. Define hyperparameters such as total interaction step Total_timestep, training extraction amount Batch_size, number of repeated training epochs, learning rate α, total number of iterations R, collection step size T, etc.

[0038] Reasoning part:

[0039] 5. Iteratively update the agent network parameters to the training network parameters, and the total number of iterations is epoch.

[0040] 6. t Input the value evaluation and action network strategy of the agent to obtain the state value function V ψt and the current policy π θt , execute action a according to the strategy t .

[0041] 7. Change action a t Return to the environment and get the environment Env feedback data group {r t+1 ,s t+1 ,info}

[0042] 8. The sample data set {t,s t ,a t ,v t ,r t+1 ,π θold ,info} is stored in the experience pool for learning.

[0043] 9. Update the current state, the number of updates is the total interaction step length Total_timestep. The inference iteration round is completed, Total_timestep data groups are collected, and the training process begins.

[0044] Training part:

[0045] 10. Randomly extract Batch_size time-continuous data groups {t,s t ,a t ,v t ,r t+1 ,π θold,info},t∈[k,k+Batch_size-1],k∈[0,Data_size-Batch_size]

[0046] 11. Perform epoch training iterations, where each iteration performs multiple mini-batch training, the number of times is Batch_size / mini_batch_sizee.

[0047] 12. Calculate the advantage function, strategy ratio, clipping coefficient, limit function, gradient at each time t in each small batch training, and get it according to the back propagation of the neural network Then use the Adam optimizer to update the network parameters, and end a small batch training iteration.

[0048] 13. After reaching the number of small batch training times, start the next batch training iteration.

[0049] 14. After epoch training iterations, the training iteration is completed and the next total iteration begins: inference + training

[0050] 15. After reaching the set total number of iterations, the overall network training ends, the data group in the database is output, and the score-time graph is obtained.

Claims

1. A new deep reinforcement learning method based on policy function, named DCCP (Dynamic Clipping Proximal Policy), is characterized by using adaptive dynamic clipping rate and dynamic clipping function with hyperparameters, applying logarithmic policy method, and then optimizing the expression of loss function to achieve rapid policy convergence. The DCCP method includes three loss functions, namely policy loss function, state value loss function and entropy loss function.

2. The strategy loss function L optimized according to claim 1 policy : A logarithmic strategy method is used to obtain a better strategy ratio function. A dynamic clipping value with hyperparameters and adaptability is defined to ensure that the overall learning algorithm gradually approaches an excellent strategy over time. The DCCP strategy loss function is obtained based on the optimized strategy ratio function and dynamic clipping value.

3. The state value loss function L optimized according to claim 1 value : The state value loss function is defined as the maximum of two mean square errors: the first is to calculate the mean square error of the state value of the evaluation network of the training network under a given sample state to the current reward value; the second mean square error formula is obtained by clipping the old state value function saved in the sample.

4. The loss function of the entropy function optimized according to claim 1:

5. The total loss function optimized according to claim 1 is the sum of the strategy loss function, the state value loss function and the state value loss function:

Images