A reinforcement learning method of adaptive dynamic discount factor

Abstract

The application discloses a reinforcement learning method of an adaptive dynamic discount factor, and comprises the following steps: constructing an action network, a value network and an adaptive discount factor network; randomly sampling trajectory data from an experience replay pool as a training set and a verification set; fixing the parameters of the adaptive discount factor network, and determining an adaptive discount factor under a current reward; updating the parameters of the action network and the value network based on the training set; fixing the parameters of the updated action network and the value network, and updating the parameters of the adaptive discount factor network based on the verification set; and alternately and iteratively optimizing the parameters of the action network, the value network and the adaptive discount factor network. The application fully considers the far-sighted differences of different actions, allocates a separate discount factor for each reward, and improves the performance and robustness of the reinforcement learning algorithm.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of reinforcement learning, and more particularly to a reinforcement learning method with adaptive dynamic discount factor. BACKGROUND

[0002] With the success of deep reinforcement learning in automatic driving, agent decision-making and agent control, various deep reinforcement learning algorithms have been developed and applied. At present, there is little research on dynamic discount factor in reinforcement learning, and most existing reinforcement learning methods use a fixed discount factor by default, and calculate the cumulative reward using the exponential backtracking function specified in the traditional Markov reward process. Under such a framework, the discount factor as a hyperparameter is artificially adjusted and fixed, and usually needs to be manually adjusted repeatedly to achieve relatively good results, which cannot adapt to increasingly complex reinforcement learning tasks and environments.

[0003] For the optimization design of the discount factor, there are exponential backtracking form discount factor, hyperbolic function form discount factor, meta-learning method, etc. However, these algorithms change the influence of the discount factor in the time dimension or the influence of the discount factor in the time difference, and the discount factor itself is still fixed in the process of calculating the cumulative reward, which cannot distinguish the difference in the time required for different actions to generate benefits in complex reinforcement learning tasks, and cannot distinguish between long-term and short-term benefits. Since the same discount factor is assigned to all rewards, it is not reasonable to distinguish between "long-term" and "short-term" rewards.

[0004] Therefore, how to provide a reinforcement learning method with adaptive dynamic discount factor that can consider the long-term and short-term nature of rewards and assign a separate dynamic discount factor to each reward is a problem that needs to be solved by those skilled in the art. SUMMARY

[0005] Therefore, the present application provides a reinforcement learning method with adaptive dynamic discount factor, which introduces a measure of the "far-sightedness" of actions, fully considers the influence of long-term and short-term benefits of performing certain actions in real tasks, and assigns a separate discount factor to each reward, thereby improving the performance of existing reinforcement learning algorithms and improving robustness.

[0006] To achieve the above purpose, the present application adopts the following technical solutions:

[0007] A reinforcement learning method with adaptive dynamic discount factor, comprising:

[0008] S1, constructing a reinforcement learning model and an experience replay pool; the reinforcement learning model comprises an action network, a value network and an adaptive discount factor network;

[0009] S2, initialize parameters of the action network, the value network and the adaptive discount factor network;

[0010] S3, randomly sample trajectory data from the experience replay pool as a training set and a validation set;

[0011] S4, obtain an adaptive discount factor under a current reward according to a relationship between a discount factor corresponding to a certain range before each reward in the sampled trajectory data and a historical decision trajectory of an agent;

[0012] Fix the parameters of the adaptive discount factor network, and update the parameters of the action network and the value network based on the training set;

[0013] S5, fix the updated parameters of the action network and the value network, and update the parameters of the adaptive discount factor network based on the adaptive discount factor;

[0014] S6, determine whether a maximum number of iterations is reached, if yes, stop the iterative optimization process; if no, return to S3 to continue alternating optimization of the parameters of the action network, the value network and the adaptive discount factor network.

[0015] Further, the experience replay pool in S1 stores time sequence trajectory tuples wherein, represents a state sequence from an initial state to a terminal state, represents an action sequence from the initial state to the terminal state, represents a reward sequence from the initial state to the terminal state, represents a selected action probability sequence from the initial state to the terminal state.

[0016] Further, in S4, the process of updating the parameters of the action network is as follows:

[0017] S411, take the sampled trajectory data as a training set τ t , extract a state sequence from it, and input each state s t in time dimension to the action network π θ and the value network , to obtain an agent's selected action probability π θ (a t |s t ) at each time, an importance sampling coefficient α, and an agent's value estimation

[0018] S412, fix the parameters of the adaptive discount factor network, and obtain an adaptive discount factor γ from the sampled trajectory data τt The state sequence and the action sequence are extracted from the sampled trajectory data, and the state sequence and the action sequence are sequentially input into the adaptive discount factor network in the time dimension, to obtain the adaptive discount factor γ corresponding to the reward at each time t ;

[0019] S413, collect the adaptive discount factor γ of the reward at each time t , and extract the reward sequence from the sampled trajectory data, calculate the value target value of the agent in the state s t performing the action a t at each time, and take the difference between the value target value and the corresponding value estimate as the advantage estimate of the action a t in the state s t .

[0020] S414, calculate the loss function of the action network according to the selected action probability of the agent at each time and the advantage estimate, and update the parameters of the action network through stochastic gradient descent.

[0021] Further, in S414, the expression of the loss function of the action network is:

[0022]

[0023] The formula for updating the parameters of the action network iteratively is:

[0024]

[0025] Wherein, ρ represents the step size, represents the gradient of the action network parameters.

[0026] Further, the process of updating the parameters of the value network in S4 is:

[0027] S421, take the sampled trajectory data as the training set τ t , and extract the state sequence therefrom, and sequentially input each state s t into the value network in the time dimension, to obtain the value estimate of the agent for the current state at each time

[0028] S422, fix the parameters of the adaptive discount factor network, extract the state sequence and the action sequence from the sampled trajectory data τ t , and sequentially input the state sequence and the action sequence into the adaptive discount factor network in the time dimension, to obtain the adaptive discount factor γ corresponding to the reward at each time t .

[0029] S423, collect the adaptive discount factor γ of each time reward that has been obtained t , and extract the reward sequence from the sampled trajectory data τ t , calculate the value target value of the agent in state s t at each time action a t .

[0030] S424, calculate the loss function of the value network according to the value target value of the agent at each time and the value estimate, and update the parameters of the value network through stochastic gradient descent.

[0031] Further, in S424, the expression of the loss function of the value network is:

[0032]

[0033] The formula for updating the value network is:

[0034]

[0035] Where ρ is the step size, , which represents the gradient of the value network parameters.

[0036] Further, in S5, the process of updating the parameters of the adaptive discount factor network is:

[0037] S511, fix the parameters of the action network and the value network, take the sampled trajectory data as the validation set τ v , and extract the state sequence from it, and input the state s t in time dimension to the action network π θ and the value network , get the action probability π θ (a t |s t ) of the agent at each time, the importance sampling coefficient α, and the value estimate of the agent to the current state

[0038] S512, extract the state sequence and action sequence from the sampled trajectory data, and input the state sequence and action sequence in time dimension to the adaptive discount factor network, get the adaptive discount factor γ t of each time reward.

[0039] S513, collect the adaptive discount factor γ of each time reward t , and extract the reward sequence from the sampled trajectory data, calculate the value target value of the agent in state st the value target value of the agent performing action a in state s t the value target value of the agent performing action a in state s the difference between the value target value and the corresponding value estimate as a judgment of the advantage estimate of the agent performing action a in state s t the value target value of the agent performing action a in state s t the value target value of the agent performing action a in state s

[0040] S514, calculate the loss function of the adaptive discount factor network according to the selected action probability and the advantage estimate of the agent at each time, and update the parameters of the adaptive discount factor network through stochastic gradient descent.

[0041] Further, in S514, the expression of the loss function of the adaptive discount factor network is:

[0042]

[0043] The formula for updating the parameters of the adaptive discount factor network iteratively is:

[0044]

[0045] Where ρ represents the step size, represents the gradient of the adaptive discount factor network parameters, represents the gradient of the action network and the value network.

[0046] The calculation formula of the value target value of the agent performing action a in state s t the value target value of the agent performing action a in state s t the value target value of the agent performing action a in state s

[0047]

[0048] Where, represents the cumulative sum of the discounted rewards after time step t; i represents the i-th time step of the agent; t represents the current time step; represents the discount factor at time step i; r i represents the reward obtained by the agent at time step i.

[0049] Further, the adaptive discount factor network includes a deep residual network, a self-attention network, and a multi-layer perceptron; the deep residual network and the self-attention network perform feature extraction and encoding operations on the input state sequence and action sequence, model functions according to the relationship between the state sequence, the action sequence, and the discount factor, and update the functions iteratively through double-layer optimization meta-learning to constantly construct new cumulative reward backtracking forms; the multi-layer perceptron extracts the adaptive discount factor under the current reward from the constructed new cumulative reward backtracking form.​

[0050] Compared with the prior art, the self-adaptive dynamic discount factor reinforcement learning method provided by the application can convert the cumulative reward calculation of the fixed discount factor into the cumulative reward calculation of the dynamic discount factor through the design of the inner and outer optimization cycles under the architecture of the inner and outer double cycles, consider that the reward currently obtained has a certain correlation with the previous action and state sequence, for example, if most of the actions before the agent receives the reward are long-sighted, then the reward is likely to be a long-term reward, and a larger discount factor is required to ensure that it can be returned; if most of the actions before the agent receives the reward are short-sighted, then the reward is likely to be a short-term reward, and a smaller discount factor is required to prevent the return from being too long. Based on this, the inner cycle realizes the optimization update of the action and value network of the agent under the reinforcement learning framework, and the outer cycle optimization assigns the discount factor to different rewards according to the long-sightedness of the actions within a certain range before the rewards, assigns a larger discount factor to the long-sightedness, and assigns a smaller discount factor to the small long-sightedness, fully considers the long-sightedness difference of different actions, and realizes the optimization update of the self-adaptive discount factor network for different rewards.

[0051] The application is applicable to any value-based reinforcement learning algorithm, effectively improves the performance of the general reinforcement learning algorithm by assigning a separate dynamic discount factor to each reward, solves the problem of unreasonable fixed discount factor under actual task conditions, and has better convergence and robustness and stronger performance. BRIEF DESCRIPTION OF DRAWINGS

[0052] In order to more clearly illustrate the technical solutions in the embodiments of the application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description only constitute the embodiments of the application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of the provided drawings.

[0053] Figure 1 The flowchart of the self-adaptive dynamic discount factor reinforcement learning method provided by the application;

[0054] Figure 2 The update flowchart of the action network provided by the application;

[0055] Figure 3 The update flowchart of the value network provided by the application;

[0056] Figure 4 The update flowchart of the self-adaptive discount factor network provided by the application. DETAILED DESCRIPTION

[0057] With reference to the accompanying drawings: clearly and fully describe the technical solutions in the embodiments of the present application, obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of the present application.

[0058] As shown in the figure, the embodiments of the present application disclose a reinforcement learning method of adaptive dynamic discount factor, comprising: Figure 1

[0059] S1, construct a reinforcement learning model and an experience replay pool; the reinforcement learning model comprises an action network, a value network and an adaptive discount factor network;

[0060] S2, initialize the parameters of the action network, the value network and the adaptive discount factor network;

[0061] S3, randomly sample trajectory data from the experience replay pool as a training set and a validation set;

[0062] S4, according to the relationship between the adaptive discount factor corresponding to each reward in a certain range in the sampled trajectory data and the historical decision trajectory of the agent, obtain the adaptive discount factor under the current reward;

[0063] Fix the parameters of the adaptive discount factor network, update the parameters of the action network and the value network based on the training set;

[0064] S5, fix the parameters of the updated action network and value network, update the parameters of the adaptive discount factor network based on the adaptive discount factor;

[0065] S6, judge whether the maximum iteration number is reached, if yes, stop the iteration optimization process; if not, return to S3 to continue the alternate optimization of the parameters of the action network, the value network and the adaptive discount factor network.

[0066] The main architecture of the adaptive discount factor reinforcement learning method based on double-layer optimization of the present application mainly comprises an action network, a value network and an adaptive discount factor network, which can be described as the following double-layer optimization problem:

[0067]

[0068] s.t.

[0069]

[0070] Wherein, indicates the value network parameter, indicates the action network parameter, indicates the dynamic discount factor network parameter, indicates the adaptive discount factor, and indicates the discount factor. t ​denotes the trajectories sampled from the training set v denotes the trajectories sampled from the validation set, Ω(·) denotes the regularization term, which is the entropy of the action output probability distribution in this paper, and a denotes the regularization term coefficient, L(·) denotes the loss function under the actor-critic reinforcement learning framework, and the specific loss form can be written as the weighted sum of the action network loss and the value network loss, as follows:

[0071] L(·) = loss actor + βloss critic

[0072] The above double-layer optimization can be roughly summarized as the following two stages: 1) fixing the adaptive discount factor network parameters unchanged, under the sampled training set trajectories, the action network and value network parameters are simultaneously optimized through the actor-critic reinforcement learning loss function until the two network parameters converge; 2) when the action network and value network converge to the optimal, fixing the action network and value network parameters unchanged, under the sampled validation set trajectories, the network parameter update is realized by using the direct gradient and indirect gradient of the adaptive discount factor network through the actor-critic reinforcement learning loss function. The optimization process of alternating the two stages until the fixed optimization times are reached.

[0073] The reinforcement learning algorithm of the adaptive discount factor aims at the deficiency of the traditional reinforcement learning algorithm using a fixed discount factor, considers that the rewards at different times have different "far-sighted" characteristic values, i.e. corresponding discount factors, and establishes the relationship between the current reward "far-sighted" characteristic value and the previous historical decision trajectory of the agent, models this relationship through the adaptive discount factor deep network, and updates and iterates through the double-layer optimization meta-learning method to construct a new cumulative reward backtracking form, better fit the state value, and thus improve the reinforcement learning ability of the agent.

[0074] The present application can distinguish the difference in time required for different actions to produce benefits in a complex reinforcement learning task, and can distinguish whether the obtained reward is long-term benefit or short-term benefit through the two-stage double-layer optimization process. For example, in the commonly used benchmark task of multi-agent reinforcement learning, StarCraft Multi Agent Challenge (SMAC), when the agent performs action a1 "attack", it can cause damage to the enemy target and obtain reward r1 immediately, which should belong to short-term reward, and a smaller discount factor should be assigned to ensure that reward r1 is strongly related to action a1. When the agent performs action a2 "upgrade technology tree", it may not receive obvious reward in a short time, but in the long run, this action can improve the combat level of our side, so the reward r2 obtained in the future belongs to long-term reward, and a larger discount factor should be assigned to ensure that the obtained reward r2 is related to action a2, so that the agent learns that performing action a2 "upgrade technology tree" cannot obtain immediate effect, but can enhance the combat power of our side in the long run and obtain a higher value estimate. If the same discount factor is assigned to all rewards, the "long-term" and "short-term" of the reward cannot be distinguished, which is obviously unreasonable.

[0075] Therefore, in the dynamic discount factor reinforcement learning algorithm based on double-layer optimization, the present application converts the cumulative reward calculation method of the fixed discount factor into the cumulative reward calculation of the dynamic discount factor through the inner and outer double loops, considers that the current obtained reward has certain correlation with the previous action and state sequence, for example, if most of the actions before the agent receives the reward are far-sighted, then the reward is likely to be long-term reward, and a larger discount factor is needed to ensure that it can be returned; if most of the actions before the agent receives the reward are short-sighted, then the reward is likely to be short-term reward, and a smaller discount factor is needed to prevent long return. Based on this, the inner loop optimization uses the traditional reinforcement learning framework, and the outer loop optimization assigns a discount factor to different rewards according to the far-sightedness of the actions within a certain range before the reward: a larger discount factor is assigned to a larger far-sightedness, and a smaller discount factor is assigned to a smaller far-sightedness.

[0076] Next, the corresponding steps are further described.

[0077] In one specific embodiment, in S1, the time sequence trajectory tuple stored in the experience replay pool is wherein, represents a state sequence from the initial state to the terminal state, represents an action sequence from the initial state to the terminal state, represents a reward sequence from the initial state to the terminal state, This represents the sequence of probabilities of selected actions from the initial state to the final state.

[0078] To improve sampling efficiency and maximize the use of sample trajectory data, this embodiment of the invention sets up an experience replay pool to store the trajectories sampled from the interaction between the agent and the environment. This is compared to the (s) stored in DQN. t ,a t ,s t+1 ,r t+1 Unlike temporal difference tuples, because the adaptive discount factor changes the reward backtracking form, the temporal difference (TD) formula is no longer applicable. A Monte Carlo (MC) update form is used, and the data stored in the experience replay pool is... Time-series trajectory tuple.

[0079] The trajectory data sampled in S3 serves as both the training and validation sets. These sets are obtained separately from the experience replay pool, and they have no overlap. During training, cross-validation is used to improve the generalization ability of the reinforcement learning algorithm. In practice, to further improve training efficiency, the training and validation sets can be swapped for repeated training and optimization.

[0080] In one specific embodiment, the action network includes components such as a self-attention encoding network, a multilayer perceptron, a recurrent neural network, and activation functions. It takes the current state information of the agent as input and outputs the probability distribution of all executable actions of the agent.

[0081] The action network preprocesses the input state information through a self-attention encoding network, considers the influence of long-term and short-term historical observations and actions on the current state's action probability output through a recurrent neural network, and sends the processed information to a multilayer perceptron to extract relevant information. Finally, after activation function and softmax probability normalization, the policy output is formed.

[0082] like Figure 2 As shown, in S4, the process of updating the parameters of the action network is as follows:

[0083] S411. Sample trajectories from the experience replay pool as the training set τ t From the sampled trajectory data Extracting state sequences And divide each state s according to the time dimension t Successive Input Action Network π θ and value network In this process, the probability π of the agent's chosen action at each moment is obtained. θ (a t |s t Importance sampling coefficient α and the agent's value estimate of the current state. Importance sampling coefficient α = π θ (a t |s t ) / q t ;

[0084] S412, the parameters of the fixed adaptive discount factor network are obtained from the sampled trajectory data τ. t Extracting state sequences With action sequence And sort the state sequence by time dimension With action sequence By successively inputting the data into the adaptive discount factor network, the adaptive discount factor γ corresponding to the reward at each time step is obtained. t γ t =λ(s) 1:t ,a 1:t ), where λ represents the adaptive discount factor network parameters, s t Indicates the state at time t, a t R represents the action at time t. t q represents the reward at time t. t s represents the probability of the action selected at time t; 1:t ,a 1:t It represents the sequence of states and actions from the initial moment to time t.

[0085] S413, Collect the adaptive discount factor γ of the reward at each moment. t and from the sampled trajectory data Extracting reward sequences Calculate the agent's state s at each time step t Next, execute action a t Value target value And the difference between this target value and the corresponding value estimate. As a judgment in state s t Next, execute action a t Advantage estimation

[0086] At each moment, the agent is in state s. t Next, execute action a t Value target value The calculation formula is:

[0087]

[0088] In the above formula, where, This represents the cumulative sum of the discount rewards after time step t; i represents the i-th time step of the agent; t represents the current time step; r represents the discount factor at time step i; irepresents the reward obtained by the agent at time step i.

[0089] S414, update the parameters of the action network according to the selected action probability of the agent at each time θ (a t |s t ) and the advantage estimate Calculate the loss function of the action network, and update the parameters of the action network by stochastic gradient descent.

[0090] wherein the expression of the loss function of the action network is:

[0091]

[0092] The formula for updating the parameters of the action network iteratively is:

[0093]

[0094] wherein p represents the step size, represents the gradient of the action network parameters.

[0095] In another embodiment, the value network includes components such as self-attention encoding network, multi-layer perception network and recurrent neural network, inputs the current state information of the agent, and outputs the value estimate of the agent for the current state. The self-attention encoding network, the recurrent neural network in the value network and the corresponding part structure and parameters in the action network module are shared, the input state information is preprocessed through the self-attention encoding network, the influence of long-term and short-term historical observations and actions on the time axis on the value output for the current state is considered through the recurrent neural network, and the processed information is delivered to the multi-layer perception to extract relevant information, and finally the value estimate for the corresponding state is directly regressed.

[0096] As shown in FIG. 4B, the process of updating the parameters of the value network S4 is as follows: Figure 3

[0097] S421, sample a trajectory from the experience replay pool as a training set τ t , extract the state sequence from the sampled trajectory data , and input each state s t to the value network in time dimension, to obtain the value estimate of the agent for the current state at each time

[0098] S422, fix the parameters of the adaptive discount factor network, extract the state sequence t and the action sequence from the sampled trajectory data τ and input the state sequence​​ With action sequence By successively inputting the data into the adaptive discount factor network, the adaptive discount factor γ corresponding to the reward at each time step is obtained. t γ t =λ(s) 1:t ,a 1:t );

[0099] S423, Collect the adaptive discount factor γ of the reward acquired at every moment. t And from the sampled trajectory data τ t Extracting reward sequences Calculate the agent's state s at each time step t Next, execute action a t Value target value

[0100] S424. Based on the agent's target value at each moment. With value estimation Calculate the loss function of the value network and update the parameters of the value network using stochastic gradient descent.

[0101] The expression for the loss function of the value network is as follows:

[0102]

[0103] The formula for updating and iterating the value network is:

[0104]

[0105] Where ρ is the step size. This indicates that the gradient of the value network parameters is calculated.

[0106] In one specific embodiment, the adaptive discount factor network consists of a deep residual network, a self-attention network, and a multilayer perceptron. It takes the agent's previous state and action sequences as input and outputs an adaptive discount factor for the current reward backtracking. The deep residual network and the self-attention network perform feature extraction and encoding operations on the input state and action sequences, model a function based on the relationship between the state and action sequences and the discount factor, and update and iterate the function through a two-layer optimization meta-learning approach, continuously constructing new cumulative reward backtracking forms. The multilayer perceptron extracts the adaptive discount factor for the current reward from the constructed new cumulative reward backtracking forms.

[0107] like Figure 4 As shown, in S5, the process of updating the parameters of the adaptive discount factor network is as follows:

[0108] S511. Sample trajectories from the experience replay pool as the validation set τv The parameters of the fixed action network and value network are derived from the sampled validation set trajectory data. Extracting state sequences And classify the state s according to the time dimension t Successive Input Action Network π θ With value network In this process, the probability π of the agent's chosen action at each moment is obtained. θ (a t |s t Importance sampling coefficient α and the agent's value estimate of the current state. Wherein, the importance sampling coefficient α = π θ (a t |s t ) / q t ;

[0109] S512, from the sampled trajectory data Extract state sequences and action sequences, and sort the state sequences by time dimension. With action sequence By successively inputting the data into the adaptive discount factor network, the adaptive discount factor γ corresponding to the reward at each time step is obtained. t =λ(s) 1:t ,a 1:t );

[0110] S513, Collect the adaptive discount factor γ of the reward at each moment. t And extract the reward sequence from the sampled trajectory data. Calculate the agent's state s at each time step t Next, execute action a t Value target value The difference between this target value and the corresponding value estimate is used as the criterion for judgment in state s. t Next, execute action a t Advantage estimation At each moment, the agent is in state s. t Next, execute action a t Value target value

[0111] S514. Based on the probability π of the agent's selected action at each moment... θ (a t |s t ) and advantage estimation Calculate the loss function of the adaptive discount factor network and update the parameters of the adaptive discount factor network using stochastic gradient descent.

[0112] The expression for the loss function of the adaptive discount factor network is:

[0113]

[0114] The formula for updating iterations of the parameters of the adaptive discount factor network is:

[0115]

[0116] where ρ represents the step size, denotes the gradient of the adaptive discount factor network parameters, denotes the gradient of the action network and the value network. Indirect gradient can be obtained by the implicit function theorem.

[0117] The present application fully considers the influence of long-term and short-term benefits of performing an action in a real task by introducing the measurement of the action "far-sightedness", and assigns a separate discount factor for each reward. It can improve the performance of existing reinforcement learning algorithms and improve robustness.

[0118] The various embodiments described in this specification are presented by way of example, and each embodiment describes a specific implementation of the general principles described herein. The embodiments are not intended to limit or restrict the scope of the application to the embodiments disclosed, but rather, the scope of the application is to be determined entirely by the claims, which are to be interpreted in accordance with established doctrines of claim interpretation. The embodiments disclosed herein are intended to be used in combination with one another, and the description of the various embodiments is intended to be used in combination with one another. The various embodiments disclosed herein are intended to be used in combination with one another, and the description of the various embodiments is intended to be used in combination with one another.

[0119] The above description of disclosed embodiments enables a person skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A reinforcement learning method with adaptive dynamic discount factor, characterized in that, The application is applied to a multi-agent reinforcement learning task, a StarCraft multi-agent task, and comprises: S1, constructing a reinforcement learning model and an experience replay pool; the reinforcement learning model comprises an action network, a value network and an adaptive discount factor network; the experience replay pool stores trajectories sampled by an agent interacting with an environment; the stored in the experience replay pool is a time sequence trajectory pair wherein, denotes a state sequence from an initial state to a terminal state, denotes an action sequence from an initial state to a terminal state, denotes a reward sequence from an initial state to a terminal state, denotes a selected action probability sequence from an initial state to a terminal state; The adaptive discount factor network is composed of a deep residual network, a self-attention network and a multi-layer perception, inputs a state sequence and an action sequence of an agent, and outputs an adaptive discount factor for a reward rollback of a current time; S2, initializing parameters of the action network, the value network and the adaptive discount factor network; S3, randomly sampling trajectory data from the experience replay pool as a training set and a validation set; S4, obtaining an adaptive discount factor under a current reward according to a relationship between a discount factor corresponding to a certain range before each reward and a historical decision trajectory of an agent; if most actions before an agent receives a reward are long-sighted, a larger discount factor is needed to ensure that the reward can be returned; if most actions before an agent receives a reward are short-sighted, a smaller discount factor is needed to prevent the reward from being returned for too long; long-sighted actions include "upgrading a technology tree", and short-sighted actions include "attacking"; Fixing parameters of the adaptive discount factor network, updating parameters of the action network and the value network based on the training set; S5, fixing updated parameters of the action network and the value network, updating parameters of the adaptive discount factor network based on the validation set; S6, judging whether a maximum iteration number is reached, if yes, stopping an iteration optimization process; if no, returning to S3 to continue alternately optimizing parameters of the action network, the value network and the adaptive discount factor network.

2. The reinforcement learning method with adaptive dynamic discount factor according to claim 1, wherein, In S4, a process of updating parameters of the action network is as follows: S411. Use the sampled trajectory data as the training set. And extract the state sequence from it, and sort each state according to the time dimension. The action network is input sequentially and the value network In this process, the probability of the agent's selected action at each moment is obtained. Importance sampling coefficient And the agent's value estimation of the current state. ; S412, fix the parameters of the adaptive discount factor network, extract the state sequence and the action sequence from the sampled trajectory data and input the state sequence and the action sequence into the adaptive discount factor network in the time dimension to obtain the adaptive discount factor corresponding to the reward at each time point ; S413, collect the adaptive discount factor of the reward of each moment , and extract the reward sequence from the sampled trajectory data , calculate the value target value of the agent in the state performing the action at each moment, and take the difference between the value target value and the corresponding value estimate as the advantage estimate of the agent in the state performing the action ;​ S414, calculating a loss function of the action network according to a selected action probability of an agent at each time and an advantage estimation, and updating parameters of the action network through stochastic gradient descent.

3. The reinforcement learning method with an adaptive dynamic discount factor according to claim 2, wherein, In S414, an expression of the loss function of the action network is as follows: ； A formula for updating parameters of the action network iteratively is as follows: ； wherein, denotes a step size, denotes taking the gradient with respect to the action network parameters.

4. The reinforcement learning method with adaptive dynamic discount factor according to claim 1, wherein, In S4, a process of updating parameters of the value network is as follows: S421. Use the sampled trajectory data as the training set. And extract the state sequence from it, and sort each state according to the time dimension. The value network is input sequentially In this process, the agent obtains a value estimate of the current state at each moment. ; S422, fix the parameters of the adaptive discount factor network, extract the state sequence and the action sequence from the sampled trajectory data , and input the state sequence and the action sequence into the adaptive discount factor network in the time dimension to obtain the adaptive discount factor corresponding to the reward at each time point ; S423、collecting an adaptive discount factor of each moment reward that has been acquired , and extracting a reward sequence from the sampled trajectory data , calculating a value target value of an action of the agent in a state at each moment ; ; S424, calculating a loss function of the value network according to a value target value of an agent at each time and a value estimation, and updating parameters of the value network through stochastic gradient descent.

5. The reinforcement learning method of claim 4, wherein, In S424, an expression of the loss function of the value network is as follows: ； The formula for updating iterations of the value network is ; wherein, is a step size, denotes taking the gradient with respect to the value network parameters.

6. The reinforcement learning method with an adaptive dynamic discount factor according to claim 1, wherein, In S5, a process of updating parameters of the adaptive discount factor network is as follows: S511, fix the parameters of the action network and the value network, and take the sampled trajectory data as a verification set , and extract the state sequence therefrom and arrange the states in a time dimension , input the action network sequentially , and the value network , to obtain the probability of the action selected by the agent at each time , the importance sampling coefficient , and the value estimate of the agent for the current state ; S512, extract the state sequence and the action sequence from the sampled trajectory data, and input the state sequence and the action sequence into the adaptive discount factor network in time dimension, to obtain the adaptive discount factor corresponding to the reward at each time point ; S513, Collect adaptive discount factors for rewards at every moment. It extracts the reward sequence from the sampled trajectory data and calculates the agent's state at each moment. Next action Value target value The difference between this target value and the corresponding value estimate is used as the criterion for judging the state. Next action Advantage estimation ; S514, calculating a loss function of the adaptive discount factor network according to a selected action probability of an agent at each time and an advantage estimation, and updating parameters of the adaptive discount factor network through stochastic gradient descent.

7. The reinforcement learning method of claim 6, wherein, In S514, an expression of the loss function of the adaptive discount factor network is as follows: ； A formula for updating parameters of the adaptive discount factor network iteratively is as follows: ； wherein, denotes a step size, denotes taking the gradient with respect to the adaptive discount factor network parameters, denotes taking the gradient with respect to the action network and the value network.

8. The reinforcement learning method of claim 2 or 3 or 6, wherein, At each time step, the agent is in a state The value target value of the action The calculation formula is: ； wherein, represents a cumulative sum of discounted rewards after time step t; i represents the i-th time step of the agent; t represents the current time step; represents a discount factor at time step i; represents a reward obtained by the agent at time step i.

9. The reinforcement learning method with an adaptive dynamic discount factor according to claim 1, wherein, The deep residual network and the self-attention network perform feature extraction and encoding operations on the input state sequence and action sequence, model functions according to the relationship between the state sequence, the action sequence and the discount factor, and iteratively update the functions through double-layer optimization meta-learning to constantly construct new cumulative reward backtracking forms; the multi-layer perceptron extracts an adaptive discount factor under the current reward from the constructed new cumulative reward backtracking form.

Images