Method and system for optimizing deep reinforcement learning based on composite priority experience playback

Abstract

The invention provides a method and system for optimizing deep reinforcement learning based on composite priority experience playback. The method comprises the following steps: acquiring a current training network and a target network; the current training network comprises a current strategy network and a current value network, and the target network comprises a target strategy network and a target value network; acquiring an experience tuple based on an action selection strategy of the current strategy network; calculating a composite priority of the experience tuple, and enabling a life cycle pointer to update an experience playback area according to the experience tuple and the composite priority of the experience tuple; and sampling from the experience playback area to update the network parameters of the current training network, and updating the network parameters of the current training network to a target network. According to the method, the time sequence difference error of the instant reward and the experience is introduced as the priority evaluation standard of the new experience, the priority of the new experience is improved through the playback times, and the replacement efficiency of the experience playback area is improved through the lifetime pointer.

Description

Technical Field

[0001] The present application relates to the technical field of deep reinforcement learning, and in particular to a method and system for optimizing deep reinforcement learning based on composite priority experience replay. Background Art

[0002] As an important branch of artificial intelligence, deep reinforcement learning (DRL) combines the representation learning ability of deep neural networks and the decision-making optimization mechanism of reinforcement learning, enabling it to autonomously learn optimal strategies in complex, dynamic, and uncertain environments. Therefore, deep reinforcement learning has been widely used in many fields such as autonomous driving, energy scheduling, and industrial system control, and has achieved remarkable results.

[0003] In many deep reinforcement learning algorithms, experience replay (ER) technology is widely used to improve the utilization of experience and break the temporal correlation of experience. However, there is a problem with the experience replay mechanism based on uniform sampling and batch update, that is, the experience with high value but small quantity is not efficiently utilized. To solve this problem, the Prioritized Experience Replay (PER) mechanism introduces Temporal Difference Errors (TD-errors) and assigns priority to each experience according to the TD-errors of the experience. The larger the absolute value of TD-errors, the higher the priority of the experience and the greater the probability of being sampled. At the same time, PER combines the ER mechanism to alleviate the problem of too large experience replay area by directly putting the latest experience into a small sample to train the neural network. However, the traditional PER mechanism mainly divides the experience by value based on the temporal difference error of the experience, which easily ignores other important factors in deep reinforcement learning, resulting in the convergence speed and convergence performance of deep reinforcement learning to be further improved. Therefore, how to consider complex factors in priority experience replay to further optimize the convergence speed and convergence performance of deep reinforcement learning is a technical problem that needs to be urgently solved by technical personnel in this field. Summary of the invention

[0004] The purpose of this application is to provide a method and system for optimizing deep reinforcement learning based on composite priority experience replay, which is used to solve the current technical problem of how to consider composite factors in priority experience replay to further improve the convergence speed and convergence performance of deep reinforcement learning.

[0005] The first aspect of the present application provides a method for optimizing deep reinforcement learning based on composite priority experience replay, the method comprising:

[0006] Acquire a current training network and a target network; the current training network includes a current policy network and a current value network, and the target network includes a target policy network and a target value network;

[0007] Acquire an experience tuple based on the action selection strategy of the current strategy network;

[0008] Calculating the composite priority of the experience tuple so that the life cycle pointer updates the experience playback area according to the experience tuple and the composite priority of the experience tuple;

[0009] Sampling is performed from the experience playback area to update the network parameters of the current training network, and the network parameters of the current training network are updated to the target network.

[0010] In some embodiments of the first aspect of the present application, before obtaining the experience tuple, the method further includes:

[0011] Initialize environment hyperparameters;

[0012] Initializing network parameters of the current training network and the target network;

[0013] Initialize random noise, experience playback area, playback count array and the life cycle pointer; the playback count array is the same size as the experience playback area and is used to store the number of times the experience at each position in the experience playback area is sampled; the life cycle pointer points to the position of the experience with the minimum composite priority in the experience playback area.

[0014] In some embodiments of the first aspect of the present application, acquiring the experience tuple based on the action selection strategy of the current policy network includes:

[0015] Obtaining the current environment state corresponding to the current time step to input into the current strategy network;

[0016] According to the corresponding current action and random noise, the current strategy action is obtained and output to the environment to obtain the corresponding current reward value and the corresponding environment state of the next time step;

[0017] The current environment state, the current strategy action, the current reward value, and the environment state corresponding to the next time step are used as the current experience tuple.

[0018] In some embodiments of the first aspect of the present application, calculating the composite priority of the experience tuple so that the lifecycle pointer updates the experience playback area according to the experience tuple and the composite priority of the experience tuple includes:

[0019] Calculating a temporal difference error of the current experience tuple to calculate a composite priority evaluation standard of the current experience tuple based on the temporal difference error and the immediate reward;

[0020] Calculate the composite priority of the current experience tuple based on the number of experience playbacks and the composite priority evaluation criteria of the current experience tuple;

[0021] Replace the current experience tuple to the position pointed to by the life cycle pointer, and set the composite priority of the current experience tuple to the highest priority;

[0022] Reset the number of experience replays at the location pointed to by the life cycle pointer to 0.

[0023] In some embodiments of the first aspect of the present application, calculating the timing differential error includes:

[0024] δ i =r i +γQ′(s i+1 ,π′(s i+1 |θ π′ )+n ou )-Q(s i ,a i |θ Q )

[0025] Among them, δ i represents the time series difference error of the i-th experience, r i represents the reward value obtained, γ is the reward discount factor, θ π′ represents the current policy network π′(s t ,θ) parameter, θ Q Represents the current value network Q(s t ,a t ,θ) parameter, π′(s i+1 |θ μ′ ) represents the target policy network π′(s t ,θ) in s i+1 The action generated by ou represents OU random noise, Q(s i ,a i |θ Q ) represents the Q value calculated by the current value network, Q′(s i+1 ,π′(s i+1 |θ μ′ )+n ou ) is the expected Q value calculated by the target value network.

[0026] In some embodiments of the first aspect of the present application, calculating the composite priority evaluation criterion includes:

[0027] Γ i =||δ i |+λ*ri |

[0028] Among them, Γ i represents the composite priority evaluation of the i-th experience, |δ i | is the time series difference error of the i-th experience, r i represents the reward value obtained, and λ is a parameter used to adjust the weight of the immediate reward in the composite priority evaluation criterion.

[0029] In some embodiments of the first aspect of the present application, calculating the composite priority includes:

[0030] p i =μ C(i) Γ i +∈

[0031] Among them, p i represents the priority of the i-th experience, Γ i represents the composite priority evaluation criteria of the i-th experience, C(i) represents the number of playbacks of the i-th experience, μ represents the playback discount factor, and ∈ is a very small normal number.

[0032] In some embodiments of the first aspect of the present application, sampling from the experience replay area to update the network parameters of the current training network, and updating the network parameters to the target network includes:

[0033] Determining a sampling probability of experience according to the composite priority;

[0034] Sampling from the experience playback area based on the sampling probability and updating a playback count array;

[0035] Update the network parameters of the current policy network and the network parameters of the current value network based on the sampled experience group;

[0036] Updating the composite priority and the composite priority evaluation of the sampled experience group in the experience playback area;

[0037] The network parameters of the current value network are updated to the target value network, and the network parameters of the current policy network are updated to the target policy network.

[0038] In some embodiments of the first aspect of the present application, determining the empirical sampling probability according to the composite priority includes:

[0039]

[0040] Among them, P i represents the sampling probability of the i-th experience, and α represents the priority adjustment coefficient.

[0041] A second aspect of the present application provides a system for deep reinforcement learning based on composite priority experience replay optimization, including: an acquisition module, used to acquire a current training network and a target network; the current training network includes a current policy network and a current value network, and the target network includes a target policy network and a target value network;

[0042] An experience accumulation module, used for acquiring experience tuples based on the action selection strategy of the current strategy network;

[0043] An updating module, used for calculating the composite priority of the experience tuple, so that the life cycle pointer updates the experience playback area according to the experience tuple and the composite priority of the experience tuple;

[0044] The sampling training module is used to sample from the experience playback area to update the network parameters of the current training network, and update the network parameters of the current training network to the target network.

[0045] In summary, the method and system for optimizing deep reinforcement learning based on composite priority experience replay provided by the present application have the following beneficial effects: on the one hand, by introducing timely rewards and temporal difference errors of experience as priority evaluation criteria for new experience, on the other hand, by introducing replay discount factors and the number of experience replays to improve the priority of new experience. In addition, the replacement efficiency of the experience replay area is improved by introducing a life cycle pointer. The present application solves the problem that the traditional priority experience replay mechanism does not take into account the immediate rewards and the number of replays of experience enough, and provides an effective solution for the efficient learning of deep reinforcement learning algorithms. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 Shown is a flowchart of a method for optimizing deep reinforcement learning based on composite priority experience replay in one embodiment of the present application.

[0047] Figure 2 Shown is a flowchart of a method for optimizing deep reinforcement learning based on composite priority experience replay in one embodiment of the present application.

[0048] Figure 3 Shown is a flowchart of a method for optimizing deep reinforcement learning based on composite priority experience replay in one embodiment of the present application.

[0049] Figure 4 Shown is a flowchart of a method for optimizing deep reinforcement learning based on composite priority experience replay in one embodiment of the present application.

[0050] Figure 5 Shown is a schematic diagram of the principle of a method for optimizing deep reinforcement learning based on composite priority experience replay in one embodiment of the present application.

[0051] Figure 6 Shown is a structural schematic diagram of a system for deep reinforcement learning based on composite priority experience replay optimization in one embodiment of the present application. DETAILED DESCRIPTION

[0052] The following describes the embodiments of the present application through specific examples, and those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. The present application can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0053] It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present application, and thus the drawings only show components related to the present application rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed at will, and the component layout may also be more complicated.

[0054] In addition, in this application, descriptions such as "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in this field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by this application.

[0055] The traditional priority experience replay mechanism only focuses on the temporal difference error and assigns a priority to each experience based on the temporal difference error of the experience. The larger the absolute value of the temporal difference error, the higher the priority of the experience and the greater the probability of being sampled. Given that each interaction between the agent and the environment will generate different immediate rewards, the temporal difference error will actually change with the value of the immediate reward. Therefore, in deep reinforcement learning, the immediate reward of experience can reflect its potential value and is crucial to the learning of the agent. If the traditional priority experience replay mechanism is adopted, it is likely that the experience with high immediate rewards will be ignored.

[0056] In addition, the number of experience replays also has a certain potential value. According to the PER mechanism, each experience can be replayed multiple times. The replay process is equivalent to the process of extracting information from the experience. For two experiences that have been updated with roughly the same temporal difference error, the experience with fewer replays is more likely to squeeze out valuable information, especially those experiences that have not been replayed repeatedly.

[0057] However, the traditional PER mechanism mainly divides the value of experience based on the temporal difference error of experience, which easily ignores the experience with higher immediate rewards and fewer revisits, affecting the convergence speed and performance of deep reinforcement learning. Therefore, how to consider complex factors in the priority experience replay to further optimize the convergence speed and performance of deep reinforcement learning is a technical problem that needs to be solved urgently by those skilled in the art.

[0058] To solve the existing problems, this application proposes a method and system for optimizing deep reinforcement learning based on composite priority experience replay. On the one hand, it introduces the timely reward and the temporal difference error of experience as the priority evaluation standard for new experience, and on the other hand, it introduces the replay discount factor and the number of experience replays to improve the priority of new experience. In addition, the replacement efficiency of the experience replay area is improved by introducing the life cycle pointer. This application solves the problem that the traditional priority experience replay mechanism does not take the immediate reward and replay times of experience into consideration, and provides an effective solution for the efficient learning of deep reinforcement learning algorithms.

[0059] It should be noted that the present application can be applied to any deep reinforcement learning algorithm based on priority experience replay. The following embodiment combines it with the deep deterministic policy gradient algorithm (DDPG) as an example to illustrate its complete learning process.

[0060] Figure 1 A flow chart of a method for optimizing deep reinforcement learning based on composite priority experience replay in one embodiment of the present application is shown. Figure 1 As shown, the method comprises the following steps:

[0061] S1. Obtain a current training network and a target network; the current training network includes a current policy network and a current value network, and the target network includes a target policy network and a target value network.

[0062] Specifically, the current training network and the target network have exactly the same network structure. The target network is used to calculate the target value, and by reducing the fluctuation of the estimate, a more stable target value is provided, thereby avoiding the estimated target of the current training network from changing too frequently.

[0063] In some embodiments, the current training network includes an Actor network π(s t ,θ) and Critic network Q(s t ,a t ,θ). The target network includes a target strategy network π′(s t ,θ) and the target value network Q′(s t ,a t ,θ).

[0064] Among them, Actor network π(s t ,θ) as the current policy network can make current decisions to control the agent to make current actions, while the Critic network Q(s t ,a t ,θ) as the current value network can evaluate the expected benefits of the current action made by the agent (obtain the Q value). In the deep reinforcement learning process of the agent, the policy gradient update is actually performed by updating the network parameters of the current policy network (making the action benefit Q value output by the current value network as high as possible) to minimize the loss of the benefit evaluation of the current value network (evaluate the action benefit as accurately as possible), thereby enabling the agent to make the best action.

[0065] In some embodiments, after obtaining the current training network and the target network, the environment hyperparameters should be initialized, and the network parameters of the current training network and the target network should be initialized, as well as the random noise, the experience playback area, the playback count array and the life cycle pointer.

[0066] The environmental hyperparameters include: number of samples, network learning rate, experience playback area capacity, number of training rounds, reward discount factor, control usage priority, control correction degree, control correction degree growth rate, and playback discount factor. In some embodiments, the number of samples K = 64, the current strategy network learning rate α = 5*10 -4 , the current value network learning rate is β = 1*10 -3 , experience playback area capacity N = 10 4 , the number of training rounds M = 50, the reward discount factor γ = 0.95. In the priority experience replay mechanism, the priority adjustment coefficient α = 0.6, the importance sampling weight β = 0.4, and the importance sampling weight growth rate β rate =1.01, playback discount factor μ=0.95.

[0067] The network parameters of the current training network and the target network include the number of network layers and nodes, activation function, and optimizer. In some embodiments, a fully connected Critic network Q and an Actor network π with two hidden layers are initialized, wherein the first layer has 128 hidden units and the second layer has 128 hidden units, a linear rectifier function (RELU for short) is used as the activation function between layers, and RMSprop is used as the network optimizer.

[0068] Among them, the playback count array C is the same size as the experience playback area R, and is used to store the number of times the experience at each position in the experience playback area is sampled; the life cycle pointer L points to the position of the experience with the minimum composite priority in the experience playback area.

[0069] S2. Acquire experience tuples based on the action selection strategy of the current strategy network.

[0070] In deep reinforcement learning, the action selection strategy of the current policy network is used to select appropriate actions based on the state of the environment, and interact with the environment through these actions to obtain experience tuples. These experience tuples are a crucial source of data in the training process of reinforcement learning algorithms. The agent accumulates these experiences by continuously interacting with the environment, and then uses them to adjust the strategy to train the current training network and the target network. Figure 2 A flow chart of a method for optimizing deep reinforcement learning based on composite priority experience replay in one embodiment of the present application is shown. Figure 2 As shown, obtaining the experience tuple based on the action selection strategy of the current strategy network includes the following steps:

[0071] S21. Obtain the current environment state corresponding to the current time step to input into the current strategy network.

[0072] S22. According to the corresponding current action and random noise, the current strategy action is obtained and output to the environment to obtain the corresponding current reward value and the environment state corresponding to the next time step.

[0073] S23. Taking the current environment state, the current strategy action, the current reward value, and the environment state corresponding to the next time step as the current experience tuple.

[0074] In some embodiments, the current environment state s corresponding to the current time step t is obtained. t The current policy network selects the corresponding current action according to the current environment state, and obtains the current policy action a by combining the current action and random OU noise Φ t Output to the environment to obtain the corresponding current reward value r t , and the corresponding environmental state s in the next time stept+1 . Then the current experience tuple can be expressed as (s t ,a t ,r t ,s t+1 ).

[0075] In fact, when the current policy network in the above embodiment of the present application selects an action, it does not directly output the current action based on the current environment state, but adds random OU noise Φ on the basis of the current action to obtain the current policy action a output to the environment. t , in order to further optimize the effect of deep reinforcement learning.

[0076] S3. Calculate the composite priority of the experience tuple, so that the life cycle pointer updates the experience playback area according to the experience tuple and the composite priority of the experience tuple.

[0077] After obtaining the experience tuple, it is also necessary to calculate the composite priority of the experience tuple according to the determined criteria to determine the importance of each experience tuple, and store them in the experience replay pool through the life cycle pointer, so as to sort the experience tuples according to the composite priority and adjust the sampling probability of the experience tuple in the experience replay pool for sampling in subsequent training. Figure 3 A flow chart of a method for optimizing deep reinforcement learning based on composite priority experience replay in one embodiment of the present application is shown. Figure 3 As shown, calculating the composite priority of the experience tuple so that the life cycle pointer updates the experience playback area according to the experience tuple and the composite priority of the experience tuple includes the following steps:

[0078] S31. Calculate the temporal difference error of the current experience tuple to calculate the composite priority evaluation standard of the current experience tuple based on the temporal difference error and the immediate reward.

[0079] Specifically, when calculating the composite priority of the experience tuple, it is necessary to first determine the composite priority evaluation criteria to determine the importance of each experience tuple. In the embodiment of the present application, the temporal difference error and the immediate reward are added to the composite priority evaluation criteria to evaluate the importance of each experience tuple, thereby avoiding assigning a priority to each experience based on a single temporal difference error, thereby improving the convergence speed and convergence performance of deep reinforcement learning.

[0080] Specifically, calculating the timing difference error includes:

[0081] δ i =r i +γQ ′ (s i+1 ,π′(s i+1 |θ π′)+n ou )-Q(s i ,a i |θ Q ) (1)

[0082] Among them, δ i represents the time series difference error of the i-th experience, r i represents the reward value obtained, γ is the reward discount factor, θ π′ represents the current policy network π′(s t ,θ) parameter,θ Q Represents the current value network Q(s t ,a t ,θ) parameter, π′(s i+1 |θ μ′ ) represents the target policy network π′(s t ,θ) in s i+1 The action generated by ou represents OU random noise, Q(s i ,a i |θ Q ) represents the Q value calculated by the current value network, Q′(s i+1 ,π′(s i+1 |θ μ′ )+n ou ) is the expected Q value calculated by the target value network.

[0083] Furthermore, the composite priority evaluation criteria for calculating the current experience tuple based on the temporal difference error and the instant reward include:

[0084] Γ i =||δ i |+λ*r i | (2)

[0085] Among them, Γ i represents the composite priority evaluation criteria of the i-th experience, |δ i | is the time series difference error of the i-th experience, r i represents the reward value obtained, and λ is a parameter used to adjust the weight of the immediate reward in the composite priority evaluation criterion.

[0086] S32, calculating the composite priority of the current experience tuple based on the number of experience playbacks and the composite priority evaluation standard of the current experience tuple.

[0087] Since the number of experience replays has a certain potential value, after determining the composite priority evaluation criteria, it is necessary to introduce the number of replays and the replay discount factor to improve the priority of the new experience. Specifically, calculating the composite priority of the current experience tuple includes:

[0088] p i =μ C(i) Γ i +∈ (3)

[0089] Among them, p i represents the priority of the i-th experience, Γ i represents the composite priority evaluation criteria of the i-th experience, C(i) represents the number of playbacks of the i-th experience, μ represents the playback discount factor, and ∈ is a very small normal number.

[0090] S33. Replace the current experience tuple to the position pointed to by the lifecycle pointer, and set the composite priority of the current experience tuple to the highest priority.

[0091] S34. Reset the number of experience replays at the location pointed to by the life cycle pointer to 0.

[0092] After obtaining the experience tuple and its corresponding composite priority, it is necessary to store it in the experience playback area to accumulate experience and provide a basis for subsequent sampling training.

[0093] Specifically, the life cycle pointer is used to update the experience playback area according to the experience tuple and the composite priority of the experience tuple. The life cycle pointer L points to the position of the experience with the lowest composite priority in the experience playback area, so as to ensure that each time a new experience is inserted, the experience with the lowest priority will be replaced. This helps to keep the experience in the experience playback area more diverse, avoid some experience tuples from not being updated for a long time, and thus accelerate the learning of the intelligent agent.

[0094] After obtaining the current experience tuple (s t ,a t ,r t ,s t+1 ), replace the current experience tuple to this position (the position pointed to by the life cycle pointer), and based on the highest priority p of the current experience tuple t =max i<t p i , ensuring that the current experience tuple has sufficient influence. When the replacement is completed, the playback times C(L) of the position is reset to 0, which indicates that the current experience tuple is a new experience and can provide new information for the next update.

[0095] By replacing the lifecycle pointer, the agent can more efficiently use high-priority experience in training while avoiding over-reliance on certain experience. In addition, by resetting the number of replays, the experience in the replay pool is guaranteed to be updated in a timely manner and used fairly, which helps to accelerate the deep reinforcement learning process and improve performance.

[0096] S4. Sampling is performed from the experience playback area to update the network parameters of the current training network, and the network parameters of the current training network are updated to the target network.

[0097] Specifically, sampling experience from the experience replay area for training can break the time correlation of experience, make the training process more stable, and improve the efficiency of experience utilization. Figure 4 A flow chart of a method for optimizing deep reinforcement learning based on composite priority experience replay in one embodiment of the present application is shown. Figure 4 As shown, step S4 includes the following steps:

[0098] S41. Determine an empirical sampling probability according to the composite priority.

[0099] Specifically, determining the sampling probability of each experience tuple according to the composite priority can adjust the probability of each experience being sampled according to its priority, thereby updating the experiences that are most valuable for learning more frequently.

[0100] Specifically, determining the sampling probability of experience according to the composite priority includes:

[0101]

[0102] Among them, P i represents the sampling probability of the i-th experience, and α represents the priority adjustment coefficient.

[0103] S42: Sampling is performed from the experience playback area based on the sampling probability, and a playback count array is updated.

[0104] According to the sampling probability of experience, experience is selected from the experience playback area for sampling, and the playback count array is updated after sampling to record the playback times of the sampled experience. This can ensure that more valuable or important experiences are sampled first, and at the same time, the frequency of use of each experience is tracked through the playback count array, providing a basis for subsequent replacement or update, thereby optimizing the efficiency and effect of experience playback, and further improving the convergence speed and convergence performance of deep reinforcement learning.

[0105] In some embodiments, the playback count array is updated by adding one to the playback times of the sample experience and assigning it to the updated playback times, that is, C(i)=C(i)+1.

[0106] S43. Update the network parameters of the current policy network and the network parameters of the current value network based on the sampled experience group.

[0107] Specifically, the sampled experience is used to update the network parameters of the current policy network, thereby maximizing the expected cumulative return to optimize the parameters of the policy network. In some embodiments, the network parameters of the current policy network can be updated by the policy gradient method, as follows:

[0108]

[0109] Formula (5) represents the current environment state S t =s i When the calculation is in a specific state s i The policy gradient under , by sampling multiple state-action pairs in the experience replay area, calculating the gradient and averaging it, a stable gradient estimate can be obtained.

[0110] in, is the gradient of the current value network, reflecting the current strategy π(s i ,θ) in state S t and action a i The gradient information under is the gradient of the current policy network, reflecting the policy in state S when the parameters θ of the current policy network are adjusted t The rate of change under .

[0111] Specifically, the Q parameter of the current value network is updated by minimizing the loss function, so that the current value network can evaluate the action benefit as accurately as possible. The details are as follows:

[0112]

[0113] in, It is expressed as the square value of the time series difference error of the i-th experience, w i is the relevant weight of the i-th experience. In the weighted loss function, the loss of each experience can be calculated according to its corresponding weight w i Make adjustments.

[0114] S44. Update the composite priority and the composite priority evaluation of the sampled experience group in the experience playback area.

[0115] S45: Update the network parameters of the current value network to the target value network, and update the network parameters of the current policy network to the target policy network.

[0116] In some embodiments, the current value network Q(s) is periodically updated. t ,a i ,θ) is soft updated to the target value network Q′(s t ,a i ,θ), the current measurement network π(st ,θ) is soft-updated to the target policy network π′(s t ,θ), which can make the training process more stable and reduce the amount of calculation.

[0117] It should be noted that when the maximum number of training rounds is reached, the training is terminated and the final target network is output, otherwise the process returns to step S2 to continue acquiring experience tuples. In some embodiments, if an interaction termination signal is obtained from the environment, the training is terminated.

[0118] Figure 5 The schematic diagram of the principle of the method for optimizing deep reinforcement learning based on composite priority experience replay described in the embodiment of the present application is shown. Figure 5 As shown, the Actor network serves as the current strategy network, and outputs the current strategy action to the environment according to the current action corresponding to the current state of the environment and the random noise, obtains the corresponding current reward value and the environment state corresponding to the next time step, and the Critic network serves as the current value network to evaluate the benefit of the current strategy action and output the Q value. At the same time, an experience tuple is formed according to the current state, the current strategy action, the reward value and the next state, so as to replace the current experience tuple to the position pointed to by the life cycle pointer and store it in the experience playback area. In addition, the temporal differential error of the current experience tuple is calculated to calculate the composite priority evaluation, and finally the composite priority of the current experience tuple is obtained, which is set as the highest priority. Sampling is performed from the experience playback area according to the sampling probability obtained by the composite priority calculation, and the playback count array is updated. Based on the sampled experience group, the network parameters of the current strategy network and the network parameters of the current value network are updated, and the composite priority and the composite priority evaluation of the sampled experience group in the experience playback area are updated; the Critic network Q(s t ,a i ,θ) parameter soft update to its target network Q′(s t ,a i ,θ), Actor networkπ(s t ,θ) parameter soft update to its target network π′(s t ,θ). If the interaction termination signal is obtained from the environment, the training process is completed.

[0119] Therefore, the method for optimizing deep reinforcement learning based on composite priority experience replay provided by the present application can improve the priority of new experience by introducing timely rewards and temporal difference errors of experience as the priority evaluation criteria for new experience, and by introducing replay discount factors and the number of experience replays. In addition, the present application improves the replacement efficiency of the experience replay area by introducing a life cycle pointer. The present application solves the problem that the traditional priority experience replay mechanism does not take into account the immediate rewards and the number of replays of experience, and provides an effective solution for the efficient learning of deep reinforcement learning algorithms.

[0120] The protection scope of the method for optimizing deep reinforcement learning based on composite priority experience replay in the embodiment of the present application is not limited to the execution order of the steps listed in the present embodiment. All solutions implemented by adding, reducing or replacing steps in the prior art based on the principles of the present application are included in the protection scope of the present application.

[0121] An embodiment of the present application also provides a system based on deep reinforcement learning with compound priority experience replay optimization. The system based on deep reinforcement learning with compound priority experience replay optimization can implement the method based on deep reinforcement learning with compound priority experience replay optimization described in the present application. However, the implementation device of the system based on deep reinforcement learning with compound priority experience replay optimization described in the present application includes but is not limited to the structure of the system based on deep reinforcement learning with compound priority experience replay optimization listed in the present embodiment. All structural deformations and replacements of the prior art made according to the principles of the present application are included in the protection scope of the present application.

[0122] Figure 6 The schematic diagram of the structure of the system based on composite priority experience replay optimization deep reinforcement learning described in the embodiment of the present application is shown as follows: Figure 6 As shown, the system 4 for optimizing deep reinforcement learning based on composite priority experience replay includes an acquisition module 41, an experience accumulation module 42, an update module 43 and a sampling training module 44.

[0123] An acquisition module 41 is used to acquire a current training network and a target network; the current training network includes a current policy network and a current value network, and the target network includes a target policy network and a target value network;

[0124] An experience accumulation module 42, configured to obtain an experience tuple based on the action selection strategy of the current strategy network;

[0125] An updating module 43, used for calculating the composite priority of the experience tuple, so that the life cycle pointer updates the experience playback area according to the experience tuple and the composite priority of the experience tuple;

[0126] The sampling training module 44 is used to sample from the experience playback area to update the network parameters of the current training network, and update the network parameters of the current training network to the target network.

[0127] It should be noted that the structures and principles of the acquisition module, the generation module, the indicator module and the search module, as well as the beneficial effects achieved by the device are the same as those in the above embodiment, and will not be described in detail here.

[0128] It should be noted that if an interaction termination signal is obtained from the environment, the training ends.

[0129] In the several embodiments provided in the present application, it should be understood that the disclosed system, device or method can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of modules / units is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or units can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules or units, which can be electrical, mechanical or other forms.

[0130] The modules / units described as separate components may or may not be physically separated, and the components displayed as modules / units may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules / units may be selected according to actual needs to achieve the purpose of the embodiments of the present application. For example, the functional modules / units in the various embodiments of the present application may be integrated into one processing module, or each module / unit may exist physically separately, or two or more modules / units may be integrated into one module / unit.

[0131] Those of ordinary skill in the art should further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0132] Based on the above embodiments, the embodiments of the present application also provide a computer-readable storage medium. A person of ordinary skill in the art can understand that all or part of the steps in the method for implementing the above embodiments can be completed by instructing the processor through a program, and the program can be stored in a computer-readable storage medium, and the storage medium is a non-transitory medium, such as a random access memory, a read-only memory, a flash memory, a hard disk, a solid-state hard disk, a magnetic tape, a floppy disk, an optical disc, and any combination thereof. The above storage medium can be any available medium that a computer can access or a data storage device such as a server or a data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a digital video disc (DVD)), or a semiconductor medium (e.g., a solid-state disk (SSD)), etc.

[0133] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0134] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0135] The descriptions of the processes or structures corresponding to the above-mentioned figures have different emphases. For parts that are not described in detail in a certain process or structure, please refer to the relevant descriptions of other processes or structures.

[0136] The above embodiments are merely illustrative of the principles and effects of the present application and are not intended to limit the present application. Anyone familiar with the technology may modify or change the above embodiments without violating the spirit and scope of the present application. Therefore, all equivalent modifications or changes made by a person of ordinary skill in the art without departing from the spirit and technical ideas disclosed in the present application shall still be covered by the claims of the present application.

Claims

1. A method for optimizing deep reinforcement learning based on compound priority experience replay, characterized in that: The method comprises: Acquire a current training network and a target network; the current training network includes a current policy network and a current value network, and the target network includes a target policy network and a target value network; Acquire an experience tuple based on the action selection strategy of the current strategy network; Calculating the composite priority of the experience tuple so that the life cycle pointer updates the experience playback area according to the experience tuple and the composite priority of the experience tuple; Sampling is performed from the experience playback area to update the network parameters of the current training network, and the network parameters of the current training network are updated to the target network.

2. The method for deep reinforcement learning based on composite priority experience replay optimization according to claim 1, characterized in that: Before obtaining the experience tuple, the method further includes: Initialize environment hyperparameters; Initializing network parameters of the current training network and the target network; Initialize random noise, experience playback area, playback count array and the life cycle pointer; the playback count array is the same size as the experience playback area and is used to store the number of times the experience at each position in the experience playback area is sampled; the life cycle pointer points to the position of the experience with the minimum composite priority in the experience playback area.

3. The method for deep reinforcement learning based on composite priority experience replay optimization according to claim 1, characterized in that: The action selection strategy based on the current strategy network obtains the experience tuple including: Obtaining the current environment state corresponding to the current time step to input into the current strategy network; According to the corresponding current action and random noise, the current strategy action is obtained and output to the environment to obtain the corresponding current reward value and the corresponding environment state of the next time step; The current environment state, the current strategy action, the current reward value, and the environment state corresponding to the next time step are used as the current experience tuple.

4. The method for deep reinforcement learning based on composite priority experience replay optimization according to claim 1, characterized in that: Calculating the composite priority of the experience tuple so that the life cycle pointer updates the experience playback area according to the experience tuple and the composite priority of the experience tuple includes: Calculating a temporal difference error of the current experience tuple to calculate a composite priority evaluation standard of the current experience tuple based on the temporal difference error and the immediate reward; Calculate the composite priority of the current experience tuple based on the number of experience playbacks and the composite priority evaluation criteria of the current experience tuple; Replace the current experience tuple to the position pointed to by the life cycle pointer, and set the composite priority of the current experience tuple to the highest priority; Reset the number of experience replays at the location pointed to by the life cycle pointer to 0.

5. The method for deep reinforcement learning based on composite priority experience replay optimization according to claim 4 is characterized in that: Calculating the timing differential error includes: d i =r i +γQ′(s i+1 ,π′(s i+1 |θ π′ )+n ou )-Q(s i ,a i |θ Q ) Among them, δ i represents the time series difference error of the i-th experience, r i represents the reward value obtained, γ is the reward discount factor, θ π′ represents the current policy network π′(s t ,θ) parameter,θ Q Represents the current value network Q(s t ,a t ,θ) parameter, π′(s i+1 |θ μ′ ) represents the target policy network π′(s t ,θ) in s i+1 The action generated by ou represents OU random noise, Q(s i ,a i |θ Q ) represents the Q value calculated by the current value network, Q′(s i+1 ,π′(s i+1 |θ μ′ )+n ou ) is the expected Q value calculated by the target value network.

6. The method for deep reinforcement learning based on composite priority experience replay optimization according to claim 4, characterized in that: The calculation of the composite priority evaluation criteria includes: C i =||d i |+λ*r i | Among them, Γ i represents the composite priority evaluation of the i-th experience, |δ i | is the time series difference error of the i-th experience, r i represents the reward value obtained, and λ is a parameter used to adjust the weight of the immediate reward in the composite priority evaluation criterion.

7. The method for deep reinforcement learning based on composite priority experience replay optimization according to claim 4, characterized in that: Calculating the composite priority includes: p i =μ C(i) C i +∈ Among them, p i represents the priority of the i-th experience, Γ i represents the composite priority evaluation criteria of the i-th experience, C(i) represents the number of playbacks of the i-th experience, μ represents the playback discount factor, and ∈ is a very small normal number.

8. The method for deep reinforcement learning based on composite priority experience replay optimization according to claim 1, characterized in that: Sampling from the experience playback area to update the network parameters of the current training network, and updating the network parameters to the target network includes: Determining a sampling probability of experience according to the composite priority; Sampling from the experience playback area based on the sampling probability, and updating a playback count array; Update the network parameters of the current policy network and the network parameters of the current value network based on the sampled experience group; Updating the composite priority and the composite priority evaluation of the sampled experience group in the experience playback area; The network parameters of the current value network are updated to the target value network, and the network parameters of the current policy network are updated to the target policy network.

9. The method for deep reinforcement learning based on composite priority experience replay optimization according to claim 8, characterized in that: Determining the sampling probability of experience according to the composite priority includes: Among them, P i represents the sampling probability of the i-th experience, and α represents the priority adjustment coefficient.

10. A system based on compound priority experience replay optimization deep reinforcement learning, characterized in that: The system comprises: An acquisition module, used to acquire a current training network and a target network; the current training network includes a current policy network and a current value network, and the target network includes a target policy network and a target value network; An experience accumulation module, used for acquiring experience tuples based on the action selection strategy of the current strategy network; An updating module, used for calculating the composite priority of the experience tuple, so that the life cycle pointer updates the experience playback area according to the experience tuple and the composite priority of the experience tuple; The sampling training module is used to sample from the experience playback area to update the network parameters of the current training network, and update the network parameters of the current training network to the target network.

Images