Reinforcement learning network training method and device based on multi-pool prioritized experience replay

Abstract

The application discloses a kind of based on the reinforcement learning network training method and device of multi-pool priority experience playback, constructs the multi-pool framework based on experience playback pool and shared pool;Sample in the experience playback pool corresponding to each agent is sampled into shared cache area according to sampling probability, sample in shared cache area is placed into shared pool, and sample extracted from shared pool by each agent is stored in respective cache area;K-means algorithm is used in shared pool to cluster, and clustering result is obtained to clean sample in shared pool;Reinforcement learning network is trained in stages, and agent extracts sample from its corresponding experience playback pool or cache area to learn in different stages, the TD error of sample in experience playback pool and shared pool is updated, and sample in shared pool is cleaned according to updated TD error;Repeat the above several steps to make agent better explore optimal strategy in asynchronous environment.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of reinforcement learning training, and in particular to a reinforcement learning network training method and device based on multi-pool prioritized experience replay. BACKGROUND

[0002] Reinforcement learning (RL) is a technique in which an agent explores its optimal strategy in an environment in a trial-and-error manner, and the goal is to maximize the cumulative reward of a game. As one of the key technologies in the field of artificial intelligence, the reinforcement learning algorithm has potential applicability in many engineering problems and can be widely applied in fields such as autonomous vehicles, computer games, robotics, etc.

[0003] In the early training process of the reinforcement learning network, the agent uses the experience obtained from a single interaction with the environment to update its model, which means that this interaction information will be discarded after one update, thereby causing two problems: (1) the obtained experience has a strong time correlation, which is not conducive to the training of the model; (2) the quickly discarded interaction information can be rare experience that is useful later. Therefore, it is particularly important to make full use of historical experience for model convergence. The experience replay mechanism can well solve these two problems, which stores the experience obtained from the interaction of the agent with the environment in the experience replay pool, and the agent can update the model according to these mixed experience stored previously and obtained recently. On the one hand, this mechanism solves the problem of time correlation between training data; on the other hand, some important experience is also used to update the model multiple times, which effectively improves the learning efficiency of the agent.

[0004] Many recent works have proved the necessity of introducing experience replay in the training process of the reinforcement learning network. The DQN algorithm stabilizes the training of the value function represented by the deep neural network by using the experience replay mechanism. The DDPG algorithm also adopts the experience replay mechanism, i.e. by sampling state transition samples from the environment through the exploration strategy, and storing the samples into the experience replay pool, and sampling a small batch of samples from the experience replay pool each time for updating, which can reduce the number of interactions between the agent and the environment.

[0005] Although the above reinforcement learning networks can effectively explore the optimal policy of the environment, they all use uniform sampling, i.e., without considering the importance of the samples, so that the probability of sampling the samples is the same. Moreover, in the traditional reinforcement learning method, the priority experience replay effectively accelerates the training speed, but in the asynchronous reinforcement learning environment, the priority experience replay needs to be studied. Intuitively, there are two ways to apply the traditional non-asynchronous priority experience replay mechanism to asynchronous reinforcement learning: (1) each agent has its own experience replay pool, and the priority experience replay mechanism is used in the respective experience replay pool; (2) each agent uses the same shared experience replay pool, and the priority experience replay mechanism is used in the shared experience replay pool. The first way makes the agents unable to share the excellent experience learned by each other, resulting in slow training speed; and the second way makes each agent need to frequently interact with the data between processes. Therefore, the traditional priority experience replay is not suitable for use in asynchronous reinforcement learning. SUMMARY

[0006] In view of the above technical problems, the embodiments of the present application aim to provide a reinforcement learning network training method and device based on multi-pool priority experience replay, to solve the technical problems mentioned in the background section, to realize the priority experience replay mechanism in the asynchronous environment, and to enable the agents to better explore the optimal policy in the environment.

[0007] In a first aspect, the present application provides a reinforcement learning network training method based on multi-pool priority experience replay, comprising the following steps:

[0008] S1, constructing a multi-pool framework, the multi-pool framework comprising a shared pool and an experience replay pool corresponding to each agent, the samples in the experience replay pool comprising an identifier of the environment entering a terminal state and a TD error, constructing a cache area for each agent and a shared cache area;

[0009] S2, sampling the samples in the experience replay pool corresponding to each agent into the shared cache area according to the sampling probability, putting the samples in the shared cache area into the shared pool, and storing the samples drawn from the shared pool by each agent in the respective cache area;

[0010] S3, clustering in the shared pool using the K-means algorithm to obtain a clustering result, and cleaning the samples in the shared pool according to the clustering result;

[0011] S4, training the reinforcement learning network in stages, the agents drawing samples from the experience replay pool or the cache area corresponding thereto for learning in different stages, updating the TD error of the samples in the experience replay pool and the shared pool, and cleaning the samples in the shared pool according to the updated TD error;

[0012] S5, repeating cross-executing steps S2, S3 and S4.

[0013] As preferred, step S4 specifically comprises:

[0014] The training of the reinforcement learning network is divided into a first phase and a second phase:

[0015] In the first phase, each agent learns from the samples drawn from the respective corresponding experience replay pool, recalculates the TD error of the drawn samples and updates them;

[0016] In the second phase, each agent learns from the samples drawn from the respective corresponding buffer, recalculates the TD error of all samples in the shared pool and updates them, and cleans up the samples in the shared pool according to the updated TD error.

[0017] As preferred, cleaning up the samples in the shared pool according to the updated TD error specifically comprises:

[0018] The samples are sorted according to the updated TD error from small to large, and the samples corresponding to the minimum value of the updated TD error are deleted starting from the number of the first cleaning ratio.

[0019] As preferred, in each time step, each agent in the reinforcement learning network interacts with the respective corresponding environment, and the generated samples are put into the experience replay pool. The samples in the experience replay pool are defined as <s t ,a t ,s t+1 ,r t ,done, TD-error>, wherein s t and a t represent the state and action at time t, s t+1 represents the state at time t+1, r t represents the reward at time t, done represents the identification of the environment entering the termination state, and TD-error represents the TD error.

[0020] As preferred, the TD error is calculated as follows:

[0021] δ t =r t +γQ(s t+1 ,a t+1 )-Q(s t ,a t );

[0022] wherein a t+1 represents the action at time t+1, γ represents the discount factor, and Q represents the state value function.

[0023] As preferred, the sampling probability is:

[0024]

[0025] Wherein, p i =|δ t +ε| is the priority of sample i according to the TD error ranking, ε is a non-zero positive value, and α represents the weight value for determining the priority sampling, and when α=0, the uniform sampling mode is used.

[0026] As preferred, step S3 specifically comprises:

[0027] The state s t and the action a t of each sample in the shared pool are spliced to obtain a spliced sample, and the spliced sample forms a spliced sample point set.

[0028] The K-means algorithm is used to cluster the spliced samples in the spliced sample point set to obtain k clusters.

[0029] The spliced samples in each cluster are cleaned according to a second cleaning ratio.

[0030] In a second aspect, the present application provides a reinforcement learning network training device based on multi-pool priority experience replay, comprising:

[0031] A framework construction module is configured to construct a multi-pool framework, the multi-pool framework comprising a shared pool and an experience replay pool corresponding to each agent, the samples in the experience replay pool comprising an identifier of the environment entering a terminal state and a TD error, a cache area is constructed for each agent, and a shared cache area is constructed;

[0032] A sampling module is configured to sample the samples in the experience replay pool corresponding to each agent into the shared cache area according to a sampling probability, put the samples in the shared cache area into the shared pool, and store the samples drawn from the shared pool by each agent in the respective cache area;

[0033] A first cleaning module is configured to cluster the samples in the shared pool using the K-means algorithm to obtain a clustering result, and clean the samples in the shared pool according to the clustering result;

[0034] A second cleaning module is configured to train the reinforcement learning network in stages, and in different stages, the agents draw samples from the experience replay pool or the cache area corresponding thereto for learning, update the TD errors of the samples in the experience replay pool and the shared pool, and clean the samples in the shared pool according to the updated TD errors;

[0035] A repeating module is configured to repeatedly cross-execute the sampling module, the first cleaning module, and the second cleaning module.

[0036] In a third aspect, the present application provides an electronic device, comprising one or more processors; a storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any implementation manner of the first aspect.

[0037] In a fourth aspect, the present application provides a computer readable storage medium, having stored thereon a computer program, when the computer program is executed by a processor, the method as described in any implementation manner of the first aspect is implemented.

[0038] Compared with the prior art, the present application has the following beneficial effects:

[0039] (1) The reinforcement learning network training method based on multi-pool priority experience replay provided by the present application adopts a multi-pool framework in an asynchronous environment, and can solve the problem of data communication between different agents.

[0040] (2) The reinforcement learning network training method based on multi-pool priority experience replay provided by the present application adopts an effective self-cleaning mechanism and a self-updating mechanism, and solves the problems of lack of diversity of data and TD error lag in priority experience replay.

[0041] (3) The reinforcement learning network training method based on multi-pool priority experience replay provided by the present application has a cache area corresponding to each agent, so as to store the samples extracted from the shared pool into the cache area, each agent extracts samples from the corresponding experience replay pool or the cache area in different training stages for learning, and a shared cache area is constructed to store the samples extracted from the experience replay pool, so as to form an efficient data interaction structure, and greatly reduce the cost of process communication. BRIEF DESCRIPTION OF DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0043] Figure 1 is an exemplary device architecture diagram to which an embodiment of the present application can be applied;

[0044] Figure 2 is a flowchart of the reinforcement learning network training method based on multi-pool priority experience replay of the embodiment of the present application;

[0045] Figure 3 is a network framework diagram of the reinforcement learning network training method based on multi-pool priority experience replay of the embodiment of the present application;

[0046] Figure 4 Structure diagram of multiple buffer areas of the embodiment of the application's reinforcement learning network training method based on multi-pool prioritized experience replay;

[0047] Figure 5 Learning curve diagram of the comparative algorithm in 1 million time steps;

[0048] Figure 6 Comparison result diagram of effectiveness of the self-cleaning mechanism of the embodiment of the application's reinforcement learning network training method based on multi-pool prioritized experience replay;

[0049] Figure 7 Comparison result diagram of effectiveness of the self-updating mechanism of the embodiment of the application's reinforcement learning network training method based on multi-pool prioritized experience replay;

[0050] Figure 8 Schematic diagram of the embodiment of the application's reinforcement learning network training device based on multi-pool prioritized experience replay;

[0051] Figure 9 Structure schematic diagram of a computer device suitable for implementing the electronic device of the embodiment of the application. DETAILED DESCRIPTION

[0052] In order to make the objects, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0053] Figure 1 An exemplary device architecture 100 to which the reinforcement learning network training method based on multi-pool prioritized experience replay or the reinforcement learning network training device based on multi-pool prioritized experience replay of the embodiments of the present application can be applied is shown.

[0054] As shown in Figure 1 , the device architecture 100 can include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a communication link medium between the terminal devices 101, 102, 103 and the server 105. The network 104 can include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0055] The user can use the terminal devices 101, 102, and 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various applications can be installed on the terminal devices 101, 102, and 103, such as data processing applications, file processing applications, etc.

[0056] The terminal devices 101, 102, and 103 can be hardware or software. When the terminal devices 101, 102, and 103 are hardware, they can be various electronic devices, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, etc. When the terminal devices 101, 102, and 103 are software, they can be installed in the above-mentioned electronic devices. They can be implemented as multiple software or software modules (for example, software or software modules used to provide distributed services) or as a single software or software module. No specific limitation is made herein.

[0057] The server 105 can be a server that provides various services, such as a background data processing server that processes files or data uploaded by the terminal devices 101, 102, and 103. The background data processing server can process the obtained files or data to generate processing results.

[0058] It should be noted that the method for training a reinforcement learning network based on multi-pool prioritized experience replay provided by the embodiments of the present application can be executed by the server 105 or the terminal devices 101, 102, and 103, and accordingly, the device for training a reinforcement learning network based on multi-pool prioritized experience replay can be arranged in the server 105 or the terminal devices 101, 102, and 103.

[0059] It should be understood that Figure 1 The number of terminal devices, networks, and servers in the above-mentioned device architecture is only illustrative. Any number of terminal devices, networks, and servers can be provided according to the needs of implementation. In the case where the data to be processed does not need to be obtained from a remote location, the above-mentioned device architecture can not include a network, but only a server or a terminal device.

[0060] Figure 2 A method for training a reinforcement learning network based on multi-pool prioritized experience replay is shown, which includes the following steps:

[0061] S1, a multi-pool framework is constructed, which includes a shared pool and an experience replay pool corresponding to each agent. The samples in the experience replay pool include an identifier of an environment entering a terminal state and a TD error. A cache area is constructed for each agent, and a shared cache area is constructed.

[0062] In a specific embodiment, at each time step, each agent in the reinforcement learning network interacts with its corresponding environment, and the generated samples are placed into the experience replay pool. Samples in the experience replay pool are defined as <s> t ,a t ,s t+1 ,r t ,done,TD-error>,where,s t and a t s represents the state and action at time t. t+1 Represents the state at time t+1, r t t represents the reward at time t, done indicates that the environment has entered the termination state, and TD-error indicates the TD error.

[0063] In a specific embodiment, the TD error is calculated using the following formula:

[0064] δ t =r t +γQ(s t+1 ,a t+1 )-Q(s t ,a t );

[0065] Among them, a t+1 Let represent the action at time t+1, γ represent the discount factor, and Q represent the state value function.

[0066] For details, please refer to Figure 3 Within the multi-pool framework, each agent corresponds to an experience replay pool, also known as a local pool. Each agent interacts and learns with its corresponding environment, storing the learned samples in its respective experience replay pool. (Reference) Figure 4 A buffer of size m2 is constructed for each agent, so each agent only needs to sample within its own buffer, avoiding resource conflicts; each agent draws from the shared pool P. g Each batch of m1 samples is extracted and placed into its respective buffer. If a buffer is full, the oldest batch of samples is replaced. Additionally, a shared buffer is constructed to store each experience replay pool P. li When samples are transmitted and are available in the shared buffer, they are stored in the shared pool P. g middle.

[0067] Specifically, to reduce the cost of inter-process communication, in the first phase of the training process, each agent A... li They all directly draw from their respective experience replay pools P li Sampling is performed during the training process. Once a predefined condition is met, each agent A... lidirectly from the buffer area (which is from the shared pool P g Sampling learning is performed on the extracted samples. The predefined condition can specifically set a hyperparameter, for example, the hyperparameter is set to 100,000 time steps, that is, the first 100,000 time steps are performed on the samples in the respective experience replay pool P li The samples in each experience replay pool P li are beneficial to training, which can be sampled by other agents A i for sampling learning, thereby accelerating the speed and effect of network convergence.

[0068] S2, samples in the experience replay pool P

[0069] In specific embodiments, the sampling probability is:

[0070]

[0071] where p i is the priority of the sample i according to the TD error, ε is a very small value, and the purpose is to prevent the probability of sampling the sample with a TD error of 0 from being 0, and α represents the weight value for determining the priority sampling. When α = 0, it means that the uniform sampling method is used.

[0072] Specifically, each agent interacts with its respective environment for learning, and the obtained samples are stored in the respective experience replay pool P li Each agent corresponds to an experience replay pool P li , and samples in the experience replay pool P g are sampled into the shared buffer area according to a certain probability. A self-cleaning mechanism for the shared pool P g and a self-updating mechanism for the shared pool P li and the experience replay pool P g of each agent are started. The samples in the shared buffer area are placed in the shared pool P t Each agent periodically extracts samples from the shared pool and stores them in the respective buffer area. Each agent extracts learning samples from the respective buffer area for training to explore the optimal strategy.

[0073] S3, the K-means algorithm is used in the shared pool to perform clustering, and the clustering result is obtained. The samples in the shared pool are cleaned according to the clustering result.

[0074] In specific embodiments, step S3 specifically includes:

[0075] The state s of each sample in the shared pool is t and action a t are spliced to obtain spliced samples and constitute a spliced sample point set.

[0076] The K-means algorithm is used to cluster the spliced samples in the spliced sample point set to obtain k clusters.

[0077] The spliced samples in each cluster are cleaned according to a second cleaning ratio.

[0078] Specifically, for the self-cleaning mechanism of the shared pool P g , on the one hand, the K-means algorithm is used, and on the other hand, the TD error is used. The K-means algorithm is used to cluster the spliced samples in the spliced sample point set to obtain k clusters, and then each cluster is cleaned according to a certain ratio.

[0079] S4, the reinforcement learning network is trained in stages, and in different stages, the agent extracts samples from the corresponding experience replay pool or cache area for learning, updates the TD error of the samples in the experience replay pool and the shared pool, and cleans the samples in the shared pool according to the updated TD error.

[0080] In specific embodiments, step S4 specifically includes:

[0081] The training of the reinforcement learning network is divided into a first stage and a second stage:

[0082] In the first stage, each agent extracts samples from the corresponding experience replay pool for learning, recalculates the TD error of the extracted samples and updates them;

[0083] In the second stage, each agent extracts samples from the corresponding cache area for learning, recalculates the TD error of all samples in the shared pool and updates them, and cleans the samples in the shared pool according to the updated TD error.

[0084] In specific embodiments, the samples in the shared pool are cleaned according to the updated TD error, specifically including:

[0085] The samples are sorted according to the updated TD error from small to large, and the samples corresponding to the minimum value of the updated TD error are deleted starting from the number of the first cleaning ratio.

[0086] Specifically, through phased training, samples are extracted from the experience replay pool or buffer area corresponding to the agent for learning. Since the buffer area stores samples extracted from the shared pool, the process communication time can be reduced. In the first phase of training, each agent extracts samples for a mini-batch from the respective experience replay pool P li li To avoid repeated scanning of the entire experience replay pool P li , only the sampled samples are updated for TD error. Specifically, TD error refers to the difference between the estimated value of the state-action value function and the actual value. The larger the TD error, the greater the gap between the estimated value of the state value function and the actual value. Therefore, the sample needs to be learned. In addition, the TD error of a sample will gradually change with the update of the network.

[0087] In order to enable each agent A li to extract high-information learning samples in the second phase of training, the TD error of all samples in the shared pool P g is updated, so that some samples with large TD error but small TD error after updating can be removed, thereby ensuring that the samples beneficial to the current time step training can be saved in the shared pool P g with limited capacity.

[0088] S5, repeating the cross-execution of steps S2, S3 and S4.

[0089] Specifically, the two cleaning steps in steps S3 and S4 above belong to the self-cleaning mechanism and can be repeatedly crossed. The sampling step of step S2 and the cleaning step of the self-cleaning mechanism are also repeatedly crossed, forming a cyclic training process to ultimately explore the optimal strategy.

[0090] The following simulation experiment is used to illustrate the present scheme.

[0091] The effectiveness of the reinforcement learning network training method based on multi-pool priority experience replay proposed in the embodiments of the present application is verified by comparing it with six mainstream reinforcement learning algorithms: TD3, DDPG, PPO, VPG, A3C, and SAC. Seven different environments in MuJoCo are selected for testing.

[0092] The parameter settings involved in the experiment are as follows: the experience replay pool capacity is 1000000, the exploration noise is 0.25, the discount factor is 0.99, the learning rate is 0.0001, k in step S3 is 10, the size of the mini-batch in step S4 is 256, m1 in step S1 is 1000, and m2 is 2.

[0093] To measure the performance of the algorithms, each algorithm is run for 1 million time steps in the given environment. Every 5000 time steps, the algorithm is tested in the test environment. During the testing, the exploration noise is removed and each algorithm is evaluated for at least 10 experiments to obtain more accurate results.

[0094] Figure 5 The learning curves of the proposed reinforcement learning network training method based on multi-pool prioritized experience replay (MP-TD3) and other state-of-the-art deep reinforcement learning algorithms proposed by the embodiments of the present application are shown. Among them, the off-policy based algorithms such as MP-TD3, TD3, SAC and DDPG rank in the top three in 20 out of 21 rankings in 7 experiments; while the on-policy based algorithms such as PPO, VPG and A3C rank in the top three in only 3 out of 21 rankings in 7 experiments. In addition, in the first 500,000 time steps, the off-policy based algorithms rank in the top three in 19 out of 21 rankings in 7 experiments, while the on-policy based algorithms rank in the top three in only 4 out of 21 rankings in 7 experiments. This shows that the off-policy based deep reinforcement learning algorithms have more advantages in performance and convergence speed, because these algorithms are trained based on historical data. On the one hand, this mechanism can reduce the correlation between training data; on the other hand, samples with high value can be repeatedly sampled, thereby improving the learning efficiency and stability of the algorithm.

[0095] In addition, the performance of 7 algorithms, including 3 on-policy algorithms and 4 off-policy algorithms, is compared. The results show that the number of times the off-policy algorithm ranks first in all environments is 6, 1, 2, 0, respectively, while the number of times the on-policy algorithm ranks first is 1, 0, 0. If only the performance in the first 500,000 time steps is considered, the number of times the off-policy algorithm ranks first is 5, 2, 2, 0, while the on-policy algorithm still has 1, 0, 0. This shows that the off-policy algorithm has a clear advantage in reinforcement learning tasks.

[0096] Among the four off-policy algorithms, MP-TD3 performs the best, achieving 6 first rankings. In contrast, SAC, TD3 and DDPG achieve 2, 1, 0 first rankings, respectively. The reason why MP-TD3 can achieve excellent results is mainly due to the fact that they adopt the double Q truncation mechanism and the delayed update mechanism to alleviate the overestimation problem and the policy degradation problem. While MP-TD3 has better performance than TD3, it is mainly due to the fact that it adopts the asynchronous update mechanism and the multi-pool prioritized experience replay mechanism.

[0097] It is noted that in the Reacher environment (see Figure 5 (f)), the final cumulative reward values obtained by the other five algorithms are very close, and there is no significant difference, except for the VPG and A3C algorithms. This may be because the Reacher environment is relatively simple, and most algorithms can quickly find the optimal strategy.

[0098] In order to explore the effectiveness of the proposed algorithm, the effectiveness of the self-cleaning mechanism and the self-updating mechanism is further explored respectively, from Figure 6 It can be seen from that in most environments, the effect of MP-TD3 is obviously improved compared with MP-TD3 without clean, which may be because the proposed self-cleaning mechanism can make a large number of redundant samples in the shared pool be cleaned. Compared with the traditional TD3 algorithm, the samples obtained by MP-TD3 from the experience replay pool are more diverse, which helps the deep learning model to converge faster. From Figure 7 It can be seen from that in most environments, the effect of MP-TD3 is obviously improved compared with MP-TD3 without self-cleaning mechanism, which may be because the self-updating mechanism makes the samples in the shared pool can keep the latest TD error at all times, and then helps the agent to select the sample with the largest global TD error for training when selecting training samples, effectively accelerating the convergence speed of the deep reinforcement learning model.

[0099] In the asynchronous environment, the reinforcement learning network training method based on multi-pool priority experience replay of the embodiment of the application proposes a multi-pool framework, so that the problem of data communication between different agents is solved; then, an effective self-cleaning mechanism and a self-updating mechanism are proposed, which solve the problems of lack of diversity of data and lag of TD error in priority experience replay; in addition, an efficient data interaction structure is proposed, which greatly reduces the cost of process communication.

[0100] Further referring to Figure 8 , as an implementation of the method shown in the above figures, the application provides an embodiment of a reinforcement learning network training device based on multi-pool priority experience replay. The device embodiment corresponds to the method embodiment shown in Figure 2 , and the device can be applied to various electronic devices.

[0101] The embodiment of the application provides a reinforcement learning network training device based on multi-pool priority experience replay, which comprises:

[0102] The framework construction module 1 is configured to construct a multi-pool framework, the multi-pool framework comprising a shared pool and an experience replay pool corresponding to each agent, the samples in the experience replay pool comprising an identifier of the environment entering a terminal state and a TD error, constructing a buffer for each agent and constructing a shared buffer;

[0103] The sampling module 2 is configured to sample the samples in the experience replay pool corresponding to each agent into the shared buffer according to a sampling probability, put the samples in the shared buffer into the shared pool, and store the samples drawn from the shared pool by each agent in the buffer corresponding to the agent;

[0104] The first cleaning module 3 is configured to perform clustering on the samples in the shared pool by using a K-means algorithm to obtain a clustering result, and clean the samples in the shared pool according to the clustering result;

[0105] The second cleaning module 4 is configured to train the reinforcement learning network in stages, and in different stages, the agents draw samples from the experience replay pool or the buffer corresponding to the agent for learning, update the TD errors of the samples in the experience replay pool and the shared pool, and clean the samples in the shared pool according to the updated TD errors;

[0106] The repeating module 5 is configured to repeatedly cross-perform the sampling module, the first cleaning module and the second cleaning module.

[0107] Reference will be made to Figure 9 , which shows a structural schematic diagram of a computer device 900 of an electronic device (for example, a server or a terminal device) suitable for implementing the embodiments of the present application. Figure 1 The electronic device shown is only an example, and should not bring any limitation to the functions and use range of the embodiments of the present application. Figure 9 The electronic device shown is only an example, and should not bring any limitation to the functions and use range of the embodiments of the present application.

[0108] As shown in Figure 9 , the computer device 900 comprises a central processing unit (CPU) 901 and a graphics processor (GPU) 902, which can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 903 or programs loaded from a storage portion 909 into a random access memory (RAM) 904. In the RAM 904, various programs and data required for the operation of the device 900 are also stored. The CPU 901, the GPU 902, the ROM 903 and the RAM 904 are connected to each other through a bus 905. An input / output (I / O) interface 906 is also connected to the bus 905.

[0109] The following components are connected to the I / O interface 906: an input part 907 including a keyboard, a mouse, etc.; an output part 908 including a display such as a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage part 909 including a hard disk, etc.; and a communication part 910 including a network interface card such as a LAN card, a modem, etc. The communication part 910 performs communication processing via a network such as the Internet. A drive 911 can also be connected to the I / O interface 906 as necessary. A removable media 912 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc. is attached to the drive 911 as necessary, so that a computer program read out therefrom is installed in the storage part 909 as necessary.

[0110] In particular, the processes described above with reference to the flowcharts can be implemented as a computer software program according to embodiments of the present disclosure. For example, embodiments of the present disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for executing the methods illustrated by the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network by the communication part 910, and / or installed from the removable media 912. When the computer program is executed by the central processing unit (CPU) 901 and the graphics processor (GPU) 902, the above-described functions defined in the methods of the present application are performed.

[0111] Note that the computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium or any combination thereof. The computer-readable medium can be, for example but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a computer-readable medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present context, a computer-readable medium can be any tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device. In the present context, a computer-readable signal medium can include a computer-readable program code in a baseband or propagated as carrier waves in a propagated data signal associating with a carrier wave. Such a propagated signal can take a wide variety of forms including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer-readable signal medium can also be any computer-readable medium that can be used to carry or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. The program contained in the computer-readable medium can be transmitted in any suitable format including, but not limited to, wireless, wireline, optical fiber cable, RF, or any suitable combination thereof.

[0112] Computer program code for carrying out operations of the present application can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0113] The computer readable medium described above can carry one or more programs which, when executed by the electronic device, cause the electronic device to: construct a multi-pool framework, the multi-pool framework comprising a shared pool and an experience replay pool corresponding to each agent, samples in the experience replay pool comprising an identifier of an environment entering a terminal state and a TD error, construct a cache area for each agent and construct a shared cache area; sample samples in the experience replay pool corresponding to each agent into the shared cache area according to a sampling probability, place samples in the shared cache area into the shared pool, and store samples drawn from the shared pool by each agent in the cache area corresponding to the agent; perform clustering on samples in the shared pool by using a K-means algorithm to obtain a clustering result, and clean samples in the shared pool according to the clustering result; train the reinforcement learning network in stages, and in different stages, agents learn from samples in the experience replay pool or the cache area corresponding to the agent, update the TD error of samples in the experience replay pool and the shared pool, and clean samples in the shared pool according to the updated TD error; repeatedly perform the above two steps to clean samples in the shared pool, and then repeatedly perform the above three steps.

[0114] The modules involved in the embodiments of the present application can be implemented in a software manner or in a hardware manner. The described modules can also be arranged in a processor.

[0115] As another aspect, the present application also provides a computer readable medium, which can be included in the electronic device described in the above embodiments, or can exist independently without being assembled into the electronic device. The computer readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to: construct a multi-pool framework, the multi-pool framework comprising a shared pool and an experience replay pool corresponding to each agent, samples in the experience replay pool comprising an identifier of an environment entering a terminal state and a TD error, construct a cache area for each agent and construct a shared cache area; sample samples in the experience replay pool corresponding to each agent into the shared cache area according to a sampling probability, place samples in the shared cache area into the shared pool, and store samples drawn from the shared pool by each agent in the cache area corresponding to the agent; perform clustering on samples in the shared pool by using a K-means algorithm to obtain a clustering result, and clean samples in the shared pool according to the clustering result; train the reinforcement learning network in stages, and in different stages, agents learn from samples in the experience replay pool or the cache area corresponding to the agent, update the TD error of samples in the experience replay pool and the shared pool, and clean samples in the shared pool according to the updated TD error; repeatedly perform the above two steps to clean samples in the shared pool, and then repeatedly perform the above three steps.

[0116] The above description is only the preferred embodiment of the present application and the explanation of the technical principles. It should be understood by those skilled in the art that the scope of the protection of the present application is not limited to the technical solutions formed by the specific combinations of the above technical features. It should also cover other technical solutions formed by the combinations of the above technical features or their equivalent features without departing from the concept of the present application. For example, the technical solutions formed by the mutual replacement of the above features and the technical features with similar functions disclosed (but not limited to) in the present application.

Claims

1. A reinforcement learning network training method based on multi-pool prioritized experience replay, characterized in that, The method comprises the following steps: S1, constructing a multi-pool framework, wherein the multi-pool framework comprises a shared pool and an experience replay pool corresponding to each agent, samples in the experience replay pool comprise an identifier of an environment entering a terminal state and a TD error, a buffer area is constructed for each agent, and a shared buffer area is constructed; S2, sampling samples in the experience replay pool corresponding to each agent into the shared buffer area according to a sampling probability, putting samples in the shared buffer area into the shared pool, and storing samples extracted from the shared pool by each agent in the buffer area of the agent; S3, performing clustering on samples in the shared pool by using a K-means algorithm to obtain a clustering result, and cleaning samples in the shared pool according to the clustering result; S4, training a reinforcement learning network in stages, wherein in different stages, the agent extracts samples from the experience replay pool or the buffer area corresponding to the agent for learning, the TD error of samples in the experience replay pool and the shared pool is updated, and samples in the shared pool are cleaned according to the updated TD error; The step S4 specifically comprises: dividing the training of the reinforcement learning network into a first stage and a second stage; In the first stage, each of the agents learns from the samples drawn from the respective experience replay pool, recalculates and updates the TD error of the drawn samples, and in each time step, each agent in the reinforcement learning network interacts with the respective corresponding environment, and the generated samples are put into the experience replay pool, and the samples in the experience replay pool are defined as <s t , a t , s t+1 , r t , done, TD-error>, wherein s t and a t represent the state and action at time t, s t+1 represents the state at time t+1, r t represents the reward at time t, done represents an identifier indicating that the environment enters a terminal state, and TD-error represents a TD error. in the second stage, each agent extracts samples from the buffer area corresponding to the agent for learning, re-calculates and updates the TD error of all samples in the shared pool, and cleans samples in the shared pool according to the updated TD error, specifically comprising: sorting samples according to the updated TD error from small to large, and deleting samples from the smallest value of the updated TD error according to the number of a first cleaning ratio; S5, repeatedly performing steps S2, S3 and S4.

2. The method of claim 1, wherein, The TD error is calculated by using the following formula: delta t = r t + gamma * Q(s t+1 , a t+1 ) - Q(s t , a t ); where a t+1 denotes the action at time t + 1, γ denotes the discount factor, and Q denotes the state value function. 3.The method of claim 2, wherein, The sampling probability is: where p i = |δ t + ε| is the priority of sample i according to the TD error ranking, ε is a non-zero positive value, and α represents the weight value for determining the priority sampling. When α = 0, it indicates that a uniform sampling method is used. 4.The method of claim 1, wherein, The step S3 specifically comprises: state s of each sample in the shared pool t and action a t stitching, obtaining a stitched sample, and constituting a stitched sample point set; performing clustering on spliced samples in the spliced sample point set by using a K-means algorithm to obtain k clusters; cleaning spliced samples in each cluster according to a second cleaning ratio.

5. A reinforcement learning network training device based on multi-pool priority experience replay, characterized in that, comprise: a framework construction module configured to construct a multi-pool framework, wherein the multi-pool framework comprises a shared pool and an experience replay pool corresponding to each agent, samples in the experience replay pool comprise an identifier of an environment entering a terminal state and a TD error, a buffer area is constructed for each agent, and a shared buffer area is constructed; a sampling module configured to sample samples in the experience replay pool corresponding to each agent into the shared buffer area according to a sampling probability, put samples in the shared buffer area into the shared pool, and store samples extracted from the shared pool by each agent in the buffer area of the agent; a first cleaning module configured to perform clustering on samples in the shared pool by using a K-means algorithm to obtain a clustering result, and clean samples in the shared pool according to the clustering result; The second cleaning module is configured to train the reinforcement learning network in stages, in different stages, the agent learns from the corresponding experience replay pool or buffer area respectively, the TD error of the samples in the experience replay pool and the shared pool is updated, and the samples in the shared pool are cleaned according to the updated TD error. The second cleaning module specifically includes: The training of the reinforcement learning network is divided into a first stage and a second stage: In the first stage, each of the agents learns from the samples drawn from the respective experience replay pool, recalculates and updates the TD error of the drawn samples, and in each time step, each agent in the reinforcement learning network interacts with the respective corresponding environment, and the generated samples are put into the experience replay pool, and the samples in the experience replay pool are defined as <s t , a t , s t+1 , r t , done, TD-error>, wherein s t and a t represent the state and action at time t, s t+1 represents the state at time t+1, r t represents the reward at time t, done represents the identification that the environment enters the terminal state, and TD-error represents the TD error. In the second stage, each agent learns from the corresponding buffer area, recalculates and updates the TD error of all samples in the shared pool, and cleans the samples in the shared pool according to the updated TD error, specifically including: The samples are sorted according to the updated TD error from small to large, and the first cleaning ratio of the number of samples is deleted from the smallest value of the updated TD error corresponding to the sample. The repeating module is configured to repeatedly cross-execute the sampling module, the first cleaning module and the second cleaning module. 6.An electronic device, comprising: one or more processors; a storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, so that the one or more processors implement the method of any one of claims 1-4.

7. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the method of any one of claims 1-4.

Images