Multi-agent reinforcement learning method for optimizing experience storage and reuse

Abstract

The present invention discloses a multi-agent reinforcement learning method for optimizing experience storage and experience reuse, comprising: constructing an experience buffer pool based on the LRU (Least Recently Used) mechanism for experience storage; designing a multi-agent experience collection method based on network weight sharing; adopting a hybrid sampling method based on experience priority to provide a higher retrieval rate for higher priority data, and adopting B + A tree data structure stores prioritized experiences. Compared to traditional methods, the proposed method significantly reduces the correlation between sample data and further enriches sample types. It also boasts a high retrieval rate for experience samples, good biological interpretability, and greater robustness to sample noise, enabling intelligent agents to perform better in complex environments and tasks.

Description

Technical Field

[0001] The present invention belongs to the field of deep reinforcement learning technology, and in particular to a multi-agent experience replay method that combines a least recently used mechanism and a priority hybrid sampling mechanism. Background Art

[0002] In recent years, artificial neural networks, inspired by the structure and information transfer mechanisms of biological neural networks, have become a hot research topic in machine learning. Deep learning methods have demonstrated excellent performance in a wide range of fields, including computer vision, intelligent recommendation, and autonomous driving. Meanwhile, reinforcement learning, another important branch of machine learning, has been widely applied to sequential decision-making problems. Deep reinforcement learning methods combine deep learning and reinforcement learning. Their key feature is that they leverage deep learning's ability to perceive and abstract high-dimensional state information to control the interaction of intelligent agents with their environment, leading to continuous trial and error.

[0003] As a weakly supervised learning method, deep reinforcement learning lacks effective human intervention and control. The agent requires extensive environmental interactions to achieve a satisfactory control strategy. In practical applications, deep reinforcement learning suffers from issues such as lack of rationality in the agent's action output, sparse environmental reward signals, and difficulty in designing reward functions. Therefore, extensive environmental interactions are impractical in real-world settings, and low sample utilization is a major bottleneck in the widespread application of deep reinforcement learning algorithms in real-world scenarios.

[0004] Experience replay is a key approach to addressing the low sample utilization rate in deep reinforcement learning algorithms. This method uses an experience buffer pool to store samples generated at each time step. Each experience sample contains the current state, the action taken in that state, the reward given by the environment, and the next state the agent transitions to after executing the action. During network training, stored samples are randomly selected from the experience buffer pool. This mechanism reuses experience sample data during neural network training through experience pooling. Furthermore, by randomly selecting experience, it overcomes the issues of correlated data and non-stationary distribution.

[0005] However, different experience samples in the experience pool contribute differently to network weight updates. Traditional storage methods based on a first-in, first-out mechanism and uniform sampling ignore the importance of different samples. All experience samples have the same probability of being sampled. Using a uniform sampling strategy can lead to difficulties in network convergence and easily lead to problems such as falling into local minima. Furthermore, frequently using outdated experience samples to update the agent's decision model may affect the agent's maximum expected reward during subsequent training, thereby increasing the agent's learning difficulty and prolonging the learning time.

[0006] In deep reinforcement learning, the importance of experience can be measured by the absolute value of the time difference error. When the absolute value of the time difference error is larger, the estimate made by the current state value network is more different from the true value, and the contribution to the weight of the back propagation of the neural network is also greater; when the time difference error is smaller, it means that the sample has little influence on the calculation of the reverse gradient. The current priority experience replay method based on the greedy strategy has the following disadvantages: (1) If the time difference error of a certain sample is not high, it only means that the amount of information it provides for feedback during training is low, but it does not mean that it has no contribution to the update of the weight. If a greedy algorithm is used, some important information contained in the sample may be missed. (2) Using a greedy algorithm will cause the strategy to only select samples with large time difference errors for repeated training, which will cause overfitting of the results. (3) The order of magnitude of the experience pool capacity in the deep reinforcement learning algorithm is generally set to be large. When using a linear storage method, when extracting experience, it is necessary to search by comparing one by one, and the time complexity of the search algorithm is high.

[0007] Therefore, it is necessary to improve the experience storage and sampling methods in deep reinforcement learning algorithms, which can further improve the training efficiency and versatility of deep reinforcement learning algorithms. Summary of the Invention

[0008] The purpose of the present invention is to address the problems existing in the above-mentioned prior art and provide a deep reinforcement learning method that optimizes the experience replay storage method and sampling strategy. This method can significantly improve the convergence speed and stability of the deep reinforcement learning algorithm.

[0009] The technical solution to achieve the purpose of the present invention is: a multi-agent reinforcement learning method for optimizing experience storage and experience reuse, the method comprising the following steps:

[0010] Step 1: Initialize all parameters of the current Q network ω, all parameters of the target Q network ω′=ω, and the capacity N of the experience pool, set the number of samples m for batch gradient descent, and the parameter update time step T of the target Q network;

[0011] Step 2: Conduct multi-agent experience collection based on multi-threading: By sharing network weights, use multi-threading technology to deploy multiple agents to interact with the environment simultaneously. At each time point, agents in different threads adopt different exploration strategies based on probability to obtain experience samples from the environment.

[0012] Step 3: Calculate the absolute value of the time difference error (TD error) of the experience sample and insert the experience into the global shared experience pool with TD error as the index. At the same time, update the time attribute values ​​of all experiences in the global shared experience pool based on the least recently used mechanism.

[0013] Step 4: When updating the current training strategy, m experience samples are sampled from the global shared experience pool using a hybrid sampling method, and the time attributes of the experience in the experience pool are updated according to the least recently used mechanism; where m is the number of samples for setting batch gradient descent;

[0014] Step 5: Use the m experience samples obtained by sampling to train the current Q network. After the training is completed, recalculate the TD error of all experience samples in the experience pool, update the priority of the experience in the global shared experience pool, and determine whether the number of training steps reaches the preset maximum value. If not, return to step 3, otherwise end the process.

[0015] A multi-agent reinforcement learning system for optimizing experience storage and experience reuse, the system comprising sequentially executing:

[0016] Initialization module, used to initialize all parameters of the current Q network ω, all parameters of the target Q network ω′ = ω, and the capacity N of the experience pool, set the number of samples m for batch gradient descent, and the parameter update time step T of the target Q network;

[0017] The multi-agent experience collection module is used for multi-threaded multi-agent experience collection: through network weight sharing, multiple agents are deployed using multi-threading technology to interact with the environment simultaneously. At each time point, agents in different threads adopt different exploration strategies based on probability to obtain experience samples from the environment.

[0018] The experience pool construction module is used to calculate the absolute value of the time difference error (TD error) of the experience sample and insert the experience into the global shared experience pool with TD error as the index. At the same time, the time attribute values ​​of all experiences in the global shared experience pool are updated based on the least recently used mechanism.

[0019] The update module is used to sample m experience samples from the global shared experience pool using a hybrid sampling method when updating the current training strategy, and update the time attributes of the experience in the experience pool according to the least recently used mechanism; where m is the number of samples used to set the batch gradient descent;

[0020] The training module is used to train the current Q network using the m experience samples obtained by sampling. After the training is completed, the TD error of all experience samples in the experience pool is recalculated, the priority of the experience in the global shared experience pool is updated, and it is determined whether the number of training steps has reached the preset maximum value. If not, it returns to the experience pool construction module.

[0021] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the following steps when executing the computer program:

[0022] Step 1: Initialize all parameters of the current Q network ω, all parameters of the target Q network ω′=ω, and the capacity N of the experience pool, set the number of samples m for batch gradient descent, and the parameter update time step T of the target Q network;

[0023] Step 2: Conduct multi-agent experience collection based on multi-threading: By sharing network weights, use multi-threading technology to deploy multiple agents to interact with the environment simultaneously. At each time point, agents in different threads adopt different exploration strategies based on probability to obtain experience samples from the environment.

[0024] Step 3: Calculate the absolute value of the time difference error (TD error) of the experience sample and insert the experience into the global shared experience pool with TD error as the index. At the same time, update the time attribute values ​​of all experiences in the global shared experience pool based on the least recently used mechanism.

[0025] Step 4: When updating the current training strategy, m experience samples are sampled from the global shared experience pool using a hybrid sampling method, and the time attributes of the experience in the experience pool are updated according to the least recently used mechanism; where m is the number of samples for setting batch gradient descent;

[0026] Step 5: Use the m experience samples obtained by sampling to train the current Q network. After the training is completed, recalculate the TD error of all experience samples in the experience pool, update the priority of the experience in the global shared experience pool, and determine whether the number of training steps reaches the preset maximum value. If not, return to step 3, otherwise end the process.

[0027] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the following steps:

[0028] Step 1: Initialize all parameters of the current Q network ω, all parameters of the target Q network ω′=ω, and the capacity N of the experience pool, set the number of samples m for batch gradient descent, and the parameter update time step T of the target Q network;

[0029] Step 2: Conduct multi-agent experience collection based on multi-threading: By sharing network weights, use multi-threading technology to deploy multiple agents to interact with the environment simultaneously. At each time point, agents in different threads adopt different exploration strategies based on probability to obtain experience samples from the environment.

[0030] Step 3: Calculate the absolute value of the time difference error (TD error) of the experience sample and insert the experience into the global shared experience pool with TD error as the index. At the same time, update the time attribute values ​​of all experiences in the global shared experience pool based on the least recently used mechanism.

[0031] Step 4: When updating the current training strategy, m experience samples are sampled from the global shared experience pool using a hybrid sampling method, and the time attributes of the experience in the experience pool are updated according to the least recently used mechanism; where m is the number of samples for setting batch gradient descent;

[0032] Step 5: Use the m experience samples obtained by sampling to train the current Q network. After the training is completed, recalculate the TD error of all experience samples in the experience pool, update the priority of the experience in the global shared experience pool, and determine whether the number of training steps reaches the preset maximum value. If not, return to step 3, otherwise end the process.

[0033] Compared with the existing technology, the present invention has the following significant advantages: 1) Compared with the first-in-first-out mechanism, the least recently used mechanism can increase the retention time of important experience samples in the experience pool, thereby accelerating network training; 2) The priority mixed sampling method can avoid the defect of low learning efficiency under the uniform sampling method and the overfitting problem caused by the priority sampling method based on the greedy strategy; 3) The multi-agent experience collection method updates parameters through a parallel calculation method, which improves the diversity of experience samples and accelerates the convergence speed of the neural network; 4) By using the B+ tree data structure to organize the storage of experience samples, the linear time complexity of searching for experience samples can be reduced to logarithmic time complexity, which fully accelerates the retrieval speed of a specific experience sample in a large-capacity experience pool.

[0034] The present invention is further described in detail below with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1A schematic diagram of a reinforcement learning method for optimizing an experience replay sampling strategy in one embodiment.

[0036] Figure 2 A schematic diagram of a multi-agent experience collection method based on multi-threading in one embodiment.

[0037] Figure 3 Schematic diagram of a hybrid sampling method based on empirical sample priority in one embodiment. DETAILED DESCRIPTION

[0038] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present application is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are merely used to explain the present application and are not intended to limit the present application. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0039] In one embodiment, the present invention provides a multi-agent experience replay method that combines the least recently used mechanism and the priority hybrid sampling mechanism. Experience replay is an important optimization strategy in deep reinforcement learning algorithms. The present invention can improve the convergence speed and stability of reinforcement learning algorithms, thereby improving the overall performance of reinforcement learning tasks. Figure 1 As shown, the method of the present invention comprises the following steps:

[0040] Step 1: Initialize all parameters of the current Q network ω, initialize all parameters of the target Q network ω′=ω, initialize the capacity N of the experience pool, set the number of samples m for batch gradient descent, and the parameter update time step T of the target network;

[0041] Step 2: By sharing network weights, we deploy multiple agents using multi-threading technology to interact with the environment simultaneously. At each time point, agents in different threads adopt different exploration strategies based on probability to obtain experience samples from the environment.

[0042] Step 3: Calculate the absolute value of the TD error (temporal differential error) of the experience sample and insert the experience into the global shared experience pool with TD error as the index. At the same time, update the time attribute values ​​of all experiences in the experience pool based on the least recently used mechanism.

[0043] Step 4: When updating the current training strategy, m experience samples are sampled from the globally shared experience replay pool using a hybrid sampling method, and the time attributes of the experience in the experience pool are updated according to the least recently used mechanism; where m is the number of samples used to set the batch gradient descent.

[0044] Step 5: Use the sampled m experience samples to train the current Q network. After the training is completed, recalculate the TD error of all experience samples in the experience pool, update the priority of the experience in the global shared experience pool, and determine whether the number of training steps reaches the preset maximum value. If not, return to step 3.

[0045] In the embodiment of the present invention, the experience obtained by the agent in the interaction with the environment at time point t is a five-tuple {s t ,a t ,r t ,s t+1 ,is_end}, where s t is the state of the environment where the Agent is located at time t, and is also the input of the current Q network. t Represents the action performed by the Agent at time t. The environment transfers to the state s at the next moment according to the Agent's state and action t+1 And give the environmental reward r obtained by this action t s, is_end indicates whether the current state is already the terminal state.

[0046] Furthermore, in one embodiment, in combination Figure 2 The multi-agent experience collection method based on multi-threading described in step 2 has the following specific process:

[0047] First, deploy a public neural network model, and open n threads based on the public model. Each thread has the same network structure as the public neural network. Each thread will independently interact with the environment to obtain experience data. These threads do not interfere with each other and run independently.

[0048] Each individual sampling agent interacts with its own environment to generate experience samples, and the experience samples generated by all sampling agents are uniformly stored in a shared experience buffer pool. Each agent adopts an ε-greedy strategy to select actions. In each state, the action with the maximum state-action value function is selected with a probability of 1-ε, and a random action is selected with a probability of ε. In the same state, the action selected by each agent is uncertain, thereby obtaining different experience samples.

[0049] Furthermore, in one embodiment, step 3 calculates the absolute value of the temporal differential error (TD error) of the experience sample, inserts the experience into a global shared experience pool indexed by the TD error, and simultaneously updates the time attributes of all experiences in the experience pool based on the least recently used mechanism. The specific process includes:

[0050] Step 3-1, calculate the time difference error δ of the current time t experience sample t :

[0051] δ t =Q * (s t ,a t )-y t

[0052] Where s t is the state of the agent's environment at time t, a t represents the action performed by the agent at time t, y t represents the TD target at time t, Q * (s t ,a t ) is in state s t Take action a t , is the approximation of the maximum cumulative future reward expectation of the current Q network calculated using the Monte Carlo approximation method, as follows:

[0053]

[0054]

[0055] Where r t is the real-time reward obtained after the agent interacts with the environment at time t, γ is the discounted return rate, and s t+1 The new state that the Agent transfers to after executing the action;

[0056] Step 3-2, take the absolute value of the calculated time difference error |δ t | is used as the priority weight of the experience sample. If the time difference error cannot be calculated for the experience quintuple, its priority is set to the preset maximum value. Then, the priority of the experience sample is inserted as an index into the B + In the global shared experience pool with the tree as the storage structure, the non-leaf nodes of the tree only store the priority of the experience samples, while the leaf nodes store both the priority of the experience and the experience quintuple itself;

[0057] The time attribute of each experience in the experience pool is given by the least recently used principle. The time attribute of the experience currently inserted into the experience pool is set to 0, and the time attribute values ​​of all other experiences in the experience pool are increased by 1 at the same time. When the capacity of the experience pool has reached the upper limit, the experience with the largest time attribute value in the experience pool will be discarded.

[0058] Furthermore, in one embodiment, in combination Figure 3 , the empirical mixed sampling method described in step 4, the sample data with higher priority can obtain a higher sampling rate. The specific process is as follows:

[0059] Step 4-1: Given that the capacity of the global shared experience pool is N, a line segment of length 1 is divided into N segments, such as Figure 3 I1, I2, I3...I N As shown, each segment corresponds to an experience record in the experience pool. The length of the line segment corresponding to each experience record is different. The line segment corresponding to the high-priority experience record is long, and the line segment corresponding to the low-priority experience record is short. The line segment length of each experience record i is determined by the following formula:

[0060]

[0061] Where, δ i It represents the time difference error of the empirical record i, ε is a positive number that is infinitely close to 0 to avoid the situation where the line length is 0, and the exponent value is 3 / 4 as the empirical value.

[0062] Step 4-2: Before sampling, divide the line segment of length 1 into M equal parts, as Figure 3 S1, S2, S3...S M-1 S M As shown, here M>>N, ensuring that the line segment corresponding to each experience record is divided into corresponding small blocks. Each of the M copies falls on the line segment corresponding to a specific experience record, and longer line segments are divided into more small blocks. When sampling, m locations are uniformly sampled from the M locations. This ensures that higher-priority experience records have a higher retrieval rate.

[0063] In step 4-3, the priority of the extracted experience is used as the index value to find the corresponding experience record from the global shared experience pool, and its time attribute value is updated to 0. At the same time, the time attribute values ​​of the remaining experience samples are all increased by 1.

[0064] Furthermore, in one embodiment, in combination Figure 1 In step 5, the training of the current Q network using m experience samples adopts the variable learning rate stochastic gradient descent method. The specific process is as follows:

[0065] Step 5-1, take out the experience samples from the m experience samples in turn, and use the target Q network to calculate the gradient value g corresponding to the i-th experience sample i :

[0066]

[0067] Where ω is the parameter of the current Q network, s i ,a i are the state of the agent in the i-th experience sample and the action selected by the agent in the current state;

[0068] Step 5-2, using the gradient value g i Update the parameters of the current Q network, and the update rules are:

[0069]

[0070] Among them, α′ is the learning rate, which is a variable parameter and is defined as:

[0071] α′=α·(n·p i ) -β

[0072] In the formula, α is the initial setting value of the learning rate, which is a hyperparameter that needs to be given manually before network training. i is the sampling probability of the i-th experience sample, β∈(0,1). At the beginning of training, β is a positive real number close to 0. As the training iteration proceeds, β gradually approaches 1. After each training, the time difference error δ of the experience sample is calculated using the Q network after parameter update. i Update and update the storage structure of the global shared experience pool at the same time;

[0073] Step 5-3, according to the set time step T, update the parameters of the target Q network every time step T. The update rules are as follows:

[0074] ω target ←ω(when t=nT).

[0075] A multi-agent reinforcement learning system for optimizing experience storage and experience reuse, the system comprising sequentially executing:

[0076] Initialization module, used to initialize all parameters of the current Q network ω, all parameters of the target Q network ω′ = ω, and the capacity N of the experience pool, set the number of samples m for batch gradient descent, and the parameter update time step T of the target Q network;

[0077] The multi-agent experience collection module is used for multi-threaded multi-agent experience collection: through network weight sharing, multiple agents are deployed using multi-threading technology to interact with the environment simultaneously. At each time point, agents in different threads adopt different exploration strategies based on probability to obtain experience samples from the environment.

[0078] The experience pool construction module is used to calculate the absolute value of the time difference error (TD error) of the experience sample and insert the experience into the global shared experience pool with TD error as the index. At the same time, the time attribute values ​​of all experiences in the global shared experience pool are updated based on the least recently used mechanism.

[0079] The update module is used to sample m experience samples from the global shared experience pool using a hybrid sampling method when updating the current training strategy, and update the time attributes of the experience in the experience pool according to the least recently used mechanism; where m is the number of samples used to set the batch gradient descent;

[0080] The training module is used to train the current Q network using the m experience samples obtained by sampling. After the training is completed, the TD error of all experience samples in the experience pool is recalculated, the priority of the experience in the global shared experience pool is updated, and it is determined whether the number of training steps has reached the preset maximum value. If not, it returns to the experience pool construction module.

[0081] Regarding the specific limitations of the multi-agent reinforcement learning system for optimizing experience storage and experience reuse, please refer to the limitations of the multi-agent reinforcement learning method for optimizing experience storage and experience reuse above, which will not be repeated here. The various modules in the above-mentioned multi-agent reinforcement learning method for optimizing experience storage and experience reuse can be implemented in whole or in part by software, or by a combination of software and hardware. Based on this understanding, the above-mentioned modules can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to the above modules. Matters not covered in the present invention are known technologies.

[0082] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the following steps are performed:

[0083] Step 1: Initialize all parameters of the current Q network ω, all parameters of the target Q network ω′=ω, and the capacity N of the experience pool, set the number of samples m for batch gradient descent, and the parameter update time step T of the target Q network;

[0084] Step 2: Conduct multi-agent experience collection based on multi-threading: By sharing network weights, use multi-threading technology to deploy multiple agents to interact with the environment simultaneously. At each time point, agents in different threads adopt different exploration strategies based on probability to obtain experience samples from the environment.

[0085] Step 3: Calculate the absolute value of the time difference error (TD error) of the experience sample and insert the experience into the global shared experience pool with TD error as the index. At the same time, update the time attribute values ​​of all experiences in the global shared experience pool based on the least recently used mechanism.

[0086] Step 4: When updating the current training strategy, m experience samples are sampled from the global shared experience pool using a hybrid sampling method, and the time attributes of the experience in the experience pool are updated according to the least recently used mechanism; where m is the number of samples for setting batch gradient descent;

[0087] Step 5: Use the m experience samples obtained by sampling to train the current Q network. After the training is completed, recalculate the TD error of all experience samples in the experience pool, update the priority of the experience in the global shared experience pool, and determine whether the number of training steps reaches the preset maximum value. If not, return to step 3, otherwise end the process.

[0088] For the specific limitations of each step, please refer to the limitations of the multi-agent reinforcement learning method for optimizing experience storage and experience reuse mentioned above, which will not be repeated here.

[0089] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented:

[0090] Step 1: Initialize all parameters of the current Q network ω, all parameters of the target Q network ω′=ω, and the capacity N of the experience pool, set the number of samples m for batch gradient descent, and the parameter update time step T of the target Q network;

[0091] Step 2: Conduct multi-agent experience collection based on multi-threading: By sharing network weights, use multi-threading technology to deploy multiple agents to interact with the environment simultaneously. At each time point, agents in different threads adopt different exploration strategies based on probability to obtain experience samples from the environment.

[0092] Step 3: Calculate the absolute value of the time difference error (TD error) of the experience sample and insert the experience into the global shared experience pool with TD error as the index. At the same time, update the time attribute values ​​of all experiences in the global shared experience pool based on the least recently used mechanism.

[0093] Step 4: When updating the current training strategy, m experience samples are sampled from the global shared experience pool using a hybrid sampling method, and the time attributes of the experience in the experience pool are updated according to the least recently used mechanism; where m is the number of samples for setting batch gradient descent;

[0094] Step 5: Use the m experience samples obtained by sampling to train the current Q network. After the training is completed, recalculate the TD error of all experience samples in the experience pool, update the priority of the experience in the global shared experience pool, and determine whether the number of training steps reaches the preset maximum value. If not, return to step 3, otherwise end the process.

[0095] For the specific limitations of each step, please refer to the limitations of the multi-agent reinforcement learning method for optimizing experience storage and experience reuse mentioned above, which will not be repeated here.

[0096] Compared with traditional methods, the multi-agent experience replay method proposed in the present invention, which combines the least recently used mechanism and the priority hybrid sampling mechanism, further enriches the sample types on the basis of fully reducing the correlation between sample data, has a high degree of biological interpretability, and performs better when facing complex environments and tasks.

[0097] The basic principles, main features, and advantages of the present invention are shown and described above. Those skilled in the art should understand that the present invention is not limited to the foregoing embodiments. The foregoing embodiments and descriptions are merely illustrative of the principles of the present invention. Various changes and modifications may be made to the present invention without departing from the spirit and scope of the present invention. Such changes and modifications are intended to fall within the scope of the present invention. The scope of protection claimed in the present invention is defined by the appended claims and their equivalents.

Claims

1. A multi-agent reinforcement learning method for optimizing experience storage and experience reuse, characterized in that: The method comprises the following steps: Step 1: Initialize all parameters of the current Q network ω, all parameters of the target Q network ω′=ω, and the capacity N of the experience pool, set the number of samples m for batch gradient descent, and the parameter update time step T of the target Q network; Step 2: Conduct multi-agent experience collection based on multi-threading: By sharing network weights, use multi-threading technology to deploy multiple agents to interact with the environment simultaneously. At each time point, agents in different threads adopt different exploration strategies based on probability to obtain experience samples from the environment. Step 3: Calculate the absolute value of the time difference error (TD error) of the experience sample and insert the experience into the global shared experience pool with TD error as the index. At the same time, update the time attribute values ​​of all experiences in the global shared experience pool based on the least recently used mechanism. Step 4: When updating the current training strategy, m experience samples are sampled from the global shared experience pool using a hybrid sampling method, and the time attributes of the experience in the experience pool are updated according to the least recently used mechanism; where m is the number of samples for setting batch gradient descent; Step 5: Use the m experience samples obtained by sampling to train the current Q network. After the training is completed, recalculate the TD error of all experience samples in the experience pool, update the priority of the experience in the global shared experience pool, and determine whether the number of training steps reaches the preset maximum value. If not, return to step 3, otherwise end the process.

2. The multi-agent reinforcement learning method for optimizing experience storage and experience reuse according to claim 1, characterized in that: In step 2, the experience gained by the agent in interacting with the environment at time point t is a five-tuple {s t ,a t ,r t ,s t+1 ,is_end}, where s t is the state of the environment where the Agent is located at time t, and is also the input of the current Q network. t Represents the action performed by the Agent at time t. The environment transfers to the state s at the next moment according to the Agent's state and action t+1 And give the environmental reward r obtained by this action t , is_end indicates whether the current state is already the terminal state.

3. The multi-agent reinforcement learning method for optimizing experience storage and experience reuse according to claim 1 or 2, characterized in that: The multi-agent experience collection based on multi-threading described in step 2 specifically includes the following steps: Deploy a public neural network model and create n threads based on it. Each thread has the same network structure and parameters as the public neural network model. Each thread independently interacts with the environment to obtain empirical data. These threads do not interfere with each other and run independently. Each individual sampling agent interacts with its own environment to generate experience samples, and the experience samples generated by all sampling agents are uniformly stored in a shared experience buffer pool. Each agent adopts an ε-greedy strategy to select actions. Specifically, in each state, the action with the maximum state-action value function is selected with a probability of 1-ε, and a random action is selected with a probability of ε.

4. The multi-agent reinforcement learning method for optimizing experience storage and experience reuse according to claim 3, characterized in that: Step 3 calculates the absolute value of the temporal differential error (TD error) of the experience sample and inserts the experience into the global shared experience pool indexed by TD error. At the same time, the time attribute values ​​of all experiences in the global shared experience pool are updated based on the least recently used mechanism. The specific process includes: Step 3-1, calculate the time difference error δ of the current time t experience sample t : δ t =Q * (s t ,a t )-y t Where s t is the state of the agent's environment at time t, a t represents the action performed by the agent at time t, y t represents the TD target at time t, Q * (s t ,a t ) is in state s t Take action a t , is the approximation of the maximum cumulative future reward expectation of the current Q network calculated using the Monte Carlo approximation method, as follows: Where r t is the real-time reward obtained after the agent interacts with the environment at time t, γ is the discounted return rate, and s t+1 The new state that the Agent transfers to after executing the action; Step 3-2, take the absolute value of the calculated time difference error |δ t | is used as the priority weight of the experience sample. If the time difference error of the experience quintuple cannot be calculated at present, its priority is set to the preset maximum value, and then the priority of the experience sample is inserted as an index into the B + In the global shared experience pool with a tree as the storage structure, the non-leaf nodes of the tree only store the priority of the experience samples, while the leaf nodes store both the priority of the experience and the experience quintuple itself; The time attribute of each experience in the global shared experience pool is derived from the principle of least recent use: the time attribute of the experience currently inserted into the global shared experience pool is set to 0, and at the same time the time attribute values ​​of all other experiences in the global shared experience pool are increased by 1. When the capacity of the experience pool reaches the upper limit, the experience with the largest time attribute value will be discarded.

5. The multi-agent reinforcement learning method for optimizing experience storage and experience reuse according to claim 4, characterized in that: The hybrid sampling method described in step 4 is specifically to obtain a higher sampling rate for sample data with higher priority. The hybrid sampling method is used to sample m experience samples and update the time attribute of the experience in the experience pool according to the least recently used mechanism. The specific implementation process is as follows: Step 4-1: Given the capacity of the global shared experience pool as N, a line segment of length 1 is divided into N segments, each of which corresponds to an experience sample in the experience pool. The line segment lengths corresponding to each experience sample are different. The line segments corresponding to high-priority experience samples are long, while the line segments corresponding to low-priority experience samples are short. The line segment length of each experience sample i is determined by the following formula: Where, δ i represents the time difference error of the empirical sample, k is the number of empirical samples, ε is a positive number that is infinitely close to 0 to avoid the situation where the line length is 0, and the exponent value is 3 / 4 as the empirical value; Step 4-2: Before sampling, divide the line segment of length 1 into M equal parts, where M>>N. Each of the M parts will fall on a line segment corresponding to an empirical sample, and the line segment with a higher priority will be divided into more small pieces. During sampling, m positions are uniformly sampled from the M positions. In step 4-3, the priority of the extracted experience is used as the index value to find the corresponding experience sample from the global shared experience pool, and its time attribute value is updated to 0. At the same time, the time attribute values ​​of the remaining experience samples are all increased by 1.

6. The multi-agent reinforcement learning method for optimizing experience storage and experience reuse according to claim 5, characterized in that: In step 5, the current Q network is trained using the m empirical samples obtained by sampling. Specifically, a batch gradient descent method based on a variable learning rate is used. The steps are as follows: Step 5-1, take out the experience samples from the m experience samples in turn, and use the target Q network to calculate the gradient value g corresponding to the i-th experience sample i : Where ω is the parameter of the current Q network, s i ,a i are the state of the agent in the i-th experience sample and the action selected by the agent in the current state; Step 5-2, using the gradient value g i Update the parameters of the current Q network, and the update rules are: Among them, α′ is the learning rate, which is a variable parameter and is defined as: α′=α·(n·p i ) -β In the formula, α is the initial setting value of the learning rate, which is a hyperparameter that needs to be given manually before network training. i is the sampling probability of the i-th experience sample, β∈(0,1). At the beginning of training, β is a positive real number close to 0. As the training iteration proceeds, β gradually approaches 1. After each training, the time difference error δ of the experience sample is calculated using the Q network after parameter update. i Update and update the storage structure of the global shared experience pool at the same time; Step 5-3, according to the set time step T, update the parameters of the target Q network every time step T. The update rules are as follows: ω target ←ω(when t=nT)。 7. A multi-agent reinforcement learning system that optimizes experience storage and experience reuse, characterized by: The system includes sequentially executing: Initialization module, used to initialize all parameters of the current Q network ω, all parameters of the target Q network ω′ = ω, and the capacity N of the experience pool, set the number of samples m for batch gradient descent, and the parameter update time step T of the target Q network; The multi-agent experience collection module is used for multi-threaded multi-agent experience collection: through network weight sharing, multiple agents are deployed using multi-threading technology to interact with the environment simultaneously. At each time point, agents in different threads adopt different exploration strategies based on probability to obtain experience samples from the environment. The experience pool construction module is used to calculate the absolute value of the time difference error (TD error) of the experience sample and insert the experience into the global shared experience pool with TD error as the index. At the same time, the time attribute values ​​of all experiences in the global shared experience pool are updated based on the least recently used mechanism. The update module is used to sample m experience samples from the global shared experience pool using a hybrid sampling method when updating the current training strategy, and update the time attributes of the experience in the experience pool according to the least recently used mechanism; where m is the number of samples used to set the batch gradient descent; The training module is used to train the current Q network using the m experience samples obtained by sampling. After the training is completed, the TD error of all experience samples in the experience pool is recalculated, the priority of the experience in the global shared experience pool is updated, and it is determined whether the number of training steps has reached the preset maximum value. If not, it returns to the experience pool construction module.

8. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images