Multi-agent collaboration method and device based on priority experience playback

Abstract

The invention relates to the technical field of reinforcement learning, and provides a multi-agent collaboration method and device based on priority experience playback. The method comprises the following steps: completing multi-round strategy training by updating a data set; wherein the strategy training of each round comprises the following steps: acquiring a data set through interaction between an intelligent agent and an environment; based on sample quality, constructing an optimal subspace according to the data set; key points are obtained through screening in the optimal subspace, and multi-agent training is conducted through the key points. According to the method, data sampling is performed based on sample quality, and optimal subspace and key point screening is performed, so that the problem of difficult exploration caused by sparse rewards in multi-agent reinforcement learning is relieved, and the exploration efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the field of reinforcement learning technology, and in particular to a multi-agent collaboration method and device based on priority experience replay. Background Art

[0002] As the third paradigm of machine learning, reinforcement learning (RL) overcomes the constraint of supervised learning that requires a large number of labeled data sets, and has achieved good performance in solving the problems of difficult and inaccurate modeling. Since Q-learning, reinforcement learning has rapidly developed from single-agent algorithms to multi-agent algorithms. To date, multi-agent reinforcement learning has been successfully applied in logistics, Go, robotics, and games.

[0003] However, in a multi-agent environment, sparse rewards make exploration more difficult. Since the agent cannot obtain environmental rewards in a short period of time, the agent cannot judge the goodness of the action taken for a period of time, and cannot be used to guide the subsequent exploration of the agent, resulting in a sharp decline in exploration efficiency, which hinders the widespread application of multi-agent reinforcement learning in the real world.

[0004] In view of this, overcoming the defects of the prior art is an urgent problem to be solved in this technical field. Summary of the invention

[0005] The technical problem to be solved by the present invention is to provide a multi-agent collaboration method and device based on priority experience replay to alleviate the problem of exploration difficulty caused by sparse rewards in multi-agent reinforcement learning.

[0006] The present invention adopts the following technical solution:

[0007] In a first aspect, the present invention provides a multi-agent collaboration method based on priority experience replay, which completes multiple rounds of strategy training by updating a data set; wherein each round of strategy training includes:

[0008] Obtaining data sets through the interaction between the agent and the environment;

[0009] Based on the sample quality, the optimal subspace is constructed according to the data set;

[0010] The key points are screened in the optimal subspace and used to train multi-agents.

[0011] Preferably, the step of acquiring a data set through interaction between an agent and an environment specifically includes:

[0012] In each round, the environment state is reset to the initial state, and the initial joint observation of all agents is obtained. In each time step of a round, the transfer tuple of the corresponding time step is obtained through the interaction between the agent and the environment, and each transfer tuple is stored to obtain a data set;

[0013] Each round includes at least one time step, and the initial joint observation of the agents in one round is used as the joint observation of the first time step to perform interactions within the first time step.

[0014] Preferably, in each time step of a round, obtaining the transfer tuple of the corresponding time step through the interaction between the agent and the environment specifically includes:

[0015] At time step t, the current state is used as the starting state s t , each agent has a starting state s t Select the action to be performed at the current time step according to the preset strategy, and combine the actions selected by all agents into a joint action

[0016] Perform joint action a t , the state after the joint action is executed is taken as the successor state s t+1 , and the agent receives the environmental reward r t and the joint observation o at the next time step t+1 ;

[0017] Set the starting state s of the current time step to t , the joint observation o at the current time step t , joint action a t , successor state s t +1 , the joint observation o at the next time step t+1 and environmental rewards t The transition tuple (s) that constitutes the current time step t ,o t ,a t ,s t+1 ,o t +1 ,r t ).

[0018] Preferably, after performing the joint action, each agent observes the environment to obtain the local observation of the agent at the next time step, and the local observations of all agents at the next time step constitute the joint observation of the next time step. t+1 .

[0019] Preferably, constructing the optimal subspace based on the sample quality and the data set specifically includes:

[0020] The data set is batch sampled based on sample quality; wherein, every M rounds, N data subsets are constructed, wherein K samples with reward values ​​are sampled from the data set of each round as a data subset;

[0021] Each data subset is regarded as a subspace, the entropy of each subspace is calculated, and the subspace with the maximum entropy is selected as the optimal subspace.

[0022] Preferably, the determinant of the matrix corresponding to each subspace is used as the entropy of the subspace.

[0023] Preferably, the step of screening out key points in the optimal subspace specifically includes:

[0024] Statistically optimal subspace S i The counts of each column vector in the equation are calculated, and the column vector with the smallest count is selected as the key point, that is, g = argmin count (S i ); wherein the optimal subspace includes multiple transfer tuples (s t ,o t ,a t ,s t+1 ,o t+1 ,r t ), s t is the starting state at time step t, o t is the joint observation at time step t, a t is the joint action at time step t, s t+1 is the successor state of time step t, r t is the environment reward at time step t, o t+1 is the joint observation at time step t+1, and the column vector is s t .

[0025] Preferably, the multi-agent training using key points specifically includes:

[0026] When the agent explores a key point, it is given an additional reward.

[0027] In a second aspect, the present invention further provides a multi-agent collaboration device based on priority experience replay, which is used to implement the multi-agent collaboration method based on priority experience replay described in the first aspect, and the device comprises:

[0028] At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the processor to execute the multi-agent collaboration method based on prioritized experience replay as described in the first aspect.

[0029] In a third aspect, the present invention further provides a non-volatile computer storage medium, wherein the computer storage medium stores computer executable instructions, and the computer executable instructions are executed by one or more processors to complete the method described in the first aspect.

[0030] In a fourth aspect, a chip is provided, comprising: a processor and an interface, for calling and running a computer program stored in the memory from a memory, and executing any method of the first aspect.

[0031] In a fifth aspect, a computer program product comprising instructions is provided, which, when executed on a computer or a processor, causes the computer or the processor to execute any method as described in the first aspect.

[0032] The present invention performs data sampling based on sample quality and screens the optimal subspace and key points, thereby alleviating the problem of exploration difficulty caused by sparse rewards in multi-agent reinforcement learning and improving exploration efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments of the present invention. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0034] Figure 1 It is a flowchart of a first multi-agent collaboration method based on priority experience playback provided by an embodiment of the present invention;

[0035] Figure 2 is a schematic diagram of a multi-agent collaboration method based on priority experience playback provided by an embodiment of the present invention;

[0036] Figure 3 is a schematic diagram of another multi-agent collaboration method based on priority experience playback provided by an embodiment of the present invention;

[0037] Figure 4 is a flow chart of a second multi-agent collaboration method based on priority experience playback provided by an embodiment of the present invention;

[0038] Figure 5 is a flowchart of a third multi-agent collaboration method based on priority experience playback provided by an embodiment of the present invention;

[0039] Figure 6 is a flowchart of a fourth multi-agent collaboration method based on priority experience playback provided by an embodiment of the present invention;

[0040] Figure 7 is a flowchart of a fifth multi-agent collaboration method based on priority experience playback provided by an embodiment of the present invention;

[0041] Figure 8 is a flowchart of a sixth multi-agent collaboration method based on priority experience playback provided by an embodiment of the present invention;

[0042] Fig. 9 is a flowchart of a seventh multi-agent collaboration method based on priority experience playback provided by an embodiment of the present invention;

[0043] Fig.10 is a flowchart of an eighth multi-agent collaboration method based on priority experience playback provided by an embodiment of the present invention;

[0044] Fig.11 is a schematic diagram of another multi-agent collaboration method based on priority experience playback provided by an embodiment of the present invention;

[0045] Fig.12 is a flowchart of a ninth multi-agent collaboration method based on priority experience playback provided by an embodiment of the present invention;

[0046] Fig.13 is a flowchart of a tenth multi-agent collaboration method based on priority experience playback provided by an embodiment of the present invention;

[0047] Fig.14 is a flowchart of the eleventh multi-agent collaboration method based on priority experience playback provided by an embodiment of the present invention;

[0048] Fig.15 It is a schematic diagram of the architecture of a multi-agent collaborative device based on priority experience playback provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0050] Unless the context requires otherwise, throughout the specification and claims, the term "including" is to be interpreted as open inclusion, that is, "including, but not limited to". In the description of the specification, the terms "one embodiment", "some embodiments", "exemplary embodiments", "examples", "specific examples" or "some examples" and the like are intended to indicate that specific features, structures, materials or characteristics associated with the embodiment or example are included in at least one embodiment or example of the present disclosure. The schematic representation of the above terms does not necessarily refer to the same embodiment or example. In addition, the specific features, structures, materials or characteristics may be included in any one or more embodiments or examples in any appropriate manner, that is, although they may be carried in the embodiments or examples of the above terms due to reasons such as the order and position of appearance, it is not limited to that they can be carried in combination by one embodiment or example.

[0051] In the description of the present invention, the terms "first" and "second" are used only for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present disclosure, unless otherwise specified, the meaning of "multiple" is two or more. In addition, for example, the same type of nouns may be described as two independent individuals by adding "A" and "B" at the end. In this case, the corresponding features defined as "A" and "B" are only used to distinguish the same type of individuals for description purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features.

[0052] In the description of the present invention, the expression "A and / or B" (where A and B are used to formally represent specific characteristic contents) will be involved, and the corresponding expressions include the following three combinations: only A, only B, and a combination of A and B.

[0053] As used herein, "about," "substantially," or "approximately" includes the stated value and an average value that is within an acceptable range of deviation from the particular value as determined by one of ordinary skill in the art taking into account the measurements in question and the errors associated with the measurement of the particular quantity (i.e., the limitations of the measurement system).

[0054] In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0055] Embodiment 1:

[0056] In order to solve the sparse reward problem during multi-agent exploration, a variety of exploration methods have been proposed. For example, the successor representation method (full name in English: Successor representation) and the reinforcement learning exploration method based on embedded state space (full name in English: Count-Based Exploration via Embedded State Space for Deep Reinforcement Learning) both use count-based exploration strategies to encourage agents to visit states that are rarely visited; the SAC (full name: Soft Actor-Critic) algorithm and the KME (full name: k-Means Maximum Entropy Exploration) algorithm promote the agent's full exploration of the environment based on the maximum entropy principle.

[0057] These methods can essentially be regarded as reinforcement learning methods based on intrinsic motivation. Reinforcement learning methods based on intrinsic motivation introduce intrinsic reward mechanisms to motivate agents to explore the environment autonomously, thereby improving learning efficiency. This method encourages agents to continue learning in the absence of clear external rewards, and is widely used in fields such as robot learning, game AI, personalized education, and multi-agent systems. However, it still faces challenges such as designing an effective intrinsic reward mechanism and balancing exploration and utilization. In order to solve this problem, Example 1 of the present invention provides a multi-agent collaboration method based on priority experience replay, which completes multiple rounds of strategy training by updating data sets; wherein, Figure 1 As shown, each round of strategy training includes:

[0058] In step 201, a data set is obtained through the interaction between the agent and the environment; in order to train multiple agents, it is first necessary to obtain sample data for training to generate a data set. In this embodiment, the agent selects actions to interact with the environment according to the overall strategy, and obtains the subsequent state of the environment state and the environment reward after the agent performs different actions to form a transfer tuple. After obtaining the transfer tuples of the agent at different time steps, all the obtained transfer tuples can be used as the experience pool data set, and each transfer tuple is used as a sample data in the experience pool data set.

[0059] In actual implementation, if necessary, the interaction between the agent and the environment can be completed through reinforcement learning. Figure 2 Online reinforcement learning (On-line RL), in which the agent and the environment are constantly interacting, and the dataset is continuously expanded as the interaction proceeds; or using Figure 3 On-policy RL, in which the sample data comes from the current policy and each sample data is used only once.

[0060] In step 202, based on the sample quality, an optimal subspace is constructed according to the data set.

[0061] In step 203, key points are screened in the optimal subspace and used to train the multi-agent.

[0062] This embodiment performs data sampling based on sample quality and screens the optimal subspace and key points, thereby alleviating the problem of difficult exploration caused by sparse rewards in multi-agent reinforcement learning and improving exploration efficiency.

[0063] In a specific implementation, the data set is obtained through the interaction between the agent and the environment, such as Figure 4 As shown, specifically including:

[0064] In step 301, in each round, the environment state is reset to the initial state, and the initial joint observations of all agents are obtained; that is, the local observations of all agents in the initial state constitute the initial joint observations of the agents.

[0065] In step 302, within each time step of a round, the transfer tuple of the corresponding time step is obtained through the interaction between the agent and the environment, and each transfer tuple is stored to obtain a data set; wherein each round includes at least one time step, and the initial joint observation of the agent of a round is used as the joint observation of the first time step, and the interaction within the first time step is performed.

[0066] In each time step of a round, the transfer tuple of the corresponding time step is obtained through the interaction between the agent and the environment, such as Figure 5 As shown, specifically including:

[0067] In step 401, at time step t, the current state is used as the starting state s t , each agent has a starting state s t Select the action to be performed at the current time step according to the preset strategy, and combine the actions selected by all agents into a joint action The preset strategy is obtained by those skilled in the art based on demand analysis, and is used to train the agent to access the target and collect data to obtain the maximum environmental reward.

[0068] In step 402, a joint action a is performed. t , the state after the joint action is executed is taken as the successor state s t+1 , and the agent receives the environmental reward r t and the joint observation o at the next time step t+1 .

[0069] In step 403, the initial state s of the current time step is t , the joint observation o at the current time step t , joint action a t , successor state s t+1 , the joint observation o at the next time step t+1 and environmental rewards t The transition tuple (s) that constitutes the current time step t ,o t ,a t ,s t+1 ,o t+1 ,r t ).

[0070] After performing the joint action, each agent observes the environment to obtain the local observation of the agent at the next time step, and the local observations of all agents at the next time step constitute the joint observation of the next time step. t+1 .

[0071] In an optional implementation, the optimal subspace is constructed based on the sample quality according to the data set, such as Figure 6 As shown, specifically including:

[0072] In step 501, batch sampling is performed on the data set based on sample quality; wherein, every M rounds, N data subsets are constructed, wherein K samples with reward values ​​are sampled from the data set of each round as a data subset.

[0073] In step 502, each data subset is taken as a subspace, the entropy of each subspace is calculated, and the subspace with the maximum entropy is selected as the optimal subspace. The determinant of the matrix corresponding to each subspace is taken as the entropy of the subspace.

[0074] In an optional implementation manner, the step of screening the key points in the optimal subspace specifically includes: statistically analyzing the optimal subspace S i The counts of each column vector in the equation are calculated, and the column vector with the smallest count is selected as the key point, that is, g = argmin count (S i ); wherein the optimal subspace includes multiple transfer tuples (s t ,o t ,a t ,s t+1 ,o t+1 ,r t ), s t is the starting state at time step t, o t is the joint observation at time step t, a t is the joint action at time step t, s t+1 is the successor state at time step t, rt is the environment reward at time step t, o t+1 is the joint observation at time step t+1, and the column vector is s t .

[0075] In a specific application scenario, the use of key points for multi-agent training specifically includes: when the agent explores the key point, giving the agent an additional reward.

[0076] This embodiment samples high-quality samples by the priority experience playback method, constructs the optimal subspace based on the maximum entropy exploration principle, selects key points based on the counting method, and emphasizes the access to key points through additional rewards under the framework of on-policy reinforcement learning, and achieves the purpose of training strategy by continuously updating the data set, thereby solving the pain point that agents in multi-agent reinforcement learning are difficult to explore in the face of sparse rewards, and realizing collaborative exploration between multi-agents and overall optimization of the system. Compared with the historical patent with publication number CN118504651A, entitled "A multi-agent collaborative method and device based on maximum entropy", the historical patent does not care about the sample quality and randomly samples from the data set, and then uses the sampled data to construct an augmented matrix (regarded as the space before the projection operation); then uses the numerical calculation method to project the above space to the space with maximum entropy, that is, the optimal projection space; finally, the count-based method is used in the abstract space to select the shared target. The samples used to construct the subspace in this embodiment are selected based on the sample quality, and the number of subspaces formed is controlled by hyperparameters, and then the entropy of each subspace is calculated, and the optimal subspace is selected as the premise for screening key points. In other words, the space where the key states in historical patents are located is an abstract space, and the shared target of screening may be a state that has never been visited; in this case, the space where the key points are located is an actual subspace, and the key points of screening are also samples that have been visited.

[0077] Embodiment 2:

[0078] The present invention is based on the method described in Example 1, combined with specific application scenarios, and uses technical descriptions in related scenarios to illustrate the implementation process of the present invention in characteristic scenarios.

[0079] like Figure 7 As shown, the multi-agent collaboration method based on priority experience playback provided in this embodiment specifically includes:

[0080] In step 601, a data set is acquired through the interaction between the agent and the environment.

[0081] In step 602, the optimal subspace is constructed based on the sample quality. Specifically, first, batch sampling is performed on the data set based on the sample quality: every M rounds, K samples with reward values ​​in the data set are sampled each time to construct N data subsets; then, the optimal subspace is constructed: each data subset is regarded as a subspace, the entropy of each subspace is calculated, and the subspace with the maximum entropy is selected as the optimal subspace.

[0082] In step 603, key points are screened in the optimal subspace, that is, using a count-based exploration method, the element with the least count in the optimal subspace is selected as the key point.

[0083] In step 604, training is performed using the key points. In specific implementation, training is performed based on the acquired key points in every M rounds. When the agent accesses the key points, an additional reward is given to the agent.

[0084] In step 605, the strategy training is completed by updating the data set. In the specific implementation, the data set is updated through the continuous interaction between the agent and the environment, so as to obtain the best convergence strategy for the agent to access the terminal state from the initial state - to obtain the maximum cumulative discounted reward.

[0085] This embodiment uses the method described in steps 601 to 605 to train multiple agents as a whole, construct an optimal subspace based on the maximum entropy principle, screen key points based on a counting method, and emphasize the visit of key points through additional rewards, thereby alleviating the pain point of sparse rewards in multi-agent reinforcement learning.

[0086] Before training, the target strategy π of each agent is initialized. i (i.e., the preset strategy in Example 1); then, the overall strategy of all agents That is, each agent has a target strategy. For multiple agents, the entire algorithm is regarded as the joint strategy of all agents, that is, the overall strategy. After completing the target strategy initialization, it is also necessary to complete the interaction between the agent and the environment based on the overall strategy, and use the transfer tuples obtained from the interaction to expand the experience pool data set D, that is, obtain the data set through the interaction between the agent and the environment, such as Figure 8 As shown, specifically including:

[0087] In step 701, at the beginning of each round, the environment state is reset to the initial state and the joint observations of all agents are obtained.

[0088] When exploring, the agent and the environment need to interact for multiple rounds, each round is represented by i, i∈[1...E]. When entering each round i, the environment state needs to be reset to s first. 1 , and observe the first state O when the environment is initialized1 .

[0089] At the same time, each agent observes the initial state to obtain the local observation of the agent, and the local observations of all agents are combined into the initial joint observation. The local observation obtained by the yth agent is expressed as The first joint observation of n agents is expressed as

[0090] In step 702, at each time step, each agent selects its own action according to the strategy and the global state, and the actions selected by all agents are combined into a joint action.

[0091] In the specific implementation, each round includes at least one time step, each time step is denoted as t, t∈(1,2…H), and the agent performs an action in each time step.

[0092] At each time step t, t∈(1,2…H), according to the global state s t , each agent uses the target strategy to select its own actions to form a joint action. The action selected by the yth agent according to the target strategy is expressed as The joint action consisting of the actions selected by n agents is expressed as

[0093] In step 703, after performing the joint action at each time step, the starting state is transformed into the successor state, and the agent obtains the environment reward and the next joint observation.

[0094] At each time t, before executing the joint action, the global state of the environment is s at the current time step t , the joint observation o at the current time step t After executing the joint action, the global state of the environment changes to the successor state s t+1 , all agents receive a total environmental reward r t , the joint observation of all agents is transformed into the next joint observation o t+1 .

[0095] In step 704, the starting state, joint observation, joint action, successor state, next joint observation and environmental reward of each time step are combined into a transfer tuple of the current time step, which is stored to obtain an experience playback data set. After obtaining the global state and joint observation before and after the execution of the joint action, as well as the corresponding environmental reward, the corresponding parameters can be combined into a transfer tuple of the time step. The transfer tuple can be expressed as (s t ,o t ,a t ,s t+1 ,o t+1 ,r t).

[0096] In this embodiment, the method described in step 701 to step 704 is used. After completing the interaction of all time steps in all rounds, the transfer tuples in each time step can be obtained, and all transfer tuples are stored to continuously expand the data set D. After obtaining the data set D, the samples in the data set D can be used to repeatedly train the agent according to the subsequent steps, and new transfer tuples are obtained after each training to update the data set D, that is, to perform strategy training.

[0097] The constructing of the optimal subspace based on sample quality specifically includes: splicing the state and action of each transfer tuple in each data subset into a state-action vector to form multiple augmented subspaces, and selecting one of the multiple augmented subspaces as the optimal subspace, wherein the splicing of the state and action of each transfer tuple in each data subset into a state-action vector to form multiple augmented subspaces, such as Fig. 9 As shown, specifically including:

[0098] In step 801, the data set is sampled preferentially based on the sample quality to obtain a data subset, wherein, every M rounds, K samples with reward values ​​in the data set are sampled each time to obtain N data subsets.

[0099] In step 802, a subspace is constructed using the data subset. That is, the state in each transfer tuple is regarded as a column vector and a matrix is ​​constructed, which is the subspace. In the specific implementation, for each transfer tuple (s t ,o t ,a t ,s t +1 ,o t+1 ,r t ), the state s in the transition tuple t As a column vector, construct the matrix J.

[0100] Select one of the multiple augmented subspaces as the optimal subspace and filter the key points in the optimal subspace, such as Fig.10 As shown, specifically including:

[0101] In step 901, the entropy of each subspace is calculated and the subspace with the maximum entropy is selected as the optimal subspace based on the maximum entropy principle. That is, the optimal subspace is obtained based on the maximum entropy principle, and the key points are screened in the optimal subspace based on the counting method. According to the maximum entropy principle, the greater the entropy of the exploration space, the more complete the exploration. Therefore, the entropy of each subspace is calculated and the subspace with the maximum entropy is selected as the optimal subspace, that is, the target space where the key points are located. In the specific implementation, the determinant of the matrix corresponding to each subspace is used as the entropy of the subspace. For the N subspaces constructed, the entropy values ​​are calculated to be μ1, μ2,…, μi ,…,μ N . Compare and select the subspace with the largest entropy value: i = argmax i∈1,2,…,N (μ i ).

[0102] In step 902, the count of each column vector in the optimal subspace is counted, and the state vector with the minimum count is selected as the key point.

[0103] Statistical target space S i The counts of each column vector in , and the column vector with the minimum count is selected as the key point. That is, g = argmin coont (S i ).

[0104] After steps 301 to 402 provided in this embodiment, key points can be screened out.

[0105] Among them, count() means that the subspace S i Count by column, argmin count () indicates taking out S i The column vector with the least count in is taken as the key point.

[0106] After obtaining the key point g, the effect of the interaction between the agent and the environment during each training can be judged according to the terminal state. When the agent explores the key point, the agent will receive additional rewards.

[0107] In the above process, the method of multi-agent collaborative training includes the following sub-steps: obtaining samples to expand the data set, performing preferential experience replay on the data set based on sample quality to obtain multiple data subsets, calculating the entropy of each data subset and selecting the subspace with the maximum entropy as the optimal subspace, selecting key points in the optimal subspace based on counting methods, and using key points to complete the collaborative training of multi-agents.

[0108] In a specific application scenario, the complete training process of multi-agent is as follows: Fig.11 As shown in Figure 1, the training process of the agent includes: data sampling, entropy calculation, key point screening and collaborative training. When the agent interacts with the environment, at each time step t, the agent performs action a t , so that the environment state changes from the initial state s t Change to the successor state s t+1 , and obtain the corresponding reward value r t In the next time step, the successor state s in the previous time step is t+1 As the starting state s of this time step tAfter each round, the updated dataset D is used to train the target policy π, and additional rewards are used to emphasize the visit of key points in the process until the multi-agent training is completed.

[0109] This embodiment implements data sampling based on sample quality and the maximum entropy principle to identify key points, alleviating the problem of difficult exploration caused by sparse rewards in multi-agent reinforcement learning. The maximum entropy exploration principle is used to construct the optimal subspace, the counting-based exploration method is used to screen key points, and the additional rewards are used to emphasize the exploration of key points. This improves the exploration efficiency. In addition, the additional rewards emphasize the exploration of key points, solving the problem of difficult collaborative exploration of agents.

[0110] Embodiment 3:

[0111] Considering that there may be multi-target exploration scenarios in actual use, in order to cope with this scenario, this embodiment also provides a preferred implementation method based on Embodiment 1 and Embodiment 2. Embodiment 1 of the present invention provides a multi-agent collaboration method based on priority experience playback, such as Fig.12 As shown, including:

[0112] In step 1001, agents are reallocated to each target according to the loss function at every preset number of rounds; wherein, multiple agents allocated to one target form an agent group; the preset number is obtained by those skilled in the art based on empirical analysis. In order to realize agent collaborative exploration in a multi-target reinforcement learning environment, it is necessary to group agents and targets, allocate agents (make some agents as professional agents of the target); and in order to rationalize the allocation, a loss function is constructed and minimized; when a target reaches the final state, the agent allocated to the target is released so that it can serve other targets.

[0113] In addition, in order to rationalize the use of resources and prevent a certain target from occupying a certain agent for a long time, it is necessary to regroup the agent and the target, and release the agent occupied by the target so that it can be used for exploration by other agents. At the same time, in order not to make the grouping process occupy too much computing resources, it is planned to regroup once every M rounds.

[0114] In step 1002, according to the goal of each agent grouping, the target strategy of each agent is determined, and each agent is trained for multiple rounds using the target strategy, that is, multiple rounds of strategy training are completed by updating the data set in Example 1, wherein the target strategy is used as the preset strategy in Example 1. The target strategy is used to train the agent to access the target and collect data to obtain the maximum environmental reward.

[0115] In a specific implementation, the target strategy of each agent is determined according to the target of each agent group, which can be: setting a corresponding strategy for each target in advance, and taking the strategy of the target of the agent group to which the agent belongs as the target strategy of the agent.

[0116] This embodiment dynamically groups agents and targets (i.e., allocates agents) based on the constructed loss function, solves the difficult exploration problem in multi-target reinforcement learning, and realizes the collaborative allocation among agents. It also considers the optimal allocation of resources, so as to regroup agents and targets in a timely manner, and realizes efficient training of multiple agents.

[0117] In an optional implementation, the loss function is Where N is the number of targets participating in the allocation, k v is the number of agents required to participate in the vth goal of the assignment, dist vu is the distance between the vth target and the corresponding uth agent involved in the allocation; When dist vu The smaller it is, the closer the agent u is to the target v; x is the horizontal coordinate of the u-th agent, u y is the ordinate of the u-th agent, v x is the horizontal coordinate of the vth target, v y is the ordinate of the vth target. The smaller the value of the loss function, the better the allocation method.

[0118] Considering that in actual use, the number of agents may not completely match the number required by the target, in order to cope with various scenarios, this embodiment also provides a preferred implementation method, that is, according to the loss function, the agents are reallocated to each target, such as Fig.13 As shown, specifically including:

[0119] In step 1101, when the total number of agents When the total number of agents can meet the needs of all unfinished targets, all unfinished targets are used as targets for allocation, and the targets for allocation are allocated agents to minimize the value of the loss function.

[0120] In step 1102, when the total number of agents When the total number of agents cannot meet the needs of all unfinished targets, w targets are selected from all unfinished targets as targets to be allocated, and agents to minimize the value of the loss function; wherein w can be predetermined by those skilled in the art, or a target selection strategy can be predetermined by those skilled in the art, and w targets are selected according to the target selection strategy.

[0121] Among them, K i is the target of the i-th participant in the allocation, K v is the number of agents required for the vth unfinished target, and M is the number of unfinished targets. The unfinished target means that the target has not reached the required position, that is, the agent has not reached the required position of the target.

[0122] There is another situation: P <K v , that is, the total number of agents is less than the number of agents required for the vth unfinished goal. At this time, the vth unfinished goal does not participate in the allocation of agents; when P <min(K v ), that is, when the total number of agents is less than the number of agents required for each goal, no allocation of agents is performed and the exploration task cannot be completed.

[0123] Among them, the above method selects w targets from all unfinished targets as targets to be allocated, and allocates agents to minimize the value of the loss function, such as Fig.14 As shown, specifically including:

[0124] In step 1201, multiple combinations are taken from unfinished goals; wherein the number of goals included in each combination is the maximum number corresponding to the required number of agents less than or equal to P; wherein P is the total number of agents; the number of goals included in each combination is the maximum number corresponding to the required number of agents less than or equal to P, that is, the total number of agents required for all goals in the combination is just less than or equal to P, and when any other goal is added to the combination, the number of agents required in the combination exceeds P.

[0125] In step 1202, each combination is used as a target to participate in the allocation, and the value of the loss function under different agent allocation methods is calculated; the agent allocation method is to allocate which agents to which targets.

[0126] In step 1203, the combination corresponding to the minimum loss function value is used as the target for participating in the allocation, and the agent allocation method corresponding to the minimum loss function value is used to allocate the agents; step 1203 can be understood as selecting the one that minimizes the loss function value among all possible combinations and all agent allocation methods under all combinations.

[0127] In step 1204, alternatively, taking the combination with the largest number of target quantities as the target participating in the allocation, the agent allocation is performed using the agent allocation method when the loss function value corresponding to the target participating in the allocation is minimized. The step 1204 can be understood as first determining the combination that can explore the most targets, and then determining the optimal agent allocation method under this combination.

[0128] In a specific application scenario, that is, when the number of agents required for each unfinished target is the same, where w is the number of targets participating in the allocation, k is the number of agents required for each unfinished target, and P%k represents the remainder of P divided by k, represents rounding down.

[0129] Taking another case as an example, that is, the number of agents required for all targets is the same and is k. Then, reallocating agents for each target according to the loss function specifically includes:

[0130] If P < k, the exploration task will not be able to be completed.

[0131] If P ≥ k×N, the agents required for the target are sufficient. At this time, the loss function for selecting k×N agents for allocation is Minimize the loss function to allocate k agents for each agent v to complete the grouping of agents and targets; when a certain target (assumed to be numbered N) reaches the final state, release the agents allocated for this target, and minimize Allocate k×(N - 1) agents for the remaining N - 1 targets.

[0132] If k < P < k×N, it is not possible to allocate sufficient agents for all targets. At this time, only targets can be allocated, and k×w agents are selected for allocation; the loss function is Minimize the loss function to allocate k agents for each agent v to complete the grouping of agents and targets; every M rounds, release the agents allocated for all targets and regroup. If a certain target reaches the required position, allocate k×w′ agents for w′ targets among the remaining N - 1 targets; minimize Allocate k×(N - 1) agents for the remaining N - 1 targets.

[0133] It should be noted here that the methods in Embodiment 1 and Embodiment 2 are applicable in this embodiment and will not be elaborated here.

[0134] Embodiment 4:

[0135] As Fig.15, is a schematic diagram of the architecture of a multi-agent collaboration device based on priority experience playback according to an embodiment of the present invention. The multi-agent collaboration device based on priority experience playback according to this embodiment includes one or more processors 21 and a memory 22. Fig.15 A processor 21 is taken as an example.

[0136] The processor 21 and the memory 22 may be connected via a bus or other means. Fig.15 The example of connecting through bus is taken in the following.

[0137] The memory 22 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs and non-volatile computer executable programs, such as the multi-agent collaboration method based on priority experience replay in Example 1. The processor 21 executes the multi-agent collaboration method based on priority experience replay by running the non-volatile software programs and instructions stored in the memory 22.

[0138] The memory 22 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage devices. In some embodiments, the memory 22 may optionally include a memory remotely arranged relative to the processor 21, and these remote memories may be connected to the processor 21 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0139] The program instructions / modules are stored in the memory 22, and when executed by the one or more processors 21, the multi-agent collaboration method based on priority experience playback in the above-mentioned embodiment 1 is executed.

[0140] It is worth noting that the information interaction, execution process, etc. between the modules and units within the above-mentioned devices and systems are based on the same concept as the processing method embodiment of the present invention. The specific contents can be found in the description of the method embodiment of the present invention and will not be repeated here.

[0141] A person skilled in the art may understand that all or part of the steps in the various methods of the embodiments may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the storage medium may include: a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc.

[0142] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A multi-agent collaboration method based on priority experience replay, characterized in that: Multiple rounds of strategy training are completed by updating the data set. Each round of strategy training includes: Obtaining data sets through the interaction between the agent and the environment; Based on the sample quality, the optimal subspace is constructed according to the data set; The key points are screened in the optimal subspace and used to train multi-agents.

2. The multi-agent collaboration method based on priority experience playback according to claim 1 is characterized in that: The step of obtaining a data set through interaction between an agent and an environment specifically includes: In each round, the environment state is reset to the initial state, and the initial joint observation of all agents is obtained. In each time step of a round, the transfer tuple of the corresponding time step is obtained through the interaction between the agent and the environment, and each transfer tuple is stored to obtain a data set; Each round includes at least one time step, and the initial joint observation of the agents in one round is used as the joint observation of the first time step to perform interactions within the first time step.

3. The multi-agent collaboration method based on priority experience playback according to claim 2 is characterized in that: In each time step of a round, the transfer tuple of the corresponding time step is obtained through the interaction between the agent and the environment, specifically including: At time step t, take the current state as the starting state s t , each agent has a starting state s t Select the action to be performed at the current time step according to the preset strategy, and combine the actions selected by all agents into a joint action Perform joint action a t , the state after the joint action is executed is taken as the successor state s t+1 , and the agent receives the environmental reward r t and the joint observation o at the next time step t+1 ; Set the starting state s of the current time step to t , the joint observation o at the current time step t , joint action a t , successor state s t+1 , the joint observation o at the next time step t+1 and environmental rewards t The transition tuple (s) that constitutes the current time step t , o t , a t ,s t+1 , o t+1 , r t ).

4. The multi-agent collaboration method based on priority experience playback according to claim 3 is characterized in that: After performing the joint action, each agent observes the environment to obtain the local observation of the agent at the next time step. The local observations of all agents at the next time step constitute the joint observation of the next time step. t+1 .

5. The multi-agent collaboration method based on priority experience playback according to claim 1 is characterized in that: The optimal subspace is constructed based on the sample quality and the data set, specifically including: The data set is batch sampled based on sample quality; wherein, every M rounds, N data subsets are constructed, wherein K samples with reward values ​​are sampled from the data set of each round as a data subset; Each data subset is regarded as a subspace, the entropy of each subspace is calculated, and the subspace with the maximum entropy is selected as the optimal subspace.

6. The multi-agent collaboration method based on priority experience playback according to claim 5 is characterized in that: The determinant of the matrix corresponding to each subspace is taken as the entropy of the subspace.

7. The multi-agent collaboration method based on priority experience playback according to claim 1 is characterized in that: The key points are obtained by screening in the optimal subspace, specifically including: Statistically optimal subspace S i The counts of each column vector in the equation are calculated, and the column vector with the smallest count is selected as the key point, that is, g = argmin count (S i ); wherein the optimal subspace includes multiple transfer tuples (s t , o t , a t ,s t+1 , o t+1 , r t ), st is the starting state of time step t, o t is the joint observation at time step t, a t is the joint action at time step t, s t+1 is the successor state of time step t, r t is the environment reward at time step t, o t+1 is the joint observation at time step t+1, and the column vector is s t .

8. The multi-agent collaboration method based on priority experience playback according to claim 1 is characterized in that: The multi-agent training using key points specifically includes: When the agent explores a key point, it is given an additional reward.

9. A non-volatile computer storage medium, characterized in that: The computer storage medium stores computer executable instructions, which are executed by one or more processors to complete the multi-agent collaboration method based on prioritized experience replay as described in any one of claims 1-8.

10. A multi-agent collaborative device based on priority experience playback, characterized in that: include: at least one processor; And, a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the processor to execute the multi-agent collaboration method based on prioritized experience replay as described in any one of claims 1-8.

Images