Multi-agent multi-target exploration method and device based on dynamic marshalling

Abstract

The invention relates to the technical field of reinforcement learning, and provides a multi-agent multi-target exploration method and device based on dynamic marshalling. The method comprises the following steps of: redistributing an intelligent agent for each target according to a loss function every a preset number of rounds; wherein a plurality of agents distributed to a target form an agent group; and determining a target strategy of each agent according to the target of each agent marshalling, and performing multi-round training on each agent by using the target strategy. According to the invention, intelligent agent distribution is carried out based on the constructed loss function, the problem of difficult exploration in multi-objective reinforcement learning is solved, and collaborative distribution among intelligent agents is realized. And optimal allocation of resources is also considered, regrouping of the agents and the targets is carried out timely, and efficient training of multiple agents is realized.

Description

Technical Field

[0001] The present invention relates to the field of reinforcement learning technology, and in particular to a multi-agent multi-objective exploration method and device based on dynamic grouping. Background Art

[0002] Reinforcement learning (RL), as the third paradigm of machine learning, overcomes the constraint of supervised learning that requires a large number of labeled data sets, and has achieved good performance in solving the problems of difficult and inaccurate modeling. Since Q-learning, reinforcement learning has rapidly developed from single-agent algorithms to multi-agent algorithms. To date, multi-agent reinforcement learning has been successfully applied in logistics, Go, robotics, and games.

[0003] However, in a multi-objective reinforcement learning environment where a single agent cannot complete the task independently, as the number of objectives increases, the collaborative exploration of the agents becomes difficult to allocate. This makes the exploration difficulty increase dramatically, and in extreme cases, the target strategy will not converge. Problems such as difficult allocation and slow exploration hinder the widespread application of multi-agent reinforcement learning in the real world.

[0004] In view of this, how to overcome the defects of the existing technology and realize the collaborative training of multiple agents and multiple objectives is an urgent problem to be solved in this technical field. Summary of the invention

[0005] The technical problem to be solved by the present invention is to provide a multi-agent multi-objective exploration method based on dynamic grouping to achieve multi-agent multi-objective collaborative training.

[0006] The present invention adopts the following technical solution:

[0007] The present invention provides a multi-agent multi-objective exploration method based on dynamic grouping, comprising:

[0008] After a preset number of rounds, agents are reallocated to each target based on the loss function; multiple agents assigned to one target form an agent group;

[0009] According to the goal of each agent grouping, the target strategy of each agent is determined, and each agent is trained for multiple rounds using the target strategy.

[0010] Preferably, the loss function is

[0011] Where N is the number of targets participating in the allocation, k v is the number of agents required to participate in the vth goal of the assignment, dist vuis the distance between the vth target and the corresponding uth agent involved in the allocation; u x is the horizontal coordinate of the u-th agent, u y is the ordinate of the u-th agent, v x is the horizontal coordinate of the vth target, v y is the vertical coordinate of the vth target.

[0012] Preferably, the reallocation of agents to each target according to the loss function specifically includes:

[0013] When the total number of agents When all unfinished targets are used as targets for allocation, agents to minimize the value of the loss function;

[0014] When the total number of agents , then from all unfinished goals, w goals are selected as the goals to be allocated, and the goals to be allocated are allocated agents to minimize the value of the loss function;

[0015] Among them, K i is the target of the i-th participant in the allocation, K v is the number of agents required for the vth unfinished goal, and M is the number of unfinished goals.

[0016] Preferably, the method selects w targets from all unfinished targets as targets to be allocated, and allocates Agents, minimize the value of the loss function, including:

[0017] From the unfinished goals, take out multiple combinations; the number of goals contained in each combination is the maximum number of the required number of agents less than or equal to P; where P is the total number of agents;

[0018] Take each combination as the target of participation in allocation and calculate the value of the loss function under different agent allocation methods;

[0019] Use the combination corresponding to the minimum loss function value as the target for participation in allocation, and use the agent allocation method corresponding to the minimum loss function value to allocate agents;

[0020] Alternatively, the combination containing the largest number of targets is used as the target for allocation, and the agents are allocated using the agent allocation method with the smallest loss function value corresponding to the targets for allocation.

[0021] Preferably, when the number of agents required for each unfinished goal is the same, Where w is the number of goals to be allocated, k is the number of agents required for each uncompleted goal, and P%k represents the remainder of P divided by k. Represents round down.

[0022] Preferably, the method of using the target strategy to train each agent for multiple rounds specifically includes:

[0023] In each round, the environment state is reset to the initial state, and the initial joint observation of all agents is obtained. In each time step of a round, the agent uses the target strategy to interact with the environment to obtain the transfer tuple of the corresponding time step, and each transfer tuple is stored to obtain the experience replay data set;

[0024] Each round includes at least one time step, and the initial joint observation of the agents in one round is used as the joint observation of the first time step to perform interactions within the first time step.

[0025] Preferably, in each time step of a round, the agent uses the target strategy to interact with the environment to obtain the transfer tuple of the corresponding time step, specifically including:

[0026] At time step t, the current state is used as the starting state s t , each agent has a starting state s t Select the action to be performed at the current time step according to the target strategy, and combine the actions selected by all agents into a joint action

[0027] Perform joint action a t , the state after the joint action is executed is taken as the successor state s t+1 , and the agent receives the environmental reward r t and the joint observation o at the next time step t+1 ;

[0028] Set the starting state s of the current time step to t , the joint observation o at the current time step t , joint action a t , successor state s t +1 , the joint observation o at the next time step t+1 and environmental rewards t The transition tuple (s) that constitutes the current time step t ,o t ,a t ,s t+1 ,o t +1 ,r t ).

[0029] Preferably, after performing the joint action, each agent observes the environment to obtain the local observation of the agent at the next time step, and the local observations of all agents at the next time step constitute the joint observation of the next time step. t+1 .

[0030] In a second aspect, the present invention further provides a multi-agent multi-objective exploration device based on dynamic grouping, which is used to implement the multi-agent multi-objective exploration method based on dynamic grouping described in the first aspect, and the device comprises:

[0031] At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the processor to execute the multi-agent multi-objective exploration method based on dynamic grouping as described in the first aspect.

[0032] In a third aspect, the present invention further provides a non-volatile computer storage medium, wherein the computer storage medium stores computer executable instructions, and the computer executable instructions are executed by one or more processors to complete the method described in the first aspect.

[0033] In a fourth aspect, a chip is provided, comprising: a processor and an interface, for calling and running a computer program stored in the memory from a memory, and executing any method of the first aspect.

[0034] In a fifth aspect, a computer program product comprising instructions is provided, which, when executed on a computer or a processor, causes the computer or the processor to execute any method as described in the first aspect.

[0035] The present invention dynamically groups agents and targets (i.e., allocates agents) based on the constructed loss function, solves the difficult exploration problem in multi-target reinforcement learning, and realizes the collaborative allocation among agents. It also considers the optimal allocation of resources, so as to regroup agents and targets in a timely manner and realize the efficient training of multiple agents. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments of the present invention. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0037] Figure 1 It is a flowchart of a first multi-agent multi-objective exploration method based on dynamic grouping provided by an embodiment of the present invention;

[0038] Figure 2 is a flow chart of a second multi-agent multi-objective exploration method based on dynamic grouping provided by an embodiment of the present invention;

[0039] Figure 3 is a flowchart of a third multi-agent multi-objective exploration method based on dynamic grouping provided by an embodiment of the present invention;

[0040] Figure 4 is a schematic diagram of a multi-agent multi-objective exploration method based on dynamic grouping provided by an embodiment of the present invention;

[0041] Figure 5 is a schematic diagram of another multi-agent multi-objective exploration method based on dynamic grouping provided by an embodiment of the present invention;

[0042] Figure 6 is a flowchart of a fourth multi-agent multi-objective exploration method based on dynamic grouping provided by an embodiment of the present invention;

[0043] Figure 7 is a flowchart of a fifth multi-agent multi-objective exploration method based on dynamic grouping provided by an embodiment of the present invention;

[0044] Figure 8 is a flowchart of a sixth multi-agent multi-objective exploration method based on dynamic grouping provided by an embodiment of the present invention;

[0045] Fig. 9 is a schematic diagram of another multi-agent multi-objective exploration method based on dynamic grouping provided by an embodiment of the present invention;

[0046] Fig.10 It is a schematic diagram of the architecture of a multi-agent multi-objective exploration device based on dynamic grouping provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0048] Unless the context requires otherwise, throughout the specification and claims, the term "including" is to be interpreted as open inclusion, that is, "including, but not limited to". In the description of the specification, the terms "one embodiment", "some embodiments", "exemplary embodiments", "examples", "specific examples" or "some examples" and the like are intended to indicate that specific features, structures, materials or characteristics associated with the embodiment or example are included in at least one embodiment or example of the present disclosure. The schematic representation of the above terms does not necessarily refer to the same embodiment or example. In addition, the specific features, structures, materials or characteristics may be included in any one or more embodiments or examples in any appropriate manner, that is, although they may be carried in the embodiments or examples of the above terms due to reasons such as the order and position of appearance, it is not limited to that they can be carried in combination by one embodiment or example.

[0049] In the description of the present invention, the terms "first" and "second" are used only for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present disclosure, unless otherwise specified, the meaning of "multiple" is two or more. In addition, for example, the same type of nouns may be described as two independent individuals by adding "A" and "B" at the end. In this case, the corresponding features defined as "A" and "B" are only used to distinguish the same type of individuals for description purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features.

[0050] In the description of the present invention, the expression "A and / or B" (where A and B are used to formally represent specific characteristic contents) will be involved, and the corresponding expressions include the following three combinations: only A, only B, and a combination of A and B.

[0051] As used herein, "about," "substantially," or "approximately" includes the stated value and an average value that is within an acceptable range of deviation from the particular value as determined by one of ordinary skill in the art taking into account the measurements in question and the errors associated with the measurement of the particular quantity (i.e., the limitations of the measurement system).

[0052] In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0053] Embodiment 1:

[0054] Embodiment 1 of the present invention provides a multi-agent multi-objective exploration method based on dynamic grouping, such as Figure 1 As shown, including:

[0055] In step 201, agents are reallocated to each target according to the loss function at every preset number of rounds; wherein multiple agents allocated to one target form an agent group; the preset number is obtained by those skilled in the art based on empirical analysis. In order to realize agent collaborative exploration in a multi-target reinforcement learning environment, it is necessary to group agents and targets, allocate agents (make some agents as professional agents of the target); and in order to rationalize the allocation, a loss function is constructed and minimized; when a target reaches the final state, the agent allocated to the target is released so that it can serve other targets.

[0056] In addition, in order to rationalize the use of resources and prevent a certain target from occupying a certain agent for a long time, it is necessary to regroup the agent and the target, and release the agent occupied by the target so that it can be used for exploration by other agents. At the same time, in order not to make the grouping process occupy too much computing resources, it is planned to regroup once every M rounds.

[0057] In step 202, the target strategy of each agent is determined according to the goal of each agent grouping, and each agent is trained for multiple rounds using the target strategy.

[0058] In a specific implementation, the target strategy of each agent is determined according to the target of each agent group, which can be: setting a corresponding strategy for each target in advance, and taking the strategy of the target of the agent group to which the agent belongs as the target strategy of the agent.

[0059] This embodiment dynamically groups agents and targets (i.e., allocates agents) based on the constructed loss function, solves the difficult exploration problem in multi-target reinforcement learning, and realizes the collaborative allocation among agents. It also considers the optimal allocation of resources, so as to regroup agents and targets in a timely manner, and realizes efficient training of multiple agents.

[0060] In an optional implementation, the loss function is Where N is the number of targets participating in the allocation, k v is the number of agents required to participate in the vth goal of the assignment, dist vu is the distance between the vth target and the corresponding uth agent involved in the allocation; When dist vu The smaller it is, the closer the agent u is to the target v; x is the horizontal coordinate of the u-th agent, u y is the ordinate of the u-th agent, v x is the horizontal coordinate of the vth target, v yis the ordinate of the vth target. The smaller the value of the loss function, the better the allocation method.

[0061] Considering that in actual use, the number of agents may not completely match the number required by the target, in order to cope with various scenarios, this embodiment also provides a preferred implementation method, that is, according to the loss function, the agents are reallocated to each target, such as Figure 2 As shown, specifically including:

[0062] In step 301, when the total number of agents When the total number of agents can meet the needs of all unfinished targets, all unfinished targets are used as targets for allocation, and the targets for allocation are allocated agents to minimize the value of the loss function.

[0063] In step 302, when the total number of agents When the total number of agents cannot meet the needs of all unfinished targets, w targets are selected from all unfinished targets as targets to be allocated, and agents to minimize the value of the loss function; wherein w can be predetermined by those skilled in the art, or a target selection strategy can be predetermined by those skilled in the art, and w targets are selected according to the target selection strategy.

[0064] Among them, K i is the target of the i-th participant in the allocation, K v is the number of agents required for the vth unfinished target, and M is the number of unfinished targets. The unfinished target means that the target has not reached the required position, that is, the agent has not reached the required position of the target.

[0065] There is another situation: P <K v , that is, the total number of agents is less than the number of agents required for the vth unfinished goal. At this time, the vth unfinished goal does not participate in the allocation of agents; when P <min(K v ), that is, when the total number of agents is less than the number of agents required for each goal, no allocation of agents is performed and the exploration task cannot be completed.

[0066] Among them, the above method selects w targets from all unfinished targets as targets to be allocated, and allocates agents to minimize the value of the loss function, such as Figure 3 As shown, specifically including:

[0067] In step 401, multiple combinations are taken out from the unfinished goals; wherein, the number of goals included in each combination is the maximum number corresponding to the required number of agents less than or equal to P; wherein, P is the total number of agents; the number of goals included in each combination is the maximum number corresponding to the required number of agents less than or equal to P, that is, the total number of agents required for all goals in the combination is exactly less than or equal to P, and when any other goal is added to the combination, the number of agents required in the combination exceeds P.

[0068] In step 402, with each combination as the goals participating in the allocation, calculate the values of the loss function under different agent allocation methods; the agent allocation method is which agents are allocated to which goals.

[0069] In step 403, use the combination corresponding to the minimum loss function value as the goals participating in the allocation, and use the agent allocation method corresponding to the minimum loss function value to allocate agents; step 403 can be understood as selecting the one that minimizes the value of the loss function among all possible combinations and all agent allocation methods under all combinations.

[0070] In step 404, alternatively, use the combination with the largest number of goals as the goals participating in the allocation, and use the agent allocation method corresponding to the minimum loss function value of the goals participating in the allocation to allocate agents. Step 404 can be understood as first determining the combination that can explore the most goals, and then determining the optimal agent allocation method under this combination.

[0071] In a specific application scenario, that is, when the number of agents required for each unfinished goal is the same, wherein, w is the number of goals participating in the allocation, k is the number of agents required for each unfinished goal, and P % k represents the remainder of P divided by k, represents rounding down.

[0072] Here is also an example in one case, that is, the number of agents required for all goals is the same and is k, then reallocating agents to each goal according to the loss function specifically includes:

[0073] If P < k, the exploration task will not be able to be completed.

[0074] If P ≥ k × N, the agents required for the goals are sufficient. At this time, the loss function for selecting k × N agents for allocation is Minimize the loss function to allocate k agents to each agent v to complete the grouping of agents and goals; when a certain goal (assumed to be numbered N) reaches the final state, release the agents allocated to this goal, and minimize Allocate \(k\times(N - 1)\) agents to the remaining \(N - 1\) targets.

[0075] If \(k\lt P\lt k\times N\), then not enough agents can be allocated to all targets. In this case, only \( targets can be allocated, and \(k\times w\) agents are selected for allocation; the loss function is Minimize the loss function by allocating \(k\) agents to each agent \(v\) to complete the grouping of agents and targets; every \(M\) rounds, release the agents allocated to all targets and regroup. If a target reaches the required position, then allocate \(k\times w'\) agents to \(w'\) of the remaining \(N - 1\) targets; minimize Allocate \(k\times(N - 1)\) agents to the remaining \(N - 1\) targets.

[0076] In a specific application scenario, the multi-round training of each agent using the target policy specifically includes:

[0077] In each round, reset the environmental state to the initial state and obtain the initial joint observations of all agents. In each time step of a round, the agents use the target policy to interact with the environment to obtain the transition tuples corresponding to the time steps, and store each transition tuple to obtain an experience replay data set; among them, the target policy is used to train the agents to access the targets and collect data to obtain the maximum environmental reward.

[0078] In actual implementation, according to needs and necessity, the interaction process between the agents and the environment is completed through reinforcement learning. The specifically selected reinforcement learning method: such as Figure 4 Online reinforcement learning (English full name: On-line RL), in this method, the agents and the environment are always interacting, and as the interaction progresses, the data set is continuously expanded; Figure 5 On-policy reinforcement learning (English full name: On-policy RL), in this method, the sample data comes from the current policy and each sample data is used only once.

[0079] Among them, each round includes at least one time step. Using the initial joint observations of the agents in a round as the joint observations in the first time step, the interaction in the first time step is carried out.

[0080] Among them, in each time step of a round, the agents use the target policy to interact with the environment to obtain the transition tuples corresponding to the time steps, as Figure 6 shown, specifically including:

[0081] In step 501, at time step \(t\), use the current state as the starting state \(s\) t , and each agent according to the starting state \(s\) tSelect the action to be performed at the current time step according to the target strategy, and combine the actions selected by all agents into a joint action

[0082] In step 502, a joint action a is performed. t , the state after the joint action is executed is taken as the successor state s t+1 , and the agent receives the environmental reward r t and the joint observation o at the next time step t+1 .

[0083] In step 503, the initial state s of the current time step is t , the joint observation o at the current time step t , joint action a t , successor state s t+1 , the joint observation o at the next time step t+1 and environmental rewards t The transition tuple (s) that constitutes the current time step t ,o t ,a t ,s t+1 ,o t+1 ,r t ).

[0084] After performing the joint action, each agent observes the environment to obtain the local observation of the agent at the next time step, and the local observations of all agents at the next time step constitute the joint observation of the next time step. t11 .

[0085] Embodiment 2:

[0086] The present invention is based on the method described in Example 1, combined with specific application scenarios, and uses technical descriptions in related scenarios to illustrate the implementation process of the present invention in characteristic scenarios.

[0087] This embodiment takes a multi-agent multi-objective reinforcement learning environment (P≥N) with P agents and N objectives of size L×L as an example, and each objective requires k agents to collaborate to complete the task of the objective.

[0088] The multi-agent multi-objective exploration method based on dynamic grouping provided in this embodiment is as follows: Figure 7 As shown, specifically including:

[0089] In step 601, the agent and the target are grouped: a loss function is constructed, and the loss function is minimized to group the agent and the target. When a certain target reaches the final state, the agent and the target are regrouped.

[0090] In order to achieve collaborative exploration of intelligent agents in a multi-objective reinforcement learning environment, it is necessary to group the agents and objectives and allocate the agents (making several agents serve as professional agents for the objectives). In order to rationalize the allocation, a loss function is constructed and minimized. When a certain objective reaches the final state, the agent allocated to that objective is released so that it can serve other objectives.

[0091] In step 602, in the grouping state, the agent obtains the transfer tuple based on the interaction between the target strategy and the environment. After storage, a data set is obtained; wherein, the agent and the target are regrouped every M rounds.

[0092] In order to train multiple agents, it is first necessary to obtain sample data for training. In this embodiment, the agent selects actions to interact with the environment according to the target strategy, and obtains the subsequent state of the environment state and the environment reward after the agent performs different actions to form a transition tuple. By storing the transition tuples of the agent at different time steps, the data set can be obtained.

[0093] In actual implementation, the interaction between the agent and the environment is completed through reinforcement learning as needed and necessary. The specific reinforcement learning method used is: Figure 4 Online reinforcement learning (On-line RL), in which the agent and the environment are constantly interacting, and as the interaction proceeds, the dataset is continuously expanded; Figure 5 This is On-policy RL. In this method, the sample data comes from the current policy and each sample data is used only once.

[0094] In step 603, the data set is updated to complete the training of the agent.

[0095] In specific implementation, the data set is updated through continuous interaction between the agent and the environment, so as to obtain the best convergence strategy for the agent to visit the terminal state from the initial state - obtaining the maximum cumulative discounted reward.

[0096] This embodiment uses the method described in steps 601 to 603 to construct a loss function and minimize the loss function to dynamically group the agents and the targets. In the grouped state, multiple agents are trained as a whole and collaborative training of the agents is achieved by continuously updating the data set, thereby solving the problem of the difficulty in allocating collaborative exploration of agents in multi-objective reinforcement learning.

[0097] The construction of the loss function and minimization of the loss function to group the agent and the target specifically include:

[0098] The environmental state is the positions of all agents and targets, temporarily represented by (x, y), where 0 ≤ x < L and 0 ≤ y < L. Then, the distance between an agent u, 1 ≤ u ≤ P and a target v, 1 ≤ v ≤ N can be expressed as The sum of distances of the k agents assigned to target v is

[0099] Using the sum of the distances from the agents to the targets as the loss function l.

[0100] If P < k, the exploration task cannot be completed.

[0101] If P ≥ k × N, there are enough agents for the targets. At this time, the loss function for selecting k × N agents for allocation is Minimizing the loss function assigns k agents to each agent v to complete the grouping of agents and targets; when a certain target (assumed to be numbered N) reaches the final state, release the agents assigned to this target, and minimize Assign k × (N - 1) agents to the remaining N - 1 targets.

[0102] If k < P < k × N, not enough agents can be assigned to all targets. At this time, only w targets can be assigned, and k × w agents are selected for allocation; the loss function is Minimizing the loss function assigns k agents to each agent v to complete the grouping of agents and targets; every M rounds, release the agents assigned to all targets and regroup. If a certain target reaches the required position, then assign k × w' agents to w' targets among the remaining N - 1 targets; minimize Assign k × (N - 1) agents to the remaining N - 1 targets.

[0103] Every M (i.e., the preset quantity in Embodiment 1) rounds, the regrouping of agents and targets is carried out to rationalize the utilization of resources, prevent a certain target from occupying a certain agent for a long time, so the regrouping of agents and targets is required to release the agent occupied by this target for use in the exploration of other agents. At the same time, in order not to make the regrouping process occupy too much computing resources, it is stipulated that every M rounds, a regrouping is carried out.

[0104] When training, first initialize the target policy π of each agent i ; then, the overall policy of all agents That is, each agent has a target policy. For multiple agents, the entire algorithm is regarded as the joint policy of all agents, that is, the overall policy.

[0105] After completing the target strategy initialization, it is also necessary to complete the interaction between the agent and the environment based on the overall strategy (that is, each agent uses the target strategy to interact with the environment), and use the transfer tuples obtained from the interaction to update the data set D. Figure 8 As shown, each transfer tuple can be obtained through the interaction between the overall strategy and the environment in the following way:

[0106] In step 701, at the beginning of each round, the environment state is reset to the initial state and the joint observations of all agents are obtained.

[0107] When exploring, the agent and the environment need to interact for multiple rounds, each round is represented by i, i∈[1...E]. When entering each round i, the environment state needs to be reset to s first. 1 , and observe the first state o when the environment is initialized 1 .

[0108] At the same time, each agent observes the initial state to obtain the local observation of the agent, and the local observations of all agents are combined into the initial joint observation. The local observation obtained by the jth agent is expressed as The first joint observation of n agents is expressed as

[0109] In step 702, at each time step, each agent selects its own action according to the target strategy and the global state, and the actions selected by all agents are combined into a joint action.

[0110] In the specific implementation, each round includes at least one time step, each time step is denoted as t, t∈(1,2…H), and the agent performs an action in each time step.

[0111] At each time step t, t∈(1,2…H), according to the global state s t , each agent uses the target strategy to select its own actions to form a joint action. The yth agent selects actions according to the target strategy and is expressed as The joint action consisting of the actions selected by n agents is expressed as

[0112] In step 703, after executing the joint action at each time step, the starting state is transformed into the successor state (i.e., the state of the agent after executing the joint action is taken as the successor state), and the agent obtains the environmental reward and the next joint observation.

[0113] At each time t, before executing the joint action, the global state of the environment is s at the current time step t , the joint observation o at the current time step tAfter executing the joint action, the global state of the environment changes to the successor state s t+1 , all agents receive a total environmental reward r t , the joint observation of all agents is transformed into the next joint observation o t+1 .

[0114] In step 704, the starting state, joint observation, joint action, successor state, next joint observation and environmental reward of each time step are combined into a transfer tuple of the current time step, which is stored to obtain an experience replay data set.

[0115] After obtaining the global state and joint observation before and after the joint action is executed, as well as the corresponding environmental rewards, the corresponding parameters can be combined into the transfer tuple of the time step. The transfer tuple can be expressed as (s t ,o t ,a t ,s t+1 ,o t+1 ,r t ).

[0116] After completing the interactions of all time steps in all rounds through steps 701 to 704 provided in this embodiment, the transfer tuples in each time step can be obtained, and all transfer tuples are stored to obtain the data set D.

[0117] After obtaining the data set D, the agent's exploration strategy from the initial state to the terminal state can be determined; in subsequent training, new transfer tuples are continuously obtained and the data set D is updated to complete the strategy training.

[0118] In the above process, the multi-agent multi-objective exploration method based on dynamic grouping includes: grouping agents and objectives based on minimizing the constructed loss function; in the grouping state, the agents obtain data sets based on strategy interaction with the environment; and continuously updating the data sets to complete training.

[0119] In a specific scenario, the complete training process of multi-agent is as follows: Fig. 9 As shown in Figure 2, the training process of each agent includes: grouping the agent with the target and updating the data set. When the agent interacts with the environment, at each time step t, the agent performs an action a. t , so that the environment state changes from the initial state s t Change to the successor state s t+1 , and obtain the corresponding reward value r t In the next time step, the successor state s in the previous time step is t+1 As the starting state s of this time step tAfter each round, consider whether to regroup the agents and targets, and use the updated experience pool dataset D to train the target strategy π until the multi-agent training is completed.

[0120] This embodiment dynamically groups agents and targets based on the constructed loss function, solves the difficult exploration problem in multi-objective reinforcement learning and the difficult problem of difficult allocation of agent collaborative exploration in a multi-objective reinforcement learning environment, and realizes collaborative allocation among agents. It also considers the optimal allocation of resources, and realizes efficient training of multiple agents by regrouping agents and targets in a timely manner, thereby realizing collaborative exploration among multiple agents and optimization of the overall system.

[0121] Embodiment 3:

[0122] like Fig.10 , is a schematic diagram of the architecture of a multi-agent multi-objective exploration device based on dynamic grouping according to an embodiment of the present invention. The multi-agent multi-objective exploration device based on dynamic grouping according to this embodiment includes one or more processors 21 and a memory 22. Fig.10 A processor 21 is taken as an example.

[0123] The processor 21 and the memory 22 may be connected via a bus or other means. Fig.10 The example of connecting through bus is taken in the following.

[0124] The memory 22 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs and non-volatile computer executable programs, such as the multi-agent multi-objective exploration method based on dynamic grouping in Example 1. The processor 21 executes the multi-agent multi-objective exploration method based on dynamic grouping by running the non-volatile software programs and instructions stored in the memory 22.

[0125] The memory 22 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage devices. In some embodiments, the memory 22 may optionally include a memory remotely arranged relative to the processor 21, and these remote memories may be connected to the processor 21 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0126] The program instructions / modules are stored in the memory 22, and when executed by the one or more processors 21, the multi-agent multi-objective exploration method based on dynamic grouping in the above-mentioned embodiment 1 is executed.

[0127] It is worth noting that the information interaction, execution process, etc. between the modules and units within the above-mentioned devices and systems are based on the same concept as the processing method embodiment of the present invention. The specific contents can be found in the description of the method embodiment of the present invention and will not be repeated here.

[0128] A person skilled in the art may understand that all or part of the steps in the various methods of the embodiments may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the storage medium may include: a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc.

[0129] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A multi-agent multi-objective exploration method based on dynamic grouping, characterized in that: include: After a preset number of rounds, agents are reallocated to each target based on the loss function; multiple agents assigned to one target form an agent group; According to the goal of each agent grouping, the target strategy of each agent is determined, and each agent is trained for multiple rounds using the target strategy.

2. The multi-agent multi-objective exploration method based on dynamic grouping according to claim 1 is characterized in that: The loss function is Where N is the number of targets participating in the allocation, k v is the number of agents required to participate in the vth goal of the assignment, dist vu is the distance between the vth target and the corresponding uth agent involved in the allocation; u x is the horizontal coordinate of the u-th agent, u y is the ordinate of the u-th agent, v x is the horizontal coordinate of the vth target, v y is the vertical coordinate of the vth target.

3. The multi-agent multi-objective exploration method based on dynamic grouping according to claim 1 is characterized in that: The reallocation of agents to each target according to the loss function specifically includes: When the total number of agents When all unfinished targets are used as targets for allocation, agents to minimize the value of the loss function; When the total number of agents , then from all unfinished goals, w goals are selected as the goals to be allocated, and the goals to be allocated are allocated agents to minimize the value of the loss function; Among them, K i is the target of the i-th participant in the allocation, K v is the number of agents required for the vth unfinished goal, and M is the number of unfinished goals.

4. The multi-agent multi-objective exploration method based on dynamic grouping according to claim 1 is characterized in that: The method selects w targets from all unfinished targets as targets to be allocated, and allocates Agents, minimize the value of the loss function, including: From the unfinished goals, take out multiple combinations; the number of goals contained in each combination is the maximum number of the required number of agents less than or equal to P; where P is the total number of agents; Take each combination as the target of participation in allocation and calculate the value of the loss function under different agent allocation methods; Use the combination corresponding to the minimum loss function value as the target for participation in allocation, and use the agent allocation method corresponding to the minimum loss function value to allocate agents; Alternatively, the combination containing the largest number of targets is used as the target for allocation, and the agents are allocated using the agent allocation method with the smallest loss function value corresponding to the targets for allocation.

5. The multi-agent multi-objective exploration method based on dynamic grouping according to claim 4 is characterized in that: When the number of agents required for each unfinished goal is the same, Where w is the number of goals to be allocated, k is the number of agents required for each uncompleted goal, and P%k represents the remainder of P divided by k. Represents round down.

6. The multi-agent multi-objective exploration method based on dynamic grouping according to claim 1 is characterized in that: The target strategy is used to train each agent for multiple rounds, specifically including: In each round, the environment state is reset to the initial state, and the initial joint observation of all agents is obtained. In each time step of a round, the agent uses the target strategy to interact with the environment to obtain the transfer tuple of the corresponding time step, and each transfer tuple is stored to obtain the experience replay data set; Each round includes at least one time step, and the initial joint observation of the agents in one round is used as the joint observation of the first time step to perform interactions within the first time step.

7. The multi-agent multi-objective exploration method based on dynamic grouping according to claim 6 is characterized in that: In each time step of a round, the agent uses the target strategy to interact with the environment to obtain the transfer tuple of the corresponding time step, which specifically includes: At time step t, the current state is used as the starting state s t , each agent has a starting state s t Select the action to be performed at the current time step according to the target strategy, and combine the actions selected by all agents into a joint action Perform joint action a t , the state after the joint action is executed is taken as the successor state s t+1 , and the agent receives the environmental reward r t and the joint observation o at the next time step t+1 ; Set the starting state s of the current time step to t , the joint observation o at the current time step t , joint action a t , successor state s t+1 , the joint observation o at the next time step t+1 and environmental rewards t The transition tuple (s) that constitutes the current time step t ,o t ,a t ,s t+1 ,o t+1 ,r t ).

8. The multi-agent multi-objective exploration method based on dynamic grouping according to claim 7 is characterized in that: After performing the joint action, each agent observes the environment to obtain the local observation of the agent at the next time step. The local observations of all agents at the next time step constitute the joint observation of the next time step. t+1 .

9. A non-volatile computer storage medium, characterized in that: The computer storage medium stores computer executable instructions, which are executed by one or more processors to complete the multi-agent multi-objective exploration method based on dynamic grouping as described in any one of claims 1-8.

10. A multi-agent multi-objective exploration device based on dynamic grouping, characterized in that: include: at least one processor; And, a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the processor to execute the multi-agent multi-objective exploration method based on dynamic grouping as described in any one of claims 1-8.

Images