Multi-agent cooperation method based on general cooperation knowledge
Through the multi-task learning method, agents share cooperative decision-making knowledge in different task fields, solving the problem that multi-agent reinforcement learning collaboration methods in the existing technology are difficult to apply in different environments, realizing collaboration and coordination between agents, and reducing computing resource requirements.
Patent Information
- Application Number
- CN202411655649.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-19
- Publication Date
- 2025-03-07
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
The existing multi-agent reinforcement learning collaboration methods are difficult to apply in different environments, and training multi-agent reinforcement learning networks requires a large amount of computing resources.
A multi-task learning method is adopted to simultaneously learn the common cooperative decision-making knowledge of agents in multiple different tasks through a policy network, and apply this knowledge to other cooperative tasks in the same or different fields to promote collaboration and coordination among agents.
Cooperation and coordination among agents in different task areas is realized, reducing the computing resource requirements for training multi-agent reinforcement learning networks.
Smart Images
Figure CN119578452A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of reinforcement learning, and in particular relates to a multi-agent collaboration method based on general collaboration knowledge. Background Art
[0002] Multi-agent reinforcement learning is an important research direction in multi-agent systems, and the collaboration problem between agents is a key research issue in multi-agent reinforcement learning. Most of the existing multi-agent reinforcement learning collaboration methods focus on the collaboration problem in one field, and do not consider applying the collaboration algorithm to different fields, such as applying the collaboration algorithm in the StarCraft SMAC environment to the Google Football GRF environment.
[0003] However, training a multi-agent reinforcement learning network is difficult and requires a lot of computing resources to interact with different environmental variables, extract state representations, and acquire decision-making knowledge.
[0004] Therefore, in order to solve the above-mentioned training difficulty problem, it is urgent to propose a multi-agent collaboration method based on general cooperation knowledge. Summary of the invention
[0005] To solve the above technical problems, the present invention proposes a multi-agent collaboration method based on general cooperation knowledge, which adopts a multi-task learning method to learn multiple tasks at the same time and then obtain a task-sharing strategy network. The purpose is to use a strategy network to learn and extract the common cooperation decision-making knowledge of agents in multiple different tasks, and then apply the extracted knowledge to other cooperation tasks in the same or different fields, so as to promote collaboration and coordination between agents and solve the problems existing in the above-mentioned prior art.
[0006] To achieve the above object, the present invention provides a multi-agent collaboration method based on general cooperation knowledge, comprising the following steps:
[0007] Determine the mission area and select a number of cooperative missions;
[0008] Constructing a multi-task training framework, the training framework including a perception network and a decision network set for each cooperative task, the decision network including a shared decision layer and a task-specific decision layer;
[0009] Based on each cooperative task, the multi-task training framework is trained, and based on the trained decision network, a task sharing decision network model is constructed;
[0010] The task sharing decision network model is transferred to a new task domain for training to complete the collaboration of multiple agents in the current task.
[0011] Optionally, the training method of the multi-task training framework adopts a reinforcement learning algorithm VDN.
[0012] Optionally, the structure of the perception network includes two layers of fully connected networks and one layer of LSTM network.
[0013] Optionally, the shared decision layer and the task-specific decision layer each include two layers of fully connected networks.
[0014] Optionally, based on each cooperative task, the process of training the multi-task training framework using a reinforcement learning algorithm includes:
[0015] Assign an equal initial weight to each cooperative task;
[0016] Calculate the loss of each cooperative task;
[0017] Normalize the loss of each cooperative task and calculate the weight of the standardized loss;
[0018] Weight the weights of the standardized loss to obtain the total weighted loss;
[0019] The network parameters are updated based on the total weighted loss.
[0020] Optionally, the process of migrating the task sharing decision network model to a new task domain to complete the multi-agent collaboration in the current task includes:
[0021] Adjusting a multi-task training framework in a new task domain into a single-task training framework, wherein the single-task training framework includes a perception network and a decision network;
[0022] Migrating the task-sharing decision network model to the position of the decision network in the current new task domain;
[0023] When the single-task training framework in a new task domain is trained, the parameters of the task-sharing decision network model are fixed and do not participate in the training, thereby completing the collaboration of multiple agents in the current task.
[0024] Optionally, the multi-task training framework is trained in a cyclic training manner.
[0025] The present invention also provides a multi-agent collaboration system based on general collaboration knowledge, which is used to implement a multi-agent collaboration method based on general collaboration knowledge, including: a framework construction module, a framework training module and a model migration module;
[0026] The framework building module is used to determine the task domain, select several cooperative tasks, set up a perception network and a decision network for each cooperative task, and build a multi-task training framework. The decision network includes a shared decision layer and a task-specific decision layer;
[0027] The framework training module is used to train the multi-task training framework based on each cooperative task using a reinforcement learning algorithm, and to construct a task sharing decision network model based on the trained decision network;
[0028] The model migration module migrates the task sharing decision network model to a new task domain for training to complete the collaboration of multiple agents in the current task.
[0029] The present invention also provides an electronic device, comprising: a memory and a processor; the memory is used to store programs; the processor is used to execute the programs to implement the various steps of the multi-agent collaboration method based on general cooperation knowledge.
[0030] The present invention also provides a readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the various steps of the multi-agent collaboration method based on general cooperation knowledge are implemented.
[0031] Compared with the prior art, the present invention has the following advantages and technical effects:
[0032] The present invention provides an agent collaboration method for multi-agent reinforcement learning, which constructs a multi-task training framework, wherein the training framework includes a perception network and a decision network set for each cooperative task, and the decision network includes a shared decision layer and a task-specific decision layer; based on each cooperative task, the multi-task training framework is trained, and based on the trained decision network, a task-sharing decision network model is constructed; the task-sharing decision network model is migrated to a new task domain for training, and the collaboration of multiple agents in the current task is completed. The present invention adopts a multi-task learning method, and simultaneously learns multiple tasks to obtain a task-sharing strategy network; the common cooperative decision-making knowledge of agents in multiple different tasks can be extracted through a strategy network learning; the learned shared strategy network can be applied to other cooperative tasks in the same or different fields for training, so as to promote collaboration and coordination between agents. BRIEF DESCRIPTION OF THE DRAWINGS
[0033] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:
[0034] Figure 1 A schematic diagram of a multi-agent collaboration method according to an embodiment of the present invention;
[0035] Figure 2 It is a schematic diagram of the overall framework of an embodiment of the present invention;
[0036] Figure 3This is a diagram of the LSTM architecture of an embodiment of the present invention. DETAILED DESCRIPTION
[0037] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.
[0038] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.
[0039] Embodiment 1
[0040] This embodiment provides a multi-agent collaboration method based on general collaboration knowledge, including the following steps:
[0041] Determine the mission area and select a number of cooperative missions;
[0042] Construct a multi-task training framework, including a perception network and a decision network for each cooperative task, where the decision network includes a shared decision layer and a task decision layer;
[0043] Based on each cooperative task, the multi-task training framework is trained, and based on the trained decision network, a task sharing decision network model is constructed;
[0044] The task sharing decision network model is transferred to a new task domain for training to complete the collaboration of multiple agents in the current task.
[0045] like Figure 1 As shown in the figure, the multi-agent collaboration method based on general cooperation knowledge includes the following steps:
[0046] Step 100, preliminarily constructing a multi-task oriented reinforcement learning training framework, i.e., a multi-task training framework;
[0047] The step 100 further comprises the following steps:
[0048] Step 110, determine the task domain. In this embodiment, three cooperative tasks in the StarCraft SMAC environment are selected as pre-training tasks, namely 3s_vs_5z, 2c_vs_64zg, and 5m_vs_6m;
[0049] Step 120, for different tasks in step 110, training is performed in a cyclic manner, and each task is trained in turn in each round;
[0050] Step 130, based on the training method in step 120, selecting the reinforcement learning algorithm VDN as the training algorithm;
[0051] The VDN (Value Decomposition Network) algorithm is an algorithm for multi-agent reinforcement learning (MARL). In multi-agent reinforcement learning, multiple agents need to work together in an environment to complete a specific task. The core idea of the VDN algorithm is to decompose the joint action value function of multiple agents into the sum of the individual value functions of each agent, thereby achieving effective collaborative learning. The VDN algorithm is based on the following principles:
[0052] Decomposition: VDN assumes that the joint action-value function can be decomposed into the sum of the individual value functions of each agent.
[0053] Centralized Training and Decentralized Execution: VDN uses global information to learn individual value functions during training, but in the execution phase, each agent takes actions based only on local observations.
[0054] The key formula of the VDN algorithm is as follows:
[0055] Estimation of the joint value function:
[0056]
[0057] in, It is a joint state. is the joint action of all agents, s i is the local state of the ith agent, a i is the action of the ith agent. Q i (s i ,a i ) is the agent i in state s i Take action a i The individual value function of . is an estimate of the joint value function.
[0058] Loss function:
[0059] Calculate the loss function, usually using the mean squared error (MSE) to measure the difference between the predicted joint value function and the target value:
[0060]
[0061] Where N is the number of experience replay samples, r j is the reward for the jth sample, γ is the discount factor, is the next state.
[0062] optimization:
[0063] Optimize the loss function through the gradient descent method and update the parameters of the value network:
[0064]
[0065] Among them, θ i is the value network parameter of the ith agent, and α is the learning rate.
[0066] Training process:
[0067] 1. Data collection: The agent performs actions in the environment and collects experience data (s, a, R, s').
[0068] 2. Value function update: Use the collected data to calculate the loss function and update the parameters of the value network through gradient descent.
[0069] 3. Policy update: Usually a value-based method (such as Q-learning) is used to update the policy.
[0070] The VDN algorithm simplifies the multi-agent learning problem through this decomposition method, allowing each agent to learn independently without considering the strategies or value functions of other agents. This is very useful when dealing with large multi-agent systems because it can reduce the complexity of learning.
[0071] Step 200, refine the training framework and set up the perception network and decision network;
[0072] The step 200 further comprises the following steps:
[0073] Step 210, according to the number of tasks selected in step 110, the number of perception networks and decision networks is set. Since the observation space of each task may be inconsistent, each task needs to have its own specific perception network. Similarly, the action space of each task is also different, so each task needs to have a specific decision network.
[0074] like Figure 2 As shown, Figure 2It is the overall framework, in which the upper part is the network architecture of the pre-training part. Each task corresponds to its own perception layer network, which is called a task-specific perception network. The decision network follows. Due to the needs of multi-task training, the decision network is divided into two parts. The first part is a shared decision network shared by all tasks, and the second part is a task-specific decision network. Each task has its own unique decision network.
[0075] Step 220, set the structure of the perception network, the perception network is a two-layer fully connected network plus an LSTM network; the LSTM architecture is as follows Figure 3 , the state processing flow of the entire perception layer is:
[0076] Get the initial state:
[0077] Initialize the environment to get a state s t ;
[0078] Feature extraction:
[0079] Secondly, feature extraction is performed based on the state St. t Input the first fully connected layer, and the output feature after one activation layer is F' t =ReLU(Dense1(s t )), where Dense1 is the weight and bias of the first fully connected layer and ReLU is the activation function.
[0080] The feature F' obtained after processing the first fully connected layer t Input the second fully connected layer, and the output feature after one activation layer is F″ t =ReLU(Dense2(F' t )), where Dense2 is the weight and bias of the second fully connected layer, and ReLU is the activation function.
[0081] Status indicates:
[0082] like Figure 3 As shown, the core calculation process of LSTM is as follows:
[0083] Forget gate: f t =σ(W f ·[h t-1 ,F″ t ]+b f )
[0084] Input gate: i t =σ(W i ·[h t-1 ,F″ t ]+b i )
[0085] Output gate: o t =σ(W o ·[h t-1 ,F″ t ]+b o )
[0086] New cell state candidate values:
[0087] Update cell status:
[0088] Output hidden state: h t =o t ·tanh(c t )
[0089] Among them, W f ,W i ,W o ,W c are the weight matrices of the forget gate, input gate, output gate, and cell state, respectively, and b f ,b i ,b o ,b c is the corresponding bias term, σ is the sigmoid activation function, tanh is the hyperbolic tangent activation function, and h t-1 is the hidden state of the previous time step, c t-1 is the cell state at the previous time step.
[0090] Output:
[0091] The hidden state h of LSTM t Pass to the decision-making layer.
[0092] Step 230, setting the structure of the decision network, dividing the decision network into two parts;
[0093] like Figure 2 As shown in the figure, the decision layer is divided into a shared decision layer and a task-specific decision layer. The shared decision layer consists of two layers of fully connected networks, and the task-specific decision layer also consists of two layers of fully connected networks. The input dimensions of these four layers of fully connected networks are 256, 256, 128, and 64 respectively. The decision layer takes the output of the perception layer as input, and finally obtains an output after being processed by the neural network.
[0094] Step 300, pre-training to obtain a task-sharing decision network;
[0095] The step 300 further comprises the following steps:
[0096] Step 310, during the pre-training process, the task-specific network will only be trained for the specific task, and the shared decision network will be trained for every task. Figure 2 As shown, Figure 2 The upper part is the pre-training framework. Each task has its own perception layer and the final decision layer, while the shared decision layer in the middle is shared by all tasks. During training, each task will optimize this shared decision network.
[0097] Step 320, in order to make the network learn more smoothly in multiple tasks, a method of dynamically adjusting the loss weight based on the loss value is adopted, wherein tasks with larger losses are assigned smaller weights, thereby reducing their impact on the total loss;
[0098] The basic steps are as follows:
[0099] 1. At the initial moment, each task is assigned an initial weight, which is equal to 1.0.
[0100] 2. Calculate the loss for each task:
[0101] losses=[loss_task1, loss_task2,..., loss_taskN];
[0102] 3. Calculate the normalization and weight of the loss: Normalize the loss of each task and calculate the weight. Here, the inverse of the loss can be used as the weight, but in order to avoid the weight becoming infinite, the following formula can be used:
[0103] normalized_losses=[loss / sum(losses)for loss in losses];
[0104] weights=[1 / (loss+epsilon)for loss in normalized_losses];
[0105] Here, epsilon is a small positive number used to avoid division by zero.
[0106] 4. Calculate weighted loss:
[0107] Compute the weighted total loss using the calculated weights:
[0108] weighted_losses=[w**lfor w, l in zip(weights, losses)],
[0109] total_weighted_loss=sum(weighted_losses);
[0110] Update network parameters.
[0111] Use the calculated total weighted loss to update the parameters of the network:
[0112] total_weighted_loss.backward(),
[0113] optimizer.step().
[0114] The principle of this method is based on the following considerations:
[0115] Balanced influence: If the loss of one task is much larger than other tasks, it will dominate the gradient update, which may cause the learning of other tasks to be neglected. By reducing the weight of large losses, the influence of different tasks on the learning process can be balanced.
[0116] Avoid extreme cases: When the loss is very large, directly using the inverse of the loss may cause the weights to become very large, which may cause numerical instability. This can be avoided by using normalization of the loss and adding a small positive epsilon.
[0117] Dynamic Adjustment: As training progresses, the losses for different tasks change, and dynamically adjusting weights ensures that the network continues to focus on all tasks.
[0118] Step 330: After training, a shared decision network is obtained and the network is saved for subsequent migration.
[0119] Based on the three tasks selected in step 110, the models finally saved include a model of a shared decision network and three task-specific decision network models. The saved models include the parameters of each layer of the neural network. Since the state of each task is different, the perception network is not universal, so the perception network is not saved.
[0120] Step 400, migrating the pre-trained task sharing decision network model to the new cooperative task;
[0121] The step 400 further includes the following steps:
[0122] Step 410: Based on the multi-task training framework in step 100, modify it into a single-task training framework. Figure 2 As shown in the lower middle part, there is a perception layer. In order to be consistent with the pre-training framework, the decision layer is also divided into two parts.
[0123] Step 420, such as Figure 2 As shown, based on the training framework in step 410, the perception layer network parameters and the second half decision layer network parameters are initialized before training, the shared decision network model obtained based on step 330 is migrated to the position of the first half decision network of the current new task, and the parameters of the shared decision network model are fixed and do not participate in training, so as to promote collaboration between intelligent agents in the current task.
[0124] This embodiment provides an agent collaboration method for multi-agent reinforcement learning, which adopts a multi-task learning method to learn multiple tasks at the same time and then obtain a task-sharing policy network; the common cooperative decision-making knowledge of agents in multiple different tasks can be extracted through a policy network learning; the learned shared policy network can be applied to other cooperative tasks in the same or different fields for training, so as to achieve the purpose of promoting collaboration and coordination between agents.
[0125] This embodiment also provides a multi-agent collaboration system based on universal collaboration knowledge, which is used to implement a multi-agent collaboration method based on universal collaboration knowledge. The use method of the system can be mutually referenced with the collaboration method provided in the above embodiment in terms of effect, including: a framework construction module, a framework training module and a model migration module;
[0126] The framework construction module is used to determine the task domain, select several cooperative tasks, set up a perception network and a decision network for each cooperative task, and build a multi-task training framework. The decision network includes a shared decision layer and a task-specific decision layer.
[0127] The framework training module is used to train the multi-task training framework based on each cooperative task using a reinforcement learning algorithm, and to construct a task sharing decision network model based on the trained decision network;
[0128] The model transfer module transfers the task sharing decision network model to a new task domain for training to complete the multi-agent collaboration in the current task.
[0129] Embodiment 2
[0130] The present invention also provides an electronic device, comprising: a memory and a processor; the memory is used to store programs; the processor is used to execute the programs to implement the various steps of the multi-agent collaboration method based on general cooperation knowledge.
[0131] Embodiment 3
[0132] The present invention also provides a readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the various steps of the multi-agent collaboration method based on general cooperation knowledge are implemented.
[0133] The above are only preferred specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.
Claims
1. A multi-agent collaboration method based on general cooperation knowledge, characterized in that: The following steps are involved: Determine the mission area and select a number of cooperative missions; Construct a multi-task training framework, including a perception network and a decision network for each cooperative task, where the decision network includes a shared decision layer and a task decision layer; Based on each cooperative task, the multi-task training framework is trained, and based on the trained decision network, a task sharing decision network model is constructed; The task sharing decision network model is transferred to a new task domain for training to complete the multi-agent collaboration in the current task.
2. The method according to claim 1, characterized in that The training method of the multi-task training framework adopts the reinforcement learning algorithm VDN.
3. The method according to claim 1, characterized in that The structure of the perception network includes two layers of fully connected networks and one layer of LSTM network.
4. The method according to claim 1, characterized in that: The shared decision layer and the task-specific decision layer each include two layers of fully connected networks.
5. The method according to claim 1, characterized in that Based on each cooperative task, the process of training the multi-task training framework using a reinforcement learning algorithm includes: Assign an equal initial weight to each cooperative task; Calculate the loss of each cooperative task; Normalize the loss of each cooperative task and calculate the weight of the standardized loss; Weight the weights of the standardized loss to obtain the total weighted loss; The network parameters are updated based on the total weighted loss.
6. The method according to claim 1, characterized in that The process of migrating the task sharing decision network model to a new task domain and completing the multi-agent collaboration in the current task includes: Adjusting a multi-task training framework in a new task domain into a single-task training framework, wherein the single-task training framework includes a perception network and a decision network; Migrating the task-sharing decision network model to the position of the decision network in the current new task domain; When the single-task training framework in the new task domain is trained, the parameters of the task sharing decision network model are fixed and do not participate in the training, thereby completing the collaboration of multiple agents in the new task domain.
7. The method according to claim 5, characterized in that The multi-task training framework is trained in a cyclical manner.
8. A multi-agent collaboration system based on general collaboration knowledge, characterized in that: A method for implementing a multi-agent collaboration method based on general collaboration knowledge as described in any one of claims 1 to 7, comprising: a framework building module, a framework training module and a model migration module; The framework construction module is used to determine the task domain, select several cooperative tasks, set up a perception network and a decision network for each cooperative task, and build a multi-task training framework. The decision network includes a shared decision layer and a task decision layer. The framework training module is used to train the multi-task training framework based on each cooperative task using a reinforcement learning algorithm, and to construct a task sharing decision network model based on the trained decision network; The model migration module migrates the task sharing decision network model to a new task domain for training to complete the collaboration of multiple agents in the current task.
9. An electronic device, characterized in that: include: Memory and processor; The memory is used to store programs; The processor is used to execute the program to implement the various steps of the multi-agent collaboration method based on general cooperation knowledge as described in any one of claims 1-7.
10. A readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the various steps of the multi-agent collaboration method based on general cooperation knowledge as described in any one of claims 1 to 7 are implemented.