A multi-agent policy model transfer method and system with invariant subtask semantics

Abstract

The present invention discloses a multi-agent strategy model migration method and system with unchanged sub-task semantics. The present invention encodes the multi-agent task to be executed into an executable sub-task through an extensible sub-task encoder, and assigns the sub-task to each agent in executing the multi-agent task. Then, the adaptive action decoder uses the assigned sub-tasks and the observation data of the current agent to calculate the specific action of the agent's interaction with the environment; when the multi-agent task to be executed changes, the extensible sub-task encoder and the adaptive action decoder can ensure that the assigned sub-tasks have consistent and extensible semantics among various multi-agent tasks, and at the same time, the decomposed sub-tasks give the tasks independence, thereby realizing the model migration of the multi-agent strategy model among various multi-agent tasks. The present invention can realize the model migration of the multi-agent strategy model among various multi-agent tasks.

Description

Technical Field

[0001] The present invention belongs to the field of machine learning and intelligent computing, and specifically relates to a model migration method and system for a multi-agent strategy model in a multi-agent game confrontation scenario. Background Art

[0002] With the continuous development of deep learning technology, machine learning has been widely applied in numerous research fields, such as autonomous driving, the Industrial Internet of Things, and intelligent robotics, playing an important role. Multi-Agent Reinforcement Learning (MARL) is a current research hotspot in intelligent computing. It focuses on how to enable multiple agents to collaborate and compete in a shared environment, achieving higher-level decision-making and behavior through interaction and learning. Many works in this field have made significant progress, including value-based algorithms such as the QMIX algorithm and policy-based algorithms such as the MAPPO algorithm. However, these works primarily focus on the asymptotic performance of models in a single task, neglecting their transferability across tasks. Training an optimal model on a single task requires millions of interactions with the environment, and the number of interactions increases further when dealing with large-scale multi-agent systems. Transferring a model trained on a source task to multiple target tasks can reduce training costs by tens of times. As the number of target tasks increases, the training cost reduction achieved by model transfer becomes even more significant.

[0003] In the field of multi-agent reinforcement learning, knowledge reuse is a common approach for model generalization across tasks. Recent knowledge reuse methods for online multi-agent reinforcement learning can be roughly divided into two categories: network design-based methods and task embedding-based methods. Network design-based methods implicitly reuse knowledge extracted from the source task to the target task, leveraging specialized network structures such as Transformers or GNNs to construct a model structure that is general across tasks. However, the knowledge implicitly learned from the source task may not be suitable for the target task, which can lead to inefficient knowledge reuse. Task embedding-based methods achieve knowledge reuse by calculating the similarity between tasks using learned embedding vectors that represent the tasks. However, accurately mapping tasks to task embeddings requires a large number of task samples. Therefore, current knowledge reuse methods for online multi-agent reinforcement learning still have limitations, namely, inefficient knowledge reuse and a reliance on a large number of task samples.

[0004] Currently, Chinese invention patent CN 111695690A discloses a multi-agent adversarial decision-making method based on cooperative reinforcement learning and transfer learning. Its transfer learning method, based on a decay function, allows agents to reuse previous experience with a gradually decreasing probability. However, the reused experience may not necessarily be suitable for the target task, resulting in poor task knowledge transfer. Chinese invention patent CN 11185009A discloses a task scheduling method for a mobile edge computing system based on transfer and reinforcement learning. This method transfers task features by introducing horizontal links into multi-agent reinforcement learning. However, this link layer becomes difficult to train as the number of multi-agents increases, making it difficult to achieve efficient task knowledge reuse. Chinese invention patent CN 113218400A discloses a multi-agent navigation algorithm based on deep reinforcement learning. This algorithm integrates the A* algorithm into the PPO algorithm and uses a designed reward and penalty function to achieve a deep fusion of the two algorithms to enhance the transfer capability of the multi-agent model. This method primarily utilizes task reuse knowledge to design the reward and penalty function for the target task to enhance the transfer capability of the multi-agent model, but its scope of application is relatively limited. Summary of the Invention

[0005] In response to the technical problems existing in the prior art, the purpose of the present invention is to provide a multi-agent strategy model migration method and system with unchanged sub-task semantics. The present invention encodes the multi-agent task to be executed into an executable sub-task through an extensible sub-task encoder, and assigns the sub-task to each agent in executing the multi-agent task, and then uses an adaptive action decoder to calculate the specific action of the agent's interaction with the environment using the assigned sub-task and the observation data of the current agent. When the multi-agent task to be executed changes, the extensible sub-task encoder and adaptive action decoder of the present invention can ensure that the assigned sub-tasks have consistent and extensible semantics among various multi-agent tasks, and at the same time, the decomposed sub-tasks give task independence, thereby realizing the model migration of the multi-agent strategy model among various multi-agent tasks. The scalable subtask encoder generates subtasks based on the history of {subtask, entity observation}. Due to the cross-task universality of {subtask, entity observation}, the generated subtasks are task-independent; the adaptive action decoder is based on the consistent abstract semantics of the subtask across tasks and the impact of the subtask on all entities. It adaptively calculates the actual semantics of the subtask, which is called consistent and scalable semantics. Based on this actual semantics, the specific actions of the intelligent agent to interact with the environment are calculated.

[0006] The technical solution of the multi-agent strategy model migration system with unchanged subtask semantics of the present invention is as follows.

[0007] This technical solution is as Figure 1As shown, the technical development mainly includes two parts: an extensible subtask encoder and an adaptive action decoder. The multi-agent policy model migration system is jointly constructed by the extensible subtask encoder and the adaptive action decoder. The workflow of the system starts with the extensible subtask encoder, which assigns subtasks to each agent based on the {subtask, entity observation} history that is common across tasks, rather than the {action, observation} history with task dependencies, thereby avoiding the overfitting of the subtask decision process to the source task to a certain extent. Then, the adaptive action decoder calculates the specific actions of interacting with the environment by utilizing the assigned subtasks and the current entity observations. Within the decoder, the adaptive subtask semantic module plays a core role, ensuring that the subtasks have consistent and extensible semantics across tasks based on the impact of the subtasks on the entities. Through the extensible subtask encoder and the adaptive subtask semantic module proposed in the present invention, the decomposed subtasks are given task independence, thereby achieving the generalization of the model across various tasks.

[0008] like Figure 2 As shown in the figure, the scalable subtask encoder mainly includes a cognitive encoder, a Gaussian inner product component, an environmental cognition component, a GRU trajectory encoder, and a Gumbel normalization function. The cognitive encoder mainly encodes the entity observation vector into a vector of the mean and variance of a Gaussian distribution. The entity observation vector of the agent consists of the agent's observations of each entity in the environment. The Gaussian inner product component and the environmental cognition component mainly convert the vector of the mean and variance of the Gaussian distribution into an entity observation embedding vector. The GRU trajectory encoder mainly uses the entity observation embedding vector and the hidden history representation vector of the previous moment to calculate the hidden history representation vector of the current moment. The Gumbel normalization function uses the reparameterized operator Gumbel-Softmax to calculate the subtask.

[0009] like Figure 3 As shown in the figure, the adaptive action encoder mainly includes an adaptive subtask semantic module and an execution action calculation module. The adaptive subtask semantic module mainly uses the attention mechanism to calculate the cross-task subtask semantic vector. The adaptive subtask semantic module uses the subtask as the query in the attention mechanism and the entity observation vector as the key and value in the attention mechanism, and then models the subtask semantic vector as the weighted sum of the entity observation vector. The execution action calculation module mainly uses the entity state embedding vector and the subtask semantic vector to calculate the basic action that an agent currently needs to perform. The basic action that agent i currently needs to perform is calculated using the entity state embedding vector of agent i and the cross-task subtask semantic vector of agent i.

[0010] The specific steps are as follows:

[0011] 1) Initialize the cross-task general model structure (referring to converting environmental observations into a cross-task general structure) to obtain the operating environment of the present invention: First, define the agents controlled by the multi-agent reinforcement learning strategy and the task-built-in agents as entities, and then use n, m, n ally ,n enemy Denotes the number of agents, entities, allies, and enemies respectively. Based on this, the following equation holds: n = n ally +1,m=n+n enemy , and finally define the observation o of agent i i can be constructed as a concatenation of m entity observations: i =[o i,1 ,o i,2 ,…,o i,m ], where o i,1 is the observation of agent i on itself and the environment, and the rest is the observation of agent i on the other m-1 entities. A multi-agent strategy model is constructed, including an extensible subtask encoder and an adaptive action decoder; the extensible subtask encoder includes a cognitive encoder, a Gaussian inner product component, an environmental cognitive component, a GRU trajectory encoder, and a Gumbel normalization function; the adaptive action decoder includes an adaptive subtask semantic module and an execution action calculation module;

[0012] 2) Use the e The parameterized cognitive encoder (MLP, multi-layer perceptron) transforms the entity observation vector o of agent i to the j-th entity at time t i t ,j Encoded as the mean vector of a Gaussian distribution and variance vector

[0013]

[0014] 3) Use the Gaussian inner product component to convert the mean vector of the Gaussian distribution and variance vector Convert to entity observation vector The specific calculation formula is as follows:

[0015]

[0016] 4) Use the c The parameterized environment recognition component (MLP) further encodes the entity observation vector Embedding vector for entity observation As shown in the following formula:

[0017]

[0018] 5) Using the θ-based h The parameterized GRU (Gated Recurrent Unit) trajectory encoder is used to obtain the hidden history representation vector of the agent at the current moment t The trajectory encoder embeds the agent's observations into Subtask assigned to agent i at the previous time t-1 and the hidden history representation vector As input, a new hidden history representation vector for the current time t is generated As its history indicates:

[0019]

[0020] 6) Use the Gumbel-Softmax operator of discrete random variables to represent the hidden history vector at the current time t Calculate the subtask assigned to agent i at the current time t

[0021]

[0022] in ( Represents an n k dimensional real number space) is an n k -dimensional one-hot vector, n k is a hyperparameter representing the total number of subtasks. The Gumbel-Softmax operator allows the system to train the subtask assignment process through gradient backpropagation.

[0023] 7) In the adaptive subtask semantic module, the one-hot subtask Encoded as embedding vector The calculation formula is as follows:

[0024]

[0025] 8) Subtask embedding vector As a query, the entity observation vector As keys and values: Where W Q ,W K ,W V is a learnable matrix, Based on this, we construct adaptive subtask semantics as follows:

[0026]

[0027] where d K yes feature dimension.

[0028] In order to make the action decision process universal across tasks, we extend the model structure of ASN (Action Semantic Network). Since the alignment between actions and entity observations may be inconsistent with the history, we use the self-attention mechanism to As input to build adaptive action semantics

[0029]

[0030] in Represents the semantics of the action of agent i interacting with the mth entity at time t.

[0031] 9) By comparing the similarity of subtask semantics and action semantics, we calculate the value or probability of the interaction action that determines the interaction between agent i and entity j at time t in the current observation:

[0032]

[0033] Then according to the value or probability Determine the interaction action between agent i and entity j at time t, that is, select the action with the highest value / probability as the interaction action between agent i and entity j at time t. Similarity is composed of trainable MLP, Split / Mean-Pooling, and Based on the similarity, we sample actions (training phase) or select the action corresponding to the maximum similarity (inference phase) as the agent action to interact with the environment.

[0034] 11) Model training: The agent interacts with the environment to obtain data trajectories. Based on the different basic algorithms used (value-based algorithms such as QMIX or policy-based algorithms such as MAPPO), the multi-agent policy model is trained using the loss function of the corresponding algorithm.

[0035] 12) Model transfer: Migrating the multi-agent policy model trained on scenario A to scenario B means loading the model trained on scenario A onto scenario B, and then fine-tuning it (performing a small amount of interaction on scenario B and training using the data obtained from the interaction) or directly deploying it (interaction only, no training), thereby reducing the cost of obtaining a usable model on scenario B.

[0036] Compared with the prior art, the present invention has the following positive effects:

[0037] (1) Traditional multi-agent reinforcement learning algorithm systems use task-dependent {action, observation} historical information to make decisions. The present invention uses cross-task common {subtask, entity observation} historical information to make decisions, making it easier to migrate strategy models across tasks.

[0038] (2) Compared with the traditional multi-agent reinforcement learning algorithm system that uses an end-to-end policy transfer learning model from state input to action output, the present invention designs an extensible subtask encoder to extract relevant task information from environmental information to assign subtasks to each agent to be performed in the current stage. The added calculation of the unique-hot subtask logic layer reduces the difficulty of policy model transfer training.

[0039] (3) Compared with the traditional multi-agent reinforcement learning algorithm system, the adaptive action decoder designed in the present invention can ensure that the assigned subtasks have consistent and scalable semantics among various multi-agent tasks. At the same time, the decomposed subtasks give tasks independence, thereby realizing the model migration of the multi-agent strategy model among various multi-agent tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 The overall architecture diagram of the multi-agent policy model migration system with semantically unchanged subtasks.

[0041] Figure 2 This is the architecture diagram of the scalable subtask encoder.

[0042] Figure 3 This is the architecture diagram of the adaptive motion decoder. DETAILED DESCRIPTION

[0043] The present invention will be described in further detail below with reference to the accompanying drawings. The examples given are only used to explain the present invention and are not used to limit the scope of the present invention.

[0044] Consider the following test scenario: In the StarCraft multi-agent experimental environment (SMAC), the source domain task is set to 3m vs. 5m, where m represents the number of ranged marines, 3m is the number of friendly agents, and 5m is the number of enemy agents controlled by the environment's AI. The target domain task is set to 8m vs. 11m, where 8m is the number of friendly agents and 11m is the number of enemy agents controlled by the environment's AI.

[0045] The input to the multi-agent policy model is the agent's observation of the environment and all entities within it. The output of the multi-agent policy model varies depending on the phase. During the interaction phase, the output is the agent's actions used to interact with the environment. During the model update phase, in addition to the actions, the output also includes the expected value of the action in the current state.

[0046] The implementation steps are as follows:

[0047] 1) Initialization of the model structure that is universal across tasks: First, define the agents controlled by the multi-agent reinforcement learning strategy and the agents built into the task as entities, and then use n,m,n ally ,n enemy Represents the number of agents, entities, red teams, and blue teams (red teams and blue teams are allies and enemies in the StarCraft scenario). Based on this, the following equation holds: n = n ally +1,m=n+n enemy , and finally define the observation o of agent i i can be constructed as a concatenation of m entity observations: i =[o i,1 ,o i,2 ,…,o i,m ], where o i,1 is the observation of agent i on itself and the environment, and the rest is the observation of agent i on other m-1 entities. i,j,j=1,2,…,m are all composed of the same form: {attribute1,attribute2,…}. In the StarCraft scenario, {attribute1,attribute2,…} is actually {available,relativeDistance,relativeX,relativeY,health,shield,unitType}. The physical meaning of each attribute is as follows. Available: For agent i, whether entity j is available for interaction. The actual form of interaction is determined by the type of agent i and the relationship between agent i and entity j. For example, if agent i is an attacking unit, it can only interact with entity j if they are enemies and entity j is within agent i's attack range. That is, agent i can attack entity j. If agent i is a healing unit, it can only interact with entity j if they are allies and entity j is within agent i's healing range. That is, agent i can heal entity j. RelativeDistance: The relative distance between agent i and entity j. relativeX: The horizontal coordinate of entity j in a rectangular coordinate system defined with agent i as the origin. relativeY: The vertical coordinate of entity j in a rectangular coordinate system defined with agent i as the origin. health: Entity j's current relative health, i.e., the ratio of its current health to its maximum health. shield: Entity j's current relative shield value, i.e., the ratio of its current shield value to its maximum shield. unitType: Entity j's unit type, represented as a one-hot vector encoding.

[0048] 2) Use the e The parameterized cognitive encoder (MLP, multi-layer perceptron) embeds the entity observation into a vector Encoded as the mean vector of a Gaussian distribution and variance vector

[0049]

[0050] 3) Use the Gaussian inner product component to convert the mean vector of the Gaussian distribution and variance vector Convert to entity observation embedding vector The specific calculation formula is as follows:

[0051]

[0052] 4) Use the c The parameterized context recognition component (MLP) further encodes the entity observation embedding vector Embedding vector for entity observation As shown in the following formula:

[0053]

[0054] 5) Using the θ-based h The parameterized GRU (Gated Recurrent Unit) trajectory encoder is used to obtain the hidden history representation vector of the agent at the current moment t The GRU trajectory encoder embeds the agent's observations into Subtask at the previous time t-1 and historical representation As input, generate a new hidden state As its history indicates:

[0055]

[0056] 6) Use the Gumbel-Softmax operator of discrete random variables to represent the current history Compute subtasks for agent i

[0057]

[0058] in is an n k -dimensional one-hot vector, n k is a hyperparameter representing the total number of subtasks. The Gumbel-Softmax operator allows the system to train the subtask assignment process through gradient backpropagation.

[0059] 7) In the adaptive subtask semantic module, the one-hot subtask Encoded as embedding vector The calculation formula is as follows:

[0060]

[0061] 8) As a query, the entity observation vector and subtask embedding vectors As keys and values: Where W Q ,W K ,W V is a learnable matrix,

[0062] Based on this, we construct adaptive subtask semantics as follows:

[0063]

[0064] where d K yes feature dimension.

[0065] 9) To make the action decision process universal across tasks, we extend the model structure of ASN. Since the alignment between actions and entity observations may be inconsistent with the history, we construct adaptive action semantics by using the self-attention mechanism and taking entity observations as input.

[0066]

[0067] 10) By comparing the similarity of subtask semantics and action semantics, we calculate the value or probability of each action in the current observation:

[0068]

[0069] Where similarity is a trainable MLP, and The choice of Split or Mean-Pooling is based on the actual meaning of the action. If the action interacts with the environment, choose Split; if the action interacts with the entity, choose Mean-Pooling.

[0070] 11) Based on the different stages, different methods are needed to process the above results. In the stage of interaction with the environment during the training process, considering the exploration-exploitation dilemma in reinforcement learning, it is necessary to use action probability Random sampling is used to select actions to interact with the environment; in the multi-agent strategy model update phase of the training process, based on the action probability Calculate the actor loss function to maximize the expected discounted cumulative reward based on the action value The critic loss function is calculated to minimize the gap between the actual value and the expected value of the action; during the inference process, considering the need to select the optimal action to show the best performance of the model, the action probability is used to calculate the critic loss function. The action with the highest probability value is selected to interact with the environment, instead of random sampling during training.

[0071] 12) Model transfer: Migrating the multi-agent policy model trained on scenario A to scenario B means loading the model trained on scenario A onto scenario B, and then fine-tuning it (performing a small amount of interaction on scenario B and training with the data obtained from the interaction) or directly deploying it (interaction only, no training), thereby reducing the cost of obtaining a usable model on scenario B.

[0072] While specific embodiments of the present invention have been disclosed for illustrative purposes, intended to facilitate understanding and implementation of the present invention, those skilled in the art will appreciate that various substitutions, variations, and modifications are possible without departing from the spirit and scope of the present invention and the appended claims. Therefore, the present invention should not be limited to the disclosure of the preferred embodiments, and the scope of protection claimed in the present invention shall be determined by the scope of the claims.

Claims

1. A method for migrating a multi-agent strategy model with invariant subtask semantics, comprising the following steps: 1) Initialize the cross-task general model structure and obtain the agent's operating environment; in the cross-task general model, the agents controlled by the multi-agent reinforcement learning strategy and the task-built-in agents are defined as entities, n represents the number of agents, m represents the number of entities, and n represents the number of entities. ally Indicates the number of allies of each agent, n enemy Represents the number of enemies for each agent, n=n ally +1,m=n+n enemy ; Define the entity observation vector o of agent i i =[o i,1 ,o i,2 ,…,o i,m ], o i,1 is the observation of agent i on itself and the environment, o i,m The observation of the m-1th entity by agent i is constructed; a multi-agent strategy model is constructed, including an extensible subtask encoder and an adaptive action decoder; the extensible subtask encoder includes a cognitive encoder, a Gaussian inner product component, an environmental cognitive component, a GRU trajectory encoder, and a Gumbel normalization function; the adaptive action decoder includes an adaptive subtask semantic module and an execution action calculation module; 2) For the entity observation vector of agent i at time t The cognitive encoder transforms the entity observation vector The entity observation vector of agent i for the jth entity Encoded as the mean vector of a Gaussian distribution and variance vector 3) The Gaussian inner product component converts the mean vector of the Gaussian distribution into and variance vector Convert to entity observation vector 4) The environmental cognition component transforms the entity observation vector Encoded as entity observation embedding vector 5) The GRU trajectory encoder embeds the vector according to the entity observation Subtask assigned to agent i at time t-1 and the hidden history representation vector Get the hidden history representation vector of agent i at time t 6) The Gumbel normalization function uses the Gumbel-Softmax operator of discrete random variables based on the hidden history representation vector at time t Calculate the subtasks assigned to agent i at time t 7) The adaptive subtask semantic module converts the subtask Encoded as subtask embedding vector 8) The execution action calculation module embeds the subtask into the vector As query and entity observation vectors in the attention mechanism As keys and values ​​in the attention mechanism, construct adaptive subtask semantic vectors Represents the action semantics of the interaction between agent i and the mth entity at time t; then the adaptive subtask semantic vector and the jth action semantics Similarity As the value of the corresponding action or probability Determine the interaction action between agent i and entity j at time t, j = 1, ..., m; 9) training the multi-agent strategy model based on data trajectories obtained from the interaction between the agent and the environment; 10) Transfer the multi-agent strategy model trained in step 9) to the target scenario.

2. The method according to claim 1, characterized in that The scalable subtask encoder generates subtasks based on the cross-task common {subtask, entity observation} history and assigns them to agents; each agent observes each entity in the environment according to the received subtask, obtains the entity observation vector and sends it to the adaptive action decoder; the adaptive action decoder calculates the specific action of the corresponding agent's interaction with the environment based on each agent's subtask and its corresponding entity observation vector.

3. The method according to claim 2, characterized in that in, W Q ,W K ,W V is the learnable matrix in the attention mechanism; Represents the semantics of the action of agent i interacting with entity m at time t.

4. The method according to claim 1, 2 or 3, characterized in that: in n k -dimensional one-hot vector, n k is a hyperparameter representing the total number of subtasks.

5. The method according to claim 1, 2 or 3, characterized in that: The model trained in step 10) is transferred to the target scene and fine-tuned on the target scene.

6. The method according to claim 1, 2 or 3, characterized in that: The cognitive encoder is a multi-layer perceptron.

7. A multi-agent strategy model migration system with invariant subtask semantics, characterized by: Includes initialization module, multi-agent strategy model, and migration module; The multi-agent strategy model includes an extensible subtask encoder and an adaptive action decoder; the extensible subtask encoder includes a cognitive encoder, a Gaussian inner product component, an environmental cognitive component, a GRU trajectory encoder, and a Gumbel normalization function; the adaptive action decoder includes an adaptive subtask semantic module and an execution action calculation module; The initialization module is used to initialize the cross-task general model structure and obtain the operating environment of the agent; wherein, the agents controlled by the multi-agent reinforcement learning strategy and the task-built-in agents in the cross-task general model are defined as entities, n represents the number of agents, m represents the number of entities, and n represents the number of entities. ally Indicates the number of allies of each agent, n enemy Represents the number of enemies for each agent, n=n ally +1,m=n+n enemy ; Define the entity observation vector o of agent i i =[o i,1 ,o i,2 ,…,o i,m ], o i,1 is the observation of agent i on itself and the environment, o i,m is the observation of agent i on the m-1th entity; The cognitive encoder is used to observe the entity vector of agent i at time t The entity observation vector The entity observation vector of agent i for the jth entity Encoded as the mean vector of a Gaussian distribution and variance vector The Gaussian inner product component is used to convert the mean vector of the Gaussian distribution and variance vector Convert to entity observation vector The environmental cognition component is used to transform the entity observation vector Encoded as entity observation embedding vector The GRU trajectory encoder is used to embed the vector according to the entity observation Subtask assigned to agent i at time t-1 and the hidden history representation vector Get the hidden history representation vector of agent i at time t The Gumbel normalization function is used to use the Gumbel-Softmax reparameterization operator of discrete random variables based on the hidden history representation vector at time t Calculate the subtasks assigned to agent i at time t The adaptive subtask semantic module is used to convert the subtask Encoded as subtask embedding vector The execution action calculation module is used to embed the subtask into the vector As query and entity observation vectors in the attention mechanism As keys and values ​​in the attention mechanism, construct adaptive subtask semantic vectors Represents the action semantics of the interaction between agent i and the mth entity at time t; then the adaptive subtask semantic vector and the jth action semantics Similarity As the value of the corresponding action or probability Determine the interaction action between agent i and entity j at time t, j = 1, ..., m; The migration module is used to train the multi-agent strategy model based on the data trajectory obtained by the interaction between the agent and the environment; and then migrate the trained multi-agent strategy model to the target scene.

8. A server, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, the computer program is configured to be executed by the processor, and the computer program includes instructions for executing each step of the method according to any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images