Multi-agent model and training system and method

Abstract

The present invention discloses a multi-agent model and a training system and method. An improved MADDPG model is used as the multi-agent model. A simulation module and an expert module are introduced during the model training process. The strategy number and the next action sequence are used as the multi-agent model output. The action sequence is corrected and updated by the expert module, thereby facilitating flexible communication between the multi-agents and consideration of global information. The system can also make good use of existing expert experience, thereby improving the training speed and making the trained agents more stable.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology and relates to intelligent agent training, and in particular to a multi-agent model and its training system and method. Background Art

[0002] Existing agent training mostly uses traditional ideas, inputs the current observation sequence, outputs the next action sequence, and then uses the action sequence to execute and generate in the game script. For example, application number CN201811492348.1 discloses an optimization method, device, terminal equipment and storage medium for game agent training. It discloses an agent model based on an action network and a critic network. The model inputs the current observation sequence, obtains the action sequence, and then obtains a new observation sequence and current reward after execution in the script. When the reward is greater than a certain target value, the training ends. However, traditional agent training is mostly single-agent, and the trained network is relatively stable. In addition, the observation space of a single agent is smaller than that of a multi-agent, and it is easier to train. The reward design of a single agent is simpler and is not suitable for direct migration to multi-agent training.

[0003] Although improvements have been made to the training of multi-agent models (Multi-Agent Deep Deterministic Policy Gradient, MADDPG model), such as adding a separate module to the MADDPG model to ensure communication between agents, application number CN202111240522.5 discloses a multi-agent deep reinforcement learning algorithm that adds a leader network module to the MADDPG model to enhance communication between agents, increase training stability, and improve the robustness of the results.

[0004] However, existing multi-agent systems have relatively large action and observation spaces. Most agents interact or influence each other to some degree, resulting in complex and sparse reward design. Furthermore, the network must simultaneously consider the agent's own state and its communication with other agents. Consequently, current multi-agent training results are far from ideal. Furthermore, due to the large action and observation spaces faced by multi-agent training, learning the agents' actions relies solely on extensive hardware resources and extensive training, failing to effectively leverage existing game strategies and expert knowledge to aid network training. Summary of the Invention

[0005] In view of the current technical status that it is difficult to achieve effective training of multi-agents, the purpose of this invention is to provide a new multi-agent model that can have global situation estimation capabilities.

[0006] Another object of the present invention is to provide a multi-agent training system and method for the above-mentioned multi-agent model, which is based on the improved MADDPG model, introduces expert experience and voting mechanism, and makes the trained agents more stable while improving the training speed.

[0007] In order to achieve the above objectives, the present invention adopts the following technical solutions.

[0008] The multi-agent model provided by the present invention includes multiple agents set up in parallel. The multi-agent model in the present invention is based on the traditional MADDPG model (see Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments Ryan Lowe, et al.arXiv:1706.02275v4[cs.LG], 14Mar2020). What is different from the agents in the traditional MADDPG model is the Actor strategy network and the Critic evaluation network. In the present invention, the Actor strategy network and the Critic evaluation network of the agent have the same structure, both including six fully connected layers, and a residual layer is introduced after every two layers of the fully connected network. ReLU activation function is set after the fully connected layer and the residual layer. Each agent takes the current observation sequence as input and the next action as output. The current observation sequence includes the agent's state and position information for the current frame, the agent's state and relative position information, the enemy agent's state and relative position information detected by the current agent, and the enemy agent's state and relative position information detected by the agent's own agent (referred to as observation content). The state information includes speed, health, ammunition inventory, etc.

[0009] The multi-agent model training system provided by the present invention includes a simulation module, a multi-agent training module, and an expert module connected in communication; multiple agents in the multi-agent model constitute one's own side, and the multiple agents are added to the confrontation scenario of the simulation module;

[0010] The simulation module is used to generate the current observation sequence, current environment reward and cumulative reward of each agent;

[0011] The multi-agent training module is used to output a strategy number sequence and a first action sequence based on the current observation sequence of each agent from the simulation module;

[0012] The expert module is used to select a mapping strategy corresponding to one of the strategy numbers as the execution strategy through a voting strategy based on the strategy number sequence from the multi-agent training module; at the same time, based on the execution strategy and the observation sequence of each agent from the simulation module, a second action sequence is obtained; and then the first action sequence and the second action sequence are weighted averaged to obtain the next action sequence of the multi-agent, which is fed back to the simulation module.

[0013] In the present invention, a conventional Unity simulation platform (such as the Unity 3D engine) is used as a simulation module. The simulation module includes a virtual simulation confrontation scene built in imitation of a real scene (such as a mountain road, a valley road map, etc.). In the present invention, multiple agents in a multi-agent model constitute one's own side, and multiple agents are added to the virtual simulation confrontation scene; the simulation module can also set the enemy behavior through its own automatic control function script; both sides of the confrontation can move freely in the scene, and the mission goal of the one's own side is to cooperate as a team to annihilate the enemy, thereby forming a simulated confrontation environment.

[0014] Each agent's observation sequence includes the concatenation of the current agent's state and position information, the state and relative position information of other agents on its side, the state and relative position information of the enemy agents detected by the current agent, and the state and relative position information of enemy agents detected by other agents on its side (referred to as the observation content). This state information includes speed, health, ammunition inventory, etc. This state and position information can be directly obtained by the simulation module.

[0015] The environment reward is the reward received by each agent during the game between the agent and the enemy in the simulation environment. The current environment reward is the reward received by each agent for the current frame's action, and the cumulative reward is the accumulated reward received by each agent for the historical frame's action.

[0016] The simulation module also includes a reward feedback submodule and a splicing submodule. The reward feedback submodule is used to calculate the current environmental reward and cumulative reward of each agent based on the reward value corresponding to the pre-designed target (such as the reward value obtained by hitting the enemy). The splicing submodule is used to splice the observation content of each agent to obtain the observation sequence of each agent.

[0017] The multi-agent training module includes training submodules corresponding to multiple agents. The training submodule network in the present invention is the agent network. The Actor strategy network of the training submodule outputs a strategy number corresponding to the formulated strategy while outputting the next action. Each training subnetwork takes the observation sequence as input and the strategy number and the next action as output. The strategy numbers of all agents are spliced together to form a strategy number sequence, and the next actions of all agents are spliced together to form a first action sequence. The strategy in the present invention is to collect, analyze and count existing confrontation experience, military strategic situation analysis, game strategy analysis, etc. to obtain some general strategy rules for instructing agents to cooperate; and formulate a unique strategy number for each strategy. The training submodule here introduces the residual network idea to connect the features of each two layers, which can reduce the loss of initial information and prevent network degradation. This makes the agent have stronger classification ability and stability.

[0018] In the present invention, when the multi-agent includes four fire vehicles and one support vehicle, the strategy is formulated as follows:

[0019] (1) Three of your own firepower vehicles are arranged in a vertical snake formation;

[0020] (2) The three fire vehicles of the own side are arranged in a long snake formation;

[0021] (3) Four of your own firepower vehicles are arranged in a vertical snake formation;

[0022] (4) The four fire vehicles of the own side are arranged in a long snake formation;

[0023] (5) The firepower vehicles of the team move closer together to surround the support vehicle;

[0024] (6) The firepower vehicle of the own side explores in the four directions of east, west, south and north;

[0025] (7) Random walk.

[0026] The strategy code and corresponding strategy are stored in a storage module. The storage module also stores the observation sequence threshold range associated with the strategy and the next action of each agent associated with the strategy. A list is constructed of the strategy number, strategy, observation sequence threshold range associated with the strategy, and the next action of each agent associated with the strategy.

[0027] The expert module includes a strategy formulation submodule and an action inference submodule. The strategy formulation submodule is used to select a strategy number mapping (i.e., the strategy represented by the strategy code) as the execution strategy based on the strategy number sequence from the multi-agent training module through a voting strategy. The voting strategy here uses the conventional Moore voting algorithm (see Robert S. Boyer and J. Strother Moore, MJRTY-A Fast Majority Vote Algorithm, 1982). Furthermore, the strategy formulation submodule uses the current observation sequence of each agent from the simulation module, based on a constructed list, to obtain the corresponding strategy number, and adds this to the strategy number sequence output by the multi-agent training module as the strategy set to be selected. The action inference submodule determines the next action of each agent based on the current observation sequence from the simulation module and the execution strategy selected by voting, and forms a second action sequence. The first action sequence and the second action sequence are then weighted averaged to obtain the next action sequence of each agent. In one implementation, the action inference submodule averages the corresponding items in the first action sequence and the second action sequence, and uses the average of each item as the next action sequence of the multi-agent.

[0028] The simulation module obtains the next action sequence based on the received multi-agent next action sequence and the current observation sequence, and then obtains a new observation sequence, environmental reward and cumulative reward in the next round.

[0029] The simulation module, multi-agent training module and expert module communicate with each other through socket processes, so that distributed training can be achieved.

[0030] The present invention further provides a method for training a multi-agent model, which is performed using the above training system according to the following steps:

[0031] S1 multi-agent training module initialization;

[0032] S2 simulation module initialization;

[0033] The S3 simulation module generates the current observation sequence, current environment reward and cumulative reward of each agent and sends them to the multi-agent training module and the expert module;

[0034] The S4 multi-agent training module outputs the strategy number sequence and the first action sequence based on the current observation sequence of each agent from the simulation module and sends them to the expert module;

[0035] The S5 expert module selects a strategy number mapping from the strategy number sequence from the multi-agent training module through a voting strategy as the execution strategy. At the same time, based on the execution strategy and the observation sequence of each agent from the simulation module, it obtains the second action sequence. It then performs a weighted average of the first and second action sequences to obtain the next action sequence of the multi-agent and feeds it back to the simulation module.

[0036] Repeat steps S3-S5 until the cumulative reward of any agent exceeds the set threshold or the friendly agent completely wins (i.e., the enemy is annihilated), ending this round of training. After that, return to step S2, and each agent returns to the initial state and continues training again until the winning rate of the friendly agent in the last 100 games reaches 95%.

[0037] In the above step S1, the initialization of the multi-agent training module mainly refers to the network parameters of the training sub-module, which can be achieved by random assignment.

[0038] In step S2 above, the simulation module is initialized, primarily referring to initializing the state and position information of each agent, including each agent's initial position, initial direction, initial speed, initial health, initial number of projectiles, etc. During the simulation, a human-computer interface can be used to visualize the simulation environment and the state and position information of each simulated agent.

[0039] In summary, the multi-agent reinforcement learning training method provided by the present invention has the advantages of the traditional multi-agent reinforcement learning training method MADDPG, which can be trained centrally and executed independently. It also adds a well-designed expert module and uses existing tactical strategies to guide the actions of agents. It can better and faster utilize prior knowledge and enable the training network to learn beneficial strategies as soon as possible.

[0040] Compared with the prior art, the present invention has the following beneficial effects:

[0041] 1) The training system and method of the multi-agent model provided by the present invention have the advantages of existing multi-agent training, can be centralized for training and decentralized for execution, assist in flexible communication between multiple agents and consideration of global information, and at the same time can make good use of existing expert experience and show good stability in rapid training.

[0042] 2) The multi-agent training method of the present invention simplifies the output of the agent training network, using strategy numbers and relatively few action behavior sequences as the output of the agent network. The strategy comes from existing expert experience, and the final action sequence not only includes the actions of a single agent, but most importantly, guides the overall coordination of multiple agents and the macro-overall actions. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1Schematic diagram of the training system structure of the multi-agent model provided in Example 2 of the present invention.

[0044] Figure 2 Schematic diagram of the multi-agent training module.

[0045] Figure 3 A flowchart of the training method for the multi-agent model provided in Example 3 of the present invention. DETAILED DESCRIPTION

[0046] The technical solutions of the various embodiments of the present invention are clearly and completely described in conjunction with the accompanying drawings. It is obvious that the described embodiments are only some embodiments of the present invention, not all embodiments. All other embodiments derived by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts belong to the present invention.

[0047] Example 1

[0048] The multi-agent model provided in this embodiment includes multiple agents set up in parallel, with each agent forming its own side. The multi-agent model in this invention is based on the traditional MADDPG model (see Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments Ryan Lowe, et al. arXiv:1706.02275v4[cs.LG], 14 Mar 2020) and is essentially the same as the traditional MADDPG model, differing in the actor policy network and critic evaluation network in each agent.

[0049] In this embodiment, the agent's Actor policy network and Critic evaluation network have the same structure, consisting of six fully connected layers and two residual layers. A residual layer is introduced after every two fully connected layers, and a ReLU activation function is used after each fully connected layer and each residual layer. In this embodiment, the convolutional layer serves as the residual layer, and the sum of the residual layer's input and output serves as the input to the next fully connected layer. Each agent takes its current observation sequence as input and outputs its corresponding next action as output.

[0050] The current observation sequence includes the agent's state and position information for the current frame, the state and relative position information of other friendly agents, the state and relative position information of the enemy detected by the current agent, and the state and relative position information of the enemy detected by other friendly agents (referred to as observation content). State information includes speed, health, ammunition inventory, etc.

[0051] In this embodiment, the agents are designed to represent multiple vehicle types (firepower vehicles, support vehicles, etc.). In this embodiment, the multi-agent model includes four firepower vehicles and one support vehicle. The firepower vehicle has a detection range of 1600m, an ammunition capacity of 50 rounds, and a communication range of 3000m; the support vehicle has a detection range of 1400m, an ammunition capacity of 500 rounds, and a communication range of 3000m. The firepower and support vehicles have an initial health of 100 and a speed range of 0-5m / s.

[0052] Example 2

[0053] like Figure 1 As shown, the training system of the multi-agent model provided in this embodiment includes a communication-connected simulation module, a multi-agent training module, a storage module (not shown in the figure) and an expert module.

[0054] 1. Simulation module

[0055] The simulation module is used to generate the current observation sequence, current environment reward and cumulative reward of each agent.

[0056] In this example, the conventional Unity simulation platform (here, the Unity 3D engine) is used as the simulation module. The simulation module is connected to the multi-agent training module and the expert module through the Unity simulation platform interface. All three modules are equipped with communication submodules, and the modules communicate with each other through socket processes, enabling distributed training.

[0057] The simulation module includes a virtual simulation confrontation scene built in imitation of real scenes (such as mountain roads, valley maps, etc.), and multiple intelligent agents are added to the virtual simulation confrontation scene; in this embodiment, multiple intelligent agents in the multi-agent model constitute our side, and multiple intelligent agents are added to the virtual simulation confrontation scene; the simulation module can also set the enemy behavior through its own automatic control function script (such as Unity's script-random walk); both sides of the confrontation can move freely in the scene, and our side's mission goal is to cooperate as a team to annihilate the enemy, thereby forming a simulated confrontation environment.

[0058] Each agent's observations include the current agent's state and position, the state and relative position of other agents on its side, the state and relative position of enemy agents detected by the current agent, and the state and relative position of enemy agents detected by other agents on its side. State information includes speed, health, ammunition reserves, etc.

[0059] The environment reward is the reward received by each agent during the game between the agent and the enemy in the simulation environment. The current environment reward is the reward received by each agent for the current frame's action, and the cumulative reward is the accumulated reward received by each agent for the historical frame's action. Where R i represents the cumulative return of the i-th agent, γ represents the return superimposed discount coefficient at time t, represents the environmental return of the i-th agent at time t.

[0060] The above simulation module also includes a reward feedback submodule and a splicing submodule.

[0061] The reward feedback submodule is used to calculate the current environment reward and cumulative reward of each agent based on the reward value corresponding to the pre-designed target (such as the reward value obtained by hitting the enemy). The splicing submodule is used to splice the observation content of each agent to obtain the observation sequence of each agent.

[0062] 2. Multi-agent training module

[0063] The multi-agent training module is used to output a strategy number sequence and a first action sequence based on the current observation sequence of each agent from the simulation module.

[0064] The multi-agent training module includes training submodules corresponding to multiple agents. The training submodule network is the agent network. The Actor strategy network and Critic evaluation network structure are as follows: Figure 2 As shown, each includes six fully connected layers and two residual layers; a residual layer is introduced after every two fully connected layers; and a ReLU activation function is set after each fully connected layer and residual layer. In this embodiment, the convolutional layer serves as the residual layer; and the sum of the input and output of the residual layer serves as the input of the next fully connected layer. The Actor policy network of the training submodule outputs a policy number in addition to the next action. The training submodule network also includes a target policy network and a target evaluation network, which are used to calculate the policy loss and update the network parameters during the training process.

[0065] Each training submodule takes the current observation sequence of each agent as input and the strategy number and next action as output. The strategy numbers of all agents are concatenated to form a strategy number sequence, and the next actions of all agents are concatenated to form the first action sequence.

[0066] The strategy of this invention is to collect, analyze and compile statistics on existing confrontation experience, military strategic situation analysis, game strategy analysis, etc. to obtain some general strategy rules to instruct intelligent agents to cooperate; and to formulate a unique strategy number for each strategy.

[0067] In this embodiment, the strategy is formulated as follows:

[0068] (1) Three of your own firepower vehicles are arranged in a vertical snake formation;

[0069] (2) The three fire vehicles of the own side are arranged in a long snake formation;

[0070] (3) Four of your own firepower vehicles are arranged in a vertical snake formation;

[0071] (4) The four fire vehicles of the own side are arranged in a long snake formation;

[0072] (5) The firepower vehicles of the team move closer together to surround the support vehicle;

[0073] (6) The firepower vehicle of the own side explores in the four directions of east, west, south and north;

[0074] (7) Random walk.

[0075] like Figure 2 As shown, during the training process, for the ith agent, the observation sequence (s i ) Input the Actor strategy network and get the next action (a i ) and the strategy number (p i ); Next action (a i ) and the observation sequence (s i ) and input the Critic evaluation network to obtain the output action evaluation (q i ); using the next action (a i ) and action evaluation (q i ) as well as the target action of the target policy network and the target action evaluation of the target evaluation network to calculate the policy loss, and then use the policy loss to correct the parameters of the Actor policy network and the Critic evaluation network.

[0076] 3. Storage module

[0077] The strategy code and corresponding strategy are stored in the storage module. The storage module also stores the observation sequence threshold range associated with the strategy and the next action of each agent associated with the strategy. It also constructs a list of the strategy number, strategy, observation sequence threshold range associated with the strategy, and the next action of each agent associated with the strategy.

[0078] Table 1. Strategies, observation sequences, and action lists

[0079]

[0080] 4. Expert Module

[0081] The expert module is used to select a mapping strategy corresponding to one of the strategy numbers as the execution strategy through a voting strategy based on the strategy number sequence from the multi-agent training module; at the same time, it obtains the second action sequence based on the execution strategy and the observation sequence of each agent from the simulation module; then the first action sequence and the second action sequence are weighted averaged to obtain the next action sequence of the multi-agent, and fed back to the simulation module.

[0082] The expert module includes a strategy formulation submodule and an action inference submodule. The strategy formulation submodule uses the current observation sequence of each agent from the simulation module and, based on the constructed list, obtains the corresponding strategy number. This number is then added to the strategy number sequence output by the multi-agent training module as the set of strategies to be selected. The strategy formulation submodule then selects a strategy number mapping (i.e., the strategy represented by the strategy code) from the set of strategies to be selected as the execution strategy through a voting strategy. The voting strategy here uses the conventional Moore voting algorithm (see Robert S. Boyer and J. Strother Moore, MJRTY-A Fast Majority Vote Algorithm, 1982).

[0083] The action inference submodule determines the next action for each agent based on the constructed list, using the current observation sequence from the simulation module and the voting execution strategy. This determines the second action sequence and then takes a weighted average of the first and second action sequences to determine the next action sequence for each agent. In one implementation, the action inference submodule averages the corresponding items in the first and second action sequences, using the average as the next action sequence for the multi-agent.

[0084] The simulation module obtains the next action sequence based on the received multi-agent next action sequence and the current observation sequence, and then obtains a new observation sequence, environmental reward and cumulative reward in the next round.

[0085] The simulation module, multi-agent training module and expert module communicate through socket processes based on their respective communication submodules, which can realize distributed training.

[0086] Example 3

[0087] The training method of the multi-agent model provided in this embodiment is as follows: Figure 3 As shown, the training system provided in Example 2 is used to perform the following steps:

[0088] S1 multi-agent training module initialization.

[0089] The initialization of the multi-agent training module mainly refers to the network parameters of the training sub-module, which can be achieved through random assignment.

[0090] S2 simulation module initialization.

[0091] The simulation module initialization mainly refers to the initialization of the state information and position information of each intelligent agent.

[0092] After receiving the initialization signal (e.g., "reset": 1), the simulation module initializes all agents, including the initial position, initial direction, initial speed, initial health, initial number of shells, etc. of each agent.

[0093] During the simulation process, the human-computer interaction interface can be used to visualize the state and position information of the simulation environment and each simulation agent, and reset the current environment reward and cumulative reward to zero.

[0094] The S3 simulation module generates the current observation sequence of each agent, the current environment reward and the cumulative rewards of both parties, and sends them to the multi-agent training module and the expert module.

[0095] When the simulation module receives the initialization signal, it initializes the data with the state information and position information of each agent as the first current observation sequence, and resets the current environment reward and cumulative reward to zero.

[0096] In the subsequent cyclic training process, based on the current actions of each agent, the current observation sequence, current environment reward and cumulative reward of each agent are generated.

[0097] The simulation module sends the generated current observation sequence, current environment reward and cumulative reward of each agent to the multi-agent training module and expert module through the communication submodule.

[0098] The S4 multi-agent training module outputs the strategy number sequence and the first action sequence based on the current observation sequence of each agent from the simulation module and sends them to the expert module;

[0099] In this step, the current observation sequence of each agent (s1, s2, ..., s i ,…,s N ) as input, and the next action sequence (a1, a2, ..., a i ,…,,a N , as the first action sequence) and the strategy number sequence (p1, p2, ..., p i ,…,p N ) is output and sent to the expert module through the communication submodule.

[0100] The following takes the i-th intelligent agent as an example to illustrate the network parameter update process in the intelligent agent.

[0101] The gradient of the critic evaluation network of the i-th agent can be expressed as:

[0102]

[0103] Where, J(μ i ) represents the R that satisfies the above objective function i, that is, J(μ i )=E[R i ]; E[] represents the expected value; μ i (a i |s i ) represents the Actor strategy network of the i-th agent; θ i Represents the Actor strategy network parameters of the i-th agent; Represents the action evaluation obtained by the Critic evaluation network of the i-th agent.

[0104] The critic evaluation network loss of the i-th agent is:

[0105]

[0106]

[0107] Where r i represents the current environment report of the i-th agent (time t is omitted here); γ represents the reward superimposed discount coefficient (time t is omitted here); The target evaluation network outputs the target action evaluation of the i-th agent. The specific acquisition process is: after the Actor strategy network iterates M steps, the observation sequence (s′) of the M-th step is i ) Input the target policy network to obtain the target action (a′ i ), and then the observation sequence (s′ i ) and target action (a′ i ) and input the target evaluation network to obtain the output target action evaluation (q′ i ), in order to distinguish the Actor strategy network from the target strategy network, it is recorded here as Therefore, the target action in the target policy network and the target action evaluation in the target evaluation network are updated once every M steps of the actor policy network iteration.

[0108] Critic evaluation network parameters ω i The update is based on L(θ i ) is obtained using conventional back-propagation.

[0109] The update direction of the Actor policy network of the i-th agent is to use the gradient value of the critic to transform in the negative direction of its gradient value (the fastest increasing direction) even if the value obtained by the critic evaluation network is larger:

[0110] That is, the loss function of the Actor strategy network is Loss = -L(θ i ).

[0111] Therefore, the network parameter θ in the Actor strategy network iThe update is obtained by conventional back propagation based on Loss.

[0112] The network parameters θ′ in the target policy network i The update process is as follows: After each iteration M steps of the Actor strategy network, according to the updated θ i , then according to the following formula: θ′ i ←τθ i +(1-τ)θ′ i , τ represents the network parameter update discount factor.

[0113] Network parameter ω′ in the target evaluation network i The updating process is as follows: after each M-step iteration of the Critic evaluation network, the updated ω i , then follow the formula: ω′ i ←τω i +(1-τ)ω′ i , indicating that τ represents the network parameter update discount factor.

[0114] The S5 expert module selects one of the strategy number mappings as the execution strategy based on the strategy number sequence from the multi-agent training module through a voting strategy; at the same time, it obtains the second action sequence based on the execution strategy and the observation sequence of each agent from the simulation platform; then it performs a weighted average of the first action sequence and the second action sequence to obtain the next action sequence of the multi-agent, which is fed back to the simulation module.

[0115] In this step, the strategy formulation submodule first uses the current observation sequence of each agent from the simulation module, and obtains the corresponding strategy number based on the predicted sequence threshold range and strategy mapping relationship in the constructed list, and adds it to the strategy number sequence output by the multi-agent training module as the strategy set to be selected; then the strategy formulation submodule selects one of the strategy number mappings (that is, the strategy represented by the strategy code) as the execution strategy based on the strategy set to be selected through the Moore voting algorithm.

[0116] The action inference submodule determines each agent's next action based on the constructed list, based on the current observation sequence from the simulation module and the execution strategy selected by voting. This determines the second action sequence. The current observation sequence determines each agent's current position; combined with the next action requirements corresponding to the execution strategy, the target position of each agent can be determined. Therefore, the next action of each agent can be determined based on its current and target positions. The next actions of all agents constitute the second action sequence.

[0117] Then the action calculation submodule averages the corresponding items in the first action sequence and the second action sequence, and uses the average value of each item as the next action sequence of the multi-agent, which is fed back to the simulation module.

[0118] Repeat steps S3-S5 until the cumulative reward of any agent exceeds the set threshold or the friendly agent achieves complete victory (i.e., the enemy is annihilated), ending the current round of training. Then, return to step S2, and each agent returns to the initial state and continues training again until the friendly agent's win rate reaches 95% over the last 100 rounds.

[0119] Therefore, the present invention proposes a new multi-agent training optimization method and system. The method obtains the current observation sequence based on the Unity simulation platform. Based on the improved MADDPG model network, the method outputs and returns the strategy number sequence and the predicted next action sequence, which are sent to the expert module. The expert module votes for the final overall strategy based on the current observation sequence and the strategy number sequence obtained by the training module, and calculates an updated next action sequence based on the overall strategy and the predicted action sequence of the training module, and sends it to the simulation module. The simulation module acts according to the next action sequence provided by the expert module, obtaining a new current observation sequence and the current environment reward. The current environment reward is added to the cumulative reward. When the cumulative reward exceeds a set threshold or the self-agent agent completely wins, the current round of training ends and training returns to the initial state to continue. This training system not only uses the multi-agent action sequence as the training network return result, but also uses the agent strategy number as the network result, which enables the training network to have global situation estimation capabilities and improves training efficiency.

[0120] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the principles of the present invention, and it should be understood that the scope of protection of the present invention is not limited to such specific descriptions and embodiments. Those skilled in the art can make various other specific variations and combinations based on the technical teachings disclosed in the present invention without departing from the essence of the present invention, and such variations and combinations are still within the scope of protection of the present invention.

Claims

1. A multi-agent model training system, characterized in that: The system includes a simulation module with communication connections, a multi-agent training module, and an expert module. The multi-agent model includes multiple agents set up in parallel, each agent includes an actor strategy network and a critic evaluation network with the same structure. The network structure includes six fully connected layers, and a residual layer is introduced after every two fully connected layers. A ReLU activation function is set after each fully connected layer and the residual layer. The multiple agents in the multi-agent model constitute the self-team, and the multiple agents are added to the confrontation scenario of the simulation module. The simulation module is used to generate the current observation sequence, current environment reward and cumulative reward of each agent; the observation sequence of each agent includes the state information and position information of the current agent, the state information and relative position information of other agents on the same side, the state information and relative position information of the enemy detected by the current agent, and the state information and relative position information of the enemy agents detected by other agents on the same side; The multi-agent training module is used to output a strategy number sequence and a first action sequence based on the current observation sequence of each agent from the simulation module; The expert module is used to select a mapping strategy corresponding to one of the strategy numbers as the execution strategy through a voting strategy based on the strategy number sequence from the multi-agent training module; at the same time, based on the execution strategy and the observation sequence of each agent from the simulation module, a second action sequence is obtained; and then the first action sequence and the second action sequence are weighted averaged to obtain the next action sequence of the multi-agent, which is fed back to the simulation module.

2. The multi-agent model training system according to claim 1, characterized in that The simulation module also includes a reward feedback submodule and a splicing submodule; the reward feedback submodule is used to calculate the current environment reward and cumulative reward of each intelligent agent based on the reward value corresponding to the pre-designed target; The splicing submodule is used to splice the observation contents of each agent to obtain the observation sequence of each agent.

3. The multi-agent model training system according to claim 1, characterized in that The multi-agent training module includes training sub-modules corresponding to multiple agents. The Actor strategy network of the training sub-module outputs a strategy number corresponding to the formulated strategy while outputting the next action; the strategy numbers of all agents are spliced together to form a strategy number sequence, and the next predicted actions of all agents are spliced together to form a first action sequence.

4. The multi-agent model training system according to claim 3, characterized in that: The strategy number and the corresponding strategy are saved in a storage module; the storage module also saves the observation sequence threshold range related to the strategy, and the next action of each intelligent agent related to the strategy, and constructs a list of the strategy number, strategy, the observation sequence threshold range related to the strategy, and the next action of each intelligent agent related to the strategy.

5. The multi-agent model training system according to claim 4, characterized in that: The expert module includes a strategy formulation submodule and an action inference submodule; the strategy formulation submodule is used to select a mapping of one of the strategy numbers as the execution strategy through a voting strategy based on the strategy number sequence from the multi-agent training module; the action inference submodule determines the next action of each agent based on the current observation sequence from the simulation module and the execution strategy selected by voting, and forms a second action sequence.

6. The multi-agent model training system according to claim 5, characterized in that: The strategy formulation submodule also uses the current observation sequence of each agent from the simulation module to obtain the corresponding strategy number based on the constructed list, and adds it to the strategy number sequence output by the multi-agent training module as the strategy set to be selected.

7. The multi-agent model training system according to claim 5 or 6, characterized in that: The action calculation submodule performs weighted averaging on the first action sequence and the second action sequence to obtain the next action sequence of each agent.

8. A method for training a multi-agent model, characterized in that: The training system according to any one of claims 1 to 7 is used to perform the following steps: S1 multi-agent training module initialization; S2 simulation module initialization; The S3 simulation module generates the current observation sequence, current environment reward, and cumulative reward for each agent and sends them to the multi-agent training module and the expert module. Each agent's observation sequence includes the concatenated state information and position information of the current agent, the state information and relative position information of other agents on the same team, the state information and relative position information of the enemy detected by the current agent, and the state information and relative position information of the enemy agents detected by other agents on the same team. S4 The multi-agent training module outputs the strategy number sequence and the first action sequence based on the current observation sequence of each agent from the simulation module and sends them to the expert module; The S5 expert module uses a voting strategy to select a mapping strategy corresponding to a strategy number from the multi-agent training module as the execution strategy. It also obtains a second action sequence based on the execution strategy and the observation sequence of each agent from the simulation module. It then takes a weighted average of the first and second action sequences to obtain the next action sequence for the multi-agent, which is then fed back to the simulation module. Repeat steps S3-S5 until the cumulative reward of any agent exceeds the set threshold or the agent on your side wins completely, ending this round of training. After that, return to step S2, and each agent returns to the initial state and continues training.

Images