A heterogeneous task optimization method and system for multi-agent game confrontation

Abstract

The present invention relates to the field of multi-agent control technology, and in particular to a heterogeneous task optimization method and system for multi-agent game confrontation. The method comprises obtaining observable information of the intelligent agents of both sides of the game, generating a state set in the current environment using an actor-critic network structure based on the observable information of the intelligent agents of both sides of the game, disrupting and renumbering the data sequence in the experience buffer area, and extracting data to train the actor-critic network structure in combination with cognitive theory. The present invention obtains observable information of the intelligent agents of both sides of the game first, and then transmits the state observation data to a central controller. After training, the intelligent agent no longer needs to communicate with the central controller, and can generate the optimal action based on its own local observation state through its own action strategy function.

Description

Technical Field

[0001] The present invention relates to the field of multi-agent control technology, and in particular to a heterogeneous task optimization method and system for multi-agent game confrontation. Background Art

[0002] Multi-agent game confrontation refers to the realization of all-round monitoring, detection and operation of the game environment through collaborative planning among multiple agents, such as aerial drone patrols, maritime unmanned boat search and rescue, maritime unmanned boat strikes, etc. It has the characteristics of clusters such as centerlessness, group control, and high emergence. Multi-agent clusters can realize the allocation method of heterogeneous tasks through collaborative operations, thereby improving task execution efficiency and completion accuracy.

[0003] Although multi-agent game confrontation technology has many advantages, it still faces many challenges. How to enable agents to perform heterogeneous tasks in a complex environment and ensure that the agent cluster completes the task to maximize the benefits is a difficult problem that the multi-agent system needs to solve in game confrontation. Summary of the invention

[0004] In order to solve the above-mentioned problems, the present invention provides a heterogeneous task optimization method and system for multi-agent game confrontation.

[0005] In the first aspect, the present invention provides a heterogeneous task optimization method for multi-agent game confrontation, which adopts the following technical solution:

[0006] A heterogeneous task optimization method for multi-agent game confrontation, comprising:

[0007] S1: Obtain observable information of the two agents in the game. The observable information is represented by a set of one-hot codes {i, t, l, h, c}, including the agent's unique ID, team ID, position, health value, and cooldown time;

[0008] S2: Based on the observable information of the two game agents, the actor-critic network structure is used to generate the state set in the current environment. The actor network is used to output the action of the current agent in the current state, and the critic network is used to combine the current state of the agent and the action taken in the current state to calculate the corresponding state-action value loss, and the corresponding data is stored in the experience buffer;

[0009] S3: The data in the experience buffer are shuffled and renumbered, and data is extracted to train the actor-critic network structure in combination with cognitive theory. The actor network adjusts its action strategy according to the value loss given by the critic network, and the critic adjusts its evaluation strategy according to the reward given by the system and the value given by other critic networks. Combined with the adaptive learning strategy, a practical and reliable actor-critic network is finally obtained, thereby maximizing the benefits of collaborative planning of the intelligent cluster.

[0010] S4: By using the trained agent model, each agent model uses the action strategy function to generate the best action based on its own local observation data and historical experience, thereby ensuring that the agent completes the adversarial task.

[0011] Furthermore, S2 also includes using the actor network to select actions, wherein the actor network selects actions according to a stored strategy, the critic network evaluates actions in the form of a time difference error, and the actor-critic network drives all learning between the actor network and the critic network according to the time difference error calculated by the current value function.

[0012] Furthermore, the data storage method in S3 includes storing the observable information of the two game agents obtained within a fixed step size into the experience buffer, and using random combinations to disrupt and renumber the data order to break the correlation between the data, thereby stabilizing the training process.

[0013] Furthermore, the data taken in S3 is combined with cognitive theory for training, including generating different network structures for different intelligent agents, each intelligent agent actor network is evaluated by all critic networks during training, and expert guidance is introduced into the intelligent agent training to accelerate the convergence speed of the intelligent agent training and increase the benefits of training.

[0014] Furthermore, the cognitive theory in S3 includes that when encountering a new situation and having to take some action, the agent compares the internal representation of the current state with the stored representation of past situations, selects the associated action based on the past situation that is most similar to the current one, introduces external expert guidance, and utilizes external rewards to drive learning, thereby improving the overall benefits of the agent game confrontation system.

[0015] Furthermore, the adaptive learning strategy in S3 includes dynamically adjusting the learning rates of the actor and the critic according to the current learning state and the historical learning situation during the training process of the actor-critic network structure to improve the convergence speed and stability of the algorithm.

[0016] Furthermore, the state-action value loss in S2 includes, when the critic network calculates the value loss, for the situation in which the agent in the multi-agent task cannot perform certain actions, generating invalid work, by shielding the invalid actions so that they do not participate in the action probability calculation.

[0017] Second, a heterogeneous task optimization system for multi-agent game confrontation includes:

[0018] The data acquisition module is configured to obtain observable information of the two agents of the game. The observable information is represented by a set of one-hot codes {i, t, l, h, c}, including the unique ID, team ID, position, health value and cooling time of the agent;

[0019] The experience storage module is configured to generate a state set in the current environment based on the observable information of the two game agents, using the actor-critic network structure, wherein the actor network is used to output the action of the current agent in the current state, and the critic network is used to calculate the corresponding state-action value loss in combination with the current state of the agent and the action taken in the current state, and the corresponding data is stored in the experience buffer area;

[0020] The model training module is configured to shuffle and renumber the data in the experience buffer, extract data and train the actor-critic network structure in combination with cognitive theory, wherein the actor network adjusts its action strategy according to the value loss given by the critic network, and the critic adjusts its evaluation strategy according to the reward given by the system and the value given by other critic networks. The practical and reliable actor-critic network is finally obtained by combining the adaptive learning strategy, thereby maximizing the benefits of the collaborative planning of the intelligent agent cluster;

[0021] The task execution module is configured to use the trained agent models, each of which uses an action strategy function to generate the best action based on its own local observation data and historical experience, thereby ensuring that the agent completes the adversarial task.

[0022] In a third aspect, the present invention provides a computer-readable storage medium storing a plurality of instructions, wherein the instructions are suitable for being loaded and executed by a processor of a terminal device to implement the heterogeneous task optimization method for multi-agent game confrontation.

[0023] In a fourth aspect, the present invention provides a terminal device comprising a processor and a computer-readable storage medium, wherein the processor is used to implement various instructions; the computer-readable storage medium is used to store multiple instructions, wherein the instructions are suitable for being loaded by the processor and executing the heterogeneous task optimization method for multi-agent game confrontation.

[0024] In summary, the present invention has the following beneficial technical effects:

[0025] The present invention first obtains observable information of the intelligent agents of both parties in the game, and then transmits the state observation data to the central controller. After the central controller obtains the global state, it trains the model, combines expert experience and adaptive learning strategy to obtain the optimal decentralized control strategy, and then transmits it to the intelligent agent. After training, the intelligent agent no longer needs to communicate with the central controller, and can generate the optimal action based on its own local observation state through its own action strategy function. It can not only ensure that each of our intelligent agents can accurately and efficiently complete the confrontation task, but also maximize the benefits of our intelligent agent cluster collaborative planning.

[0026] The present invention is based on the benefits obtained by each of our intelligent agents taking actions. After generating the actor-critic network, the convergence speed of the intelligent agent training and the benefits of training are accelerated by introducing expert guidance. The internal representation of the current state is compared with the stored representation of the past situation, and the associated action is selected according to the past situation that is most similar to the current one. The external reward is used to drive learning, thereby ensuring that each of our intelligent agents performs tasks accurately and efficiently, and maximizing the benefits of each of our intelligent agents. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 1 is a schematic diagram of the comparison of benefits between the optimized algorithm of Example 1 of the present invention and other algorithms;

[0028] Figure 2 1. It is a schematic diagram of the comparison of the winning rate of the optimized model of Example 1 of the present invention and other models;

[0029] Figure 3 1 is a diagram of the task execution process of the optimization method of Example 1 of the present invention; wherein (a) is a schematic diagram of the initial distribution; (b) is a schematic diagram of the joint attack; (c) is a schematic diagram of the state analysis; and (d) is a schematic diagram of the confrontation result. DETAILED DESCRIPTION

[0030] The present invention is further described in detail below in conjunction with the accompanying drawings.

[0031] Example 1

[0032] Reference Figure 1 , a heterogeneous task optimization method for multi-agent cluster game confrontation in this embodiment includes the following steps:

[0033] S1: Obtain observable information of both agents in the game. The observable information is represented by a set of one-hot codes {i, t, l, h, c}, including the agent's unique ID, team ID, position, health value and cooldown time. The model that controls an agent can also see other agents within its visual range (3×3 surrounding area);

[0034] S2: Within a certain step length, based on the observable information of the two game agents, generate a state set in the current environment, use the actor network to output the action of the current agent in the current state, use the critic network to combine the current state of the agent and the action taken in the current state to calculate the corresponding state-action value loss, and store the corresponding data in the experience buffer;

[0035] S3: The data in the experience buffer is shuffled and renumbered, and data is extracted from it for training in combination with cognitive theory. The actor network adjusts its action strategy according to the value loss given by the critic network, and the critic adjusts its evaluation strategy according to the reward given by the system and the value given by other critic networks. Combined with the adaptive learning strategy, a practical and reliable actor-critic network is finally obtained, thereby maximizing the benefits of collaborative planning of the intelligent cluster.

[0036] S4: By using the trained agent model, each agent can use its own action strategy function to generate the best action based on its local observation data and historical experience, thereby ensuring that each of our agents can complete the adversarial task accurately and efficiently.

[0037] The observable information of the two game agents in S1 of this embodiment includes:

[0038] The unique ID, team ID, location, health, cooldown time of the agent, and other agents within its visual range (3×3 surrounding area); specifically, the set of agent cluster attributes is represented by a set of one-hot codes as {i, t, l, h, c}, where i represents the unique ID of the agent, which is used to distinguish each different agent, t represents the ID of the team to which the agent belongs, which is used to indicate the agent cluster to which each agent belongs, l represents the location of the agent, h represents the health of the agent, which is used to determine whether the agent is alive, and c represents the cooldown time for the agent to generate an attack behavior, which is used to determine whether the agent should take an attack behavior.

[0039] Specifically, the game confrontation environment is set as a rectangular two-dimensional plane environment. Both agents are set in a 15×15 grid. Each cluster consists of 4 agents. One agent is selected as the central agent, and the remaining 3 agents are evenly sampled in a 5×5 square grid around the central agent. In each time step, the agent can do one of the following: move one grid in one of the four directions; attack another agent by specifying an ID (there are m attack actions, each attack action corresponds to an enemy agent); or do nothing. If agent B is within the shooting range of agent A (the 3×3 area around it), agent A can choose to attack agent B. If A chooses to attack B, B's health will be reduced by 1. The agent needs a cooling time step after attacking, during which the agent cannot attack. All agents start with 3 health points and die when their health reaches 0. If all agents on one side die, the other side wins, and the winner will receive a positive incentive reward. The game ends when one team wins or when neither team wins (a tie) within 40 time steps. If the team loses or the game ends in a tie, the model will receive a reward of -1. In addition, it will receive a reward of -0.1 times the total health of the enemy team, which will encourage our agents to attack the enemy agents.

[0040] The set of all agents used in S1 of this embodiment is specifically: The observable information of the agent can be represented by a set of one-hot codes, where each category corresponds to a binary vector, only one position is 1, and the rest are 0. It is represented as follows:

[0041]

[0042] The set of all agents within the agent's visual range can be expressed as:

[0043]

[0044] Where v represents the set of all agents within the visible range of an agent, Represents the observable information vector of the nth agent within the visible range. Since agents with the same power attributes can communicate with each other and share the set of all agents within the visible range, the shared agent set is represented by express.

[0045] The actor-critic network structure used in S2 of this embodiment is specifically:

[0046] The actor is used to select actions. The method by which the actor selects actions is not based on the current value function, but on the stored strategy. The critic evaluates the quality of the action in the form of temporal difference error. The actor-critic network drives all learning between the actor and the critic based on the temporal difference error calculated by the current value function.

[0047] The optimization objective function of the actor network is as follows:

[0048]

[0049] make ,but is the advantage function, and the n-step time difference method is used to solve it. It is expressed as follows:

[0050]

[0051] in, Represents the optimal strategy, since the optimization objective function means the time series difference error value When the error value is greater than 0, the probability of selecting the action is increased. When it is less than 0, the probability of selecting the action is reduced, so the goal is to minimize the loss function .

[0052] Similar to the processing in the actor network, the critic network is an evaluation network for evaluating the value of an action. That is, the input is the state and action. The critic network splits the Q function into the sum of the value function and the advantage function. , from which the loss function of the critic network is as follows:

[0053]

[0054] in, The expression transformation is as follows:

[0055]

[0056] That is, the objective function is weighted only by the advantage function, the V(s) function only plays the role of the mean function to reduce the variance of the policy gradient, and the value function is regressed and optimized with the discounted return as the goal.

[0057] The state-action value loss in S2 of this embodiment is specifically:

[0058] In multi-agent tasks, it may happen that the agent cannot perform certain actions. When the critic network calculates the value loss, these invalid actions will be blocked so that they do not participate in the action probability calculation.

[0059] The invalid action masking function is expressed as:

[0060]

[0061] In the formula, is the agent in the survival state, For all agents. In order to prevent invalid actions generated when an agent dies from continuing to affect the network, this function is used to mask invalid actions when updating parameters. During forward execution and back propagation, only the parameters of surviving agents are used for effective updates.

[0062] The invalid value shielding function is expressed as:

[0063]

[0064] In the formula, is the importance sampling weight, is the floating important sampling weight. When the agent dies during the confrontation process, the value generated is called invalid value. Invalid value will affect the training accuracy of the critic network. After shielding the invalid value, use and The minimum value in the sample updates the relevant data. Sampling ensures the use of Increasing the ability to explore can ensure the accuracy of the critic network and avoid falling into local optimality.

[0065] The data caching method in S3 of this embodiment is specifically as follows:

[0066] The observable information of the two intelligent agents in the game obtained within a fixed step size is stored in the experience buffer. The data order is shuffled and renumbered in the form of random combination to break the correlation between the data and stabilize the training process.

[0067] Specifically, during the training process, after each round of intelligent agent game confrontation, the valid data that has blocked invalid actions and invalid values ​​will be stored in the data buffer. Within a fixed step time interval, the valid data collected in the data buffer will be randomly arranged and renumbered to achieve multiple utilization of limited data, effectively solving the problem of local optimality caused by insufficient data.

[0068] In S3 of this embodiment, the intelligent agent is trained by combining cognitive theory, specifically,

[0069] Different network structures are generated for different intelligent agents. During training, each intelligent agent actor must be evaluated by all critics. Expert guidance is introduced into the training of intelligent agents. By introducing expert guidance combined with cognitive theory, the convergence speed of intelligent agent training can be accelerated and the benefits of training can be increased.

[0070] In this application, experts provide informative and interpretable rewards for training models. In each round of game competition, a potential reward function is provided , a target strategy , a learner's strategy , and a learning algorithm The main goal is to Design a new reward function , so that Using the algorithm Update strategy To quantify this goal, we define a reward informativeness criterion: , this criterion can adapt to the agent's current strategy and can be optimized under specified structural constraints to obtain interpretable rewards. The reward design problem can be formulated as follows:

[0071]

[0072] Considering the invariance constraints and structural constraints, the above formula can be expressed as:

[0073]

[0074] For some , by removing the constant term, we define the following expert-guided reward design:

[0075]

[0076] in, The reward for expert guidance combined with cognitive theory is set. The expert's knowledge can be transferred to the intelligent agent using this reward function. When the intelligent agent reaches the expert level, the weight of this function will be reduced accordingly so that the intelligent agent can conduct more extensive exploration to obtain higher benefits.

[0077] The cognitive theory in S3 of this embodiment is specifically:

[0078] When encountering a new situation and having to take some action, the agent will compare the internal representation of the current state with the stored representation of past situations, select the associated action based on the past situation that is most similar to the current one, introduce external expert guidance, and use external rewards to drive learning, thereby improving the overall benefits of the agent game adversarial system.

[0079] Given the recognition target label function , we can intuitively assume that, in most cases, the posterior probability of the critic discriminator should be equal to or greater than However, the discriminator The posterior probability that may be returned Lower than , especially in the early training stages when neither the policy nor the discriminator is optimized. Since we expect near , so integrate the prior knowledge and define:

[0080]

[0081] Therefore, the reward function defined by introducing the guidance of cognitive theory experts is designed as follows:

[0082]

[0083] The adaptive learning strategy in S3 of this embodiment is specifically:

[0084] During the agent-centric training process, the learning rates of the actor and critic are dynamically adjusted according to the current learning state and historical learning situation to improve the convergence speed and stability of the algorithm.

[0085] The learning rate of the i-th dimension at time t It is expressed as:

[0086]

[0087] in, is a vector, It represents the cumulative sum of squares of the gradient of the loss function for the i-th dimension parameter from time 0 to time t. is a constant term used to smooth the formula. It is expressed as:

[0088]

[0089] Therefore, the adaptive learning rate It is expressed as:

[0090] .

[0091] Simulation experiment

[0092] The game confrontation environment is set as a rectangular two-dimensional plane environment. Both agents are set in a 15×15 grid. Each cluster consists of 4 agents. One agent is selected as the central agent, and the 5×5 square grid around the central agent is uniformly sampled to place the remaining 3 agents. In each time step, the agent can perform one of the following operations: move one grid in one of the four directions; attack another agent by specifying an ID (there are m attack actions, each attack action corresponds to an enemy agent) or do nothing. Under the premise that there is no obvious difference in the speed and equipment of the enemy and our agents, the experiment evaluates the impact of the heterogeneous task optimization method of the multi-agent cluster game confrontation of this application on other algorithms under different numbers of enemy and our agents. Multi-agent proximal strategy optimization-optimization algorithm (mappo-ae) and multi-agent deep deterministic policy gradient-optimization algorithm (maddpg-ae) are algorithms processed by the optimization method proposed in this application, and multi-agent proximal strategy optimization (mappo) and multi-agent deep deterministic policy gradient (maddpg) are algorithms that have not been processed by the optimization method.

[0093] Figure 1 The comparison of the optimized algorithm of this application with other algorithms shows that the convergence speed and final benefits of the processed MAPPO-AE and MADDPG-AE algorithms are higher than those of the algorithms that have not been processed by the optimization method. During the training process, the convergence speed and benefits of the MAPPO-AE algorithm have always been higher than those of the MAPPO algorithm. Although the benefits of the MADDPG-AE algorithm are lower than those of the MADDPG algorithm at 2×10^6 steps, as time goes by, at 4.5×10^6 steps, MADDPG-AE begins to obtain benefits that exceed those of MADDPG, and the final benefits are higher than those of MADDPG.

[0094] Figure 2 This is a win rate comparison chart of the optimized model of the present application and other models. It can be seen that the win rate of the model optimized by the optimization method of the present application is higher than that of the unoptimized model. The win rate of the mappo model is increased from 86% before optimization to 98% after optimization, and the win rate of the maddpg model is increased from 52% before optimization to 69% after optimization.

[0095] Figure 3 This is a diagram of a simulation process of the optimization method of the present application, in which our agents A1, A2, A3, and A4 are represented by white, and the enemy agents E1, E2, E3, and E4 are represented by gray.

[0096] Depend on Figure 3From (a), we can see that our agent cluster selects A1 as the center point and randomly generates A2, A3, and A4 in the 5×5 grid around A1. The enemy agent cluster selects E1 as the center point and randomly generates E2, E3, and E4 in the 5×5 grid around E1. Figure 3 In (b), we can see that at this time, A1, A2, A3, and A4 choose to jointly attack the enemy agent cluster, and E2 and E3 in the enemy agent cluster choose to attack our agent, and E1 and E4 gradually approach E2 and E3; Figure 3 From (c), we can see that our agents A1, A2, and A4 successfully eliminated the enemy agents E2 and E3, and the enemy agents E2 and E3 successfully eliminated our agent A1 and caused some damage to A2 and A4. Figure 3 As shown in (d), A2 and A4 successfully eliminated E1, and E1 and E4 successfully eliminated A4. At this time, A3 and A2 jointly attacked E4, and our intelligent agent cluster won.

[0097] Example 2

[0098] This embodiment provides a heterogeneous task optimization system for multi-agent game confrontation, including:

[0099] The data acquisition module is configured as follows:

[0100] A computer-readable storage medium stores a plurality of instructions, wherein the instructions are suitable for being loaded and executed by a processor of a terminal device to implement a heterogeneous task optimization method for multi-agent game confrontation.

[0101] A terminal device includes a processor and a computer-readable storage medium, wherein the processor is used to implement various instructions; the computer-readable storage medium is used to store multiple instructions, wherein the instructions are suitable for being loaded by the processor and executing the heterogeneous task optimization method for multi-agent game confrontation.

[0102] The above are all preferred embodiments of the present invention, and are not intended to limit the protection scope of the present invention. Therefore, any equivalent changes made based on the structure, shape, and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A heterogeneous task optimization method for multi-agent game confrontation, characterized in that: include: S1: Obtain observable information of the two agents in the game. The observable information is represented by a set of one-hot codes {i, t, l, h, c}, including the agent's unique ID, team ID, position, health value, and cooldown time; S2: Based on the observable information of the two game agents, the actor-critic network structure is used to generate the state set in the current environment. The actor network is used to output the action of the current agent in the current state, and the critic network is used to combine the current state of the agent and the action taken in the current state to calculate the corresponding state-action value loss, and the corresponding data is stored in the experience buffer; S3: The data in the experience buffer are shuffled and renumbered, and data is extracted to train the actor-critic network structure in combination with cognitive theory. The actor network adjusts its action strategy according to the value loss given by the critic network, and the critic adjusts its evaluation strategy according to the reward given by the system and the value given by other critic networks. Combined with the adaptive learning strategy, a practical and reliable actor-critic network is finally obtained, thereby maximizing the benefits of collaborative planning of the intelligent cluster. S4: By using the trained agent model, each agent model uses the action strategy function to generate the best action based on its own local observation data and historical experience, thereby ensuring that the agent completes the adversarial task; Among them, the actor-critic network structure used in S2 is specifically: The actor is used to select actions. The method by which the actor selects actions is not based on the current value function, but on the stored strategy. The critic evaluates the quality of the action in the form of temporal difference error. The actor-critic network drives all learning between the actor and the critic based on the temporal difference error calculated by the current value function. The optimization objective function of the actor network is expressed as: make ,but is the advantage function, and the n-step time difference method is used to solve it. It is expressed as: in, Represents the optimal strategy, since the optimization objective function means the time series difference error value When the error value is greater than 0, the probability of selecting the action is increased. When it is less than 0, the probability of selecting the action is reduced, so the goal is to minimize the loss function ; The critic network is an evaluation network for evaluating the value of an action. The input is the state and action. The critic network splits the Q function into the sum of the value function and the advantage function. , from which the loss function of the critic network is as follows: in, The expression transformation is as follows: That is, the objective function is weighted by the advantage function, the V(s) function acts as a mean function to reduce the variance of the policy gradient, and the value function is optimized for regression with discounted returns as the goal; The state-action value loss in S2 is: In multi-agent tasks, there may be situations where the agent cannot perform actions. When the critic network calculates the value loss, these invalid actions will be blocked so that they do not participate in the action probability calculation; The invalid action masking function is expressed as: In the formula, is the agent in the survival state, for all agents; to prevent invalid actions generated when an agent dies from continuing to affect the network, this function is used to mask invalid actions when updating parameters. During forward execution and backpropagation, only the parameters of surviving agents are used for effective updates; The invalid value shielding function is expressed as: In the formula, is the importance sampling weight, is the floating important sampling weight; when the agent dies during the confrontation process, the value generated is called invalid value, which will affect the training accuracy of the critic network. After shielding the invalid value, use and The minimum value in the sample updates the relevant data. Sampling ensures the use of Increasing the ability to explore can ensure the accuracy of the critic network and avoid falling into local optimality.

2. The heterogeneous task optimization method of multi-agent game confrontation according to claim 1 is characterized in that: The S2 also includes using the actor network to select actions, wherein the actor network selects actions according to a stored strategy, the critic network evaluates actions in the form of a time difference error, and the actor-critic network drives all learning between the actor network and the critic network according to the time difference error calculated by the current value function.

3. The heterogeneous task optimization method of multi-agent game confrontation according to claim 2 is characterized in that: The data storage method in S3 includes storing the observable information of the two game agents obtained within a fixed step size into the experience buffer, and using random combinations to disrupt and renumber the data order to break the correlation between the data, thereby stabilizing the training process.

4. The heterogeneous task optimization method of multi-agent game confrontation according to claim 3 is characterized in that: The data taken in S3 is combined with cognitive theory for training, including generating different network structures for different intelligent agents, each intelligent agent actor network accepts the evaluation of all critic networks during training, and introduces expert guidance to the intelligent agent training to accelerate the convergence speed of the intelligent agent training and increase the benefits of training.

5. The heterogeneous task optimization method of multi-agent game confrontation according to claim 4 is characterized in that: The cognitive theory in S3 includes that when encountering a new situation and having to take some action, the agent compares the internal representation of the current state with the stored representation of past situations, selects the associated action based on the past situation that is most similar to the current one, introduces external expert guidance, and uses external rewards to drive learning, thereby improving the overall benefits of the agent game confrontation system.

6. A heterogeneous task optimization method for multi-agent game confrontation according to claim 5, characterized in that: The adaptive learning strategy in S3 includes dynamically adjusting the learning rates of the actor and the critic according to the current learning state and the historical learning situation during the training process of the actor-critic network structure to improve the convergence speed and stability of the algorithm.

7. The heterogeneous task optimization method of multi-agent game confrontation according to claim 6 is characterized in that: The state-action value loss in S2 includes that when the critic network calculates the value loss, in the case where the agent cannot perform certain actions in the multi-agent task, invalid work is generated, and the invalid actions are shielded so that they do not participate in the action probability calculation.

8. A heterogeneous task optimization system for multi-agent game confrontation, executing a heterogeneous task optimization method for multi-agent game confrontation as described in any one of claims 1 to 7, characterized in that: include: The data acquisition module is configured to obtain observable information of the two agents of the game. The observable information is represented by a set of one-hot codes {i, t, l, h, c}, including the unique ID, team ID, position, health value and cooling time of the agent; The experience storage module is configured to generate a state set in the current environment based on the observable information of the two game agents, using the actor-critic network structure, wherein the actor network is used to output the action of the current agent in the current state, and the critic network is used to calculate the corresponding state-action value loss in combination with the current state of the agent and the action taken in the current state, and the corresponding data is stored in the experience buffer area; The model training module is configured to shuffle and renumber the data in the experience buffer, extract data and train the actor-critic network structure in combination with cognitive theory, wherein the actor network adjusts its action strategy according to the value loss given by the critic network, and the critic adjusts its evaluation strategy according to the reward given by the system and the value given by other critic networks. The practical and reliable actor-critic network is finally obtained by combining the adaptive learning strategy, thereby maximizing the benefits of the collaborative planning of the intelligent agent cluster; The task execution module is configured to use the trained agent models, each of which uses an action strategy function to generate the best action based on its own local observation data and historical experience, thereby ensuring that the agent completes the adversarial task.

9. A computer-readable storage medium storing a plurality of instructions, characterized in that: The instructions are suitable for being loaded by a processor of a terminal device and executing the method according to claim 1 .

10. A terminal device, comprising a processor and a computer-readable storage medium, wherein the processor is used to implement each instruction; and the computer-readable storage medium is used to store multiple instructions, characterized in that: The instructions are suitable for being loaded by a processor and executing the method as claimed in claim 1 .

Images