A method and device for training intelligent agents based on inter-cluster confrontation

Abstract

The present application provides an agent training method and device based on inter-cluster confrontation, which belongs to the field of agent technology. The embodiment of the present application first uses multiple opponent pools to perform personalized training on multiple target agent clusters, and can train the target extended agent cluster and the opponent extended agent cluster; then, by controlling the agents in the target extended agent cluster and the opponent extended agent cluster to perform confrontation training, and controlling the trained agents of the opposing parties to perform simulation confrontation, the simulation confrontation score of each agent can be obtained, and then according to the agent parameters of the agents with the highest simulation confrontation scores, the target balanced agents under different target optimization objectives and different target constraints can be finally obtained. In this way, the generalization ability of the target balanced agent can be effectively improved, and even in the case of completely unknown opponents, the target balanced agent can achieve better confrontation effects.

Description

Technical Field

[0001] The present application relates to the field of intelligent agent technology, and in particular to an intelligent agent training method and device based on inter-cluster confrontation. Background Art

[0002] Game confrontation between intelligent agents refers to the adversarial behavior based on strategy and decision-making between two or more intelligent agents in the field of artificial intelligence. In the game confrontation, each intelligent agent tries to maximize its own interests by formulating the best strategy, and must also predict and respond to the behavior of its opponent.

[0003] With the continuous emergence of artificial intelligence technology, researchers have applied artificial intelligence technology to the training of intelligent agents, especially the intelligent agent training method based on reinforcement learning, and have achieved relatively good operating results in decision-making problems.

[0004] In the process of game confrontation, it is necessary to consider various aspects of the opponent, such as the opponent's intelligence level, available tactics, confrontation style and tendency, etc. However, in the current reinforcement learning-based intelligent agent training methods, the opponent's confrontation methods are usually based on fixed rules and have only a single style / tendency.

[0005] Based on the simulation training environment of fixed rules or single style / tendency opponents, the trained intelligent agents can only deal with opponents with these rules or single style / tendency, resulting in limited generalization ability of the intelligent agents and difficulty in dealing with complex and changeable adversarial tasks. Especially when the opponent is completely unknown, it is difficult for operators to select suitable intelligent agents to perform corresponding adversarial tasks, which leads to poor adversarial effect of the intelligent agents. Summary of the invention

[0006] The present application provides an agent training method and device based on inter-cluster confrontation to solve the problem that the generalization ability of the trained agent in the related art is limited and it is difficult to cope with complex and changeable confrontation tasks without fully understanding the opponent.

[0007] In order to solve the above problems, this application adopts the following technical solutions:

[0008] In a first aspect, an embodiment of the present application provides an agent training method based on inter-cluster confrontation, the method comprising:

[0009] Determine any one of the two adversarial parties as the target party, and construct multiple target party agent clusters of the target party based on multiple target party optimization objectives and multiple target party constraints of the target party under the target task boundary; the target party agent clusters include multiple initial target party agents, and different initial target party agents have different target party optimization objectives and / or different target party constraints;

[0010] The other party of the two opposing parties relative to the target party is determined as an opponent party, and multiple opponent party agent clusters of the opponent party are combined to obtain multiple opponent party pools of the target party agent cluster; wherein different opponent party pools are composed of different opponent party agent clusters, and each of the opponent party agent clusters includes multiple initial opponent party agents;

[0011] For any target agent cluster, control the target agent cluster to perform adversarial training with multiple opponent agent clusters in the opponent pool, so as to obtain multiple personalized agents corresponding to each initial target agent in the target agent cluster, and obtain a personalized target agent corresponding to the initial target agent based on agent parameters of the multiple personalized agents;

[0012] The personalized target party agent and multiple personalized agents corresponding to each initial target party agent are stored in the target party extended agent cluster of the target party, and the personalized opponent party agent and multiple personalized agents corresponding to each initial opponent party agent are stored in the opponent party extended agent cluster of the opponent party;

[0013] Controlling the agents in the target party's extended agent cluster and the opponent party's extended agent cluster to perform adversarial training to update the agents in the target party's extended agent cluster and the opponent party's extended agent cluster;

[0014] For any agent in the updated target party extended agent cluster, control the agent to perform simulation confrontation with the agent in the updated opponent party extended agent cluster to obtain a simulation confrontation score for each agent in the target party extended agent cluster;

[0015] For multiple agents in the target-side extended agent cluster having the same target-side optimization objective and target-side constraint conditions, a target-side balanced agent having the target-side optimization objective and the target-side constraint conditions is obtained based on the agent parameters of a preset proportion of agents with high simulation adversarial scores among the multiple agents, so as to obtain a target-side balanced agent corresponding to each initial target-side agent.

[0016] In one embodiment of the present application, the step of controlling the agents in the target extended agent cluster and the opponent extended agent cluster to perform adversarial training to update the agents in the target extended agent cluster and the opponent extended agent cluster includes:

[0017] Repeat the following training steps for the preset number of repetitions:

[0018] Controlling the agents in the target party's extended agent cluster to perform preset rounds of simulation confrontation with the agents in the opponent party's extended agent cluster to obtain simulation confrontation results for each agent;

[0019] For any agent, if the simulation confrontation result of the agent meets the winning condition, the winning index of the agent is increased; if the simulation confrontation result meets the normal condition, the winning index of the agent is kept unchanged; if the simulation confrontation result meets the failure condition, the winning index of the agent is reduced, and if the winning index of the agent is reduced to a preset value, the agent is deleted from the target party's extended agent cluster or the opponent party's extended agent cluster;

[0020] The agents in the target extended agent cluster and the agents in the opponent extended agent cluster are controlled to perform adversarial training to update the agent parameters of the agents in the target extended agent cluster and the opponent extended agent cluster.

[0021] In one embodiment of the present application, after the step of deleting the agent from the target party's extended agent cluster or the opponent party's extended agent cluster, the method further includes:

[0022] In the target party extended agent cluster or the opponent party extended agent cluster, obtaining multiple agents having the same optimization objectives and constraints as the deleted agent;

[0023] Copy the agent with the largest winning index among multiple agents to obtain the first twin agent to supplement the deleted agent; or,

[0024] Based on the agent parameters of multiple agents, a second twin agent is constructed to supplement the deleted agents.

[0025] In one embodiment of the present application, the step of controlling the agents in the target party's extended agent cluster to perform preset rounds of simulation confrontation with the agents in the opponent party's extended agent cluster to obtain the simulation confrontation result of each agent includes:

[0026] For any agent in the target party's extended agent cluster, control the agent to respectively conduct preset rounds of simulation confrontation with agents in the opponent party's extended agent cluster to obtain simulation confrontation results for each agent;

[0027] The simulation confrontation result includes the individual winning rate of the agent against each agent in the opponent's extended agent cluster and the average winning rate of all agents in the opponent's extended agent cluster.

[0028] In an embodiment of the present application, the method further includes:

[0029] Determine the number of wins for which the single-unit win rate of the intelligent agent is greater than a win rate threshold;

[0030] Determine a first winning number threshold based on the total number of agents in the opponent's extended agent cluster and the first ratio; determine that the simulation confrontation result of the agent meets the winning condition when the winning number is greater than or equal to the first winning number threshold;

[0031] Based on the total number of agents in the opponent's extended agent cluster and the second ratio, a second winning number threshold is determined; when the winning number is greater than or equal to the second winning number threshold and less than the first winning number threshold, and the average winning rate is greater than or equal to the average winning rate threshold, it is determined that the simulation confrontation result of the agent meets the general condition;

[0032] When the number of wins is less than the second number of wins threshold, or the average winning rate is less than the average winning rate threshold, it is determined that the simulation confrontation result of the intelligent agent meets the failure condition.

[0033] In one embodiment of the present application, for any target agent cluster, controlling the target agent cluster to perform adversarial training with multiple opponent agent clusters in the opponent pool to train multiple personalized agents corresponding to each initial target agent in the target agent cluster, includes:

[0034] Copying the target agent cluster to obtain a plurality of personalized agent clusters corresponding to the plurality of opponent pools; each of the personalized agent clusters includes a plurality of initial target agents in the target agent cluster;

[0035] For any of the initial target agents in any of the personalized agent clusters, repeat the following steps: randomly selecting an initial opponent agent from the opponent agent cluster in the opponent pool corresponding to the personalized agent cluster to perform adversarial training with the initial target agent to update the agent parameters of the initial target agent;

[0036] When it is determined that the initial target party agent meets a preset training cutoff condition, a personalized agent obtained by training the initial target party agent based on the opponent pool is obtained.

[0037] In one embodiment of the present application, the step of obtaining a personalized target party agent corresponding to the initial target party agent based on agent parameters of the plurality of personalized agents includes:

[0038] Determine the average value of the agent parameters of the plurality of personalized agents corresponding to the initial target agent as the agent parameter of the personalized target agent corresponding to the initial target agent; or

[0039] For any personalized agent corresponding to the initial target agent, a weight parameter of the personalized agent is determined based on a ratio between the number of opponent agent clusters in the opponent pool corresponding to the personalized agent and the total number of opponent agent clusters in all opponent pools; and agent parameters of the personalized target agent corresponding to the initial target agent are determined based on the weight parameters and agent parameters corresponding to each of the plurality of personalized agents.

[0040] In one embodiment of the present application, after the step of obtaining the target-side balancing agent corresponding to each initial target-side agent, the method further includes:

[0041] Acquire target party information input by a user; wherein the target party information includes a target party optimization target and / or a target party constraint condition input by the target party;

[0042] Based on the target party information, at least one target equalization agent is determined from a plurality of target party equalization agents; wherein the at least one target equalization agent is used to generate a countermeasure against an opponent of the target party.

[0043] In an embodiment of the present application, the method further includes:

[0044] For any agent in the target party's expanded agent cluster, whenever the agent completes the training step, the training times of the agent are increased by 1;

[0045] After the step of obtaining the target-side equalization agent corresponding to each initial target-side agent, the method further includes:

[0046] Acquire target party information and counterparty information input by a user; wherein the target party information includes a target party optimization target and / or a target party constraint condition input by the target party, and the counterparty information includes a counterparty optimization target and / or a counterparty constraint condition input by the counterparty;

[0047] Based on the target party information, at least one target equalization agent is determined from a plurality of target party equalization agents, and at least one target agent with the largest number of training times is determined from an updated target party extended agent cluster;

[0048] Based on the counterparty information, determining at least one target counterparty agent in the updated counterparty extended agent cluster;

[0049] Controlling at least one of the target agents and at least one of the target equalization agents to respectively perform simulation confrontation with at least one of the target opponent agents in a target simulation confrontation environment, and generating a confrontation evaluation table according to a preset format;

[0050] The confrontation evaluation table is outputted.

[0051] In a second aspect, based on the same inventive concept, an embodiment of the present application provides an agent training device based on inter-cluster confrontation, the device comprising:

[0052] An agent cluster construction module is used to determine any of the two adversarial parties as a target party, and to construct multiple target party agent clusters of the target party based on multiple target party optimization objectives and multiple target party constraints of the target party under the target task boundary; the target party agent cluster includes multiple initial target party agents, and different initial target party agents have different target party optimization objectives and / or different target party constraints;

[0053] An opponent pool construction module is used to determine the other party of the two opposing parties relative to the target party as the opponent, and combine multiple opponent agent clusters of the opponent to obtain multiple opponent pools of the target party agent cluster; wherein different opponent pools are composed of different opponent agent clusters, and each opponent agent cluster includes multiple initial opponent agents;

[0054] The first training module is used for controlling any target agent cluster to perform adversarial training with multiple opponent agent clusters in the opponent pool, so as to obtain multiple personalized agents corresponding to each initial target agent in the target agent cluster, and obtain a personalized target agent corresponding to the initial target agent based on agent parameters of the multiple personalized agents;

[0055] An agent storage module, used to store the personalized target agent and multiple personalized agents corresponding to each initial target agent into the target extended agent cluster of the target party, and store the personalized opponent agent and multiple personalized agents corresponding to each initial opponent agent into the opponent extended agent cluster of the opponent;

[0056] A second training module is used to control the agents in the target party's extended agent cluster and the opponent party's extended agent cluster to perform adversarial training to update the agents in the target party's extended agent cluster and the opponent party's extended agent cluster;

[0057] A simulation confrontation module is used to control any agent in the updated target party extended agent cluster to perform simulation confrontation with an agent in the updated opponent party extended agent cluster, so as to obtain a simulation confrontation score for each agent in the target party extended agent cluster;

[0058] A parameter determination module is used to obtain a target-side balanced intelligent agent with the target-side optimization objective and the target-side constraint condition for multiple intelligent agents in the target-side extended intelligent agent cluster, based on the intelligent agent parameters of a preset proportion of intelligent agents with high simulation adversarial scores among the multiple intelligent agents, so as to obtain a target-side balanced intelligent agent corresponding to each initial target-side intelligent agent.

[0059] Compared with the prior art, this application has the following advantages:

[0060] An agent training method based on inter-cluster confrontation is provided in an embodiment of the present application. First, agent clusters of the two opposing parties are constructed according to the optimization objectives and constraints corresponding to each of the two opposing parties. Then, for any target agent cluster of the target party, multiple opponent pools are constructed. Since the opponent agent cluster in each opponent pool includes multiple opponent agents with different opponent optimization objectives and different opponent constraints, personalized agents that can adapt to different opponent styles and personalized target agents with strong comprehensive capabilities can be trained by conducting confrontation training between the target agent cluster and the opponent agent clusters in multiple opponent pools, and then a target extended agent cluster of the target party and an opponent extended agent cluster of the opponent are obtained. By controlling the agents that can adapt to different styles in the target extended agent cluster and the opponent extended agent cluster to conduct confrontation training, and controlling the trained agents of the two opposing parties to conduct simulated confrontation, a simulated confrontation score of each agent can be obtained, and then according to the agent parameters of the agent with the highest simulation confrontation score, a target balanced agent with different target optimization objectives and different target constraints can be finally obtained. In this way, the generalization ability of the target party's balancing agent can be effectively improved, so that the target party's balancing agent can better adapt to various opponents with different styles and tendencies. When facing complex and changeable confrontation tasks, even if it has no understanding of the opponent, the target party's balancing agent can achieve better confrontation results. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0062] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0063] Figure 1 This is a flowchart of the steps of an intelligent agent training method based on inter-cluster confrontation in one embodiment of the present application.

[0064] Figure 2 It is a schematic diagram of the confrontation between intelligent agent clusters in one embodiment of the present application.

[0065] Figure 3 This is a schematic diagram of adversarial training of a personalized agent cluster in one embodiment of the present application.

[0066] Figure 4 It is a module diagram of an intelligent agent training device based on inter-cluster confrontation in one embodiment of the present application. DETAILED DESCRIPTION

[0067] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0068] It should be noted that game confrontation can occur in many different scenarios and has applications in machine learning, game theory, multi-agent systems, and reinforcement learning. Game confrontation has important applications in adversarial learning, chess games, security defense, and network attack and defense.

[0069] For example, in the field of drones, as the application areas of unmanned aerial vehicles increase, the security threats brought by unmanned aerial vehicles are also gradually increasing. For example, in important places such as airports, hydro (nuclear) power stations, important meetings, large gatherings and sports events, due to the requirements of site safety, unmanned aerial vehicles are often banned from flying. When unauthorized unmanned aerial vehicles enter important places, they need to be countered and hunted down to reduce the risks and losses they bring. As a result, a series of technologies for directional attack, countermeasures and capture of unmanned aerial vehicles have been developed, such as unmanned aerial vehicle detection and attack integrated systems, capture and defense systems, etc. The drone capture device in the above system is usually equipped with an intelligent agent, which can generate corresponding confrontation plans according to the current situation (number of resources, resource location, resource capacity, opponent's location, type, etc.) and the purpose of confrontation (strike, drive away, damage target drones, etc.).

[0070] In traditional intelligent agent training methods, it is usually based on rule construction (the code is mainly conditional judgment, such as if-then logic code) or traditional optimization algorithms. Due to the high real-time requirements of game confrontation, greedy algorithms can generally be used, but the algorithm can easily fall into local optimality, and the confrontation effect is not significantly improved compared to the rule-based method; with the continuous emergence of artificial intelligence technology, researchers have applied artificial intelligence technology to decision-making intelligent agent construction methods, especially the decision-making intelligent agent construction method based on reinforcement learning, which has achieved relatively good operating results in decision-making problems.

[0071] The intelligent agent training method based on reinforcement learning first constructs the intelligent agent network structure according to the input, output and problem complexity of the intelligent agent required in the confrontation scenario, and then constructs a simulation training environment consistent with the intelligent agent application environment, such as a drone confrontation environment or a twin simplified environment of the drone confrontation environment. Finally, the reinforcement learning algorithm is used to train the intelligent agent network in the simulation training environment until convergence, and a decision-making intelligent agent can be obtained.

[0072] The essence of the confrontation problem is game confrontation, and the opponent's situation needs to be considered during the confrontation, such as the opponent's intelligence level, available tactics, confrontation style and tendencies, etc. However, in the general construction of decision-making agents based on reinforcement learning, the opponent / opponent's confrontation methods in the simulation training environment are basically based on rules or a single style / tendency.

[0073] Based on the simulation training environment of rules or single style / tendency opponents, the trained intelligent agents have limited generalization ability and can only deal with the rule opponents or single style / tendency opponents. However, confrontation problems are often much more complicated. For example, in drone confrontation, the drones to be captured may have different flight capabilities and escape strategies, or the operators of the drones may be people with different styles, different operating habits or different operating levels, which leads to poor confrontation performance of the intelligent agents of the drone capture device and cannot adapt to opponents with changing styles. At the same time, in actual confrontation scenarios, the information about the opponent's drones that can be obtained is usually small, and there is often a situation where the opponent is completely unknown. At this time, it will be difficult for the operator to select a suitable intelligent agent to perform the corresponding confrontation task, which leads to poor confrontation effect of the intelligent agent.

[0074] In view of the problem that the generalization ability of the intelligent agents trained in the relevant technology is limited and it is difficult to cope with complex and changeable adversarial tasks without understanding the opponent, the present application aims to provide an intelligent agent training method based on inter-cluster adversarial. First, multiple intelligent agent clusters corresponding to the two adversarial parties are constructed, and personalized training is performed on any target party intelligent agent cluster to train personalized intelligent agents that can adapt to different opponent styles and personalized target party intelligent agents with strong comprehensive capabilities, and then obtain the target party extended intelligent agent cluster of the target party and the opponent party extended intelligent agent cluster of the opponent party; then, by controlling the intelligent agents that can adapt to different styles in the target party extended intelligent agent cluster and the opponent party extended intelligent agent cluster to perform adversarial training, and controlling the trained intelligent agents of the two adversarial parties to perform simulated confrontation, the simulated confrontation score of each intelligent agent can be obtained, and then according to the intelligent agent parameters of the intelligent agent with the highest simulated confrontation score, the target party balanced intelligent agent under different target party optimization objectives and different target party constraints can be finally obtained. In this way, the generalization ability of the target party's balancing agent can be effectively improved, so that the target party's balancing agent can better adapt to various opponents with different styles and tendencies. When facing complex and changeable confrontation tasks, even if it has no understanding of the opponent, the target party's balancing agent can achieve better confrontation results.

[0075] Reference Figure 1 , showing an agent training method based on inter-cluster confrontation in the present application, the method may include the following steps:

[0076] S101: Determine any one of the two adversarial parties as the target party, and construct multiple target party agent clusters of the target party based on multiple target party optimization objectives and multiple target party constraints under the target task boundary.

[0077] In this embodiment, the target party agent cluster includes multiple initial target party agents, and different initial target party agents have different target party optimization objectives and / or different target party constraints.

[0078] It should be noted that the two opposing parties refer to the two parties participating in the game confrontation. For example, the two opposing parties can be defined as the red party and the blue party, or the two opposing parties are the attacker and the defender. In the field of drones, the target party can refer to the drone capture device, and the opponent party refers to the drone to be captured.

[0079] In this embodiment, according to the training requirements, any one of the two adversarial parties can be determined as the target party. Once one of the two adversarial parties is determined as the target party, the other party of the two adversarial parties relative to the target party can be used as the opponent party. For the target party, multiple target party optimization objectives and multiple target party constraints of the target party under the target task boundary are determined; for the opponent party, multiple opponent party optimization objectives and multiple opponent party constraints of the opponent party under the target task boundary are determined.

[0080] In this embodiment, the target task boundary represents the boundary conditions for the opposing parties. The opposing parties need to make corresponding confrontation strategies within the target task boundary. Specifically, the target task boundary includes the problem input and problem output of the opposing parties and the confrontation resources that the opposing parties can dispatch, such as the resource teams and equipment that can be dispatched, and the confrontation means that can be used.

[0081] In this embodiment, the optimization target represents the strategic target when the intelligent agent formulates the confrontation strategy, which can reflect the intention of the operator / commander. For example, the optimization target may include: the fastest time to complete the goal, the least number of waves to complete the goal, the least number of equipment to complete the goal, the lowest resource consumption to complete the goal, the highest consumption ratio to complete the goal, the least team to call to complete the goal, and / or the use of more resource points to complete the goal. Specifically, the optimization target may include the target optimization target and the opponent optimization target.

[0082] In this embodiment, the constraint conditions represent the limiting conditions when the agent formulates the confrontation strategy, which can reflect the subjective constraints of the operator / commander or the resource constraints under the influence of objective resource conditions. For example, the constraint conditions may include: the type of equipment used, which resource teams to call, which resource points to use, whether to adopt a collaborative method, how to collaborate, success rate requirements, total wave requirements and / or time period requirements, etc. Specifically, the constraint conditions may include target constraints and opponent constraints.

[0083] In this embodiment, different target optimization goals correspond to different target agent clusters. That is, for each target optimization goal, a target agent cluster is constructed, and each target agent cluster includes multiple initial target agents, and different initial target agents have different target constraints. In this way, for any initial target agent, the initial target agent has a unique corresponding target optimization goal and target constraint.

[0084] In an example, for the target party, there are N target party optimization objectives and P target party constraints, then there are N target party agent clusters and N×P initial target party agents.

[0085] S102: Determine the other party of the two opposing parties relative to the target party as the opponent party, and combine multiple opponent party agent clusters of the opponent party to obtain multiple opponent pools of the target party agent cluster.

[0086] In this embodiment, different opponent pools are composed of different opponent agent clusters, and each opponent agent cluster includes multiple initial opponent agents.

[0087] It should be noted that the construction process of the opponent's intelligent agent cluster is the same as the construction process of the target party's intelligent agent cluster, that is, for any opponent's optimization objective, an opponent's intelligent agent cluster under the opponent's optimization objective is constructed, and based on multiple opponent constraints, multiple initial opponent intelligent agents in each opponent's intelligent agent cluster are constructed; among which different opponent optimization objectives correspond to different opponent intelligent agent clusters.

[0088] It should be noted that the structure of the neural network adopted by each intelligent agent can be selected in multiple ways, according to the complexity of the task. The constraints can be introduced into the intelligent agent by adding a shielding matrix at the output end or by using reinforcement learning feedback values ​​during training. The former requires adjusting the network structure, while the latter does not need to change the network structure.

[0089] In this embodiment, each agent can be labeled with features, which include agent labels, task boundaries corresponding to the agent, constraints, optimization goals, round conditions, etc., where the round conditions correspond to training rounds.

[0090] Reference Figure 2 , showing a schematic diagram of the confrontation between intelligent clusters. Define the target party (drone capture device) as the red party and the opponent party (drone) as the blue party, then Figure 2A schematic diagram showing the confrontation between 4 red agent clusters and 3 blue agent clusters with different optimization goals under the same target task boundary. Among them, Agent_H1, Agent_H2, Agent_H3, and Agent_H4 are red agent clusters; Agent_L1, Agent_L2, and Agent_L3 are blue agent clusters; the target task boundary is: the red team cooperates with the blue team to confront the red team, the red team hopes to capture the blue team to achieve the goal, and the blue team has a certain ability to protect itself and has a certain ability to resist the capture of the red team.

[0091] The common point of Agent_H1, Agent_H2, Agent_H3, and Agent_H4 is that they can receive input information from the Red Team and output Red Team strategy information. The input information includes information such as Red Team collaborative operations, Blue Team situation information (such as type, location, number, flight capability, etc.), and Red Team optimization goals under the target mission boundary; Red Team strategy information includes: Red Team's action plan, such as the task team called, the equipment used, the target to be captured, the timing, etc. It should be noted that the difference between the above four Red Team agent clusters is that they have different Red Team constraints.

[0092] The task boundaries in the feature labels of the red agents in Agent_H1, Agent_H2, Agent_H3, and Agent_H4 are the same, but the optimization goals are different. For example, the red agent in Agent_H1 optimizes the fastest time under the premise of completing the goal, the red agent in Agent_H2 optimizes the least number of waves under the premise of completing the goal, the red agent in Agent_H3 optimizes the least number of equipment under the premise of completing the goal, and the red agent in Agent_H4 optimizes the least cost under the premise of completing the goal. The constraint feature labels of the red agents in each red agent cluster are also different.

[0093] Agent_L1, Agent_L2, and Agent_L3 are blue agent clusters. Similar to the red agent clusters, the blue agents in them all have the same point of receiving blue input information and outputting blue strategy information. The blue input information includes the red situation information, the blue optimization target under the target task boundary, and other information; the blue strategy information includes: the blue action plan, such as escape strategy and action plan, interception strategy and action plan, formation strategy and action plan, etc. It should be noted that the difference between the above three blue agent clusters is that they have different blue constraints.

[0094] Similarly, the task boundaries in the feature labels of the blue agents in Agent_L1, Agent_L2, and Agent_L3 are the same, but the optimization goals are different. For example, the blue agent in Agent_L1 optimizes the highest survival rate under the premise of completing the goal, the blue agent in Agent_L2 optimizes the lowest cost under the premise of completing the goal, and the blue agent in Agent_L3 optimizes the maximum use of reusable soft kills under the premise of completing the goal. The constraint feature labels of the blue agents in each blue agent cluster are also different.

[0095] Continue with Figure 2 As an example of the confrontation scenario shown in the figure, the target party is the red party. Then, the three blue party agent clusters of the blue party can be combined to obtain the red party agent cluster (C 1 3+C 2 3+C 3 3=) 7 opponent pools. Specifically, the 7 opponent pools can be included in the following set:

[0096] Opponent Pool 1: {Agent_L1}.

[0097] Opponent Pool 2: {Agent_L2}.

[0098] Opponent Pool 3: {Agent_L3}.

[0099] Opponent pool 4: {Agent_L1, Agent_L2}.

[0100] Opponent pool 5: {Agent_L1, Agent_L3}.

[0101] Opponent Pool 6: {Agent_L2, Agent_L3}.

[0102] Opponent Pool 7: {Agent_L1, Agent_L2, Agent_L3}.

[0103] It should be noted that if the blue side has k blue agent clusters, then for the red side, at most C 1 k +C 2 k +C 3 k +…+C k k A pool of opponents.

[0104] S103: For any target party intelligent agent cluster, control the target party intelligent agent cluster to perform adversarial training with the opponent party intelligent agent clusters in multiple opponent pools respectively, so as to train and obtain multiple personalized intelligent agents corresponding to each initial target party intelligent agent in the target party intelligent agent cluster, and obtain the personalized target party intelligent agent corresponding to the initial target party intelligent agent based on the intelligent agent parameters of the multiple personalized intelligent agents.

[0105] In this embodiment, in order to enable any initial target party agent in any target party agent cluster to adapt to different opponents, personalized training will be carried out for each initial target party agent, that is, different opponents are controlled to conduct adversarial training with the initial target party agent respectively, so that each initial target party agent can be trained to obtain multiple personalized agents.

[0106] In this embodiment, after training to obtain multiple personalized agents corresponding to the initial target party agent, the agent parameters of the multiple personalized agents will be comprehensively considered to obtain the agent parameters of the personalized target party agent corresponding to the initial target party agent. Since each personalized agent is adapted to the opponent style in the corresponding opponent pool, the personalized target party agent obtained by comprehensively considering the agent parameters of the multiple personalized agents corresponding to the initial target party agent has a strong comprehensive ability.

[0107] In one example, the red side has N optimization goals and P red side constraints, then there are N red side agent clusters, and N×P initial red side agents. Multiple blue side agent clusters of the blue side are combined to obtain 10 opponent pools. After the above training process, the red side can be expanded to obtain 10×N×P personalized agents. After the parameters of the 10×N×P personalized agents are integrated, N×P personalized red side agents corresponding to the N×P initial red side agents are finally obtained. Among them, each personalized red side agent has a unique red side optimization goal and red side constraint, and each personalized red side agent can effectively fight against opponents with different blue side optimization goals and / or different blue side constraints.

[0108] In a specific implementation, in S103, for any target agent cluster, controlling the target agent cluster to perform adversarial training with opponent agent clusters in multiple opponent pools to obtain multiple personalized agents corresponding to each initial target agent in the target agent cluster, may specifically include the following sub-steps:

[0109] S103-1: Copy the target agent cluster to obtain multiple personalized agent clusters corresponding to multiple opponent pools.

[0110] It should be noted that after any target agent cluster is replicated, each personalized agent cluster includes multiple initial target agents in the target agent cluster. In other words, by training each opponent pool separately, multiple personalized decision-making agent clusters with the same number as the opponent pool can be obtained.

[0111] Reference Figure 3 , showing a schematic diagram of adversarial training of personalized agent clusters. Figure 3 The explanation is based on Agent_H1 in the above four red agent clusters. Since Agent_H1 has a 7-point opponent pool, 7 personalized agent clusters can be obtained, namely Agent_H1_1, Agent_H1_2, Agent_H1_3, Agent_H1_4, Agent_H1_5, Agent_H1_6, and Agent_H1_7, which are 7 personalized agent clusters obtained by Agent_H1 selecting different opponent pools.

[0112] In this embodiment, a personalized feature label is set for each personalized agent in each personalized agent cluster, and the personalized feature label includes personalized agent label, task boundary, constraint condition, optimization target, opponent situation and round situation, etc.

[0113] By analogy, the remaining three Red Agent clusters can be obtained: Agent_H2_1, Agent_H2_2, ..., Agent_H2_7, Agent_H3_1, Agent_H3_2, ..., Agent_H3_7, Agent_H4_1, Agent_H4_2, Agent_H4_3, ..., Agent_H4_7. In this way, a total of 28 Red Agent personalized agent clusters can be obtained.

[0114] If you need to train a personalized blue agent, the training method is similar to that of the red agent. The following example illustrates this:

[0115] Taking the training of the blue agent cluster Agent_L1 as an example, we first construct the following (C 1 4+C 2 4+C 3 4+C 4 4=) 15 opponent pools.

[0116] Opponent Pool 1: {Agent_H1}.

[0117] Opponent Pool 2: {Agent_H2}.

[0118] Opponent Pool 3: {Agent_H3}.

[0119] Opponent Pool 4: {Agent_H4}.

[0120] Opponent Pool 5: {Agent_H1, Agent_H2}.

[0121] Opponent Pool 6: {Agent_H1, Agent_H3}.

[0122] Opponent Pool 7: {Agent_H1, Agent_H4}.

[0123] Opponent pool 8: {Agent_H2, Agent_H3}.

[0124] Opponent Pool 9: {Agent_H2, Agent_H4}.

[0125] Opponent Pool 10: {Agent_H3, Agent_H4}.

[0126] Opponent pool 11: {Agent_H1, Agent_H2, Agent_H3}.

[0127] Opponent pool 12: {Agent_H1, Agent_H2, Agent_H4}.

[0128] Opponent Pool 13: {Agent_H1, Agent_H3, Agent_H4}.

[0129] Opponent pool 14: {Agent_H2, Agent_H3, Agent_H4}.

[0130] Opponent Pool 15: {Agent_H1, Agent_H2, Agent_H3, Agent_H4}.

[0131] It should be noted that if the red side has s red agent clusters, then for the blue side, at most C 1 s+C 2 s+C 3 s+…+C s s opponent pools.

[0132] For each opponent pool, training is carried out separately, and 15 blue side personalized agent clusters can be obtained, including Agent_L1_1, Agent_L1_2, ..., Agent_L1_15, and so on: Agent_L2_1, Agent_L2_2, ..., Agent_L2_15, Agent_L3_1, Agent_L3_2, ..., Agent_L3_15. Each cluster still contains personalized agents with different constraints, and there are a total of 45 blue side personalized agent clusters.

[0133] S103-2: For any initial target agent in any personalized agent cluster, repeat the following steps: randomly select an initial opponent agent from the opponent agent cluster in the opponent pool corresponding to the personalized agent cluster to perform adversarial training with the initial target agent to update the agent parameters of the initial target agent.

[0134] In this embodiment, after obtaining multiple personalized agent clusters of the target party, iterative training will be performed in a layered adversarial manner, that is, each personalized agent cluster will carry out multiple rounds of adversarial training with the opponent agent cluster in the corresponding opponent pool.

[0135] For example, continue to refer to Figure 3 For the initial target agent in Agent_H1, only one initial opponent agent in Agent_L1 will be randomly selected for adversarial training, while for Agent_H7, one initial opponent agent will be randomly selected from Agent_L1, Agent_L2, and Agent_L3 for adversarial training.

[0136] Specifically, during the adversarial process, for the initial target agent in each personalized agent cluster, the initial target agent needs to undergo adversarial training with different opponent agents in the opponent pool to continuously update and optimize its agent parameters, so as to finally obtain the corresponding personalized agent.

[0137] It should be noted that since each personalized agent is trained based on different opponent pools, each personalized agent can effectively fight against opponents with the same fighting style as the corresponding opponent pool. For example, a personalized agent with a certain red constraint in Agent_H1 can effectively fight against various blue agents with different blue constraints in Agent_L1.

[0138] S103-3: When it is determined that the initial target party agent meets the preset training cutoff condition, a personalized agent is obtained by training the initial target party agent based on the opponent pool.

[0139] In this embodiment, during the iterative training process, if it is determined that the initial target party agent meets the preset training cutoff condition, the current agent parameters of the initial target party agent are fixed to obtain a personalized agent that completes the adversarial training.

[0140] In this embodiment, the training cutoff condition can be set as: the training round of the initial target party agent reaches the round threshold and / or the update change amplitude of the agent parameter of the initial target party agent is less than the amplitude threshold.

[0141] In a specific implementation, the step of obtaining the personalized target party agent corresponding to the initial target party agent based on the agent parameters of the multiple personalized agents in S103 may specifically include the following sub-steps:

[0142] S103-4: Determine the average value of the agent parameters of the multiple personalized agents corresponding to the initial target agent as the agent parameter of the personalized target agent corresponding to the initial target agent.

[0143] For example, for an initial target agent in Agent_H1, after layered adversarial training, corresponding personalized agents are trained in Agent_H1_1, Agent_H1_2, Agent_H1_3, Agent_H1_4, Agent_H1_5, Agent_H1_6, and Agent_H1_7 respectively. The average value of the agent parameters of the seven personalized agents can be used as the agent parameters of the initial target agent to obtain the personalized target agent corresponding to the initial target agent.

[0144] S103-5: For any personalized agent corresponding to the initial target agent, determine the weight parameter of the personalized agent based on the ratio between the number of opponent agent clusters in the opponent pool corresponding to the personalized agent and the total number of opponent agent clusters in all opponent pools; and determine the agent parameters of the personalized target agent corresponding to the initial target agent based on the weight parameters and agent parameters corresponding to each of the multiple personalized agents.

[0145] In this embodiment, considering that each personalized agent faces different opponents, that is, the number of opponent agent clusters in the opponent pool corresponding to each personalized agent may be different, the comprehensive performance of each personalized agent also has certain differences. For example, the comprehensive performance of a personalized agent trained based on three opponent agent clusters is usually better than that of a personalized agent trained based on one or two opponent agent clusters. Therefore, a weighted calculation will be performed on multiple personalized agents to obtain more accurate agent parameters of the personalized target agent.

[0146] Continue with Figure 2 Taking the confrontation scenario shown in the figure as an example, we can get the 7 opponent pools of the red agent cluster, including:

[0147] Agent_H1_1's opponent pool 1: {Agent_L1}.

[0148] Agent_H1_2's opponent pool 2: {Agent_L2}.

[0149] Agent_H1_3's opponent pool 3: {Agent_L3}.

[0150] Agent_H1_4's opponent pool 4: {Agent_L1, Agent_L2}.

[0151] Agent_H1_5's opponent pool 5: {Agent_L1, Agent_L3}.

[0152] Agent_H1_6's opponent pool 6: {Agent_L2, Agent_L3}.

[0153] Agent_H1_7's opponent pool 7: {Agent_L1, Agent_L2, Agent_L3}.

[0154] Therefore, it can be determined that the total number of opponent agent clusters in all opponent pools is 12, then the weight parameters of personalized agents in Agent_H1_1, Agent_H1_2 and Agent_H1_3 are all 1 / 12; the weight parameters of personalized agents in Agent_H1_4, Agent_H1_5 and Agent_H1_6 are all 2 / 12, that is, 1 / 6; the weight parameters of personalized agents in Agent_H1_7 are all 3 / 12, that is, 1 / 4.

[0155] In this embodiment, after calculating the weight parameters of multiple personalized agents corresponding to the initial target agent in multiple personalized agent clusters, the agent parameters of the multiple personalized agents can be weighted summed according to the weight parameters to obtain the agent parameters of the personalized target agent.

[0156] S104: storing the personalized target agent and multiple personalized agents corresponding to each initial target agent into the target extended agent cluster of the target party, and storing the personalized opponent agent and multiple personalized agents corresponding to each initial opponent agent into the opponent extended agent cluster of the opponent.

[0157] Exemplarily, for any initial target party agent in Agent_H1, after hierarchical adversarial training, 7 personalized agents can be trained in Agent_H1_1, Agent_H1_2, Agent_H1_3, Agent_H1_4, Agent_H1_5, Agent_H1_6, and Agent_H1_7 respectively, and based on the above 7 personalized agents, 1 personalized target party agent can be obtained; then, the 1 personalized target party agent and 7 personalized agents corresponding to the initial target party agent can be stored in the target party's target party extended agent cluster; similarly, for any initial opponent party agent in any opponent party agent cluster, the personalized opponent party agent and multiple personalized agents corresponding to the initial opponent party agent can be obtained, and then the opponent party extended agent cluster of the opponent can be obtained.

[0158] S105: Control the agents in the target party's extended agent cluster and the opponent party's extended agent cluster to perform adversarial training to update the agents in the target party's extended agent cluster and the opponent party's extended agent cluster.

[0159] In this embodiment, after each agent completes personalized training and obtains multiple personalized agents, each personalized agent will be controlled to perform further balanced training to update the agent parameters of the agents in the target extended agent cluster and the opponent extended agent cluster.

[0160] In this embodiment, taking an agent in the target party's extended agent cluster as an example, since the agent undergoes adversarial training with different agents in the opponent party's extended agent cluster, after adversarial training, the agent can effectively fight against agents of different styles.

[0161] S106: For any agent in the updated target extended agent cluster, control the agent to perform a simulation confrontation with the agent in the updated opponent extended agent cluster to obtain a simulation confrontation score for each agent in the target extended agent cluster.

[0162] In this embodiment, by controlling the agents to conduct simulated confrontation, the confrontation effect of each agent can be effectively verified. In other words, the higher the simulation confrontation score of the agent, the stronger its generalization ability is, and the better confrontation effect can be achieved when confronting different opponent agents.

[0163] It should be noted that the simulation confrontation score can be calculated based on a preset scoring rule. For example, different expected confrontation results can be set for agents with different optimization objectives and / or different constraints, and then the simulation confrontation score of the agent is determined based on the similarity between the simulation confrontation result of the agent and the expected confrontation result. The higher the similarity, the higher the simulation confrontation score.

[0164] S107: For multiple agents in the target-side extended agent cluster having the same target-side optimization objective and target-side constraints, a target-side balanced agent having the target-side optimization objective and target-side constraints is obtained based on the agent parameters of a preset proportion of agents with high simulation adversarial scores among the multiple agents, so as to obtain a target-side balanced agent corresponding to each initial target-side agent.

[0165] It should be noted that, for the target side, if there are N target side optimization goals and P target side constraints, then N×P initial target side agents can be constructed, and then after personalized training and balanced training, N×P target side balanced agents can be trained. Among them, different target side balanced agents have different target side optimization goals and / or different multiple target side constraints, that is, each target side balanced agent has a unique target side optimization goal and target side constraint.

[0166] In this embodiment, since multiple agents with the same target optimization objectives and target constraints undergo adversarial training and simulation confrontation during the balanced training phase, in order to fully ensure the generalization ability and comprehensive performance of the target balanced agent, the agent parameters of the agent with a high simulation confrontation score, that is, the agent with a good simulation confrontation performance, are selected to calculate the corresponding agent parameters of the target balanced agent.

[0167] In a specific implementation, the average value of the agent parameters of a preset proportion of agents with high simulation confrontation scores among multiple agents can be determined as the agent parameters of the target party's balanced agent.

[0168] In this embodiment, personalized training and balanced training are performed on any initial target agent in any target agent cluster, so that each trained target balanced agent can better adapt to various opponents with different styles and tendencies. When facing complex and changeable confrontation tasks, even when the opponent is completely unknown, the target balanced agent can achieve better confrontation effect.

[0169] In a feasible implementation, in S105, the following training steps may be repeated according to a preset number of repetitions:

[0170] S105-1: Control the agents in the target party's extended agent cluster to conduct preset rounds of simulation confrontation with the agents in the opponent party's extended agent cluster to obtain the simulation confrontation result of each agent.

[0171] It should be noted that the preset rounds can be set according to actual needs. For example, if computing power and time allow, a larger preset round can be used, such as setting the preset rounds to 100 times.

[0172] In the specific implementation, in order to fully verify the confrontation effect of each intelligent agent, any intelligent agent in the target party's extended intelligent agent cluster can be controlled to conduct preset rounds of simulation confrontation with the intelligent agents in the opponent's extended intelligent agent cluster to obtain the simulation confrontation results of each intelligent agent.

[0173] In this embodiment, the simulation confrontation result may specifically include the individual winning rate of the agent against each agent in the opponent's extended agent cluster and the average winning rate of all agents in the opponent's extended agent cluster.

[0174] Exemplarily, there are 200 agents in the opponent's extended agent cluster. For any agent in the target's extended agent cluster, the agent will be controlled to conduct 100 simulation confrontations with the 200 agents in the hand's extended agent cluster, and the win or loss results of each simulation confrontation will be counted; where the win or loss result is victory or failure. For example, agent a1 conducts 100 simulation confrontations with agent b1 in the hand's extended agent cluster, and wins 85 times. The individual winning rate of agent a1 relative to agent b1 is 85%; based on the individual winning rate of agent a1 against all agents in the opponent's extended agent cluster, the average winning rate of agent a1 against all agents in the opponent's extended agent cluster can be calculated.

[0175] S105-2: For any intelligent agent, when the simulation confrontation result of the intelligent agent meets the winning condition, the winning index of the intelligent agent is increased; when the simulation confrontation result meets the normal condition, the winning index of the intelligent agent is kept unchanged; when the simulation confrontation result meets the failure condition, the winning index of the intelligent agent is reduced, and when the winning index of the intelligent agent is reduced to a preset value, the intelligent agent is deleted from the target party's extended intelligent agent cluster or the opponent party's extended intelligent agent cluster.

[0176] In this embodiment, since there are agents with poor confrontation effects in the target party's extended agent cluster, in order to avoid meaningless multiple training of these agents, a winning index will be set for each agent. If the simulation confrontation result of the agent in a certain iteration training meets the winning condition, that is, it means that the agent performs well, then the winning index of the agent is increased; if the simulation confrontation result of the agent in a certain iteration training meets the ordinary condition, that is, it means that although the agent performs ordinary, it has certain training value, then the winning index of the agent is kept unchanged; if the simulation confrontation result of the agent in a certain iteration training meets the failure condition, that is, it means that the agent performs poorly, then the winning index of the agent is reduced.

[0177] In one example, after each round of adversarial training, if the agent meets the winning condition, the winning index is increased by 5; if the agent meets the normal condition, the winning index remains unchanged; if the agent meets the failure condition, the winning index is reduced by 1. In addition, the preset value can be set to 0, and an initial winning index greater than or equal to 1 is set for each agent. For example, it can be set to 1. When the winning index of an agent decreases to 0, the agent is deleted from the target party's extended agent cluster or the opponent party's extended agent cluster. By repeating the above process according to the preset number of repetitions, the winning index of each agent can be updated, and then the agents with poor training effects can be continuously eliminated from the target party's extended agent cluster or the opponent party's extended agent cluster. Among them, the number of repetitions can be set to 1000 times.

[0178] In this embodiment, in order to realize iterative calculation of the winning index, the agent training method based on inter-cluster confrontation may further include the following steps:

[0179] S201: Determine the number of wins where the single-unit win rate of the intelligent agent is greater than the win rate threshold.

[0180] In this embodiment, the winning rate threshold can be set to 85%. That is, for any agent in the target party's extended agent cluster, whenever the winning rate of the agent relative to an agent in the opponent party's extended agent cluster is greater than 85%, the number of wins is increased by 1. For example, the total number of agents in the opponent party's extended agent cluster is 200, and if the winning rate of 180 agents is greater than 85%, the number of wins is 180.

[0181] S202: Determine a first winning number threshold based on the total number of agents in the opponent's extended agent cluster and the first ratio; when the winning number is greater than or equal to the first winning number threshold, determine that the simulated confrontation result of the agent meets the winning condition.

[0182] Exemplarily, the first ratio can be set to 80%. If the total number of agents in the opponent's extended agent cluster is 200, the first winning number threshold is 160. If the winning number of a certain agent reaches 160, it means that the agent's individual winning rate against more than 80% of the agents in the opponent's extended agent cluster is greater than 85%, that is, the agent can achieve a high winning rate against most of the opponent's agents, and then determine that the agent's simulation confrontation results meet the winning conditions to increase its winning index.

[0183] S203: Determine a second winning number threshold based on the total number of agents in the opponent's extended agent cluster and the second ratio; when the winning number is greater than or equal to the second winning number threshold and less than the first winning number threshold, and the average winning rate is greater than or equal to the average winning rate threshold, determine that the simulation confrontation result of the agent meets the normal conditions.

[0184] In this embodiment, the average winning rate threshold can be set to 60%, and the second ratio is less than the first ratio, for example, it can be set to 20%. If the total number of agents in the opponent's extended agent cluster is 200, the second winning number threshold is 40. In other words, even if the number of winnings of a certain agent against the opponent's extended agent cluster does not reach 160, as long as the number of winnings reaches 40 and the average winning rate reaches 60%, it means that the agent has a certain confrontation potential, and it can continue to be trained and optimized, and then determine that the simulation confrontation results of the agent meet the general conditions to keep its winning index unchanged.

[0185] S204: When the number of wins is less than the second winning number threshold, or the average winning rate is less than the average winning rate threshold, it is determined that the simulation confrontation result of the intelligent agent meets the failure condition.

[0186] In this embodiment, if the number of wins of an agent against the opponent's extended agent cluster is less than the second win number threshold, it means that it cannot achieve a high winning rate against most of the opponent's agents, and the agent is considered to have failed in the current round of training; if the average winning rate of the agent is less than the average winning rate threshold, it means that its overall winning rate against the opponent's extended agent cluster is low, and the agent is also considered to have failed in the current round of training to reduce its winning index.

[0187] In this embodiment, in order to maintain the number of agents in the target party's extended agent cluster or the opponent party's extended agent cluster, after the step of deleting the agent from the target party's extended agent cluster or the opponent party's extended agent cluster in S105-2, the agent training method based on inter-cluster confrontation may further include the following steps:

[0188] S301: In the target party's extended agent cluster or the opponent party's extended agent cluster, obtain multiple agents that have the same optimization objectives and constraints as the deleted agent.

[0189] In this embodiment, since there are multiple agents under each specific combination of optimization objectives and constraints, after an agent with poor confrontation effect is eliminated, a twin agent can be obtained based on multiple agents with the same optimization objectives and constraints to supplement the deleted agent.

[0190] S302-A: Copy the agent with the largest winning index among multiple agents to obtain the first twin agent to supplement the deleted agent.

[0191] In this embodiment, the agent with the largest winning index, that is, the agent with the best performance in confrontation, can be directly selected from the remaining multiple agents and copied to obtain the first twin agent for supplementation.

[0192] It should be noted that when the agents in the target extended agent cluster and the agents in the opponent extended agent cluster conduct preset rounds of simulation confrontation, each round of simulation confrontation will change the situation and other simulation confrontation environments so that the agents can cope with different confrontation scenarios, where the situation information in the simulation confrontation environment can be obtained through random sampling. In other words, in different simulation confrontation environments, the same agent may make different decisions, and then update different agent parameters.

[0193] S302-B: Based on the agent parameters of multiple agents, a second twin agent is constructed to supplement the deleted agent.

[0194] In this embodiment, the second twin agent can also be obtained by comprehensively considering the agent parameters of multiple agents. Specifically, the average value of the agent parameters of multiple agents can be determined as the agent parameters of the second twin agent; or, based on the winning indicators of multiple agents, the agent parameters of multiple agents can be weighted and summed to obtain the agent parameters of the second twin agent.

[0195] In this embodiment, by supplementing the deleted agents, it is possible to ensure that there are corresponding agents that can participate in adversarial training under each specific combination of optimization objectives and constraints, and ultimately obtain the target side balanced agents corresponding to each initial target side agent.

[0196] S105-3: Control the agents in the target extended agent cluster to perform adversarial training with the agents in the opponent extended agent cluster to update the agent parameters of the agents in the target extended agent cluster and the opponent extended agent cluster.

[0197] It should be noted that, unlike simulation adversarial training which is used to output simulation adversarial scores, adversarial training is used to reversely update the agent parameters of the agent according to the results of adversarial training to achieve iterative optimization of the agent parameters.

[0198] In a specific implementation, for any agent in any target extended agent cluster, the adversarial training for the agent can be terminated if the agent meets the preset training cutoff condition, until all agents in the target extended agent cluster and the opponent extended agent cluster complete the training, and an updated target extended agent cluster and an updated opponent extended agent cluster are obtained.

[0199] In a feasible implementation, after S107, the agent training method based on inter-cluster confrontation may further include the following steps:

[0200] S401: Acquire target party information input by the user.

[0201] It should be noted that the target party information includes the target party optimization target and / or target party constraint condition input by the user for the target party. The user may specifically be a commander or operator of the drone capture device, and the following description will be based on the operator.

[0202] In this embodiment, if there are N target optimization objectives and P target constraints, N×P target balancing agents can be trained in the end, where different target balancing agents have unique target optimization objectives and target constraints, and each target balancing agent can effectively fight against opponents with different specific opponent optimization objectives and / or different opponent constraints.

[0203] In one example, the operator, as the drone capturer, knows the target information such as the capture strategy and flight capability of the drone capture device, and can then input the corresponding target optimization objectives and / or target constraints to screen out at least one corresponding target balancing agent from multiple target balancing agents.

[0204] S402: Based on the target party information, determine at least one target balancing agent from multiple target party balancing agents.

[0205] It should be noted that at least one target equalization agent is used to generate a countermeasure against the opponent of the target party. When the target party is a drone capture device, the opponent party is the drone to be captured.

[0206] In this embodiment, the operator can select a corresponding target balancing agent based on the target party information.

[0207] For example, if the operator inputs the target optimization goal and the target constraint conditions at the same time, a unique target balancing intelligent agent can be determined; if the operator only inputs the target optimization goal and there are 10 target constraints, 10 target balancing intelligent agents with the same target optimization goal but different target constraints can be selected from multiple target balancing intelligent agents.

[0208] In this embodiment, when there are multiple target balancing agents, multiple sets of confrontation plans can be generated, and the operator can make further selections from the multiple sets of confrontation plans based on experience, or directly confront the opponent according to the multiple sets of confrontation plans.

[0209] In this embodiment, the operator selects a suitable target balancing agent to perform the corresponding confrontation task based on actual needs without knowing the opponent at all. Since the screened target balancing agent has strong robustness and comprehensive ability, when facing complex and changeable confrontation tasks, the target balancing agent can achieve better confrontation effect even without knowing the opponent at all.

[0210] In a feasible implementation, the agent training method based on inter-cluster confrontation may further include the following steps:

[0211] S501: For any agent in the target party's expanded agent cluster, each time the agent completes a training step, the number of training times of the agent is increased by 1.

[0212] In this embodiment, in the process of controlling the agents in the target party's extended agent cluster and the opponent party's extended agent cluster to conduct adversarial training, the number of training times for any agent in the target party's extended agent cluster will be counted. The greater the number of training times of the agent, the better and more stable the adversarial performance in the entire adversarial training. For example, when the number of repetitions is set to 1000, if the number of training times reaches 1000 times, it means that the agent has always existed in the target party's extended agent cluster and has never been eliminated; if each native agent in the target party's extended agent cluster has been eliminated, the number of training times of the subsequent added agents will be less than 1000 times.

[0213] In this embodiment, after S107, the agent training method based on inter-cluster confrontation may further include the following steps:

[0214] S601: Acquire the target party information and the counterparty party information input by the user.

[0215] It should be noted that the target party information includes the target party optimization target and / or target party constraint conditions input for the target party, and the counterparty information includes the counterparty optimization target and / or counterparty constraint conditions input for the counterparty.

[0216] In this embodiment, if the operator knows part of the opponent's information, the operator can input the known opponent's information in addition to the target party information. For example, when the opponent is a drone, the opponent's information can include the drone's flight capabilities and possible escape strategies or confrontation strategies.

[0217] S602: Based on the target party information, at least one target equalization agent is determined from among multiple target party equalization agents, and at least one target agent with the largest number of training times is determined from among the updated target party extended agent cluster.

[0218] It should be noted that the method of selecting the target balancing agent is referred to S401-S402 and will not be repeated here.

[0219] In this embodiment, while at least one target balancing agent is obtained by screening based on the target party information, at least one target agent with the largest number of training times is also determined in the updated target party extended agent cluster.

[0220] S603: Based on the counterparty information, determine at least one target counterparty agent in the updated counterparty extended agent cluster.

[0221] In this embodiment, the operator can select at least one target counterparty agent in the updated counterparty extended agent cluster based on known counterparty information or analyzed counterparty information.

[0222] S604: Control at least one target agent and at least one target equalization agent to respectively perform a simulation confrontation with at least one target opponent agent in a target simulation confrontation environment, and generate a confrontation evaluation table according to a preset format.

[0223] In this embodiment, considering that different target agents and different target balancing agents usually have different confrontation performances when facing different target opponent agents, in order to ensure that the finally selected agent can achieve better confrontation effects in actual confrontations, each target agent will be controlled to conduct a simulated confrontation with each target opponent agent. At the same time, each target balancing agent will also be controlled to conduct a simulated confrontation with each target opponent agent.

[0224] It should be noted that, unlike the confrontation training stage in S105, when the agents in the target extended agent cluster and the agents in the opponent extended agent cluster are controlled to conduct a preset round of simulation confrontation, it is necessary to change the situation and other information in the simulation confrontation environment. In this embodiment, the target agent or the target equalization agent will generate a target simulation confrontation environment, wherein the target simulation confrontation environment has a fixed target situation. In other words, the operator can determine the target situation according to the current actual confrontation scenario to construct the required target simulation confrontation environment for the agent to conduct simulation confrontation in the target simulation confrontation environment.

[0225] In this embodiment, based on the simulation confrontation results of each target agent and each target equalization agent, the simulation confrontation results of each target agent and each target equalization agent can be quantitatively scored, and then a confrontation evaluation table in a preset format can be generated.

[0226] S605: Outputting a confrontation evaluation table.

[0227] In this embodiment, by outputting a confrontation evaluation table, the operator can intuitively and comprehensively understand the confrontation situation between our intelligent agent and the opponent's intelligent agent, and then select the required intelligent agent from at least one target intelligent agent and at least one target balancing intelligent agent according to actual needs to generate a confrontation plan for the opponent.

[0228] In this embodiment, before the actual confrontation, at least one target intelligent agent and at least one target equalization intelligent agent with better confrontation performance are respectively subjected to simulated confrontation with at least one target opponent intelligent agent in a target simulation confrontation environment to simulate the actual confrontation scenario, and then the intelligent agent with the best performance in the simulated confrontation is put into use. In this way, the actual confrontation effect of the drone capture device can be effectively improved.

[0229] In a feasible implementation, in order to improve the training effect of the intelligent agent in the personalized training stage, before S103, the intelligent agent training method based on inter-cluster confrontation may further include the following steps:

[0230] S701: Initialize training for multiple initial target agents in the target agent cluster to determine initial agent parameters corresponding to each of the multiple initial target agents.

[0231] In this embodiment, since the agent parameters of each initial target agent are obtained randomly when constructing multiple initial target agents in each target agent cluster, if game adversarial training is directly conducted between agents, there may be problems such as poor training effect and long convergence time. Therefore, before conducting adversarial training between the two adversaries, each initial target agent will be initialized and trained so that each initial target agent has a certain adversarial ability, thereby obtaining the training efficiency and training effect of subsequent adversarial training.

[0232] In this embodiment, unlike adversarial training with an agent, the initialization training is only for the agent itself, and the opponents in the training environment are all constructed based on basic rules.

[0233] It should be noted that the basic rules include our rules and opponent rules. Among them, the opponent rules include the location constraints of the target group, the formation constraints of the target group, the flight capability constraints, the counterattack capability constraints, and the capability constraints of subordinates and vassals; our rules include position constraints, team resource quantity and capability constraints, equipment quantity and capability constraints, information assurance capability constraints, etc. The above-mentioned related constraints have corresponding optional constraint value ranges.

[0234] In the specific implementation, under the set task boundaries, reinforcement learning is used to train the initial intelligent agent. During each training, the situation, resources, confrontation, etc. of the enemy and us in the training environment appear randomly within the constrained numerical range of the basic rules.

[0235] It should be noted that the initialization training is for both adversaries, that is, the target party can be either of the two adversaries. Taking the red and blue parties as an example, the initial agent parameters of the red party's initial red agent and the initial agent parameters of the blue party's initial blue agent can be obtained by training respectively.

[0236] In the specific implementation, the corresponding training method is selected according to the structural design of the intelligent agent.

[0237] In a feasible implementation, S701 may specifically include the following sub-steps:

[0238] S701-A1: When the agent structures of multiple initial target party agents in the target party agent cluster are inconsistent, the multiple initial target party agents in the target party agent cluster are independently trained respectively until the multiple initial target party agents meet the preset first training cutoff condition, and the initial agent parameters corresponding to each of the multiple initial target party agents are determined.

[0239] In this embodiment, if the agent structures of multiple initial target-side agents in the target-side agent cluster are inconsistent, independent training is adopted for training.

[0240] Continue to refer to Figure 1 For example, Agent_H1 is a red agent cluster that optimizes the fastest time under the premise of completing the goal. When training a red agent in the red agent cluster Agent_H1, a simulation training environment is constructed through our basic rules and the opponent's basic rules. That is, the opponent and environment of the red agent to be trained are constructed through basic rules. The part that changes in each training is to randomly select specific values ​​within the rule constraint value range. For example, the relative position of positions in the formation of the target group has a certain range. For example, the interception strategies for offensive equipment include 1 intercept 1, 1 intercept 2, 1 intercept 3, etc. The training values ​​are randomly selected each time. Until convergence, a red agent in the red agent cluster Agent_H1 is obtained.

[0241] Initialization training is performed for the red and blue sides respectively, and the initial agent parameters of the initial red agents in the four red agent clusters Agent_H1, Agent_H2, Agent_H3, and Agent_H4 and the initial agent parameters of the initial blue agents in the three blue agent clusters Agent_L1, Agent_L2, and Agent_L3 can be obtained.

[0242] In a feasible implementation manner, S701 may further specifically include the following sub-steps:

[0243] S701-B1: When the agent structures of multiple initial target agents in the target agent cluster are consistent, remove the shielding matrix of any initial target agent to obtain the initial simplified agent corresponding to the target agent cluster, and determine the simplified optimization target of the initial simplified agent based on multiple target optimization targets.

[0244] It should be noted that the shielding matrix represents the target constraint conditions of the initial target agent. By removing the shielding matrix of the initial target agent, it means removing the target constraint conditions corresponding to the initial target agent.

[0245] In this embodiment, considering that when the agent structures of multiple initial target agents in the target agent cluster are consistent, the only difference is that the shielding matrices of the initial target agents are different, therefore, by removing the shielding matrices of multiple initial target agents, the same initial simplified agent can be obtained.

[0246] In this embodiment, the simplified optimization target used in the reinforcement learning training can be the average of the target optimization targets corresponding to each of the multiple initial target intelligent agents.

[0247] S701-B2: Based on the simplified optimization objective and the preset first reinforcement learning algorithm, the initial simplified intelligent agent is trained until the initial simplified intelligent agent meets the preset first training cutoff condition, thereby obtaining a trained first simplified intelligent agent.

[0248] In this embodiment, a simulation training environment is constructed through our basic rules and the opponent's basic rules, that is, the opponent and environment of the initial simplified intelligent agent to be trained are constructed through basic rules. The changing part in each training is the random selection of specific values ​​within the rule constraint value range. For example, the relative positions of positions in the formation of the target group have a certain range. For example, the interception strategies for offensive equipment include 1 block 1, 1 block 2, 1 block 3, etc., and the values ​​are randomly selected for each training.

[0249] In this embodiment, a first simplified agent is trained by continuously training in a simulation training environment until a preset first training cutoff condition is met, wherein the first training cutoff condition may include that the number of training rounds reaches a first round threshold and / or that the update change amplitude of the agent parameter is less than a first amplitude threshold.

[0250] S701-B3: Based on the target optimization objectives corresponding to each of the multiple target agent clusters and the preset second reinforcement learning algorithm, the first simplified agent is trained respectively until the first simplified agent meets the first training cutoff condition, thereby obtaining multiple second simplified agents corresponding one-to-one to the multiple target agent clusters.

[0251] In this embodiment, after the first simplified agent is trained, it will be further trained in the same simulation training environment as above with the target optimization goal of each target agent cluster to obtain a simplified second simplified agent for each target agent cluster.

[0252] S701-B4: For any second simplified agent, the agent parameters of the second simplified agent are used as the agent parameters of multiple initial target agents in the corresponding target agent cluster, and each initial target agent is trained based on a preset third reinforcement learning algorithm until the initial target agent meets the first training cutoff condition, thereby obtaining the initial agent parameters of the initial target agent.

[0253] It should be noted that the first reinforcement learning algorithm, the second reinforcement learning algorithm and the third reinforcement learning algorithm may be the same or different and may be selected from a preset reinforcement learning algorithm library according to actual needs.

[0254] In this embodiment, after training a simplified second simplified agent for each target agent cluster, the parameters of the second simplified agent correspond to the initial parameters of the initial target agent in the target agent cluster, and the initial target agent in each target agent cluster is retrained to obtain the initial target agent with target constraints. Taking Agent_H1 as an example, the parameters of the simplified second simplified agent trained for Agent_H1 are used as the initial values ​​of the parameters of the initial red agent, and retrained to obtain the initial red agent with constraints.

[0255] In this embodiment, initialization training is performed for the red and blue sides respectively, and the initial agent parameters of the initial red agents in the four red agent clusters Agent_H1, Agent_H2, Agent_H3, and Agent_H4 and the initial agent parameters of the initial blue agents in the three blue agent clusters Agent_L1, Agent_L2, and Agent_L3 can be obtained.

[0256] In this embodiment, by initializing and training multiple initial target agents in the target agent cluster, the prototypes of multiple initial target agents can be constructed, so that each initial target agent has a certain confrontation ability, that is, the initial agent parameters of each initial target agent have a certain feasibility. Compared with directly letting the confrontation parties conduct confrontation training of agents, the training efficiency and training effect of confrontation training can be effectively improved, so that the agent parameters can converge quickly.

[0257] In a feasible implementation, the agent training method based on inter-cluster confrontation may further include the following steps:

[0258] S801: When multiple personalized agent clusters are trained serially, determine a trained historical personalized agent cluster for an untrained current personalized agent cluster.

[0259] It should be noted that when computing resources are sufficient, multiple personalized agent clusters can be trained in parallel; when computing resources are insufficient, serial training is required. However, serial training is time-consuming. Therefore, in order to improve the efficiency of serial training, the parameter inheritance strategy will be used to optimize the agent parameters of the current personalized agent cluster that has not been trained.

[0260] For example, take the training of Agent_H1_4 as an example. Agent_L1 and Agent_L2 are included in the opponent pool of Agent_H1_4. In order to train a certain red agent in Agent_H1, the parameters of the agent in Agent_H1 are used as the initial parameters of Agent_H1_4 for training. The opponent is randomly selected from Agent_L1 and Agent_L2. The parameters of Agent_H1_4 are evolved using samples obtained from the training environment. This training method is time-consuming. The agent initialization method based on parameter inheritance does not use the parameters of the agent in Agent_H1 as the initial state of the corresponding agent parameters in Agent_H1_4, but through inheritance. In other words, the current personalized agent cluster that has not been trained can inherit the relevant agent parameters from the historical personalized agent cluster that has been trained, and then the initial target agent in the current personalized agent cluster can be optimized before adversarial training.

[0261] S802: For any historical personalized agent cluster, if the opponent pool of the historical personalized agent cluster includes any opponent agent cluster in the opponent pool of the current personalized agent cluster, the historical personalized agent cluster is determined to be an associated personalized agent cluster of the current personalized agent cluster, and the ratio between the number of opponent agent clusters in the opponent pool where the associated personalized agent cluster is located and the total number of opponent agent clusters in the opponent pool of all associated personalized agent clusters is determined as the association weight of the personalized agents in the associated personalized agent cluster.

[0262] It should be noted that for the current personalized agent cluster, there may be one or more historical personalized agent clusters. Therefore, it is necessary to first determine an associated personalized agent cluster associated with the current personalized agent cluster from one or more historical personalized agent clusters.

[0263] For example, continue to refer to Figure 3 , if Agent_H1_1, Agent_H1_2, Agent_H1_3, Agent_H1_4, Agent_H1_5, Agent_H1_6 and Agent_H1_7 are trained serially from top to bottom, then for Agent_H1_1, Agent_H1_2 and Agent_H1_3, there is no associated personalized agent cluster, but for Agent_H1_4, Agent_H1_5, Agent_H1_6 and Agent_H1_7, there is one or more associated personalized agent clusters.

[0264] In one example, Agent_H1_4's opponent pool includes Agent_L1 and Agent_L2, then all personalized agent clusters that contain their opponent pool subsets are associated personalized agent clusters related to Agent_H1_4. Since Agent_H1_1's opponent pool is Agent_L1 and Agent_H1_2's opponent pool is Agent_L2, Agent_H1_4's associated personalized agent clusters are Agent_H1_1 and Agent_H1_2. By calculating the ratio between the number of opponent agent clusters in the opponent pool where the associated personalized agent cluster is located and the total number of opponent agent clusters in the opponent pool of all associated personalized agent clusters, it can be obtained that the associated weights of Agent_H1_11 and Agent_H1_2 are both 1 / (1+1), that is, 1 / 2.

[0265] In another example, taking the initial target agent in the training of Agent_H1_7 as an example, the agent initialization method based on parameter inheritance is adopted. The opponent pool of Agent_H1_7 includes Agent_L1, Agent_L2 and Agent_L3. Then Agent_H1_1~Agent_H1_6 are all the associated personalized agent clusters of Agent_H1_7. And because:

[0266] The opponent pool of Agent_H1_1 contains the opponent Agent_L1.

[0267] The opponent pool of Agent_H1_2 contains the opponent Agent_L2.

[0268] The opponent pool of Agent_H1_3 contains the opponent Agent_L3.

[0269] The opponent pool of Agent_H1_4 includes opponents Agent_L1 and Agent_L2.

[0270] The opponent pool of Agent_H1_5 includes opponents Agent_L1 and Agent_L3.

[0271] The opponent pool of Agent_H1_6 includes opponents Agent_L2 and Agent_L3.

[0272] Therefore, the opponent pool of Agent_H1_1 includes the opponent Agent_L1, the opponent pool of Agent_H1_2 includes the opponent Agent_L2, the opponent pool of Agent_H1_3 includes the opponent Agent_L3, the opponent pool of Agent_H1_4 includes the opponents Agent_L1 and Agent_L2, the opponent pool of Agent_H1_5 includes the opponents Agent_L1 and Agent_L3, and the opponent pool of Agent_H1_6 includes the opponents Agent_L2 and Agent_L3. It can be counted that the total number of opponent agent clusters in the opponent pools of all associated personalized agent clusters is 9, and the following association weight information can be obtained:

[0273] The weights of Agent_H1_1, Agent_H1_2, and Agent_H1_3 are all 1 / 9.

[0274] The weights of Agent_H1_4, Agent_H1_5, and Agent_H1_6 are all 2 / 9.

[0275] S803: For any initial target party agent in the current personalized agent cluster, determine the initial agent parameters of the initial target party agent based on the associated weights and agent parameters of the personalized agents corresponding to the initial target party agent in the associated personalized agent cluster.

[0276] In this embodiment, after calculating the association weight of the personalized agent corresponding to the initial target party agent in the associated personalized agent cluster, the initial agent parameters of the initial target party agent can be determined according to the association weight and the corresponding agent parameters.

[0277] For example, taking an initial target party agent in Agent_H1_4 as an example, since the association weights of Agent_H1_11 and Agent_H1_2 are both 1 / 2, the agent parameters of the personalized agents corresponding to the initial target party agent in Agent_H1_11 and Agent_H1_2 can be weighted and summed according to the association weights to obtain the initial agent parameters of the initial target party agent.

[0278] It should be noted that if the current personalized agent cluster does not have an associated personalized agent cluster, the agent parameters in the current personalized agent cluster are the initial agent parameters obtained through the initialization training of S701; if the current personalized agent cluster has an associated personalized agent cluster, the initial agent parameters obtained through the initialization training of S701 are further optimized to obtain the initial agent parameters for adversarial training.

[0279] In this embodiment, by adopting a parameter inheritance-based training strategy to perform serial training on the personalized agent cluster, the training efficiency can be effectively improved.

[0280] Second, refer to Figure 4 The embodiment of the present application provides an intelligent agent training device 400 based on inter-cluster confrontation, and the intelligent agent training device 400 based on inter-cluster confrontation includes:

[0281] The agent cluster construction module 401 is used to determine any of the two adversarial parties as the target party, and construct multiple target party agent clusters of the target party based on multiple target party optimization objectives and multiple target party constraints of the target party under the target task boundary; the target party agent cluster includes multiple initial target party agents, and different initial target party agents have different target party optimization objectives and / or different target party constraints;

[0282] The opponent pool construction module 402 is used to determine the other party of the opposing parties relative to the target party as the opponent, and combine multiple opponent agent clusters of the opponent to obtain multiple opponent pools of the target party agent cluster; wherein different opponent pools are composed of different opponent agent clusters, and each opponent agent cluster includes multiple initial opponent agents;

[0283] The first training module 403 is used to control the target agent cluster to perform adversarial training with the opponent agent clusters in multiple opponent pools for any target agent cluster, so as to obtain multiple personalized agents corresponding to each initial target agent in the target agent cluster, and obtain personalized target agents corresponding to the initial target agent based on the agent parameters of the multiple personalized agents;

[0284] Agent storage module 404, used to store the personalized target agent and multiple personalized agents corresponding to each initial target agent into the target extended agent cluster of the target party, and store the personalized opponent agent and multiple personalized agents corresponding to each initial opponent agent into the opponent extended agent cluster of the opponent;

[0285] The second training module 405 is used to control the agents in the target party's extended agent cluster and the opponent party's extended agent cluster to perform adversarial training to update the agents in the target party's extended agent cluster and the opponent party's extended agent cluster;

[0286] The simulation confrontation module 406 is used to control the agent to conduct a simulation confrontation with the agent in the updated opponent's extended agent cluster for any agent in the updated target party's extended agent cluster, so as to obtain a simulation confrontation score for each agent in the target party's extended agent cluster;

[0287] The parameter determination module 407 is used to obtain a target-side balanced agent with the target-side optimization objective and target-side constraint conditions for multiple agents in the target-side expanded agent cluster based on the agent parameters of a preset proportion of agents with high simulation adversarial scores among the multiple agents, so as to obtain a target-side balanced agent corresponding to each initial target-side agent.

[0288] In one embodiment of the present application, the second training module 405 includes:

[0289] The first simulation confrontation submodule is used to control the intelligent agents in the target party's extended intelligent agent cluster to conduct a preset round of simulation confrontation with the intelligent agents in the opponent party's extended intelligent agent cluster to obtain the simulation confrontation result of each intelligent agent;

[0290] The winning index update submodule is used to increase the winning index of any agent when the simulation confrontation result of the agent meets the winning condition; keep the winning index of the agent unchanged when the simulation confrontation result meets the normal condition; reduce the winning index of the agent when the simulation confrontation result meets the failure condition, and delete the agent from the target party's extended agent cluster or the opponent party's extended agent cluster when the winning index of the agent is reduced to a preset value;

[0291] The first adversarial training submodule is used to control the agents in the target party's extended agent cluster to perform adversarial training with the agents in the opponent party's extended agent cluster to update the agent parameters of the agents in the target party's extended agent cluster and the opponent party's extended agent cluster.

[0292] In one embodiment of the present application, the agent training device 400 based on inter-cluster confrontation further includes:

[0293] An agent acquisition module, used to acquire multiple agents having the same optimization objectives and constraints as the deleted agent in the target party's extended agent cluster or the opponent party's extended agent cluster;

[0294] The first agent supplement module is used to copy the agent with the largest winning index among multiple agents to obtain the first twin agent to supplement the deleted agent;

[0295] The second agent supplement module is used to construct a second twin agent based on the agent parameters of multiple agents to supplement the deleted agent.

[0296] In one embodiment of the present application, the first simulation confrontation submodule includes:

[0297] The first simulation confrontation unit is used to control any agent in the target party's extended agent cluster to conduct preset rounds of simulation confrontation with agents in the opponent party's extended agent cluster to obtain a simulation confrontation result for each agent;

[0298] The simulation confrontation results include the individual winning rate of the agent against each agent in the opponent's extended agent cluster and the average winning rate of all agents in the opponent's extended agent cluster.

[0299] In one embodiment of the present application, the agent training device 400 based on inter-cluster confrontation further includes:

[0300] A winning number determination module is used to determine the number of winnings of the intelligent agent whose single winning rate is greater than the winning rate threshold;

[0301] A first condition determination module is used to determine a first winning number threshold based on the total number of agents in the opponent's extended agent cluster and the first ratio; when the winning number is greater than or equal to the first winning number threshold, it is determined that the simulated confrontation result of the agent meets the winning condition;

[0302] A second condition determination module is used to determine a second winning number threshold based on the total number of agents in the opponent's extended agent cluster and the second ratio; when the winning number is greater than or equal to the second winning number threshold and less than the first winning number threshold, and the average winning rate is greater than or equal to the average winning rate threshold, determine that the simulated confrontation result of the agent meets the general condition;

[0303] The third condition determination module is used to determine that the simulation confrontation result of the intelligent agent meets the failure condition when the number of wins is less than the second number of wins threshold, or the average winning rate is less than the average winning rate threshold.

[0304] In one embodiment of the present application, the first training module 403 includes:

[0305] An agent cluster replication submodule is used to replicate the target agent cluster to obtain multiple personalized agent clusters corresponding to multiple opponent pools; each personalized agent cluster includes multiple initial target agents in the target agent cluster;

[0306] The second adversarial training submodule is used to repeat the following steps for any initial target agent in any personalized agent cluster: randomly select an initial opponent agent from the opponent agent cluster in the opponent pool corresponding to the personalized agent cluster to perform adversarial training with the initial target agent to update the agent parameters of the initial target agent;

[0307] The condition determination submodule is used to obtain a personalized agent trained by the initial target party agent based on the opponent pool when it is determined that the initial target party agent meets the preset training cutoff condition.

[0308] In an embodiment of the present application, the first training module 403 further includes:

[0309] A first parameter determination submodule is used to determine the average value of the agent parameters of the multiple personalized agents corresponding to the initial target party agent as the agent parameter of the personalized target party agent corresponding to the initial target party agent;

[0310] The second parameter determination submodule is used to determine the weight parameter of the personalized intelligent agent for any personalized intelligent agent corresponding to the initial target intelligent agent based on the ratio between the number of opponent intelligent agent clusters in the opponent pool corresponding to the personalized intelligent agent and the total number of opponent intelligent agent clusters in all opponent pools; and determine the agent parameters of the personalized target intelligent agent corresponding to the initial target intelligent agent based on the weight parameters and agent parameters corresponding to each of the multiple personalized intelligent agents.

[0311] In one embodiment of the present application, the agent training device 400 based on inter-cluster confrontation further includes:

[0312] A first information acquisition module is used to acquire target party information input by a user; wherein the target party information includes a target party optimization target and / or a target party constraint condition input by the target party;

[0313] The first agent determination module is used to determine at least one target equalization agent from multiple target equalization agents based on target party information; wherein the at least one target equalization agent is used to generate a confrontation plan for the opponent of the target party.

[0314] In one embodiment of the present application, the agent training device 400 based on inter-cluster confrontation further includes:

[0315] The training times accumulation module is used to expand any agent in the agent cluster for the target party. Every time the agent completes the training step, the training times of the agent are increased by 1;

[0316] A second information acquisition module is used to acquire target party information and counterparty information input by a user; wherein the target party information includes a target party optimization target and / or a target party constraint condition input by the target party, and the counterparty information includes a counterparty optimization target and / or a counterparty constraint condition input by the counterparty;

[0317] A second agent determination module is used to determine at least one target equalization agent from multiple target equalization agents based on the target party information, and determine at least one target agent with the largest number of training times from the updated target party extended agent cluster;

[0318] A third agent determination module, configured to determine at least one target counterparty agent in the updated counterparty extended agent cluster based on counterparty information;

[0319] An evaluation table generation module, used to control at least one target agent and at least one target equalization agent to respectively perform simulation confrontation with at least one target opponent agent in a target simulation confrontation environment, and generate a confrontation evaluation table according to a preset format;

[0320] The evaluation table output module is used to output the adversarial evaluation table.

[0321] It should be noted that the specific implementation of the intelligent agent training device 400 based on inter-cluster confrontation in the embodiment of the present application refers to the specific implementation of the intelligent agent training method based on inter-cluster confrontation proposed in the first aspect of the aforementioned embodiment of the present application, and will not be repeated here.

[0322] It will be appreciated by those skilled in the art that the embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the embodiments of the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the embodiments of the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0323] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0324] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0325] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0326] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0327] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0328] The above is a detailed introduction to the intelligent agent training method and device based on inter-cluster confrontation provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. An agent training method based on inter-cluster confrontation, characterized in that: The method comprises: Determine any one of the two adversarial parties as the target party, and construct multiple target party agent clusters of the target party based on multiple target party optimization objectives and multiple target party constraints of the target party under the target task boundary; the target party agent cluster includes multiple initial target party agents, and different initial target party agents have different target party optimization objectives and / or different target party constraints; the target party is a drone capture device; The other party of the two opposing parties relative to the target party is determined as an opponent party, and multiple opponent party agent clusters of the opponent party are combined to obtain multiple opponent party pools of the target party agent cluster; wherein different opponent party pools are composed of different opponent party agent clusters, and each of the opponent party agent clusters includes multiple initial opponent party agents; the opponent party is a drone to be captured; For any target agent cluster, control the target agent cluster to perform adversarial training with multiple opponent agent clusters in the opponent pool, so as to obtain multiple personalized agents corresponding to each initial target agent in the target agent cluster, and obtain a personalized target agent corresponding to the initial target agent based on agent parameters of the multiple personalized agents; The personalized target party agent and multiple personalized agents corresponding to each initial target party agent are stored in the target party extended agent cluster of the target party, and the personalized opponent party agent and multiple personalized agents corresponding to each initial opponent party agent are stored in the opponent party extended agent cluster of the opponent party; Controlling the agents in the target party's extended agent cluster and the opponent party's extended agent cluster to perform adversarial training to update the agents in the target party's extended agent cluster and the opponent party's extended agent cluster; For any agent in the updated target party extended agent cluster, control the agent to perform simulation confrontation with the agent in the updated opponent party extended agent cluster to obtain a simulation confrontation score for each agent in the target party extended agent cluster; For multiple agents in the target-side extended agent cluster having the same target-side optimization objective and target-side constraint conditions, a target-side balanced agent having the target-side optimization objective and the target-side constraint conditions is obtained based on the agent parameters of a preset proportion of agents with high simulation confrontation scores among the multiple agents, so as to obtain a target-side balanced agent corresponding to each initial target-side agent; the target-side balanced agent is used to generate a confrontation plan for the drone to be captured.

2. According to the method of claim 1, the method is characterized in that: The step of controlling the agents in the target party's extended agent cluster and the opponent party's extended agent cluster to perform adversarial training to update the agents in the target party's extended agent cluster and the opponent party's extended agent cluster includes: Repeat the following training steps for the preset number of repetitions: Controlling the agents in the target party's extended agent cluster to perform preset rounds of simulation confrontation with the agents in the opponent party's extended agent cluster to obtain simulation confrontation results for each agent; For any agent, if the simulation confrontation result of the agent meets the winning condition, the winning index of the agent is increased; if the simulation confrontation result meets the normal condition, the winning index of the agent is kept unchanged; if the simulation confrontation result meets the failure condition, the winning index of the agent is reduced, and if the winning index of the agent is reduced to a preset value, the agent is deleted from the target party's extended agent cluster or the opponent party's extended agent cluster; The agents in the target extended agent cluster and the agents in the opponent extended agent cluster are controlled to perform adversarial training to update the agent parameters of the agents in the target extended agent cluster and the opponent extended agent cluster.

3. The method for training an intelligent agent based on inter-cluster confrontation according to claim 2, characterized in that: After the step of deleting the agent from the target party's extended agent cluster or the opponent party's extended agent cluster, the method further comprises: In the target party extended agent cluster or the opponent party extended agent cluster, obtaining multiple agents having the same optimization objectives and constraints as the deleted agent; Copy the agent with the largest winning index among multiple agents to obtain the first twin agent to supplement the deleted agent; or, Based on the agent parameters of multiple agents, a second twin agent is constructed to supplement the deleted agents.

4. The method for training an intelligent agent based on inter-cluster confrontation according to claim 2, characterized in that: The step of controlling the agents in the target party's extended agent cluster to perform preset rounds of simulation confrontation with the agents in the opponent party's extended agent cluster to obtain the simulation confrontation result of each agent includes: For any agent in the target party's extended agent cluster, control the agent to respectively conduct preset rounds of simulation confrontation with agents in the opponent party's extended agent cluster to obtain simulation confrontation results for each agent; The simulation confrontation result includes the individual winning rate of the agent against each agent in the opponent's extended agent cluster and the average winning rate of all agents in the opponent's extended agent cluster.

5. The method for training an intelligent agent based on inter-cluster confrontation according to claim 4, characterized in that: The method further comprises: Determine the number of wins for which the single-unit win rate of the intelligent agent is greater than a win rate threshold; Determine a first winning number threshold based on the total number of agents in the opponent's extended agent cluster and the first ratio; determine that the simulation confrontation result of the agent meets the winning condition when the winning number is greater than or equal to the first winning number threshold; Based on the total number of agents in the opponent's extended agent cluster and the second ratio, a second winning number threshold is determined; when the winning number is greater than or equal to the second winning number threshold and less than the first winning number threshold, and the average winning rate is greater than or equal to the average winning rate threshold, it is determined that the simulation confrontation result of the agent meets the general condition; When the number of wins is less than the second number of wins threshold, or the average winning rate is less than the average winning rate threshold, it is determined that the simulation confrontation result of the intelligent agent meets the failure condition.

6. The method for training an intelligent agent based on inter-cluster confrontation according to claim 1, characterized in that: For any target agent cluster, the step of controlling the target agent cluster to perform adversarial training with multiple opponent agent clusters in the opponent pool to obtain multiple personalized agents corresponding to each initial target agent in the target agent cluster, includes: Copying the target agent cluster to obtain a plurality of personalized agent clusters corresponding to the plurality of opponent pools; each of the personalized agent clusters includes a plurality of initial target agents in the target agent cluster; For any of the initial target agents in any of the personalized agent clusters, repeat the following steps: randomly selecting an initial opponent agent from the opponent agent cluster in the opponent pool corresponding to the personalized agent cluster to perform adversarial training with the initial target agent to update the agent parameters of the initial target agent; When it is determined that the initial target party agent meets a preset training cutoff condition, a personalized agent obtained by training the initial target party agent based on the opponent pool is obtained.

7. The method for training an intelligent agent based on inter-cluster confrontation according to claim 1, characterized in that: The step of obtaining a personalized target party agent corresponding to the initial target party agent based on agent parameters of the plurality of personalized agents comprises: Determine the average value of the agent parameters of the plurality of personalized agents corresponding to the initial target agent as the agent parameter of the personalized target agent corresponding to the initial target agent; or For any personalized agent corresponding to the initial target agent, a weight parameter of the personalized agent is determined based on a ratio between the number of opponent agent clusters in the opponent pool corresponding to the personalized agent and the total number of opponent agent clusters in all opponent pools; and agent parameters of the personalized target agent corresponding to the initial target agent are determined based on the weight parameters and agent parameters corresponding to each of the plurality of personalized agents.

8. The method for training an intelligent agent based on inter-cluster confrontation according to claim 1, characterized in that: After the step of obtaining the target-side equalization agent corresponding to each initial target-side agent, the method further includes: Acquire target party information input by a user; wherein the target party information includes a target party optimization target and / or a target party constraint condition input by the target party; Based on the target party information, at least one target equalization agent is determined from a plurality of target party equalization agents; wherein the at least one target equalization agent is used to generate a countermeasure against an opponent of the target party.

9. The method for training an intelligent agent based on inter-cluster confrontation according to claim 2, characterized in that: The method further comprises: For any agent in the target party's expanded agent cluster, whenever the agent completes the training step, the training times of the agent are increased by 1; After the step of obtaining the target-side equalization agent corresponding to each initial target-side agent, the method further includes: Acquire target party information and counterparty information input by a user; wherein the target party information includes a target party optimization target and / or a target party constraint condition input by the target party, and the counterparty information includes a counterparty optimization target and / or a counterparty constraint condition input by the counterparty; Based on the target party information, at least one target equalization agent is determined from a plurality of target party equalization agents, and at least one target agent with the largest number of training times is determined from an updated target party extended agent cluster; Based on the counterparty information, determining at least one target counterparty agent in the updated counterparty extended agent cluster; Controlling at least one of the target agents and at least one of the target equalization agents to respectively perform simulation confrontation with at least one of the target opponent agents in a target simulation confrontation environment, and generating a confrontation evaluation table according to a preset format; The confrontation evaluation table is outputted.

10. An agent training device based on inter-cluster confrontation, characterized in that: The device comprises: An agent cluster construction module is used to determine any of the two opposing parties as a target party, and to construct multiple target party agent clusters of the target party based on multiple target party optimization objectives and multiple target party constraints of the target party under the target task boundary; the target party agent cluster includes multiple initial target party agents, and different initial target party agents have different target party optimization objectives and / or different target party constraints; the target party is a drone capture device; An opponent pool construction module is used to determine the other party of the two opposing parties relative to the target party as the opponent, and combine multiple opponent agent clusters of the opponent to obtain multiple opponent pools of the target agent cluster; wherein different opponent pools are composed of different opponent agent clusters, and each opponent agent cluster includes multiple initial opponent agents; the opponent is a drone to be captured; The first training module is used for controlling any target agent cluster to perform adversarial training with multiple opponent agent clusters in the opponent pool, so as to obtain multiple personalized agents corresponding to each initial target agent in the target agent cluster, and obtain a personalized target agent corresponding to the initial target agent based on agent parameters of the multiple personalized agents; An agent storage module, used to store the personalized target agent and multiple personalized agents corresponding to each initial target agent into the target extended agent cluster of the target party, and store the personalized opponent agent and multiple personalized agents corresponding to each initial opponent agent into the opponent extended agent cluster of the opponent; A second training module is used to control the agents in the target party's extended agent cluster and the opponent party's extended agent cluster to perform adversarial training to update the agents in the target party's extended agent cluster and the opponent party's extended agent cluster; A simulation confrontation module is used to control any agent in the updated target party extended agent cluster to perform simulation confrontation with an agent in the updated opponent party extended agent cluster, so as to obtain a simulation confrontation score for each agent in the target party extended agent cluster; A parameter determination module is used to obtain a target-side balanced intelligent agent with the same target-side optimization goal and target-side constraint conditions for multiple intelligent agents in the target-side extended intelligent agent cluster, based on the intelligent agent parameters of a preset proportion of intelligent agents with high simulation confrontation scores among the multiple intelligent agents, so as to obtain a target-side balanced intelligent agent corresponding to each initial target-side intelligent agent; the target-side balanced intelligent agent is used to generate a confrontation plan for the drone to be captured.

Images