A method and device for training intelligent agents based on layered confrontation

Abstract

The present application provides an agent training method and device based on hierarchical confrontation, which belongs to the field of agent technology. The embodiment of the present application can construct an agent cluster of the two opposing parties according to the optimization objectives and constraints corresponding to each of the opposing parties, and by constructing multiple opponent pools and multiple personalized agent clusters corresponding to the multiple opponent pools, the initial target party agent in each personalized agent cluster can be trained against the opponent party agent with different optimization objectives and different constraints, thereby obtaining a personalized agent that can adapt to different opponent styles, and finally, according to the agent parameters of multiple personalized agents, the agent parameters of the initial target party agent can be obtained, thereby obtaining a personalized target party agent with strong robustness and comprehensive ability. In this way, the generalization ability of the agent can be effectively improved, and even when faced with complex and changeable confrontation tasks, better confrontation effects can be achieved.

Description

Technical Field

[0001] The present application relates to the field of intelligent agent technology, and in particular to an intelligent agent training method and device based on layered confrontation. Background Art

[0002] Game confrontation between intelligent agents refers to the adversarial behavior based on strategy and decision-making between two or more intelligent agents in the field of artificial intelligence. In the game confrontation, each intelligent agent tries to maximize its own interests by formulating the best strategy, and must also predict and respond to the behavior of its opponent.

[0003] With the continuous emergence of artificial intelligence technology, researchers have applied artificial intelligence technology to the training of intelligent agents, especially the intelligent agent training method based on reinforcement learning, and have achieved relatively good operating results in decision-making problems.

[0004] In the process of game confrontation, it is necessary to consider various aspects of the opponent, such as the opponent's intelligence level, available tactics, confrontation style and tendency, etc. However, in the current reinforcement learning-based intelligent agent training methods, the opponent's confrontation methods are usually based on fixed rules and have only a single style / tendency.

[0005] In a simulation training environment based on fixed rules or opponents with a single style / tendency, the trained intelligent agent can only deal with opponents with these rules or opponents with a single style / tendency, resulting in the limited generalization ability of the intelligent agent and making it difficult to cope with complex and changeable adversarial tasks. Summary of the invention

[0006] The present application provides a method and device for training an intelligent agent based on layered adversarial systems to solve the problem that the intelligent agents trained in related technologies have limited generalization capabilities and are unable to cope with complex and changeable adversarial tasks.

[0007] In order to solve the above problems, this application adopts the following technical solutions:

[0008] In a first aspect, an embodiment of the present application provides an agent training method based on layered confrontation, the method comprising:

[0009] Determine any one of the two opposing parties as a target party, and determine multiple target party optimization objectives and multiple target party constraints for the target party under the target task boundary;

[0010] For any target optimization target, a target agent cluster of the target under the target optimization target is constructed, and based on a plurality of target constraint conditions, a plurality of initial target agents in the target agent cluster are constructed; wherein different target optimization targets correspond to different target agent clusters;

[0011] The other party of the two opposing parties relative to the target party is determined as an opponent party, and multiple opponent party agent clusters of the opponent party are combined to obtain multiple opponent party pools of the target party agent cluster; wherein different opponent party pools are composed of different opponent party agent clusters;

[0012] Copying the target agent cluster to obtain a plurality of personalized agent clusters corresponding to the plurality of opponent pools; each of the personalized agent clusters includes a plurality of initial target agents in the target agent cluster;

[0013] For any personalized agent cluster, the personalized agent cluster is subjected to adversarial training with an opponent agent cluster in an opponent pool corresponding to the personalized agent cluster, so as to train and obtain a personalized agent corresponding to each initial target agent in the personalized agent cluster;

[0014] For any of the initial target party agents, agent parameters of the initial target party agent are determined based on agent parameters of a plurality of the personalized agents corresponding to the initial target party agent, so as to obtain a personalized target party agent corresponding to the initial target party agent.

[0015] In one embodiment of the present application, the step of determining a plurality of target party constraints of the target party under the target task boundary includes:

[0016] Acquire a constraint condition set of the target party; the constraint condition set includes a plurality of constraint condition subsets, and different constraint condition subsets are used to represent different constraint conditions of the target party;

[0017] Multiple constraint condition subsets in the constraint condition set are combined to obtain multiple target party constraints of the target party under the target task boundary.

[0018] In one embodiment of the present application, for any initial target party agent in the target party agent cluster, before the step of respectively performing adversarial training on the initial target party agent with initial opponent party agents in the plurality of opponent party agents in the opponent party pool to obtain a personalized target party agent corresponding to the initial target party agent, the method further includes:

[0019] Initialization training is performed on the multiple initial target-side intelligent agents in the target-side intelligent agent cluster to determine initial intelligent agent parameters corresponding to each of the multiple initial target-side intelligent agents.

[0020] In one embodiment of the present application, the step of performing initialization training on the plurality of initial target agents in the target agent cluster to determine initial agent parameters corresponding to each of the plurality of initial target agents includes:

[0021] In the case that the agent structures of multiple initial target party agents in the target party agent cluster are inconsistent, the multiple initial target party agents in the target party agent cluster are independently trained respectively until the multiple initial target party agents meet a preset first training cutoff condition, and the initial agent parameters corresponding to each of the multiple initial target party agents are determined.

[0022] In one embodiment of the present application, the step of performing initialization training on the plurality of initial target agents in the target agent cluster to determine initial agent parameters corresponding to each of the plurality of initial target agents includes:

[0023] In the case where the agent structures of multiple initial target-side agents in the target-side agent cluster are consistent, removing the shielding matrix of any of the initial target-side agents to obtain the initial simplified agent corresponding to the target-side agent cluster, and determining the simplified optimization target of the initial simplified agent based on the multiple target-side optimization targets; wherein the shielding matrix represents the target-side constraint condition of the initial target-side agent;

[0024] Based on the simplified optimization objective and a preset first reinforcement learning algorithm, the initial simplified intelligent agent is trained until the initial simplified intelligent agent meets a preset first training cutoff condition, thereby obtaining a trained first simplified intelligent agent;

[0025] Based on the target optimization objectives corresponding to each of the multiple target agent clusters and a preset second reinforcement learning algorithm, the first simplified agent is trained respectively until the first simplified agent meets the first training cutoff condition, thereby obtaining multiple second simplified agents corresponding to the multiple target agent clusters one by one;

[0026] For any second simplified agent, the agent parameters of the second simplified agent are used as the agent parameters of multiple initial target agents in the corresponding target agent cluster, and each of the initial target agents is trained based on a preset third reinforcement learning algorithm until the initial target agent meets the first training cutoff condition, thereby obtaining the initial agent parameters of the initial target agent.

[0027] In one embodiment of the present application, for any personalized agent cluster, the steps of conducting adversarial training between the personalized agent cluster and an opponent agent cluster in an opponent pool corresponding to the personalized agent cluster to train and obtain personalized agents corresponding to each initial target agent in the personalized agent cluster include:

[0028] For any of the initial target agents in any of the personalized agent clusters, repeat the following steps: randomly select an initial opponent agent from the opponent agent cluster in the opponent pool corresponding to the personalized agent cluster to perform adversarial training with the initial target agent, and update the agent parameters of the initial target agent based on the adversarial result;

[0029] When it is determined that the initial target party agent meets the preset second training cutoff condition, a personalized agent obtained by training the initial target party agent based on the opponent pool is obtained.

[0030] In an embodiment of the present application, the method further includes:

[0031] In the case where the plurality of personalized agent clusters are trained serially, determining a trained historical personalized agent cluster for an untrained current personalized agent cluster;

[0032] For any of the historical personalized agent clusters, if the opponent pool of the historical personalized agent cluster includes any opponent agent cluster in the opponent pool of the current personalized agent cluster, the historical personalized agent cluster is determined to be an associated personalized agent cluster of the current personalized agent cluster, and the ratio between the number of opponent agent clusters in the opponent pool where the associated personalized agent cluster is located and the total number of opponent agent clusters in the opponent pool of all the associated personalized agent clusters is determined as the association weight of the personalized agents in the associated personalized agent cluster;

[0033] For any initial target party agent in the current personalized agent cluster, the initial agent parameters of the initial target party agent are determined based on the associated weights and agent parameters of the personalized agents corresponding to the initial target party agent in the associated personalized agent cluster.

[0034] In one embodiment of the present application, for any of the initial target party agents, the step of determining the agent parameters of the initial target party agent based on the agent parameters of the plurality of personalized agents corresponding to the initial target party agent includes:

[0035] Determine the average value of the agent parameters of the plurality of personalized agents corresponding to the initial target agent as the agent parameter of the initial target agent; or

[0036] For any personalized agent corresponding to the initial target agent, a weight parameter of the personalized agent is determined based on the ratio between the number of opponent agent clusters in the opponent pool corresponding to the personalized agent and the total number of opponent agent clusters in all opponent pools; and the agent parameters of the initial target agent are determined based on the weight parameters and agent parameters corresponding to each of the multiple personalized agents.

[0037] In one embodiment of the present application, for any of the initial target party agents, after the step of determining the agent parameters of the initial target party agent based on the agent parameters of the plurality of personalized agents corresponding to the initial target party agent to obtain the personalized target party agent corresponding to the initial target party agent, the method further includes:

[0038] Acquire target party information and / or counterparty information input by a user; wherein the target party information includes a target party optimization target and / or a target party constraint condition input by the target party, and the counterparty information includes a counterparty optimization target and / or a counterparty constraint condition input by the counterparty;

[0039] Based on the target party information and / or the opponent party information, one or more target agents are determined from the multiple personalized target party agents; wherein the target agent is used to generate a confrontation plan for the opponent party.

[0040] In the second aspect, based on the same inventive concept, the embodiment of the present application provides an intelligent agent training device based on layered confrontation, the device comprising:

[0041] A condition determination module, used to determine any one of the two adversarial parties as a target party, and determine a plurality of target party optimization objectives and a plurality of target party constraint conditions of the target party under the target task boundary;

[0042] An agent construction module is used to construct a target agent cluster of the target under the target optimization target for any target optimization target, and to construct multiple initial target agents in the target agent cluster based on multiple target constraint conditions; wherein different target optimization targets correspond to different target agent clusters;

[0043] An opponent pool determination module is used to determine the other party of the opposing parties relative to the target party as the opponent, and combine multiple opponent agent clusters of the opponent to obtain multiple opponent pools of the target party agent cluster; wherein different opponent pools are composed of different opponent agent clusters;

[0044] A copy module, used to copy the target agent cluster to obtain a plurality of personalized agent clusters corresponding to the plurality of opponent pools; each of the personalized agent clusters includes a plurality of initial target agents in the target agent cluster;

[0045] An adversarial training module is used to perform adversarial training on any personalized agent cluster, comparing the personalized agent cluster with an opponent agent cluster in an opponent pool corresponding to the personalized agent cluster, so as to train and obtain personalized agents corresponding to each initial target agent in the personalized agent cluster;

[0046] The parameter determination module is used to determine the agent parameters of any of the initial target party agents based on the agent parameters of multiple personalized agents corresponding to the initial target party agent, so as to obtain the personalized target party agent corresponding to the initial target party agent.

[0047] In an embodiment of the present application, the condition determination module includes:

[0048] A set acquisition submodule, used to acquire the constraint condition set of the target party; the constraint condition set includes multiple constraint condition subsets, and different constraint condition subsets are used to represent different constraint conditions of the target party;

[0049] The constraint condition combination submodule is used to combine multiple constraint condition subsets in the constraint condition set to obtain multiple target party constraints of the target party under the target task boundary.

[0050] In one embodiment of the present application, the intelligent agent training device based on layered confrontation further includes:

[0051] The initialization training module is used to perform initialization training on the multiple initial target party agents in the target party agent cluster to determine the initial agent parameters corresponding to each of the multiple initial target party agents.

[0052] In one embodiment of the present application, the initialization training module includes:

[0053] The first initialization training submodule is used to independently train multiple initial target party agents in the target party agent cluster when the agent structures of multiple initial target party agents in the target party agent cluster are inconsistent, until the multiple initial target party agents meet a preset first training cutoff condition, and determine the initial agent parameters corresponding to each of the multiple initial target party agents.

[0054] In one embodiment of the present application, the initialization training module includes:

[0055] The agent simplification submodule is used to remove the shielding matrix of any of the initial target-side agents in the target-side agent cluster when the agent structures of the multiple initial target-side agents are consistent, obtain the initial simplified agent corresponding to the target-side agent cluster, and determine the simplified optimization target of the initial simplified agent based on the multiple target-side optimization targets; wherein the shielding matrix represents the target-side constraint condition of the initial target-side agent;

[0056] A first training submodule, configured to train the initial simplified intelligent agent based on the simplified optimization objective and a preset first reinforcement learning algorithm until the initial simplified intelligent agent meets a preset first training cutoff condition, thereby obtaining a trained first simplified intelligent agent;

[0057] A second training submodule is used to train the first simplified agent based on the target optimization objectives corresponding to each of the multiple target agent clusters and a preset second reinforcement learning algorithm, until the first simplified agent meets the first training cutoff condition, thereby obtaining multiple second simplified agents corresponding to the multiple target agent clusters one by one;

[0058] The third training submodule is used to use the agent parameters of any second simplified agent as the agent parameters of multiple initial target agents in the corresponding target agent cluster, and train each of the initial target agents based on a preset third reinforcement learning algorithm until the initial target agent meets the first training cutoff condition, thereby obtaining the initial agent parameters of the initial target agent.

[0059] In one embodiment of the present application, the adversarial training module includes:

[0060] The iterative training submodule is used to repeat the following steps for any of the initial target agents in any of the personalized agent clusters: randomly selecting an initial opponent agent from the opponent agent cluster in the opponent pool corresponding to the personalized agent cluster to perform adversarial training with the initial target agent, and updating the agent parameters of the initial target agent based on the adversarial result;

[0061] The agent parameter determination submodule is used to obtain a personalized agent trained by the initial target agent based on the opponent pool when it is determined that the initial target agent meets a preset second training cutoff condition.

[0062] In one embodiment of the present application, the intelligent agent training device based on layered confrontation further includes:

[0063] A historical personalized agent cluster determination module is used to determine a trained historical personalized agent cluster for an untrained current personalized agent cluster when a plurality of the personalized agent clusters are trained in series;

[0064] an association weight determination module, for determining, for any of the historical personalized agent clusters, that the historical personalized agent cluster is an associated personalized agent cluster of the current personalized agent cluster if the opponent pool of the historical personalized agent cluster includes any opponent agent cluster in the opponent pool of the current personalized agent cluster, and determining the ratio between the number of opponent agent clusters in the opponent pool where the associated personalized agent cluster is located and the total number of opponent agent clusters in the opponent pool of all the associated personalized agent clusters as the association weight of the personalized agents in the associated personalized agent cluster;

[0065] The initial agent parameter determination module is used to determine the initial agent parameters of any initial target agent in the current personalized agent cluster based on the associated weights and agent parameters of the personalized agents corresponding to the initial target agent in the associated personalized agent cluster.

[0066] In an embodiment of the present application, the parameter determination module includes:

[0067] A first parameter determination submodule is used to determine the average value of the agent parameters of the plurality of personalized agents corresponding to the initial target agent as the agent parameter of the initial target agent; or

[0068] The second parameter determination submodule is used to determine the weight parameter of any personalized agent corresponding to the initial target agent based on the ratio between the number of opponent agent clusters in the opponent pool corresponding to the personalized agent and the total number of opponent agent clusters in all opponent pools; and determine the agent parameters of the initial target agent based on the weight parameters and agent parameters corresponding to each of the multiple personalized agents.

[0069] In one embodiment of the present application, the intelligent agent training device based on layered confrontation includes:

[0070] An information acquisition module, used to acquire target party information and / or counterparty information input by a user; wherein the target party information includes a target party optimization target and / or a target party constraint condition input by the target party, and the counterparty information includes a counterparty optimization target and / or a counterparty constraint condition input by the counterparty;

[0071] An agent determination module is used to determine one or more target agents from a plurality of personalized target agents based on the target party information and / or the opponent party information; wherein the target agent is used to generate a countermeasure plan for the opponent party.

[0072] Compared with the prior art, this application has the following advantages:

[0073] An agent training method based on hierarchical confrontation is provided in an embodiment of the present application. First, agent clusters of the two opposing parties are constructed according to the optimization objectives and constraints corresponding to each other. Then, for any target agent cluster of the target party, multiple opponent pools are constructed. Since the opponent agent cluster in each opponent pool includes multiple opponent agents with different opponent optimization objectives and different opponent constraints, multiple personalized agents corresponding to the initial target agent can be trained by conducting adversarial training on the target agent cluster and the opponent agent clusters in the multiple opponent pools respectively. Since each personalized agent adapts to the opponent style in the corresponding opponent pool, the personalized target agent obtained by comprehensively considering the agent parameters of the multiple personalized agents corresponding to the initial target agent has strong comprehensive ability and can better adapt to various opponents with different styles and tendencies. In this way, the generalization ability of the agent can be effectively improved, and better confrontation effect can be achieved when facing complex and changeable confrontation tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0075] Figure 1 This is a flowchart of the steps of a method for training an intelligent agent based on layered confrontation in one embodiment of the present application.

[0076] Figure 2 It is a schematic diagram of the confrontation between intelligent agent clusters in one embodiment of the present application.

[0077] Figure 3 This is a schematic diagram of adversarial training of a personalized agent cluster in one embodiment of the present application.

[0078] Figure 4 It is a module diagram of an intelligent agent training device based on layered confrontation in one embodiment of the present application. DETAILED DESCRIPTION

[0079] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0080] It should be noted that game confrontation can occur in many different scenarios and has applications in machine learning, game theory, multi-agent systems, and reinforcement learning. Game confrontation has important applications in adversarial learning, chess games, security defense, and network attack and defense.

[0081] For example, in the field of drones, as the application areas of unmanned aerial vehicles increase, the security threats brought by unmanned aerial vehicles are also gradually increasing. For example, in important places such as airports, hydro (nuclear) power stations, important meetings, large gatherings and sports events, due to the requirements of site safety, unmanned aerial vehicles are often banned from flying. When unauthorized unmanned aerial vehicles enter important places, they need to be countered and hunted down to reduce the risks and losses they bring. As a result, a series of technologies for directional attack, countermeasures and capture of unmanned aerial vehicles have been developed, such as unmanned aerial vehicle detection and attack integrated systems, capture and defense systems, etc. The drone capture device in the above system is usually equipped with an intelligent agent, which can generate corresponding confrontation plans according to the current situation (number of resources, resource location, resource capacity, opponent's location, type, etc.) and the purpose of confrontation (strike, drive away, damage target drones, etc.).

[0082] For example, in the field of games, there are usually human-machines fighting against players. As an intelligent agent, the human-machine needs to generate corresponding confrontation strategies based on the player's operation. Specifically, the intelligent agent generates a decision plan based on the situation (number of resources, resource location, resource capacity, opponent's location, type, etc.) and the purpose of the confrontation (occupying a certain area, destroying a certain target, etc.). The decision plan may include moving the target person / monster / vehicle to the target location, and / or using the target skill trick on the target person.

[0083] In traditional intelligent agent training methods, it is usually based on rule construction (the code is mainly conditional judgment, such as if-then logic code) or traditional optimization algorithms. Due to the high real-time requirements of game confrontation, greedy algorithms can generally be used, but the algorithm can easily fall into local optimality, and the confrontation effect is not significantly improved compared to the rule-based method; with the continuous emergence of artificial intelligence technology, researchers have applied artificial intelligence technology to decision-making intelligent agent construction methods, especially the decision-making intelligent agent construction method based on reinforcement learning, which has achieved relatively good operating results in decision-making problems.

[0084] The intelligent agent training method based on reinforcement learning first constructs the intelligent agent network structure according to the input, output and problem complexity of the intelligent agent required in the confrontation scenario, and then constructs a simulation training environment consistent with the intelligent agent application environment, such as a drone confrontation environment or a twin simplified environment of the drone confrontation environment. Finally, the reinforcement learning algorithm is used to train the intelligent agent network in the simulation training environment until convergence, and a decision-making intelligent agent can be obtained.

[0085] The essence of the confrontation problem is game confrontation, and the opponent's situation needs to be considered during the confrontation, such as the opponent's intelligence level, available tactics, confrontation style and tendencies, etc. However, in the general construction of decision-making agents based on reinforcement learning, the opponent / opponent's confrontation methods in the simulation training environment are basically based on rules or a single style / tendency.

[0086] Based on the simulation training environment of rules or single style / tendency opponents, the trained intelligent agents have limited generalization ability and can only deal with the rule opponents or single style / tendency opponents. However, confrontation problems are often much more complicated. For example, in drone confrontation, the drones to be captured may have different flight capabilities and escape strategies, or the operators of the drones may be people with different styles, different operating habits or different operating levels, which leads to poor confrontation performance of the intelligent agents of the drone capture device and cannot adapt to opponents with changing styles. At the same time, in actual confrontation scenarios, the information about the opponent's drones that can be obtained is usually small, and there is often a situation where the opponent is completely unknown. At this time, it will be difficult for the operator to select a suitable intelligent agent to perform the corresponding confrontation task, which leads to poor confrontation effect of the intelligent agent.

[0087] In view of the problem that the generalization ability of the intelligent agents trained in the related technologies is limited and it is difficult to cope with complex and changeable adversarial tasks, this application aims to provide an intelligent agent training method based on hierarchical adversarial, by constructing multiple opponent pools and multiple personalized intelligent agent clusters corresponding to the multiple opponent pools, so that the initial target party intelligent agent in each personalized intelligent agent cluster can be trained against the opponent party intelligent agents with different optimization goals and different constraints, thereby obtaining a personalized intelligent agent that can adapt to different opponent styles, and finally, according to the intelligent agent parameters of the multiple personalized intelligent agents, the intelligent agent parameters of the initial target party intelligent agent can be obtained, thereby obtaining a personalized target party intelligent agent with strong robustness and comprehensive ability. In this way, the generalization ability of the intelligent agent can be effectively improved, and even when facing complex and changeable adversarial tasks, a better adversarial effect can be achieved.

[0088] Reference Figure 1 , shows an agent training method based on layered confrontation in the present application, which may include the following steps:

[0089] S101: Determine any one of the two adversarial parties as the target party, and determine multiple target party optimization objectives and multiple target party constraints of the target party under the target task boundary.

[0090] It should be noted that the two opposing parties refer to the two parties participating in the game confrontation. For example, the two opposing parties can be defined as the red party and the blue party, or the two opposing parties are the attacker and the defender. In the field of drones, the target party can refer to the drone capture device, and the opponent party refers to the drone to be captured.

[0091] In this embodiment, according to the training requirements, any one of the two adversarial parties can be determined as the target party. Once one of the two adversarial parties is determined as the target party, the other party of the two adversarial parties relative to the target party can be used as the opponent party. For the target party, multiple target party optimization objectives and multiple target party constraints of the target party under the target task boundary are determined; for the opponent party, multiple opponent party optimization objectives and multiple opponent party constraints of the opponent party under the target task boundary are determined.

[0092] In this embodiment, the target task boundary represents the boundary conditions for the opposing parties. The opposing parties need to make corresponding confrontation strategies within the target task boundary. Specifically, the target task boundary includes the problem input and problem output of the opposing parties and the confrontation resources that the opposing parties can dispatch, such as the resource teams and equipment that can be dispatched, and the confrontation means that can be used.

[0093] In this embodiment, the optimization target represents the strategic target when the intelligent agent formulates the confrontation strategy, which can reflect the intention of the operator / commander. For example, the optimization target may include: the fastest time to complete the goal, the least number of waves to complete the goal, the least number of equipment to complete the goal, the lowest resource consumption to complete the goal, the highest consumption ratio to complete the goal, the least team to call to complete the goal, and / or the use of more resource points to complete the goal. Specifically, the optimization target may include the target optimization target and the opponent optimization target.

[0094] In this embodiment, the constraint conditions represent the limiting conditions when the agent formulates the confrontation strategy, which can reflect the subjective constraints of the operator / commander or the resource constraints under the influence of objective resource conditions. For example, the constraint conditions may include: the type of equipment used, which resource teams to call, which resource points to use, whether to adopt a collaborative method, how to collaborate, success rate requirements, total wave requirements and / or time period requirements, etc. Specifically, the constraint conditions may include target constraints and opponent constraints.

[0095] S102: For any target optimization objective, a target agent cluster of the target under the target optimization objective is constructed, and based on multiple target constraints, multiple initial target agents in the target agent cluster are constructed.

[0096] In this embodiment, different target optimization goals correspond to different target agent clusters. That is, for each target optimization goal, a target agent cluster is constructed, and each target agent cluster includes multiple initial target agents, and different initial target agents have different target constraints. In this way, for any initial target agent, the initial target agent has a unique corresponding target optimization goal and target constraint.

[0097] In an example, for the target party, there are N target party optimization objectives and P target party constraints, then there are N target party agent clusters and N×P initial target party agents.

[0098] S103: Determine the other party of the two opposing parties relative to the target party as the opponent party, and combine multiple opponent party agent clusters of the opponent party to obtain multiple opponent pools of the target party agent cluster.

[0099] In this embodiment, different opponent pools are composed of different opponent agent clusters.

[0100] It should be noted that the construction process of the opponent's intelligent agent cluster is the same as the construction process of the target party's intelligent agent cluster, that is, for any opponent's optimization objective, an opponent's intelligent agent cluster under the opponent's optimization objective is constructed, and based on multiple opponent constraints, multiple initial opponent intelligent agents in each opponent's intelligent agent cluster are constructed; among which different opponent optimization objectives correspond to different opponent intelligent agent clusters.

[0101] It should be noted that the structure of the neural network adopted by each intelligent agent can be selected in multiple ways, according to the complexity of the task. The constraints can be introduced into the intelligent agent by adding a shielding matrix at the output end or by using reinforcement learning feedback values ​​during training. The former requires adjusting the network structure, while the latter does not need to change the network structure.

[0102] In this embodiment, each agent can be labeled with features, which include agent labels, task boundaries corresponding to the agent, constraints, optimization goals, round conditions, etc., where the round conditions correspond to training rounds.

[0103] Reference Figure 2 , showing a schematic diagram of the confrontation between intelligent clusters. Define the target party (drone capture device) as the red party and the opponent party (drone) as the blue party, then Figure 2A schematic diagram showing the confrontation between 4 red agent clusters and 3 blue agent clusters with different optimization goals under the same target task boundary. Among them, Agent_H1, Agent_H2, Agent_H3, and Agent_H4 are red agent clusters; Agent_L1, Agent_L2, and Agent_L3 are blue agent clusters; the target task boundary is: the red team cooperates with the blue team to confront the red team, the red team hopes to capture the blue team to achieve the goal, and the blue team has a certain ability to protect itself and has a certain ability to resist the capture of the red team.

[0104] The common point of Agent_H1, Agent_H2, Agent_H3, and Agent_H4 is that they can receive input information from the Red Team and output Red Team strategy information. The input information includes information such as Red Team collaborative operations, Blue Team situation information (such as type, location, number, flight capability, etc.), and Red Team optimization goals under the target mission boundary; Red Team strategy information includes: Red Team's action plan, such as the task team called, the equipment used, the target to be captured, the timing, etc. It should be noted that the difference between the above four Red Team agent clusters is that they have different Red Team constraints.

[0105] The task boundaries in the feature labels of the red agents in Agent_H1, Agent_H2, Agent_H3, and Agent_H4 are the same, but the optimization goals are different. For example, the red agent in Agent_H1 optimizes the fastest time under the premise of completing the goal, the red agent in Agent_H2 optimizes the least number of waves under the premise of completing the goal, the red agent in Agent_H3 optimizes the least number of equipment under the premise of completing the goal, and the red agent in Agent_H4 optimizes the least cost under the premise of completing the goal. The constraint feature labels of the red agents in each red agent cluster are also different.

[0106] Agent_L1, Agent_L2, and Agent_L3 are blue agent clusters. Similar to the red agent clusters, the blue agents in them all have the same point of receiving blue input information and outputting blue strategy information. The blue input information includes the red situation information, the blue optimization target under the target task boundary, and other information; the blue strategy information includes: the blue action plan, such as escape strategy and action plan, interference strategy and action plan, formation strategy and action plan, etc. It should be noted that the difference between the above three blue agent clusters is that they have different blue constraints.

[0107] Similarly, the task boundaries in the feature labels of the blue agents in Agent_L1, Agent_L2, and Agent_L3 are the same, but the optimization goals are different. For example, the blue agent in Agent_L1 optimizes the highest survival rate under the premise of completing the goal, the blue agent in Agent_L2 optimizes the lowest cost under the premise of completing the goal, and the blue agent in Agent_L3 optimizes the maximum use of reusable soft kills under the premise of completing the goal. The constraint feature labels of the blue agents in each blue agent cluster are also different.

[0108] Continue with Figure 2 As an example of the confrontation scenario shown in the figure, the target party is the red party. Then, the three blue party agent clusters of the blue party can be combined to obtain the red party agent cluster (C 1 4+C 2 4+C 3 4+C 4 4=) 7 opponent pools. Specifically, the 7 opponent pools can be included in the following set:

[0109] Opponent Pool 1: {Agent_L1}.

[0110] Opponent Pool 2: {Agent_L2}.

[0111] Opponent Pool 3: {Agent_L3}.

[0112] Opponent pool 4: {Agent_L1, Agent_L2}.

[0113] Opponent pool 5: {Agent_L1, Agent_L3}.

[0114] Opponent Pool 6: {Agent_L2, Agent_L3}.

[0115] Opponent Pool 7: {Agent_L1, Agent_L2, Agent_L3}.

[0116] It should be noted that if the blue side has k blue agent clusters, then for the red side, at most C 1 k +C 2 k +C 3 k +…+C k k A pool of opponents.

[0117] S104: Copy the target agent cluster to obtain multiple personalized agent clusters corresponding to multiple opponent pools.

[0118] It should be noted that after any target agent cluster is replicated, each personalized agent cluster includes multiple initial target agents in the target agent cluster. In other words, by training each opponent pool separately, multiple personalized decision-making agent clusters with the same number as the opponent pool can be obtained.

[0119] Reference Figure 3 , showing a schematic diagram of adversarial training of personalized agent clusters. Figure 3 The explanation is based on Agent_H1 in the above four red agent clusters. Since Agent_H1 has a 7-point opponent pool, 7 personalized agent clusters can be obtained, namely Agent_H1_1, Agent_H1_2, Agent_H1_3, Agent_H1_4, Agent_H1_5, Agent_H1_6, and Agent_H1_7, which are 7 personalized agent clusters obtained by Agent_H1 selecting different opponent pools.

[0120] In this embodiment, a personalized feature label is set for each personalized agent in each personalized agent cluster, and the personalized feature label includes personalized agent label, task boundary, constraint condition, optimization target, opponent situation and round situation, etc.

[0121] By analogy, the remaining three Red Agent clusters can be obtained: Agent_H2_1, Agent_H2_2, ..., Agent_H2_7, Agent_H3_1, Agent_H3_2, ..., Agent_H3_7, Agent_H4_1, Agent_H4_2, Agent_H4_3, ..., Agent_H4_7. In this way, a total of 28 Red Agent personalized agent clusters can be obtained.

[0122] If you need to train a personalized blue agent, the training method is similar to that of the red agent. The following example illustrates this:

[0123] Taking the training of the blue agent cluster Agent_L1 as an example, we first construct the following (C 1 4+C 2 4+C 3 4+C 4 4=) 15 opponent pools.

[0124] Opponent Pool 1: {Agent_H1}.

[0125] Opponent Pool 2: {Agent_H2}.

[0126] Opponent Pool 3: {Agent_H3}.

[0127] Opponent Pool 4: {Agent_H4}.

[0128] Opponent Pool 5: {Agent_H1, Agent_H2}.

[0129] Opponent Pool 6: {Agent_H1, Agent_H3}.

[0130] Opponent Pool 7: {Agent_H1, Agent_H4}.

[0131] Opponent pool 8: {Agent_H2, Agent_H3}.

[0132] Opponent Pool 9: {Agent_H2, Agent_H4}.

[0133] Opponent Pool 10: {Agent_H3, Agent_H4}.

[0134] Opponent pool 11: {Agent_H1, Agent_H2, Agent_H3}.

[0135] Opponent pool 12: {Agent_H1, Agent_H2, Agent_H4}.

[0136] Opponent Pool 13: {Agent_H1, Agent_H3, Agent_H4}.

[0137] Opponent pool 14: {Agent_H2, Agent_H3, Agent_H4}.

[0138] Opponent Pool 15: {Agent_H1, Agent_H2, Agent_H3, Agent_H4}.

[0139] It should be noted that if the red side has s red agent clusters, then for the blue side, at most C 1 s+C 2 s+C 3 s+…+C s s opponent pools.

[0140] For each opponent pool, training is carried out separately, and 15 blue side personalized agent clusters can be obtained, including Agent_L1_1, Agent_L1_2, ..., Agent_L1_15, and so on: Agent_L2_1, Agent_L2_2, ..., Agent_L2_15, Agent_L3_1, Agent_L3_2, ..., Agent_L3_15. Each cluster still contains personalized agents with different constraints, and there are a total of 45 blue side personalized agent clusters.

[0141] S105: For any personalized agent cluster, the personalized agent cluster is subjected to adversarial training with an opponent agent cluster in an opponent pool corresponding to the personalized agent cluster, so as to train and obtain personalized agents corresponding to each initial target agent in the personalized agent cluster.

[0142] In this embodiment, after obtaining multiple personalized agent clusters of the target party, a layered adversarial approach will be adopted for training, that is, each personalized agent cluster will carry out adversarial training with the opponent agent cluster in the corresponding opponent pool.

[0143] For example, continue to refer to Figure 3 For Agent_H1, it will only perform adversarial training with Agent_L1, while for Agent_H7, it needs to perform adversarial training with Agent_L1, Agent_L2, and Agent_L3.

[0144] Specifically, during the adversarial process, for the initial target agent in each personalized agent cluster, the initial target agent needs to undergo adversarial training with different opponent agents in the opponent pool to continuously update and optimize its agent parameters, so as to finally obtain the corresponding personalized agent.

[0145] It should be noted that since each personalized agent is trained based on different opponent pools, each personalized agent can effectively fight against opponents with the same fighting style as the corresponding opponent pool. For example, a personalized agent with a certain red constraint in Agent_H1 can effectively fight against various blue agents with different blue constraints in Agent_L1.

[0146] S106: For any initial target party agent, based on agent parameters of multiple personalized agents corresponding to the initial target party agent, determine the agent parameters of the initial target party agent to obtain a personalized target party agent corresponding to the initial target party agent.

[0147] In this embodiment, after multiple personalized agents corresponding to the initial target agent are obtained through training, the agent parameters of the multiple personalized agents are comprehensively considered to obtain the final agent parameters of the initial target agent.

[0148] For example, after training to obtain the Red Party personalized agent cluster including Agent_H1_1, Agent_H1_2, Agent_H1_3, Agent_H1_4, Agent_H1_5, Agent_H1_6, and Agent_H1_7, new parameters of the corresponding agent in the Red Party agent cluster Agent_H1 can be calculated based on the agent parameters of the Red Party personalized agents in the 7 Red Party personalized agent clusters.

[0149] In one example, the red side has N optimization goals and P red side constraints, then there are N red side agent clusters, and N×P initial red side agents. Multiple blue side agent clusters of the blue side are combined to obtain 10 opponent pools. After the above training process, the red side can expand to obtain 10×N personalized agent clusters and 10×N×P personalized agents. After the parameters of the 10×N×P personalized agents are integrated, N×P personalized red side agents corresponding to the N×P initial red side agents are finally obtained. Among them, each personalized red side agent has a unique red side optimization goal and red side constraint, and each personalized red side agent can effectively fight against opponents with different blue side optimization goals and / or different blue side constraints.

[0150] The embodiment of the present application provides an agent training method based on hierarchical confrontation. By constructing multiple opponent pools and multiple personalized agent clusters corresponding to the multiple opponent pools, the initial target agent in each personalized agent cluster can be trained against the opponent agent with different optimization goals and different constraints, thereby obtaining a personalized agent that can adapt to different opponent styles. Finally, based on the agent parameters of the multiple personalized agents, the agent parameters of the initial target agent can be obtained, thereby obtaining a personalized target agent with strong robustness and comprehensive ability. In this way, the generalization ability of the agent can be effectively improved, and even when facing complex and changeable confrontation tasks, a good confrontation effect can be achieved.

[0151] In a feasible implementation manner, the step of determining multiple target party constraints of the target party under the target task boundary in S101 may specifically include the following sub-steps:

[0152] S101-1: Obtain the constraint set of the target party.

[0153] In this embodiment, the constraint condition set includes multiple constraint condition subsets, and different constraint condition subsets are used to represent different constraint conditions of the target party.

[0154] It should be noted that each constraint has an optional specific value or value range, each corresponding to a constraint subset. For example, a constraint subset may include a subset of equipment usage type values, a subset of resource team calls, a subset of positions used, a subset of whether to use a collaborative approach and how to collaborate, a subset of success rate requirements, a subset of total wave requirements, and a subset of time period requirements. All constraint subsets constitute a constraint set.

[0155] Exemplarily, taking the target party's equipment usage type value subset as an example, each element in the subset is the equipment usage type or a combination of multiple types allowed by the operator / commander.

[0156] In an example, a drone capture device acts as the capture party of a drone. The usage type value subset corresponding to the capture equipment configured by it is {a, b, (a, b), c}, where a corresponds to the use of only equipment a, b and c correspond to the use of only equipment b or equipment c respectively, and (a, b) corresponds to equipment a and equipment b being used together. The usage type value subset of this equipment does not include (b, c), (a, c), and (a, b, c), indicating that the operator subjectively disagrees with the combined use of the above equipment. The equipment usage type value subset can be set according to specific application problems.

[0157] It should be noted that if the equipment usage type value subset is {a, b, (a, b), c}, then the number of elements in the equipment usage type value subset is 4. In this way, the sum of the number of elements in each subset in the constraint condition set can be obtained by counting, which is M.

[0158] S101 - 2: Combine multiple constraint condition subsets in the constraint condition set to obtain multiple target party constraints of the target party under the target task boundary.

[0159] In this embodiment, multiple constraint condition subsets of the target party are combined, that is, all elements in all constraint condition subsets are combined. Then, for the target party, at most C 1 M +C 2 M +C 3 M +…+C M M target constraints.

[0160] It should be noted that C 1 M +C 2 M +C 3 M +…+CM M It is the maximum number of target constraints that the operator / commander can select in each confrontation. The operator / commander can delete and select. In specific applications, some combinations of target constraints will not be selected under a certain target optimization goal, and the number will be further reduced. The reduced number is P. P is the number of target constraints that the operator / commander can select under the target optimization goal in the actual application scenario. Then, P initial target agents can be constructed in the target agent cluster corresponding to a certain target optimization goal.

[0161] In a feasible implementation, before S105, the agent training method based on layered confrontation may further include the following steps:

[0162] S201: Initialize training for multiple initial target agents in the target agent cluster to determine initial agent parameters corresponding to each of the multiple initial target agents.

[0163] In this embodiment, since the agent parameters of each initial target agent are obtained randomly when constructing multiple initial target agents in each target agent cluster, if game adversarial training is directly conducted between agents, there may be problems such as poor training effect and long convergence time. Therefore, before conducting adversarial training between the two adversaries, each initial target agent will be initialized and trained so that each initial target agent has a certain adversarial ability, thereby obtaining the training efficiency and training effect of subsequent adversarial training.

[0164] In this embodiment, unlike adversarial training with an agent, the initialization training is only for the agent itself, and the opponents in the training environment are all constructed based on basic rules.

[0165] It should be noted that the basic rules include our rules and opponent rules. Among them, the opponent rules include the location constraints of the target group, the formation constraints of the target group, the flight capability constraints, the counterattack capability constraints, and the capability constraints of subordinates and vassals; our rules include position constraints, team resource quantity and capability constraints, equipment quantity and capability constraints, information assurance capability constraints, etc. The above-mentioned related constraints have corresponding optional constraint value ranges.

[0166] In the specific implementation, under the set task boundaries, reinforcement learning is used to train the initial intelligent agent. During each training, the situation, resources, confrontation, etc. of the enemy and us in the training environment appear randomly within the constrained numerical range of the basic rules.

[0167] It should be noted that the initialization training is for both adversaries, that is, the target party can be either of the two adversaries. Taking the red and blue parties as an example, the initial agent parameters of the red party's initial red agent and the initial agent parameters of the blue party's initial blue agent can be obtained by training respectively.

[0168] In the specific implementation, the corresponding training method is selected according to the structural design of the intelligent agent.

[0169] In a feasible implementation, S201 may specifically include the following sub-steps:

[0170] S201-A1: When the agent structures of multiple initial target party agents in the target party agent cluster are inconsistent, the multiple initial target party agents in the target party agent cluster are independently trained respectively until the multiple initial target party agents meet the preset first training cutoff condition, and the initial agent parameters corresponding to each of the multiple initial target party agents are determined.

[0171] In this embodiment, if the agent structures of multiple initial target-side agents in the target-side agent cluster are inconsistent, independent training is adopted for training.

[0172] Continue to refer to Figure 1 For example, Agent_H1 is a red agent cluster that optimizes the fastest time under the premise of completing the goal. When training a red agent in the red agent cluster Agent_H1, a simulation training environment is constructed through our basic rules and the opponent's basic rules. That is, the opponent and environment of the red agent to be trained are constructed through basic rules. The part that changes in each training is to randomly select specific values ​​within the rule constraint value range. For example, the relative position of positions in the formation of the target group has a certain range. For example, the interception strategies for offensive equipment include 1 intercept 1, 1 intercept 2, 1 intercept 3, etc. The training values ​​are randomly selected each time. Until convergence, a red agent in the red agent cluster Agent_H1 is obtained.

[0173] Initialization training is performed for the red and blue sides respectively, and the initial agent parameters of the initial red agents in the four red agent clusters Agent_H1, Agent_H2, Agent_H3, and Agent_H4 and the initial agent parameters of the initial blue agents in the three blue agent clusters Agent_L1, Agent_L2, and Agent_L3 can be obtained.

[0174] In a feasible implementation, S201 may further include the following sub-steps:

[0175] S201-B1: When the agent structures of multiple initial target agents in the target agent cluster are consistent, remove the shielding matrix of any initial target agent to obtain the initial simplified agent corresponding to the target agent cluster, and determine the simplified optimization target of the initial simplified agent based on multiple target optimization targets.

[0176] It should be noted that the shielding matrix represents the target constraint conditions of the initial target agent. By removing the shielding matrix of the initial target agent, it means removing the target constraint conditions corresponding to the initial target agent.

[0177] In this embodiment, considering that when the agent structures of multiple initial target agents in the target agent cluster are consistent, the only difference is that the shielding matrices of the initial target agents are different, therefore, by removing the shielding matrices of multiple initial target agents, the same initial simplified agent can be obtained.

[0178] In this embodiment, the simplified optimization target used in the reinforcement learning training can be the average of the target optimization targets corresponding to each of the multiple initial target intelligent agents.

[0179] S201-B2: Based on the simplified optimization objective and the preset first reinforcement learning algorithm, the initial simplified intelligent agent is trained until the initial simplified intelligent agent meets the preset first training cutoff condition, thereby obtaining a trained first simplified intelligent agent.

[0180] In this embodiment, a simulation training environment is constructed through our basic rules and the opponent's basic rules, that is, the opponent and environment of the initial simplified intelligent agent to be trained are constructed through basic rules. The changing part in each training is the random selection of specific values ​​within the rule constraint value range. For example, the relative positions of positions in the formation of the target group have a certain range. For example, the escape strategies for offensive equipment include 1 block 1, 1 block 2, 1 block 3, etc., and the values ​​are randomly selected for each training.

[0181] In this embodiment, a first simplified agent is trained by continuously training in a simulation training environment until a preset first training cutoff condition is met, wherein the first training cutoff condition may include that the number of training rounds reaches a first round threshold and / or that the update change amplitude of the agent parameter is less than a first amplitude threshold.

[0182] S201-B3: Based on the target optimization objectives corresponding to each of the multiple target agent clusters and the preset second reinforcement learning algorithm, the first simplified agent is trained respectively until the first simplified agent meets the first training cutoff condition, thereby obtaining multiple second simplified agents corresponding one-to-one to the multiple target agent clusters.

[0183] In this embodiment, after the first simplified agent is trained, it will be further trained in the same simulation training environment as above with the target optimization goal of each target agent cluster to obtain a simplified second simplified agent for each target agent cluster.

[0184] S201-B4: For any second simplified agent, the agent parameters of the second simplified agent are used as the agent parameters of multiple initial target agents in the corresponding target agent cluster, and each initial target agent is trained based on a preset third reinforcement learning algorithm until the initial target agent meets the first training cutoff condition, thereby obtaining the initial agent parameters of the initial target agent.

[0185] It should be noted that the first reinforcement learning algorithm, the second reinforcement learning algorithm and the third reinforcement learning algorithm may be the same or different and may be selected from a preset reinforcement learning algorithm library according to actual needs.

[0186] In this embodiment, after training a simplified second simplified agent for each target agent cluster, the parameters of the second simplified agent correspond to the initial parameters of the initial target agent in the target agent cluster, and the initial target agent in each target agent cluster is retrained to obtain the initial target agent with target constraints. Taking Agent_H1 as an example, the parameters of the simplified second simplified agent trained for Agent_H1 are used as the initial values ​​of the parameters of the initial red agent, and retrained to obtain the initial red agent with constraints.

[0187] In this embodiment, initialization training is performed for the red and blue sides respectively, and the initial agent parameters of the initial red agents in the four red agent clusters Agent_H1, Agent_H2, Agent_H3, and Agent_H4 and the initial agent parameters of the initial blue agents in the three blue agent clusters Agent_L1, Agent_L2, and Agent_L3 can be obtained.

[0188] In this embodiment, by initializing and training multiple initial target agents in the target agent cluster, the prototypes of multiple initial target agents can be constructed, so that each initial target agent has a certain confrontation ability, that is, the initial agent parameters of each initial target agent have a certain feasibility. Compared with directly letting the confrontation parties conduct confrontation training of agents, the training efficiency and training effect of confrontation training can be effectively improved, so that the agent parameters can converge quickly.

[0189] In a feasible implementation, S105 may further include the following sub-steps:

[0190] S105-1: For any initial target agent in any personalized agent cluster, repeat the following steps: randomly select an initial opponent agent from the opponent agent cluster in the opponent pool corresponding to the personalized agent cluster to perform adversarial training with the initial target agent, and update the agent parameters of the initial target agent based on the adversarial results.

[0191] In this embodiment, in the process of selecting a certain opponent pool to train and obtain a personalized agent, the opponent is randomly selected from the opponent pool, that is, the initial opponent agent in the opponent agent cluster in the opponent pool is randomly selected. Figure 3 , each time the initial target agent in the Agent_H1_4 personalized agent cluster is trained, the opponent randomly selects an initial opponent agent from the opponent pool 4: {Agent_L1, Agent_L2} for adversarial training, and other rules in the simulation training environment still randomly select specific values ​​within the rule range.

[0192] S105-2: When it is determined that the initial target party agent meets the preset second training cutoff condition, a personalized agent is obtained by training the initial target party agent based on the opponent pool.

[0193] In this embodiment, during the iterative training process, if it is determined that the initial target agent meets the preset second training cutoff condition, the current agent parameters of the initial target agent are fixed to obtain a personalized agent that completes the adversarial training.

[0194] In this embodiment, the second training cutoff condition can be set as: the training round of the initial target agent reaches the second round threshold and / or the update change amplitude of the agent parameter of the initial target agent is less than the second amplitude threshold.

[0195] In a feasible implementation, the agent training method based on layered adversarial learning may further include the following steps:

[0196] S301: When multiple personalized agent clusters are trained serially, determine a trained historical personalized agent cluster for an untrained current personalized agent cluster.

[0197] It should be noted that when computing resources are sufficient, multiple personalized agent clusters can be trained in parallel; when computing resources are insufficient, serial training is required. However, serial training is time-consuming. Therefore, in order to improve the efficiency of serial training, the parameter inheritance strategy will be used to optimize the agent parameters of the current personalized agent cluster that has not been trained.

[0198] For example, take the training of Agent_H1_4 as an example. Agent_L1 and Agent_L2 are included in the opponent pool of Agent_H1_4. In order to train a certain red agent in Agent_H1, the parameters of the agent in Agent_H1 are used as the initial parameters of Agent_H1_4 for training. The opponent is randomly selected from Agent_L1 and Agent_L2. The parameters of Agent_H1_4 are evolved using samples obtained from the training environment. This training method is time-consuming. The agent initialization method based on parameter inheritance does not use the parameters of the agent in Agent_H1 as the initial state of the corresponding agent parameters in Agent_H1_4, but through inheritance. In other words, the current personalized agent cluster that has not been trained can inherit the relevant agent parameters from the historical personalized agent cluster that has been trained, and then the initial target agent in the current personalized agent cluster can be optimized before adversarial training.

[0199] S302: For any historical personalized agent cluster, if the opponent pool of the historical personalized agent cluster includes any opponent agent cluster in the opponent pool of the current personalized agent cluster, the historical personalized agent cluster is determined to be an associated personalized agent cluster of the current personalized agent cluster, and the ratio between the number of opponent agent clusters in the opponent pool where the associated personalized agent cluster is located and the total number of opponent agent clusters in the opponent pool of all associated personalized agent clusters is determined as the association weight of the personalized agents in the associated personalized agent cluster.

[0200] It should be noted that for the current personalized agent cluster, there may be one or more historical personalized agent clusters. Therefore, it is necessary to first determine an associated personalized agent cluster associated with the current personalized agent cluster from one or more historical personalized agent clusters.

[0201] For example, continue to refer to Figure 3 , if Agent_H1_1, Agent_H1_2, Agent_H1_3, Agent_H1_4, Agent_H1_5, Agent_H1_6 and Agent_H1_7 are trained serially from top to bottom, then for Agent_H1_1, Agent_H1_2 and Agent_H1_3, there is no associated personalized agent cluster, but for Agent_H1_4, Agent_H1_5, Agent_H1_6 and Agent_H1_7, there is one or more associated personalized agent clusters.

[0202] In one example, Agent_H1_4's opponent pool includes Agent_L1 and Agent_L2, then all personalized agent clusters that contain their opponent pool subsets are associated personalized agent clusters related to Agent_H1_4. Since Agent_H1_1's opponent pool is Agent_L1 and Agent_H1_2's opponent pool is Agent_L2, Agent_H1_4's associated personalized agent clusters are Agent_H1_1 and Agent_H1_2. By calculating the ratio between the number of opponent agent clusters in the opponent pool where the associated personalized agent cluster is located and the total number of opponent agent clusters in the opponent pool of all associated personalized agent clusters, it can be obtained that the associated weights of Agent_H1_11 and Agent_H1_2 are both 1 / (1+1), that is, 1 / 2.

[0203] In another example, taking the initial target agent in the training of Agent_H1_7 as an example, the agent initialization method based on parameter inheritance is adopted. The opponent pool of Agent_H1_7 includes Agent_L1, Agent_L2 and Agent_L3. Then Agent_H1_1~Agent_H1_6 are all the associated personalized agent clusters of Agent_H1_7. And because:

[0204] The opponent pool of Agent_H1_1 contains the opponent Agent_L1.

[0205] The opponent pool of Agent_H1_2 contains the opponent Agent_L2.

[0206] The opponent pool of Agent_H1_3 contains the opponent Agent_L3.

[0207] The opponent pool of Agent_H1_4 includes opponents Agent_L1 and Agent_L2.

[0208] The opponent pool of Agent_H1_5 includes opponents Agent_L1 and Agent_L3.

[0209] The opponent pool of Agent_H1_6 includes opponents Agent_L2 and Agent_L3.

[0210] Therefore, the opponent pool of Agent_H1_1 includes the opponent Agent_L1, the opponent pool of Agent_H1_2 includes the opponent Agent_L2, the opponent pool of Agent_H1_3 includes the opponent Agent_L3, the opponent pool of Agent_H1_4 includes the opponents Agent_L1 and Agent_L2, the opponent pool of Agent_H1_5 includes the opponents Agent_L1 and Agent_L3, and the opponent pool of Agent_H1_6 includes the opponents Agent_L2 and Agent_L3. It can be counted that the total number of opponent agent clusters in the opponent pools of all associated personalized agent clusters is 9, and the following association weight information can be obtained:

[0211] The weights of Agent_H1_1, Agent_H1_2, and Agent_H1_3 are all 1 / 9.

[0212] The weights of Agent_H1_4, Agent_H1_5, and Agent_H1_6 are all 2 / 9.

[0213] S303: For any initial target party agent in the current personalized agent cluster, determine the initial agent parameters of the initial target party agent based on the associated weight and agent parameters of the personalized agent corresponding to the initial target party agent in the associated personalized agent cluster.

[0214] In this embodiment, after calculating the association weight of the personalized agent corresponding to the initial target party agent in the associated personalized agent cluster, the initial agent parameters of the initial target party agent can be determined according to the association weight and the corresponding agent parameters.

[0215] For example, taking an initial target party agent in Agent_H1_4 as an example, since the association weights of Agent_H1_11 and Agent_H1_2 are both 1 / 2, the agent parameters of the personalized agents corresponding to the initial target party agent in Agent_H1_11 and Agent_H1_2 can be weighted and summed according to the association weights to obtain the initial agent parameters of the initial target party agent.

[0216] It should be noted that if the current personalized agent cluster does not have an associated personalized agent cluster, the agent parameters in the current personalized agent cluster are the initial agent parameters obtained through the initialization training of S201; if the current personalized agent cluster has an associated personalized agent cluster, the initial agent parameters obtained through the initialization training of S201 are further optimized to obtain the initial agent parameters for adversarial training.

[0217] In this embodiment, by adopting a parameter inheritance-based training strategy to perform serial training on the personalized agent cluster, the training efficiency can be effectively improved.

[0218] In a feasible implementation, S106 may specifically include the following sub-steps:

[0219] S106-A: Determine the average value of the agent parameters of the multiple personalized agents corresponding to the initial target party agent as the agent parameter of the initial target party agent.

[0220] In this embodiment, the agent parameters of the initial target agent can be calculated by taking the average value.

[0221] For example, for an initial target agent in Agent_H1, after layered adversarial training, corresponding personalized agents are trained in Agent_H1_1, Agent_H1_2, Agent_H1_3, Agent_H1_4, Agent_H1_5, Agent_H1_6, and Agent_H1_7 respectively. The average value of the agent parameters of the seven personalized agents can be used as the agent parameters of the initial target agent to obtain the personalized target agent corresponding to the initial target agent.

[0222] S106-B: For any personalized agent corresponding to the initial target agent, determine the weight parameter of the personalized agent based on the ratio between the number of opponent agent clusters in the opponent pool corresponding to the personalized agent and the total number of opponent agent clusters in all opponent pools; and determine the agent parameters of the initial target agent based on the weight parameters and agent parameters corresponding to each of the multiple personalized agents.

[0223] In this embodiment, considering that each personalized agent faces different opponents, that is, the number of opponent agent clusters in the opponent pool corresponding to each personalized agent may be different, the comprehensive performance of each personalized agent also has certain differences. For example, the comprehensive performance of a personalized agent trained based on three opponent agent clusters is usually better than that of a personalized agent trained based on one or two opponent agent clusters. Therefore, a weighted calculation will be performed on multiple personalized agents to obtain more accurate agent parameters of the initial target agent.

[0224] Continue with Figure 2 Taking the confrontation scenario shown in the figure as an example, we can get the 7 opponent pools of the red agent cluster, including:

[0225] Agent_H1_1's opponent pool 1: {Agent_L1}.

[0226] Agent_H1_2's opponent pool 2: {Agent_L2}.

[0227] Agent_H1_3's opponent pool 3: {Agent_L3}.

[0228] Agent_H1_4's opponent pool 4: {Agent_L1, Agent_L2}.

[0229] Agent_H1_5's opponent pool 5: {Agent_L1, Agent_L3}.

[0230] Agent_H1_6's opponent pool 6: {Agent_L2, Agent_L3}.

[0231] Agent_H1_7's opponent pool 7: {Agent_L1, Agent_L2, Agent_L3}.

[0232] Therefore, it can be determined that the total number of opponent agent clusters in all opponent pools is 12, then the weight parameters of personalized agents in Agent_H1_1, Agent_H1_2 and Agent_H1_3 are all 1 / 12; the weight parameters of personalized agents in Agent_H1_4, Agent_H1_5 and Agent_H1_6 are all 2 / 12, that is, 1 / 6; the weight parameters of personalized agents in Agent_H1_7 are all 3 / 12, that is, 1 / 4.

[0233] In this embodiment, after calculating the weight parameters of multiple personalized agents corresponding to the initial target agent in multiple personalized agent clusters, the agent parameters of the multiple personalized agents can be weighted summed according to the weight parameters to obtain the agent parameters of the initial target agent, and then obtain the final personalized target agent.

[0234] In a feasible implementation, after S106, the agent training method based on layered confrontation may further include the following steps:

[0235] S107: Acquire the target party information and / or counterparty party information input by the user.

[0236] It should be noted that the target party information includes the target party optimization target and / or target party constraint conditions input for the target party, and the opponent party information includes the opponent party optimization target and / or opponent party constraint conditions input for the opponent party. Among them, the user can specifically be a commander or operator of the drone capture device.

[0237] S108: Based on the target party information and / or the counterparty party information, determine one or more target agents from a plurality of personalized target party agents.

[0238] It should be noted that the target agent is used to generate adversarial plans against the opponent.

[0239] In this embodiment, if there are N target optimization objectives and P target constraints, N×P personalized target agents can be trained in the end, where different personalized target agents have unique target optimization objectives and target constraints, and each personalized target agent can effectively fight against opponents with different specific opponent optimization objectives and / or different opponent constraints.

[0240] In this embodiment, the commander / operator can select a corresponding personalized target agent based on the target party information and the opponent party information.

[0241] Specifically, the operator can comprehensively consider the opponent's optimization goals that the opponent may adopt (corresponding to the opponent's intelligent agents that will be used), the opponent's optimization goals that the opponent may not adopt (corresponding to the opponent's intelligent agents that will not be used), and the target optimization goals and target constraints that the target party hopes to adopt (corresponding to the different target party red intelligent agents that are hoped to be adopted), and select one or more target intelligent agents from N×P personalized target intelligent agents to generate confrontation plans to confront the opponent. It should be noted that when there are multiple target intelligent agents, multiple sets of confrontation plans can be generated, and the commander / operator can make further selections from multiple sets of confrontation plans based on experience, or directly confront the opponent separately according to multiple sets of confrontation plans.

[0242] In one example, the operator, as the drone capture party, knows the target party information such as the capture strategy and flight capability of the drone capture device, and can then filter out one or more corresponding target party agents from multiple personalized target party agents by inputting the target party information related to the drone capture device.

[0243] Second, refer to Figure 4 The embodiment of the present application provides an intelligent agent training device 400 based on layered confrontation, and the intelligent agent training device 400 based on layered confrontation includes:

[0244] The condition determination module 401 is used to determine any one of the two opposing parties as the target party, and determine multiple target party optimization objectives and multiple target party constraint conditions of the target party under the target task boundary;

[0245] The agent construction module 402 is used to construct a target agent cluster of the target under the target optimization target for any target optimization target, and to construct multiple initial target agents in the target agent cluster based on multiple target constraint conditions; wherein different target optimization targets correspond to different target agent clusters;

[0246] The opponent pool determination module 403 is used to determine the other party of the opposing parties relative to the target party as the opponent, and combine multiple opponent agent clusters of the opponent to obtain multiple opponent pools of the target party agent cluster; wherein different opponent pools are composed of different opponent agent clusters;

[0247] A copy module 404 is used to copy the target agent cluster to obtain multiple personalized agent clusters corresponding to multiple opponent pools; each personalized agent cluster includes multiple initial target agents in the target agent cluster;

[0248] The adversarial training module 405 is used to perform adversarial training on any personalized agent cluster, between the personalized agent cluster and the opponent agent cluster in the opponent pool corresponding to the personalized agent cluster, so as to train and obtain the personalized agents corresponding to each initial target agent in the personalized agent cluster;

[0249] The parameter determination module 406 is used to determine the agent parameters of the initial target agent for any initial target agent based on the agent parameters of multiple personalized agents corresponding to the initial target agent, so as to obtain the personalized target agent corresponding to the initial target agent.

[0250] In an embodiment of the present application, the condition determination module 401 includes:

[0251] A set acquisition submodule is used to acquire a constraint condition set of a target party; the constraint condition set includes multiple constraint condition subsets, and different constraint condition subsets are used to represent different constraint conditions of the target party;

[0252] The constraint combination submodule is used to combine multiple constraint subsets in the constraint set to obtain multiple target constraint conditions of the target under the target task boundary.

[0253] In one embodiment of the present application, the agent training device 400 based on layered confrontation further includes:

[0254] The initialization training module is used to perform initialization training on multiple initial target party agents in the target party agent cluster to determine the initial agent parameters corresponding to each of the multiple initial target party agents.

[0255] In one embodiment of the present application, the initialization training module includes:

[0256] The first initialization training submodule is used to independently train multiple initial target party agents in the target party agent cluster when the agent structures of multiple initial target party agents in the target party agent cluster are inconsistent, until the multiple initial target party agents meet the preset first training cutoff condition, and determine the initial agent parameters corresponding to each of the multiple initial target party agents.

[0257] In one embodiment of the present application, the initialization training module includes:

[0258] The agent simplification submodule is used to remove the shielding matrix of any initial target agent in the target agent cluster when the agent structures of multiple initial target agents are consistent, obtain the initial simplified agent corresponding to the target agent cluster, and determine the simplified optimization target of the initial simplified agent based on multiple target optimization targets; wherein the shielding matrix represents the target constraint condition of the initial target agent;

[0259] A first training submodule is used to train the initial simplified intelligent agent based on the simplified optimization objective and the preset first reinforcement learning algorithm until the initial simplified intelligent agent meets the preset first training cutoff condition, thereby obtaining a trained first simplified intelligent agent;

[0260] A second training submodule is used to train the first simplified agent based on the target optimization objectives corresponding to each of the multiple target agent clusters and a preset second reinforcement learning algorithm, until the first simplified agent meets the first training cutoff condition, thereby obtaining multiple second simplified agents corresponding to the multiple target agent clusters one by one;

[0261] The third training submodule is used to use the agent parameters of the second simplified agent as the agent parameters of multiple initial target agents in the corresponding target agent cluster for any second simplified agent, and train each initial target agent based on a preset third reinforcement learning algorithm until the initial target agent meets the first training cutoff condition, thereby obtaining the initial agent parameters of the initial target agent.

[0262] In one embodiment of the present application, the adversarial training module 405 includes:

[0263] The iterative training submodule is used to repeat the following steps for any initial target agent in any personalized agent cluster: randomly select an initial opponent agent from the opponent agent cluster in the opponent pool corresponding to the personalized agent cluster to perform adversarial training with the initial target agent, and update the agent parameters of the initial target agent based on the adversarial results;

[0264] The agent parameter determination submodule is used to obtain a personalized agent obtained by training the initial target agent based on the opponent pool when it is determined that the initial target agent meets the preset second training cutoff condition.

[0265] In one embodiment of the present application, the agent training device 400 based on layered confrontation further includes:

[0266] A historical personalized agent cluster determination module is used to determine a trained historical personalized agent cluster for an untrained current personalized agent cluster when multiple personalized agent clusters are trained in series;

[0267] An association weight determination module is used for determining, for any historical personalized agent cluster, that the historical personalized agent cluster is an associated personalized agent cluster of the current personalized agent cluster if the opponent pool of the historical personalized agent cluster includes any opponent agent cluster in the opponent pool of the current personalized agent cluster, and determining the ratio between the number of opponent agent clusters in the opponent pool where the associated personalized agent cluster is located and the total number of opponent agent clusters in the opponent pool of all associated personalized agent clusters as the association weight of the personalized agents in the associated personalized agent cluster;

[0268] The initial agent parameter determination module is used to determine the initial agent parameters of any initial target agent in the current personalized agent cluster based on the associated weights and agent parameters of the personalized agents corresponding to the initial target agent in the associated personalized agent cluster.

[0269] In an embodiment of the present application, the parameter determination module 406 includes:

[0270] The first parameter determination submodule is used to determine the average value of the agent parameters of multiple personalized agents corresponding to the initial target party agent as the agent parameter of the initial target party agent; or

[0271] The second parameter determination submodule is used to determine the weight parameters of the personalized agent for any personalized agent corresponding to the initial target agent based on the ratio between the number of opponent agent clusters in the opponent pool corresponding to the personalized agent and the total number of opponent agent clusters in all opponent pools; and determine the agent parameters of the initial target agent based on the weight parameters and agent parameters corresponding to each of the multiple personalized agents.

[0272] In one embodiment of the present application, the agent training device 400 based on layered confrontation includes:

[0273] An information acquisition module, used to acquire target party information and / or counterparty information input by a user; wherein the target party information includes a target party optimization target and / or a target party constraint condition input by the target party, and the counterparty information includes a counterparty optimization target and / or a counterparty constraint condition input by the counterparty;

[0274] The agent determination module is used to determine one or more target agents from multiple personalized target agents based on target party information and / or opponent party information; wherein the target agent is used to generate a confrontation plan for the opponent party.

[0275] It should be noted that the specific implementation of the intelligent agent training device 400 based on layered confrontation in the embodiment of the present application refers to the specific implementation of the intelligent agent training method based on layered confrontation proposed in the first aspect of the embodiment of the present application, and will not be repeated here.

[0276] It will be appreciated by those skilled in the art that the embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the embodiments of the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the embodiments of the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0277] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0278] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0279] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0280] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0281] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0282] The above is a detailed introduction to the intelligent agent training method and device based on hierarchical confrontation provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A method for training an intelligent agent based on layered confrontation, characterized in that: The method comprises: Determine any one of the two opposing parties as a target party, and determine multiple target party optimization objectives and multiple target party constraints for the target party under the target task boundary; the target party is a drone capture device; For any target optimization target, a target agent cluster of the target under the target optimization target is constructed, and based on a plurality of target constraint conditions, a plurality of initial target agents in the target agent cluster are constructed; wherein different target optimization targets correspond to different target agent clusters; The other party of the two opposing parties relative to the target party is determined as an opponent party, and multiple opponent party agent clusters of the opponent party are combined to obtain multiple opponent party pools of the target party agent cluster; wherein different opponent party pools are composed of different opponent party agent clusters; and the opponent party is a drone to be captured; Copying the target agent cluster to obtain a plurality of personalized agent clusters corresponding to the plurality of opponent pools; each of the personalized agent clusters includes a plurality of initial target agents in the target agent cluster; For any personalized agent cluster, the personalized agent cluster is subjected to adversarial training with an opponent agent cluster in an opponent pool corresponding to the personalized agent cluster, so as to train and obtain a personalized agent corresponding to each initial target agent in the personalized agent cluster; For any of the initial target party intelligent agents, based on the intelligent agent parameters of the multiple personalized intelligent agents corresponding to the initial target party intelligent agent, the intelligent agent parameters of the initial target party intelligent agent are determined to obtain a personalized target party intelligent agent corresponding to the initial target party intelligent agent; the personalized target party intelligent agent is used to generate a countermeasure plan for the drone to be captured; For any personalized agent cluster, the step of conducting adversarial training between the personalized agent cluster and an opponent agent cluster in an opponent pool corresponding to the personalized agent cluster to obtain a personalized agent corresponding to each initial target agent in the personalized agent cluster, comprises: For any of the initial target agents in any of the personalized agent clusters, repeat the following steps: randomly select an initial opponent agent from the opponent agent cluster in the opponent pool corresponding to the personalized agent cluster to perform adversarial training with the initial target agent, and update the agent parameters of the initial target agent based on the adversarial result; When it is determined that the initial target party agent satisfies a preset second training cutoff condition, obtaining a personalized agent obtained by training the initial target party agent based on the opponent pool; For any of the initial target party agents, the step of determining the agent parameters of the initial target party agent based on the agent parameters of the plurality of personalized agents corresponding to the initial target party agent comprises: Determine the average value of the agent parameters of the plurality of personalized agents corresponding to the initial target agent as the agent parameter of the initial target agent; or For any personalized agent corresponding to the initial target agent, a weight parameter of the personalized agent is determined based on the ratio between the number of opponent agent clusters in the opponent pool corresponding to the personalized agent and the total number of opponent agent clusters in all opponent pools; and the agent parameters of the initial target agent are determined based on the weight parameters and agent parameters corresponding to each of the multiple personalized agents.

2. The method for training an intelligent agent based on layered confrontation according to claim 1, characterized in that: The step of determining a plurality of target party constraints of the target party under the target task boundary comprises: Acquire a constraint condition set of the target party; the constraint condition set includes a plurality of constraint condition subsets, and different constraint condition subsets are used to represent different constraint conditions of the target party; Multiple constraint condition subsets in the constraint condition set are combined to obtain multiple target party constraints of the target party under the target task boundary.

3. The method for training an intelligent agent based on layered confrontation according to claim 1, characterized in that: Before the step of performing adversarial training on any initial target agent in the target agent cluster and initial opponent agents in the plurality of opponent pools to obtain a personalized target agent corresponding to the initial target agent, the method further comprises: Initialization training is performed on the multiple initial target-side intelligent agents in the target-side intelligent agent cluster to determine initial intelligent agent parameters corresponding to each of the multiple initial target-side intelligent agents.

4. The method for training an intelligent agent based on layered confrontation according to claim 3, characterized in that: The step of performing initialization training on the plurality of initial target-side agents in the target-side agent cluster to determine initial agent parameters corresponding to each of the plurality of initial target-side agents comprises: In the case that the agent structures of multiple initial target party agents in the target party agent cluster are inconsistent, the multiple initial target party agents in the target party agent cluster are independently trained respectively until the multiple initial target party agents meet a preset first training cutoff condition, and the initial agent parameters corresponding to each of the multiple initial target party agents are determined.

5. The method for training an intelligent agent based on layered confrontation according to claim 3, characterized in that: The step of performing initialization training on the plurality of initial target-side agents in the target-side agent cluster to determine initial agent parameters corresponding to each of the plurality of initial target-side agents comprises: In the case where the agent structures of multiple initial target-side agents in the target-side agent cluster are consistent, removing the shielding matrix of any of the initial target-side agents to obtain the initial simplified agent corresponding to the target-side agent cluster, and determining the simplified optimization target of the initial simplified agent based on the multiple target-side optimization targets; wherein the shielding matrix represents the target-side constraint condition of the initial target-side agent; Based on the simplified optimization objective and a preset first reinforcement learning algorithm, the initial simplified intelligent agent is trained until the initial simplified intelligent agent meets a preset first training cutoff condition, thereby obtaining a trained first simplified intelligent agent; Based on the target optimization objectives corresponding to each of the multiple target agent clusters and a preset second reinforcement learning algorithm, the first simplified agent is trained respectively until the first simplified agent meets the first training cutoff condition, thereby obtaining multiple second simplified agents corresponding to the multiple target agent clusters one by one; For any second simplified agent, the agent parameters of the second simplified agent are used as the agent parameters of multiple initial target agents in the corresponding target agent cluster, and each of the initial target agents is trained based on a preset third reinforcement learning algorithm until the initial target agent meets the first training cutoff condition, thereby obtaining the initial agent parameters of the initial target agent.

6. The method for training an intelligent agent based on layered confrontation according to claim 1, characterized in that: The method further comprises: In the case where the plurality of personalized agent clusters are trained serially, determining a trained historical personalized agent cluster for an untrained current personalized agent cluster; For any of the historical personalized agent clusters, if the opponent pool of the historical personalized agent cluster includes any opponent agent cluster in the opponent pool of the current personalized agent cluster, the historical personalized agent cluster is determined to be an associated personalized agent cluster of the current personalized agent cluster, and the ratio between the number of opponent agent clusters in the opponent pool where the associated personalized agent cluster is located and the total number of opponent agent clusters in the opponent pool of all the associated personalized agent clusters is determined as the association weight of the personalized agents in the associated personalized agent cluster; For any initial target party agent in the current personalized agent cluster, the initial agent parameters of the initial target party agent are determined based on the associated weights and agent parameters of the personalized agents corresponding to the initial target party agent in the associated personalized agent cluster.

7. The method for training an intelligent agent based on layered confrontation according to claim 1, characterized in that: After the step of determining the agent parameters of any of the initial target party agents based on the agent parameters of the plurality of personalized agents corresponding to the initial target party agent to obtain the personalized target party agent corresponding to the initial target party agent, the method further comprises: Acquire target party information and / or counterparty information input by a user; wherein the target party information includes a target party optimization target and / or a target party constraint condition input by the target party, and the counterparty information includes a counterparty optimization target and / or a counterparty constraint condition input by the counterparty; Based on the target party information and / or the opponent party information, one or more target agents are determined from the multiple personalized target party agents; wherein the target agent is used to generate a confrontation plan for the opponent party.

8. An intelligent agent training device based on layered confrontation, characterized in that: The device comprises: A condition determination module is used to determine any one of the two opposing parties as a target party, and determine multiple target party optimization targets and multiple target party constraint conditions of the target party under the target task boundary; the target party is a drone capture device; An agent construction module is used to construct a target agent cluster of the target under the target optimization target for any target optimization target, and to construct multiple initial target agents in the target agent cluster based on multiple target constraint conditions; wherein different target optimization targets correspond to different target agent clusters; An opponent pool determination module is used to determine the other party of the two opposing parties relative to the target party as the opponent, and combine multiple opponent agent clusters of the opponent to obtain multiple opponent pools of the target agent cluster; wherein different opponent pools are composed of different opponent agent clusters; and the opponent is a drone to be captured; A copy module, used to copy the target agent cluster to obtain a plurality of personalized agent clusters corresponding to the plurality of opponent pools; each of the personalized agent clusters includes a plurality of initial target agents in the target agent cluster; An adversarial training module is used to perform adversarial training on any personalized agent cluster, comparing the personalized agent cluster with an opponent agent cluster in an opponent pool corresponding to the personalized agent cluster, so as to train and obtain personalized agents corresponding to each initial target agent in the personalized agent cluster; A parameter determination module, for determining the agent parameters of any of the initial target agent based on the agent parameters of the plurality of personalized agents corresponding to the initial target agent, so as to obtain a personalized target agent corresponding to the initial target agent; the personalized target agent is used to generate a countermeasure against the drone to be captured; The adversarial training module includes: The iterative training submodule is used to repeat the following steps for any of the initial target agents in any of the personalized agent clusters: randomly selecting an initial opponent agent from the opponent agent cluster in the opponent pool corresponding to the personalized agent cluster to perform adversarial training with the initial target agent, and updating the agent parameters of the initial target agent based on the adversarial result; An agent parameter determination submodule, configured to obtain a personalized agent obtained by training the initial target agent based on the opponent pool when it is determined that the initial target agent satisfies a preset second training cutoff condition; The parameter determination module comprises: A first parameter determination submodule is used to determine the average value of the agent parameters of the plurality of personalized agents corresponding to the initial target agent as the agent parameter of the initial target agent; or The second parameter determination submodule is used to determine the weight parameter of any personalized agent corresponding to the initial target agent based on the ratio between the number of opponent agent clusters in the opponent pool corresponding to the personalized agent and the total number of opponent agent clusters in all opponent pools; and determine the agent parameters of the initial target agent based on the weight parameters and agent parameters corresponding to each of the multiple personalized agents.

Images