Clustered adversarial method and system based on expert knowledge assisted deep reinforcement learning

Abstract

The application provides a cluster confrontation method and system based on expert knowledge assisted deep reinforcement learning. By introducing an expert knowledge base and imitation learning technology, combining deep reinforcement learning, the efficiency of the collaborative decision of the agent is optimized, thereby improving the initial strategy learning speed of the system and the overall combat effectiveness. The method aims to provide an effective initial strategy acquisition mechanism, accelerate the early strategy learning of the agent by using the expert knowledge base, reduce the training time, and optimize the strategy of the agent in a complex dynamic environment through a multi-agent deep reinforcement learning algorithm to improve the collaborative combat capability. The scheme can greatly shorten the time required for initial strategy learning, obtain a more optimized strategy by combining deep reinforcement learning, ensure the efficiency of strategy optimization, and further ensure the real-time performance in large-scale cluster confrontation.

Description

Technical Field

[0001] This invention relates to the field of deep reinforcement learning technology, and more specifically, to a swarm adversarial method and system based on expert knowledge-assisted deep reinforcement learning. Background Technology

[0002] With the development of unmanned systems and multi-agent technologies, the application scenarios of collaborative operations among multiple intelligent agents are gradually increasing. For example, drone swarms and unmanned underwater vehicle swarms are widely used in military, surveillance, and disaster relief scenarios. These systems typically involve multiple autonomous intelligent agents performing collaborative tasks in dynamic and complex environments, which places higher demands on the intelligence level of swarm adversarial game systems. Especially under conditions of incomplete information, complex adversarial environments, and high real-time requirements, how to achieve optimal decision-making in adversarial tasks through effective cooperation among intelligent agents has become one of the core issues in current research on swarm adversarial game systems.

[0003] Current mainstream technologies largely rely on deep reinforcement learning and multi-agent reinforcement learning methods to optimize the decision-making behavior of multiple agents in complex environments. Patent CN109657802B proposes a hybrid reinforcement learning method combining expert systems and deep learning. This method uses multi-agent agents sharing network parameters as experts to provide guidance, improving the system's adaptability and learning efficiency in multi-task scenarios. Patent CN115903903A proposes a reinforcement learning method based on common goods game theory for collaborative combat in UAV swarms. This method enhances the collaborative capabilities of UAV swarms by introducing optimization of common resources among multiple agents. Patent CN115525058A proposes an adversarial method for multi-agent unmanned underwater vehicles, primarily using deep reinforcement learning to optimize the collaborative decision-making strategy of the underwater vehicles. Patent CN117787444A proposes a method for rapidly integrating different intelligent algorithms, solving the efficiency problems caused by redundant code and poor algorithm collaboration during the integration process.

[0004] However, the technical solution proposed in patent CN109657802B mainly targets transfer learning between single-task and expert tasks, failing to effectively address the real-time decision-making problem of multi-agent systems in highly dynamic adversarial environments. Furthermore, the expert system design in this patent is relatively complex, relying on the correlation between multiple tasks and offering weak support for initial strategy optimization in independent adversarial tasks. The solution proposed in patent CN115903903A has certain advantages in optimizing group collaboration, but its limitation lies in its failure to effectively solve the problem of acquiring the initial strategy of the agents. It relies on a cooperative game model between agents and cannot cope with adversarial tasks with incomplete information and extremely high real-time requirements. While the solution proposed in patent CN115525058A solves the problem of cooperative adversarial interaction between underwater vehicles, its support for initial strategy acquisition and the real-time response capability of the cluster adversarial game system is insufficient. Especially in large-scale clusters, the training convergence time of agents is long, making it unsuitable for the real-time requirements of large-scale complex adversarial scenarios. The solution proposed in patent CN117787444A effectively improves the management and scheduling capabilities of the algorithm, but it is mainly applicable to the algorithm integration and resource management level. It does not explore the optimization of adversarial decision-making in cluster adversarial game systems in depth and fails to effectively cope with the needs of multi-agent collaborative decision-making in dynamic and complex environments.

[0005] Through research and analysis of existing technical solutions in terms of initial strategy learning, real-time response capabilities, and strategy optimization in cluster adversarial game systems, the following technical bottlenecks were identified:

[0006] Initial policy learning problem: Cluster adversarial game systems typically need to start from scratch, gradually optimizing their policies through a large number of training samples and experience accumulation. Without prior knowledge, initial policy learning is usually inefficient, time-consuming, and slow in policy convergence. This problem is particularly prominent in multi-agent cooperative combat environments, where collaboration between agents relies on highly complex policy design. Real-time decision-making problem: For cluster adversarial game systems, especially in large-scale cluster environments, collaborative decision-making between agents needs to be completed within a very short time. Traditional deep reinforcement learning algorithms often fail to meet real-time requirements when handling large-scale cluster decision-making due to high computational complexity and slow convergence speed. Information incompleteness in cooperative combat: In complex and dynamic adversarial environments, the environmental information acquired by a single agent is usually local and incomplete. How to enable multiple agents to cooperate effectively and make optimal decisions under such circumstances is one of the problems that current technologies struggle to solve. Summary of the Invention

[0007] The purpose of this invention is to provide a swarm adversarial method and system based on expert knowledge-assisted deep reinforcement learning, which optimizes the agent's strategy in complex dynamic environments and improves collaborative combat capabilities.

[0008] In a first aspect, the present invention provides a cluster adversarial method based on expert knowledge-assisted deep reinforcement learning, the method comprising:

[0009] An adversarial model for an agent cluster is constructed based on a partially observable Markov decision process. The agent cluster includes multiple agents and multiple pre-protection points. The adversarial model includes a scenario model, an environment model, and a task model. Adversarial strategies of agents adversarially competing in the adversarial model based on preset rules are collected, and an expert knowledge base is formed based on these strategies. Adversarial strategies are collected from the expert knowledge base, and a constructed policy network is trained based on these strategies. The initial strategy of the adversarial agents is output based on the trained policy network. A selected deep reinforcement learning model is trained based on the initial strategy to obtain a trained deep reinforcement learning model. The optimal strategy of the adversarial agents is obtained based on the trained deep reinforcement learning model, and the optimal strategy guides the adversarial agents to perform adversarial operations.

[0010] In an optional implementation, the step of constructing the scene model includes:

[0011] Set the regional information of the adversarial zone, and set the number of agents in each faction and the number of pre-protection points; set the adversarial tasks of the agents in each faction based on the number of pre-protection points.

[0012] In an optional implementation, the step of constructing the environment model includes:

[0013] Set the environmental attributes of the adversarial environment, and set the characteristics of each agent and each pre-protection point; set the end condition for each adversarial round based on the number of pre-protection points.

[0014] In an optional implementation, the task model includes a state-space model, which is constructed in the following manner:

[0015] Based on the three-dimensional coordinates of each agent, the component velocities along each coordinate axis of the Cartesian coordinate system, and the health value, the state information of each agent is constructed; based on the three-dimensional coordinates of each pre-protection point and its health value, the state information of each pre-protection point is constructed; and a state space model is constructed based on the state information of each agent and the state information of each pre-protection point.

[0016] In an optional implementation, the task model includes an action space model, and the step of constructing the action space model includes:

[0017] A flight motion space model is constructed for each agent, which is based on the acceleration components of the agent in each coordinate axis direction in every two adjacent time steps; a functional motion space model is constructed for each agent, which is based on the functional attributes of different types of agents; and an action space model is obtained based on the flight motion space model and the functional motion space model.

[0018] In an optional implementation, the task model includes a state transition model, which is constructed in the following manner:

[0019] Health value transfer information for each agent is constructed, which is obtained based on the agent's current health value, the health value loss caused by agents from different factions, and the health value loss caused by agents from the same faction; position transfer information and velocity transfer information for each agent are constructed; and a state transition model is constructed based on the health value transfer information, position transfer information, and velocity transfer information.

[0020] In an optional implementation, the task model includes an observation space model, and the step of constructing the observation space model includes:

[0021] For each agent, obtain the state information of agents of the same camp, the state information of agents of different camps, and the state information of pre-protection points; construct an observation space model based on the state information of agents of the same camp, the state information of agents of different camps, and the state information of pre-protection points.

[0022] In an optional implementation, the task model includes a reward function model, which is constructed in the following manner:

[0023] For each agent, an individual reward function is constructed based on the agent's type; a group reward function is constructed based on the distance between agents of different factions and the pre-protection point, as well as the lowest health value of agents of the same faction; and a reward function model for the agent is constructed based on the individual reward function and the group reward function.

[0024] In an optional implementation, the adversarial strategy includes paired state information and action information; the step of training the constructed policy network based on the adversarial strategy includes:

[0025] The state information included in the adversarial strategy is input into the constructed policy network to obtain the action probability distribution output by the policy network; a cross-entropy loss function is constructed based on the action probability distribution output by the policy network and the action information included in the adversarial strategy, and the training of the policy network is guided by the cross-entropy loss function to minimize the cross-entropy loss function until a preset iteration stopping condition is met.

[0026] Secondly, the present invention provides a swarm adversarial system based on expert knowledge-assisted deep reinforcement learning, the system comprising:

[0027] The system comprises a construction module for building an adversarial model of an agent cluster based on a partially observable Markov decision process. The agent cluster includes multiple agents and multiple pre-protected points. The adversarial model includes a scenario model, an environment model, and a task model. A collection module is used to collect adversarial strategies of agents adversarially competing in the adversarial model based on preset rules, and to form an expert knowledge base based on these strategies. A training module is used to collect adversarial strategies from the expert knowledge base, train the constructed policy network based on these strategies, and output the initial strategy of the adversarial agents based on the trained policy network. The training module is also used to train a selected deep reinforcement learning model based on the initial strategy to obtain a trained deep reinforcement learning model, and to obtain the optimal strategy of the agents adversarially competing in the adversarial model based on the trained deep reinforcement learning model. The optimal strategy guides the adversarial agents to perform adversarial operations.

[0028] This application provides a swarm adversarial method and system based on expert knowledge-assisted deep reinforcement learning. It collects adversarial strategies to form an expert knowledge base through multiple swarm adversarial interactions between agents. Then, it trains agents based on these adversarial strategies and outputs initial strategies, enabling agents to quickly acquire basic adversarial strategies and significantly shortening training time. Compared to existing technologies, it achieves rapid acquisition of initial strategies and reduces the complexity of strategy learning. Furthermore, based on imitation learning, it selects an appropriate multi-agent deep reinforcement learning algorithm to address different adversarial scenarios. This allows for further optimization of agent strategies, ensuring their decision-making capabilities in both continuous and discrete action spaces, effectively addressing decision-making needs in situations with incomplete information, dynamic environmental changes, and large-scale swarm adversarial tasks. Moreover, when adapted to real-world scenarios, this solution can quickly generate optimal strategies based on the trained deep reinforcement learning model, guiding agents to execute adversarial operations and meeting high real-time requirements. Attached Figure Description

[0029] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments of the present invention will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0030] Figure 1 A flowchart illustrating a cluster adversarial method based on expert knowledge-assisted deep reinforcement learning provided in an embodiment of the present invention;

[0031] Figure 2 A schematic diagram illustrating expert knowledge-assisted deep reinforcement learning provided for an embodiment of the present invention;

[0032] Figure 3 A flowchart for policy network training provided in an embodiment of the present invention;

[0033] Figure 4 A schematic diagram illustrating the generation of the expert knowledge base provided in the embodiments of the present invention;

[0034] Figure 5 This is a schematic diagram illustrating policy optimization using the SQME-QPLEX deep reinforcement learning model, provided in an embodiment of the present invention.

[0035] Figure 6 A schematic diagram of the SQME-QPLEX deep reinforcement learning model framework provided in an embodiment of the present invention;

[0036] Figure 7 This is a functional block diagram of a cluster adversarial system based on expert knowledge-assisted deep reinforcement learning provided in an embodiment of the present invention.

[0037] Figure 8 This is a structural block diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0038] The technical solutions of the present invention will now be described with reference to the accompanying drawings in the embodiments of the present invention.

[0039] Please see Figure 1 This is a flowchart of a swarm adversarial method based on expert knowledge-assisted deep reinforcement learning provided in an embodiment of the present invention. This swarm adversarial method can be executed by a swarm adversarial system based on expert knowledge-assisted deep reinforcement learning. This system can be implemented in software and / or hardware and can be configured in an electronic device, such as a computer. The detailed steps of this swarm adversarial method based on expert knowledge-assisted deep reinforcement learning are described below.

[0040] S11, Construct an adversarial model for an agent cluster based on a partially observable Markov decision process. The agent cluster includes multiple agents and multiple pre-protection points. The adversarial model includes a scenario model, an environment model, and a task model.

[0041] S12, collect the adversarial strategies of the agents that engage in adversarial battles based on preset rules in the adversarial model, and form an expert knowledge base based on the adversarial strategies.

[0042] S13, collect adversarial strategies from the expert knowledge base, train the constructed policy network based on the adversarial strategies, and output the initial policy of the adversarial agent based on the trained policy network.

[0043] S14, the selected deep reinforcement learning model is trained based on the initial strategy to obtain the trained deep reinforcement learning model.

[0044] S15. Based on the trained deep reinforcement learning model, obtain the optimal strategy of the agent that is adversarial under the adversarial model, and guide the adversarial agent to perform adversarial operations based on the optimal strategy.

[0045] In this embodiment, the intelligent agent cluster includes multiple intelligent agents and multiple pre-protection points. The multiple intelligent agents belong to different factions and are of different types. Among them, the different factions include the red team and the blue team, and the different types include soft-kill type, reconnaissance type, and attack type.

[0046] Each intelligent agent can be such as a drone agent or an unmanned underwater vehicle agent, and can be applied to scenarios such as drone swarm collaborative operations, unmanned underwater vehicle swarm confrontation, and other multi-agent collaborative tasks. For example, drone swarms can perform interception, reconnaissance, and attack missions in complex battlefield environments. In marine environments, multiple unmanned underwater vehicles can coordinate reconnaissance and confrontation missions. Alternatively, it can be used in various mission scenarios requiring cooperation among multiple intelligent agents, such as collaborative logistics drone swarms and unmanned system collaboration in disaster relief.

[0047] During adversarial execution, intelligent agents adhere to relevant adversarial models, which include scenario models, environment models, and task models. The task model primarily includes models such as state space models, action space models, observation space models, state transition models, and reward function models. This can be understood as each agent determining action information according to a strategy based on local and observational information obtained from the state space and observation space models. This action information satisfies the action space model, and the agent exchanges this information with the environment to obtain a reward, which satisfies the reward function model. The environment then transitions to the next state according to the state transition model.

[0048] In other words, during the adversarial process, the intelligent agent performs the adversarial action according to the set task model under the set scenario model and environment model.

[0049] The construction of the scenario model is mainly achieved in the following ways: setting the regional information of the adversarial area, setting the number of agents of each camp and the number of pre-protection points; setting the adversarial tasks of agents of each camp based on the number of pre-protection points.

[0050] Specifically, the scene model is mainly used to describe a region of a specified size, within which there are N p (N p ≥0) fixed pre-protection points, N red (N red ≥1) Red Team (also known as Team A) agents and N blue (N blue ≥1) Blue Team (also known as B Team) faction agents.

[0051] When N p When N ≥ 0, A's objective is to protect the pre-protected points from being destroyed by B while simultaneously destroying all of B's ​​agents. B's objective is to protect B's agents from being completely destroyed by A while destroying at least one of A's pre-protected points. When N p When the value is 0, the objective of the adversarial mission for both sides A and B is symmetrical: to destroy all enemy agents while protecting one's own agents from being completely destroyed by the enemy agents. Both sides A and B have three types of agents to choose from: reconnaissance agents, attack agents, and soft-kill agents.

[0052] Furthermore, the environment model in the adversarial model can be constructed in the following way: set the environmental attributes of the adversarial environment, and set the characteristics of each agent and each pre-protection point; set the end condition for each adversarial round based on the number of pre-protection points.

[0053] In this embodiment, the environmental attributes of the adversarial environment include the length l, width w, and height v of the environmental region. Furthermore, the time interval between two time steps can be set to τ seconds, and the number of simulation steps in one round (adversarial round) can be κ. The pre-protection point is defined as follows (when N... p Characteristics when ≥0): For any pre-protected point p, its characteristics are defined as {x i ,y i ,z i ,h0}, where x i ∈[0,l],y i ∈[0,w],z i ∈[0,v] represents the three-dimensional coordinates of the pre-protection point, and h0 represents the initial health value of the pre-protection point.

[0054] In addition, the characteristics of each agent can be set in the following ways:

[0055] Set N blue Let B be the total number of agents. N represents the number of soft-kill agents, reconnaissance agents, and attack agents on side B, respectively. red Let A be the total number of intelligent agents. These represent the number of soft-kill agents, reconnaissance agents, and attack agents on side A, respectively. N is the total number of agents on both sides, satisfying: N = N blue +N red .

[0056] When setting the end conditions for each round, the following methods are mainly used:

[0057] When N p When N > 0, the game round ends when any one of the following three conditions is met: ① When any of A's pre-protected points is destroyed; ② When all of B's ​​soft-kill agents and attack agents are destroyed; ③ When the total duration reaches kτ seconds. p When = 0, the game round ends when any one of the following three conditions is met: ① When all of Player A's soft-kill agents and attack agents are destroyed; ② When all of Player B's soft-kill agents and attack agents are destroyed; ③ When the total duration reaches κτ seconds. Furthermore, the specific settings for the task model in the adversarial model will be described in detail later.

[0058] Please refer to the following: Figure 2 Building upon this, to ensure that each agent possesses a sound initial adversarial strategy, this embodiment first collects the adversarial strategies of agents engaging in adversarial combat according to preset rules within the adversarial model. These collected adversarial strategies can serve as expert knowledge to construct an expert knowledge base. The preset rules here refer to guiding rules governing their observations, actions, and states. In other words, it can be understood that the strategies of agents engaging in adversarial combat under these preset rules are superior. Specifically, a basic expert knowledge base can be formed by collecting the opponent's superior adversarial strategies through adversarial combat between one agent and an opponent based on preset rules and possessing basic decision-making capabilities for tasks such as interception, encirclement, and expulsion.

[0059] Building upon this foundation, adversarial strategies are collected from an expert knowledge base. Specifically, paired adversarial strategies, composed of state and action information, are acquired from the expert knowledge base. Using behavioral cloning technology and imitation learning, the constructed policy network is trained based on these acquired adversarial strategies, thereby enabling the policy network to acquire the agent's initial strategies for interception, encirclement, and expulsion.

[0060] In this embodiment, the algorithm library contains deep reinforcement learning models constructed using different deep reinforcement learning algorithms, including but not limited to multi-agent deep deterministic policy gradient algorithms, multi-agent proximal policy optimization algorithms, and population deep reinforcement learning algorithms, etc. (e.g.) Figure 2 Algorithms 1 to m are used in the implementation. During implementation, a suitable deep reinforcement learning model can be selected based on requirements. Building upon imitation learning, the selected deep reinforcement learning model is trained based on the obtained initial policy. The trained deep reinforcement learning model can then optimize the input initial policy. Through continuous interaction with the environment, the agents (Agent 1 to Agent N)... n ) Continuously receiving reward feedback (such as environmental information) Reward Information The strategy is iteratively optimized during the process, enabling the cluster adversarial game system to make optimal decisions in complex dynamic environments, surpassing the level of expert knowledge. This ensures its decision-making capabilities in both continuous and discrete action spaces.

[0061] To verify the effectiveness of the final adversarial strategy and the real-time performance of the agent system, this embodiment allows for multi-round simulated training of adversarial tasks under the adversarial model. During the simulation, the number of pre-protection points for both sides' agents (A and B) and for A can be freely adjusted according to the task. The agent strategy can be verified and optimized through multiple rounds of simulated adversarial tasks. Real-time verification of the adversarial simulation can be performed to ensure that the agent system can complete decisions and execute actions within seconds. Furthermore, the strategy network can be adjusted based on feedback data from the adversarial simulation to further improve the system's operational efficiency and mission success rate. Through multiple rounds of training, the agent system can effectively cope with complex and dynamic adversarial environments, ensuring the accuracy and real-time performance of decisions. Finally, the output of the trained deep reinforcement learning model can be transferred to other swarm adversarial environments to verify its generalization ability.

[0062] The swarm adversarial method based on expert knowledge-assisted deep reinforcement learning provided in this embodiment optimizes the collaborative decision-making efficiency of agents by introducing an expert knowledge base and imitation learning techniques, combined with deep reinforcement learning. This improves the initial policy learning speed and overall combat effectiveness of the system. The method aims to provide an effective initial policy acquisition mechanism, accelerate early policy learning of agents using an expert knowledge base, reduce training time, and optimize agent policies in complex dynamic environments through multi-agent deep reinforcement learning algorithms, thereby enhancing collaborative combat capabilities.

[0063] The following explains the specific methods for constructing the task model in the adversarial model. The task model includes the state space model, action space model, state transition model, observation space model, reward function model, etc.

[0064] In this embodiment, a multi-agent adversarial task is modeled using a partially observable Markov decision process. The agents can make action decisions based on local observations in an environment with incomplete information and gradually optimize their strategies through interaction with the environment.

[0065] For any agent g in any faction, its flight feature set is defined as:

[0066] F f ={color,id,type,d min ,v max ,a max ,ω max}#(1)

[0067] `color` represents the faction to which the agent belongs, with values ​​{red, blue}; `id` represents the unique ID of the agent, with values ​​{1, 2, ..., N}; `type` represents the type of the agent.

[0068]

[0069] d min Represents the minimum collision avoidance constant; v max Indicates the maximum speed of the agent; a max ω represents the maximum acceleration of the agent. max This represents the agent's maximum turning angular velocity. Its set of functional characteristics is defined as follows:

[0070]

[0071] d soft ,θ soft a represents the length and angle of the soft-kill range of the soft-kill agent. soft ∈(0,1) represents the health value loss per second for the agent; d find ,θ find The r represents the length and angle of the reconnaissance range of the reconnaissance agent. hard It is the strike radius of the attacking agent, a hard =1 indicates the health loss caused by a single attack.

[0072] Based on this, the steps to construct a state-space model can be implemented in the following ways:

[0073] Based on the three-dimensional coordinates of each agent, the component velocities along each coordinate axis of the Cartesian coordinate system, and the health value, the state information of each agent is constructed; based on the three-dimensional coordinates of each pre-protection point and its health value, the state information of each pre-protection point is constructed; and a state space model is constructed based on the state information of each agent and the state information of each pre-protection point.

[0074] For any agent g from either side, its state information is defined as follows:

[0075]

[0076] x t ∈[0,l]、y t ∈[0,w]、z t ∈[0,v], representing the three-dimensional coordinates of the target point; These represent the three-dimensional velocity components of the agent along each coordinate axis of the Cartesian coordinate system; h t ∈[0,1] represents the agent's current health value.

[0077] For any pre-protected point p, its state information is defined as follows:

[0078] s p ={x,y,z,h t}#(4)

[0079] x, y, z represent the three-dimensional coordinates of the pre-protection point; h t ∈[0,h0] represents the health value of the pre-protection point at this time.

[0080] Based on the above, the global state space model of the agent is as follows:

[0081]

[0082] Furthermore, the steps for constructing the action space model can be achieved as follows: constructing a flight action space model for each agent, which is based on the acceleration components of the agent in each coordinate axis direction in every two adjacent time steps; constructing a functional action space model for each agent, which is based on the functional attributes of different types of agents; and obtaining the action space model based on the flight action space model and the functional action space model.

[0083] Specifically, define It is the joint action space of all intelligent agents. For have:

[0084] A={A f A l}#(6)

[0085] Among them, A f The flight action space model of an intelligent agent is represented as follows:

[0086]

[0087] and Let x, y, and z represent the acceleration components of the agent in the x, y, and z coordinate axes between step t and step t+1, respectively, satisfying the following formula:

[0088]

[0089] A l The functional action space model representing an intelligent agent is defined as follows:

[0090]

[0091] c t Indicates whether the agent soft-kills a target in front, and is a boolean value; b t Indicates whether the agent attacks the target; is a boolean value, and satisfies:

[0092]

[0093] The above formula means that an attacking agent can only attack once per round.

[0094] In summary, the action space models for the three types of intelligent agents can be set as follows:

[0095]

[0096] In addition, the steps to construct the state transition model can be implemented in the following ways: construct the health value transition information of each agent, which is obtained by the agent's current health value, the health value loss caused by agents of different factions, and the health value loss caused by agents of the same faction; construct the position transition information and velocity transition information of each agent; and construct the state transition model based on the health value transition information, position transition information, and velocity transition information.

[0097] In this embodiment, the state transition function It can be represented as:

[0098]

[0099] The characteristics of S′ changing over time are analyzed. The distance function is defined as follows:

[0100]

[0101] Where x1, y1, z1, x2, y2, z2 represent the coordinate values ​​of the two agents on the three coordinate axes, respectively.

[0102] The damage distribution function of the attacking agent is:

[0103]

[0104] Then the health value of the k-th agent at step t is:

[0105]

[0106] in, It is the index of the set of all Blue Team attack agents.

[0107] Let the characteristic function δ(x) be:

[0108]

[0109] Let the clip function clip(x,a,b) be:

[0110]

[0111] When the i-th agent is not destroyed in step t+1, its velocity transfer information and position transfer information can be obtained by the following formulas:

[0112]

[0113] It should be noted that the speed must meet the maximum speed limit during the transfer process.

[0114] Simultaneously, the agent satisfies the maximum turning angular velocity limit during the transfer process, denoted as:

[0115]

[0116] Let the normalized outer product of the speeds before and after the turn be:

[0117]

[0118] but:

[0119]

[0120] Where K = 1 - cosα.

[0121] The health value transfer information of an agent can be constructed as follows:

[0122]

[0123] in, This represents the health loss caused by an attack from an enemy agent (an agent from a different faction). Assume this agent is from the red faction. Let the set of indices of all agents attacked by the blue side be represented as:

[0124]

[0125] This represents the health loss caused by friendly (or allied) attacks. Let the set of indices of all agents attacked by the Red side be represented as:

[0126]

[0127] The health loss caused by soft-kill attacks from enemy agents is denoted as:

[0128]

[0129] d=d(x it ,y it ,z it ,x jt ,y jt ,z jt )#(28)

[0130] The mean is 0 and the variance is σ. 2 The probability density function of a Gaussian distribution. It can be represented as:

[0131]

[0132] The health loss caused by friendly AI's soft-kill collateral damage can be represented as:

[0133]

[0134] The final state transition model of the agent consists of the aforementioned health value transition information, position transition information, and velocity transition information. Furthermore, the construction of the observation space model in the adversarial model can be achieved in the following ways:

[0135] For each agent, obtain the state information of agents of the same camp, agents of different camps, and the state information of pre-protected points; construct an observation space model based on the state information of agents of the same camp, agents of different camps, and the state information of pre-protected points.

[0136] Specifically, at time t, if at least one pre-protected point is destroyed, the game ends:

[0137]

[0138] When all of the Blue Team's attacking agents are destroyed:

[0139]

[0140] When the total duration reaches kτ seconds.

[0141] Suppose that at time t, the set of feature information of the agents still alive in the blue and red teams is:

[0142]

[0143] Here, 'c' represents the faction of the intelligent agent. This indicates that at time t, the c faction of the intelligent agent possesses type = i and The number of intelligent agents, This indicates that at time t, the c faction possesses... The total number of intelligent agents satisfies the following condition:

[0144]

[0145] For the red team's intelligent agent, its observation space model at time t can be represented as:

[0146]

[0147] This represents the state information of the red agent, namely:

[0148]

[0149] It is the set of indices of the surviving agents of the Red side, any have:

[0150]

[0151] This indicates that the red team has obtained the state information of the blue team's agent. The red team obtains the state information of the blue team's agent through the detection results of all types of agents, therefore it is denoted as:

[0152]

[0153] This represents the index of the ID of the state information of the blue team's agent obtained by all types of red team agents. For any... There exists i∈I red It is sufficient to satisfy one of the following equations:

[0154]

[0155] This indicates the status information of the pre-protected point, namely:

[0156]

[0157] In summary, the observation space model of the red agent at time t is as follows:

[0158]

[0159] The definition of state information and the construction of the observation space model of the blue agent are similar to those of the red agent, so they will not be described in detail here.

[0160] In this embodiment, the adversarial model also includes a reward function model, which can be constructed in the following way: for each agent, an individual reward function is constructed based on the type of the agent; a group reward function is constructed based on the distance between agents of different factions and the pre-protection point and the lowest health value of agents of the same faction; and the reward function model of the agent is constructed based on the individual reward function and the group reward function.

[0161] In this embodiment, the reward function can be a Reward function, which can be expressed as:

[0162]

[0163] If the state information of the agent at time t is s t The selected action is a t Then, the individual reward function for the three heterogeneous agents at time t is constructed as follows:

[0164] For soft-kill agents:

[0165]

[0166] in, This represents the set of enemy agents within the soft-kill range of this agent. This refers to the set of AI agents of the same faction within the soft-kill range of this AI agent. Indicates the relationship with g it The resulting loss of health points. d This is the reward constant for soft-killing one enemy agent.

[0167] For reconnaissance agents:

[0168]

[0169] in, Let r represent the set of enemy agents within the detection range of this agent at time t. s The reward constant for successfully detecting an enemy agent.

[0170] Regarding the attack on intelligent agents:

[0171]

[0172] in, Let represent the set of enemy agents within the attack range of this agent at time t. Let r be the health value lost by the i-th agent at time t. a The reward constant obtained for successfully attacking an enemy agent.

[0173] In addition to considering the individual reward function of the agents, it is also necessary to consider the team reward function that encourages cooperation among the agents. Let the team reward function r... team The structure is as follows:

[0174]

[0175] The average distance between all surviving Blue Team agents at time t and the nearest target point is defined as:

[0176]

[0177] r dh It is a distance constant; The lowest health value of the agent at time t is defined as follows:

[0178]

[0179] r h It is a health value constant; z t Let r be the time interval at time t, indicating whether the round has ended. If the round has ended, the value is 1; otherwise, it is 0. episode This is the final reward given at the end of the round. δ red Let δ be the characteristic function of the faction. If it is the red side, then δ red =1, then the blue side is δ red =-1.

[0180] The reward function model for each agent at time t is then as follows:

[0181]

[0182] Based on the above approach, an adversarial model can be specifically constructed for the intelligent agent cluster to rely on during the adversarial process. Under this adversarial model and guided by the optimal strategy obtained from the final trained deep reinforcement learning model, adversarial operations are performed.

[0183] In this embodiment, the deep reinforcement learning model is trained based on an initial policy, which is obtained by training the policy network with adversarial policies collected from an expert knowledge base. For details, please refer to the relevant documentation. Figure 3 The policy network constructed based on adversarial policy pairs can be implemented in the following ways:

[0184] S21, input the state information included in the adversarial strategy into the constructed policy network to obtain the action probability distribution output by the policy network.

[0185] S22, construct the cross-entropy loss function based on the action probability distribution output by the policy network and the action information included in the adversarial policy.

[0186] S23, the training of the policy network is guided by the cross-entropy loss function to minimize the cross-entropy loss function until the preset iteration stopping condition is met.

[0187] Please refer to the following: Figure 4 In this embodiment, the player's agent collects the opponent's strategy data by engaging in combat within a rule-based environment. The player can observe the opponent's agent's state and actions. For any state... Our own intelligent agent can observe the enemy intelligent agent U e Actions performed The collected adversarial strategies can form expert knowledge data and be stored in the expert knowledge base.

[0188] With the enemy's observation data and existing expert knowledge data, the agent is trained to learn from the expert knowledge data by minimizing the cross-entropy loss function. The method used is Behavior Cloning (BC) imitation learning.

[0189] Behavior cloning (BC) is a supervised machine learning method. The input is state information *s*, and the output is action information *a*. A policy network is trained through supervised learning, and the output of the policy network is the probability distribution π of different actions to take given states. θ (a|s) is used to make its output consistent with the expert's actions.

[0190] In this embodiment, the constructed cross-entropy loss function is:

[0191]

[0192] in:

[0193] θ is a parameter of the policy network;

[0194] π θ (a|s) is the probability distribution of the policy network outputting action a after being given state information s;

[0195] π E (a|s) is the action information about action a given state information s in the adversarial strategy;

[0196] D is the dataset (i.e., adversarial strategy data) in the expert knowledge base.

[0197] By minimizing the cross-entropy loss function mentioned above, the policy network can be trained to make its actions in various states as consistent as possible with the selection of expert knowledge data.

[0198] In this embodiment, through imitation learning based on an expert knowledge base, the agent can quickly acquire a high-quality initial policy in the initial stage. The initial policy learned in the first stage is then... As input for the second stage.

[0199] Please refer to the following: Figure 5 The appropriate algorithm is selected from multiple algorithms, and the deep reinforcement learning model under the algorithm is trained. During the training process, each agent can share information. The input is the observation information of the agent and the output is the action information of the agent.

[0200] Figure 5 Algorithm 1 is a sub-goal-based Q-learning framework (SQME-QPLEX) proposed in this invention. It combines a maximum entropy exploration strategy and is built upon dual-adversarial multi-agent Q-learning. This framework can address the challenges of homogeneous and heterogeneous adversarial games and comprises three functional modules: a state-conditional sub-goal generator, a structured agent encoder guided by sub-goals, and an ε-entropy action selection strategy. These modules collectively enhance the collaborative ability, adaptability, and exploration efficiency in large-scale partially observable environments. The SQME-QPLEX algorithm architecture is as follows: Figure 6 The diagram shows five key components: a hybrid adversarial network 601 for perceiving sub-targets, an ε-entropy action selection strategy 602, a centralized transformation module 603, a sub-target conditional structured agent network 604, and a sub-target generator 605. Under this configuration, the agent aims to maximize expected reward while coordinating with teammates and adversaries, under conditions of partially observable and decentralized execution.

[0201] To achieve cooperation under partially observable conditions, a sub-objective mechanism is introduced to simultaneously regulate representation learning and value estimation. At each time step t, the centralized sub-objective generator g... Φ 605 will observe the joint Encoded as a shared auxiliary information vector in This auxiliary information vector serves as the prior input for the decentralized strategy generation.

[0202] Each agent i receives a structured local observation, which consists of semantically distinct sub-modules:

[0203]

[0204] Each of these elements reflects a different aspect of the environment or agent configuration. To encode these heterogeneous inputs into feature representations with sub-target perception capabilities, a modular encoding scheme is employed. Specifically, and These represent local, task-independent features, which are processed through a standard multilayer perceptron (MLP) shared among all agents. and This includes interaction-related information, such as relative position or role, which requires context conditioning to achieve collaborative behavior. Therefore, these modules are encoded by a hypernetwork-regulated multilayer perceptron (HyperMLP), whose parameters are derived from the sub-target auxiliary information vector z. t Dynamically generated. The final embedding vectors of each module are concatenated into a unified sub-target perceptual feature representation:

[0205]

[0206] Among them, f self and f move It is a standard multilayer perceptron network (MLP), and f ally with f enemy It is made by z t HyperMLP with dynamically adjustable parameters.

[0207] Subsequently, this feature is represented Actions in the previous moment and the current sub-target z t Together, they are fed into a gated recurrent unit (GRU) to model temporal dependencies:

[0208]

[0209] Then, the Q-function of each agent is decomposed using an adversarial structure:

[0210]

[0211] Where f V It is a standard MLP, f A This is a HyperMLP with sub-target perception capabilities. The shared sub-target vector z t It participates in the feature extraction process and also influences the value function decomposition process, thereby providing a consistent global cooperative signal for all agents. The process is described in 604.

[0212] To promote efficient collaboration in heterogeneous multi-agent environments, a value decomposition framework for perceiving sub-goals is proposed. This framework introduces a global sub-goal adjustment mechanism based on the adversarial decomposition structure of QPLEX. Specifically, the global joint action-value function Q...tot (τ,a;z t ):

[0213] Q tot (τ,a;z t ) = V tot (τ;z t )+A tot (τ,a;z t )#(55)

[0214] in, This represents the sub-target vector generated by the centralized state.

[0215] Global value function part V tot It is the individual value function V that is adjusted through sub-goals. i (τ i The transformation parameters are obtained by performing a linear transformation and summing them, and are shared among all agents.

[0216] V i (τ i ;z t ) = w i (s t ,z t )·V i (τ i )+b i (s t ,z t )#(56)

[0217]

[0218] Among them, w i (·) and b i (·) depends on the sub-goal z t The transformation function is used to ensure the flexibility and monotonicity of individual values.

[0219] For the joint advantage function part A tot A multi-head attention mechanism is used to calculate the importance weight λ of the perceived sub-targets. i The format is as follows:

[0220]

[0221] Where each λ i The importance weight is >0 to ensure the monotonicity required to satisfy the consistency of greedy actions. This importance weight is based on the joint action-observation history and sub-objective z. t The calculations are performed on the above, thus giving the coordination strategy context-awareness and permutation sensitivity.

[0222] To enhance exploration capabilities in partially observable and adversarial uncertainties, an ε-entropy action selection strategy 602 is employed. This strategy combines entropy-regularized Q-values ​​with an ε-greedy sampling mechanism, striking a balance between guided randomness and undirected randomization to promote robust policy learning.

[0223] At each time step t, agent i is based on its local trajectory τ i and current sub-target z t Construct a sub-goal adjustment strategy, defined as follows:

[0224]

[0225] Where, α H >0 is a temperature coefficient, and annealing is performed during the training process.

[0226] The entropy of the induced strategy is defined as follows:

[0227]

[0228] Adding the entropy reward term to the Q value yields the regularized action value function:

[0229]

[0230] This regularization value encourages policy exploration in regions of high uncertainty by favoring policies with higher entropy values.

[0231] During training, an ε-greedy sampling mechanism is further applied based on the regularized Q-value:

[0232]

[0233] Where r follows a uniform distribution U(0,1), and ε undergoes stepwise annealing during training.

[0234] It is important to note that entropy regularization is only enabled during training to facilitate exploration; however, during the testing phase, the agent's action choices are deterministic.

[0235]

[0236] The test no longer includes entropy terms or ε random noise.

[0237] The SQME-QPLEX framework is implemented through end-to-end training. Its goal is to minimize a sub-objective-adjusted temporal difference (TD) loss applied to the joint action-value function. Given a transition tuple (τ,a,r,τ′,done), its TD objective is defined as follows:

[0238]

[0239] Among them, Q′ tot This represents a slowly updating target hybrid network. It is a sub-target vector generated from the next state.

[0240] Based on a mini-batch of transfer samples of size B, the TD loss function is defined as:

[0241]

[0242] in B represents the batch size.

[0243] The initial strategy provides a solid foundation for training the deep reinforcement learning model, enabling the agent, guided by the model, to perform well in adversarial environments and gradually improve its combat capabilities. During training, the value function of the agent performing adversarial simulations in dynamic adversarial environments is used as the objective function to guide the training. Training stops when the iterations converge or the maximum number of iterations is reached.

[0244] The cluster adversarial scheme based on expert knowledge-assisted deep reinforcement learning provided in this embodiment has at least the following advantages compared to existing technologies:

[0245] The efficiency of initial policy learning is improved. Traditional reinforcement learning methods typically require a large number of training samples to gradually optimize the agent's policy, resulting in slow convergence. This invention, by introducing an expert knowledge base, allows the agent to directly acquire the initial policy through imitation learning, significantly reducing the time required for policy learning.

[0246] This invention combines policy optimization with a deep reinforcement learning library. Building upon the initial policy generated through imitation learning, it integrates mainstream deep reinforcement learning algorithms into a library to further optimize the agent's collaborative decision-making capabilities. This combination enables the agent to optimize policies in both continuous and discrete action spaces under conditions of incomplete information and dynamic, complex environments, ensuring efficient collaboration between agents and enabling them to cope with diverse environments.

[0247] Real-time performance assurance in large-scale cluster warfare. Traditional cluster warfare systems often fail to meet real-time requirements when facing large-scale clusters due to high computational complexity. This invention, verified through a simulation platform, ensures that the decision response time of the agent in large-scale cluster warfare tasks does not exceed 1 second, thus meeting real-time requirements.

[0248] The efficiency of strategy optimization. Through multiple rounds of adversarial training on the simulation platform, the agent can continuously optimize its strategy based on feedback data, ensuring high adaptability in different adversarial scenarios and improving the success rate of tasks.

[0249] Based on the same inventive concept, please refer to Figure 7 This invention also provides a functional module diagram of a swarm adversarial system based on expert knowledge-assisted deep reinforcement learning. This embodiment can divide the swarm adversarial system based on expert knowledge-assisted deep reinforcement learning into functional modules according to the above method embodiments. For example, each function can be divided into its own functional modules, or two or more functions can be integrated into one processing module. The integrated module can be implemented in hardware or as a software functional module. It should be noted that the module division in this embodiment is illustrative and only represents one logical functional division; other division methods may be used in actual implementation.

[0250] For example, when dividing functional modules according to their respective functions, Figure 7 The swarm adversarial system based on expert knowledge-assisted deep reinforcement learning shown is only a schematic diagram of a device. This system may include a construction module, a collection module, and a training module. The functions of each module in this system will be described in detail below.

[0251] The system comprises a construction module for building an adversarial model of an agent cluster based on a partially observable Markov decision process. The agent cluster includes multiple agents and multiple pre-protected points. The adversarial model includes a scenario model, an environment model, and a task model. A collection module is used to collect adversarial strategies of agents adversarially competing in the adversarial model based on preset rules, and to form an expert knowledge base based on these strategies. A training module is used to collect adversarial strategies from the expert knowledge base, train the constructed policy network based on these strategies, and output the initial strategy of the adversarial agents based on the trained policy network. The training module is also used to train a selected deep reinforcement learning model based on the initial strategy to obtain a trained deep reinforcement learning model, and to obtain the optimal strategy of the agents adversarially competing in the adversarial model based on the trained deep reinforcement learning model. The optimal strategy guides the adversarial agents to perform adversarial operations.

[0252] It is understood that the above-mentioned construction module, collection module and training module can be used to execute S11 to S15. For detailed implementation of the construction module, collection module and training module, please refer to the relevant content of S11 to S15 above.

[0253] The swarm adversarial system based on expert knowledge-assisted deep reinforcement learning provided in this embodiment has the same, similar or corresponding technical features as the swarm adversarial method based on expert knowledge-assisted deep reinforcement learning in the above embodiments, and has the same technical effect. For relevant content about this system, please refer to the description of the above methods. This embodiment will not repeat it here.

[0254] Please see Figure 8 This is a structural block diagram of an electronic device provided in an embodiment of the present invention. The electronic device can be a computer device, etc., and includes a memory, a processor, and a communication module. The memory, processor, and communication module are electrically connected directly or indirectly to each other to achieve data transmission or interaction. For example, these components can be electrically connected to each other through one or more communication buses or signal lines.

[0255] The memory is used to store computer programs or data. The memory can be, but is not limited to, Random Access Memory (RAM), Read Only Memory (ROM), Programmable Read-Only Memory (PROM), Erasable Programmable Read-Only Memory (EPROM), and Electrically Erasable Programmable Read-Only Memory (EEPROM). The processor is used to read / write the data or programs stored in the memory and execute the swarm adversarial method based on expert knowledge-assisted deep reinforcement learning provided in any embodiment of this invention. The communication module is used to establish communication connections between the electronic device and other communication terminals via a network and to send and receive data via the network.

[0256] It should be understood that, Figure 8 The structure shown is only a schematic diagram of an electronic device; the electronic device may also include components that are larger than those shown. Figure 8 The more or fewer components shown, or having the same Figure 8 The different configurations shown.

[0257] Furthermore, this embodiment of the invention also provides a computer-readable storage medium storing machine-executable instructions. When these machine-executable instructions are executed, they implement the swarm adversarial method based on expert knowledge-assisted deep reinforcement learning provided in the above embodiments. Specifically, the computer-readable storage medium can be a general-purpose storage medium, such as a removable disk or hard disk. When the computer program on the computer-readable storage medium is run, it can execute the aforementioned swarm adversarial method based on expert knowledge-assisted deep reinforcement learning. The processes involved in the execution of the machine-readable storage medium and its executable instructions can be referred to the relevant descriptions in the above method embodiments, and will not be detailed here.

[0258] In the embodiments provided by this invention, it should be understood that the disclosed apparatus and method can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some communication interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0259] Furthermore, the units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs. Moreover, the functional modules in the various embodiments of the present invention can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0260] It should be noted that if the functionality is implemented as a software module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0261] The above description is merely an embodiment of the present invention and is not intended to limit the scope of protection of the present invention. For those skilled in the art, the present invention can have various modifications and variations. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A cluster adversarial method based on expert knowledge-assisted deep reinforcement learning, characterized in that, The method includes: An adversarial model for an agent cluster is constructed based on a partially observable Markov decision process. The agent cluster includes multiple agents and multiple pre-protection points. The adversarial model includes a scenario model, an environment model, and a task model. The agents are UAV agents or unmanned underwater vehicle agents. The adversarial strategies of the agents that engage in adversarial combat based on preset rules in the adversarial model are collected, and an expert knowledge base is formed based on the adversarial strategies. In the process of adversarial combat, the agents perform adversarial combat according to the set task model under the set scenario model and environment model. The preset rules refer to the guiding rules for their observation, actions and states. The better adversarial strategies of the opponent are collected by one agent against the opponent who is based on the preset rules and has the basic decision-making ability of interception, encirclement and expulsion tasks, and the expert knowledge base is formed. Adversarial strategies are collected from the expert knowledge base, and the constructed policy network is trained based on the adversarial strategies using imitation learning. The initial policy of the adversarial agent is then output based on the trained policy network. The selected deep reinforcement learning model is trained based on the initial strategy to obtain a trained deep reinforcement learning model. Based on the trained deep reinforcement learning model, the optimal strategy for the adversarial agent in the adversarial model is obtained. The optimal strategy guides the adversarial agent to perform adversarial operations. The deep reinforcement learning model includes deep reinforcement learning models constructed using different deep reinforcement learning algorithms, and the deep reinforcement learning model includes sub-objectives based on a maximum entropy exploration strategy. Learning framework The learning framework is built on dual adversarial multi-agent learning, including a sub-goal generator and -Entropy action selection strategy, the sub-target generator will combine observations Encoded as a shared auxiliary information vector ,in This auxiliary information vector serves as the prior input for the decentralized strategy generation; Each agent Receive a structured local observation, which consists of semantically distinct sub-modules: and Representing local, task-independent features, these are processed using a standard multilayer perceptron (MLP) shared among all agents. and Information related to the interaction, including relative position or role, is encoded by a HyperMLP multilayer perceptron regulated by a hypernetwork, the parameters of which are derived from the sub-target auxiliary information vector. Dynamically generated, the final embedding vectors of each module are concatenated into a unified sub-target perceptual feature representation: in, and It is a standard multilayer perceptron network (MLP). and It is by HyperMLP with dynamically adjustable parameters; Represent this feature Actions in the previous moment and current sub-goals Together, they are input into the gated recurrent unit (GRU) to model temporal dependencies: The Q-function of each agent is decomposed using an adversarial structure: in, It is a standard MLP. It is a HyperMLP with sub-target perception capabilities. Let i be the local trajectory of agent i; The Entropy action selection strategy combined with entropy regularization Value and Greedy sampling mechanism determines the actions of the agent.

2. The cluster adversarial method based on expert knowledge-assisted deep reinforcement learning according to claim 1, characterized in that, The steps for constructing the scene model include: Set the area information of the adversarial zone, and set the number of agents of each faction and the number of pre-protection points; The adversarial tasks for each faction's agents are set based on the number of pre-protected points.

3. The cluster adversarial method based on expert knowledge-assisted deep reinforcement learning according to claim 1, characterized in that, The steps for constructing the environment model include: Set the environmental attributes of the adversarial environment, and set the characteristics of each intelligent agent and the characteristics of each pre-protection point; The end condition for each round of the confrontation is set based on the number of pre-protected points.

4. The cluster adversarial method based on expert knowledge-assisted deep reinforcement learning according to claim 1, characterized in that, The task model includes a state-space model, which is constructed in the following way: Based on the three-dimensional coordinates of each intelligent agent, the component velocities in each coordinate axis direction of the Cartesian coordinate system, and the health value, the state information of each intelligent agent is constructed. Based on the three-dimensional coordinates and health values ​​of each pre-protection point, the status information of each pre-protection point is constructed. A state space model is constructed based on the state information of each intelligent agent and the state information of each pre-protection point.

5. The cluster adversarial method based on expert knowledge-assisted deep reinforcement learning according to claim 1, characterized in that, The task model includes an action space model, and the steps for constructing the action space model include: Construct a flight action space model for each agent, which is based on the acceleration components of the agent in each coordinate axis direction in every two adjacent time steps; Construct a functional action space model for each of the aforementioned intelligent agents, wherein the functional action space model is constructed based on the functional attributes of different types of intelligent agents; Based on the flight motion space model and the functional motion space model, the motion space model is obtained.

6. The cluster adversarial method based on expert knowledge-assisted deep reinforcement learning according to claim 1, characterized in that, The task model includes a state transition model, which is constructed in the following way: Construct health value transfer information for each agent, which is obtained based on the agent's current health value, the health value loss caused by agents from different factions, and the health value loss caused by agents from the same faction. Construct position transfer information and velocity transfer information for each of the aforementioned intelligent agents; A state transition model is constructed based on the health value transfer information, location transfer information, and velocity transfer information.

7. The cluster adversarial method based on expert knowledge-assisted deep reinforcement learning according to claim 1, characterized in that, The task model includes an observation space model, and the steps for constructing the observation space model include: For each agent, obtain the state information of agents of the same faction, the state information of agents of different factions, and the state information of pre-protected points; An observation space model is constructed based on the state information of agents from the same camp, the state information of agents from different camps, and the state information of pre-protected points.

8. The cluster adversarial method based on expert knowledge-assisted deep reinforcement learning according to claim 1, characterized in that, The task model includes a reward function model, which is constructed in the following way: For each agent, an individual reward function for that agent is constructed based on the agent's type; A group reward function is constructed based on the distance between agents of different factions and the pre-protection point, as well as the lowest health value of agents of the same faction. The reward function model of the agent is constructed based on the individual reward function and the group reward function.

9. The cluster adversarial method based on expert knowledge-assisted deep reinforcement learning according to claim 1, characterized in that, The adversarial strategy includes paired state information and action information; The step of training the constructed policy network based on the adversarial strategy includes: The state information included in the adversarial strategy is input into the constructed strategy network to obtain the action probability distribution output by the strategy network; A cross-entropy loss function is constructed based on the action probability distribution output by the policy network and the action information included in the adversarial strategy. The training of the policy network is then guided by the cross-entropy loss function to minimize the cross-entropy loss function until a preset iteration stopping condition is met.

10. A swarm adversarial system based on expert knowledge-assisted deep reinforcement learning, characterized in that, The system is used to implement the cluster adversarial method based on expert knowledge-assisted deep reinforcement learning as described in any one of claims 1-9, the system comprising: A construction module is used to build an adversarial model of an agent cluster based on a partially observable Markov decision process. The agent cluster includes multiple agents and multiple pre-protection points. The adversarial model includes a scenario model, an environment model, and a task model. The collection module is used to collect the adversarial strategies of the agents that are adversarial in the adversarial model based on preset rules, and to form an expert knowledge base based on the adversarial strategies. The training module is used to collect adversarial strategies from the expert knowledge base, train the constructed policy network based on the adversarial strategies, and output the initial policy of the adversarial agent based on the trained policy network. The training module is also used to train the selected deep reinforcement learning model based on the initial policy to obtain the trained deep reinforcement learning model, and to obtain the optimal policy of the agent that is adversarial under the adversarial model based on the trained deep reinforcement learning model, and to guide the adversarial agent to perform adversarial operations based on the optimal policy.

Images