Cluster confrontation method and system based on expert knowledge assisted deep reinforcement learning

Abstract

The invention provides a cluster confrontation method and system based on expert knowledge-assisted deep reinforcement learning, and the method and system improve the initial strategy learning speed and overall combat effectiveness of the system by introducing an expert knowledge base and an imitation learning technology and combining deep reinforcement learning to optimize the collaborative decision-making efficiency of an intelligent agent. The method aims at providing an effective initial strategy acquisition mechanism, utilizing an expert knowledge base to accelerate the early strategy learning of the agents, reducing the training time, optimizing the strategies of the agents in a complex dynamic environment through a multi-agent deep reinforcement learning algorithm, and improving the cooperative combat ability. According to the scheme, the time required for initial strategy learning can be greatly shortened, a more optimized strategy is obtained in combination with deep reinforcement learning, the high efficiency of strategy tuning is guaranteed, and then the real-time guarantee in large-scale cluster confrontation is guaranteed.

Description

Technical Field

[0001] The present invention relates to the technical field of deep reinforcement learning, and more particularly, to a cluster confrontation method and system based on expert knowledge-assisted deep reinforcement learning. Background Art

[0002] With the development of unmanned systems and multi-agent technologies, the application scenarios of multiple agents collaborating in combat are gradually increasing. For example, drone swarms, unmanned underwater vehicle swarms, etc. are widely used in military, surveillance, disaster relief and other scenarios. Such systems usually involve multiple autonomous agents performing collaborative tasks in a dynamic and complex environment, which poses higher requirements for the intelligence level of the cluster confrontation game system. Especially in the case of incomplete information, complex confrontation environment and high real-time requirements, how to achieve the optimal decision-making of confrontation tasks through effective cooperation between agents has become one of the core issues in the current research of cluster confrontation game systems.

[0003] Most of the current mainstream technologies rely on deep reinforcement learning and multi-agent reinforcement learning methods to optimize the decision-making behaviors of multi-agents in complex environments. Patent CN109657802B proposes a hybrid reinforcement learning method combining an expert system and deep learning, and uses a multi-head agent sharing network parameters as an expert for guidance, which improves the adaptability and learning efficiency of the system in multi-task scenarios. Patent CN115903903A proposes a reinforcement learning method based on public goods game for the collaborative combat of drone swarms, and improves the collaborative ability of drone swarms by introducing the optimization of public resources among multi-agents. Patent CN115525058A proposes an adversarial method for multi-agent unmanned underwater vehicles, mainly using deep reinforcement learning to optimize the collaborative decision-making strategy of the underwater vehicles. Patent CN117787444A proposes a method for quickly integrating different intelligent algorithms, which solves the efficiency problems caused by redundant code and poor algorithm collaboration during the integration process.

[0004] However, the technical solution proposed in Patent CN109657802B mainly focuses on transfer learning between single tasks and expert tasks, and fails to effectively solve the real-time decision-making problem of multi-agent in a highly dynamic adversarial environment. In addition, the expert system design in this patent is relatively complex, relying on the correlation between multi-tasks, and has weak support for the initial strategy optimization in independent adversarial tasks. The solution proposed in Patent CN115903903A has certain advantages in optimizing group cooperation, but its limitation lies in that it fails to effectively solve the problem of obtaining the initial strategy of agents, relying on the cooperative game model between agents, and cannot cope with adversarial tasks with incomplete information and extremely high real-time requirements. The solution proposed in Patent CN115525058A solves the problem of cooperative confrontation between submarines, but it has insufficient support for the acquisition of the initial strategy and the real-time response ability of the cluster confrontation game system. Especially in large-scale clusters, the training convergence time of agents is long, and it cannot meet the real-time requirements of large-scale complex confrontation scenarios. The solution proposed in Patent CN117787444A effectively improves the management and scheduling ability of the algorithm, but it is mainly applicable to the algorithm integration and resource management level, and does not deeply explore the optimization of the confrontation decision-making of the cluster confrontation game system, and fails to effectively respond to the multi-agent cooperative decision-making requirements in a dynamic and complex environment.

[0005] Through research and analysis of the existing technical solutions in aspects such as the initial strategy learning, real-time response ability, and strategy optimization of the cluster confrontation game system, it is found that the following several technical bottlenecks still exist in the existing solutions:

[0006] Initial strategy learning problem: The cluster confrontation game system usually needs to start from scratch and gradually optimize its strategy through a large number of training samples and experience accumulation. In the absence of prior knowledge, the learning of the initial strategy is usually inefficient, with a long training time and slow strategy convergence. This problem is particularly prominent in the multi-agent cooperative combat environment, where the cooperation between agents depends on a highly complex strategy design. Decision real-time problem: For the cluster confrontation game system, especially in a large-scale cluster environment, the cooperative decision-making between agents needs to be completed in a very short time. Traditional deep reinforcement learning algorithms often cannot meet the real-time requirements due to high computational complexity and slow convergence speed when dealing with large-scale cluster decisions. Information incompleteness in cooperative combat: In a complex and dynamic adversarial environment, the environmental information obtained by a single agent is usually local and incomplete. How to enable multi-agents to effectively cooperate and make optimal decisions in this situation is one of the problems difficult to solve by the existing technologies. Summary of the Invention

[0007] The purpose of the embodiments of the present invention is to provide a cluster confrontation method and system based on expert knowledge-assisted deep reinforcement learning to optimize the strategies of agents in a complex and dynamic environment and improve the cooperative combat ability.

[0008] In a first aspect, the present invention provides a cluster confrontation method based on expert knowledge-assisted deep reinforcement learning. The method includes:

[0009] Construct an adversarial model for an agent cluster based on a partially observable Markov decision process. The agent cluster includes multiple agents and multiple pre-protection points. The adversarial model includes a scenario model, an environment model, and a task model; collect the adversarial strategies of the agents that conduct confrontation based on preset rules in the adversarial model, and form an expert knowledge base based on the adversarial strategies; collect adversarial strategies from the expert knowledge base, train the constructed policy network based on the adversarial strategies, and output the initial strategies of the agents for confrontation based on the trained policy network; train a selected deep reinforcement learning model based on the initial strategies to obtain a trained deep reinforcement learning model, and obtain the optimal strategies of the agents for confrontation under the adversarial model based on the trained deep reinforcement learning model, and guide the agents for confrontation to execute confrontation operations based on the optimal strategies.

[0010] In an optional implementation manner, the steps of constructing the scenario model include:

[0011] Set the area information of the confrontation area, and set the number of agents in each camp and the number of pre-protection points; set the confrontation tasks of the agents in each camp based on the number of pre-protection points.

[0012] In an optional implementation manner, the steps of constructing the environment model include:

[0013] Set the environmental attributes of the confrontation environment, and set the characteristics of each agent and the characteristics of each pre-protection point; set the end condition for each confrontation round based on the number of pre-protection points.

[0014] In an optional implementation manner, the task model includes a state space model, and the state space model is constructed in the following manner:

[0015] Construct the state information of each agent based on the three-dimensional coordinates of each agent, the component velocities in the directions of each coordinate axis in a rectangular coordinate system, and the health value; construct the state information of each pre-protection point based on the three-dimensional coordinates of each pre-protection point and its health value; construct a state space model according to the state information of each agent and the state information of each pre-protection point.

[0016] In an optional implementation manner, the task model includes an action space model. The steps of constructing the action space model include:

[0017] Construct the flight action space model for each agent, where the flight action space model is constructed based on the acceleration components of the agent in each coordinate axis direction at every two adjacent time steps; construct the functional action space model for each agent, where the functional action space model is constructed based on the functional attributes of different types of agents; based on the flight action space model and the functional action space model, obtain the action space model.

[0018] In an alternative embodiment, the task model includes a state transition model, and the state transition model is constructed in the following manner:

[0019] Construct the health value transfer information for each agent, where the health value transfer information is obtained based on the current health value of the agent, the health value loss caused by agents in different camps, and the health value loss caused by agents in the same camp; construct the position transfer information and velocity transfer information for each agent; based on the health value transfer information, position transfer information, and velocity transfer information, construct the state transition model.

[0020] In an alternative embodiment, the task model includes an observation space model, and the steps for constructing the observation space model include:

[0021] For each agent, obtain the state information of agents in the same camp, the state information of agents in different camps, and the state information of the pre-protection point; based on the state information of agents in the same camp, the state information of agents in different camps, and the state information of the pre-protection point, construct the observation space model.

[0022] In an alternative embodiment, the task model includes a reward function model, and the reward function model is constructed in the following manner:

[0023] For each agent, construct the individual reward function of the agent based on the type of the agent; construct the group reward function based on the distance between agents in different camps and the pre-protection point and the minimum health value of agents in the same camp; based on the individual reward function and the group reward function, construct the reward function model of the agent.

[0024] In an alternative embodiment, the adversarial strategy includes paired state information and action information; the steps for training the policy network based on the adversarial strategy pair include:

[0025] Input the state information included in the adversarial strategy into the constructed policy network to obtain the action probability distribution output by the policy network; construct a cross-entropy loss function based on the action probability distribution output by the policy network and the action information included in the adversarial strategy, and use the cross-entropy loss function to guide the training of the policy network to minimize the cross-entropy loss function until a preset iteration stop condition is satisfied.

[0026] In a second aspect, the present invention provides a cluster adversarial system based on expert knowledge-assisted deep reinforcement learning, the system comprising:

[0027] A construction module, configured to construct an adversarial model of an agent cluster based on a partially observable Markov decision process, the agent cluster comprising a plurality of agents and a plurality of pre-protection points, the adversarial model comprising a scenario model, an environment model, and a task model; a collection module, configured to collect the adversarial strategies of the agents that perform adversarial actions based on preset rules in the adversarial model, and form an expert knowledge base based on the adversarial strategies; a training module, configured to collect adversarial strategies from the expert knowledge base, and train the constructed policy network based on the adversarial strategies, and output an initial strategy of the agents that perform adversarial actions based on the trained policy network; the training module is further configured to train a selected deep reinforcement learning model based on the initial strategy to obtain a trained deep reinforcement learning model, and obtain an optimal strategy of the agents that perform adversarial actions under the adversarial model based on the trained deep reinforcement learning model, and guide the agents that perform adversarial actions to execute adversarial operations based on the optimal strategy.

[0028] The cluster adversarial method and system based on expert knowledge-assisted deep reinforcement learning provided in this application collect adversarial strategies through multiple cluster adversarial actions among agents to form an expert knowledge base, and then train based on the adversarial strategies in the expert knowledge base and output an initial strategy, enabling agents to quickly obtain basic adversarial strategies and significantly shortening the training time. Compared with the prior art, it can achieve the rapid acquisition of the initial strategy and reduce the complexity of strategy learning. In addition, based on imitation learning, a suitable multi-agent deep reinforcement learning algorithm is selected to cope with different adversarial scenarios. In this way, the strategy of the agent can be further optimized to ensure its decision-making ability in continuous and discrete action spaces, and it can effectively meet the decision-making requirements in information-incomplete, dynamically changing environments, and large-scale cluster adversarial tasks. Moreover, when adapted to the actual scenario, the optimal strategy can be quickly generated based on the trained deep reinforcement learning model, so as to guide the agent to execute adversarial operations and meet the high real-time requirements. Description of the Drawings

[0029] To more clearly illustrate the technical solutions of the embodiments of the present invention, the following will briefly introduce the drawings required for use in the embodiments of the present invention. It should be understood that the following drawings only show certain embodiments of the present invention and should not be regarded as limiting the scope. For those of ordinary skill in the art, without creative efforts, other related drawings can also be obtained based on these drawings.

[0030] Figure 1 Flowchart of the cluster confrontation method based on expert knowledge-assisted deep reinforcement learning provided by the embodiments of the present invention;

[0031] Figure 2 Schematic diagram of the expert knowledge-assisted deep reinforcement learning provided by the embodiments of the present invention;

[0032] Figure 3 Flowchart of the training of the policy network provided by the embodiments of the present invention;

[0033] Figure 4 Schematic diagram of the generation of the expert knowledge base provided by the embodiments of the present invention;

[0034] Figure 5 Schematic diagram of policy optimization using the SQME-QPLEX deep reinforcement learning model provided by the embodiments of the present invention;

[0035] Figure 6 Schematic diagram of the framework of the SQME-QPLEX deep reinforcement learning model provided by the embodiments of the present invention;

[0036] Figure 7 Block diagram of the functional modules of the cluster confrontation system based on expert knowledge-assisted deep reinforcement learning provided by the embodiments of the present invention;

[0037] Figure 8 Block diagram of the structure of the electronic device provided by the embodiments of the present invention. Detailed implementation manners

[0038] The following will describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention.

[0039] Please refer to Figure 1 , flowchart of the cluster confrontation method based on expert knowledge-assisted deep reinforcement learning provided by the embodiments of the present invention. This cluster confrontation method based on expert knowledge-assisted deep reinforcement learning can be executed by a cluster confrontation system based on expert knowledge-assisted deep reinforcement learning. This cluster confrontation system based on expert knowledge-assisted deep reinforcement learning can be implemented by software and / or hardware and can be configured in an electronic device, which can be a computer device. The detailed steps of this cluster confrontation method based on expert knowledge-assisted deep reinforcement learning are introduced as follows.

[0040] S11. Construct an adversarial model for an intelligent agent cluster based on a partially observable Markov decision process. The intelligent agent cluster includes multiple intelligent agents and multiple pre-protection points. The adversarial model includes a scenario model, an environment model, and a task model.

[0041] S12. Collect the adversarial strategies of the intelligent agents that conduct adversarial actions based on preset rules in the adversarial model, and form an expert knowledge base based on the adversarial strategies.

[0042] S13. Extract adversarial strategies from the expert knowledge base, train the constructed policy network based on the adversarial strategies, and output the initial strategies of the adversarial intelligent agents based on the trained policy network.

[0043] S14. Train the selected deep reinforcement learning model based on the initial strategies to obtain a trained deep reinforcement learning model.

[0044] S15. Obtain the optimal strategies of the intelligent agents that conduct adversarial actions under the adversarial model based on the trained deep reinforcement learning model, and guide the adversarial intelligent agents to execute adversarial operations based on the optimal strategies.

[0045] In this embodiment, the intelligent agent cluster includes multiple intelligent agents and multiple pre-protection points. The multiple intelligent agents include intelligent agents belonging to different camps and different types. Among them, different camps include the red camp and the blue camp, and different types include, for example, soft kill type, reconnaissance type, and strike type.

[0046] Each intelligent agent can be, for example, an unmanned aerial vehicle (UAV) intelligent agent, an unmanned underwater vehicle (UUV) intelligent agent, etc., and can be applied to scenarios such as UAV cluster cooperative operations, UUV cluster confrontation, and other multi-intelligent agent cooperative tasks. For example, UAV clusters perform tasks such as interception, reconnaissance, and attack in complex battlefield environments. In the marine environment, cooperative reconnaissance and confrontation tasks among multiple UUVs. Or used in various task scenarios that require cooperation among multiple intelligent agents, such as logistics UAV cluster cooperation, unmanned system cooperation in disaster relief, etc.

[0047] During the process of conducting adversarial actions, the intelligent agents satisfy the relevant adversarial model, which includes a scenario model, an environment model, and a task model. The task model mainly includes models such as a state space model, an action space model, an observation space model, a state transition model, and a reward function model. It can be understood that each intelligent agent obtains local information and observation information according to the state space model and the observation space model, determines action information according to a certain strategy, the action information satisfies the action space model, exchanges with the environment to obtain the reward given by the environment, the reward satisfies the reward function model, and the environment transfers to the next state according to the state transition model.

[0048] That is, during the adversarial process executed by the agent, under the set scenario model and environment model, the agent executes the adversarial process according to the set task model.

[0049] Among them, the scenario model is mainly constructed in the following way: setting the area information of the adversarial area, and setting the number of agents in each camp and the number of pre-protection points; setting the adversarial tasks of the agents in each camp based on the number of pre-protection points.

[0050] Specifically, the scenario model is mainly used to describe a region of a specified size, in which there are N p (N p ≥0) fixed pre-protection points, N red (N red ≥1) red-side (also called Party A) camp agents and N blue (N blue ≥1) blue-side (also called Party B) camp agents.

[0051] When N p ≥0, the goal of the adversarial task of Party A is to protect the pre-protection points from being destroyed by Party B and at the same time destroy all Party B agents. The goal of the adversarial task of Party B is to destroy at least 1 pre-protection point of Party A on the premise of protecting Party B agents from being completely destroyed by Party A. When N p =0, the goals of the adversarial tasks of both Party A and Party B are symmetric, that is, to destroy all the enemy agents on the premise of protecting their own agents from being completely destroyed by the enemy agents. Among them, both Party A and Party B have three types of agents to choose from: reconnaissance agents, strike agents, and soft-kill agents.

[0052] In addition, the environment model in the adversarial model can be constructed in the following way: setting the environmental attributes of the adversarial environment, and setting the characteristics of each agent and the characteristics of each pre-protection point; setting the end condition of each adversarial round based on the number of pre-protection points.

[0053] In this embodiment, the environmental attributes of the adversarial environment include the length l, width w, height v, etc. of the environmental area. In addition, the time interval between two time steps can be set to τ seconds, and the number of simulation steps in one round (adversarial round) is κ. The characteristics of the pre-protection points (when N p ≥0) are defined in the following way: for any pre-protection point p, its characteristics are defined as {x i , y i , z i , h0}, where x i ∈[0, l], y i ∈[0, w], z i ∈[0, v], representing the three-dimensional coordinates of the pre-protection point, and h0 represents the initial health value of the pre-protection point.

[0054] In addition, the characteristics of each agent are set as follows:

[0055] Set N blue as the total number of agents on Party B's side, which are the numbers of Party B's soft kill agents, reconnaissance agents, and strike agents respectively. N red is the total number of agents on Party A's side, which are the numbers of Party A's soft kill agents, reconnaissance agents, and strike agents respectively. N is the total number of agents on both sides, satisfying: N = N blue + N red .

[0056] When setting the end condition of each round, it is mainly achieved through the following methods:

[0057] When N p > 0, when any one of the following three conditions is met, the game round ends: ① When any one of Party A's pre-protected points is destroyed; ② When all of Party B's soft kill agents and strike agents are destroyed; ③ When the total duration reaches kτ seconds. When N p = 0, when any one of the following three conditions is met, the game round ends: ① When all of Party A's soft kill agents and strike agents are destroyed; ② When all of Party B's soft kill agents and strike agents are destroyed; ③ When the total duration reaches κτ seconds. In addition, the specific setting method of the task model in the confrontation model will be described in detail later.

[0058] Please refer to Figure 2 . On this basis, in order to enable each agent to have a better initial confrontation strategy, in this embodiment, the confrontation strategies of the agents that conduct confrontations according to preset rules in the confrontation model can be collected first. The collected confrontation strategies can be used as expert knowledge to form an expert knowledge base. The preset rules here can refer to the guiding rules for their observations, actions, states, etc. That is to say, it can be understood that the strategies of the agents that conduct confrontations under the preset rules are relatively optimal. Specifically, the relatively optimal confrontation strategies of the opponent can be collected by having one side's agents confront an opponent that has the basic decision-making capabilities of tasks such as interception, encirclement, and expulsion based on the preset rules to form a basic expert knowledge base.

[0059] On this basis, the confrontation strategies are collected from the expert knowledge base. Specifically, the paired confrontation strategies composed of state information - action information are collected from the expert knowledge base. The behavior cloning technology can be used to train the constructed policy network in an imitation learning manner based on the collected confrontation strategies, so that the policy network has the ability to obtain the initial strategies of the agents for interception, encirclement, expulsion, etc.

[0060] In this embodiment, the algorithm library has deep reinforcement learning models constructed using different deep reinforcement learning algorithms, including but not limited to multi-agent deep deterministic policy gradient algorithm, multi-agent proximal policy optimization algorithm, population deep reinforcement learning algorithm, etc. (such as Figure 2 Algorithms 1 to m in n ). During implementation, an appropriate deep reinforcement learning model can be selected based on requirements. On the basis of imitation learning, the selected deep reinforcement learning model is trained based on the obtained initial policy, and the input initial policy can be optimized based on the trained deep reinforcement learning model. Through continuous interaction with the environment, the agents (Agent 1 to Agent N Reward information ) iteratively optimize the policy during the process of continuously receiving reward feedback (such as environmental information

[0061] ), so that the cluster confrontation game system can make optimal decisions in a complex dynamic environment, exceeding the level of expert knowledge. Ensure its decision-making ability in continuous and discrete action spaces.

[0062] In this embodiment, the cluster confrontation method based on expert knowledge-assisted deep reinforcement learning optimizes the collaborative decision-making efficiency of agents by introducing an expert knowledge base and imitation learning technology, combined with deep reinforcement learning, thereby improving the initial policy learning speed and overall combat effectiveness of the system. The purpose of this method is to provide an effective initial policy acquisition mechanism, use the expert knowledge base to accelerate the early policy learning of agents, reduce the training time, and at the same time optimize the policies of agents in complex dynamic environments through multi-agent deep reinforcement learning algorithms to improve the collaborative combat ability.

[0063] The following describes the specific method for constructing the task model in the confrontation model. Among them, the task model includes a state space model, an action space model, a state transition model, an observation space model, a reward function model, etc.

[0064] In this embodiment, the adversarial tasks of multiple agents are modeled through a partially observable Markov decision process. The agents can make action decisions based on local observations in an environment with incomplete information and gradually optimize their strategies through interactions with the environment.

[0065] For an agent g in any camp, define its flight feature set as:

[0066] F f = {color, id, type, d min , v max , a max , ω max} #(1)

[0067] color represents the camp to which the agent belongs, and the value set is {red, blue}; id represents the unique number of the agent, and the value set is {1, 2,..., N}; type represents the type of the agent:

[0068]

[0069] d min represents the minimum collision avoidance constant; v max represents the maximum speed of the agent; a max represents the maximum acceleration of the agent; ω max represents the maximum turning angular velocity of the agent. Define its functional feature set as:

[0070]

[0071] d soft , θ soft represents the length and angle of the soft kill range of the soft kill agent, a soft ∈(0, 1) represents the health value loss caused to the agent per second; d find , θ find represents the length and angle of the reconnaissance range of the reconnaissance agent; r hard is the strike radius of the strike agent, a hard = 1, representing the health value loss caused by a single strike.

[0072] On this basis, the steps of constructing the state space model can be specifically implemented in the following ways:

[0073] Based on the three-dimensional coordinates of each agent, the component velocities in the directions of each coordinate axis in the rectangular coordinate system, and the health value, construct the state information of each agent; based on the three-dimensional coordinates of each pre-protection point and its health value, construct the state information of each pre-protection point; construct the state space model according to the state information of each agent and the state information of each pre-protection point.

[0074] For any agent \(g\) in both camps, its state information is defined as:

[0075]

[0076] x t \(\in [0, l]\), y t \(\in [0, w]\), z t \(\in [0, v]\), representing the three - dimensional coordinates of the target point; Respectively represent the three - dimensional velocity components of the agent in each axis direction of the rectangular coordinate system; h t \(\in [0, 1]\), representing the health value of the agent at this time.

[0077] For any pre - protected point \(p\), its state information is defined as:

[0078] s p \(=\{x, y, z, h\) t \}\#(4)

[0079] x, y, z represent the three - dimensional coordinates of the pre - protected point; h t \(\in [0, h0]\), representing the health value of the pre - protected point at this time.

[0080] Based on the above, the global state - space model of the agent is:

[0081]

[0082] In addition, the steps to construct the action - space model can be achieved in the following way: construct the flight - action space model of each agent, where the flight - action space model is constructed based on the acceleration components of the agent in each axis direction at every two adjacent time steps; construct the functional - action space model of each agent, where the functional - action space model is constructed based on the functional attributes of different types of agents; based on the flight - action space model and the functional - action space model, obtain the action - space model.

[0083] Specifically, define as the joint - action space of all agents. For there is:

[0084] A = \{A f , A l \}\#(6)

[0085] Among them, A f represents the flight - action space model of the agent, defined as:

[0086]

[0087] and respectively represent the acceleration components of the agent in the x, y, and z axis directions between the t-th step and the (t + 1)-th step, and satisfy the following formula:

[0088]

[0089] A l represents the functional action space model of the agent, and its definition is as follows:

[0090]

[0091] c t represents whether the agent soft-kills the target in front, and is a bool-type data; b t represents whether the agent strikes the target, and is a bool-type data, and satisfies:

[0092]

[0093] The above formula means that a striking agent can only strike once in a round.

[0094] In summary, the action space models of the three types of agents can be set as follows:

[0095]

[0096] In addition, the steps to construct the state transition model can be achieved in the following way: construct the health value transfer information of each agent, which is obtained from the current health value of the agent, the health value loss caused by agents in different camps, and the health value loss caused by agents in the same camp; construct the position transfer information and speed transfer information of each agent; construct the state transition model based on the health value transfer information, position transfer information, and speed transfer information.

[0097] In this embodiment, the state transition function can be expressed as:

[0098]

[0099] Analyze the time-varying characteristics in S'. Among them, the distance function is defined as follows:

[0100]

[0101] Among them, x1, y1, z1, x2, y2, z2 respectively represent the coordinate values of two agents on the three coordinate axes.

[0102] The strike damage distribution function of the striking agent is:

[0103]

[0104] The health value of the k-th agent at the t-th step is as follows:

[0105]

[0106] where, is the subscript of the set of all blue-side attacking agents.

[0107] The indicator function δ(x) is defined as:

[0108]

[0109] The truncation function clip(x,a,b) is defined as:

[0110]

[0111] When the i-th agent is not destroyed at the (t + 1)-th step, its speed transfer information and position transfer information can be obtained by the following formulas:

[0112]

[0113] It should be noted that the speed needs to satisfy the maximum speed limit during the transfer process.

[0114] At the same time, the agent satisfies the limit of the maximum turning angular velocity during the transfer process. Denote the turning angle as:

[0115]

[0116] Denote the outer product of the normalized speeds before and after turning as:

[0117]

[0118] Then:

[0119]

[0120] where, K = 1 - cosα.

[0121] The health value transfer information of the agent can be constructed as follows:

[0122]

[0123] where, represents the health value loss caused by the attack of the enemy agents (agents of different camps). Assume that this agent is a red-side camp agent, and represents the set of subscripts of all blue-side attacking agents, then it is expressed as:

[0124]

[0125] Denote the health value loss caused by friendly agent (agents of the same camp) friendly fire as Denote the set of all the subscripts of the red side's striking agents, which is denoted as:

[0126]

[0127] Denote the health value loss caused by the soft kill of enemy agents, denoted as:

[0128]

[0129] d = d(x it , y it , z it , x jt , y jt , z jt )#(28)

[0130] is the probability density function of a Gaussian distribution with mean 0 and variance σ 2 . Then can be expressed as:

[0131]

[0132] Denote the health value loss caused by the soft kill friendly fire of friendly agents, which can be expressed as:

[0133]

[0134] Then the final state transition model of the agent is composed of the above health value transition information, position transition information, and velocity transition information. In addition, the construction of the observation space model in the confrontation model can be achieved through the following methods:

[0135] For each agent, obtain the state information of agents in the same camp, the state information of agents in different camps, and the state information of the pre-protection points; construct the observation space model based on the state information of agents in the same camp, the state information of agents in different camps, and the state information of the pre-protection points.

[0136] Specifically, at time t, if at least one pre-protection point is destroyed, the game ends:

[0137]

[0138] When all the blue side's striking agents are destroyed:

[0139]

[0140] When the total duration reaches kτ seconds.

[0141] Suppose that at time t, the set of feature information of the agents still alive in the blue and red camps is:

[0142]

[0143] where c represents the camp of the agent, represents the number of agents with type = i in camp c at time t and the number of agents, represents the total number of agents with in camp c at time t, satisfying the condition:

[0144]

[0145] For the agents in the red camp, the observation space model at the t-th moment can be expressed as:

[0146]

[0147] represents the state information of the red agents, that is:

[0148]

[0149] is the set of subscripts of the red agents that are still alive. For any there is:

[0150]

[0151] represents the state information of the blue agents obtained by the red agents. The channels through which the red agents obtain the state information of the blue agents include the detection results of all types of agents. Therefore, it is denoted as:

[0152]

[0153] represents the id subscript of the state information of the blue agents obtained by all types of red agents. For any there exists i ∈ I red such that one of the following equations is satisfied:

[0154]

[0155] represents the state information of the pre-protection point, that is:

[0156]

[0157] In summary, the observation space model of the red - side agent at time t is as follows:

[0158]

[0159] The definition of the state information of the blue - side agent and the construction of the observation space model are similar to those of the red - side agent, so they will not be elaborated here.

[0160] In this embodiment, the adversarial model further includes a reward function model, which can be constructed in the following way: for each agent, an individual reward function of the agent is constructed based on the type of the agent; a group reward function is constructed based on the distance between agents of different camps and the pre - protected point and the minimum health value of agents of the same camp; the reward function model of the agent is constructed according to the individual reward function and the group reward function.

[0161] In this embodiment, the reward function can be the Reward function, and the Reward function can be expressed as:

[0162]

[0163] If the state information of the agent at time t is s t , and the action selected is a t , then the individual reward functions for the three heterogeneous agents at time t are constructed as follows:

[0164] For the soft - kill agent:

[0165]

[0166] Among them, represents the set of enemy agents within the soft - kill range of this agent, represents the set of agents of the same camp within the soft - kill range of this agent, represents the health value loss caused to g it . r d is the reward constant for soft - killing one enemy agent.

[0167] For the reconnaissance agent:

[0168]

[0169] Among them, represents the set of enemy agents within the reconnaissance range of this agent at time t, and r s is the reward constant for successfully detecting one enemy agent.

[0170] For the strike agent:

[0171]

[0172] Among them, represents the set of enemy agents within the strike range of the agent at time t, is the health value lost by the i-th agent at time t, r a is the reward constant obtained for successfully striking an enemy agent.

[0173] In addition to considering the individual reward function of the agent, it is also necessary to consider the team reward function that promotes agent cooperation. The team reward function r team is constructed as follows:

[0174]

[0175] represents the average distance between all surviving strike agents of the blue side and the nearest target point at time t, defined as:

[0176]

[0177] r dh is the distance constant; is the lowest health value of the agent at time t, defined as follows:

[0178]

[0179] r h is the health value constant; z t is whether the round ends at time t. If the round ends, it is 1, otherwise it is 0, r episode is the final reward given at the end of the round. δ red is the indicator function of the camp. If it is the red side, then δ red = 1, and for the blue side, δ red = -1.

[0180] Then the reward function model of each agent at time t is as follows:

[0181]

[0182] Based on the above method, the confrontation model relied on by the agent cluster during the confrontation process can be specifically constructed. Under this confrontation model and guided by the optimal strategy obtained from the finally trained deep reinforcement learning model, confrontation operations are performed.

[0183] In this embodiment, the deep reinforcement learning model is trained based on the initial strategy, and this initial strategy is obtained by training the policy network with confrontation strategies collected from the expert knowledge base. Specifically, please refer to Figure 3 . The policy network constructed based on the confrontation strategies can be implemented in the following way:

[0184] S21. Input the state information included in the adversarial strategy into the constructed policy network to obtain the action probability distribution output by the policy network.

[0185] S22. Construct a cross-entropy loss function based on the action probability distribution output by the policy network and the action information included in the adversarial strategy.

[0186] S23. Use the cross-entropy loss function to guide the training of the policy network to minimize the cross-entropy loss function until the preset iteration stop condition is met.

[0187] Please refer to Figure 4 In this embodiment, the own agent collects the opponent's strategy data by engaging in battles with the opponent in a rule-based environment. The own side can observe the states and actions of the opponent agent. For any state the own agent can observe the actions e executed by the opponent agent U The collected adversarial strategies can form expert knowledge data and be stored in the expert knowledge base.

[0188] With the opponent's observation data and the existing expert knowledge data, the agent is trained to imitate and learn the expert knowledge data by minimizing the cross-entropy loss function. The method adopted is imitation learning of behavior cloning (BC).

[0189] Behavior cloning (BC) is a supervised machine learning method. The input is the state information s, and the output is the action information a. The policy network is trained through supervised learning, and the output of the policy network is the action probability distribution π θ (a|s) of taking different actions under a given state, making its output consistent with the actions of the expert.

[0190] In this embodiment, the constructed cross-entropy loss function is:

[0191]

[0192] Where:

[0193] θ is the parameter of the policy network;

[0194] π θ (a|s) is the action probability distribution of the policy network outputting action a given the state information s;

[0195] π E (a|s) is the action information about action a given the state information s in the adversarial strategy;

[0196] D is the data set (i.e., adversarial strategy data) in the expert knowledge base.

[0197] By minimizing the above cross-entropy loss function, the training of the policy network can be completed, enabling the actions selected in various states to be as consistent as possible with those of the expert knowledge data.

[0198] In this embodiment, through imitation learning based on the expert knowledge base, the intelligent agent can quickly obtain a high-quality initial policy in the initial stage. And the initial policy learned in the first stage is used as the input for the second stage.

[0199] Please refer to Figure 5 , select a suitable algorithm from multiple algorithms to train the deep reinforcement learning model under this algorithm. Each intelligent agent can share information during the training process. The input is the observation information of the intelligent agent, and the output is the action information of the intelligent agent.

[0200] Figure 5 In, Algorithm 1 is a subgoal-based Q-learning framework (SQME-QPLEX) proposed by the present invention. Combining the maximum entropy exploration strategy, it is constructed on the basis of double adversarial multi-agent Q-learning. This framework can handle challenges in homogeneous and heterogeneous adversarial games and includes three functional modules: a state-conditioned subgoal generator, a structured agent encoder with subgoal guidance, and an ε-entropy action selection strategy. These modules together improve the collaboration ability, adaptability, and exploration efficiency in large-scale partially observable environments. The SQME-QPLEX algorithm architecture is as Figure 6 shown and includes five key components: an adversarial hybrid network 601 for perceiving subgoals, an ε-entropy action selection strategy 602, a centralized transformation module 603, a subgoal-conditioned structured agent network 604, and a subgoal generator 605. Under this setting, the intelligent agent aims to maximize the expected return while coordinating with teammates and competing against opponents under the conditions of partial observability and decentralized execution.

[0201] To achieve cooperation under partially observable conditions, a subgoal mechanism is introduced to simultaneously adjust representation learning and value estimation. At each time step t, the centralized subgoal generator g Φ 605 encodes the observation joint into a shared auxiliary information vector where this auxiliary information vector serves as the prior input for decentralized policy generation.

[0202] Each intelligent agent i receives a structured local observation, which consists of semantically different sub-modules:

[0203]

[0204] Among them, each item respectively reflects different aspects of the environment or agent configuration. To encode these heterogeneous inputs into a feature representation with sub-goal awareness, a modular encoding scheme is adopted. Specifically, and represent local, task-independent features, which are processed by a standard multi-layer perceptron (MLP) shared among all agents. While and contain interaction-related information, such as relative position or role, and this information requires context modulation to achieve collaborative behavior. Therefore, these modules are encoded by a multi-layer perceptron regulated by a hypernetwork (HyperMLP), whose parameters are dynamically generated by the sub-goal auxiliary information vector z t Finally, the embedded vectors of each module are concatenated into a unified sub-goal aware feature representation:

[0205]

[0206] where f self and f move are standard multi-layer perceptron networks (MLP), while f ally and f enemy are HyperMLPs whose parameters are dynamically regulated by z t .

[0207] Subsequently, this feature representation is input together with the action at the previous moment and the current sub-goal z t into a gated recurrent unit (GRU) for modeling temporal dependencies:

[0208]

[0209] Then, the Q-function for each agent is decomposed using an adversarial structure:

[0210]

[0211] where f V is a standard MLP, and f A is a HyperMLP with sub-goal awareness. The shared sub-goal vector z t is involved in both the feature extraction process and affects the value function decomposition process, thus providing a consistent global collaboration signal for all agents. The process is shown in 604.

[0212] To promote efficient collaboration in heterogeneous multi-agent environments, a sub-goal aware value decomposition framework is proposed. This framework introduces a global sub-goal regulation mechanism based on the adversarial decomposition structure of QPLEX. Specifically, the global joint action-value function Qtot (τ, a; z t ):

[0213] Q tot (τ, a; z t ) = V tot (τ; z t ) + A tot (τ, a; z t ) #(55)

[0214] Wherein, represents the sub-goal vector generated from the centralized state.

[0215] The global value function part V tot is obtained by linearly transforming and summing the individual value function V i (τ i ), and the transformation parameters are shared among all agents:

[0216] V i (τ i ; z t ) = w i (s t , z t ) · V i (τ i ) + b i (s t , z t ) #(56)

[0217]

[0218] Wherein, w i (·) and b i (·) are transformation functions dependent on the sub-goal z t to ensure the flexibility and monotonicity of the individual value.

[0219] For the joint advantage function part A tot , the multi-head attention mechanism is adopted to calculate the importance weight λ i of the perceived sub-goal, and the form is as follows:

[0220]

[0221] Wherein, each λ i > 0 to ensure the monotonicity required to satisfy the greedy action consistency. The importance weight is calculated on the joint action-observation history and the sub-goal z t so that the coordination strategy has context awareness and permutation sensitivity.

[0222] To enhance the exploration ability in partially observable and adversarial uncertainty environments, an ε-entropy action selection strategy 602 is adopted. This strategy combines the entropy-regularized Q-value with the ε-greedy sampling mechanism, achieving a balance between guided randomness and un-directed randomization and promoting robust policy learning.

[0223] At each time step t, agent i constructs a sub-goal-conditioned policy based on its local trajectory τ i and the current sub-goal z t as defined below:

[0224]

[0225] where α H > 0 is a temperature coefficient that is annealed during training.

[0226] The entropy of the induced policy is defined as follows:

[0227]

[0228] An entropy reward term is added to the Q-value to obtain the regularized action-value function:

[0229]

[0230] This regularized value encourages policy exploration in high-uncertainty regions by preferring policies with higher entropy values.

[0231] During training, based on the regularized Q-value, the ε-greedy sampling mechanism is further applied:

[0232]

[0233] where r follows a uniform distribution U(0,1) and ε is annealed gradually during training.

[0234] Note that entropy regularization is only enabled during training to promote exploration; while in the test phase, the agent's action selection is deterministic:

[0235]

[0236] The entropy term or ε random noise is not included during testing.

[0237] The SQME-QPLEX framework is implemented through end-to-end training, whose goal is to minimize a sub-goal-conditioned temporal difference (TD) loss acting on the joint action-value function. Given a transition tuple (τ, a, r, τ′, done), its TD target is defined as follows:

[0238]

[0239] Among them, Q' tot represents a slowly updated target mixing network, which is a sub-goal vector generated from the next state.

[0240] Based on a mini-batch transition sample of size B, its TD loss function is defined as:

[0241]

[0242] where B represents the batch size.

[0243] The initial policy provides a good foundation for the training of the deep reinforcement learning model, enabling the agent ultimately guided by the deep reinforcement learning model to perform well in the adversarial environment and gradually improve its combat level. During the training process of the deep reinforcement learning model, the value function when the agent executes the adversarial simulation in the dynamic adversarial environment is used as the objective function to guide the training, and finally the training stops when the iteration reaches convergence or the maximum number of iterations is reached.

[0244] The cluster confrontation scheme based on expert knowledge-assisted deep reinforcement learning provided by this embodiment has at least the following advantages compared with the prior art:

[0245] Improved learning efficiency of the initial policy. Traditional reinforcement learning methods usually require a large number of training samples to gradually optimize the agent's policy, with a slow convergence speed. By introducing the expert knowledge base in the present invention, the agent can directly obtain the initial policy through imitation learning, greatly shortening the time required for policy learning.

[0246] Combined with the policy optimization of the deep reinforcement learning library. On the basis of generating the initial policy through imitation learning, the present invention combines mainstream deep reinforcement learning algorithms to form an algorithm library, further optimizing the collaborative decision-making ability of the agent. This combination enables the agent to optimize the policy in continuous and discrete action spaces in the case of incomplete information and dynamic and complex environments, ensuring the efficient cooperation between agents and enabling it to cope with different environments.

[0247] Guarantee of real-time performance in large-scale cluster confrontation. Traditional cluster confrontation game systems often cannot meet the real-time requirements due to high computational complexity when facing large-scale clusters. The present invention is verified through the simulation platform to ensure that the decision response time of the agent in large-scale cluster confrontation tasks does not exceed 1 second, meeting the real-time requirements.

[0248] High efficiency of policy tuning. Through multiple rounds of adversarial training on the simulation platform, the agent can continuously optimize the policy according to the feedback data, ensuring efficient adaptability in different adversarial scenarios and increasing the task success rate.

[0249] Based on the same inventive concept, please refer to Figure 7 , an embodiment of the present invention further provides a schematic diagram of functional modules of a cluster confrontation system based on expert knowledge-assisted deep reinforcement learning. In this embodiment, the functional modules of the cluster confrontation system based on expert knowledge-assisted deep reinforcement learning can be divided according to the above method embodiment. For example, corresponding functional modules can be divided for each function, or two or more functions can be integrated into one processing module. The above integrated module can be implemented in the form of hardware or in the form of a software functional module. It should be noted that the division of modules in the embodiment of the present invention is illustrative, only a logical function division, and there may be other division methods in actual implementation.

[0250] For example, in the case of dividing corresponding functional modules for each function, Figure 7 the shown cluster confrontation system based on expert knowledge-assisted deep reinforcement learning is only a schematic diagram of a device. The cluster confrontation system based on expert knowledge-assisted deep reinforcement learning may include a construction module, a collection module, and a training module. The functions of each functional module of the cluster confrontation system based on expert knowledge-assisted deep reinforcement learning will be elaborated in detail below.

[0251] The construction module is used to construct an adversarial model of an intelligent agent cluster based on the partially observable Markov decision process. The intelligent agent cluster includes multiple intelligent agents and multiple pre-protection points. The adversarial model includes a scenario model, an environment model, and a task model. The collection module is used to collect the adversarial strategies of the intelligent agents that conduct confrontation based on preset rules in the adversarial model, and form an expert knowledge base based on the adversarial strategies. The training module is used to collect adversarial strategies from the expert knowledge base, train the constructed policy network based on the adversarial strategies, and output the initial strategies of the intelligent agents that conduct confrontation based on the trained policy network. The training module is further used to train the selected deep reinforcement learning model based on the initial strategies, obtain a trained deep reinforcement learning model, and obtain the optimal strategies of the intelligent agents that conduct confrontation under the adversarial model based on the trained deep reinforcement learning model, and guide the intelligent agents that conduct confrontation to execute confrontation operations based on the optimal strategies.

[0252] It can be understood that the above construction module, collection module, and training module can be used to execute the above S11 to S15. The detailed implementation manners of the construction module, collection module, and training module can refer to the content related to the above S11 to S15.

[0253] The cluster confrontation system based on expert knowledge-assisted deep reinforcement learning provided in this embodiment has the same, similar or corresponding technical features as the cluster confrontation method based on expert knowledge-assisted deep reinforcement learning in the above embodiment, and has the same technical effects. For the relevant content of this system, reference can be made to the description related to the above method, and this embodiment will not be elaborated here.

[0254] Please refer to Figure 8 , which is a structural block diagram of an electronic device provided by an embodiment of the present invention. The electronic device can be a computer device, etc. The electronic device includes a memory, a processor, and a communication module. Each element of the memory, the processor, and the communication module is directly or indirectly electrically connected to each other to achieve data transmission or interaction. For example, these elements can be electrically connected to each other through one or more communication buses or signal lines.

[0255] Among them, the memory is used to store computer programs or data. The memory can be, but is not limited to, a random access memory (RAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), etc. The processor is used to read / write the data or programs stored in the memory and execute the cluster confrontation method based on expert knowledge-assisted deep reinforcement learning provided in any embodiment of the present invention. The communication module is used to establish a communication connection between the electronic device and other communication terminals through a network and is used to transmit and receive data through the network.

[0256] It should be understood that Figure 8 The structure shown is only a schematic diagram of the structure of the electronic device. The electronic device may also include more or fewer components than Figure 8 shown, or have a different configuration from Figure 8 shown. [[ID=ig]]

[0257] Furthermore, an embodiment of the present invention also provides a computer-readable storage medium storing machine-executable instructions, which, when executed, implement the cluster confrontation method based on expert knowledge-assisted deep reinforcement learning provided in the above embodiment. Specifically, the computer-readable storage medium can be a general storage medium, such as a removable disk, a hard disk, etc. When the computer program on the computer-readable storage medium runs, it can execute the above-mentioned cluster confrontation method based on expert knowledge-assisted deep reinforcement learning. Regarding the process involved when the machine-executable instructions in the computer-readable storage medium run, reference can be made to the relevant descriptions in the above method embodiments and will not be elaborated here.

[0258] In the embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of the units is only a logical function division, and there may be other division methods in actual implementation. For another example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed mutual coupling or direct coupling or communication connection can be through some communication interfaces. The indirect coupling or communication connection of the devices or units can be in an electrical, mechanical or other form.

[0259] In addition, the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they can be located in one place or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment. Furthermore, the functional modules in each embodiment of the present invention can be integrated together to form an independent part, or each module can exist alone, or two or more modules can be integrated to form an independent part.

[0260] It should be noted that if a function is implemented in the form of a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or a part of this technical solution can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: various media that can store program codes such as USB flash drives, mobile hard disks, read-only memories (ROM), random access memories (RAM), magnetic disks, or optical discs.

[0261] The above are only embodiments of the present invention and are not intended to limit the protection scope of the present invention. For those skilled in the art, the present invention can have various changes and modifications. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A cluster confrontation method based on expert knowledge-assisted deep reinforcement learning, characterized in that, The method includes: Constructing an adversarial model for an agent cluster based on a partially observable Markov decision process, where the agent cluster includes multiple agents and multiple pre-protection points, and the adversarial model includes a scenario model, an environment model, and a task model; Collecting the adversarial strategies of the agents that conduct adversarial operations based on preset rules in the adversarial model, and forming an expert knowledge base based on the adversarial strategies; Collecting adversarial strategies from the expert knowledge base, training the constructed policy network based on the adversarial strategies, and outputting the initial strategies of the agents for adversarial operations based on the trained policy network; Training the selected deep reinforcement learning model based on the initial strategies to obtain a trained deep reinforcement learning model, obtaining the optimal strategies of the agents for adversarial operations under the adversarial model based on the trained deep reinforcement learning model, and guiding the agents for adversarial operations to execute adversarial operations based on the optimal strategies.

2. The cluster confrontation method based on expert knowledge-assisted deep reinforcement learning according to claim 1, characterized in that The steps of constructing the scenario model include: Setting the regional information of the adversarial region, and setting the number of agents in each camp and the number of pre-protection points; Setting the adversarial tasks of the agents in each camp based on the number of pre-protection points.

3. The cluster confrontation method based on expert knowledge-assisted deep reinforcement learning according to claim 1, characterized in that The steps of constructing the environment model include: Setting the environmental attributes of the adversarial environment, and setting the characteristics of each agent and the characteristics of each pre-protection point; Setting the end condition for each adversarial round based on the number of pre-protection points.

4. The cluster confrontation method based on expert knowledge-assisted deep reinforcement learning according to claim 1, characterized in that The task model includes a state space model, and the state space model is constructed as follows: Constructing the state information of each agent based on the three-dimensional coordinates of each agent, the velocity components in the directions of each coordinate axis in the rectangular coordinate system, and the health value; Constructing the state information of each pre-protection point based on the three-dimensional coordinates of each pre-protection point and its health value; Constructing the state space model according to the state information of each agent and the state information of each pre-protection point.

5. The cluster confrontation method based on expert knowledge-assisted deep reinforcement learning according to claim 1, characterized in that The task model includes an action space model, and the steps of constructing the action space model include: Constructing a flight action space model for each agent, where the flight action space model is constructed based on the acceleration components of the agent in the directions of each coordinate axis at every two adjacent time steps; Constructing a functional action space model for each agent, where the functional action space model is constructed based on the functional attributes of different types of agents; Obtaining the action space model based on the flight action space model and the functional action space model.

6. The cluster confrontation method based on expert knowledge-assisted deep reinforcement learning according to claim 1, characterized in that The task model includes a state transition model, and the state transition model is constructed as follows: Constructing the health value transfer information of each agent, where the health value transfer information is obtained based on the current health value of the agent, the health value loss caused by agents in different camps, and the health value loss caused by agents in the same camp; Constructing the position transfer information and velocity transfer information of each agent; Constructing the state transition model based on the health value transfer information, position transfer information, and velocity transfer information.

7. The cluster confrontation method based on expert knowledge-assisted deep reinforcement learning according to claim 1, characterized in that The task model includes an observation space model, and the steps of constructing the observation space model include: For each agent, obtain the status information of agents in the same camp, the status information of agents in different camps, and the status information of pre-protection points; Construct an observation space model based on the status information of agents in the same camp, the status information of agents in different camps, and the status information of pre-protection points.

8. The cluster confrontation method based on expert knowledge-assisted deep reinforcement learning according to claim 1, characterized in that, The task model includes a reward function model, and the reward function model is constructed in the following way: For each agent, construct an individual reward function of the agent based on the type of the agent; Construct a group reward function based on the distance between agents in different camps and pre-protection points and the minimum health value of agents in the same camp; Construct the reward function model of the agent according to the individual reward function and the group reward function.

9. The cluster confrontation method based on expert knowledge-assisted deep reinforcement learning according to claim 1, characterized in that The adversarial strategy includes paired status information and action information; The step of training the constructed policy network based on the adversarial strategy includes: Input the status information included in the adversarial strategy into the constructed policy network to obtain the action probability distribution output by the policy network; Construct a cross-entropy loss function based on the action probability distribution output by the policy network and the action information included in the adversarial strategy, and use the cross-entropy loss function to guide the training of the policy network to minimize the cross-entropy loss function until a preset iteration stop condition is met.

10. A cluster confrontation system based on expert knowledge-assisted deep reinforcement learning, characterized in that, The system includes: A construction module for constructing an adversarial model of an agent cluster based on a partially observable Markov decision process. The agent cluster includes multiple agents and multiple pre-protection points, and the adversarial model includes a scenario model, an environment model, and a task model; A collection module for collecting the adversarial strategies of agents that conduct adversarial operations based on preset rules in the adversarial model, and forming an expert knowledge base based on the adversarial strategies; A training module for collecting adversarial strategies from the expert knowledge base, training the constructed policy network based on the adversarial strategies, and outputting the initial strategies of the agents that conduct adversarial operations based on the trained policy network; The training module is further configured to train a selected deep reinforcement learning model based on the initial strategy to obtain a trained deep reinforcement learning model, obtain the optimal strategies of the agents that conduct adversarial operations under the adversarial model based on the trained deep reinforcement learning model, and guide the agents that conduct adversarial operations to perform adversarial operations based on the optimal strategies.

Images