Red and blue game confrontation-oriented constrained collaborative multi-agent reinforcement learning method

Abstract

The invention relates to a red-blue game confrontation-oriented constrained collaborative multi-agent reinforcement learning method, which comprises the following steps of: constructing a virtual game scene and a scenario confrontation blue target, and decomposing a game confrontation task into a target collaborative allocation problem and a maneuvering strategy generation problem; respectively establishing a Markov game process of a target collaborative allocation problem and a Markov game process of a maneuvering strategy generation problem; establishing a constraint collaborative multi-agent reinforcement learning model used for solving a Markov game process of a target collaborative allocation problem, and inputting game situation information to obtain a target allocation scheme; and establishing a constraint collaborative experience-first multi-agent reinforcement learning model of a Markov game process for solving a maneuvering strategy generation problem, and inputting game situation information and maneuvering state information to obtain an optimal path of an attack target. The method can improve the utilization rate of the training sample, reduce the training time, improve the sample efficiency, and enhance the environment learning ability of the intelligent agent.

Description

Technical Field

[0001] The present invention belongs to the field of reinforcement learning technology and proposes a constrained collaborative multi-agent reinforcement learning method for red-blue game confrontation. It mainly involves the modeling and construction of the simulation environment for game confrontation tasks, as well as the design of multi-agent reinforcement learning methods. Background Art

[0002] Game confrontation is a theory and technology that studies the strategic interaction between multiple intelligent agents in a competitive environment. It is widely used in fields such as network security and artificial intelligence. Its core goal is to analyze and optimize the strategies of the two (or multiple) adversarial parties in a dynamic environment, so that one party can gain an advantage in the game or find an equilibrium point.

[0003] With the rapid development of information technology, red-blue adversarial game technology has spawned diverse applications across many fields. In cybersecurity, adversarial game technology can simulate the game between attackers and defenders in areas such as intrusion detection and software defense. In economics and finance, adversarial game technology can simulate adversarial strategies in competitive markets, such as high-frequency trading and risk hedging. Game adversarial game technology primarily encompasses different types, including zero-sum games, non-zero-sum games, and mixed cooperative-adversarial games. Commonly used methods include game theory analysis, reinforcement learning, and deep learning.

[0004] Game theory analysis, the most classic equilibrium-solving method, uses methods such as Nash equilibrium and minimax strategies. While effective for small-scale problems, it suffers from high computational complexity, an inability to adapt to dynamic environments, and a lack of autonomous learning in complex adversarial environments. Specifically, traditional game theory is computationally expensive to find equilibrium points in high-dimensional state spaces, making it difficult to scale to large-scale adversarial scenarios. In such adversarial games, finding mixed-strategy Nash equilibria is often an NP-hard problem. Furthermore, classical game theory typically assumes a static or known environment, making it difficult to adapt to the dynamics of the blue team's strategy. In a typical red-blue adversarial scenario, the opponent's strategy constantly adjusts, making fixed equilibrium strategies difficult to adapt to. Finally, traditional game theory relies on mathematical modeling and cannot automatically learn optimal strategies from the environment, making it difficult to efficiently infer optimal countermeasures in environments with incomplete information.

[0005] Single-agent reinforcement learning (SRL) learns optimal strategies through environmental interaction, addressing the shortcomings of game theory analysis. However, in adversarial environments, it still faces challenges such as difficulty coping with multi-agent interactions, insufficient stability, and difficulty balancing collaboration and adversarial forces. Specifically, SRL assumes a static environment or is influenced only by external randomness. In adversarial games, however, the opponent's strategy is constantly changing. In typical multi-agent game environments, SRL cannot effectively model the strategy evolution of other agents. Furthermore, due to the opponent's constantly changing behavior, SRL is prone to policy instability during training. For example, in an adaptive adversary environment, a single agent may learn conflicting strategies in different training cycles. In mixed game environments (including both competition and cooperation), SRL cannot effectively learn collaborative mechanisms among multiple agents. Summary of the Invention

[0006] In order to solve the above technical problems existing in the prior art, the present invention proposes a constrained collaborative multi-agent reinforcement learning method for red-blue game confrontation, comprising the following steps:

[0007] Step 1: Model the Red Agent collaborative confrontation problem, including constructing a virtual game scenario and assuming a confrontation with the Blue Agent. This task is then decomposed into a target collaborative allocation problem and a maneuver strategy generation problem.

[0008] Step 2: Establish the Markov game process for the target collaborative allocation problem and the Markov game process for the maneuver strategy generation problem respectively;

[0009] Step 3: Under the red-blue game situation, a constrained collaborative multi-agent reinforcement learning model for solving the target collaborative allocation problem is established for the Markov game process. The game situation information is input to obtain the target allocation solution.

[0010] Step 4: Establish a constrained collaborative experience-first multi-agent reinforcement learning model for the Markov game process for solving the maneuver strategy generation problem. Train the constrained collaborative experience-first multi-agent reinforcement learning model according to the target allocation scheme obtained in step 3, input game situation information and maneuver state information, and obtain the optimal path to attack the target.

[0011] Furthermore, step 1 specifically includes:

[0012] The target collaborative allocation problem is defined as:

[0013]

[0014] Where n represents the number of blue targets; m represents the number of red agents; v i Indicates the value of the blue team's goal; pij represents the probability of the red agent j confronting the blue target i; b ij Indicates whether the red agent j is assigned to the blue target i. If so, then b ij =1, otherwise, b ij =0;

[0015] Define the maneuver strategy generation problem, including:

[0016] Given a red agent set L, which contains a finite number of red agents L i ∈L(i=1,…,N L ), for any three-dimensional path node t, the red agent L i The spatial position of Blue team target T j The spatial position of All the offensive tasks to be executed are grouped into a task set U. Each task U in the set U i ∈U(i=1,…,N L ), U i The result is that for each red agent L i The three-dimensional attack path Tr corresponding to the blue team's target planning k (1≤k≤n),

[0017]

[0018] Where, represents the initial position of the red agent, Represents the three-dimensional path nodes that the red agent passes through in sequence, Indicates the final position of the red agent, that is, the target space position of the blue agent;

[0019] The coordinate system is established with the center of mass of the red agent as the origin, and the kinematic equation of the red agent i is:

[0020]

[0021] Among them, v i represents the speed of the red agent, γ i represents the angle between the velocity direction and the XOY plane, ψ i Represents the angle between the velocity direction and the X-axis;

[0022] The dynamic equation of the red agent i is:

[0023]

[0024] in, Represents the directional control overload of the red agent, and g is the acceleration due to gravity.

[0025] Furthermore, the Markov game process for establishing the target collaborative allocation problem in step 2 includes:

[0026] Given m blue target points, the i-th target point is represented by its two-dimensional coordinates The threat level and the remaining integrity level are composed of the threat level of the i-th target point, including the blue side's defense level Number and scale of blue team targets Importance of blue team's target Game task objectives The remaining completeness is determined by Indicates that the state of the target point i at time t is:

[0027]

[0028] At time t, the observation matrix received by the red agent i from the environment is:

[0029]

[0030] in, for Medium Elements other than , stack(·) represents the row stacking operation, the action a of the red agent i i =j is defined as a confrontation with target j, and all agents share a joint confrontation reward, which is proportional to the sum of the changes in the blue agent's health before and after the confrontation:

[0031]

[0032] in, ρ is a parameter;

[0033] The Markov game process for modeling the maneuver strategy generation problem includes:

[0034] Add the speed information of the Red Team agent based on the coordinate state of the simulation entity Define the Red Agent State and the blue team intercepting the agent state At time t, the observation matrix received by the red agent i from the environment is:

[0035]

[0036] Among them, stack(·) represents the row stacking operation, is the coordinate of the target point;

[0037] Define the cooperative game confrontation action set: Define the red agent action set as a continuous action of direction angle adjustment:

[0038]

[0039] The joint actions of the multi-agent group are:

[0040] The reward function for the collaborative game scenario is defined as:

[0041]

[0042] in, Represents the cumulative distance between each red agent and its target point;

[0043] The maneuver shielding mechanism for the threat area at time t is established as follows:

[0044]

[0045] in, Represents taking action a at the current time t t The state that the agent reaches later, D((x T ,y T ),r) represents the threat area, (x T ,y T ) is the coordinate of the center of the circle, r is the radius;

[0046] By statically avoiding the penalty r p Drive the agent to avoid the fixed threat area, the static avoidance penalty r p for:

[0047]

[0048] The overall reward function is defined as:

[0049] r=r a +λr p ,

[0050] Where λ is the coefficient.

[0051] Furthermore, step 3 includes:

[0052] When training the independent policy network-centralized policy evaluation network, introduce clipping instead of collaborative loss function for:

[0053]

[0054] Where θ is the parameter of the independent policy network, r t (θ) is the ratio of the new strategy to the old strategy, is the generalization advantage estimate, clip(·) is the clipping function, ε is the truncation factor, ε∈[0,1], for:

[0055]

[0056] Where, δ t is the time difference error at time t, γ is the discount factor, λ is the learning rate, δ t+1 is the time difference error at time t+1, T s -1 time difference error, δ t for:

[0057] δ t =r t +γV φ (s t+1 )-V φ (S t )

[0058] Where r t For reward, V φ (s t+1 ) is the value function of the centralized strategy judging the network at time t+1, V φ (s t ) is the value function of the centralized strategy judging the network at time t, V φ (s t )for:

[0059]

[0060] In the formula, the parameters of the centralized strategy evaluation network are represented by φ, and the centralized strategy evaluation network is updated by gradient descent and loss function. The loss function for:

[0061]

[0062] Where y(t) is the target value function, y(t) = r t +γV φ (s t+1 ), the overall objective function for:

[0063]

[0064] In the formula, c1 and c2 are coefficients, S[π θ ](s t ) is state s t Next strategy π θ Entropy;

[0065] After the training is completed, the trained model is executed, and the game situation information is used as input to obtain the target allocation plan.

[0066] Furthermore, step 4 includes:

[0067] When establishing a constrained collaborative experience-first multi-agent reinforcement learning model for the Markov game process to solve the maneuver strategy generation problem, an independent strategy network-centralized strategy evaluation network is used, and the multi-agent strategy containing N agents is denoted as π=[π1,…,π N ], the corresponding strategy parameters are recorded as θ=[θ1,…,θ N ], the gradient of each agent’s independent policy network is:

[0068]

[0069] in, Represents the global action-value function, the input is the action combination of all agents [a1,…,a N ] and global observation state information x=[s1,…,S N ], the output is the Q value of each agent;

[0070] The maneuver strategy generation multi-agent is set as:

[0071] Each Red Agent searches for an independent strategy network for the resource optimization agent, including the strategy network and network parameters Policy Target Network and network parameters Represents the local observation state of each agent; all agents share the global value network Q(S t ,A t ∣θ θ ) and network parameters θ Q , global value target network Q′(S t ,A t ∣θ Q ) and network parameters θ Q′ , the global observation state and global action are expressed as The policy gradient of each agent is:

[0072]

[0073] Among them, the experience replay pool D includes the four-tuple [S t ,S t+1 ,A t ,R t ], the reward obtained by each agent for this action is expressed as

[0074] The loss function of the global action-value network is:

[0075]

[0076]

[0077] The target network parameters are soft-updated using the updated network parameters. The update expression is:

[0078] θ Q′ ←τθ Q +(1-τ)θ Q′

[0079]

[0080] The constrained collaborative experience-first multi-agent reinforcement learning model is trained according to the target allocation scheme obtained in step 3: after receiving the relative position of the target point, the relative position of the red multi-agents, and the position of the blue intercepting agent, each red agent maintains an independent policy network. First, feature extraction is performed on the position information, the feature vector is input into the independent policy network, and a deterministic maneuver action plan is output; then, the red agent interacts with the environment, performs maneuvers, and receives rewards and next state vectors returned by the environment; the received rewards are compared with the estimated results of the global value network and the global value network is updated; the reward output of the global value network is used to calculate the policy gradient of each agent, and finally, the independent policy network of each red agent is updated;

[0081] Input the game situation information and maneuver status information to obtain the optimal path to attack the target.

[0082] Furthermore, add the following term to the loss function in step 4:

[0083] L=L(r)+ω·cos(θ(v,p tar -p missile ))

[0084] Among them, ω and θ are parameters, p tar and p missile are the position of the target point and the position of the red team agent respectively.

[0085] Compared with the prior art, the present invention has the following beneficial effects:

[0086] Regarding the task allocation problem: Compared with single-agent reinforcement learning, the method proposed in the present invention innovatively combines a centralized training distributed execution architecture with single-agent proximal strategy optimization, designs a collaborative multi-agent reinforcement learning method, and applies it to the multi-red agent target allocation task; Compared with multiple single-agent methods, the present invention uses a centralized value network to generate a global state-action value containing global information, so that each agent network can obtain global information during the training process. In this way, each agent can make independent actions with the goal of collaboration by observing the state and behavior of other agents, thereby achieving cooperation between agents. Ultimately, the multi-agent group can observe the confrontation capabilities of other red agents and the defense level and threat level of the blue agent's target on the basis of modeling its own confrontation capabilities, physical location and other information, and make confrontation actions to cooperate with other red agents, thereby improving the success rate of confrontation and overall effectiveness.

[0087] When it comes to online maneuver strategy generation, it's crucial to consider inter-agent coordination within a rapidly changing dynamic environment, as well as physical constraints such as the agents' own maneuverability thresholds and the rationality of their planned routes. Therefore, within the proposed dual-team heterogeneous adversarial simulation platform, this paper further proposes an online generation model for group maneuver strategies based on constrained collaborative multi-agent reinforcement learning.

[0088] Compared to single-agent reinforcement learning methods, the constrained collaborative multi-agent reinforcement learning method proposed in this paper adopts a centralized training and distributed execution strategy. This allows for access to all agent states and actions during training, thus better managing the collaborative relationships among multiple agents. During execution, each Red Agent makes independent decisions. This design effectively balances information sharing and independence, enabling Red Agents to better adapt in complex environments.

[0089] Compared to multiple single-agent reinforcement learning approaches, the model proposed in this paper uses a centralized global value network that accesses the states and actions of all agents, allowing for more accurate assessment of the value of joint actions. Furthermore, this centralized design enables a more rational strategic division of labor among agents, resulting in superior sample efficiency and convergence speed compared to multiple single-agent approaches.

[0090] To speed up the solution, this paper sets physical constraints such as maneuver constraints and adds a penalty term to the reward function to inhibit actions that deviate significantly from the target. This helps the Red Agent choose actions that are more likely to move towards the target during action selection.

[0091] Finally, the present invention proposes the use of an experience-prioritized replay module to improve the Red Agent's utilization of training samples in complex game scenarios. This module reuses the temporal difference error of each action step as a parameter for adjusting the weight of high-quality experience. By reweighting the experience buffer, the agent is encouraged to prioritize high-quality experience in the buffer, thereby improving training sample efficiency. Ablation experiments demonstrate that this module significantly reduces training time, improves sample efficiency, and enhances the agent's ability to learn from the environment in a multi-agent maneuver strategy generation task. BRIEF DESCRIPTION OF THE DRAWINGS

[0092] Figure 1 is a flow chart of the target allocation method of the present invention;

[0093] Figure 2 It is a flow chart of the maneuver strategy generation method in the present invention. DETAILED DESCRIPTION

[0094] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0095] The present invention provides a constrained collaborative multi-agent reinforcement learning method for red-blue game confrontation. By establishing a three-dimensional game confrontation scenario and an agent kinematic and dynamic model based on real physical constraints, it accurately describes the agent's action selection and execution process, defines multi-agent reinforcement learning elements as key information in the game confrontation process, and designs a constrained collaborative multi-agent reinforcement learning method to solve it, thereby achieving a high win rate in red-blue game confrontation. Specifically:

[0096] First, a three-dimensional simulation platform for dual-team heterogeneous game confrontation is established, the movement rules of intelligent agents are defined according to real physical laws, and the game confrontation entities between the red and blue teams are modeled.

[0097] Secondly, we assign subtasks to collaborative goals, define key elements of multi-agent reinforcement learning, and establish a collaborative multi-agent reinforcement learning method. Key elements in multi-agent reinforcement learning—agents, actions, and states—are specifically defined as information about the blue team's goal, goal value, and damage level, among other game scenario information. Next, we design an overall reward function for different value goals to encourage agents to compete against high-value goals.

[0098] Building on the single-agent approach, a multi-agent reinforcement learning method based on a centralized training distributed execution architecture is constructed. Unlike traditional fully distributed reinforcement learning architectures, the CTDE (Centralized Training Distributed Execution) architecture considers global information about the game confrontation scenario during the training phase. Each agent can obtain the position and loss information of other teammates through a global network. Therefore, agents trained in this way can make intelligent decisions based on the global game confrontation environment and oriented towards inter-agent cooperation.

[0099] Third, focusing on the subtasks of maneuver strategy generation, we redefine the key elements of multi-agent reinforcement learning and establish a constrained collaborative multi-agent reinforcement learning method. We introduce a prioritized experience replay mechanism to improve the agents' utilization of high-quality execution experience, thereby increasing the sample efficiency of training.

[0100] The present invention first models the collaborative confrontation problem of the red agent, including constructing a virtual game scene and assuming confrontation with the blue agent's goal, decomposing the game confrontation task into the blue agent's goal allocation problem and the maneuver strategy generation problem for modeling; then, under the game situation between the red and blue parties, a goal allocation method based on constrained collaborative multi-agent reinforcement learning is established to obtain a goal allocation scheme; finally, according to the obtained goal allocation scheme, a maneuver strategy generation method based on constrained collaborative multi-agent reinforcement learning is established to obtain the optimal path to attack the target.

[0101] like Figure 1 and Figure 2 As shown, the detailed scheme of the present invention comprises the following steps:

[0102] S1. Define the target collaborative allocation problem and the maneuver strategy generation problem.

[0103] First, let's define the problem of target collaborative allocation. Target collaborative allocation is a classic operations research problem. Consider a static target allocation problem. Given the value of a target and the probability of a candidate agent attacking each target, the remaining value of the target is minimized by assigning the red agent to the target to be attacked. This problem can be formalized as:

[0104]

[0105] Where: n represents the number of targets; m represents the number of red agents; v i Indicates the value of the target; p ij represents the probability of the red agent j confronting the target i; b ij Indicates whether agent j is assigned to target i. If so, then b ij =1, otherwise b ij= 0. The general principles that should be followed in target collaborative allocation are: 1) Value priority principle: prioritize targets with high value; 2) Local optimality principle: assign targets to the Red Team agent that is most beneficial to it; 3) Load balancing principle: allocate targets reasonably to each Red Team agent, and balance the task load as much as possible.

[0106] Assume that there are N blue targets in the game confrontation scenario, and our side has M red agents. Each target requires one or more red agents to attack. Assume that there is no mutual dependence between the targets. N} is a set of N targets, {att(T K ),def(T K )} is the attack capability and breakthrough capability required for the K-th target, U={U1,U2,…,U M} is the set of red agents, {att(U S ),def(U S )} is the attack capability and breakout capability of the Sth red agent. Based on the above data, the mapping relationship between the red agent and the target can be constructed.

[0107] Next, we define the problem of the maneuver strategy generation subtask. Given a set L of red agents, which contains a finite number (|L|=N L ) of the Red Agent L i ∈L(i=1,…,N L To simplify the problem, this paper constructs it in three-dimensional space: for any path node t, the red agent L i The spatial position of Target T j The spatial position of The Red Agent is abstracted as a uniformly moving mass point without considering the constraints of complex maneuverability. Finally, all the offensive tasks to be executed are grouped into a task set U. Each task U in the set U i ∈U(i=1,…,N L ),U i The result is that each red party agent L i The attack path Tr corresponding to the target planning k (1≤k≤n), so each task U i It is composed of the path position nodes of the Red Agent, different U i Allowed to have the same spatial position

[0108]

[0109] Where: represents the initial position of the Red Agent; etc. represent the path nodes that the red agent passes through in sequence; Indicates the final position of the Red Agent, i.e. the target space position. Attack path Tr k This is an ordered collection of path nodes that define the Red Agent's maneuver path from its starting state to its final state, satisfying specific constraints. Based on the corresponding method, a model established using target scenarios is used to calculate target threat coefficients and assign targets. A planning space is established based on the actual terrain of the game scenario, completing the Red Agent's path planning under target allocation decisions.

[0110] S2. Modeling the Markov game process of the target collaborative allocation problem and the maneuver strategy generation problem.

[0111] First, we model the key elements for solving the collaborative goal allocation: state, action, and reward. First, we model the state of the blue team's goal. Given m blue team target points, the i-th target point is represented by its two-dimensional coordinates The threat level and the remaining integrity level are composed. For the i-th target point, its threat level includes the blue side's defense level Number of blue agents Importance of blue team's target Mission Special Objective And other target point information. The remaining completeness represents the health of the blue agent, which is The state of the target point i at time t is:

[0112]

[0113] Among them, the defense level and mission objectives of the blue team generally do not change over time. Consider a red team agent group consisting of n friendly agents. The state of the i-th red team agent is determined by its adversarial capability matrix and two-dimensional coordinates Among them, the Red Agent's confrontation ability matrix is ​​a D L ×A L dimensional matrix, where D L is the blue side's defense level dimension, A L The dimension of our confrontation level. Element A in the confrontation capability matrix agent (i,j) represents the use of A agent (·,j)-level Red Agent targets A agent The effect of the confrontation with the target of (i,·) defense level. At time t, the red agent i receives an observation matrix from the environment:

[0114]

[0115] in, for Medium For elements other than , stack(·) represents a row stacking operation. Action a of agent i i =j is defined as a confrontation with target j. All agents share a joint confrontation reward, which is proportional to the sum of the changes in the health of the blue target before and after the confrontation:

[0116]

[0117] in, ρ is a parameter. Combined with the state definition, this joint adversarial reward can correctly model the value priority principle, local optimal principle, and load balancing principle required by the task.

[0118] Secondly, we establish a Markov game process modeling and reinforcement learning method for the red-blue game against the blue team's maneuver strategy generation problem. The key elements of the game process are defined in the following steps:

[0119] First, based on the simulation entity coordinate state, add the red party agent speed information Define the state of our red agent and the blue team intercepting the agent state At time t, the red agent i receives an observation matrix from the environment:

[0120]

[0121] Among them, stack(·) represents the row stacking operation, is the coordinate of the target point.

[0122] Next, we define the cooperative game action set. In a cooperative game scenario, to avoid the threat zone, the Red Agent primarily changes its azimuth and path direction by adjusting the power source. Therefore, controlling lateral overload can achieve path direction switching. This invention defines the Red Agent action set as a continuous series of azimuth adjustment actions:

[0123]

[0124] The joint action of the multi-agent group is: According to the above definition, the Red Agent is able to continue moving in the direction of velocity and maneuver sideways.

[0125] Model the agent entities in the three-dimensional game confrontation space of the maneuver strategy generation task. Establish a coordinate system with the center of mass of the red agent as the origin, and the kinematic equation of the red agent i can be formalized as:

[0126]

[0127] Among them, v i represents the speed of the red agent, γ i represents the angle between the velocity direction and the XOY plane, ψ i represents the angle between the velocity direction and the X-axis. In the collaborative breakout scenario, the Red Agent's maneuvering decision mainly depends on adjusting the maneuvering direction angle. Therefore, the dynamic equation of the Red Agent i can be formalized as:

[0128]

[0129] in, Represents the directional control overload of the red agent, that is, the y-axis direction control overload, and g is the gravitational acceleration.

[0130] Modeling is used to solve the key elements of maneuver strategy generation: state, action, and reward. In this task, the reward function aims to drive the Red Agent to approach the target and avoid the Interceptor Agent. When the Red Agent is very close to the target (which can be considered as the Red Agent hitting the target), the reward function should provide a larger reward. Therefore, the reward function is defined as follows:

[0131]

[0132] in, represents the cumulative distance between each Red Agent and its target point. Furthermore, when a Red Agent is too close to an Interceptor Agent (considered to have hit the Red Agent), the Red Agent receives a penalty and is blocked, no longer participating in subsequent simulations. When all Red Agents are blocked, it can be considered that all Red Agents have been intercepted, and the simulation ends. A hybrid expert knowledge-assisted threat avoidance rule is established from the perspectives of maneuver behavior blocking and spontaneous learning. First, the threat area maneuver action blocking mechanism at time t is established using the following formula:

[0133]

[0134] in, Represents taking action a at the current time t t After that, the agent reaches the state, D((x T ,y T ), r) represents the threat area, which is represented by the coordinates of the center of the circle (x T ,y T ) and its radius r. The above formula means that when the red agent is about to take a specific action that puts it in the threat zone, the action will be blocked as unselectable. In addition, by static avoidance penalty r p Further drive the agent to avoid fixed threat areas:

[0135]

[0136] If our Red Agent enters the air defense threat zone, it will receive a large negative penalty and will be blocked from participating in subsequent simulations. If all agents are blocked, the simulation ends and the mission is considered a failure. The settlement reward is a large negative failure penalty. The overall reward function is: r = r a +λr p , where λ is a coefficient. The above reward function design can meet the path effectiveness principle, self-safety principle, and energy conservation principle required by the maneuver strategy generation task. The three principles are:

[0137] 1) Principle of effective path: the planned path should be effective;

[0138] 2) Self-safety principle: The multi-agent formation should be able to safely pass through threat areas, avoid obstacles, and avoid intercepting agents;

[0139] 3) Energy conservation principle: The Red Agent's maneuvering strategy is rationally planned to ensure the shortest path and conserve resources.

[0140] Next, we define the mobile threat. First, we define two Blue intercepting agents, each with the goal of getting as close to the Red agent as possible. If the Red agent gets too close to the intercepting agent, it receives a large negative penalty and is blocked from participating in subsequent simulations. Next, we define the mobile threat zone. Each Blue target also has a certain ability to intercept the Red agent, and its success rate in intercepting the Red agent is inversely proportional to its relative distance from the Red agent.

[0141] S3. Establish a constrained collaborative multi-agent reinforcement learning model for solving target collaborative allocation.

[0142] The present invention proposes a framework diagram of collaborative multi-agent reinforcement learning based on CTDE. Figure 1 The present invention adopts an independent policy network-centralized policy evaluation network architecture and introduces a new pruning substitution function to avoid excessive modification of the target value.

[0143] New clipping replacement collaborative loss function during training for:

[0144]

[0145] Where: θ is the parameter of the independent policy network, r t (θ) is the ratio of the new strategy to the old strategy; For General Advantage Estimation (GAE), clip(·) is a clipping function, which is introduced to limit r t(θ), ε is the truncation factor, ε∈[0,1]. The generalization advantage estimate is used to estimate the advantage of taking action a relative to the average action in state s. Its mathematical expression is:

[0146]

[0147] Where: δ t is the time difference error at time t, γ is the discount factor, λ is the learning rate, δ t+1 is the time difference error at time t+1, T s -1 moment time difference error.

[0148] The mathematical expression of the time difference error at time t is:

[0149] δ t =r t +γV φ (s t+1 )-V φ (S t )

[0150] Where: r t For reward, V φ (s t+1 ) is the value function of the centralized strategy judging the network at time t+1, V φ (s t ) is the value function of the centralized strategy evaluation network at time t, which is expressed as:

[0151]

[0152] The parameters of the centralized policy evaluation network are represented by φ. The centralized policy evaluation network is updated by gradient descent and loss function. The loss function The expression is:

[0153]

[0154] Where y(t) is the target value function, expressed as y(t)=r t +γV φ (s t+1 ). The loss function combines the policy proxy term and the value function error term, and uses entropy addition to increase the exploration power, so the overall objective function It can be described as:

[0155]

[0156] In the formula, c1 and c2 are coefficients, S[π θ ](s t ) is state s t Next strategy πθ entropy.

[0157] The model framework proposed in this invention includes multiple agents, each of which executes the Proximal Policy Optimization (PPO) method, and guides the training of independent policy networks through a centralized policy evaluation network that can observe global information. The input of the independent policy network is the local observation of each red agent, and the output is a list of probabilities corresponding to each action in the action space. These probabilities constitute a distribution, so actions can be sampled. The centralized policy evaluation network can evaluate whether the actions taken by the independent policy network meet the expected feedback, and obtain the optimal strategy after the training is completed. During the execution phase, there is no need for internal information exchange between the agents, nor is there any need for random exploration. The optimal action can be taken based on their own local observations.

[0158] After the training is completed, the trained model is executed and the situation information is used as input to obtain the target allocation solution.

[0159] S4. Establish a constrained collaborative experience-first multi-agent reinforcement learning model for solving maneuver strategy generation.

[0160] After obtaining the target allocation scheme, a constrained experience-priority multi-agent reinforcement learning method is established to solve the maneuver strategy generation subtask, and physical constraints are used to improve the rationality of the generated strategy.

[0161] The multi-agent strategy containing N agents is denoted as π=[π1,…,π N ], the corresponding strategy parameters are recorded as θ=[θ1,…,θ N ], then the policy gradient of each agent can be expressed as follows:

[0162]

[0163] in Represents the global action-value function, the input is the action combination of all agents [a1,…,a N ] and global observation state information x=[s1,…,S N ], and the output is the Q value of each agent.

[0164] The multi-agent setting of maneuver strategy generation is as follows: Each red agent searches for the strategy network of the resource optimization agent (independent strategy network) including the strategy network and network parameters Policy Target Network and network parameters Represents the local observation state of each agent; all agents share the global value network (centralized strategy evaluation network) Q(S t,A t ∣θ θ ) and network parameters θ Q , global value target network Q′(S t ,A t ∣θ Q ) and network parameters θ Q′ , the global observation state and global action are expressed as Then the policy gradient of each agent can be expressed as follows:

[0165]

[0166] The experience replay pool D includes the four-tuple [S t ,S t+1 ,A t ,R t ], the reward obtained by each agent for this action is expressed as Then the loss function of the global action-value network can be expressed as follows:

[0167]

[0168]

[0169] Finally, the updated network parameters are used to soft-update the target network parameters. The update expression is as follows:

[0170] θ Q′ ←τθ Q +(1-τ)θ Q′

[0171]

[0172] After receiving the relative position of the target point, the relative positions of our multi-agent team, and the position of the Blue intercepting agent, each Red agent maintains an independent policy network. It first extracts features from this position information and feeds the feature vector into its own independent policy network, which outputs a deterministic maneuver plan. It then interacts with the environment, executes the maneuver, and receives rewards and next-state vectors from the environment. The received rewards are compared with the estimates of the global value network, which then updates the global value network. The reward output from the global value network is used to calculate the policy gradient for each agent, and finally, each Red agent's independent policy network is updated.

[0173] In order to speed up the solution, the present invention sets the maneuver constraint and adds the following terms to the loss function:

[0174] L=L(r)+ω·cos(θ(v,p tar -p agent ))

[0175] Among them, ω and θ are parameters, p tar and p agent are the position of the target point and the position of the Red Agent, respectively. By penalizing actions that deviate significantly from the target, the Red Agent is able to choose actions that are more likely to move towards the target during action selection.

[0176] To further improve sample utilization, this paper proposes an experience-prioritized replay module. Because multi-agent maneuverable combat scenarios are complex and varied, agents need to make numerous attempts in each scenario to learn the appropriate strategy. Therefore, this paper proposes an experience-prioritized replay method, assigning different priorities to historical experiences in the experience buffer. Experiences with larger temporal differential errors are defined as high-quality experiences, and these high-quality experiences are given a higher sampling probability.

[0177] The following introduces a specific testing method of an embodiment of the present invention. First, a reinforcement learning simulation environment is established according to steps S1 and S2; then, a constrained collaborative multi-agent reinforcement learning method is established according to steps S3 and S4.

[0178] 1. Example test environment:

[0179] The test software environment is Pytorch 1.12.1+cu116, the CPU is Intel core i5-13600KF 3.50GHz, the memory is 32GB, and the hard disk storage is 1TB.

[0180] 2. Specific settings of the embodiment

[0181] This invention sets up 12 Red Agents and 8 Blue Agents. The Red Agent's goal is to reach the Blue Agent's position and gain an advantage. If the advantage is greater than 0, the goal is considered achieved and the Blue Agent's target is destroyed. For the target collaborative allocation subtask, the threat level and importance level of the Blue Agent's target are set as shown in Table 1:

[0182] Table 1: Threat Level and Importance Level of Blue Force Targets

[0183] Target number 0 1 2 3 4 5 6 7 Threat Level Low middle middle middle middle high high high Importance level 10 7 7 5 5 6 6 6

[0184] The confrontation capabilities of the Red Team agent are set as shown in Table 2:

[0185] Table 2: Red Team’s Countermeasures

[0186] Agent 0 1 2 3 4 5 6 7 8 9 10 11 Countermeasures high high high high Low Low Low Low middle middle middle middle

[0187] The obtained relative confrontation advantage matrix is ​​shown in Table 3:

[0188] Table 3: Agent relative adversarial advantage matrix

[0189]

[0190] For the maneuver strategy generation subtask, the game confrontation scenario simulation environment was set to a 50-unit × 50-unit area with a height of 200 units. To meet the performance requirements, two fixed threat zones, 2 units × 2 units with infinite height, were set at (x = 2, y = 22.5) and (x = -2, y = 22.5). The fixed threat zones were shaped like cubes. If a Red agent touched a fixed threat zone, it was considered intercepted by the Blue agent and blocked from participating in the subsequent simulation. Two Blue agents were set to intercept the Red agent and treat it as a mobile threat. Their initial positions were randomly generated near the Blue agent's target point. The Blue agent was assumed to have a certain interception capability for the Red agent, with the interception probability inversely proportional to its distance from the Red agent. Agents s1-s7 were defined as having interception capability, while s0 was defined as not having interception capability. The experiment was divided into three parts: with an intercepting agent in the presence of a fixed threat, without an intercepting agent in the presence of a fixed threat zone, and without an intercepting agent in the absence of a fixed threat zone.

[0191] 3. Benchmark Methodology

[0192] In order to evaluate the effectiveness of the present invention, the ant colony algorithm was selected as the preset benchmark method for experiments.

[0193] 4. Experimental results of the embodiment

[0194] The experimental results of the existing mainstream solutions and the present invention are shown in Table 4 and Table 5. From the results, the present invention has a significantly higher winning rate than other mainstream methods.

[0195] Table 4: Quantitative experimental comparison (agent hit rate / target damage rate)

[0196]

[0197] Table 5: Quantitative experiment of constraint experience-first reinforcement learning (agent hit rate / target damage rate)

[0198]

[0199] The beneficial effects of the present invention are as follows:

[0200] Due to the complexity of the two-team game confrontation scenario and the high dynamics of the environment, in order to improve the winning rate of the game confrontation, the present invention proposes a constrained collaborative multi-agent reinforcement learning method for red-blue game confrontation. For the target allocation subtask, an online target allocation technology based on multi-agent deep reinforcement learning is adopted, combined with a lightweight multi-agent reinforcement learning method, to solve the problem of a sharp increase in task complexity caused by the increase in the number of agents, and to achieve high efficiency in strategy generation and decision-making. For the maneuver strategy generation subtask, a model framework containing physical constraints and adversarial constraints is designed to ensure the executability and practical applicability of the generated strategy; a collaborative constrained experience-priority multi-agent deep deterministic policy gradient model is constructed to achieve efficient strategy generation and optimization. The model structure of the present invention is simple, the architecture is convenient, and it can be used and promoted in a wide range.

[0201] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0202] The above content is only for explaining the technical idea of ​​the present invention and cannot be used to limit the protection scope of the present invention. Any changes made on the basis of the technical solution in accordance with the technical idea proposed by the present invention shall fall within the protection scope of the claims of the present invention.

Claims

1. A constrained collaborative multi-agent reinforcement learning method for red-blue game confrontation, characterized by: The following steps are involved: Step 1: Model the Red Agent collaborative confrontation problem, including constructing a virtual game scenario and assuming a confrontation with the Blue Agent. This task is then decomposed into a target collaborative allocation problem and a maneuver strategy generation problem. Step 2: Establish the Markov game process for the target collaborative allocation problem and the Markov game process for the maneuver strategy generation problem respectively; Step 3: Under the red-blue game situation, a constrained collaborative multi-agent reinforcement learning model for solving the target collaborative allocation problem is established for the Markov game process. The game situation information is input to obtain the target allocation solution. Step 4: Establish a constrained collaborative experience-first multi-agent reinforcement learning model for the Markov game process for solving the maneuver strategy generation problem. Train the constrained collaborative experience-first multi-agent reinforcement learning model according to the target allocation scheme obtained in step 3, input game situation information and maneuver state information, and obtain the optimal path to attack the target.

2. The method according to claim 1, characterized in that Step 1 specifically includes: The target collaborative allocation problem is defined as: Where n represents the number of blue targets; m represents the number of red agents; v i Indicates the value of the blue team's goal; p ij represents the probability of the red agent j confronting the blue target i; b ij Indicates whether the red agent j is assigned to the blue target i. If so, then b ij =1, otherwise, b ij =0; Define the maneuver strategy generation problem, including: Given a red agent set L, which contains a finite number of red agents L i ∈L(i=1,…,N L ), for any three-dimensional path node t, the red agent L i The spatial position of Blue team target T j The spatial position of All the offensive tasks to be executed are grouped into a task set U. Each task U in the set U i ∈U(i=1,…,N L ), U i The result is that for each red agent L i The three-dimensional attack path Tr corresponding to the blue team's target planning k (1≤k≤n), Where, represents the initial position of the red agent, Represents the three-dimensional path nodes that the red agent passes through in sequence, Indicates the final position of the red agent, that is, the target space position of the blue agent; The coordinate system is established with the center of mass of the red agent as the origin, and the kinematic equation of the red agent i is: Among them, v i represents the speed of the red agent, γ i represents the angle between the velocity direction and the XOY plane, ψ i Represents the angle between the velocity direction and the X-axis; The dynamic equation of the red agent i is: in, Represents the directional control overload of the red agent, and g is the acceleration due to gravity.

3. The method according to claim 2, characterized in that The Markov game process for establishing the target collaborative allocation problem described in step 2 includes: Given m blue target points, the i-th target point is represented by its two-dimensional coordinates The threat level and the remaining integrity level are composed of the threat level of the i-th target point, including the blue side's defense level Number and scale of blue team targets Importance of blue team's target Game task objectives The remaining completeness is determined by Indicates that the state of the target point i at time t is: At time t, the observation matrix received by the red agent i from the environment is: in, for Medium Elements other than , stack(·) represents the row stacking operation, the action a of the red agent i i =j is defined as a confrontation with target j, and all agents share a joint confrontation reward, which is proportional to the sum of the changes in the blue agent's health before and after the confrontation: in, ρ is a parameter; The Markov game process for modeling the maneuver strategy generation problem includes: Add the speed information of the Red Team agent based on the coordinate state of the simulation entity Define the Red Agent State and the blue team intercepting the agent state At time t, the observation matrix received by the red agent i from the environment is: Among them, stack(·) represents the row stacking operation, is the coordinate of the target point; Define the cooperative game confrontation action set: Define the red agent action set as a continuous action of direction angle adjustment: The joint actions of the multi-agent group are: The reward function for the collaborative game scenario is defined as: in, Represents the cumulative distance between each red agent and its target point; The maneuver shielding mechanism for the threat area at time t is established as follows: in, Represents taking action a at the current time t t The state that the agent reaches later, D((x T ,y T ),r) represents the threat area, (x T ,y T ) is the coordinate of the center of the circle, r is the radius; By statically avoiding the penalty r p Drive the agent to avoid the fixed threat area, the static avoidance penalty r p for: The overall reward function is defined as: r=r a +λr p , Where λ is the coefficient.

4. The method according to claim 3, characterized in that Step 3 includes: When training the independent policy network-centralized policy evaluation network, introduce clipping instead of collaborative loss function for: Where θ is the parameter of the independent policy network, r t (θ) is the ratio of the new strategy to the old strategy, is the generalization advantage estimate, clip(·) is the clipping function, ε is the truncation factor, ε∈[0,1], for: Where, δ t is the time difference error at time t, γ is the discount factor, λ is the learning rate, δ t+1 is the time difference error at time t+1, T s -1 time difference error, δ t for: δ t =r t +γV φ (s t+1 )-V φ (S t ) Where r t For reward, V φ (s t+1 ) is the value function of the centralized strategy judging the network at time t+1, V φ (s t ) is the value function of the centralized strategy to judge the network at time t, V φ (s t )for: In the formula, the parameters of the centralized strategy evaluation network are represented by φ, and the centralized strategy evaluation network is updated by gradient descent and loss function. The loss function for: Where y(t) is the target value function, y(t) = r t +γV φ (s t+1 ), the overall objective function for: In the formula, c1 and c2 are coefficients, S[π θ ](s t ) is state s t Next strategy π θ Entropy; After the training is completed, the trained model is executed, and the game situation information is used as input to obtain the target allocation plan.

5. The method according to claim 4, characterized in that Step 4 includes: When establishing a constrained collaborative experience-first multi-agent reinforcement learning model for the Markov game process to solve the maneuver strategy generation problem, an independent strategy network-centralized strategy evaluation network is used, and the multi-agent strategy containing N agents is denoted as π=[π1,…,π N ], the corresponding strategy parameters are recorded as θ=[θ1,…,θ N ], the gradient of each agent’s independent policy network is: in, Represents the global action-value function, the input is the action combination of all agents [a1,…,a N ] and global observation state information x=[s1,…,S N ], the output is the Q value of each agent; The maneuver strategy generation multi-agent is set as: Each Red Agent searches for an independent strategy network for the resource optimization agent, including the strategy network and network parameters Policy Target Network and network parameters Represents the local observation state of each agent; all agents share the global value network Q(S t ,A t ∣θ θ ) and network parameters θ Q , global value target network Q′(S t ,A t ∣θ Q ) and network parameters θ Q′ , the global observation state and global action are expressed as The policy gradient of each agent is: Among them, the experience replay pool D includes the four-tuple [S t ,S t+1 ,A t ,R t ], the reward obtained by each agent for this action is expressed as The loss function of the global action-value network is: The target network parameters are soft-updated using the updated network parameters. The update expression is: The constrained collaborative experience-first multi-agent reinforcement learning model is trained according to the target allocation scheme obtained in step 3: after receiving the relative position of the target point, the relative position of the red multi-agents, and the position of the blue intercepting agent, each red agent maintains an independent policy network. First, feature extraction is performed on the position information, the feature vector is input into the independent policy network, and a deterministic maneuver action plan is output; then, the red agent interacts with the environment, performs maneuvers, and receives rewards and next state vectors returned by the environment; the received rewards are compared with the estimated results of the global value network and the global value network is updated; the reward output of the global value network is used to calculate the policy gradient of each agent, and finally, the independent policy network of each red agent is updated; Input the game situation information and maneuver status information to obtain the optimal path to attack the target.

6. The method according to claim 5, characterized in that Add the following term to the loss function in step 4: L=L(r)+ω·cos(θ(v,p tar -p missile )) Among them, ω and θ are parameters, p tar and p missile are the position of the target point and the position of the red team agent respectively.

Images