An intelligent cooperative confrontation strategy generation method based on a heterogeneous unmanned aerial vehicle group

Abstract

The application discloses a kind of intelligent cooperative confrontation strategy generation methods based on heterogeneous unmanned aerial vehicle group, belong to multi-agent deep reinforcement learning field.The application includes the following steps: constructing cooperative confrontation scene, constructing blue side confrontation rule strategy model, constructing red side heterogeneous multi-agent reinforcement learning model, intelligent strategy training and generation.The application is directed to intelligent cooperative confrontation strategy generation method problem, abstracts electromagnetic confrontation and fire confrontation mechanism, and innovatively constructs heterogeneous multi-agent reinforcement learning model, and realizes intelligent confrontation strategy generation by data training.

Description

Technical Field

[0001] This invention relates to a method for generating intelligent cooperative adversarial strategies based on heterogeneous drone swarms, belonging to the field of multi-agent deep reinforcement learning. Background Technology

[0002] Limited by the payload capacity of drones, a single drone cannot simultaneously handle both firepower strikes and electromagnetic countermeasures. Therefore, drone swarms are currently the primary means of achieving mission objectives, especially when drones within a swarm carry different payloads and form tactical synergy based on adversarial strategies to achieve these objectives. The generation method of adversarial strategies is therefore of paramount importance. Furthermore, compared to the lack of flexibility in rule-based models built on expert experience, intelligent collaborative adversarial strategy generation methods oriented towards data and digital twin battlefields can adaptively generate adversarial strategies, making them a hot topic in current intelligent strategy generation research.

[0003] Patent CN111024080B proposes a method for planning the reconnaissance path of a mobile UAV swarm, but this method lacks consideration for enemy firepower or electromagnetic interception. Patent CN112595174B proposes a method and device for multi-UAV tactical decision-making in dynamic environments, focusing on fire control countermeasures, but lacking an electromagnetic countermeasures domain, making it difficult to respond to mission requirements from electromagnetic sensing to fire control. "QMIX: Monotonic Value Function Factorization for Deep Multi-Agent Reinforcement Learning" is a representative application of multi-agent reinforcement learning, innovatively achieving overall value decomposition through a hypernetwork, thus correctly handling the relationship between individual actions and overall value. However, each UAV only inputs its own local information and relies on overall state information to form cooperation, making it inflexible in engineering practice. "A Method for UAV Swarm Formation Based on Cooperative Electronic Countermeasures" transforms the cooperative electromagnetic countermeasures problem of UAV swarms into a multi-objective optimization problem and optimizes it using a genetic algorithm, but weakens the impact of maneuvering on the mission in the scenario. Summary of the Invention

[0004] To enhance the ability to generate intelligent cooperative adversarial strategies for heterogeneous UAV swarms, this invention proposes a method for generating intelligent cooperative adversarial strategies based on heterogeneous UAV swarms, which enables heterogeneous UAV swarms carrying different mission payloads to achieve cooperative adversarial tasks.

[0005] The technical solution adopted in this invention is as follows:

[0006] A method for generating intelligent cooperative adversarial strategies based on heterogeneous UAV swarms includes the following steps:

[0007] Step 1: Construct a cooperative combat scenario, including the number of entities and scenario location, observation rules, firepower strike rules, electromagnetic countermeasure rules, maneuver rules, observation data, and reward function;

[0008] Step 2: Construct the blue team's adversarial rules, including the horizontal and vertical actions of each blue team entity;

[0009] Step 3: Construct a heterogeneous multi-agent reinforcement learning model for the red team;

[0010] Step 4: Based on the cooperative adversarial scenario and the blue side's adversarial rules, train the red side's heterogeneous multi-agent reinforcement learning model to generate intelligent cooperative adversarial strategies.

[0011] Furthermore, the specific method of step 1 is as follows:

[0012] Step 1-1: Determine the number of entities and scene location

[0013] A two-dimensional rectangular adversarial scenario is used, with the projection range on the X-axis of the scenario being [0, X]. max The projection range of the two-dimensional rectangular adversarial scene on the Y-axis is [0, Y]. max The entities in the adversarial scenario are divided into the red team and the blue team, each including 1 jammer, 2 fighters and 1 base; the position of the red team's jammer is x0, y0, the position of the red team's first fighter is x1, y1, the position of the red team's second fighter is x2, y2, the position of the red team's base is x3, y3, the position of the blue team's jammer is x′0, y′0, the position of the blue team's first fighter is x′1, y′1, the position of the blue team's second fighter is x′2, y′2, the position of the blue team's base is x′3, y′3;

[0014] Step 1-2: Constructing Observation Rules

[0015] The observation distance threshold is set to θ1, and the observation angle threshold is set to θ2. For the nth red fighter jet, when the relationship with the blue entity m satisfies the following condition, the nth red fighter jet can detect the target and use the blue entity's position to construct the observation data:

[0016]

[0017]

[0018] a=[x′ m,t -x n,t ,y′ m,t -y n,t ]

[0019] b = [x n,t -x n,t-1 ,y n,t-y n,t-1 ]

[0020] Where, x n,t and y n,t Let x be the position of the nth red fighter in step t. n,t-1 and y n,t-1 Let x′ be the position of the nth red fighter in step t-1. m,t and y′ m,t Let m be the position of the blue square entity at step t;

[0021] For the m-th blue fighter jet, when the relationship with the red entity n satisfies the following condition, the m-th blue fighter jet can detect the target and use the red entity's position to construct observation data:

[0022]

[0023]

[0024] a = [x n,t -x′ m,t ,y n,t -y′ m,t ]

[0025] b = [x' m,t -x′ m,t-1 ,y′ m,t -y′ m,t-1 ]

[0026] Where, x′ m,t and y′ m,t Let x′ be the position of the m-th Blue Team fighter in step t. m,t-1 and y′ m,t-1 Let x be the position of the m-th Blue Team fighter in step t-1. n,t and y n,t Let n be the position of the red entity at step t;

[0027] Steps 1-3: Constructing Firepower Strike Rules

[0028] The fire strike range threshold is set to θ3, and the fire strike angle threshold is set to θ4. For the red team's nth fighter jet, when the relationship with the blue team's entity m satisfies the following condition, the red team's nth fighter jet can strike the target and remove the blue team entity from the scene:

[0029]

[0030]

[0031] a=[x′ m,t -x n,t ,y′ m,t -yn,t ]

[0032] b = [x n,t -x n,t-1 ,y n,t -y n,t-1 ]

[0033] Where, x n,t and y n,t Let x be the position of the nth red fighter in step t. n,t-1 and y n,t-1 Let x′ be the position of the nth red fighter in step t-1. m,t and y′ m,t Let m be the position of the blue square entity at step t;

[0034] For the m-th blue fighter jet, when the relationship between it and the red entity n satisfies the following condition, the m-th blue fighter jet can engage the target and remove the red entity from the scene:

[0035]

[0036]

[0037] a = [x n,t -x′ m,t ,y n,t -y′ m,t ]

[0038] b = [x' m,t -x′ m,t-1 ,y′ m,t -y′ m,t-1 ]

[0039] Where, x′ m,t and y′ m,t Let x′ be the position of the m-th Blue Team fighter in step t. m,t-1 and y′ m,t-1 Let x be the position of the m-th Blue Team fighter in step t-1. n,t and y n,t Let n be the position of the red entity at step t;

[0040] Steps 1-4: Constructing Electromagnetic Countermeasures Rules

[0041] The electromagnetic countermeasures distance threshold is set to θ5, and the electromagnetic countermeasures angle threshold is set to θ6. For the red jammer n, when the relationship with the blue entity m satisfies the following condition, the red jammer n can suppress the observation distance, making the observation distance threshold half of its original value:

[0042]

[0043]

[0044] a=[x′ m,t -x n,t ,y′ m,t -y n,t ]

[0045] b = [x n,t -x n,t-1 ,y n,t -y n,t-1 ]

[0046] Where, x n,t and y n,t Let x be the position of the red team's jammer n in step t. n,t-1 and y n,t-1 Let x′ be the position of the red team's jammer n in step t-1. m,t and y′ m,t Let m be the position of the blue square entity at step t;

[0047] For the blue jammer m, when the relationship between it and the red entity n satisfies the following condition, the blue jammer n can suppress the observation range, making the observation range threshold half of its original value:

[0048]

[0049]

[0050] a = [x n,t -x′ m,t ,y n,t -y′ m,t ]

[0051] b = [x' m,t -x′ m,t-1 ,y′ m,t -y′ m,t-1 ]

[0052] Where, x′ m,t and y′ m,t Let x′ be the position of the blue team's jammer m in step t. m,t-1 and y′ m,t-1 Let x be the position of the blue team's jammer m in step t-1. n,t and y n,t Let n be the position of the red entity at step t;

[0053] Steps 1-5: Constructing Mobility Rules

[0054] Except for the base, which is immovable, all entities are controlled by speed v. x and v y Perform position update; set velocity threshold to θ vThe location update rule is

[0055]

[0056]

[0057] Steps 1-6: Constructing Observational Data

[0058] According to the observation rules, the targets that each entity n can see are assembled into a list O. n Used to interact with intelligent models;

[0059] Steps 1-7: Construct the reward function

[0060] Set a zero-sum game objective: The blue team cooperates to attack the red team's base, while the red team protects its own base;

[0061] When the Red team's base is destroyed, the round ends, the done flag changes to True, the Red team's reward_r is -150, and the Blue team's reward_b is 150.

[0062] When all of the Blue side's attacking forces are eliminated, the round ends, the done flag changes to True, the Red side's reward_r is 150, and the Blue side's reward_b is -150.

[0063] When a blue unit is destroyed, the round does not end, the done flag remains False, the red team's reward_r is 75, and the blue team's reward_b is -75.

[0064] In other cases, if the round does not end, the "done" flag remains False, the blue side's reward_b is -1, and the red side's reward_r is...

[0065]

[0066] Furthermore, the specific rules for the blue team's confrontation in step 2 include:

[0067] Lateral movement of the blue team's jammer for:

[0068]

[0069] Longitudinal movement of the blue team's jammer for:

[0070]

[0071] Lateral maneuvers of the first attack aircraft of the Blue Team for:

[0072]

[0073] Longitudinal movement of Blue Team's first attack aircraft for:

[0074]

[0075] Lateral maneuvers of the Blue Team's second attack aircraft for:

[0076]

[0077] Vertical movement of the Blue Team's second attack aircraft for:

[0078]

[0079] Furthermore, the specific method for step 3 is as follows:

[0080] Step 3-1, Construct the state space:

[0081] Through a cooperative combat scenario, the observation space o1=[x′0,y′0,x′1,y′1,x′2,y′2,x′3,y′3] of the first Red fighter and the observation space o2=[x′0,y′0,x′1,y′1,x′2,y′2,x′3,y′3] of the second Red fighter are obtained and concatenated with our information o3=[x0,y0,x1,y1,x2,y2,x3,y3] to form the state space s=[o1,o2,o3].

[0082] Step 3-2, Construct a deep network:

[0083] The deep network consists of four fully connected neural networks, Net1 to Net4. Net1 extracts features from the state space; its input dimension is the same as the state space dimension *s*, and its output dimension is half the state space dimension *s*. Net2, Net3, and Net4 have identical structures. Based on the extracted state space features, Net2, Net3, and Net4 map the extracted features to form value vectors Q1, Q2, and Q3 corresponding to the entity's actions. The input dimension of this network is half the state space dimension *s*, and its output dimension is [0, θ]. v ]、[0,-θ v ]、[θ v [,0] and [-θ] v The four dimensions of [,0]:

[0084]

[0085] Step 3-3, Construct the action selection strategy:

[0086] Set the action selection threshold θ7, and randomly select a value ε from [0,1].

[0087] When ε > θ7, each entity will be in the range [0, θ v ]、[0,-θ v ]、[θ v [,0] and [-θ] v Randomly mapped from [0, 0] to the action space and

[0088] When ε≤θ7, the largest number, argmaxQ1, argmaxQ2, and argmaxQ3, is obtained from the value vectors Q1, Q2, and Q3 of each entity's action. Then, these numbers are mapped to the action space from [0,0.1], [0,-0.1], [0.1,0], and [-0.1,0]. and middle;

[0089] Steps 3-4: Construct an experience storage space:

[0090] Action space and After execution, the updated state space s′ and the red team's reward reward_r will be obtained from the scene; the state space s and the action spaces of each entity will be... and The updated state space s′ and the red team's reward reward_r are concatenated into an experience vector exp and stored in the variable storage;

[0091] Steps 3-5: Construct the value difference calculation formula and update the neural network parameters:

[0092] The empirical vector exp is randomly read from the variable storage, the loss is calculated according to the value difference formula, and the parameters of the four fully connected neural networks Net1-Net4 are updated using the Adam algorithm in the PyTorch deep learning framework.

[0093] The value difference calculation formula for loss is:

[0094]

[0095] in, This represents selecting the entity action value vectors Q1, Q2, and Q3 obtained by all agents from the updated state space s′, where the largest Q value is chosen from these vectors. n (s′) and sum them to form the group value; This represents the selection of action a from the value vectors Q1, Q2, and Q3 of each entity's actions obtained by all agents from the current state s. n The corresponding value is calculated, and the summation yields the current group value; η is the attenuation coefficient.

[0096] Furthermore, the specific method for training the heterogeneous multi-agent reinforcement learning model of the red side in step 4 is as follows:

[0097] Step 4-1: Set the maximum number of rounds in the battle (Num_Episode)

[0098] Step 4-2: Set the maximum step size per round, Num_Step

[0099] Step 4-3: Execute steps 3-3, 3-4, and 3-5, and increment the step counter by 1; when the step counter reaches the maximum step size Num_Step, the counter step is cleared to zero, and the round counter epi is incremented by 1; when the round counter epi reaches the maximum number of rounds Num_Episode, training terminates.

[0100] The beneficial effects of this invention are as follows:

[0101] 1. This invention constructs a collaborative adversarial scenario for generating adversarial strategies, and provides a detailed blue side adversarial rule strategy model. It also innovatively proposes a multi-agent reinforcement learning deep network model, which, after training, can generate intelligent collaborative adversarial strategies based on heterogeneous UAV swarms.

[0102] 2. This invention enables heterogeneous drone swarms carrying different mission payloads to achieve cooperative combat missions, thereby improving the level of cooperation among heterogeneous drone swarms. Attached Figure Description

[0103] Figure 1 This is a flowchart of a method according to an embodiment of the present invention.

[0104] Figure 2 This is a schematic diagram of a game scenario in an embodiment of the present invention.

[0105] Figure 3 This is a network structure diagram in an embodiment of the present invention.

[0106] Figure 4 This is a trajectory diagram illustrating the intelligent collaborative confrontation of a heterogeneous drone swarm in an embodiment of the present invention. Detailed Implementation

[0107] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0108] A multi-UAV cooperative penetration method based on heterogeneous multi-agent reinforcement learning, the overall process of which is as follows: Figure 1 As shown. Specifically, it includes the following steps:

[0109] Step 1: Construct a cooperative combat scenario, including the number of entities and their locations, attack rules, electromagnetic countermeasure rules, maneuver rules, and reward functions.

[0110] Step 1-1: Number of Entities and Scene Location

[0111] The dimensions of the two-dimensional rectangular adversarial scenario are the X-axis and the Y-axis. The projection range on the X-axis is [0, X...]. max ], X max =10, and its projection range on the Y-axis is [0,Y]. max ], Y max =10. In the adversarial scenario, the entities are divided into the Red Team and the Blue Team, each including 1 jammer, 2 fighter jets, and 1 base. Therefore, the total number of entities in the scenario is 8. The Red Team's jammer is located at x0 and y0, the Red Team's fighter jet 1 at x1 and y1, the Red Team's fighter jet 2 at x2 and y2, and the Red Team's base at x3 and y3. The Blue Team's jammer is located at x′0 and y′0, the Blue Team's fighter jet 1 at x′1 and y′1, the Blue Team's fighter jet 2 at x′2 and y′2, and the Blue Team's base at x′3 and y′3. A diagram of the game scenario is shown below. Figure 2 As shown.

[0112] Steps 1-2: Observation Rules

[0113] The observation distance threshold is set to θ1 = 2, and the observation angle threshold is set to θ2 = 30°.

[0114] For the red fighter jet n, when the relationship with the blue entity m satisfies the following condition, the red fighter jet n can detect the target and use the blue entity's position to construct observation data.

[0115]

[0116]

[0117] a=[x′ m,t -x n,t ,y′ m,t -y n,t ]

[0118] b = [x n,t -x n,t-1 ,y n,t -y n,t-1 ]

[0119] Where x n,t and y n,t Let x be the position of the red fighter n at step t.n,t-1 and y n,t-1 Let x′ be the position of the red fighter n in step t-1. m,t and y′ m,t Let m be the position of the blue square entity at step t.

[0120] For the blue fighter m, when the relationship with the red entity n satisfies the following condition, the blue fighter m can detect the target and use the red entity's position to construct observation data.

[0121]

[0122]

[0123] a = [x n,t -x′ m,t ,y n,t -y′ m,t ]

[0124] b = [x' m,t -x′ m,t-1 ,y′ m,t -y′ m,t-1 ]

[0125] Where x′ m,t and y′ m,t Let x′ be the position of Blue Team fighter m at step t. m,t-1 and y′ m,t-1 Let x be the position of Blue Force fighter m in step t-1. n,t and y n,t Let n be the position of the red entity n at step t.

[0126] Steps 1-3: Firepower Strike Rules

[0127] The fire strike distance threshold is set to θ3 = 2, and the fire strike angle threshold is set to θ4 = 30°.

[0128] For a red fighter jet n, when the relationship between it and the blue entity m satisfies the following condition, the red fighter jet n can strike the target and remove the blue entity from the scene.

[0129]

[0130]

[0131] a=[x′ m,t -x n,t ,y′ m,t -y n,t ]

[0132] b = [x n,t -x n,t-1 ,yn,t -y n,t-1 ]

[0133] Where x n,t and y n,t Let x be the position of the red fighter n at step t. n,t-1 and y n,t-1 Let x′ be the position of the red fighter n in step t-1. m,t and y′ m,t Let m be the position of the blue square entity at step t.

[0134] For the blue fighter m, when the relationship between it and the red entity n satisfies the following condition, the blue fighter m can strike the target and remove the red entity from the scene.

[0135]

[0136]

[0137] a = [x n,t -x′ m,t ,y n,t -y′ m,t ]

[0138] b = [x' m,t -x′ m,t-1 ,y′ m,t -y′ m,t-1 ]

[0139] Where x′ m,t and y′ m,t Let x′ be the position of Blue Team fighter m at step t. m,t-1 and y′ m,t-1 Let x be the position of Blue Force fighter m in step t-1. n,t and y n,t Let n be the position of the red entity n at step t.

[0140] Steps 1-4: Electromagnetic Countermeasures Rules

[0141] The electromagnetic countermeasure distance threshold is set to θ5 = 3, and the electromagnetic countermeasure angle threshold is set to θ6 = 30°.

[0142] For the red jammer n, when the relationship with the blue entity m satisfies the following condition, the red jammer n can suppress the observation distance threshold set to θ1, and the observation distance threshold becomes θ1←θ1 / 2.

[0143]

[0144]

[0145] a=[x′ m,t-x n,t ,y′ m,t -y n,t ]

[0146] b = [x n,t -x n,t-1 ,y n,t -y n,t-1 ]

[0147] Where x n,t and y n,t Let x be the position of the red team's jammer n in step t. n,t-1 and y n,t-1 Let x′ be the position of the red team's jammer n in step t-1. m,t and y′ m,t Let m be the position of the blue square entity at step t.

[0148] For the blue jammer m, when the relationship between it and the red entity n satisfies the following condition, the blue jammer n can suppress the observation distance threshold set to θ1, and the observation distance threshold becomes θ1←θ1 / 2.

[0149]

[0150]

[0151] a = [x n,t -x′ m,t ,y n,t -y′ m,t ]

[0152] b = [x' m,t -x′ m,t-1 ,y′ m,t -y′ m,t-1 ]

[0153] Where x′ m,t and y′ m,t Let x′ be the position of the blue team's jammer m in step t. m,t-1 and y′ m,t-1 Let x be the position of the blue team's jammer m in step t-1. n,t and y n,t Let n be the position of the red entity n at step t.

[0154] Steps 1-5: Maneuvering Rules

[0155] Except for the base, which is immovable, all entities are controlled by speed v. x and v y Perform position update; set velocity threshold to θ v =0.5, the position update rule is

[0156]

[0157]

[0158] Steps 1-6: Construction of Observational Data

[0159] According to the observation rules, the targets that each entity n can see are assembled into a list O. n It is used to interact with intelligent models.

[0160] Steps 1-7: Constructing the reward function

[0161] Set a zero-sum game objective: the blue team cooperates to attack the red team's base, while the red team protects its own base.

[0162] When the Red team's base is destroyed, the round ends, the done flag changes to True, the Red team's reward_r is -150, and the Blue team's reward_b is 150.

[0163] When all of the Blue team's attacking forces are eliminated, the round ends, the done flag changes to True, the Red team's reward_r is 150, and the Blue team's reward_b is -150.

[0164] When a blue unit is destroyed, the round does not end, the done flag remains False, the red team's reward_r is 75, and the blue team's reward_b is -75.

[0165] In other cases, if the round does not end, the "done" flag remains False, the blue side's reward_b is -1, and the red side's reward_r is...

[0166]

[0167] Step 2: Construct the blue team's adversarial strategy model separately.

[0168] Lateral movement of the blue team's jammer for:

[0169]

[0170] Longitudinal movement of the blue team's jammer for:

[0171]

[0172] Lateral movement of Blue Team Attack Aircraft 1 for:

[0173]

[0174] Vertical movement of Blue Team Attack Aircraft 1 for:

[0175]

[0176] Lateral movement of Blue Team Attack Aircraft 2 for:

[0177]

[0178] Vertical movement of Blue Team Attack Aircraft 2 for:

[0179]

[0180] Step 3: Construct a heterogeneous multi-agent reinforcement learning model for the red team.

[0181] Step 3-1: Construct the state space

[0182] In a cooperative combat scenario, the observation space o1=[x′0,y′0,x′1,y′1,x′2,y′2,x′3,y′3] of Red Team fighter 1, the observation space o2=[x′0,y′0,x′1,y′1,x′2,y′2,x′3,y′3] of Red Team fighter 2, and our information o3=[x0,y0,x1,y1,x2,y2,x3,y3] are concatenated to form the state space s=[o1,o2,o3].

[0183] Step 3-2: Constructing a Deep Network

[0184] The deep network consists of four fully connected neural networks, Net1 to Net4. Net1 extracts features from the state space; its input dimension is the same as the state space dimension *s*, and its output dimension is half the state space dimension *s*. Net2, Net3, and Net4 have identical structures. Based on the extracted state space features, Net2, Net3, and Net4 map the extracted features to form value vectors Q1, Q2, and Q3 corresponding to the entity's actions. The input dimension of this network is half the state space dimension *s*, and its output dimension is [0, θ]. v ]、[0,-θ v ]、[θ v [,0] and [-θ] v The network structure has four dimensions: [0, 1], [2], [3], [4], [5], [ Figure 3 As shown.

[0185]

[0186] Step 3-3: Constructing an action selection strategy

[0187] Set the action selection threshold to θ7, and randomly select a value ε = 0.9 from [0,1].

[0188] When ε > θ7, each entity will be in the range [0, θ v ]、[0,-θ v ]、[θ v [,0] and [-θ] v Randomly mapped from [0, 0] to the action space and

[0189] When ε≤θ7, the maximum index is obtained from the value vectors Q1, Q2, and Q3 of each entity's action, namely argmaxQ1, argmaxQ2, and argmaxQ3, and then from [0,θ7]... v ]、[0,-θ v ]、[θ v [,0] and [-θ] v The corresponding mapping in [0] to the action space and middle.

[0190] Steps 3-4: Constructing an experience storage space

[0191] Action space and After execution, the updated state space s′ and the red team's reward reward_r will be obtained from the scene. The state space s and the action spaces of each entity will then be... and The updated state space s′ and the red team's reward reward_r are concatenated into an experience vector exp and stored in storage.

[0192] Steps 3-5: Construct the value difference calculation formula and update the neural network parameters

[0193] The empirical vector exp is randomly read from storage, the value difference loss is calculated, and the parameters of the neural network are updated based on the value difference.

[0194]

[0195] in, This represents selecting the individual with the largest Q from the value vectors Q1, Q2, and Q3 of each entity action obtained by all agents from the updated state space s′. n (s′) and sum them to form the group value; This represents the selection of action a from the value vectors Q1, Q2, and Q3 of each entity's actions obtained by all agents from the current state s. n The corresponding value is calculated, and the summation yields the current group value; η is the attenuation coefficient set to 0.9.

[0196] Step 4: Construct intelligent strategy training and effect evaluation.

[0197] Step 4-1: Set the maximum number of rounds in the battle: Num_Episode = 100000

[0198] Step 4-2: Set the maximum step size per round, Num_Step = 20

[0199] Step 4-3: Training Model Generation Strategy

[0200] Execute steps 3-3, 3-4, and 3-5, and increment the step counter by 1. When the step counter reaches the maximum step size Num_Step, the step counter is reset to zero, and the round counter epi is incremented by 1. Training terminates when the round counter epi reaches the maximum number of rounds Num_Episode.

[0201] In summary, this invention constructs a cooperative adversarial scenario, targets the blue team's adversarial rule-strategy model, and builds a heterogeneous multi-agent reinforcement learning model for the red team. Through intelligent strategy training and generation, an intelligent cooperative adversarial strategy based on a heterogeneous drone swarm is formed, thereby completing the intelligent adversarial task of the heterogeneous drone swarm. The trajectory of the intelligent adversarial process of the heterogeneous drone swarm in the cooperative adversarial scenario of this invention is shown in the following example. Figure 4 As shown.

[0202] In summary, this invention addresses the problem of generating intelligent collaborative adversarial strategies, abstracts the mechanisms of electromagnetic and firepower confrontation, innovatively constructs a heterogeneous multi-agent reinforcement learning model, and achieves intelligent adversarial strategy generation through data training.

Claims

1. A method for generating intelligent cooperative adversarial strategies based on heterogeneous UAV swarms, characterized in that, Includes the following steps: Step 1: Construct a cooperative combat scenario, including the number of entities and scenario location, observation rules, firepower strike rules, electromagnetic countermeasures rules, maneuver rules, observation data, and reward function; the specific method is as follows: Step 1-1: Determine the number of entities and scene locations A two-dimensional rectangular adversarial scenario is used, and the projection range of the two-dimensional rectangular adversarial scenario on the X-axis is: The projection range on the Y-axis of the two-dimensional rectangular adversarial scene is The entities in the adversarial scenario are divided into red and blue teams, each consisting of one jammer, two fighter jets, and one base; the location of the red team's jammer is... , The position of the first Red Team fighter jet is , The position of the second Red Team fighter jet is , The location of the Red Team's base is , The location of the blue team's jammer is , The position of the first Blue Team fighter jet is , The position of the Blue Team's second fighter jet is , The location of the Blue Team's base is , ; Step 1-2: Constructing Observation Rules Observation distance threshold set to The observation angle threshold is set to For the nth red fighter jet, when the relationship between it and the blue entity m satisfies the following condition, the nth red fighter jet can detect the target and use the blue entity's position to construct observation data: in, and Let t be the position of the nth red fighter jet. and Let t-1 be the position of the nth red fighter. and Let m be the position of the blue square entity at step t; For the m-th blue fighter jet, when the relationship with the red entity n satisfies the following condition, the m-th blue fighter jet can detect the target and use the red entity's position to construct observation data: in, and Let t be the position of the m-th Blue Team fighter jet. and This represents the position of the m-th Blue Team fighter jet in step t-1. and Let n be the position of the red entity at step t; Steps 1-3: Constructing Firepower Strike Rules Firepower strike range threshold set The firepower strike angle threshold is set to For the nth red fighter jet, when the relationship between it and the blue entity m satisfies the following condition, the nth red fighter jet can attack the target and remove the blue entity from the scene: in, and Let t be the position of the nth red fighter jet. and Let t-1 be the position of the nth red fighter. and Let m be the position of the blue square entity at step t; For the m-th blue fighter jet, when the relationship between it and the red entity n satisfies the following condition, the m-th blue fighter jet can engage the target and remove the red entity from the scene: in, and Let t be the position of the m-th Blue Team fighter jet. and This represents the position of the m-th Blue Team fighter jet in step t-1. and Let n be the position of the red entity at step t; Steps 1-4: Constructing Electromagnetic Countermeasures Rules Electromagnetic countermeasures distance threshold set Electromagnetic countermeasure angle threshold set to For the red jammer n, when the relationship between the red jammer n and the blue entity m satisfies the following condition, the red jammer n can suppress the observation range, making the observation range threshold half of its original value: in, and Let n be the position of the red team's jammer in step t. and Let n be the position of the red team's jammer in step t-1. and Let m be the position of the blue square entity at step t; For the blue jammer m, when the relationship between it and the red entity n satisfies the following condition, the blue jammer n can suppress the observation range, making the observation range threshold half of its original value: in, and The position of the blue team's jamming device m is given by step t. and The position of the blue team's jammer m in step t-1. and Let n be the position of the red entity in step t; Steps 1-5: Constructing Mobility Rules Except for the base, which is immovable, all entities are controlled by speed. and Perform position update; set speed threshold to The location update rule is Steps 1-6: Constructing Observational Data Based on the observation rules, the targets that each entity n can see are assembled into a list. Used to interact with intelligent models; Steps 1-7: Construct the reward function Set a zero-sum game objective: The blue team cooperates to attack the red team's base, while the red team protects its own base; When the Red team's base is destroyed, the round ends, the done flag changes to True, the Red team's reward_r is -150, and the Blue team's reward_b is 150. When all of the Blue side's attacking forces are eliminated, the round ends, the done flag changes to True, the Red side's reward_r is 150, and the Blue side's reward_b is -150. When a blue unit is destroyed, the round does not end, the done flag remains False, the red team's reward_r is 75, and the blue team's reward_b is -75. In other cases, if the round does not end, the "done" flag remains False, the blue side's reward_b is -1, and the red side's reward_r is... ； Step 2: Construct the blue team's adversarial rules, including the horizontal and vertical actions of each blue team entity; the blue team's adversarial rules specifically include: Lateral movement of the blue team's jammer for: Longitudinal movement of the blue team's jammer for: Lateral maneuvers of the first attack aircraft of the Blue Team for: Longitudinal movement of Blue Team's first attack aircraft for: Lateral maneuvers of the Blue Team's second attack aircraft for: Vertical movement of the Blue Team's second attack aircraft for: ； Step 3: Construct a heterogeneous multi-agent reinforcement learning model for the red team; the specific method is as follows: Step 3-1, Construct the state space: By conducting cooperative combat scenarios, the observation space of the Red Team's first fighter jet was obtained. The observation space of the second fighter jet of the Red Team and information from our side The components are spliced ​​together to form a state space. ; Step 3-2, Construct a deep network: The deep network consists of four fully connected neural networks, Net1 to Net4. Net1 is used to extract features from the state space, and its input dimension is the same as that of the state space. Maintaining consistent dimensions, the output dimension is the state space. Half the dimension; Net2, Net3, and Net4 have the same structure, and Net2, Net3, and Net4 form the value vector of the corresponding entity action based on the extracted state space feature mapping. , and The input dimension of this network is the state space. Half the dimension, the output dimension is [0, ]、[0, - ]、[ , 0] and [- The four dimensions of [, 0]: Step 3-3, Construct the action selection strategy: Set action selection threshold Randomly select a value from [0, 1]. ; when At that time, each entity will be from [0, ]、[0, - ]、[ , 0] and [- Randomly mapped from [0, 0] to the action space [ , ]、[ , ]and[ , ]; when At that time, from the value vector of each entity's action , and The largest number is obtained from the middle, that is , and Then, the values ​​from [0, 0.1], [0, -0.1], [0.1, 0], and [-0.1, 0] are mapped to the action space. , ]、[ , ]and[ , ]middle; Steps 3-4: Construct an experience storage space: Action space =[ , ]、 = [ , ]and = [ , After execution, the updated state space will be obtained from the scene. And the red team's reward_r; [This will change the state space] Action space of each entity [ , ]、[ , ]and[ , Updated state space The experience vector is formed by concatenating the reward_r of the red team with the experience vector. And store it in the variable storage; Steps 3-5: Construct the value difference calculation formula and update the neural network parameters: Randomly read the experience vector from the variable storage. Based on the loss formula for value difference, the Adam algorithm of the PyTorch deep learning framework is used to update the parameters of the four fully connected neural networks Net1-Net4. The value difference calculation formula for loss is: in, This indicates that all agents are in the updated state space. The value vector of each entity's action. , and Among them, select the largest individual. And sum up to form group value; This indicates that all agents are in the current state. The value vector of each entity's action. , and In the middle, choose their respective actions. The corresponding value is calculated, and the summation yields the current group value. The attenuation coefficient; Step 4: Based on the cooperative adversarial scenario and the blue side's adversarial rules, train the red side's heterogeneous multi-agent reinforcement learning model to generate intelligent cooperative adversarial strategies.

2. The method for generating intelligent cooperative adversarial strategies based on heterogeneous UAV swarms according to claim 1, characterized in that, The specific method for training the heterogeneous multi-agent reinforcement learning model of the red side in step 4 is as follows: Step 4-1: Set the maximum number of rounds in the battle (Num_Episode) Step 4-2: Set the maximum step size per round, Num_Step Step 4-3: Execute steps 3-3, 3-4, and 3-5, and increment the step counter by 1; when the step counter reaches the maximum step size Num_Step, the counter step is cleared to zero, and the round counter epi is incremented by 1; when the round counter epi reaches the maximum number of rounds Num_Episode, training terminates.

Images