Multi-uav cooperative countermeasure decision method and system

Abstract

This invention relates to a method and system for multi-UAV cooperative adversarial decision-making. The method includes the following steps: constructing a multi-UAV cooperative air combat adversarial decision-making environment by establishing a multi-UAV air combat adversarial motion model and an air combat situation assessment model; establishing a distributed partially observable Markov decision process model for the multi-UAV cooperative adversarial decision-making problem based on the action space, local observations, and states of each UAV in the adversarial decision-making environment; designing a multi-UAV cooperative adversarial reward function and a HASAC algorithm network space; and training and generating a multi-UAV cooperative adversarial strategy model based on the interaction between the HASAC algorithm network space and the multi-UAV cooperative adversarial decision-making environment. This invention addresses the multi-UAV cooperative air combat adversarial problem by designing specific global states for the UAVs involved in multi-UAV cooperative adversarial decision-making. Compared to directly concatenating the observation vectors of each UAV, this reduces the dimensionality of the global state and improves training efficiency.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of multi-unmanned aerial vehicle decision-making, and particularly relates to a multi-unmanned aerial vehicle cooperative countermeasure decision-making method and system. BACKGROUND

[0002] The purpose of unmanned aerial vehicle countermeasure decision-making is to obtain a superior air combat situation through real-time decision-making in a high-dynamic situation air combat environment, to have a position condition for implementing strikes on the enemy, and to destroy the use condition of the enemy weapons. The combat capability of a single unmanned aerial vehicle is limited, and multi-vehicle cooperative countermeasure decision-making can complete air combat tasks through mutual cooperation and mutual cooperation of unmanned aerial vehicles, and become the main scenario of future combat.

[0003] Unmanned aerial vehicle countermeasure decision-making algorithms can be divided into algorithms based on expert knowledge, game theory, optimization theory, and deep reinforcement learning. Among them, the rule base designed based on expert knowledge has high modeling cost and poor scalability, and cannot meet the demand of intelligent combat; the algorithms based on game theory and optimization theory are mostly used in one-to-one unmanned aerial vehicle countermeasure decision-making problems, and the solving time increases explosively with the increase of the number of unmanned aerial vehicles, leading to difficulty in solving and poor real-time performance. The current multi-vehicle cooperative countermeasure decision-making problem is often divided into target allocation and single-vehicle countermeasure decision-making, and the combat scenario is often a simple two-dimensional scene, which has poor applicability.

[0004] Deep reinforcement learning algorithm can generate real-time and accurate unmanned aerial vehicle countermeasure decision-making method without expert experience, but it is currently mostly used in single-vehicle countermeasure problems. Compared with single-vehicle countermeasure problems, multi-vehicle cooperative countermeasure is a multi-agent system, and the strategy learning is not only related to the agent itself, but also related to the strategy of the teammates and opponents, and there are difficulties such as non-stationarity, high state-action dimension, partial observability, and cooperative exploration. Based on the centralized training and distributed execution framework, the non-stationarity problem can be alleviated. The multi-agent reinforcement learning algorithm based on parameter sharing can only be used in homogeneous agent problems, although it alleviates the computational burden of solving, but hinders the learning of joint strategies between agents, and the exploratory nature of the solution space is insufficient, which may converge to a suboptimal Nash equilibrium. The heterogeneous multi-agent reinforcement learning algorithm based on the introduction of maximum entropy can be applied to heterogeneous unmanned aerial vehicles, has strong exploratory nature, avoids falling into a suboptimal Nash equilibrium, and generates a real-time and accurate multi-vehicle cooperative countermeasure strategy model. SUMMARY

[0005] In order to overcome the above problems existing in the prior art, the present application provides a multi-unmanned aerial vehicle cooperative countermeasure decision-making method and system, a multi-vehicle cooperative air combat decision-making method based on HASAC (Heterogeneous-Agent Soft Actor-Critic) algorithm, which is based on multi-agent reinforcement learning and can realize real-time and dynamic decision-making of heterogeneous multi-unmanned aerial vehicles, and is used to solve the above problems existing in the prior art.

[0006] A multi-UAV cooperative adversarial decision-making method, the method comprising the following steps:

[0007] Step 1. Construct a multi-UAV collaborative air combat decision-making environment by establishing a multi-UAV air combat confrontation motion model and an air combat situation assessment model;

[0008] Step 2. Based on the action space and local observations and states of each UAV in the adversarial decision-making environment, establish a distributed partially observable Markov decision process model for the multi-UAV cooperative adversarial decision-making problem;

[0009] Step 3. Design the multi-machine cooperative adversarial reward function and the HASAC algorithm network space;

[0010] Step 4. Based on the interaction between the HASAC algorithm network space and the multi-UAV cooperative adversarial decision-making environment, train and generate a multi-UAV cooperative adversarial strategy model. The multi-UAV includes multiple UAVs of our side and multiple UAVs of the enemy, where our side is the red side and the enemy is the blue side.

[0011] In addition to the aspects and any possible implementations described above, a further implementation is provided, wherein step 1 specifically includes:

[0012] Step 11. Analyze the forces acting on each drone and establish a motion model of the mass point;

[0013] Step 12. Analyze the interrelationships between multiple UAVs and establish a relative motion model for the multiple UAVs;

[0014] Step 13. Establish an air combat situation assessment model for UAVs based on their angle, speed, altitude, and distance.

[0015] In addition to the aspects and any possible implementations described above, a further implementation is provided in which the particle motion model includes kinematic and dynamic equations, specifically as follows:

[0016]

[0017]

[0018] In the formula, The UAV in inertial coordinate system The coordinates of the axis, the magnitude of the velocity, the inclination angle of the track, the deflection angle of the track, and the acceleration due to gravity; For the drone along the direction of its flight speed, this represents the drone's tangential overload; The vertical vector and the flight velocity vector represent the normal overload of the UAV. This indicates the roll angle of the drone around its velocity axis.

[0019] In addition to the aspects described above and any possible implementations, a further implementation is provided in which the relative motion model of the UAV is established as follows:

[0020]

[0021]

[0022]

[0023]

[0024]

[0025]

[0026]

[0027]

[0028]

[0029]

[0030]

[0031]

[0032]

[0033]

[0034]

[0035]

[0036]

[0037] In the formula, The set of numbers for the Red Team's drones. , The set of numbers for the blue team's drones. Red team's first UAV velocity vector , Blue Square UAV velocity vector , Red Team No. The drone's teammate drone velocity vector , ,Right now , indicating the teammate's number; For the red team A drone and the blue team A drone in Distance between axes; Indicates the first Red team drone and Red teammate drone exist Distance between axes; Red side's first A drone and the blue team relative position vector of the drone Red Team No. The drone and the red team's drone relative position vector Red team's first A drone and the blue team Deflection angle of the drone For the red team UAV velocity vector and relative position vector The included angle; the red team's first A drone and the blue team Departure angle of the drone For the blue team UAV velocity vector and relative position vector The included angle; the red team's first A drone and its teammate drone deviation angle For the red team UAV velocity vector and relative position vector The included angle; the red team's first A drone and its teammate drone breakaway angle drones for teammates velocity vector and relative position vector The included angle, Indicates the red team's first The drone relative to the blue team's first The relative distance between the drones Indicates the red team's first The drone relative to its teammate drone The magnitude of the relative distance, They represent the red team's first... The speed, trajectory tilt angle, and trajectory deviation angle of the drone. They represent the blue team's number 1 and 2 respectively. The speed, trajectory tilt angle, and trajectory deviation angle of the drone. They represent the red team's first... drone teammate drone Speed ​​magnitude, track inclination angle, track deviation angle; They represent the red team's first... The position of the drone in the inertial coordinate system. They represent the blue team's number 1 and 2 respectively. The position of the drone in the inertial coordinate system. They represent the red team's first... drone teammate drone Position in the inertial coordinate system.

[0038] In addition to the aspects and any possible implementations described above, a further implementation is provided, wherein step 13 specifically includes: converting multi-machine cooperative confrontation into target allocation and single-machine confrontation, wherein the target allocation is based on situation assessment, and in the case of mutual threat between the red and blue teams, the allocated target poses the least threat to the red team, while the red team poses a greater threat than the target. The expression for the situation assessment is as follows:

[0039]

[0040] in, For the red team The drone relative to the blue team's first Situational assessment values ​​of the drone. , , , They represent the red team's first... The drone relative to the blue team's first The situational advantages of using drones in terms of angle, altitude, speed, and distance. For the corresponding weights, satisfying .

[0041] As described above and in any possible implementation, a further implementation is provided, wherein step 2 specifically includes: the multi-UAV cooperative adversarial decision-making model is implemented based on a distributed partially observable Markov decision process, wherein the distributed partially observable Markov process uses tuples. To describe, among which, express A collection of drones belonging to the Red Team; This is the state space of the Red Team's drone; It is the joint action space for all Red Team drones. Indicates the red team's first The operational space of the drone ; It is the red team's first Local observations from a global perspective using a drone. It is the joint reward function for all red team drones cooperating against the blue team. It is a state transition function. It is a discount factor; Red team's first Local observations by the drone included the Red Team's... Information about the drone itself, the blue team's drone, and teammates.

[0042] As described above, and in any possible implementation, a further implementation is provided, wherein the design of the multi-machine cooperative adversarial reward function is... The sum of the rewards earned by each of the Red Team's drones against the Blue Team is, i.e. ,in, Indicates the red team's first The reward for deploying a drone to fight against all blue team drones.

[0043] In addition to the aspects and any possible implementations described above, a further implementation is provided, wherein the establishment of the HASAC algorithm network space specifically includes: adopting a centralized training distributed execution framework, including n policy networks, two value networks and two target value networks, with each red team UAV corresponding to one policy network, each policy network having the same structure and being independent of each other, used to approximate the UAV decision model, and the value networks being used to evaluate the quality of the policy networks' actions under given observations.

[0044] In addition to the aspects and any possible implementations described above, a further implementation is provided, wherein step 4 specifically includes: taking the observations of each red team UAV at the current moment as the input to the policy network of each red team UAV, outputting the actions of each red team UAV under the current observations, and simultaneously returning the observations, global states, and joint rewards of each red team UAV at the next moment from the interactive environment; storing the observations and global states of each red team UAV at the current moment, the joint actions of each red team UAV, and the observations and global states and joint rewards of each red team UAV at the next moment into an experience pool connected to the HASAC algorithm network space.

[0045] The present invention also provides a multi-UAV cooperative combat decision-making system, the system being used to implement the method, the system comprising:

[0046] The module is used to build a multi-UAV collaborative air combat decision-making environment by establishing a multi-UAV air combat confrontation motion model and an air combat situation assessment model;

[0047] The first module is used to establish a distributed partially observable Markov decision process model for the multi-UAV cooperative adversarial decision problem based on the action space, local observations and states of each UAV in the adversarial decision environment.

[0048] The second module is used to design the multi-machine cooperative adversarial reward function and the network space of the HASAC algorithm.

[0049] The generation module is used to train and generate a multi-drone cooperative adversarial strategy model based on the interaction between the HASAC algorithm network space and the multi-drone cooperative adversarial decision-making environment. The multi-drone includes multiple friendly drones and multiple enemy drones, with the friendly side as the red side and the enemy side as the blue side.

[0050] Compared with the prior art, the present invention has the following beneficial effects:

[0051] This invention proposes a multi-aircraft cooperative adversarial decision-making method based on heterogeneous multi-agent reinforcement learning, building upon UAV motion modeling and air combat situation assessment. First, a three-degree-of-freedom UAV point mass model and situation assessment model are established. Second, a multi-aircraft cooperative adversarial decision-making model is established based on a distributed partially observable Markov decision process model, designing the actions, states, observations, reward functions, and network for multi-aircraft cooperative adversarial operations. Finally, the HASAC algorithm is used as the heterogeneous multi-agent reinforcement learning algorithm to train the network, generating the multi-aircraft cooperative adversarial decision-making model. This method offers the following advantages:

[0052] (1) This invention addresses the problem of multi-aircraft cooperative air combat. It designs a specific global state for the decision-making agent of multi-aircraft cooperative combat. Compared with directly splicing the observation vectors of each agent, it reduces the dimension of the global state and improves training efficiency.

[0053] (2) This invention uses the HASAC algorithm as a heterogeneous multi-agent reinforcement learning algorithm, introduces the maximum entropy, increases the randomness of action exploration, avoids falling into suboptimal Nash equilibrium, and adopts sequential update of the policy network of each agent. The trained policy network forms a joint policy to realize real-time decision-making for multi-machine collaborative adversarial. Attached Figure Description

[0054] Figure 1 This is a schematic diagram of the multi-machine collaborative adversarial decision-making framework of the present invention;

[0055] Figure 2 This is a schematic diagram illustrating the relative motion relationships of the multiple unmanned aerial vehicles (UAVs) according to the present invention.

[0056] Figure 3 This is a diagram of the heterogeneous multi-agent reinforcement learning framework for multi-machine cooperative adversarial decision-making in this invention;

[0057] Figure 4 This is a schematic diagram of the strategy network structure of the present invention;

[0058] Figure 5 This is a schematic diagram of the value network structure of the present invention;

[0059] Figure 6This is a schematic diagram of the multi-machine cooperative combat simulation curves (equilibrium situation) of the present invention;

[0060] Figure 7 This is a schematic diagram of the multi-machine cooperative combat simulation curve (advantageous situation) of the present invention;

[0061] Figure 8 This is a schematic diagram of the multi-machine cooperative combat simulation curve (disadvantageous situation) of the present invention. Detailed Implementation

[0062] To better understand the technical solution of this invention, the content of this invention includes, but is not limited to, the specific embodiments described below. Similar technologies and methods should be considered within the scope of protection of this invention. To make the technical problems to be solved, the technical solutions, and advantages of this invention clearer, a detailed description will be provided below in conjunction with the accompanying drawings and specific embodiments.

[0063] It should be understood that the embodiments described in this invention are merely some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without inventive effort are within the scope of protection of this invention.

[0064] The terminology used in the embodiments of this invention is for the purpose of describing particular embodiments only and is not intended to limit the invention. The singular forms “a,” “the,” and “the” as used in the embodiments of this invention and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.

[0065] This invention provides a multi-UAV cooperative combat decision-making method, the method comprising the following steps:

[0066] Step 1. Construct a multi-UAV collaborative air combat decision-making environment by establishing a multi-UAV air combat confrontation motion model and an air combat situation assessment model;

[0067] Step 2. Based on the action space, local observations, and states of each UAV in the adversarial decision-making environment, establish a distributed partially observable Markov decision process model for the multi-UAV cooperative adversarial decision-making problem;

[0068] Step 3. Design the multi-machine cooperative adversarial reward function and the HASAC algorithm network space;

[0069] Step 4. Based on the interaction between the HASAC algorithm network space and the multi-UAV cooperative adversarial decision-making environment, train and generate a multi-UAV cooperative adversarial strategy model. The multi-UAV includes multiple UAVs of our side and multiple UAVs of the enemy, where our side is the red side and the enemy is the blue side.

[0070] Preferably, step 1 specifically includes:

[0071] Step 11. Analyze the forces acting on each drone and establish a motion model of the mass point;

[0072] Step 12. Analyze the interrelationships between multiple UAVs and establish a relative motion model for the multiple UAVs;

[0073] Step 13. Establish an air combat situation assessment model for UAVs based on their angle, speed, altitude, and distance.

[0074] Preferably, the particle motion model includes kinematic and dynamic equations, specifically as follows:

[0075]

[0076]

[0077] In the formula, The UAV in inertial coordinate system The coordinates of the axis, the magnitude of the velocity, the inclination angle of the track, the deflection angle of the track, and the acceleration due to gravity; For the drone along the direction of its flight speed, this represents the drone's tangential overload; The vertical vector and the flight velocity vector represent the normal overload of the UAV. This indicates the roll angle of the drone around its velocity axis.

[0078] Preferably, the relative motion model of the UAV is established as follows:

[0079]

[0080]

[0081]

[0082]

[0083]

[0084]

[0085]

[0086]

[0087]

[0088]

[0089]

[0090]

[0091]

[0092]

[0093]

[0094]

[0095]

[0096] In the formula, The set of numbers for the Red Team's drones. , The set of numbers for the blue team's drones. Red team's first UAV velocity vector , Blue Square UAV velocity vector , Red Team No. The drone's teammate drone velocity vector , ,Right now , indicating the teammate's number; For the red team A drone and the blue team A drone in Distance between axes; Indicates the first Red team drone and Red teammate drone exist Distance between axes; Red side's first A drone and the blue team relative position vector of the drone Red Team No. The drone and the red team's drone relative position vector Red team's first A drone and the blue team Deflection angle of the drone For the red team UAV velocity vector and relative position vector The included angle; the red team's first A drone and the blue team Departure angle of the drone For the blue team UAV velocity vector and relative position vector The included angle; the red team's first A drone and its teammate drone deviation angle For the red team UAV velocity vector and relative position vector The included angle; the red team's first A drone and its teammate drone breakaway angle drones for teammates velocity vector and relative position vector The included angle, Indicates the red team's first The drone relative to the blue team's first The relative distance between the drones Indicates the red team's first The drone relative to its teammate drone The magnitude of the relative distance, They represent the red team's first... The speed, trajectory tilt angle, and trajectory deviation angle of the drone. They represent the blue team's number 1 and 2 respectively. The speed, trajectory tilt angle, and trajectory deviation angle of the drone. They represent the red team's first... drone teammate drone Speed ​​magnitude, track inclination angle, track deviation angle; They represent the red team's first... The position of the drone in the inertial coordinate system. They represent the blue team's number 1 and 2 respectively. The position of the drone in the inertial coordinate system. They represent the red team's first... drone teammate drone Position in the inertial coordinate system.

[0097] Preferably, step 13 specifically includes: converting multi-machine cooperative combat into target allocation and single-machine combat. The target allocation is based on situation assessment, ensuring that, under the condition that the red and blue teams threaten each other, the allocated target poses the least threat to the red team and the red team poses a greater threat than the target. The expression for the situation assessment is as follows:

[0098]

[0099] in, For the red team The drone relative to the blue team's first Situational assessment values ​​of the drone. , , , They represent the red team's first... The drone relative to the blue team's first The situational advantages of using drones in terms of angle, altitude, speed, and distance. For the corresponding weights, satisfying .

[0100] Preferably, step 2 specifically includes: the multi-UAV cooperative adversarial decision-making model is implemented based on a distributed partially observable Markov decision process, wherein the distributed partially observable Markov process uses tuples. To describe, among which, express A collection of drones belonging to the Red Team; This is the state space of the Red Team's drone; It is the joint action space for all Red Team drones. Indicates the red team's first The operational space of the drone ; It is the red team's first Local observations from a global perspective using a drone. It is the joint reward function for all red team drones cooperating against the blue team. It is a state transition function. It is a discount factor; Red team's first Local observations by the drone included the Red Team's... Information about the drone itself, the blue team's drone, and teammates.

[0101] Preferably, the design of the multi-machine cooperative adversarial reward function The sum of the rewards earned by each of the Red Team's drones against the Blue Team is, i.e. ,in, Indicates the red team's first The reward for deploying a drone to fight against all blue team drones.

[0102] Preferably, the establishment of the HASAC algorithm network space specifically includes: adopting a centralized training and distributed execution framework, including n policy networks, two value networks and two target value networks. Each red team UAV corresponds to one policy network. Each policy network has the same structure and is independent of each other, and is used to approximate the UAV decision model. The value network is used to evaluate the quality of the policy network's actions under given observations.

[0103] Preferably, step 4 specifically includes: taking the observations of each red team UAV at the current moment as the input to the policy network of each red team UAV, outputting the actions of each red team UAV under the current observation, and at the same time, the interactive environment returns the observations and global state of each red team UAV and the joint reward at the next moment; storing the observations and global state of each red team UAV at the current moment, the joint actions of each red team UAV, and the observations and global state and joint reward of each red team UAV at the next moment into an experience pool connected to the HASAC algorithm network space.

[0104] like Figure 1 , Figure 2 , Figure 3 , Figure 4 and Figure 5 As shown, in multi-drone cooperative air combat, based on perception information, this invention assumes that the red and blue UAVs are the opposing sides, with our side being the red side and the opponent being the blue side. Both sides can obtain the position, speed, and attitude information of themselves, their opponents, and their teammates. Only the decision-making stage is considered, performing situational assessment and maneuver decisions. The overall framework for air combat decision-making is as follows: Figure 1 As shown. Air combat decision-making is a crucial aspect of air combat, and the specific implementation process of this invention is as follows:

[0105] S1: Multi-aircraft Cooperative Air Combat Decision Environment

[0106] Assuming our drones are the red force and the opponent's drones are the blue force, we establish a single drone point motion model and a multi-drone relative motion model. Combining these with air combat situational factors, we establish an air combat situational assessment model.

[0107] S1-1 Establishing a Multi-UAV Air Combat Motion Model

[0108] a. Single UAV motion model

[0109] By simplifying and deriving the forces acting on the UAV, a three-degree-of-freedom point mass model is established, and the kinematic and dynamic equations are as follows:

[0110] (1)

[0111] (2)

[0112] In the formula, The UAV in inertial coordinate system The coordinates of the axis, the magnitude of the velocity, the inclination angle of the track, the deflection angle of the track, and the acceleration due to gravity; For the drone along the direction of its flight speed, this represents the drone's tangential overload; The vertical vector and the flight velocity vector represent the normal overload of the UAV. This indicates the roll angle of the drone around its velocity axis.

[0113] b. Multi-UAV relative motion model

[0114] In multi-machine combat, the red team has A drone, numbered set The blue team has A drone, numbered set Red team's first The relative motion relationship between the drone and the blue team's drone and the red team's teammate's drone is as follows: Figure 2 As shown. Red team's [number]th [unit / item]. UAV velocity vector ( ), Blue Square UAV velocity vector ( ), Red Team The drone's teammate drone velocity vector subscript ,Right now This indicates the number of the teammate's drone. For the red team A drone and the blue team A drone in Distance between axes; Indicates the first Red team drone and Red teammate drone exist Distance between axes; Red side's first A drone and the blue team relative position vector of the drone Red Team No. The drone and the red team's drone relative position vector Red team's first A drone and the blue team Deflection angle of the drone For the red team UAV velocity vector and relative position vector The included angle; the red team's first A drone and the blue team Departure angle of the drone For the blue team UAV velocity vector and relative position vector The included angle; the red team's first A drone and its teammate drone deviation angle For the red team UAV velocity vector and relative position vector The included angle; the red team's first A drone and its teammate drone breakaway angle drones for teammates velocity vector and relative position vector The included angle, Indicates the red team's first The drone relative to the blue team's first The relative distance between the drones Indicates the red team's first The drone relative to its teammate drone The relative distance is calculated using the following formula:

[0115] (3)

[0116] (4)

[0117] (5)

[0118] (6)

[0119] (7)

[0120] (8)

[0121] (9)

[0122] (10)

[0123] (11)

[0124] (12)

[0125] (13)

[0126] (14)

[0127] (15)

[0128] (16)

[0129] (17)

[0130] (18)

[0131] (19)

[0132] In the formula, They represent the red team's first... The speed, trajectory tilt angle, and trajectory deviation angle of the drone. They represent the blue team's number 1 and 2 respectively. The speed, trajectory tilt angle, and trajectory deviation angle of the drone. They represent the red team's first... drone teammate drone Speed ​​magnitude, track inclination angle, track deviation angle; They represent the red team's first... The position of the drone in the inertial coordinate system. They represent the blue team's number 1 and 2 respectively. The position of the drone in the inertial coordinate system. They represent the red team's first... drone teammate drone Position in the inertial coordinate system.

[0133] S1-2 Design of UAV Air Combat Situation Assessment Model

[0134] In multi-aircraft cooperative combat, multi-target allocation is achieved based on situational assessment, transforming the multi-aircraft cooperative combat problem into a target allocation and single-aircraft combat problem. Target allocation is based on situational assessment, considering the threat posed by the red team to the blue team and the threat posed by the blue team to the red team, ensuring that the allocated target poses the least threat to the user while the user poses a greater threat than the target. Situational assessment focuses on the angle, speed, altitude, and distance factors of the red and blue UAVs, calculated as follows:

[0135] (20)

[0136] in, For the red team The drone relative to the blue team's first Situational assessment values ​​of the drone. , , , They represent the red team's first... The drone relative to the blue team's first The situational advantages of using drones in terms of angle, altitude, speed, and distance. For the corresponding weights, satisfying .

[0137] The angle advantage function is designed as follows:

[0138] (twenty one)

[0139] The height advantage function is designed as follows:

[0140] (22), where, Indicates the red team's first The drone relative to the blue team's first Z-axis coordinate difference between the two drones This represents the optimal altitude advantage difference between the red and blue drones.

[0141] The speed advantage function is designed as follows:

[0142] (twenty three)

[0143] In the formula, Indicates the red team's first The drone relative to the blue team's first The speed difference between the drones This indicates the maximum and minimum speeds of the drone.

[0144] The distance advantage design is as follows:

[0145] (twenty four)

[0146] In the formula, Indicates the red team's first The drone relative to the blue team's first The relative distance between the drones This indicates the maximum and minimum range for attacks by drone-borne weapons.

[0147] In the above formulas (20)-(24) Replace with - Then the red team's number is calculated. drone teammate drone The corresponding parameters.

[0148] S2 establishes a distributed, partially observable Markov decision process model for the multi-machine cooperative adversarial decision problem, corresponding to the model in S1. Figure 3 The environment consists of two parts: the drone model and the situation assessment. In step S2, this is modeled using a distributed partially observable Markov process—modeling the entire multi-drone adversarial decision-making problem. The distributed partially observable Markov process is composed of tuples. describe.

[0149] In multi-machine cooperative adversarial decision-making problems express A collection of drones belonging to the Red Team; This is the state space of the Red Team's drone; It is the joint action space for all Red Team drones. Indicates the red team's first The operational space of the drone ; It is the red team's first Local observations from a global perspective using a drone. It is a state transition function. It is a discount factor. In time... The Red Team's drone was observing Execute action at time ( , ) is the first (Strategy of deploying Red Team drones) to obtain joint reward function and the state in the next moment and observation Joint reward function The solution is found in S3, abbreviated as R. Its joint objective is to learn the red strategy. To maximize the expected total benefit, we introduce a term maximizing entropy into the joint objective, with the objective as follows:

[0150] (25)

[0151] in, It is a temperature constant that balances rewards and maximizing entropy. Let T represent the expected value of the round, and T represent the time when the round ends. Representation strategy The entropy.

[0152] Each Red Team drone has its own local observation capabilities. observation by drones Including the red team's first Information about the drone itself Blue Team UAV Information Teammate information . Including the red team's first The speed, trajectory tilt angle, and trajectory deviation angle of the drone; Including the red team's first The relative angle and distance between the drone and each blue team drone, as well as the speed, trajectory tilt angle and trajectory deflection angle of the blue team drones; Including the red team's first The relative angle and distance between the drone and its teammates' drones, as well as the speed, inclination, and yaw of the teammates' drones, are shown in the subscript. , representing the teammate's number, as shown in the following formula:

[0153] in

[0154] ,

[0155] ,

[0156] ,

[0157] in, They represent the first The speed, trajectory tilt angle, and trajectory deviation angle of the Red Team's UAV; They represent the first The speed, trajectory tilt angle, and trajectory deviation angle of the blue team's drone; They represent the first Teammate drones of the Red Team Speed, track inclination, and track deviation; They represent the first The red team's drone and the blue team's first The deviation angle, separation angle, and relative positional deviation between the drones; Indicates the red team's first The drone and the red team's drone The deviation angle, separation angle, and relative positional deviation between them.

[0158] Global state The global state information for multi-drone collaborative combat environment is obtained by combining the observation information of each red team UAV, removing redundant information. Compared with directly splicing observations, this reduces the state dimension and helps to accelerate training to convergence.

[0159]

[0160] in, Indicates the red team's drone Remove the observation information and the red team's first The same elements were found in the observation information from the drone.

[0161] Action space Designed as a continuous action space, the Red Team's UAVs coordinate actions ,in, .

[0162] S3 Design Multi-Machine Cooperative Battle Reward Function

[0163] The multi-aircraft cooperative air combat joint reward function in step S2 The sum of the rewards earned by each of the Red Team's drones against the Blue Team is, i.e. , Indicates the red team's first The reward for a drone that fights against all blue team drones is as follows:

[0164]

[0165] in, Indicates the red team's first The drone relative to the blue team Rewards for drones, including short-term rewards. and long-term rewards ,Right now .

[0166] It is a dense reward system, designed based on situational assessment to create short-term rewards, expressed as:

[0167] (26)

[0168] in, For the red team The drone relative to the blue team's first Rewards are given based on the angle, altitude, speed, and distance of the drone. Weights are assigned to the angle, height, speed, and distance rewards.

[0169] (27)

[0170] (28)

[0171] In the formula, Indicates the red team's first The drone relative to the blue team's first The difference in z-axis coordinates between the drones.

[0172] (29)

[0173] (30)

[0174] It is a sparse reward, it is the red team's first A drone and the blue team The reward value for the end of a drone battle. The end markers for these two drone battles are divided into two groups: the red team's [number]th ... A drone hit the blue team's first The drone may be intercepted by the Blue Team. A drone hit, the Red team's first The drone exceeded the simulation boundary, and the blue team's first If the drone exceeds the simulation boundary and the maximum simulation step size per round, the calculation is as follows:

[0175] (31)

[0176] Red Team A drone hit the blue team's first The following constraints must be met when deploying a drone.

[0177] (32)

[0178] In the formula, Indicates the red team's first The drone relative to the blue team's first The relative distance between the drones This indicates the maximum and minimum range for attacks by drone-borne weapons.

[0179] The drone must satisfy the following equation within the simulation boundary:

[0180] (33)

[0181] The simulation step size per round used in formula (31) The following formula must be satisfied:

[0182] (34)

[0183] In the formula, It is the maximum simulation step size per round.

[0184] S4 Design the HASAC algorithm network space

[0185] The HASAC algorithm employs a centralized training and distributed execution framework, consisting of n policy networks, two value networks, and two target value networks, where n represents the number of red team drones. Each policy network corresponds to one red team drone. Each red team drone is connected to a policy network to approximate the drone decision-making model and generate actions based on the red team drone's local observations. Their policy network structures are identical, but each policy network learns independently, and network parameters are not shared. For example, the red team's... Strategic network for drones The input is the red team's first... A drone in Observation of time The output is the distribution of UAV decision actions under the current observations. The value network is used to approximate a given global state. All Red Team drones will conduct joint operations. The quality of the input is the state and action, and the output is the Q-value. Two value networks with identical structures are trained independently. The smaller Q-value is used for policy network updates to mitigate the Q-value overestimation problem. This is an inherent characteristic of reinforcement learning algorithms; when using the value network to estimate the Q-value, estimation errors during training or other factors can cause the learned Q-value function to overestimate the true Q-value. A target value network is used to stabilize the training process. Its network structure is identical to the value networks, and the target value network parameters are updated based on a weighted average of the value network parameters. The policy network takes observations as input and outputs decision actions based on those observations. Each policy network interacts with the environment, storing the collected training data in the experience pool. That is, the action commands of the red team's UAV are applied to the red team's UAV model. Subsequently, the entire environment performs state updates, collects data, and returns the environment's o', s', and R, which are stored in the experience pool along with the previous o, s, and a. Secondly, during the training process, calculating the loss function requires the estimated Q(s, a) value of the current state and the estimated Q(s', a') value of the next state. At this time, the estimated Q(s', a') of the next state uses s' extracted from the experience pool, and a' is generated using the policy network input o'. This part is described in detail in step S5.

[0186] S5 uses HASAC to train and generate a multi-machine cooperative adversarial decision-making model.

[0187] When performing adversarial training based on the HASAC algorithm, a centralized training and distributed execution framework is adopted. After initializing parameters such as the network and experience pool, data is collected and network parameters are updated based on the interaction between the policy network and the environment. Figure 3 As shown, the specific training process is as follows:

[0188] 1) Obtain from the interactive environment Constant observation The policy network will be used respectively Enter the red team's number observation by drones Output the red team's first... The actions of the drone All actions of the Red Team's drones constitute a joint action. ;

[0189] 2) All actions The input environment returns the observation for the next time step. and global state Combined rewards with the performance of actions Among them, actions The red team's n drones are each applied to update their state. The blue team's decision-making model transforms the multi-adversarial decision problem into a one-to-one adversarial decision problem, where each blue team drone is assigned a red team target based on target allocation. The target allocation model outputs the red team targets attacked by the blue team drones based on situation assessment, and the Minimax model outputs the decision actions of the blue team drones against the red team targets, updating the state of the blue team drones. Target allocation is based on the situation assessment model's evaluation of the threat posed by one red team drone and one blue team drone to the opponent, ensuring that the assigned red team target poses a low threat to the blue team, while the blue team drone poses a high threat to the red team target. Observations of the red team drones are extracted through observation and a global state processor. and global state value And calculate the joint reward based on the reward function. ;

[0190] 3) Put the current Observations of each Red Team drone at any given moment and global state The coordinated actions of all Red Team drones The observation of each Red Team drone at the next moment and global state Joint awards Forming tuples Stored in the experience pool;

[0191] 4) Then, randomly sample data from the experience pool. As training data for the value network, policy network, and target value network, the Adam optimizer is used to update the value network, target value network, and policy network of each red team drone.

[0192] The specific process of updating the value network is as follows: two value networks input sampled data and Output status Next action The estimated Q value and smaller The value is used as the predicted Q-value in the loss function; the two target value networks are based on the next time-step state of the sampled data. The policy network observes the input at the next time step. Output at time Predicting the estimated Q value at the next time step Based on rewards Entropy regularization term and Calculate the target Q value, and use the loss function loss to update the value network parameters, where the two value network parameters are updated independently;

[0193] Update the policy network: Randomly generate a set of n red team drones with their numbers arranged in order. Update the policy network of each red team drone in the order of arrangement. When updating the current red team drone policy network parameters, consider the previously updated red team drone policy networks. The loss function that guides the policy network update takes the smaller Q value of the estimated Q values ​​of the two value networks and the entropy regularization term as input.

[0194] Update the target value network: Optimize the parameters of the target value network through soft updates at regular intervals.

[0195] Repeat the above steps until all networks gradually converge. That is, when the reward per round does not increase significantly and the round duration does not decrease significantly over a period of time, each strategy network is used as a red team multi-drone cooperative adversarial decision-making model. Its input is the local observation of each red team drone at the current moment, and its output is the decision action executed by each red team drone under the current observation.

[0196] To facilitate understanding of the above technical solutions of the present invention, the following specific examples will be used to describe the above technical solutions of the present invention in detail.

[0197] Taking a two-on-two air combat scenario as an example, the red team uses the HASAC algorithm for adversarial training, while the blue team uses traditional decision-making algorithms, including target assignment and the Minimax algorithm. Target assignment is designed based on the situational assessment values ​​of the blue team's drones and each of the red team's drones, constructing a threat matrix between the red team and the blue team, aiming to maximize the threat to enemy targets while minimizing its own risk. This is solved using linear programming. Target assignment transforms multi-aircraft cooperative combat into one-on-one combat, and the blue team's one-on-one combat decision-making algorithm is the Minimax algorithm. A multi-aircraft cooperative adversarial decision-making framework based on heterogeneous multi-agent reinforcement learning is as follows: Figure 3 As shown.

[0198] The simulation environment parameters are set as follows: the simulation boundary is 20km*20km*10km, the parameters for both the red and blue UAVs are the same, the speed range is 80m / s to 400m / s, and the track inclination range is... The range of track deviation angle is The weapon's attack range is 150m~900m, its tangential overload range is -3g to 3g, its normal overload range is -5g to 5g, and its roll angle range around the velocity axis is... .

[0199] The training algorithm parameters are set as follows: the experience pool size is 1,000,000, the experience replay batch size is 1,000, the discount factor is 0.99, the soft update coefficient is 0.005, the maximum number of steps per round is 500, and the learning rate of each network is 0.0005.

[0200] In the first group of experiments, the initial position of the red team was even; in the second group of experiments, the initial position of the red team was advantageous; and in the third group of experiments, the initial position of the red team was disadvantageous. The specific states of the red and blue teams are shown in Table 1.

[0201] Observations in

[0202]

[0203] Right now

[0204]

[0205] Global state It was based on the local observations of two Red Team UAVs, and the data was pieced together. and Duplicate elements are removed to reduce the global state dimension. express Remove the middle Duplicate elements, express Remove the middle Duplicate elements.

[0206]

[0207] Each Red Team UAV has a 27-dimensional observation space and a 42-dimensional state space. , These represent the speed, track inclination angle, and track deviation angle of the two red team drones, respectively. , These represent the speed, trajectory inclination angle, and trajectory deviation angle of the two blue team drones, respectively. They represent the red team's first... The relative angle and relative position information between the first drone and the first drone of the blue team; Indicates the first The relative angle and relative position information between the red team's drone and the blue team's second drone; Indicates the first The relative angle and relative position information between the red team's drone and its teammates' drones, among which... Indicates the teammate's number.

[0208] , , , The parameters are shown in Table 1, relative angles , The calculation is obtained using the formulas (14), (15), (16), and (17) mentioned earlier; , , The relative position information is calculated using the formulas (6), (7), (8), (9), (10), and (11) mentioned earlier.

[0209] Each Red Team drone has a 3D motion space, and the Red Team drones coordinate their actions. ,in, .

[0210] Joint reward function

[0211]

[0212] in,

[0213]

[0214]

[0215]

[0216] When conducting adversarial training based on the HASAC algorithm, a centralized training and distributed execution framework is adopted. Each red team drone has its own policy network, and all red team drones have a centralized value network and target value network. The structures of the policy network and value network are as follows: Figure 4 and Figure 5 As shown. To alleviate the Q-value overestimation problem, two independent value networks are designed, and two target value networks are introduced to stabilize the training process.

[0217] After initializing network, experience pool, and other parameters, data is collected based on the interaction between the policy network and the environment to update network parameters. The specific training process is as follows: Figure 3 As shown:

[0218] 1) Obtain from the interactive environment Constant observation Strategy Network 1 inputs the observations of the first drone of the Red Team. Output the actions of the first drone of the Red Team. Strategy Network 2 input: observations of the second drone of the Red Team. Output the actions of the second drone of the Red Team. To form a joint action ;

[0219] 2) The action The inputs are given in the interactive environment, and the environment returns the observations for the next time step. and global state Combined rewards with the performance of actions Among them, actions The system operates on both Red 1 and Red 2 drone models to update the state of the Red drones. The Blue team's decision-making model transforms the two-on-two adversarial decision problem into a one-on-one adversarial decision problem, where each Blue drone is assigned a Red target. The target assignment model outputs the Red targets attacked by the Blue drones based on situational assessment, and the Minimax outputs the Blue drones' decision actions against the Red targets, updating the Blue drones' states. Target assignment is based on the situational assessment model's evaluation of the threat posed by one Red drone and one Blue drone to the opponent, ensuring that the assigned Red target poses a low threat to the Blue team, while the Blue drone poses a high threat to the Red target. Observations of the Red drones are extracted through observation and a global state processor. and global state value And calculate the joint reward based on the reward function. ;

[0220] 3) Then, the current The observation of the two Red Team drones at that moment and global state The joint operation of the two Red Team drones The next moment's observation of the two red team drones and global state Joint awards Forming tuples Stored in the experience pool;

[0221] 4) Then, randomly sample data from the experience pool. As training data for the value network, policy network, and target value network, the value network, target value network, and policy networks of the two red team drones are updated based on the Adam optimizer.

[0222] Updating the value network: Input sampling data for two value networks and Output status Next action The estimated Q value and smaller The predicted Q-value is used in the loss function; the two target value networks are based on the next time-step state of the sampled data. The policy network observes the input at the next time step. Output at time Predicting the estimated Q value at the next time step Based on rewards Entropy regularization term and Calculate the target Q value, and update the value network parameters using the loss function of minimizing the mean squared error between the target Q value and the predicted Q value, wherein the two value network parameters are updated independently.

[0223] Update the policy network: Randomly generate a set of red team drone numbers and update the policy networks of the two red team drones according to the order of the arrangement. When updating the current red team drone policy network parameters, the previously updated red team drone policy networks are considered. The loss function that guides the policy network update takes the smaller Q value of the estimated Q values ​​of the two value networks and the entropy regularization term as input.

[0224] Update the target value network: Optimize the parameters of the target value network through soft updates at regular intervals.

[0225] Repeat the above steps until all networks gradually converge. Each policy network is used as a red team multi-UAV cooperative adversarial decision-making model. Its input is the local observation of each red team UAV under the current environment, and its output is the action performed by each red team UAV under the current observation.

[0226] Table 1. State Quantities of UAVs for Red and Blue Teams under Three Initial Situations

[0227]

[0228] Simulation results are as follows Figure 6 , Figure 7 and Figure 8 As shown, under three basic initial situations, the red team's generation strategy model can hit all enemy drones in a short time, and the two red team drones cooperate to hit one blue drone each. In the initial balanced situation, both red and blue teams quickly approach each other, then both red team drones decelerate to turn around with the minimum turning radius and hit the two blue team drones respectively. In the initial advantageous situation, the red team accelerates to quickly approach the blue team drones, while the blue team drones, although attempting to turn, are shot down by the two red team drones due to their high turning speed and large turning radius. In the initial disadvantageous situation, the red team drones first attempt to turn around to reverse the disadvantage. At this time, the blue team drones accelerate to approach the opponent, and later, due to their high speed, they rush ahead of the red team drones. At this point, the two red team drones again perform a minimum turning maneuver, enabling them to hit the two blue team drones respectively.

[0229] As a disclosed embodiment, the present invention also provides a multi-UAV cooperative adversarial decision-making system, the system being used to implement the method, the system comprising:

[0230] The module is used to build a multi-UAV collaborative air combat decision-making environment by establishing a multi-UAV air combat confrontation motion model and an air combat situation assessment model;

[0231] The first module is used to establish a distributed partially observable Markov decision process model for the multi-UAV cooperative adversarial decision problem based on the action space, local observations and states of each UAV in the adversarial decision environment.

[0232] The second module establishes the design of the multi-machine collaborative adversarial reward function and the network space of the HASAC algorithm.

[0233] The generation module is used to train the observation of each UAV based on the HASAC training multi-UAV cooperative adversarial decision model and generate a multi-UAV cooperative adversarial strategy model. The multi-UAV includes multiple friendly UAVs and multiple enemy UAVs, where the friendly side is the red side and the enemy side is the blue side.

[0234] The foregoing description illustrates and describes several preferred embodiments of the present invention. However, as previously stated, it should be understood that the present invention is not limited to the forms disclosed herein and should not be construed as excluding other embodiments. It can be used in various other combinations, modifications, and environments, and can be altered within the scope of the inventive concept described herein through the foregoing teachings or techniques or knowledge in related fields. Any modifications and variations made by those skilled in the art that do not depart from the spirit and scope of the present invention should be within the protection scope of the appended claims.

Claims

1. A multi-UAV cooperative adversarial decision-making method, characterized in that, The multiple drones include multiple friendly drones and multiple enemy drones, where friendly drones are designated as the red team and enemy drones as the blue team. The method includes the following steps: Step 1. Construct a multi-UAV cooperative air combat decision-making environment by establishing a multi-UAV air combat confrontation motion model and an air combat situation assessment model, specifically including: Step 11. Analyze the forces acting on each drone and establish a motion model of the mass point; Step 12. Analyze the interrelationships between multiple UAVs and establish a relative motion model of the multiple UAVs. The relative motion model of the UAVs is established as follows: In the formula, The set of numbers for the Red Team's drones. , The set of numbers for the blue team's drones. Red team's first UAV velocity vector , Blue Square UAV velocity vector , Red Team No. The drone's teammate drone velocity vector , , indicating the teammate's number; For the red team A drone and the blue team A drone in Distance between axes; Indicates the red team's first The drone and the red team's drone exist Distance between axes; Red side's first A drone and the blue team relative position vector of the drone Red Team No. The drone and the red team's drone relative position vector Red team's first A drone and the blue team Deflection angle of the drone For the red team UAV velocity vector and relative position vector The included angle; the red team's first A drone and the blue team Departure angle of the drone For the blue team UAV velocity vector and relative position vector The included angle; the red team's first A drone and its teammate drone deviation angle For the red team UAV velocity vector and relative position vector The included angle; the red team's first A drone and its teammate drone breakaway angle drones for teammates velocity vector and relative position vector The included angle, Indicates the red team's first The drone relative to the blue team's first The relative distance between the drones Indicates the red team's first The drone relative to its teammate drone The magnitude of the relative distance, They represent the red team's first... The speed, trajectory tilt angle, and trajectory deviation angle of the drone. They represent the blue team's number 1 and 2 respectively. The speed, trajectory tilt angle, and trajectory deviation angle of the drone. They represent the red team's first... drone teammate drone Speed ​​magnitude, track inclination angle, track deviation angle; They represent the red team's first... The position of the drone in the inertial coordinate system. They represent the blue team's number 1 and 2 respectively. The position of the drone in the inertial coordinate system. They represent the red team's first... drone teammate drone Position in the inertial coordinate system; Step 13. Establish an air combat situation assessment model for the UAV based on its angle, speed, altitude, and distance; Step 2. Based on the action space, local observations, and states of each UAV in the adversarial decision-making environment, establish a distributed partially observable Markov decision process model for the multi-UAV cooperative adversarial decision-making problem; Step 3. Design the multi-drone cooperative adversarial reward function and the HASAC algorithm network space. The HASAC algorithm adopts a centralized training and distributed execution framework, consisting of n policy networks, two value networks, and two target value networks, where n represents the number of red team drones. Each policy network corresponds to one red team drone, and each red team drone is connected to a policy network to approximate the drone decision-making model and generate the actions made by the red team drone under local observations. The policy networks have the same structure, learn independently, and do not share network parameters. The value network is used to evaluate the quality of the policy network's actions under given observations. The target value network is used to stabilize the training process, and its network structure is the same as that of the value network. The parameter updates of the target value network are obtained by weighted averaging of the value network parameters. Step 4. Based on the interaction between the HASAC algorithm network space and the multi-UAV cooperative adversarial decision-making environment, train and generate a multi-UAV cooperative adversarial strategy model, specifically including: 1) Obtain from the interactive environment Constant observation The policy network will be used respectively Enter the red team's number observation by drones Output the red team's first... The actions of the drone All actions of the Red Team's drones constitute a joint action. ; 2) All actions The input environment returns the observation for the next time step. and global state Combined rewards with the performance of actions Among them, actions The system operates on n red team drones, updating their states. The blue team's decision-making model transforms the multi-adversarial decision problem into a one-to-one adversarial decision problem, assigning each blue team drone to a red team target. The target assignment model outputs the red team targets attacked by the blue team drones based on situational assessment. The Minimax model outputs the decision actions of the blue team drones against the red team targets, updating their states. Observations of the red team drones are extracted through observation and a global state processor. and global state value And calculate the joint reward based on the reward function. ; 3) Put the current Observations of each Red Team drone at any given moment and global state The coordinated actions of all Red Team drones The observation of each Red Team drone at the next moment and global state Joint awards Forming tuples Stored in the experience pool; 4) Then, randomly sample data from the experience pool. As training data for the value network, policy network, and target value network, the Adam optimizer is used to update the value network, target value network, and policy network of each red team drone. Repeat the above steps until all networks gradually converge. That is, when the reward per round does not increase significantly and the round duration does not decrease significantly over a period of time, each strategy network is used as a red team multi-drone cooperative adversarial decision-making model. Its input is the local observation of each red team drone at the current moment, and its output is the decision action executed by each red team drone under the current observation.

2. The multi-UAV cooperative adversarial decision-making method according to claim 1, characterized in that, The particle motion model includes kinematic and dynamic equations, specifically as follows: In the formula, The UAV in inertial coordinate system The coordinates of the axis, the magnitude of the velocity, the inclination angle of the track, the deflection angle of the track, and the acceleration due to gravity; For the drone along the direction of its flight speed, this represents the drone's tangential overload; The vertical vector and the flight velocity vector represent the normal overload of the UAV. This indicates the roll angle of the drone around its velocity axis.

3. The multi-UAV cooperative adversarial decision-making method according to claim 1, characterized in that, Step 13 specifically includes: converting multi-machine cooperative combat into target allocation and single-machine combat. The target allocation is based on situation assessment. Under the condition that the red and blue teams threaten each other, the allocated target is minimized compared to the user, and the user is more threatening than the target. The expression for the situation assessment is as follows: in, For the red team The drone relative to the blue team's first Situational assessment values ​​of the drone. , , , They represent the red team's first... The drone relative to the blue team's first The situational advantages of using drones in terms of angle, altitude, speed, and distance. For the corresponding weights, satisfying .

4. The multi-UAV cooperative adversarial decision-making method according to claim 1, characterized in that, Step 2 specifically includes: the multi-UAV cooperative adversarial decision-making model is implemented based on a distributed partially observable Markov decision process, wherein the distributed partially observable Markov process uses tuples. To describe, among which, express A collection of drones belonging to the Red Team; This is the state space of the Red Team's drone; It is the joint action space for all Red Team drones. Indicates the red team's first The operational space of the drone ; It is the red team's first Local observations from a global perspective using a drone. It is the joint reward function for all red team drones cooperating against the blue team. It is a state transition function. It is a discount factor; Red team's first Local observations by the drone included the Red Team's... Information about the drone itself, the blue team's drone, and teammates.

5. The multi-UAV cooperative adversarial decision-making method according to claim 1, characterized in that, Establish a collaborative adversarial reward function The sum of the rewards earned by each of the Red Team's drones against the Blue Team is, i.e. ,in, Indicates the red team's first The reward for deploying a drone to fight against all blue team drones.

6. A multi-UAV cooperative combat decision-making system, characterized in that, The system is used to implement the method according to any one of claims 1-5, the system comprising: The module is used to build a multi-UAV collaborative air combat decision-making environment by establishing a multi-UAV air combat confrontation motion model and an air combat situation assessment model; The first module is used to establish a distributed partially observable Markov decision process model for the multi-UAV cooperative adversarial decision problem based on the action space, local observations and states of each UAV in the adversarial decision environment. The second module is used to design the multi-machine cooperative adversarial reward function and the network space of the HASAC algorithm. The generation module is used to train and generate a multi-drone cooperative adversarial strategy model based on the interaction between the HASAC algorithm network space and the multi-drone cooperative adversarial decision-making environment. The multi-drone includes multiple friendly drones and multiple enemy drones, with the friendly side as the red side and the enemy side as the blue side.

Images