Multi-unmanned aerial vehicle cooperative confrontation decision-making method and system

Abstract

The invention relates to a multi-unmanned aerial vehicle cooperative confrontation decision-making method and system. The method comprises the following steps: constructing a multi-unmanned aerial vehicle cooperative air combat confrontation decision-making environment by establishing a multi-unmanned aerial vehicle air combat confrontation motion model and an air combat situation evaluation model; establishing a distributed partially observable Markov decision process model of the multi-unmanned aerial vehicle cooperative confrontation decision problem according to the action space and the local observation and state of each unmanned aerial vehicle in the confrontation decision environment; designing a multi-machine collaborative adversarial reward function and an HASAC algorithm network space; and training and generating a multi-unmanned-aerial-vehicle cooperative confrontation strategy model based on interaction between a network space and a multi-unmanned-aerial-vehicle cooperative confrontation decision-making environment of an HASAC algorithm. According to the method, for the multi-aircraft cooperative air combat confrontation problem, the specific global state is designed for the multi-aircraft cooperative confrontation decision-making unmanned aerial vehicle, and compared with direct splicing of observation vectors of all unmanned aerial vehicles, the global state dimension is reduced, and the training efficiency is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of multi-UAV decision-making, and in particular relates to a multi-UAV collaborative confrontation decision-making method and system. Background Art

[0002] The purpose of UAV confrontation decision-making is to make real-time decisions, conduct maneuvering flights in an air combat environment with highly dynamic changes, gain an advantageous air combat situation, and have the conditions to strike the enemy while destroying the conditions for the use of enemy weapons. The combat capability of a single UAV is limited, and multi-aircraft collaborative confrontation decision-making can complete air combat missions through the cooperation and collaboration of UAVs, which will become the main scenario for future combat.

[0003] UAV confrontation decision-making algorithms can be divided into algorithms based on expert knowledge, game theory, optimization theory, deep reinforcement learning, etc. Among them, the rule base designed based on expert knowledge has high modeling cost and poor scalability, and can no longer meet the needs of intelligent combat; algorithms based on game theory and optimization theory are mostly used in one-on-one UAV confrontation decision-making problems, and their solution time increases explosively with the increase in the number of UAVs, resulting in difficulty in solving and poor real-time performance. At present, multi-machine collaborative confrontation decision-making problems are often decomposed into target allocation and single-machine confrontation decision-making, and the combat scenarios are often simple two-dimensional scenarios with poor applicability.

[0004] Deep reinforcement learning algorithms can generate real-time and accurate decision-making methods for drone confrontation without expert experience, but they are currently mostly used in single-machine confrontation problems. Compared with single-machine confrontation problems, multi-machine collaborative confrontation is a multi-agent system. Its strategy learning is not only related to the agent itself, but also to the strategies of teammates and opponents. There are difficulties such as non-stationarity, high state-action dimension, partial observability and collaborative exploration. Based on centralized training and distributed execution framework, non-stationarity problems can be alleviated. Multi-agent reinforcement learning algorithms based on parameter sharing can only be used in homogeneous agent problems. Although it alleviates the computational burden of solving, it will hinder the learning of joint strategies between agents, lack exploration of the solution space, and may converge to suboptimal Nash equilibrium. Heterogeneous multi-agent reinforcement learning algorithms based on the introduction of maximum entropy can be applied to heterogeneous drones, have strong exploration, avoid falling into suboptimal Nash equilibrium, and generate real-time and accurate multi-machine collaborative confrontation strategy models. Summary of the invention

[0005] In order to overcome the above-mentioned problems existing in the prior art, the present invention provides a multi-UAV collaborative confrontation decision-making method and system, a multi-UAV collaborative air combat decision-making method based on the HASAC (Heterogeneous-Agent Soft Actor-Critic) algorithm, and multi-agent reinforcement learning, which can realize real-time dynamic decision-making of heterogeneous multi-UAV confrontation, and is used to solve the above-mentioned problems existing in the prior art.

[0006] A multi-UAV collaborative confrontation decision-making method, the method comprising the steps of: Step 1. Construct a multi-UAV collaborative air combat confrontation decision-making environment by establishing a multi-UAV air combat confrontation motion model and an air combat situation assessment model; Step 2. Establish a distributed partially observable Markov decision process model for the multi-UAV collaborative confrontation decision problem based on the action space, local observation, and state of each UAV in the confrontation decision environment; Step 3. Design the multi-machine collaborative adversarial reward function and HASAC algorithm network space; Step 4. Based on the interaction between the HASAC algorithm cyberspace and the multi-UAV collaborative confrontation decision-making environment, train and generate a multi-UAV collaborative confrontation strategy model, where the multi-UAVs include multiple UAVs of our side and multiple UAVs of the enemy side, where our side is the red side and the enemy side is the blue side.

[0007] According to the above aspects and any possible implementation manner, an implementation manner is further provided, wherein step 1 specifically includes: Step 11. Analyze the forces on each drone and establish a particle motion model; Step 12: Analyze the relationship between multiple UAVs and establish a relative motion model of multiple UAVs; Step 13. Establish a drone air combat situation assessment model based on the drone's angle, speed, altitude and distance.

[0008] According to the above aspects and any possible implementation, an implementation is further provided, wherein the particle motion model includes kinematic and dynamic equations, specifically as follows: In the formula, The UAV is in the inertial coordinate system. Axis coordinates, velocity magnitude, track inclination, track deviation and gravitational acceleration; It is the flight speed direction of the UAV, indicating the tangential overload of the UAV; The vertical and flight speed vector represents the normal overload of the drone. Indicates the roll angle of the drone around the velocity axis.

[0009] According to the above aspects and any possible implementation, an implementation is further provided, wherein the relative motion model of the drone is established as follows: In the formula, The number of Red Team drones, a collection of serial numbers , The number of blue drones, a set of serial numbers , Red Square Drone speed vector , , Lan Fangdi Drone speed vector , , Red Square Teammate drone The velocity vector , ,Right now , represents the teammate number; For the Red A drone and the blue square Drone in The distance between the axes; Indicates Red team drone and Red team teammate drone exist The distance between the axes; the red square A drone and the blue square The relative position vector of the drone , Red Square drone and the Red Team's drone The relative position vector ; Red Square A drone and the blue square The deviation angle of the drone For the Red The velocity vector and relative position vector of a drone The angle of the red square A drone and the blue square Departure angle of the drone For Lan Fangdi The velocity vector and relative position vector of a drone The angle of the red square A drone and its teammate drones The deviation angle For the Red The velocity vector and relative position vector of a drone The angle of the red square A drone and its teammate drones Breakaway angle Drone for teammates Velocity vector and relative position vector The angle of Indicates the red side The drone is relative to the blue side The relative distance between the two drones, Indicates the red side The drone relative to its teammate drones The relative distance, Respectively represent the red The speed, inclination and deviation of the drone. Respectively represent the blue square The speed, inclination and deviation of the drone. Respectively represent the red Drone Teammate Drone The speed, track inclination and track deviation; Respectively represent the red The position of the UAV in the inertial coordinate system, Respectively represent the blue square The position of the UAV in the inertial coordinate system, Respectively represent the red Drone Teammate Drone Position in the inertial coordinate system.

[0010] According to the above aspects and any possible implementation, an implementation is further provided, wherein step 13 specifically includes: converting the multi-machine coordinated confrontation into target allocation and single-machine confrontation, wherein the target allocation is based on situation assessment, and in the case of mutual threat between the red party and the blue party, the allocated target is least threatening to itself, and the threat itself is greater than the target. The expression of the situation assessment is as follows:

[0011] in, For the Red The drone is relative to the blue side The situation assessment value of the UAV, , , , Respectively represent the red The drone is relative to the blue side The situational advantages of the drone’s angle, height, speed, and distance, is the corresponding weight, satisfying .

[0012] According to the above aspects and any possible implementation, an implementation is further provided, wherein step 2 specifically includes: the multi-UAV cooperative confrontation decision model is implemented based on a distributed partially observable Markov decision process, and the distributed partially observable Markov process adopts a tuple To describe, express A collection of Red Army drones; is the state space of the Red Team’s drone; is the joint action space of all red drones, Indicates the red side The action space of the drone, ; It is the Red Square The local observation of a UAV in a global state, is the joint reward function of all red drones working together against the blue drone, is the state transition function, is the discount factor; The local observation of the drone includes the Red Army The drone's own information, the blue team's drone information and teammates' information.

[0013] According to the above aspects and any possible implementation, a further implementation is provided, wherein the multi-machine collaborative confrontation reward function is designed is the sum of the rewards obtained by each drone of the red team against the blue team, that is, ,in, Indicates the red side Reward for having one drone fight against all blue drones.

[0014] According to the aspects and any possible implementation methods described above, an implementation method is further provided, wherein the establishment of the HASAC algorithm network space specifically includes: adopting a centralized training distributed execution framework, including n policy networks, two value networks and two target value networks, each red team drone corresponds to a policy network, each of the policy networks has the same structure and is independent of each other, and is used to approximate the drone decision model, and the value network is used to evaluate how well the policy network performs actions under given observations.

[0015] According to the aspects and any possible implementation methods described above, an implementation method is further provided, wherein step 4 specifically includes: using the observations of each Red Team UAV at the current moment as the input of the policy network of each Red Team UAV, outputting the actions of each Red Team UAV under the observation at the current moment, and at the same time, the interactive environment returns the observations and global states of each Red Team UAV at the next moment, as well as the joint rewards; storing the observations and global states of each Red Team UAV at the current moment, the joint actions of each Red Team UAV, and the observations and global states of each Red Team UAV at the next moment, as well as the joint rewards in an experience pool connected to the HASAC algorithm network space.

[0016] The present invention also provides a multi-UAV collaborative confrontation decision-making system, which is used to implement the method described above, and includes: A construction module is used to build a multi-UAV collaborative air combat confrontation decision environment by establishing a multi-UAV air combat confrontation motion model and an air combat situation assessment model; The first building module is used to establish a distributed partially observable Markov decision process model for the multi-UAV collaborative confrontation decision-making problem based on the action space, local observations and states of each UAV in the confrontation decision-making environment; The second module is used to design the multi-machine collaborative confrontation reward function and the HASAC algorithm network space; A generation module is used to train and generate a multi-drone collaborative confrontation strategy model based on the interaction between the HASAC algorithm cyberspace and the multi-drone collaborative confrontation decision-making environment. The multi-drones include multiple drones of our side and multiple drones of the enemy, where our side is the red side and the enemy is the blue side.

[0017] Compared with the prior art, the present invention has the following beneficial effects: Based on the UAV motion modeling and air combat situation assessment, the present invention proposes a multi-machine collaborative confrontation decision-making method based on heterogeneous multi-agent reinforcement learning. First, a three-degree-of-freedom UAV particle model and a situation assessment model are established. Secondly, a multi-machine collaborative confrontation decision-making model is established based on a distributed partially observable Markov decision process model, and the actions, states, observations, and reward functions and networks of multi-machine collaborative confrontation are designed. Finally, the HASAC algorithm is used as a heterogeneous multi-agent reinforcement learning algorithm to train the network and generate a multi-machine collaborative confrontation decision-making model. It has the following beneficial effects: (1) The present invention is aimed at the problem of multi-aircraft coordinated air combat confrontation. A specific global state is designed for the multi-aircraft coordinated confrontation decision-making intelligent agent. Compared with directly splicing the observation vectors of each intelligent agent, the global state dimension is reduced and the training efficiency is improved.

[0018] (2) The present invention adopts the HASAC algorithm as the heterogeneous multi-agent reinforcement learning algorithm, introduces maximum entropy, increases the randomness of action exploration, avoids falling into suboptimal Nash equilibrium, and adopts sequential updating of the strategy network of each agent. The trained strategy network forms a joint strategy to achieve real-time decision-making in multi-machine collaborative confrontation. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is a schematic diagram of the multi-machine collaborative confrontation decision framework of the present invention; Figure 2 A schematic diagram of the relative motion relationship of multiple UAVs of the present invention; Figure 3 This is a framework diagram of the multi-machine collaborative confrontation decision-making heterogeneous multi-agent reinforcement learning of the present invention; Figure 4 It is a schematic diagram of the strategy network structure of the present invention; Figure 5 It is a schematic diagram of the value network structure of the present invention; Figure 6 It is a schematic diagram of a multi-machine collaborative confrontation simulation curve diagram (balance of power situation) of the present invention; Figure 7 It is a schematic diagram of a multi-machine collaborative confrontation simulation curve diagram (advantage situation) of the present invention; Figure 8 It is a schematic diagram of the multi-machine collaborative confrontation simulation curve diagram (disadvantage situation) of the present invention. DETAILED DESCRIPTION

[0020] In order to better understand the technical solution of the present invention, the content of the present invention includes but is not limited to the specific implementation methods described below, and similar technologies and methods should be considered to be within the scope of protection of the present invention. In order to make the technical problems, technical solutions and advantages to be solved by the present invention clearer, the following will be described in detail with reference to the accompanying drawings and specific embodiments.

[0021] It should be clear that the embodiments described in the present invention are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0022] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit the present invention. The singular forms "a", "said" and "the" used in the embodiments of the present invention and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings.

[0023] The present invention provides a multi-UAV collaborative confrontation decision-making method, the method comprising the steps of: Step 1. Construct a multi-UAV collaborative air combat confrontation decision-making environment by establishing a multi-UAV air combat confrontation motion model and an air combat situation assessment model; Step 2. Establish a distributed partially observable Markov decision process model for the multi-UAV collaborative confrontation decision-making problem based on the action space, local observations and states of each UAV in the confrontation decision-making environment; Step 3. Design the multi-machine collaborative adversarial reward function and HASAC algorithm network space; Step 4. Based on the interaction between the HASAC algorithm cyberspace and the multi-UAV collaborative confrontation decision-making environment, train and generate a multi-UAV collaborative confrontation strategy model, where the multi-UAVs include multiple UAVs of our side and multiple UAVs of the enemy side, where our side is the red side and the enemy side is the blue side.

[0024] Preferably, the step 1 specifically includes: Step 11. Analyze the forces on each drone and establish a particle motion model; Step 12: Analyze the relationship between multiple UAVs and establish a relative motion model of multiple UAVs; Step 13. Establish a drone air combat situation assessment model based on the drone's angle, speed, altitude and distance.

[0025] Preferably, the particle motion model includes kinematic and dynamic equations, specifically as follows: In the formula, The UAV is in the inertial coordinate system. Axis coordinates, velocity magnitude, track inclination, track deviation and gravitational acceleration; It is the flight speed direction of the UAV, indicating the tangential overload of the UAV; The vertical and flight speed vector represents the normal overload of the drone. Indicates the roll angle of the drone around the velocity axis.

[0026] Preferably, the relative motion model of the drone is established as follows: In the formula, The number of Red Team drones, a collection of serial numbers , The number of blue drones, a set of serial numbers , Red Square Drone speed vector , , Lan Fangdi Drone speed vector , , Red Square Teammate drone The velocity vector , ,Right now , represents the teammate number; For the Red A drone and the blue square Drone in The distance between the axes; Indicates Red team drone and Red team teammate drone exist The distance between the axes; the red square A drone and the blue square The relative position vector of the drone , Red Square drone and the Red Team's drone The relative position vector ; Red Square A drone and the blue square The deviation angle of the drone For the Red The velocity vector and relative position vector of a drone The angle of the red square A drone and the blue square Departure angle of the drone For Lan Fangdi The velocity vector and relative position vector of a drone The angle of the red square A drone and its teammate drones The deviation angle For the Red The velocity vector and relative position vector of a drone The angle of the red square A drone and its teammate drones Breakaway angle Drone for teammates Velocity vector and relative position vector The angle of Indicates the red side The drone is relative to the blue side The relative distance between the two drones, Indicates the red side The drone relative to its teammate drones The relative distance, Respectively represent the red The speed, inclination and deviation of the drone. Respectively represent the blue square The speed, inclination and deviation of the drone. Respectively represent the red Drone Teammate Drone The speed, track inclination and track deviation; Respectively represent the red The position of the UAV in the inertial coordinate system, Respectively represent the blue square The position of the UAV in the inertial coordinate system, Respectively represent the red Drone Teammate Drone Position in the inertial coordinate system.

[0027] Preferably, step 13 specifically includes: converting the multi-machine coordinated confrontation into target allocation and single-machine confrontation, wherein the target allocation is based on situation assessment, and in the case of mutual threat between the red party and the blue party, the allocated target poses the least threat to itself, and the threat itself poses a greater threat to the target. The expression of the situation assessment is as follows:

[0028] in, For the Red The drone is relative to the blue side The situation assessment value of the UAV, , , , Respectively represent the red The drone is relative to the blue side The situational advantages of the drone’s angle, height, speed, and distance, is the corresponding weight, satisfying .

[0029] Preferably, the step 2 specifically includes: the multi-UAV cooperative confrontation decision model is implemented based on a distributed partially observable Markov decision process, and the distributed partially observable Markov process adopts a tuple To describe, among them, express A collection of Red Army drones; is the state space of the Red Team’s drone; is the joint action space of all red drones, Indicates the red side The action space of the drone, ; It is the Red Square The local observation of a UAV in a global state, is the joint reward function of all red drones working together against the blue drone, is the state transition function, is the discount factor; The local observation of the drone includes the Red Army The drone's own information, the blue team's drone information and teammates' information.

[0030] Preferably, the design of multi-machine collaborative confrontation reward function is the sum of the rewards obtained by each drone of the red team against the blue team, that is, ,in, Indicates the red side Reward for having one drone fight against all blue drones.

[0031] Preferably, the establishment of the HASAC algorithm network space specifically includes: adopting a centralized training distributed execution framework, including n policy networks, two value networks and two target value networks, each red team drone corresponds to a policy network, each of the policy networks has the same structure and is independent of each other, and is used to approximate the drone decision model, and the value network is used to evaluate the quality of the policy network's execution of actions under given observations.

[0032] Preferably, step 4 specifically includes: using the observation of each Red UAV at the current moment as the input of the policy network of each Red UAV, outputting the action of each Red UAV under the observation at the current moment, and at the same time, the interactive environment returns the observation and global state of each Red UAV at the next moment and the joint reward; storing the observation and global state of each Red UAV at the current moment, the joint action of each Red UAV and the observation and global state of each Red UAV at the next moment, and the joint reward in an experience pool connected to the HASAC algorithm network space.

[0033] like Figure 1 , Figure 2 , Figure 3 , Figure 4 and Figure 5 As shown, in the multi-machine coordinated air combat confrontation, the two sides are based on the perception information. The present invention assumes that the red and blue drones are the confrontational sides, the red side is our side, and the opponent is the blue side. Both sides can obtain the position, speed and posture information of themselves, the opponent, and teammates. Only the confrontation in the decision-making link is considered to perform situation assessment and maneuver decision-making. The overall framework of air combat decision-making is as follows Figure 1 As shown. Among them, air combat decision-making is the key link of air combat confrontation, and the specific implementation process of the present invention is as follows: S1: Multi-aircraft collaborative air combat confrontation decision-making environment Assuming that our drone is the red drone and the opponent is the blue drone, a single drone particle motion model and a multi-drone relative motion model are established. Combined with the air combat situation elements, an air combat situation assessment model is established.

[0034] S1-1 Establishing a multi-UAV air combat confrontation motion model a. Single UAV motion model By simplifying and deducing the forces acting on the drone, a three-degree-of-freedom point model is established. The kinematic and dynamic equations are as follows: (1) (2) In the formula, The UAV is in the inertial coordinate system. Axis coordinates, velocity magnitude, track inclination, track deviation and gravitational acceleration; It is the flight speed direction of the UAV, indicating the tangential overload of the UAV; The vertical and flight speed vector represents the normal overload of the drone. Indicates the roll angle of the drone around the velocity axis.

[0035] b. Multi-UAV relative motion model In multi-machine confrontation, the red side has Drones, numbered collection , Blue has Drones, numbered collection , Red Square The relative motion relationship between the UAV and the blue UAV and the red teammate UAV is as follows: Figure 2 As shown. Drone speed vector ( )、Blue Fangdi Drone speed vector ( )、Red Square Teammate drone The velocity vector ,Subscript ,Right now , indicating the ID of the teammate’s drone. For the Red A drone and the blue square Drone in The distance between the axes; Indicates Red team drone and Red team teammate drone exist The distance between the axes; the red square A drone and the blue square The relative position vector of the drone , Red Square drone and the Red Team's drone The relative position vector ; Red Square A drone and the blue square The deviation angle of the drone For the Red The velocity vector and relative position vector of a drone The angle of the red square A drone and the blue square Departure angle of the drone For Lan Fangdi The velocity vector and relative position vector of a drone The angle of the red square A drone and its teammate drones The deviation angle For the Red The velocity vector and relative position vector of a drone The angle of the red square A drone and its teammate drones Breakaway angle Drone for teammates Velocity vector and relative position vector The angle of Indicates the red side The drone is relative to the blue side The relative distance between the two drones. Indicates the red side The drone relative to its teammates The relative distance is calculated as follows:

[0036] (3) (4) (5) (6) (7) (8) (9) (10) (11) (12) (13) (14) (15) (16) (17) (18) (19) In the formula, Respectively represent the red The speed, inclination and deviation of the drone. Respectively represent the blue square The speed, inclination and deviation of the drone. Respectively represent the red Drone Teammate Drone The speed, track inclination and track deviation; Respectively represent the red The position of the UAV in the inertial coordinate system, Respectively represent the blue square The position of the UAV in the inertial coordinate system, Respectively represent the red Drone Teammate Drone Position in the inertial coordinate system.

[0037] S1-2 Design of UAV air combat situation assessment model In the process of multi-machine coordinated confrontation, multi-machine confrontation multi-target allocation is realized based on situation assessment, which converts the multi-machine coordinated confrontation problem into target allocation and single-machine confrontation problem. Target allocation is based on situation assessment, considering the threat of the red side to the blue side and the threat of the blue side to the red side, so that the allocated target poses the least threat to itself, and the threat itself poses a greater threat to the target. Among them, the situation assessment focuses on the angle, speed, height, and distance elements of the red and blue drones, and is calculated as follows: (20) in, For the Red The drone is relative to the blue side The situation assessment value of the UAV, , , , Respectively represent the red The drone is relative to the blue side The situational advantages of the drone’s angle, height, speed, and distance, is the corresponding weight, satisfying .

[0038] The angle advantage function is designed as follows: (twenty one) The height advantage function is designed as follows: (22), where Indicates the red side The drone is relative to the blue side The z-axis coordinate difference of the drones, It represents the optimal height advantage difference between the red and blue drones.

[0039] The speed advantage function is designed as follows: (twenty three) In the formula, Indicates the red side The drone is relative to the blue side The speed difference between the two drones is Indicates the maximum and minimum speed of the drone.

[0040] The distance advantage design is as follows: (twenty four) In the formula, Indicates the red side The drone is relative to the blue side The relative distance between the two drones. Indicates the maximum and minimum distances of the drone's onboard weapons attack.

[0041] Substituting the above formulas (20)-(24) into Replace with - , then the red side can be calculated Drone Teammate Drone The corresponding parameters.

[0042] S2 establishes a distributed partially observable Markov decision process model for multi-machine collaborative confrontation decision-making problems. S1 corresponds to Figure 3 The UAV model and situation assessment in the environment are modeled by a distributed partially observable Markov process in this step S2 - the modeling of the entire multi-machine confrontation decision-making problem, the distributed partially observable Markov process consists of the tuple describe.

[0043] In the multi-machine collaborative confrontation decision-making problem, express A collection of Red Army drones; is the state space of the Red Team’s drone; is the joint action space of all red drones, Indicates the red side The action space of the drone, ; It is the Red Square The local observation of a UAV in a global state, is the state transition function, is the discount factor. , the Red Army drone is observing Execute actions when ( , ) is The strategy of flying a red drone) to obtain the joint reward function and the state at the next moment and observation ,Joint reward function In S3, it is referred to as R. Its joint goal is to learn the red side strategy In order to maximize the expected total benefit, the maximum entropy term is introduced in the joint objective, and its objective is as follows: (25) in, is the temperature constant for the trade-off between reward and maximizing entropy. represents the mathematical expectation, T represents the time when the round ends, Representation strategy The entropy of .

[0044] Each Red Team UAV has its own local observation. Observation by drone Including the Red Drone's own information , Blue drone information Teammate information . Including the Red The speed, inclination and deviation of the UAV; Including the Red The relative angle and distance elements of the UAV to each blue UAV, as well as the speed, track inclination and track deviation of the blue UAV; Including the Red The relative angle and distance elements of the first UAV to each teammate UAV, as well as the speed, track inclination and track deviation of the teammate UAV, subscript , represents the teammate number, as follows: in , , , in, Respectively represent The speed, inclination and deviation of the Red Army UAV; Respectively represent The speed, track inclination and track deviation of the blue team’s UAV; Respectively represent Teammate drone that flies the red drone speed, track inclination and track deviation; Respectively represent The Red Team UAV and the Blue Team Deviation angle, breakaway angle, and relative position deviation between drones; Indicates the red side drone and red teammate drone The deviation angle, breakaway angle and relative position deviation between them.

[0045] Global State In order to obtain the global state information of the multi-machine collaborative confrontation environment, redundant information is removed based on the combination of observation information of each red team drone. Compared with direct observation splicing, it reduces the state dimension and is conducive to accelerating training to convergence.

[0046]

[0047] in, Indicates the Red Team teammate drone Remove the red party's observation information The same exact elements of the information were observed by two drones.

[0048] Action Space Designed as a continuous action space, the Red Army drones jointly move ,in, .

[0049] S3 Design of multi-machine collaborative adversarial reward function The joint reward function of multi-aircraft coordinated air combat in step S2 is the sum of the rewards obtained by each drone of the red team against the blue team, that is, , Indicates the red side The reward for a drone to fight against all blue drones is as follows

[0050] in, Indicates the red side The drone is relative to the blue side Rewards for drones, including short-term rewards and long-term rewards ,Right now .

[0051] is a dense reward, which is a short-term reward designed based on situation assessment and is expressed as: (26) in, For the Red The drone is relative to the blue side Drone angle, height, speed, and distance rewards, The weights corresponding to the angle, height, speed, and distance rewards.

[0052] (27) (28) In the formula, Indicates the red side The drone is relative to the blue side The z-axis coordinate difference of the two drones.

[0053] (29)

[0054] (30) It is a sparse reward, and it is the red side A drone and the blue square The reward value for the end of the battle between the two drones. Among them, the two drone battle end marks are divided into the red side A drone hit the blue team The drone may be captured by the blue team The Red Army hit The blue team’s drones have exceeded the simulation boundary. The number of drones that exceed the simulation boundary and the maximum simulation step size per round is calculated as follows: (31)

[0055] Red Square A drone hit the blue team The drone must satisfy the following constraints (32) In the formula, Indicates the red side The drone is relative to the blue side The relative distance between the two drones. Indicates the maximum and minimum distances of the drone's onboard weapons attack.

[0056] The drone must satisfy the following equation within the simulation boundary: (33) The simulation step length per round used in formula (31) The following formula must be satisfied: (34) In the formula, is the maximum simulation step size per round.

[0057] S4 Design of HASAC algorithm network space The HASAC algorithm uses a centralized training distributed execution framework, which consists of n policy networks, two value networks, and two target value networks. n represents the number of red drones, and each policy network corresponds to a red drone. Each red drone is connected to a policy network to approximate the drone decision model and generate the actions of the red drone under local observation. The policy network structure is the same, but each policy network learns independently and the network parameters are not shared. Drone strategy network The input is the red square Drone in Observation of the moment , the output is the distribution of drone decision actions under the current observation. The value network is used to approximate the given global state All Red Army drones are ordered to perform joint actions The input is state and action, and the output is Q value. The two value networks have the same structure and are trained independently. The smaller Q value is used to update the policy network to alleviate the problem of overestimation of Q value, that is, the reinforcement learning algorithm has its own characteristics. When the value network is used to estimate the Q value, due to estimation errors or other factors during the training process, the learned Q value function overestimates the true Q value. The target value network is used to stabilize the training process. The network structure is the same as the value network. The parameter update of the target value network is based on the weighted average of the value network parameters. The policy network inputs observations and outputs decision actions under the observations; each policy network interacts with the environment and stores the collected training data in the experience pool, that is, the action instructions of the Red Team UAV are applied to the Red Team UAV model. Subsequently, the entire environment updates the state to collect data and returns the environment's o', s', R, and stores them together with the previous o, s, and a in the experience pool. Secondly, during the training process, the calculation of the loss function loss requires the estimated Q (s, a) value of the current state and the estimated Q (s', a') value of the next state. At this time, the s' used for the estimated Q (s', a') of the next state is extracted from the experience pool, and a' is generated by the policy network input o'. This part is described in detail in step S5.

[0058] S5 Based on HASAC training and generation of multi-machine collaborative confrontation decision model When conducting adversarial training based on the HASAC algorithm, a centralized training and distributed execution framework is used. After initializing the network, experience pool and other parameters, data is collected based on the interaction between the policy network and the environment and the network parameters are updated. Figure 3 As shown, the specific training process is as follows: 1) Obtain from the interactive environment Always observe , respectively, the policy network Enter the red square Observation by drone , output the red side Drone action , all Red Army drone actions constitute a joint action ; 2) All actions Enter the interactive environment, and the environment returns the observation at the next moment and global state and the joint reward for executing the action Among them, the action They act on the n red drone models respectively to update the status of the red drones; the blue decision model converts the multi-adversarial decision problem into a one-to-one confrontation decision problem in which each blue drone is assigned a red target based on target allocation. The target allocation model outputs the red target attacked by the blue drone based on situation assessment, and Minimax outputs the decision action of the blue drone against the red target to update the status of the blue drone. The target allocation evaluates the threat of a red drone and a blue drone to the opponent when they confront each other based on the situation assessment model, so that the assigned red target poses a small threat to the blue team, and the blue drone poses a large threat to the red target. The observation value of the red drone is extracted through the observation and global state processor and global state value , and calculate the joint reward based on the reward function ; 3) Set the current Observations of each Red Army drone at the time and global state , the joint action of all Red Army drones , the observation of each Red Army drone at the next moment and global state , Joint Rewards Composing tuples Deposit into the experience pool; 4) Then, randomly sample data from the experience pool As training data for the value network, policy network, and target value network, the value network, target value network, and the policy network of each red drone are updated based on the Adam optimizer; The specific process of updating the value network is as follows: two value networks input sampled data and , output status Next action The estimated Q value of , and the smaller The value is used as the predicted Q value in the loss function; the two target value networks are based on the state of the sampled data at the next moment And the policy network observes the next moment when inputting Output Predict the estimated Q value at the next moment ; Reward-based , entropy regularization term and Calculate the target Q value and use the loss function loss to minimize the mean square error between the target Q value and the predicted Q value to update the value network parameters, where the two value network parameters are updated independently; Update the policy network: randomly generate a group of n Red Team drones and number them, and update the policy network of each Red Team drone in the order of arrangement. When updating the parameters of the current Red Team drone policy network, consider the previously updated Red Team drone policy network; the loss function guiding the update of the policy network takes the smaller Q value of the estimated Q values ​​of the two value networks and the entropy regularization term as input.

[0059] Update the target value network: Optimize the target value network parameters through soft update at certain intervals.

[0060] Repeat the above steps until all networks gradually converge, that is, when there is no obvious increase in the reward per round and no obvious shortening of the round confrontation time within a period of time, each strategy network is used as the Red Team multi-UAV collaborative confrontation decision model, whose input is the local observation of each Red Team UAV at the current moment, and the output is the decision action performed by each Red Team UAV under the current observation.

[0061] In order to facilitate the understanding of the above technical solutions of the present invention, the above technical solutions of the present invention are described in detail through the following specific examples.

[0062] Taking a two-on-two air combat confrontation as an example, the red team uses the HASAC algorithm for confrontation training, and the blue team uses traditional decision-making algorithms, including target allocation and minimax algorithm. Target allocation is designed based on the situation assessment values ​​of the blue team's drones and the red team's drones. The threat matrix of the red team to the blue team and the threat matrix of the blue team to the red team are constructed. The goal is to maximize the threat to enemy targets and minimize its own risks, and solve them through linear programming. Target allocation transforms multi-machine collaborative confrontation into one-on-one confrontation. The blue team's one-on-one confrontation decision algorithm is the Minimax algorithm. The multi-machine collaborative confrontation decision-making framework based on heterogeneous multi-agent reinforcement learning is as follows: Figure 3 shown.

[0063] The simulation environment parameters are set as follows: the simulation boundary is 20km*20km*10km, the following parameters of the red and blue drones are the same, the speed range is 80m / s~400m / s, and the track inclination range is , the track deviation range is , the weapon attack range is 150m~900m, the action amount tangential overload range is -3g~3g, the normal overload is -5g~5g, and the roll angle range around the velocity axis is .

[0064] The parameters of the training algorithm are set as follows: the experience pool size is set to 1000000, the experience replay batch size is set to 1000, the discount factor is set to 0.99, the soft update coefficient is set to 0.005, the maximum number of steps per round is set to 500, and the learning rate of each network is set to 0.0005.

[0065] In the first group of experiments, the initial situation of the red side was a balanced situation, in the second group of experiments, the initial situation of the red side was an advantageous situation, and in the third group of experiments, the initial situation of the red side was a disadvantageous situation. The specific status of the red and blue sides is shown in Table 1.

[0066] Observations in

[0067] Right now

[0068]

[0069] Global State It is spliced ​​based on the local observations of two Red Army drones. and Remove duplicate elements in the , and reduce the dimension of the global state. express Remove Repeating elements, express Remove Repeating elements.

[0070]

[0071] Each Red Team UAV has a 27-dimensional observation space and a 42-dimensional state space, where: , Respectively represent the speed, track inclination and track deviation of the two red drones; , Respectively represent the speed, track inclination, and track deviation of the two blue drones; Respectively represent the red The relative angle and relative position information between the first UAV and the first UAV of the blue team; Indicates The relative angle and relative position information between the first red drone and the second blue drone; Indicates The relative angle and relative position information between the red team drone and the red team teammate drone, where Indicates the teammate number.

[0072] , , , The parameters are shown in Table 1. The relative angle , The calculation is done using the previous formulas (14)(15)(16)(17); , , The relative position information is calculated using the above formulas (6)(7)(8)(9)(10)(11).

[0073] The action space of each Red UAV is 3D, and the Red UAVs jointly move ,in, .

[0074] Joint Reward Function

[0075] in,

[0076]

[0077]

[0078] When conducting adversarial training based on the HASAC algorithm, a centralized training and distributed execution framework is adopted. Each red drone has a policy network, and all red drones have a centralized value network and a target value network. The structures of the policy network and the value network are as follows: Figure 4 and Figure 5 In order to alleviate the problem of overestimation of Q value, two independent value networks are designed. At the same time, two target value networks are introduced to stabilize the training process.

[0079] After initializing the network, experience pool and other parameters, the policy network interacts with the environment to collect data and update the network parameters. The specific training process is as follows: Figure 3 As shown: 1) Obtain from the interactive environment Always observe , the strategy network 1 inputs the observation of the first drone of the red side , output the action of the first drone of the Red Team ; Policy Network 2 inputs the observation of the second Red Army drone , output the action of the second drone of the Red Team , forming a joint action ; 2) Action Enter the interactive environment separately, and the environment returns the observation at the next moment and global state and the joint reward for executing the action Among them, the action It acts on the Red 1 UAV model and the Red 2 UAV model to update the status of the Red UAV. The Blue decision model converts the two-to-two confrontation decision problem into a one-to-one confrontation decision problem in which each Blue UAV is assigned a Red target based on target allocation. The target allocation model outputs the Red target attacked by the Blue UAV based on situation assessment. Minimax outputs the decision action of the Blue UAV against the Red target to update the status of the Blue UAV. The target allocation evaluates the threat of a Red UAV and a Blue UAV to the opponent when they confront each other based on the situation assessment model, so that the assigned Red target poses a small threat to the Blue UAV, and the Blue UAV poses a large threat to the Red target. The observation value of the Red UAV is extracted through the observation and global state processor and global state value , and calculate the joint reward based on the reward function ; 3) Then, the current Observation of two Red Army drones at the moment and global state , the joint action of two Red Army drones , the observation of the two red drones at the next moment and global state , Joint Rewards Composing tuples Deposit into the experience pool; 4) After that, randomly sample data from the experience pool As training data for the value network, policy network, and target value network, the value network, target value network, and the policy networks of the two red drones are updated based on the Adam optimizer; Update value network: two value networks input sample data and , output status Next action The estimated Q value of , and the smaller As the predicted Q value in the loss function; the two target value networks are based on the state of the sampled data at the next moment And the policy network observes the next moment when inputting Output Predict the estimated Q value at the next moment ; Reward-based , entropy regularization term and Calculate the target Q value, and update the value network parameters by minimizing the mean square error between the target Q value and the predicted Q value as the loss function, where the two value network parameters are updated independently; Update the policy network: randomly generate a set of Red UAV number arrangements, and update the policy networks of the two Red UAVs in the order of the arrangements. When updating the parameters of the current Red UAV policy network, the previously updated Red UAV policy network is considered; the loss function guiding the update of the policy network takes the smaller Q value of the estimated Q values ​​of the two value networks and the entropy regularization term as input.

[0080] Update the target value network: Optimize the target value network parameters through soft update at certain intervals.

[0081] Repeat the above steps until all networks gradually converge, and use each policy network as the Red Team multi-UAV collaborative confrontation decision-making model. Its input is the local observation of each Red Team UAV in the current environment, and its output is the action performed by each Red Team UAV under the current observation.

[0082] Table 1 The state scale of the red and blue drones under three initial situations

[0083] The simulation results are as follows Figure 6 , Figure 7 and Figure 8 As shown in the figure, under the three basic initial situations, the red side's strategy generation model can hit all enemy aircraft in a short time, and the two red drones cooperate to hit one blue drone respectively. When the initial situation is balanced, the red and blue sides first quickly approach the opponent, and then the two red drones slow down to complete the U-turn with the minimum turning radius, and hit the two blue drones respectively; when the initial situation is advantageous, the red side accelerates and quickly approaches the blue drone, and although the blue drone performs a turning maneuver, it is shot down by the two red drones due to the large turning speed and turning radius; when the initial situation is disadvantageous, the red drone first performs a turning maneuver to reverse the disadvantageous situation. At this time, the blue drone accelerates to approach the opponent. Later, due to the excessive speed of the blue drone, it rushes to the front of the red drone. At this time, the two red drones perform the minimum turning maneuver again and can hit the two blue drones respectively.

[0084] As a disclosed embodiment, the present invention further provides a multi-UAV collaborative confrontation decision-making system, the system is used to implement the method, and the system includes: A construction module is used to build a multi-UAV collaborative air combat confrontation decision environment by establishing a multi-UAV air combat confrontation motion model and an air combat situation assessment model; The first building module is used to establish a distributed partially observable Markov decision process model for the multi-UAV collaborative confrontation decision-making problem based on the action space, local observations and states of each UAV in the confrontation decision-making environment; The second module is to design the multi-machine collaborative confrontation reward function and the HASAC algorithm network space; A generation module is used to generate a multi-drone collaborative confrontation strategy model based on the observations of each drone obtained by the multi-drone collaborative confrontation decision model trained by HASAC, wherein the multi-drone includes multiple drones of our side and multiple drones of the enemy, wherein our side is the red side and the enemy is the blue side.

[0085] The above description shows and describes several preferred embodiments of the present invention, but as mentioned above, it should be understood that the present invention is not limited to the form disclosed herein, and should not be regarded as excluding other embodiments, but can be used in various other combinations, modifications and environments, and can be modified within the scope of the application concept described herein through the above teachings or the technology or knowledge of the relevant field. The changes and modifications made by those skilled in the art shall not depart from the spirit and scope of the present invention, and shall be within the scope of protection of the claims attached to the present invention.

Claims

1. A multi-UAV collaborative confrontation decision-making method, characterized in that: The method comprises the steps of: Step 1. Construct a multi-UAV collaborative air combat confrontation decision-making environment by establishing a multi-UAV air combat confrontation motion model and an air combat situation assessment model; Step 2. Establish a distributed partially observable Markov decision process model for the multi-UAV collaborative confrontation decision problem based on the action space, local observations and states of each UAV in the confrontation decision environment; Step 3. Design the multi-machine collaborative adversarial reward function and HASAC algorithm network space; Step 4. Based on the interaction between the HASAC algorithm cyberspace and the multi-UAV collaborative confrontation decision-making environment, train and generate a multi-UAV collaborative confrontation strategy model, where the multi-UAVs include multiple UAVs of our side and multiple UAVs of the enemy side, where our side is the red side and the enemy side is the blue side.

2. The multi-UAV cooperative confrontation decision-making method according to claim 1 is characterized in that: The step 1 specifically includes: Step 11. Analyze the forces on each drone and establish a particle motion model; Step 12: Analyze the relationship between multiple UAVs and establish a relative motion model of multiple UAVs; Step 13. Establish a drone air combat situation assessment model based on the drone's angle, speed, altitude and distance.

3. The multi-UAV cooperative confrontation decision-making method according to claim 2 is characterized in that: The particle motion model includes kinematic and dynamic equations, which are specifically as follows: In the formula, The UAV is in the inertial coordinate system. Axis coordinates, velocity magnitude, track inclination, track deviation and gravitational acceleration; It is the flight speed direction of the UAV, indicating the tangential overload of the UAV; The vertical and flight speed vector represents the normal overload of the drone. Indicates the roll angle of the drone around the velocity axis.

4. The multi-UAV cooperative confrontation decision-making method according to claim 2 is characterized in that: The relative motion model of the UAV is established as follows: In the formula, The number of Red Team drones, a collection of serial numbers , The number of blue drones, a set of serial numbers , Red Square Drone speed vector , , Lan Fangdi Drone speed vector , , Red Square Teammate drone The velocity vector , , represents the teammate number; For the Red A drone and the blue square Drone in The distance between the axes; Indicates Red team drone and Red team teammate drone exist The distance between the axes; the red square A drone and the blue square The relative position vector of the drone , Red Square drone and the Red Team's drone The relative position vector ; Red Square A drone and the blue square The deviation angle of the drone For the Red The velocity vector and relative position vector of a drone The angle of the red square A drone and the blue square Departure angle of the drone For Lan Fangdi The velocity vector and relative position vector of a drone The angle of the red square A drone and its teammate drones The deviation angle For the Red The velocity vector and relative position vector of a drone The angle of the red square A drone and its teammate drones Breakaway angle Drone for teammates Velocity vector and relative position vector The angle of Indicates the red side The drone is relative to the blue side The relative distance between the two drones. Indicates the red side The drone relative to its teammates The relative distance, Respectively represent the red The speed, inclination and deviation of the drone. Respectively represent the blue square The speed, inclination and deviation of the drone. Respectively represent the red Drone Teammate Drone The speed, track inclination and track deviation; Respectively represent the red The position of the UAV in the inertial coordinate system, Respectively represent the blue square The position of the UAV in the inertial coordinate system, Respectively represent the red Drone Teammate Drone Position in the inertial coordinate system.

5. The multi-UAV cooperative confrontation decision-making method according to claim 2 is characterized in that: Step 13 specifically includes: converting the multi-machine coordinated confrontation into target allocation and single-machine confrontation, wherein the target allocation is based on situation assessment, and in the case of mutual threat between the red party and the blue party, the allocated target is least threatening to itself, and the threat itself is greater than the target. The expression of the situation assessment is as follows: ， in, For the Red The drone is relative to the blue side The situation assessment value of the UAV, , , , Respectively represent the red The drone is relative to the blue side The situational advantages of the drone’s angle, height, speed, and distance, is the corresponding weight, satisfying .

6. The multi-UAV cooperative confrontation decision-making method according to claim 2 is characterized in that: The step 2 specifically includes: the multi-UAV cooperative confrontation decision model is implemented based on a distributed partially observable Markov decision process, and the distributed partially observable Markov process adopts a tuple To describe, among them, express A collection of Red Army drones; is the state space of the Red Team’s drone; is the joint action space of all red drones, Indicates the red side The action space of the drone, ; It is the Red Square The local observation of a UAV in a global state, is the joint reward function of all red drones working together against the blue drone, is the state transition function, is the discount factor; The local observation of the drone includes the Red Army The drone's own information, the blue team's drone information and teammates' information.

7. The multi-UAV cooperative confrontation decision-making method according to claim 2 is characterized in that: The establishment of the design of collaborative adversarial reward function is the sum of the rewards obtained by each drone of the red team against the blue team, that is, in, Indicates the red side Reward for having one drone fight against all blue drones.

8. The multi-UAV cooperative confrontation decision-making method according to claim 3 is characterized in that: The establishment of the HASAC algorithm network space specifically includes: adopting a centralized training distributed execution framework, including n policy networks, two value networks and two target value networks, each red team drone corresponds to a policy network, each of the policy networks has the same structure and is independent of each other, and is used to approximate the drone decision model, and the value network is used to evaluate the quality of the policy network's execution of actions under given observations.

9. The multi-UAV cooperative confrontation decision-making method according to claim 2 is characterized in that: The step 4 specifically includes: using the observations of each red drone at the current moment as the input of the policy network of each red drone, outputting the actions of each red drone under the observation at the current moment, and at the same time, the interactive environment returns the observations and global states of each red drone at the next moment, as well as the joint rewards; storing the observations and global states of each red drone at the current moment, the joint actions of each red drone, and the observations and global states of each red drone at the next moment, as well as the joint rewards in an experience pool connected to the HASAC algorithm network space.

10. A multi-UAV collaborative confrontation decision-making system, characterized in that: The system is used to implement the method according to any one of claims 1 to 9, and the system comprises: A construction module is used to build a multi-UAV collaborative air combat confrontation decision environment by establishing a multi-UAV air combat confrontation motion model and an air combat situation assessment model; The first building module is used to establish a distributed partially observable Markov decision process model for the multi-UAV collaborative confrontation decision-making problem based on the action space, local observations and states of each UAV in the confrontation decision-making environment; The second module is used to design the multi-machine collaborative confrontation reward function and the HASAC algorithm network space; A generation module is used to train and generate a multi-drone collaborative confrontation strategy model based on the interaction between the HASAC algorithm cyberspace and the multi-drone collaborative confrontation decision-making environment. The multi-drones include multiple drones of our side and multiple drones of the enemy, where our side is the red side and the enemy is the blue side.

Images