UAV swarm air combat decision-making method combining mimicry physics method and DDQN

Abstract

The present invention discloses a drone cluster air combat decision-making method combining the mimetic physics method with DDQN, and relates to the field of air combat decision-making technology. The method first establishes a particle motion model of the drone, the relative geometric position relationship between the drone and its neighboring individuals, and between the drone and the target; secondly, the mimetic physics method is used to describe the forces between the drone and the surrounding individuals; then, a continuous state space of the drone cluster system is established to describe the complex coupling relationship between the drone cluster individuals; then, a reward function is established by comprehensively considering factors such as the cluster drones approaching the target area, launching an attack, the effectiveness of joint strikes, and being destroyed in air combat, thereby ensuring mutual cooperation between drones; finally, the DDQN algorithm is used to train the drone cluster to obtain the optimal air combat strategy. The present invention enables the drone cluster to autonomously realize complex air combat behaviors such as cluster search, cooperative roundup, and friendly support, and has good cluster combat efficiency.

Description

Technical Field

[0001] The present invention belongs to the field of UAV air combat decision-making, in particular to a UAV cluster air combat decision-making method combining a mimic physics method with DDQN. Background Art

[0002] UAV swarm air combat maneuver decision-making refers to the completion of combat tasks such as target search, friendly support, and cooperative strikes through autonomous decision-making after the swarm system is deployed in the combat area. Compared with traditional multi-UAV combat, the UAV swarm system, as an organic whole, does not rely on the dispatch of the central decision-making system, and is a higher level of intelligent performance.

[0003] According to whether they rely on decision models, decision methods can be divided into model-based analytical methods and model-free methods represented by expert systems and reinforcement learning. Analytical methods mainly include matrix games, differential games, etc., that is, first establish an air combat model, such as influence diagrams, strategy evaluation functions, zero-sum games, etc., and then solve the optimal strategy through adaptive planning, differential evolution algorithms, simulated annealing algorithms, etc. However, since the information in the actual combat environment often has complex characteristics such as uncertainty and incompleteness, it is difficult to establish an accurate decision model. Therefore, model-free methods have become a research hotspot in recent years. Reinforcement learning is a learning algorithm that does not require the preparation of sample sets in advance. It is more suitable for military fields where there are no samples or sample collection is difficult. Chen Xia et al. established a Markov decision process (MDP) model for drone swarm air combat and used heuristic reinforcement learning to train large-scale drone swarms. The drone swarms showed a high level of intelligence in air combat. However, the computational complexity of this method will explode exponentially with the increase of the state space dimension, and it is no longer applicable to continuous state space problems. The deep reinforcement learning algorithm uses the good nonlinear representation ability of deep neural networks to fit the value function and strategy in the reinforcement learning model, and uses gradient descent and other methods to optimize the neural network parameters. It has been widely used in the field of cluster decision-making. The DeepMind team used the deep Q-network (DQN) for multi-agent matrix games, and proved through three multi-agent game cases that the agents can obtain better decision-making capabilities through learning with the DQN algorithm. The OpenAI team used the Actor-Critic algorithm for multi-agent decision-making problems in a cooperative-competitive environment, and used the information exchange mechanism to solve the problem of algorithm non-convergence caused by dynamic changes in the environment in multi-agent decision-making problems. Summary of the invention

[0004] The purpose of the present invention is to provide a UAV cluster air combat decision-making method combining artificial physics method with Double DQN (AP-DDQN), which solves the problem of UAV cluster autonomously realizing complex air combat behaviors such as group search, cooperative round-up, and friendly support.

[0005] To achieve the above object, the present invention adopts the following technical solutions:

[0006] The UAV swarm air combat decision-making method combining the mimic physics method with DDQN includes:

[0007] Step 1: In the UAV cluster system, a particle motion model of the UAV is established, and the relative geometric position relationship between the UAV and its neighbor individuals, and between the UAV and the target is established.

[0008] Step 2: For a drone swarm system consisting of N drones, define the drone UAV i The state perception space Ω i for in is the distance from UAV i The nearest N uav A collection of neighbor individuals. is the distance from UAV i The nearest N aim A collection of targets.

[0009] Step 3: Calculate the UAV of the swarm system based on the mimicry physics method j ∈Ω i UAV i Interaction force F i , including the collision avoidance force and speed consistency force between drones.

[0010] Step 4: For any time t, calculate the UAV i Status

[0011] Step 5: To ensure the success rate of target destruction and the coordination between drones, the joint damage effectiveness of the target is used to represent the cooperative relationship between drones. The average return of the drone cluster is used to represent the air combat decision-making ability of the drones in cooperation with each other.

[0012] Step 6: After each drone performs an action, the total reward is obtained by summing up the energy consumption, approaching the target, the joint strike reward after attacking the target, and the reward for the drone being destroyed.

[0013] Step 7: Use the DDQN algorithm to train the drone cluster to converge the air combat maneuver strategy, obtain the optimal strategy, calculate the cluster benefit and count the air combat time.

[0014] The present invention has the following advantages:

[0015] 1. Based on the actual air combat environment, the state perception space of the UAV is defined, which improves the calculation speed of the UAV in air combat decision-making. The mimetic physical force is used to describe the effects of other individuals around the UAV. At the same time, based on the relative geometric positions between the UAV and its neighboring individuals, and the relative geometric positions between the UAV and the target, the continuous state space of the UAV cluster system is established to describe the complex coupling relationship between individuals in the UAV cluster.

[0016] 2. After the Red Army drones enter the target area and form attack conditions, the joint damage efficiency of the target is used to represent the cooperative relationship between the cluster drones, ensuring the success rate of target destruction and the coordination between the drones.

[0017] 3. It is proposed to obtain the total reward by summing up the energy consumption, the reward obtained by the drone in the process of approaching the target area, the joint strike return after attacking the target, and the reward for the drone being destroyed, thus ensuring the mutual cooperation between drones. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 Flow chart of the method of the present invention.

[0019] Figure 2 It is the cluster average profit curve during the training process of the AP-DDQN algorithm of the present invention, the classic DDQN algorithm, and the Actor-Critic algorithm.

[0020] Figure 3 This is the individual round profit curve during the training process of the AP-DDQN algorithm of the present invention.

[0021] Figure 4 This is the individual round profit curve during the DDQN algorithm training process.

[0022] Figure 5 This is a drone swarm support situation.

[0023] Figure 6 A situation where a swarm of drones cooperate to round up a target.

[0024] Figure 7 The figure shows the comparison of the task completion efficiency of the AP-DDQN algorithm, the DDQN algorithm and the Actor-Critic algorithm of the present invention. DETAILED DESCRIPTION

[0025] The technical solution of the present invention is described in detail with reference to the accompanying drawings.

[0026] like Figure 1 As shown, the drone cluster air combat decision-making method combining the mimic physics method with DDQN of the present invention specifically includes the following steps:

[0027] Step 1: In the UAV cluster system, establish the particle motion model of the UAV, and establish the relative geometric position relationship between the UAV and its neighbor individuals, and between the UAV and the target.

[0028] (1.1) Considering the individual drone as a particle, the particle motion model of the drone is established as

[0029]

[0030] In the formula, v i =||V i || is the speed, satisfying v min <v i <v max , v min and v max are the minimum and maximum speeds of the drone, is the velocity direction angle, satisfying N is the total number of drones.

[0031] (1.2) Based on the relationship between two drones in the drone cluster, the relative geometric position relationship between the drone and its neighbor individuals and the relative geometric position relationship between the drone and the target are established, specifically:

[0032] Assume that there are UAVs in the UAV swarm j UAV i neighbor individuals, then the UAV i Individual UAVs with their neighbors j The relative geometric position relationship between them is:

[0033]

[0034] In the formula, is the distance vector between the UAVs, is the angle between the speed directions of the two machines, V ij is the relative speed of the two machines, For UAV i The angle between the velocity vector and the line of sight of the two aircraft is, is the Euclidean distance between drones.

[0035] UAV i with target aim m The relative geometric position relationship between

[0036]

[0037] In the formula, UAV i with target aim a The distance vector between i UAV i The position of P m Aim m location.

[0038] Step 2: For a drone swarm system consisting of N drones, define the drone UAV i State-aware space in is the distance from UAV i The nearest N uav A collection of neighbor individuals. is the distance from UAV i The nearest N aim A collection of targets.

[0039] State perception space Ω i The following conditions must be met:

[0040] (a) UAV i Neighbor individual UAV j Satisfies j∈N i , if the number of neighboring drones is less than N uav ,but The vacant positions in the middle are filled by optional neighbor drones. If the number of neighbor individuals is 0, then

[0041] (b) Distance to UAV i The nearest N aim The targets include the targets known before the cluster is deployed and the targets within the drone’s sensing area. If the number of known targets is 0, then

[0042] Step 3: Calculate the UAVs in the swarm system based on the mimicry physics method i ∈Ω i UAV i Interaction force F i , including the force of consistent speed between drones and collision avoidance force The specific process is:

[0043]

[0044] In the formula, satisfy:

[0045]

[0046] in, is the relative position vector of the two UAVs, is the average relative speed of the two UAVs, r S and r A are the collision avoidance distance and alignment distance respectively, k p and k v They are the aggregation coefficient and the consistent motion coefficient, respectively, satisfying:

[0047]

[0048] satisfy:

[0049]

[0050] in, is the relative position vector of the two UAVs, r S and r A are the collision avoidance distance and alignment distance, respectively, k r is the collision avoidance coefficient, satisfying:

[0051]

[0052] Here, q is an exponential factor.

[0053] Step 4: For any time t, calculate the UAV i Status The specific process is:

[0054] (4.1) UAV i Observable UAV j ∈Ω i The relative geometric relationship information of is obtained by formula (2);

[0055] (4.2) UAV i with the target aim m The relative geometric position relationship between them is obtained by formula (3);

[0056] (4.3)F i UAV in the swarm system j ∈Ω i UAV i The interaction force is obtained by formula (4).

[0057] Step 5: To ensure the success rate of target destruction and the coordination between drones, the joint damage effectiveness of the target is used to represent the cooperative relationship between drones. The average return of the drone cluster rounds is used to represent the air combat decision-making ability of the drones in coordination with each other.

[0058] (5.1) Joint damage effectiveness of the target W des for

[0059]

[0060] In the formula, Indicates the UAV numbered i i The probability of damaging the target is taken here n is the number of Red Team UAVs in the target area. des =1 indicates that the target is destroyed. des When <1, it means that due to the small number of attacking drones, the target cannot be destroyed, but only a certain degree of damage is caused, and the damage degree is W des .

[0061] (5.2) Average round revenue of drone cluster R ave for

[0062]

[0063] In the formula, R i UAV i The individual round benefits satisfy:

[0064]

[0065] Among them, let the drone cluster from being deployed to destroying all targets in the combat area be one air combat round, t end The moment when an air combat round is completed, r i (t) for return.

[0066] In the formula, the higher the round benefit of a single drone, the better the drone has the ability to make air combat decisions. The higher the average round benefit of a cluster, the better the cooperation between drones.

[0067] Step 6: After each drone performs an action, the total reward is obtained by summing up the energy consumption, approaching the target, the joint strike reward after attacking the target, and the reward for the drone being destroyed.

[0068] (6.1) The energy consumption reward of the drone per unit time is set to r a , here we take r a =-0.01.

[0069] (6.2) The reward for the drone to approach the target is r app , specifically:

[0070]

[0071] (6.3) After the UAV enters the target area and launches an attack on the target, the target strike return based on the target joint damage efficiency is r kill , specifically:

[0072] r kill =k kill ·W des (10)

[0073] In the formula, k kill is the target damage reward gain, where k is taken kill =10,W des The combined lethality of the target,

[0074] It is obtained by formula (7).

[0075] This method ensures the mutual coordination between drones, that is, the benefit of a single drone depends not only on whether it completes its own mission, but also on whether the individuals in the cluster cooperate with each other, which is conducive to improving the destructive effectiveness of the strike mission.

[0076] (6.4) If the drone leaves the combat zone, it is considered destroyed. The negative reward for the drone being destroyed if the combat mission fails is r bekilled , here we take r bekilled =-10.

[0077] (6.5) The total reward r is

[0078] r=r a +r app +r kill +r bekilled (11)

[0079] The significance of the above reward function is that it allows the drone to make better decisions under current conditions based on the observed states of other drones and its own state.

[0080] Step 7: Use the DDQN algorithm to train the drone cluster to converge the air combat maneuver strategy, obtain the optimal strategy, calculate the cluster benefit and count the air combat time.

[0081] (7.1) UAV i Calculate the current state according to step 4

[0082] (7.2) Select action a according to the ε-greedy strategy i(t), according to formula (11), we get the reward r i (t), and calculate the state (7.3) Stored in memory pool D

[0083] Forj=1,..,N do

[0084] Sample a batch of size n from the memory pool D batch Sample Data <s,a,r i , s′>

[0085] calculate

[0086] Let the objective function be (∑[y j -Q t (a, s′, ψ)] 2 ) / n batch , perform gradient descent and update the network parameters ψ=ψ+Δψ

[0087] end for

[0088] (7.4) Update the target neural network parameters, that is,

[0089] (7.5) Use the trained network for air combat decision-making and calculate the status of the drones in the current drone cluster according to step 4 And input the network, output the actions that the drone can take. When the number of remaining targets is 0, calculate the cluster benefit. Here, the action set A that the drone can take is defined as

[0090] A={ac, de, lat, rat, ldt, rdt, lt, rt} (12)

[0091] In the formula, ac represents accelerated flight, de represents decelerated flight, lat represents accelerated flight with a left turn, rat represents accelerated flight with a right turn, ldt represents decelerated flight with a left turn, rdt represents decelerated flight with a right turn, lt represents a left turn, and rt represents a left turn.

[0092] In order to verify the feasibility and effectiveness of the method, the present invention is further described in detail below with reference to examples.

[0093] A drone swarm air combat decision-making simulation environment was built in the Linux environment, based on Python programming language, pygame1.9.4 graphical interface library and tensorflow 1.13 deep learning framework. The computer configuration is Intel Core i5-3210M processor, NVIDIA GeForce GTX 750 Ti graphics card, 8GB memory. The combat area is set as a two-dimensional continuous space of 10km×6km. A drone swarm consisting of N=15 red drones launches an attack on 5 targets in the environment in the combat area. The red early warning aircraft has sent information about two random targets to each drone in the swarm system, and the remaining targets need to be searched and discovered by drones.

[0094] The parameters of a single UAV and the air combat environment are shown in Table 1, and the hyperparameters of the AP-DDQN algorithm are shown in Table 2.

[0095] Table 1 Parameter settings of a single UAV and air combat environment

[0096]

[0097]

[0098] Table 2 AP-DDQN algorithm hyper parameters

[0099]

[0100] Figure 2 The average cluster reward curves during the training of the AP-DDQN algorithm, the classic DDQN algorithm, and the Actor-Critic algorithm of the present invention. The solid line in the figure represents the average cluster round reward after the AP-DDQN algorithm converges, the short dashed line represents the average cluster round reward after the classic DDQN algorithm converges, and the long dashed line represents the average cluster round reward after the Actor-Critic algorithm converges. It can be seen from the figure that in 1000 rounds of training, both the AP-DDQN algorithm and the DDQN algorithm can converge well, but the average cluster round reward after the AP-DDQN algorithm converges is greater than the average cluster round reward after the DDQN algorithm converges, and the Actor-Critic algorithm is always in a divergent state.

[0101] Figure 3 is the individual round profit curve during the training process of the AP-DDQN algorithm of the present invention, Figure 4The individual round benefit curve during the DDQN algorithm training process. As can be seen from the figure, the individual reward of the DDQN algorithm can reach a relatively high level, but it is still lower than the individual reward of the drone obtained by the AP-DDQN algorithm training. This is obviously because the DDQN algorithm did not take advantage of the cluster combat during the training process. In the AP-DDQN algorithm, drones can better achieve individual cooperation based on the distribution and interaction of neighboring drones, thereby improving the overall benefits of the cluster.

[0102] Figure 5 The figure shows a situation of drone cluster support. It can be seen from the figure that when a single drone cannot guarantee the damage effect on the target, it will choose to circle near the target until the red side reinforcements arrive and the cluster converges to launch an attack together, thus showing the phenomenon of cluster collaboration.

[0103] Figure 6 The figure shows a situation of cooperative hunting by drones. It can be seen from the figure that when a large number of drones form a team to attack a target, as the target approaches, the drones separate from each other to form a cooperative hunting. The advantage of this is that on the one hand, it can avoid collisions between drones, and on the other hand, it increases the exploration area of ​​the team, which is conducive to searching for new targets and better reflects the advantages of cluster combat.

[0104] Figure 7 The figure shows a comparison of the task completion efficiency of the AP-DDQN algorithm of the present invention, the classic DDQN algorithm and the Actor-Critic algorithm. It can be seen from the figure that due to the advantages of the cluster air combat maneuvering strategy obtained by the AP-DDQN algorithm training in group attack and target search, the average time used to complete the task in 100 rounds is 132.46s; the average time for the drone cluster trained by the DDQN algorithm to complete the task is 175.26s; the average time for the drone cluster trained by the Actor-Critic algorithm to complete the task is 215.53s. If it is considered that the task fails when the duration exceeds 300s, there are still cases where the task cannot be completed in the training results of the Actor-Critic algorithm. Obviously, the AP-DDQN algorithm better exerts the cluster performance and improves the combat efficiency of the drone cluster.

Claims

1. The UAV swarm air combat decision-making method combining the mimic physics method and DDQN is characterized by: The following steps are involved: Step 1: In the UAV swarm system, establish the particle motion model of the UAV, and establish the relative geometric position relationship between the UAV and its neighbor individuals, and between the UAV and the target. Specifically: (1.1) Assume that UAV j UAV i Neighbor individuals, UAVs i The position in the plane is Speed The velocity direction angle is UAV j The position in the plane is Speed The velocity direction angle is Defining UAVs in a UAV swarm i Individual UAVs with their neighbors j The relative geometric position between them is: In the formula, is the distance vector between UAVs, is the angle between the speed directions of the two machines, V ij is the relative speed of the two machines, For UAV i The angle between the velocity vector and the line of sight of the two aircraft is, is the Euclidean distance between UAVs; (1.2) Definition of UAV i with the target aim m The relative geometric relationship between for Where P i UAV j The position of P m Aim m location; Step 2: For a drone swarm system consisting of N drones, define the drone UAV i The state perception space Ω i for in is the distance from UAV i The nearest N uav A collection of neighbor individuals. is the distance from UAV i The nearest N aim A collection of targets; Step 3: Calculate the UAV of the swarm system based on the mimicry physics method j ∈Ω i UAV i Interaction force F i , including the collision avoidance force and speed consistency force between drones. The specific process is: In the formula, satisfy: in, is the relative position vector of the two UAVs, is the average relative speed of the two UAVs, r S and r A are the collision avoidance distance and alignment distance, respectively, k p and k v They are the aggregation coefficient and the consistent motion coefficient, respectively, satisfying: satisfy: in, is the relative position vector of the two UAVs, r S and r A are the collision avoidance distance and alignment distance, respectively, k r is the collision avoidance coefficient, satisfying: Here, q is the exponential factor; Step 4: For any time t, calculate the UAV i Status Specifically: (4.1) UAV i Observable UAV j ∈Ω i The relative geometric relationship information of is obtained by formula (1); (4.2) UAV i with the target aim m The relative geometric position relationship between them is obtained by formula (2); (4.3)F i UAV in the swarm system j ∈Ω i UAV i The interaction force includes the speed consistency force and the collision avoidance force between drones. for: in, is the relative position vector of the two UAVs, is the average relative speed of the two UAVs, k p and k v are the aggregation coefficient and the uniform motion coefficient, respectively. r S and r A They are collision avoidance distance and alignment distance respectively; Collision avoidance force for: in, is the relative position vector of the two UAVs, k r is the collision avoidance coefficient; Step 5: To ensure the success rate of target destruction and the coordination between drones, the joint damage effectiveness of the target is used to represent the cooperative relationship between drones. The average return of the drone cluster rounds is used to represent the air combat decision-making ability of the drones in coordination with each other. Step 6: After each drone performs an action, the total reward is obtained by summing up the energy consumption, the approach to the target, the joint strike reward after attacking the target, and the reward for the drone being destroyed; Step 7: Use the DDQN algorithm to train the drone cluster to converge the air combat maneuver strategy, obtain the optimal strategy, calculate the cluster benefit and count the air combat time.

2. The drone swarm air combat decision-making method combining the mimic physics method and DDQN as claimed in claim 1 is characterized in that: In step 5, the cooperative relationship between the UAVs is represented by the joint damage effectiveness of the target, where the joint damage effectiveness of the target W des satisfy: In the formula, Indicates the UAV numbered i i The probability of damaging the target is taken here n is the number of Red Team UAVs in the target area; when W des =1 indicates that the target is destroyed. des When <1, it means that due to the small number of attacking drones, the target cannot be destroyed, but only a certain degree of damage is caused, and the damage degree is W des .

3. The drone cluster air combat decision-making method combining the mimic physics method and DDQN as claimed in claim 1 is characterized in that: In step 6, after each drone performs an action, the total reward r is obtained by summing up the energy consumption, the approach to the target, the joint attack reward after launching an attack on the target, and the reward for the drone being destroyed, which is specifically: (3.1) The energy consumption reward of the drone per unit time is set to r a , here we take r a =-0.01; (3.2) The reward for the drone to approach the target is r app , specifically (3.3) After the UAV enters the target area and launches an attack on the target, the target strike return based on the target joint damage efficiency is r kill , specifically: r kill =k kill ·W des (10) In the formula, k kill is the target damage reward gain, where k is taken kill =10,W des The joint destruction efficiency of the target is guaranteed by this formula. The benefit of a single UAV depends not only on whether it completes its own mission, but also on whether the individuals in the cluster cooperate with each other, which is conducive to improving the destruction efficiency of the strike mission. (3.4) If the drone leaves the combat zone, it is considered destroyed. The negative reward for the drone being destroyed if the combat mission fails is r bekilled , here we take r bekilled = -10; (3.5) The total reward r is r＝r a +r app +r kill +r bekilled (11) The significance of the above reward function is that it allows the drone to make better decisions under current conditions based on the observed states of other drones and its own state.

Images