Multi-uav cooperative air combat maneuvering decision-making method based on reinforcement learning

Abstract

The application discloses a multi-UAV cooperative air combat maneuver decision-making method based on reinforcement learning, which comprises the following steps: step 1: taking the average position and average speed of a reference UAV group and the maneuver strategy of an enemy group as inputs to perform strategy learning, and outputting the maneuver strategy of the reference UAV group; and step 2: taking the average position and average speed of a UAV group to be trained and the maneuver strategy of the reference UAV group as inputs to perform strategy learning through a neural network based on a DDPG algorithm, and outputting the maneuver strategy of the UAV group to be trained. The application can enable the UAV to perform autonomous situation awareness and tactical planning, and complete an attack on the enemy in cooperation with autonomous maneuver, thereby solving the problem of autonomous decision-making of the UAV in a complex high-dynamic environment.

Description

Technical Field

[0001] This invention belongs to the field of unmanned aerial vehicle (UAV) game-based combat technology, and particularly relates to a multi-UAV multi-target cooperative air combat maneuver decision-making method based on reinforcement learning. Background Technology

[0002] With the advancement of combat methods, the scale of operations is also expanding, leading to a continuous increase in human resource costs. In pursuit of the goal of zero casualties, "non-contact" warfare is the primary mode of combat on the future battlefield. Unmanned Aerial Vehicles (UAVs), with their low cost, high mobility, high stealth, and lack of pilot operation, are increasingly being used to replace manned aircraft in military missions such as reconnaissance, surveillance, and target engagement, representing a typical example of "non-contact" combat equipment. Because of the limitations in the mission execution and combat capabilities of a single UAV, and the rapid popularization of artificial intelligence technology in recent years, multi-UAV collaborative combat and intelligent warfare have become research hotspots. Multi-UAV collaborative combat refers to the process by which multiple UAVs utilize algorithms for environmental perception, situational assessment, task allocation, and maneuver decision-making to carry out combat missions. Multi-UAV collaborative combat is usually accompanied by inter-UAV coordination, such as collaborative search, collaborative reconnaissance, collaborative attack, and collaborative countermeasures.

[0003] Autonomous decision-making is inseparable from artificial intelligence. Over the past decade, AI technology has developed rapidly, overcoming the technological barriers between science and application, and has become one of the world's recognized cutting-edge technologies. Deep reinforcement learning technology enables unmanned aerial vehicles (UAVs) to perform autonomous situational awareness and tactical planning, coordinating with autonomous maneuvers to complete attacks against the enemy, thus solving the problem of autonomous decision-making for UAVs in complex and highly dynamic environments. For UAV combat scenarios in three-dimensional continuous space, some scholars have studied single-aircraft maneuver decision-making based on specific tactics using deep reinforcement learning. However, for multi-aircraft autonomous combat and autonomous coordination, there are still no mature research results in this field. Summary of the Invention

[0004] The purpose of this invention is to provide a multi-UAV cooperative air combat maneuver decision-making method based on reinforcement learning, which enables UAVs to make autonomous decisions in complex and highly dynamic environments, achieve multi-UAV group confrontation, and carry out synchronous attacks.

[0005] This invention adopts the following technical solution: a multi-UAV multi-target cooperative air combat maneuver decision-making method based on reinforcement learning, comprising:

[0006] Step 1: Using the average position and average speed of the reference UAV swarm and the maneuvering strategy of the enemy swarm as input, perform strategy learning and output the maneuvering strategy of the reference UAV swarm.

[0007] Step 2: Based on the DDPG algorithm, the average position and average velocity of the UAV swarm to be trained, and the maneuvering strategy of the reference UAV swarm are used as inputs to perform policy learning through a neural network, and the maneuvering strategy of the UAV swarm to be trained is output.

[0008] The reward function of the DDPG algorithm is:

[0009] In the formula, R t In order to evaluate the final outcome of the air battle, To enable real-time evaluation of the effectiveness of multi-machine collaboration, As a perspective for rewards and punishments, For distance-based rewards and penalties, w3, w4, and w5 are all proportional factors.

[0010] Furthermore, in step 2, the average position and average velocity of the swarm of drones to be trained are represented by their relative states, defined as s = [s1, s2, s3, s4, s5, s6, s7].

[0011] Wherein, s1-s7 are defined as:

[0012]

[0013]

[0014]

[0015]

[0016] In the formula, a2 and b2 are two positive numbers, and a2 = 2b2, v U,x v represents the velocity of the drone swarm to be trained along the x-axis. avr,x for The average velocity of each drone along the x-axis, v U,y v represents the velocity of the drone swarm to be trained on the y-axis. avr,y for The average y-axis velocity of each drone within the system, v U,z v represents the speed of the drone swarm to be trained along the z-axis. avr,z for The average speed of each drone along the z-axis, v max For the maximum operational speed of the drone swarm to be trained, v min The minimum operating speed for the drone swarm to be trained.

[0017] In the formula, D UT,x The distance from the current position of the drone swarm to the target T i The distance on the x-axis, D avr,x for The average distance from each drone to the target along the x-axis, x max For the maximum boundary of the drone swarm combat to be trained on the x-axis, x min The minimum boundary on the x-axis for the operation of the drone swarm to be trained.

[0018] In the formula, D UT,y The distance from the current position of the drone swarm to the target T i The distance on the y-axis, D avr,y for The average distance of each drone to the target along the y-axis, y max For the maximum boundary of the drone swarm to be trained for combat on the y-axis, y min The minimum boundary of the y-axis for the unmanned aerial vehicle swarm to be trained for combat.

[0019] In the formula, D UT,z The distance from the current position of the drone swarm to the target T i The distance D along the z-axis avr,z for The average distance from each drone to the target along the z-axis, z max For the maximum boundary of the drone swarm to be trained for combat on the z-axis, z min The minimum boundary of the unmanned aerial vehicle swarm combat system on the z-axis is given.

[0020] In the formula, Let i be the velocity of the drone at time t. and The included angle.

[0021] Furthermore, in step 2, the output maneuver strategy of the drone swarm to be trained is the acceleration of the drone in the three spatial dimensions, and its calculation formula is as follows:

[0022] a t =[a x ,a y ,a z ],

[0023] In the formula, a x Let a be the acceleration of the drone swarm along the x-axis in space. y Let a be the acceleration of the drone swarm on the y-axis. z Let be the acceleration of the drone swarm along the z-axis.

[0024] Furthermore,

[0025]

[0026] Where C2 is a constant greater than 0.

[0027] Furthermore,

[0028] Where ι1 and ι2 are both weighting factors;

[0029] in,

[0030] In the formula, For the unmanned aerial vehicle group to be trained at time t and Coordination of other drone routes within the area

[0031] in, For the unmanned aerial vehicle group to be trained at time t-1 and Coordination of other drone routes within the area For the drone i to be trained and Coordination of other drone routes within the area A swarm of K drones. The current position of the drone i to be trained. The location of the attack target for the drone i to be trained. For reference, the current position of drone j For reference, the location of the target being attacked by the drone;

[0032] in,

[0033] In the formula, For the drone group at time t and Speed ​​coordination with other drones within the system For the drone group at time t-1 and Speed ​​coordination with other drones within the system For drones i and Speed ​​coordination with other drones within the system The current speed of the drone i to be trained For reference, the current speed of drone j is given, and K is the number of drones in the drone swarm.

[0034] Furthermore,

[0035] in, This represents the velocity of reference drone j at time t. and The included angle, This is the current location of drone i. Let i be the location of the target being attacked by drone i. This represents the velocity of reference drone j at time t-1. and The included angle.

[0036] Furthermore,

[0037] in, The position of the drone i to be trained at time t is relative to the target T. i distance, Let i be the position of the drone at time t. Let D be the position of the target of drone i at time t. UT,t-1 Let the position of the drone i to be trained at time t-1 be the distance from the target T. i The distance.

[0038] The beneficial effects of this invention are:

[0039] 1. This invention is based on a multi-UAV autonomous collaborative air combat maneuver strategy for multiple targets, which is based on target allocation and time constraints. By analyzing different combat scenarios, the key elements of the battlefield environment are extracted and corresponding mathematical models are established. A reasonable autonomous maneuver strategy for UAVs is designed to improve the success rate of multi-UAV strikes on the battlefield.

[0040] 2. This invention enables UAVs to perform autonomous situational awareness and tactical planning, and to complete attacks on the enemy in conjunction with autonomous maneuvering, thus solving the problem of autonomous decision-making for UAVs in complex and highly dynamic environments.

[0041] 3. This invention adds time constraints to the multi-aircraft autonomous and coordinated air combat strategy against a single target, enabling simultaneous attacks by multiple aircraft against a single target, and achieving full coverage strikes against the target through target allocation;

[0042] 4. In the multi-aircraft versus multi-target air combat maneuver strategy, this invention further improves the reward function of reinforcement learning and designs a multi-target allocation algorithm, transforming multi-aircraft versus multi-target air combat into several multi-aircraft versus single-target air combats for grouped confrontation. Attached Figure Description

[0043] Figure 1 This is a schematic diagram of the process of the present invention;

[0044] Figure 2 This invention relates to a multi-machine versus single-target synchronous maneuver strategy training method.

[0045] Figure 3 This is the redistribution process of the present invention. Detailed Implementation

[0046] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0047] This invention discloses a multi-UAV cooperative air combat maneuver decision-making method based on reinforcement learning, such as... Figure 1 As shown, it includes:

[0048] Step 1: Using the average position and average speed of the reference UAV swarm and the maneuvering strategy of the enemy swarm as input, perform strategy learning and output the maneuvering strategy of the reference UAV swarm.

[0049] Step 2: Based on the DDPG algorithm, the average position and average velocity of the UAV swarm to be trained and the maneuvering strategy of the reference UAV swarm are used as inputs to learn the strategy through a neural network, and the maneuvering strategy of the UAV swarm to be trained is output.

[0050] In step 2, the average position and average velocity of the drone swarm to be trained are represented by their relative states, defined as s = [s1, s2, s3, s4, s5, s6, s7].

[0051] Wherein, s1-s7 are defined as:

[0052]

[0053]

[0054]

[0055]

[0056] In the formula, a2 and b2 are two positive numbers, and a2 = 2b2, v U,x v represents the velocity of the drone swarm to be trained along the x-axis. avr,x for The average velocity of each drone along the x-axis, v U,y v represents the velocity of the drone swarm to be trained on the y-axis. avr,y for The average y-axis velocity of each drone within the system, v U,z v represents the speed of the drone swarm to be trained along the z-axis. avr,z for The average speed of each drone along the z-axis, v max For the maximum operational speed of the drone swarm to be trained, v min The minimum operating speed for the drone swarm to be trained.

[0057] In the formula, D UT,x The distance from the current position of the drone swarm to the target T i The distance on the x-axis, D avr,x for The average distance from each drone to the target along the x-axis, x max For the maximum boundary of the drone swarm combat to be trained on the x-axis, x min The minimum boundary on the x-axis for the operation of the drone swarm to be trained.

[0058] In the formula, D UT,yThe distance from the current position of the drone swarm to the target T i The distance on the y-axis, D avr,y for The average distance of each drone to the target along the y-axis, y max For the maximum boundary of the drone swarm to be trained for combat on the y-axis, y min The minimum boundary of the y-axis for the unmanned aerial vehicle swarm to be trained for combat.

[0059] In the formula, D UT,z The distance from the current position of the drone swarm to the target T i The distance D along the z-axis avr,z for The average distance from each drone to the target along the z-axis, z max For the maximum boundary of the drone swarm to be trained for combat on the z-axis, z min The minimum boundary of the unmanned aerial vehicle swarm combat system on the z-axis is given.

[0060] In the formula, Let i be the velocity of the drone at time t. and The included angle.

[0061] In step 2, the output maneuver strategy of the drone swarm to be trained is the acceleration of the drone in the three spatial dimensions, and its calculation formula is as follows:

[0062] a t =[a x ,a y ,a z ],

[0063] In the formula, a x Let a be the acceleration of the drone swarm along the x-axis in space. y Let a be the acceleration of the drone swarm on the y-axis. z Let be the acceleration of the drone swarm along the z-axis.

[0064] The reward function of the DDPG algorithm is:

[0065] In the formula, R t In order to evaluate the final outcome of the air battle, To enable real-time evaluation of the effectiveness of multi-machine collaboration, As a perspective for rewards and punishments, For distance-based rewards and penalties, w3, w4, and w5 are all proportional factors.

[0066] in,

[0067] Where C2 is a constant greater than 0.

[0068] in,

[0069] Where ι1 and ι2 are both weighting factors;

[0070] in,

[0071] In the formula, For the unmanned aerial vehicle group to be trained at time t and Coordination of other drone routes within the area

[0072] in, For the unmanned aerial vehicle group to be trained at time t-1 and Coordination of other drone routes within the area For the drone i to be trained and Coordination of other drone routes within the area A swarm of K drones. The current position of the drone i to be trained. The location of the attack target for the drone i to be trained. For reference, the current position of drone j For reference, the location of the target being attacked by the drone;

[0073] in,

[0074] In the formula, For the drone group at time t and Speed ​​coordination with other drones within the system For the drone group at time t-1 and Speed ​​coordination with other drones within the system For drones i and Speed ​​coordination with other drones within the system The current speed of the drone i to be trained For reference, the current speed of drone j is given, and K is the number of drones in the drone swarm.

[0075] in,

[0076] in, This represents the velocity of reference drone j at time t. and The included angle, This is the current location of drone i. Let i be the location of the target being attacked by drone i. This represents the velocity of reference drone j at time t-1. and The included angle.

[0077] in,

[0078] in, The position of the drone i to be trained at time t is relative to the target T. i distance, Let i be the position of the drone at time t. Let D be the position of the target of drone i at time t. UT,t-1 Let the position of the drone i to be trained at time t-1 be the distance from the target T. i The distance.

[0079] This invention addresses the issue of full target coverage in synchronous multi-drone engagements against a single target. Before the air battle begins, our drones determine the number of enemy targets through pre-reconnaissance. Then, our multiple drones are divided into several coordinated groups, each targeting a specific enemy target. The target allocation process includes two parts: cost function calculation and model solving. The cost matrix calculation involves threat assessment and pursuit strength evaluation.

[0080] This invention outlines the factors to consider and the overall situational awareness assessment method for multi-aircraft synchronous attacks. Analyzing and assessing the overall status of multiple aircraft through collaborative elements is fundamental to achieving simultaneous arrival and synchronous attacks. Under time synchronization constraints, the flight status of friendly aircraft must also be monitored during UAV operations to ensure all UAVs approach the target simultaneously for a synchronized attack.

[0081] First, consider the distance and speed between multiple drones. Our N drones form a group. They were divided into M groups. For collaboration Composed of K drones, the multiple drones did not strictly control their formation during flight, and the distance from each drone to the target was also different.

[0082] Let the current position of drone i be... speed is The target of drone i is denoted as T. i The location of the target attacked by drone i is denoted as use Evaluate the flight path coordination between drone i and other drones.

[0083]

[0084] In the formula, For drone i, the coordination of its flight path with other drones.

[0085] Represents the distance from the drone's current position to the target T. i distance, represent The average distance from each drone to the target.

[0086] Equation (1) represents the distance from UAV i to target T. i straight-line distance and The difference between the average straight-line distance from all other drones to the target. The closer it is to 0, the better the drone i is to... The higher the coordination between the flight paths of other drones within the drone's range.

[0087] Similarly, using Evaluation of drones i and Speed ​​coordination with other drones within the system.

[0088]

[0089] In the formula, For drones i and Speed ​​coordination with other drones within the system The closer it is to 0, the better the drone i is to... The higher the speed coordination with other drones within the system.

[0090] represent The average speed of each drone within the system.

[0091] Therefore, it can be used Evaluation of drones i and The overall level of coordination The closer to 0, the more the drone i and The more synchronized the operation of other drones within the drone pod, the better. It can also be used... measure Overall coordination, η Σ The closer The more consistent the overall pace, the better.

[0092]

[0093] In addition to considering the overall multi-drone configuration, this invention also needs to consider the relationship between the individual drone and the target. Represents the velocity of drone i at time t. and The angle between the two points is denoted as the target direction angle.

[0094]

[0095] To measure the overall status of drones during air combat. Evaluate the drone and The closer the overall coordination level is to 0, the better. t,iIt is used to assess the situation of the drone and enemy aircraft; the larger the size, the better.

[0096] This invention also needs to pay attention to the relative state between the UAV and the enemy aircraft, and the state between friendly aircraft, using v U,x -v avr,x v U,y -v avr,y v U,z -v avr,z D UT,x -D avr,x D UT,y -D avr,y D UT,z -D avr,z , The relative state of the drone is described. Similarly, to facilitate network learning, the state information is normalized in this invention, where a2 and b2 are two positive numbers satisfying a2 = 2b2. Therefore, the state space of reinforcement learning can be defined as s = [s1, s2, s3, s4, s5, s6, s7].

[0097] Based on the drone's motion model, the reinforcement learning action output a t The formula for the acceleration of the UAV in the three spatial dimensions is (5).

[0098] a t =[a x ,a y ,a z (5)

[0099] To achieve autonomous decision-making in a continuous three-dimensional air combat environment for unmanned aerial vehicles (UAVs), this invention employs the DDPG algorithm to continuously control the UAV's motion. DDPG uses an Actor-Critic dual neural network to train and optimize the UAV's maneuvering behavior. The Actor network generates maneuvering strategies, taking the UAV's state as input and outputting its maneuvering actions. The DDPG algorithm uses an OU process to generate noise for environmental exploration. To ensure the stability of network training, the DDPG algorithm uses two methods for sample collection and network training: fixed Q-targets and experience replay.

[0100] The reward function is defined as follows: It consists of four parts.

[0101] Part 1 R t This is an evaluation of the final outcome of the air battle. (Regarding the coordination team) Define a deviation threshold T for simultaneous arrivals. th ,Will The first UAV to meet the requirements for attacking the enemy is designated as U.f The time used is recorded as The last UAV that meets the requirements for attacking the enemy is denoted as U. e , time t a,e If t exists a,e -t a,f ≤T th If multiple drones meet the requirements for a synchronized attack, then the "synchronized arrival" condition is satisfied. Similarly, if t a,e -t a,f >T th Therefore, the condition "no synchronous arrival" is considered true. The remaining intermediate state transition process R... t =0. Then the first part R of the reward function is... t Record

[0102]

[0103] Where C2 is a constant greater than 0.

[0104] Part Two Used for immediate evaluation of the effects of multi-machine collaboration, denoted as This indicates the changes in the situation of drones during air combat.

[0105]

[0106] if This indicates that the overall situational awareness of the multi-aircraft group at time t is better than that at time t-1. The UAV starts from s... t-1 to s t Maneuver strategy a t-1 It is reasonable to give positive rewards. Conversely, if This invention applies a negative penalty. ι1 and ι2 are weighting factors, representing the degree of influence of different factors on the overall state assessment of the multi-machine system. In addition, and η vt The scope and meaning of these two terms differ greatly, and this invention requires adjusting the degree of influence of both on the state assessment through ι1 and ι2.

[0107] Part Three It is a reward and punishment from a certain angle, recorded as

[0108] The ultimate goal of drones is to approach targets and carry out attacks, while minimizing threats from enemy aircraft during the attack; therefore, flight course is extremely important.

[0109] Part Four It is a distance-based reward or punishment, recorded as When the speed of all UAVs in the cooperative group is zero, the requirement of speed coordination is satisfied, but the target can never be approached. When the distance between all UAVs in the cooperative group and the target is consistent, but the speed direction deviates from the target, although the requirement of path coordination is satisfied, the target point can never be reached.

[0110] Therefore, if a UAV only focuses on its own cooperative relationship with other UAVs, it is likely to face the problems of stagnation or overall deviation. Adding and can enable multiple UAVs to approach the target synchronously while maintaining cooperation. Proportional factors w3, w4 and w5 are used to coordinate the three reward functions. Considering that the importance of rewards varies under different air combat conditions, the present invention adopts a strategy evaluation method based on segmented weight adjustment, and sets w4 as a piecewise function,

[0111]

[0112] wherein W3 and W4 are constants, and 0<W3<W4. When the UAV should first consider the internal cooperation of multiple UAVs, so that and D avr tend to be consistent, and in order to prevent multiple UAVs from stagnation, use to motivate UAVs to approach the target. When the enemy aircraft is within the maximum attackable range of our side, the UAV should consider adjusting and and is smaller, w4 should be larger.

[0113] The present invention uses DDPG for policy learning, because the state input of the reinforcement learning algorithm involves the average position and average velocity information of all friendly aircraft, and the logic block diagram is shown in Figure 2.

[0114] Three UAVs are loaded in the simulation environment, which are respectively recorded as enemy aircraft E with a maneuver strategy πE; reference UAV Uj, whose position and velocity are used to represent the average position Pavr and average velocity vavr of other our UAVs in actual combat, and the air combat maneuver strategy uses neural network Q(s,a,θμ); UAV Ui to be trained, whose maneuver strategy is the neural network Q2(s,a,θμ2) to be trained. At this time, training Ui to perform synchronous attack with Uj in this environment is equivalent to training Ui to perform synchronous attack with other UAVs in the cooperative group. Then during the training process, v U,x -v avr,x , v U,y -v avr,y , v U,z -v avr,z , D UT,x -D avr,x , D UT,y-D avr,y D UT,z -D avr,z , The normalized values ​​are used as the state input.

[0115] The cost function is a crucial basis for target allocation and optimization, typically incorporating factors such as enemy aircraft threat, friendly energy efficiency, and environmental constraints. Since this paper focuses on the maneuvering decision-making and cooperative control of UAVs, and target allocation is secondary to maneuvering decision-making, it will not be considered in a complex manner. Regarding enemy aircraft threat, only three factors are considered: heading angle, shortest path, and arrival time.

[0116] Threat quantification by target orientation angle:

[0117] For UAV i and enemy aircraft j, based on the relevant data and guidelines provided by the research institute, the threat quantification function for the target entry angle is defined as follows:

[0118]

[0119] This is a value with π as the mean. Let be the normal distribution function of the variance. When the target's entry angle is π, i.e. This indicates that an enemy aircraft is flying directly towards our drone, at which point the threat level is highest. The threat level gradually decreases as the approach angle decreases, reaching its lowest point when the target's approach angle is 0 degrees. This indicates that the enemy aircraft is flying away from our drone, at which point the threat level is lowest. This reflects the enemy aircraft's flight direction relative to the drone. The overall variation characteristics of the curve are determined by the hyperparameter k1. The larger k1 is, the steeper the curve.

[0120] Target shortest path threat quantification:

[0121] use Represents the shortest path to the target, where Radial distance, Let be the approach angle. The shortest path is calculated using the enemy-friendly distance and the approach angle. As both the enemy-friendly distance and the approach angle increase, the threat level of the enemy aircraft increases accordingly. The shortest path threat level function follows a normal distribution, and its functional form is defined as follows:

[0122]

[0123] This is in μ D The mean, Let μ be the normal distribution function of the variance. Generally, in air combat, this paper considers the greatest threat to be when the enemy-friendly distance is 0, where μ... D Typically, it is 0. The smaller the shortest path value, the greater the threat level. The larger the k2 value, the steeper the graph and the lower the threat differentiation.

[0124] Target arrival time threat quantification:

[0125] use Indicates the near-boundary time of the target arrival, where, The enemy target's flight speed is considered. The time to approach is an organic combination of flight speed and distance-based threat level. The faster the enemy target's flight speed or the closer it is, the shorter its arrival time and the higher its threat level. The time to approach threat quantification model for enemy targets is as follows:

[0126]

[0127] Where parameters The valid range of values ​​is t max For the maximum attack time, when At that time, the enemy aircraft was far enough away from our drone that it exceeded the maximum range for sustainable flight, and was therefore deemed to lack deterrent capability. It is 0. When The shorter the time frame, the faster the enemy aircraft is expected to approach our drone. At that time, the threat is greatest. Considering the above three factors, using... Measure the threat index between drones and enemy aircraft.

[0128] The pursuit was well-assessed.

[0129] The advantage of pursuit is a comprehensive indicator for measuring the effectiveness of our attack. It is related not only to the weapon system, the state and performance of the incoming target, but also to the weapon system's ability to attack subsequent incoming targets. It is determined by the shortest distance relative to the target; generally, the distance between the target and the drone is less than the maximum attack range D. max When the target enters the drone's no-escape attack zone, the attack success rate is maximized. The attack advantage at target distance is denoted as...

[0130]

[0131] Where D ij d represents the path distance of the i-th UAV relative to the j-th aerial target. i,max D represents the maximum traversable distance of drone i. max This indicates the maximum attack range of drone i.

[0132] Target assignment solution:

[0133] 1) Find the optimal solution to the relaxation problem of integer programming. If the relaxation problem has no feasible solution, then the original problem has no solution. If the optimal solution to the relaxation problem satisfies the integer requirement, then the optimal solution to the integer programming problem is obtained; otherwise, proceed to the next step.

[0134] 2) For any non-negative integer solution variable xi, add constraints in the relaxation problem. This creates two new relaxation problems, called branches. The new relaxation problems have the characteristic that when the original problem seeks a minimum, the objective value is a lower bound of the branch problem. From the problem statement, we know that xi is either 0 or 1, so we directly set all variables to 0 and check if all constraints are satisfied. If satisfied, we terminate; otherwise, we proceed to the next step.

[0135] 3) Let a certain variable xj be 0 or 1 (fixed variable), where i ≠ j, and other variables are free variables (equal to 0 or 1), and see if the constraint conditions are met.

[0136] ① If the condition is not met, terminate the branch. And the current solution is infeasible.

[0137] ② If satisfied, calculate the objective function value, update the lower or upper bound of the current objective function value, and continue branching. The branching method also fixes one variable, while other variables are free variables.

[0138] ③ The solution process continues until all branches have stopped or all free variables have become fixed variables. The solution that best maximizes the objective function is then identified.

[0139] Multi-aircraft vs. multi-target air combat autonomous coordinated maneuver strategy:

[0140] Research was completed on multi-aircraft target allocation and autonomous maneuver strategies for simultaneous multi-aircraft attacks on single targets. Multi-aircraft attacks on multiple targets also need to consider constraints such as inter-aircraft communication and collision avoidance, while also ensuring full coverage of targets; therefore, coordination rules and target redistribution rules were introduced.

[0141] Multi-aircraft vs. multi-target air combat rules:

[0142] Multi-machine collaboration rules

[0143] The multi-machine collaboration process still does not consider inter-machine constraints and adversarial constraints, mainly using "aggregation" and "separation" rules.

[0144] a. Separation rules

[0145] use This represents the position of the enemy target at time t, if the distance between the two drones is less than the predefined minimum safe distance d. min There is a potential threat of collision. In other words, if... Then the drone's next flight target point needs to be modified.

[0146]

[0147] To ensure that drones approach enemy targets while maintaining sufficient distance between them to avoid collisions, λ1 and λ2 are positive weighting coefficients, and their selection should ensure... Approximately 1, in the formula, express The current location of the drone j inside.

[0148] b. Clustering rules

[0149] Clustering refers to the behavior of drones converging towards and assembling with other drones during their movement. Chapter Two introduces a distributed communication strategy where each drone only communicates with drones within a communication radius R. c While communicating with other drones within the same formation can reduce communication costs, it doesn't guarantee that drones can obtain status information from other aircraft in the formation. This is detrimental to multi-aircraft operations and could potentially cause drones to stray from the formation. Furthermore, if the distance between a drone and the formation exceeds the maximum communication radius, communication will be lost. Therefore, the flight range of drones must be limited. To meet communication requirements, multi-aircraft cooperative air combat should be restricted to a predefined radius R. m Within the sphere. For each drone, before initiating aggregation, the position information of each of the other drones is obtained from the perception network. The center of the communication sphere is calculated by averaging the positions of all drones (excluding itself).

[0150]

[0151] Where N is the total number of UAVs. Then, the current flight status of the UAV itself and the overall flight status of the multiple UAVs are assessed. If the UAV... i and The distance between them is less than R m Then, there's no need to move towards the multi-aircraft center; they can approach the enemy target directly. If the UAV... i and The distance between them is greater than R m This requires the UAV to approach the target while simultaneously converging towards the center of the multi-UAV group, thus requiring UAVs to... i The next navigation target point will be adjusted.

[0152]

[0153] Where λ3 and λ4 are positive weighting coefficients, their selection should ensure... Approaching 1. Besides ensuring normal communication between drones, swarming behavior allows multiple drones to form a unified force during air combat, preventing situations where drones are outnumbered and lack friendly support. Furthermore, swarming behavior enables drones to form an attack network, enhancing multi-drone cooperation and overall combat effectiveness.

[0154] This invention realizes a multi-machine target allocation and multi-machine synchronous attack on a single target autonomous maneuver strategy. The multi-machine attack on multiple targets also needs to consider constraints such as inter-machine communication and inter-machine collision avoidance, and also take into account the full coverage of the target. Therefore, a coordination rule and a target redistribution rule are introduced.

[0155] c: Reassignment rules

[0156] The following contingency scenarios should be considered during combat operations:

[0157] 1) Our drones suddenly malfunctioned and crashed, and some enemy aircraft were not assigned to attack.

[0158] 2) The number of enemy targets suddenly increases, requiring our team to be split up.

[0159] 3) The number of our air combat drones will be increased by several units, and the coordination team will expand the fleet.

[0160] When unexpected situations occur during air combat, target redistribution is necessary to ensure full coverage of the targets. This invention divides these unexpected situations into three major scenarios.

[0161] Multi-drone splitting scenario: After pre-allocation, the drones deploy for t seconds and suddenly k new targets are added. At this time, it is necessary to split the existing collaborative groups and add k new collaborative groups to attack the newly added targets.

[0162] Multi-drone reconfiguration scenario: After pre-allocation, the drones deploy for t seconds. Our side then dispatches k additional drones to participate in the operation. This scenario focuses on testing how to reconfigure and supplement existing multi-drone groups when our side has added combat forces, thereby increasing the number of drones in a single collaborative group and improving the success rate of the operation.

[0163] Multi-drone fault-tolerant scenario: After pre-allocation, the drones deploy for t seconds, and k of our drones are intercepted and damaged, requiring reassignment. Figure 3 As shown.

[0164] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A multi-UAV cooperative air combat maneuver decision-making method based on reinforcement learning, characterized in that, include: Step 1: Using the average position and average speed of the reference UAV swarm and the maneuvering strategy of the enemy swarm as input, perform strategy learning and output the maneuvering strategy of the reference UAV swarm. Step 2: Based on the DDPG algorithm, the average position and average velocity of the UAV swarm to be trained, and the maneuvering strategy of the reference UAV swarm are used as inputs to perform policy learning through a neural network, and the maneuvering strategy of the UAV swarm to be trained is output. The reward function of the DDPG algorithm is as follows: In the formula, In order to evaluate the final outcome of the air battle, To enable real-time evaluation of the effectiveness of multi-machine collaboration, As a perspective for rewards and punishments, For distance-based rewards and punishments, , , All are scaling factors; in, ; in, All are weighting factors; in, ， In the formula, For the unmanned aerial vehicle group to be trained at time t and Coordination of flight paths with other drones within the area; in, For the unmanned aerial vehicle group to be trained at time t-1 and Coordination of other drone routes within the area For the drones to be trained and Coordination of other drone routes within the area for A swarm of drones For the drones to be trained Current location For the drones to be trained The location of the attack target. For reference, the current position of drone j For reference, the location of the target being attacked by the drone; in, , In the formula, For the drone group at time t and Speed ​​coordination with other drones within the system For the drone group at time t-1 and Speed ​​coordination with other drones within the system For drones and Speed ​​coordination with other drones within the system For the drones to be trained Current speed, For reference, the current speed of drone j The number of drones in the drone swarm; in, ; in, Indicates reference drone j in Speed ​​in a moment and The included angle, For drones Current location For drones The location of the attack target. Indicates reference drone j in -1 Moment Velocity and The included angle; in, ; in, For the drones to be trained The position from the target at time t distance, For drones At the position at time t For drones The position of the attack target at time t. The position of the drone i to be trained at time t-1 is relative to the target. The distance.

2. The multi-UAV multi-target cooperative air combat maneuver decision-making method based on reinforcement learning according to claim 1, characterized in that, in, In step 2, the average position and average velocity of the swarm of drones to be trained are represented by their relative states, where the relative state is defined as s = [ , , , , , , ], in, - Defined as: ， In the formula, a2 and b2 are two positive numbers, and a2 = 2b2. For the drone swarm to be trained The speed of the shaft, Each drone in The average shaft velocity, For the drone swarm to be trained The speed of the shaft, Each drone in The average shaft velocity, For the drone swarm to be trained The speed of the shaft, Each drone in The average shaft velocity, The maximum operational speed of the drone swarm to be trained. The minimum operating speed for the drone swarm to be trained. In the formula, The current position of the drone swarm to be trained to the target exist Distance between axes The distance from each drone to the target is within The average value of the axis, For training drone swarm operations The maximum boundary of the axis, For training drone swarm operations The minimum boundary of the axis, In the formula, The current position of the drone swarm to be trained to the target exist Distance between axes The distance from each drone to the target is within The average value of the axis, For training drone swarm operations The maximum boundary of the axis, For training drone swarm operations The minimum boundary of the axis, In the formula, The current position of the drone swarm to be trained to the target exist Distance between axes The distance from each drone to the target is within The average value of the axis, For training drone swarm operations The maximum boundary of the axis, For training drone swarm operations The minimum boundary of the axis, In the formula, For drones i in Speed ​​in a moment and The included angle.

3. The multi-UAV multi-target cooperative air combat maneuver decision-making method based on reinforcement learning according to claim 1, characterized in that, in, In step 2, the output maneuver strategy of the drone swarm to be trained is the acceleration of the drone in the three spatial dimensions, and its calculation formula is: ， In the formula, Let x be the acceleration of the drone swarm along the x-axis in space. Let be the acceleration of the drone swarm on the y-axis. Let be the acceleration of the drone swarm along the z-axis.

4. The multi-UAV multi-target cooperative air combat maneuver decision-making method based on reinforcement learning according to claim 1, characterized in that, ； in, It is a constant greater than 0.

Images