Multi-uav air combat strategy generation method and device and computer equipment

Abstract

This application relates to a method, apparatus, and computer device for generating multi-UAV air combat strategies in the field of UAV technology. Based on MATD3 reinforcement learning, this method introduces an attention mechanism into the policy network and evaluation network to handle the dynamic changes in the number of UAVs. During training, agent crossover and mutation operators are introduced to avoid the sharp performance degradation caused by parameter exchange in traditional evolutionary multi-agent reinforcement learning. The network parameters of each UAV are treated as genes, and each UAV group as a chromosome. The introduction of agent-level crossover and mutation operators avoids disrupting the nonlinear relationship between network parameters and outputs and effectively maintains population diversity. The attention mechanism helps UAVs focus more on situations closely related to them, ignoring irrelevant information and improving the algorithm's scalability. This method can solve the problem of dynamic changes in the number of UAVs during air combat, while improving the efficiency of multi-UAV air combat.

Description

Technical Field

[0001] This application relates to the field of unmanned aerial vehicle (UAV) technology, and in particular to a method, apparatus, and computer device for generating multi-UAV air combat strategies. Background Technology

[0002] With the rapid development of unmanned aerial vehicles (UAVs) and artificial intelligence (AI), autonomous multi-UAV combat will become the main mode of future air warfare. Unlike single UAVs, multiple UAVs can complete more complex tasks and have greater survivability. For example, when one UAV is unable to complete a task, other UAVs can take its place to complete the predetermined task, making the system highly fault-tolerant. Intelligent decision-making methods for multi-UAV air combat have attracted widespread attention from many scholars.

[0003] In recent years, multi-agent reinforcement learning (MARL) has been widely used in autonomous multi-UAV adversarial systems. However, because their training is usually conducted in a self-play manner, existing algorithms often develop policies that form loops, hindering their ability to achieve better adversarial results. Furthermore, in complex air combat, the number of UAVs is dynamically changing, a problem that traditional MARL struggles to address. Summary of the Invention

[0004] Therefore, it is necessary to provide a method, apparatus, and computer equipment for generating multi-UAV air combat strategies that can effectively avoid strategy loops and handle different numbers of UAVs during combat, in order to address the aforementioned technical problems.

[0005] A method for generating multi-UAV air combat strategies, the method comprising:

[0006] Construct a second-order UAV motion model in three-dimensional space and a multi-UAV cooperative air combat environment.

[0007] In a multi-UAV cooperative air combat environment, initialize K parallel UAV groups, each UAV group consisting of N UAVs, and each UAV can obtain the global state of the battlefield; where K and N are integers greater than 0.

[0008] An evolutionary MATD3 architecture with attention mechanism is constructed for each UAV; the evolutionary MATD3 architecture with attention mechanism is an architecture obtained by combining the evolutionary algorithm with the MATD3 architecture with attention mechanism, which includes: a policy network and two evaluation networks with the same structure.

[0009] Each of the K parallel drone groups is paired with all the drone groups to obtain K(K+1) / 2 different pairings.

[0010] The two drone groups in each pair are cross-linked at two points, and adversarial training is performed on each pair. The obtained experience is stored in the experience replay buffer. A batch of data is sampled from the experience replay buffer, and the gradient descent method is used to update the parameters of the policy network and evaluation network of each drone in each pair based on the sampled data, generating a new K(K+1) drone policy.

[0011] In the new K(K+1) group UAV strategy, the optimal K group UAVs are selected to enter the next stage for continued training until the preset conditions are met, thus generating a multi-UAV cooperative air combat strategy.

[0012] Based on the multi-UAV collaborative air combat strategy, each UAV is controlled to complete the multi-UAV collaborative air combat mission.

[0013] A multi-UAV air combat strategy generation device, the device comprising:

[0014] The motion model and air combat environment construction module is used to build second-order UAV motion models and multi-UAV cooperative air combat environments in three-dimensional space.

[0015] The initialization module is used to initialize K parallel drone groups in a multi-drone cooperative air combat environment. Each drone group consists of N drones, and each drone can obtain the global state of the battlefield; where K and N are integers greater than 0.

[0016] The E-MATD3 reinforcement learning architecture building module is used to build an evolutionary MATD3 architecture with attention mechanism for each UAV. The evolutionary MATD3 architecture with attention mechanism is an architecture obtained by combining evolutionary algorithm with MATD3 with attention mechanism. The network structure of the architecture includes: a policy network and two evaluation networks with the same structure.

[0017] The E-MATD3 evolutionary training module is used to pair each of the K parallel drone groups with all other drone groups, resulting in K(K+1) / 2 different pairings. The two drone groups in each pairing are then cross-trained at two points, and adversarial training is performed on each pairing. The resulting experience is stored in an experience replay buffer. A batch of data is sampled from the experience replay buffer, and gradient descent is used to update the parameters of the policy network and evaluation network of each drone in each pairing based on the sampled data, generating new K(K+1) drone policies. The optimal K drone policies are selected from the new K(K+1) drone policies to proceed to the next stage of training until preset conditions are met, thus generating a multi-drone cooperative air combat strategy.

[0018] The multi-UAV collaborative air combat mission completion module is used to control each UAV according to the multi-UAV collaborative air combat strategy to complete the multi-UAV collaborative air combat mission.

[0019] A computer device includes a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps of any of the methods described above.

[0020] The aforementioned method, apparatus, and computer equipment for generating multi-UAV air combat strategies introduce an attention mechanism into the policy and evaluation networks to handle the dynamic changes in the number of UAVs, building upon the classic MATD3 reinforcement learning approach. During training, agent crossover and mutation operators are incorporated to avoid the sharp performance degradation caused by parameter exchange in traditional evolutionary multi-agent reinforcement learning. This method treats the neural network parameters of each UAV as genes and each UAV group as a chromosome, introducing agent-level crossover and mutation operators to avoid disrupting the nonlinear relationship between neural network parameters and outputs, effectively maintaining population diversity. The attention mechanism helps UAVs focus more on closely related situations, ignoring irrelevant information and improving the algorithm's scalability. This method can solve the problem of dynamic changes in the number of UAVs during air combat, while simultaneously improving the efficiency of multi-UAV air combat. Attached Figure Description

[0021] Figure 1 This is a flowchart illustrating a method for generating multi-UAV air combat strategies in one embodiment;

[0022] Figure 2 In another embodiment, the position and attitude angle of the drone are displayed in three-dimensional space;

[0023] Figure 3 This is another embodiment of a multi-UAV in-visibility air combat scenario;

[0024] Figure 4 This is a schematic diagram of a policy network with an attention mechanism in another embodiment;

[0025] Figure 5 The learning curves of different algorithms in a multi-drone tracking scenario are shown in another embodiment.

[0026] Figure 6 The following is a description of the maneuvering trajectory of a UAV trained using different algorithms in a tracking scenario in another embodiment, where (a) is MADDPG (5 to 5), (b) is MATD3 (5 to 5), (c) is ATT-MATD3 (5 to 5), and (d) is ATT-MATD3 (10 to 10).

[0027] Figure 7For a cross-comparison experiment of all algorithms in another embodiment;

[0028] Figure 8 This is a maneuver trajectory during the confrontation process in another embodiment;

[0029] Figure 9 For the attention weights of ally_5 in the initial state in another embodiment;

[0030] Figure 10 This is a structural block diagram of a multi-UAV air combat strategy generation device in one embodiment;

[0031] Figure 11 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0032] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0033] Multi-drone air combat can be described as a Markov game, which can be represented by tuples (N, S1, ..., S). N , a1,…,a N ,T,R i ,…,R N S = [S1, ..., S2], where N is the number of drones in the system. N ] is the joint state space of all UAVs, A = [A1, ..., A2] N ] is their joint action space, S i In This represents the state information observed by UAV i at time t. UAV i, based on its policy μ... i Execute action Based on all the actions and state transition functions T performed by the UAVs, the environment transitions to the next state S×T→S', with a corresponding reward r. i t =S×A∈R is returned to drone i. This process continues until the terminal state is reached. Let Γ be the expected cumulative reward of drone i, with a discount factor of Γ∈[0,1]. The goal of each drone is to maximize its expected reward.

[0034] In one embodiment, such as Figure 1 As shown, a method for generating multi-UAV air combat strategies is provided, which includes the following steps:

[0035] Step 100: Construct a second-order UAV motion model and a multi-UAV cooperative air combat environment in three-dimensional space.

[0036] Specifically, a multi-drone cooperative air combat environment is a scenario where multiple drones engage in close-range air combat. In such an environment, the objective of each drone is to inflict as much damage as possible on the opponent while minimizing its own damage.

[0037] The position and attitude angles displayed by the drone in three-dimensional space are as follows: Figure 2 As shown.

[0038] The flight constraints that all UAVs need to meet are shown in Table 1.

[0039] Table 1: Flight Constraints that UAVs Must Meet

[0040]

[0041]

[0042] Step 102: In a multi-UAV cooperative air combat environment, initialize K parallel UAV groups, each UAV group consisting of N UAVs, and each UAV can obtain the global state of the battlefield; where K and N are integers greater than 0.

[0043] Specifically, each drone is equipped with radar and other sensors to obtain status information of adversaries and allies, such as speed and relative position.

[0044] Initialize K parallel drone groups, consisting of {G1, G2, ..., G...} K} indicates. Unmanned aerial vehicle group G i N drones The network consists of several drones, where i = 1, 2, ..., K. Each drone is modeled as an agent, and each agent includes a policy network and two evaluation networks. Each drone group can be viewed as a chromosome in an evolutionary algorithm, and each drone in the group can be considered as a gene on the chromosome. Initially, the parameters of each drone's policy network and the two evaluation networks are typically randomly populated.

[0045] Since the multi-UAV cooperative air combat scenario constructed in this embodiment is within the visual range, the global state is fully observable. In a specific embodiment, the red and blue teams in the multi-UAV cooperative air combat environment each consist of five UAVs with identical performance. This method uses agent-specific global state, which includes all global state information and necessary agent-specific features. The battlefield global state obtainable by UAV i is as follows: Figure 3 As shown, the state s of its drone i i,0 The state of the opponent observed by the drone [s] i,1 ,s i,2 ,…,si,m ] and the status of allies observed by drone i [s i,m+1 ,s i,m+2 ,…,s i,m+n ].

[0046] Step 104: Construct an evolutionary MATD3 architecture with attention mechanism for each UAV; the evolutionary MATD3 architecture with attention mechanism is an architecture obtained by combining the evolutionary algorithm with the MATD3 architecture with attention mechanism, which includes: a policy network and two evaluation networks with the same structure.

[0047] Specifically, traditional algorithms are typically trained in a competitive environment through adversarial self-play, which provides agents with perfect learning opportunities. That is, regardless of the agent's strength, an environment composed of other agents of comparable strength provides the agent with appropriate challenges to avoid getting stuck. However, self-play can lead to policy loops, preventing performance improvements. Therefore, this invention proposes an attention-based evolutionary MATD3 architecture (E-MATD3 architecture) that combines evolutionary algorithms with an attention-based MATD3 (ATT-MATD3). Instead of training through self-adversarial play, this method initializes a swarm of K drones and improves their policies through evolutionary selection. The optimal response is calculated for all drone groups within the swarm. Therefore, this training paradigm not only solves the policy loop problem but also retains the learning opportunities. After initialization, the evolutionary selection process includes stages such as crossover, mutation, and selection.

[0048] MATD3 is a multi-agent variant of dual-delay deep deterministic policy gradient, applicable to continuous action spaces. The main idea is to address the non-stationarity problem caused by applying independent reinforcement learning in multi-agent environments using a centralized training and distributed execution (CTDE) paradigm. To mitigate the overestimation of Q-values ​​in MADDPG, MATD3 introduces three key techniques:

[0049] 1) Truncated Double Q Learning. MATD3 learns two evaluation networks to assess state-action values, with the smaller one used to update the policy network. Compared to standard evaluation network updates, truncated double Q learning does not introduce any additional overestimation.

[0050] 2) Delayed policy updates. If the value estimate is inaccurate, the policy network's performance will deteriorate. Therefore, the policy network and the target network are updated only after the evaluation network has been updated d times. This allows the update to be based on a value estimate with low variance.

[0051] 3) Target policy smoothing and regularization. Since deterministic policies may overfit narrow peaks in the value estimates, the network updates are smoothed by adding small noise to the target action.

[0052] Existing MARL algorithms cannot be directly applied to multi-drone air combat environments because their input networks cannot handle changes in the number of drones during combat. This problem is addressed by using an attention-based neural architecture to parameterize the policy and evaluation networks. Furthermore, the ATT-MATD3 architecture creates permutation-invariant embeddings of the input states, ensuring that the policy and value functions remain invariant for the order of adversary and ally.

[0053] The E-MATD3 architecture maintains a population of drone groups, where each group needs to compute the optimal response against all other drone groups, not itself, to avoid policy loops. Inspired by evolutionary algorithms, novel crossover and mutation operators are designed in the training of multi-agent reinforcement learning to maintain population diversity and enhance policy robustness. The E-MATD3 architecture can generalize to dynamically changing numbers of drones. To prevent crashed drones from causing bias in weight coefficient calculation, a death mask is used to enhance the attention mechanism, where the weight coefficient of crashed drones is set to zero.

[0054] The structure diagram of the policy network in the MATD3 reinforcement learning architecture with attention mechanism is shown below. Figure 4 As shown.

[0055] Step 106: Pair each of the K parallel drone groups with all the drone groups to obtain K(K+1) / 2 different pairings.

[0056] Specifically, the evolutionary selection process includes stages such as crossover, mutation, and selection.

[0057] Crossover is the most crucial stage in evolutionary algorithms, preserving population diversity. However, direct crossover between two neural networks by exchanging parameters often disrupts the hierarchical structure, leading to a catastrophic performance degradation. Therefore, a novel crossover operator is proposed, where E-MATD3 first performs hybrid matching on K parallel groups: for each group G... i It is trained on all K groups of drones (including its own drone group), resulting in a total of K(K+1) / 2 different pairings. Then, a random number of drones are selected and swapped between parent groups. This mixed matching and swapping process can be viewed as a crossover operator in standard evolutionary algorithms, without disrupting the hierarchical relationships of the nonlinear neural network.

[0058] Step 108: Perform two-point crossover on the two drone groups in each pairing and conduct adversarial training on each pairing. Store the obtained experience in the experience replay buffer. Sample a batch of data from the experience replay buffer and use gradient descent to update the parameters of the policy network and evaluation network of each drone in each pairing based on the sampled data, generating a new K(K+1) drone policy.

[0059] Specifically, in the policy network, the global state of the air combat environment acquired by the drone itself is used as input. After being processed by the attention mechanism of the policy network, the global attention embeddings of the adversary and the allies are obtained. The adversary global attention embeddings and the allies global attention embeddings are then concatenated and processed by two layers of neural networks to output the actions of each drone.

[0060] The E-MATD3 architecture utilizes gradient descent to update the parameters of the policy and evaluation networks because random parameter perturbations are typically ineffective for high-dimensional policies. This use of gradient descent to update the parameters of the policy and evaluation networks is the mutation operator employed in this invention. Due to the randomness of the sample data, this mutation operator also maintains sufficient diversity.

[0061] Step 110: Select the optimal drone strategy from the new K(K+1) group. K The group of drones will proceed to the next stage of training until the preset conditions are met, thus generating a multi-drone cooperative air combat strategy.

[0062] Specifically, after crossover and mutation, the best K groups among the newly generated K(K+1) drone strategies can survive. The optimal K groups of drones are then selected from the new K(K+1) drone strategies to proceed to the next stage of training until preset conditions are met.

[0063] The preset condition is to reach a preset number of iterations.

[0064] Step 112: Based on the multi-UAV cooperative air combat strategy, control each UAV to complete the multi-UAV cooperative air combat mission.

[0065] Specifically, the multi-UAV cooperative air combat strategy refers to the action space of each UAV in the UAV group. The action space is defined as [T, L, β], where the speed of the UAV is mainly controlled by thrust T, and the direction is controlled by lift L and roll β.

[0066] Compared to other algorithms that use discrete actions to control UAV maneuvers, the action space [T, L, β] used in this invention is continuous, which enables more precise control of the UAV.

[0067] The aforementioned multi-UAV air combat strategy generation method introduces an attention mechanism into the policy and evaluation networks to handle the dynamic changes in the number of UAVs, building upon the classic MATD3 reinforcement learning approach. During training, agent crossover and mutation operators are incorporated to avoid the sharp performance degradation caused by parameter exchange in traditional evolutionary multi-agent reinforcement learning. This method treats the neural network parameters of each UAV as genes and each UAV group as a chromosome, introducing agent-level crossover and mutation operators to avoid disrupting the nonlinear relationship between neural network parameters and outputs, effectively maintaining population diversity. The attention mechanism helps UAVs focus more on closely related situations, ignoring irrelevant information and improving the algorithm's scalability. This method can solve the problem of dynamic changes in the number of UAVs during air combat, while simultaneously improving the efficiency of multi-UAV air combat.

[0068] In one embodiment, step 100 includes: constructing a multi-UAV cooperative air combat environment based on an actual air combat scenario using computer simulation; setting the speed direction of the UAVs to be consistent with their heading; constructing a three-dimensional spatial coordinate system, wherein the three-dimensional spatial coordinate system is established with the east as the positive x-axis, the north as the positive y-axis, and the vertical upward as the positive z-axis; constructing a second-order UAV motion model in the three-dimensional spatial coordinate system, wherein the second-order UAV motion model includes the UAV's dynamics module and kinematic model; the UAV's dynamics model is as follows:

[0069]

[0070]

[0071]

[0072] Where T is the engine thrust, L is the lift of the UAV, α, β, and γ are the yaw, roll, and pitch angles of the UAV, respectively, and v is the speed of the UAV. is the derivative of the drone's velocity, and g is the acceleration due to gravity.

[0073] The kinematic model of the drone is as follows:

[0074] x=vcosγsinα (4)

[0075] y=vcosγcosα (5)

[0076] z = vsinγ (6)

[0077] Where x, y, and z are the positions of the drone on the three axes, respectively.

[0078] Specifically, such as Figure 2 As shown, the constructed three-dimensional spatial coordinate system is the ground coordinate system O. xyzThe ground coordinate system O xyz It is an inertial coordinate system.

[0079] In one embodiment, the global state of the battlefield acquired by UAV i in step 102 includes: UAV i's own state s i,0 The state of the opponent observed by the drone [s] i,1 ,s i,2 ,…,s i,m ] and the status of allies observed by drone i [s i,m+1 ,s i,m+2 ,…,s i,m+n ]; where, the drone i's own state s i,0 This includes: the speed and attitude angle of drone i itself; the state of each opponent observed by drone i, including: the distance d between drone i and the m-th opponent. im The relative position R between drone i and the m-th opponent im The relative velocity V of the m-th opponent im The heading cross angle of the m-th opponent The azimuth of the m-th opponent and the angle between the m-th opponent Where m is an integer greater than 0 and less than or equal to the number of opponents; the state of each ally observed by drone i includes: the distance d between drone i and the nth ally. in The relative position R between drone i and the nth ally in And the relative velocity V of the nth ally in , where n is an integer greater than 0 and less than or equal to the number of allies.

[0080] Specifically, the battlefield situation information that UAVs can obtain includes, for example: Figure 3 As shown, this includes the self-state s of drone i. i,0 The observed state of the opponent [s] i,1 ,s i,2 ,…,s i,m ] and the observed status of allies [s i,m+1 ,s i,m+2 ,…,s i,m+n ].

[0081] 1) The state s of UAV i i,0 This includes: its own velocity and attitude angle.

[0082] 2) The state of the opponent m observed by drone i includes: the distance d between drone i and the m-th opponent. im The relative position R between drone i and the m-th opponent im The relative velocity V of the m-th opponent im =Vm -V i The heading cross angle of the m-th opponent The azimuth of the m-th opponent Angle with the m-th opponent Among them, V m V is the absolute speed of the m-th opponent. i It is the absolute speed of drone i.

[0083] 3) The state of ally n includes distance d. in The relative position R between drone i and the nth ally in And the relative velocity V of the nth ally in .

[0084] The drone's attack zone is a cone with an angle of A = π / 3 and a generatrix length of L = 500 meters. When an enemy drone falls into this zone, it will be destroyed with a certain probability.

[0085] In one embodiment, the policy network of UAV i in step 104 is expressed as follows:

[0086]

[0087]

[0088]

[0089]

[0090] Among them, a i f is the action of drone i at the current moment. i It is a two-layer neural network. It is the global attention embedding of the adversary observed by the drone i. It is the global attention embedding of drone i, W q W k and W v These are the parameters that the policy network needs to learn, and d is... The dimension of the vector (introducing d in the formula prevents the input value of softmax from being too large), mask(·) is the function that implements the death mask function (mask(·) sets the score of the crashed drone to an infinitesimal value to prevent its state from affecting the training of the policy network), m is an integer greater than 0 and less than or equal to the number of opponents, n is an integer greater than 0 and less than or equal to the number of allies, α i,j It is the attention coefficient, s i,0 This refers to the state of drone i itself, where s is the state of drone i when j = 1, 2, ..., m. i,jThis refers to the state of the opponent observed by drone i, where s is the state of the opponent when j = m+1, m+2, ..., m+n. i,j This is the status of the ally as observed by the drone.

[0091] The evaluation network for drone i is expressed as follows:

[0092]

[0093] in, Let g be the output of the b-th evaluation network for drone i, where b = 1, 2, ... i It is a two-layer fully connected network (g i (Using a weighted sum of the attention embeddings of each ally as input and outputting the Q-value), W Q W K and W V These are the parameters that the network learns, d. e It is W V e j The dimension of e j The output of the embedding function is obtained by taking the current state and action of UAV j as input, e. i The current state and action of UAV i are the inputs to the embedding function, and the output is obtained by e(·). e(·) is the embedding function (which has a similar structure to the policy network). The only difference is that action a... j Also included in its input), s j Let a be the state of drone j at the current moment. j The current action of drone j.

[0094] In one embodiment, step 108 includes: performing two-point crossover on the two drone groups in each pairing, conducting adversarial training on each pairing, storing the obtained experience in an experience replay buffer, and sampling a batch of data from the experience replay buffer; based on the sampled data, using gradient descent to update the parameters of the policy network and evaluation network of each drone in each pairing, generating new K(K+1) drone policies; wherein, the update formula for the evaluation network parameters is:

[0095]

[0096]

[0097] in, It is an evaluation network The parameters are j = 1, 2, D is the experience replay buffer that records the experience of all drones, and s and a are the state and action values ​​of the drone group at the current moment, respectively. Let μ'(s') be the evaluation target network and policy target network of UAV i, k = 1, 2, y be the update target of the evaluation network, a' be the action of the UAV group at the next time step, s' be the state of the UAV group at the next time step, and r be the action of the UAV group at the next time step. i Γ is the reward value at the current moment, and Γ is the discount factor.

[0098] The gradient of the policy network is:

[0099]

[0100] in, It is the gradient of the policy network, μ i It is the policy network of drone i. It is the first evaluation network for drone i, μ i (a i |s i ) is the drone i in state s i Choose action a i The probabilities, a1, a2, ..., a N These are the actions of drone 1, drone 2, ..., drone N in the drone group.

[0101] In one embodiment, step 110 includes: calculating the TrueSkill score of each group of UAV strategies to obtain the score of each group of UAV strategies; sorting the K(K+1) groups of UAV strategies in ascending order of scores, selecting the best K groups of UAVs to enter the next stage for continued training until the preset conditions are met, thereby generating a multi-UAV cooperative air combat decision-making method.

[0102] Specifically, the TrueSkill points for each drone group are calculated through a round-robin tournament across all drone groups.

[0103] In one embodiment, the method employs a sparse reward setting in the reward function, where each drone's goal is to gain more rewards by cooperating with allies to destroy opponents. Specifically, destroying an enemy drone earns a reward of +5, attacking an enemy drone earns a reward of +0.2, and being destroyed incurs a penalty of -0.1.

[0104] It should be understood that, although Figure 1 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 1At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but may be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but may be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0105] In a verification embodiment, this embodiment was conducted in a constructed multi-UAV cooperative air combat environment, where UAVs take actions in a continuous three-dimensional world with discrete time steps. To demonstrate the performance of E-MATD3, the method of the present invention (E-MATD3) is compared with three existing MARL methods: (1) MADDPG algorithm; (2) MATD3 method; (3) MATD3 algorithm with attention mechanism (ATT-MATD3). For the method of the present invention E-MATD3, the number of UAV groups K is set to 5, and the number of iterations S is set to 3. Common hyperparameters used in this experiment are shown in Table 2.

[0106] Table 2. Hyperparameters used in the experiment

[0107] Hyperparameters value Optimizer Adam Learning rate 3e-4 Experience replay buffer size 1e6 Number of training set samples 256 Number of hidden layers 3 Hidden layer size 256 Discount coefficient Γ 0.95 Policy update interval 2

[0108] (1) Training process

[0109] In reinforcement learning, agents improve their policies by interacting with their environment and maximizing the expected cumulative rewards they receive. However, training with sparse rewards in complex multi-UAV combat environments is a challenging problem. To mitigate this, a paradigm similar to curriculum learning is referenced, in which agents are first trained in a tracking scenario and then transferred to a combat scenario. The goal of the UAV tracking scenario is to reduce the distance to the nearest adversary, so its reward is set to the inverse of that distance. Once the agents have learned the tracking skills, they are transferred to more complex combat scenarios for further training, where the reward function has been described in previous embodiments. In this embodiment, all baseline algorithms are first prepared for 100,000 sets in the tracking scenario and then transferred to the combat scenario for 90,000 sets of training. Our E-MATD3 algorithm is trained for 40,000, 30,000, and 20,000 iterations in each iteration of the combat scenario. The pseudocode for the E-MATD3 algorithm trained in the combat scenario is shown in Table 3.

[0110] Table 3. Pseudocode of the E-MATD3 algorithm

[0111]

[0112]

[0113] (2) Experimental Results

[0114] 1) Tracking experimental results in the scenario

[0115] In this scenario, the drones are trained to track randomly moving adversaries, and all drones start at random positions. Since the training of the E-MATD3 algorithm is identical to that of ATT-MATD3 in this case, its curves are not plotted repeatedly. Figure 5 The results show that the MATD3 algorithm outperforms MADDPG, demonstrating the effectiveness of the improvement techniques applied to MADDPG. Furthermore, the ATT-MATD3 algorithm is the best among the three. This is because the attention mechanism allows the policy to focus more on valuable information while ignoring distracting information.

[0116] In addition to the average reward, the maneuver trajectories of drones controlled by different algorithms in the tracking scenario were also plotted. To ensure a fair comparison of the maneuver strategies of each algorithm, the initial position and maneuver strategy of the opponent drone were the same in each experiment. Figure 6 The figures show the maneuver trajectories of drones controlled by different algorithms, where (a) is MADDPG (5-on-5), (b) is MATD3 (5-on-5), (c) is ATT-MATD3 (5-on-5), and (d) is ATT-MATD3 (10-on-10). Figure 6 The movement trajectories of the opponent (oppo) are represented by dashed lines, while the movement trajectories of the ally (ally) are represented by dashed lines. From Figure 6 As can be seen in (c), the UAV controlled by the ATT-MATD3 algorithm quickly caught up with the nearest opponent, while some UAVs controlled by the MADDPG and MATD3 methods lost their targets, such as... Figure 6 ally_5 in (a) and Figure 6 ally_2 in (b)

[0117] The joint state space and action space of this strategy grow exponentially with the number of drones in the environment, making MARL training extremely difficult. To test the scalability of ATT-MATD3, we converted the model trained in a 5-on-5 scenario to a 10-on-10 scenario. Figure 6 In (d), each red drone successfully chased the nearest blue drone, demonstrating that ATT-MATD3 allows for training on smaller problems with fewer drones and execution on larger problems.

[0118] (3) Experimental results in adversarial scenarios

[0119] After completing training in tracking scenarios, drones equipped with tracking skills will continue training in multi-drone air combat environments. To ensure the generalization of maneuver strategies, we randomly generate the initial states of the drones within a certain range. Figure 7 The results of 200 rounds of cross-comparison experiments show that the ATT-MATD3 algorithm outperforms MADDPG and MATD3, indicating that attention mechanisms are practical in multi-UAV air combat. The algorithm with the highest win rate is E-MATD3, suggesting that the evolution of UAV swarms can effectively resolve strategic loops caused by self-games.

[0120] Figure 8 The diagram shows the maneuver trajectory of the red team's drone (ally) controlled by the E-MATD3 algorithm relative to the blue team's drone (opponent) controlled by the MADDPG algorithm. Figure 8 The maneuver trajectories of the Red team's UAVs are represented by _____, and the maneuver trajectories of the opponents are represented by _____. In the diagram, Ally_1, Ally_2, Ally_3, Ally_4, and Ally_5 represent five allies, and oppo_1, oppo_2, oppo_3, oppo_4, and oppo_5 represent five opponents. Ally_3 selects the nearest oppo_5 as its target and flies towards it. After destroying oppo_5, Ally_3 acts as a decoy, drawing the fire of oppo_1, and then cooperates with Ally_5 to eliminate it. On the other side of the battlefield, Ally_1 and Ally_4 cooperate to destroy oppo_4, and then Ally_2 and Ally_5 attack oppo_2 together. Finally, the Red team wins the air battle. Throughout the process, strategies such as drawing fire and cross-target attacks were employed, indicating that E-MATD3's strategy effectively achieved tactical cooperation between UAVs.

[0121] To observe how the attention mechanism works in our algorithm, we visualize the attention weights that ally_5 assigns to adversary and allied drones in the initial state. For example... Figure 9 As shown, ally_5 assigns higher weight to oppo_1 and oppo_5 when aggregating adversary information, and pays more attention to ally_3 when aggregating ally information. Closer-range drones will attract more attention because their status and actions are crucial to the overall situation.

[0122] In one embodiment, such as Figure 10 As shown, a multi-UAV air combat strategy generation device is provided, including: a motion model and air combat environment construction module, an initialization module, an E-MATD3 reinforcement learning architecture construction module, an E-MATD3 evolutionary training module, and a multi-UAV cooperative air combat mission completion module, wherein:

[0123] The motion model and air combat environment construction module is used to build second-order UAV motion models and multi-UAV cooperative air combat environments in three-dimensional space.

[0124] The initialization module is used to initialize K parallel drone groups in a multi-drone cooperative air combat environment. Each drone group consists of N drones, and each drone can obtain the global state of the battlefield; where K and N are integers greater than 0.

[0125] The E-MATD3 reinforcement learning architecture building module is used to build an evolutionary MATD3 architecture with attention mechanism for each UAV. The evolutionary MATD3 architecture with attention mechanism is an architecture obtained by combining evolutionary algorithm with MATD3 with attention mechanism. The network structure of this architecture includes: a policy network and two evaluation networks with the same structure.

[0126] The E-MATD3 evolutionary training module is used to pair each of the K parallel drone groups with all other drone groups, resulting in K(K+1) / 2 different pairings. The two drone groups in each pairing are then cross-trained at two points, and adversarial training is performed on each pairing. The resulting experience is stored in an experience replay buffer. A batch of data is sampled from the experience replay buffer, and gradient descent is used to update the parameters of the policy network and evaluation network of each drone in each pairing based on the sampled data, generating new K(K+1) drone policies. The optimal K drone policies are selected from the new K(K+1) drone policies to proceed to the next stage of training until preset conditions are met, thus generating a multi-drone cooperative air combat strategy.

[0127] The multi-UAV collaborative air combat mission completion module is used to control each UAV according to the multi-UAV collaborative air combat strategy to complete the multi-UAV collaborative air combat mission.

[0128] In one embodiment, the motion model and air combat environment construction module is also used to construct a multi-UAV cooperative air combat environment based on actual air combat scenarios using computer simulation; set the speed direction of the UAV to be consistent with the heading of the UAV; construct a three-dimensional spatial coordinate system, which is established with the east as the positive x-axis, the north as the positive y-axis, and the vertical upward as the positive z-axis; construct a second-order UAV motion model in the three-dimensional spatial coordinate system, which includes the dynamics module and kinematic model of the UAV; the dynamics model of the UAV is shown in equations (1)-(3). The kinematic model of the UAV is shown in equations (4)-(6).

[0129] In one embodiment, the global battlefield state acquired by UAV i in the initialization module includes: UAV i's own state s i,0 The state of the opponent observed by the drone [s]i,1 ,s i,2 ,…,s i,m ] and the status of allies observed by drone i [s i,m+1 ,s i,m+2 ,…,s i,m+n ]; where, the drone i's own state s i,0 This includes: the speed and attitude angle of drone i itself; the state of each opponent observed by drone i, including: the distance d between drone i and the m-th opponent. im The relative position R between drone i and the m-th opponent im The relative velocity V of the m-th opponent im The heading cross angle of the m-th opponent The azimuth of the m-th opponent and the angle between the m-th opponent Where m is an integer greater than 0 and less than or equal to the number of opponents; the state of each ally observed by drone i includes: the distance d between drone i and the nth ally. in The relative position R between drone i and the nth ally in And the relative velocity V of the nth ally in , where n is an integer greater than 0 and less than or equal to the number of allies.

[0130] In one embodiment, the policy network of UAV i in the E-MATD3 reinforcement learning architecture building module is expressed as Equations (7)-(10). The evaluation network of UAV i is expressed as Equation (11).

[0131] In one embodiment, the evolutionary training module of E-MATD3 is further used to perform two-point crossover of the two drone groups in each pairing, and to perform adversarial training on each pairing. The obtained experience is stored in the experience replay buffer, and a batch of data is sampled from the experience replay buffer. Based on the sampled data, the gradient descent method is used to update the parameters of the policy network and evaluation network of each drone in each pairing, generating a new K(K+1) drone policy. The update formulas for the evaluation network parameters are shown in equations (12) and (13). The gradient of the policy network is shown in equation (14).

[0132] In one embodiment, the evolutionary training module of E-MATD3 is also used to calculate the TrueSkill score of each group of UAV strategies to obtain the score of each group of UAV strategies; sort the K(K+1) groups of UAV strategies in ascending order of score, select the best K groups of UAVs to enter the next stage to continue training until the preset conditions are met, and generate a multi-UAV cooperative air combat decision-making method.

[0133] In one embodiment, the reward function in the device employs a sparse reward setting, where each drone's goal is to gain more rewards by cooperating with allies to destroy opponents. Destroying an enemy drone earns a reward of +5, attacking an enemy drone earns a reward of +0.2, and being destroyed incurs a penalty of -0.1.

[0134] Specific limitations regarding the multi-UAV air combat strategy generation device can be found in the limitations of the multi-UAV air combat strategy generation method described above, and will not be repeated here. Each module in the aforementioned multi-UAV air combat strategy generation device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0135] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 11 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements a method for generating multi-UAV air combat strategies. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the computer device's casing, or an external keyboard, touchpad, or mouse.

[0136] Those skilled in the art will understand that Figure 11 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0137] In one embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps in the above method embodiment.

[0138] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0139] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A method for multi-UAV air combat strategy generation, characterized in that, The method includes: Constructing a second-order UAV motion model in three-dimensional space and a multi-UAV cooperative air combat environment; Initialization in a multi-UAV cooperative air combat environment K Several parallel drone groups, each drone group consisting of N Each drone can obtain a global view of the battlefield; among them, K and N It is an integer greater than 0; An evolutionary MATD3 architecture with an attention mechanism is constructed for each UAV. This architecture combines an evolutionary algorithm with the attention-based MATD3 algorithm, and includes a policy network and two identical evaluation networks. i The policy network is expressed as follows: in, The drone at the current moment i The action, It is a two-layer neural network. It is a drone i Observed global attention embedding of the adversary It is a drone i Observed global attention embedding of allies, These are the parameters that the policy network needs to learn. d yes dimensionality The function to implement the death mask functionality, m The number of opponents is an integer greater than 0 and less than or equal to the number of opponents. n An integer greater than 0 and less than or equal to the number of allies. It is the attention coefficient. It is a drone i Its own state, when hour, It is a drone i The observed state of the opponent, when hour, It is a drone i The observed status of allies; Drones i The expression form of the evaluation network is: in, For drones i The b The output of the evaluation network, b =1,2, It is a two-layer fully connected network. These are the parameters that the network learns to evaluate its performance. yes dimensionality For the current moment, the drone j The state and action are the inputs to the embedded function, and the output is obtained from them. For the current moment, the drone i The state and action are the inputs to the embedded function, and the output is obtained from them. For embedded functions, For the current moment, the drone j state, Current moment drone j The action; Will K Each drone group in a parallel drone group is paired with all other drone groups to obtain... Different pairings; two drones in each group of pairs are crossed at two points, and each group of pairs is trained in confrontation, the experience obtained is stored in an experience replay buffer, a batch of data is sampled from the experience replay buffer, and the parameters of the strategy network and the evaluation network of each drone in each group of pairs are updated by using gradient descent method according to the sampled data, and new group drone strategies are generated. In the new select the optimal strategy for the group of UAVs K The group of UAVs enters the next stage to continue training until the preset condition is met, and a multi-UAV cooperative air combat strategy is generated. Based on the multi-UAV collaborative air combat strategy, each UAV is controlled to complete the multi-UAV collaborative air combat mission.

2. The method of claim 1, wherein, Constructing a second-order UAV motion model in three-dimensional space and a multi-UAV cooperative air combat environment, including: Based on actual air combat scenarios, a multi-UAV collaborative air combat environment is constructed using computer simulation. Set the drone's speed direction to be consistent with its heading; A three-dimensional spatial coordinate system is constructed, wherein the three-dimensional spatial coordinate system is established with the east as the positive x-axis, the north as the positive y-axis, and the vertical upward as the positive z-axis; A second-order UAV motion model is constructed in a three-dimensional spatial coordinate system. The second-order UAV motion model includes the UAV's dynamics module and kinematic model. The dynamic model of the UAV is as follows: wherein, T is the engine thrust, L is the lift of the drone, and are the yaw angle, roll angle and pitch angle of the drone, respectively, is the velocity of the drone, is the derivative of the velocity of the drone, g is the gravitational acceleration; The kinematic model of the drone is as follows: wherein x , y and z especially the position of the drone in three axes.

3. The method according to claim 1, characterized in that, Initialization in a multi-UAV cooperative air combat environment K Several parallel drone groups, each drone group consisting of N The system deploys drones, each capable of acquiring a global battlefield status. The drones are involved in the process. i The acquired global battlefield status includes: drones i own state drones i Observed opponent's state and drones i Observed state of allies ; wherein the unmanned aerial vehicle i its own state comprises the unmanned aerial vehicle i its own speed and attitude angle drones i The observed status of each adversary includes: drones i With the m The distance between each opponent drones i With the m Relative positions between opponents , No. m The relative speed of each opponent , No. m The opponent's heading cross angle , No. m The azimuth of each opponent and the m Angle between opponents ,in m An integer greater than 0 and less than or equal to the number of opponents; drones i The observed status of each ally includes: drones i With the n The distance between allies drones i With the n The relative positions of the allies and the n The relative speed of each ally ,in n An integer greater than 0 and less than or equal to the number of allies.

4. The method according to claim 1, characterized in that, The two drone pairs in each pair are cross-trained at two points, and adversarial training is performed on each pair. The gained experience is stored in an experience replay buffer. A batch of data is sampled from the experience replay buffer, and gradient descent is used to update the parameters of the policy network and evaluation network of each drone in each pair based on the sampled data, generating new... Group drone strategies include: The two drone groups in each pair are cross-pointed, and adversarial training is performed on each pair. The obtained experience is stored in the experience replay buffer, and a batch of data is sampled from the experience replay buffer. Based on the sampled data, gradient descent is used to update the parameters of the policy network and evaluation network for each drone in each pairing, generating new... Group drone strategy; the formula for evaluating the update of network parameters is: in, It is an evaluation network The parameters, j =1, 2, D It is an experience playback buffer that records all drone experiences. These are the current status and action values ​​of the drone group. and It is a drone i Evaluation target network and policy target network, k= 1,2, It is an evaluation of the network's update goals. This refers to the actions of the drone crew in the next moment. This is the status of the drone group at the next moment. It is the reward value at the current moment. It is the discount factor; The gradient of the policy network is: in, It is the gradient of the policy network. It is a drone i The policy network, It is the first evaluation network for drones. It is a drone i In state Select action The probability, These are drones 1, 2, ..., 3 in the drone group. N The action.

5. The method according to claim 1, characterized in that, In the new Choose the optimal strategy for the group of drones. K The group of drones will proceed to the next phase of training until preset conditions are met, generating a multi-drone cooperative air combat strategy, including: Calculate the TrueSkill score for each drone strategy to obtain the score for each drone strategy. Will The drone strategies are sorted by score from smallest to largest, and the optimal strategy is selected. K The group of drones will proceed to the next stage of training until the preset conditions are met, thereby generating a multi-drone collaborative air combat decision-making method.

6. The method according to claim 1, characterized in that, The reward function in the method adopts a sparse reward setting. The goal of each drone is to obtain more rewards by cooperating with allies to destroy opponents. Specifically, destroying an enemy drone earns a reward of +5, attacking an enemy drone earns a reward of +0.2, and being destroyed incurs a penalty of -0.

1.

7. A multi-UAV air combat strategy generation device, characterized in that, The device includes: The motion model and air combat environment construction module is used to construct second-order UAV motion models and multi-UAV cooperative air combat environments in three-dimensional space. The initialization module is used to initialize in a multi-UAV cooperative air combat environment. K There are several parallel drone groups, each consisting of N drones, and each drone can obtain the global state of the battlefield; among them, K and N It is an integer greater than 0; The E-MATD3 reinforcement learning architecture building module is used to construct an evolutionary MATD3 architecture with an attention mechanism for each UAV. This attention-based evolutionary MATD3 architecture combines an evolutionary algorithm with the attention-based MATD3 algorithm. The network structure of this architecture includes: a policy network and two identical evaluation networks. The UAV... i The policy network is expressed as follows: in, The drone at the current moment i The action, It is a two-layer neural network. It is a drone i Observed global attention embedding of the adversary It is a drone i Observed global attention embedding of allies, These are the parameters that the policy network needs to learn. d yes dimensionality The function to implement the death mask functionality, m The number of opponents is an integer greater than 0 and less than or equal to the number of opponents. n An integer greater than 0 and less than or equal to the number of allies. It is the attention coefficient. It is a drone i Its own state, when hour, It is a drone i The observed state of the opponent, when hour, It is a drone i The observed status of allies; drones i The evaluation network is expressed as follows: in, For drones i The b The output of the evaluation network, b =1,2, It is a two-layer fully connected network. These are the parameters that the network learns to evaluate its performance. yes dimensionality For the current moment, the drone j The state and action are the inputs to the embedded function, and the output is obtained from them. For the current moment, the drone i The state and action are the inputs to the embedded function, and the output is obtained from them. For embedded functions, For the current moment, the drone j state, Current moment drone j The action; The E-MATD3 evolutionary training module is used to... K Each drone group in a parallel drone group is paired with all other drone groups to obtain... Different pairings are used; the two drone groups in each pair are cross-trained at two points, and adversarial training is performed on each pair. The obtained experience is stored in an experience replay buffer. A batch of data is sampled from the experience replay buffer, and based on the sampled data, gradient descent is used to update the parameters of the policy network and evaluation network of each drone in each pair, generating new... Group drone strategy; in the new Choose the optimal strategy for the group of drones. K The group of drones will enter the next stage to continue training until the preset conditions are met, and a multi-drone cooperative air combat strategy will be generated. The multi-UAV collaborative air combat mission completion module is used to control each UAV according to the multi-UAV collaborative air combat strategy to complete the multi-UAV collaborative air combat mission.

8. The apparatus according to claim 7, characterized in that, The motion model and air combat environment construction module is also used to construct a multi-UAV cooperative air combat environment based on actual air combat scenarios through computer simulation; and to set the speed direction of the UAV to be consistent with the UAV's heading. A three-dimensional spatial coordinate system is constructed, with the east as the positive x-axis, the north as the positive y-axis, and the vertically upward as the positive z-axis. A second-order UAV motion model is constructed within this three-dimensional spatial coordinate system, comprising a dynamics module and a kinematic model of the UAV. The dynamics model of the UAV is as follows: in, T It is engine thrust. L It's the lift from the drone. and These are the yaw angle, roll angle, and pitch angle of the drone, respectively. It's the speed of the drone. It is the derivative of the drone's speed. g It is gravitational acceleration; The kinematic model of the drone is as follows: in, x , y and z Especially the position of the drone on the three axes.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the method of any one of claims 1 to 6.

Images