Attention network migration-based multi-agent reinforcement learning air combat decision-making method

By adopting the attention network-based migration method in multi-agent reinforcement learning, the problem of insufficient convergence speed and robustness of model in multi-drone air combat missions is solved, and efficient combat capability and rapidity are achieved in large-scale scenarios Training effect.

CN118917171AActive Publication Date: 2024-11-08BEIHANG UNIV

Patent Information

Application Number
CN202410888880.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-04
Publication Date
2024-11-08
Estimated Expiration
2044-07-04

AI Technical Summary

Technical Problem

The existing multi-agent reinforcement learning methods are difficult to effectively improve the convergence speed and robustness of the model in multi-drone air combat missions, and it is difficult to realize the efficient combat capabilities of the agent in complex scenarios.

Method used

Using the attention network-based migration method, the Actor-Critic network is trained in a small-scale multi-air air combat scenario, and the Transformer module is used to process global information, and then the trained model is migrated to a large-scale scenario to perform battlefield attention information. Analysis to improve the combat capabilities of the agent.

Benefits of technology

The convergence speed and robustness of the model of reinforcement learning training have been significantly improved, the combat capability of the agent in complex environments has been improved, and faster training efficiency and higher winning rate have been achieved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118917171A_ABST
    Figure CN118917171A_ABST
Patent Text Reader

Abstract

The invention discloses a multi-agent reinforcement learning air combat decision-making method based on attention network migration. The method can realize migration training from a small-scale air combat (2vs. 2) strategy model to a large-scale (5vs. 5) strategy model. According to the method, the Transform module in the small-scale air combat model is reused, the understanding and expression of the small-scale air combat model on the global situation are transplanted to the large-scale air combat scene, strategy convergence can be achieved through a small amount of training, and the training stability and efficiency of the large-scale air combat model are remarkably improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of computer force generation and artificial intelligence, and specifically to a multi-agent reinforcement learning air combat decision-making method based on attention network migration constructed based on a K multi-agent reinforcement learning method and its application. Background Art

[0002] Tactical-level multi-UAV air combat missions have become a hot topic of research in the military field at home and abroad due to their potential application value.

[0003] Reinforcement-deep learning technology provides an effective way to achieve the autonomous mission capability of drone swarms. A series of multi-agent training frameworks based on reinforcement-deep learning have been invented, such as QMIX, Q-Trans, MAPPO, etc. These learning frameworks are oriented to multi-agent systems and make a lot of innovations in basic principles or structural design to improve the learning efficiency and quality of multi-agent collaborative strategies. The QMIX network simplifies the value distribution problem between individual agents and the collective based on the principle of value decomposition and simplifies the learning structure. The Q-Trans network introduces a transformation function to express how individual agents adjust their strategies according to the behavior of other agents, thereby capturing the complex interactions between agents. MAPPO uses the advantage function to estimate the expected return of each agent's action relative to other possible actions, promotes agents to learn better independent strategies, and at the same time ensures the consistency of the entire agent system strategy through a coordination mechanism to achieve a balance between independence and coordination. In addition, the Transformer network has repeatedly proved its unique advantages in expressing high-dimensional situations in many types of applications, so it has also been used in multi-aircraft air combat decision-making problems and achieved good results. Summary of the invention

[0004] In order to solve the defects in the prior art, the present invention discloses a multi-agent reinforcement learning air combat decision-making system based on attention network migration, and its technical solution is as follows:

[0005] It includes the following modules: multi-aircraft near-visual range air combat simulation platform, multi-agent reinforcement learning network design and small-scale multi-aircraft air combat scenario training module, large-scale multi-aircraft air combat scenario training based on attention network migration, and battlefield attention information analysis module. Its features are:

[0006] Multi-aircraft near-visual-range air combat simulation platform: Construct a multi-aircraft air combat scenario, where aircraft gain the advantage of firing on targets through tactical maneuvers, with short-range air-to-air missiles and machine guns as the main weapons; the two sides of the air combat are defined as the red side and the blue side; the red side's aircraft is our side, and the tactical strategy is obtained through learning; the blue side's aircraft, as the opponent, adopts tactics of different difficulty levels based on random action sampling, air combat scripts and virtual self-games to respond; the above scenarios are modeled to define aircraft characteristics; at the same time, according to the partially observable Markov game theory, the state space, action space and reward function are calculated through the aircraft's position, speed and weapon information.

[0007] Multi-agent reinforcement learning network design and small-scale (2.vs.2) multi-aircraft air combat scenario training module: Design Actor-Critic network and reinforcement learning algorithm. The multi-agent reinforcement learning air combat decision-making method based on attention network migration is characterized by: the step 2 includes the following contents: the network used is the Actor-Critic architecture. For the Actor and Critic network, the input information includes: self information O t,i , Friendly Information t,f ,Enemy InformationO t,j And global information O t,full . t,i ,O t,f ,O t,j After a fully connected network, a high-dimensional tensor is obtained, which converts the global information O t,full After the embedding operation, it is input into the LSTM network for time series processing. The obtained tensor is connected to the Transformer module and spliced ​​with the high-dimensional tensor as the decision information of the Actor and Critic network. A 2.vs.2 air combat environment is set in the environment, and the red side's tactical strategy is learned by self-play; the size of the geographical scene is 50x50 kilometers; a total of 10,000 episodes are set in the training process, each episode is an air combat process, and the upper limit is 2,000 decision steps; the hyperparameters are set as follows: the learning rate of the Actor and Critic network lr = 0.0001, the discount factor γ = 0.95, the MAPPO hyperparameter shear coefficient ∈ = 0.2, the optimizer is Adam, and the batch size is 2000; the strategies adopted by the blue side aircraft are divided into four levels (L1-L4), the higher the level, the more complex and diverse the blue side's behavior; the trained red side agent has a stronger ability to adapt to complex environments; the trained agent strategy π i ,π j ; The specific classification of blue strategy is as follows:

[0008] L1: static, the blue team is in a stationary state;

[0009] L2: random, the blue plane takes random actions within the allowed action space;

[0010] L3: script, the blue aircraft engages the nearest red aircraft and after engaging, moves away from the red aircraft;

[0011] L4: Self-play, the Red aircraft uses the Blue strategy trained in L3 to fight against the opponent.

[0012] Large-scale (5.vs.5) multi-aircraft air combat scenario training and battlefield attention information analysis module based on attention network migration: The model trained in small-scale scenarios is reused and put into large-scale multi-level air combat scenarios for training. The output information from the attention network of the two scenarios is analyzed to obtain the battlefield attention level of each aircraft to the enemy and friendly forces.

[0013] The present invention discloses a multi-agent reinforcement learning air combat decision-making method based on attention network migration, and the method is based on the above system.

[0014] The present invention also discloses a non-volatile storage medium, characterized in that the non-volatile storage medium includes a stored program, wherein when the program is running, the device where the non-volatile storage medium is located is controlled to execute the above method.

[0015] The present invention also discloses an electronic device, characterized in that it comprises a processor and a memory; the memory stores computer-readable instructions, and the processor is used to execute the computer-readable instructions, wherein the computer-readable instructions execute the above method when they are executed.

[0016] Beneficial Effects

[0017] 1) Establish a multi-aircraft near-visual-range air combat simulation platform: --- Use the two-dimensional fixed-wing aircraft dynamics equation to model the aircraft. Based on Markov game theory, set the state space, action space and reward function of the dynamic characteristics and weapon carrying characteristics for the multi-aircraft air combat decision-making task.

[0018] 2) A multi-agent reinforcement learning algorithm framework based on attention network transfer is proposed. This method reuses and reconstructs the agent decision network that has been trained and converged in the 2.vs.2 air combat scenario, and loads it into the complex 5.vs.5 air combat scenario for training. It is proved that this method can improve the model convergence speed and robustness of reinforcement learning training, as well as the combat capability of the agent.

[0019] 3) A battlefield situation analysis method based on attention network is proposed. This method processes the battlefield attention global observation information output by the trained attention network to obtain the battlefield attention of the intelligent agent to other enemy and friendly forces in the battlefield, and then analyze the battlefield situation. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 It is a schematic diagram describing the functional components of the present invention and their interrelationships;

[0021] Figure 2 It is a schematic diagram describing the close-range air combat characteristics of two-dimensional aircraft;

[0022] Figure 3 It is a schematic diagram describing a multi-aircraft air combat simulation platform;

[0023] Figure 4 It is a schematic diagram describing the decision network;

[0024] Figure 5 This is a diagram describing the reward function curve for small-scale air combat training;

[0025] Figure 6 It is a diagrammatic illustration of the test success rate graph for small-scale air combat training;

[0026] Figure 7 It is a schematic diagram describing the trajectory of a small-scale air combat;

[0027] Figure 8 It is a schematic diagram describing the battlefield attention heat map at a certain moment in the air battle;

[0028] Fig. 9 This is a schematic diagram describing the training curve of a large-scale multi-aircraft air combat scenario for attention network migration;

[0029] Fig.10 This is a schematic diagram describing the success rate graph of a large-scale multi-aircraft air combat scenario test of attention network migration;

[0030] Fig.11 It is a schematic diagram describing the trajectory diagram of a large-scale multi-machine scenario;

[0031] Fig.12 It is a schematic diagram describing the battlefield attention heat map of the trajectory map of a large-scale multi-machine scene.

[0032] Fig.13 Schematic diagram of the process of the present invention. DETAILED DESCRIPTION

[0033] Example 1

[0034] The multi-agent reinforcement learning air combat decision-making system based on attention network transfer includes the following modules: a multi-aircraft near-visual-range air combat simulation platform, a multi-agent reinforcement learning network design and a small-scale multi-aircraft air combat scenario training module, a large-scale multi-aircraft air combat scenario training based on attention network transfer, and a battlefield attention information analysis module.

[0035] See also Figure 1 As shown. Multi-aircraft near-visual-range air combat simulation platform: Construct a multi-aircraft air combat scenario, where aircraft gain the advantage of firing on targets through tactical maneuvers, with short-range air-to-air missiles and machine guns as the main weapons; the two sides of the air combat are defined as the red side and the blue side; the red side aircraft is our side, and the tactical strategy is obtained through learning; the blue side aircraft is the opponent, and adopts tactics with different difficulty levels based on random action sampling, air combat scripts and virtual self-games to respond; the above scenarios are modeled to define aircraft characteristics; at the same time, according to the partially observable Markov game theory, the state space, action space and reward function are calculated through the position, speed and weapon information of the aircraft.

[0036] Multi-agent reinforcement learning network design and small-scale multi-aircraft air combat scenario training module: Design Actor-Critic network and reinforcement learning algorithm. For the multi-agent reinforcement learning air combat decision-making method based on attention network transfer, the network used is the Actor-Critic architecture. For the Actor and Critic network, the input information includes: self information O t,i , Friendly Information t,f ,Enemy InformationO t,j And global information O t,full . t,i ,O t,f ,O t,j After a fully connected network, a high-dimensional tensor is obtained, which converts the global information O t,full After the embedding operation, it is input into the LSTM network for time series processing. The obtained tensor is connected to the Transformer module and spliced ​​with the high-dimensional tensor as the decision information of the Actor and Critic network. A 2.vs.2 air combat environment is set in the environment, and the red side's tactical strategy is learned by self-play; the size of the geographical scene is 50x50 kilometers; a total of 10,000 episodes are set in the training process, each episode is an air combat process, and the upper limit is 2,000 decision steps; the hyperparameters are set as follows: the learning rate of the Actor and Critic network lr = 0.0001, the discount factor γ = 0.95, the MAPPO hyperparameter shear coefficient ∈ = 0.2, the optimizer is Adam, and the batch size is 2000; the strategies adopted by the blue side aircraft are divided into four levels (L1-L4), the higher the level, the more complex and diverse the blue side's behavior; the trained red side agent has a stronger ability to adapt to complex environments; the trained agent strategy πi ,π j ; The specific classification of blue strategy is as follows:

[0037] L1: static, the blue team is in a stationary state;

[0038] L2: random, the blue plane takes random actions within the allowed action space;

[0039] L3: script, the blue aircraft engages the nearest red aircraft and after engaging, moves away from the red aircraft;

[0040] L4: Self-play, the Red aircraft uses the Blue strategy trained in L3 to fight against the opponent.

[0041] Large-scale multi-aircraft air combat scenario training and battlefield attention information analysis module based on attention network migration: Large-scale (5.vs.5) multi-aircraft air combat scenario training and battlefield attention information analysis module based on attention network migration: From the designed air combat simulation platform, a close-range air combat scenario between five red aircraft and five blue aircraft is constructed. The model trained in the small-scale scenario is used to randomly expand and increase the input layer weights of the decision network. Ensure that its input dimension is consistent with the dimension of the red aircraft. At the same time, reuse the parameters of the trained LSTM network and Transfomer network and put them into large-scale multi-level air combat scenarios for training. Analyze the output information from the attention network of the two scenarios to obtain the battlefield attention level of each aircraft to the enemy and friendly forces.

[0042] Example 2

[0043] See also Figure 2-13 The multi-agent reinforcement learning air combat decision-making method based on attention network transfer is based on the following modules: multi-aircraft near-visual range air combat simulation platform, multi-agent reinforcement learning network design and small-scale multi-aircraft air combat scene training, large-scale multi-aircraft air combat scene training based on attention network transfer and battlefield attention information analysis, including the following steps:

[0044] Step 1: Use the aircraft dynamics equation to build a simulation environment for close-range dogfight missions. At the same time, read the aircraft's position, weapons and other information from the simulation environment to define the state space, action space and reward function of multi-agent reinforcement learning.

[0045] Multi-aircraft near-visual-range air combat simulation platform: Construct a multi-aircraft air combat scenario, where aircraft gain the advantage of firing on targets through tactical maneuvers, with short-range air-to-air missiles and machine guns as the main weapons; the two sides of the air combat are defined as the red side and the blue side; the red side's aircraft is our side, and the tactical strategy is obtained through learning; the blue side's aircraft, as the opponent, adopts tactics of different difficulty levels based on random action sampling, air combat scripts and virtual self-games to respond; the above scenarios are modeled to define aircraft characteristics; at the same time, according to the partially observable Markov game theory, the state space, action space and reward function are calculated through the aircraft's position, speed and weapon information.

[0046] Step 2: Improve the known Actor-Critic network, add the LSTM time series processing module based on global information and the Transformer architecture, use it as the decision network for multi-intelligence reinforcement learning, and then train it with the MAPPO algorithm. Set up a small-scale multi-aircraft air combat scenario, and use the designed network and algorithm to train the close-range air combat simulation scenario between two red aircraft and two blue aircraft.

[0047] Multi-agent reinforcement learning network design and small-scale (2.vs.2) multi-aircraft air combat scenario training module: Design Actor-Critic network and reinforcement learning algorithm. A multi-agent reinforcement learning air combat decision-making method based on attention network transfer. The network used is the Actor-Critic architecture. For the Actor and Critic network, the input information includes: self information O t,i , Friendly Information t,f ,Enemy InformationO t,j And global information O t,full . t,i ,O t,f ,O t,j After a fully connected network, a high-dimensional tensor is obtained, which converts the global information O t,fullAfter the embedding operation, it is input into the LSTM network for time series processing. The obtained tensor is connected to the Transformer module and spliced ​​with the high-dimensional tensor as the decision information of the Actor and Critic network. A 2.vs.2 air combat environment is set in the environment, and the red side's tactical strategy is learned by self-play; the size of the geographical scene is 50x50 kilometers; a total of 10,000 episodes are set in the training process, each episode is an air combat process, and the upper limit is 2,000 decision steps; the hyperparameters are set as follows: the learning rate of the Actor and Critic network lr = 0.0001, the discount factor γ = 0.95, the MAPPO hyperparameter shear coefficient ∈ = 0.2, the optimizer is Adam, and the batchsize = 2000; the strategies adopted by the blue side aircraft are divided into four levels (L1-L4), the higher the level, the more complex and diverse the blue side's behavior; the trained red side agent has a stronger ability to adapt to complex environments; the trained agent strategy π i ,π j ; The specific classification of blue side strategies is as follows:

[0048] L1: static, the blue team is in a stationary state;

[0049] L2: random, the blue plane takes random actions within the allowed action space;

[0050] L3: script, the blue aircraft engages the nearest red aircraft and after engaging, moves away from the red aircraft;

[0051] L4: Self-play, the Red aircraft uses the Blue strategy trained in L3 to fight against the opponent.

[0052] Step 3: Migrate the policy network trained in step 2, reuse and extract the output information of the attention network in the close combat scenario between multiple red aircraft and multiple blue aircraft, and perform attention analysis on the information to obtain the battlefield attention level of each agent to the enemy and friendly forces.

[0053] Large-scale (5.vs.5) multi-aircraft air combat scenario training and battlefield attention information analysis module based on attention network migration: The model trained in the small-scale scenario is reused and put into the large-scale multi-level air combat scenario for training. The output information from the attention network of the two scenarios is analyzed to obtain the battlefield attention level of each aircraft to the enemy and friendly forces. Specifically, the heat value range of the red side's aircraft to each other aircraft in the battlefield is [-0.5, 0.5].

[0054] The working principle of the present invention is further described below.

[0055] 1: Multi-aircraft near-visual range air combat simulation platform

[0056] The multi-aircraft air combat scenario of the present invention refers to an air combat within visual range, in which tactical maneuvers are used to gain the advantage of firing on the target, with short-range air-to-air missiles and machine guns as the main weapons. The two sides of the air combat are defined as the red side and the blue side. Among them, the red side aircraft is our side, and the tactical strategy is obtained through learning; the blue side aircraft is the opponent, and the script-based tactics with different difficulty levels are used to respond.

[0057] Based on the establishment of 2D air combat aircraft model, it can launch air-to-air missiles and aircraft cannon weapons, with the following characteristics:

[0058] Steering angular velocity range: [0,5] degrees / second;

[0059] Speed ​​range: [200,500] m / s;

[0060] Short-range missile weapons: Maximum number: 8, range [0,111] km; launch cone angle (centered on the aircraft's longitudinal axis) range [-60,60] degrees: single-shot damage probability: 0.75;

[0061] Machine gun weapons: Maximum number: 400; range [0,2] kilometers, firing cone angle range [-60,60] degrees, maximum continuous firing time 200 seconds.

[0062] 2: Multi-agent reinforcement learning network design and small-scale (2.vs.2) multi-aircraft air combat scenario training.

[0063] The multi-aircraft air combat is modeled as a partially observable Markov Game (POMG) process. From the perspective of the Red Army, the air combat process can be represented as a six-tuple (S, O i (i∈N),A i (i∈n),P,R,γ). Among them, S is the air combat state of the red and blue sides, Oi is the state observation of the i-th red aircraft, and N>=1 is the number of red aircraft. i represents the action space of the i-th red plane. Joint action A = A 1 ×…×A n , which represents the joint action of all red aircraft in a certain decision step. P:S×A→Δ(S) is the state transition probability, which represents the transition probability of the red aircraft group from state s→s' after taking joint action a in state s∈S. represents the reward function, i.e., the immediate reward obtained by the Red Team aircraft group when taking action a in state s and entering the new state s'. γ∈[0,1] is the discount factor, which is used to accumulate and calculate the long-term decision reward.

[0064] As a typical sequential decision-making problem, air combat aircraft observe the environment, make tactical decisions and execute (in a simulation environment) (e.g. Figure 2 ), while receiving and accumulating rewards. In the partially observable Markov game model, the observation space of each aircraft consists of three parts:

[0065] 1) Self-information, expressed as:

[0066] O t,i =[x,y,v,h,Aoff i,j ,AA i,j ,ATA i,j ,Aoff i,f ,AA i,f ,ATA i,f ,d i,j ,d i,f ,c,m,Ω,l]

[0067] Where, subscript t represents the current time, subscript i∈N represents the i-th aircraft of the red side, and N is the total number of aircraft of the red side; subscript j∈M represents the j-th aircraft of the blue side, and M is the total number of aircraft of the blue side. Subscript f∈N -i represents the fth friendly aircraft of aircraft i. (x,y) represents the 2D position of the aircraft (x,y), v is the current speed, c is the number of remaining cannons, and m is the number of remaining missiles. Ω represents whether the next missile is ready to be launched, and l represents whether the aircraft is shooting. h represents the current steering angle of the aircraft, and Aoff i,j represents the angle of attack of blue plane j relative to red plane i, AA i,j ATA is the azimuth of blue plane j relative to red plane i. i,j It represents the radar tracking angle of blue aircraft j relative to red aircraft i. Other symbols with multiple subscripts can be understood in the same way.

[0068] 2) Friendly information, expressed as:

[0069] O t,f =[v,h,Aoff f,i ,AA f,i ,ATA f,i ,d j,i ,c,m]

[0070] 3) Air combat opponent information, expressed as:

[0071] O t,j =[v,h,Aoff j,i ,AA j,i ,ATA j,i ,d j,i ,c,m]

[0072] Connect the observation information of all red aircraft to obtain the global observation O t,full :

[0073] O t,full =O t,i ∪O t,f ∪O t,j

[0074] Action space definition: [h, v, c, r], where h represents the turning angle of the aircraft, v represents the flight speed of the aircraft, c∈[0,1] represents whether to fire the artillery, and r∈[0,1] represents whether to fire the missile. Actions h and v are discretized to simplify the action space, and we have:

[0075] h∈[-6,…,6], respectively represents the discrete action space of turning the aircraft to a steering angle within the range of [-180,180]deg / s:

[0076] v = [0,…,9]: represents the discrete action space of setting the aircraft speed in the range of [200,500] m / s.

[0077] Reward function definition:

[0078] In air-to-air combat, facing the tail of the opponent is a relatively advantageous shooting scenario for one's own side. Therefore, we use AA, D and the remaining guns and missiles of the aircraft participating in the attack to encourage the agent to attack and improve its attack efficiency. AA is the azimuth of the red aircraft to the blue aircraft, and D is the distance between the two.

[0079]

[0080] Among them, C max ,S max Indicates the maximum number of artillery and missiles carried by the aircraft, C remain ,S remain represents the remaining amount of artillery and missiles after defeating the opponent. This reward function encourages the aircraft to turn to the tail of the opponent to attack and try to destroy the opponent with the least weapons. When the aircraft flies out of the environment boundary, it will be punished f =-5, penalty r for destroying friendly aircraft m = -2. The total reward for each simulation step is: r = r a +r f +r m .

[0081] The network used is the Actor-Critic architecture. Figure 3 As shown, for the Critic network, the input information is O t,i , O t,f , O t,jIn addition to the individual input channels, the global observation O t,full As a separate channel, it is input into the Transformer module for processing. In this method, this structure is believed to better express the relationship between individual situation and global situation, improve the sensitivity of individual decision-making to global situation, and enhance coordination. A similar structure is also used in the Actor network. Transformer has the advantage of strong expression ability of high-dimensional data space, so the training framework uses Transformer to process the input global information, hoping to better learn high-dimensional features closely related to tactical decisions from high-dimensional observation data. The addition of the long short-term memory network (LSTM) module is to further capture the correlation of observation information in time series and enhance historical cognition ability. Finally, a shared fully connected layer is set in the network to share parameters between the actor and critic networks to further improve the consistency of multi-member decision-making.

[0082] The training is conducted for the 2vs.2 air combat scenario. The MAPPO (Multi-Agent PPO) method is used for training under the above-designed decision network. The blue team uses four different strategies to train the red team's aircraft. The geographical scene size is 50x50 kilometers. A total of 10,000 episodes are set in the training process, each episode is an air combat process, and the upper limit is 2,000 decision steps. Figure 4 The training curve and Figure 5 The winning rate graph. From the two figures, we can see that the rewards of the red agent of the four strategies trained can reach the convergence level. At the same time, the winning rate is as high as more than 70%. This proves the effectiveness of this decision network.

[0083] 3: Large-scale (5.vs.5) multi-aircraft air combat scenario training and battlefield attention information analysis based on attention network migration.

[0084] exist Figure 3 In the structure shown, the Transformer module is used to process global observation information, hoping to better capture global features and provide support for individual aircraft decision-making. Therefore, a simple idea is: if the Transformer module has learned to better express the situational features of the 2vs.2 air combat scenario, then can reusing the Transformer module parameters speed up the learning of larger-scale air combat strategies? Based on this idea, the present invention reuses the Transformer module parameters trained in the 2vs.2 air combat scenario in the 5vs.5 scenario to avoid training from scratch, hoping to improve training efficiency while ensuring training quality.

[0085] The attention information of the trained attention network is analyzed and transferred to a larger battlefield environment. Through the transferred attention network, the agent can quickly obtain battlefield situation information that needs attention in different environments and tasks, improving the convergence efficiency and robustness of its model. The model obtained by transfer training is compared with the model trained from scratch. Figure 8 and Fig. 9 . Compared with the reward curve trained from scratch, the training curve based on the 2.vs.2 model attention parameter sharing shows that the episode reward converges faster and the reward value after convergence is smoother. This proves that the model has better convergence speed and robustness. At the same time, the winning rate of the model trained from scratch was 71.0% after testing, while the winning rate of the model obtained after transfer learning was as high as 82.0%, proving that the model has better combat capability.

[0086] From the Multihead_attention network in the above network structure, o full As the network input, multiply it with the trained attention weight to obtain the battlefield situation information with attention, and add the situation information of each aircraft to obtain the attention information heat map of each aircraft. Fig.10 and Fig.11 .For the destroyed aircraft targets, their attention is 0, such as r1 in 10. For the red side r5 aircraft, the target that it pays the most attention to is b7, so it launches a missile at this target; at the same time, the blue side aircraft b9 is located at the tail of aircraft r5 and the distance is very close, so r5 also maintains a high attention to b9 (attention value is 0.49).

[0087] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions only describe the principles of the present invention. The present invention may be subject to various changes and improvements without departing from the spirit and scope of the present invention. These changes and improvements fall within the scope of the present invention. The scope of protection claimed by the present invention is defined by the attached claims and their equivalents.

Claims

1. A multi-agent reinforcement learning air combat decision-making system based on attention network transfer, including the following modules: a multi-aircraft near-visual range air combat simulation platform, a multi-agent reinforcement learning network design and a small-scale multi-aircraft air combat scenario training module, a large-scale multi-aircraft air combat scenario training and battlefield attention information analysis module based on attention network transfer, which is characterized by: The multi-aircraft near-visual-range air combat simulation platform: uses two-dimensional fixed-wing aircraft dynamics equations to model aircraft dynamics and weapon carrying characteristics, sets the aircraft's flight speed and heading angle, and models the aircraft's weapon characteristics of launching short-range air-to-air missiles and artillery. It models air combat based on partially observable Markov game theory, and sets the state space, action space, and reward function of multi-aircraft air combat; The multi-agent reinforcement learning network design and small-scale multi-aircraft air combat scenario training module are improved on the basis of the known Actor-Critic network, and the LSTM network and the multi-head attention mechanism network are introduced; the Mappo algorithm is adopted, and multiple aircraft of the red side and multiple aircraft of the blue side are set to fight in close range in the multi-aircraft near-visual range air combat simulation platform, and the improved Actor-Critic network is introduced; The large-scale multi-aircraft air combat scenario training and battlefield attention information analysis module based on attention network migration reuses the model trained in small-scale scenarios and puts it into large-scale multi-level air combat scenarios for training; analyzes the output information from the attention networks of both small-scale scenarios and large-scale multi-level air combat scenarios to obtain the degree of battlefield attention of each aircraft to enemy and friendly forces.

2. The multi-agent reinforcement learning air combat decision-making system based on attention network transfer according to claim 1 is characterized by: The multi-aircraft near-visual-range air combat simulation platform uses the dynamics of two-dimensional fixed-wing aircraft and the characteristics of weapons carried to model the following contents: construct a multi-aircraft air combat scenario, in which the aircraft obtains the advantage of firing on the target through tactical maneuvers, with short-range air-to-air missiles and machine guns as the main weapons; the two sides of the air combat are defined as red and blue; the red aircraft is our side, and the tactical strategy is obtained through learning; the blue aircraft is the opponent, and adopts tactics with different difficulty levels based on random action sampling, air combat scripts and virtual self-games to respond; the above scenario is modeled to define the characteristics of the aircraft; and at the same time, according to the partially observable Markov game theory, the state space, action space and reward function are calculated through the position, speed and weapon information of the aircraft.

3. The multi-agent reinforcement learning air combat decision-making system based on attention network transfer according to claim 1 is characterized by: The multi-agent reinforcement learning network design and small-scale multi-aircraft air combat scenario training module include the following contents: The network used is the well-known Actor-Critic architecture, and the input information includes: self information O t,i , Friendly Information t,f ,Enemy InformationO t,j And global information O t,full ; O t,i ,O t,f ,O t,j After a fully connected network, a high-dimensional tensor is obtained, which converts the global information Ot ,full After the embedding operation, it is input into the LSTM network for time series processing, and the obtained tensor is connected to the Transformer module and spliced ​​with the high-dimensional tensor as the decision information of the Actor and Critic networks; training is carried out for the close-range air combat simulation scenario between the two red aircraft and the two blue aircraft; and the multi-agent proximal policy optimization algorithm MAPPO is used for training under the decision network framework.

4. The multi-agent reinforcement learning air combat decision-making system based on attention network transfer according to claim 1 is characterized by: The large-scale multi-aircraft air combat scenario training and attention analysis module based on attention network migration includes the following contents: the Transformer module parameters trained in the close-range dogfight scenario between two red aircraft and two blue aircraft are reused in the close-range dogfight scenario between five red aircraft and five blue aircraft for re-training to improve the model convergence speed.

5. A multi-agent reinforcement learning air combat decision-making method based on attention network transfer can realize multi-agent air combat decision-making training, and realize transfer learning of different tasks and battlefield attention analysis by transferring the attention network. The method is based on the system described in claim 1 and is characterized by: Step 1: Use the aircraft dynamics equation to build a simulation environment for close-range dogfight missions. At the same time, read the aircraft's position and weapon information from the simulation environment, and define the state space, action space, and reward function of multi-agent reinforcement learning. Step 2: Improve the known Actor-Critic network by adding a LSTM time series processing module based on global information and a Transformer architecture, use it as a decision network for multi-intelligence reinforcement learning, and then train it with the MAPPO algorithm; set up a small-scale multi-aircraft air combat scenario, and use the designed network and algorithm to train a close-range air combat simulation scenario between two red aircraft and two blue aircraft; Step 3: Migrate the policy network trained in step 2, reuse and extract the output information of the attention network in the close combat scenario between the five red planes and the five blue planes, and perform attention analysis on the information to obtain the degree of battlefield attention of each agent to the enemy and friendly forces.

6. The multi-agent reinforcement learning air combat decision-making method based on attention network transfer according to claim 5 is characterized by: The step 1 includes the following contents: using short-range air-to-air missiles and machine guns as the main weapons, defining the two sides of the air combat as the red side and the blue side, the red side aircraft as our side, and obtaining the tactical strategy through learning; the blue side aircraft as the opponent, using tactics based on random action sampling, air combat scripts and virtual self-game with different difficulty levels to respond; the modeling equation of the aircraft in the two-dimensional plane is described as follows: Among them: x, y are the horizontal and vertical coordinates of the aircraft, u, v are the horizontal and vertical speeds of the aircraft, r, F is the aircraft's heading angular velocity and heading angle; x ,F y It is the force in the horizontal and vertical directions. It builds a 2D air combat aircraft model that can launch air-to-air missiles and cannon weapons. The characteristics are as follows: Steering angular velocity range: [0,5] degrees / second; Speed ​​range: [200,500] m / s; Short-range missile weapons: Maximum number: 8, range [0,111] km; launch cone angle range [-60,60] degrees centered on the aircraft's longitudinal axis: single-shot kill probability: 0.75; Machine gun weapons: Maximum number: 400; range: [0,2] km, firing cone range: [-60,60] degrees, maximum continuous firing time: 200 seconds.

7. The multi-agent reinforcement learning air combat decision-making method based on attention network migration according to claim 5 is characterized by: The step 2 includes the following contents: the network used is a well-known Actor-Critic architecture; For Actor and Critic networks, the input information includes: self information O t,i , Friendly Information t,f ,Enemy InformationO t,j And global information O t,full ; O t,i ,O t,f ,O t,j After a fully connected network, a high-dimensional tensor is obtained, which converts the global information O t,full After the embedding operation, it is input into the LSTM network for time series processing, and the obtained tensor is connected to the Transformer module and spliced ​​with the high-dimensional tensor as the decision information of the Actor and Critic network; a 2.vs.2 air combat environment is set in the environment, and the red side's tactical strategy is learned by self-play; the size of the geographical scene is 50x50 kilometers; a total of 10,000 episodes are set in the training process, each episode is an air combat process, and the upper limit is 2,000 decision steps; the hyperparameters are set as follows: the learning rate of the Actor and Critic network lr = 0.0001, the discount factor γ = 0.95, the MAPPO hyperparameter shear coefficient ∈ = 0.2, the optimizer is Adam, and the batch size is 2000; the strategies adopted by the blue side aircraft are divided into four levels (L1-L4), and the higher the level, the more complex and diverse the blue side's behavior; the trained red side agent has a stronger ability to adapt to complex environments; the trained agent strategy π is extracted from the Actor network of the trained decision network i ,π j ; The specific classification of blue strategy is as follows: L1: static, the blue team is in a stationary state; L2: random, the blue plane takes random actions within the allowed action space; L3: script, the blue aircraft engages the nearest red aircraft and after engaging, moves away from the red aircraft; L4: Self-play, the Red aircraft uses the Blue strategy trained in L3 to fight against the opponent.

8. The multi-agent reinforcement learning air combat decision-making method based on attention network transfer according to claim 5 is characterized by: Migrate the policy network trained in step 3, and reuse the Transformer module parameters trained in the close-range dogfight scenario between two red planes and two blue planes for re-training in the close-range dogfight scenario between five red planes and five blue planes to improve the model convergence speed; extract the output information of the attention network, and perform attention analysis on the information to obtain the degree of battlefield attention of each agent to the enemy and friendly forces.

9. A non-volatile storage medium, characterized in that: The non-volatile storage medium includes a stored program, wherein the program controls the device where the non-volatile storage medium is located to execute the method according to any one of claims 5 to 8 when the program is executed.

10. An electronic device, characterized in that: It comprises a processor and a memory; the memory stores computer-readable instructions, and the processor is used to execute the computer-readable instructions, wherein the computer-readable instructions execute the method described in any one of claims 5 to 8 when executed.

Citation Information

Patent Citations

  • Stand-alone air combat decision-making method based on curriculum type reinforcement learning

    CN116415646A

  • Multi-unmanned aerial vehicle air combat strategy generation method and device and computer equipment

    CN116430888A

  • Aircraft intelligent collaborative confrontation decision-making method and system based on reinforcement learning

    CN116578113A

  • Close-range air combat maneuver decision-making method based on improved DDPG

    CN116661475A

  • Motorized penetration method combining transfer learning and deep reinforcement learning

    CN117348391A

Cited By

  • Simulation training method, device and equipment for land battle unmanned equipment

    CN119398130A

  • System combat ability dynamic game comprehensive evaluation method based on deep learning

    CN120124859A

  • A dynamic game comprehensive evaluation method for system combat capability based on deep learning

    CN120124859B

  • Multi-aircraft collaborative attack target allocation decision-making method based on near-end strategy optimization

    CN120372818A

  • Defense strategy self-generation method and system for intelligent device cluster

    CN120768612A