A multi-UAV collaborative adversarial learning method based on reinforcement learning

Abstract

The present invention discloses a multi-UAV collaborative confrontation learning method based on reinforcement learning, designs a feature aggregation extraction module based on a communication mechanism between multiple agents and combined with a long short-term memory network, accurately and efficiently extracts battlefield situation features; adopts a hierarchical reinforcement learning method, introduces a bottom-level maneuvering control module and an upper-level combat decision-making module, respectively realizes bottom-level control and upper-level decision-making, and performs training based on an efficient environment parallel module to improve the efficiency of combat strategy training in complex combat scenarios; in addition, the generalization ability of combat strategies is improved through a multi-agent game training module based on population evolution diversity. The present invention can effectively extract battlefield situation features, conduct efficient collaborative combat training, thereby adapting to complex and changeable battlefield environments and improving the generalization ability of autonomous collaborative combat of UAVs.

Description

Technical Field

[0001] The present invention relates to a multi-UAV collaborative confrontation learning method based on reinforcement learning, specifically to an intelligent unmanned air combat decision-making system and method based on multi-agent hierarchical reinforcement learning, belonging to the technical field of UAV cluster collaboration and autonomous air combat decision-making in complex air combat scenarios. Background Art

[0002] In recent years, with the rapid development of Unmanned Combat Aerial Vehicle (UCAV) technology, intelligent unmanned air combat systems have gradually become the core force in modern military confrontation. Traditional air combat decision-making systems mainly rely on expert knowledge, by encoding tactical rules into fixed operating instructions, or using fuzzy systems, evolutionary learning and other technologies to generate response strategies. These methods can play a certain role in tasks with clear rules and simple scenarios, and show relatively stable performance in certain specific military applications. However, with the increasing complexity of air combat environments, especially in the context of multi-agent collaborative operations, the limitations of traditional methods are becoming increasingly apparent. Specifically, traditional tactical rules and strategy generation methods lack sufficient adaptability to changes in the environment and hostile behavior, and are difficult to cope with changes in teammates, unknown opponents, and complex and changing battlefield conditions in modern air combat, which seriously limits the application effect of these systems in actual combat.

[0003] In order to cope with the complex air combat environment, reinforcement learning technology has gradually been introduced into the development of intelligent unmanned air combat systems. Reinforcement learning gradually learns the optimal decision-making strategy through the continuous interaction between the intelligent agent and the environment, which overcomes the reliance on fixed rules and expert knowledge in traditional methods to a certain extent. By utilizing a large amount of interaction data, reinforcement learning algorithms can dynamically adapt to changes in the environment, thereby demonstrating decision-making capabilities that surpass humans in systems such as AlphaGo and AlphaStar. However, despite this, existing reinforcement learning-based UCAV systems are still mainly limited to simple and idealized air combat environments. These systems usually assume that the environmental laws are unchanged, the interaction objects are fixed, and the task types are single. This assumption leads to obvious deficiencies in existing systems when dealing with complex and changeable air combat environments in reality. For example, when UCAV systems face challenges such as information asymmetry and changes in enemy and friendly situations in a multi-agent environment, traditional reinforcement learning methods are difficult to effectively optimize strategies, resulting in unstable or even ineffective performance of the system in actual combat.

[0004] In addition, in many-to-many air combat environments, classic collaborative multi-agent reinforcement learning methods face multiple challenges in information feature extraction, large decision action space, and insufficient generalization of opponent strategies. First, the situation information in air combat environments is complex and changeable, and agents can usually only obtain local observation data, which are often interfered by noise and adversarial factors and are difficult to accurately reflect the actual battlefield situation. In this case, agents are easily affected by unstable information during strategy learning and execution, which makes it difficult for the learning process to converge to the global optimal solution, and the final strategy performance is poor. Secondly, with the increase in the number of agents and the complexity of tasks, the decision action space grows exponentially. When faced with such a huge search space, traditional reinforcement learning methods often find it difficult to effectively explore the optimal strategy, and the strategy optimization process is time-consuming and inefficient. Finally, the opponent strategies in open environments are diverse and unpredictable. Existing reinforcement learning methods are usually trained in a single scenario. The resulting strategies have poor generalization ability when facing new opponents and are difficult to adapt to the changing battlefield environment. The existence of these problems makes intelligent unmanned air combat systems face major challenges in practical applications. It is urgent to develop a multi-agent collaborative decision-making method that can cope with complex and changing environments and improve strategy generalization capabilities. Summary of the invention

[0005] Purpose of the invention: The present invention aims to solve the problems faced by multi-agent air combat decision-making systems in complex many-to-many environments, such as difficulty in extracting situational features, large decision-making action space, and poor strategy generalization ability. By proposing a multi-UAV collaborative adversarial learning method based on reinforcement learning and providing multi-agent air combat decision-making based on a hierarchical collaborative reinforcement learning method, the present invention improves the system's decision-making efficiency, information processing capability, and strategy generalization performance, thereby adapting to complex and changeable air combat scenarios.

[0006] Technical solution: A multi-UAV collaborative adversarial learning method based on reinforcement learning, including the following steps:

[0007] Step 1: According to the current training task, initialize the underlying maneuver control module or the upper-level combat decision module, randomly initialize the parameters of the neural network and optimizer, or load the checkpoint to continue training.

[0008] The training tasks can be divided into "bottom-level training" and "upper-level training". You can choose one of these two tasks to execute. The former selects the bottom-level maneuver control module, and the latter selects the upper-level decision-making module. Both the bottom-level maneuver control module and the upper-level combat decision-making module contain neural network strategies. Optimizing the parameters of these neural networks requires the use of optimizers.

[0009] Generally, the bottom-level maneuver control module is trained first, and the parameters of the neural network and optimizer of the bottom-level maneuver control module are initialized. After the training task of the bottom-level maneuver control module is completed, the training task of the upper-level combat decision module is executed, and the parameters of the neural network and optimizer of the upper-level combat decision module are initialized.

[0010] For a training task, the timed archiving of the parameters of the neural network and optimizer during the training process is the checkpoint, which is a safety measure to deal with unknown emergencies such as power outages during training. When starting a training task, if there is already a checkpoint, you can directly load the checkpoint to continue the previously interrupted training process, thereby saving time and resources. Step 2, if the current training task is to optimize the upper-level combat decision-making module, initialize the multi-agent game training module with diversified population evolution. The multi-agent game training module with diversified population evolution is used to improve the actual combat effect of the upper-level combat decision-making module. Otherwise, proceed to the next step.

[0011] Step 3: Start the efficient environment parallel module, deploy the simulation environment to all available working nodes, and run it in parallel on all working nodes. The master node transmits the strategies of the two adversaries to all working nodes, and the parallel environment is ready.

[0012] The strategies of both sides are: In the "bottom-level training" task, the task goal is to train our bottom-level control strategy and learn how to control the drone. The enemy's strategy is to hover in place and not interfere with the training of our bottom-level control strategy. In the "upper-level training" task, the task goal is to train the upper-level combat strategy and learn coordinated combat. The enemy's strategy is determined by the multi-agent game training module based on population evolution diversity. It may be a copy of our strategy, other expert strategies, etc. Regardless of the task, the simulation environment requires both strategies to make decisions in order to operate normally.

[0013] Step 4: All simulation environments start running.

[0014] Step 5: For a single simulation environment, take a step forward and use the reward function of the underlying maneuver control module or the upper-level combat decision module to calculate the reward, and use the feature aggregation extraction module to process the drone's observation. Taking the simulation environment one step further means that the simulation environment gives the drone's observation information in the current environment, the red and blue strategies (the opposing sides) give control instructions based on the corresponding observation information, the environment is simulated based on the control instructions and the current state, and the changes in the environment are simulated and applied.

[0015] Step 6: The strategies of both adversaries make decisions based on the observations and give instructions to the drone.

[0016] Repeat steps 5 and 6 above until a single simulation environment ends and a complete trajectory is obtained. A trajectory refers to the sequence of states and rewards generated after one or more agents perform a series of actions in the environment in chronological order. A complete trajectory is the trajectory generated from the start of the environment to the natural termination or truncation of the environment.

[0017] Step 8: The working node transmits the complete trajectory generated by a single simulation environment to the master node.

[0018] Repeat steps 5 to 7 above until all environments have completed sampling and data transmission.

[0019] Step 8: The master node organizes all sampled data and prepares for training according to the training task.

[0020] Step 9, according to the current training task, use the optimization method of the underlying maneuver control module to perform reinforcement learning training; or use the optimization method of the upper-level combat decision-making module, and combine the optimization of the upper-level combat decision-making module with the multi-agent game training module based on population evolution diversity to perform reinforcement learning training.

[0021] Step 10: If the current task is an upper-level decision-making module training task, update the opponent pool of the multi-agent game training module based on population evolution diversity.

[0022] Repeat steps 2 to 10 above until the model converges or reaches the maximum number of iterations; the model is a bottom-level maneuver control module or an upper-level combat decision-making module.

[0023] Efficient environment parallel module: By using virtual containers to encapsulate the simulation environment, running it in parallel on all working nodes, parallel sampling of multiple environments is achieved; this module first encapsulates each simulation environment as an independent Docker virtual container; with the help of the Ray distributed framework, the environment of multiple working nodes is started and data is transmitted in parallel. In one iteration, the master node transmits a copy of the combat strategy to all working nodes, starts the specified environment in parallel, and interactively samples the combat strategy and the environment. After the simulation environment completes the sampling, the trajectory is transmitted to the master node. After the master node waits for a certain period of time, it uses the collected data to perform reinforcement learning training optimization strategy. The multi-UAV collaborative adversarial learning method is executed once as one iteration.

[0024] Feature aggregation and extraction module: By combining communication methods and long short-term memory networks (LSTM), the observation information of drones is effectively processed and aggregated to improve the collaboration and decision-making capabilities of multi-drone systems in air confrontation. The specific implementation of the module is as follows: First, the observation information of the drone is processed and divided into five parts, including its own physical information, missile warning information, local observation of enemy aircraft information, weapon information, and attack list information. A communication-based feature aggregation mechanism is introduced. Each drone sends its own physical information and local observation of enemy aircraft information to other drones in the same camp, and then aggregates the received communication information. For enemy aircraft information, the module aggregates the observation results of multiple drones through mean pooling and adds an identification bit to indicate whether friendly observation information is used. The aggregated information includes the relative north-east distance of the enemy aircraft and the sine and cosine values ​​of the relative azimuth and pitch angles. The vector obtained after integration is further used to improve the generalization of the strategy. After completing the preliminary communication aggregation operation, the module further extracts features from the aggregated observation information through the policy network. To this end, the feature extraction module first uses a multi-layer perceptron to perform preliminary processing on various types of observation information and converts it into latent vectors. Then, the latent vectors are spliced ​​to achieve deep aggregation of information. The module uses LSTM to further process historical observation data and extract abstract representations of battlefield information including the self, team, and enemy. Finally, these processed representation vectors are input into the upper-level combat decision module.

[0025] Bottom-level maneuvering control module: responsible for the attitude and speed control of the drone. There are two ways to implement the bottom-level maneuvering control module. The first is a bottom-level maneuvering control method based on reinforcement learning: first, receive input information from the drone sensor; in order to improve the generalization of the model, we try to minimize the absolute amount of information in the input during design, so that the bottom-level network has translation invariance and horizontal rotation invariance, so as to better adapt to various environmental changes. The input features are processed through a power series connected neural network (PSCN); the output of the bottom-level network is used to control the specific actions of the drone. In order to optimize the output space, some parameters such as thrust and overload are pre-set to larger values, while the heading angle, acceleration, and pitch angle parameters are discretized, thereby reducing the output space of the maneuvering number. During the training process, by normalizing the input and randomly sampling the target speed and attitude, the basic reward function during the training process is:

[0026] R=200(d vel / 400+d pitch / 90+d yaw / 360)-0.01p roll -0.1p turn +50×r reach

[0027] Among them, d vel ,d pitch ,d yaw Respectively represent the changes in the target speed, pitch angle and heading angle of the drone compared with the previous moment, p roll represents the penalty for the roll angle changing too much for 5 consecutive time steps, p turn It is the penalty given when the heading angle is greater than 30 degrees from the target, if the overload does not exceed 3G, G represents the acceleration due to gravity, and r reach It is the speed bonus given when reaching the target point. In order to prevent the underlying strategy from over-relying on climb and dive maneuvers to adjust the heading angle, resulting in the inability to achieve a fast turn, when the difference between the heading angle and the target angle is greater than 30 degrees, the heading angle bonus factor will be multiplied by 2, and if the turning direction is opposite to the target direction, an additional penalty will be imposed regardless of whether the overload exceeds 3G. turn The underlying control problem is treated as a single-agent reinforcement learning problem and trained using the Proximal Policy Optimization (PPO) algorithm. Specifically, the algorithm optimizes the actor network π θ (a|s), and introduce the critic network V w (s) The estimated state value function is used to assist in calculating more accurate actor network gradients. The parameters θ of the actor network are optimized by maximizing the expectation formula.

[0028] During training and actual combat, the maneuver number is updated every 5 steps, and the strategy keeps executing the previous maneuver number before the new maneuver number is output. Through the above design and optimization, the module can effectively control the attitude and speed of the drone, and show strong adaptability and flexibility in complex air confrontation environments.

[0029] The bottom-level maneuvering control module in this embodiment also provides another implementation scheme, namely, a bottom-level control method based on macro-actions and skills. In this scheme, macro-actions corresponding to multiple bottom-level maneuvering numbers are directly designed using prior knowledge, including five basic operations: level flight acceleration, left turn, right turn, climb and dive. On the basis of the basic macro-actions, a skill of turning around after firing is also implemented. The specific implementation of this skill includes: first firing at the enemy aircraft in the field of view, and then after maintaining the current heading acceleration for 8 seconds, starting to perform a 180-degree stable turning operation, then accelerating and gradually adjusting the difference with the target angle to achieve the effect of quickly avoiding enemy missiles.

[0030] Upper-level combat decision module: Generates the maneuver and attack instructions of the UAV through a multi-agent reinforcement learning algorithm to achieve efficient air confrontation decision-making. This module first uses the representation vector generated by the feature aggregation extraction module to construct the corresponding upper-level combat strategy network for different underlying control methods. PSCN is used to process high-dimensional input features. For the underlying maneuver control obtained by reinforcement learning training, the upper-level combat strategy network outputs a three-dimensional Gaussian distribution, which represents the target heading angle, target pitch angle, and target speed respectively. The mean of the output is limited to the interval [-1,1] by the hyperbolic tangent function, and inverse normalized to obtain the actual physical control value. These output values ​​are combined with other necessary state information as the input of the underlying maneuver control strategy to further achieve precise control of the UAV. In order to solve the problem of large firing intervals caused by the low frequency of upper-level strategy calls, a continuous firing strategy suitable for hierarchical structure is designed. Specifically, the upper-level strategy uses the representation vector processed by PSCN to output a four-dimensional vector, each dimension represents a Bernoulli distribution, which is used to determine the probability of launching an attack on a certain enemy aircraft. According to the real-time physical conditions of the battlefield, the enemy aircraft that cannot be attacked are masked, and the attack targets are determined by sampling from the remaining Bernoulli distribution. The sampling results are sorted to generate a firing list, and the underlying firing rules control the launch of missiles, target switching, and cooldown timing. For the underlying control based on macro actions and skills, the policy network outputs the distribution of discrete actions. In the strategy learning based on multi-agent reinforcement learning, the multi-agent proximal policy optimization algorithm (MAPPO) is used to optimize the upper-level combat strategy. In order to solve the problem of low training efficiency caused by sparse rewards, a complex reward function including individual rewards and team rewards is designed. The reward of agent (i.e., drone) i at time t includes the firing angle reward , Firing Penalty , Scan Rewards , Scanning Penalty , Search Rewards , Retreat Reward , Survival Reward and energy rewards , and team rewards include damage penalties , Destruction Reward And rewards for victory Specifically, the reward function at a certain time step is:

[0031]

[0032] The goal of algorithm optimization is the policy network. In order to enhance the learning effect of the policy, the critic network is introduced By inputting global state information to predict the cumulative discounted return of each drone under the current strategy, the stability of the strategy learning process is improved. In a training of the upper-level combat decision module, a batch of data is sampled from the data collected in the efficient parallel environment to optimize the loss functions of the policy network and the critic network respectively. The optimization goal of the policy network is:

[0033]

[0034] in is the importance weight, that is, the ratio of the probability of the current strategy to the probability of the strategy producing a certain action at the time of sampling, is the advantage function calculated by the generalized advantage estimation (GAE), clip(•) is the truncation function, ϵ is a small positive number, and the truncated importance weights are used to minimize the deviation between the objective function and the original objective function, encouraging the strategy to maintain a certain stability when exploring new behavior strategies; the critic network parameters The optimization goal is: ,in is the critic network’s estimate of the state value, is the cumulative discounted return. In addition, we also introduce the entropy regularization term in the optimization of the policy network To promote the exploration of strategies, the regular term is defined as: in is the action probability distribution output by the strategy, The entropy function encourages the strategy to maintain a certain degree of randomness and avoids premature convergence to suboptimal strategies or over-determined behavior patterns. Using the above method, the upper-level combat decision module learns combat strategies through multi-agent reinforcement learning algorithms, generates maneuvering instructions and firing instructions, and controls drones as strategy output instructions, thereby achieving efficient collaborative combat in complex air confrontation environments.

[0035] Multi-agent game training module based on population evolution diversity: By introducing an opponent pool in the training process of the upper-level combat decision-making module, the robustness of the strategy is enhanced, and the actual combat performance of the upper-level combat decision-making module is improved. This module introduces an opponent pool mechanism in the training process to enhance the adaptability and robustness of the strategy when facing different types of opponents. Specifically, this module designs a set of opponent populations, including its own model, historical model, rule-based expert opponent, and adversarial population opponent. During the training process, self-game training is carried out, against its own historical model, against the rule baseline, and against the adversarial population opponent. When selecting opponents, sampling is performed according to the probability of the adversary winning rate to enable the model to deal with opponents of different styles and learn more general game strategies. In order to further improve the training efficiency, the module designs a step-by-step strategy learning method based on curriculum learning. First, the combat position is initialized through additional expert rules to guide the red agent to cruise to the east of the center of the battlefield. When the red agent approaches the blue enemy aircraft and the distance meets the conditions, the strategy control is handed over to the reinforcement learning strategy. Considering that it takes a long time to directly conduct strategy learning based entirely on reinforcement learning, and the exploration space is too large, the initial training adopts a state initialization method based on expert rule guidance. Specifically, the latitude and longitude state of the red party agent guided by the expert rule is tested at the time of engagement, and random perturbations within a certain range are performed based on this state to directly enter the engagement state, thereby promoting efficient learning of the strategy. In the later stage of training, in order to better simulate the real battlefield environment, the module combines expert rules with real initialization states. The environment first calls the expert rules to guide the red party agent to approach the enemy aircraft, and after approaching, the reinforcement learning strategy takes over to collect samples and learn strategies. Through this step-by-step strategy learning method, the module can more effectively improve the performance of the strategy while shortening the training time, and enhance the adaptability and robustness to different opponents. In the field of attack and defense confrontation, the red and blue sides are generally used to distinguish the confrontation teams, which is only used for distinction and has no special meaning. In the description of the present invention, for convenience, without loss of generality, our agent is called the red party and the enemy agent is called the blue party.

[0036] A multi-UAV cooperative confrontation learning system based on reinforcement learning includes five modules: an efficient environment parallel module, a feature aggregation extraction module, a bottom-level maneuvering control module, an upper-level combat decision module and a multi-agent game training module based on population evolution diversity. Among them, both the bottom-level maneuvering control module and the upper-level combat decision module include the feature aggregation extraction module, which are distributedly trained through the efficient environment parallel module. First, the bottom-level maneuvering control module is trained to accurately convert the UAV control parameters into output instructions, and then the upper-level combat decision module is trained to control the UAV by outputting the UAV control parameters and combining the bottom-level maneuvering control module to achieve cooperative combat, and the multi-agent game training module based on population evolution diversity is introduced to improve the actual combat performance of the upper-level combat decision module.

[0037] The implementation process and method of the system are the same and will not be repeated here.

[0038] Beneficial effects: Compared with the prior art, the multi-UAV collaborative confrontation learning method based on reinforcement learning of the present invention optimizes situation feature extraction through multi-agent communication mechanism and long short-term memory module, significantly improving the accuracy and efficiency of information processing. The present invention designs a key quantity guidance strategy and macro action skill strategy based on reinforcement learning, which greatly simplifies the control process and improves the response speed and accuracy of decision-making. By implementing a game training framework of population confrontation, the present invention effectively improves the generalization ability of the strategy and can adapt to changing hostile strategies and complex battlefield environments. Experimental results show that the present invention maintains a high winning rate when confronting different opponent strategies, which is superior to traditional rule-based methods and single-agent decision-making methods, showing broad application prospects, especially in the fields of intelligent control and collaborative decision-making of unmanned combat aircraft. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 A multi-agent hierarchical reinforcement learning decision framework diagram according to an embodiment of the present invention;

[0040] Figure 2 A schematic diagram of an adversarial training game framework according to an embodiment of the present invention;

[0041] Figure 3 This is a diagram of the underlying mobile control strategy network architecture of an embodiment of the present invention;

[0042] Figure 4 This is a diagram of a combat strategy network architecture based on underlying mobile control according to an embodiment of the present invention;

[0043] Figure 5 The figure is a complete flow chart of an embodiment of the present invention. DETAILED DESCRIPTION

[0044] The present invention is further explained below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention. After reading the present invention, various equivalent forms of modifications to the present invention by those skilled in the art all fall within the scope defined by the claims attached to this application.

[0045] like Figure 5 The multi-UAV collaborative adversarial learning method based on reinforcement learning comprises the following steps:

[0046] Step 1: According to the current training task, initialize the underlying maneuver control module or the upper-level combat decision module, randomly initialize the parameters of the neural network and optimizer, or load the checkpoint to continue training.

[0047] Step 2: If the current training task is to optimize the upper-level combat decision-making module, initialize the multi-agent game training module with diverse population evolution, which is used to improve the actual combat effect of the upper-level combat decision-making module. Otherwise, proceed to the next step.

[0048] Step 3: Start the efficient environment parallel module, deploy the simulation environment to all available working nodes, and run it in parallel on all working nodes. The master node transmits the strategies of the two adversaries to all working nodes, and the parallel environment is ready.

[0049] Step 4: All simulation environments start running.

[0050] Step 5: For a single simulation environment, take a step further and use the reward function of the underlying maneuver control module or the upper-level combat decision module to calculate the reward, and use the feature aggregation extraction module based on the long short-term memory network to process the drone's observations.

[0051] Step 6: The strategies of both adversaries make decisions based on the observations and give instructions to the drone.

[0052] Repeat steps 5 and 6 above until a single simulation environment ends and a complete trajectory is obtained. A trajectory refers to the sequence of states and rewards generated after one or more agents perform a series of actions in the environment in chronological order. A complete trajectory is the trajectory generated from the start of the environment to the natural termination or truncation of the environment.

[0053] Step 8: The working node transmits the complete trajectory generated by a single simulation environment to the master node.

[0054] Repeat steps 5 to 7 above until all environments have completed sampling and data transmission.

[0055] Step 8: The master node organizes all sampled data and prepares for training according to the training task.

[0056] Step 9, according to the current training task, use the optimization method of the underlying maneuver control module to perform reinforcement learning training; or use the optimization method of the upper-level combat decision-making module, and combine the optimization of the upper-level combat decision-making module with the multi-agent game training module based on population evolution diversity to perform reinforcement learning training.

[0057] Step 10: If the current task is an upper-level decision-making module training task, update the opponent pool of the multi-agent game training module based on population evolution diversity.

[0058] Repeat steps 2 to 10 above until the model converges or reaches the maximum number of iterations; the model is a bottom-level maneuver control module or an upper-level combat decision-making module.

[0059] Efficient environment parallel module: By using virtual containers to encapsulate the simulation environment and running it in parallel on all working nodes, efficient parallel sampling of multiple environments can be achieved, thereby significantly improving the efficiency of reinforcement learning training. Specifically, this module first encapsulates each simulation environment as an independent Docker virtual container to avoid mutual interference and resource competition between different environments, and ensure that multiple environments can run in parallel. With the help of the Ray distributed framework, the parallel startup and data transmission of multiple working nodes are realized. In one iteration, the master node transmits a copy of the battle strategy to all working nodes, and starts the specified environment in parallel. The battle strategy and the environment are interactively sampled. After the simulation environment completes sampling, the trajectory is immediately transmitted to the master node. After the master node waits for a certain period of time, it uses the collected data to optimize the reinforcement learning training strategy.

[0060] Feature aggregation and extraction module: By combining communication methods and long short-term memory networks (LSTM), the observation information of drones is effectively processed and aggregated to improve the collaboration and decision-making capabilities of multi-drone systems in air confrontation. The specific implementation method of the module first processes the observation information of drones and divides the observation information into five parts, including their own physical information, missile warning information, local observation of enemy aircraft information, weapon information, and attack list information. In order to enhance the accuracy and collaboration of observation information, the module introduces a communication-based feature aggregation mechanism. Each drone sends its own physical information and local observation of enemy aircraft information to other drones in the same camp, and then aggregates the received communication information. Especially for enemy aircraft information, the module aggregates the observation results of multiple drones through mean pooling and adds an identification bit of whether friendly observation information is used. The aggregated information includes the relative north sky east distance of the enemy aircraft and the sine and cosine values ​​of the relative azimuth and pitch angles. The vector obtained after integration is further used to improve the generalization of the strategy. After completing the preliminary communication aggregation operation, the module further extracts features from the aggregated observation information through the policy network. To this end, the feature extraction module first uses a multi-layer perceptron to perform preliminary processing on various types of observation information and converts it into latent vectors. Then, the latent vectors are spliced ​​to achieve deep aggregation of information. In order to overcome the local observability problem and make full use of historical information, the module uses LSTM to further process historical observation data and extract an abstract representation of battlefield information including itself, the team, and the enemy. Finally, these processed representation vectors are input into the upper-level combat decision-making module to assist the UAV system in making more intelligent and coordinated decisions in a dynamic and complex air confrontation environment.

[0061] The bottom-level maneuvering control module is responsible for the attitude and speed control of the drone. We have designed two implementation methods. The first is a bottom-level maneuvering control method based on reinforcement learning. Specifically, Figure 3As shown in , the module first receives input information from the drone sensor. In order to improve the generalization of the model, we try to minimize the absolute amount of information in the input during design, so that the underlying network has translation invariance and horizontal rotation invariance, so as to better adapt to various environmental changes. The input features are processed by the power series connection neural network (PSCN), which reduces the redundancy of parameters while ensuring the depth and breadth of the network. The output of the underlying network includes 4-dimensional maneuver number, 21-dimensional heading angle, 4-dimensional pitch angle and 13-dimensional speed command, which are used to control the specific actions of the drone respectively. In order to optimize the output space, some parameters such as thrust and overload are pre-set to larger values, while the heading angle, acceleration and pitch angle parameters are discretized, thereby reducing the output space of the maneuver number. During the training process, the input is normalized and the target speed and attitude are randomly sampled, so that the network can learn the strategy of reaching a given target point at a higher speed more quickly. The basic reward function during training is:

[0062] R=200(d vel / 400+d pitch / 90+d yaw / 360)-0.01p roll -0.1p turn +50×r reach

[0063] Among them, d vel ,d pitch ,d yaw Respectively represent the changes in the target speed, pitch angle and heading angle of the drone compared with the previous moment, p roll represents the penalty for the roll angle changing too much for 5 consecutive time steps, p turn It is the penalty given when the heading angle is greater than 30 degrees from the target, if the overload does not exceed 3G, G represents the acceleration due to gravity, and r reach It is the speed bonus given when reaching the target point. In order to prevent the underlying strategy from over-relying on climb and dive maneuvers to adjust the heading angle, resulting in the inability to achieve a fast turn, when the difference between the heading angle and the target angle is greater than 30 degrees, the heading angle bonus factor will be multiplied by 2, and if the turning direction is opposite to the target direction, an additional penalty will be imposed regardless of whether the overload exceeds 3G. turn The underlying control problem is treated as a single-agent reinforcement learning problem, and the widely verified proximal policy optimization (PPO) algorithm is used for training. Specifically, the algorithm optimizes the actor network π θ (a|s), and introduce the critic network V w (s) The estimated state value function is used to assist in calculating a more accurate actor network gradient. The parameters θ of the actor network are optimized by maximizing the expected formula:

[0064]

[0065] Among them, assuming that k training iterations have been completed, then θ k represents the parameters before optimization, θ k+1 represents the optimized parameters, o t ,a t is the observation and action of the tth step in a trajectory in the training data, is the importance weight, that is, the ratio of the probability of the current strategy to the probability of the strategy producing a certain action at the time of sampling, is the advantage function calculated by the generalized advantage estimation (GAE), clip(.) is the truncation function, ϵ is a small positive number not exceeding 0.3, and is generally taken as 0.1. The truncated importance weight is used to minimize the deviation between the objective function and the original objective function, while encouraging the strategy to maintain a certain stability when exploring new behavior strategies; the parameter w of the critic network is optimized by minimizing the error:

[0066]

[0067] Among them, w k+1 represents the optimized critic network parameters, V w (S t ) is the critic network’s estimate of the state value, is the cumulative discounted return, represents expectation, that is, the goal of optimizing the critic network parameter w is to minimize the expected error of its estimated state value. During training and actual combat, the maneuver number is updated every 5 steps, and the strategy keeps executing the previous maneuver number before the new maneuver number is output. Through the above design and optimization, the module can effectively control the attitude and speed of the drone, and show strong adaptability and flexibility in complex air confrontation environments.

[0068] The underlying maneuvering control module in this embodiment also provides another implementation scheme, namely, the underlying control method based on macro actions and skills. In this scheme, we use prior knowledge to directly design macro actions corresponding to multiple underlying maneuvering numbers, including five basic operations: level flight acceleration, left turn, right turn, climb and dive. Specifically, each macro action calls a different maneuvering number and sets related parameters. Level flight acceleration calls the acceleration and deceleration maneuvering number, sets the speed to the current speed plus 0.2 Mach, and the thrust limit is 100% of the maximum value; both left and right turns call the stable turning maneuvering number, set the turning angle to 90 degrees, and call the leveling maneuvering number before turning to increase the turning speed and overload. Dive and climb respectively call the dive and fastest climb maneuvering numbers, and set the corresponding pitch angle and thrust parameters. In order to meet more complex tactical needs, we have also implemented a skill of turning around after firing on the basis of basic macro actions. The specific implementation of this skill includes: first, fire at the enemy aircraft in the field of view, then maintain the current heading and accelerate for 8 seconds, start a stable 180-degree turn, then accelerate and gradually adjust the difference in angle with the target to achieve the effect of quickly avoiding enemy missiles.

[0069] Upper-level combat decision module: Generates the maneuver and attack instructions of the drone through the multi-agent reinforcement learning algorithm to achieve efficient air confrontation decision-making. This module first uses the representation vector generated by the feature aggregation extraction module to build the corresponding upper-level combat strategy network for different underlying control methods. Use PSCN to process high-dimensional input features, such as Figure 4As shown in Figure 1, for the underlying maneuver control obtained through reinforcement learning training, the upper-level combat strategy network outputs a three-dimensional Gaussian distribution, which represents the target heading angle, target pitch angle, and target speed. The mean of the output is limited to the interval through the hyperbolic tangent function, and inverse normalization is performed to obtain the actual physical control value. These output values ​​are combined with other necessary state information as the input of the underlying maneuver control strategy to further achieve precise control of the UAV. In order to solve the problem of large firing intervals caused by the low frequency of upper-level strategy calls, we designed a continuous firing strategy suitable for hierarchical structures. Specifically, the upper-level strategy uses the representation vector processed by PSCN to output a four-dimensional vector, each dimension represents a Bernoulli distribution, which is used to determine the probability of launching an attack on a certain enemy aircraft. According to the real-time physical situation of the battlefield, the enemy aircraft that cannot be attacked is masked, and the attack target is determined by sampling from the remaining Bernoulli distribution. The sampling results are sorted to generate a firing list, and the underlying firing rules control the missile launch, target switching, and cooling timing. For the underlying control based on macro actions and skills, the strategy network outputs the distribution of discrete actions. In the strategy learning based on multi-agent reinforcement learning, we use the multi-agent proximal policy optimization algorithm (MAPPO) to optimize the upper-level combat strategy. To solve the problem of low training efficiency caused by sparse rewards, we designed a complex reward function that includes individual rewards and team rewards. The reward of agent (i.e., drone) i at time t includes the firing angle reward , Firing Penalty , Scan Rewards , Scanning Penalty , Search Rewards , Retreat Reward , Survival Reward and energy rewards , and team rewards include damage penalties , Destruction Reward And rewards for victory Specifically, the reward function at a certain time step is:

[0070]

[0071] The goal of algorithm optimization is the policy network. In order to enhance the learning effect of the policy, the critic network is introduced By inputting global state information to predict the cumulative discounted return of each drone under the current strategy, the stability of the strategy learning process is improved. In a training of the upper-level combat decision module, a batch of data is sampled from the data collected in the efficient parallel environment to optimize the loss functions of the policy network and the critic network respectively. The optimization goal of the policy network is:

[0072]

[0073] in is the importance weight, that is, the ratio of the probability of the current strategy to the probability of the strategy producing a certain action at the time of sampling, is the advantage function calculated by the generalized advantage estimation (GAE), clip(•) is the truncation function, ϵ is a small positive number, and the truncated importance weights are used to minimize the deviation between the objective function and the original objective function, encouraging the strategy to maintain a certain stability when exploring new behavior strategies; the critic network parameters The optimization goal is: ,in is the critic network’s estimate of the state value, is the cumulative discounted return. In addition, we also introduce the entropy regularization term in the optimization of the policy network To promote the exploration of strategies, the regular term is defined as: in is the action probability distribution output by the strategy, The entropy function encourages the strategy to maintain a certain degree of randomness and avoids premature convergence to suboptimal strategies or over-determined behavior patterns. Using the above method, the upper-level combat decision module learns combat strategies through multi-agent reinforcement learning algorithms, generates maneuvering instructions and firing instructions, and controls drones as strategy output instructions, thereby achieving efficient collaborative combat in complex air confrontation environments.

[0074] Multi-agent game training module based on population evolution diversity: By introducing the opponent pool in the training process of the upper-level combat decision-making module, the robustness of the strategy is enhanced and the actual combat performance of the upper-level combat decision-making module is improved. Figure 2As shown in Figure 1, this module introduces an opponent pool mechanism during the training process to enhance the adaptability and robustness of the strategy when facing different types of opponents. Specifically, this module designs a group of opponent populations, including the self-model, the historical model, the rule-based expert opponent, and the adversary population opponent. During the training process, we conduct self-game training with a probability of 25%, the probability of competing with the self-historical model with a probability of 25%, the probability of competing with the rule baseline with a probability of 20%, and the probability of competing with the adversary population opponent with a probability of 30%. When selecting opponents, we sample according to the probability of winning the competition to enable the model to deal with opponents of different styles and learn more general game strategies. In order to further improve the training efficiency, the module designs a step-by-step strategy learning method based on curriculum learning. First, the combat position is initialized through additional expert rules to guide the red agent to cruise to the east of the center of the battlefield. When the red agent approaches the blue enemy aircraft and the distance meets the conditions, the strategy control is handed over to the reinforcement learning strategy. Considering that it takes a long time to directly conduct strategy learning based entirely on reinforcement learning and the exploration space is too large, the initial training adopts the state initialization method guided by expert rules. Specifically, the longitude and latitude state of the Red Agent guided by the expert rules is tested at the time of engagement, and random perturbations within a certain range are performed based on this state to directly enter the engagement state, thereby promoting efficient learning of the strategy. In the later stages of training, in order to better simulate the real battlefield environment, the module combines the expert rules with the real initialization state. The environment first calls the expert rules to guide the Red Agent to approach the enemy aircraft, and after approaching, the reinforcement learning strategy takes over to collect samples and learn the strategy. Through this step-by-step strategy learning method, the module can more effectively improve the performance of the strategy while shortening the training time, and enhance the adaptability and robustness to different opponents.

[0075] The multi-UAV cooperative confrontation learning system based on reinforcement learning includes five modules: efficient environment parallel module, feature aggregation extraction module, bottom-level maneuvering control module, upper-level combat decision module and multi-agent game training module based on population evolution diversity. Among them, both the bottom-level maneuvering control module and the upper-level combat decision module contain feature aggregation extraction modules, which are distributedly trained through the efficient environment parallel module. First, the bottom-level maneuvering control module is trained to accurately convert the UAV control parameters into output instructions, and then the upper-level combat decision module is trained to control the UAV by outputting the UAV control parameters and combining the bottom-level maneuvering control module to achieve cooperative combat, and the multi-agent game training module based on population evolution diversity is introduced to improve the actual combat performance of the upper-level combat decision module.

Claims

1. A multi-UAV collaborative adversarial learning method based on reinforcement learning, characterized in that: The steps include: Step 1: Initialize the underlying maneuver control module or the upper-level combat decision module according to the current training task, randomly initialize the parameters of the neural network and optimizer, or load the checkpoint to continue training; Step 2: If the current training task is to optimize the upper-level combat decision-making module, then initialize the multi-agent game training module with diversified population evolution, which is used to improve the actual combat effect of the upper-level combat decision-making module; otherwise, proceed to the next step; Step 3: Start the efficient environment parallel module, deploy the simulation environment to all available working nodes, and run it in parallel on all working nodes. The master node transmits the strategies of the two adversaries to all working nodes, and the parallel environment is ready. Step 4, all simulation environments start running; Step 5: For a single simulation environment, take a step further and use the reward function of the underlying maneuver control module or the upper-level combat decision module to calculate the reward, and use the feature aggregation extraction module to process the drone's observations; Step 6: The strategies of both adversaries make decisions based on the observations and give instructions to the drones; Repeat steps 5 and 6 above until a single simulation environment finishes running and a complete trajectory is obtained; Step 7, the working node transmits the complete trajectory generated by a single simulation environment to the master node; Repeat steps 5 to 7 above until all environments have completed sampling and data transmission; Step 8: The master node organizes all sampled data and prepares for training according to the training task; Step 9: According to the current training task, the optimization method of the underlying maneuver control module is used for reinforcement learning training; or the optimization method of the upper-level combat decision module is used, and the optimization of the upper-level combat decision module is combined with the multi-agent game training module based on population evolution diversity for reinforcement learning training; Step 10: If the current task is the upper-level decision-making module training task, update the opponent pool of the multi-agent game training module based on population evolution diversification; Repeat steps 2 to 10 above until the model converges or reaches the maximum number of iterations; the model is a bottom-level maneuver control module or an upper-level combat decision-making module.

2. The multi-UAV collaborative adversarial learning method based on reinforcement learning according to claim 1 is characterized in that: The efficient environment parallel module uses virtual containers to encapsulate the simulation environment and runs in parallel on all working nodes to achieve parallel sampling of multiple environments. First, each simulation environment is encapsulated as an independent Docker virtual container; With the help of the Ray distributed framework, the environment of multiple working nodes can be started in parallel and data can be transmitted. In one iteration of the multi-UAV collaborative adversarial learning method, the master node transmits a copy of the combat strategy to all working nodes, starts the specified environment in parallel, and interactively samples the combat strategy and the environment. After the simulation environment completes the sampling, the trajectory is transmitted to the master node, and the master node uses the collected data to perform reinforcement learning training to optimize the strategy.

3. The multi-UAV collaborative adversarial learning method based on reinforcement learning according to claim 1 is characterized in that: The feature aggregation extraction module processes and aggregates the observation information of the UAV by combining the communication method and the long short-term memory network. The specific implementation method is as follows: first, the observation information of the UAV is processed, and the observation information is divided into five parts including its own physical information, missile warning information, local observation of enemy aircraft information, weapon information and attack list information; a communication-based feature aggregation mechanism is introduced; each UAV sends its own physical information and local observation of enemy aircraft information to other UAVs in the same camp, and then aggregates the received communication information. For enemy aircraft information, the module aggregates the observation results of multiple UAVs through mean pooling and adds Whether the identification bit of friendly observation information is utilized; the aggregated information includes the relative north-east distance of the enemy aircraft and the sine and cosine values ​​of the relative azimuth and pitch angles; after completing the initial communication aggregation operation, the module further extracts features from the aggregated observation information through the policy network; the feature extraction module first uses a multi-layer perceptron to perform preliminary processing on various types of observation information and converts the observation information into latent vectors; then, the latent vectors are spliced ​​to achieve deep aggregation of information; the module uses LSTM to extract an abstract representation of battlefield information including itself, the team, and the enemy based on historical observation data; finally, the representation vector is input into the upper-level combat decision-making module.

4. The multi-UAV collaborative adversarial learning method based on reinforcement learning according to claim 1 is characterized in that: There are two ways to implement the underlying maneuver control module. The first is a reinforcement learning-based underlying maneuver control method: first, receive input information from the drone sensor; process the input features through a power series connected neural network; the output of the underlying network is used to control the specific actions of the drone; during the training process, the input is normalized and the target speed and posture are randomly sampled. The basic reward function during the training process is: R=200(d vel / 400+d pitch / 90+d yaw / 360)-0.01p roll -0.1p turn +50×r reach Among them, d vel ,d pitch ,d yaw Respectively represent the changes in the target speed, pitch angle and heading angle of the drone compared with the previous moment, p roll represents the penalty for the roll angle changing too much for 5 consecutive time steps, p turn It is the penalty given when the heading angle is greater than 30 degrees from the target, if the overload does not exceed 3G, G represents the acceleration due to gravity, and r reach It is a speed bonus given when reaching the target point.

5. The multi-UAV collaborative adversarial learning method based on reinforcement learning according to claim 4 is characterized in that: There are two implementation methods for the bottom-level maneuvering control module. The second implementation scheme is a bottom-level control method based on macro actions and skills. It directly designs macro actions corresponding to multiple bottom-level maneuvering numbers using prior knowledge, including five basic operations: level flight acceleration, left turn, right turn, climb and dive. In addition to the basic macro actions, a skill for evading by turning after firing is also implemented; the specific implementation of this skill includes: first firing at the enemy aircraft in the field of view, then starting to execute the turning operation after maintaining the current heading acceleration, and then accelerating and gradually adjusting the difference in angle with the target to achieve the effect of quickly evading enemy missiles.

6. The multi-UAV collaborative adversarial learning method based on reinforcement learning according to claim 1 is characterized in that: The upper-level combat decision module: generates the maneuver and attack instructions of the UAV through a multi-agent reinforcement learning algorithm to achieve efficient air confrontation decision-making; firstly, the characterization vector generated by the feature aggregation extraction module is used to construct the corresponding upper-level combat strategy network for different underlying control methods; the PSCN is used to process the high-dimensional input features, and for the underlying maneuver control obtained by reinforcement learning training, the upper-level combat strategy network outputs a three-dimensional Gaussian distribution, which respectively represents the target heading angle, target pitch angle and target speed; the mean value of the output is limited to the interval [-1,1] through the hyperbolic tangent function, and inverse normalization is performed to obtain the actual physical control value; the output of the upper-level combat decision module is used as the input of the underlying maneuver control strategy module to realize the control of the UAV; Using the representation vector processed by PSCN, a four-dimensional vector is output, where each dimension represents a Bernoulli distribution to determine the probability of launching an attack on a certain enemy aircraft. According to the real-time physical situation of the battlefield, the enemy aircraft that cannot be attacked are masked, and the attack targets are determined by sampling from the remaining Bernoulli distribution. The sampling results are sorted to generate a firing list, and the underlying firing rules control the missile launch, target switching and cooling timing operations. For the underlying control based on macro-actions and skills, the policy network outputs the distribution of discrete actions; in the policy learning based on multi-agent reinforcement learning, the multi-agent proximal policy optimization algorithm is used to optimize the upper-level combat strategy.

7. The multi-UAV collaborative adversarial learning method based on reinforcement learning according to claim 6 is characterized in that: A complex reward function including individual rewards and team rewards is designed in the upper-level combat decision-making module; the reward of a certain UAV at a certain time step includes firing angle reward, firing penalty, scanning reward, being scanned penalty, search reward, retreat reward, survival reward and energy reward, while the team reward includes damage penalty, destruction reward and combat result reward; the reward function at a certain time step is the weighted sum of individual reward and team reward.

8. The multi-UAV collaborative adversarial learning method based on reinforcement learning according to claim 6 is characterized in that: The target of optimization of the multi-agent reinforcement learning algorithm is the policy network. The critic network is introduced to predict the cumulative discounted return of each drone under the current strategy by inputting global state information. In one training of the upper-level combat decision module, a batch of data is sampled from the data collected in the efficient parallel environment, and the loss functions of the policy network and the critic network are optimized respectively. The upper-level combat decision module learns the combat strategy through the multi-agent reinforcement learning algorithm, generates maneuver instructions and firing instructions, and controls the drone as the policy output instructions.

9. The multi-UAV collaborative adversarial learning method based on reinforcement learning according to claim 1 is characterized in that: The multi-agent game training module based on population evolution diversification introduces an opponent pool mechanism in the training process. This module designs a group of opponent populations, including self-models, historical models, rule-based expert opponents, and adversarial population opponents. During the training process, self-game training is carried out respectively, against its own historical model, against the rule baseline, and against opponents from the adversarial population; when selecting opponents, sampling is carried out according to the probability of winning the adversary; the module designs a step-by-step strategy learning method based on curriculum learning. First, the combat position is initialized through expert rules to guide the red agent to cruise to the east of the center of the battlefield; when the red agent approaches the blue enemy aircraft and the distance meets the conditions, the strategy control is handed over to the reinforcement learning strategy.

10. A multi-UAV cooperative confrontation learning system based on reinforcement learning that implements the method according to any one of claims 1 to 9, characterized in that: It includes five modules: an efficient environment parallel module, a feature aggregation and extraction module, a bottom-level maneuvering control module, an upper-level combat decision-making module, and a multi-agent game training module based on population evolution diversity. Among them, both the bottom-level maneuvering control module and the upper-level combat decision-making module contain feature aggregation and extraction modules. They are distributedly trained through the efficient environment parallel module. First, the bottom-level maneuvering control module is trained to accurately convert the UAV control parameters into output instructions. Then, the upper-level combat decision-making module is trained to control the UAV by outputting the UAV control parameters and combining the bottom-level maneuvering control module to achieve coordinated combat. In addition, a multi-agent game training module based on population evolution diversity is introduced to improve the actual combat performance of the upper-level combat decision-making module.

Images