Unmanned aerial vehicle reinforcement learning confrontation method based on multi-agent proximal policy optimization

Abstract

This invention discloses a UAV reinforcement learning adversarial method based on multi-agent proximal policy optimization. The method utilizes a UAV swarm adversarial training and control system based on multi-agent proximal policy optimization to achieve UAV reinforcement learning adversarial training. The overall architecture of the UAV swarm adversarial training and control system follows a "centralized training, distributed execution" paradigm, mainly consisting of four parts: an environment interaction interface module, a distributed policy generation module, a centralized value evaluation module, and a policy optimization and update module. By introducing self-game theory and robustness enhancement strategies, this invention forms a stable, efficient, and transferable UAV intelligent decision-making scheme suitable for complex adversarial environments. It solves the technical problems of existing multi-agent deep deterministic policy gradient algorithms in UAV swarm adversarial tasks, such as policy convergence difficulties, poor adaptability to discrete tactical actions, and low training efficiency due to input dimension explosion in large-scale formations.

Description

Technical Field

[0001] This invention belongs to the field of intelligent control and autonomous system technology, and relates to a reinforcement learning adversarial method for unmanned aerial vehicles (UAVs). Specifically, it relates to an adversarial decision-making and control method for UAVs based on proximal policy optimization (PPO) in a multi-agent scenario. Background Technology

[0002] Deep Reinforcement Learning (DRL) has been widely applied in Unmanned Aerial Vehicle (UAV) control tasks, especially suitable for scenarios where the model is unknown, the dynamics are complex, or it is difficult to accurately model. DRL adaptively learns optimal policies through interaction with the environment and has demonstrated good results in tasks such as UAV path planning, dynamic target interception, obstacle avoidance control, and multi-UAV formation. In the field of continuous motion control, policy gradient methods, represented by Trust Region Policy Optimization (TRPO) and Proximal Policy Optimization (PPO), balance training stability and sample efficiency, becoming the most representative reinforcement learning methods in current engineering practice. As UAV applications gradually evolve to multi-UAV cooperative and adversarial tasks, the effectiveness of single-agent reinforcement learning is limited, and multi-agent reinforcement learning (MARL) has become an important technical means to achieve complex cooperative and adversarial behavior modeling.

[0003] In the development of MARL, the Centralized Training with Decentralized Execution (CTDE) paradigm has been widely adopted. This paradigm utilizes global state, joint actions, and centralized value assessment information during the training phase to mitigate the adverse effects of environmental non-stationarity during multi-agent learning. During the execution phase, each UAV relies solely on its own local observations for real-time autonomous decision-making, achieving a combination of theoretical feasibility and engineering deployability. Based on this, several representative MARL methods have been proposed. For example, Multi-Agent Proximal Policy Optimization (MAPPO), within the CTDE framework, combines the PPO's cut-update mechanism and uses a centralized value network for advantage estimation, significantly improving training stability and generalization ability under high-dimensional continuous control and partially observable conditions. Multi-Agent Deep Deterministic Policy Gradient (MADDPG) handles continuous action scenarios through a centralized commentator structure; Counterfactual Multi-Agent Policy Gradient (COMA) introduces a counterfactual baseline to alleviate the credit allocation problem; Value-Decomposition Networks (VDN) and Hybrid Q-Networks (QMIX) enable scalable multi-agent collaborative decision-making by structurally decomposing centralized value functions. These algorithms have made key progress in cooperative control, adversarial games, and multi-machine dynamic decision-making. Furthermore, to address the strong nonstationarity caused by the continuous changes in adversarial agent policies, researchers have proposed frameworks such as Self-Play, Population-Based Training, and Policy Space Response Oracle (PSRO) to obtain robust policies closer to game equilibrium through iterative optimal responses, policy library management, and opponent pairing training. Meanwhile, Robust Adversarial Reinforcement Learning (RARL) improves the policy's resilience to disturbances in uncertain environments by introducing perturbations or adversaries into the training process to optimize worst-case performance.In conjunction with these algorithm systems, high-fidelity UAV simulation platforms such as Multi-Agent Particle Environment (MPE), StarCraft Multi-Agent Challenge (SMAC), AirSim, and Gazebo / PX4-SITL provide a rich experimental foundation for algorithm verification and performance evaluation.

[0004] While the aforementioned technologies have significantly advanced the development of multi-UAV adversarial decision-making, several unresolved challenges remain when facing real-world UAV adversarial tasks. First, the constantly changing strategies of each UAV in adversarial scenarios lead to strong non-stationarity in the learning process, making strategy convergence difficult and prone to training oscillations. Second, the complex reward coupling in multi-UAV collaborative-competitive tasks makes it difficult for a single agent's reward to reflect its true contribution, exacerbating the credit allocation problem. Third, UAV systems possess highly nonlinear dynamic characteristics and are affected by partial observability, sensor noise, airframe constraints, and real-time requirements, resulting in problems such as incomplete states, shrinking action feasible domains, and easily violated safety boundaries in policy learning. Furthermore, limited communication bandwidth between UAVs, unstable links, and susceptibility to interference make it difficult to balance the effectiveness and communication cost of information sharing strategies, further increasing the difficulty of learning collaborative and adversarial behaviors. More importantly, discrepancies in dynamic modeling, differences in external disturbances, and sensor uncertainties still exist between the simulation environment and the real physical environment, leading to performance degradation and even security risks when transferring strategies from simulation to reality. While existing MARL, PPO extension methods and adversarial training frameworks have made progress in certain aspects, they have not yet formed a solution that can systematically address the comprehensive challenges of multiple constraints, partial observation, limited communication, high real-time requirements, and high adversarial nature in UAV multi-agent adversarial missions.

[0005] The aforementioned prior art laid the methodological foundation for the present invention’s approach of “stable and efficient training under the PPO / CTDE framework for multi-agent adversarial scenarios of UAVs, combined with self-game and robustness mechanisms”, but still has not systematically solved the comprehensive challenges unique to UAV scenarios, such as multiple constraints, partial observability, communication / bandwidth limitations, real-time performance and simulation-reality transfer.

[0006] The MADDPG algorithm, as the closest technical solution to this invention, is suitable for continuous actions and hybrid cooperative-adversarial scenarios. Each agent outputs continuous actions; a centralized Critic inputs the global state / joint actions to minimize TD error; experience replay is off-policy and includes historical step-size information; network updates only update the target network and use soft updates; noise-generating actions are added during the exploration phase. This algorithm has the following problems:

[0007] (1) The MADDPG algorithm adopts a deterministic policy output. When dealing with the discrete action space (such as target selection, maneuver mode switching, weapon launch decision, etc.) commonly encountered in UAV combat, it is necessary to introduce heavy parameterization techniques such as Gumbel-Softmax for approximation. This not only increases the complexity of the algorithm implementation, but also causes the policy gradient estimation deviation due to the approximation error, making it difficult to accurately capture refined tactical actions.

[0008] (2) The Off-Policy mechanism relies heavily on historical data in the experience replay pool for training. In the game environment of drone confrontation, which is highly non-stationary and the opponent's strategy is dynamically changing, there is often a huge difference between the distribution of historical data and the current strategy distribution. This can easily lead to overestimation of Q value and training divergence, making it difficult for the strategy to converge to the Nash equilibrium point.

[0009] (3) MADDPG’s centralized critic network requires all agents’ actions as input. As the size of the drone formation increases, the dimension of the input space grows linearly or even exponentially, which significantly increases computational overhead and reduces training efficiency, making it difficult to adapt to the needs of large-scale swarm adversarial tasks. Summary of the Invention

[0010] To address the technical challenges of existing Multi-Agent Deep Deterministic Policy Gradient (MADDPG) algorithms in UAV swarm adversarial tasks, such as policy convergence difficulties, poor adaptability to discrete tactical actions, and low training efficiency due to input dimension explosion in large-scale formations, this invention provides a UAV reinforcement learning adversarial method based on multi-agent proximal policy optimization. This method applies Multi-Agent Reinforcement Learning (MARL) to adversarial and cooperative game-based tasks. By introducing self-game theory and robustness enhancement strategies, it forms a stable, efficient, and transferable intelligent decision-making scheme for UAVs suitable for complex adversarial environments, thereby overcoming the shortcomings of existing technologies in UAV scenarios.

[0011] The objective of this invention is achieved through the following technical solution:

[0012] A UAV swarm adversarial training and control system based on multi-agent proximal policy optimization comprises four parts: an environment interaction interface module, a distributed policy generation module (Actor), a centralized value evaluation module (Critic), and a policy optimization and update module.

[0013] The environment interaction interface module is responsible for collecting multi-source heterogeneous data in the simulation environment in real time, transforming the original physical quantities into normalized local observation vectors and global state vectors through feature engineering, and decoding discrete action commands into simulation control signals.

[0014] The distributed strategy generation module is deployed on each UAV agent and is responsible for using a parameter-shared random strategy network to independently output discrete action probability distributions based on their respective local observation vectors, thereby achieving decentralized tactical decision-making.

[0015] The centralized value assessment module only runs during the training phase and is responsible for using the global state vector to perform an overall value assessment of the current battlefield situation, providing a low-variance benchmark signal for strategy updates.

[0016] The policy optimization and update module is responsible for calculating the advantage function based on the trajectory data collected online using generalized advantage estimation (GAE), and optimizing the objective function by maximizing the pruned proximal policies, while simultaneously updating the parameters of the policy network and the value network, thereby achieving continuous evolution of the cluster collaborative policy while ensuring training stability.

[0017] A method for implementing UAV reinforcement learning adversarial attacks based on multi-agent proximal policy optimization using the above system includes the following steps:

[0018] Step 1: Utilize the environment interaction interface module to collect multi-source heterogeneous data from the simulation environment in real time. Through feature engineering, transform the original physical quantities into normalized local observation vectors and global state vectors, and decode discrete action commands into simulation control signals. The specific steps are as follows:

[0019] Step 11: The environment interaction interface module subscribes to and obtains battlefield situation data in real time through the communication interface provided by the simulation platform, including the dynamic state of friendly UAVs, enemy target information detected by airborne sensors, and weapon system status.

[0020] Step 12: After acquiring the raw data, the environment interaction interface module performs feature engineering processing to transform heterogeneous physical quantities into local observation vectors and global state vectors suitable for neural network input. The local observation vectors contain only the perception information of a single UAV and the local situation within a limited range, used to simulate the single-UAV perception capability under communication-limited conditions in a real battlefield, and serve as the input to the distributed strategy generation module. The global state vectors are composed of the absolute positions, velocities, survival status, and global alarm information of all enemy and friendly entities on the battlefield, used to provide a "God's-eye view" for the centralized value assessment module during the training phase.

[0021] Step 13: The environment interaction interface module receives the discrete action index output by the distributed strategy generation module and maps it into low-level control commands that the simulation platform can recognize.

[0022] Step 2: The distributed strategy generation module receives local observation vectors from the environment interaction interface module, performs feature extraction and inference through a deep neural network, outputs the probability distribution of the action space, and generates specific tactical maneuver commands based on this distribution. The specific steps are as follows:

[0023] Step 21: The distributed policy generation module constructs policy functions using an Actor network architecture based on a multilayer perceptron (MLP). ,in Represents network parameters, for Local observation at a given moment For output actions;

[0024] Step 22: During the inference phase, the policy network forward propagates to calculate the action probability distribution. The distributed policy generation module constructs a classification distribution based on this distribution and randomly samples from it to determine the discrete action index at the current moment. ;

[0025] Step 23: The generated motion index is then mapped to specific flight control parameters and sent to the simulation environment for execution;

[0026] Step 3: The value assessment module acquires data using the global interface provided by the simulation environment. Employing a centralized training mechanism, it uses global battlefield situational information to assess the value of the current environmental state. The specific steps are as follows:

[0027] Step 31: The input to the value assessment module is defined as a high-dimensional global state vector. The vector is composed of two parts: the first part is a concatenation of the local observation vectors of all UAV agents performing the mission, ensuring that the commentator network can understand the micro state of each individual; the second part is a global battlefield macro information vector, which contains the key statistical indicators of the entire system, including the total number of missiles launched, the total number of enemy aircraft detected, the total number of threat warnings, and the normalized value of mission time progress.

[0028] Step 32: Construct a value assessment module using a multilayer perceptron structure: The input layer receives the global state vector, which is then processed through three fully connected hidden layers for feature extraction and transformation. The output layer consists of a single neuron and directly outputs a scalar value. , representing the current global state The expected cumulative discounted return that the intelligent agent cluster can obtain in the future;

[0029] Step 33: Define the parameters of the value assessment module as follows Its loss function Represented as:

[0030]

[0031] in, For the network to the first The predicted value of a sample state. The target return value is calculated based on the generalized advantage estimation.

[0032] Step 4: The strategy update and optimization module iteratively updates the network parameters of the distributed strategy generation module and the centralized value assessment module based on the batch flight and adversarial data collected in the experience replay cache. The specific steps are as follows:

[0033] Step 41: The strategy update and optimization module randomly samples batch sample data from the experience pool, including observation state, global state, action index, action log probability, reward value, and value estimate.

[0034] Step 42: For the sampled data, the policy update and optimization module uses the generalized dominance estimation algorithm to calculate the dominance function value at each time step. and target return value ;

[0035] Step 43: To ensure the monotonicity and stability of policy updates, the policy update optimization module uses the clipped surrogate objective function from the Proximal Policy Optimization (PPO) algorithm to update the Actor network parameters. The specific steps are as follows:

[0036] Step 431: Calculate the ratio of the old and new strategies. That is, the ratio of the action probability output by the current policy network to the action probability output by the old policy at the time of sampling:

[0037]

[0038] Step 432, based on Construct the policy loss function :

[0039]

[0040] in, It is a generalized advantage estimate calculated by the centralized valuation module. To truncate hyperparameters, Here is the entropy regularization coefficient. This is a truncation function. For policy entropy;

[0041] Step 433: Construct the value loss function :

[0042]

[0043] in, This represents the value network's prediction of the current global state. This refers to the cumulative return on discounts calculated based on GAE.

[0044] Step 434: Weight and fuse the policy loss, value loss, and entropy regularization term to construct the overall objective loss function:

[0045]

[0046] in, The value loss coefficient, It is the entropy coefficient;

[0047] Step 435: After calculating the total objective loss, the policy update optimization module uses the Adaptive Moment Estimation (Adam) optimizer to backpropagate and update the network parameters.

[0048] Compared with the prior art, the present invention has the following advantages:

[0049] 1. To address the issue that the MADDPG algorithm, due to its deterministic strategy, requires reparameterization to approximate discrete actions, leading to gradient estimation bias and imprecise tactical action capture, this invention designs a distributed strategy generation module. This module adopts a parameter-sharing stochastic strategy network architecture, directly performs probabilistic modeling of discrete macro actions such as target locking and maneuver evasion, and outputs a Softmax distribution. Combined with an innovative pulse command decoding mechanism, it eliminates approximation errors and achieves accurate decision-making and execution of refined tactical actions.

[0050] 2. To address the issues of Q-value overestimation and training divergence that arise in highly non-stationary game environments due to the offline policy mechanism of the MADDPG algorithm, this invention constructs a policy optimization and update module. This module is based on an online policy sampling mechanism, utilizes generalized advantage estimation (GAE) to calculate the policy gradient, and introduces a clipped proximal policy optimization (PPO-Clip) objective function. By strictly constraining the difference between the old and new policies to construct a trust domain, it effectively suppresses the impact of data distribution shifts caused by environmental non-stationarity, ensuring robust convergence of the policy during dynamic adversarial processes.

[0051] 3. To address the problem of the curse of dimensionality caused by the surge in input dimensionality with the formation size due to the need to input the joint actions of all agents in the centralized commentator network of the MADDPG algorithm, this invention proposes a centralized value evaluation module. This module decouples the value function from the joint action space and uses only the dimensionality-reduced global battlefield situation vector as input to evaluate the value of the current state, reducing the input space complexity from exponential to linear, and significantly improving the training efficiency and algorithm scalability of large-scale UAV swarms.

[0052] 4. Compared to adversarial strategies relying on manually generated rules, this invention introduces a multi-agent proximal policy optimization (MAPPO) method based on "centralized training and distributed execution," supplemented by specific observation coding, discrete macro-action design, rising-edge command decoding, security constraint rewards, and curriculum learning. This enables the trained group strategy to consistently generate higher combat effectiveness and better security in the simulation platform. Common problems with rule-based strategies in complex situations include: multi-aircraft target allocation conflicts, cooperative failures caused by non-stationarity during adversarial processes, chattering and invalid launches due to mismatch between missile / fire control commands and the simulation's "rising-edge" semantics, slow maneuvering under missile warning / lock conditions, and poor generalization under different initial positions. To address these shortcomings, the various technical features of this invention have a clear causal relationship with their inherent technical effects:

[0053] (1) By combining a centralized value network (Critic inputs the global state) with a decentralized policy network (Actor inputs their respective local observations) that shares parameters, the "non-stationarity" of the environment is alleviated during the training phase. The Critic can explicitly aggregate the global constraints of the formation and the enemy situation, and learn the global optimal trend of "deduplicated attack, dispersed envelope, and coordinated distance control". Therefore, during the execution phase, the phenomenon of the same enemy target being repeatedly locked by multiple aircraft is naturally reduced, the relative distance between friendly aircraft is more in line with the safety threshold, and the target allocation tends to be balanced. This effect is not an empirical accidental result, but an inevitable learning bias brought about by the modeling assumption of MAPPO (centralized evaluation). In the code, this is reflected in the MAPPO algorithm where the Critic receives the global state and the Actor shares parameters for multiple friendly aircraft.

[0054] (2) By imposing structural constraints (search, engagement, locking onto and firing at the first N targets, evasion) on the discrete macro action space, the difficulty of strategy search and the exploration dimension are significantly reduced, enabling PPO-Clip to stably converge to an executable combat rhythm even with limited samples. At the same time, the fire control and SOCtrlInfo are decoded by "rising edge" in the action generation instruction function of the environment encapsulation module, avoiding chattering or multiple false triggers caused by continuous high-frequency identical instructions, directly reducing the probability of invalid launches and out-of-band instructions. The missile utilization rate and the proportion of effective actions related to hits increase with training. This effect comes from the strict matching of the simulation interface semantics at the action decoding level, which is an inevitable improvement brought about by the consistency of the engineering interface.

[0055] (3) In the reward function, positive rewards are added for kills, negative rewards are added for destruction and threat loitering, small rewards are added for effective engagement and shooting within the window, and penalties are added for invalid launches and long-range shooting. Constraints are also added to the formation safety distance, forcing the strategy to optimize in the expected solution domain: triggering evasion macro actions earlier when missile / lockout alarms occur, shortening engagement paths, standardizing launch windows, and reducing redundant ammunition consumption. Since the reward is the only path to the optimization goal, these behavioral improvements are not heuristic accidents, but an inevitable chain of "change in optimization goal - change in strategy distribution - change in behavioral statistics", which is clearly reflected in the reward items of the environment encapsulation module. Learning starts from low difficulty and gradually increases the enemy situation intensity, improving the exploration difficulty under sparse rewards, which is reflected in reaching stable hit and survival indicators faster with the same number of training steps. This comes from the training arrangement of gradual shrinking of distribution migration (Curriculum) rather than parameter accidents.

[0056] (4) Due to parameter sharing and feature normalization, a single Actor can be reused under different initial positions and enemy situation scales, reducing overfitting and improving generalization ability. During deployment, our strategy directly loads the Actor weights to execute decisions, and the corresponding rule-based strategy backoff ensures that the task is not interrupted in extreme cases such as model not being loaded, abnormal observations, or inference timeouts. This makes the system have engineering availability and robustness: it can still maintain controllable flight and safe actions in the event of hardware jitter, missing model files, or individual abnormal observations, which is an improvement in availability directly resulting from fault-tolerant design.

[0057] (5) From the perspective of measurable indicators, the combined effect of the above technical features should show stable improvement in the following dimensions: increased survival rate of our side under the same scenario and seed, increased number of enemy aircraft destroyed, shortened first effective kill time, improved missile hit rate, decreased invalid launch rate, decreased proportion of repeated attacks on the same target, reduced near miss / collision events, improved unlock success rate under missile warning conditions, and reduced control command jitter count. To ensure "seeking truth from facts", it is recommended to use the evaluation mode of the trainer in conjunction with simulation playback, fix the random seed, run no less than 30 independent rounds, and record the mean and 95% confidence interval; Group A / B respectively use the rule baseline and the strategy of this invention, with simulation parameters and initial positions kept consistent, only the decision module is changed; all KPIs are directly calculated from the simulation log to avoid subjective human evaluation. Attached Figure Description

[0058] Figure 1 Here is a diagram of the MADDPG algorithm architecture;

[0059] Figure 2 Flowchart for optimizing reinforcement learning adversarial algorithms for multi-agent proximal policies;

[0060] Figure 3 Here is a structural diagram of the policy generation module (Actor Network);

[0061] Figure 4 This is a structural diagram of the Critic Network module for value assessment.

[0062] Figure 5 This is a screenshot of the aerial combat where our side shot down the first enemy aircraft.

[0063] Figure 6 This is a screenshot of the aerial combat where our side shot down the second enemy aircraft.

[0064] Figure 7 This is a screenshot of the aerial combat where our side shot down the third enemy aircraft.

[0065] Figure 8 This is a screenshot of the aerial combat where our side shot down the fourth enemy aircraft. Detailed Implementation

[0066] The technical solution of the present invention will be further described below with reference to the accompanying drawings, but it is not limited thereto. Any modifications or equivalent substitutions to the technical solution of the present invention that do not depart from the spirit and scope of the technical solution of the present invention should be covered within the protection scope of the present invention.

[0067] This invention provides a drone swarm adversarial training and control system based on Multi-Agent Proximity Policy Optimization (MAPPO), such as... Figure 2As shown, its overall architecture follows the "Centralized Training, Distributed Execution" (CTDE) paradigm, mainly consisting of four parts: an environment interaction interface module, a distributed policy generation module (Actor), a centralized value evaluation module (Critic), and a policy optimization and update module.

[0068] The environment interaction interface module is responsible for collecting multi-source heterogeneous data in the simulation environment in real time, transforming the original physical quantities into normalized local observation vectors and global state vectors through feature engineering, and decoding discrete action commands into simulation control signals.

[0069] The distributed strategy generation module is deployed on each UAV agent and adopts a parameter-shared random strategy network. It independently outputs discrete action probability distributions based on its local observation vectors to achieve decentralized tactical decision-making.

[0070] The centralized value assessment module only runs during the training phase. It uses the global state vector to perform an overall value assessment of the current battlefield situation, providing a low-variance benchmark signal for strategy updates.

[0071] The policy optimization and update module calculates the advantage function based on the online collected trajectory data and uses generalized advantage estimation (GAE). It also optimizes the objective function by maximizing the pruned proximal policies and updates the parameters of the policy network and value network simultaneously, thereby achieving continuous evolution of the cluster collaborative policy while ensuring training stability.

[0072] The detailed process for each module is as follows:

[0073] 1. Environment Interaction Interface Module:

[0074] The environment interaction interface module serves as a bridge connecting the simulation environment and reinforcement learning algorithms, primarily responsible for data acquisition, preprocessing, and command conversion and execution. This module first subscribes to and acquires battlefield situational data in real time through the communication interface provided by the simulation platform. This includes the dynamic state of friendly UAVs (latitude, longitude, altitude, speed, heading, overload, etc.), enemy target information detected by airborne sensors (relative distance, azimuth, radial velocity, etc.), and weapon system status (remaining missile count, fire control lock status, alarm information, etc.). After acquiring the raw data, this module performs feature engineering processing, transforming heterogeneous physical quantities into standardized tensors suitable for neural network input.

[0075] Specifically, it constructs two types of feature vectors: one is a local observation vector, which contains only the perception information of a single UAV and the local situation within a limited range, used to simulate the single-UAV perception capability under communication-limited conditions in real battlefields, serving as input to the distributed strategy generation module; the other is a global state vector, which concatenates the absolute position, velocity, survival status, and global alarm information of all enemy and friendly entities on the battlefield, used to provide a "God's-eye view" for the centralized value assessment module during the training phase. All continuous features are normalized to accelerate neural network convergence; discrete features are represented using one-hot encoding or embedding. In addition, this module also undertakes the decoding and execution of action commands. It receives discrete action indices output by the distributed strategy generation module and maps them to low-level control commands recognizable by the simulation platform. This module has built-in "rising edge" command decoding logic, which, by monitoring the state changes of the action sequence, transforms the continuous attack intent into a pulse-type fire control trigger signal (isNTSAssigned set), effectively avoiding invalid launches or system deadlocks caused by signal jitter, and ensuring the accurate execution of tactical actions in the simulation environment.

[0076] 2. Distributed strategy generation module:

[0077] The distributed policy generation module, serving as the decision-making core for each individual agent in the UAV swarm, aims to achieve fully distributed execution logic. Its main function is to receive local observation vectors from the environment interaction interface module, extract and infer features through a deep neural network, output a probability distribution of the action space, and generate specific tactical maneuver commands based on this distribution. This module is deployed on each friendly UAV through parameter sharing or independent parameters, ensuring that each UAV retains independent tactical decision-making capabilities even in communication-constrained or adversarial environments.

[0078] The distributed policy generation module uses an Actor network architecture based on a multilayer perceptron (MLP) to construct policy functions. ,in Represents network parameters, for Local observation at a given moment This is for outputting actions. For example... Figure 3As shown, the Actor network input layer is configured with 50 dimensions, corresponding to the normalized local observation vector output by the environment interface. This vector includes the Actor's own flight status (latitude, longitude, altitude, speed, Euler angles, overload, etc.), multi-target enemy situation information detected by the airborne radar (relative distance, azimuth, elevation angle), weapon system status (missile reserve, fire control lock status), and warning information. The network hidden layer consists of three fully connected layers. The first and second layers each have 128 hidden units, while the third layer has 64. Each fully connected layer is followed by a ReLU nonlinear activation function to enhance the model's ability to fit nonlinear high-dimensional battlefield situation features. The network output layer has 7 dimensions, corresponding to 7 discretized tactical actions. The output layer uses the Softmax function to normalize the logistic values, generating the execution probability distribution of each action in the current state.

[0079] During the inference phase, the policy network calculates the action probability distribution through forward propagation. The module constructs a classification distribution based on this distribution and randomly samples from it to determine the discrete action index at the current moment. This mechanism introduces necessary exploratory elements while ensuring policy determinism. The generated action indexes are then mapped to specific flight control parameters (such as heading angle, target speed, overload limits, and fire control commands) and issued to the simulation environment for execution. The module's parameter updates follow the core idea of ​​the Proximal Policy Optimization (PPO) algorithm, aiming to maximize the objective function while limiting the policy update step size to prevent performance collapse. A policy ratio is defined. The ratio of the probabilities of the new and old strategies:

[0080]

[0081] To ensure training stability, this module employs a clipped objective function. For each training batch, the following loss function is minimized:

[0082]

[0083] in, It is a generalized advantage estimate (GAE) calculated by the centralized value assessment module, used to evaluate actions. The degree to which it is better than the average level. This is used to truncate hyperparameters to limit the magnitude of policy updates. For policy entropy, is the entropy regularization coefficient, which is used to encourage agents to explore and prevent premature convergence to local optima.

[0084] 3. Value Assessment Module:

[0085] The value assessment module, acting as the "Critic" component in the algorithm architecture, employs a centralized training mechanism to evaluate the value of the current environmental state using global battlefield situational information. Its core function lies in calculating the state value function. This module provides low-variance gradient update signals to the policy generation module (Actor), thereby addressing the training non-convergence problem caused by limited local observations in multi-agent environments. This module only runs during the training phase, acquiring data through the global interface provided by the simulation environment, and does not participate in computation during the derivation or practical application phases.

[0086] The input to the value assessment module is defined as a high-dimensional global state vector. Its dimensions are set to 220 based on the simulation environment. This vector is composed of two parts: the first part is a concatenation of the local observation vectors of all UAV agents performing the task, with a total dimension of 200, ensuring that the commentator network can understand the micro-state of each individual; the second part is a global battlefield macro-information vector, with a dimension of 20, containing key statistical indicators of the entire system, specifically including the total number of missile launches, the total number of enemy aircraft detected, the total number of threat warnings, and the normalized value of mission time progress. By introducing global state, this module can effectively eliminate the influence of environmental non-stationarity among multiple agents.

[0087] The value assessment network is constructed using a multilayer perceptron structure. For example... Figure 4 As shown, the input layer receives a 220-dimensional global state vector, which is then processed and transformed through three fully connected hidden layers. The first and second hidden layers each have 128 neurons, while the third hidden layer has 64 neurons. Linear ReLU is used as the activation function across all hidden layers to enhance the network's ability to fit complex, nonlinear battlefield value surfaces. The output layer consists of a single neuron, has no activation function, and directly outputs a scalar value. , representing the current global state The expected cumulative discounted return that the intelligent agent cluster can obtain in the future.

[0088] The parameter optimization objective of this module is to minimize the mean squared error (MSE) between the predicted value and the actual return. The parameters of the value network are defined as follows: Its loss function Represented as:

[0089]

[0090] in, For the network to the first The predicted value of a sample state. This module outputs the target return value calculated based on generalized advantage estimation. It was further used to calculate the advantage function. as follows:

[0091]

[0092] in, For timing difference error, As a discount factor, These are the GAE smoothing parameters. Through the above calculations, this module transmits global value information to the policy generation module, guiding it to update its policy towards higher-value states, thereby enhancing collaborative combat capabilities.

[0093] 4. Strategy Update and Optimization Module:

[0094] The strategy update and optimization module, as the core component of the algorithm training loop, is responsible for iteratively updating the network parameters of the distributed policy generation module (Actor) and the centralized value evaluation module (Critic) based on batch flight and adversarial data collected from the experience replay cache. The module first randomly samples batch data from the experience pool, including observation state, global state, action index, action log probability, reward value, and value estimate. For the sampled data, the module uses the generalized advantage estimation algorithm to calculate the advantage function value at each time step. and target return value Advantage function The target reward value is used to measure the quality of a specific action relative to the average level under a specific condition. This serves as a monitoring signal for value network updates.

[0095] To ensure the monotonicity and stability of policy updates, this module uses the clipped surrogate objective function from the proximal policy optimization (PPO) algorithm to update the Actor network parameters. First, calculate the ratio of the old to the new strategies. That is, the ratio of the action probability output by the current policy network to the action probability output by the old policy at the time of sampling:

[0096]

[0097] Based on this ratio, a policy loss function is constructed. This function takes the smaller value between the original advantage objective and the truncated advantage objective to limit the magnitude of policy updates and prevent performance collapse caused by excessively large parameter steps. The formula is as follows:

[0098]

[0099] in, To truncate hyperparameters, For the truncation function, the ratio is limited to Within the interval. Furthermore, to encourage the agent to explore the environment and prevent premature convergence to local optima, an entropy regularization term is introduced into the loss function. .

[0100] Value network parameters The update aims to minimize the mean squared error between the estimated state value and the true target reward. The module constructs the value loss function. as follows:

[0101]

[0102] in This represents the value network's prediction of the current global state. This represents the cumulative discounted reward calculated based on GAE. This loss is minimized using a gradient descent algorithm, enabling the Critic network to more accurately assess the global battlefield situation.

[0103] This module weights and fuses the policy loss, value loss, and entropy regularization term to construct the overall objective loss function:

[0104]

[0105] in, The value loss coefficient, Here, represents the entropy coefficient. After calculating the total loss, the module uses the Adaptive Moment Estimator (Adam) to update the network parameters via backpropagation. To prevent gradient explosion from causing training instability, a gradient clipping strategy is implemented during backpropagation, limiting the gradient norm of all parameters to within the maximum threshold. This update process is repeated multiple times in each training round to fully utilize the collected data samples, achieving rapid convergence of the strategy and improved performance.

[0106] Example:

[0107] To make the objectives, technical solutions, and advantages of this invention clearer, the following will provide a detailed description of this embodiment in conjunction with natural scene images.

[0108] The system development platform is Windows 11 operating system, the GPU is an NVIDIA GeForce RTX2060 GPU, the program is written in Python 3.10, and uses the PyTorch 1.12 framework.

[0109] This embodiment is based on a simulation platform, in which four of our drones engage in combat against four enemy targets.

[0110] I. The system consists of the following modules:

[0111] Communication interface layer: Real-time situation information (self, friendly aircraft, enemy aircraft, alarms, etc.) is obtained through DataService, and control commands are sent through SendData.

[0112] The core strategy layer, red_strategy.py, implements distributed target allocation, fire control locking / firing, evasion, and search logic.

[0113] Global state management: A dictionary structure is used to maintain the missile launch records, target allocation, evasion status, etc. of each aircraft, ensuring information isolation and division of labor among multiple aircraft.

[0114] Target allocation mechanism: Each aircraft is assigned the easiest enemy target to hit based on distance and angle priority, avoiding multiple aircraft attacking the same target repeatedly.

[0115] The static relationship is reflected in the fact that each of our aircraft maintains its own status independently, but targets are distributed through a global allocation table to ensure uniform fire coverage.

[0116] II. Method Details:

[0117] Situational awareness: In each simulation step, our aircraft acquires real-time information such as its own position, speed, heading, and warnings as well as those of the enemy aircraft.

[0118] Target allocation: Each aircraft prioritizes the nearest target that has not been assigned by other aircraft based on the currently visible enemy targets. If all targets have been assigned, the nearest target is selected.

[0119] Fire control decision: If the target is locked and no missile has been launched, launch the missile immediately (maintain current altitude, reduce overload, and avoid the missile's upward trajectory). If the target is not locked, trigger the lock command (rising edge control to prevent repeated locking). Record the target after launch to prevent repeated launches.

[0120] Evasion maneuver: Upon receiving a missile warning, our aircraft immediately evades laterally and increases speed, continuing the evasion for 20 steps before resuming the attack.

[0121] Search and approach: When no target is assigned or the target is not visible, our aircraft will perform a sector search according to the preset offset and time changes to ensure omnidirectional coverage.

[0122] Command issuance: All decisions are issued through the SendData structure, and the simulation platform executes the command and proceeds to the next step.

[0123] III. Feasibility Analysis:

[0124] Depend on Figures 5-8Screenshots from the simulated combat platform show that distributed target allocation significantly reduces repeated attacks on the same target by multiple aircraft, improving missile utilization and strike efficiency. Fire control commands maintain the current altitude, preventing missiles from initially pitching up, shortening flight paths, and increasing the probability of a hit. The evasion mechanism ensures that friendly aircraft can quickly escape danger zones when threatened, reducing the probability of being destroyed. The search mechanism ensures that the swarm can quickly cover the battlefield when there are no targets, improving detection and engagement efficiency. Global state management and rising edge control ensure command standardization, avoiding simulation interface jitter and invalid launches. This method is entirely based on the simulation platform interface; all logic can be engineered and implemented, and it has been running stably in actual simulation environments, demonstrating high feasibility.

Claims

1. A UAV swarm counter-training and control system based on multi-agent proximal policy optimization, characterized in that The system comprises four parts: an environment interaction interface module, a distributed policy generation module, a centralized value assessment module, and a policy optimization and update module. The environment interaction interface module is responsible for collecting multi-source heterogeneous data in the simulation environment in real time, transforming the original physical quantities into normalized local observation vectors and global state vectors through feature engineering, and decoding discrete action commands into simulation control signals. The distributed strategy generation module is deployed on each UAV agent and is responsible for using a parameter-shared random strategy network to independently output discrete action probability distributions based on their respective local observation vectors, thereby achieving decentralized tactical decision-making. The centralized value assessment module only runs during the training phase and is responsible for using the global state vector to perform an overall value assessment of the current battlefield situation, providing a low-variance benchmark signal for strategy updates. The strategy optimization and update module is responsible for calculating the advantage function based on the trajectory data collected online using generalized advantage estimation, and optimizing the objective function by maximizing the pruned proximal policies, while simultaneously updating the parameters of the policy network and value network, thereby achieving continuous evolution of the cluster collaborative strategy while ensuring training stability.

2. A method for UAV reinforcement learning adversarial training based on multi-agent proximal policy optimization using the system described in claim 1, characterized in that... The method includes the following steps: Step 1: Use the environment interaction interface module to collect multi-source heterogeneous data in the simulation environment in real time, transform the original physical quantities into normalized local observation vectors and global state vectors through feature engineering, and decode discrete action commands into simulation control signals. Step 2: The distributed strategy generation module receives local observation vectors from the environment interaction interface module, performs feature extraction and inference through a deep neural network, outputs the probability distribution of the action space, and generates specific tactical maneuver commands based on the distribution. Step 3: The value assessment module uses the global interface provided by the simulation environment to acquire data, adopts a centralized training mechanism, and uses global battlefield situation information to assess the value of the current environmental state. Step 4: The strategy update and optimization module iteratively updates the network parameters of the distributed strategy generation module and the centralized value assessment module based on the batch flight and combat data collected in the experience replay cache.

3. The multi-agent proximal policy optimization based drone reinforcement learning adversarial method according to claim 2, characterized in that The specific steps of step 1 are as follows: Step 11: The environment interaction interface module subscribes to and obtains battlefield situation data in real time through the communication interface provided by the simulation platform, including the dynamic state of friendly UAVs, enemy target information detected by airborne sensors, and weapon system status. Step 12: After obtaining the raw data, the environment interaction interface module performs feature engineering processing to transform the heterogeneous physical quantities into local observation vectors and global state vectors suitable for neural network input. Step 13: The environment interaction interface module receives the discrete action index output by the distributed strategy generation module and maps it into low-level control commands that the simulation platform can recognize.

4. The UAV reinforcement learning adversarial method based on multi-agent proximal policy optimization according to claim 3, characterized in that... The battlefield situation data includes the dynamic state of friendly UAVs, enemy target information detected by airborne sensors, and weapon system status.

5. The UAV reinforcement learning adversarial method based on multi-agent proximal policy optimization according to claim 3, characterized in that... The local observation vector contains only the perception information of a single UAV and the local situation within a limited range. It is used to simulate the perception capability of a single UAV under communication-limited conditions in a real battlefield and serves as the input to the distributed strategy generation module. The global state vector is composed of the absolute position, velocity, survival status and global alarm information of all enemy and friendly entities on the battlefield. It is used to provide a God's-eye view for the centralized value assessment module during the training phase.

6. The multi-agent proximal policy optimization based drone reinforcement learning adversarial method according to claim 2, characterized in that The specific steps of step 2 are as follows: Step 21, the distributed policy generation module constructs a policy function using an actor network architecture based on a multi-layer perceptron wherein denotes a network parameter, is a local observation at time t, is an output action; Step 22, in the reasoning stage, the policy network is forward-propagated to calculate the action probability distribution The distributed policy generation module constructs a classification distribution according to the distribution, and randomly samples from the classification distribution to determine the discrete action index at the current time ; Step 23: The generated motion index is then mapped to specific flight control parameters and sent to the simulation environment for execution.

7. The multi-agent proximal policy optimization based drone reinforcement learning adversarial method according to claim 2, characterized in that The specific steps of step 3 are as follows: Step 31, the input of the value assessment module is defined as a high-dimensional global state vector ; Step 32: Construct a value assessment module using a multilayer perceptron structure: The input layer receives the global state vector, which is then processed through three fully connected hidden layers for feature extraction and transformation. The output layer consists of a single neuron and directly outputs a scalar value. , representing the current global state The expected cumulative discounted return that the intelligent agent cluster can obtain in the future; Step 33: Define the parameters of the value assessment module as follows Its loss function Represented as: in, For the network to the first The predicted value of a sample state. The target return value is calculated based on the generalized advantage estimation.

8. The UAV reinforcement learning adversarial method based on multi-agent proximal policy optimization according to claim 7, characterized in that... The It consists of two parts: the first part is a concatenation of local observation vectors of all UAV agents performing the mission, ensuring that the commentator network can understand the micro state of each individual; the second part is a global battlefield macro information vector, which contains key statistical indicators of the entire system, including the total number of missiles launched, the total number of enemy aircraft detected, the total number of threat warnings, and the normalized value of mission time progress.

9. The UAV reinforcement learning adversarial method based on multi-agent proximal policy optimization according to claim 2, characterized in that... The specific steps of step 4 are as follows: Step 41: The strategy update and optimization module randomly samples batch sample data from the experience pool, including observation state, global state, action index, action log probability, reward value, and value estimate. Step 42: For the sampled data, the policy update and optimization module uses the generalized dominance estimation algorithm to calculate the dominance function value at each time step. and target return value ; Step 43: To ensure the monotonicity and stability of the policy update, the policy update optimization module uses the truncated objective function from the near-end policy optimization algorithm to update the Actor network parameters. .

10. The UAV reinforcement learning adversarial method based on multi-agent proximal policy optimization according to claim 2, characterized in that... The specific steps of step 43 are as follows: Step 431: Calculate the ratio of the old and new strategies. That is, the ratio of the action probability output by the current policy network to the action probability output by the old policy at the time of sampling: Step 432, based on Construct the policy loss function : in, It is a generalized advantage estimate calculated by the centralized valuation module. To truncate hyperparameters, Here is the entropy regularization coefficient. This is a truncation function. For policy entropy; Step 433: Construct the value loss function : in, This represents the value network's prediction of the current global state. This refers to the cumulative return on discounts calculated based on GAE. Step 434: Weight and fuse the policy loss, value loss, and entropy regularization term to construct the overall objective loss function: in, The value loss coefficient, It is the entropy coefficient; Step 435: After calculating the total objective loss, the policy update optimization module uses an adaptive moment estimation optimizer to backpropagate and update the network parameters.

Images