Aircraft intelligent evasion guidance method and system based on generative adversarial imitation learning

Abstract

The application discloses an aircraft intelligent evasion guidance method and system based on a generative adversarial imitation learning. Firstly, a three-dimensional aircraft evasion simulation environment is constructed, and evasion demonstration data is generated based on program maneuver or differential game expert strategy. Secondly, a GAIL framework including a strategy network, a value network and a double-branch discriminator is constructed, and the strategy network is pre-trained by using behavior cloning. Then, fusion GAIL training is performed through mechanisms such as progressive GAIL reward introduction, PPO strategy optimization, adaptive supervision decay and EMA strategy smoothing. Finally, the trained strategy model is deployed in an online guidance system, and evasion control instructions are output according to the real-time state of a red aircraft. The GAIL imitation learning framework is adopted, and strategy learning is guided relying on expert demonstration data, so that a complex multi-objective reward function does not need to be artificially designed, and the ability of strategy autonomous optimization and surpassing the performance of experts is reserved.

Description

Technical Field

[0001] This invention relates to the field of aircraft guidance technology, specifically to an intelligent evasion guidance method and system for aircraft based on generative adversarial learning. Background Technology

[0002] In aircraft guidance technology, optimizing the evasion guidance of the red force aircraft is crucial. Faced with increasingly sophisticated blue force defense systems, red force aircraft need intelligent evasion capabilities, enabling them to autonomously evade multiple blue force aircraft in complex threat environments while maintaining precise target navigation. Traditional programmed maneuver evasion methods rely on pre-set maneuver strategies and lack the ability to adapt to real-time threat situations.

[0003] The application of existing reinforcement learning methods in evasion guidance faces the following challenges: First, the reward function of pure reinforcement learning methods is difficult to design, making it difficult to accurately describe the complex target in the evasion process; second, reinforcement learning agents trained directly from scratch have difficulty converging and have low sample efficiency; third, it is difficult to guarantee a smooth transition of the control strategy during the evasion-cruise switching process.

[0004] Traditional reinforcement learning methods, such as PPO, learn policies directly in the avoidance environment, which suffers from unstable training and difficulty in learning fine maneuvers. While differential game theory can provide theoretically optimal avoidance policies, it incurs significant real-time computational overhead in complex 3D scenes.

[0005] The purpose of this invention is to address the shortcomings of existing reinforcement learning-based intelligent avoidance methods, such as unstable training and low action learning accuracy. This invention proposes a GAIL-based intelligent avoidance guidance method and system for aircraft, which uses expert demonstration data to guide strategy learning, achieving efficient and stable avoidance strategy training and providing algorithmic support for intelligent avoidance of aircraft. Summary of the Invention

[0006] The purpose of this invention is to provide an intelligent evasion guidance method and system for aircraft based on generative adversarial learning, so as to solve the problems existing in the prior art.

[0007] To achieve the above objectives, the present invention provides the following technical solution: an intelligent evasion guidance method for aircraft based on generative adversarial learning, comprising:

[0008] Step 1: Construct a 3D aircraft avoidance simulation environment, generate avoidance demonstration data based on expert strategies, and model the avoidance process as a Markov decision process;

[0009] Step 2: Construct the GAIL framework, which includes a policy network, a value network, and a two-branch discriminator, and pre-train the policy network using behavior cloning;

[0010] Step 3: Perform integrated GAIL training based on the simulation environment, expert data, and the GAIL framework;

[0011] Step 4: Load the trained strategy model and output evasion control commands based on the real-time status of the red force's aircraft to achieve online intelligent evasion guidance.

[0012] Furthermore, step 1 specifically includes:

[0013] 1-1: Establish a three-dimensional kinematic model of the Red Force's aircraft, and construct kinematic equations and attitude update equations based on position, velocity, trajectory tilt angle, trajectory deflection angle and aerodynamic parameters;

[0014] 1-2: Establish a model of the Blue Force aircraft using proportional guidance law and determine the rules for calculating guidance acceleration;

[0015] 1-3: The design includes a 20-dimensional observation space containing normalized position, normalized velocity, ballistic inclination angle, ballistic deflection angle, angle of attack, inclination angle, altitude error, and heading error, as well as a two-dimensional continuous motion space, and completes the normalization mapping.

[0016] 1-4: Design a phased reward function, which includes four parts: immediate penalty, avoidance phase reward, cruise phase reward, and terminal reward;

[0017] 1-5: Set up the avoidance judgment and cruise recovery mechanism. Enter the avoidance phase based on the threat distance judgment, and start the cruise recovery controller after the avoidance is completed;

[0018] 1-6: Based on the procedural maneuvering expert strategy and the differential game expert strategy, evasion demonstration data is generated, and the expert action execution is exponentially smoothed.

[0019] Furthermore, step 2 specifically includes:

[0020] 2-1: The policy network adopts a three-layer fully connected backbone network, with Alpha head and Gamma head mapped to the corresponding action range respectively, and LayerNorm normalization is added after each layer;

[0021] 2-2: The value network adopts the same backbone structure as the policy network, and outputs a single-valued state-value function estimate;

[0022] 2-3: Construct a dual-branch discriminator. The main branch evaluates the similarity between the overall behavior and the expert behavior, while the Gamma branch specifically evaluates the accuracy of the tilt angle control action. The outputs of the two branches are weighted and fused to obtain the final discrimination result.

[0023] 2-4: Configure the AdamW optimizer for the policy network, value network, and discriminator. The policy network adopts a learning rate scheduling strategy of cosine annealing and restart.

[0024] 2-5: Perform behavioral cloning pre-training, calculate the separable supervised loss including the mean square error of the angle of attack, the mean square error of the tilt angle, and the consistency of the tilt angle direction, and use early stopping mechanism and gradient pruning to synchronously update the EMA policy network.

[0025] Furthermore, step 3 specifically includes:

[0026] 3-1: A progressive GAIL reward introduction mechanism is adopted, in which the GAIL reward weight increases linearly from 0 to a set maximum value in the first N warmup rounds;

[0027] 3-2: Collect experience data in stages. During the avoidance stage, the policy network samples and outputs actions. During the non-avoidance stage, zero actions are output. Only during the avoidance stage is the GAIL reward calculated. Successful task trajectories are stored in a buffer for self-imitation and learning.

[0028] 3-3: The generalized dominance estimation method is used to calculate the time series difference error and dominance function, thereby reducing the variance of policy gradient estimation;

[0029] 3-4: In each iteration, the discriminator is updated, and the label smoothing technique is used to calculate the weighted total loss of the main branch and the Gamma branch;

[0030] 3-5: Based on the PPO algorithm, the policy network and value network are jointly updated, and the total loss function is constructed by integrating PPO pruning loss, value function loss, entropy regularization loss, expert supervision loss, and success trajectory self-imitation loss.

[0031] 3-6: An adaptive supervision coefficient adjustment mechanism is adopted to dynamically adjust the weight allocation of expert supervision and autonomous exploration based on the strategy success rate and action error;

[0032] 3-7: After each policy network parameter update, the EMA policy is updated smoothly and synchronously.

[0033] 3-8: Select the best model based on a comprehensive evaluation of multiple indicators. If the best model has not been updated for several consecutive rounds and the performance meets the target, perform a training rollback to prevent performance crash.

[0034] Furthermore, step 4 specifically includes:

[0035] 4-1: Load the EMA strategy model and observation normalization parameters, perform standardization and pruning on the 20-dimensional real-time state vector of the red force's spacecraft, input the strategy model, output control commands in deterministic mode, and drive the actuators after staged smoothing by a second-order filter.

[0036] 4-2: Conduct multi-scenario tests through Monte Carlo simulation, apply random perturbations to the initial position of the Blue Team's aircraft, and statistically analyze the mission success rate, avoidance success rate, minimum miss distance, cumulative reward, terminal altitude error, and heading error.

[0037] Intelligent evasion guidance system for aircraft based on generative adversarial imitation learning includes:

[0038] The evasion simulation environment module is used to construct a three-dimensional aircraft evasion simulation environment, establish the kinematic model of the red team's aircraft and the blue team's interceptor model, and design the observation space, action space, phased reward function and evasion-cruise switching mechanism.

[0039] The expert data generation module is used to generate evasion demonstration data based on programmatic expert strategies and differential game expert strategies, and to complete the expert action index smoothing process.

[0040] The GAIL training module is used to build the GAIL framework, which includes a policy network, a value network, and a two-branch discriminator. It performs behavior cloning pre-training and fusion GAIL training to output the optimal policy model.

[0041] The online guidance module is used to load the trained EMA strategy model, preprocess the real-time flight status, output smooth evasion control commands, and complete online guidance and performance verification.

[0042] Furthermore, the GAIL training module includes:

[0043] The network building unit is used to build a three-layer fully connected policy network and value network, construct a two-branch discriminator containing a main branch and a Gamma branch, and configure the activation functions and normalization layers of each network.

[0044] The pre-training unit is used to perform behavior cloning pre-training, calculate the separate supervised loss, and realize early stopping judgment, gradient pruning, and synchronous update of the EMA policy network.

[0045] The fusion training unit is used to perform progressive GAIL reward introduction, empirical data collection, generalized advantage function estimation, discriminator update, joint policy and value network update, adaptive supervision coefficient adjustment, EMA policy smoothing, optimal model selection, and training rollback.

[0046] Furthermore, the online guidance module includes:

[0047] The data preprocessing unit is used to perform normalization and numerical clipping on the real-time state vector of the red team's aircraft.

[0048] The policy reasoning unit is used to input the preprocessed observation vector into the EMA policy model so that the deterministic mode outputs normalized control commands.

[0049] The command smoothing unit is used to smooth the control commands in stages using a second-order filter, so as to achieve a continuous transition of commands during the avoidance and cruise phases.

[0050] The performance verification unit is used to perform Monte Carlo simulation tests, and to statistically analyze and output the strategy's task success rate, avoidance success rate, miss distance distribution, and end-point control accuracy.

[0051] Compared with the prior art, the beneficial effects of the present invention are:

[0052] This invention employs the GAIL imitation learning framework, relying on expert demonstration data to guide policy learning. It eliminates the need for manually designing complex multi-objective reward functions while retaining the policy's ability to autonomously optimize and surpass expert performance. A dual-branch discriminator structure is used: the main branch learns overall avoidance behavior, while the Gamma branch specifically learns fine-grained tilt angle control actions, addressing the shortcomings of traditional GAIL in learning attitude control precision. A three-pronged mechanism—behavioral cloning pre-training, progressive GAIL reward introduction, and adaptive supervised decay—is integrated to achieve a smooth transition from expert imitation to autonomous exploration, avoiding training oscillations and crashes, and significantly improving sample efficiency. Monte Carlo random perturbation testing demonstrates strong generalization ability in diverse threat scenarios, with both avoidance and mission success rates meeting engineering application standards. It also supports both programmed maneuvering and differential strategy expert policy generation data, adapting to the avoidance and guidance needs of aircraft across multiple models and threat scenarios. The decoupled module design facilitates future functional expansion. Attached Figure Description

[0053] Figure 1 This is a flowchart of the intelligent evasion guidance method for aircraft of the present invention.

[0054] Figure 2 This is a diagram of the overall architecture of the GAIL framework of this invention.

[0055] Figure 3 This is a diagram of the strategy network structure of the present invention.

[0056] Figure 4 This is a structural diagram of the dual-branch discriminator of the present invention.

[0057] Figure 5 This is a flowchart of the GAIL training process of the present invention.

[0058] Figure 6 This represents the reward change curve during the agent's training process.

[0059] Figure 7 This is the success rate curve during the training process of the agent.

[0060] Figure 8 This is a 3D trajectory diagram of a single evasion test.

[0061] Figure 9 A comparison chart of strategy output actions and expert actions.

[0062] Figure 10 The diagram shows the off-target displacement distribution from the Monte Carlo simulation.

[0063] Figure 11 A statistical chart showing the success rate of Monte Carlo simulations. Detailed Implementation

[0064] The technical solutions of the present invention will be clearly and completely described below with reference to the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0065] Please see Figure 1-11 This invention provides an intelligent evasion guidance method for aircraft based on generative adversarial learning, comprising:

[0066] Step 1: Construct a 3D aircraft avoidance simulation environment, generate avoidance demonstration data based on expert strategies, and model the avoidance process as a Markov decision process;

[0067] Step 2: Construct the GAIL framework, which includes a policy network, a value network, and a two-branch discriminator, and pre-train the policy network using behavior cloning;

[0068] Step 3: Perform integrated GAIL training based on the simulation environment, expert data, and the GAIL framework;

[0069] Step 4: Load the trained strategy model and output evasion control commands based on the real-time status of the red force's aircraft to achieve online intelligent evasion guidance.

[0070] Example 1: Intelligent Escape Guidance Method for Aircraft Based on Generative Adversarial Imitation Learning

[0071] Step 1: Simulation Environment Construction and Expert Data Generation

[0072] Step 1-1: Three-dimensional kinematic modeling of the Red Team's aircraft

[0073] The position of the Red Team's spacecraft in three-dimensional space is determined by... Description, in which , For horizontal coordinates, Here are the altitude coordinates. The aircraft's speed is... The trajectory inclination angle is The ballistic deflection angle is The kinematic equations are:

[0074]

[0075]

[0076]

[0077] The attitude update equations consist of lift Y, side force Z, thrust P, gravity mg, and angle of attack. Sideslip angle Inclination angle Determined by aerodynamic parameters. The rate of change of the trajectory inclination angle is:

[0078]

[0079] The rate of change of ballistic deflection angle is:

[0080]

[0081] Aerodynamic parameters: air density ρ = 1.0 kg / m³, lift curve slope The aircraft has a mass m = 600 kg, gravitational acceleration g = 9.8 m / s², and a reference area S = 0.5 m². Lift calculation formula:

[0082]

[0083] Steps 1-2: Blue Team Aircraft Model

[0084] Two blue team aircraft are set up, employing proportional guidance laws with a guidance coefficient N=4. The initial position of blue team aircraft 1 is... The initial position of Blue Team's aircraft 2 is The speeds are all 4 × 340 m / s. The formula for calculating guidance acceleration is:

[0085]

[0086]

[0087] Steps 1-3: Design of Observation Space and Action Space

[0088] Construct a 20-dimensional observation space, including: normalized position, normalized velocity, ballistic inclination angle, ballistic deflection angle, and angle of attack. Inclination angle Errors include altitude and heading. The motion space is a two-dimensional continuous motion, with the motion range normalized to [-1, 1]. Actual commands are converted using a scaling factor.

[0089]

[0090] Steps 1-4: Design of Phased Reward Function

[0091] The reward function employs a phased design, providing different reward signals based on the aircraft's current flight phase. The overall reward consists of four parts: penalty, evasion phase reward, cruise phase reward, and terminal reward.

[0092] First, define the immediate penalty: when an aircraft is intercepted, a penalty is imposed. A penalty will be imposed when the aircraft crashes. A penalty is imposed when the aircraft flies out of the simulation boundary. The current turn will terminate immediately upon triggering the aforementioned penalty conditions.

[0093] (1) Evasion Phase Reward: When the aircraft is in an evasion state (Pen_on=1), the reward consists of the following components:

[0094] ① Successful avoidance reward: For each successful avoidance of a blue team aircraft (missed target judgment), a one-time reward of +30 is given. The reward is calculated separately for each blue team aircraft, with a maximum cumulative reward of +60.

[0095] ② Threat Distance Maintenance Bonus: When the nearest threat distance For distances greater than 500m, a reward proportional to the threat distance is granted:

[0096]

[0097] in This is an avoidance distance threshold. The reward encourages aircraft to maintain a safe distance from threats during avoidance maneuvers.

[0098] ③ Active maneuver reward: When the aircraft's angle of attack and bank angle deflection meet the requirements At that time, a maneuver reward of +0.002 is given to encourage proactive maneuvering behavior during the avoidance phase.

[0099] ④ Survival bonus: A fixed survival bonus of +0.001 is given for each physical step.

[0100] ⑤ Altitude soft constraint penalty: To prevent the aircraft from deviating too much from its flight altitude during evasive maneuvers, a soft altitude boundary is set. When the height exceeds this range, a linear penalty is applied based on the deviation amount.

[0101]

[0102] Where Δh is the height deviation beyond the soft boundary (unit: m).

[0103] (2) Cruise Phase Reward: When the aircraft is in cruise mode At that time, the reward consists of the following components:

[0104] ① Basic Survival Bonus: A fixed bonus of +0.0005 is given for each physical step.

[0105] ② Proximity Reward Rewards are given based on the rate at which the aircraft approaches the target point. The approach rate is defined as the projected component of the aircraft's velocity in the direction of the target.

[0106]

[0107] when (Approaching the target) ;when (When far from the target) The penalty for moving away from the target is greater than the reward for moving closer, to emphasize the importance of course correction.

[0108] ③ Heading accuracy bonus After both blue team aircraft are successfully avoided, a tiered reward is given based on the magnitude of the heading error |ψ_e|.

[0109]

[0110] ④ High-precision reward After both blue team aircraft were successfully avoided, based on the altitude error... Tiered rewards will be given:

[0111]

[0112] in The target altitude is set at the cruise target altitude. The aforementioned tiered reward design encourages the agent to quickly return to a precise cruise state after completing the avoidance maneuver.

[0113] (3) Terminal Rewards: Terminal rewards are given based on the reason for task completion, as defined below:

[0114] ① Mission Success Reward: A success reward is given when the aircraft successfully avoids all blue team aircraft and resumes stable cruise (meeting altitude error <100m, heading error <1° and approaching the target for 100 consecutive steps):

[0115] Among them, time reward Additional rewards are given based on the speed at which the task is completed; the earlier the task is completed, the higher the reward.

[0116]

[0117] Precision reward Additional rewards will be given based on the accuracy of the flight status upon mission completion: when the altitude error is less than 50m. Accumulate +50; when the heading error is less than 0.5°, The cumulative increase is +50, up to a maximum of +100.

[0118] ② Penalties for intercepted terminals:

[0119]

[0120] ③ Recovery Timeout Penalty: If stable cruise cannot be restored within the specified time (200 seconds) after the evasion operation ends:

[0121]

[0122] ④ Total timeout penalty: Simulation time exceeds the maximum duration hour:

[0123]

[0124] Steps 1-5: Avoidance Judgment and Cruise Recovery Mechanism

[0125] When the nearest threat distance is less than the avoidance distance threshold (25km) and the blue team's aircraft is still active, the avoidance phase is initiated. After the avoidance phase ends, the enhanced PID cruise recovery controller is activated.

[0126] Steps 1-6: Expert Data Generation

[0127] In the program maneuvering expert strategy, the evasion acceleration is dynamically adjusted based on the threat distance:

[0128]

[0129] Differential game expert strategy employs variable strength avoidance:

[0130]

[0131] Exponential smoothing of expert actions:

[0132]

[0133] Step 2: GAIL Framework Construction and Behavior Cloning Pre-training

[0134] Step 2-1, Strategy Network Design

[0135] The policy network employs a three-layer fully connected backbone network (256-256-128). The first two layers use the GELU activation function, and the last layer uses the Tanh activation function. LayerNorm normalization is added after each layer. The alpha head is mapped to the expert action range via a sigmoid function.

[0136]

[0137] in The Gamma head is mapped to Tanh. .

[0138] Step 2-2, Value Network Design

[0139] The value network uses the same backbone structure (256-256-128) as the policy network and outputs a single value function estimate V(s).

[0140] Steps 2-3: Design of the dual-branch discriminator

[0141] To address the issue of insufficient accuracy in learning tilt angle actions in the conventional GAIL framework, this invention constructs a dual-branch discriminator that includes a main branch and an auxiliary Gamma branch.

[0142] The main branch network is used to evaluate the similarity between the agent's overall behavior (including position and pose changes) and the expert's behavior. Its input is a joint vector formed by concatenating the current state observation vector *s* and the action vector *a* performed by the agent. This joint vector passes through two fully connected hidden layers (256-128), and finally outputs a scalar logical value. This value, after being activated by the Sigmoid function, yields the main branch similarity probability.

[0143] .

[0144] The Gamma branch network is used to focus on evaluating the fineness of tilt angle control actions. Its input consists of the current state observation vector *s* and the tilt angle component in the action vector. The concatenated joint vector. This joint vector passes through two fully connected hidden layers with a small number of nodes (128-64), and finally outputs a scalar logical value. This value, after being activated by the Sigmoid function, yields the Gamma branch similarity probability.

[0145] .

[0146] (2) Discriminator output and reward calculation: In this step, the final output of the discriminator is calculated. Probability of the main branch and Gamma branch probability The weighted fusion is obtained, and the calculation formula is:

[0147]

[0148] in, This is a weighting coefficient (set to 0.6 here) used to balance the emphasis on overall behavior imitation and fine motor imitation.

[0149] Based on the discriminator output described above, the GAIL reward function is defined. for:

[0150]

[0151] in, This is a constant (set to 0.01 here) to prevent numerical calculation overflow. The physical meaning of this reward function is: the more likely the action 'a' produced by the agent is to be misjudged by the discriminator as an expert action (i.e., the higher the probability of misjudging it by the discriminator), the higher the probability of misjudging it by the discriminator as an expert action (i.e., the higher the probability of misjudging it by the discriminator). The closer it is to 1, the higher the reward value it receives.

[0152] Steps 2-4: Optimizer Configuration

[0153] The policy network employs the AdamW optimizer and a hierarchical learning rate strategy is designed to adapt to the training requirements of different network modules: the learning rate for the backbone layer is [missing information]. The learning rates for the Alpha head and Gamma head are... The standard deviation parameter learning rate is The value network uses the AdamW optimizer with a learning rate of [missing information]. The discriminator uses the AdamW optimizer with a learning rate of [missing information]. The weight decay coefficients of the above optimizers are all set to... The learning rate scheduling of the policy network adopts a cosine annealing restart strategy, with an initial period of... Cycle multiplication factor The minimum learning rate is .

[0154] Steps 2-5: Behavioral Cloning Pre-training

[0155] Up to 800 rounds of behavior cloning pre-training are performed, with 256 samples uniformly sampled from expert data in each round, and a segregated supervised loss is calculated. The goal of behavior cloning pre-training is to make the output actions of the policy network as close as possible to the expert actions, providing good initialization parameters for subsequent GAIL training. The supervised loss consists of three parts, calculated using a weighted summation method:

[0156]

[0157] in, The weighting coefficient represents the mean square error loss of the angle of attack deviation. ; The weighting coefficient is the mean square error loss of the tilt angle deviation. ; The Gamma direction consistency loss is defined as the loss in the expert action. For salient samples, calculate the symbol matching rate between the policy output and the expert actions. Weighting coefficient This directional consistency loss ensures that the policy network not only learns the magnitude of the tilt angle, but also correctly determines whether it tilts to the left or right.

[0158] An early stopping mechanism is employed during training: training is terminated early if no loss occurs for 100 consecutive epochs or if the Alpha MSE shows a significant improvement (improvement <1%). The gradient pruning threshold is set to 0.5. The EMA policy network is updated synchronously after each parameter update. After training, the optimal model parameters from the BC phase are loaded and used as the initial state for subsequent GAIL training.

[0159] Step 3: Fusion GAIL Training

[0160] Step 3-1, Progressive GAIL Introduction

[0161] To avoid spurious reward signals interfering with strategy stability due to the immaturity of the discriminator in the early stages of GAIL training, this invention employs a gradual GAIL introduction strategy. In the first N_warmup=50 iterations, the GAIL reward weight w_gail(t) linearly increases from 0 to a set maximum GAIL reward coefficient λ_gail=0.4.

[0162]

[0163]

[0164] In each training step, the total combined reward obtained by the agent is a weighted fusion of the GAIL reward and the environment reward:

[0165]

[0166] Step 3-2, Experience Collection

[0167] During reinforcement learning training, each iteration collects complete experience data for four episodes through interaction with the simulation environment, storing it in an experience buffer for subsequent policy updates and model optimization. The action decision logic employs a phased control strategy: when the agent is in the avoidance phase (Pen_on=1), it uses the current policy network to randomly sample and output actions, distributed by a Gaussian distribution. The actual action 'a' is obtained through sampling. When the agent is in the non-avoidance phase (Pen_on=0), zero action is uniformly output, and no additional control commands are generated, which is strictly consistent with the logic during expert data collection. The GAIL reward is only calculated in the avoidance phase (Pen_on=1), and the GAIL reward is set to 0 in the non-avoidance phase. This design avoids the discriminator's incorrect evaluation of behavior in the non-avoidance phase. In addition, when the agent successfully completes the avoidance task, the complete trajectory (state-action sequence) of that round is stored in the success trajectory buffer, with a maximum of 20 successful trajectories retained for subsequent self-imitation learning.

[0168] Step 3-3: Generalized Dominance Estimation

[0169] The generalized advantage estimation (GAE) method is used to calculate the advantage function at each time step to reduce the variance of the policy gradient estimation. The specific calculation process is as follows: First, the time-series difference error δ_t is calculated:

[0170]

[0171]

[0172] Steps 3-4: Discriminator Update

[0173] The discriminator is updated twice per iteration. During each update, a batch of samples is randomly sampled from both the empirical data and expert data collected for the policy. To improve training stability and prevent overfitting, this invention employs label smoothing technology, setting the expert sample label to 0.9 and the policy sample label to 0.1. The total loss function of the dual-branch discriminator is a weighted sum of the main branch loss and the Gamma branch loss:

[0174]

[0175] Steps 3-5: Joint update of policy network and value network

[0176] This step jointly optimizes and updates the policy network and the value network. The update process uses Proximal Policy Optimization (PPO) as the core framework of reinforcement learning, while fusing expert supervision signals and successful trajectory self-imitation signals into a unified loss function to achieve a dynamic balance between training stability and policy exploration capability. Specifically, in each training iteration, K traversal-style random mini-batch updates are performed on all the experience data collected in step 3-2, where K=4 and the mini-batch size M=128. Each mini-batch update includes the following steps:

[0177] (1) Calculation of loss in PPO pruning strategy

[0178] For each set of empirical samples in the current mini-batch ,in For state observation vectors, The action to be performed at this time step. This refers to the standardized advantage function value calculated using the generalized advantage estimation in step 3-3. First, calculate the actions... Under the current strategy Old strategy during data collection The importance sampling ratio is obtained from the logarithmic probability below:

[0179]

[0180] To enhance numerical stability, a safety truncation is applied to the above ratios:

[0181]

[0182] Based on this, the loss of the PPO pruning strategy is defined as:

[0183]

[0184] In the formula This refers to the PPO pruning range parameter. The physical meaning of this pruning objective function is: when the policy update makes the probability ratio... The deviation from 1 exceeds When the advantage function contributes to the gradient, it will be truncated, thereby limiting the single-step update magnitude of the policy and effectively preventing policy collapse.

[0185] (2) Calculation of value function loss

[0186] The value network V(s) is used to estimate the expected cumulative reward of the current state. The value function loss is defined as the value network output. Target return value calculated with GAE Mean square error between:

[0187]

[0188] (3) Calculation of policy entropy regularization loss

[0189] To encourage the policy to maintain a moderate level of exploration during training and avoid premature convergence to a suboptimal solution, a policy entropy regularization term is introduced. The average entropy of the policy on a mini-batch set of states is defined as:

[0190]

[0191] Where H(·) is the differential entropy of the Gaussian policy distribution. The entropy loss is the negative of the average entropy. This ensures that the optimization direction of the total loss includes the objective of maximizing policy entropy.

[0192] (4) Calculation of expert supervision loss

[0193] In each mini-batch update, an amount equal to the current mini-batch is randomly sampled from the expert demonstration dataset generated in step 1. Expert State-Action Pair The expert status Input the current policy network to obtain the mean of the predicted actions. ,in For angle of attack channel output, This is the output for the tilt angle channel. The expert-monitored loss uses a multi-channel design, consisting of the mean square error loss from the angle-of-attack channel. Mean square error loss of tilt angle channel Consistency loss with tilt angle direction The weighted average consists of three parts:

[0194]

[0195] The definitions of each sub-item of the loss are as follows. The angle of attack channel loss is the mean square error between the strategy-predicted angle of attack and the expert's angle of attack:

[0196]

[0197] The tilt angle channel loss is the mean square error between the strategy-predicted tilt angle and the expert-predicted tilt angle.

[0198]

[0199] Tilt angle direction consistency loss is used to ensure that the policy network can correctly determine the evasion turning direction (tilt left or right). First, in the current batch of expert actions, those with an absolute tilt angle greater than a threshold are selected. The significant maneuver samples constitute the significant maneuver set. .when At that time, the sign agreement rate between the predicted tilt angle and the expert tilt angle is calculated. and directional loss :

[0200] η

[0201]

[0202] In the formula This is an indicator function that takes the value 1 when the condition is true and 0 otherwise. When the number of significant maneuver samples is insufficient (|Ω|≤5), let... The weighting coefficients for each loss term are taken as follows: , , The angle-of-attack channel is assigned the highest weight coefficient, reflecting the dominant role of angle-of-attack accuracy in the aircraft's evasive maneuverability.

[0203] (5) Calculation of loss from self-imitation of success trajectory

[0204] To further solidify the effective avoidance behavior learned through the strategy, this invention introduces a self-imitation learning mechanism based on successful trajectories. During the experience acquisition process in step 3-2, whenever the avoidance task is successfully completed in a given interaction round, the complete state-action trajectory for that round is stored in a successful trajectory buffer (maximum capacity 20 trajectories, using a first-in, first-out elimination strategy). During strategy updates, the most successful trajectory is randomly sampled from the buffer. State-action pairs When the effective number of samples At that time, calculate the self-imitation loss:

[0205]

[0206] In the formula The policy network predicts actions for successful trajectory states (including two channels: angle of attack and tilt angle). This represents the squared L2 norm of the vector. When the buffer is empty or there are fewer than 10 valid samples, this loss term is not included in the total loss calculation.

[0207] (6) Construction and parameter update of the total loss function

[0208] Combining the above loss terms, the total loss function for the policy network and the value network is defined as follows:

[0209]

[0210] The physical meanings and values ​​of each parameter in the formula are as follows: The time-varying supervision coefficient controls the relative weight distribution between PPO reinforcement learning exploration and expert supervision. The initial value is 0.9, and it is adaptively decayed during training based on policy performance (see steps 3-6 for the decay mechanism). The lower limit of decay is 0.4. The loss coefficient of the value function; This is the entropy regularization coefficient; These are the weighting coefficients for the successful trajectory self-imitation. The design idea behind the above total loss function is: in the early stages of training... When the value approaches 0.9, strategy updates primarily mimic expert behavior, and PPO explores weights. The lower threshold ensures that the policy can be reliably learned within the state space covered by the expert trajectory; as training progresses and performance improves, As the PPO exploration weight gradually decreases to 0.4, it increases to 0.6, giving the policy network more room for autonomous exploration and enabling further optimization and improvement based on expert policies.

[0211] During the gradient calculation and parameter update phase, a backpropagation is performed on the total loss, simultaneously obtaining the gradients of the policy network and the value network. After backpropagation, the following operations are performed sequentially: ① Detect and clear outlier gradient values ​​(NaN or Inf) parameter by parameter to prevent the propagation of numerical anomalies; ② Perform global gradient norm pruning on the policy network and the value network respectively, with a pruning threshold G_max=0.5; ③ Update the model parameters of the policy network and the value network respectively through their independent AdamW optimizers; ④ Immediately after each parameter update, perform EMA policy smoothing update synchronously (see steps 3-7 for details).

[0212] Steps 3-6: Adaptive Supervision Coefficient Adjustment

[0213] Supervision coefficient An adaptive adjustment strategy is adopted, with the initial value... minimum value Basic attenuation factor The adjustment logic is as follows:

[0214] (1) When the Alpha MSE deteriorates significantly (greater than 1.5 times the historical moving average and greater than 0.001), increase the supervision coefficient: To prevent strategies from deviating too far from expert behavior;

[0215] (2) When the moving average success rate is greater than the threshold (0.4) and the Alpha MSE is better than the historical average, the decay supervision coefficient is: Gradually relax expert constraints and give strategies more room for independent exploration;

[0216] (3) The supervision coefficient remains unchanged in other cases.

[0217] This adaptive mechanism enables a smooth transition from expert imitation to autonomous strategies: in the early stages of training, the supervision coefficient is high (close to 0.9), and the strategy mainly imitates the expert; as performance improves, the supervision coefficient gradually decreases (to a minimum of 0.4), and the strategy can autonomously explore and improve based on the expert's experience.

[0218] Steps 3-7: EMA strategy smoothing

[0219] Each policy network parameter After the update, the network parameters of the Exponential Moving Average (EMA) strategy will be updated synchronously. :

[0220]

[0221] Steps 3-8: Optimal Model Selection and Training Rollback

[0222] The optimal model checkpoint is dynamically updated using a multi-indicator comprehensive evaluation mechanism. The update is triggered when one of the following three conditions is met:

[0223] (1) The success rate of the moving average has improved by more than 3 percentage points compared to the current best.

[0224] (2) The success rate is comparable to the best (difference < 2%) and the average reward is increased by more than 5;

[0225] (3) Alpha MSE improved by more than 10% and the success rate was less than 5% away from the best.

[0226] If the optimal model is not updated for 25 consecutive iterations and the current moving average success rate is below 20%, a training rollback operation is performed to restore the parameters of the policy network, EMA network, value network, and discriminator to the optimal checkpoint state, effectively preventing performance crashes during training.

[0227] Step 4: Online intelligent avoidance guidance and verification

[0228] Step 4-1: Online evasion guidance application

[0229] Load the trained EMA policy model ( ) and the observation normalization parameters saved during the training phase Real-time acquisition of the 20-dimensional flight state vector of the red team's aircraft. Standardization process:

[0230]

[0231] The standardized observation vector (trimmed to) (Range) Input to the EMA policy network to output control commands in a deterministic mode, i.e., directly take the mean output of the policy network without random sampling:

[0232]

[0233] The normalized actions output by the strategy are converted into physical control commands. The Alpha head output is mapped to the expert action range [-0.05, 0.15] via a Sigmoid function:

[0234]

[0235] The Gamma header output is mapped to the range [-1, 1] via Tanh, and then multiplied by a scaling factor to obtain the actual tilt angle command:

[0236]

[0237] Control commands are smoothed by a second-order filter before driving the aircraft's actuators. During the avoidance phase, a low-bandwidth filter is used to ensure the smoothness of control commands and avoid abrupt maneuver transitions; during the cruise phase, a high-bandwidth filter is used to achieve rapid attitude recovery. The two-phase transition employs a cosine transition weight function with a transition time of 0.5 seconds to ensure the continuity of control commands during phase transitions. During the online inference phase, the policy network only performs forward propagation, resulting in minimal computational overhead (single inference < 1ms), meeting the computational requirements of real-time guidance.

[0238] Step 4-2, Monte Carlo test verification

[0239] The trained EMA strategy was validated using Monte Carlo simulation testing (100 independent simulations by default). Gaussian random perturbations (standard deviation of 2000m) were introduced into the initial positions of the two blue team aircraft to evaluate the strategy's generalization ability under different threat scenarios. Statistical metrics included: overall mission success rate, avoidance success rate, minimum miss distance distribution statistics, cumulative reward statistics (mean, standard deviation, maximum, minimum), and the distribution of terminal altitude error and heading error accuracy of successful trajectories. The robustness and generalization performance of the strategy were verified through large-scale random testing.

[0240] Example 2: Intelligent Avoidance Guidance System for Aircraft Based on Generative Adversarial Imitation Learning

[0241] The system implementation of the corresponding method in this embodiment.

[0242] The evasion simulation environment module integrates the red team's kinematics model, the blue team's interception model, the observation / action space, phased rewards, and evasion-cruise switching logic, providing a standardized simulation scenario for training and validation.

[0243] Expert Data Generation Module: Invokes program maneuvering and differential strategy to generate high-quality evasion trajectories, completes motion smoothing, and outputs a standardized expert demonstration dataset.

[0244] The GAIL training module comprises three main sub-units: network construction, behavior cloning pre-training, and fusion-based GAIL training, which outputs the most robust EMA policy model.

[0245] Online guidance module: Completes real-time state preprocessing, strategy reasoning, command smoothing, and performance verification, and deploys the trained model to the aircraft guidance system to achieve autonomous intelligent avoidance.

[0246] Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. An intelligent evasion guidance method for aircraft based on generative adversarial learning, characterized in that: include: Step 1: Construct a 3D aircraft avoidance simulation environment, generate avoidance demonstration data based on expert strategies, and model the avoidance process as a Markov decision process; Step 2: Construct the GAIL framework, which includes a policy network, a value network, and a two-branch discriminator, and pre-train the policy network using behavior cloning; Step 3: Perform integrated GAIL training based on the simulation environment, expert data, and the GAIL framework; Step 4: Load the trained strategy model and output evasion control commands based on the real-time status of the red force's aircraft to achieve online intelligent evasion guidance.

2. The intelligent evasion guidance method for aircraft based on generative adversarial learning according to claim 1, characterized in that: Step 1 specifically includes: 1-1: Establish a three-dimensional kinematic model of the Red Force's aircraft, and construct kinematic equations and attitude update equations based on position, velocity, trajectory tilt angle, trajectory deflection angle and aerodynamic parameters; 1-2: Establish a model of the Blue Force aircraft using proportional guidance law and determine the rules for calculating guidance acceleration; 1-3: The design includes a 20-dimensional observation space containing normalized position, normalized velocity, ballistic inclination angle, ballistic deflection angle, angle of attack, inclination angle, altitude error, and heading error, as well as a two-dimensional continuous motion space, and completes the normalization mapping. 1-4: Design a phased reward function, which includes four parts: immediate penalty, avoidance phase reward, cruise phase reward, and terminal reward; 1-5: Set up the avoidance judgment and cruise recovery mechanism. Enter the avoidance phase based on the threat distance judgment, and start the cruise recovery controller after the avoidance is completed; 1-6: Based on the procedural maneuver expert strategy and the differential game expert strategy, evasion demonstration data is generated, and the expert action execution is exponentially smoothed.

3. The intelligent evasion guidance method for aircraft based on generative adversarial learning according to claim 2, characterized in that: Step 2 specifically includes: 2-1: The policy network adopts a three-layer fully connected backbone network, with Alpha head and Gamma head mapped to the corresponding action range respectively, and LayerNorm normalization is added after each layer; 2-2: The value network adopts the same backbone structure as the policy network, and outputs a single-valued state-value function estimate; 2-3: Construct a dual-branch discriminator. The main branch evaluates the similarity between the overall behavior and the expert behavior, while the Gamma branch specifically evaluates the accuracy of the tilt angle control action. The outputs of the two branches are weighted and fused to obtain the final discrimination result. 2-4: Configure the AdamW optimizer for the policy network, value network, and discriminator. The policy network adopts a learning rate scheduling strategy of cosine annealing and restart. 2-5: Perform behavioral cloning pre-training, calculate the separable supervised loss including the mean square error of the angle of attack, the mean square error of the tilt angle, and the consistency of the tilt angle direction, and use early stopping mechanism and gradient pruning to synchronously update the EMA policy network.

4. The intelligent evasion guidance method for aircraft based on generative adversarial learning according to claim 3, characterized in that: Step 3 specifically includes: 3-1: A progressive GAIL reward introduction mechanism is adopted, in which the GAIL reward weight increases linearly from 0 to a set maximum value in the first N warmup rounds; 3-2: Collect experience data in stages. During the avoidance stage, the policy network samples and outputs actions. During the non-avoidance stage, zero actions are output. Only during the avoidance stage is the GAIL reward calculated. Successful task trajectories are stored in a buffer for self-imitation and learning. 3-3: The generalized dominance estimation method is used to calculate the time series difference error and dominance function, thereby reducing the variance of policy gradient estimation; 3-4: In each iteration, the discriminator is updated, and the label smoothing technique is used to calculate the weighted total loss of the main branch and the Gamma branch; 3-5: Based on the PPO algorithm, the policy network and value network are jointly updated, and the total loss function is constructed by integrating PPO pruning loss, value function loss, entropy regularization loss, expert supervision loss, and success trajectory self-imitation loss. 3-6: An adaptive supervision coefficient adjustment mechanism is adopted to dynamically adjust the weight allocation of expert supervision and autonomous exploration based on the strategy success rate and action error; 3-7: After each policy network parameter update, the EMA policy is updated smoothly and synchronously. 3-8: Select the best model based on a comprehensive evaluation of multiple indicators. If the best model has not been updated for several consecutive rounds and the performance meets the target, perform a training rollback to prevent performance crash.

5. The intelligent evasion guidance method for aircraft based on generative adversarial learning according to claim 4, characterized in that: Step 4 specifically includes: 4-1: Load the EMA strategy model and observation normalization parameters, perform standardization and pruning on the 20-dimensional real-time state vector of the red force's spacecraft, input the strategy model, output control commands in deterministic mode, and drive the actuators after staged smoothing by a second-order filter. 4-2: Conduct multi-scenario tests through Monte Carlo simulation, apply random perturbations to the initial position of the Blue Team's aircraft, and statistically analyze the mission success rate, avoidance success rate, minimum miss distance, cumulative reward, terminal altitude error, and heading error.

6. An intelligent evasion guidance system for aircraft based on generative adversarial learning, characterized in that, include: The evasion simulation environment module is used to construct a three-dimensional aircraft evasion simulation environment, establish the kinematic model of the red team's aircraft and the blue team's interceptor model, and design the observation space, action space, phased reward function and evasion-cruise switching mechanism. The expert data generation module is used to generate evasion demonstration data based on programmatic expert strategies and differential game expert strategies, and to complete the expert action index smoothing process. The GAIL training module is used to build the GAIL framework, which includes a policy network, a value network, and a two-branch discriminator. It performs behavior cloning pre-training and fusion GAIL training to output the optimal policy model. The online guidance module is used to load the trained EMA strategy model, preprocess the real-time flight status, output smooth evasion control commands, and complete online guidance and performance verification.

7. The intelligent evasion guidance system for aircraft based on generative adversarial learning according to claim 6, characterized in that: The GAIL training module includes: The network building unit is used to build a three-layer fully connected policy network and value network, construct a two-branch discriminator containing a main branch and a Gamma branch, and configure the activation functions and normalization layers of each network. The pre-training unit is used to perform behavior cloning pre-training, calculate the disjoint supervised loss, and realize early stopping judgment, gradient pruning, and synchronous update of the EMA policy network. The fusion training unit is used to perform progressive GAIL reward introduction, empirical data collection, generalized advantage function estimation, discriminator update, joint policy and value network update, adaptive supervision coefficient adjustment, EMA policy smoothing, optimal model selection, and training rollback.

8. The intelligent evasion guidance system for aircraft based on generative adversarial learning according to claim 6, characterized in that: The online guidance module includes: The data preprocessing unit is used to perform normalization and numerical clipping on the real-time state vector of the red team's aircraft. The policy reasoning unit is used to input the preprocessed observation vector into the EMA policy model so that the deterministic mode outputs normalized control commands. The command smoothing unit is used to smooth the control commands in stages using a second-order filter, so as to achieve a continuous transition of commands during the avoidance and cruise phases. The performance verification unit is used to perform Monte Carlo simulation tests, and to statistically analyze and output the strategy's task success rate, avoidance success rate, miss distance distribution, and end-point control accuracy.

Images