Hierarchical iterative adversarial policy network training method and application thereof

Abstract

The application discloses a layered iterative countermeasure strategy network training method and application thereof, and comprises the following steps: S1, updating an evasion strategy network according to a current evasion result reward; S2, updating a launch strategy network according to a current missile launch result reward, wherein R F The application further discloses a hit rate reward function R M When the distance between the enemy aircraft and the aircraft is not greater than the maximum distance at which the missile radar can lock the enemy aircraft, the real-time output value of the launch guidance model after training of the current generation is acquired, and the real-time output value of the launch guidance model is taken as the function value of R M ; S3, repeating S1-S2 until a preset convergence condition is reached, and obtaining the trained layered iterative countermeasure strategy network.

Description

Technical Field

[0001] This invention relates to the field of reinforcement learning technology, and more specifically to a hierarchical iterative adversarial strategy training system and method based on deep reinforcement learning. Background Technology

[0002] In today's intelligent era, with the rapid development of artificial intelligence and aviation technologies, unmanned aerial vehicles (UAVs) are playing an increasingly important role in modern air combat. Traditional manned aircraft are limited by factors such as pilot physiological limits, reaction speed, and psychological stress in air combat. UAVs, however, can overcome these limitations, withstand higher G-forces, and perform higher-risk missions. They can also be deployed in large numbers at relatively low cost, thus adapting to more novel combat modes. In future air combat environments, UAVs not only need to be able to cooperate with manned aircraft but also need to have the ability to execute missions independently. Therefore, endowing UAVs with real-time, autonomous tactical decision-making capabilities is crucial.

[0003] Air combat decision-making is a complex dynamic game process involving various information exchanges between opposing sides and environmental changes, placing extremely high demands on the adaptability and robustness of decision-making algorithms. Traditional air combat decision-making relies heavily on empirical rules, which struggles to meet the real-time and accuracy requirements of complex air combat environments. Deep reinforcement learning technology, on the other hand, establishes feedback loops between the agent and the environment, allowing UAVs to continuously learn through trial and error in simulated air combat environments. This enables them to learn tactical strategies suitable for highly dynamic environments through extensive adversarial processes.

[0004] However, despite the powerful policy learning capabilities demonstrated by deep reinforcement learning, the effectiveness of agent training largely depends on the design of the reward function. This is especially true in dynamic and high-dimensional air combat scenarios, where designing a reward function that accurately guides the agent to formulate reasonable maneuver strategies is often quite challenging. Traditional reward function design typically relies on the prior knowledge of domain experts, requiring the manual design of a series of quantitative indicators reflecting air combat objectives (such as shooting down enemy aircraft, gaining a favorable attack position, and avoiding threats). This not only demands extensive domain knowledge but also easily leads to inconsistencies between training objectives and actual combat objectives, thus limiting the upper limit of agent performance. Summary of the Invention

[0005] In view of this, the present invention provides a hierarchical iterative adversarial strategy training system and method based on deep reinforcement learning, which at least solves the problem in the prior art that the training effect of deep reinforcement learning agents in air combat game scenarios is highly dependent on the design of the reward function.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] A hierarchical iterative adversarial strategy network training method includes the following steps:

[0008] S1: Train the reward function based on the current enemy missile avoidance strategy and evasion strategy. Obtain the evasion result reward corresponding to the current evasion strategy for enemy aircraft missiles, and update the evasion strategy network based on the current evasion result reward;

[0009] S2: Train the reward function based on the current missile launch strategy and launch strategy. Obtain the missile launch result reward corresponding to the current missile launch strategy, and update the launch strategy network based on the current missile launch result reward. Including hit rate reward function When the distance between friendly and enemy aircraft is no greater than the maximum distance at which the missile radar can lock onto the enemy aircraft, the real-time output value of the current generation of trained launch guidance model is obtained, and the real-time output value of the launch guidance model is used as... The function value;

[0010] S3: Repeat S1-S2 until the preset convergence condition is met to obtain the trained hierarchical iterative adversarial strategy network.

[0011] Preferably, both the avoidance strategy network and the launch strategy network include an Actor network and a Critic network, respectively. The specific training process for the Actor network and the Critic network includes:

[0012] ① Initialize the network parameters of the Actor network and the Critic network, and initialize the preset data threshold and the preset maximum number of training steps;

[0013] ② Both Actor networks and Critic networks obtain state information from the environment. ;

[0014] ③ The Actor network outputs the action-motion type based on the state information. ,get Then execute the corresponding maneuver and receive the corresponding reward information. Then obtain the next state information of the current environment. Among them, reward information In the evasion strategy network, according to Acquired, in the launch strategy network according to Obtain;

[0015] ④ Transfer the currently acquired data group ( , , , Stored in the experience replay area;

[0016] ⑤ When the data stored in the experience playback area does not reach the preset data threshold, continue to acquire data groups ( , , , (until the preset data threshold is reached);

[0017] When the data stored in the experience replay area reaches the preset data threshold, data is randomly sampled from the experience replay area to update the network parameters of the Actor network and the Critic network.

[0018] ⑥ Repeat steps ②-⑤ until the maximum number of training steps is reached, and obtain the trained avoidance strategy network and launch strategy network.

[0019] Preferably, during the training process of the evasion strategy network, the training of the first generation of evasion strategies is based on a preset rule-based launch strategy, which provides the missile source for the training of the evasion strategy; the second generation and subsequent evasion strategies are trained in the missile evasion stage of the two-aircraft game confrontation scenario, using the launch strategy obtained from the previous generation as a coach.

[0020] During the training of the launch strategy network, the training scenario for the first generation of launch strategies is a preset enemy aircraft chasing scenario. Under this preset enemy aircraft chasing scenario, the enemy aircraft randomly performs maneuvers, and the launch strategy network only chases and locks onto the enemy aircraft, maintaining the lock-on state without launching missiles. The second generation and later launch strategies are trained in the enemy aircraft chasing phase of a two-aircraft game confrontation scenario.

[0021] Preferred training reward function for avoidance strategy Specifically:

[0022] ;

[0023] in, For out-of-bounds penalty functions, To avoid the result reward function;

[0024] Launch strategy training reward function Including the initial launch training reward function and the launch training reward function of descendants :

[0025] ;

[0026] ;

[0027] in, For distance reward function, For the attack angle reward function, For the hit rate reward function, To continuously pursue the reward function, To successfully hit the reward function.

[0028] Preferred out-of-bounds penalty function and the reward function for avoiding outcomes Specifically:

[0029] ;

[0030] ;

[0031] In the formula, The flight altitude of the intelligent agent; The preset flight altitude range for the intelligent agent;

[0032] Distance reward function for:

[0033] ;

[0034] In the formula, The distance between enemy and friendly aircraft; Minimum safe launch distance; This is the maximum distance at which the missile radar can lock onto enemy aircraft;

[0035] Attack Angle Reward Function :

[0036] ;

[0037] In the formula, The lock-on angle for the intelligent agent against the target enemy aircraft; This is the maximum lock-on angle at which the missile radar can lock onto enemy aircraft;

[0038] Hit rate reward function for:

[0039] ;

[0040] In the formula, This is the real-time output value of the launch guidance model;

[0041] Continuous pursuit of reward function for:

[0042] ;

[0043] Successfully hit the reward function for:

[0044] .

[0045] Preferably, the loss function of the Actor network is:

[0046] ;

[0047] in, As expected, This represents the ratio of the probabilities of the new and old strategies. and They represent the old and new strategies respectively. Status complete The probability is used for importance sampling. The dominance function measures the action. In state The relative value below, clip(·) represents the clip function, used to... Limited to Within the scope, to prevent excessively large policy updates, among which... For limiting parameters;

[0048] The loss function of the Critic network is:

[0049] ;

[0050] in, This indicates the current value network's state information. The value of the predicted state, This represents the target state value calculated using discounted returns and generalized advantage estimation.

[0051] Preferably, both the Actor network and the Critic network consist of one input layer, two hidden layers, and one output layer. Each hidden layer contains 256 neurons and uses the ReLU activation function.

[0052] The input features of the evasion strategy network include: enemy missile and aircraft distance scalar, enemy missile and aircraft horizontal position difference, enemy missile altitude, aircraft altitude, enemy missile Euler angle, aircraft Euler angle, enemy missile velocity vector and aircraft velocity vector;

[0053] The input features of the launch strategy network include: enemy and friendly distance scalar, enemy and friendly horizontal position difference, enemy altitude, friendly altitude, enemy Euler angle, friendly Euler angle, enemy velocity vector, friendly velocity vector, enemy linear velocity scalar, friendly linear velocity scalar, and friendly lock-on angle against the enemy.

[0054] The Actor network outputs for both the evasion and launch strategies are integers in the range [0, 6], representing 7 possible actions: stable flight, climb, dive, left roll, right roll, left yaw, and right yaw. The Critic network output is a one-dimensional state value estimate.

[0055] Preferably, the launch guidance model is trained using a supervised learning training method, which includes:

[0056] The original dataset is obtained from the missile engagement simulation dataset. The original dataset is divided into training set, validation set and test set, and the dataset is normalized. Each row in the original dataset represents a sample data. Each sample data includes a feature vector representing the engagement state and its corresponding binary label, which is used to indicate whether the missile launched in this state hits the target.

[0057] The training set data is divided into multiple mini-batches, and one batch of data is used in each iteration. Each iteration process includes:

[0058] Perform forward propagation on the training set samples;

[0059] Calculate the loss function;

[0060] Perform backpropagation and update network parameters;

[0061] After each training cycle, the current model performance is evaluated using the validation set, and the loss value on the validation set is continuously monitored. Training stops when the validation loss decreases by less than a specified threshold for more than a preset number of decreases.

[0062] When training terminates, the model parameters with the lowest validation loss are used as the final launch guidance model, and the final launch guidance model is evaluated using a test set.

[0063] Preferably, the launch guidance model uses a feedforward deep neural network as a binary classification discriminant model, including:

[0064] Input layer: Used to obtain the original dataset. The input features of the launch guidance model include: enemy and aircraft distance scalar, enemy and aircraft horizontal position difference, enemy aircraft altitude, aircraft altitude, enemy aircraft Euler angle, aircraft Euler angle, enemy aircraft velocity vector, aircraft velocity vector, enemy aircraft linear velocity scalar, aircraft linear velocity scalar, and aircraft lock angle against the enemy aircraft.

[0065] Hidden layers: Contain two fully connected hidden layers, each followed by a LeakyReLU activation function and Dropout regularization operation;

[0066] Output layer: 1 neuron, activation function is sigmoid, the output value is compressed to the (0,1) interval, as the predicted probability of missile hitting the target.

[0067] An application of a hierarchical iterative adversarial strategy network training method for UAV control, which includes three stages:

[0068] Phase 1 is the pursuit phase, which is the default phase when our aircraft first enters the combat state. The launch strategy output by the trained launch strategy network controls the launch, and missiles are launched immediately when the enemy aircraft is successfully locked, or when the real-time hit rate predicted by the trained launch guidance model is greater than a specified threshold.

[0069] Phase 2 is the phase of moving away from enemy aircraft. It uses a rule-based autopilot mode to adjust the heading angle in real time to the opposite direction of the enemy aircraft, and flies horizontally after reaching the set altitude to move away from the enemy aircraft as far as possible.

[0070] Phase 3 is the missile evasion phase, controlled by the evasion strategy output by the trained evasion strategy network.

[0071] As can be seen from the above technical solution, compared with the prior art, the present invention discloses a hierarchical iterative adversarial strategy network training method and its application, which has the following beneficial effects:

[0072] To address the issue that the training performance of deep reinforcement learning agents in air combat scenarios is highly dependent on the design of the reward function, this invention proposes a novel hierarchical iterative adversarial strategy training framework based on deep reinforcement learning, HIAPTF. This framework innovatively introduces a launch guidance model and uses its real-time output as the reward signal, enabling the reward function to adapt and evolve. Through iterative training, it indirectly optimizes the entire policy system, allowing the agent to autonomously learn more fundamental winning strategies, thereby improving the agent's learning efficiency and generalization ability in complex air combat environments. Attached Figure Description

[0073] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0074] Figure 1 A flowchart of a hierarchical iterative adversarial strategy network training method provided by the present invention;

[0075] Figure 2 A schematic diagram of air combat geometry provided for an embodiment of the present invention;

[0076] Figure 3 This is a schematic diagram of drone operation provided in an embodiment of the present invention;

[0077] Figure 4 Training curves for avoidance strategies provided in embodiments of the present invention;

[0078] Figure 5 Training curves for the launch strategy provided in this embodiment of the invention;

[0079] Figure 6 This is a training curve diagram of the PPO agent provided in an embodiment of the present invention;

[0080] Figure 7 Heatmaps of the net win rate of each generation of intelligent agents provided in embodiments of the present invention;

[0081] Figure 8 The adversarial trajectory diagram of the 5th generation intelligent agent (our side) and the 1st generation intelligent agent (enemy side) provided for embodiments of the present invention;

[0082] Figure 9 A schematic diagram illustrating the guidance effect of the launch guidance model provided in an embodiment of the present invention;

[0083] Figure 10 This is a comparison chart of training curves for the first-generation launch strategies with and without guided reward signals, provided in an embodiment of the present invention. Detailed Implementation

[0084] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0085] This invention provides a method for training a hierarchical iterative adversarial strategy network (HIAPTF), which includes the following steps:

[0086] S1: Train the reward function based on the current enemy missile avoidance strategy and evasion strategy. Obtain the evasion result reward corresponding to the current evasion strategy for enemy aircraft missiles, and update the evasion strategy network based on the current evasion result reward;

[0087] S2: Train the reward function based on the current missile launch strategy and launch strategy. Obtain the missile launch result reward corresponding to the current missile launch strategy, and update the launch strategy network based on the current missile launch result reward. Including hit rate reward function When the distance between friendly and enemy aircraft is no greater than the maximum distance at which the missile radar can lock onto the enemy aircraft, the real-time output value of the current generation of trained launch guidance model is obtained, and the real-time output value of the launch guidance model is used as... The function value;

[0088] S3: Repeat S1-S2 until the preset convergence condition is met to obtain the trained hierarchical iterative adversarial strategy network.

[0089] It should be noted that:

[0090] The hierarchical iterative adversarial strategy training framework designed in this invention is as follows: Figure 1 As shown, the framework includes three policy models: two maneuver policy models trained using deep reinforcement learning and one launch guidance model trained using supervised learning. The overall control of the UAV agent is primarily governed by two core maneuver policies: evasion and launch. The launch guidance model controls the timing of missile launches and supports the training of the launch policy. The training process for these three policy models is progressive and can enhance performance through iterative training.

[0091] The air combat scenario of this invention focuses on 1v1 close-range air combat, and establishes a six-degree-of-freedom air combat geometric model. The schematic diagram of the air combat geometric relationship is shown below. Figure 2 As shown in the image. The aircraft at the bottom are our fighters, and the aircraft at the top are enemy fighters. This is the velocity vector of my machine. Let be the velocity vector of the target enemy aircraft. Let the positions of our aircraft and the target enemy aircraft in the navigation coordinate system be respectively... and Then the line-of-sight vector of the enemy aircraft relative to our aircraft is... The direction is from our aircraft to the enemy aircraft, and its magnitude is the distance between us and the enemy. .

[0092] To further implement the above technical solution, both the evasion strategy network and the launch strategy network include an Actor network and a Critic network, respectively. The specific training process for the Actor network and the Critic network includes:

[0093] ① Initialize the network parameters of the Actor network and the Critic network, and initialize the preset data threshold and the preset maximum number of training steps;

[0094] ② Both Actor networks and Critic networks obtain state information from the environment. ;

[0095] ③ The Actor network outputs the action-motion type based on the state information. ,get Then execute the corresponding maneuver and receive the corresponding reward information. Then obtain the next state information of the current environment. Among them, reward information In the evasion strategy network, according to Acquired, in the launch strategy network according to Obtain;

[0096] ④ Transfer the currently acquired data group ( , , , Stored in the experience replay area;

[0097] ⑤ When the data stored in the experience playback area does not reach the preset data threshold, continue to acquire data groups ( , , , (until the preset data threshold is reached);

[0098] When the data stored in the experience replay area reaches the preset data threshold, data is randomly sampled from the experience replay area to update the network parameters of the Actor network and the Critic network.

[0099] ⑥ Repeat steps ②-⑤ until the maximum number of training steps is reached, and obtain the trained avoidance strategy network and launch strategy network.

[0100] To further implement the above technical solution, during the training process of the evasion strategy network, the training of the first generation of evasion strategies is based on a preset rule-based launch strategy, which provides the missile source for the training of the evasion strategy; the second generation and subsequent evasion strategies are trained in the missile evasion stage of the two-aircraft game confrontation scenario, using the launch strategy obtained from the previous generation as a coach.

[0101] During the training of the launch strategy network, the training scenario for the first generation of launch strategies is a preset enemy aircraft chasing scenario. Under this preset enemy aircraft chasing scenario, the enemy aircraft randomly performs maneuvers, and the launch strategy network only chases and locks onto the enemy aircraft, maintaining the lock-on state without launching missiles. The second generation and later launch strategies are trained in the enemy aircraft chasing phase of a two-aircraft game confrontation scenario.

[0102] It should be noted that:

[0103] Training scenarios for evasion strategies require the participation of enemy missiles. However, since a usable launch strategy model was not available at the beginning of training, the initial evasion strategy training relied on a rule-based launch strategy. This strategy could chase the enemy aircraft and launch missiles immediately upon the missile radar locking onto the enemy, thus providing a missile source for the evasion strategy training. Second-generation and later evasion strategies used the launch strategy trained in the previous generation as a coach, training in the missile evasion phase of a two-aircraft game-like confrontation scenario.

[0104] The key reward for launch strategy training comes from the reward signal provided by the launch guidance model, but the training scenarios for the first and subsequent generations are set differently. The training scenario for the first generation launch strategy is a single enemy aircraft chase scenario. In this scenario, the enemy aircraft performs random maneuvers, and the agent needs to chase and lock onto the enemy aircraft as much as possible, maintaining the lock-on state without launching missiles. The purpose of this scenario is to teach the agent basic chase maneuvers and enemy aircraft lock-on capabilities. The second generation and later launch strategies are trained in the enemy aircraft chase phase of a two-aircraft game-like confrontation scenario, aiming to improve the missile hit rate.

[0105] To further implement the above technical solution and circumvent the policy training reward function Specifically:

[0106] ;

[0107] in, For out-of-bounds penalty functions, To avoid the result reward function;

[0108] Launch strategy training reward function Including the initial launch training reward function and the launch training reward function of descendants :

[0109] ;

[0110] ;

[0111] in, For distance reward function, For the attack angle reward function, For the hit rate reward function, To continuously pursue the reward function, To successfully hit the reward function.

[0112] To further implement the above technical solution, an out-of-bounds penalty function is needed. and the reward function for avoiding outcomes Specifically:

[0113] ;

[0114] ;

[0115] In the formula, The flight altitude of the intelligent agent; The preset flight altitude range for the intelligent agent;

[0116] Distance reward function for:

[0117] ;

[0118] In the formula, The distance between enemy and friendly aircraft; Minimum safe launch distance; This is the maximum distance at which the missile radar can lock onto enemy aircraft;

[0119] Attack Angle Reward Function :

[0120] ;

[0121] In the formula, The lock-on angle for the intelligent agent against the target enemy aircraft; This is the maximum lock-on angle at which the missile radar can lock onto enemy aircraft;

[0122] Hit rate reward function for:

[0123] ;

[0124] In the formula, This is the real-time output value of the launch guidance model;

[0125] Continuous pursuit of reward function for:

[0126] ;

[0127] Successfully hit the reward function for:

[0128] .

[0129] It should be noted that:

[0130] This embodiment limits the flight altitude range of the intelligent agent to 0-11km. Exceeding the altitude boundary results in damage. To ensure the agent can prevent it from going out of bounds, a penalty is imposed when the agent's flight altitude is within 1 km of the boundary. This invention takes... =1km, =10km.

[0131] The main purpose of the evasion strategy is to enable the agent to make the enemy missile miss the target through reasonable maneuvering, that is, to evade the enemy missile. Therefore, the agent is rewarded or punished once based on the result of whether the evasion is successful or not.

[0132] The launch strategy also needs to have the ability to prevent out-of-bounds errors, therefore, it is also based on the out-of-bounds penalty function. Punish intelligent agents that are about to go out of bounds.

[0133] To encourage agents to approach enemy aircraft in order to keep them within the lock-on range of our missile radar, a distance reward function is set. This invention takes... =0.6km, =6km.

[0134] To enable the agent to obtain the launch window when the missile radar locks onto the enemy aircraft, and to encourage the agent to point its nose at the enemy aircraft, an attack angle reward function is set. This invention takes... =15°.

[0135] To maximize the hit rate of missiles launched by the intelligent agent, at the distance between enemy and friendly aircraft... Smaller than the maximum distance at which a missile can lock onto an enemy aircraft At that time, the real-time output values ​​of the launch guidance model will be used. As a reward signal.

[0136] During the initial launch strategy training, to encourage the agent to continuously pursue the enemy aircraft, a reward function is provided for the behavior of continuously locking onto the enemy aircraft. .

[0137] When training launch strategies for the second generation and beyond in a two-aircraft game scenario, a one-time reward is given for the agent's action of launching a missile and successfully hitting the enemy aircraft. .

[0138] To further implement the above technical solution, the loss function of the Actor network is:

[0139] ;

[0140] in, As expected, This represents the ratio of the probabilities of the new and old strategies. and They represent the old and new strategies respectively. Status complete The probability is used for importance sampling. The dominance function measures the action. In state The relative value below, clip(·) represents the clip function, used to... Limited to Within the scope, to prevent excessively large policy updates, among which... For limiting parameters;

[0141] The loss function of the Critic network is:

[0142] ;

[0143] in, This indicates the current value network's state information. The value of the predicted state, This represents the target state value calculated using discounted returns and generalized advantage estimation.

[0144] To further implement the above technical solution, both the Actor network and the Critic network consist of one input layer, two hidden layers, and one output layer. Each hidden layer contains 256 neurons and uses the ReLU activation function.

[0145] The input features of the evasion strategy network include: enemy missile and aircraft distance scalar, enemy missile and aircraft horizontal position difference, enemy missile altitude, aircraft altitude, enemy missile Euler angle, aircraft Euler angle, enemy missile velocity vector and aircraft velocity vector;

[0146] The input features of the launch strategy network include: enemy and friendly distance scalar, enemy and friendly horizontal position difference, enemy altitude, friendly altitude, enemy Euler angle, friendly Euler angle, enemy velocity vector, friendly velocity vector, enemy linear velocity scalar, friendly linear velocity scalar, and friendly lock-on angle against the enemy.

[0147] The Actor network outputs for both the evasion and launch strategies are integers in the range [0, 6], representing 7 possible actions: stable flight, climb, dive, left roll, right roll, left yaw, and right yaw. The Critic network output is a one-dimensional state value estimate.

[0148] To further implement the above technical solution, the launch guidance model is trained using a supervised learning method, which includes the following:

[0149] The original dataset is obtained from the missile engagement simulation dataset. The original dataset is divided into training set, validation set and test set, and the dataset is normalized. Each row in the original dataset represents a sample data. Each sample data includes a feature vector representing the engagement state and its corresponding binary label, which is used to indicate whether the missile launched in this state hits the target.

[0150] The training set data is divided into multiple mini-batches, and one batch of data is used in each iteration. Each iteration process includes:

[0151] Perform forward propagation on the training set samples;

[0152] Calculate the loss function;

[0153] Perform backpropagation and update network parameters;

[0154] After each training cycle, the current model performance is evaluated using the validation set, and the loss value on the validation set is continuously monitored. Training stops when the validation loss decreases by less than a specified threshold for more than a preset number of decreases.

[0155] When training terminates, the model parameters with the lowest validation loss are used as the final launch guidance model, and the final launch guidance model is evaluated using a test set.

[0156] To further implement the above technical solution, the launch guidance model adopts a feedforward deep neural network as a binary classification discrimination model, including:

[0157] Input layer: Used to obtain the original dataset. The input features of the launch guidance model include: enemy and aircraft distance scalar, enemy and aircraft horizontal position difference, enemy aircraft altitude, aircraft altitude, enemy aircraft Euler angle, aircraft Euler angle, enemy aircraft velocity vector, aircraft velocity vector, enemy aircraft linear velocity scalar, aircraft linear velocity scalar, and aircraft lock angle against the enemy aircraft.

[0158] Hidden layers: Contain two fully connected hidden layers, each followed by a LeakyReLU activation function and Dropout regularization operation;

[0159] Output layer: 1 neuron, activation function is sigmoid, the output value is compressed to the (0,1) interval, as the predicted probability of missile hitting the target.

[0160] It should be noted that:

[0161] The essence of the launch guidance model is a binary classification model trained using supervised learning, with the output value of its final sigmoid layer serving as the real-time predicted missile hit rate. The dataset collection scenario for training this model is similar to that for evasion strategies: a launch strategy agent launches missiles at an evasion strategy agent, and the state data at launch time and the label indicating whether the missile ultimately hits are recorded as the training dataset. One function of this model is to control the missile launch timing; for example, it can be set to launch a missile only when the real-time predicted missile hit rate is greater than 20%. Furthermore, a more important function of this model is to use the output value as a reward signal to participate in the training of the launch strategy, guiding the launch strategy to maximize its missile hit rate.

[0162] The key reward for launch strategy training comes from the reward signal provided by the launch guidance model, but the training scenarios for the first and subsequent generations are set differently. The training scenario for the first generation launch strategy is a single enemy aircraft chase scenario. In this scenario, the enemy aircraft performs random maneuvers, and the agent needs to chase and lock onto the enemy aircraft as much as possible, maintaining the lock-on state without launching missiles. The purpose of this scenario is to teach the agent basic chase maneuvers and enemy aircraft lock-on capabilities. The second generation and later launch strategies are trained in the enemy aircraft chase phase of a two-aircraft game-like confrontation scenario, aiming to improve the missile hit rate.

[0163] In this embodiment, the neural networks used in the hierarchical iterative adversarial strategy network are all fully connected neural networks. The Actor and Critic networks for the avoidance and launch strategies each consist of one input layer, two hidden layers, and one output layer. Each hidden layer contains 256 neurons and uses the ReLU activation function. The input layer dimension of the avoidance strategy network is 17, and the input layer dimension of the launch strategy network is 20. The corresponding state feature inputs are shown in Table 1. The outputs of the Actor networks for both the avoidance and launch strategies are integers in the range [0, 6], representing seven possible actions: stable flight, climb, dive, left roll, right roll, left yaw, and right yaw. The output of the Critic network is a one-dimensional state value estimate. The launch guidance model network consists of one input layer, two hidden layers, and one output layer. The dimension and state feature input of the input layer are consistent with the launch strategy. The number of neurons in the hidden layers are 128 and 64, respectively, using the LeakyReLU activation function. The network output is a continuous value in the range [0, 1], using the Sigmoid activation function.

[0164] Table 1 State Feature Input

[0165] ;

[0166] An application of a hierarchical iterative adversarial strategy network training method for UAV control, which includes three stages:

[0167] Phase 1 is the pursuit phase, which is the default phase when our aircraft first enters the combat state. The launch strategy output by the trained launch strategy network controls the launch, and missiles are launched immediately when the enemy aircraft is successfully locked, or when the real-time hit rate predicted by the trained launch guidance model is greater than a specified threshold.

[0168] Phase 2 is the phase of moving away from enemy aircraft. It uses a rule-based autopilot mode to adjust the heading angle in real time to the opposite direction of the enemy aircraft, and flies horizontally after reaching the set altitude to move away from the enemy aircraft as far as possible.

[0169] Phase 3 is the missile evasion phase, controlled by the evasion strategy output by the trained evasion strategy network.

[0170] It should be noted that:

[0171] The operation of a drone consists of three stages, as illustrated in the diagram below. Figure 3 As shown.

[0172] The invention will be further illustrated below through simulation experiments:

[0173] The simulation experiments of this invention are conducted based on the Harfang3D open-source air combat simulation platform. This platform is a reinforcement learning research platform specifically customized for fighter jet control tasks, allowing experimenters to customize various settings such as scenarios, tasks, entity parameters, and simulation parameters. Based on this platform, this invention designs multiple scenarios for agent training, evaluation, and dataset collection. All scenarios are set in the airspace above the sea, with no other obstacles on the sea surface affecting the flight of fighter jets and missiles.

[0174] In the two-aircraft combat scenario constructed in this invention, both sides use an unmanned combat aerial vehicle (UCAV) with identical performance for close-range dogfights. Each UCAV has exactly one missile. By default, upon successfully locking onto an enemy aircraft, the missile is launched immediately. A hit is considered successful if the missile and target are within 10 meters. The UCAV is destroyed if it is hit by an enemy missile or exceeds its altitude limit. Successfully evading the enemy missile is considered a successful evasion. The UCAV's altitude is limited to 0-11 km, its maximum speed is 400 m / s, and its engine thrust is kept at maximum. The missile radar has a maximum lock-on range of 6 km and a maximum lock-on angle of 15°. The missile radar successfully locks onto the enemy aircraft within the maximum lock-on range and angle for 3 seconds.

[0175] In the initial situation, both aircraft have an initial speed of 300 m / s, an initial roll angle and pitch angle of 0°, and initial heading angles pointing towards each other. Each time the initial positions are randomized, one aircraft's initial horizontal position is at the center of the map, while the other aircraft's initial horizontal position is randomized within a horizontal circular area centered on the other aircraft. The inner and outer radii of this circle are 6 km and 9 km respectively, and the initial altitudes of both aircraft are randomized between 4 km and 8 km.

[0176] The round ends when there are no missiles in flight within the airspace, and one of the following three conditions is met simultaneously: both sides have launched missiles, at least one side's aircraft has been destroyed, or the round lasts for more than 2 minutes. If our aircraft survives and the enemy aircraft is destroyed at the end of the round, our side wins; otherwise, the enemy wins. If both sides' aircraft survive or are destroyed at the end of the round, the round is considered a draw.

[0177] This invention performs five rounds of iterative training on each of the three policy models in HIAPTF, gradually improving the performance of each policy model. The training scenario for the first-generation evasion policy is a one-way attack scenario, where the evasion policy acts as the target, absorbing enemy missiles. The training steps are as follows: The first step involves training a rule-based launch strategy, which chases and quickly targets enemy aircraft, locking onto and launching missiles in 100% of rounds. Training for evasion strategies from the 2nd to 5th generations takes place during the missile evasion phase of a two-aircraft game scenario, using the previous generation's launch strategy as the training instructor. Each strategy is trained for 10 rounds based on the previous generation's evasion strategy. 7 10 6 10 5 10 4 Step. The training curve for the avoidance strategy is shown in the figure. Figure 4 As shown.

[0178] The training for the first-generation launch strategy was conducted in a single scenario involving chasing randomly maneuvering enemy aircraft. The goal was to learn basic chasing maneuvers and the ability to lock onto enemy aircraft. The training steps were 4 × 10. 8 Step 2. Training for launch strategies from generation 2 to 5 is conducted during the pursuit phase of a two-aircraft game scenario. The goal is to improve missile hit rate. The opponents are the latest generation of evasion strategies and the previous generation of launch strategies. Training is conducted 2.5 × 10⁻⁶ times based on the previous generation launch strategy. 7 10 6 10 5 10 4 Step. The training curve of the launch strategy is shown in the figure. Figure 5 As shown.

[0179] To fully evaluate the performance of HIAPTF, a single-policy model was directly trained using the PPO algorithm. The network structure of this model is basically the same as the policy model in HIAPTF, except that the input layer dimension is changed to 33 dimensions. Its input features integrate the 30 deduplicated state features of the evasion and launch policies in Table 1. When some state features are unavailable, the input is 0. Three additional flag features are added: whether the enemy aircraft is alive, whether there is an enemy missile in flight, and whether our aircraft has launched a missile. The reward function during PPO agent training integrates... and Except for the reward signal from the launch guidance model, all other rewards are used. The training scenario is also a two-player game, with the opponent being an agent composed of the HIAPTF first-generation avoidance strategy and a rule-based launch strategy with a 100% locked launch rate. The training curve of the PPO agent is shown in the figure. Figure 6 As shown.

[0180] The performance evaluation of the agents is also conducted in a two-player game scenario. Each evaluation involves 10,000 rounds of competition between the two sides. In every two rounds (e.g., rounds 1 and 2, rounds 3 and 4), the initial positions of the two sides in the second round are the same as the initial positions in the first round, ensuring the overall fairness of the competition between the two sides.

[0181] Table 2 shows the adversarial results of the various generations of agents trained using HIAPTF and the PPO agent. The launch rate is the ratio of rounds in which missiles were launched to the total number of rounds; the hit rate is the ratio of rounds in which missiles were launched and hit the enemy aircraft to the total number of rounds in which missiles were launched; the evasion rate is the ratio of rounds in which enemy missiles were successfully evaded to the total number of rounds in which the enemy launched missiles; and the net win rate is the ratio of the difference between the number of winning rounds and the number of losing rounds to the total number of rounds.

[0182] Table 2. Adversarial Results between HIAPTF Agent and PPO Agent

[0183] ;

[0184] As shown in Table 2, each generation of the HIAPTF agent outperforms the PPO agent in all key metrics. In terms of launch rate and hit rate, the HIAPTF agent is more consistently able to identify and utilize effective launch opportunities, with launch rates exceeding 96% across all generations and hit rates increasing from 58.95% to 65.90%. Regarding avoidance rate, higher-generation HIAPTF agents demonstrate stronger survivability, reaching a maximum of 98.32%. This verifies that HIAPTF can achieve decoupled learning and simultaneous enhancement of offensive and defensive capabilities through iterative evolution. In terms of win rate and net win rate, the HIAPTF agent achieves a significant advantage. With increasing iteration generations, its win rate against the PPO agent gradually improves, reaching a maximum net win rate of 62.51%, demonstrating HIAPTF's significant advantage in enhancing the overall combat capabilities of UAVs.

[0185] To verify the generalization ability of the HIAPTF agent against all previous generations of agents, pairwise adversarial matches were conducted between agents from different generations. The net win rate of each side was calculated, and the net win rate heatmap is shown below. Figure 7 As shown. Taking the 4th row and 2nd column as an example, the value in this box represents the net win rate of the 4th generation agent against the 2nd generation agent as 20.88%. From... Figure 7 As can be seen, the net win rate of the HIAPTF agent against all previous generation agents is positive and increases with the increase of the generation gap. This indicates that compared with the previous generation agents, the later generation agents have stronger generalization ability, and the iterative training method can improve the win rate of the agent.

[0186] Figure 8This is a trajectory diagram of a confrontation between our fifth-generation HIAPTF agent and the enemy's first-generation HIAPTF agent. In this round, both sides reached a lock-on condition and launched missiles simultaneously. During the missile evasion phase, our agent first flew straight for a short distance, then made a lateral turn, and finally dived downwards as the enemy missile approached, performing two U-shaped maneuvers in different directions throughout the phase, ultimately successfully evading the enemy missile. After entering the missile evasion phase, the enemy immediately dived downwards, then pulled up as our missile approached, performing a U-shaped maneuver throughout the phase, but was ultimately hit by our missile. It can be seen that both sides were able to perform U-shaped maneuvers, which is an effective evasion maneuver. However, our agent was able to perform evasion maneuvers in a more complex and periodic manner, thereby consuming more of the missile's kinetic energy and ultimately successfully evading the missile and winning the confrontation. This shows that through confrontation training with opponents of varying intelligence levels, agents can learn more complex and effective maneuvering strategies.

[0187] To test the effectiveness of the launch guidance model in guiding missile launch timing, the system was set to launch missiles only when the real-time predicted missile hit rate exceeded a specified threshold, while the enemy would launch missiles immediately upon locking onto the friendly aircraft. Both sides used first-generation HIAPTF agents with identical settings. The friendly adversarial data, with different hit rate thresholds, was analyzed as follows: Figure 9 As shown in the figure. The missile survival rate refers to the ratio of rounds in which our side did not launch missiles and our aircraft survived to the total number of rounds.

[0188] from Figure 9 As can be seen, the hit rate increases with the increase of the threshold, reflecting the effective hit rate prediction capability of the launch guidance model. However, the higher the hit rate requirement, the more likely the agent will abandon launch opportunities in more rounds due to failing to meet the requirement, leading to a decrease in the launch rate. This decrease in the launch rate, in turn, leads to a decrease in the net win rate, because even missiles launched with a low predicted hit rate have a chance to hit the enemy aircraft. Missing the first launch opportunity to achieve a high hit rate and evade enemy missiles often results in a second launch opportunity in many scenarios. Therefore, in such scenarios, if only the win rate is considered, the missile should be launched immediately upon seizing the opportunity, or only a lower hit rate requirement should be set. One advantage of setting a higher hit rate is a higher missile retention rate, which may have some tactical significance.

[0189] To examine the impact of the reward signal provided by the launch guidance model on the training of the launch strategy model, this invention additionally trained a first-generation launch strategy model without the guidance reward signal. The training process was identical except for this difference. A comparison of the training curves for the first-generation launch strategies with and without the guidance reward signal is shown in the figure below. Figure 10As shown, the training curves of the two are similar, indicating that the artificial reward function designed in this invention can also enable the launch strategy to achieve basic training results.

[0190] To further explore the impact of guidance reward signals on the combat effectiveness of the launch strategy model, a pairwise game was conducted between three launch strategies: the initial launch strategy with guidance reward signals (A), the initial launch strategy without guidance reward signals (B), and the rule-based launch strategy with a 100% locked launch rate (C). All three launch strategies were paired with the same initial evasion strategy. The results of the game are shown in Table 3.

[0191] Table 3. Results of countermeasures against launch strategies to verify the effect of the launch guidance model's reward signal.

[0192]

[0193] As shown in Table 3, the initial launch strategy with guided reward signals outperforms the initial launch strategy without guided reward signals in terms of launch rate, hit rate, and net win rate. Furthermore, it can globally defeat the rule-based launch strategy, which the initial launch strategy without guided reward signals cannot achieve. This indicates that the reward signals provided by the launch guidance model have a positive effect on the training of the launch strategy model, guiding the launch strategy to learn more efficiently how to find high-hit-rate launch opportunities, highlighting the important role of the launch guidance model in HIAPTF.

[0194] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A hierarchical iterative adversarial strategy network training method, characterized in that, Includes the following steps: S1: Train the reward function based on the current enemy missile avoidance strategy and evasion strategy. Obtain the evasion result reward corresponding to the current evasion strategy for enemy aircraft missiles, and update the evasion strategy network based on the current evasion result reward; S2: Train the reward function based on the current missile launch strategy and launch strategy. Obtain the missile launch result reward corresponding to the current missile launch strategy, and update the launch strategy network based on the current missile launch result reward. Including hit rate reward function When the distance between friendly and enemy aircraft is no greater than the maximum distance at which the missile radar can lock onto the enemy aircraft, the real-time output value of the current generation of trained launch guidance model is obtained, and the real-time output value of the launch guidance model is used as... The function value; S3: Repeat S1-S2 until the preset convergence condition is met to obtain the trained hierarchical iterative adversarial strategy network; Avoidance strategy training reward function Specifically: ； in, For out-of-bounds penalty functions, To avoid the result reward function; Out-of-bounds penalty function and the reward function for avoiding outcomes Specifically: ； ； In the formula, The flight altitude of the intelligent agent; The preset flight altitude range for the intelligent agent; Launch strategy training reward function Including the initial launch training reward function and the launch training reward function of descendants : ； ； in, For distance reward function, For the attack angle reward function, For the hit rate reward function, To continuously pursue the reward function, To successfully hit the reward function; Distance reward function for: ； In the formula, The distance between enemy and friendly aircraft; Minimum safe launch distance; This is the maximum distance at which the missile radar can lock onto enemy aircraft; Attack Angle Reward Function : ； In the formula, The lock-on angle for the intelligent agent against the target enemy aircraft; This is the maximum lock-on angle at which the missile radar can lock onto enemy aircraft; Hit rate reward function for: ； In the formula, This is the real-time output value of the launch guidance model; Continuous pursuit of reward function for: ； Successfully hit the reward function for: 。 2. The hierarchical iterative adversarial strategy network training method according to claim 1, characterized in that, Both the evasion strategy network and the launch strategy network include an Actor network and a Critic network, respectively. The specific training process for the Actor network and the Critic network includes: ① Initialize the network parameters of the Actor network and the Critic network, and initialize the preset data threshold and the preset maximum number of training steps; ② Both Actor networks and Critic networks obtain state information s from the environment. t ; ③ The Actor network outputs the action-motion type a based on the state information. t , and get a t Then execute the corresponding maneuver and obtain the corresponding reward information r. t Then obtain the next state information s of the current environment. t+1 Among them, reward information r t In the evasion strategy network, according to Acquired, in the launch strategy network according to Obtain; ④ Transfer the currently acquired data group (s) t a t r t s t+1 Stored in the experience replay area; ⑤ When the data stored in the experience playback area does not reach the preset data threshold, continuously acquire data groups (s) t a t r t s t+1 (until the preset data threshold is reached); When the data stored in the experience replay area reaches the preset data threshold, data is randomly sampled from the experience replay area to update the network parameters of the Actor network and the Critic network. ⑥ Repeat steps ②-⑤ until the maximum number of training steps is reached, and obtain the trained avoidance strategy network and launch strategy network.

3. The hierarchical iterative adversarial strategy network training method according to claim 1, characterized in that, During the training of the evasion strategy network, the first generation of evasion strategies is trained according to a preset rule-based launch strategy, which provides the missile source for the training of the evasion strategy. The second generation and subsequent evasion strategies are trained using the launch strategy obtained from the previous generation as a coach in the missile evasion phase of the two-machine game confrontation scenario. During the training of the launch strategy network, the training scenario for the first generation of launch strategies is a preset enemy aircraft chasing scenario. Under this preset enemy aircraft chasing scenario, the enemy aircraft randomly performs maneuvers, and the launch strategy network only chases and locks onto the enemy aircraft, maintaining the lock-on state without launching missiles. The second generation and later launch strategies are trained in the enemy aircraft chasing phase of a two-aircraft game confrontation scenario.

4. The hierarchical iterative adversarial strategy network training method according to claim 2, characterized in that, The loss function of the Actor network is: ； in, [· As expected, This represents the ratio of the probabilities of the new and old strategies. and They represent the old and new strategies in s respectively. t Status completed a t The probability is used for importance sampling. The dominance function measures the action. In state The relative value below, clip(·) represents the clip function, used to... Limited to [1- ,1+ Within the scope, to prevent excessively large policy updates, among which... For limiting parameters; The loss function of the Critic network is: ； in, This indicates the current value network's state information s t The value of the predicted state, This represents the target state value calculated using discounted returns and generalized advantage estimation.

5. The hierarchical iterative adversarial strategy network training method according to claim 2, characterized in that, Both the Actor network and the Critic network consist of one input layer, two hidden layers, and one output layer. Each hidden layer contains 256 neurons and uses the ReLU activation function. The input features of the evasion strategy network include: enemy missile and aircraft distance scalar, enemy missile and aircraft horizontal position difference, enemy missile altitude, aircraft altitude, enemy missile Euler angle, aircraft Euler angle, enemy missile velocity vector and aircraft velocity vector; The input features of the launch strategy network include: enemy and friendly distance scalar, enemy and friendly horizontal position difference, enemy altitude, friendly altitude, enemy Euler angle, friendly Euler angle, enemy velocity vector, friendly velocity vector, enemy linear velocity scalar, friendly linear velocity scalar, and friendly lock-on angle against the enemy. The Actor network outputs for both the evasion and launch strategies are integers in the range [0, 6], representing 7 possible actions: stable flight, climb, dive, left roll, right roll, left yaw, and right yaw. The Critic network output is a one-dimensional state value estimate.

6. The hierarchical iterative adversarial strategy network training method according to claim 2, characterized in that, The launch guidance model is trained using a supervised learning method, which includes the following: The original dataset is obtained from the missile engagement simulation dataset. The original dataset is divided into training set, validation set and test set, and the dataset is normalized. Each row in the original dataset represents a sample data. Each sample data includes a feature vector representing the engagement state and its corresponding binary label, which is used to indicate whether the missile launched in this state hits the target. The training set data is divided into multiple mini-batches, and one batch of data is used in each iteration. Each iteration process includes: Perform forward propagation on the training set samples; Calculate the loss function; Perform backpropagation and update network parameters; After each training cycle, the current model performance is evaluated using the validation set, and the loss value on the validation set is continuously monitored. Training stops when the validation loss decreases by less than a specified threshold for more than a preset number of decreases. When training terminates, the model parameters with the lowest validation loss are used as the final launch guidance model, and the final launch guidance model is evaluated using a test set.

7. The hierarchical iterative adversarial strategy network training method according to claim 5, characterized in that, The launch guidance model employs a feedforward deep neural network as a binary classification discriminant model, including: Input layer: Used to obtain the original dataset. The input features of the launch guidance model include: enemy and aircraft distance scalar, enemy and aircraft horizontal position difference, enemy aircraft altitude, aircraft altitude, enemy aircraft Euler angle, aircraft Euler angle, enemy aircraft velocity vector, aircraft velocity vector, enemy aircraft linear velocity scalar, aircraft linear velocity scalar, and aircraft lock angle against the enemy aircraft. Hidden layers: Contain two fully connected hidden layers, each followed by a LeakyReLU activation function and Dropout regularization operation; Output layer: 1 neuron, activation function is sigmoid, the output value is compressed to the (0,1) interval, as the predicted probability of missile hitting the target.

8. An application of a hierarchical iterative adversarial strategy network training method, based on any one of claims 1-7, for unmanned aerial vehicle (UAV) control, characterized in that, The operation of a drone consists of three stages: Phase 1 is the pursuit phase, which is the default phase when our aircraft first enters the combat state. The launch strategy output by the trained launch strategy network controls the launch, and missiles are launched immediately when the enemy aircraft is successfully locked, or when the real-time hit rate predicted by the trained launch guidance model is greater than a specified threshold. Phase 2 is the phase of moving away from enemy aircraft. It uses a rule-based autopilot mode to adjust the heading angle in real time to the opposite direction of the enemy aircraft, and flies horizontally after reaching the set altitude to move away from the enemy aircraft as far as possible. Phase 3 is the missile evasion phase, controlled by the evasion strategy output by the trained evasion strategy network.

Images