Layered iterative adversarial strategy network training method and application thereof

Abstract

The invention discloses a hierarchical iterative adversarial strategy network training method and application thereof. The method comprises the following steps: S1, updating an evasion strategy network according to a current evasion result award; s2, the launching strategy network is updated according to the current missile launching result reward, RF comprises a hit rate reward function RM, when the distance between the fighter plane and the foe plane is not larger than the maximum distance that the missile radar can lock the enemy plane, the real-time output value of the launching guidance model after current generation training is obtained, and the real-time output value of the launching guidance model is obtained; taking a real-time output value of the launching guidance model as a function value of the RM; and S3, repeating the steps S1 to S2 until a preset convergence condition is reached, and obtaining a trained layered iterative adversarial strategy network.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of reinforcement learning, in particular to a hierarchical iterative adversarial strategy training system based on deep reinforcement learning and a method thereof. BACKGROUND

[0002] In the current intelligent era, with the rapid development of artificial intelligence technology and aviation technology, unmanned aerial vehicles (UAVs) play an increasingly important role in modern air combat. Traditional manned aircraft are limited by factors such as pilot physiological limits, reaction speed, and psychological stress in air combat, while UAVs can break through these bottlenecks, withstand higher overload, and perform higher-risk tasks, and can be deployed in large numbers at relatively low cost, thus adapting to more new combat modes. In the future air combat environment, UAVs not only need to be able to cooperate with manned aircraft, but also need to have the ability to independently perform tasks. Therefore, it is crucial to endow UAVs with real-time and autonomous tactical decision-making capabilities.

[0003] Air combat decision-making is a complex dynamic game process involving various information interactions and environmental changes between the enemy and the self, and poses extremely high requirements on the adaptability and robustness of decision-making algorithms. Traditional air combat decision-making relies heavily on experience rules, which cannot meet the real-time and accuracy requirements in complex air combat environments. Deep reinforcement learning technology, however, establishes a feedback loop between the agent and the environment, allowing UAVs to continuously experiment and learn from themselves in a simulated air combat environment, and thus learn tactical strategies suitable for high dynamic environments in a large number of adversarial processes.

[0004] However, although deep reinforcement learning technology exhibits strong strategy learning capabilities, the effectiveness of agent training is largely dependent on the design of the reward function, especially in dynamic and high-dimensional air combat game scenarios. It is often difficult to design a reward function that can accurately guide the agent to form a reasonable maneuvering strategy. Traditional reward function design usually relies on the prior knowledge of domain experts and requires careful manual design of a series of quantitative indicators reflecting air combat objectives (such as shooting down enemy aircraft, occupying a favorable attack position, and avoiding threats). This not only requires a large amount of domain knowledge, but also easily causes inconsistencies between training objectives and actual combat objectives, thereby limiting the upper limit of agent performance. SUMMARY

[0005] Therefore, the present application provides a hierarchical iterative adversarial strategy training system based on deep reinforcement learning and a method thereof to at least solve the problem that the training effectiveness of a deep reinforcement learning agent in an air combat game scenario is highly dependent on the design of a reward function in the prior art.

[0006] To achieve the above-mentioned purpose, the present application adopts the following technical solutions: A hierarchical iterative adversarial strategy network training method, comprising the following steps: S1: According to the current missile evasion strategy and the evasion strategy training reward function Get the evasion result reward corresponding to the current evasion enemy aircraft missile strategy, update the evasion strategy network according to the current evasion result reward; S2: According to the current missile launch strategy and launch strategy training reward function Get the missile launch result reward corresponding to the current missile launch strategy, update the launch strategy network according to the current missile launch result reward, wherein Including hit rate reward function When the distance between the enemy and the aircraft is not greater than the maximum distance that the missile radar can lock the enemy aircraft, get the real-time output value of the launch guidance model after the current generation training, and take the real-time output value of the launch guidance model as the function value of ; S3: Repeat S1-S2 until the preset convergence condition is reached, and get the trained hierarchical iterative adversarial strategy network.

[0007] Preferably, the evasion strategy network and the launch strategy network each include an Actor network and a Critic network, and the specific training process of the Actor network and the Critic network includes: ①Initialize the network parameters of the Actor network and the Critic network, and initialize the preset data threshold and the preset maximum training step number; ②The Actor network and the Critic network each obtain state information from the environment; ③The Actor network outputs action-manipulation type according to the state information, gets after executing the corresponding maneuver, and gets the corresponding reward information , then obtains the next state information of the current environment; wherein the reward information is obtained in the evasion strategy network according to , and is obtained in the launch strategy network according to ; ④Store the current obtained data group ( , , , ) to the experience replay area; ⑤When the data stored in the experience replay area does not reach the preset data threshold, continuously obtain the data group ( , , , ), until the preset data threshold is reached; When the data stored in the experience replay area reaches a preset data threshold, the network parameters of the Actor network and the Critic network are updated by randomly sampling data from the experience replay area; ⑥Repeat steps ②-⑤ until the maximum number of training steps is reached, obtaining the trained evasion strategy network and launch strategy network.

[0008] Preferably, in the training process of the evasion strategy network, the training of the initial evasion strategy is based on a preset rule launch strategy, which provides the source of the missile for the training of the evasion strategy; the second generation and subsequent evasion strategies are trained in the evasion missile phase of the two-machine game confrontation scene according to the launch strategy obtained by the previous generation training as a coach; In the training process of the launch strategy network, the training scene of the initial launch strategy is a preset pursuit enemy scene, in which the enemy aircraft randomly performs maneuvers, and the launch strategy network only pursues and locks the enemy aircraft, maintains the locked state but does not launch the missile; the second generation and subsequent launch strategies are trained in the pursuit enemy aircraft phase of the two-machine game confrontation scene.

[0009] Preferably, the evasion strategy training reward function Specifically, ; Wherein, is the out-of-bound penalty function, is the evasion result reward function; The launch strategy training reward function includes the initial launch training reward function and the subsequent launch training reward function : ; ; Wherein, is the distance reward function, is the attack angle reward function, is the hit rate reward function, is the continuous pursuit reward function, is the successful hit reward function.

[0010] Preferably, the out-of-bound penalty function and the evasion result reward function Specifically, ; ; In the formula, is the height of the intelligent agent; The preset flight altitude range for the intelligent agent; Distance reward function for: ; In the formula, The distance between enemy and friendly aircraft; Minimum safe launch distance; This is the maximum distance at which the missile radar can lock onto enemy aircraft; Attack Angle Reward Function : ; In the formula, The lock-on angle for the intelligent agent against the target enemy aircraft; This is the maximum lock-on angle at which the missile radar can lock onto enemy aircraft; Hit rate reward function for: ; In the formula, This is the real-time output value of the launch guidance model; Continuous pursuit of reward function for: ; Successfully hit the reward function for: .

[0011] Preferably, the loss function of the Actor network is: ; in, As expected, This represents the ratio of the probabilities of the new and old strategies. and They represent the old and new strategies respectively. Status complete The probability is used for importance sampling. The dominance function measures the action. In state The relative value below, clip(·) represents the clip function, used to... Limited to Within the scope, to prevent excessively large policy updates, among which... For limiting parameters; The loss function of the Critic network is: ; in, This indicates the current value network's state information. predicted state value, target state value calculated by discounted return and generalized advantage estimation.

[0012] Preferably, both the Actor network and the Critic network are composed of 1 input layer, 2 hidden layers, and 1 output layer, each hidden layer containing 256 neurons, and using ReLU activation function. The input features of the evasion strategy network include: the distance scalar between the enemy missile and the aircraft, the horizontal position difference between the enemy missile and the aircraft, the height of the enemy missile, the height of the aircraft, the Euler angle of the enemy missile, the Euler angle of the aircraft, the velocity vector of the enemy missile, and the velocity vector of the aircraft. The input features of the launch strategy network include: the distance scalar between the enemy aircraft and the aircraft, the horizontal position difference between the enemy aircraft and the aircraft, the height of the enemy aircraft, the height of the aircraft, the Euler angle of the enemy aircraft, the Euler angle of the aircraft, the velocity vector of the enemy aircraft, the velocity vector of the aircraft, the linear velocity scalar of the enemy aircraft, the linear velocity scalar of the aircraft, and the lock angle of the aircraft against the enemy aircraft. The output of the Actor network of the evasion strategy and the launch strategy is an integer in the range of [0, 6], representing 7 possible actions: steady flight, climb, dive, left roll, right roll, left yaw, and right yaw. The output of the Critic network is a state value estimate in one dimension.

[0013] Preferably, the launch guidance model is trained using a supervised learning training method, which includes the following specific contents: The original dataset is obtained from the missile combat simulation dataset, and the original dataset is divided into a training set, a validation set, and a test set, and the dataset is normalized. Each row in the original dataset represents a sample data, and each sample data includes a feature vector representing the combat state and its corresponding binary label, which indicates whether the launched missile hits the target in that state. The training set data is divided into multiple small batches, and one batch of data is used for each iteration. Each iteration process includes: Forward propagation of the training set samples; Calculate the loss function; Backpropagation and update of network parameters; After each training cycle, the validation set is used to evaluate the performance of the current model, and the loss value on the validation set is continuously monitored. When the continuous decrease in validation loss is less than a specified threshold for more than a preset number of times, the training stops. When the training is terminated, the model parameters with the lowest validation loss are used as the final launch guidance model, and the final launch guidance model is evaluated through the test set.

[0014] Preferably, the launch guidance model uses a feedforward deep neural network as a binary classification discrimination model, which includes: Input layer: used to obtain the original data set, the input features of the launch guidance model include: enemy and my machine distance scalar, enemy and my machine horizontal position difference, enemy height, my machine height, enemy Euler angle, my machine Euler angle, enemy speed vector, my machine speed vector, enemy linear speed scalar, my machine linear speed scalar and my machine locking angle for enemy; Hidden layer: contains two fully connected hidden layers, each hidden layer is connected with LeakyReLU activation function and Dropout regularization operation; Output layer: 1 neuron, the activation function is Sigmoid, the output value is compressed to the interval (0, 1), as the prediction probability of missile hitting target.

[0015] An application of a hierarchical iterative adversarial policy network training method is used for unmanned aerial vehicle control, and the control of the unmanned aerial vehicle includes three stages: Stage 1 is the pursuit of enemy aircraft stage, which is the default stage when the aircraft enters the combat state, and is controlled by the launch strategy output by the trained launch strategy network, and the missile is launched immediately when the enemy aircraft is successfully locked or when the real-time hit rate predicted by the trained launch guidance model is greater than the specified threshold; Stage 2 is the stage of moving away from the enemy aircraft, using a rule-based automatic driving mode, adjusting the heading angle to the opposite direction of the enemy aircraft in real time, and flying horizontally after reaching the set height, and moving away from the enemy aircraft as far as possible; Stage 3 is the evading missile stage, which is controlled by the evading strategy output by the trained evading strategy network.

[0016] According to the technical solution, compared with the prior art, the application provides a hierarchical iterative adversarial policy network training method and its application, which has the following beneficial effects: In view of the problem that the training effect of the deep reinforcement learning intelligent agent in the air combat game scene is highly dependent on the reward function design, the application proposes a new hierarchical iterative adversarial policy training framework based on deep reinforcement learning HIAPTF. The framework innovatively introduces a launch guidance model, and uses its real-time output value as a reward signal, so that the reward function has the ability of self-adaptation and evolution, and through iterative training, the whole strategy system is indirectly optimized, so that the intelligent agent can learn more essential winning keys, and the learning efficiency and generalization of the intelligent agent in the complex air combat game environment are improved. BRIEF DESCRIPTION OF DRAWINGS

[0017] In order to make the technical solutions in the embodiments of the present application or the prior art clearer, the accompanying drawings needed in the embodiments or prior art description will be briefly introduced below. Obviously, the accompanying drawings in the following description only represent some of the embodiments of the present application, and all other embodiments obtained by those of ordinary skill in the art without creative efforts based on these drawings also belong to the protection scope of the present application.

[0018] Figure 1 A flowchart of a hierarchical iterative adversarial policy network training method provided by the present application is shown in the figure. Figure 2 An air combat geometric relationship schematic diagram provided by an embodiment of the present application is shown in the figure. Figure 3 An unmanned aerial vehicle control schematic diagram provided by an embodiment of the present application is shown in the figure. Figure 4 An evasive strategy training curve diagram provided by an embodiment of the present application is shown in the figure. Figure 5 A launch strategy training curve diagram provided by an embodiment of the present application is shown in the figure. Figure 6 A PPO agent training curve diagram provided by an embodiment of the present application is shown in the figure. Figure 7 An adversarial net win rate heat map of agents of each generation provided by an embodiment of the present application is shown in the figure. Figure 8 A trajectory diagram of a 5th generation agent (our side) and a 1st generation agent (enemy side) provided by an embodiment of the present application is shown in the figure. Figure 9 A guidance effect schematic diagram of a launch guidance model provided by an embodiment of the present application is shown in the figure. Figure 10 A training curve comparison diagram of a first generation launch strategy with and without a guidance reward signal provided by an embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0019] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments only represent some of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the protection scope of the present application.

[0020] The present application provides a hierarchical iterative adversarial policy network training method, wherein the hierarchical iterative adversarial policy network is abbreviated as HIAPTF, and includes the following steps: S1: According to the current evasive enemy aircraft missile strategy and the evasive strategy training reward function obtaining an evasion result reward corresponding to the current evasion enemy aircraft missile strategy, and updating the evasion strategy network according to the current evasion result reward; S2: obtaining a current missile launch strategy and a launch strategy training reward function obtaining a missile launch result reward corresponding to the current missile launch strategy, and updating the launch strategy network according to the current missile launch result reward, wherein including a hit rate reward function When the distance between the enemy and the aircraft is not greater than the maximum distance at which the missile radar can lock the enemy aircraft, the real-time output value of the current generation of the trained launch guidance model is obtained, and the real-time output value of the launch guidance model is taken as The function value of the function; S3: repeating S1-S2 until a preset convergence condition is reached, and obtaining a trained hierarchical iterative adversarial strategy network.

[0021] It should be noted that: The hierarchical iterative adversarial strategy training framework designed by the present application is as shown in Figure 1 The framework contains three strategy models, two maneuver strategy models trained using deep reinforcement learning and one launch guidance model trained using supervised learning. For the overall control of the unmanned aerial vehicle agent, the two core maneuver strategies of evasion strategy and launch strategy are mainly used for control, and the launch guidance model is used for controlling the timing of missile launch and supporting the training of the launch strategy. The training processes of the three strategy models are progressive, and the performance can be enhanced through iterative training.

[0022] The air combat scene of the present application focuses on 1v1 close-range air combat, and a six-degree-of-freedom air combat geometric model is established, and the air combat geometric relationship diagram is as shown in Figure 2 The lower plane is the aircraft, and the upper plane is the enemy aircraft, is the speed vector of the aircraft, is the target enemy aircraft speed vector. Let the positions of the aircraft and the target enemy aircraft in the navigation coordinate system be and , then the line-of-sight vector of the enemy aircraft relative to the aircraft is , the direction is from the aircraft to the enemy aircraft, and the size is the distance between the enemy and the aircraft. .

[0023] In order to further implement the above technical scheme, the evasion strategy network and the launch strategy network each include an Actor network and a Critic network, and the specific training process of the Actor network and the Critic network includes: ①Initialize the network parameters of the Actor network and the Critic network, and initialize the preset data threshold and the preset maximum number of training steps; ② Both Actor networks and Critic networks obtain state information from the environment. ; ③ The Actor network outputs the action-motion type based on the state information. ,get Then execute the corresponding maneuver and receive the corresponding reward information. Then obtain the next state information of the current environment. Among them, reward information In the evasion strategy network, according to Acquired, in the launch strategy network according to Obtain; ④ Transfer the currently acquired data group ( , , , Stored in the experience replay area; ⑤ When the data stored in the experience playback area does not reach the preset data threshold, continue to acquire data groups ( , , , (until the preset data threshold is reached); When the data stored in the experience replay area reaches the preset data threshold, data is randomly sampled from the experience replay area to update the network parameters of the Actor network and the Critic network. ⑥ Repeat steps ②-⑤ until the maximum number of training steps is reached, and obtain the trained avoidance strategy network and launch strategy network.

[0024] To further implement the above technical solution, during the training process of the evasion strategy network, the training of the first generation of evasion strategies is based on a preset rule-based launch strategy, which provides the missile source for the training of the evasion strategy; the second generation and subsequent evasion strategies are trained in the missile evasion stage of the two-aircraft game confrontation scenario, using the launch strategy obtained from the previous generation as a coach. During the training of the launch strategy network, the training scenario for the first generation of launch strategies is a preset enemy aircraft chasing scenario. Under this preset enemy aircraft chasing scenario, the enemy aircraft randomly performs maneuvers, and the launch strategy network only chases and locks onto the enemy aircraft, maintaining the lock-on state without launching missiles. The second generation and later launch strategies are trained in the enemy aircraft chasing phase of a two-aircraft game confrontation scenario.

[0025] It should be noted that: The training scene of the evading strategy needs the participation of the enemy missile, and there is no available launch strategy model at the beginning of the training, so the training of the initial evading strategy uses a rule-based launch strategy to provide the source of the missile for the training of the evading strategy, that is, the rule-based launch strategy can pursue the enemy and launch the missile as soon as the missile radar locks the enemy. The second generation and later evading strategies use the launch strategy trained by the previous generation as a coach to train in the evading missile phase of the two-machine game confrontation scene.

[0026] The key reward of the launch strategy training comes from the reward signal provided by the launch guidance model, but the training of the initial generation and the later generations sets different training scenes. The training scene of the initial launch strategy is a single pursuit enemy scene, in which the enemy randomly performs maneuvers, and the agent needs to pursue and lock the enemy as much as possible, maintain the locked state but not launch the missile. The purpose of this scene is to enable the agent to learn the basic pursuit maneuver and the ability to lock the enemy. The second generation and later launch strategies are trained in the pursuit enemy phase of the two-machine game confrontation scene to improve the hit rate of the missile.

[0027] In order to further implement the above technical solutions, the evading strategy training reward function Specifically, ; Among them, is the out-of-bound penalty function, is the evading result reward function; The launch strategy training reward function includes the initial launch training reward function and the later launch training reward function : ; ; Among them, is the distance reward function, is the attack angle reward function, is the hit rate reward function, is the continuous pursuit reward function, is the successful hit reward function.

[0028] In order to further implement the above technical solutions, the out-of-bound penalty function and the evading result reward function are specifically: ; ; In the formula, is the flight height of the agent; The preset flight height interval of the intelligent agent; Distance reward function For: In the formula, The distance between the enemy aircraft and the intelligent agent; The minimum safe launch distance; The maximum distance at which the missile radar can lock the enemy aircraft; Attack angle reward function : In the formula, The locking angle of the intelligent agent to the target enemy aircraft; The maximum locking angle at which the missile radar can lock the enemy aircraft; Hit rate reward function For: In the formula, The real-time output value of the launch guidance model; Continuous pursuit reward function For: Successful hit reward function For: .

[0029] It should be noted that: The flight height interval of the intelligent agent is limited to 0-11km in this embodiment, and if the intelligent agent exceeds the height boundary, it is determined to be damaged. In order to enable the intelligent agent to have the ability to prevent out-of-bound, a penalty is given when the flight height of the intelligent agent is within 1km of the boundary. The present application takes =1km, =10km.

[0030] The main purpose of the evasion strategy is to make the intelligent agent miss the enemy missile by reasonable maneuvering, that is, to evade the enemy missile, so a one-time reward or penalty is given to the intelligent agent according to the result of whether the evasion is successful or not.

[0031] The launch strategy also needs to have the ability to prevent out-of-bound, so the intelligent agent that is about to go out of bound is also punished according to the out-of-bound penalty function .

[0032] In order to enable the enemy aircraft to be within the locking distance of the missile radar, the intelligent agent is encouraged to approach the enemy aircraft, and the distance reward function is set. The present application takes =0.6km, =6km.

[0033] ​​​​In order to make the agent obtain the launch window of the missile radar locking the enemy aircraft, the agent is encouraged to point the nose to the enemy aircraft, and the attack angle reward function is set. = 15°.

[0034] In order to maximize the hit rate of the missile launched by the agent, when the distance between the enemy aircraft and the agent is less than the maximum distance at which the missile can lock the enemy aircraft , the real-time output value of the launch guidance model is used as a reward signal.

[0035] In the training of the first generation launch strategy, in order to encourage the agent to continuously pursue the enemy aircraft, the continuous pursuit reward function is given to the behavior of continuously locking the enemy aircraft .

[0036] In the training of the second generation and later launch strategy in the two-machine game confrontation scene, the agent launches a missile and successfully hits the enemy aircraft, and a one-time reward is given to the behavior .

[0037] In order to further implement the above technical solutions, the loss function of the Actor network is: ; Wherein, is the expectation, the probability ratio of the new and old strategies, and respectively represent the probabilities of the new and old strategies completing in state , used for importance sampling, is the advantage function, which measures the relative value of action in state , clip(·) represents the clip function, which limits to range, to prevent the update amplitude of the strategy from being too large, wherein is the limit parameter; The loss function of the Critic network is: ; Wherein, represents the predicted state value of the current value network to state information , and represents the target state value calculated by discount return and generalized advantage estimation.

[0038] In order to further implement the above technical solutions, the Actor network and the Critic network are both composed of 1 input layer, 2 hidden layers and 1 output layer, each hidden layer contains 256 neurons, and adopts ReLU activation function.​ The input features of the evasive strategy network include: enemy missile and own aircraft distance scalar, enemy missile and own aircraft horizontal position difference, enemy missile height, own aircraft height, enemy missile Euler angle, own aircraft Euler angle, enemy missile velocity vector, and own aircraft velocity vector; The input features of the launch strategy network include: enemy aircraft and own aircraft distance scalar, enemy aircraft and own aircraft horizontal position difference, enemy aircraft height, own aircraft height, enemy aircraft Euler angle, own aircraft Euler angle, enemy aircraft velocity vector, own aircraft velocity vector, enemy aircraft linear velocity scalar, own aircraft linear velocity scalar, and own aircraft lock angle for the enemy aircraft; The Actor network outputs of the evasive strategy and the launch strategy are integers in [0, 6], representing 7 possible actions: steady flight, climb, dive, left roll, right roll, left yaw, and right yaw. The output of the Critic network is a one-dimensional state value estimate.

[0039] To further implement the above technical solutions, the launch guidance model is trained using a supervised learning training method, and the specific content includes: An original data set is obtained from a missile combat simulation data set, the original data set is divided into a training set, a validation set, and a test set, and the data set is normalized; each row in the original data set represents a sample data, and each sample data includes a feature vector representing a combat state and a corresponding binary label, which is used to indicate whether the launched missile hits the target in the state; The training set data is divided into multiple small batches, and one batch of data is used for each iteration, and each iteration process includes: Forward propagation is performed on the training set samples; The loss function is calculated; Back propagation is performed and the network parameters are updated; After each training cycle, the validation set is used to evaluate the performance of the current model, and the loss value on the validation set is continuously monitored. When the continuous decrease of the validation loss is less than a specified threshold for more than a preset number of times, the training stops; When the training is terminated, the model parameters with the lowest validation loss are used as the final launch guidance model, and the final launch guidance model is evaluated through the test set.

[0040] To further implement the above technical solutions, the launch guidance model uses a feedforward deep neural network as a binary classification discrimination model, which includes: The input layer is used to obtain the original data set, and the input features of the launch guidance model include: enemy aircraft and own aircraft distance scalar, enemy aircraft and own aircraft horizontal position difference, enemy aircraft height, own aircraft height, enemy aircraft Euler angle, own aircraft Euler angle, enemy aircraft velocity vector, own aircraft velocity vector, enemy aircraft linear velocity scalar, own aircraft linear velocity scalar, and own aircraft lock angle for the enemy aircraft; Hidden layer: contains two fully connected hidden layers, each hidden layer is followed by a LeakyReLU activation function and a Dropout regularization operation; Output layer: 1 neuron, activation function is Sigmoid, compresses the output value to the interval (0, 1) as the predicted probability of the missile hitting the target.

[0041] It should be noted that: The essence of the launch guidance model is a binary classification model trained using supervised learning, and the output value of the last sigmoid layer is used as the real-time predicted missile hit rate. The data set required for training this model is collected in a similar scenario to the training scenario of the evasion strategy, that is, the launch strategy agent launches a missile at the evasion strategy agent, and records the state data at the time of launch and whether the missile hits the final label as the training data set. One of the functions of this model is to control the timing of missile launch, for example, it can be set to launch the missile only when the real-time predicted missile hit rate is greater than 20%. In addition, the more important function of this model is to use the output value as a reward signal to participate in the training of the launch strategy to guide the launch strategy to maximize its missile hit rate.

[0042] The key reward for the launch strategy training comes from the reward signal provided by the launch guidance model, but the training of the initial generation and the offspring will set different training scenarios. The training scenario of the initial launch strategy is a single pursuit of the enemy plane scenario, in which the enemy plane randomly performs maneuvers, and the agent needs to pursue and lock the enemy plane as much as possible, maintain the locked state but do not launch the missile. The purpose of this scenario is to enable the agent to learn the basic pursuit maneuver and the ability to lock the enemy plane. The launch strategy of the 2nd generation and beyond is trained in the pursuit of the enemy plane stage in the two-machine game scenario, striving to improve the hit rate of the missile.

[0043] The neural networks used in the layered iterative countermeasure strategy network in this embodiment are all fully connected neural networks, wherein the Actor networks and the Critic networks of the evasion strategy and the launch strategy each consist of 1 input layer, 2 hidden layers and 1 output layer, each hidden layer contains 256 neurons, and a ReLU activation function is used. The input layer dimension of the network of the evasion strategy is 17, and the input layer dimension of the network of the launch strategy is 20, and the corresponding state feature inputs are as shown in Table 1. The output of the Actor network of the evasion strategy and the launch strategy is an integer in [0, 6], representing 7 possible actions: stable flight, climb, dive, left roll, right roll, left yaw and right yaw, and the output of the Critic network is a state value estimate in 1 dimension. The network of the launch guidance model consists of 1 input layer, 2 hidden layers and 1 output layer, the dimension of the input layer and the state feature input are consistent with those of the launch strategy, the number of neurons in the hidden layers is 128 and 64 respectively, a LeakyReLU activation function is used, and the output of the network is a continuous value in [0, 1], and a Sigmoid activation function is used.

[0044] Table 1 State feature input ; An application of a layered iterative countermeasure strategy network training method is used for unmanned aerial vehicle control, and the control of the unmanned aerial vehicle includes three stages: Stage 1 is a pursuit enemy aircraft stage, which is the default stage when the aircraft enters the combat state, is controlled by the launch strategy output by the trained launch strategy network, and launches a missile immediately when the enemy aircraft is successfully locked or when the real-time hit rate predicted by the trained launch guidance model is greater than a specified threshold; Stage 2 is a stage of moving away from the enemy aircraft, which uses a rule-based autopilot mode, adjusts the heading angle to the opposite direction of the enemy aircraft in real time, and flies horizontally after reaching the set height to move away from the enemy aircraft as far as possible; Stage 3 is a stage of evading missiles, which is controlled by the evasion strategy output by the trained evasion strategy network.

[0045] It should be noted that: The control of the unmanned aerial vehicle consists of three stages, and a schematic diagram is as shown in Figure 3 .

[0046] The application will be further described below through a simulation experiment: The simulation experiment of the application is based on a Harfang 3D open-source simulation air combat platform, which is a reinforcement learning research platform specially customized for fighter control tasks, allowing experimenters to customize scene, task, entity parameters and simulation parameters and other settings. The application designs multiple scenes based on the platform for agent training, evaluation and data set collection, all of which are set in the airspace above the sea area, and there are no other obstacles on the sea level to affect the flight of the warplanes and missiles.

[0047] In the two-machine game confrontation scene constructed by the application, each of the two parties uses a performance-identical unmanned combat aircraft for close combat, each aircraft has and only has one missile, and the default condition is to immediately launch the missile when successfully locking the enemy aircraft. If the distance between the missile and the target is less than 10 m, it is determined to be a hit. The warplane is damaged by the enemy missile or exceeds the altitude limit, so that the enemy missile is off target, which is evasion success. The altitude of the warplane is limited within 0~11km, the maximum speed is 400m / s, and the engine thrust remains maximum. The maximum locking distance of the missile radar is 6km, the maximum locking angle is 15°, and the condition for the missile radar to successfully lock the enemy aircraft is to maintain 3s within the maximum locking distance and angle.

[0048] In the initial situation, the initial speed of the warplanes of the two parties is 300m / s, the initial roll angle and pitch angle are both 0°, and the initial heading angle points to each other. When the initial position is randomized each time, the initial horizontal position of one warplane is the center point of the map, and the initial horizontal position of the other warplane is randomized within a horizontal annular region centered on the warplane, the inner radius and outer radius of the annular region are 6km and 9km respectively, and the initial altitudes of the two warplanes are randomized within 4km~8km.

[0049] The determination condition of the end of the round is that there is no missile flying in the airspace, and at the same time, one of the following three conditions is met: both parties have launched a missile, at least one warplane is damaged or the round time exceeds 2 minutes. If the warplane of the home party survives while the enemy warplane is damaged at the end of the round, it is determined that the home party wins this round, otherwise the enemy party wins. If both warplanes survive or are damaged at the end of the round, it is determined that the two parties are in a draw.

[0050] The application iteratively trains the three strategy models in the HIAPTF for 5 rounds, gradually improving the performance of each strategy model. The training scene of the first generation of evasion strategy is a one-way hit scene, and the evasion strategy is used as the hit party to withstand the enemy missile, and the training step is The first step involves training a rule-based launch strategy, which chases and quickly targets enemy aircraft, locking onto and launching missiles in 100% of rounds. Training for evasion strategies from the 2nd to 5th generations takes place during the missile evasion phase of a two-aircraft game scenario, using the previous generation's launch strategy as the training instructor. Each strategy is trained for 10 rounds based on the previous generation's evasion strategy. 7 10 6 10 5 10 4 Step. The training curve for the avoidance strategy is shown in the figure. Figure 4 As shown.

[0051] The training for the first-generation launch strategy was conducted in a single scenario involving chasing randomly maneuvering enemy aircraft. The goal was to learn basic chasing maneuvers and the ability to lock onto enemy aircraft. The training steps were 4 × 10. 8 Step 2. Training for launch strategies from generation 2 to 5 is conducted during the pursuit phase of a two-aircraft game scenario. The goal is to improve missile hit rate. The opponents are the latest generation of evasion strategies and the previous generation of launch strategies. Training is conducted 2.5 × 10⁻⁶ times based on the previous generation launch strategy. 7 10 6 10 5 10 4 Step. The training curve of the launch strategy is shown in the figure. Figure 5 As shown.

[0052] To fully evaluate the performance of HIAPTF, a single-policy model was directly trained using the PPO algorithm. The network structure of this model is basically the same as the policy model in HIAPTF, except that the input layer dimension is changed to 33 dimensions. Its input features integrate the 30 deduplicated state features of the evasion and launch policies in Table 1. When some state features are unavailable, the input is 0. Three additional flag features are added: whether the enemy aircraft is alive, whether there is an enemy missile in flight, and whether our aircraft has launched a missile. The reward function during PPO agent training integrates... and Except for the reward signal from the launch guidance model, all other rewards are used. The training scenario is also a two-player game, with the opponent being an agent composed of the HIAPTF first-generation avoidance strategy and a rule-based launch strategy with a 100% locked launch rate. The training curve of the PPO agent is shown in the figure. Figure 6 As shown.

[0053] The performance evaluation of the agents is also conducted in a two-player game scenario. Each evaluation involves 10,000 rounds of competition between the two sides. In every two rounds (e.g., rounds 1 and 2, rounds 3 and 4), the initial positions of the two sides in the second round are the same as the initial positions in the first round, ensuring the overall fairness of the competition between the two sides.

[0054] The results of the confrontation between each generation of HIAPTF-trained agents and PPO agents are shown in Table 2. The launch rate refers to the ratio of rounds in which missiles were launched to the total number of rounds, the hit rate refers to the ratio of rounds in which missiles were launched and hit the enemy aircraft to the number of rounds in which missiles were launched, the evasion rate refers to the ratio of rounds in which the enemy's missiles were successfully evaded to the number of rounds in which the enemy launched missiles, and the net win rate refers to the ratio of the difference between the number of winning rounds and the number of losing rounds to the total number of rounds.

[0055] Table 2 Confrontation results of HIAPTF agents and PPO agents ; As can be seen from Table 2, each generation of HIAPTF agents is superior to PPO agents in all key indicators. In terms of launch rate and hit rate, HIAPTF agents can more stably identify and utilize effective launch opportunities, with a launch rate of more than 96% and a hit rate of 58.95% to 65.90%; in terms of evasion rate, higher generation HIAPTF agents exhibit stronger survival ability, with a maximum of 98.32%. This verifies that HIAPTF can achieve decoupled learning and synchronous enhancement of attack and defense capabilities through iterative evolution. In terms of win rate and net win rate, HIAPTF agents have obvious advantages, with the confrontation win rate against PPO agents gradually increasing with the increase of iteration generations, and the highest net win rate reaching 62.51%, which reflects the significant advantage of HIAPTF in improving the comprehensive confrontation capability of unmanned aerial vehicles.

[0056] To verify the generalization of HIAPTF agents against all previous generations of agents, each generation of HIAPTF agents is pitted against each other, and the net win rates of the two opposing sides are counted. The confrontation net win rate heat map is shown in Figure 7 . Taking the 4th row and 2nd column as an example, the value in this block indicates that the net win rate of the 4th generation agent against the 2nd generation agent is 20.88%. As can be seen from Figure 7 , the net win rates of HIAPTF agents against all previous generations of agents are positive, and increase with the increase of the intergenerational gap, which indicates that compared with previous generations of agents, later generations of agents have stronger generalization, and the iterative training method can improve the win rate of the agent.

[0057] Figure 8is the confrontation trajectory graph between the 5th generation HIAPTF agent of our side and the 1st generation HIAPTF agent of the enemy side, in this round, both sides reach the lock condition at the same time and launch missiles. In the missile evasion stage, our side first flies straight for a short distance, then turns sideways, and finally dives downward when the enemy missile is approaching, making two U-shaped maneuvers in different directions throughout the stage, and finally successfully evades the enemy missile. After the enemy enters the missile evasion stage, it immediately dives downward, and pulls up when the missile of our side is approaching, making a U-shaped maneuver throughout the stage, but is still hit by the missile of our side. It can be seen that both sides can perform U-shaped maneuvers, which is an effective evasion maneuver, but our side can perform the evasion maneuver in a more complex and periodic manner, thereby consuming the kinetic energy of the missile to a greater extent, and finally successfully evading the missile and winning the confrontation. This shows that through confrontation training with more opponents of different intelligence levels, the agent can learn more complex and effective maneuvering strategies.

[0058] In order to test the guidance effect of the launch guidance model on the launch timing of the missile, it is provided that our side launches the missile only when the real-time predicted hit rate is greater than a specified threshold, while the enemy side still launches the missile as soon as it locks onto our plane, both sides being the initial generation HIAPTF agent and the rest of the settings being the same, the confrontation data of our side with different hit rate threshold requirements being as shown in Figure 9 The hit rate is the ratio of the number of rounds in which the missile of our side hits the enemy plane to the total number of rounds.

[0059] From Figure 9 it can be seen that as the threshold increases, the hit rate also increases, reflecting the effective hit rate prediction ability of the launch guidance model. However, the higher the hit rate requirement, the more the agent will give up the launch opportunity in more rounds due to not meeting the requirement, resulting in a decrease in the launch rate. And the decrease in the launch rate will lead to a decrease in the net win rate, because even in the case of low predicted hit rate, the launched missile has a chance to hit the enemy plane, and in many scenarios, missing the first launch opportunity makes it difficult to have a second launch opportunity. Therefore, in such scenarios, if only the win rate is considered, the missile should be launched immediately, or only a lower hit rate requirement should be set, and one of the benefits of setting a higher hit rate is that there will be a higher missile retention rate, which may have some significance in certain tactical aspects.

[0060] In order to test the influence of the reward signal provided by the launch guidance model on the training of the launch strategy model, the present application additionally trains an initial generation launch strategy model without the guidance of the reward signal, and the training process is identical except for this difference. The training curve comparison chart of the initial generation launch strategy with and without the guidance of the reward signal is as shown in Figure 10The training curves of the two are similar in trend, which shows that the artificial reward function designed by the application can also make the launch strategy obtain basic training effect.

[0061] To further explore the influence of the guidance reward signal on the combat effectiveness of the launch strategy model, the three launch strategies of the initial launch strategy with guidance reward signal (A), the initial launch strategy without guidance reward signal (B), and the rule launch strategy with 100% lock launch rate (C) are used for two-by-two game confrontation, and the three launch strategies are matched with the same initial evasion strategy. The confrontation results are shown in Table 3.

[0062] Table 3 Launch strategy confrontation results for testing the effect of launch guidance model reward signal

[0063] As can be seen from Table 3, the initial launch strategy with guidance reward signal is superior to the initial launch strategy without guidance reward signal in terms of launch rate, hit rate and net win rate, and can defeat the rule launch strategy globally, while the initial launch strategy without guidance reward signal cannot do so. This shows that the reward signal provided by the launch guidance model has a positive effect on the training of the launch strategy model, and can guide the launch strategy to learn how to find high hit rate launch opportunities more efficiently, highlighting the important role of the launch guidance model in the HIAPTF.

[0064] The above examples are only used to illustrate the technical solutions of the present application, but not to limit them; although the present application has been described in detail with reference to the foregoing examples, those skilled in the art should understand that they can modify the technical solutions recorded in the foregoing examples, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A hierarchical iterative adversarial strategy network training method, characterized in that, Includes the following steps: S1: Train the reward function based on the current enemy missile avoidance strategy and evasion strategy. Obtain the evasion result reward corresponding to the current evasion strategy for enemy aircraft missiles, and update the evasion strategy network based on the current evasion result reward; S2: Train the reward function based on the current missile launch strategy and launch strategy. Obtain the missile launch result reward corresponding to the current missile launch strategy, and update the launch strategy network based on the current missile launch result reward. Including hit rate reward function When the distance between friendly and enemy aircraft is no greater than the maximum distance at which the missile radar can lock onto the enemy aircraft, the real-time output value of the current generation of trained launch guidance model is obtained, and the real-time output value of the launch guidance model is used as... The function value; S3: Repeat S1-S2 until the preset convergence condition is met to obtain the trained hierarchical iterative adversarial strategy network.

2. The hierarchical iterative adversarial strategy network training method according to claim 1, characterized in that, Both the evasion strategy network and the launch strategy network include an Actor network and a Critic network, respectively. The specific training process for the Actor network and the Critic network includes: ① Initialize the network parameters of the Actor network and the Critic network, and initialize the preset data threshold and the preset maximum number of training steps; ② Both Actor networks and Critic networks obtain state information from the environment. ; ③ The Actor network outputs the action-motion type based on the state information. ,get Then execute the corresponding maneuver and receive the corresponding reward information. Then obtain the next state information of the current environment. Among them, reward information In the evasion strategy network, according to Acquired, in the launch strategy network according to Obtain; ④ Transfer the currently acquired data group ( , , , Stored in the experience replay area; ⑤ When the data stored in the experience playback area does not reach the preset data threshold, continue to acquire data groups ( , , , (until the preset data threshold is reached); When the data stored in the experience replay area reaches the preset data threshold, data is randomly sampled from the experience replay area to update the network parameters of the Actor network and the Critic network. ⑥ Repeat steps ②-⑤ until the maximum number of training steps is reached, and obtain the trained avoidance strategy network and launch strategy network.

3. The hierarchical iterative adversarial strategy network training method according to claim 1, characterized in that, During the training of the evasion strategy network, the first generation of evasion strategies is trained according to a preset rule-based launch strategy, which provides the missile source for the training of the evasion strategy. The second generation and subsequent evasion strategies are trained using the launch strategy obtained from the previous generation as a coach in the missile evasion phase of the two-machine game confrontation scenario. During the training of the launch strategy network, the training scenario for the first generation of launch strategies is a preset enemy aircraft chasing scenario. Under this preset enemy aircraft chasing scenario, the enemy aircraft randomly performs maneuvers, and the launch strategy network only chases and locks onto the enemy aircraft, maintaining the lock-on state without launching missiles. The second generation and later launch strategies are trained in the enemy aircraft chasing phase of a two-aircraft game confrontation scenario.

4. The hierarchical iterative adversarial strategy network training method according to claim 3, characterized in that, Avoidance strategy training reward function Specifically: ； in, For out-of-bounds penalty functions, To avoid the result reward function; Launch strategy training reward function Including the initial launch training reward function and the launch training reward function of descendants : ； ； in, For distance reward function, For the attack angle reward function, For the hit rate reward function, To continuously pursue the reward function, To successfully hit the reward function.

5. The hierarchical iterative adversarial strategy network training method according to claim 4, characterized in that, Out-of-bounds penalty function and the reward function for avoiding outcomes Specifically: ； ； In the formula, The flight altitude of the intelligent agent; The preset flight altitude range for the intelligent agent; Distance reward function for: ； In the formula, The distance between enemy and friendly aircraft; Minimum safe launch distance; This is the maximum distance at which the missile radar can lock onto enemy aircraft; Attack Angle Reward Function : ； In the formula, The lock-on angle for the intelligent agent against the target enemy aircraft; This is the maximum lock-on angle at which the missile radar can lock onto enemy aircraft; Hit rate reward function for: ； In the formula, This is the real-time output value of the launch guidance model; Continuous pursuit of reward function for: ； Successfully hit the reward function for: 。 6. The hierarchical iterative adversarial strategy network training method according to claim 2, characterized in that, The loss function of the Actor network is: ； in, As expected, This represents the ratio of the probabilities of the new and old strategies. and They represent the old and new strategies respectively. Status complete The probability is used for importance sampling. The dominance function measures the action. In state The relative value below, clip(·) represents the clip function, used to... Limited to Within the scope, to prevent excessively large policy updates, among which... For limiting parameters; The loss function of the Critic network is: ； in, This indicates the current value network's state information. The value of the predicted state, This represents the target state value calculated using discounted returns and generalized advantage estimation.

7. The hierarchical iterative adversarial strategy network training method according to claim 2, characterized in that, Both the Actor network and the Critic network consist of one input layer, two hidden layers, and one output layer. Each hidden layer contains 256 neurons and uses the ReLU activation function. The input features of the evasion strategy network include: enemy missile and aircraft distance scalar, enemy missile and aircraft horizontal position difference, enemy missile altitude, aircraft altitude, enemy missile Euler angle, aircraft Euler angle, enemy missile velocity vector and aircraft velocity vector; The input features of the launch strategy network include: enemy and friendly distance scalar, enemy and friendly horizontal position difference, enemy altitude, friendly altitude, enemy Euler angle, friendly Euler angle, enemy velocity vector, friendly velocity vector, enemy linear velocity scalar, friendly linear velocity scalar, and friendly lock-on angle against the enemy. The Actor network outputs for both the evasion and launch strategies are integers in the range [0, 6], representing 7 possible actions: stable flight, climb, dive, left roll, right roll, left yaw, and right yaw. The Critic network output is a one-dimensional state value estimate.

8. The hierarchical iterative adversarial strategy network training method according to claim 2, characterized in that, The launch guidance model is trained using a supervised learning method, which includes the following: The original dataset is obtained from the missile engagement simulation dataset. The original dataset is divided into training set, validation set and test set, and the dataset is normalized. Each row in the original dataset represents a sample data. Each sample data includes a feature vector representing the engagement state and its corresponding binary label, which is used to indicate whether the missile launched in this state hits the target. The training set data is divided into multiple mini-batches, and one batch of data is used in each iteration. Each iteration process includes: Perform forward propagation on the training set samples; Calculate the loss function; Perform backpropagation and update network parameters; After each training cycle, the current model performance is evaluated using the validation set, and the loss value on the validation set is continuously monitored. Training stops when the validation loss decreases by less than a specified threshold for more than a preset number of decreases. When training terminates, the model parameters with the lowest validation loss are used as the final launch guidance model, and the final launch guidance model is evaluated using a test set.

9. The hierarchical iterative adversarial strategy network training method according to claim 7, characterized in that, The launch guidance model employs a feedforward deep neural network as a binary classification discriminant model, including: Input layer: Used to obtain the original dataset. The input features of the launch guidance model include: enemy and aircraft distance scalar, enemy and aircraft horizontal position difference, enemy aircraft altitude, aircraft altitude, enemy aircraft Euler angle, aircraft Euler angle, enemy aircraft velocity vector, aircraft velocity vector, enemy aircraft linear velocity scalar, aircraft linear velocity scalar, and aircraft lock angle against the enemy aircraft. Hidden layers: Contain two fully connected hidden layers, each followed by a LeakyReLU activation function and Dropout regularization operation; Output layer: 1 neuron, activation function is sigmoid, the output value is compressed to the (0,1) interval, as the predicted probability of missile hitting the target.

10. An application of a hierarchical iterative adversarial strategy network training method, based on any one of claims 1-9, for unmanned aerial vehicle (UAV) control, characterized in that, The operation of a drone consists of three stages: Phase 1 is the pursuit phase, which is the default phase when our aircraft first enters the combat state. The launch strategy output by the trained launch strategy network controls the launch, and missiles are launched immediately when the enemy aircraft is successfully locked, or when the real-time hit rate predicted by the trained launch guidance model is greater than a specified threshold. Phase 2 is the phase of moving away from enemy aircraft. It uses a rule-based autopilot mode to adjust the heading angle in real time to the opposite direction of the enemy aircraft, and flies horizontally after reaching the set altitude to move away from the enemy aircraft as far as possible. Phase 3 is the missile evasion phase, controlled by the evasion strategy output by the trained evasion strategy network.

Images