An air combat strategy evolution method based on priority virtual self-game

Abstract

The application relates to the technical field of unmanned aerial vehicle control and artificial intelligence, in particular to an air combat strategy evolution method based on a priority virtual self-game, which comprises the following steps: constructing a strategy pool containing pre-training initial seed strategies, historical pursuit strategies and Elo rating points and a self-game pursuit environment; screening opponent strategies from the strategy pool based on the priority virtual self-game, training an unmanned aerial vehicle model and the opponent strategies in the pursuit environment, executing double-track reasoning, generating a current pursuit strategy by a pursuit intelligent agent and collecting track data; updating strategy network parameters according to the track data, updating historical strategies and Elo rating points in the strategy pool at a preset interval, removing redundant strategies with the lowest Elo rating points when the capacity is exceeded, and generating a current pursuit strategy of the unmanned aerial vehicle by the updated pursuit intelligent agent. Thus, the problems of insufficient generalization ability of intelligent agents and unstable training in the unmanned aerial vehicle pursuit game in the related art are solved.

Description

Technical Field

[0001] This application relates to the fields of unmanned aerial vehicle (UAV) control and artificial intelligence technology, and in particular to an air combat strategy evolution method based on priority virtual self-game. Background Technology

[0002] In recent years, intelligent drone pursuit and escape game decision-making technology based on deep reinforcement learning has developed rapidly and has been widely applied in scenarios such as drone autonomous combat. However, during the training process for drone pursuit and escape games, related technologies are prone to problems such as unstable agent training and insufficient generalization ability, making it difficult to adapt to dynamically changing game environments and affecting the practicality and reliability of drone pursuit and escape strategies. Summary of the Invention

[0003] This application provides a method, apparatus, electronic device and storage medium for air combat strategy evolution based on priority virtual self-game, to solve the problems of insufficient generalization ability and unstable training of intelligent agents in UAV pursuit and escape game in related technologies, and to overcome the pain point of inefficient cold start in initial training.

[0004] The first aspect of this application provides a method for air combat strategy evolution based on priority virtual self-game, comprising the following steps: constructing a self-game pursuit and escape environment and a strategy pool, wherein the strategy pool stores initial seed strategies obtained by pre-training with air combat expert experience and historical adversarial data, historical pursuit and escape strategies generated during training, and Elo rating scores corresponding to each strategy; selecting opponent strategies for the current training stage from the strategy pool based on priority virtual self-game, training a UAV model to interact with the opponent strategies in the self-game pursuit and escape environment, performing dual-track inference to collect trajectory data of the UAV model, and generating the current pursuit and escape strategy using the pursuit and escape agent model of the UAV model during the interaction process; iteratively updating the strategy network parameters of the pursuit and escape agent model according to the trajectory data, updating at least one of the historical pursuit and escape strategies and Elo rating scores in the strategy pool at preset intervals, removing redundant strategies with the lowest Elo rating scores when overcapacity occurs, and generating the current pursuit and escape strategy of the UAV using the iteratively updated pursuit and escape agent model.

[0005] Based on the aforementioned technical means, this embodiment constructs a policy pool containing pre-trained initial seed policies and Elo ratings. It employs a priority virtual self-game mechanism to dynamically select suitable opponent policies, combines dual-track inference to collect trajectory data, iteratively updates the agent's policy network parameters, and updates the policy pool at preset intervals, i.e., over-capacity removal of the lowest-scoring policy, forming a closed-loop evolutionary process. This effectively solves the problems of unstable agent training, insufficient generalization ability, and inefficient initial cold start, enhances the robustness of the policy in dynamic adversarial environments, enables autonomous evolution of UAV pursuit strategies, and improves the practicality and reliability of real-world applications.

[0006] Optionally, the opponent's strategy for the current training phase is selected from the strategy pool based on the priority virtual self-game, including: traversing the strategy pool and constructing a score set based on the Elo ratings of all extracted historical pursuit strategies; calculating the non-normalized weight of each historical pursuit strategy based on the score set, calculating the normalized probability of the corresponding historical pursuit strategy based on the non-normalized weight, generating a probability distribution based on the normalized probability of each historical pursuit strategy; performing discrete random sampling from the strategy pool based on the probability distribution to determine the opponent's strategy index for the current training phase, determining the opponent's strategy for the current training phase based on the opponent's strategy index, and freezing the network parameters of the opponent's strategy.

[0007] Based on the aforementioned technical means, this embodiment constructs a scoring set based on Elo ratings, calculates the normalized probability of each historical pursuit strategy and forms a probability distribution, and then determines the opponent's strategy through discrete random sampling according to the probability distribution, thus achieving dynamic and adaptive selection of opponent strategies; simultaneously, it freezes the opponent strategy network parameters to ensure the stability of the opponent strategies during training. Therefore, the current agent can always play against opponents of a suitable skill level, avoiding both the inefficiency of training caused by random sampling and the sparse rewards caused by a single strong opponent, thereby improving training efficiency and stability and enhancing the generalization ability of the agent's strategies.

[0008] Optionally, in a self-playing pursuit environment, a drone model is trained to interact with an adversary strategy, including: dividing the self-playing pursuit environment into multiple parallel inference threads; loading the adversary strategy into the adversary agent model of each inference thread; performing dual-track inference on the drone model and the adversary agent model through each inference thread; wherein, on the training side, the drone model receives the current situation observation vector, calculates the action probability distribution through forward propagation of a policy network with gradients, and performs random sampling to maintain exploration; on the adversary side, the adversary agent model receives the situation observation vector from the adversary's perspective, performs forward inference through a policy network with frozen parameters, and outputs action commands; inputting the action commands of both parties into a flight simulation engine in the self-playing pursuit environment, and using the flight simulation engine to calculate the flight state, relative geometric relationship, and payload attack judgment result at the next moment.

[0009] Based on the aforementioned technical means, this embodiment of the application divides the self-playing chase environment into multiple parallel inference threads to achieve dual-track asynchronous inference on the training and adversary sides. The training side employs a gradient-based policy network for forward propagation and random sampling to ensure exploratory nature and the effectiveness of policy updates. The adversary side uses a parameter-frozen policy network for forward inference to ensure adversarial stability. Finally, a flight simulation engine is used to uniformly calculate the state and adversarial results. Thus, while ensuring training efficiency and parallelism, the realism of the game interaction and the effectiveness of policy learning are improved, enabling the agent to complete chase strategy learning in a more stable and efficient parallel environment.

[0010] Optionally, the policy network parameters of the pursuit agent model are iteratively updated based on trajectory data, including: calculating the state value at each time step based on trajectory data; calculating the temporal differential residual based on the state value reward signal at each time step; recursively calculating the advantage value based on the temporal differential residual; calculating the probability ratio of the updated policy distribution to the old policy distribution on actions; calculating the truncation loss based on the advantage value and the probability ratio; obtaining the mean square loss of the value network prediction error and the policy entropy regularization term of the pursuit agent model; calculating the total loss based on the truncation loss, the mean square loss of the value network prediction error, and the policy entropy regularization term; and updating the policy network parameters of the pursuit agent model based on the total loss.

[0011] Based on the aforementioned technical means, this embodiment calculates state value, temporal difference residual, and advantage value using trajectory data. It then constructs a truncated loss by combining the probability ratio of the old and new policies, and integrates the mean square loss of the value network with the policy entropy regularization term to obtain the total loss. This is used to update the policy network parameters. Therefore, while ensuring the stability of policy updates and avoiding training oscillations caused by large updates, it maintains policy exploration, improves policy learning efficiency and convergence reliability, and enables the pursuit agent model to more stably optimize its policy in dynamic games.

[0012] Optionally, updating at least one of the historical pursuit strategies and Elo ratings in the strategy pool based on the training results of the current training phase includes: if the training results of the current training phase indicate that a preset league evaluation interval has been reached, then triggering a full-pool league mechanism, which includes: controlling the drone model to engage in multiple rounds of combat with the historical pursuit strategies in the strategy pool, and updating the Elo ratings of each strategy in the strategy pool based on the results of the combat; if the training results of the current training phase indicate that a preset entry interval has been reached, then triggering a strategy pool update, which includes: storing the latest pursuit strategy of the drone model into the strategy pool, and removing the redundant strategy with the lowest Elo rating when the strategy pool is overloaded.

[0013] Based on the aforementioned technical means, this application embodiment triggers the full-pool league mechanism and strategy pool update by setting league evaluation intervals and pool entry intervals, respectively: During league evaluation, multiple rounds of battles are conducted between the drone model and historical pursuit strategies in the strategy pool to dynamically update the Elo rating of each strategy, thereby achieving dynamic quantification of strategy strength; when the pool entry condition is met, the latest strategy is stored in the strategy pool, and redundant strategies with the lowest Elo rating are removed when the pool is overloaded, maintaining the diversity and effectiveness of the strategy pool. This provides a foundation for selecting high-quality opponents suitable for the current training stage in priority virtual self-play, further improving training stability and strategy evolution quality.

[0014] Optionally, the drone model is controlled to engage in multiple rounds of combat against historical pursuit strategies in the strategy pool, and the Elo rating of each strategy in the strategy pool is updated based on the results of the combat. This includes: traversing the historical pursuit strategies in the strategy pool, loading the historical pursuit strategies as evaluation opponents into the simulation environment in sequence, conducting several evaluation rounds for each group of adversaries, and recording the adversary results; identifying the Elo rating of the current strategy and the Elo rating of the opponent's strategy in each evaluation round of the adversary results, calculating the expected win rate of the current strategy based on the Elo rating of the current strategy and the Elo rating of the opponent's strategy; identifying the win-loss relationship in each evaluation round of the adversary results, calculating the actual score based on the win-loss relationship in each evaluation round, and updating the Elo rating of each strategy in the strategy pool based on the actual score and the expected win rate.

[0015] Based on the aforementioned technical means, this embodiment of the application iterates through the historical pursuit strategies in the strategy pool and conducts multiple rounds of adversarial competition, recording the adversarial results. It then calculates the expected win rate by combining the Elo ratings of the current strategy and the opponent's strategy, determines the actual score based on the win-loss relationship, and finally dynamically updates the Elo rating of each strategy based on the actual score and the expected win rate. This achieves a precise quantitative assessment of the strength of historical pursuit strategies, enabling the Elo rating to truly reflect the game-playing ability of each strategy. This provides a reliable basis for selecting opponents in subsequent priority virtual self-play, further improving the targeting of training and the effectiveness of strategy evolution.

[0016] Optionally, after iteratively updating the policy network parameters of the pursuit agent model based on trajectory data, a loop control and opponent replacement mechanism is executed. The loop control and opponent replacement mechanism includes: performing a modulo operation on the number of training rounds and a preset opponent update interval threshold; if the number of training rounds meets the opponent update interval threshold, it is determined that the opponent strategy needs to be changed, the sampling probability is recalculated based on the latest Elo rating in the policy pool, and a new historical pursuit strategy is randomly selected according to the sampling probability and loaded into the self-play pursuit environment; if the number of training rounds does not meet the opponent update interval threshold, the opponent strategy parameters in the self-play pursuit environment are kept unchanged, and distributed game data collection is performed for multiple rounds using the current opponent strategy until the next parameter update condition is met; in each loop, it is synchronously checked whether the preset training termination condition is met. If the termination condition is met, the training loop is stopped, the currently trained policy network parameters are extracted, and the policy network parameters are solidified into the pursuit agent model.

[0017] Based on the aforementioned technical means, this embodiment of the application achieves periodic dynamic replacement of opponent strategies through modulo operation of the number of training rounds and the opponent update interval threshold. It also recalculates the sampling probability based on the latest Elo rating and extracts new historical pursuit strategies as opponents. If the replacement condition is not met, the current opponent is retained, and distributed game data is continuously collected until the parameters are updated. Simultaneously, the training termination condition is checked in the loop, and the strategy network parameters are solidified when the condition is met. This ensures that the agent continuously faces suitable and diverse opponents during training, avoiding overfitting caused by a single opponent over a long period, improving the richness of game data and the comprehensiveness of strategy learning, and guaranteeing efficient and stable convergence of the training process.

[0018] The second aspect of this application provides an air combat strategy evolution device based on priority virtual self-game, comprising: a construction module for constructing a self-game pursuit and escape environment and a strategy pool, wherein the strategy pool stores initial seed strategies obtained by pre-training with air combat expert experience and historical adversarial data, historical pursuit and escape strategies generated during training, and Elo rating scores corresponding to each strategy; a collection module for selecting opponent strategies in the current training phase from the strategy pool based on priority virtual self-game, training a UAV model to interact with the opponent strategies in the self-game pursuit and escape environment, performing dual-track inference to collect trajectory data of the UAV model, and generating the current pursuit and escape strategy using the pursuit and escape agent model of the UAV model during the interaction process; and a generation module for iteratively updating the strategy network parameters of the pursuit and escape agent model according to the trajectory data, updating at least one of the historical pursuit and escape strategies and Elo rating scores in the strategy pool at preset intervals, removing redundant strategies with the lowest Elo rating scores when the pool is overloaded, and generating the current pursuit and escape strategy of the UAV using the iteratively updated pursuit and escape agent model.

[0019] Optionally, the acquisition module is further configured to: traverse the policy pool, construct a score set based on the Elo ratings of all extracted historical escape strategies; calculate the non-normalized weight of each historical escape strategy based on the score set, calculate the normalized probability of the corresponding historical escape strategy based on the non-normalized weight, generate a probability distribution based on the normalized probability of each historical escape strategy; perform discrete random sampling from the policy pool based on the probability distribution, determine the opponent policy index for the current training phase, determine the opponent policy for the current training phase based on the opponent policy index, and freeze the network parameters of the opponent policy.

[0020] Optionally, the acquisition module is further used to: divide the self-game pursuit environment into multiple parallel inference threads; load the adversary policy into the adversary agent model of each inference thread, and perform dual-track inference on the UAV model and the adversary agent model through each inference thread. On the training side, the UAV model receives the current situation observation vector, calculates the action probability distribution through forward propagation of the policy network with gradients, and performs random sampling to maintain exploration. On the adversary side, the adversary agent model receives the situation observation vector from the adversary's perspective, performs forward inference through the policy network with frozen parameters, and outputs action commands. The action commands of both parties are input into the flight simulation engine in the self-game pursuit environment, and the flight simulation engine calculates the flight state, relative geometric relationship, and payload attack judgment result at the next moment.

[0021] Optionally, the generation module is further configured to: calculate the state value at each time step based on the trajectory data; calculate the temporal differential residual based on the state value reward signal at each time step; recursively calculate the advantage value based on the temporal differential residual; calculate the probability ratio of the updated policy distribution to the old policy distribution on actions; calculate the truncation loss based on the advantage value and the probability ratio; obtain the mean square loss of the value network prediction error and the policy entropy regularization term of the pursuit agent model; calculate the total loss based on the truncation loss, the mean square loss of the value network prediction error, and the policy entropy regularization term; and update the policy network parameters of the pursuit agent model based on the total loss.

[0022] Optionally, the generation module is further configured to: trigger a full-pool league mechanism if the training results of the current training phase indicate that a preset league evaluation interval has been reached; the full-pool league mechanism includes: controlling the drone model to engage in multiple rounds of combat with historical pursuit strategies in the strategy pool, and updating the Elo rating of each strategy in the strategy pool based on the results of the combat; if the training results of the current training phase indicate that a preset entry interval has been reached, trigger a strategy pool update; the strategy pool update includes: storing the latest pursuit strategy of the drone model into the strategy pool, and removing the redundant strategy with the lowest Elo rating when the strategy pool is overloaded.

[0023] Optionally, the generation module is further configured to: traverse the historical pursuit strategies in the strategy pool, load the historical pursuit strategies as evaluation opponents into the simulation environment in sequence, conduct several evaluation rounds for each group of adversaries, and record the adversary results; identify the Elo rating of the current strategy and the Elo rating of the opponent's strategy in each evaluation round of the adversary results, calculate the expected win rate of the current strategy based on the Elo rating of the current strategy and the Elo rating of the opponent's strategy; identify the win-loss relationship of each evaluation round in the adversary results, calculate the actual score based on the win-loss relationship of each evaluation round, and update the Elo rating of each strategy in the strategy pool based on the actual score and the expected win rate.

[0024] Optionally, it also includes: a determination module, used to execute a loop control and opponent replacement mechanism after iteratively updating the policy network parameters of the pursuit agent model based on trajectory data. The loop control and opponent replacement mechanism includes: performing a modulo operation on the number of training rounds and a preset opponent update interval threshold; if the number of training rounds meets the opponent update interval threshold, it is determined that the opponent strategy needs to be changed, the sampling probability is recalculated based on the latest Elo rating in the policy pool, and a new historical pursuit strategy is randomly selected according to the sampling probability and loaded into the self-play pursuit environment; if the number of training rounds does not meet the opponent update interval threshold, the opponent strategy parameters in the self-play pursuit environment are kept unchanged, and distributed game data collection is performed for multiple rounds using the current opponent strategy until the next parameter update condition is met; in each loop, it is synchronously checked whether the preset training termination condition is met. If the termination condition is met, the training loop is stopped, the currently trained policy network parameters are extracted, and the policy network parameters are solidified into the pursuit agent model.

[0025] A third aspect of this application provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor. The processor executes the program to implement the air combat strategy evolution method based on priority virtual self-game as described in the above embodiments.

[0026] A fourth aspect of this application provides a computer-readable storage medium having a computer program stored thereon, which is executed by a processor to implement the air combat strategy evolution method based on priority virtual self-game as described in the above embodiments.

[0027] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0028] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0029] Figure 1 This is a flowchart of an air combat strategy evolution method based on priority virtual self-game, according to an embodiment of this application. Figure 2 This is a strategy network structure diagram provided according to an embodiment of this application; Figure 3 This is a value network structure diagram provided according to an embodiment of this application; Figure 4 This is a flowchart illustrating an air combat strategy evolution method based on priority virtual self-game, according to an embodiment of this application. Figure 5This is a schematic diagram of the average round reward convergence curve during the training process of the agent model provided according to the embodiments of this application; Figure 6 This is a schematic diagram of the evolution curve of the Elo score of the agent model provided in the embodiments of this application during the training process; Figure 7 This is a schematic diagram of the win rate matrix cross-validation during the training process of the agent model provided in the embodiments of this application; Figure 8 This is a block diagram illustrating an air combat strategy evolution device based on priority virtual self-game, according to an embodiment of this application. Figure 9 This is a schematic diagram of the electronic structure provided according to an embodiment of this application. Detailed Implementation

[0030] The embodiments of this application are described in detail below. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0031] Currently, significant progress has been made in intelligent drone pursuit and escape game decision-making technology based on deep reinforcement learning. By constructing a high-fidelity simulation environment, the agent can continuously learn end-to-end strategies from situational awareness to maneuver decision-making through trial and error in interaction with the environment. However, in drone pursuit and escape game scenarios, due to the high uncertainty and nonlinearity of the environmental situation, rule-based or expert system-based methods often cannot cover all possible game states. Therefore, reinforcement learning methods with autonomous learning and evolution capabilities have become a key approach to improve the intelligence level of drone pursuit and escape game. Related technologies have the following shortcomings when using reinforcement learning to train intelligent agents for drone pursuit and escape game, especially for high-dimensional game tasks: First, training based on fixed expert strategies leads to "overfitting" and capability limitations. Many studies usually set the opponent of the agent as a fixed expert system built on a specific behavior tree or state machine. Although it can provide stable guidance signals in the early stage of training, the agent tends to learn "speculative strategies" that target the loopholes in the specific rules, rather than general chasing and escaping game capabilities. Once the parameters or logic of the opponent change slightly, the performance of the agent will drop significantly. Moreover, the capability limit of the fixed expert strategy is the capability limit of the agent, and it is impossible to evolve a better strategy that surpasses the cognition of human experts through training. Secondly, naive self-games face the problems of "catastrophic forgetting" and "cyclical restraint". To break through the limitations of expert strategies, the introduction of self-game mechanisms has become an inevitable choice. However, the simple "self-play", that is, the agent only plays against the latest version of itself, is prone to training instability. Since the chase game is usually non-transitive, in the process of the agent continuously optimizing the current strategy to defeat the previous generation strategy, it often loses the ability to defend against the early strategy, resulting in "catastrophic forgetting". This causes the training curve to oscillate violently, and the model is difficult to converge to the Nash equilibrium point. Third, the opponent selection mechanism is singular and inefficient. To solve the forgetting problem, it is usually necessary to maintain a policy pool containing historical policies. However, how to select opponents from the huge policy pool in each round of training becomes a new problem. If random uniform sampling is used, the training efficiency is extremely low because a large number of early weak policies cannot provide effective gradient information for the current agent. If only the strongest opponent is selected, it is easy to lead to sparse rewards and dampen the agent's exploration enthusiasm. Related technologies lack a dynamic mechanism that can quantitatively evaluate the strength of historical policies and adaptively select appropriate challenging opponents according to the current agent level.

[0032] In summary, the relevant technologies have shortcomings in terms of universality, stability, efficiency, and strategy evolution capabilities in training drone-based pursuit and escape game agents. They are difficult to adapt to unknown dynamic game environments and cannot efficiently train drone pursuit and escape strategies with strong generalization capabilities and the ability to evolve autonomously and stably.

[0033] The following describes, with reference to the accompanying drawings, a method, apparatus, electronic device, and storage medium for UAV strategy evolution based on priority-fictitious self-play, according to embodiments of this application. Addressing the problems mentioned in the background art, such as overfitting, poor generalization ability, and susceptibility to cyclical restraint and catastrophic forgetting in deep reinforcement learning under fixed-rule opponents, as well as the inefficient and unsuitable opponent selection mechanism, this application provides an air combat strategy evolution method based on priority-fictitious self-play. This method establishes a strategy pool containing historical strategy versions and uses the Elo (Elo Rating System) to quantitatively evaluate the strength of each strategy. During training, a probability distribution is constructed based on the PFSP (Prioritized Fictitious Self-Play) algorithm, prioritizing historical strategies that provide effective gradients and prevent forgetting as training partners. Specifically, this method constructs a virtual opponent database with historical memory through a policy pool, accurately quantifies and evaluates the game-playing ability of strategies using the Elo rating system, and constructs a dynamically adaptable opponent probability distribution using the PFSP sampling algorithm. This achieves a shift from blind confrontation to precise lesson learning, effectively ensuring the monotonic increase in the agent's game-playing ability and enhanced robustness during long-term evolution. In its implementation, a policy pool is constructed in a six-degree-of-freedom high-fidelity simulation environment to store diverse historical strategies. An evaluation system based on Elo ratings is established, and the ratings of each strategy are dynamically updated through periodic full-pool leagues. The strategy is iterated using PPO (Proximal Policy Optimization) and GRU (Gated Recurrent Unit) algorithms, and optimized new strategies are periodically injected into the policy pool. Ultimately, this achieves continuous evolution and enhanced robustness of the agent in unknown adversarial environments.

[0034] Specifically, Figure 1 This is a flowchart illustrating an air combat strategy evolution method based on priority virtual self-game, provided as an embodiment of this application.

[0035] like Figure 1 As shown, this air combat strategy evolution method based on priority virtual self-game includes the following steps: In step S101, a self-playing pursuit environment and a strategy pool are constructed. The strategy pool stores the initial seed strategy obtained by pre-training with the experience of air combat experts and historical confrontation data, the historical pursuit strategies generated during the training process, and the Elo rating score corresponding to each strategy.

[0036] It is understood that the embodiments of this application construct a self-playing chase environment and store historical chase strategies and corresponding Elo ratings in a strategy pool, providing basic support for subsequent dynamic selection of suitable opponents and the realization of iterative evolution of strategies. This alleviates problems such as strategy overfitting, insufficient generalization ability, easy to fall into cyclical restraint and catastrophic forgetting, and inefficient opponent selection in related technologies, and lays the foundation for stable training and strategy evolution of intelligent agents in unknown dynamic game environments.

[0037] It should be noted that the initial seed strategy refers to the initial input strategy in the strategy pool, which is pre-trained based on classic maneuver rules, offensive and defensive tactics, and historical confrontation data summarized by experts in the field of air combat. It has basic air combat capabilities and can overcome the inefficiency problem of cold start in the early stage of training. The historical pursuit strategy refers to the pursuit strategy version generated iteratively during the training process and verified by game theory. It includes the strategy network parameters and corresponding game performance characteristics of different training stages. The Elo rating is a scoring system used to quantitatively evaluate the game ability of each pursuit strategy. The initial value is preset to a unified benchmark and is dynamically updated through the results of multiple rounds of combat. The score directly reflects the relative strength of the strategy and provides a quantitative basis for the selection of opponent strategies.

[0038] Specifically, the self-play pursuit environment is constructed based on a six-degree-of-freedom nonlinear dynamic model, using JSBSim (an open-source flight dynamics simulation engine) as the underlying physics engine. It includes an aerodynamics solution module and a load guidance solution module, and its motion state equations are as follows: , in, Vacuum speed, For the angle of attack, Sideslip angle, These are the airframe roll rate, pitch rate, and yaw rate, respectively. These represent the thrust components of the engine in three directions along the engine's axis. For aerodynamic lift, For aerodynamic drag, For lateral aerodynamic forces, For the quality of drones, It is the acceleration due to gravity. For the inclination angle of the flight path, The tilt angle is defined; simultaneously, the policy network and value network of the training agent are initialized. The policy network is used to output maneuver commands, and the value network is used to evaluate the value of the current game situation.

[0039] The strategy pool manages data using a key-value pair structure or a queue structure. It is a limited-capacity queue. Initially, it stores untrained initial strategies or preset baseline rule strategies as seed strategies. The initial Elo rating of all strategies is set to a preset baseline value. In this embodiment, the initial value is 1000 points. Each historical pursuit strategy stored during subsequent training corresponds to a unique Elo rating. The maximum capacity of the strategy pool can be preset to 30, which is used to dynamically maintain the diverse historical strategies and their strength quantification scores generated during training, providing a data foundation for subsequent steps such as priority virtual self-play sampling and full-pool league evaluation.

[0040] In step S102, the opponent strategy for the current training phase is selected from the strategy pool based on priority virtual self-game. The drone model is trained to interact with the opponent strategy in the self-game pursuit environment. The trajectory data of the drone model is collected by dual-track inference. During the interaction, the pursuit agent model of the drone model is used to generate the current pursuit strategy.

[0041] It is understood that the embodiments of this application select opponent strategies that are suitable for the current training stage from the strategy pool based on priority virtual self-game, and interact with the drone model in a self-game chase environment. The chase agent model generates the current chase strategy and collects trajectory data, thereby achieving dynamic and adaptive opponent selection. This avoids overfitting and insufficient generalization ability caused by fixed opponents. At the same time, it alleviates the problems of cyclical restraint and catastrophic forgetting by interacting with diverse historical strategies, and provides effective and rich training data for subsequent policy network parameter updates, thereby improving training efficiency and policy learning quality.

[0042] Specifically, firstly, opponent strategies are selected from the strategy pool based on a priority virtual self-game algorithm. Then, the current Elo ratings of all historical pursuit strategies are extracted from the strategy pool to form a score set. A weighting function based on the Elo scores is introduced to calculate the non-normalized weight of each historical strategy. The weighting function is configured to give higher weights to strategies with higher scores to prioritize the selection of challenging opponents. Finally, the Softmax function is used to convert the weights into a probability distribution with a sum of 1. The probability calculation formula is as follows: ,in, For the first Elo ratings for historical fugitive repatriation strategies. For the scoring weighting function, this embodiment can use a linear mapping or a function based on the current agent's Elo score. To evaluate, To adjust the temperature coefficient for the degree of exploration, The current strategy pool contains the total number of historical pursuit strategies. Then, based on this probability distribution, discrete random sampling is performed to determine the opponent strategy index. The network parameters of this opponent strategy are frozen and deployed in the opponent control channel of the game environment. After selecting the opponent strategies, the self-play pursuit environment is divided into multiple parallel inference threads. The opponent strategies are loaded into the opponent agent model of each inference thread, and dual-track inference is performed.

[0043] The structures of policy networks and value networks are as follows: Figure 2 and Figure 3 As shown. The UAV model on the training side receives the current situation observation vector, which is: ,in

[0044] The flight state consists of 9 dimensions, including flight altitude. Aircraft attitude angle roll , up and down The sine and cosine values, and the three-axis velocity components under the mechanical system. Real airspeed , The observations of the target intelligent agent are in 6 dimensions, including relative approach velocity. Relative height difference Front view Target azimuth Relative distance and left and right relative positions (Left side is -1, right side is 1); UAV self-payload observation is

[0045]

[0046] A total of 62 dimensions, of which This represents the number of our effective payloads. For the target number of payloads, each payload has 5 states, including the first... Approach speed of each payload relative to the UAV model Front view Azimuth and relative distance and left and right relative positions The maximum number of payloads for both sides is 6; if the number is less than 6, zero-padding is used to fill the gaps. The observed data is processed using a 2-layer fully connected layer with 128 hidden neurons, followed by the ReLU activation function, expressed as follows:

[0047] Layer normalization is performed on the features. Specifically, for each feature, normalization and affine transformation are performed. The specific calculation method is as follows:

[0048] in The mean of the input data. To determine the variance, the input data is first normalized to a distribution with a mean of 0 and a standard deviation of 1, and then a learnable parameter is used. and An affine transformation is performed to restore the expressive power of the input data. After layer normalization, a gated recurrent unit (GPU) network (128 hidden layer neurons) is used to process the observed input, restoring the hidden state from the previous time step. Compared with the current state The process involves fusion and calculation to arrive at the final result. and new implicit states The policy network outputs an action probability distribution and performs random sampling to maintain exploratory behavior, with actions drawn from a predefined set of maneuvers. Choose from, among which To track actions, For attack actions, For escape actions, the adversary agent model receives situational observation vectors from its own perspective, performs forward inference through a parameter-frozen policy network, and outputs action commands. No computational graph is constructed to reduce memory usage and computational overhead. The value network shares the same structure as the policy network; the only difference is that the final output is an estimate of the value function of the current state.

[0049] Both parties input their action commands into the six-degree-of-freedom flight simulation engine, which calculates the flight state, relative geometry, and payload attack determination results for the next moment. Simultaneously, it collects quadruple data consisting of the current state, executed actions, environmental reward value, and termination flag bit at each time step, packages it, and stores it in a shared experience replay buffer, completing trajectory data acquisition. The environmental reward value is calculated using a composite reward function. , in, Rewards are based on the outcome of the game: a win earns a positive reward, a loss earns a negative reward, and a draw earns zero reward. The reward is calculated based on the front-view perspective; the smaller the deviation, the higher the reward. A reward is given for flight safety; a positive reward is given for no collision, and a negative reward is given for a collision. Load release rewards are given based on the following: reasonable load release is rewarded with a positive reward, and ineffective load release is rewarded with a negative reward. The target payload threat reward is calculated based on the threat level of the adversary's payload; the lower the threat, the higher the reward.

[0050] The reward for the front-view perspective is: ,in This is the angle between the direction my aircraft's nose points and the line connecting the target.

[0051] Flight safety rewards are: , in, For flight altitude, The downward vertical velocity, This is the indicator function. The load release reward is: .

[0052] The reward for unlocking the advantage is: , Release quantity reward This is used to constrain payload consumption, encourage the launch of the first payload, penalize continuous releases, and penalize attacks from optimal attack positions. However, this behavior of missing an opportunity can be specifically expressed as:

[0053] in This represents the current cumulative number of payloads launched.

[0054] The target payload threat reward is: ,in, For the first The relative approach velocity of the target loads This is a truncation function.

[0055] In this embodiment, the process of selecting opponent strategies for the current training phase from the strategy pool based on priority virtual self-game includes: traversing the strategy pool and constructing a score set based on the Elo ratings of all extracted historical pursuit strategies; calculating the non-normalized weight of each historical pursuit strategy based on the score set, calculating the normalized probability of the corresponding historical pursuit strategy based on the non-normalized weight, generating a probability distribution based on the normalized probability of each historical pursuit strategy; performing discrete random sampling from the strategy pool based on the probability distribution to determine the opponent strategy index for the current training phase, determining the opponent strategy for the current training phase based on the opponent strategy index, and freezing the network parameters of the opponent strategy.

[0056] It is understood that the embodiments of this application construct a scoring set based on Elo ratings, calculate the normalized probability of each historical pursuit strategy and form a probability distribution, and then determine the opponent strategy by discrete random sampling according to the probability distribution, thereby realizing dynamic and adaptive selection of opponent strategies; at the same time, the network parameters of the opponent strategy are frozen to ensure the stability of the opponent strategy during training. Thus, the current agent can always play against opponents of a suitable level, avoiding both the inefficiency of training caused by random sampling and the sparse rewards caused by a single strong opponent, thereby improving training efficiency and stability and enhancing the generalization ability of the agent's strategy.

[0057] Specifically, firstly, all historical escape strategies in the strategy pool are traversed, and the Elo rating corresponding to each historical escape strategy is extracted and integrated to form a complete rating set: , in, This represents the total number of strategies currently in the strategy pool. The corresponding current Elo rating is then calculated. Subsequently, the non-normalized weights for each historical pursuit strategy are calculated based on this rating set, using a weighting function based on the Elo score. The calculation is then performed. Next, normalized probabilities are calculated based on the non-normalized weights, and then normalized using the Softmax function to obtain a probability distribution that sums to 1. The specific calculation formula is as follows: ,in, This is a temperature coefficient used to adjust the level of exploration. When sampling tends to be uniform, The time tends to select only the optimal strategy, and the embodiments of this application set To balance exploration and exploitation, a discrete random sampling operation is then performed based on this probability distribution to determine the opponent's policy index for the current training phase using the following method. : Based on this index, the corresponding historical pursuit strategy is extracted from the strategy pool as the current opponent. Finally, after loading the network parameters of the opponent's strategy into the opponent agent model, its network parameters are frozen, the opponent agent is set to inference mode, the Dropout layer is turned off, and the gradient computation graph is truncated (requires_grad=False). This ensures that in subsequent game interactions, only the parameters of the training agent (drone model) are updated, while the opponent's strategy parameters remain unchanged. This avoids the impact of changes in the opponent's strategy on training stability and provides a stable adversarial environment for the training process.

[0058] In this embodiment, training a drone model to interact with an opponent's strategy in a self-playing pursuit environment includes: dividing the self-playing pursuit environment into multiple parallel inference threads; loading the opponent's strategy into the opponent's agent model in each inference thread; performing dual-track inference on the drone model and the opponent's agent model through each inference thread; wherein, on the training side, the drone model receives the current situation observation vector, calculates the action probability distribution through forward propagation of a policy network with gradients, and performs random sampling to maintain exploration; on the opponent side, the opponent's agent model receives the situation observation vector from the opponent's perspective, performs forward inference through a policy network with frozen parameters, and outputs action commands; inputting the action commands of both parties into a flight simulation engine in the self-playing pursuit environment, and solving the flight state, relative geometric relationship, and payload attack judgment result at the next moment through the flight simulation engine.

[0059] It is understood that this embodiment of the application achieves dual-track asynchronous inference on the training and adversary sides by dividing the self-playing chase environment into multiple parallel inference threads. The training side uses a gradient-based policy network for forward propagation and random sampling to ensure exploration and the effectiveness of policy updates. The adversary side uses a policy network with frozen parameters for forward inference to ensure adversarial stability. Finally, a flight simulation engine is used to uniformly calculate the state and adversarial results. Thus, while ensuring training efficiency and parallelism, the realism of the game interaction and the effectiveness of policy learning are improved, enabling the agent to complete the chase strategy learning in a more stable and efficient parallel environment.

[0060] Specifically, the self-game chasing environment is first parallelized, and initialization is performed. Multiple parallel simulation processes or threads are used to achieve synchronous adversarial scenarios in multiple environments, improving data acquisition throughput and enabling the implementation of adversary strategies selected in the previous steps. Parameters are distributed and loaded into the adversary agent model of each thread, constructing the following parallel environment: ,in, For the current policy network, These are the current training parameters (drone pursuit agent model parameters) that need to be updated. For the first sampled from the policy pool The policy network parameters of each historical strategy are used to achieve a "one-to-many" training mode, enabling the agent to fight against opponents of various styles simultaneously to improve generalization.

[0061] Subsequently, a dual-track inference mechanism is executed in each inference thread, within the same simulation time step. On the training side, the drone model is used as our intelligent agent in exploratory mode, receiving current situation observation vectors. The observation vector consists of 9 dimensions of the agent's flight state, 6 dimensions of the target's relative motion state, and up to 62 dimensions of the target payload state. Invalid payload states are zero-padded to ensure a fixed input dimension. The agent calculates the discrete action probability distribution through forward propagation of a policy network with gradients, and obtains actions by random sampling based on this distribution. Simultaneously record the logarithmic probability of the action. and valuation This is used for gradient calculation in subsequent policy network parameter updates.

[0062] The adversary agent on the opposing side receives the situation observation vector from its perspective. Forward inference is performed through a policy network with frozen parameters. At this time, the network is in eval mode, with Dropout (random deactivation) and BatchNorm (batch normalization) updates disabled, and directly outputs deterministic actions or random actions without gradients. This process only performs forward propagation and does not construct a computational graph, effectively reducing memory usage and computational overhead. Finally, the action commands of both parties are synchronously input into the six-degree-of-freedom flight simulation engine of the self-playing chase environment. The engine calculates the flight state at the next moment, the relative geometric relationship between the two parties, and the payload attack judgment result, providing a foundation for subsequent reward calculation and trajectory data collection, while supporting the continuous game interaction.

[0063] In step S103, the policy network parameters of the pursuit agent model are iteratively updated based on the trajectory data. At least one of the historical pursuit policies and Elo ratings in the policy pool is updated at a preset interval. When the capacity is exceeded, the redundant policy with the lowest Elo rating is removed. The current pursuit policy of the UAV is generated using the pursuit agent model that has been iteratively updated.

[0064] It is understood that the embodiments of this application iteratively update the policy network parameters of the pursuit agent model through trajectory data, update the historical pursuit policies and Elo rating scores in the policy pool according to the training results, and then use the updated pursuit agent model to generate the current pursuit policy of the drone. In this way, the policy is continuously optimized during training, the effectiveness of the policy pool is dynamically maintained, and policy overfitting, cyclical restraint and catastrophic forgetting are avoided. This improves the generalization ability and robustness of the policy in unknown dynamic game environment, and provides key support for the continuous evolution of the agent policy.

[0065] It should be noted that the preset interval refers to the pre-set trigger conditions for updating the strategy pool, including the league evaluation interval (used to update the Elo rating) and the pool entry interval (used to store new strategies). The interval length can be flexibly configured according to training needs. Redundant strategies refer to historical chasing strategies in the strategy pool that exceed the preset generation capacity, or whose Elo rating is too low and whose game ability is no longer suitable for the current training stage. Removing them can maintain the efficiency and targeting of the strategy pool.

[0066] Specifically, firstly, trajectory data, including quadruples, is collected and stored in the experience playback buffer based on step S102. And metadata, the experience replay buffer is represented as: ,in, For single trajectory data, The total number of trajectories stored in the buffer. for The situation observation vector at time t, for Actions performed at all times for The environmental reward value at each time step. Based on this trajectory data, the policy network parameters of the pursuit agent model are iteratively updated. The update is completed using a near-end policy optimization algorithm. Specifically, the state value, temporal difference residual, and advantage value at each time step are calculated to construct the truncation loss, value network mean square loss, and policy entropy regularization term. After fusing them to obtain the total loss, the parameters are backpropagated to update them.

[0067] After the parameters are updated, the updated pursuit agent model receives the situational observation vector of the current self-playing pursuit environment, calculates the action probability distribution through forward propagation of the policy network, and obtains maneuver actions by performing random sampling or deterministic output, generating the current pursuit strategy for the UAV. Simultaneously, based on the training results of the current training phase, it determines whether the preset league evaluation interval or pool entry interval has been reached: if the league evaluation interval is reached, the full-pool league mechanism is triggered, traversing all historical pursuit strategies in the policy pool and loading them sequentially as evaluation opponents into the simulation environment. Each group of opponents undergoes several evaluation rounds, recording the results, calculating the expected win rate and actual score of the current strategy, and dynamically updating the Elo rating of each strategy in the policy pool according to the Elo update formula; if the pool entry interval is reached, the policy pool is updated, and the latest pursuit strategy of the currently trained UAV model is added. Store the latest Elo rating of the current agent. The initial Elo score for this strategy is stored in the strategy pool. If the current capacity of the strategy pool has not reached the preset limit, the pooling operation is performed. If the policy pool has reached the preset maximum capacity The algorithm iterates through the Elo ratings of all historical policies in the policy pool, removing redundant policies with the lowest Elo ratings. If policies have the same Elo rating, the oldest policy is removed according to a first-in, first-out (FIFO) principle. This process ensures the survival of the fittest in the policy pool, maintaining its diversity and high quality, and providing reliable support for subsequent opponent selection. Furthermore, after parameter updates, a loop control and opponent replacement mechanism are executed to ensure efficient and stable convergence during training. The entry into the pool is triggered by modular arithmetic. ,in, This represents the current cumulative number of training rounds. This is the preset interval for updating the pool.

[0068] In this embodiment, the policy network parameters of the pursuit agent model are iteratively updated based on trajectory data, including: calculating the state value at each time step based on trajectory data; calculating the temporal differential residual based on the state value reward signal at each time step; recursively calculating the advantage value based on the temporal differential residual; calculating the probability ratio of the updated policy distribution to the old policy distribution in actions; calculating the truncation loss based on the advantage value and the probability ratio; obtaining the mean square loss of the value network prediction error and the policy entropy regularization term of the pursuit agent model; calculating the total loss based on the truncation loss, the mean square loss of the value network prediction error, and the policy entropy regularization term; and updating the policy network parameters of the pursuit agent model based on the total loss. The hyperparameters of the PPO algorithm are shown in Table 1, and the hyperparameters related to self-game are shown in Table 2.

[0069] Table 1

[0070] Table 2

[0071] Understandably, this embodiment calculates state value, temporal difference residual, and advantage value using trajectory data, constructs a truncated loss by combining the probability ratio of the old and new policies, and integrates the mean square loss of the value network and the policy entropy regularization term to obtain the total loss, thereby updating the policy network parameters. Thus, while ensuring the stability of policy updates and avoiding training oscillations caused by large updates, it maintains policy exploration, improves policy learning efficiency and convergence reliability, and enables the pursuit agent model to optimize policies more stably in dynamic games.

[0072] Specifically, the state value at each time step is first calculated using trajectory data, and then analyzed through a commentator network of the pursuit agent model. The state at each time step in the trajectory Perform value prediction to obtain state value ,in The parameters are defined for the Critic network; subsequently, the state value at each time step and the corresponding reward signal are used. (Calculated from the composite reward function), calculate the time-series difference residuals: ,in, As a discount factor, in the embodiments of this application =0.99, used to adjust the weight of future rewards; then, based on the generalized dominance estimation algorithm, the dominance value is recursively calculated according to the time-series difference residuals: ,in, The smoothing factor is used in the embodiments of this application. Used to balance bias and variance. This represents the total time step for a single trajectory.

[0073] Next, the updated policy distribution and the old policy distribution are calculated in terms of action. The probability ratio above: ,in, For the updated strategy, the Actor network parameters are: , The old strategy before the update has the following Actor network parameters: ;Calculate the cutoff loss based on the advantage value and probability ratio, and construct the cutoff objective function for near-end policy optimization: ,in, As a preset truncation hyperparameter, in the embodiments of this application This is used to limit the policy update magnitude and prevent training from crashing due to excessively large update steps. This represents the expectation over all time steps. Simultaneously, the mean squared loss of the value network prediction error of the pursuit agent model is obtained: ,in, For target value, The parameters of the Critic network before the update; calculate the policy entropy regularization term: This is used to encourage agents to explore and prevent premature convergence to local optima. For all possible actions.

[0074] Finally, the total loss is calculated based on the truncation loss, the mean squared loss of the value network prediction error, and the policy entropy regularization term: ,in, As a value loss weighting coefficient, in the embodiments of this application =0.1, since the loss function of the value network is often on the order of magnitude larger than that of the policy network, therefore, is chosen. This makes the two orders of magnitude comparable, which facilitates optimization; The entropy regularization coefficient is used because the policy entropy is a regularization term and cannot account for too large a proportion. Therefore, in the embodiments of this application... =0.001; The Adam optimizer is used, and the gradient of the total loss function is calculated using the backpropagation algorithm. And update the policy network parameters (Actor network) and value network parameters (Critic network) of the pursuing agent model according to the gradient, that is: , ,in, The learning rate is 3e-4. Through multiple rounds of mini-batch iterations, the parameters of the policy network are updated iteratively. In this embodiment, the underlying PPO algorithm can be replaced with SAC (Soft Actor-Critic) or TD3 (Twin Delayed Deep Deterministic Policy Gradient).

[0075] In this embodiment, after iteratively updating the policy network parameters of the pursuit agent model based on trajectory data, a loop control and opponent replacement mechanism is executed. The loop control and opponent replacement mechanism includes: performing a modulo operation on the number of training rounds and a preset opponent update interval threshold; if the number of training rounds meets the opponent update interval threshold, it is determined that the opponent strategy needs to be changed, the sampling probability is recalculated based on the latest Elo rating in the policy pool, and a new historical pursuit strategy is randomly selected according to the sampling probability and loaded into the self-play pursuit environment; if the number of training rounds does not meet the opponent update interval threshold, the opponent strategy parameters in the self-play pursuit environment are kept unchanged, and distributed game data collection is performed for multiple rounds using the current opponent strategy until the next parameter update condition is met; in each loop, it is synchronously checked whether the preset training termination condition is met. If the termination condition is met, the training loop is stopped, the currently trained policy network parameters are extracted, and the policy network parameters are solidified into the pursuit agent model.

[0076] It is understood that the embodiments of this application achieve periodic dynamic replacement of opponent strategies through modulo operations of the number of training rounds and the opponent update interval threshold, and recalculate the sampling probability based on the latest Elo rating to extract new historical pursuit strategies as opponents; if the replacement condition is not met, the current opponent is retained, and distributed game data is continuously collected until the parameters are updated. At the same time, the training termination condition is checked synchronously in the loop, and the policy network parameters are solidified when the condition is met. Thus, the agent continuously faces suitable and diverse opponents during the training process, avoiding overfitting caused by a single opponent for a long time, improving the richness of game data and the comprehensiveness of strategy learning, and ensuring efficient and stable convergence of the training process.

[0077] It should be noted that the number of training rounds refers to the total number of complete game rounds completed by the agent in the self-playing chase environment. Each complete interaction process from environment initialization to game termination is counted as one training round. This number monotonically increases with the training progress and is used to control the training progress, trigger opponent replacement, and determine training termination. The opponent update interval threshold is a pre-set round number threshold used to control the frequency of opponent strategy replacement. When the cumulative number of training rounds reaches an integer multiple of this threshold, the system determines that the opponent needs to be replaced and resamples new historical chase strategies from the strategy pool to avoid overfitting caused by the agent playing against a single opponent for a long time. The training termination condition is a pre-set set of training stop judgment rules, including the cumulative number of training rounds reaching a preset maximum value, the agent's policy Elo rating score remaining stable in the target range for multiple consecutive rounds, the policy network parameter update magnitude being less than a preset convergence threshold, and the game performance index reaching a preset target for multiple consecutive rounds. When any of these conditions are met, the training loop terminates and the final solidified chase agent model is output.

[0078] Specifically, after updating the policy network parameters of the pursuit agent model, the system first uses a monotonically increasing counter maintained in global memory. The counter retrieves the cumulative number of training rounds completed so far. It increments upon receiving the "Done" signal after each complete game simulation.

[0079] The cumulative number of training rounds is then compared with a preset opponent update interval threshold. In the embodiments of this application Perform a modulo operation to determine if the opponent's strategy needs to be changed: if the modulo result is 0, then... If the opponent strategy needs to be changed, the process jumps to the opponent sampling step. Based on the latest Elo rating of each historical pursuit strategy in the strategy pool, the priority virtual self-game algorithm is re-executed to calculate the non-uniform sampling probability distribution. The calculation process is the same as in step S102, using the Softmax function, combined with the weighting function and temperature coefficient. Discrete random sampling is performed according to the recalculated sampling probability to extract new historical pursuit strategies. The network parameters of these strategies are loaded into the parallel inference threads of the self-game pursuit environment. After freezing the parameters, the next round of training begins. If the modulo result is not 0, it is determined that there is no need to change the opponent strategy. The current opponent strategy parameters in the self-game pursuit environment remain unchanged, and the process jumps directly to the distributed game data collection step. The current opponent strategy is used to continue to collect distributed game data for multiple rounds, accumulating more trajectory data until the conditions for the next strategy network parameter update are met (such as collecting a preset number of trajectory data or completing a preset number of rounds).

[0080] During each loop, the system synchronously checks whether the preset training termination conditions are met. These conditions include reaching the maximum number of training rounds. In the embodiments of this application If any one of the following conditions is met—policy win rate convergence, policy Elo rating reaching a preset target value—the training cycle is immediately stopped, the policy network parameters (Actor and Critic network weights) of the currently trained pursuit agent model are extracted, and the policy with the highest Elo rating in the policy pool is selected: ,in, The set of all historical escape strategies in the strategy pool. For strategy The current Elo rating score is obtained; the parameters of the optimal strategy are serialized and saved as a binary model file (e.g., .pt format), and then embedded into the pursuit agent model as the final converged optimal pursuit agent model for subsequent UAV-borne real-time inference.

[0081] In this embodiment of the application, updating at least one of the historical pursuit strategies and Elo ratings in the strategy pool based on the training results of the current training phase includes: if the training results of the current training phase determine that a preset league evaluation interval has been reached, then a full-pool league mechanism is triggered. The full-pool league mechanism includes: controlling the drone model to engage in multiple rounds of combat with the historical pursuit strategies in the strategy pool, and updating the Elo ratings of each strategy in the strategy pool based on the results of the combat; if the training results of the current training phase determine that a preset entry interval has been reached, then a strategy pool update is triggered. The strategy pool update includes: storing the latest pursuit strategy of the drone model into the strategy pool, and removing the redundant strategy with the lowest Elo rating when the strategy pool is overcapacity.

[0082] It is understood that the embodiments of this application trigger the full-pool league mechanism and strategy pool update respectively by setting league evaluation intervals and pool entry intervals: during league evaluation, the Elo rating of each historical strategy is updated through multiple rounds of matches to achieve dynamic quantification of strategy strength; when the pool entry conditions are met, the latest strategy is stored and redundant strategies are removed according to capacity limits to maintain the effectiveness and timeliness of the strategy pool. Thus, the strategy pool always contains diverse opponents suitable for the current training stage, providing a reliable opponent selection basis for priority virtual self-play, further improving training stability and strategy evolution quality.

[0083] Specifically, the current training progress is monitored in real time. Based on training results such as the number of training rounds and policy updates, it is determined whether the preset league evaluation interval or pool entry interval has been reached. These two intervals can be set independently, such as a league evaluation interval of every 50 training rounds and a pool entry interval of every 10 training rounds. If the preset league evaluation interval is reached, the full-pool league mechanism is immediately triggered. All historical pursuit strategies in the strategy pool are traversed, and each historical pursuit strategy is used as an evaluation opponent in turn. This strategy is then loaded into the self-playing pursuit environment, and the currently trained drone model is controlled to engage in multiple rounds of combat with the evaluation opponent (each adversary consists of several evaluation rounds). The system records the outcome of each round of combat. After each round, it calculates the actual scores of the current drone model strategy and the strategies of each evaluated opponent based on the outcome of the combat. It then calculates the expected win rate by combining the current Elo rating scores of both sides. Finally, it dynamically updates the Elo rating scores of each historical pursuit strategy in the strategy pool according to the Elo rating score update formula, thus completing the update of the Elo rating scores in the strategy pool.

[0084] If the preset entry interval is reached, the policy pool update mechanism is triggered. First, the latest pursuit and escape policy of the current drone model (the policy after parameter updates) is stored, and the latest Elo rating of the current drone model is extracted and used as the initial rating label for the policy. Then, it is checked whether the current storage capacity of the policy pool has reached the preset maximum capacity threshold. In the embodiments of this application =30. If the maximum capacity has not been reached, the strategy and its rating tag are directly stored in the strategy pool to complete the strategy pool update; if the maximum capacity has been reached, the survival of the fittest rule is triggered, and the Elo rating scores of all historical escape strategies in the strategy pool are traversed to identify the historical strategy with the lowest Elo rating score, i.e., the index of the strategy with the lowest rating: If multiple strategies have the same lowest Elo rating, the strategy with the longest duration is removed according to the first-in, first-out (FIFO) principle. and the corresponding Elo score The replacement operation is performed to maintain a constant pool capacity. After freeing up storage space, the newly generated policy and its rating label are stored in the policy pool. This ensures that the strategy pool is always kept within its maximum capacity and contains diverse, high-quality historical strategies. Additionally, the implementation of this application allows for adjustment of the calculation parameters for the Elo rating score.

[0085] In this embodiment, the control of the drone model involves multiple rounds of combat against historical pursuit strategies in the strategy pool, and updating the Elo rating of each strategy in the strategy pool based on the win / loss results. This includes: traversing the historical pursuit strategies in the strategy pool, loading each historical pursuit strategy as an evaluation opponent into the simulation environment, conducting at least one evaluation round for each group of adversaries, and recording the adversary results; identifying the Elo rating of the current strategy and the Elo rating of the opponent's strategy in each evaluation round of the adversary results, calculating the expected win rate of the current strategy based on the Elo rating of the current strategy and the Elo rating of the opponent's strategy; identifying the win / loss relationship in each evaluation round of the adversary results, calculating the actual score based on the win / loss relationship in each evaluation round, and updating the Elo rating of each strategy in the strategy pool based on the actual score and the expected win rate.

[0086] Understandably, this application embodiment traverses the historical pursuit strategies in the strategy pool and conducts multiple rounds of adversarial competition, recording the adversarial results. It then calculates the expected win rate by combining the Elo ratings of the current strategy and the opponent's strategy, determines the actual score based on the win-loss relationship, and finally dynamically updates the Elo rating of each strategy based on the actual score and the expected win rate. This achieves a precise quantitative assessment of the strength of historical pursuit strategies, enabling the Elo rating to truly reflect the game-playing ability of each strategy. This provides a reliable basis for selecting opponents in subsequent priority virtual self-play, further improving the targeting of training and the effectiveness of strategy evolution.

[0087] Specifically, firstly, all historical pursuit strategies in the strategy pool are traversed to form a historical strategy set. Each historical pursuit strategy in the set is then used as an evaluation opponent and loaded one by one into the simulation thread of the self-playing pursuit environment, ensuring that each confrontation involves only the current drone model (current strategy) and one evaluation opponent (historical strategy). Each confrontation lasts at least one evaluation round. To ensure evaluation accuracy, multiple evaluation rounds can be set, such as... In each evaluation round, both sides engage in game interaction according to a dual-track reasoning mechanism. The flight simulation engine calculates the outcome of the confrontation, recording the win / loss relationship (win, loss, or draw) for each evaluation round throughout the process. Subsequently, the current strategy score and the opponent's strategy score for each evaluation round are identified in the confrontation results, i.e., the current Elo rating of the current drone model strategy. And the current Elo rating for assessing the opponent's strategy. Based on these two ratings, and following the logistic distribution assumption, calculate the expected win rate of the current strategy against the opponent's strategy: Where 400 is the standard scaling factor of the Elo rating system, making the difference in rating scores and the expected win rate logistically correlated; if Then the expected win rate The larger the score difference, the closer the expected win rate is to 1 (current strategy is superior) or 0 (opponent's strategy is superior).

[0088] Identify the win / loss outcome of each evaluation round, calculate the actual score of the current strategy. In a single match, a win for the current strategy earns 1 point, a draw earns 0.5 points, and a loss earns 0 points. After completing all evaluation rounds, calculate the average actual score of the current strategy. ,in, For the first The actual score of the current strategy in each evaluation round. Finally, based on the actual score and expected win rate, the Elo ratings of both players are updated using the Elo rating iterative update equation, the update formula being: , ,in, These are the updated Elo ratings for the current strategy and the opponent's strategy, respectively. K is a weight adjustment factor used to control the impact of a single evaluation on the total score. In this application, K can be set to 32. The updated Elo ratings for each historical pursuit strategy are synchronously written back to the strategy pool, overwriting the original scores, and used as the latest calculation basis for the subsequent priority virtual self-game algorithm when sampling opponents.

[0089] The air combat strategy evolution method based on priority virtual self-game proposed in this application constructs a strategy pool with Elo ratings and employs a priority virtual self-game mechanism to dynamically select opponent strategies suitable for the current training stage, avoiding training oscillations caused by overfitting to a single opponent or random opponent selection. Trajectory data is collected through game interaction to iteratively update the agent's policy network parameters and simultaneously update the strategy pool and Elo ratings, forming a closed-loop evolution process. This effectively solves the problems of unstable agent training and insufficient generalization ability, making the strategy more robust in dynamic adversarial environments. Simultaneously, it enables the autonomous evolution of UAV pursuit strategies, improving the practicality and reliability of the strategies in real-world applications.

[0090] The following will elaborate on the proposed drone pursuit strategy evolution method based on priority virtual self-game through a specific embodiment, such as... Figure 4 As shown, the specific steps are as follows: In step one, the self-play chasing environment and the strategy pool are constructed and initialized.

[0091] First, a self-playing pursuit and escape environment based on six-degree-of-freedom nonlinear dynamics is constructed, using JSBSim as the underlying physics engine. This engine includes aerodynamic and load guidance calculation modules, providing high-fidelity simulation support for subsequent pursuit and escape game scenarios. Simultaneously, a strategy pool is initialized. This pool employs a finite-capacity queue or key-value pair structure, initially storing seed strategies pre-trained using air combat expert experience and historical adversarial data. An initial Elo rating (e.g., 1000 points) is assigned to all strategies for subsequent dynamic maintenance of historical strategies and their strength quantification scores.

[0092] In step two, dynamic opponent sampling is performed based on the PFSP algorithm.

[0093] The algorithm uses a priority virtual self-game approach to select suitable opponent strategies from the strategy pool for the current training phase. Specifically, it iterates through the strategy pool to extract the current Elo rating of all historical strategies, calculates non-normalized weights using a weighting function, converts them into a probability distribution using a Softmax function, and finally determines the opponent strategy index by discrete random sampling according to this distribution. The network parameters are then frozen and deployed into the game environment.

[0094] In step three, distributed data acquisition and dual-track inference are performed.

[0095] The self-playing pursuit environment is divided into multiple parallel inference threads, continuously interacting with the simulation environment to collect training data. On the training side, the friendly UAV model receives situational observation vectors, calculates the action probability distribution through forward propagation of a policy network with gradients, and randomly samples actions. On the adversary side, the adversary agent outputs action commands through forward inference of a policy network with frozen parameters, without constructing a computation graph to reduce computational overhead. Both sides' actions are input into a six-DOF flight simulation engine, which calculates the state and adversarial outcome at the next moment, and collects quadruples of data containing state, action, reward, and termination flag, storing them in an experience replay buffer.

[0096] In step four, the policy network parameters are updated and near-end optimization is performed.

[0097] Based on trajectory data from the experience replay buffer, a near-end policy optimization algorithm is used to iteratively update the policy network parameters of the pursuit agent model. Specifically, the state value, temporal difference residual, and advantage value at each time step are calculated. A truncation loss, a value network mean squared loss, and a policy entropy regularization term are constructed, fused to obtain the total loss, and then backpropagated to update the parameters of the executor network and the critic network.

[0098] In step five, the policy pool is updated and the best and worst performers are eliminated.

[0099] When training reaches the preset entry interval, the policy pool is updated. The latest pursuit policy after training is completed is stored, and it is stored in the policy pool with the current agent's Elo rating score as the initial score label. If the policy pool has reached the preset maximum capacity, the Elo rating scores of all policies in the pool are traversed, and the redundant policy with the lowest score is removed (if the scores are the same, the oldest policy is removed according to the first-in-first-out principle), so as to maintain the diversity and high quality of the policy pool.

[0100] In step six, the entire pool league is evaluated and the Elo rating is updated.

[0101] When the training reaches the preset league evaluation interval, the full-pool league mechanism is triggered. All historical strategies in the strategy pool are traversed, and each strategy is used as an evaluation opponent to engage in multiple rounds of battles with the current agent. The win and loss results of each round are recorded, the expected win rate and actual score of the current strategy are calculated, and the Elo rating of each strategy in the strategy pool is dynamically updated according to the Elo update formula to achieve a precise quantitative evaluation of the strength of the strategies.

[0102] Finally, the loop control and termination are performed. After each parameter update, it is checked whether the cumulative number of training rounds has reached the preset opponent update interval threshold. If it has, step two is repeated to change the opponent; if not, distributed data collection continues using the current opponent. Simultaneously, in each loop, it is checked whether the training termination conditions are met (such as reaching the maximum number of training rounds, policy win rate convergence, Elo rating reaching the target value, etc.). If these conditions are met, the training loop stops, the current policy network parameters are extracted and solidified, and the final optimal pursuit agent model is output.

[0103] Figure 5 The curve depicting the fluctuation of the average round reward during training is shown. Since this embodiment employs a self-play adversarial mode, where the two players are each other's opponents, the curve reflects the net winning advantage of the current strategy relative to the opponent's strategy. As can be seen from the figure, the curve exhibits dramatic oscillations in the early and middle stages of training, corresponding to the intense offensive and defensive transitions and tactical suppression processes between the two players in the strategy space. Notably, in the later stages of training, the average reward gradually converges and stabilizes around 0 with slight oscillations. This phenomenon aligns with the Nash equilibrium property of games, indicating that a dynamic equilibrium has been reached between the agent and an opponent of equal strength. The strategy possesses strong robustness and defensive capabilities, making it difficult for the opponent to easily exploit vulnerabilities to obtain high rewards, further verifying the completeness of the final generated strategy.

[0104] Figure 6 This paper demonstrates the changing trend of the agent's Elo rating during training based on a priority virtual self-game mechanism. Although there were brief numerical fluctuations in the middle of training due to changes in the policy exploration space, the overall trend shows a significant and continuous increase in the agent's Elo score with the increase in training iterations. The smoothed curve shows that the Elo score steadily increased from approximately 1000 points in the initial stage to over 1300 points at the end of training, with a peak value approaching 1400 points. This continuous growth trend proves that the training framework proposed in this application can effectively overcome the policy cycle restraint problem in self-game, enabling the agent to achieve effective iteration of tactical strategies and significant enhancement of combat performance in continuous adversarial exercises, thus verifying the convergence and effectiveness of the algorithm.

[0105] Figure 7This diagram displays the cross-adversarial win rate matrix of the policy models at different stages of training. The horizontal axis represents the number of iterations of the historical strategy as the opponent, and the vertical axis represents the number of iterations of the current test strategy. The values ​​in the cells represent the probability that the strategy on the vertical axis defeats the strategy on the horizontal axis; darker colors indicate higher win rates. Experimental results show that although the strategy experienced some performance fluctuations in the middle of training, as training continued: Significant final convergence: The latest policy model achieved a win rate exceeding 50% against all historical versions, reaching a maximum of 80%, verifying that the algorithm can effectively overcome policy forgetting and achieve a steady improvement in combat effectiveness; Improvement relative to the initial state: Compared to the untrained initial strategy, the final strategy's win rate stabilized at around 80%, proving the effectiveness of the training framework.

[0106] In summary, the embodiments of this application have at least the following beneficial effects. (1) Breaking through the limitations of fixed rules to achieve policy generalization and high-performance policy generation. Compared with the training methods for fixed expert systems or state machines in related technologies, this application abandons the static training mode and uses a self-game mechanism to make the agents compete against each other and evolve together. This not only eliminates the ability ceiling set by artificial rules, enabling the agents to generate complex strategies (such as extreme energy maneuvers) that have not been preset by human experts, but also significantly improves the generalization ability and robustness of the strategy when facing unknown opponents, effectively solving the problem of "overfitting".

[0107] (2) Effectively overcomes the problems of "catastrophic forgetting" and "cyclic restraint" in self-games. Addressing the training oscillation problem inherent in naive self-games in related technologies, this application introduces a historical memory mechanism based on a policy pool. By forcing the agent to periodically engage with older versions of policies from the evolutionary history, it ensures that the new policy, while enhancing its offensive capabilities, does not lose its defensive capabilities against earlier basic policies. This mechanism fundamentally breaks the cyclic restraint trap similar to "rock-paper-scissors" in non-transitive games, guiding the policy to converge stably towards the Nash equilibrium point.

[0108] (3) Precise course training based on ability quantification significantly improves convergence efficiency. Compared with the problem of wasted computing power caused by random selection of opponents in related technologies, this application innovatively introduces the Elo rating score and the priority virtual self-game sampling algorithm. The system can accurately quantify the relative strength of each strategy based on the Elo score and construct a dynamic course with "moderate challenge". By prioritizing the sampling of opponents that match the current agent's strength or have specific strategic value, the training is always kept in the efficient "zone of proximal development", which greatly shortens the cultivation cycle of high-level agents.

[0109] (4) Supports parallel adversarial training with multiple styles, enhancing policy richness. The training architecture of this application supports the simultaneous loading of multiple different historical opponent policies in a single parameter update. This means that the agent can simultaneously obtain feedback from multiple policy styles (such as aggressive, conservative, and avoidant) within the same time step. This high-concurrency and diverse data input further enriches the sample distribution of the experience replay pool, enabling the trained agent to have more comprehensive offensive and defensive capabilities and avoiding policy style homogenization.

[0110] Next, referring to the accompanying drawings, an air combat strategy evolution device based on priority virtual self-game proposed according to an embodiment of this application is described.

[0111] Figure 8 This is a block diagram of an air combat strategy evolution device based on priority virtual self-game according to an embodiment of this application.

[0112] like Figure 8 As shown, the air combat strategy evolution device 80 based on priority virtual self-game includes: a construction module 801, a collection module 802, and a generation module 803.

[0113] The construction module 801 is used to construct a self-playing pursuit environment and a strategy pool. The strategy pool stores the initial seed strategy obtained by pre-training with the experience of air combat experts and historical confrontation data, the historical pursuit strategies generated during the training process, and the Elo rating score corresponding to each strategy. The acquisition module 802 is used to select the opponent strategy in the current training stage from the strategy pool based on priority virtual self-play. The drone model is trained to play against the opponent strategy in the self-playing pursuit environment, and the trajectory data of the drone model is collected by performing dual-track inference. During the interaction, the current pursuit strategy is generated using the pursuit agent model of the drone model. The generation module 803 is used to iteratively update the strategy network parameters of the pursuit agent model according to the trajectory data, update at least one of the historical pursuit strategies and Elo rating scores in the strategy pool at preset intervals, remove the redundant strategy with the lowest Elo rating score when the capacity is exceeded, and generate the current pursuit strategy of the drone using the pursuit agent model that has been iteratively updated.

[0114] In this embodiment, the acquisition module 802 is further configured to: traverse the policy pool, construct a score set based on the Elo ratings of all extracted historical escape strategies; calculate the non-normalized weight of each historical escape strategy based on the score set, calculate the normalized probability of the corresponding historical escape strategy based on the non-normalized weight, generate a probability distribution based on the normalized probability of each historical escape strategy; perform discrete random sampling from the policy pool based on the probability distribution, determine the opponent policy index for the current training phase, determine the opponent policy for the current training phase based on the opponent policy index, and freeze the network parameters of the opponent policy.

[0115] In this embodiment, the acquisition module 802 is further used to: perform simulation in a simulation environment and collect data for training. The self-game pursuit environment is divided into multiple parallel inference threads; the opponent's policy is loaded into the opponent's agent model in each inference thread, and dual-track inference is performed on the UAV model and the opponent's agent model through each inference thread. On the training side, the UAV model receives the current situation observation vector, calculates the action probability distribution through forward propagation of the policy network with gradients, and performs random sampling to maintain exploration; on the opponent side, the opponent's agent model receives the situation observation vector from the opponent's perspective, performs forward inference through the policy network with frozen parameters, and outputs action commands; the action commands of both parties are input into the flight simulation engine in the self-game pursuit environment, and the flight simulation engine calculates the flight state, relative geometric relationship, and payload attack judgment result at the next moment. Before training, the data in each dimension of the observation is divided by the maximum value and normalized to the range [0,1] to facilitate the training of the neural network.

[0116] In this embodiment, the generation module 803 is further configured to: calculate the state value at each time step based on the trajectory data; calculate the temporal differential residual based on the state value reward signal at each time step; recursively calculate the advantage value based on the temporal differential residual; calculate the probability ratio of the updated policy distribution to the old policy distribution on actions; calculate the truncation loss based on the advantage value and the probability ratio; obtain the mean square loss of the value network prediction error and the policy entropy regularization term of the pursuit agent model; calculate the total loss based on the truncation loss, the mean square loss of the value network prediction error, and the policy entropy regularization term; and update the policy network parameters of the pursuit agent model based on the total loss.

[0117] In this embodiment of the application, the generation module 803 is further configured to: if the training results of the current training phase determine that a preset league evaluation interval has been reached, then trigger a full-pool league mechanism. The full-pool league mechanism includes: controlling the drone model to engage in multiple rounds of combat with historical pursuit strategies in the strategy pool, and updating the Elo rating of each strategy in the strategy pool based on the results of the combat; if the training results of the current training phase determine that a preset entry interval has been reached, then trigger a strategy pool update. The strategy pool update includes: storing the latest pursuit strategy of the drone model into the strategy pool, and removing the redundant strategy with the lowest Elo rating when the strategy pool is overcapacity.

[0118] In this embodiment, the generation module 803 is further configured to: traverse the historical pursuit strategies in the strategy pool, load the historical pursuit strategies as evaluation opponents into the simulation environment in sequence, conduct at least one evaluation round for each group of adversaries, and record the adversary results; identify the Elo rating of the current strategy and the Elo rating of the opponent's strategy in each evaluation round of the adversary results, calculate the expected win rate of the current strategy based on the Elo rating of the current strategy and the Elo rating of the opponent's strategy; identify the win-loss relationship of each evaluation round in the adversary results, calculate the actual score based on the win-loss relationship of each evaluation round, and update the Elo rating of each strategy in the strategy pool based on the actual score and the expected win rate.

[0119] In this embodiment of the application, the apparatus 80 further includes a determination module.

[0120] The determination module is used to execute a loop control and opponent replacement mechanism after iteratively updating the policy network parameters of the pursuit agent model based on trajectory data. The loop control and opponent replacement mechanism includes: performing a modulo operation on the number of training rounds and a preset opponent update interval threshold; if the number of training rounds meets the opponent update interval threshold, it is determined that the opponent strategy needs to be changed, the sampling probability is recalculated based on the latest Elo rating in the policy pool, and a new historical pursuit strategy is randomly selected according to the sampling probability and loaded into the self-game pursuit environment; if the number of training rounds does not meet the opponent update interval threshold, the opponent strategy parameters in the self-game pursuit environment are kept unchanged, and distributed game data collection is carried out for multiple rounds using the current opponent strategy until the next parameter update condition is reached; in each loop, it is synchronously checked whether the preset training termination condition is met. If the termination condition is met, the training loop is stopped, the currently trained policy network parameters are extracted, and the policy network parameters are solidified into the pursuit agent model.

[0121] It should be noted that the foregoing explanation of the air combat strategy evolution method based on priority virtual self-game also applies to the air combat strategy evolution device based on priority virtual self-game in this embodiment, and will not be repeated here.

[0122] The air combat strategy evolution device based on priority virtual self-game proposed in this application constructs a strategy pool with Elo ratings and employs a priority virtual self-game mechanism to dynamically select opponent strategies suitable for the current training stage, avoiding training oscillations caused by overfitting to a single opponent or random opponent selection. Trajectory data is collected through game interaction to iteratively update the agent's policy network parameters and simultaneously update the strategy pool and Elo ratings, forming a closed-loop evolution process. This effectively solves the problems of unstable agent training and insufficient generalization ability, making the strategy more robust in dynamic adversarial environments. Simultaneously, it enables the autonomous evolution of UAV pursuit strategies, improving the practicality and reliability of the strategies in real-world applications.

[0123] Figure 9 A schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device may include: The memory 901, the processor 902, and the computer program stored on the memory 901 and capable of running on the processor 902.

[0124] When the processor 902 executes the program, it implements the air combat strategy evolution method based on priority virtual self-game provided in the above embodiments.

[0125] Furthermore, electronic devices also include: Communication interface 903 is used for communication between memory 901 and processor 902.

[0126] The memory 901 is used to store computer programs that can run on the processor 902.

[0127] The memory 901 may include high-speed RAM (Random Access Memory) memory, and may also include non-volatile memory, such as at least one disk storage.

[0128] If the memory 901, processor 902, and communication interface 903 are implemented independently, then the communication interface 903, memory 901, and processor 902 can be interconnected via a bus to complete communication between them. The bus can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 9 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0129] Optionally, in a specific implementation, if the memory 901, processor 902, and communication interface 903 are integrated on a single chip, then the memory 901, processor 902, and communication interface 903 can communicate with each other through an internal interface.

[0130] The processor 902 may be a CPU (Central Processing Unit), an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of this application.

[0131] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the above-described air combat strategy evolution method based on priority virtual self-game.

[0132] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0133] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0134] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or N executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0135] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any of the following techniques known in the art, or a combination thereof: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (FPGAs), field-programmable gate arrays (FPGAs), etc.

[0136] Those skilled in the art will understand that all or part of the steps of the methods implementing the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0137] Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of this application.

Claims

1. An evolutionary method for air combat strategies based on priority virtual self-game, characterized in that, Includes the following steps: Construct a self-playing pursuit environment and strategy pool. The strategy pool stores the initial seed strategy obtained by pre-training with the experience of air combat experts and historical confrontation data, the historical pursuit strategies generated during the training process, and the Elo rating score corresponding to each strategy. Based on priority virtual self-play, the opponent strategy for the current training stage is selected from the strategy pool. The drone model is trained to play against the opponent strategy in the self-play pursuit environment. The trajectory data of the drone model is collected by dual-track inference. During the interaction, the pursuit agent model of the drone model is used to generate the current pursuit strategy. The policy network parameters of the pursuit agent model are iteratively updated based on the trajectory data. At least one of the historical pursuit policies and Elo ratings in the policy pool is updated at preset intervals. When the pool is overloaded, the redundant policy with the lowest Elo rating is removed. The current pursuit policy of the drone is generated using the iteratively updated pursuit agent model.

2. The air combat strategy evolution method based on priority virtual self-game as described in claim 1, characterized in that, The step of selecting opponent strategies from the strategy pool based on priority virtual self-game includes: Traverse the strategy pool and construct a score set based on the Elo rating of all extracted historical pursuit strategies; The non-normalized weight of each historical pursuit strategy is calculated based on the scoring set, the normalized probability of the corresponding historical pursuit strategy is calculated based on the non-normalized weight, and a probability distribution is generated based on the normalized probability of each historical pursuit strategy. Discrete random sampling is performed from the policy pool according to the probability distribution to determine the opponent policy index for the current training phase. The opponent policy for the current training phase is determined based on the opponent policy index, and the network parameters of the opponent policy are frozen.

3. The air combat strategy evolution method based on priority virtual self-game as described in claim 1, characterized in that, Training a drone model to engage in a game-like chase environment with the opponent's strategy includes: The self-playing chase environment is divided into multiple parallel deduction threads; The adversary policy is loaded into the adversary agent model of each inference thread. Dual-track inference is performed on the UAV model and the adversary agent model through each inference thread. On the training side, the UAV model is trained to receive the current situation observation vector, calculate the action probability distribution through forward propagation of a policy network with gradients, and perform random sampling to maintain exploration. On the adversary side, the adversary agent model receives the situation observation vector from the adversary's perspective, performs forward inference through a policy network with frozen parameters, and outputs action commands. The action commands of both parties are input into the flight simulation engine in the self-game pursuit environment, and the flight simulation engine calculates the flight state, relative geometric relationship and load attack judgment result at the next moment.

4. The air combat strategy evolution method based on priority virtual self-game as described in claim 1, characterized in that, The iterative update of the policy network parameters of the pursuit agent model based on the trajectory data includes: The state value at each time step is calculated based on the trajectory data, the temporal differential residual is calculated based on the state value reward signal at each time step, and the advantage value is recursively calculated based on the temporal differential residual. Calculate the probability ratio of the updated policy distribution to the old policy distribution in terms of actions, calculate the truncation loss based on the advantage value and probability ratio, and obtain the mean square loss of the value network prediction error and the policy entropy regularization term of the pursuit agent model. The total loss is calculated based on the truncation loss, the mean square loss of the value network prediction error, and the policy entropy regularization term. The policy network parameters of the pursuit agent model are then updated based on the total loss.

5. The air combat strategy evolution method based on priority virtual self-game as described in claim 1, characterized in that, Updating at least one of the historical pursuit strategies and Elo ratings in the policy pool based on the training results of the current training phase includes: If the training results of the current training phase indicate that the preset league evaluation interval has been reached, the full-pool league mechanism is triggered. The full-pool league mechanism includes: controlling the drone model to engage in multiple rounds of battles with the historical pursuit strategies in the strategy pool, and updating the Elo rating of each strategy in the strategy pool based on the battle results. If the preset entry interval is reached based on the training results of the current training phase, a policy pool update is triggered. The policy pool update includes: storing the latest pursuit and escape policy of the UAV model into the policy pool, and removing the redundant policy with the lowest Elo rating when the policy pool is overcapacity.

6. The air combat strategy evolution method based on priority virtual self-game as described in claim 5, characterized in that, The system controls the drone model to engage in multiple rounds of combat against historical pursuit strategies in the strategy pool, and updates the Elo rating of each strategy in the strategy pool based on the results of the battles, including: Iterate through the historical pursuit strategies in the strategy pool, load the historical pursuit strategies as evaluation opponents into the simulation environment in turn, conduct at least one evaluation round for each group of adversaries, and record the adversary results. Identify the Elo rating of the current strategy and the Elo rating of the opponent's strategy for each evaluation round in the adversarial results, and calculate the expected win rate of the current strategy based on the Elo rating of the current strategy and the Elo rating of the opponent's strategy. Identify the win / loss relationship for each evaluation round in the adversarial results, calculate the actual score based on the win / loss relationship for each evaluation round, and update the Elo rating of each strategy in the strategy pool based on the actual score and the expected win rate.

7. The air combat strategy evolution method based on priority virtual self-game as described in claim 1, characterized in that, After iteratively updating the policy network parameters of the pursuit agent model based on the trajectory data, a cyclic control and opponent replacement mechanism is executed, which includes: The number of training rounds is moduloed by a preset opponent update interval threshold. If the number of training rounds meets the opponent update interval threshold, it is determined that the opponent strategy needs to be changed. The sampling probability is recalculated based on the latest Elo rating in the strategy pool, and a new historical pursuit strategy is randomly selected according to the sampling probability and loaded into the self-play pursuit environment. If the number of training rounds does not meet the opponent update interval threshold, the opponent strategy parameters in the self-play chasing environment remain unchanged, and distributed game data collection is carried out for multiple rounds using the current opponent strategy until the next parameter update condition is met. In each loop, it is synchronously checked whether the preset training termination condition is met. If the termination condition is met, the training loop is stopped, the policy network parameters of the current training are extracted, and the policy network parameters are solidified into the pursuit agent model.

8. An air combat strategy evolution device based on priority virtual self-game, characterized in that, include: The construction module is used to build a self-playing pursuit environment and a strategy pool. The strategy pool stores the initial seed strategy obtained by pre-training with the experience of air combat experts and historical confrontation data, the historical pursuit strategies generated during the training process, and the Elo rating score corresponding to each strategy. The acquisition module is used to select the opponent strategy in the current training stage from the strategy pool based on priority virtual self-game, train the drone model to play against the opponent strategy in the self-game pursuit environment, perform dual-track inference to collect the trajectory data of the drone model, generate the current pursuit strategy using the pursuit agent model of the drone model during the interaction process, and collect the trajectory data of the drone model. The generation module is used to iteratively update the policy network parameters of the pursuit agent model based on the trajectory data, update at least one of the historical pursuit policies and Elo ratings in the policy pool at preset intervals, remove the redundant policy with the lowest Elo rating when the capacity is exceeded, and generate the current pursuit policy of the UAV using the iteratively updated pursuit agent model.

9. An electronic device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, the processor executing the program to implement the air combat strategy evolution method based on priority virtual self-game as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program or instructions stored thereon, characterized in that, When the computer program or instructions are executed, they implement the air combat strategy evolution method based on priority virtual self-game as described in any one of claims 1-7.

Images