Fighter flight control method based on adversarial skill embedding and hierarchical reinforcement learning

Abstract

The present invention discloses a fighter flight control method based on adversarial skill embedding and hierarchical reinforcement learning. The method designs an adversarial skill embedding learning framework composed of an adversarial generative algorithm and a deep reinforcement learning algorithm. The method uses expert flight data to automatically learn the mapping of underlying control instructions to macro maneuvering skills by an AI algorithm. By reusing these macro maneuvering skills, the method significantly reduces the learning difficulty of downstream air combat tasks, greatly compresses the convergence time of the reinforcement learning algorithm, improves the game performance of air combat intelligent agents, and gives air combat intelligent agents higher flexibility and rapid migration capabilities, thereby exploring a new technical path for the practical training and deployment of air combat intelligent agents.

Description

Technical Field

[0001] The present invention belongs to the field of high-precision (with aerodynamic model and flight control system) fighter flight simulation, and in particular to a fighter flight control method based on adversarial skill embedding and hierarchical reinforcement learning. Background Art

[0002] As a highly sophisticated electromechanical system, a fighter jet requires complex mapping and conversion of its underlying control instructions (ailerons, elevators, rudders, throttles, etc.) to a specific flight trajectory. In addition, there are various limitations of the aircraft structure and the pilot's ability to withstand the test. Therefore, human pilots need to undergo a long period of training before they can fly a fighter jet to complete various tasks such as reconnaissance, deterrence, and within-visual-range or beyond-visual-range air combat.

[0003] On the other hand, due to the limitations of human pilots in terms of experience, skills and reaction speed, the maneuverability of fighter jets is often not fully utilized in intense combat scenarios such as air combat, and there is still a lot of room for improvement in winning power. In the past decade, artificial intelligence (AI) technology has achieved explosive growth. AI agents have achieved remarkable results that surpass human levels in various fields such as Go games, large-scale video games, mathematical operations, and nuclear fusion plasma control. The application of AI in the military field has become a focus of attention.

[0004] The potential application scenarios of AI air combat agents are very rich. For example, taking over the low-level maneuvers of fighter jets in air combat allows human pilots to focus on high-level decision-making; controlling unmanned wingmen to form formations with manned lead aircraft for coordinated operations, responsible for reconnaissance, luring, protection and other tasks; controlling drone swarms to complete tasks such as dense formations, coordinated close approaches, and operations within and outside visual range. In addition, the AI ​​agent trained in an air combat simulation environment can control fighter jets to generate a massive and diverse database of flight actions, and can be used for the training and fine-tuning of large models for video generation, thereby strongly supporting extended application scenarios such as battlefield visualization review.

[0005] However, due to the high complexity of fighter flight control systems, the virtual environment equipped with flight control system simulation consumes a lot of computing resources and has a slow calculation speed, making it difficult to achieve the large-scale distributed interactive sampling required by deep reinforcement learning technology; on the other hand, due to the extremely high difficulty of learning fighter flight and air combat missions themselves, the algorithm needs to first learn to control basic actions such as take-off, landing, and cruising, and then gradually learn advanced maneuvers such as serpentine maneuvers. The above factors all lead to very slow convergence of the algorithm, and the final performance is often unsatisfactory. Summary of the invention

[0006] The purpose of the present invention is to solve the problems of learning difficulty, slow convergence, poor performance and poor transferability of air combat agents in a high-precision simulation environment (with an aerodynamic model and a flight control system). A fighter flight control method based on adversarial skill embedding and hierarchical reinforcement learning is proposed to improve the learning efficiency and combat performance of air combat agents, thereby significantly enhancing their practical value.

[0007] According to a first aspect of an embodiment of the present application, a fighter flight control method based on adversarial skill embedding and hierarchical reinforcement learning is provided, comprising:

[0008] Acquire an expert teaching data set, wherein the expert teaching data set includes flight trajectories of multiple fighter flight maneuvers;

[0009] According to the maneuver state transition pairs in the expert teaching data set, low-level strategies are trained based on a deep reinforcement learning algorithm and a generalized advantage estimation algorithm, wherein the low-level strategies are used to obtain corresponding underlying control instructions according to macro maneuver skills and current states;

[0010] For downstream task objectives, high-level strategies are trained based on deep reinforcement learning algorithms and generalized advantage estimation algorithms, where the high-level strategies are used to obtain corresponding macro-maneuver skills according to the current state, and the parameters of the low-level strategies remain frozen during the high-level strategy training process;

[0011] The current state of the fighter is obtained, and the corresponding underlying control instructions are obtained based on the trained high-level and low-level strategies, thereby realizing the flight control of the fighter.

[0012] In one possible implementation, the objectives of the low-level strategy are:

[0013]

[0014] where d π (s,s′) and denote the state transition distribution dominated by the low-level strategy π and the state transition distribution in the expert teaching dataset, respectively, and d JS is the operator of Jensen-Shannon divergence, s is the state, (s, s′) is the maneuver state transition pair, I(s, s′; z|π) represents the mutual information between the skill embedding z learned under the dominance of the low-level strategy π and the corresponding maneuver state transition, and β is the relative weight used to adjust the Jensen-Shannon divergence and mutual information.

[0015] In one possible implementation, the training of the low-level strategy is achieved by minimizing the Jensen-Shannon divergence and maximizing the mutual information;

[0016] The Jensen-Shannon divergence The optimization goal is:

[0017]

[0018] Where D(s,s′) is a discriminator used to distinguish whether the maneuver state transition pair (s,s′) comes from an expert teaching dataset or is generated by a low-level strategy; τ is a weight coefficient, Represents gradient calculation;

[0019] The mutual information I(s,s′; z|π) is maximized by maximizing the skill encoder q(z|s,s′), and the optimization goal is:

[0020]

[0021] In a possible implementation, a neural network is used to represent the low-level strategy. The input of the neural network is the concatenation vector of the state s and the skill embedding z, and the output is the mean vector μ π (s,z), and then obtain the Gaussian distribution of the multi-dimensional continuous space corresponding to the underlying control instructions The diagonal covariance matrix ∑ π It consists of a fixed vector of the same dimension that can receive the gradient;

[0022] The specific training steps of the low-level strategy are as follows:

[0023] 1) Collect a skill embedding sequence {z0,z1,..., ... T-1};

[0024] 2) Based on the low-level strategy π(a t |s t ,z t ) Sample an equal-length trajectory from an air combat simulation environment At each time step, the low-level strategy inputs the current state s t and skill embedding z t , output action a t , and obtain the reward function defined by the discriminator and the skill encoder

[0025]

[0026] 3) Repeat steps 1) to 2) to obtain a batch of trajectory data;

[0027] 4) Randomly select a minibatch of state transition pairs and corresponding skill embeddings (s i ,s i ′ ,z i), used to update the skill encoder q(z|s,s′);

[0028] 5) Randomly select a minibatch of state transition pairs (s) from the batch of trajectory data and the expert teaching data set i ,s i ′ ), used to update the discriminator D(s,s′);

[0029] 6) Based on the batch of trajectory data, combined with the reward function Use deep reinforcement learning algorithm and generalized advantage estimation algorithm to update the low-level policy π(a|s,z);

[0030] 7) Repeat steps 1)-6) until convergence or the preset maximum number of training steps is reached.

[0031] In a possible implementation, the distribution space p(z) where the macro-maneuver skill embedding z in the input of the low-level strategy is located is used as the action space of the high-level strategy ω(z|s), and the action of the high-level strategy is used as the input of the low-level strategy after normalization by modulus length.

[0032] In one possible implementation, the high-level policy optimization objective can be expressed as the following formula:

[0033]

[0034] in, represents the trajectory collected from the simulation environment based on the high-level strategy, represents the high-level strategy output without modulus normalization, and g represents the downstream task goal.

[0035] According to a second aspect of an embodiment of the present application, a fighter flight control device based on adversarial skill embedding and hierarchical reinforcement learning is provided, comprising:

[0036] A data set acquisition module, used to acquire an expert teaching data set, wherein the expert teaching data set includes flight trajectories of various fighter flight maneuvers;

[0037] A low-level strategy training module, used for training low-level strategies based on the deep reinforcement learning algorithm and the generalized advantage estimation algorithm according to the maneuver state transition pairs in the expert teaching data set, wherein the low-level strategy is used to obtain corresponding underlying control instructions according to the macro maneuver skills and the current state;

[0038] The high-level strategy training module trains high-level strategies based on the deep reinforcement learning algorithm and the generalized advantage estimation algorithm for downstream task objectives, wherein the high-level strategy is used to obtain the corresponding macro-maneuver skills according to the current state and reward function, and the low-level strategy gradient is frozen during the high-level strategy training process;

[0039] The instruction generation module is used to obtain the current state of the fighter and obtain the corresponding underlying control instructions based on the trained high-level and low-level strategies, thereby realizing the flight control of the fighter.

[0040] According to a third aspect of an embodiment of the present application, a computer program product is provided, comprising a computer program / instruction, wherein when the computer program / instruction is executed by a processor, the method described in the first aspect is implemented.

[0041] According to a fourth aspect of an embodiment of the present application, there is provided an electronic device, including:

[0042] one or more processors;

[0043] A memory for storing one or more programs;

[0044] When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in the first aspect.

[0045] According to a fifth aspect of an embodiment of the present application, a computer-readable storage medium is provided, on which computer instructions are stored, and when the instructions are executed by a processor, the steps of the method described in the first aspect are implemented.

[0046] The technical solution provided by the embodiments of the present application may have the following beneficial effects:

[0047] As can be seen from the above embodiments, this application uses expert teaching data to learn and store the mapping of commonly used macro-maneuvering skills in air combat to underlying control instructions (ailerons, elevators, rudders, throttles) in advance in low-level policies for reuse, thereby significantly reducing the learning burden of downstream air combat tasks. High-level policies only need to learn to call these macro-maneuvering skills in a timely manner for specific air combat tasks, without having to worry about how to implement the corresponding actions. Significantly reduce the learning difficulty of downstream air combat tasks, greatly compress the convergence time of reinforcement learning algorithms, improve the game performance of air combat agents, and give air combat agents higher flexibility and rapid migration capabilities, and explore a new technical path for the practical training and deployment of air combat agents.

[0048] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0050] Figure 1 It is a schematic diagram of a fighter flight control method based on adversarial skill embedding and hierarchical reinforcement learning according to an exemplary embodiment.

[0051] Figure 2 It is a schematic diagram showing the collection of expert teaching data from DCS World according to an exemplary embodiment.

[0052] Figure 3 It is a schematic diagram of a low-level strategy training process according to an exemplary embodiment.

[0053] Figure 4 It is a schematic diagram of a macro-maneuver skill embedding space and a high-level strategy action space according to an exemplary embodiment.

[0054] Figure 5 It is a block diagram of a fighter flight control device based on adversarial skill embedding and hierarchical reinforcement learning according to an exemplary embodiment.

[0055] Figure 6 The diagram is a schematic diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0056] Here, exemplary embodiments are described in detail, and examples thereof are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application.

[0057] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. The singular forms of "a", "said" and "the" used in this application and the appended claims are also intended to include plural forms unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0058] It should be understood that although the terms first, second, third, etc. may be used in the present application to describe various information, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0059] Glossary:

[0060] The maneuvers of a fighter jet during air combat can be seen as the result of a sequence of underlying control commands (ailerons, elevators, rudders, and throttles) mapped to the fuselage through the flight control system, or it can be broken down into a series of commonly used macro-maneuvering skills at the middle level, such as cruise control, serpentine maneuvers, dogfight maneuvers, rolling maneuvers, etc. The combination of these skills can complete any advanced air combat maneuvers, and each macro-maneuvering skill corresponds to a specific sequence of underlying control commands.

[0061] Due to the high complexity of fighter flight control systems, it is extremely difficult to learn advanced air combat maneuvers and complete combat missions directly from the underlying control instructions. The difficulty of learning air combat missions based on ready-made macro maneuvering skill sets is significantly reduced, because this is equivalent to reusing the fighter's basic flight control skills. The reinforcement learning algorithm no longer needs to learn these skills from scratch each time, thereby greatly improving the training and migration efficiency for downstream air combat missions.

[0062] The macro maneuvering skill set can be given manually based on the experience of air combat experts, or it can be automatically discovered through AI algorithms. The present invention provides a set of intelligent algorithms based on air combat expert teaching data and adversarial skill embedding, which can automatically map the underlying control instructions to the latent space embedding of the macro maneuvering skills and form a probability distribution. At the same time, a dedicated low-level policy is learned to map the observation and skill embedding of the current fighter back to the underlying control instruction sequence, ensuring that these macro maneuvering skills can be easily reused.

[0063] The present invention provides a fighter flight control method based on adversarial skill embedding and hierarchical reinforcement learning. The principle of the method is as follows: Figure 1 As shown, including:

[0064] Step S1: obtaining an expert teaching data set, wherein the expert teaching data set includes flight trajectories of various fighter flight maneuvers;

[0065] In order to use AI algorithms to automatically explore and discover the set of macro-maneuver skills and even form a complete distribution of the skill embedding space, an expert teaching dataset is needed. The dataset should contain a rich variety of flight maneuvers, and the quality and completion of these maneuvers should be at a high level. In addition, the dataset should be of a certain size to avoid fitting problems during the learning process.

[0066] DCS World is one of the most popular air combat themed video games in the world, with a large player group and communication community. The game supports saving each battle log as an .acmi file, which can be opened by the commonly used flight analysis tool Tacview for review. Various fighters participating in the air combat can be selected and the flight trajectory information - latitude and longitude coordinates, speed, roll angle, pitch angle, yaw angle, etc. at each moment can be output according to the user-defined frequency. These flight trajectories can be used to form the initial expert teaching data set. The above operation process is as follows: Figure 2 shown.

[0067] In order to meet the requirements of diversity and professionalism in air combat maneuvers, the initial data set needs to be cleaned, the monotonous level flight cruise in non-confrontational conditions is eliminated, and the high-intensity and high-frequency maneuvering action segments in air combat are retained. The data of some players with relatively average flight control and combat levels should also be appropriately eliminated, and finally an expert teaching data set containing a sufficient number of high-quality and high-richness flight trajectories is formed.

[0068] Step S2: training a low-level strategy based on a deep reinforcement learning algorithm and a generalized advantage estimation algorithm according to the maneuver state transition pairs in the expert teaching data set, wherein the low-level strategy is used to obtain corresponding underlying control instructions according to the macro maneuver skills and the current state;

[0069] In order to learn effective low-level strategies from expert teaching datasets to support the rapid migration of any downstream tasks, it is required that the learned macro-maneuvering skills are universal and complete, and not limited to mechanical imitation of specific expert trajectories or actions, so that high-level strategies can flexibly decide when to imitate which expert trajectory when performing downstream air combat tasks, that is, to achieve "similar in spirit but not in form". On the other hand, the flight trajectories in the above expert datasets come from different fighter models, and the flight control systems and underlying control instructions are all different. Conventional imitation learning methods such as behavioral cloning are difficult to apply directly. The present invention provides an adversarial skill embedding algorithm to meet the above requirements.

[0070] First, the low-level strategy can be expressed as a conditional strategy π(a|s,z) modulated by macro maneuvering skills, where s represents the state, a represents the underlying control command (aileron, elevator, rudder, throttle), and each macro maneuvering skill z is represented as a latent space An embedding within And it is sampled from the macro-maneuver skill distribution p(z) defined in this space. In order to improve the training effect and stability, the present invention Defined as a hypersphere That is, the amplitude of all skill embeddings z is constrained to unit length, and the macro-mobility skill distribution p(z) is a uniform distribution on the hypersphere. The operation of sampling from p(z) can be achieved by sampling from a standard normal distribution and then normalizing it.

[0071] Secondly, the learning of the low-level policy π(a|s,z) combines the maneuver imitation sub-goal and the unsupervised skill discovery sub-goal, thereby encouraging the low-level policy to be more efficient in the distribution of expert teaching data. Learn a wealth of reusable skills, and the specific goals are defined as follows:

[0072]

[0073] where d π (s,s′) and denote the state transition distribution dominated by the low-level strategy π and the state transition distribution in the expert teaching dataset, respectively. JS is the Jensen-Shannon divergence operator that minimizes d π (s,s′) and The Jensen-Shannon divergence between Distribution of maneuvering state transfer in .

[0074] On the one hand, the goal is to focus only on the external maneuvers of the fighter and avoid being bound to the flight control system of a specific fighter model, so that the algorithm can use heterogeneous expert data for learning; on the other hand, the maneuver state transition distribution matching is different from simply imitating the state transition of a specific maneuver, which will help the algorithm to achieve "similar in spirit but not in form" and allow interpolation and generalization in the learned skill embedding space while still maintaining The mobile action style.

[0075] I(s, s′; z|π) represents the mutual information between the skill embedding z and the corresponding maneuver state transition learned under the guidance of the low-level strategy π. Maximizing this term will encourage π to learn diverse and discriminative skill embeddings, thereby facilitating the reuse of these macro-maneuver skills, while improving the stability of the training process and avoiding "model collapse"; β is used to adjust the relative weight between the two sub-goals.

[0076] In the specific implementation, the present invention adopts an architecture similar to the Generative Adversarial Network (GAN) to approximate the effect of minimizing the Jensen-Shannon divergence. The discriminator D(s, s′) is responsible for distinguishing the maneuver state transition pair (s, s′) from the expert dataset. Or it is dominated by the low-level strategy π(a|s,z). To alleviate the instability problem during training, an additional amplitude regularization penalty term is added for the discriminator update gradient, and τ is used as an adjustable weight coefficient. Represents a gradient computation operation.

[0077]

[0078] The calculation of mutual information I(s,s′;z|π) also requires an approximate method, because Entropy of macromobility skill embedding distribution is a constant that is independent of the low-level strategy, so maximizing the mutual information I(s,s′;z|π) is equivalent to minimizing the conditional entropy The entropy value can be calculated based on the macro-mobility skill distribution p(z) and the conditional distribution p(s,s ′ |π,z), calculated by Monte Carlo sampling.

[0079] Here we introduce a new conditional distribution q(z|s,s′), according to the mathematical principle that the cross entropy of two distributions is not less than the entropy of any of them. Then maximizing mutual information I(s,s′; z|π) can be approximated by maximizing q(z|s,s′). The present invention additionally sets up a skill encoder q(z|s,s′) represented by a neural network to help achieve this goal. Its optimization goal during training is:

[0080]

[0081] Since the inputs are exactly the same, the present invention uses the same neural network backbone to represent the discriminator D(s, s′) and the skill encoder q(z|s, s′). The two only use different output heads in the output part. The discriminator part outputs a 1-dimensional sigmoid activation value, and the skill encoder part outputs a vector μ of the same dimension as the skill embedding q (s,s′), and normalize its modulus to ensure ||μ q (s,s′)||=1.

[0082] Based on the above macro-maneuver skill embedding space distribution p(z), discriminator D(s,s′) and skill encoder q(z|s,s′), the present invention proposes a low-level strategy π(a|s,z) training method based on deep reinforcement learning algorithm (proximal policy optimization, PPO) and generalized advantage estimation (GAE).

[0083] The low-level strategy π(a|s,z) is also represented by a neural network, with the input state s and the concatenated vector of the macro-maneuver skill embedding z as the input, and the output mean vector μ π(s,z), and then obtain the Gaussian distribution of the multi-dimensional continuous space corresponding to the underlying control instructions The diagonal covariance matrix ∑ π It is represented by a fixed vector of the same dimension that can receive the gradient.

[0084] The value function in the PPO algorithm is represented by an independent neural network with the same structure as the low-level policy, except that the output is a one-dimensional value. In practice, the value network and the low-level policy network backbone can also be merged, with different heads at the output end.

[0085] The specific training steps of the low-level strategy are as follows:

[0086] 1) Collect a skill embedding sequence {z0,z1,..., ... T-1}

[0087] 2) Based on the low-level strategy π(a t |s t ,z t ) Sample an equal-length trajectory from a high-precision air combat simulation environment At each time step, the low-level strategy inputs the current state s t and skill embedding z t , output action a t , and obtain the reward function defined by the discriminator and the skill encoder

[0088] 3) Repeat steps 1)-2) and collect a batch of trajectories;

[0089] 4) Randomly select a small batch (minibatch) of state transfer pairs + corresponding skill embeddings (s i ,s i ′ ,z i ), used to update the skill encoder q(z|s,s′);

[0090] 5) Randomly select a small batch of state transition pairs (s) from this batch of trajectories and the expert teaching dataset respectively. i ,s i ′ ), used to update the discriminator D(s,s′);

[0091] 6) Based on this batch of trajectory data, combined with the reward function Use PPO+GAE algorithm to update the low-level strategy π(a|s,z)

[0092] 7) Repeat steps 1)-6) until the algorithm converges or reaches the preset maximum number of training steps;

[0093] In order to further reduce the possibility of "model collapse" during the training of low-level strategies and improve its temporal maneuverability, the present invention adds a sub-goal to encourage diversity in addition to the reinforcement learning policy gradient in the low-level strategy optimization goal. That is, skill embeddings with similar cosine distances should produce similar maneuvers. This sub-goal does not change the reward function of the low-level strategy, but only provides an additional source of gradients when the strategy is updated. The coefficient w is used to adjust the weight of this component.

[0094] The training process of the low-level strategy is as follows Figure 3 As shown, the complete optimization objective can be expressed as the following formula, where γ L Denote the discount factor:

[0095]

[0096] Step S3: For downstream mission objectives such as patrolling, accompanying, driving away, and annihilating, high-level strategies are trained based on deep reinforcement learning algorithms and generalized advantage estimation algorithms, wherein the high-level strategies are used to give corresponding macro-maneuvering skills according to the current state. During the training of the high-level strategies, the parameters of the low-level strategies remain frozen.

[0097] After the low-level strategy π(a|s,z) is trained, the distribution space p(z) where the macro-maneuver skill embedding z in its input is located can be used as the action space of the high-level strategy ω(z|s) for downstream tasks. Specifically, the high-level strategy ω(z|s) represented by a neural network has a direct output of the mean vector μ ω (s), which is consistent with the diagonal covariance matrix ∑ ω (represented as another fixed vector of the same dimension that can receive gradients) together define a Gaussian distribution in the macro-skill embedding space z is obtained by sampling from the latter.

[0098] The action vector output by the high-level strategy is not a unit vector. Its distribution It is a skill embedded in the hypersphere space More space Before the high-level action is input into the low-level strategy π(a|s,z), its output action vector is first normalized, that is, mapped to the hypersphere where p(z) is located. superior.

[0099] The benefit of this design is that it facilitates the exploration-utilization balance in the reinforcement learning training process. Figure 4 As shown, when the network of the high-level strategy ω(z|s) is initialized, the output μ at the beginning of the training will be ω(s) is close to a pure zero vector, that is, The central position of p(z) ensures the maximum exploration ability within the p(z) distribution. As the training progresses, the high-level strategy output gradually deviates from The center of The corresponding skill embedding z is then the non-uniform sampling probability in p(z).

[0100] In order to avoid abnormal behaviors that may deviate from the distribution of the expert teaching data set during the training of high-level strategies, while reusing the low-level strategies, the discriminator D(s, s′) output obtained when training the low-level strategies is used as the downstream task target component r g (s t ,a t ,s t+1 ), thereby encouraging the agent to output maneuvers that are consistent with the distribution of the expert teaching dataset as much as possible while completing the target task. The coefficient ρ is used to adjust the relative weight between the two components.

[0101]

[0102] In addition, like the low-level strategy, the network parameters of the discriminator remain frozen during the training of the high-level strategy and are no longer updated. With the further constraints of the expert maneuver prior provided by the discriminator, the high-level strategy can minimize the generation of unnatural and inappropriate abnormal maneuvers during the learning process, ensuring the robustness of the high-level strategy.

[0103] In the present invention, the high-level strategy ω(z|s) is also trained using the PPO+GAE algorithm, and can also be replaced by other reinforcement learning algorithms that support continuous action space, such as SAC, DDPG, TD3, etc. The complete high-level strategy optimization objective can be expressed as the following formula:

[0104]

[0105] in, represents the trajectory collected from the simulation environment based on the high-level strategy, represents the high-level policy output without modulus normalization. g represents the downstream task goal, which can be used as part of the high-level policy input or represent the decision-making process under the constraint of the goal (through the reward function) at an abstract level.

[0106] Step S4: obtaining the current state of the fighter, and obtaining corresponding underlying control instructions based on the trained high-level strategy and low-level strategy, thereby realizing the flight control of the fighter;

[0107] Specifically, the high-level strategy outputs the macro maneuver skill z that needs to be executed currently based on the current state s. The low-level strategy inputs the macro maneuver skill z and the current state s, and outputs the corresponding underlying control instruction a, which is applied to the flight control system and ultimately drives the fighter to complete the relevant maneuver.

[0108] It should be noted that the present application does not limit the neural network structure used, and in specific implementations, different network forms may be used depending on the specific form of the input state (1-D, 2-D).

[0109] It can be seen from the above embodiments that the present application defines a low-level strategy optimization target determined by the mutual information of Jensen-Shannon divergence and skill embedding + maneuver state transition, to ensure that the learned skill embedding has sufficient diversity and avoid "model collapse"; a discriminator and skill encoder represented by a neural network are used to define an operational algorithm implementation method that approaches the above optimization target; a regularized penalty term for updating the gradient amplitude of the discriminator is used to improve the stability of adversarial skill embedding learning; the skill embedding at the input end of the low-level strategy is resampled at each time step, and at the same time, the skill embedding is updated in the update. A sub-goal that encourages maneuver diversity is added to the low-level policy gradient (skill embeddings with similar cosine distances should produce similar maneuvers) to ensure that the low-level policy has sufficient temporal maneuverability; a skill embedding space based on a hypersphere, a matching high-level policy action space, and a high-level policy network initialization method are designed to ensure that the optimal exploration-utilization balance is achieved during the high-level policy training process; by reusing the low-level policy to provide a mapping of macro-maneuver skills to underlying control instructions, and reusing the discriminator to provide additional reward function components, the high-level policy can learn more efficiently and output more natural actions when facing downstream tasks. In summary, this application significantly reduces the learning difficulty of downstream air combat tasks, greatly compresses the convergence time of reinforcement learning algorithms, improves the game performance of air combat agents, and gives air combat agents higher flexibility and rapid migration capabilities, exploring a new technical path for the practical training and deployment of air combat agents.

[0110] Corresponding to the aforementioned embodiment of the fighter flight control method based on adversarial skill embedding and hierarchical reinforcement learning, the present application also provides an embodiment of a fighter flight control device based on adversarial skill embedding and hierarchical reinforcement learning.

[0111] Figure 5 is a block diagram of a fighter flight control device based on adversarial skill embedding and hierarchical reinforcement learning according to an exemplary embodiment. Figure 5 , the device may include:

[0112] A data set acquisition module 21 is used to acquire an expert teaching data set, wherein the expert teaching data set includes flight trajectories of various fighter flight maneuvers;

[0113] A low-level strategy training module 22 is used to train low-level strategies based on the deep reinforcement learning algorithm and the generalized advantage estimation algorithm according to the maneuver state transition pairs in the expert teaching data set, wherein the low-level strategy is used to obtain corresponding underlying control instructions according to the macro maneuver skills and the current state;

[0114] A high-level strategy training module 23 is used to train high-level strategies based on a deep reinforcement learning algorithm and a generalized advantage estimation algorithm for downstream task objectives, wherein the high-level strategies are used to obtain corresponding macro-maneuvering skills according to the current state, and the low-level strategy gradients are frozen during the high-level strategy training process;

[0115] The instruction generation module 24 is used to obtain the current state of the fighter and obtain the corresponding low-level control instructions based on the trained high-level strategy and low-level strategy, so as to realize the flight control of the fighter.

[0116] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0117] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present application scheme. A person of ordinary skill in the art can understand and implement it without paying any creative work.

[0118] Accordingly, the present application also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the fighter flight control method based on adversarial skill embedding and hierarchical reinforcement learning as described above.

[0119] Accordingly, the present application also provides an electronic device, comprising: one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the above-mentioned fighter flight control method based on adversarial skill embedding and hierarchical reinforcement learning. Figure 6 As shown, it is a hardware structure diagram of any device with data processing capability in which a deep learning data set access system provided by an embodiment of the present invention is located, except Figure 6In addition to the processor, memory and network interface shown, any device with data processing capability in which the apparatus in the embodiment is located may also include other hardware, generally based on the actual functions of the device with data processing capability, which will not be described in detail.

[0120] Accordingly, the present application also provides a computer-readable storage medium on which computer instructions are stored, and when the instructions are executed by the processor, the fighter flight control method based on adversarial skill embedding and hierarchical reinforcement learning as described above is implemented. The computer-readable storage medium can be an internal storage unit of any device with data processing capabilities described in any of the aforementioned embodiments, such as a hard disk or memory. The computer-readable storage medium can also be an external storage device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), an SD card, a flash card (FlashCard), etc. equipped on the device. Further, the computer-readable storage medium can also include both an internal storage unit of any device with data processing capabilities and an external storage device. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capabilities, and can also be used to temporarily store data that has been output or is to be output.

[0121] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the contents disclosed herein. The present application is intended to cover any variations, uses or adaptations of the present application, which follow the general principles of the present application and include common knowledge or customary technical means in the art that are not disclosed in the present application.

[0122] It should be understood that the present application is not limited to the exact construction that has been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof.

Claims

1. A fighter flight control method based on adversarial skill embedding and hierarchical reinforcement learning, characterized in that: include: Acquire an expert teaching data set, wherein the expert teaching data set includes flight trajectories of multiple fighter flight maneuvers; According to the maneuver state transition pairs in the expert teaching data set, low-level strategies are trained based on a deep reinforcement learning algorithm and a generalized advantage estimation algorithm, wherein the low-level strategies are used to obtain corresponding underlying control instructions according to macro maneuver skills and current states; For downstream task objectives, high-level strategies are trained based on deep reinforcement learning algorithms and generalized advantage estimation algorithms, where the high-level strategies are used to obtain corresponding macro-maneuver skills according to the current state, and the parameters of the low-level strategies remain frozen during the high-level strategy training process; Obtain the current state of the fighter, and obtain the corresponding low-level control instructions based on the trained high-level and low-level strategies, so as to realize the flight control of the fighter; Among them, the optimization goal of the low-level strategy is: where d π (s,s′) and They represent the state transition distribution dominated by the low-level strategy π and the state transition distribution in the expert teaching dataset, respectively. The expert teaching data distribution, D JS is the operator of Jensen-Shannon divergence, s is the state, (s, s′) is the maneuver state transition pair, I(s, s′; z|π) represents the mutual information between the skill embedding z learned under the dominance of the low-level strategy π and the corresponding maneuver state transition, and β is the relative weight used to adjust the Jensen-Shannon divergence and mutual information; The distribution space p(z) where the macro-maneuver skill embedding z in the input of the low-level strategy is located is used as the action space of the high-level strategy ω(z|s), and the action of the high-level strategy is used as the input of the low-level strategy after being normalized by the modulus length; The high-level policy optimization objective can be expressed as the following formula: in, represents the trajectory collected from the simulation environment based on the high-level strategy, represents the high-level strategy output without modulus normalization, g represents the downstream task goal, and r g (s y ,a y ,s t+1 ) is the downstream task target component, and ρ is the coefficient used to adjust the relative weight between the two components.

2. The method according to claim 1, characterized in that Training the low-level strategy is achieved by minimizing the Jensen-Shannon divergence and maximizing the mutual information; The Jensen-Shannon divergence The optimization goal is: Where D(s,s′) is a discriminator used to distinguish whether the maneuver state transition pair (s,s′) comes from an expert teaching dataset or is generated by a low-level strategy; τ is a weight coefficient, Represents gradient calculation; The mutual information I(s,s′; z|π) is maximized by maximizing the skill encoder q(z|s,s′), and the optimization goal is: Among them, p(z) is the macro maneuver skill distribution, μ q (s,s′) is the output of the skill encoder.

3. The method according to claim 1, characterized in that: The input of the neural network is the concatenation vector of the state s and the skill embedding z, and the output is the mean vector μ π (s,z), and then obtain the Gaussian distribution of the multi-dimensional continuous space corresponding to the underlying control instructions The diagonal covariance matrix ∑ π Represented by a fixed vector of the same dimension that can receive gradients; The specific training steps of the low-level strategy are as follows: 1) Collect a skill embedding sequence {z0,z1,..., ... T-1 }; 2) Based on the low-level strategy π(a t |s t ,z t ) Sample an equal-length trajectory from an air combat simulation environment At each time step, the low-level strategy inputs the current state s t and skill embedding z t , output action a t , and obtain the reward function defined by the discriminator and the skill encoder 3) Repeat steps 1) to 2) to obtain a batch of trajectory data; 4) Randomly select a minibatch of state transition pairs and corresponding skill embeddings (s i ,s i ′ ,z i ), used to update the skill encoder q(z|s,s′); 5) Randomly select a minibatch of state transition pairs (s) from the batch of trajectory data and the expert teaching data set i ,s i ′ ), used to update the discriminator D(s,s′); 6) Based on the batch of trajectory data, combined with the reward function Use deep reinforcement learning algorithm and generalized advantage estimation algorithm to update the low-level policy π(a|s,z); 7) Repeat steps 1)-6) until convergence or the preset maximum number of training steps is reached.

4. A fighter flight control device based on adversarial skill embedding and hierarchical reinforcement learning, characterized in that: Implementing the method according to any one of claims 1 to 3, comprising: A data set acquisition module, used to acquire an expert teaching data set, wherein the expert teaching data set includes flight trajectories of various fighter flight maneuvers; A low-level strategy training module, used for training low-level strategies based on the deep reinforcement learning algorithm and the generalized advantage estimation algorithm according to the maneuver state transition pairs in the expert teaching data set, wherein the low-level strategy is used to obtain corresponding underlying control instructions according to the macro maneuver skills and the current state; The high-level strategy training module trains high-level strategies based on the deep reinforcement learning algorithm and the generalized advantage estimation algorithm for downstream task objectives, wherein the high-level strategy is used to obtain the corresponding macro-maneuver skills according to the current state and reward function, and the low-level strategy gradient is frozen during the high-level strategy training process; The instruction generation module is used to obtain the current state of the fighter and obtain the corresponding underlying control instructions based on the trained high-level and low-level strategies, thereby realizing the flight control of the fighter.

5. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the method according to any one of claims 1 to 3 is implemented.

6. An electronic device, characterized in that: include: one or more processors; A memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 3.

7. A computer-readable storage medium having computer instructions stored thereon, characterized in that: When the instruction is executed by a processor, the steps of the method according to any one of claims 1 to 3 are implemented.

Images