An air combat decision-making method and system based on generative adversarial imitation learning

Abstract

The present invention discloses an air combat decision-making method and system based on generative adversarial imitation learning, which belongs to the field of aviation technology. The method comprises: obtaining battlefield situation information, processing the battlefield situation information through a pre-trained air combat decision-making model, and obtaining a decision result for guiding the aircraft. The air combat decision-making model is obtained by the following steps: obtaining typical combat opponent data, performing enhancement processing on the typical combat opponent data, and obtaining a red-blue adversarial data set; using a generative adversarial imitation learning algorithm, and obtaining an air combat decision-making model based on the red-blue adversarial data set training. The present invention solves the modal collapse problem of the generative adversarial imitation learning training process through algorithm design, and can ensure that the generated strategies are diverse and generalizable; solves the problem of low sample utilization efficiency through data enhancement methods, and can obtain better training results based on training with fewer samples; and still has decision-making robustness in complex dynamic environments.

Description

Technical Field

[0001] This invention relates to the field of aviation technology, and in particular to an air combat decision-making method and system based on generative adversarial imitation learning. Background Technology

[0002] In the field of flight control, the learning and imitation of expert flight strategies has always been an important research direction. Traditional flight strategy learning methods mainly rely on the experience of experts and manual adjustments, which is not only time-consuming and labor-intensive, but also makes it difficult to guarantee the generalization ability and robustness of the strategy.

[0003] In recent years, with the rapid development of artificial intelligence technology, imitation learning, as a method of learning strategies from expert demonstration data, has received widespread attention.

[0004] Imitation learning can be divided into two categories: behavioral cloning and imitation learning based on inverse reinforcement learning (IRL). Behavioral cloning learns policies by directly cloning the actions of experts in various states, but this method is susceptible to cascading errors, resulting in poor robustness and generalization ability of the policies. Inverse reinforcement learning learns policies by deriving the reward function behind the expert's policy. Although this method solves the cascading error problem to some extent, its high computational cost makes it difficult to apply to large-scale problems.

[0005] Generative Adversarial Imitation Learning (GAIL) is a method that combines Generative Adversarial Networks (GANs) and imitation learning, enabling the learning of policies directly from expert demonstration data without relying on explicit reward functions.

[0006] GAIL employs adversarial training between the generator and discriminator, enabling the generator to produce behavioral trajectories similar to expert policies, thus achieving imitation of expert strategies. Compared to traditional imitation learning methods, GAIL possesses stronger representational capabilities and computational efficiency, allowing it to handle complex, large-scale problems.

[0007] However, the GAIL method also has some problems:

[0008] 1. Insufficient diversity and generalization ability: In flight missions, different flight environments and mission requirements can lead to a diversity of expert policies. GAIL may experience mode collapse during training, meaning that the generated policies lack diversity and cannot cover the full distribution of expert policies, resulting in insufficient generalization ability.

[0009] 2. Low sample utilization efficiency: Data acquisition during flight missions is costly. GAIL requires a large number of interactive samples during training to estimate the expected cumulative reward of the policy, resulting in low sample utilization efficiency and increased training costs.

[0010] 3. Robustness in complex environments: Flight missions typically involve complex dynamic environments and various uncertainties. How to ensure the robustness and stability of strategies in such environments is an urgent problem to be solved. Summary of the Invention

[0011] This invention provides an air combat decision-making method and system based on generative adversarial imitation learning, so as to at least partially solve the above-mentioned problems.

[0012] To achieve the above object, the technical solution of the present invention is:

[0013] In a first aspect, the present invention provides an air combat decision-making method based on generative adversarial imitation learning. The method includes: acquiring battlefield situation information, processing the battlefield situation information through a pre-trained air combat decision-making model, and obtaining decision results for guiding the aircraft. The decision results include one or more of the aircraft's maneuvering actions, ACM mode, and missile firing actions.

[0014] The air combat decision-making model is obtained through the following steps:

[0015] Data on typical adversaries is acquired, and the data on typical adversaries is enhanced to obtain a red-blue confrontation dataset.

[0016] Using a generative adversarial imitation learning algorithm, a network model capable of simulating the behavioral characteristics and decision-making of a typical combat opponent is trained based on the red-blue adversarial dataset, namely the air combat decision model.

[0017] In some preferred embodiments, the typical adversary data includes coarse-grained sparse expert strategy data and fine-grained dense expert strategy data, wherein the coarse-grained sparse expert strategy data includes air combat rules and state-action pairs.

[0018] The step of augmenting the typical adversary data to obtain the red-blue adversary dataset includes:

[0019] The air combat rules are expanded using a decision tree model to generate new state-action pairs, which are then merged with the state-action pairs contained in the coarse-grained sparse expert strategy data to obtain the first dataset.

[0020] Using the behavior cloning algorithm, a behavior cloning network is trained based on the fine-grained dense expert policy data to obtain a behavior cloning network; the behavior cloning network is then used to augment the fine-grained dense expert policy data to obtain a second dataset.

[0021] The union of the first dataset and the second dataset constitutes the red-blue adversarial dataset.

[0022] In some preferred embodiments, battlefield situation information includes the aircraft's coordinates, the flight status of both friendly and enemy forces, geometric situation, and the aircraft's payload status.

[0023] In some preferred embodiments, the local coordinates include east-west coordinates, north-south coordinates, and flight altitude; the flight status of both sides includes airspeed, indicated airspeed, Mach number, attitude quaternion, and engagement time; the geometric situation includes enemy-friendly distance vector, enemy-friendly distance scalar, enemy-friendly velocity vector, enemy-friendly altitude difference, entry angle, and azimuth angle; and the payload status includes chaff status, remaining number of chaff, missile status, and remaining number of missiles.

[0024] In some preferred embodiments, the maneuvers include level flight, level flight acceleration and deceleration, climb, turn, dive, traverse, half roll, S-maneuver, and oblique loop; the ACM modes include horizontal sweep mode selection, vertical sweep mode selection, fixed axis mode selection, and seeker unlock; and the missile firing actions include infrared missile launch, chaff launch, and infrared chaff launch.

[0025] In some preferred embodiments, the training process of the air combat decision model is based on an air combat simulation environment, which is constructed through the following steps:

[0026] First, a six-degree-of-freedom kinematic dynamics model of the aircraft is constructed. This model is established in the body coordinate system and the airflow coordinate system, considering the dynamic equations of the aircraft's center of mass movement. With the aircraft's center of mass as the origin, in the body coordinate system, we have:

[0027]

[0028] Among them, F x F y F z , q, and r are the projections of the net external force acting on the aircraft onto the three axes of the body coordinate system, respectively; p, q, and r are the projections of the aircraft angular velocity onto the three axes of the body coordinate system, respectively; and u, v, and w are the components of the aircraft airspeed in the body coordinate system, respectively.

[0029] External forces acting on the center of mass include engine thrust T and aerodynamic force F. a The projection of gravity G, the net external force, onto the three axes of the body coordinate system can be specifically expressed as:

[0030]

[0031] Where L, D, and Y represent the projections of aerodynamic forces onto the three axes of the airflow coordinate system, namely lift, drag, and side force, respectively; α and β are the angle of attack and sideslip angle, respectively; and φ, θ, and ψ are the attitude angles in the body coordinate system.

[0032] Expressing the velocity in the body coordinate system using V, α, and β in the airflow system yields:

[0033]

[0034] In summary, the translational dynamic equations in the airflow coordinate system can be obtained as follows:

[0035]

[0036] Considering the rotational dynamics equations about the aircraft's center of mass, in the body coordinate system, we have:

[0037]

[0038] Where L, M, and N are the torques about the three axes, and I is the moment of inertia, the above equation can be derived and rearranged to obtain:

[0039]

[0040] in:

[0041]

[0042]

[0043]

[0044]

[0045] Considering the kinematic equations of the aircraft's center of mass movement, differentiating both sides of the velocity expression with respect to time, and then transforming the equations to the geodetic coordinate system using a coordinate transformation matrix, we obtain the aircraft's navigation motion equations as follows:

[0046]

[0047] Considering the kinematic equations of rotation about the aircraft's center of mass, we can obtain the following from the relationship between the rotational angular velocity and Euler angles in the body coordinate system:

[0048]

[0049] By supplementing the calculation equations for aerodynamic forces and aerodynamic torques, the six-degree-of-freedom kinematic dynamics model of the aircraft based on the above formulas is closed. After closure, by giving the control quantities that can influence the forces and torques at each moment, the state of the aircraft at any moment can be solved, thereby realizing high-precision numerical simulation of the six-degree-of-freedom kinematic dynamics model of the aircraft.

[0050] In some preferred embodiments, the step of training a network model capable of simulating the behavioral characteristics and decision-making of a typical combat adversary using a generative adversarial imitation learning algorithm based on the red-blue adversarial dataset includes:

[0051] Random initialization strategy π θ It interacts with the air combat simulation environment and obtains the trajectory under the current strategy, which means that it uses a random maneuvering strategy to perform close-range air combat.

[0052] Randomly initialize the discriminator network D φ And distinguish samples from agents (s) t , a t s t+1 )~D π With samples from the said red-blue adversarial dataset {s t , a t s t+1 )~D e ;

[0053] A reward function is defined using the discriminator's prediction. In each round of training, the discriminator is first updated, and then the agent is trained using the RL algorithm to maximize this reward, thereby deceiving the discriminator.

[0054] Simultaneously, the learning process of the generator and discriminator is iterated until the loss function of the discriminator decreases to the minimum, the network model converges, and the air combat decision model is obtained.

[0055] In some preferred embodiments, the reward function is r(s, a) = -ln(1-D(s, a)), where D(s, a) represents the probability that the discriminator classifies the state-action pair as an expert.

[0056] In some preferred embodiments, when training the air combat decision model, behavioral constraint inputs applicable to the local machine are also included, which include kinematic and dynamic constraints and tactical action constraints.

[0057] Secondly, the present invention also provides an air combat intelligent decision-making system that combines imitation learning and reinforcement learning, including a user input module, a learning and training module, a data processing module, and a real-time adversarial decision-making module.

[0058] The user input module is used to receive typical adversary data and perform enhancement processing on the typical adversary data to obtain a red-blue confrontation dataset.

[0059] The learning and training module is used to train the network based on the red-blue adversarial dataset and obtain an air combat decision model.

[0060] The data processing module is used to receive information sent by the air combat simulation environment and process it into battlefield situation information adapted to the air combat decision model.

[0061] The real-time adversarial decision-making module is used to acquire the battlefield situation information and call the functions of the air combat decision-making model to process the battlefield situation information and obtain decision results to guide the machine.

[0062] The data processing module is also used to encode the decision results into decision control instructions and send them to the air combat simulation environment for execution by the local machine.

[0063] The beneficial effects of this invention using the above technical solution are as follows: The method of this invention can learn and imitate expert flight strategies from expert demonstration data, training an expert decision-making model that can make real-time decisions based on air combat conditions. This invention solves the modal collapse problem in the generative adversarial learning training process through algorithm design, ensuring the diversity and generalization of generated strategies; it also solves the problem of low sample utilization efficiency through data augmentation methods, allowing for better training results with fewer samples. Verification shows that the trained expert decision-making model still possesses robustness in complex dynamic environments. Attached Figure Description

[0064] Figure 1 This is a schematic diagram of the process of the present invention.

[0065] Figure 2 This is a flowchart of the present invention.

[0066] Figure 3 This is a schematic diagram illustrating the geometric description of one-to-one air combat in this invention.

[0067] Figure 4 This is a schematic diagram of the GAIL algorithm training process in this invention.

[0068] Figure 5 This is a schematic diagram of the framework structure of the GAIfO algorithm in this invention. Detailed Implementation

[0069] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings. It should be noted that these descriptions are for the purpose of aiding understanding the present invention, but do not constitute a limitation thereof. Furthermore, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0070] Example 1

[0071] like Figure 1-2 As shown, this embodiment of the invention provides an air combat decision-making method based on generative adversarial imitation learning, which includes the following steps:

[0072] Obtain battlefield situation information;

[0073] The pre-trained air combat decision model processes battlefield situation information to obtain decision results that guide the aircraft. These decision results include one or more of the aircraft's maneuvers, ACM mode, and missile firing actions.

[0074] In this embodiment, battlefield situation information includes the aircraft's coordinates, the flight status of both friendly and enemy forces, geometric situation, and the aircraft's payload status. Battlefield situation information is typically represented in a state space S.

[0075] Considering that the positions of the opposing sides change drastically during air combat, describing their geometric positional relationship with a static reference frame would increase the convergence difficulty during model training. This is because even if the positions and attitudes of the opposing sides are completely different under similar adversarial situations, relative state variables should be used as much as possible to describe the situational relationship between the opposing sides, thereby minimizing the dimension of the state space S.

[0076] Therefore, when constructing the state space S, from the perspective of the local machine (e.g., the pilot), we analyze the situational factors that the pilot relies on when making decisions. First, we establish a geometric coordinate system with the local machine as the origin, such as... Figure 3 As shown, from the perspective of the local machine as the main viewpoint, R is the enemy-friendly distance vector, and V... r V represents the aircraft's velocity vector. b Let q be the velocity vector of the blue team, and q be the target entry angle. The target azimuth angle.

[0077] Considering that aircraft typically have weak perception capabilities, it is difficult to obtain opponent status information effectively if they rely solely on their own radar and other equipment. This is not conducive to the training of intelligent decision-making networks. Therefore, it is assumed that aircraft can obtain opponent situation information completely through some means (such as information injection from the target range).

[0078] Based on the classification of state information during air combat, the state space S is extracted and structured as follows:

[0079] Table 1 - State Space S Construction

[0080]

[0081] In Table 1, the machine coordinates, flight states of both red and blue forces, and geometric situation are all continuous state quantities, while the payload state is a discrete state quantity. The chaff status has two options: whether the chaff is currently active or not. The missile status includes four states: unlocked, locked but not launched, launched but not hit, and launched and hit.

[0082] In this embodiment, the decision-making results used to guide the machine include one or more of the machine's maneuvers, ACM modes, and firing actions. Typically, maneuvers, ACM modes, and firing actions are represented by action space A, which is another key element in constructing a reinforcement learning environment.

[0083] An action refers to a decision made by an agent based on its current state, and the action space A refers to the set of all possible actions. The construction of the action space A must be complete and reasonable. Complete means that it should cover all means that the aircraft can use in air combat, and reasonable means that the action design must be within the prescribed constraints.

[0084] Furthermore, due to the complexity of air combat, the design of the action space A should be as streamlined as possible. Preliminary research revealed that a continuous action space A often performs poorly in air combat reinforcement learning training. Therefore, a discrete state space is constructed to minimize the dimensionality of the action space A while meeting basic control requirements. Specifically, the aircraft's decision control does not directly control its control surfaces. Instead, based on actual maneuverability, a maneuver library is constructed. During decision-making, the aircraft selects combinations of tactical actions, reducing the convergence difficulty of reinforcement learning training.

[0085] Based on the characteristics of aircraft maneuvers, the aircraft's maneuver space A is classified and designed as follows:

[0086] Table 2 - Construction of Action Space A

[0087]

[0088] In this embodiment, the air combat decision-making model is obtained through the following steps:

[0089] Obtain data on typical adversaries;

[0090] Enhanced processing of typical adversary data yields a red-blue force-on-force dataset.

[0091] By using the generative adversarial imitation learning algorithm, a network model capable of simulating the behavioral characteristics and decision-making of typical combat opponents is obtained by training on a red-blue adversarial dataset, namely, an air combat decision-making model.

[0092] The typical adversary data includes coarse-grained sparse expert strategy data and fine-grained dense expert strategy data.

[0093] Assume that typical adversary data is in the form of large-granularity descriptive statements, such as: "When the enemy is to our right rear, they will employ an S-shaped maneuver or a half-roll reversal maneuver." This data format provides, or can be converted into, labels for state-action pairs, distinguishing whether a state-action pair belongs to a subset of the state and action spaces provided by the descriptive statements in the intelligence, thus providing a foundation for supervised learning. Therefore, the information from coarse-grained sparse expert strategy data is extracted into the following form:

[0094] D Sparse ={(S1,A1),(S2,A2),...,(S n A n )}

[0095] In the formula, S i and A i These are the state space subset and action space subset corresponding to the i-th descriptive statement, respectively. The dataset D mentioned above... Sparse This represents coarse-grained sparse expert policy data representing the simulated objects.

[0096] When the data on typical adversaries is dense enough, it is processed into a trajectory dataset:

[0097] D Dense ={τ1, r2, ..., τ n}

[0098] In the formula, This refers to the i-th trajectory of the simulated object; the superscript i represents the i-th trajectory, and the subscripts 1 to m. i Represents the order in which things are sorted by time, for example... Represents the m-th trajectory sorted by time from the i-th trajectory. i Status information, Represents the mth i Individual action information. Dataset D Dense This represents dense expert policy data with fine granularity representing the simulated objects.

[0099] Coarse-grained fuzzy expert data, due to its explicit state-action mapping relationship, can be divided into two types: air combat rules and state-action pairs. For air combat rules, a decision tree model is constructed. For a state covered by intelligence information, if the intelligence indicates a unique maneuver, the corresponding branch is executed directly. If the intelligence indicates multiple responses or provides relatively vague guidance, a random branch under that node is executed. This decision tree is then used to generate a large number of new samples in the form of state-action pairs, which are then merged with the original state-action pair data to form a state-action pair dataset.

[0100] Based on this, in this embodiment, the red-blue adversarial dataset includes the following steps:

[0101] The air combat rules are expanded by using a decision tree model to generate new state-action pairs. These new state-action pairs are then merged with the state-action pairs contained in the coarse-grained sparse expert policy data to obtain a state-action pair dataset, i.e., the first dataset.

[0102] A behavior cloning network is obtained by training on fine-grained dense expert policy data using the behavior cloning algorithm.

[0103] A second dataset is obtained by augmenting the fine-grained, dense expert policy data using a behavioral cloning network.

[0104] The union of the first dataset and the second dataset constitutes the aforementioned red-blue adversarial dataset.

[0105] The red-blue adversarial dataset is randomly divided into a training set and a test set in a 9:1 ratio. The training set is used to train the air combat decision model, and the test set is used to test the trained air combat decision model.

[0106] In this embodiment, as Figure 4 As shown, the specific steps for training a network model capable of simulating the behavioral characteristics and decision-making of a typical combat opponent using a generative adversarial imitation learning algorithm based on a red-blue adversarial dataset include:

[0107] Random initialization strategy π θ It interacts with the air combat simulation environment and obtains the trajectory under the current strategy, which means that it uses a random maneuvering strategy to perform close-range air combat.

[0108] Randomly initialize the discriminator network D φ And distinguish samples from agents (s) t , a t s t+1 )~D π Samples from the Red-Blue Adversarial Dataset (s) t , a t s t+1 )~De ;

[0109] The discriminator's prediction is used to define a reward function, which is r(s,a) = -ln(1-D(s,a)), where D(s,a) represents the probability that the discriminator classifies the state-action pair as an expert. In each round of training, the discriminator is first updated, and then the agent is trained using the RL algorithm to maximize this reward, thereby deceiving the discriminator.

[0110] Simultaneously, the learning process of the generator and discriminator is iterated until the loss function of the discriminator decreases to the minimum, the network model converges, and the air combat decision model is obtained.

[0111] Similar to Generative Adversarial Networks (GANs), the training of the discriminator and the agent is alternating. The generator employs DenseNet to enhance the model's expressive power and reduce the likelihood of mode collapse. Simultaneously, adjusting the learning rates of the generator and discriminator helps the model learn better and avoid getting trapped in local optima. Real-time interactive air combat data and the network itself are stored during training. Training is complete when the expert decision-making network trained through generative adversarial imitation converges.

[0112] In terms of the overall process, GAIL follows these steps:

[0113] (1) Use the current strategy to interact with the air combat simulation environment and store the air combat trajectory of this round in the experience buffer.

[0114] (2) Update the discriminator.

[0115] (3) Update the policy network.

[0116] As shown in Table 3, the generative adversarial imitation learning algorithm flow is as follows:

[0117] Table 3 - GAIL Algorithm Pseudocode Table

[0118]

[0119] The GAIL algorithm described above is applicable to trajectory datasets consisting of state-action pairs. As analyzed above, in reality, expert data may not necessarily contain the action space A, but may only contain partial observations, such as flight trajectory information. In this case, the IL problem is transformed into the ILfO (imitation learning from observation) problem. The GAIL algorithm approach is still used, and the problem to be solved is transformed into the following optimization problem:

[0120]

[0121] The state transition pair (s, s′) represents the changes in the air combat situation at consecutive moments. It can be observed that under the GAN architecture, only the loss function in the GAIL algorithm needs to be modified to meet the problem requirements. For example... Figure 5 As shown, this visually illustrates the functional composition of the generator and discriminator when the expert data only contains state observations.

[0122] The left side is the discriminator, which scores the input continuous state trajectory. The right side is the generator, which first generates the maneuver action 'a' under the current state 's' according to the policy 'π'. After iteration in the air combat simulation environment, the generator obtains the state 's', which is then input into the discriminator for scoring. The two perform a zero-sum game to achieve the optimal outcome for both sides. The algorithm steps are as follows:

[0123] Table 4 - Pseudocode of the GAIfO Algorithm

[0124]

[0125] It can be seen that the learning processes of the generator (i.e., the decision network) and the discriminator (i.e., the implicit reward function) are iterated simultaneously. When the loss function of the discriminator drops to 0, that is, when the discriminator outputs 0.5, it indicates that the discriminator can no longer distinguish whether the strategy generated by the decision network comes from intelligence data. At this time, the value function of the decision network must reach its maximum value, indicating that the aircraft agent has fully learned the tactics, methods and behavioral characteristics of the typical combat opponent, and the task of the imitation learning stage is completed.

[0126] It is easy to understand that different types of drones can perform different maneuvers. Therefore, when training the air combat decision model, a corresponding shielding mechanism is designed, which is to adapt the behavioral constraint input to the drone itself.

[0127] Behavioral constraints inputs include two parts: kinematic and dynamic constraints, and tactical maneuver constraints. Kinematic and dynamic constraints mainly consist of various extreme performance values ​​for the corresponding UAV model, including extreme pitch and roll angles limited by the aircraft's structural strength, extreme normal and tangential overloads, maximum and minimum flight speeds, minimum and maximum altitudes, etc. Tactical maneuver constraints, based on the maneuver space design, mask maneuvers as needed.

[0128] Typically, the training process for air combat decision-making models is based on an air combat simulation environment, which is constructed through the following steps:

[0129] First, a six-degree-of-freedom kinematic dynamics model of the aircraft is constructed. This model is established in the body coordinate system and the airflow coordinate system, considering the dynamic equations of the aircraft's center of mass movement. With the aircraft's center of mass as the origin, in the body coordinate system, we have:

[0130]

[0131] Among them, F x F y F z , q, and r are the projections of the net external force acting on the aircraft onto the three axes of the body coordinate system, respectively; p, q, and r are the projections of the aircraft angular velocity onto the three axes of the body coordinate system, respectively; and u, v, and w are the components of the aircraft airspeed in the body coordinate system, respectively.

[0132] External forces acting on the center of mass include engine thrust T and aerodynamic force F. a The projection of gravity G, the net external force, onto the three axes of the body coordinate system can be specifically expressed as:

[0133]

[0134] Where L, D, and Y represent the projections of aerodynamic forces onto the three axes of the airflow coordinate system, namely lift, drag, and side force, respectively; α and β are the angle of attack and sideslip angle, respectively; and φ, θ, and ψ are the attitude angles in the body coordinate system.

[0135] Expressing the velocity in the body coordinate system using V, α, and β in the airflow system yields:

[0136]

[0137] In summary, the translational dynamic equations in the airflow coordinate system can be obtained as follows:

[0138]

[0139] Considering the rotational dynamics equations about the aircraft's center of mass, in the body coordinate system, we have:

[0140]

[0141] Where L, M, and N are the torques about the three axes, and I is the moment of inertia, the above equation can be derived and rearranged to obtain:

[0142]

[0143] in:

[0144]

[0145]

[0146]

[0147]

[0148] Considering the kinematic equations of the aircraft's center of mass movement, differentiating both sides of the velocity expression with respect to time, and then transforming the equations to the geodetic coordinate system using a coordinate transformation matrix, we obtain the aircraft's navigation motion equations as follows:

[0149]

[0150] Considering the kinematic equations of rotation about the aircraft's center of mass, we can obtain the following from the relationship between the rotational angular velocity and Euler angles in the body coordinate system:

[0151]

[0152] The above formulas form the kinematics model of a six-degree-of-freedom fixed-wing aircraft. However, the above equations are not closed and need to be supplemented with equations for calculating aerodynamic forces and aerodynamic moments to complete the closure. A common method is to calculate them using aerodynamic coefficients. Here, we use aerodynamic parameter data of a typical combat target for modeling.

[0153] Once the equations are closed, as long as the control quantities that can influence the force and torque at each moment, such as the control surface deflection angle and engine thrust, are given, the state of the aircraft at any moment can be solved. The nonlinear model composed of the above differential equations can be solved by numerical iterative algorithms such as the fourth-order Runge-Kutta algorithm, thereby achieving high-precision numerical simulation of the model.

[0154] Example 2

[0155] This embodiment also provides an air combat decision-making system based on generative adversarial imitation learning, including a user input module, a learning and training module, a data processing module, and a real-time adversarial decision-making module.

[0156] The user input module is used to receive typical adversary data and perform augmentation processing on the typical adversary data to obtain a red-blue confrontation dataset.

[0157] The learning and training module is used to train the network based on the red-blue adversarial dataset and obtain an air combat decision model;

[0158] The data processing module is used to receive information sent by the air combat simulation environment and process it into battlefield situation information that is adapted to the air combat decision-making model.

[0159] The real-time adversarial decision-making module is used to acquire battlefield situation information and call the functions of the air combat decision-making model to process the battlefield situation information and obtain decision results to guide the machine.

[0160] The data processing module is also used to encode the decision results into decision control commands and send them to the air combat simulation environment for execution by the local machine.

[0161] The embodiments of the present invention have been described in detail above with reference to the accompanying drawings, but the present invention is not limited to the described embodiments. For those skilled in the art, various changes, modifications, substitutions, and variations can be made to these embodiments without departing from the principles and spirit of the present invention, and these variations still fall within the protection scope of the present invention.

Claims

1. An air combat decision-making method based on generative adversarial imitation learning, characterized in that, The method includes: acquiring battlefield situation information, processing the battlefield situation information through a pre-trained air combat decision model, and obtaining decision results for guiding the aircraft, wherein the decision results include one or more of the aircraft's maneuvering actions, ACM mode, and missile firing actions. The air combat decision-making model is obtained through the following steps: Data on typical adversaries is acquired, and the data on typical adversaries is enhanced to obtain a red-blue confrontation dataset. Using a generative adversarial imitation learning algorithm, a network model capable of simulating the behavioral characteristics and decision-making of a typical combat opponent is trained based on the red-blue adversarial dataset, namely the air combat decision model. The typical adversary data includes coarse-grained sparse expert strategy data and fine-grained dense expert strategy data. The coarse-grained sparse expert strategy data includes air combat rules and state-action pairs. The step of augmenting the typical adversary data to obtain the red-blue adversary dataset includes: The air combat rules are expanded using a decision tree model to generate new state-action pairs, which are then merged with the state-action pairs contained in the coarse-grained sparse expert strategy data to obtain the first dataset. Using the behavior cloning algorithm, a behavior cloning network is trained based on the fine-grained dense expert policy data to obtain a behavior cloning network; the behavior cloning network is then used to augment the fine-grained dense expert policy data to obtain a second dataset. The union of the first dataset and the second dataset constitutes the red-blue adversarial dataset; The data for coarse-grained sparse expert strategies are as follows: ； In the formula, and The first The state space subset and action space subset corresponding to each descriptive statement; Fine-grained, dense expert strategy data is as follows: ； In the formula, For the simulated object's first Trajectory; superscript Representing the Trajectory, index 1~ m i Represents the order in which things are sorted by time, for example... Representing the i The first track sorted by time m i Status information, Representing the m i Individual action information.

2. The method according to claim 1, characterized in that: Battlefield situation information includes the aircraft's coordinates, the flight status of both friendly and enemy forces, geometric situation, and the aircraft's payload status.

3. The method according to claim 2, characterized in that: The local coordinates include east-west coordinates, north-south coordinates, and flight altitude. The flight status of both sides includes airspeed, indicated airspeed, Mach number, attitude quaternion, and engagement time. The geometric situation includes enemy-friendly distance vector, enemy-friendly distance scalar, enemy-friendly velocity vector, enemy-friendly altitude difference, entry angle, and azimuth angle. The payload status includes chaff status, remaining chaff quantity, missile status, and remaining missile quantity.

4. The method according to claim 1, characterized in that: The maneuvers include level flight, level flight acceleration and deceleration, climb, turn, dive, lateral cut, half roll inversion, S-maneuver, and oblique loop. The ACM modes include horizontal sweep mode selection, vertical sweep mode selection, fixed axis mode selection, and seeker unlock. The missile firing actions include infrared missile launch, chaff launch, and infrared chaff launch.

5. The method according to claim 1, characterized in that: The training process of the air combat decision-making model is based on an air combat simulation environment, which is constructed through the following steps: First, a six-degree-of-freedom kinematic dynamics model of the aircraft is constructed. This model is established in the body coordinate system and the airflow coordinate system, considering the dynamic equations of the aircraft's center of mass movement. With the aircraft's center of mass as the origin, in the body coordinate system, we have: ； in, , , These are the projections of the net external forces acting on the aircraft onto the three axes of the aircraft coordinate system. , , These are the projections of the aircraft's angular velocity onto the three axes of the body coordinate system. , , These are the components of the aircraft's airspeed in the body coordinate system; External forces acting on the center of mass include engine thrust. Aerodynamics ,gravity The projection of the net external force onto the three axes of the body coordinate system can be specifically expressed as: ； in, , , This represents the projection of aerodynamic forces onto the three axes of the airflow coordinate system, namely lift, drag, and lateral force. , For angle of attack and sideslip angle, , , The attitude angle in the body coordinate system; The velocity in the body coordinate system is expressed as the velocity in the airflow system. , , We can express this as: ； In summary, the translational dynamic equations in the airflow coordinate system can be obtained as follows: ； Considering the rotational dynamics equations about the aircraft's center of mass, in the body coordinate system, we have: ； in L , M , N For the torque about the three axes, I To determine the moment of inertia, the above equation can be derived and rearranged to obtain: ； in: ； Considering the kinematic equations of the aircraft's center of mass movement, differentiating both sides of the velocity expression with respect to time, and then transforming the equations to the geodetic coordinate system using a coordinate transformation matrix, we obtain the aircraft's navigation motion equations as follows: ； Considering the kinematic equations of rotation about the aircraft's center of mass, we can obtain the following from the relationship between the rotational angular velocity and Euler angles in the body coordinate system: ； By supplementing the calculation equations for aerodynamic forces and aerodynamic torques, the six-degree-of-freedom kinematic dynamics model of the aircraft based on the above formulas is closed. After closure, by giving the control quantities that can influence the forces and torques at each moment, the state of the aircraft at any moment can be solved, thereby realizing high-precision numerical simulation of the six-degree-of-freedom kinematic dynamics model of the aircraft.

6. The method according to claim 1, characterized in that: The step of training a network model capable of simulating the behavioral characteristics and decision-making of a typical combat opponent using a generative adversarial imitation learning algorithm based on the red-blue adversarial dataset includes: Random initialization strategy It interacts with the air combat simulation environment and obtains the trajectory under the current strategy, which means that it uses a random maneuvering strategy to perform close-range air combat. Randomly initialize the discriminator network And distinguish samples from agents. Sampling from the aforementioned Red-Blue Adversarial Dataset ; A reward function is defined using the discriminator's prediction. In each round of training, the discriminator is first updated, and then the agent is trained using the RL algorithm to maximize this reward, thereby deceiving the discriminator. Simultaneously, the learning process of the generator and discriminator is iterated until the loss function of the discriminator decreases to the minimum, the network model converges, and the air combat decision model is obtained.

7. The method according to claim 6, characterized in that: The reward function is: , This represents the probability that the discriminator classifies the state-action pair as an expert.

8. The method according to claim 1, characterized in that: The training of the air combat decision model also includes behavioral constraint inputs applicable to the machine, which include kinematic and dynamic constraints as well as tactical action constraints.

9. A system for an air combat decision-making method based on generative adversarial imitation learning as described in any one of claims 1-8, characterized in that: It includes a user input module, a learning and training module, a data processing module, and a real-time adversarial decision-making module; The user input module is used to receive typical adversary data and perform enhancement processing on the typical adversary data to obtain a red-blue confrontation dataset. The learning and training module is used to train the network based on the red-blue adversarial dataset and obtain an air combat decision model. The data processing module is used to receive information sent by the air combat simulation environment and process it into battlefield situation information adapted to the air combat decision model. The real-time adversarial decision-making module is used to acquire the battlefield situation information and call the functions of the air combat decision-making model to process the battlefield situation information and obtain decision results to guide the machine. The data processing module is also used to encode the decision results into decision control instructions and send them to the air combat simulation environment for execution by the local machine.

Images