Air combat decision-making method and system based on generative adversarial imitation learning

Abstract

The invention discloses an air combat decision-making method and system based on generative adversarial imitation learning, and belongs to the technical field of aviation, and the method comprises the steps: obtaining battlefield situation information, carrying out the processing of the battlefield situation information through a pre-trained air combat decision-making model, and obtaining a decision-making result used for guiding a local machine. The air combat decision model is obtained through the following steps of obtaining typical combat opponent data, performing enhancement processing on the typical combat opponent data, and obtaining a red-blue confrontation data set; and training based on the red-blue adversarial data set by utilizing a generative adversarial imitation learning algorithm to obtain an air combat decision model. According to the method, through algorithm design, the problem of modal collapse in the generative adversarial imitation learning training process is solved, and diversity and generalization of generated strategies can be ensured; the problem of low sample utilization efficiency is solved through the data enhancement method, and a better training result can be obtained based on less sample training; and the method still has decision robustness in a complex dynamic environment.

Description

Technical Field

[0001] The present invention relates to the field of aviation technology, and in particular to an air combat decision-making method and system based on generative adversarial imitation learning. Background Art

[0002] In the field of flight control, learning and imitating expert flight strategies has always been an important research direction. Traditional flight strategy learning methods mainly rely on expert experience and manual debugging, which is not only time-consuming and labor-intensive, but also difficult to ensure the generalization ability and robustness of the strategy.

[0003] In recent years, with the rapid development of artificial intelligence technology, imitation learning, as a method of learning strategies from expert demonstration data, has received widespread attention.

[0004] Imitation learning can be divided into two types of methods: behavioral cloning and imitation learning based on inverse reinforcement learning (IRL). The behavioral cloning method learns the strategy by directly cloning the actions of the expert in each state, but this method is easily affected by cascading errors, resulting in poor robustness and generalization of the strategy. The method based on inverse reinforcement learning learns the strategy by deriving the reward function behind the expert strategy. Although this method solves the cascading error problem to a certain extent, it has high computational cost and is difficult to apply to large-scale problems.

[0005] Generative Adversarial Imitation Learning (GAIL), as a method that combines Generative Adversarial Networks (GANs) and imitation learning, can learn strategies directly from expert demonstration data without relying on an explicit reward function.

[0006] GAIL uses adversarial training between the generator and the discriminator to enable the generator to generate behavior trajectories similar to the expert strategy, thereby imitating the expert strategy. Compared with traditional imitation learning methods, GAIL has stronger representation capabilities and computational efficiency, and can handle complex large-scale problems.

[0007] However, the GAIL method also has some problems:

[0008] 1. Insufficient diversity and generalization ability: In flight missions, different flight environments and mission requirements may lead to the diversity of expert strategies. GAIL may experience mode collapse during training, that is, the generated strategies lack diversity and cannot cover the entire distribution of expert strategies, resulting in insufficient generalization ability of the strategies.

[0009] 2. Low sample utilization efficiency: The data collection cost in flight missions is high. During the training process, GAIL requires a large number of interaction samples to estimate the expected cumulative reward of the strategy, resulting in low sample utilization efficiency and increased training costs.

[0010] 3. Robustness in complex environments: Flight missions usually involve complex dynamic environments and multiple uncertainties. How to ensure the robustness and stability of strategies in such environments is an urgent problem to be solved. Summary of the invention

[0011] The present invention provides an air combat decision-making method and system based on generative adversarial imitation learning, so as to at least partially solve the above-mentioned problems.

[0012] To achieve the above object, the technical solution of the present invention is:

[0013] In a first aspect, the present invention provides an air combat decision-making method based on generative adversarial imitation learning, the method comprising: obtaining battlefield situation information, processing the battlefield situation information through a pre-trained air combat decision-making model, and obtaining a decision result for guiding the aircraft, the decision result comprising one or more of the aircraft's maneuvering action, ACM mode, and bullet shooting action;

[0014] The air combat decision model is obtained by the following steps:

[0015] Acquire typical combat opponent data, perform enhancement processing on the typical combat opponent data, and obtain a red-blue confrontation data set;

[0016] By utilizing a generative adversarial imitation learning algorithm, a network model capable of simulating the behavioral characteristics and decisions of a typical combat opponent, namely, the air combat decision-making model, is obtained based on the red-blue adversarial data set training.

[0017] In some preferred embodiments, the typical combat opponent data includes coarse-grained sparse expert strategy data and fine-grained dense expert strategy data, and the coarse-grained sparse expert strategy data includes air combat rules and state-action pairs;

[0018] Then the step of performing enhancement processing on the typical combat opponent data to obtain the red-blue confrontation data set includes:

[0019] Expanding the air combat rules through a decision tree model and generating new state-action pairs, and merging the new state-action pairs with the state-action pairs included in the coarse-grained sparse expert strategy data to obtain a first data set;

[0020] Using a behavior cloning algorithm, a behavior cloning network is obtained based on the fine-grained dense expert strategy data training; data expansion is performed on the fine-grained dense expert strategy data through the behavior cloning network to obtain a second data set;

[0021] The union of the first data set and the second data set constitutes the red-blue adversarial data set.

[0022] In some preferred embodiments, the battlefield situation information includes the aircraft's coordinates, the flight status of both the enemy and our own aircraft, the geometric situation, and the aircraft's load status.

[0023] In some preferred embodiments, the aircraft coordinates include east-west coordinates, north-south coordinates and flight altitude, the flight states of both the enemy and our own side include flight airspeed, flight speed indication, Mach number, attitude quaternion and confrontation time, the geometric situation includes enemy and our own side distance vector, enemy and our own side distance scalar, enemy and our own side speed vector, enemy and our own side height difference, entry angle and azimuth, and the load state includes jammer state, remaining number of jammers, missile state and remaining number of missiles.

[0024] In some preferred embodiments, the maneuvering actions include level flight, level flight acceleration and deceleration, climb, turn, dive, traverse, half roll, S maneuver and oblique somersault; the ACM mode includes level sweep mode selection, vertical sweep mode selection, fixed axis mode selection and seeker unlocking; the firing actions include infrared missile launch, chaff bomb launch and infrared jamming bomb launch.

[0025] In some preferred embodiments, the training process of the air combat decision model is performed based on an air combat simulation environment, and the air combat simulation environment is constructed by the following steps:

[0026] First, a six-degree-of-freedom kinematic dynamics model of the aircraft is constructed. The model is established in the body coordinate system and the airflow coordinate system. The dynamic equation of the movement of the center of mass of the aircraft is considered. With the center of mass of the aircraft as the origin, in the body coordinate system, there is:

[0027]

[0028] Among them, F x 、F y 、F z are the projections of the resultant external forces acting on the aircraft on the three axes of the body coordinate system, p, q, r are the projections of the aircraft angular velocity on the three axes of the body coordinate system, u, v, w are the components of the aircraft airspeed in the body coordinate system;

[0029] The external forces acting on the center of mass include engine thrust T, aerodynamic force F a , gravity G, the projection of the resultant external force on the three axes of the body coordinate system can be specifically expressed as:

[0030]

[0031] Among them, L, D, and Y represent the projection of aerodynamic forces on the three axes of the airflow coordinate system, which are lift, drag, and side force, respectively; α and β are the angle of attack and sideslip angle; φ, θ, and ψ are the attitude angles in the body coordinate system;

[0032] The velocity in the body coordinate system is expressed by V, α, and β in the airflow system:

[0033]

[0034] Based on the above, the translational dynamics equation in the airflow coordinate system is as follows:

[0035]

[0036] Considering the rotational dynamics equation around the center of mass of the aircraft, in the body coordinate system, we have:

[0037]

[0038] Among them, L, M, N are the moments around the three axes, and I is the moment of inertia. By deriving and rearranging the above formula, we can get:

[0039]

[0040] in:

[0041]

[0042]

[0043]

[0044]

[0045] Considering the kinematic equation of the movement of the center of mass of the aircraft, taking the derivative of the left and right sides of the velocity expression with respect to time, and then transforming the equation to the geodetic coordinate system through the coordinate transformation matrix, the navigation motion equation of the aircraft is obtained as follows:

[0046]

[0047] Considering the kinematic equation of rotation around the center of mass of the aircraft, the relationship between the angular velocity of rotation and the Euler angle in the body coordinate system can be obtained:

[0048]

[0049] The six-degree-of-freedom kinematic dynamics model of the aircraft based on the above formula is closed by supplementing the calculation equations of aerodynamic force and aerodynamic torque. After closing, the state of the aircraft at any time can be solved by giving the control quantity that can affect the force and torque at each moment, thereby realizing high-precision numerical simulation of the six-degree-of-freedom kinematic dynamics model of the aircraft.

[0050] In some preferred embodiments, the step of using a generative adversarial imitation learning algorithm to train a network model based on the red-blue adversarial dataset to simulate the behavioral characteristics and decisions of a typical combat opponent includes:

[0051] Random initialization strategy π θ And interact with the air combat simulation environment and obtain the trajectory under the current strategy, that is, to perform close-range air combat using a random maneuver strategy;

[0052] Randomly initialize the discriminator network D φ , and distinguish the samples from the agent (s t , a t ,s t+1 )~D π Compared with the samples from the red-blue adversarial dataset {s t , a t ,s t+1 )~D e ;

[0053] Use the discriminator’s predictions to define a reward function. In each round of training, first update the discriminator, and then use the RL algorithm to train the agent to maximize this reward, thereby deceiving the discriminator.

[0054] The learning process of the generator and the discriminator is iterated at the same time until the loss function of the discriminator drops to the minimum and the network model converges, thereby obtaining the air combat decision model.

[0055] In some preferred embodiments, the reward function is r(s, a)=-ln(1-D(s, a)), where D(s, a) represents the probability that the discriminator classifies a state-action pair as an expert.

[0056] In some preferred embodiments, when training the air combat decision model, it also includes behavioral constraint input applicable to the aircraft, and the behavioral constraint input includes kinematic and dynamic constraints and tactical action constraints.

[0057] In a second aspect, the present invention also provides an air combat intelligent decision-making system combining imitation learning and reinforcement learning, including a user input module, a learning and training module, a data processing module and a real-time confrontation decision-making module;

[0058] The user input module is used to receive typical combat opponent data and perform enhancement processing on the typical combat opponent data to obtain a red-blue confrontation data set;

[0059] The learning and training module is used to perform network training according to the red-blue confrontation data set and obtain an air combat decision model;

[0060] The data processing module is used to receive information sent by the air combat simulation environment and process it into battlefield situation information adapted to the air combat decision model;

[0061] The real-time confrontation decision module is used to obtain the battlefield situation information, and call the function of the air combat decision model to process the battlefield situation information to obtain a decision result for guiding the aircraft;

[0062] The data processing module is also used to encode the decision results into decision control instructions and send them to the air combat simulation environment for execution by the local machine.

[0063] By adopting the above technical solution, the beneficial effect of the present invention is that the method of the present invention can realize the learning and imitation of the expert flight strategy in the expert demonstration data, and the expert decision model is trained to make real-time decisions for air combat status. The invention solves the problem of modal collapse in the training process of generative adversarial imitation learning through algorithm design, and can ensure that the generated strategies are diverse and generalized; the problem of low sample utilization efficiency is solved through data enhancement methods, and better training results can be obtained based on fewer sample training. It has been verified that the expert decision model obtained by training still has the robustness of decision-making in complex dynamic environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 It is a schematic diagram of the process of the present invention.

[0065] Figure 2 It is a flowchart of the present invention.

[0066] Figure 3 It is a schematic diagram of the geometric description of one-to-one air combat in the present invention.

[0067] Figure 4 Schematic diagram of the GAIL algorithm training process in the present invention.

[0068] Figure 5 It is a schematic diagram of the framework structure of the GAIfO algorithm in the present invention. DETAILED DESCRIPTION

[0069] The specific embodiments of the present invention are further described below in conjunction with the accompanying drawings. It should be noted that the description of these embodiments is used to help understand the present invention, but does not constitute a limitation of the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0070] Example 1

[0071] like Figure 1-2 As shown, an embodiment of the present invention provides an air combat decision-making method based on generative adversarial imitation learning, and the method includes the following steps:

[0072] Obtain battlefield situation information;

[0073] The battlefield situation information is processed through a pre-trained air combat decision model to obtain a decision result for guiding the aircraft, wherein the decision result includes one or more of the aircraft's maneuvering action, ACM mode, and firing action.

[0074] In this embodiment, the battlefield situation information includes the coordinates of the aircraft, the flight status of the enemy and the aircraft, the geometric situation and the load status of the aircraft. The battlefield situation information is usually represented by the state space S.

[0075] Taking into account the dramatic changes in the positions of the two opposing sides during air combat, if the geometric position relationship between the two sides is described by a stationary reference system, the convergence difficulty in the model training process will increase. This is because even if the positions and postures of the two opposing sides are in a similar confrontation situation, they may be completely different. Therefore, relative state variables should be used as much as possible to describe the situation relationship between the two opposing sides, so as to reduce the dimension of the state space S as much as possible.

[0076] Therefore, when constructing the state space S, we start from the perspective of the aircraft (for example, the pilot of the aircraft), analyze the situational factors that the pilot relies on when making decisions, and first establish a geometric coordinate system with the aircraft as the origin, such as Figure 3 As shown, the aircraft is the main perspective, R is the enemy distance vector, V r represents the aircraft velocity vector, V b is the velocity vector of the blue party, q is the target entry angle, is the target azimuth.

[0077] Considering that the aircraft's perception ability is usually weak, it is difficult to obtain the opponent's status information if it only relies on its own radar and other equipment, which is not conducive to the training of intelligent decision-making networks. Therefore, it is assumed that the aircraft can fully obtain the opponent's situation information through some means (such as information injection from the shooting range).

[0078] According to the classification of the state information in the air combat, the state space S is extracted and composed as follows:

[0079] Table 1 - State space S construction

[0080]

[0081] In Table 1, the coordinates of the aircraft, the flight status of the red and blue sides, and the geometric situation are all continuous state quantities, and the load state is a discrete state quantity. The jammer state is whether the jammer is currently effective. The missile state includes four states: unlocked, locked but not launched, launched but not hit, and launched and hit.

[0082] In this embodiment, the decision result used to guide the machine includes one or more of the machine's maneuvering action, ACM mode, and bullet shooting action. Usually, the maneuvering action, ACM mode, and bullet shooting action are represented by action space A, which is another major element in constructing a reinforcement learning environment.

[0083] Action refers to the decision made by the agent based on the current state, and action space A refers to the set of all possible actions. The construction of action space A must be complete and reasonable. Complete means that it should cover all means that the aircraft can use during air combat confrontation, and reasonable means that the action design must be within the prescribed constraints.

[0084] In addition, due to the complexity of air combat confrontation problems, the design of action space A should be as simple as possible. In the previous research, it was found that the continuous action space A often performs poorly in air combat reinforcement learning training. Therefore, a discrete state space is constructed to reduce the dimension of action space A as much as possible while meeting basic control requirements. The specific approach is that the decision control of the aircraft does not directly control its control surface, but builds a maneuvering action library of the aircraft based on its actual maneuverability. When making decisions, the aircraft makes a combination of tactical actions to reduce the convergence difficulty of reinforcement learning training.

[0085] According to the characteristics of the aircraft's movements, the aircraft's movement space A is classified and designed, and constructed as follows:

[0086] Table 2 - Action space A construction

[0087]

[0088] In this embodiment, the air combat decision model is obtained by the following steps:

[0089] Obtain data on typical combat opponents;

[0090] Enhance the data of typical combat opponents to obtain the red-blue confrontation data set;

[0091] By using the generative adversarial imitation learning algorithm and training based on the red-blue adversarial dataset, a network model capable of simulating the behavioral characteristics and decisions of typical combat opponents, namely the air combat decision-making model, is obtained.

[0092] Among them, typical combat opponent data includes coarse-grained sparse expert strategy data and fine-grained dense expert strategy data.

[0093] Assume that the data of typical combat opponents are in the form of descriptive sentences with large granularity. For example, the descriptive sentence is: when the state is that the enemy is behind us to the right, we will take an S-shaped maneuver or a half-roll maneuver. This data form provides or can be converted into a label of a state-action pair, that is, it can distinguish whether the state-action pair belongs to a subset pair of the state space and action space provided by the descriptive sentence in the intelligence, which provides a basis for supervised learning. Therefore, the information of the coarse-grained sparse expert strategy data is extracted in the following form:

[0094] D Sparse ={(S1,A1),(S2,A2),...,(S n , A n )}

[0095] In the formula, S i and A i are the state space subset and action space subset corresponding to the i-th descriptive sentence respectively. The above dataset D Sparse That is, it represents the coarse-grained sparse expert strategy data of the simulation object.

[0096] When the typical combat opponent data is dense enough, it is processed into a trajectory dataset:

[0097] D Dense ={τ1, r2, ..., τ n}

[0098] In the formula, is the ith trajectory of the simulation object; the superscript i represents the ith trajectory, and the subscripts are 1 to m i Represents a time-ordered sequence, such as Represents the mth track in the i-th track sorted by time i Status information, Represents the mth i Action information. Dataset D Dense That is, it represents the fine-grained dense expert strategy data of the simulation object.

[0099] Since the coarse-grained fuzzy expert data has a clear state-action mapping relationship, it can be divided into two types: air combat rules and state-action pairs. If it is an air combat rule, a decision tree model is constructed. For the state covered in the intelligence information, if the intelligence points out a unique maneuvering action method, it will directly enter the branch for execution. If the intelligence points out multiple response methods or gives a more vague guiding opinion, the random branch under the node will be executed. Then this decision tree is used to generate a large number of new samples in the form of state-action pairs, which are merged with the original state-action pair data to form a state-action pair data set.

[0100] On this basis, in this embodiment, the red-blue adversarial data set includes the following steps:

[0101] The air combat rules are expanded through a decision tree model to generate new state-action pairs, which are then merged with the state-action pairs contained in the coarse-grained sparse expert strategy data to obtain a state-action pair data set, i.e., a first data set;

[0102] Using the behavior cloning algorithm, the behavior cloning network is obtained based on fine-grained dense expert strategy data training;

[0103] The fine-grained dense expert strategy data is augmented through the behavior cloning network to obtain the second data set;

[0104] The union of the first data set and the second data set constitutes the above-mentioned red-blue adversarial data set.

[0105] The red-blue confrontation dataset is randomly divided into a training set and a test set in a ratio of 9:1. The training set is used to train the air combat decision-making model, and the test set is used to test the air combat decision-making model obtained through training.

[0106] In this embodiment, Figure 4 As shown in FIG. 1 , the steps of using the generative adversarial imitation learning algorithm to train a network model based on the red-blue adversarial dataset to simulate the behavioral characteristics and decision-making of a typical combat opponent include:

[0107] Random initialization strategy π θ And interact with the air combat simulation environment and obtain the trajectory under the current strategy, that is, to perform close-range air combat using a random maneuver strategy;

[0108] Randomly initialize the discriminator network D φ , and distinguish the samples from the agent (s t , a t ,s t+1 )~D π Compared with the samples from the red-blue adversarial dataset (s t , a t ,s t+1 )~De ;

[0109] Use the discriminator's prediction to define a reward function, the reward function is r(s, a) = -ln(1-D(s, a)), D(s, a) represents the probability that the discriminator classifies the state-action pair as an expert. In each round of training, the discriminator is first updated, and then the RL algorithm is used to train the agent to maximize this reward, thereby deceiving the discriminator.

[0110] The learning process of the generator and the discriminator is iterated at the same time until the loss function of the discriminator drops to the minimum and the network model converges, thereby obtaining the air combat decision model.

[0111] Like Generative Adversarial Networks (GAN), the training of the discriminator and the agent is alternating. The generator uses DenseNet to improve the expressiveness of the model and reduce the possibility of modal collapse. At the same time, by adjusting the learning rates of the generator and discriminator, the model is assisted to learn better and avoid falling into the local optimal solution. The real-time interactive air combat data and network during the training process are stored. When the expert decision network trained by generative adversarial imitation learning converges, the training is completed.

[0112] In terms of the overall process, GAIL follows the following steps:

[0113] (1) Use the current strategy to interact with the air combat simulation environment and store the trajectory of this round of air combat in the experience buffer.

[0114] (2) Update the discriminator.

[0115] (3) Update the strategy network.

[0116] As shown in Table 3, the process of generating adversarial imitation learning algorithm is as follows:

[0117] Table 3 - GAIL algorithm pseudo code table

[0118]

[0119] The above GAIL algorithm is applicable to trajectory data sets consisting of state-action pairs. As analyzed above, in reality, expert data may not necessarily contain the action space A, but may only be partial observations, such as only flight track information. In this case, the IL problem is converted to the ILfO (imitation learning from observation, ILfO) problem. At this time, the GAIL algorithm is still used, and the problem to be solved is converted into the following optimization problem:

[0120]

[0121] Among them, the state transition pair (s, s′) can represent the changes in the air combat situation at consecutive moments in the air combat. It can be found that under the GAN architecture, only the loss function in the GAIL algorithm needs to be modified to meet the problem requirements. Figure 5 As shown, the functional composition of the generator and the discriminator is intuitively described when the expert data only contains state observations.

[0122] The left side is the discriminator, which scores the input continuous state trajectory, and the right side is the generator. The generator first generates the maneuvering execution action a under the current state s according to the strategy π, and obtains the state s′ after iteration in the air combat simulation environment, and then inputs it into the discriminator for scoring. The two perform a zero-sum game to achieve the best result for both parties. The algorithm steps are as follows:

[0123] Table 4 - GAIfO algorithm pseudo code table

[0124]

[0125] It can be seen that the learning process of the generator (i.e., the decision network) and the discriminator (i.e., the implicit reward function) is iterated simultaneously. When the loss function of the discriminator drops to 0, that is, the discriminator outputs 0.5, it indicates that the discriminator can no longer distinguish whether the strategy generated by the decision network comes from intelligence data. At this time, the value function of the decision network must reach its maximum value, indicating that the aircraft agent has fully learned the tactics and behavioral characteristics of typical combat opponents, and the task of the imitation learning stage has been completed.

[0126] It is easy to understand that the maneuvers that different types of drones can perform are not exactly the same. Therefore, when training the air combat decision-making model, a corresponding shielding mechanism will also be designed, that is, a behavioral constraint input that is adapted to the machine.

[0127] The behavior constraint input includes two parts: kinematic and dynamic constraints and tactical action constraints. Kinematic and dynamic constraints are mainly composed of various extreme performance values ​​of the corresponding model of UAV, including the extreme pitch angle and roll angle limited by the structural strength of the aircraft, the extreme normal and tangential overload, the maximum and minimum flight speeds, the minimum flight altitude and the maximum climbing altitude, etc. Tactical action constraints are designed according to the action space, and the maneuvering actions are shielded as needed.

[0128] Usually, the training process of the air combat decision model is based on the air combat simulation environment, which is constructed through the following steps:

[0129] First, a six-degree-of-freedom kinematic dynamics model of the aircraft is constructed. The model is established in the body coordinate system and the airflow coordinate system. The dynamic equation of the movement of the center of mass of the aircraft is considered. With the center of mass of the aircraft as the origin, in the body coordinate system, there is:

[0130]

[0131] Among them, F x 、F y 、F z are the projections of the resultant external forces acting on the aircraft on the three axes of the body coordinate system, p, q, r are the projections of the aircraft angular velocity on the three axes of the body coordinate system, u, v, w are the components of the aircraft airspeed in the body coordinate system;

[0132] The external forces acting on the center of mass include engine thrust T, aerodynamic force F a , gravity G, the projection of the resultant external force on the three axes of the body coordinate system can be specifically expressed as:

[0133]

[0134] Among them, L, D, and Y represent the projection of aerodynamic forces on the three axes of the airflow coordinate system, which are lift, drag, and side force, respectively; α and β are the angle of attack and sideslip angle; φ, θ, and ψ are the attitude angles in the body coordinate system;

[0135] The velocity in the body coordinate system is expressed by V, α, and β in the airflow system:

[0136]

[0137] Based on the above, the translational dynamics equation in the airflow coordinate system is as follows:

[0138]

[0139] Considering the rotational dynamics equation around the center of mass of the aircraft, in the body coordinate system, we have:

[0140]

[0141] Among them, L, M, N are the moments around the three axes, and I is the moment of inertia. By deriving and rearranging the above formula, we can get:

[0142]

[0143] in:

[0144]

[0145]

[0146]

[0147]

[0148] Considering the kinematic equation of the movement of the center of mass of the aircraft, taking the derivative of the left and right sides of the velocity expression with respect to time, and then transforming the equation to the geodetic coordinate system through the coordinate transformation matrix, the navigation motion equation of the aircraft is obtained as follows:

[0149]

[0150] Considering the kinematic equation of rotation around the center of mass of the aircraft, the relationship between the angular velocity of rotation and the Euler angle in the body coordinate system can be obtained:

[0151]

[0152] Based on the above formula, the motion dynamics model of a six-degree-of-freedom fixed-wing aircraft is constructed. However, the above equation is not closed and needs to be supplemented with the calculation equations of aerodynamic force and aerodynamic torque to complete the closure. The common method is to calculate through aerodynamic coefficients. Here, the aerodynamic parameter data of typical combat targets are used for modeling.

[0153] Once the equation is closed, as long as the control variables that can affect force and torque at each moment are given, such as rudder angle, engine thrust, etc., the state of the aircraft at any moment can be solved. The above nonlinear model composed of differential equations can be solved by numerical iterative algorithms such as the fourth-order Runge-Kutta algorithm, thereby achieving high-precision numerical simulation of the model.

[0154] Example 2

[0155] This embodiment also provides an air combat decision-making system based on generative adversarial imitation learning, including a user input module, a learning and training module, a data processing module and a real-time adversarial decision-making module.

[0156] The user input module is used to receive typical combat opponent data and perform enhancement processing on the typical combat opponent data to obtain a red-blue confrontation data set.

[0157] The learning and training module is used to perform network training based on the red-blue confrontation dataset and obtain an air combat decision model;

[0158] The data processing module is used to receive the information sent by the air combat simulation environment and process it into battlefield situation information that is adapted to the air combat decision-making model.

[0159] The real-time confrontation decision-making module is used to obtain battlefield situation information, and call the functions of the air combat decision-making model to process the battlefield situation information to obtain decision results for guiding the aircraft.

[0160] The data processing module is also used to encode the decision results into decision control instructions and send them to the air combat simulation environment for execution by the local machine.

[0161] The embodiments of the present invention are described in detail above with reference to the accompanying drawings, but the present invention is not limited to the described embodiments. For those skilled in the art, various changes, modifications, substitutions and variations of these embodiments are made without departing from the principles and spirit of the present invention, and still fall within the scope of protection of the present invention.

Claims

1. An air combat decision-making method based on generative adversarial imitation learning, characterized in that: The method comprises: obtaining battlefield situation information, processing the battlefield situation information through a pre-trained air combat decision model, and obtaining a decision result for guiding the aircraft, wherein the decision result comprises one or more of the aircraft's maneuvering action, ACM mode, and bullet shooting action; The air combat decision model is obtained by the following steps: Acquire typical combat opponent data, perform enhancement processing on the typical combat opponent data, and obtain a red-blue confrontation data set; By utilizing a generative adversarial imitation learning algorithm, a network model capable of simulating the behavioral characteristics and decisions of a typical combat opponent, namely, the air combat decision-making model, is obtained based on the red-blue adversarial data set training.

2. The method according to claim 1, characterized in that: The typical combat opponent data includes coarse-grained sparse expert strategy data and fine-grained dense expert strategy data, wherein the coarse-grained sparse expert strategy data includes air combat rules and state-action pairs; Then the step of performing enhancement processing on the typical combat opponent data to obtain the red-blue confrontation data set includes: Expanding the air combat rules through a decision tree model and generating new state-action pairs, and merging the new state-action pairs with the state-action pairs included in the coarse-grained sparse expert strategy data to obtain a first data set; Using a behavior cloning algorithm, a behavior cloning network is obtained based on the fine-grained dense expert strategy data training; data expansion is performed on the fine-grained dense expert strategy data through the behavior cloning network to obtain a second data set; The union of the first data set and the second data set constitutes the red-blue adversarial data set.

3. The method according to claim 1, characterized in that: Battlefield situation information includes the aircraft's coordinates, the flight status of both the enemy and our own aircraft, the geometric situation, and the aircraft's load status.

4. The method according to claim 3, characterized in that: The coordinates of the aircraft include east-west coordinates, north-south coordinates and flight altitude; the flight status of both the enemy and our side includes flight airspeed, flight speed indication, Mach number, attitude quaternion and confrontation time; the geometric situation includes enemy and our side distance vector, enemy and our side distance scalar, enemy and our side speed vector, enemy and our side altitude difference, entry angle and azimuth; the load status includes jammer status, remaining number of jammers, missile status and remaining number of missiles.

5. The method according to claim 1, characterized in that: The maneuvers include level flight, level flight acceleration and deceleration, climb, turn, dive, traverse, half roll, S maneuver and oblique somersault; the ACM mode includes level sweep mode selection, vertical sweep mode selection, fixed axis mode selection and seeker unlocking; the firing actions include infrared missile launch, chaff bomb launch and infrared jamming bomb launch.

6. The method according to claim 1, characterized in that: The training process of the air combat decision model is carried out based on an air combat simulation environment, and the air combat simulation environment is constructed by the following steps: First, a six-degree-of-freedom kinematic dynamics model of the aircraft is constructed. The model is established in the body coordinate system and the airflow coordinate system. The dynamic equation of the movement of the center of mass of the aircraft is considered. With the center of mass of the aircraft as the origin, in the body coordinate system, there is: Among them, F x 、F y 、F z are the projections of the resultant external forces acting on the aircraft on the three axes of the body coordinate system, p, q, r are the projections of the aircraft angular velocity on the three axes of the body coordinate system, u, v, w are the components of the aircraft airspeed in the body coordinate system; The external forces acting on the center of mass include engine thrust T, aerodynamic force F a , gravity G, the projection of the resultant external force on the three axes of the body coordinate system can be specifically expressed as: Among them, L, D, and Y represent the projection of aerodynamic forces on the three axes of the airflow coordinate system, which are lift, drag, and side force, respectively; α and β are the angle of attack and sideslip angle; φ, θ, and ψ are the attitude angles in the body coordinate system; The velocity in the body coordinate system is expressed by V, α, and β in the airflow system: Based on the above, the translational dynamics equation in the airflow coordinate system is as follows: Considering the rotational dynamics equation around the center of mass of the aircraft, in the body coordinate system, we have: Among them, L, M, N are the moments around the three axes, and I is the moment of inertia. By deriving and rearranging the above formula, we can get: in: Considering the kinematic equation of the movement of the center of mass of the aircraft, taking the derivative of the left and right sides of the velocity expression with respect to time, and then transforming the equation to the geodetic coordinate system through the coordinate transformation matrix, the navigation motion equation of the aircraft is obtained as follows: Considering the kinematic equation of rotation around the center of mass of the aircraft, the relationship between the angular velocity of rotation and the Euler angle in the body coordinate system can be obtained: The six-degree-of-freedom kinematic dynamics model of the aircraft based on the above formula is closed by supplementing the calculation equations of aerodynamic force and aerodynamic torque. After closing, the state of the aircraft at any time can be solved by giving the control quantity that can affect the force and torque at each moment, thereby realizing high-precision numerical simulation of the six-degree-of-freedom kinematic dynamics model of the aircraft.

7. The method according to claim 1, characterized in that: The step of using a generative adversarial imitation learning algorithm to train a network model capable of simulating the behavioral characteristics and decision-making of a typical combat opponent based on the red-blue adversarial data set includes: Random initialization strategy π θ And interact with the air combat simulation environment and obtain the trajectory under the current strategy, that is, to perform close-range air combat using a random maneuver strategy; Randomly initialize the discriminator network D φ , and distinguish the samples from the agent (s t , a t ,s t+1 )~D π Compared with the samples from the red-blue adversarial dataset (s t , a t ,s t+1 )~D e ; Use the discriminator’s predictions to define a reward function. In each round of training, first update the discriminator, and then use the RL algorithm to train the agent to maximize this reward, thereby deceiving the discriminator. The learning process of the generator and the discriminator is iterated at the same time until the loss function of the discriminator drops to the minimum and the network model converges, thereby obtaining the air combat decision model.

8. The method according to claim 7, characterized in that: The reward function is r(s,a)=-ln(1-D(s,a)), where D(s,a) represents the probability that the discriminator classifies a state-action pair as an expert.

9. The method according to claim 1, characterized in that: When training the air combat decision model, it also includes behavioral constraint input applicable to the aircraft, and the behavioral constraint input includes kinematic and dynamic constraints and tactical action constraints.

10. An air combat decision-making system based on generative adversarial imitation learning, characterized by: It includes user input module, learning and training module, data processing module and real-time confrontation decision module; The user input module is used to receive typical combat opponent data and perform enhancement processing on the typical combat opponent data to obtain a red-blue confrontation data set; The learning and training module is used to perform network training according to the red-blue confrontation data set and obtain an air combat decision model; The data processing module is used to receive information sent by the air combat simulation environment and process it into battlefield situation information adapted to the air combat decision model; The real-time confrontation decision module is used to obtain the battlefield situation information, and call the function of the air combat decision model to process the battlefield situation information to obtain a decision result for guiding the aircraft; The data processing module is also used to encode the decision results into decision control instructions and send them to the air combat simulation environment for execution by the local machine.

Images