Air combat intelligent decision-making method and system combining imitation learning and reinforcement learning

Abstract

The invention discloses an air combat intelligent decision-making method combining imitation learning and reinforcement learning, and belongs to the field of air combat, and the method comprises the steps: processing battlefield situation information through an air combat intelligent decision-making model, and obtaining a decision-making result used for guiding a local machine. The pre-trained air combat intelligent decision-making model is obtained through the following steps: for sparse expert strategy data with coarse granularity, using a behavior cloning algorithm to train a neural network architecture for imitation learning and reinforcement learning to obtain a strategy network Q1; taking the strategy network Q1 as an initial network in a generative adversarial imitation learning algorithm, and performing imitation learning on a fine-granularity dense expert strategy to obtain a strategy network Q2; and taking the strategy network Q2 as an initial network of a decision network in the reinforcement learning algorithm framework, and carrying out network training by adopting a strategy gradient method until convergence to obtain an air combat intelligent decision model. According to the invention, the sample utilization rate can be effectively improved, and accumulative errors are reduced.

Description

Technical Field

[0001] The present invention relates to the field of aviation technology, and in particular to an air combat intelligent decision-making method and system combining imitation learning and reinforcement learning. Background Art

[0002] The rapid development of drone technology has brought innovative solutions to many fields, especially in the military field. Compared with traditional manned aircraft, drones have lower costs, lower safety risks, and can withstand greater overload. However, due to the increasing complexity of mission scenarios and the diversification of mission requirements, how to improve the autonomous decision-making ability of drones has become an urgent problem to be solved.

[0003] At present, domestic and foreign UAV autonomous decision-making methods can be roughly divided into four categories: methods based on expert systems, methods based on game theory, methods based on optimization theory, and methods based on artificial intelligence.

[0004] The expert system-based method converts the expert's experience summary into a tactical knowledge base and makes decision-making inferences based on the information in the knowledge base. This method has good interpretability, but is highly dependent on expert experience and preset rules and is not adaptable to rapidly changing battlefield environments.

[0005] Game theory-based methods use the framework and models of game theory to analyze and optimize the decision-making process in air combat. Matrix game or differential game methods are often used. However, it is difficult to obtain Nash equilibrium analytical solutions for high-confrontation and high-dynamic air combat decision-making problems.

[0006] The method based on optimization theory models air combat decision-making as a multi-objective optimization problem, and then uses an intelligent optimization algorithm to solve it. This method is prone to fall into local optimality in the process of finding the optimal strategy, and it is necessary to improve its global search optimization ability and optimization accuracy.

[0007] Artificial intelligence-based methods usually need to be data-driven and use data samples to build decision-making models. Generally, a large amount of air combat sample data is required to achieve good results. However, in practice, air combat sample data involving human confrontation is difficult to obtain in large quantities.

[0008] Although drone autonomous decision-making technology has made some progress, it still faces many challenges. In complex and dynamic environments, existing decision-making methods are not adaptable enough, and traditional rules and algorithms are difficult to effectively respond to emergencies. In order to give full play to the performance of drones, it is necessary to continue to improve them to achieve a high degree of autonomous decision-making capabilities.

[0009] With the development of artificial intelligence technology, deep reinforcement learning theory has been successfully applied in many fields. Deep reinforcement learning combines the advantages of deep learning and reinforcement learning. Its main features include: first, deep reinforcement learning uses deep neural networks to automatically extract features and can process high-dimensional state spaces, thereby effectively coping with complex environments; second, deep reinforcement learning learns strategies through interaction with the environment, making it adaptable and autonomous in decision-making, and able to optimize behavior in dynamic and uncertain environments; in addition, deep reinforcement learning has the trial-and-error mechanism inherent in reinforcement learning, and achieves self-optimization of strategies through continuous exploration and utilization balance.

[0010] However, deep reinforcement learning itself also has some problems. Since deep reinforcement learning needs to optimize the decision network in the trial and error of continuous interaction with the environment until convergence, the decision model requires a large amount of sample data to fully explore and learn the value of each state and action, resulting in low sample efficiency; at the same time, due to the large number of parameters and complex structure, the training of the decision model is very time-consuming, which also makes the decision model prone to cumulative errors. It may even lead to the air combat intelligent decision model that takes time and effort to train is neither "intelligent" nor applicable to "air combat". Summary of the invention

[0011] The present invention provides an air combat intelligent decision-making method and system combining imitation learning and reinforcement learning, so as to at least partially solve the above problems.

[0012] To achieve the above object, the technical solution of the present invention is:

[0013] In a first aspect, the present invention provides an air combat intelligent decision-making method combining imitation learning and reinforcement learning, the method comprising: obtaining battlefield situation information, and then processing the battlefield situation information through a pre-trained air combat intelligent decision-making model to obtain a decision result for guiding the aircraft, the decision result including the aircraft's maneuvering action and / or firing action;

[0014] The air combat intelligent decision model is obtained by the following steps:

[0015] Acquiring training data, wherein the training data includes expert strategy data;

[0016] Classifying the expert strategy data to obtain coarse-grained sparse expert strategy data and fine-grained dense expert strategy data;

[0017] The coarse-grained sparse expert strategy data is processed to obtain a state-action pair data set, and then the behavior cloning algorithm is used to train the neural network architecture for imitation learning and reinforcement learning to obtain a strategy network Q1 ;

[0018] The policy network Q 1 As the initial network in the generative adversarial imitation learning algorithm, the fine-grained dense expert strategy is imitated and learned to obtain the strategy network Q 2 ;

[0019] The policy network Q 2 As the initial network of the decision network in the reinforcement learning algorithm framework, the policy gradient method is used to train the network until convergence to obtain the policy network Q 3 , that is, the pre-trained air combat intelligent decision-making model.

[0020] In some preferred embodiments, the coarse-grained sparse expert strategy data include air combat rules and state-action pairs, and the step of processing the coarse-grained sparse expert strategy data to obtain a state-action pair data set includes: expanding the air combat rules through a decision tree model and generating new state-action pairs, merging them with the state-action pairs contained in the coarse-grained sparse expert strategy data to obtain the state-action pair data set.

[0021] In some preferred embodiments, when training the air combat intelligent decision-making model, it also includes behavioral constraint input and confrontation scenario input applicable to the aircraft; the behavioral constraint input includes kinematic and dynamic constraints and tactical action constraints; the confrontation scenario input includes close-range, medium-range and beyond-visual-range scenarios.

[0022] In some preferred embodiments, when performing a policy network Q 1 When training, the state S in the state-action pair data set is used as a feature, the action A is used as a tag for regression, all actions A are One-Hot encoded, and the loss function is defined Repeat the stochastic gradient descent until the action clone algorithm converges.

[0023] In some preferred embodiments, when performing the strategy network Q 3 During training, adversarial training is guided by designing reward functions. The reward function design includes reward weight design, process reward function design, boundary reward function design and attack control area reward function design. After each reward function is designed, all rewards are summarized and assigned their respective weights to complete the overall reward design.

[0024] In some preferred embodiments, the process reward weight does not exceed the result reward weight.

[0025] In some preferred embodiments, the process reward includes an angle advantage function r a and the energy advantage function re ;

[0026]

[0027] in, is the target azimuth, q is the target entry angle, and are the energy heights of the aircraft and the enemy aircraft respectively; for a UAV with a flight speed of V and a flight altitude of H, its energy height is H E =H+0.5V 2 / g;

[0028] The process reward also includes a two-machine distance reward r R and dual aircraft height bonus r H ;

[0029]

[0030] Among them, R is the enemy distance vector, H r and H b It is the flight altitude of the own aircraft and the enemy aircraft.

[0031] In some preferred embodiments, the boundary reward function is

[0032]

[0033] Among them, p 1 is the punitive reward value; x is the value that triggers the reward, a is the lower boundary of the reward value that triggers the reward, and b is the upper boundary; k is a positive real number, representing the abruptness of the reward. The larger the value, the more abrupt the reward is triggered.

[0034] In some preferred embodiments, the offensive control zone reward function is

[0035]

[0036] Among them, r 1 , r 2 are the reward components in two dimensions, r 3 is the final offensive control zone reward, the x input is the entry angle, the y input is the enemy-friend distance, (a, b), (c, d) are the reward trigger boundaries of each dimension, and k 1 , k 2 The abruptness of the rewards for each dimension.

[0037] In a second aspect, the present invention also provides an air combat intelligent decision-making system combining imitation learning and reinforcement learning, including a user input module, a learning and training module, a data processing module and a real-time confrontation decision-making module;

[0038] The user input module is used to receive behavior constraint input, confrontation scenario input and expert strategy data input; the user input module is also used to classify and expand the expert strategy data

[0039] The data processing module is used to receive the original battlefield information sent by the battlefield support environment and process it into data for the adaptation learning and training module;

[0040] The learning and training module is used to perform network training and obtain an air combat intelligent decision model based on the data provided by the user input module and the data parsing module;

[0041] The real-time confrontation decision module is used to obtain battlefield situation information, and call the function of the air combat intelligent decision model to process the battlefield situation information to obtain a decision result for guiding the aircraft;

[0042] The data processing module is also used to encode the decision results into decision control instructions and send them to the battlefield support environment for execution by the local machine.

[0043] By adopting the above technical solution, the beneficial effect of the present invention is that: based on various forms of expert strategy data, the present invention uses an imitation learning framework to pre-train the decision network, and then uses a reinforcement learning framework to retrain the decision network. The combination of the two can effectively accelerate the training process, improve sample utilization, and reduce cumulative errors. The present invention can realize autonomous air combat decision-making and output control instructions according to the real-time status information of the battlefield support environment; it can extract and learn various forms of expert strategies and expert experience; it can establish communication with various forms of battlefield support environments to realize intelligent air combat of drones, thereby significantly improving combat efficiency and decision-making capabilities; it has strong generalization capabilities and strong robustness for complex dynamic battlefield environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 It is a schematic diagram of the architecture mode of the present invention.

[0045] Figure 2 It is a schematic diagram of the data structure definition of the data processing module in the present invention.

[0046] Figure 3 It is a schematic diagram of the overall process of the present invention. DETAILED DESCRIPTION

[0047] The specific embodiments of the present invention are further described below in conjunction with the accompanying drawings. It should be noted that the description of these embodiments is used to help understand the present invention, but does not constitute a limitation of the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0048] Example 1

[0049] An embodiment of the present invention provides an air combat intelligent decision-making system combining imitation learning and reinforcement learning, including a user input module, a data processing module, a learning and training module, and a real-time confrontation decision-making module.

[0050] The present invention will be further described in detail below with reference to the accompanying drawings.

[0051] First, the intelligent air combat decision-making system combining imitation learning and reinforcement learning (hereinafter referred to as the intelligent air combat decision-making system) is based on the model-view-controller architecture pattern commonly used in software engineering. The architecture pattern is as follows Figure 1 As shown in the figure, the overall architecture is divided into four layers, including application layer, service layer, algorithm layer, and information processing layer.

[0052] Among them, the application layer is mainly aimed at typical application scenarios of intelligent air combat decision-making systems, including intelligent air combat, pilot confrontation training and other scenarios.

[0053] The service layer provides several key services involved in intelligent air combat confrontation training, specifically the training service of air combat intelligent decision-making model and the generation service of confrontation strategy. The training service of air combat intelligent decision-making model is applied to the model training stage, and the generation service of confrontation strategy is applied to the deployment and application stage of the system.

[0054] The algorithm layer includes imitation learning algorithm and reinforcement learning algorithm, which are mainly used in the training stage of air combat intelligent decision-making model.

[0055] The bottom-level information processing layer includes the analysis and processing of battlefield support environment information and user input information.

[0056] According to the above architecture, the intelligent air combat decision-making system provided by the embodiment of the present invention is designed with a user input module, a data processing module, a learning and training module and a real-time confrontation decision-making module.

[0057] It should be noted that the present invention generally requires a battlefield support environment to provide data collection and simulation training, which may be in the form of, but not limited to, digital models, semi-physical models or ground stations. In this embodiment, a self-developed air combat simulation platform is used as support, which only provides flight scenes and data, does not affect the independence of the present invention, and is not a component of the present invention.

[0058] Before introducing the construction of the intelligent air combat decision-making system provided by the embodiment of the present invention, some necessary information involved in the system is first introduced.

[0059] The drone air combat scenario is abstracted into a Markov Decision Process (MDP), and the state space S, action space A and strategy network Q are designed.

[0060] Specifically, at each moment, the battlefield support environment will have a corresponding situation, which is mathematically described as a state. Any decision made by the UAV agent depends on the understanding and analysis of the state. During air combat, the state space of the UAV agent represents all the environmental information that can be perceived and obtained, including its own flight and weapon status, as well as enemy information and external environment information obtained by the perception system.

[0061] Considering that the perception ability of drones is usually weak, it is difficult to obtain enemy status information if it only relies on its own radar and other equipment, which is not conducive to the training of air combat intelligent decision-making models. Therefore, it is assumed that the drone agent can fully obtain enemy situation information through some means (such as data link information injection). According to the classification of state information in air combat, the state space S is extracted as follows:

[0062] 1. Local status S UAV , including the aircraft's coordinates (3D), aircraft's airspeed (3D), and flight time (1D);

[0063] 2. Relative situation S Re , including enemy and friendly distance vector (3D), enemy and friendly distance scalar (1D), enemy and friendly velocity vector (3D), entry angle (1D), azimuth (1D);

[0064] 3. Load state S Ca , including jammer status (1 dimension), jammer quantity (1 dimension), missile status (1 dimension), and missile quantity (1 dimension).

[0065] Among them, the local situation S UAV and relative situation S Re All are continuous state quantities, and the load state S Ca It is a discrete state quantity. The jammer status includes whether the jammer is currently effective. The missile status includes four states: unlocked, locked but not launched, launched but not hit, and launched and hit.

[0066] Specifically, action refers to the decision made by the UAV agent based on the current state. The action space A refers to the set of all possible actions. The construction of the action space A must be complete and reasonable. Complete means that it should cover all means that the UAV agent can use in the air combat confrontation process. Reasonable means that the action design must be under the prescribed constraints. For actions that are constrained under specific states, corresponding shielding mechanisms should be designed.

[0067] Specifically, due to the complexity of air combat confrontation problems, the design of the action space A should be as simple as possible, and the dimension of the action space A should be reduced as much as possible while meeting basic control requirements. Specifically, the decision control for the UAV does not directly control its control surface, but builds a maneuvering action library based on actual maneuverability and specific mission scenarios. The UAV agent can make a combination of tactical actions when making decisions, thereby reducing the convergence difficulty of imitation learning training and reinforcement learning training.

[0068] According to the characteristics of the UAV agent's maneuvering actions, the action space A is classified and designed, and the composition is as follows:

[0069] 1. Maneuvers: including level flight, level flight acceleration and deceleration, climb, turn, dive, cross-cut, half-roll, S maneuver, and oblique somersault;

[0070] 2. Firing actions: including infrared missile launching, chaff bomb launching, and infrared jamming bomb launching.

[0071] The action space A constructed above is a discrete action space. In the design of maneuvering actions, each maneuver needs to be given target parameters (for example, turning requires a given turning direction, turning radius, slope, etc.). In order to cover as many tactical action spaces as possible, for maneuvering actions, combined with the behavioral constraints and confrontation scenarios input by the user, when designing maneuvering actions, one or more sets of discrete setting parameters are given as typical maneuvering parameters, thereby realizing the full discretization of the action space. The discretized action space A can be expressed as:

[0072] A=A m ×A s

[0073] Among them, A is the action space, A m For maneuvering, A s For the shooting action, at any decision moment, the UAV agent selects one of the maneuvering action and the shooting action to execute, and gives the corresponding execution parameters (discretization).

[0074] The decision network Q refers to a neural network architecture that can be used for imitation learning and reinforcement learning. The system provided by the embodiment of the present invention involves the training and application of an air combat intelligent decision model, which is obtained by successively performing two processes of imitation learning pre-training and reinforcement learning re-training on the decision network Q. These two processes are essentially operations on network architectures with the same structure, so it is necessary to ensure the consistency of the network structure during the training process. Specifically, a four-layer fully connected neural network needs to be established, in which the input of the input layer is a state space vector, and the output of the output layer is an action space vector. Full connection is used from the input layer to the hidden layer, between the hidden layer and the hidden layer, and between the hidden layer and the output layer.

[0075] Specifically, the input of the decision network Q is the state space vector S=[S UAN , S Re , S Ca ], the output is the entire action space A = [A m , A s ], and select the action with the highest probability as the decision output. The output is encoded and sent to the battlefield support environment for it to update the battlefield situation.

[0076] Through the above introduction, the embodiment of the present invention provides a detailed introduction to the user input module, the data processing module, the learning and training module and the real-time confrontation decision module.

[0077] Part 1: Designing the User Input Module

[0078] The input of the user input module consists of three parts: behavior constraint input, adversarial scenario input, and expert strategy data input.

[0079] The behavior constraint input includes two parts: kinematic and dynamic constraints and tactical action constraints. Kinematic and dynamic constraints are mainly composed of various extreme performance values ​​of the corresponding model of UAV, including the extreme pitch angle and roll angle limited by the structural strength of the aircraft, the extreme normal and tangential overload, the maximum and minimum flight speeds, the minimum flight altitude and the maximum climbing altitude, etc. Tactical action constraints are designed according to the action space, and the maneuvering actions are shielded as needed.

[0080] The input of confrontation scenarios includes close-range, medium-range and beyond-visual-range scenarios, and different action libraries are determined for different scenarios. The specific difference is that different types of missiles need to be used for different scenarios.

[0081] The expert strategy data of the simulated object is the basis for subsequent imitation learning. Ideally, the real motion trajectory of the air combat can be directly used as the expert strategy data of the simulated object, but the intelligence data actually obtained may not be sufficient to support the implementation of imitation learning in terms of data volume, and may not even be a trajectory composed of state-action pairs, but may be other more fuzzy and large-granularity forms. Therefore, for expert strategy data of different granularities, different schemes need to be adopted to learn the expert strategy data information therein. The present invention designs expert strategy data extraction schemes for expert strategy data of two different granularities respectively.

[0082] 1. Fine-grained and dense expert strategy data

[0083] When the expert policy data is dense enough, it is processed into a trajectory dataset of simulated objects:

[0084] D Dense={τ 1 , τ 2 , ..., τ n}

[0085] In the formula, is the ith trajectory of the simulation object; the superscript i represents the ith trajectory, and the subscripts are 1 to m i Represents a time-ordered sequence, such as Represents the mth track in the i-th track sorted by time i Status information, Represents the mth i Action information. Dataset D Dense That is, it represents the fine-grained dense expert strategy data of the simulation object.

[0086] 2. Coarse-grained sparse expert strategy data

[0087] Assume that the expert strategy data is in the form of a descriptive statement with large granularity. For example, the descriptive statement is: when the state is that the enemy is behind us to the right, we will take an S-shaped maneuver or a half-roll maneuver. This data form provides or can be converted into a label of a state-action pair, that is, it can distinguish whether the state-action pair belongs to a subset pair of the state space and action space provided by the descriptive statement in the intelligence, which provides a basis for supervised learning. Therefore, the information of the coarse-grained sparse expert strategy data of the simulation object is extracted into the following form:

[0088] D Sparse ={(S 1 ,A 1 ),(S 2 ,A 2 ),...,(S n , A n )}

[0089] In the formula, S i and A i are the state space subset and action space subset corresponding to the i-th descriptive sentence respectively. The above dataset D Sparse That is, it represents the coarse-grained sparse expert strategy data of the simulation object.

[0090] Part II: Designing Data Processing Modules

[0091] The data processing module includes two core functions: data analysis and instruction encoding. The data analysis submodule is responsible for receiving the information sent by the battlefield support environment and processing it into a specified format; the instruction encoding submodule packages the decision instructions output by the real-time confrontation decision module and sends them to the battlefield support environment.

[0092] The data processing module is used to communicate with the battlefield support environment in real time and receive battlefield information data packets under standard protocols. Data transmission needs to consider requirements such as reliability, security, efficiency, and real-time performance. Taking various factors into consideration, the standard MAVLink (Micro Air Vehicle Link) protocol is preferably used as the communication protocol.

[0093] MAVLink is a lightweight message protocol that can be used for communication between drones and between drone-mounted devices. In MAVLink, STX represents the start flag of the data packet; LEN represents the length of PAYLOAD; SEQ represents the data packet sequence number, 0 to 255. Each time a data packet is sent, the sequence number will be accumulated, and it will be repeated to detect packet loss; SYSID represents the device system-level ID, indicating its own ID identity; COMP ID represents the device component-level ID. A same system may have multiple different components that use MAVLink for communication; PAYLOAD is used to store communication data. Each frame of data consists of a 6-byte header, up to 255 bytes of carried data, and 2 bytes of check data. At the same time, a 255-byte serial check is added to ensure the reliability of data transmission. It is not only suitable for point-to-point communication, but also easily extended to multi-point communication.

[0094] The battlefield support environment uses the standard MAVLink protocol to package the battlefield raw information data and send it to the data processing module through the TCP / IP protocol. The battlefield raw information data includes the local situation information, relative situation information and load status information. Its specific form is as follows: Figure 2 As shown in part 2-1 of . The local situation information and relative situation information are set to float type, and the load status information is set to unit8 type. Since battlefield information contains the navigation information of the opposing sides in the battlefield and the original data of the drone itself, these data may be received in multiple data frames, so a battlefield information parsing process can be regarded as a concurrent process of parsing multiple data frames. The MSG of the data frame can determine the meaning of the load information (PAYLOAD) in the data frame, and store the data in the corresponding space in the input buffer according to the meaning of the data. When the battlefield information data in the input buffer is complete, the data preprocessing module reads the data from the buffer. Afterwards, the data can be preprocessed as needed to convert the original information into the required form, such as angle conversion, unit conversion, etc.

[0095] The decision control instructions generated by this system for controlling the drone are packaged into a predefined structure under the MAVLink standard protocol and sent through the instruction encoding submodule in the data processing module. The remote control instruction frame structure of the drone intelligent body is designed according to the MAVLink protocol. In order to package each action instruction into a remote control instruction frame, it is necessary to consider the size of the space occupied by each action and its parameters when designing the remote control instruction frame structure. The load information in the MAVLink protocol data frame is divided into target ID, maneuvering action, firing action and maneuvering command parameters, such as Figure 2 As shown in part 2-2 of . Among them, the target ID, maneuvering action and firing action can be set to uint8 type, and the maneuvering parameter can be set to float type. Encode the decision control command according to the defined remote control command frame structure. For specific mission scenarios, define a number for each maneuvering action in the UAV maneuvering action library in advance in the air combat intelligent decision-making system and battlefield support environment, such as 1-half roll, 2-circling maneuver, 3-S-type maneuver, etc. When applied online, convert the maneuvering action instructions in the decision control instructions into uint8 data and float data with corresponding numbers to realize the maneuvering instruction encoding. The firing instruction adopts the same method as the maneuvering action numbering, such as 1-no jamming bomb, 2-release jamming bomb, etc. The firing action number is written into the MAVLink protocol remote control command frame to realize the UAV firing action instruction encoding.

[0096] The instruction encoding submodule is also responsible for sending the encoded decision control instructions to the battlefield support environment to achieve data uplink transmission. Usually, the system and the battlefield support environment are directly connected by physical connection methods such as serial ports to achieve uplink transmission of decision control instructions; if conditions permit, the TCP / IP communication protocol can also be used to transmit decision control instructions through the network.

[0097] Part III: Designing an Imitation-Based Learning Environment

[0098] Specifically, it includes behavioral cloning algorithm and generative adversarial imitation learning algorithm.

[0099] In the first part, the user input module classifies the expert policy data. The fine-grained dense expert policy data is processed into a trajectory dataset D of the simulated object. Dense , the coarse-grained sparse expert policy data is processed into a labeled state-action dataset D Sparse .

[0100] For coarse-grained sparse expert strategy data (such as air combat rules, state-action pairs), the behavioral cloning method is used; for fine-grained data, the Generative Adversarial Imitation Learning (GAIL) method is used.

[0101] Specifically, coarse-grained fuzzy expert data can be divided into air combat rules and state-action pairs because of their clear state-action mapping relationship. If it is an air combat rule, a decision tree model is constructed. For the state covered in the intelligence information, if the intelligence indicates a unique maneuvering method, the branch is directly entered for execution. If the intelligence indicates multiple response methods or gives vague guidance, the random branch under the node is executed. Then this decision tree is used to generate a large number of new samples in the form of state-action pairs, which are merged with the original state-action pair data to form a state-action pair data set.

[0102] Then, based on the state-action pair dataset, the behavior cloning algorithm is used to train the policy network Q and obtain the policy network Q 1 When performing behavior cloning, all state-action pairs constitute the dataset D Spartse , taking state S as feature and action A as label for regression, one-hot encoding is performed on all actions A, and the loss function is defined as:

[0103]

[0104] θ represents the network parameters of the neural network pi. The smaller the loss function, the better the policy network Q 1 The closer the decision is to the tactical decision in the state-action pair dataset. For a given state-action pair dataset, stochastic gradient descent is repeated until the algorithm converges.

[0105] Decision network Q obtained through behavior cloning 1 It is an imitation learning of coarse-grained sparse expert strategy data. For fine-grained dense expert strategy data in the form of trajectory data sets, the generative adversarial imitation learning algorithm (GAIL) is used. GAIL is essentially a reverse reinforcement learning method. Through the game confrontation between the generator and the discriminator, the expert behavior itself can be directly learned and the decision network can be directly generated.

[0106] First, use the policy network Q 1 Interact with the battlefield support environment and obtain the trajectory under the current strategy. At the same time, a discriminator network D is randomly initialized to distinguish the samples (S t , A t , S t+1 )~D π Compared with the dense expert policy dataset D from the fine-grained Dense Sampling (S t , A t , S t+1 )~D e, and then use the discriminator's predictions to define a reward function:

[0107] R(S,A))=-ln(1-D(S,A))

[0108] Where D(S, A) represents the probability that the discriminator classifies a state-action pair as an expert. In each round of training, the discriminator is updated first, and then the drone agent is trained to maximize this reward to deceive the discriminator. As with the adversarial generative network algorithm, the training of the discriminator and the agent is alternating. From the overall process, GAIL follows the following steps:

[0109] (1) Use the current strategy to interact with the battlefield support environment and store the trajectory of this round of air combat in the experience buffer.

[0110] (2) Update the discriminator.

[0111] (3) Update the strategy network.

[0112] The process of generating adversarial imitation learning algorithm is as follows:

[0113]

[0114]

[0115] Part 4: Designing an Environment Based on Reinforcement Learning

[0116] Specifically including reward function design.

[0117] From the above description, we can know that after imitating the expert strategy data, a decision network Q can be generated. 2 . Although the training of the imitation learning algorithm includes both the process of "imitation" and the process of "optimization", its fundamental starting point is to extract the typical tactics and strategies of experts. Therefore, this method is not sufficient for the exploration of the state space, that is, in an air combat situation that cannot be covered by intelligence data, the decision network method obtained by the GAIL algorithm training will still "fit" to the expert strategy data as much as possible. From the perspective of human subjective cognition, although it has a certain degree of explainability, in most cases, the decision in the state space that cannot be covered by the expert strategy data is not optimal. Therefore, the embodiment of the present invention proposes to use the decision network Q 2 The idea of ​​further optimization after generation is to imitate the decision network Q obtained by learning 2 As the initial policy network, adversarial training is continued in the reinforcement learning environment, and finally the optimal decision network in the entire state space is obtained.

[0118] Among the current mainstream reinforcement learning algorithms, the Actor-Critic (AC) architecture combines the advantages of traditional reinforcement learning value-based and policy-based methods. This method has gradually become the mainstream architecture in the field of reinforcement learning. The Deep Deterministic Policy Gradient (DDPG) method proposed on this basis solves the problem of "dimensionality explosion" output of traditional methods when facing high-dimensional decision-making problems, and realizes action output in continuous space. This method performs well in the field of autonomous air combat. In addition, some researchers use reinforcement learning methods to self-evolve air combat actions to achieve innovation in tactics and methods.

[0119] Specifically, in the decision network optimization stage, it is necessary to design a process reward function to guide adversarial training. The reward function design includes four parts: reward weight design, process reward design, boundary reward design, and control area reward design.

[0120] 1. Reward weight design

[0121] Process rewards are based on human design and have a certain degree of subjectivity; although the result reward is objective, it only occurs at the end. Therefore, the weight design principle is that the weight of the process reward should not exceed the discounted result reward to ensure that the agent has sufficient motivation to choose a result-oriented maneuver plan, and the final agent strategy tends to the objective result.

[0122] Specifically, according to experience, the duration of air combat is 4 to 10 minutes. If the step size is 4 seconds and the decay weight is 0.99, the minimum decay weight of the result reward during the air combat is about 0.2, and the process reward weight does not exceed 0.2 to meet the requirements.

[0123] 2. Process Reward Design

[0124] Process rewards are designed based on pilot domain knowledge. According to the Basic Fighter Maneuvering (BFM) course description, close-range air combat is a process of balancing angle advantage and energy advantage, so the angle advantage function r is designed. a and the energy advantage function r e As a major process reward.

[0125]

[0126] In the formula, is the target azimuth, q is the target entry angle, and are the energy heights of the aircraft and the enemy aircraft respectively. For a UAV with a flight speed of V and a flight altitude of H, its energy height is defined as H E =H+0.5V2 / g.

[0127] In order to guide the intelligent agent to "bravely" engage the enemy, a dual-machine distance reward r is also designed R and dual aircraft height bonus r H , specifically:

[0128]

[0129] Where R is the distance vector between the enemy and us, H r and H b It is the flight altitude of the own aircraft and the enemy aircraft.

[0130] 3. Boundary Reward Design

[0131] During the training of the agent, dangerous behaviors such as "falling into the sea", "stalling", and "speeding" may occur due to values ​​exceeding the safety range, resulting in air combat failure. If conditional trigger rewards are set for such situations, such rewards will be sparse, discontinuous, and gradient-free, which is not conducive to training. The present invention designs a "cup-shaped" function to optimize such rewards. The function is composed of:

[0132]

[0133] In the formula, p 1 is the punitive reward value, x is the value that triggers the reward, a is the lower boundary of the reward value that triggers the reward, b is the upper boundary, and k is a positive real number representing the abruptness of the reward. The larger the value, the more abrupt the reward is triggered. This reward is continuous and differentiable, which is conducive to neural network optimization.

[0134] 4. Control zone reward design

[0135] Specifically, in air combat, there is an important area concept, called the Offensive Control Zone (OCZ). In the Offensive Control Zone, the attacker can easily track the defender, and it is difficult for the defender to get rid of the tracking unless the attacker makes a mistake. The rewards for the Offensive Control Zone are designed as follows:

[0136]

[0137] Among them, r 1 , r 2 are the reward components in two dimensions, r 3 is the final OCZ reward, the x input is the entry angle, the y input is the enemy distance, (a, b), (c, d) are the reward trigger boundaries of each dimension, and k 1 , k 2 is the abruptness of the reward in each dimension. Specifically, (a, b), (c, d) are respectively (0, 30), (5, 15), k1 , k 2 Here, the values ​​are 1.3 and 2.3 respectively.

[0138] By summarizing all the above rewards and assigning their respective weights, we can get the overall reward design.

[0139] Among many reinforcement learning algorithms, the proximal policy optimization algorithm (PPO) has the advantages of strong adaptability and stable training. The PPO algorithm is a policy gradient algorithm. The update of the policy gradient algorithm is very sensitive to the step size, and it is difficult to choose a suitable step size. Therefore, it is easy to make the difference between the new and old strategies too large during the training process, which is not conducive to the generation of effective strategies. The objective function proposed by the PPO algorithm can be iteratively updated with a small number of samples in multiple rounds of training, solving the problem of difficult to determine the step size and large update difference in the policy gradient algorithm. At the same time, the PPO algorithm combines the advantages of high data utilization efficiency of offline policy methods and the stability of online policy methods, and is very suitable for the retraining of UAV intelligent agents in air combat confrontation.

[0140] The PPO algorithm is based on the Actor-Critic framework, which includes two Actor networks and one Critic network. It collects samples to train the network in an on-policy manner, that is, the strategy for generating samples and the strategy for training are the same. Specifically, the steps for retraining the decision network based on the PPO algorithm are as follows: Figure 3 As shown in the "Reinforcement Learning" box.

[0141] The state S of the Actor network at time step t t Generate an action probability distribution, and the agent randomly samples an action A from the generated action probability distribution t , the environment depends on the action A performed by the agent t Give the agent a reward and enter the next state S t+1 , and the four-tuple t , A t , R t , S t+1 >Store it in the experience pool. After a certain amount of data is collected, stop collecting data. Calculate the loss function of the Actor network and the Critic network for back propagation. Perform multiple rounds of updates on the parameters of Actor network 1 and the Critic network. Then, regularly copy the parameters of Actor network 1 to Actor network 2. Alternate between data collection and network update to continuously improve the agent’s strategy.

[0142] Specifically, the strategy network Q obtained through imitation learning 2 ​As the initial network of Actor Network 1 and Actor Network 2, during training, the agent samples actions from the action probability distribution output by the Actor Network, executes actions to interact with the environment to collect data, and then updates the network parameters of Actor Network 1 according to the following formula.

[0143]

[0144] Where π θ represents a random strategy, Represents the advantage function estimate for a time step:

[0145] A π (s t , a t )=Q π (s t , a t )-V π (s t )

[0146] In order to improve data utilization efficiency, combined with importance sampling, two networks, Actor1 network representing the new strategy and Actor2 network representing the old strategy, are constructed. During network training, the ratio of the new and old strategies is used to correct the collected data so that the data can be reused. The parameters of the Actor1 network representing the new strategy are updated through policy gradient ascent. The Actor2 network representing the old strategy will regularly copy the parameters of the new network. In order to limit the step size of each step in the policy space and ensure that the difference in the distribution of the new and old strategies is not too large, the ratio of the new and old strategies is truncated. The improved Actor network loss function is as follows:

[0147]

[0148] Among them, r t (θ) is the ratio of the new and old strategies, ∈ is a hyperparameter, is an estimate of the advantage function.

[0149] Critic network is used to estimate the state value V(S t ) and calculate the advantage function The input of the Critic network is the current state vector, and the output is the estimated state value. The parameters are updated by minimizing the loss function of the Critic network, making the value estimation more accurate.

[0150] The algorithm flow can be described as.

[0151]

[0152] Part V: Design Learning Training Module

[0153] Based on the contents disclosed in the third and fourth parts above, the learning and training module models the interactive environment of the UAV agent and the adversarial training, and constructs the necessary elements required for inverse reinforcement learning and reinforcement learning. The input of the learning and training module is the behavior constraints, adversarial scenarios and expert strategy data output by the user input module, and the output is the trained air combat strategy network Q 3 , that is, the intelligent decision-making model of air combat.

[0154] The learning and training module stipulates the UAV intelligent body's extreme pitch and roll angles, extreme normal and tangential overloads, maximum and minimum flight speeds, minimum flight altitude and maximum climbing altitude according to behavioral constraints, and blocks maneuvers that do not meet the requirements.

[0155] The learning and training module determines different tactical action libraries according to specific confrontation scenarios. According to specific confrontation scenarios and tasks, the learning and training module can determine the confrontation means of the aircraft, whether the confrontation process is over, and provide judgment criteria for the establishment of the reward model.

[0156] The learning and training module selects a suitable imitation learning algorithm according to the form and granularity of the expert strategy data of the input simulation object, and adopts different schemes to learn the expert strategy data information.

[0157] The training process of the air combat intelligent decision-making model is a combination of imitation learning and reinforcement learning. The training process can be described as follows:

[0158] First, the performance limitations and maneuver library of the UAV are determined based on the behavioral constraints and adversarial scenario inputs.

[0159] Then, the expert strategy data input is classified. For coarse-grained sparse expert strategy data, a decision tree model is first established to generate a large amount of state-action pair data, and then the behavior cloning algorithm is used to obtain the decision network Q 1 ; Then, this decision network Q 1 As the initial network in the generative adversarial imitation learning algorithm, the fine-grained dense expert strategy data is used as the expert trajectory, and the decision network Q is trained using the generative adversarial imitation learning framework. 2 .

[0160] Finally, the decision network Q obtained by the generative adversarial imitation learning framework is 2 As the initial network of Actor1 and Actor2 networks in the reinforcement learning algorithm framework, the parameters of Actor network 1 and Critic network are used for multiple rounds of updates, and then the parameters of Actor network 1 are regularly copied to Actor network 2, alternating between data collection and network update. During the training process, the enemy aircraft adopts a fixed expert strategy and trains until convergence to obtain the decision network Q 3 .

[0161] Finally, the learning and training module outputs the final air combat intelligent decision-making model, also known as the intelligent decision-making network.

[0162] Part VI: Designing a real-time adversarial decision-making module

[0163] The real-time confrontation decision-making module is based on the current battlefield situation information and uses the air combat intelligent decision-making model trained by the learning and training module to output the decision results. The function of the real-time confrontation decision-making module is to receive battlefield information and use the air combat intelligent decision-making model to output the decision results of the UAV, and the air combat intelligent decision model comes from the training results of the learning and training module.

[0164] The real-time confrontation decision-making module is specifically divided into two parts: information integration function and intelligent decision-making function, namely the information integration sub-module and the intelligent decision-making sub-module.

[0165] Since the real-time confrontation decision module requires the output of other modules as input, the information integration submodule is responsible for integrating the battlefield information input into the format specified by the decision network and sending it to the intelligent decision submodule. The intelligent decision submodule uses the air combat intelligent decision model obtained by the learning and training module to output the decision result in real time. The decision result is transmitted to the data processing module, processed by the instruction encoding submodule into a decision control instruction, and then sent to the battlefield support environment for execution by the local machine.

[0166] The present invention has customized the framework and network parameters of the target drone intelligent decision-making submodule according to the specific mission scenario during offline training, so the intelligent decision-making submodule directly uses the results generated by offline training. In the application scenario, the input of the intelligent decision-making submodule is the information integrated by the information integration submodule, and the output of the intelligent decision-making submodule is the maneuvering and shooting actions within the specific target drone action space. The intelligent decision-making subfunction adopts a redundant design. In actual deployment, three sets of intelligent decision-making networks with the same framework and network parameters are deployed at the same time. When the information integration submodule sends the integrated one-dimensional input vector to the intelligent decision-making submodule, the three sets of intelligent decision-making networks make decisions at the same time and integrate the decision results to avoid errors in the decision results due to force majeure such as calculation errors or computer failures when only one set of intelligent decision-making networks is deployed, thereby ensuring the safety and reliability of the decision results.

[0167] Part 7: Deploy the trained air combat intelligent decision-making model to achieve intelligent air combat decision-making

[0168] In actual deployment, thanks to the modular design, the learning and training module can be disconnected. After the battlefield support environment information is received by the data processing module, it is directly sent to the real-time confrontation decision module. The outputs of the three sets of neural networks of the intelligent decision-making function are integrated and sent to the data processing module. After the command is encoded, it is sent to the battlefield support environment to realize autonomous air combat of the UAV.

[0169] Example 2

[0170] The embodiment of the present invention provides an air combat intelligent decision-making method combining imitation learning and reinforcement learning. The method is implemented based on the system provided in Example 1, and the method includes the following steps:

[0171] Obtain battlefield situation information, including the situation of the aircraft, relative situation and the state of the aircraft's load;

[0172] Then, the battlefield situation information is processed through the pre-trained air combat intelligent decision-making model to obtain the decision results for guiding the aircraft, which include the state-action pairs consisting of the aircraft's maneuvering actions and firing actions;

[0173] Among them, the pre-trained air combat intelligent decision-making model is obtained through the following steps:

[0174] Obtain training data, including historical air combat situations and corresponding expert strategy data;

[0175] Classify the expert strategy data to obtain coarse-grained sparse expert strategy data and fine-grained dense expert strategy data;

[0176] The coarse-grained sparse expert strategy data is processed to obtain a state-action pair dataset, and then the behavior cloning algorithm is used to train the neural network architecture for imitation learning and reinforcement learning to obtain the strategy network Q 1 ;

[0177] The policy network Q 1 As the initial network in the generative adversarial imitation learning algorithm, the fine-grained dense expert strategy is imitated to obtain the strategy network Q 2 ;

[0178] The policy network Q 2 As the initial network of the decision network in the reinforcement learning algorithm framework, the policy gradient method is used to train the network until convergence to obtain the policy network Q 3 , that is, the pre-trained air combat intelligent decision-making model.

[0179] The method of the embodiment of the present invention is also implemented based on the battlefield support environment, and the relevant air combat situation data is also obtained through the battlefield support environment. The specific implementation steps of the method of the present invention are disclosed in Example 1, and will not be repeated in this embodiment.

[0180] The embodiments of the present invention are described in detail above with reference to the accompanying drawings, but the present invention is not limited to the described embodiments. For those skilled in the art, various changes, modifications, substitutions and variations of these embodiments are made without departing from the principles and spirit of the present invention, and still fall within the scope of protection of the present invention.

Claims

1. An air combat intelligent decision-making method combining imitation learning and reinforcement learning, characterized in that: The method comprises: obtaining battlefield situation information, and then processing the battlefield situation information through a pre-trained air combat intelligent decision model to obtain a decision result for guiding the aircraft, wherein the decision result includes a maneuvering action and / or a firing action of the aircraft; The air combat intelligent decision model is obtained by the following steps: Acquiring training data, wherein the training data includes expert strategy data; Classifying the expert strategy data to obtain coarse-grained sparse expert strategy data and fine-grained dense expert strategy data; The coarse-grained sparse expert strategy data is processed to obtain a state-action pair data set, and then a behavior cloning algorithm is used to train a neural network architecture for imitation learning and reinforcement learning to obtain a strategy network Q1; Using the strategy network Q1 as the initial network in the generative adversarial imitation learning algorithm, performing imitation learning on the fine-grained dense expert strategy to obtain a strategy network Q2; The policy network Q2 is used as the initial network of the decision network in the reinforcement learning algorithm framework, and the policy gradient method is used to train the network until convergence to obtain the policy network Q3, that is, the pre-trained air combat intelligent decision-making model.

2. The method according to claim 1, characterized in that: The coarse-grained sparse expert strategy data includes air combat rules and state-action pairs, and the step of processing the coarse-grained sparse expert strategy data to obtain a state-action pair data set includes: expanding the air combat rules through a decision tree model and generating new state-action pairs, merging the new state-action pairs with the state-action pairs contained in the coarse-grained sparse expert strategy data to obtain the state-action pair data set.

3. The method according to claim 1, characterized in that: When training the air combat intelligent decision-making model, it also includes behavioral constraint input and confrontation scenario input applicable to the aircraft; the behavioral constraint input includes kinematic and dynamic constraints and tactical action constraints; the confrontation scenario input includes close-range, medium-range and beyond-visual-range scenarios.

4. The method according to claim 1, characterized in that: When training the policy network Q1, the state S in the state-action pair data set is used as a feature, the action A is used as a label for regression, all actions A are One-Hot encoded, and the loss function is defined Repeat the stochastic gradient descent until the action clone algorithm converges.

5. The method according to claim 1, characterized in that: When training the strategy network Q3, adversarial training is guided by designing a reward function. The reward function design includes reward weight design, process reward function design, boundary reward function design, and offensive control area reward function design. After each reward function is designed, all rewards are aggregated and assigned their respective weights to complete the overall reward design.

6. The method according to claim 5, characterized in that: The weight of process reward shall not exceed the weight of result reward.

7. The method according to claim 5, characterized in that: The process reward includes the angle advantage function r a and the energy advantage function r e ; in, is the target azimuth, q is the target entry angle, and are the energy heights of the aircraft and the enemy aircraft respectively; for a UAV with a flight speed of V and a flight altitude of H, its energy height is H E =H+0.5V 2 / g; The process reward also includes a two-machine distance reward r R and dual aircraft height bonus r H ; Among them, R is the enemy distance vector, H r and H b It is the flight altitude of the own aircraft and the enemy aircraft.

8. The method according to claim 5, characterized in that: The boundary reward function is Among them, p1 is the punitive reward value; x is the value that triggers the reward, a is the lower boundary of the reward triggered by the reward value, and b is the upper boundary; k is a positive real number, representing the abruptness of the reward. The larger the value, the more abrupt the reward is triggered.

9. The method according to claim 5, characterized in that: The offensive control zone reward function is Among them, r1 and r2 are the reward components in two dimensions respectively, r3 is the final offensive control zone reward, x input is the entry angle, y input is the enemy distance, (a, b), (c, d) are the reward trigger boundaries of each dimension respectively, and k1 and k2 are the abruptness of the reward in each dimension.

10. An air combat intelligent decision-making system combining imitation learning and reinforcement learning, characterized in that: It includes user input module, learning and training module, data processing module and real-time confrontation decision module; The user input module is used to receive behavior constraint input, confrontation scenario input and expert strategy data input; the user input module is also used to classify and expand the expert strategy data; The data processing module is used to receive the original battlefield information sent by the battlefield support environment and process it into data for the adaptation learning and training module; The learning and training module is used to perform network training and obtain an air combat intelligent decision model based on the data provided by the user input module and the data parsing module; The real-time confrontation decision module is used to obtain battlefield situation information, and call the function of the air combat intelligent decision model to process the battlefield situation information to obtain a decision result for guiding the aircraft; The data processing module is also used to encode the decision results into decision control instructions and send them to the battlefield support environment for execution by the local machine.

Images