A method and system for intelligent decision-making in air combat that combines imitation learning and reinforcement learning

Abstract

This invention discloses an intelligent air combat decision-making method combining imitation learning and reinforcement learning, belonging to the field of air combat. The method includes: processing battlefield situation information through an intelligent air combat decision-making model to obtain decision results for guiding the aircraft. The pre-trained intelligent air combat decision-making model is obtained through the following steps: for coarse-grained sparse expert policy data, a behavior cloning algorithm is used to train a neural network architecture for imitation learning and reinforcement learning to obtain a policy network Q1; the policy network Q1 is used as the initial network in a generative adversarial imitation learning algorithm to perform imitation learning on fine-grained dense expert policies to obtain a policy network Q2; the policy network Q2 is used as the initial network in a reinforcement learning algorithm framework for decision network training, and the policy gradient method is used to train the network until convergence to obtain the intelligent air combat decision-making model. This invention is based on the ability to effectively improve sample utilization and reduce cumulative error.

Description

Technical Field

[0001] This invention relates to the field of aviation technology, and in particular to an intelligent decision-making method and system for air combat that combines imitation learning and reinforcement learning. Background Technology

[0002] The rapid development of drone technology has brought innovative solutions to many fields, especially in the military. Compared with traditional manned aircraft, drones are less expensive, have lower safety risks, and can withstand greater overload. However, due to the increasing complexity of mission scenarios and the diversification of mission requirements, improving the autonomous decision-making capabilities of drones has become an urgent problem to be solved.

[0003] Currently, domestic and international methods for autonomous decision-making of unmanned aerial vehicles (UAVs) can be broadly categorized into four types: methods based on expert systems, methods based on game theory, methods based on optimization theory, and methods based on artificial intelligence.

[0004] The expert system-based approach transforms the experience of experts into a tactical knowledge base, and makes decision-making inferences based on the information in the knowledge base. This approach has good interpretability, but it is highly dependent on expert experience and pre-set rules, and is not very adaptable to rapidly changing battlefield environments.

[0005] Game theory-based methods utilize the framework and models of game theory to analyze and optimize decision-making processes in air combat. Matrix games or differential game theory are commonly used, but it is difficult to obtain analytical solutions to Nash equilibrium for highly adversarial and dynamic air combat decision-making problems.

[0006] The optimization theory-based approach models air combat decision-making as a multi-objective optimization problem, and then uses intelligent optimization algorithms to solve it. However, this approach is prone to getting stuck in local optima during the search for the optimal strategy, so it is necessary to improve its global search optimization capability and optimization accuracy.

[0007] Artificial intelligence-based methods typically require data-driven approaches, using data samples to build decision-making models. Generally, a large amount of air combat sample data is needed to achieve good results. However, in practice, it is difficult to obtain large quantities of air combat sample data from human combat scenarios.

[0008] Despite the progress made in UAV autonomous decision-making technology, several challenges remain. In complex and dynamic environments, existing decision-making methods are insufficiently adaptable, and traditional rules and algorithms struggle to effectively handle unforeseen events. To fully leverage the performance of UAVs, further improvements are needed to achieve a high degree of autonomous decision-making capability.

[0009] With the development of artificial intelligence technology, deep reinforcement learning (DRM) has been successfully applied in many fields. DRM combines the advantages of deep learning and reinforcement learning, and its main characteristics include: First, DRM uses deep neural networks to automatically extract features, enabling it to handle high-dimensional state spaces and effectively cope with complex environments; second, DRM learns policies through interaction with the environment, giving it adaptability and autonomous decision-making capabilities, allowing it to optimize behavior in dynamic and uncertain environments; furthermore, DRM possesses the inherent trial-and-error mechanism of reinforcement learning, achieving self-optimization of policies through continuous exploration and utilization of equilibrium.

[0010] However, deep reinforcement learning also has some problems. Because it requires continuous trial and error through interaction with the environment to optimize the decision network until convergence, the decision model needs a large amount of sample data to fully explore and learn the value of each state and action, leading to low sample efficiency. Simultaneously, due to its large number of parameters and complex structure, training the decision model is very time-consuming, which also makes it prone to accumulating errors. This could even result in the time-consuming and labor-intensive training of an intelligent air combat decision model that is neither "intelligent" nor applicable to "air combat." Summary of the Invention

[0011] This invention provides an intelligent decision-making method and system for air combat that combines imitation learning and reinforcement learning, so as to at least partially solve the above-mentioned problems.

[0012] To achieve the above objectives, the technical solution of the present invention is as follows:

[0013] In a first aspect, the present invention provides an air combat intelligent decision-making method that combines imitation learning and reinforcement learning. The method includes: acquiring battlefield situation information, and then processing the battlefield situation information through a pre-trained air combat intelligent decision-making model to obtain decision results for guiding the aircraft. The decision results include the aircraft's maneuvering actions and / or missile firing actions.

[0014] The air combat intelligent decision-making model is obtained through the following steps:

[0015] Acquire training data, including expert policy data;

[0016] The expert strategy data is classified to obtain coarse-grained sparse expert strategy data and fine-grained dense expert strategy data.

[0017] The coarse-grained sparse expert policy data is processed to obtain a state-action pair dataset. Then, the behavior cloning algorithm is used to train a neural network architecture for imitation learning and reinforcement learning to obtain the policy network Q1.

[0018] The policy network Q1 is used as the initial network in the generative adversarial imitation learning algorithm to perform imitation learning on the fine-grained dense expert policy, thereby obtaining the policy network Q2.

[0019] The policy network Q2 is used as the initial network of the decision network in the reinforcement learning algorithm framework. The policy gradient method is used to train the network until convergence, and the policy network Q3 is obtained, which is the pre-trained air combat intelligent decision model.

[0020] The coarse-grained sparse expert strategy data includes air combat rules and state-action pairs. The step of processing the coarse-grained sparse expert strategy data to obtain a state-action pair dataset includes: expanding the air combat rules through a decision tree model and generating new state-action pairs, merging them with the state-action pairs contained in the coarse-grained sparse expert strategy data to obtain the state-action pair dataset.

[0021] The training of the air combat intelligent decision-making model also includes behavioral constraint inputs and adversarial scenario inputs applicable to the local machine; the behavioral constraint inputs include kinematic and dynamic constraints as well as tactical action constraints; the adversarial scenario inputs include close-range, medium-range, and beyond-visual-range scenarios.

[0022] In some preferred embodiments, when training the policy network Q1, the states S in the state-action pair dataset are used as features, and actions A are used as labels for regression. One-hot encoding is performed on all actions A, and a loss function is defined. Repeat the stochastic gradient descent process until the action cloning algorithm converges.

[0023] In some preferred embodiments, when training the policy network Q3, adversarial training is guided by designing reward functions. The reward function design includes reward weight design, process reward function design, boundary reward function design, and attack control zone reward function design. After each reward function design, all rewards are aggregated and their respective weights are assigned to complete the overall reward design.

[0024] In some preferred embodiments, the process reward weight does not exceed the result reward weight.

[0025] In some preferred embodiments, the process reward includes an angle advantage function r. a and energy advantage function r e ;

[0026]

[0027] in, Let θ be the target azimuth angle and q be the target approach angle. and These are the energy altitudes of the local aircraft and the enemy aircraft, respectively; for a UAV with a flight speed of V and a flight altitude of H, its energy altitude is H. E =H + 0.5V 2 / g;

[0028] The process reward also includes a two-machine distance reward r. R and dual-machine height bonus r H ;

[0029]

[0030] Where R is the enemy-friendly distance vector, H r and H b This refers to the flight altitude of both the aircraft and the enemy aircraft.

[0031] In some preferred embodiments, the boundary reward function is

[0032]

[0033] p1 is the punitive reward value; z is the value that triggers the reward; n is the lower boundary of the reward value that triggers the reward; m is the upper boundary; k is a positive real number that represents the abruptness of the reward. The larger the value, the more abrupt the reward is triggered.

[0034] In some preferred embodiments, the reward function for the attack control zone is:

[0035]

[0036] Where r1 and r2 are the reward components in the two dimensions, r3 is the final attack control zone reward, x is the entry angle, y is the enemy-ally distance, (a, b) and (c, d) are the reward trigger boundaries in each dimension, and k1 and k2 are the abruptness of the rewards in each dimension.

[0037] Secondly, the present invention also provides an air combat intelligent decision-making system that combines imitation learning and reinforcement learning, including a user input module, a learning and training module, a data processing module, and a real-time adversarial decision-making module.

[0038] The user input module is used to receive behavioral constraint input, adversarial scenario input, and expert strategy data input; the user input module is also used to classify and expand the expert strategy data.

[0039] The data processing module is used to receive the raw battlefield information sent by the battlefield support environment and process it into data for the adaptation learning and training module.

[0040] The learning and training module is used to perform network training based on the data provided by the user input module and the data processing module and obtain an intelligent air combat decision-making model.

[0041] The real-time adversarial decision-making module is used to acquire battlefield situation information and call the functions of the air combat intelligent decision-making model to process the battlefield situation information and obtain decision results to guide the machine.

[0042] The data processing module is also used to encode the decision results into decision control instructions and send them to the battlefield support environment for execution by the local machine.

[0043] The beneficial effects of this invention, employing the above technical solution, are as follows: Based on various forms of expert strategy data, this invention uses an imitation learning framework to pre-train the decision network, and then uses a reinforcement learning framework to retrain it. This combination effectively accelerates the training process, improves sample utilization, and reduces accumulated errors. This enables the invention to achieve autonomous air combat decision-making, outputting control commands based on real-time status information of the battlefield support environment; it can extract and learn from various forms of expert strategies and experience; it can establish communication with various forms of battlefield support environments, realizing intelligent air combat for UAVs, thereby significantly improving combat efficiency and decision-making capabilities; and it possesses strong generalization ability and robustness to complex and dynamic battlefield environments. Attached Figure Description

[0044] Figure 1 This is a schematic diagram of the architecture of the present invention.

[0045] Figure 2 This is a schematic diagram illustrating the data structure definition of the data processing module in this invention.

[0046] Figure 3 This is a schematic diagram of the overall process of the present invention. Detailed Implementation

[0047] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings. It should be noted that these descriptions are for the purpose of aiding understanding the present invention, but do not constitute a limitation thereof. Furthermore, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0048] Example 1

[0049] This invention provides an intelligent air combat decision-making system that combines imitation learning and reinforcement learning, including a user input module, a data processing module, a learning and training module, and a real-time adversarial decision-making module.

[0050] The present invention will now be described in further detail with reference to the accompanying drawings.

[0051] First, the intelligent air combat decision-making system (hereinafter referred to as the intelligent air combat decision-making system) combining imitation learning and reinforcement learning is based on the model-view-controller architecture pattern commonly used in software engineering. The architecture pattern is as follows: Figure 1 As shown, the architecture is divided into four layers: application layer, service layer, algorithm layer, and information processing layer.

[0052] The application layer primarily targets typical application scenarios of intelligent air combat decision-making systems, including intelligent air combat and pilot combat training.

[0053] The service layer involves several key services related to intelligent air combat training, specifically the training service for the intelligent air combat decision-making model and the generation service for adversarial strategies. The training service for the intelligent air combat decision-making model is applied during the model training phase, while the generation service for adversarial strategies is applied during the system deployment and application phase.

[0054] The algorithm layer includes imitation learning algorithms and reinforcement learning algorithms, which are mainly used in the training phase of intelligent decision-making models for air combat.

[0055] The lowest information processing layer includes the parsing and processing of battlefield support environment information and user input information.

[0056] Based on the above architecture, the intelligent air combat decision-making system provided in this embodiment of the invention includes a user input module, a data processing module, a learning and training module, and a real-time adversarial decision-making module.

[0057] It should be noted that this invention generally requires a battlefield support environment to provide data acquisition and simulation training, the form of which includes, but is not limited to, digital models, hardware-in-the-loop models, or ground stations. In this embodiment, a self-developed air combat simulation platform is used as support. This platform only provides flight scenarios and data, and does not affect the independence of this invention, nor is it a constituent part of this invention.

[0058] Before introducing the construction of the intelligent air combat decision-making system provided in the embodiments of the present invention, some necessary information related to this system will be introduced first.

[0059] The drone air combat scenario is abstracted as a Markov Decision Process (MDP), and a state space S, an action space A, and a policy network Q are designed.

[0060] Specifically, at every moment, the battlefield support environment has a corresponding situation, which is mathematically described as a state. Any decision made by an unmanned aerial vehicle (UAV) agent depends on its understanding and analysis of this state. In air combat, the state space of an UAV agent represents all the environmental information it can perceive and acquire, including its own flight and weapon status, as well as enemy information and external environmental information acquired by its perception system.

[0061] Considering that UAVs typically have weak perception capabilities, relying solely on their own radar and other equipment makes it difficult to acquire enemy situational information effectively, which is detrimental to training intelligent air combat decision-making models. Therefore, it is assumed that the UAV agent can fully acquire enemy situational information through some means (such as information injection via data link). Based on the classification of state information in air combat, the state space S is extracted as follows:

[0062] 1. Local Status S UAV This includes the local coordinates (3D), local airspeed (3D), and flight time (1D);

[0063] 2. Relative situation S Re This includes the enemy-friendly distance vector (3D), the enemy-friendly distance scalar (1D), the enemy-friendly velocity vector (3D), the angle of entry (1D), and the azimuth angle (1D).

[0064] 3. Load condition S Ca This includes the status of the chaff (1D), the number of chaff (1D), the status of the missiles (1D), and the number of missiles (1D).

[0065] Among them, the local situation S UAV and relative situation S Re All are continuous state quantities, load state S Ca These are discrete state quantities. The status of the chaff / flare includes whether the chaff / flare is currently active, and the status of the missile includes four states: unlocked, locked but not launched, launched but not hit, and launched and hit.

[0066] Specifically, an action refers to a decision made by an unmanned aerial vehicle (UAV) agent based on its current state. Action space A refers to the set of all possible actions. The construction of action space A must be complete and reasonable. Complete means it should cover all means that the UAV agent can utilize in aerial combat, and reasonable means that action design must be within specified constraints. For actions with constraints under specific states, corresponding shielding mechanisms should be designed.

[0067] Specifically, due to the complexity of air combat, the design of the action space A should be as streamlined as possible, minimizing its dimensions while meeting basic control requirements. This can be achieved by not directly controlling the control surfaces of the UAV, but rather constructing a maneuver library based on actual maneuverability and specific mission scenarios. The UAV agent then selects combinations of tactical actions during decision-making, thereby reducing the convergence difficulty of imitation learning and reinforcement learning training.

[0068] Based on the characteristics of the maneuvering actions of unmanned aerial vehicle (UAV) agents, the action space A is classified and designed as follows:

[0069] 1. Maneuvers: including level flight, level flight acceleration and deceleration, climb, turn, dive, lateral cut, half roll inverted turn, S-maneuver, and angled loop;

[0070] 2. Launching actions: including launching infrared missiles, chaff, and infrared flares.

[0071] The action space A constructed above is a discrete action space. Since each maneuver requires specific target parameters (e.g., turning requires specifying the turning direction, turning radius, and slope) in maneuver design, to cover as many tactical action spaces as possible, one or more sets of discrete parameters are given as typical maneuver parameters when designing maneuvers, taking into account user input constraints and the adversarial scenario. This achieves full discretization of the action space. The discretized action space A can be represented as:

[0072] A = A m ×A s

[0073] Where A is the action space, A m For maneuvering, A s For the action of firing a projectile, at any decision moment, the UAV agent selects one of the maneuvering action and the action of firing a projectile to execute, and provides the corresponding execution parameters (discretization).

[0074] The decision network Q refers to a neural network architecture capable of being used for imitation learning and reinforcement learning. The system provided in this invention relates to the training and application of an intelligent air combat decision-making model. This model is obtained by performing two processes on the decision network Q: imitation learning pre-training and reinforcement learning retraining. These two processes essentially operate on a network architecture with the same structure; therefore, it is necessary to ensure the consistency of the network structure during training. Specifically, a four-layer fully connected neural network needs to be established, where the input layer's input is a state space vector and the output layer's output is an action space vector. Full connections are used between the input layer and hidden layers, between hidden layers, and between hidden layers and the output layer.

[0075] Specifically, the input to the decision network Q is the state space vector S = [S... UAV S Re S Ca The output is the entire action space A = [A] m A s A set of probability distributions is used to select the action with the highest probability as the output of the decision. This output is then encoded and sent to the battlefield support environment for updating the battlefield situation.

[0076] Based on the above description, this embodiment of the invention provides a detailed introduction to the user input module, data processing module, learning and training module, and real-time adversarial decision-making module.

[0077] Part 1: Designing the User Input Module

[0078] The user input module consists of three parts: behavioral constraint input, adversarial scenario input, and expert strategy data input.

[0079] Behavioral constraints inputs include two parts: kinematic and dynamic constraints, and tactical maneuver constraints. Kinematic and dynamic constraints mainly consist of various extreme performance values ​​for the corresponding UAV model, including extreme pitch and roll angles limited by the aircraft's structural strength, extreme normal and tangential overloads, maximum and minimum flight speeds, minimum and maximum altitudes, etc. Tactical maneuver constraints, based on the maneuver space design, mask maneuvers as needed.

[0080] The input scenarios for adversarial operations include close-range, medium-range, and beyond-visual-range scenarios, with different action libraries determined for each scenario. Specifically, different types of missiles are required for different scenarios.

[0081] The expert strategy data of the simulated object is the foundation for subsequent imitation learning. Ideally, the actual movement trajectory in air combat can be directly used as the expert strategy data of the simulated object. However, the intelligence data actually obtained may be insufficient in quantity to support the implementation of imitation learning, and may not even be a trajectory composed of state-action pairs, but rather other more ambiguous and coarse-grained forms. Therefore, different schemes are needed to learn the expert strategy data information for expert strategy data of different granularities. This invention designs expert strategy data extraction schemes for two different granularities of expert strategy data.

[0082] 1. Fine-grained, dense expert strategy data

[0083] When the expert strategy data is dense enough, it is processed into a trajectory dataset of simulated objects:

[0084] D Dense={τ1,τ2,...,τ n}

[0085] In the formula, This refers to the i-th trajectory of the simulated object; the superscript i represents the i-th trajectory, and the subscripts 1 to m. i Represents the order in which things are sorted by time, for example... Represents the m-th trajectory sorted by time from the i-th trajectory. i Status information, Represents the mth i Individual action information. Dataset D Dense This represents dense expert policy data with fine granularity representing the simulated objects.

[0086] 2. Coarse-grained sparse expert strategy data

[0087] Suppose that the expert strategy data is in the form of large-granularity descriptive statements, for example: "When the enemy is to our right rear, we will perform an S-shaped maneuver or a half-roll reversal maneuver." This data format provides, or can be converted into, labels for state-action pairs, which can distinguish whether a state-action pair belongs to a subset of the state space and action space provided by the descriptive statements in the intelligence, thus providing a foundation for supervised learning. Therefore, the information from the coarse-grained sparse expert strategy data of the simulated object is extracted into the following form:

[0088] D Sparse ={(S1, A1),(S2, A2),...,(S n A n )}

[0089] In the formula, S i and A i These are the state space subset and action space subset corresponding to the i-th descriptive statement, respectively. The dataset D mentioned above... Sparse This represents coarse-grained sparse expert policy data representing the simulated objects.

[0090] Part Two: Designing the Data Processing Module

[0091] The data processing module comprises two core functions: data parsing and instruction encoding. The data parsing submodule is responsible for receiving information sent by the battlefield support environment and processing it into a specified format; the instruction encoding submodule packages the decision instructions output by the real-time adversarial decision-making module and sends them to the battlefield support environment.

[0092] The data processing module is used for real-time communication with the battlefield support environment, receiving battlefield information data packets under standard protocols. Data transmission needs to consider requirements such as reliability, security, efficiency, and real-time performance. Taking into account multiple factors, the standard MAVLink (Micro Air Vehicle Link) protocol is preferred as the communication protocol.

[0093] MAVLink is a lightweight messaging protocol used for communication between drones and between drone-borne devices. In MAVLink, STX represents the start of data packet; LEN represents the length of the PAYLOAD; SEQ represents the sequence number (0-255), which increments with each data packet sent, used to detect packet loss; SYSID represents the device's system-level ID, indicating its identity; COMP ID represents the device component-level ID; multiple different components within the same system may use MAVLink for communication; and PAYLOAD stores the communication data. Each data frame consists of a 6-byte header, up to 255 bytes of data, and 2 bytes of checksum data, along with a 255-byte serial checksum to ensure reliable data transmission. It is suitable not only for point-to-point communication but also easily extended to multi-point communication.

[0094] The battlefield support environment uses the standard MAVLink protocol to package raw battlefield information data and sends it to the data processing module via TCP / IP. The raw battlefield information data includes local situational awareness, relative situational awareness, and payload status information, in the following specific formats: Figure 2 As shown in section 2-1, the local situation information and relative situation information are set to float type, and the payload status information is set to unit8 type. Since the battlefield information includes navigation information of both sides in the battlefield and the UAV's own raw data, this data may be received in multiple data frames. Therefore, a battlefield information parsing process can be regarded as a concurrent process of parsing multiple data frames. The MSG of the data frame can determine the meaning of the payload information (PAYLOAD) in the data frame, and store the data in the corresponding space in the input buffer according to the meaning of the data. When the battlefield information data in the input buffer is complete, the data preprocessing module reads the data from the buffer. Then, the data can be preprocessed as needed to convert the raw information into the required form, such as angle conversion, unit conversion, etc.

[0095] The decision control commands generated by this system for controlling the UAV are packaged into a predefined structure under the MAVLink standard protocol by the command encoding submodule in the data processing module and then sent. The remote control command frame structure of the UAV agent is designed according to the MAVLink protocol. To ensure that each action command can be packaged into a remote control command frame, the space occupied by each action and its parameters needs to be considered in the design of the remote control command frame structure. The payload information in the MAVLink protocol data frame is divided into target ID, maneuver action, firing action, and maneuver command parameters, such as... Figure 2 As shown in section 2-2, the target ID, maneuver, and missile firing action can be set to uint8 type, while the maneuver parameters can be set to float type. The decision control commands are encoded according to the defined remote control command frame structure. For specific mission scenarios, each maneuver in the UAV maneuver library is pre-defined with a number in the air combat intelligent decision system and battlefield support environment, such as 1-half roll, 2-circling maneuver, 3-S-shaped maneuver, etc. During online application, the maneuver commands in the decision control commands are converted into uint8 and float data with the corresponding numbers, thus achieving maneuver command encoding. Missile firing commands use the same method as the maneuver command numbers, such as 1-no chaff, 2-release chaff, etc. The missile firing action number is written into the MAVLink protocol remote control command frame, thus achieving UAV missile firing action command encoding.

[0096] The instruction encoding submodule is also responsible for sending the encoded decision control instructions to the battlefield support environment, realizing data uplink transmission. Typically, a physical connection such as a serial port is used to directly connect the system and the battlefield support environment to achieve uplink transmission of decision control instructions; however, when conditions permit, the TCP / IP communication protocol can also be used to transmit decision control instructions over a network.

[0097] Part Three: Design Based on Imitation Learning Environment

[0098] Specifically, this includes behavior cloning algorithms and generative adversarial imitation learning algorithms.

[0099] In the first part, the user input module categorizes the expert policy data. The fine-grained, dense expert policy data is processed into a trajectory dataset D of the simulated objects. Dense Coarse-grained sparse expert policy data is processed into a labeled state-action dataset D. Sparse .

[0100] For coarse-grained sparse expert policy data (such as air combat rules and state-action pairs), the behavior cloning method is used; for fine-grained data, the generative adversarial imitation learning (GAIL) method is used.

[0101] Specifically, coarse-grained fuzzy expert data, due to its explicit state-action mapping relationship, can be divided into two types: air combat rules and state-action pairs. For air combat rules, a decision tree model is constructed. For a state covered by intelligence information, if the intelligence indicates a unique maneuver, the branch is executed directly; if the intelligence indicates multiple responses or provides relatively vague guidance, a random branch under that node is executed. This decision tree is then used to generate a large number of new samples in the form of state-action pairs, which are then merged with the original state-action pair data to form a state-action pair dataset.

[0102] Then, based on the state-action pair dataset, the policy network Q is trained using the behavior cloning algorithm to obtain the policy network Q1. During behavior cloning, all state-action pairs constitute the dataset D. Sparse Using state S as the feature and action A as the label for regression, one-hot encoding is performed on all actions A, and the loss function is defined as:

[0103]

[0104] The network parameters pi represent the neural network parameters. The smaller the loss function, the closer the policy network Q1's decision is to the tactical decision in the state-action pair dataset. For a given state-action pair dataset, stochastic gradient descent is repeatedly performed until the algorithm converges.

[0105] The decision network Q1 obtained through behavior cloning is an imitation learning of coarse-grained sparse expert policy data. For fine-grained dense expert policy data in the form of trajectory datasets, the Generative Adversarial Imitation Learning (GAIL) algorithm is adopted. GAIL is essentially an inverse reinforcement learning method. Through the game between the generator and the discriminator, the expert behavior itself can be learned directly, and the decision network can be generated directly.

[0106] First, the policy network Q1 interacts with the battlefield support environment to obtain the trajectory under the current policy. Simultaneously, a discriminator network D is randomly initialized to distinguish samples (S) from the UAV agent. t A t S t+1 )~D π Compared with the fine-grained dense expert policy dataset D Dense Sampling (S) t A t S t+1 )~D e Then, a reward function is defined using the discriminator's prediction:

[0107] R(S,A)=-ln(1-D(S,A))

[0108] Where D(S, A) represents the probability that the discriminator classifies the state-action pair as an expert. In each round of training, the discriminator is first updated, and then trained on a drone agent to maximize this reward, thus fooling the discriminator. Similar to generative adversarial networks, the training of the discriminator and the agent is alternating. Overall, GAIL follows these steps:

[0109] (1) Use the current strategy to interact with the battlefield support environment and store the air combat trajectory of this round in the experience buffer.

[0110] (2) Update the discriminator.

[0111] (3) Update the policy network.

[0112] The generative adversarial imitation learning algorithm process is as follows:

[0113]

[0114]

[0115] Part Four: Designing a Reinforcement Learning-Based Environment

[0116] Specifically, this includes the design of the reward function.

[0117] As described above, a decision network Q2 can be generated after imitation learning of expert policy data. Although the training of the imitation learning algorithm includes both the process of "imitation" and the process of "optimization," its fundamental starting point is to extract typical tactics and strategies from experts. Therefore, this method is insufficient in exploring the state space. That is, in air combat situations where intelligence data cannot cover, the decision network method trained by the GAIL algorithm will still try to "fit" the expert policy data as much as possible. From a human subjective perspective, although it has a certain interpretability, in most cases, the decision in the state space not covered by expert policy data is not optimal. Therefore, this embodiment of the invention proposes a further optimization approach after the generation of the decision network Q2. The decision network Q2 obtained by imitation learning is used as the initial policy network, and adversarial training is continued in a reinforcement learning environment to finally obtain a decision network that is optimal in the entire state space.

[0118] Among the current mainstream reinforcement learning algorithms, the Actor-Critic (AC) architecture integrates the advantages of traditional value-based and policy-based reinforcement learning methods, and this method has gradually become the mainstream architecture in the field of reinforcement learning. Based on this, the Deep Deterministic Policy Gradient (DDPG) method has been proposed to solve the problem of "dimensional explosion" in output when traditional methods face high-dimensional decision-making problems, and realize action output in continuous space. This method performs well in the field of autonomous air combat. In addition, some researchers have used reinforcement learning methods to self-evolve air combat actions to achieve tactical and operational innovation.

[0119] Specifically, in the decision network optimization phase, a process reward function needs to be designed to guide adversarial training. The reward function design includes four parts: reward weight design, process reward design, boundary reward design, and control area reward design.

[0120] 1. Reward weight design

[0121] Process rewards are based on human design and have a certain degree of subjectivity; while outcome rewards are objective, they only occur at the end. Therefore, the principle of weight design is that the weight of process rewards should not exceed the discounted outcome rewards, in order to ensure that the agent has sufficient motivation to choose outcome-oriented maneuvering schemes, and ultimately the agent's strategy tends to favor objective outcomes.

[0122] Specifically, based on experience, the duration of air combat is between 4 and 10 minutes. If the step size is 4 seconds and the decay weight is 0.99, then the weight of the result reward after the minimum decay during the air combat process is about 0.2. The requirement is met if the weight of the process reward does not exceed 0.2.

[0123] 2. Process Reward Design

[0124] The process rewards are designed based on the pilot's domain knowledge. According to the Basic Fighter Maneuvering (BFM) course description, close-range air combat is a process of balancing angular and energy advantages; therefore, the angular advantage function r is designed. a and energy advantage function r e As a primary process reward.

[0125]

[0126] In the formula, Let θ be the target azimuth angle and q be the target approach angle. and Let H be the energy altitude of the drone and the enemy drone, respectively. For a drone flying at speed V and altitude H, its energy altitude is defined as H. E =H + 0.5V2 / g.

[0127] To encourage agents to "bravely" engage the enemy, a two-machine distance reward system was also designed. R and dual-machine height bonus r H Specifically:

[0128]

[0129] In the formula, R is the enemy-friendly distance vector, and H r and H b This refers to the flight altitude of both the aircraft and the enemy aircraft.

[0130] 3. Boundary Reward Design

[0131] During agent training, dangerous behaviors such as "crashing into the sea," "stall," and "overspeeding" may occur due to values ​​exceeding safe limits, leading to air combat failure. Setting conditions to trigger rewards for such situations results in sparse, discontinuous, and gradient-less rewards, which is detrimental to training. This invention designs a "cup-shaped" function to optimize these rewards. The function structure is as follows:

[0132]

[0133] In the formula, p1 is the punitive reward value; z is the value that triggers the reward; n is the lower boundary of the reward value triggering the reward; m is the upper boundary; k is a positive real number representing the abruptness of the reward, with a larger value indicating a more abrupt reward trigger. This reward is continuous and differentiable, which is beneficial for neural network optimization.

[0134] 4. Control Zone Reward Design

[0135] Specifically, in aerial combat, there is an important area concept called the Offensive Control Zone (OCZ). Within the OCZ, the attacking side can easily "tail" the defending side, and under normal circumstances, the defending side finds it difficult to shake off the attacker. The design bonuses for the OCZ are:

[0136]

[0137] Where r1 and r2 are the reward components in the two dimensions, r3 is the final OCZ reward, x is the entry angle, y is the enemy-ally distance, (a,b) and (c,d) are the reward trigger boundaries in each dimension, and k1 and k2 are the abruptness of the reward in each dimension. Specifically, (a,b) and (c,d) take values ​​of (0,30) and (5,15) respectively, and k1 and k2 take values ​​of 1.3 and 2.3 respectively.

[0138] By summing up all the above rewards and assigning their respective weights, we can obtain the overall reward design.

[0139] Among numerous reinforcement learning algorithms, Proximal Policy Optimization (PPO) stands out for its strong adaptability and stable training. PPO is a policy gradient algorithm, and policy gradient updates are highly sensitive to step size, making it difficult to choose an appropriate step size. This can easily lead to excessive differences between the old and new policies during training, hindering the development of effective policies. PPO's objective function can be iteratively updated with a small number of samples over multiple training rounds, solving the problems of difficult-to-determine step size and excessive update differences in policy gradient algorithms. Furthermore, PPO combines the high data utilization efficiency of offline policy methods with the stability of online policy methods, making it highly suitable for retraining UAV agents in air combat scenarios.

[0140] The PPO algorithm is based on the Actor-Critic framework, consisting of two Actor networks and one Critic network. It trains the network by collecting samples online (on-policy), meaning the policy used to generate the samples is the same as the policy used for training. Specifically, the retraining steps for the decision network based on the PPO algorithm are as follows: Figure 3 The content shown in the "Reinforcement Learning" box is as follows.

[0141] The Actor network is based on the state S at time step t. t Generate an action probability distribution, and the agent randomly samples an action A from the generated action probability distribution. t The environment is based on the action A performed by the intelligent agent. t Reward the agent and proceed to the next state S t+1 and the quadruple t A t R t S t+1 The data is stored in the experience pool. After a certain amount of data is collected, data collection stops. The loss functions of the Actor network and the Critic network are calculated and backpropagated. The parameters of Actor network 1 and Critic network are updated in multiple rounds. Then, the parameters of Actor network 1 are periodically copied to Actor network 2. The two steps of data collection and network update are alternated to continuously improve the agent's strategy.

[0142] Specifically, the policy network Q2 obtained through imitation learning is used as the initial network for Actor Network 1 and Actor Network 2. During training, the agent samples actions from the action probability distribution output by the Actor network, performs actions to interact with the environment and collect data, and then updates the network parameters of Actor Network 1 according to the following formula.

[0143] ​

[0144] In the formula π θ Represents a random strategy. Indicates the advantage function estimation at each time step:

[0145] A π (s t ,a t )=Q π (s t a t )-V π (s t )

[0146] To improve data utilization efficiency, two networks were constructed, Actor1 representing the new policy and Actor2 representing the old policy, incorporating importance sampling. During network training, the ratio of the new and old policies was used to correct the collected data, enabling data reuse. The parameters of Actor1, representing the new policy, are updated through policy gradient ascent. Actor2, representing the old policy, periodically copies the parameters of the new network. To limit the step size in the policy space at each step and ensure that the difference between the distributions of the new and old policies is not too large, the ratio of the new and old policies was truncated. The improved Actor network loss function is as follows:

[0147]

[0148] Where, r t (θ) represents the ratio of the old and new strategies, and ∈ is a hyperparameter. This is an estimate of the advantage function.

[0149] Critic networks are used to estimate state values ​​V(S) t ) and calculate the advantage function The Critic network takes the current state vector as input and outputs the estimated state value. Parameter updates are performed by minimizing the Critic network's loss function, resulting in more accurate value estimation.

[0150] The algorithm flow can be described as follows.

[0151]

[0152] Part Five: Designing Learning and Training Modules

[0153] Based on the information disclosed in Parts 3 and 4 above, the learning and training module models the interaction environment between the UAV intelligent agent and the adversarial training, constructing the essential elements required for inverse reinforcement learning and reinforcement learning. The input to the learning and training module consists of the behavioral constraints, adversarial scenarios, and expert policy data output by the user input module, and the output is the trained air combat policy network Q3, i.e., the air combat intelligent decision-making model.

[0154] The learning and training module specifies the limits of pitch and roll angles, limits of normal and tangential overload, maximum and minimum flight speeds, minimum flight altitudes and maximum climb altitudes of the UAV agent based on behavioral constraints, and blocks maneuvers that do not meet the requirements.

[0155] The learning and training module determines different tactical action libraries based on specific adversarial scenarios. Depending on the specific adversarial scenario and mission, the learning and training module can determine the aircraft's adversarial methods, whether the adversarial process has ended, and provides evaluation criteria for establishing the reward model.

[0156] The learning and training module selects an appropriate imitation learning algorithm based on the form and granularity of the expert policy data of the input simulated object, and adopts different schemes to learn the expert policy data information.

[0157] The training process of the air combat intelligent decision-making model combines imitation learning and reinforcement learning. The training process can be described as follows:

[0158] First, the performance limits and maneuver library of the drone are determined based on behavioral constraints and adversarial scenario inputs.

[0159] Then, the expert policy data input is classified. For coarse-grained sparse expert policy data, a decision tree model is first built to generate a large number of state-action pairs, and then the behavior cloning algorithm is used to obtain the decision network Q1. After that, this decision network Q1 is used as the initial network in the generative adversarial imitation learning algorithm, and the fine-grained dense expert policy data is used as the expert trajectory. The generative adversarial imitation learning framework is used to train the decision network Q2.

[0160] Finally, the decision network Q2 obtained from the generative adversarial imitation learning framework is used as the initial network for Actor1 and Actor2 networks in the reinforcement learning algorithm framework. The parameters of Actor1 and Critic networks are updated in multiple rounds. Then, the parameters of Actor1 are periodically copied to Actor2. The two steps of data acquisition and network update are alternately performed. During the training process, the enemy aircraft adopts a fixed expert policy. The training continues until convergence, and the decision network Q3 is obtained.

[0161] Finally, the learning and training module outputs the final air combat intelligent decision-making model, also known as the intelligent decision-making network.

[0162] Part VI. Design of Real-Time Adversarial Decision-Making Module

[0163] The real-time adversarial decision-making module outputs decision results based on the current battlefield situation information and the air combat intelligent decision-making model trained by the learning and training module. The function of the real-time adversarial decision-making module is to receive battlefield information and output the UAV's decision results using the air combat intelligent decision-making model, which comes from the training results of the learning and training module.

[0164] The real-time adversarial decision-making module is specifically divided into two parts: information integration function and intelligent decision-making function, namely the information integration sub-module and the intelligent decision-making sub-module.

[0165] Since the real-time adversarial decision-making module requires the output of other modules as input, the information integration submodule is responsible for integrating the acquired battlefield information into the format specified by the decision network and sending it to the intelligent decision-making submodule. The intelligent decision-making submodule uses the air combat intelligent decision-making model obtained by the learning and training module to output decision results in real time. These decision results are transmitted to the data processing module, where the instruction encoding submodule processes them into decision control instructions, which are then sent to the battlefield support environment for local execution.

[0166] In this invention, the framework and network parameters of the target drone's intelligent decision-making submodule have been customized according to specific task scenarios during offline training. Therefore, the intelligent decision-making submodule directly uses the results generated during offline training. In the application scenario, the input of the intelligent decision-making submodule is the information integrated by the information integration submodule, and the output of the intelligent decision-making submodule is the maneuvering and firing actions within the specific target drone's action space. The intelligent decision-making subfunction adopts a redundant design. In actual deployment, three sets of intelligent decision-making networks with the same framework and network parameters are deployed simultaneously. After the information integration submodule sends the integrated one-dimensional input vector to the intelligent decision-making submodule, the three intelligent decision-making networks make decisions simultaneously and integrate the decision results. This avoids errors in the decision results due to force majeure such as calculation errors or computer failures when only one intelligent decision-making network is deployed, ensuring the security and reliability of the decision results.

[0167] Part VII. Deploying the trained intelligent air combat decision-making model to achieve intelligent air combat decision-making.

[0168] In actual deployment, thanks to its modular design, the learning and training module can be disconnected. Information from the battlefield support environment is received by the data processing module and directly sent to the real-time adversarial decision-making module. The outputs of the three neural networks in the intelligent decision-making function are integrated and sent to the data processing module. After instruction encoding, these are sent to the battlefield support environment, enabling autonomous air combat for the UAV.

[0169] Example 2

[0170] This invention provides an intelligent air combat decision-making method that combines imitation learning and reinforcement learning. The method is based on the system provided in Embodiment 1 and includes the following steps:

[0171] Acquire battlefield situation information, including air combat situation, including the aircraft's situation, relative situation, and aircraft payload status;

[0172] Then, the battlefield situation information is processed by the pre-trained air combat intelligent decision-making model to obtain decision results to guide the aircraft. The decision results include state-action pairs consisting of the aircraft's maneuvering actions and missile firing actions.

[0173] The pre-trained air combat intelligent decision-making model is obtained through the following steps:

[0174] Acquire training data, which includes historical air combat situations and their corresponding expert strategy data;

[0175] The expert strategy data is classified to obtain coarse-grained sparse expert strategy data and fine-grained dense expert strategy data.

[0176] The coarse-grained sparse expert policy data is processed to obtain a state-action pair dataset. Then, the behavior cloning algorithm is used to train a neural network architecture for imitation learning and reinforcement learning to obtain the policy network Q1.

[0177] The policy network Q1 is used as the initial network in the generative adversarial imitation learning algorithm to imitate the fine-grained dense expert policy and obtain the policy network Q2.

[0178] The policy network Q2 is used as the initial network of the decision network in the reinforcement learning algorithm framework. The policy gradient method is used to train the network until convergence, and the policy network Q3 is obtained, which is the pre-trained air combat intelligent decision model.

[0179] The method of this invention is also implemented based on a battlefield support environment, and the relevant air combat situation data is also obtained through the battlefield support environment. The specific implementation steps of the method of this invention are disclosed in Embodiment 1, and will not be repeated in this embodiment.

[0180] The embodiments of the present invention have been described in detail above with reference to the accompanying drawings, but the present invention is not limited to the described embodiments. For those skilled in the art, various changes, modifications, substitutions, and variations can be made to these embodiments without departing from the principles and spirit of the present invention, and these variations still fall within the protection scope of the present invention.

Claims

1. An intelligent decision-making method for air combat that combines imitation learning and reinforcement learning, characterized in that, The method includes: acquiring battlefield situation information, and then processing the battlefield situation information through a pre-trained air combat intelligent decision-making model to obtain decision results for guiding the aircraft, the decision results including the aircraft's maneuvering actions and / or missile firing actions. The air combat intelligent decision-making model is obtained through the following steps: Acquire training data, including expert policy data; The expert strategy data is classified to obtain coarse-grained sparse expert strategy data and fine-grained dense expert strategy data. The coarse-grained sparse expert policy data is processed to obtain a state-action pair dataset. Then, the behavior cloning algorithm is used to train a neural network architecture for imitation learning and reinforcement learning to obtain the policy network Q1. The policy network Q1 is used as the initial network in the generative adversarial imitation learning algorithm to perform imitation learning on the fine-grained dense expert policy, thereby obtaining the policy network Q2. The policy network Q2 is used as the initial network of the decision network in the reinforcement learning algorithm framework. The policy gradient method is used to train the network until convergence, and the policy network Q3 is obtained, which is the pre-trained air combat intelligent decision model. The coarse-grained sparse expert strategy data includes air combat rules and state-action pairs. The step of processing the coarse-grained sparse expert strategy data to obtain a state-action pair dataset includes: expanding the air combat rules through a decision tree model and generating new state-action pairs, merging them with the state-action pairs contained in the coarse-grained sparse expert strategy data to obtain the state-action pair dataset. The training of the air combat intelligent decision-making model also includes behavioral constraint inputs and adversarial scenario inputs applicable to the local machine; the behavioral constraint inputs include kinematic and dynamic constraints as well as tactical action constraints; the adversarial scenario inputs include close-range, medium-range, and beyond-visual-range scenarios. When training the policy network Q1, the states S in the state-action pair dataset are used as features, and actions A are used as labels for regression. One-hot encoding is performed on all actions A, and a loss function is defined. Repeat the stochastic gradient descent process until the action cloning algorithm converges.

2. The method according to claim 1, characterized in that: When training the policy network Q3, adversarial training is guided by designing reward functions. The reward function design includes reward weight design, process reward function design, boundary reward function design, and attack control zone reward function design. After designing each reward function, all rewards are aggregated and assigned their respective weights to complete the overall reward design.

3. The method according to claim 2, characterized in that: The weight of process rewards shall not exceed the weight of outcome rewards.

4. The method according to claim 2, characterized in that: The process reward includes the angle advantage function r. a and energy advantage function r e ; in, Let θ be the target azimuth angle and q be the target approach angle. and These are the energy altitudes of the local aircraft and the enemy aircraft, respectively; for a UAV with a flight speed of V and a flight altitude of H, its energy altitude is H. E =H + 0.5V 2 / g; The process reward also includes a two-machine distance reward r. R and dual-machine height bonus r H ; Where R is the enemy-friendly distance vector, H r and H b This refers to the flight altitude of both the aircraft and the enemy aircraft.

5. The method according to claim 2, characterized in that: The boundary reward function is: Where p1 is the punitive reward value; z is the value that triggers the reward; n is the lower boundary of the reward value that triggers the reward; m is the upper boundary; k is a positive real number that represents the abruptness of the reward. The larger the value, the more abrupt the reward is triggered.

6. The method according to claim 2, characterized in that: The reward function for the attack control zone is: Where r1 and r2 are the reward components in the two dimensions, r3 is the final attack control zone reward, x is the entry angle, y is the enemy-ally distance, (a, b) and (c, d) are the reward trigger boundaries in each dimension, and k1 and k2 are the abruptness of the rewards in each dimension.

7. A system based on the air combat intelligent decision-making method combining imitation learning and reinforcement learning as described in any one of claims 1-6, characterized in that: The system includes a user input module, a learning and training module, a data processing module, and a real-time adversarial decision-making module; The user input module is used to receive behavioral constraint input, adversarial scenario input, and expert strategy data input; the user input module is also used to classify and expand the expert strategy data; The data processing module is used to receive the raw battlefield information sent by the battlefield support environment and process it into data for the adaptation learning and training module. The learning and training module is used to perform network training based on the data provided by the user input module and the data processing module and obtain an intelligent air combat decision-making model. The real-time adversarial decision-making module is used to acquire battlefield situation information and call the functions of the air combat intelligent decision-making model to process the battlefield situation information and obtain decision results to guide the machine. The data processing module is also used to encode the decision results into decision control instructions and send them to the battlefield support environment for execution by the local machine.

Images