An aircraft maneuvering flight control strategy self-learning training method and system

Abstract

The application discloses a kind of aircraft maneuvering flight control strategy self-learning training method and system, it is related to the control technical field of combat aircraft, including: based on the engine performance and flight dynamics performance of target combat aircraft, the flight dynamics model of target combat aircraft is established;Based on flight dynamics model and reinforcement learning algorithm, the reinforcement learning agent for building the control strategy of control target combat aircraft;With the initial strategy controlled combat aircraft as the opponent, the reinforcement learning agent is trained, and the trained reinforcement learning agent is obtained;Based on self-game training method, by periodically replacing the trained reinforcement learning agent as the opponent of target combat aircraft, the reinforcement learning agent is repeatedly trained to reach the target training period, and the target reinforcement learning agent is obtained.The application alleviates the technical problems that the prior art cannot obtain a wide range of effective countermeasures.

Description

Technical Field

[0001] This invention relates to the field of combat aircraft control technology, specifically to a self-learning training method and system for aircraft maneuvering flight control strategies. Background Technology

[0002] Combat aircraft maneuvering control is a combat method aimed at gaining a favorable position and achieving battlefield dominance. In actual combat, pilots not only need to master the operation of complex aircraft and the use of weapons, but also need to assess the battlefield situation based on the positions and movements of both sides' aircraft to formulate effective air combat strategies. Due to the high speed of aircraft and the rapidly changing nature of the combat situation, it is even more difficult for pilots to quickly understand the battlefield situation and formulate or change strategies. Faced with this key issue, rapid decision-making and automated decision-making in the combat domain have gradually become major research topics.

[0003] In recent years, intelligent game adversarial research has been extensive. Researchers have proposed many methods, such as expert system methods, matrix game methods, and differential game methods, which have all demonstrated a certain degree of intelligent adversarial capability in the adversarial process.

[0004] In 2019, Wang Xuan et al. explored the application of evolutionary expert system trees in UAV air combat decision-making. This research, conducted in a 2D adversarial environment with medium-range air combat as the research scenario, combined genetic algorithms with expert systems to establish an aircraft combat decision-making system based on evolutionary expert system trees. This decision-making system is constructed based on a series of rules. First, it describes a series of situations that occur during combat using if-then statements, then obtains the state identifiers corresponding to some states. The obtained state identifiers are then combined and processed by the evaluation and decision-making layers to obtain the final decision command output. Experimental results show that this decision-making system can adjust according to the battlefield situation and achieve certain winning objectives even when the user's flight performance is inferior. However, their method has significant limitations. The if-then condition identifiers are mainly based on human experience, resulting in high subjectivity. Furthermore, as the number of inputs to the decision-making system increases, the rule base grows exponentially, leading to combinatorial explosion. Additionally, strategy optimization using genetic algorithms inevitably faces the challenge of the length of the genetic algorithm's genes in the search process. This method has certain applicability in the two-dimensional space studied, but its effectiveness in the three-dimensional combat environment, which is closer to the real environment, needs further verification.

[0005] In 2022, building upon previous matrix game algorithms, Shouyi Li et al. proposed a fast algorithm for solving large-scale matrix games based on dimensionality reduction, and verified its feasibility in the field of UAV combat decision-making. The study uses the combat strategies of both sides as the smallest computational unit in the matrix game, with each side possessing an independent set of combat strategies, and assumes that each side randomly adopts a strategy during the confrontation. Based on this, the study proposes a dimension-reduction-based matrix game entanglement algorithm for large-scale matrix games and verifies the existence of Nash equilibrium in the game. However, it should be noted that this method uses adversarial strategies as the smallest test unit in the experiment, making the assumptions about adversarial strategies rather subjective. Furthermore, the assumption that both sides adopt random strategies during the confrontation is a strong constraint, making it difficult to apply in real-world adversarial environments.

[0006] Due to the large amount of computation involved in high-dimensional inputs and control of multiple continuous variables in air combat, as well as the characteristics of continuous decision-making, traditional algorithms often exhibit poor adaptability, computational complexity, and difficulty in meeting real-time requirements. Therefore, it is necessary to consider introducing more advanced methods to solve this problem.

[0007] In 2021, Dongyuan Hu effectively addressed the challenge of real-time computation by introducing reinforcement learning into the field of air combat. His research proposed a training framework combining planning algorithms, LSTM, and DQN algorithms. The aircraft's possible actions were discretized according to an action library. The planning algorithm guided the aircraft's adversarial maneuvers, and combined with the exploration mechanism of reinforcement learning, ultimately resulting in a beyond-visual-range (BVR) combat agent capable of effective maneuvers against enemy threats. However, due to the discretization of the action space according to the action library, the flexibility of aircraft maneuverability was significantly limited. This dimensionality reduction approach is effective in BVR situations where aircraft maneuverability requirements are not high, but it still has limitations in close-range air combat. Furthermore, while the reward function-based planning simplifies the reinforcement learning exploration to some extent, it also limits the breadth of action exploration.

[0008] In summary, the problems with existing technologies are:

[0009] (1) In the process of solving aircraft decision-making using traditional methods, the countermeasures and countermeasures are largely limited by the subjective prior knowledge of the rule-makers. This limits the upper limit of the countermeasures that can be obtained and makes it difficult to obtain a wide range of effective countermeasures.

[0010] (2) Existing technologies mostly use game-theoretic optimization algorithms that take combat maneuver strategies as the smallest decision-making unit. These algorithms are helpful in improving the speed of high-level decision-making, but they do not solve the problem of real-time generation of the aircraft's underlying maneuver control strategy.

[0011] (3) In the existing technology, the flight combat decision-making method based on the action library has great limitations. For example, the stability is poor during the switching phase between different actions, the adjustable range during the action completion process is small, and most of the time only one factor is considered, such as the heading or speed of the aircraft, the altitude of the aircraft, etc. There are few cases where multiple factors are considered at the same time, which does not meet the conditions for application in actual situations. Summary of the Invention

[0012] The purpose of this invention is to provide a self-learning training method and system for aircraft maneuvering flight control strategies in order to solve at least one of the above-mentioned technical problems.

[0013] In a first aspect, embodiments of the present invention provide a self-learning training method for aircraft maneuvering flight control strategies, comprising: establishing a flight dynamics model of the target combat aircraft based on the engine performance and flight dynamics performance of the target combat aircraft; constructing a reinforcement learning agent for controlling the target combat aircraft based on the flight dynamics model and a reinforcement learning algorithm; training the reinforcement learning agent with the combat aircraft controlled by the initial strategy as the adversary to obtain a trained reinforcement learning agent; and repeatedly training the reinforcement learning agent to reach a target training cycle by periodically replacing the trained reinforcement learning agent with the adversary of the target combat aircraft based on a self-game training method to obtain a target reinforcement learning agent.

[0014] Furthermore, the state space of the reinforcement learning agent includes:

[0015] [z,Δx,Δy,Δz,Δv x ,Δv y ,Δv z ,v x ,v y ,v z ,h s ,p s ,r s ,h e ,p e ]

[0016] Where z is the altitude of the aircraft, Δx, Δy, and Δz are the northward, eastward, and celestial components of the position vector difference between the aircraft and the enemy aircraft in the local coordinate system, respectively, and Δv is the altitude of the aircraft. x Δv y and Δv z These represent the northward, eastward, and celestial components of the velocity vector difference between the aircraft and the enemy aircraft in the local coordinate system, respectively. x v y and vz These represent the machine's northbound speed, eastbound speed, and skybound speed, respectively, h. s p represents the direction in which the machine head is pointing. s The pitch angle of the machine, r s h is the roll angle of the machine. e The direction in which the enemy aircraft's nose is pointing, p e The pitch angle of the enemy aircraft;

[0017] The action space of the reinforcement learning agent includes:

[0018] A t =[elevator,rudder,aileron,throttle]

[0019] Among them, elevator represents the elevator command, rudder represents the rudder command, aileron represents the aileron command, and throttle represents the throttle command.

[0020] Furthermore, the reward function for training the reinforcement learning agent includes:

[0021] R = α H *R H +α R *R R

[0022] Among them, R H R represents the single-step reward value obtained from hierarchical analysis. R α is the round reward value obtained from the radar scan judgment result. H and α R These are the coefficients for single-step reward and round reward, respectively;

[0023] R H =ω v *T v +ω a *T a +ω d *T d +ω h *T h

[0024]

[0025] In the formula, T v As a speed threat indicator, T a As an angular threat indicator, T d As a distance threat indicator, T h As a high-threat indicator, ω v ω a ω d ωh r is the weighting factor win r represents the round reward value when the player wins. loss This represents the round reward value when defeating an opponent.

[0026] Furthermore, the speed threat indicator includes:

[0027]

[0028] In the formula, v j and v i These represent the flight speeds of the local aircraft and the enemy aircraft, respectively; the angular threat indicators include:

[0029]

[0030] In the formula, q B and q R These are the angles of attack of the local aircraft and the enemy aircraft, respectively; the range threat indicators include:

[0031]

[0032] In the formula, r0 represents the distance between the local aircraft and the enemy aircraft, r j and r i These represent the maximum attack ranges of the local and enemy aircraft, respectively. u The maximum detection range of the enemy aircraft; the altitude threat indicators include:

[0033]

[0034] In the formula, h j and h i These are the flight altitudes of the aircraft and the enemy aircraft, respectively.

[0035] Furthermore, the initial strategy-controlled combat aircraft includes a combat aircraft decision model based on PID control and a fusion of preset prior knowledge.

[0036] Further, training the reinforcement learning agent includes: training the reinforcement learning agent based on the DDPG algorithm with an Actor-Critic structure; wherein, the Actor network is used to input observations, acquire actions, and generate the final actions after adding noise.

[0037]

[0038] In the formula, 'a' represents the final action, and 'π' represents the final action. θ Let θ represent the parameters of the action network, and s represent the observation of the current action. This represents the added noise; the Critic network is used to update the Q-value by calculating the difference between the current Q-value and the target Q-value.

[0039] Q 目标 =r+γQ′(s′,a′,ω′)

[0040] In the formula, r represents the reward obtained in the current step, γ represents the discount factor of the reward, Q′ represents the Critic target network, s′ and a′ represent the observation and action in the next interaction step, respectively, and ω′ is the parameter of the target network; during the update phase, the Critic network is updated using mean squared error:

[0041]

[0042] In the formula, J() represents the mean squared error loss function, and Q(s) j ,a j ,ω) represents the predicted Q-value, ω represents the network parameters, and s j and a j Let represent the observation and action of the j-th sample, respectively, and m represent the number of samples;

[0043] The Actor network is updated using the following loss gradient:

[0044]

[0045] In the formula, Represents the loss gradient function. This represents the gradient function with respect to the action. This represents the gradient function of the parameters of the action network.

[0046] Secondly, embodiments of the present invention also provide a self-learning training system for aircraft maneuvering flight control strategies, comprising: an establishment module, a construction module, a first training module, and a second training module; wherein, the establishment module is used to establish a flight dynamics model of the target combat aircraft based on the engine performance and flight dynamics performance of the target combat aircraft; the construction module is used to construct a reinforcement learning agent for controlling the control strategy of the target combat aircraft based on the flight dynamics model and a reinforcement learning algorithm; the first training module is used to train the reinforcement learning agent with the combat aircraft controlled by the initial strategy as the adversary, to obtain a trained reinforcement learning agent; the second training module is used to repeatedly train the reinforcement learning agent to reach a target training cycle by periodically replacing the trained reinforcement learning agent with the adversary of the target combat aircraft based on a self-game training method, to obtain a target reinforcement learning agent.

[0047] Thirdly, embodiments of the present invention also provide an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method provided in the embodiments of the present invention.

[0048] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing computer instructions that, when executed by a processor, implement the method provided in the embodiments of the present invention.

[0049] This invention provides a self-learning training method and system for aircraft maneuver flight control strategies. It combines reinforcement learning algorithms and self-game training methods. By standardizing and modeling multiple important situation assessment indicators of combat aircraft during combat, it establishes a reward function that conforms to the combat scenario of the aircraft. This helps to avoid decision-making fixation and overfitting caused by a single-rule opponent during training, and alleviates the technical problem of existing technologies that make it difficult to obtain widely applicable and effective adversarial strategies. Attached Figure Description

[0050] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0051] Figure 1 A flowchart illustrating a self-learning training method for aircraft maneuvering flight control strategy provided in an embodiment of the present invention;

[0052] Figure 2 This invention provides a method for constructing a local coordinate system and defining attitude angles during combat operations involving aircraft.

[0053] Figure 3 A schematic diagram illustrating the principle of a functional-level simulated radar provided for an embodiment of the present invention;

[0054] Figure 4 This invention provides a schematic diagram of the experimental results of our win rate under an initial advantageous situation, as provided in an embodiment of the invention.

[0055] Figure 5 This is a schematic diagram illustrating the experimental results of our win rate under the initial situation of equal power between ourselves and the enemy, provided by an embodiment of the present invention;

[0056] Figure 6 This invention provides a schematic diagram of the experimental results of our win rate under an initial disadvantageous situation, as provided in an embodiment of the invention.

[0057] Figure 7 This is a schematic diagram of a self-learning training system for aircraft maneuvering flight control strategy provided in an embodiment of the present invention. Detailed Implementation

[0058] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0059] Example 1

[0060] Figure 1 This is a flowchart illustrating a self-learning training method for aircraft maneuvering flight control strategies according to an embodiment of the present invention. Figure 1 As shown, the method specifically includes the following steps:

[0061] Step S102: Based on the engine performance and flight dynamics performance of the target combat aircraft, establish a flight dynamics model of the target combat aircraft.

[0062] Step S104: Based on the flight dynamics model and reinforcement learning algorithm, construct a reinforcement learning agent for the control strategy of controlling the target combat aircraft.

[0063] Step S106: Using the combat aircraft controlled by the initial strategy as the adversary, train the reinforcement learning agent to obtain a trained reinforcement learning agent.

[0064] Preferably, the initial strategy control of the combat aircraft includes a combat aircraft decision model based on PID control and a fusion of preset prior knowledge.

[0065] Specifically, in this embodiment of the invention, a decision model for an aircraft based on the fusion of PID control and expert prior knowledge is selected as the adversary. An end-to-end reinforcement learning model is constructed with rudder throttle commands and continuous action space as the decision space. By selecting the throttle position, the aircraft speed is changed; by selecting the elevator position, the aircraft pitch angle is changed; by selecting the aileron position, the aircraft roll angle is changed; and by selecting the rudder position, the aircraft coordinates its turning actions, thereby achieving changes in the aircraft's flight state, such as speed and attitude.

[0066] Step S108: Based on the self-game training method, the reinforcement learning agent is repeatedly trained by periodically replacing the trained reinforcement learning agent with the adversary of the target combat aircraft to reach the target training cycle, thereby obtaining the target reinforcement learning agent.

[0067] Specifically, based on the self-game method, a self-game adversarial environment for aircraft is built to train the basic end-to-end aircraft adversarial decision-making model. The model obtained from further training is used as an adversary to continue to compete against the basic end-to-end aircraft adversarial decision-making model, thereby improving the agent's decision-making ability. During the training process, the parameters of the trained model are periodically fixed, and the model with these parameters is used as a new adversary to continuously train the basic end-to-end aircraft adversarial decision-making model, thereby gradually improving the aircraft's adversarial strategy capabilities.

[0068] In an optional embodiment of the present invention, step S102 further includes: constructing a flight dynamics model of the target combat aircraft based on the JSBSim aircraft simulation platform.

[0069] Specifically, the experimental environment was based on the open-source simulation platform JSBSim, an open-source, cross-platform six-degree-of-freedom nonlinear flight dynamics model. Written in object-oriented C++, it supports dynamic modeling of different types of aerospace vehicles. The vehicle dynamics characteristics in the model are expressed using Extensible Markup Language (EXPLAIN), allowing users to build their own six-degree-of-freedom aircraft models and perform simulations without compiling and linking the code. The resulting aircraft simulation model includes model constraints such as engine performance, aircraft dynamics, aircraft weight, and fuel consumption calculations, which can simulate the flight conditions of real aircraft to a certain extent, and the training results also have good interpretability.

[0070] Using a rule-based adversarial strategy and PID control, a basic aircraft adversarial expert system is implemented. This expert system is capable of performing basic flight maneuvers and simple combat maneuvers, including: straight flight, climb, dive, left turn, right turn, etc.

[0071] In this embodiment, the aircraft controlled by the intelligent agent is an open-source "F-16" aircraft model, with continuous stick and rudder control as the target action output.

[0072] Figure 2 This is a diagram illustrating a method for constructing a local coordinate system and defining attitude angles during combat aircraft combat, provided by an embodiment of the present invention. The local coordinate system is constructed using a north-east-sky orientation, and ψ, θ, and φ represent the heading angle, pitch angle, and roll angle of the aircraft, respectively. Figure 3 This is a schematic diagram illustrating the principle of a functional-level simulated radar according to an embodiment of the present invention, wherein D... abλ represents the straight-line distance between the two aircraft. a This indicates the effective radar scanning range of aircraft a. Indicates the effective angle of radar scanning. This indicates the angle between the target aircraft and the direction in which the radar-scanning aircraft's nose is pointing.

[0073] Throughout the combat training, the pitch angle was randomly set within ±20 degrees, and the roll angle was randomly set within ±15 degrees, keeping the initial distance between the two aircraft less than 20km. During the training, both aircraft used the same simulation speed.

[0074] The flight combat process of aircraft is inherently rapidly changing, and the control commands of aircraft are a continuous multi-dimensional space, which to some extent increases the complexity of decision-making. However, since flight control has certain regularities, deterministic strategy algorithms can be used to find the optimal combat strategy.

[0075] Specifically, the training of the reinforcement learning part is mainly divided into two parts: environment and algorithm.

[0076] Using the JSBSim-based simulation environment, the aircraft's position, attitude, velocity, and other information can be obtained through calculation. After the environment obtains the simulation data of both sides, the states of both sides are processed according to the design of the agent's observation space to obtain the agent's single-step observation vector. This vector is then fed into the agent network for calculation to obtain the single-step action vector output. After scaling and constraint processing, the action value is fed into the simulation platform for one-step time calculation to obtain the next state. During training, such interactive processes are repeated in a single round until the round ends.

[0077] The reinforcement learning phase experiment includes an intelligent agent and an expert system based on PID control. The expert system is constructed with the help of human prior knowledge and rules about aircraft combat, and has basic maneuvering capabilities such as steering, altitude change, and speed control. The intelligent agent is a decision-making intelligent agent in the end-to-end continuous action space of rudder deflection thrust.

[0078] In this embodiment of the invention, the "F-16" is selected as the training aircraft type, and continuous stick and rudder control is used as the target action output. Regarding the observation space, the model's goal is to acquire a better combat posture during flight combat, pose a greater threat to the enemy, and ultimately shoot down the target. Considering that the information about the enemy aircraft that one aircraft can obtain during actual aircraft combat is limited, the assumption of omniscience is not adopted. Instead, only some information that can be obtained from radar and accumulated information is used. Therefore, this embodiment of the invention selects position, velocity, and attitude as the observation dimensions, and sets the state space of the reinforcement learning agent and the observation space of the training process as follows:

[0079] [z,Δx,Δy,Δz,Δv x ,Δv y ,Δv z ,v x ,v y ,v z ,h s ,p s ,r s ,h e ,p e ]

[0080] Where z is the altitude of the aircraft, Δx, Δy, and Δz are the northward, eastward, and celestial components of the position vector difference between the aircraft and the enemy aircraft in the local coordinate system, respectively, and Δv is the altitude of the aircraft. x Δv y and Δv z These represent the northward, eastward, and celestial components of the velocity vector difference between the aircraft and the enemy aircraft in the local coordinate system, respectively. x v y and v z These represent the machine's northbound speed, eastbound speed, and skybound speed, respectively, h. s p represents the direction in which the machine head is pointing. s The pitch angle of the machine, r s h is the roll angle of the machine. e The direction in which the enemy aircraft's nose is pointing, p e The pitch angle of the enemy aircraft;

[0081] The action space of a reinforcement learning agent includes:

[0082] A t =[elevator,rudder,aileron,throttle]

[0083] Among them, elevator represents the elevator command, with a minimum command of -1 and a maximum command of 1; rudder represents the rudder command, with a minimum command of -1 and a maximum command of 1; aileron represents the aileron command, with a minimum command of -1 and a maximum command of 1; and throttle represents the throttle command, with a minimum command of 0 and a maximum command of 0.9.

[0084] The reward function of a reinforcement learning agent is used to guide the agent in optimizing decision-making. In this invention, the reward function uses hierarchical analysis to estimate the situational advantages of both sides in combat, encouraging aircraft to gain a situational advantage during the confrontation. Simultaneously, based on radar simulation, the radar lock-on situation is used as an indicator for determining the end of the battle, and the battle result is reflected in the reward function, encouraging the agent to shoot down enemy aircraft. The combination of single-step and round-based reward functions solves the problem of reward sparsity during the confrontation. In summary, the reward function in this invention is expressed as:

[0085] R = α H *R H +α R *R R

[0086] Among them, R H R represents the single-step reward value obtained from hierarchical analysis. R α is the round reward value obtained from the radar scan judgment result. H and α R These are the coefficients for single-step reward and round reward, respectively.

[0087] Specifically, the single-step reward value based on the Analytic Hierarchy Process (AHP) is as follows:

[0088] R H =ω v *T v +ω a *T a +ω d *T d +ω h *T h

[0089] In the formula, T v As a speed threat indicator, T a As an angular threat indicator, T d As a distance threat indicator, T h As a high-threat indicator, ω v ω a ω d ω h This is the weighting factor.

[0090] Specifically, when quantifying the speed threat index, the main consideration is the magnitude of the speed ratio. The specific calculation of the speed threat index is expressed as follows:

[0091]

[0092] In the formula, v j and v i These represent the flight speeds of the aircraft and the enemy aircraft, respectively.

[0093] The angle threat index is highest when our attack angle is pointing in the opposite direction of the enemy's attack, while the enemy's attack angle is pointing in the opposite direction of ours. The angle threat index is lowest when our attack angle is pointing directly at the enemy target, while the enemy's attack angle is pointing in the opposite direction of ours. Specifically, the angle threat index includes:

[0094]

[0095] In the formula, q B and q R These are the angles of attack of the aircraft and the enemy aircraft, respectively.

[0096] Considering the threat posed by distance to the enemy during combat, distance threat indicators include:

[0097]

[0098] In the formula, r0 represents the distance between the local aircraft and the enemy aircraft, r j and r i These represent the maximum attack ranges of the local and enemy aircraft, respectively. u This represents the maximum detection range of enemy aircraft.

[0099] Considering the threat posed by an aircraft's altitude to an opponent's aircraft during combat, altitude threat indicators include:

[0100]

[0101] In the formula, h j and h i These are the flight altitudes of the aircraft and the enemy aircraft, respectively.

[0102] Based on the radar function of functional simulation, the result of the aircraft combat is determined, and this result is used as the result-oriented reward function for a single round. Specifically, the round reward value based on the combat result takes the following form:

[0103]

[0104] In the formula, r win r represents the round reward value when the player wins. loss This represents the round reward value when defeating an opponent.

[0105] In this embodiment of the invention, training the reinforcement learning agent includes: training the reinforcement learning agent based on the DDPG algorithm employing an Actor-Critic structure. The Actor network and the Critic network use similar neural network structures.

[0106] Specifically, the Actor network is used to take observations as input, acquire actions, and generate the final action after adding noise:

[0107]

[0108] In the formula, 'a' represents the final action, and 'π' represents the final action. θ Let be an action network, where θ represents the parameters of the action network, and s represents the observation of the current action. This indicates the added noise;

[0109] The Critic network is used to update the Q-value by calculating the difference between the current Q-value and the target Q-value.

[0110] Q 目标 =r+γQ′(s′,a′,ω′)

[0111] In the formula, r represents the reward obtained in the current step, γ represents the discount factor of the reward, Q′ represents the Critic target network, s′ and a′ represent the observation and action of the next interaction step, respectively, and ω′ is the parameter of the target network;

[0112] During the update phase, the Critic network is updated using mean squared error:

[0113]

[0114] In the formula, J() represents the mean squared error loss function, and Q(s) j ,a j ,ω) represents the predicted Q-value, ω represents the network parameters, and s j and a j Let represent the observation and action of the j-th sample, respectively, and m represent the number of samples;

[0115] The Actor network is updated using the following loss gradient:

[0116]

[0117] In the formula, Represents the loss gradient function. This represents the gradient function with respect to the action. This represents the gradient function with respect to the parameters of the action network.

[0118] Specifically, in step S108, building the self-playing adversarial environment for the aircraft involves using the same aircraft simulation model, importing the preliminary training model obtained from the reinforcement learning process, and setting the same computational step frequency. After the self-playing adversarial environment is built, the DDPG algorithm is used to train the agent. When a higher reward function is obtained, the decision model is saved, and the decision network for the mobile phone is replaced at certain intervals to continuously optimize the decision strategy.

[0119] Self-game learning refers to the process in adversarial agent training scenarios where the agent continuously engages in competition with its historical strategy versions to obtain better decision-making strategies. In reinforcement learning training, the adversary used is an expert system based on PID control, resulting in highly subjective adversarial strategies that lack variation and flexibility. While such adversaries can effectively guide the agent to acquire basic flight capabilities in the initial training phase, they lack the ability to explore more complex adversarial strategies. This invention, in a self-game adversarial environment, also uses the JSBSim platform to complete the simulation task, replacing the adversary with the same aircraft model as the agent in a reinforcement learning environment. The adversary is replaced by changing its decision-making algorithm model. Preferably, this embodiment of the invention employs periodic replacement.

[0120] To verify the effectiveness and applicability of the algorithm, this invention designs a set of adversarial evaluation systems. Experiments are designed to begin adversarial combat under conditions of our advantage, our disadvantage, and a balanced enemy. The systems evaluate the maneuverability of both sides before and after combat based on the threats posed by each side to the other in terms of altitude, speed, distance, and angle. Radar scan results are used as indicators to evaluate the outcome of a single round of adversarial combat, and the results are presented as the win rate across multiple engagements. The experimental results and data are shown below. Figures 4-6 As shown. Among them, Figure 4 This is a schematic diagram illustrating the experimental results of our win rate under an initial advantageous situation, according to an embodiment of the present invention. Figure 5 This is a schematic diagram illustrating the experimental results of our win rate under the initial situation of equal power between ourselves and the enemy, according to an embodiment of the present invention. Figure 6 This is a schematic diagram of the experimental results of our win rate under the initial disadvantageous situation provided by an embodiment of the present invention.

[0121] As described above, the embodiments of the present invention provide a self-learning training method for aircraft maneuvering flight control strategies. Compared with the prior art, the beneficial effects of the present invention include:

[0122] (1) This invention has different advantages from other algorithms applied in the field of intelligent aircraft combat in recent years. This invention has reasonably modeled the intelligent aircraft combat scenario, analyzed the rapidly changing situation in the aircraft combat environment, combined multi-dimensional continuous control during aircraft control, and finally selected reinforcement learning and self-game combat methods. It combines the research on combat strategies in non-reinforcement learning aircraft combat with the theoretical support of factors affecting the aircraft combat situation, and on this basis, it proposes innovations, adopts a combination of single-step guided reward and round target reward, and reasonably dense rewards; combined with the self-game method, it enriches the opponent's behavioral logic and generates more flexible combat strategies; the experimental results under the verification of the combat experiment show that the aircraft combat decision training method can generate a robust aircraft control decision model.

[0123] (2) In the stage of constructing the aircraft simulation model, this invention obtains simulation calculation models of different aircraft by modeling different engine parameters, flight dynamics parameters, etc. The simulation of each aircraft has high precision and high efficiency calculation effect, and the motion and control of the aircraft in the simulation are sufficiently realistic, which effectively improves the reliability and wide applicability of this method.

[0124] (3) The observations designed in the reinforcement learning part of this invention include the position, velocity, and attitude of the local aircraft, as well as the position, velocity, and some attitude information of the enemy aircraft. Certain restrictions are placed on the content of the observations, abandoning the assumption of omniscience regarding enemy aircraft information. All variables involved can be obtained through radar in actual exercises. This method of selecting observations enhances the persuasiveness of the proposed method and effectively reduces the difficulty of transferring the trained model to a real-world environment.

[0125] (4) In the core algorithm design stage, this invention breaks down the complete aircraft adversarial decision-making into two stages: basic adversarial training and decision-making capability improvement. In the initial stage, by matching and training with an expert system implemented with PID, the adversarial situation is randomly initialized within a certain range with a certain probability distribution to obtain basic flight control and decision-making capabilities. Then, through self-game, the adversarial strategy is optimized.

[0126] Example 2

[0127] Figure 7 This is a schematic diagram of a self-learning training system for aircraft maneuvering flight control strategy provided by an embodiment of the present invention. Figure 7 As shown, the system includes: a setup module 10, a construction module 20, a first training module 30, and a second training module 40.

[0128] Specifically, module 10 is established to build a flight dynamics model of the target combat aircraft based on its engine performance and flight dynamics performance.

[0129] Module 20 is used to build a reinforcement learning agent for control strategies of target combat aircraft based on flight dynamics models and reinforcement learning algorithms.

[0130] The first training module 30 is used to train the reinforcement learning agent by using the combat aircraft controlled by the initial strategy as the opponent, so as to obtain a trained reinforcement learning agent.

[0131] Specifically, the first training module 30 is also used to train the reinforcement learning agent based on the DDPG algorithm with an Actor-Critic structure;

[0132] The second training module 40 is used to train the reinforcement learning agent repeatedly by periodically replacing the target combat aircraft with the trained reinforcement learning agent as the adversary of the target combat aircraft to reach the target training cycle and obtain the target reinforcement learning agent.

[0133] The present invention also provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method provided in the embodiments of the present invention.

[0134] The present invention also provides a computer-readable storage medium storing computer instructions that, when executed by a processor, implement the method provided in the embodiments of the present invention.

[0135] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

[0136] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. A self-learning training method for aircraft maneuvering flight control strategy, characterized in that, include: Based on the engine performance and flight dynamics performance of the target combat aircraft, a flight dynamics model of the target combat aircraft is established. Based on the flight dynamics model and reinforcement learning algorithm, a reinforcement learning agent is built to control the target combat aircraft using a control strategy. Using a combat aircraft controlled by an initial strategy as the adversary, the reinforcement learning agent is trained to obtain a well-trained reinforcement learning agent. Based on the self-game training method, the target reinforcement learning agent is obtained by periodically replacing the trained reinforcement learning agent with the adversary of the target combat aircraft and repeatedly training the reinforcement learning agent to reach the target training cycle. The reward function for training the reinforcement learning agent includes: ； in, The single-step reward value obtained from hierarchical analysis. The round reward value obtained from the radar scan judgment result. and These are the coefficients for single-step reward and round reward, respectively; ； ； In the formula, As a speed threat indicator, As an angular threat indicator, As a distance threat indicator, As a high-threat indicator, , , , As a weighting factor, This represents the round reward value when the player wins. This represents the round reward value when defeating an opponent.

2. The method according to claim 1, characterized in that: The state space of the reinforcement learning agent includes: ； in, This is the altitude value of the machine. , and These represent the northward, eastward, and skyward components of the position vector difference between the local aircraft and the enemy aircraft in the local coordinate system. , and These represent the northward, eastward, and skyward components of the velocity vector difference between the aircraft and the enemy aircraft in the local coordinate system. , and These are the machine's northbound speed, eastbound speed, and skybound speed, respectively. This indicates the direction the machine head is pointing. This refers to the pitch angle of the machine. This refers to the roll angle of the machine. The direction the enemy aircraft's nose is pointing. The pitch angle of the enemy aircraft; The action space of the reinforcement learning agent includes: ； in, Indicates elevator command, Indicates rudder command, Indicates aileron command, This indicates the throttle command.

3. The method according to claim 1, characterized in that: The speed threat indicators include: ； In the formula, and These represent the flight speeds of the aircraft and the enemy aircraft, respectively. The angle threat indicators include: ； In the formula, and These are the angles of attack of the aircraft and the enemy aircraft, respectively. The distance threat indicators include: ； In the formula, Indicates the distance between this aircraft and the enemy aircraft. and These represent the maximum attack range of the local aircraft and the enemy aircraft, respectively. This represents the maximum detection range of enemy aircraft. The high threat indicators include: ； In the formula, and These are the flight altitudes of the aircraft and the enemy aircraft, respectively.

4. The method according to claim 1, characterized in that: The initial strategy control of the combat aircraft includes a combat aircraft decision model based on PID control and a fusion of preset prior knowledge.

5. The method according to claim 1, characterized in that: Training the reinforcement learning agent includes: training the reinforcement learning agent based on the DDPG algorithm employing an Actor-Critic structure; wherein, An Actor network is used to take observations as input, acquire actions, and generate the final action after adding noise. ； In the formula, 'a' represents the final action. For action networks, The parameters of the action network are represented. Indicates the observation of the current action. This indicates the added noise; The Critic network is used to update the Q-value by calculating the difference between the current Q-value and the target Q-value. ； In the formula, Indicates the target Q value. This indicates the reward obtained in the current step. The discount factor representing the reward. Indicates the Critic target network. and These represent the observation and action for the next interaction step, respectively. These are the parameters of the target network; During the update phase, the Critic network is updated using mean squared error: ； In the formula, J() represents the mean squared error loss function. Indicates the predicted Q value, Represents network parameters, s j and a j Let represent the observation and action of the j-th sample, respectively, and m represent the number of samples; The Actor network is updated using the following loss gradient: ； In the formula, Represents the loss gradient function. This represents the gradient function with respect to the action. This represents the gradient function of the parameters of the action network.

6. A self-learning training system for aircraft maneuvering flight control strategies, characterized in that, This system is used to implement a self-learning training method for an aircraft maneuvering flight control strategy as described in any one of claims 1-5; the system includes: a setup module, a construction module, a first training module, and a second training module; wherein... The establishment module is used to establish a flight dynamics model of the target combat aircraft based on its engine performance and flight dynamics performance. The construction module is used to build a reinforcement learning agent for controlling the target combat aircraft based on the flight dynamics model and reinforcement learning algorithm. The first training module is used to train the reinforcement learning agent using a combat aircraft controlled by the initial strategy as the adversary, so as to obtain a trained reinforcement learning agent. The second training module is used to train the reinforcement learning agent repeatedly by periodically replacing the trained reinforcement learning agent with the adversary of the target combat aircraft to reach the target training cycle, based on the self-game training method, so as to obtain the target reinforcement learning agent.

7. An electronic device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the method as claimed in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that, when executed by a processor, implement the method as described in any one of claims 1-5.

Images