Aircraft maneuvering flight control strategy self-learning training method and system

Abstract

The invention discloses an aircraft maneuvering flight control strategy self-learning training method and system, and relates to the technical field of combat aircraft control, and the method comprises the steps: building a flight dynamics model of a target combat aircraft based on the engine performance and flight dynamics performance of the target combat aircraft; based on the flight dynamics model and a reinforcement learning algorithm, constructing a reinforcement learning agent for controlling a control strategy of the target combat aircraft; taking the combat aircraft controlled by the initial strategy as a confronting opponent, training the reinforcement learning agent to obtain a trained reinforcement learning agent; based on a self-game training method, a trained reinforcement learning agent is periodically replaced by a confronting opponent of a target combat aircraft, the reinforcement learning agent is repeatedly trained to reach a target training period, and a target reinforcement learning agent is obtained. The technical problem that in the prior art, a wide effective confrontation strategy is difficult to obtain is solved.

Description

Technical Field

[0001] The present invention relates to the technical field of combat aircraft control, and in particular to a self-learning training method and system for aircraft maneuvering flight control strategy. Background Art

[0002] Maneuverable flight control for combat aircraft aims to gain an advantageous position and ultimately achieve a dominant battlefield position. In actual combat, pilots must not only master the control of complex aircraft and the use of weapons, but also assess the battlefield situation based on the position and movement of aircraft on both sides to formulate effective air combat strategies. Due to the rapid movement of aircraft, the combat situation is also characterized by rapid changes, making it more difficult for pilots to quickly understand the battlefield situation and formulate or change strategies. In response to this critical issue, rapid and automated decision-making in combat is becoming a major research topic.

[0003] In recent years, intelligent game confrontation has been widely studied. Researchers have proposed many methods such as expert system method, matrix game method, differential game method, etc. These methods have also demonstrated certain intelligent confrontation capabilities in the confrontation process.

[0004] In 2019, Wang Xuan et al. explored the application of evolutionary expert system trees in drone air combat decision-making. This study, using medium-range air combat as a research scenario in a two-dimensional adversarial environment, combined genetic algorithms with expert systems to establish an aircraft combat decision-making system based on evolutionary expert system trees. This decision-making system is constructed based on a series of rules. It first describes a series of combat conditions using if-then statements, then obtains state identifiers corresponding to some of these states. These state identifiers are then combined and processed by the evaluation and decision layers to produce the final decision output. Experimental results demonstrate that this decision-making system can adapt to battlefield conditions and achieve certain victory objectives even when the team's flight performance is not superior. However, their approach has significant limitations. The if-then conditional identifiers are primarily derived from human experience and are highly subjective. Furthermore, as the number of inputs to the decision-making system increases, the size of the rule base increases exponentially, leading to combinatorial explosion. Furthermore, strategy optimization using genetic algorithms cannot avoid the search difficulties inherent in the genetic algorithm's gene length. This type of method has certain applicability in the two-dimensional space studied, but its effectiveness in a three-dimensional combat environment that is closer to the real environment needs further verification.

[0005] In 2022, building on previous matrix game algorithms, Shouyi Li et al. proposed a fast algorithm for solving large-scale matrix games based on dimensionality reduction and verified its feasibility in the field of drone combat decision-making. This study used the combat strategies of the two combatants as the minimum computational unit in the matrix game. Each side possessed an independent set of combat strategies, and each side assumed to randomly adopt a strategy during the confrontation. Based on this, the study proposed a matrix game entanglement algorithm based on dimensionality reduction for large-scale matrix games and verified the existence of Nash equilibrium in the game. However, it should be noted that this method uses the adversarial strategy as the minimum test unit during the experiment, and the assumptions about the adversarial strategy are relatively subjective. In addition, the assumption that both sides adopt random strategies during the confrontation is highly restrictive, making it difficult to apply in a real-world adversarial environment.

[0006] Due to the high-dimensional input and large amount of computation required for controlling multiple continuous variables in air combat, as well as the characteristics of continuous decision-making, traditional algorithms often exhibit poor adaptability, complex calculations, and difficulty in meeting real-time requirements. This requires the introduction of more advanced methods to solve this problem.

[0007] In 2021, Dongyuan Hu effectively solved the problem of real-time computing by introducing reinforcement learning methods in the field of air combat. The study proposed a training framework that combines planning algorithms, LSTM and DQN algorithms. The actions that the aircraft can take are discretized according to the action library. With the help of the planning algorithm, the aircraft's adversarial maneuvers are guided. Combined with the exploration mechanism of reinforcement learning, they finally obtained a beyond-visual-range combat intelligent agent that can make effective maneuvers in the face of enemy threats. However, due to the discretization of the action space according to the action library, the flexibility of the aircraft's maneuvers is greatly limited. This dimensionality reduction method has certain effectiveness in beyond-visual-range situations where the requirements for aircraft maneuverability are not high, but it still has certain limitations in close-range air combat. Planning based on the reward function simplifies the exploration of reinforcement learning to a certain extent, but it also limits the breadth of action exploration.

[0008] In summary, the problems existing in the prior art are:

[0009] (1) In the process of solving aircraft decision-making using traditional methods, the confrontation strategies and confrontation methods are largely limited by the subjective prior knowledge of the rule makers, which limits the upper limit of the obtained confrontation strategies and makes it difficult to obtain a wide range of effective confrontation strategies.

[0010] (2) Existing technologies mostly use game-based optimization algorithms that use combat maneuver strategies as the minimum decision-making unit. These algorithms are helpful in improving high-level decision-making, such as solving speed, but they do not solve the problem of real-time generation of the aircraft's underlying maneuver control strategy.

[0011] (3) In the existing technology, the flight confrontation decision-making method based on the action library has great limitations. For example, the stability in the switching stage between different actions is poor, and the adjustable range during the action completion process is small. In most cases, only one factor is considered, such as the heading of the aircraft, the speed of the aircraft, the altitude of the aircraft, etc., and there are few cases where multiple factors are considered at the same time, which does not meet the conditions for application in actual situations. Summary of the Invention

[0012] The purpose of the present invention is to provide a self-learning training method and system for aircraft maneuvering flight control strategy in order to solve at least one of the above technical problems.

[0013] In the first aspect, an embodiment of the present invention provides a self-learning training method for an aircraft maneuver flight control strategy, comprising: establishing a flight dynamics model of a target combat aircraft based on the engine performance and flight dynamics performance of the target combat aircraft; building a reinforcement learning agent for controlling the control strategy of the target combat aircraft based on the flight dynamics model and a reinforcement learning algorithm; training the reinforcement learning agent with the combat aircraft controlled by the initial strategy as an opponent to obtain a trained reinforcement learning agent; based on a self-game training method, periodically replacing the trained reinforcement learning agent with the opponent of the target combat aircraft, repeatedly training the reinforcement learning agent to achieve a target training cycle, and obtaining a target reinforcement learning agent.

[0014] Furthermore, the state space of the reinforcement learning agent includes:

[0015] [z,Δx,Δy,Δz,Δv x ,Δv y ,Δv z ,v x ,v y ,v z ,h s ,p s ,r s ,h e ,p e ]

[0016] Where z is the altitude of the aircraft, Δx, Δy and Δz are the north component, east component and celestial component of the position vector difference between the aircraft and the enemy aircraft in the local coordinate system, respectively, and Δv x , Δv y and Δv z are the north component, east component and sky component of the velocity vector difference between the own aircraft and the enemy aircraft in the local coordinate system, respectively, v x 、v y and vz are the north speed, east speed and sky speed of the aircraft respectively, h s is the direction the aircraft's nose is pointing, p s is the pitch angle of the aircraft, r s is the roll angle of the aircraft, h e is the direction the enemy aircraft's nose is pointing, p e is the pitch angle of the enemy aircraft;

[0017] The action space of the reinforcement learning agent includes:

[0018] A t =[elevator,rudder,aileron,throttle]

[0019] Among them, elevator represents the elevator command, rudder represents the rudder command, aileron represents the aileron command, and throttle represents the throttle command.

[0020] Furthermore, the reward function for training the reinforcement learning agent includes:

[0021] R=α H *R H +α R *R R

[0022] Among them, R H is the single-step reward value obtained by hierarchical analysis, R R is the round reward value obtained from the radar scanning judgment result, α H and α R are the coefficients of single-step reward and round reward respectively;

[0023] R H =ω v *T v +ω a *T a +ω d *T d +ω h *T h

[0024]

[0025] Where, T v is the speed threat index, T a is the angle threat index, T d is the distance threat index, T h is a high threat indicator, ω v 、ω a 、ω d 、ωh is the weight factor, r win Indicates the round reward value when the machine wins, r loss Indicates the round reward value when winning against the opponent.

[0026] Furthermore, the speed threat indicators include:

[0027]

[0028] Where, v j and v i Respectively represent the flight speed of the own aircraft and the enemy aircraft; the angle threat indicators include:

[0029]

[0030] Where q B and q R are the attack angles of the own aircraft and the enemy aircraft respectively; the distance threat indicators include:

[0031]

[0032] In the formula, r0 represents the distance between the aircraft and the enemy aircraft, r j and r i Represents the maximum attack distance of the own aircraft and the enemy aircraft, r u is the maximum detection distance of the enemy aircraft; the altitude threat indicators include:

[0033]

[0034] Where h j and h i They are the flight altitudes of our own aircraft and the enemy aircraft respectively.

[0035] Furthermore, the combat aircraft controlled by the initial strategy includes a combat aircraft decision model fused with PID control and preset prior knowledge.

[0036] Furthermore, the reinforcement learning agent is trained, including: training the reinforcement learning agent based on a DDPG algorithm using an Actor-Critic structure; wherein the Actor network is used to input observations, obtain actions, and generate final actions after adding noise:

[0037]

[0038] Where a represents the final action, π θ is the action network, θ represents the parameters of the action network, s represents the observation of the current action, Represents the added noise; the Critic network is used to complete the update by calculating the difference between the current Q value and the target Q value:

[0039] Q 目标 =r+γQ′(s′,a′,ω′)

[0040] Where r represents the reward obtained in the current step, γ represents the discount factor of the reward, Q′ represents the critic target network, s′ and a′ represent the observation value and action of the next interaction step respectively, and ω′ is the parameter of the target network. In the update phase, the critic network is updated by the mean square error:

[0041]

[0042] Where J() represents the mean square error loss function, Q(s j ,a j ,ω) represents the predicted Q value, ω represents the network parameter, s j and a j They represent the observation value and action of the jth sample respectively, and m represents the number of samples;

[0043] The Actor network is updated with the following loss gradients:

[0044]

[0045] Where, represents the loss gradient function, represents the gradient function of the action, represents the gradient function with respect to the parameters of the action network.

[0046] In a second aspect, an embodiment of the present invention further provides a self-learning training system for an aircraft maneuverable flight control strategy, comprising: an establishment module, a construction module, a first training module and a second training module; wherein the establishment module is used to establish a flight dynamics model of the target combat aircraft based on the engine performance and flight dynamics performance of the target combat aircraft; the construction module is used to build a reinforcement learning agent for the control strategy of the target combat aircraft based on the flight dynamics model and the reinforcement learning algorithm; the first training module is used to train the reinforcement learning agent with the combat aircraft controlled by the initial strategy as the opponent to obtain a trained reinforcement learning agent; the second training module is used to repeatedly train the reinforcement learning agent based on the self-game training method, by periodically replacing the trained reinforcement learning agent with the opponent of the target combat aircraft to achieve a target training cycle, thereby obtaining a target reinforcement learning agent.

[0047] In a third aspect, an embodiment of the present invention further provides an electronic device comprising: a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor implements the method provided in the embodiment of the present invention when executing the computer program.

[0048] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and when the computer instructions are executed by a processor, the method provided in the embodiment of the present invention is implemented.

[0049] The present invention provides a self-learning training method and system for aircraft maneuver flight control strategies, which combines a reinforcement learning algorithm with a self-game training method. By standardizing and modeling multiple important situation assessment indicators of combat aircraft during the confrontation process, a reward function that conforms to the aircraft combat scenario is established. This helps to avoid decision-making fixation and overfitting caused by a single rule opponent during the training process, and alleviates the technical problem of the existing technology that it is difficult to obtain a widely effective confrontation strategy. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the implementation methods of the present application or the technical solutions in the prior art, the following is a brief introduction to the drawings required for use in the specific implementation methods or the description of the prior art. Obviously, the drawings described below are some implementation methods of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0051] Figure 1 A flowchart of a self-learning training method for an aircraft maneuvering flight control strategy provided by an embodiment of the present invention;

[0052] Figure 2 A diagram showing a method for constructing a local coordinate system and defining attitude angles during a combat aircraft confrontation process provided by an embodiment of the present invention;

[0053] Figure 3 A diagram illustrating the principle of a functional level simulation radar provided by an embodiment of the present invention;

[0054] Figure 4 A schematic diagram of a winning rate experiment result provided by an embodiment of the present invention under the initial advantageous situation of our side;

[0055] Figure 5 A schematic diagram of a winning rate experiment result of our side in an initial situation of equilibrium between the enemy and us, provided by an embodiment of the present invention;

[0056] Figure 6 A schematic diagram of a winning rate experiment result in an initial disadvantageous situation provided by an embodiment of the present invention;

[0057] Figure 7 A schematic diagram of a self-learning training system for aircraft maneuverable flight control strategies provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0058] The following will provide a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0059] Example 1

[0060] Figure 1 This is a flow chart of a self-learning training method for an aircraft maneuvering flight control strategy according to an embodiment of the present invention. Figure 1 As shown, the method specifically includes the following steps:

[0061] Step S102: establishing a flight dynamics model of the target combat aircraft based on the engine performance and flight dynamics performance of the target combat aircraft.

[0062] Step S104: Based on the flight dynamics model and the reinforcement learning algorithm, a reinforcement learning agent is constructed to implement the control strategy for controlling the target combat aircraft.

[0063] Step S106 , training the reinforcement learning agent using the combat aircraft controlled by the initial strategy as the opponent to obtain a trained reinforcement learning agent.

[0064] Preferably, the combat aircraft controlled by the initial strategy includes a combat aircraft decision model fused with PID control and preset prior knowledge.

[0065] Specifically, an embodiment of the present invention selects an aircraft decision model based on PID control and fusion of expert prior knowledge as an adversary, constructs an end-to-end reinforcement learning model with rudder throttle instructions and continuous action space as the decision space, and changes the aircraft speed by selecting the aircraft throttle stick amount, changes the aircraft pitch angle by selecting the aircraft elevator stick amount, changes the aircraft roll angle by selecting the aircraft aileron stick amount, and coordinated actions of the aircraft during coordinated turns by selecting the aircraft rudder stick amount, thereby realizing changes in the aircraft's flight states such as speed and attitude.

[0066] Step S108, based on the self-game training method, by periodically replacing the trained reinforcement learning agent with the opponent of the target combat aircraft, the reinforcement learning agent is repeatedly trained to achieve the target training cycle, and the target reinforcement learning agent is obtained.

[0067] Specifically, based on the self-game method, an aircraft self-game confrontation environment is built to train the basic end-to-end aircraft confrontation decision-making model. The model obtained by further training is used as an adversary to continue to compete with the basic end-to-end aircraft confrontation decision-making model to improve the decision-making ability of the intelligent agent. During the training process, the parameters of the trained model are periodically fixed, and the parameter model is used as a new adversary to continuously train the basic end-to-end aircraft confrontation decision-making model, thereby gradually improving the ability of the aircraft confrontation strategy.

[0068] In an optional implementation provided by an embodiment of the present invention, step S102 further includes: constructing a flight dynamics model of the target combat aircraft based on the JSBSim aircraft simulation platform.

[0069] Specifically, the experimental environment is based on the open-source simulation platform JSBSim, an open-source, cross-platform, six-degree-of-freedom nonlinear flight dynamics model. Written in the object-oriented C++ language, JSBSim supports dynamic modeling of various types of aviation and aerospace vehicles. The vehicle dynamics characteristics in this model are expressed using Extensible Markup Language (XML), allowing users to build and simulate their own six-degree-of-freedom aircraft models without compiling and linking code. The resulting aircraft simulation model incorporates model constraints such as engine performance, aircraft dynamics, aircraft weight, and fuel consumption calculations. It simulates the flight conditions of a real aircraft to a certain extent, and the training results are highly interpretable.

[0070] With the help of rule-based confrontation strategy and PID control, a basic aircraft confrontation expert system is realized. This expert system can complete basic flight actions and simple combat maneuvers, including: straight flight, climb, dive, left turn, right turn, etc.

[0071] In this embodiment, the aircraft controlled by the intelligent agent uses the open source "F-16" aircraft model, with continuous stick and rudder control as the target action output.

[0072] Figure 2 This is a diagram of a method for constructing a local coordinate system and a method for defining attitude angles during a combat aircraft confrontation process provided according to an embodiment of the present invention, wherein the local coordinate system is constructed using a north-east sky method, and ψ, θ, and φ represent the heading angle, pitch angle, and roll angle of the aircraft, respectively. Figure 3 This is a diagram illustrating the principle of a functional level simulation radar provided according to an embodiment of the present invention, wherein D abrepresents the straight-line distance between the two aircraft, λ a Indicates the effective radar scanning range of aircraft a, Indicates the effective angle of radar scanning, Indicates the angle between the target aircraft and the direction in which the nose of the radar scanning aircraft is pointing.

[0073] During the entire adversarial training process, the pitch angle is randomly set within the range of ±20 degrees, and the roll angle is randomly set within the range of ±15 degrees. The initialization distance between the two is kept less than 20km. During the training process, both aircraft use the same simulation rate.

[0074] The flight confrontation process of an aircraft itself changes rapidly, and the control instructions of the aircraft are a continuous multi-dimensional space, which increases the complexity of decision-making to a certain extent. However, since flight control has certain regularity, deterministic strategy algorithms can be used to find the optimal confrontation combat strategy.

[0075] Specifically, the training of reinforcement learning is mainly divided into two parts: environment and algorithm:

[0076] Through the JSBSim-based simulation environment, the position, attitude, speed and other information of the aircraft can be obtained through calculation. After the environment obtains the simulation data of both sides, it processes the states of the two opposing sides according to the design of the intelligent agent observation space, obtains the observation vector of the intelligent agent in a single step, and obtains the single-step action vector output after being calculated by the intelligent agent network. After the action value is scaled and constrained, it is handed over to the simulation platform for one-step calculation to obtain the next state. During the training process, similar interaction processes are repeated in a single round until the end of the round.

[0077] The reinforcement learning phase experiment involves an intelligent agent and an expert system based on PID control. The expert system is constructed with the help of humans' prior knowledge and rules of aircraft confrontation, and has basic maneuvering capabilities such as steering, altitude change, and speed control. The intelligent agent is a decision-making intelligent agent in the end-to-end rudder thrust continuous action space.

[0078] In this embodiment of the present invention, the "F-16" is selected as the aircraft type for training, and continuous stick and rudder control is used as the target action output. Regarding the observation space, the goal of the model is to obtain a better combat posture during the flight confrontation, pose a higher threat to the enemy, and ultimately shoot down the target. Further considering that the information that one aircraft can obtain about the enemy aircraft during the actual aircraft confrontation is limited, the assumption of omniscience of the environment is not adopted, and only the partial information that can be obtained from radar and accumulated information is used. Therefore, this embodiment of the present invention selects position, velocity, and attitude as the observation dimensions, and sets the state space of the reinforcement learning agent and the observation space of the training process as:

[0079] [z,Δx,Δy,Δz,Δv x ,Δv y ,Δv z ,v x ,v y ,v z ,h s ,p s ,r s ,h e ,p e ]

[0080] Where z is the altitude of the aircraft, Δx, Δy and Δz are the north component, east component and celestial component of the position vector difference between the aircraft and the enemy aircraft in the local coordinate system, respectively, and Δv x , Δv y and Δv z are the north component, east component and sky component of the velocity vector difference between the own aircraft and the enemy aircraft in the local coordinate system, respectively, v x 、v y and v z are the north speed, east speed and sky speed of the aircraft respectively, h s is the direction the aircraft's nose is pointing, p s is the pitch angle of the aircraft, r s is the roll angle of the aircraft, h e is the direction the enemy aircraft's nose is pointing, p e is the pitch angle of the enemy aircraft;

[0081] The action space of a reinforcement learning agent consists of:

[0082] A t =[elevator,rudder,aileron,throttle]

[0083] Among them, elevator represents the elevator command, the minimum command is -1, and the maximum command is 1; rudder represents the rudder command, the minimum command is -1, and the maximum command is 1; aileron represents the aileron command, the minimum command is -1, and the maximum command is 1; throttle represents the throttle command, the minimum command is 0, and the maximum command is 0.9.

[0084] The reward function of the reinforcement learning agent is used to guide the agent in optimizing its decision-making. In this invention, the reward function uses hierarchical analysis to estimate the situational advantage of both combatants, encouraging aircraft to gain a situational advantage during the confrontation. At the same time, based on radar simulation, the weapon lock status is simulated, and the radar lock status is used as an indicator for judging the end of the battle. The combat results are reflected in the reward function, encouraging the agent to shoot down the enemy aircraft. The combination of single-step reward function and round reward function solves the problem of sparse rewards during the confrontation. In summary, the reward function of this invention is expressed as:

[0085] R=α H *R H +α R *R R

[0086] Among them, R H is the single-step reward value obtained by hierarchical analysis, R R is the round reward value obtained from the radar scanning judgment result, α H and α R are the coefficients of single-step reward and round reward respectively.

[0087] Specifically, the single-step reward value based on the hierarchical analysis method is as follows:

[0088] R H =ω v *T v +ω a *T a +ω d *T d +ω h *T h

[0089] Where, T v is the speed threat index, T a is the angle threat index, T d is the distance threat index, T h is a high threat indicator, ω v 、ω a 、ω d 、ω h is the weight factor.

[0090] Specifically, when quantifying the speed threat index, the speed ratio is mainly considered. The specific speed threat index calculation is expressed as:

[0091]

[0092] Where, v j and v i Respectively indicate the flight speed of the own aircraft and the enemy aircraft;

[0093] When our attack angle is pointing in the opposite direction of the enemy, while the enemy's attack angle is pointing towards us, the angle threat index is the largest. When our attack angle is pointing towards the enemy target, while the enemy's attack angle is pointing in the opposite direction of ours, the angle threat index is the smallest. Specifically, the angle threat index includes:

[0094]

[0095] Where q B and q R are the angles of attack of your own aircraft and the enemy aircraft respectively;

[0096] Considering the threat posed to aircraft by the distance from the opponent during combat, the distance threat indicators include:

[0097]

[0098] In the formula, r0 represents the distance between the aircraft and the enemy aircraft, r j and r i Represents the maximum attack distance of the own aircraft and the enemy aircraft, r u is the maximum detection range of enemy aircraft;

[0099] Considering the threat posed by the aircraft's altitude to the adversary's aircraft during combat, altitude threat indicators include:

[0100]

[0101] Where h j and h i They are the flight altitudes of our own aircraft and the enemy aircraft respectively.

[0102] Based on the simulated radar function, the result of the aircraft confrontation is determined and used as the result-oriented reward function for a single round. Specifically, the round reward value based on the confrontation result is as follows:

[0103]

[0104] Where r win Indicates the round reward value when the machine wins, r loss Indicates the round reward value when winning against the opponent.

[0105] In an embodiment of the present invention, training a reinforcement learning agent includes training the reinforcement learning agent based on a DDPG algorithm using an Actor-Critic structure, wherein the Actor network and the Critic network use similar neural network structures.

[0106] Specifically, the Actor network is used to input observations, obtain actions, and generate the final action after adding noise:

[0107]

[0108] Where a represents the final action, π θ is the action network, θ represents the parameters of the action network, s represents the observation of the current action, represents the added noise;

[0109] The Critic network is used to complete the update by calculating the difference between the current Q value and the target Q value:

[0110] Q 目标 =r+γQ′(s′,a′,ω′)

[0111] Where r represents the reward obtained in the current step, γ represents the discount factor of the reward, Q′ represents the Critic target network, s′ and a′ represent the observation value and action of the next interaction step respectively, and ω′ is the parameter of the target network;

[0112] In the update phase, the Critic network is updated by the mean square error:

[0113]

[0114] Where J() represents the mean square error loss function, Q(s j ,a j ,ω) represents the predicted Q value, ω represents the network parameter, s j and a j They represent the observation value and action of the jth sample respectively, and m represents the number of samples;

[0115] The Actor network is updated with the following loss gradients:

[0116]

[0117] Where, represents the loss gradient function, represents the gradient function of the action, Represents the gradient function of the action network parameters.

[0118] Specifically, in step S108, building the aircraft self-playing environment involves using the same aircraft simulation model, importing the initial training model obtained through the reinforcement learning process, and setting the same algorithm step frequency. After the self-playing environment is completed, the agent continues to be trained using the DDPG algorithm. When a higher reward function is achieved, the decision model is saved and the decision network for the mobile phone is replaced periodically to continuously optimize the decision-making strategy.

[0119] Self-game learning refers to the process of obtaining a better decision-making strategy by continuously pitting the agent against its own historical strategy version model in an adversarial agent training scenario. Since the opponent used in the reinforcement learning training process is an expert system based on PID implementation, the implemented adversarial strategy is highly subjective and lacks change and flexibility. Such adversarial opponents can well guide the agent to acquire basic flight capabilities in the initial stage of training, but lack the exploration of more complex adversarial strategies. In the adversarial environment of self-game, the present invention also uses the JSBSim platform to complete the simulation task, replaces the opponent with the same aircraft model as the agent in the reinforcement learning environment, and replaces the opponent by replacing its decision-making algorithm model. Preferably, the embodiment of the present invention adopts periodic replacement.

[0120] In order to verify the effectiveness and applicability of the algorithm, the present invention has designed a confrontation evaluation system and designed experiments. The confrontation is started in the state of our advantage, our disadvantage, and the state of equilibrium between the enemy and us. The maneuverability performance of both sides before and after the battle is evaluated from the height, speed, distance, and angle threats posed by the combatants to the other side. The radar scanning results are used as indicators to evaluate the victory or defeat of a single round of enemy-us confrontation, and the winning rate of multiple confrontations is used to reflect the results. The experimental results and data obtained are shown as follows: Figure 4-Figure 6 As shown. Among them, Figure 4 This is a schematic diagram of a winning rate experiment result when our team is in an initial advantageous situation, provided according to an embodiment of the present invention. Figure 5 This is a schematic diagram of a winning rate experiment result of our side in an initial situation of equilibrium between the enemy and us, provided according to an embodiment of the present invention. Figure 6 This is a schematic diagram of the winning rate experiment results when our team is initially in a disadvantageous situation, provided according to an embodiment of the present invention.

[0121] As can be seen from the above description, the embodiment of the present invention provides a method for self-learning and training an aircraft maneuvering flight control strategy. Compared with the prior art, the present invention has the following advantages:

[0122] (1) The present invention has different advantages over other algorithms applied in the field of intelligent confrontation of aircraft in recent years. The present invention reasonably models the scenario of intelligent confrontation of aircraft, analyzes the rapidly changing situation in the aircraft confrontation environment, combines the multi-dimensional continuous control during aircraft control, and finally selects the method of reinforcement learning and self-game confrontation: it combines the research on confrontation strategies in non-reinforcement learning aircraft confrontation and the theoretical support of factors affecting the aircraft confrontation situation, and proposes innovation on this basis, adopting a combination of single-step guidance rewards and round target rewards to reasonably densely reward; combining the self-game method to enrich the opponent's behavior logic and generate a more flexible confrontation strategy; the experimental results under confrontation experiment verification show that the aircraft confrontation decision training method can produce a robust aircraft control decision model.

[0123] (2) During the aircraft simulation model construction phase, the present invention models different aircraft simulation models by modeling different engine parameters, flight dynamics parameters, and so on. The simulation of each aircraft achieves high-precision and high-efficiency calculation results, and the aircraft's motion and control in the simulation are sufficiently realistic, which effectively improves the reliability and wide applicability of the present method.

[0124] (3) The observations designed for the reinforcement learning part of this invention include the position, velocity, and attitude of the aircraft itself, as well as the position, velocity, and partial attitude information of the enemy aircraft. This restricts the observation content and abandons the assumption of omniscience regarding enemy aircraft information. All variables involved can be acquired by radar in actual exercises. This selection of observations enhances the persuasiveness of the proposed method and effectively reduces the difficulty of transferring the trained model to the real environment.

[0125] (4) In the core algorithm design phase, the present invention splits the complete aircraft confrontation decision-making into two phases: basic confrontation training and decision-making capability improvement. In the primary phase, through matching training with the expert system implemented by PID, the confrontation situation is randomly initialized within a certain probability distribution within a range to obtain basic flight control and decision-making capabilities. Then, through self-game mode, the confrontation strategy is optimized.

[0126] Example 2

[0127] Figure 7 Schematic diagram of a self-learning training system for aircraft maneuvering flight control strategy according to an embodiment of the present invention. Figure 7 As shown, the system includes: an establishment module 10 , a construction module 20 , a first training module 30 and a second training module 40 .

[0128] Specifically, the establishment module 10 is used to establish a flight dynamics model of the target combat aircraft based on the engine performance and flight dynamics performance of the target combat aircraft;

[0129] Building module 20, for building a reinforcement learning agent for controlling a target combat aircraft based on a flight dynamics model and a reinforcement learning algorithm;

[0130] A first training module 30 is configured to train a reinforcement learning agent using a combat aircraft controlled by an initial strategy as an opponent to obtain a trained reinforcement learning agent;

[0131] Specifically, the first training module 30 is further configured to train the reinforcement learning agent based on the DDPG algorithm using an Actor-Critic structure;

[0132] The second training module 40 is used to train the reinforcement learning agent repeatedly based on the self-game training method by periodically replacing the trained reinforcement learning agent with the opponent of the target combat aircraft to achieve the target training cycle and obtain the target reinforcement learning agent.

[0133] The present invention also provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method provided in the embodiment of the present invention when executing the computer program.

[0134] The present invention also provides a computer-readable storage medium, which stores computer instructions. When the computer instructions are executed by a processor, the method provided in the embodiment of the present invention is implemented.

[0135] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above and that the invention can be embodied in other specific forms without departing from the spirit or essential characteristics of the invention. Therefore, the embodiments should be considered in all respects as illustrative and non-restrictive, and the scope of the invention is defined by the appended claims, not the foregoing description, and all variations within the meaning and range of equivalents of the claims are intended to be included therein. Any reference sign in a claim should not be construed as limiting the claim to which it relates.

[0136] In addition, it should be understood that although this specification is described in terms of implementation methods, not every implementation method contains only one independent technical solution. This narrative method of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment can also be appropriately combined to form other implementation methods that can be understood by those skilled in the art.

Claims

1. A self-learning training method for aircraft maneuver flight control strategy, characterized in that: include: establishing a flight dynamics model of the target combat aircraft based on the engine performance and flight dynamics performance of the target combat aircraft; Building a reinforcement learning agent for controlling the target combat aircraft based on the flight dynamics model and reinforcement learning algorithm; The reinforcement learning agent is trained using a combat aircraft controlled by an initial strategy as an opponent to obtain a trained reinforcement learning agent; Based on the self-game training method, the trained reinforcement learning agent is periodically replaced with the opponent of the target combat aircraft, and the reinforcement learning agent is repeatedly trained to achieve the target training cycle to obtain the target reinforcement learning agent.

2. The method according to claim 1, wherein: The state space of the reinforcement learning agent includes: [z,Δx,Δy,Δz,Δv x ,Δv y ,Δv z ,v x ,v y ,v z ,h s ,p s ,r s ,h e ,p e ] Where z is the altitude of the aircraft, Δx, Δy and Δz are the north component, east component and celestial component of the position vector difference between the aircraft and the enemy aircraft in the local coordinate system, respectively, and Δv x , Δv y and Δv z are the north component, east component and sky component of the velocity vector difference between the own aircraft and the enemy aircraft in the local coordinate system, respectively, v x 、v y and v z are the north speed, east speed and sky speed of the aircraft respectively, h s is the direction the aircraft's nose is pointing, p s is the pitch angle of the aircraft, r s is the roll angle of the aircraft, h e is the direction the enemy aircraft's nose is pointing, p e is the pitch angle of the enemy aircraft; The action space of the reinforcement learning agent includes: A t =[elevator,rudder,aileron,throttle] Among them, elevator represents the elevator command, rudder represents the rudder command, aileron represents the aileron command, and throttle represents the throttle command.

3. The method according to claim 1, wherein: The reward function for training the reinforcement learning agent includes: R=a H *R H +a R *R R Among them, R H is the single-step reward value obtained by hierarchical analysis, R R is the round reward value obtained from the radar scanning judgment result, α H and α R are the coefficients of single-step reward and round reward respectively; R H =ω v *T v +oh a *T a +oh d *T d +oh h *T h Where, T v is the speed threat index, T a is the angle threat index, T d is the distance threat index, T h is a high threat indicator, ω v 、ω a 、ω d 、ω h is the weight factor, r win Indicates the round reward value when the machine wins, r loss Indicates the round reward value when winning against the opponent.

4. The method according to claim 3, wherein: The speed threat indicators include: Where, v j and v i Respectively indicate the flight speed of the own aircraft and the enemy aircraft; The angle threat indicators include: Where q B and q R are the angles of attack of your own aircraft and the enemy aircraft respectively; The distance threat indicators include: In the formula, r0 represents the distance between the aircraft and the enemy aircraft, r j and r i Represents the maximum attack distance of the own aircraft and the enemy aircraft, r u is the maximum detection range of enemy aircraft; The high threat indicators include: Where h j and h i They are the flight altitudes of our own aircraft and the enemy aircraft respectively.

5. The method according to claim 1, wherein: The combat aircraft controlled by the initial strategy includes a combat aircraft decision model based on PID control and fusion based on preset prior knowledge.

6. The method according to claim 1, wherein: Training the reinforcement learning agent includes: training the reinforcement learning agent based on a DDPG algorithm using an Actor-Critic structure; wherein, Actor network, used to input observations, obtain actions, and generate the final action after adding noise: Where a represents the final action, π θ is the action network, θ represents the parameters of the action network, s represents the observation of the current action, represents the added noise; The Critic network is used to complete the update by calculating the difference between the current Q value and the target Q value: Q 目标 =r+γQ′(s′,a′,ω′) Where Q 目标 represents the target Q value, r represents the reward obtained in the current step, γ represents the discount factor of the reward, Q′ represents the Critic target network, s′ and a′ represent the observation value and action of the next interaction step respectively, and ω′ is the parameter of the target network; In the update phase, the Critic network is updated by the mean square error: Where J() represents the mean square error loss function, Q(s j ,a j ,ω) represents the predicted Q value, ω represents the network parameter, s j and a j They represent the observation value and action of the jth sample respectively, and m represents the number of samples; The Actor network is updated with the following loss gradients: Where, represents the loss gradient function, represents the gradient function of the action, represents the gradient function with respect to the parameters of the action network.

7. An aircraft maneuver flight control strategy self-learning training system, characterized by: include: Establishing module, building module, first training module and second training module; wherein, The establishment module is used to establish a flight dynamics model of the target combat aircraft based on the engine performance and flight dynamics performance of the target combat aircraft; The building module is used to build a reinforcement learning agent for controlling the control strategy of the target combat aircraft based on the flight dynamics model and the reinforcement learning algorithm; The first training module is configured to train the reinforcement learning agent using a combat aircraft controlled by an initial strategy as an opponent to obtain a trained reinforcement learning agent; The second training module is used to periodically replace the trained reinforcement learning agent with the opponent of the target combat aircraft based on the self-game training method, repeatedly train the reinforcement learning agent to achieve the target training cycle, and obtain the target reinforcement learning agent.

8. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method according to any one of claims 1 to 6 when executing the computer program.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, and when the computer instructions are executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images