A method for designing a six-degree-of-freedom non-full-observable intelligent decision model for air combat maneuver

Abstract

The application discloses a kind of six degrees of freedom non-complete observable air combat maneuvering intelligent decision model design method, it is related to the technical field of aerial vehicle air combat intelligent decision-making, and its technical solution points are as follows: including two steps, first is the simulation interactive environment construction of decision model design, second is the reinforcement learning training of decision model.For the problem that complete state information of the other aircraft cannot be obtained in air combat confrontation, the expected attitude of the enemy aircraft is assumed to be achieved instantaneously, and the enemy aircraft motion is abstracted as a three-degree-of-freedom flight situation considering only the flight path motion to carry out training, thereby solving the problem of incomplete information in six-degree-of-freedom air combat maneuvering intelligent decision-making, ensuring the Markov property of the environment interaction process, and realizing the effective design of the reinforcement learning decision model.Based on the design method given by the patent, the deep neural network decision model obtained can directly give the rudder deflection and thrust control command of the aircraft based on the operational situation, to drive the aircraft to maneuver and seize the advantage of the confrontation situation.

Description

Technical Field

[0001] This invention relates to the field of intelligent decision-making technology for air combat of aircraft, and more specifically, to a design method for a six-degree-of-freedom, non-fully observable, close-range air combat maneuver intelligent decision-making model. Background Technology

[0002] Future warfare will be a confrontation between intelligent agents, and intelligent unmanned combat aerial vehicles (UCAVs) will undoubtedly be one of the most important forces in this conflict. They will drastically change the patterns and forms of future warfare and become a crucial weapon system determining victory or defeat. Intelligent air combat decision-making, simulating the pilot's decisions regarding aircraft control in various air combat situations, is the "soul" and "brain" of intelligent UCAVs. Since the deep reinforcement learning-based air combat AI achieved a decisive victory over human pilots in the Artificial Intelligence (AI) close-range air combat competition held by DAPPA in the United States in August 2020, intelligent air combat decision-making has received increasing attention.

[0003] Air combat is a highly complex game of strategy, characterized by high dynamics, real-time dynamics, and uncertain / incomplete information. Although the air combat battlefield has expanded from close-range to medium- and long-range with the development of air-based weapons, the simultaneous advancements in stealth and electronic countermeasures mean that a significant proportion of future air combat will still occur at close range, making close-range air combat research still relevant. Among various close-range air combat decision-making scenarios, the two-aircraft close-range game problem is the most fundamental. In this scenario, aircraft need to perform numerous tactical maneuvers to create weapon launch conditions and evade enemy attacks. Maneuver decision-making capability is a crucial aspect of evaluating the combat capability of a single combat aircraft. As the foundation of close-range air combat decision-making, intelligent maneuver decision-making is a key issue that needs to be addressed in research on intelligent decision-making in close-range air combat.

[0004] Traditional air combat maneuver decision-making methods can be broadly categorized into game theory-based methods, optimization theory-based methods, and expert system-based methods. Currently, machine intelligence has entered the era of deep learning, and research on intelligent decision-making based on deep reinforcement learning has made substantial progress. One of the main directions for the development of intelligent flight is combining deep reinforcement learning methods with unmanned aerial vehicles (UAVs).

[0005] Current research on intelligent decision-making in air combat is generally coarse-grained, typically designing intelligent decision-making models based on three-degree-of-freedom (DOF) motion models of aircraft. However, since actual aircraft move with six degrees of freedom, intelligent decision-making models based on three-DOF models sever the connection between the aircraft's outer-loop trajectory motion and inner-loop attitude motion. This prevents the utilization of the aircraft's nonlinear aerodynamic torque characteristics and nonlinear center-of-mass / around-center-of-mass motion coupling characteristics, thus failing to fully exploit the aircraft's maneuverability potential. To address this issue, intelligent decision-making machines can be designed by directly training on the aircraft's six-DOF model to obtain an end-to-end optimal decision-making controller. However, due to the incompleteness of air combat information, such as the general inability to know the enemy aircraft's angular velocity, this violates the Markov property in reinforcement learning applications, severely affecting the convergence of AI decision-making model reinforcement learning and posing difficulties for the design of six-DOF end-to-end intelligent decision-making models.

[0006] Therefore, the present invention aims to provide a design method for a six-degree-of-freedom, non-fully observable air combat maneuver intelligent decision-making model to solve the aforementioned problems. Summary of the Invention

[0007] The purpose of this invention is to provide a design method for a six-degree-of-freedom (DOF) incomplete observable air combat maneuver intelligent decision-making model. Addressing the problem of not being able to obtain complete state information of the enemy aircraft in air combat, this method assumes that the enemy aircraft's desired attitude can be achieved instantaneously. It abstracts the enemy aircraft's motion into a three-degree-of-freedom flight scenario considering only its trajectory motion for training, thereby solving the problem of incomplete information in six-DOF air combat maneuver intelligent decision-making, ensuring the Markov property of the environmental interaction process, and achieving effective design of the reinforcement learning decision-making model.

[0008] The above-mentioned technical objective of the present invention is achieved through the following technical solution: a design method for a six-degree-of-freedom incompletely observable air combat maneuver intelligent decision-making model, including a red aircraft and a blue aircraft, wherein the red aircraft is the aircraft that needs to establish a six-degree-of-freedom end-to-end maneuver intelligent decision-making model; and the blue aircraft is the opposing enemy aircraft.

[0009] The specific steps are as follows:

[0010] S1: Establish the motion dynamics model of the red machine, establish the motion dynamics model of the blue machine, and design the reward model;

[0011] S2: Reinforcement learning training is performed on the decision model.

[0012] The present invention is further configured to: establish the motion dynamics model of the red machine described in step S1, the steps of which are as follows:

[0013] S101: Define the relevant coordinate systems that describe the motion of the aircraft, including the aircraft body coordinate system, the airflow coordinate system, and the ground coordinate system;

[0014] S102: Construct the dynamic equations of the red aircraft's trajectory, as follows:

[0015]

[0016]

[0017] Where r = [xyz] T These are the aircraft's position coordinates in the ground coordinate system.

[0018] v b =[uvw] T The velocity vector in the aircraft system;

[0019] X, Y, and Z represent the aerodynamic forces of the aircraft system in the x, y, and z directions, which can be calculated from the drag D, side force Y, and lift L represented by the airflow system.

[0020] η represents the engine throttle.

[0021] T max This is the engine's maximum thrust;

[0022] g is the acceleration due to gravity;

[0023] For the direction cosine array from the aircraft system to the ground system;

[0024] S103: The equations of motion for the attitude of the red machine are as follows:

[0025]

[0026]

[0027] Where q = [q0 q1 q2 q3] T For attitude quaternions;

[0028] ω=[pqr] T This represents the angular velocity vector in the aircraft system.

[0029] I represents the aircraft inertia tensor;

[0030] l a m a With n a These are the roll, pitch, and yaw aerodynamic moments acting on the aircraft, respectively.

[0031] S104: The aerodynamic model of the Red Engine is constructed as follows:

[0032]

[0033]

[0034] Among them, C D C Y C L These are the aircraft's drag coefficient, lateral force coefficient, and lift coefficient, respectively.

[0035] C l C m C n These are the rolling moment coefficient, pitching moment coefficient, and yaw moment coefficient under the aircraft system, respectively.

[0036] For dynamic pressure;

[0037] S is the reference area;

[0038] b and c are the aircraft's lateral and longitudinal reference lengths, respectively;

[0039] The aerodynamic coefficient and aerodynamic moment coefficient models are as follows:

[0040]

[0041]

[0042] Where α is the angle of attack;

[0043] β is the sideslip angle;

[0044] C s l C s m C s n Indicates the static aerodynamic moment coefficient;

[0045] C lp C mq C np For the coefficient of the dynamic derivative;

[0046] δ e δ a With δ r These are the elevator, ailerons, and rudder;

[0047] S105: The model established by S102-S104 determines the state variables X in the six-degree-of-freedom motion of the red machine. The state variables X include the position vector r and the velocity vector v. b The attitude quaternion q and the angular velocity vector ω, and the corresponding control quantity is U = [δ e δ a δ r η] T .

[0048] The present invention is further configured such that: the step of establishing the blue aircraft motion dynamics model in step S1 is to construct the blue aircraft motion dynamics equations based on the coordinate system describing the aircraft motion defined in S101, and the steps are as follows:

[0049] S106: Construct the dynamic equations of the blue aircraft's trajectory. The equations are as follows:

[0050]

[0051]

[0052] Where r = [xyz] T These are the aircraft's position coordinates in the ground coordinate system.

[0053] V represents the velocity amplitude;

[0054] χ represents the azimuth angle of the flight path;

[0055] γ is the track inclination angle;

[0056] α is the angle of attack;

[0057] μ is the roll angle around the velocity vector;

[0058] For engine throttle;

[0059] T represents engine thrust;

[0060] Tmax is the engine's maximum thrust;

[0061] D represents resistance;

[0062] L represents lift;

[0063] m is the mass of the aircraft;

[0064] g is the acceleration due to gravity;

[0065] S107: The aerodynamic model for constructing the Blue Machine is as follows:

[0066]

[0067] in, For dynamic pressure;

[0068] ρ is the atmospheric density;

[0069] S is the reference area;

[0070] C D With C L Let C be the drag coefficient and lift coefficient of the aircraft, respectively. Ignoring the effects of control surface deflection and sideslip angle, these are functions of the angle of attack and can be written as C0. D =CD (α) and C L =C L (α);

[0071] S108: The model established by S106-S107 determines the state variable X in the three-degree-of-freedom motion of the blue machine. The state variable X is X=[xyh V γ χ] T The corresponding control quantity is U = [α μ η] T .

[0072] The present invention is further configured such that: the reward function in S1 has the following specific form:

[0073]

[0074] in, r V r h r d r α and r β These are the angle advantage reward function, speed advantage reward function, height advantage reward function, distance advantage reward function, angle of attack penalty function, and sideslip angle penalty function;

[0075] ω V ω h ω d ω α and ω β These are the weight parameters corresponding to the angle advantage reward function, speed advantage reward function, height advantage reward function, distance advantage reward function, angle of attack penalty function, and sideslip angle penalty function.

[0076] The present invention is further configured such that the angle advantage reward function is:

[0077]

[0078] in, The angle of attack for the red aircraft;

[0079] The escape angle for the blue aircraft;

[0080] The speed advantage reward function is:

[0081]

[0082] Among them, V R For the red machine speed;

[0083] V B For the speed of the blue machine;

[0084] Vopt For optimal air combat speed;

[0085] The height advantage reward function is:

[0086]

[0087] Among them, h R The height of the red machine;

[0088] h B The height of the blue machine;

[0089] h opt The optimal altitude for air combat;

[0090] The distance advantage reward function is:

[0091]

[0092] Where, d RB It is the distance between the red and blue aircraft;

[0093] d opt This is the optimal air combat distance that the Red plane expects;

[0094] d0 is a constant parameter;

[0095] abs(·) is an absolute value function;

[0096] The angle of attack penalty function is:

[0097]

[0098] Where, α ulim To allow for the upper boundary value of the angle of attack;

[0099] α dlim To allow for the lower boundary value of the angle of attack;

[0100] k is a parameter that adjusts the abruptness of the cup-shaped function. The larger the value, the more severe the cup-shaped function penalty is triggered.

[0101] δ is a positive parameter close to 0;

[0102] The sideslip angle penalty function is:

[0103]

[0104] Where, β lim To allow for the magnitude of the sideslip angle.

[0105] The present invention is further configured such that, in the reinforcement learning training of the decision model in step S2, the close-range air combat maneuver decision is abstracted into a Markov decision process, which is described as the agent observing the enemy-friendly state S at time t. t Select action A based on strategy π t The environment provides the agent with a reward, R. t Then the agent enters a new state S. t+1 ;

[0106] Corresponding to the elements of the Markov decision process, in the 6-DOF close-range air combat maneuver intelligent decision model, the input S of the neural network decision model includes enemy and friendly situation information and the aircraft's own state information. The enemy and friendly situation information includes relative distance, elevation angle, line of sight azimuth angle, and relative speed. Moreover, the enemy and friendly situation information is calculated through the position and speed information of both the enemy and friendly parties.

[0107] The aircraft's own state information includes altitude, speed, attitude, attitude angular velocity information, and angle of attack and sideslip angle information that are closely related to the stability of the aircraft itself; the action output A is the aerodynamic control surfaces of the aircraft engine throttle η, including the elevator δ. e Aileron δ a With rudder δ r .

[0108] The present invention is further configured such that: the reinforcement learning training method for the decision model in S2 is to use a multi-agent reinforcement learning algorithm of the MADDPG or MAPPO framework for "multi-dimensional continuous state input and multi-dimensional continuous action output", and to train the decision model through a learning framework of centralized training and distributed execution.

[0109] The present invention is further configured such that: the reinforcement learning training method for the decision model in S2 can be trained by using a single agent deep reinforcement learning algorithm for "multi-dimensional continuous state input and multi-dimensional continuous action output".

[0110] The invention is further configured such that the method can also be applied to the design of decision models where the opponent's attitude angle information is known. Specifically, in the design of a six-degree-of-freedom end-to-end intelligent decision model, the blue aircraft's attitude angles are used as input information for the neural network decision model. The blue aircraft's motion is abstracted into three-degree-of-freedom motion. Using the three angles of the three-degree-of-freedom trajectory motion—including the trajectory azimuth angle χ, trajectory inclination angle γ, and velocity roll angle μ—along with the angle of attack α and sideslip angle β, the blue aircraft's attitude angles, including the yaw angle, are inversely solved. Pitch angle θ and roll angle ψ; specifically implemented as follows:

[0111] L1: First, determine the direction cosine matrix from the ground coordinate system to the airflow coordinate system using the track azimuth angle χ, track inclination angle γ, and velocity roll angle μ.

[0112]

[0113] L2: Obtain the direction cosine matrix from the airflow coordinate system to the body coordinate system using the angle of attack α and sideslip angle β.

[0114]

[0115] L3: Based on and Obtain the direction cosine matrix from the ground coordinate system to the body coordinate system:

[0116]

[0117] L4: Based on yaw angle Establish a direction cosine matrix using pitch angle θ and roll angle ψ. The relationship is as follows:

[0118]

[0119] L5: The attitude angles of the blue machine were calculated as follows:

[0120]

[0121]

[0122]

[0123] The attitude angle information of the blue aircraft is obtained through conversion, and then applied to the design of the maneuver decision model.

[0124] The present invention is further configured such that: the intelligent decision model design method can also be applied to the design of a six-degree-of-freedom end-to-end intelligent decision model considering manipulation and control, wherein the manipulation and control input devices include a joystick, a foot pedal, and a throttle lever, and the control input is the displacement of these devices.

[0125] In summary, the present invention has the following beneficial effects:

[0126] 1) This invention develops a design method for a six-degree-of-freedom end-to-end intelligent decision-making model for close-range air combat maneuvers in situations where information is not fully observable. Based on a six-degree-of-freedom model of the aircraft, an end-to-end deep reinforcement learning maneuver decision-making model is designed, encompassing both combat situation and control commands. Addressing the problem of not being able to obtain complete state information of the enemy aircraft in air combat, this method assumes that the enemy aircraft's desired attitude can be achieved instantaneously. The enemy aircraft's motion is abstracted into a three-degree-of-freedom flight scenario considering only its trajectory motion for training. This solves the problem of incomplete information in six-degree-of-freedom air combat maneuver intelligent decision-making, ensures the Markov property of the environmental interaction process, and achieves effective design of the reinforcement learning decision-making model.

[0127] 2) Based on the design method given in this application, the design and application of the deep neural network decision model do not require the attitude and angular velocity information of the enemy aircraft.

[0128] 3) Based on the design method given in this application, the obtained deep neural network decision model can directly give the aircraft's rudder deflection and thrust control commands based on the combat situation, and drive the aircraft to maneuver and seize the advantage of the confrontation situation.

[0129] 4) The method provided in this patent is still applicable to the design of a six-degree-of-freedom end-to-end intelligent decision-making model that considers manipulation and control, such as the design of a decision-making model with control inputs such as joystick, pedal, and throttle. It can solve the problem of incomplete information in control maneuver decision-making.

[0130] 5) The approach of this patent to solve the problem of incomplete information in air combat maneuver decision-making can also be extended to the design of multi-aircraft air combat decision-making models. By describing the enemy aircraft as a 3-degree-of-freedom motion, the need for attitude information (including attitude angle and attitude angular velocity) is avoided, thereby effectively realizing the design of the decision-making model.

[0131] 6) The design method given in this patent can also be applied to the design of decision models where the attitude angle information of the other party is known. That is, in the design of a six-degree-of-freedom end-to-end intelligent decision model, the attitude angle of the blue aircraft can also be used as the input information of the neural network decision model. Similarly, the motion of the blue aircraft is abstracted into a three-degree-of-freedom motion. Through the three angles of the three-degree-of-freedom trajectory motion, including the trajectory azimuth angle, trajectory inclination angle and velocity roll angle, combined with the angle of attack and sideslip angle, the attitude angles of the blue aircraft, including the yaw angle, pitch angle and roll angle, can be solved inversely. Attached Figure Description

[0132] Figure 1 This is a flowchart of intelligent decision-making for air combat maneuvers based on a six-degree-of-freedom decision model in an embodiment of the present invention;

[0133] Figure 2 This is the agent training reward convergence curve in an embodiment of the present invention;

[0134] Figure 3 This is a three-dimensional trajectory diagram of close-range air combat between two aircraft in an embodiment of the present invention;

[0135] Figure 4 These are the altitude curves of both sides' aircraft during air combat in this embodiment of the invention;

[0136] Figure 5 These are the velocity amplitude curves of both sides' aircraft during air combat in an embodiment of the present invention;

[0137] Figure 6 These are the angle of attack and sideslip angle curves of the red aircraft in air combat in this embodiment of the invention;

[0138] Figure 7 This is the attitude angle curve of the red aircraft in air combat in the embodiment of the present invention;

[0139] Figure 8 This is the angular velocity curve of the red aircraft in air combat in an embodiment of the present invention;

[0140] Figure 9 This is the aerodynamic control surface curve of the red aircraft in air combat in an embodiment of the present invention;

[0141] Figure 10 This is the red engine throttle control curve in air combat in an embodiment of the present invention;

[0142] Figure 11 This is the air combat reward and return curve in an embodiment of the present invention. Detailed Implementation

[0143] The following is in conjunction with the appendix Figure 1-11 The present invention will be described in further detail below.

[0144] Example: A design method for a six-degree-of-freedom, incompletely observable, air combat maneuvering intelligent decision-making model, such as... Figure 1-11 As shown, the specific steps are as follows:

[0145] S1: Construct a simulation interactive environment. The adversarial scenario includes red and blue aircraft. The red aircraft is the one requiring a six-degree-of-freedom end-to-end maneuver intelligent decision-making model; the blue aircraft is the adversary aircraft. The construction of the simulation interactive environment includes establishing the motion dynamics model of the red aircraft, establishing the motion dynamics model of the blue aircraft, and designing a reward model.

[0146] To design a close-range air combat maneuver intelligent decision-making model for a six-degree-of-freedom motion model of an aircraft, the first step is to establish a six-degree-of-freedom motion dynamics model of the red aircraft, as follows:

[0147] S101: Define the relevant coordinate systems that describe the motion of the aircraft, including the aircraft body coordinate system, the airflow coordinate system, and the ground coordinate system;

[0148] S102: Construct the dynamic equations of the red aircraft's trajectory, as follows:

[0149]

[0150]

[0151] Where r = [xyz] T These are the aircraft's position coordinates in the ground coordinate system.

[0152] v b =[uvw] T The velocity vector in the aircraft system;

[0153] X, Y, and Z represent the aerodynamic forces of the aircraft system in the x, y, and z directions, which can be calculated from the drag D, side force Y, and lift L represented by the airflow system.

[0154] η represents the engine throttle.

[0155] T max This is the engine's maximum thrust;

[0156] g is the acceleration due to gravity;

[0157] For the direction cosine array from the aircraft system to the ground system;

[0158] S103: The equations of motion for the attitude of the red machine are as follows:

[0159]

[0160]

[0161] Where q = [q0 q1 q2 q3] T For attitude quaternions;

[0162] ω=[pqr] T This represents the angular velocity vector in the aircraft system.

[0163] I represents the aircraft inertia tensor;

[0164] l a m a With n a These are the roll, pitch, and yaw aerodynamic moments acting on the aircraft, respectively.

[0165] S104: The aerodynamic model of the Red Engine is constructed as follows:

[0166]

[0167]

[0168] Among them, C D C Y C L These are the aircraft's drag coefficient, lateral force coefficient, and lift coefficient, respectively.

[0169] C l C m C n These are the rolling moment coefficient, pitching moment coefficient, and yaw moment coefficient under the aircraft system, respectively.

[0170] For dynamic pressure;

[0171] S is the reference area;

[0172] b and c are the aircraft's lateral and longitudinal reference lengths, respectively;

[0173] The aerodynamic coefficient and aerodynamic moment coefficient models are as follows:

[0174]

[0175]

[0176] Where α is the angle of attack;

[0177] β is the sideslip angle;

[0178] C s l C s m C s n Indicates the static aerodynamic moment coefficient;

[0179] C lp C mq C np For the coefficient of the dynamic derivative;

[0180] δ e δ a With δ r These are the elevator, ailerons, and rudder;

[0181] S105: The model established by S102-S104 determines the state variables X in the six-degree-of-freedom motion of the red machine. The state variables X include the position vector r and the velocity vector v. b The attitude quaternion q and the angular velocity vector ω, and the corresponding control quantity is U = [δ e δ a δ r η] T .

[0182] To effectively design an end-to-end six-DOF dual-aircraft close-range air combat maneuvering intelligent decision-making model and avoid the Markov property violation problem caused by incomplete information in the decision-making model design, this method assumes the blue aircraft's attitude motion as ideal motion, considers only the blue aircraft's outer loop trajectory, and abstracts the blue aircraft's motion into a three-DOF point mass motion model. This cleverly avoids the need for unobservable information such as angular velocity, thus ensuring the existence of the six-DOF end-to-end decision-making model and the convergence of reinforcement learning. Assuming the aircraft's sideslip angle is zero, the steps in step S1 for establishing the blue aircraft's motion dynamics model are as follows: Based on the coordinate system describing the aircraft's motion defined in S101, the blue aircraft's motion dynamics equations are constructed.

[0183] S106: Construct the dynamic equations of the blue aircraft's trajectory. The equations are as follows:

[0184]

[0185]

[0186] Where r = [xyz] T These are the aircraft's position coordinates in the ground coordinate system.

[0187] V represents the velocity amplitude;

[0188] χ represents the azimuth angle of the flight path;

[0189] γ is the track inclination angle;

[0190] α is the angle of attack;

[0191] μ is the roll angle around the velocity vector;

[0192] For engine throttle;

[0193] T represents engine thrust;

[0194] T max This is the engine's maximum thrust;

[0195] D represents resistance;

[0196] L represents lift;

[0197] m is the mass of the aircraft;

[0198] g is the acceleration due to gravity;

[0199] S107: The aerodynamic model for constructing the Blue Machine is as follows:

[0200]

[0201] in, For dynamic pressure;

[0202] ρ is the atmospheric density;

[0203] S is the reference area;

[0204] C D With C L Let C be the drag coefficient and lift coefficient of the aircraft, respectively. Ignoring the effects of control surface deflection and sideslip angle, these are functions of the angle of attack and can be written as C0. D =C D (α) and C L =C L (α);

[0205] S108: The model established by S106-S107 determines the state variable X in the three-degree-of-freedom motion of the blue machine. The state variable X is X=[xyh V γ χ] T The corresponding control quantity is U = [α μ η] T .

[0206] The red aircraft in air combat can obtain the blue aircraft's position and speed information through airborne radar, optoelectronic equipment, and information support from the back-end command and control system. The reward function is designed based on the information available to the red aircraft, including the adversarial situation information and its own state information. To facilitate the decision-making model in making action decisions that are advantageous to itself during combat, and to quickly gain an advantage in the adversarial situation through maneuvering, the reward function design considers reward functions for angle advantage, speed advantage, altitude advantage, and distance advantage in air combat, as well as penalty functions for angle of attack and sideslip angle to maintain normal flight. The specific form of the reward function in S1 is as follows:

[0207]

[0208] in, r V r h r d r α and r β These are the angle advantage reward function, speed advantage reward function, height advantage reward function, distance advantage reward function, angle of attack penalty function, and sideslip angle penalty function;

[0209] ω V ω h ω d ω α and ω β These are the weight parameters corresponding to the angle advantage reward function, speed advantage reward function, height advantage reward function, distance advantage reward function, angle of attack penalty function, and sideslip angle penalty function.

[0210] Referring to existing situational confrontation assessment methods, the angle advantage reward function is:

[0211]

[0212] in, The angle of attack for the red aircraft;

[0213] The escape angle for the blue aircraft;

[0214] The speed advantage reward function is:

[0215]

[0216] Among them, V R For the red machine speed;

[0217] V B For the speed of the blue machine;

[0218] V opt For optimal air combat speed;

[0219] The height advantage reward function is:

[0220]

[0221] Among them, h R The height of the red machine;

[0222] h B The height of the blue machine;

[0223] h opt The optimal altitude for air combat;

[0224] The distance advantage reward function is:

[0225]

[0226] Where, d RB It is the distance between the red and blue aircraft;

[0227] d opt This is the optimal air combat distance that the Red plane expects;

[0228] d0 is a constant parameter;

[0229] abs(·) is an absolute value function;

[0230] The angle of attack penalty function is:

[0231]

[0232] Where, α ulim To allow for the upper boundary value of the angle of attack;

[0233] α dlim To allow for the lower boundary value of the angle of attack;

[0234] k is a parameter that adjusts the abruptness of the cup-shaped function. The larger the value, the more severe the cup-shaped function penalty is triggered.

[0235] δ is a positive parameter close to 0;

[0236] The sideslip angle penalty function is:

[0237]

[0238] Where, β lim To allow for the magnitude of the sideslip angle.

[0239] S2: Reinforcement learning training of the decision model;

[0240] In the design of the six-DOF end-to-end close-range air combat maneuver intelligent decision-making model, since the input information is a one-dimensional vector of a certain size, the decision-making machine model is modeled using a multi-hidden-layer feedforward deep neural network. Based on the simulation interactive environment built in S1, the aircraft's close-range air combat maneuver intelligent decision-making model is trained using deep reinforcement learning technology.

[0241] The close-range air combat maneuver decision-making is abstracted as a Markov decision process, which is basically described as the agent observing the enemy and friendly states S at time t. t Select action A based on strategy π t The environment provides the agent with a reward, R. t Then the agent enters a new state S. t+1 .

[0242] Corresponding to the elements of the Markov decision process described above, in the six-degree-of-freedom close-range air combat maneuver intelligent decision-making model, the inputs of the neural network decision-making model include enemy and friendly situation information and the aircraft's own state information. The enemy and friendly situation information includes relative distance, elevation angle, line-of-sight azimuth angle, and relative velocity, which are calculated from the position and velocity information of both sides. The aircraft's own state information includes altitude, velocity, attitude, attitude angular velocity information, and angle of attack and sideslip angle information closely related to the aircraft's stability. The action output A is the aerodynamic control surface deflection angle of the aircraft's engine throttle η, including the elevator δ. e Aileron δ a With rudder δ r .

[0243] In this embodiment, the reinforcement learning training of the aircraft's close-range six-degree-of-freedom end-to-end air combat maneuver intelligent decision-making model can be carried out using the following two methods:

[0244] 1) Employ multi-agent reinforcement learning algorithms based on the MADDPG or MAPPO framework for "multi-dimensional continuous state input and multi-dimensional continuous action output", and train the decision model through a learning framework of centralized training and distributed execution.

[0245] 2) Training is performed using single-agent deep reinforcement learning algorithms (such as DDPG, TD(3), PPO) designed for “multi-dimensional continuous state input and multi-dimensional continuous action output”.

[0246] When training using method 2), it is necessary to know the enemy aircraft's movements (i.e., [α μ η]). TTo address this problem, enemy aircraft action decisions can be obtained through expert systems, or other existing three-degree-of-freedom maneuver decision models can be used.

[0247] By continuously improving the performance of the intelligent decision-making model for air combat maneuvering through deep reinforcement learning, the deep neural network learns the end-to-end decision-making rules from the combat situation to control commands, and directly gives rudder deflection and thrust control commands to drive the aircraft to maneuver to a position with situational advantage in order to obtain a better attack position.

[0248] The approach taken in this embodiment to address incomplete information in air combat maneuver decision-making can also be extended to the design of multi-aircraft air combat decision-making models. By describing the enemy aircraft as moving in 3 degrees of freedom, the need for attitude information (including attitude angles and attitude angular velocities) is avoided, thereby effectively enabling the design of the decision-making model.

[0249] The design method presented in this embodiment can also be applied to the design of decision models where the opponent's attitude angle information is known. That is, in the design of a six-degree-of-freedom end-to-end intelligent decision model, the blue aircraft's attitude angles can also be used as input information for the neural network decision model. Similarly, the blue aircraft's motion is abstracted into three-degree-of-freedom motion. Through the three angles of the three-degree-of-freedom trajectory motion, including the trajectory azimuth angle χ, trajectory inclination angle γ, and velocity roll angle μ, combined with the angle of attack α and sideslip angle β (which is zero), the blue aircraft's attitude angles, including the yaw angle, can be solved inversely. Pitch angle θ and roll angle ψ.

[0250] The specific implementation is as follows:

[0251] L1: First, determine the direction cosine matrix from the ground coordinate system to the airflow coordinate system using the track azimuth angle χ, track inclination angle γ, and velocity roll angle μ.

[0252]

[0253] L2: Obtain the direction cosine matrix from the airflow coordinate system to the body coordinate system using the angle of attack α and sideslip angle β.

[0254]

[0255] L3: Based on and Obtain the direction cosine matrix from the ground coordinate system to the body coordinate system:

[0256]

[0257] L4: Based on yaw angle Establish a direction cosine matrix using pitch angle θ and roll angle ψ. The relationship is as follows:

[0258]

[0259] L5: The attitude angles of the blue machine were calculated as follows:

[0260]

[0261]

[0262]

[0263] The attitude angle information of the blue aircraft is obtained through conversion, and then applied to the design of the maneuver decision model.

[0264] After designing a 6-DOF close-range air combat maneuver intelligent decision-making model for the aircraft using deep reinforcement learning methods based on the above steps, Figure 1 A flowchart of the intelligent decision-making process based on the present invention is provided, and its application is implemented as follows:

[0265] M1: The aircraft obtains the position and speed information of enemy aircraft through airborne radar and optoelectronic equipment, early warning aircraft, or ground support information;

[0266] M2: Calculates situational information using enemy aircraft position and speed information along with the aircraft's own position and speed information. This information, along with the aircraft's own state information (altitude, speed, attitude, angular velocity, angle of attack, and sideslip angle), is input into the neural network decision-making machine to obtain engine throttle control commands η and rudder deflection commands, including elevator commands δ. e Aileron command δ a With rudder command δ r These command signals are sent to the aircraft;

[0267] M3: The aircraft receives throttle and rudder control commands η from the neural network decision machine, which are implemented through the engine and servo motors to control the aircraft to maneuver and gain an advantage in air combat.

[0268] In this embodiment, the method can also be applied to the design of a six-degree-of-freedom end-to-end intelligent decision-making model that considers maneuver control. For decision-making model design where the control input device is a joystick, pedal, or throttle lever, the method provided by this patent is still applicable and can solve the problem of incomplete information in control maneuver decision-making.

[0269] Working principle: The air combat scenario is assumed to be a close-range air battle between two aircraft, both F / A-18 fixed-wing aircraft with a mass of m = 15119 kg. In the six-degree-of-freedom model, the aircraft's inertia is I. xx = 31184 kg·m 2 I yy =205125kg·m 2 I zz=230414 kg·m 2 I xz = -4028.1 kg·m 2 Two aircraft engage in combat under conditions where they know each other's positions and speeds, using various maneuvers to gain a strategic advantage in order to trigger the weapons.

[0270] Based on the given six-degree-of-freedom close-range air combat maneuver decision-making model, a motion dynamics model involving both red and blue aircraft was constructed using the Python language platform. A simulation system for a one-to-one air combat game scenario was built. The TD3 deep reinforcement learning method was used for decision model design, and an Actor-Critic type deep neural network decision-making machine model was established using PyTorch. The Critic network model has 11 layers, including an input layer, an output layer, and 9 hidden layers with 256 hidden units. The transfer function of the first 10 layers is the ReLU function, and the output layer is a linear function. The Actor network model has 7 layers, including 5 hidden layers with 256 hidden units. The first 6 layers use the ReLU activation function, and the output layer uses the Tanh activation function to implement control amplitude constraints. Deep reinforcement learning was conducted on a GPU-equipped workstation. The agent obtained training samples by engaging in aerial combat against a Blue Machine driven by an expert system. After each simulation round, the sampled samples were placed into an experience pool. Subsequently, batch sampling was performed in the experience pool to train the neural network. The batch data size was 128. The RMSprop algorithm was used for optimization, and the learning rates for the Critic and Actor models were 2×10⁻⁶. -4 With 1×10 -4 . Figure 2 The graph shows the average reward obtained by the agent during training. As can be seen from the graph, the average reward of the agent generally increases slowly throughout the training process, indicating that the agent's decision-making ability is continuously improving. When the number of training iterations reaches 65,980, the average reward of the agent reaches its peak and then tends to stabilize.

[0271] Close-range air combat simulation is conducted based on a six-degree-of-freedom end-to-end dual-aircraft close-range air combat intelligent decision-making model. Figure 3 A three-dimensional trajectory diagram of close-range air combat between two aircraft is presented. The diagram shows that the red aircraft, based on a six-degree-of-freedom end-to-end decision model, enters a tail-chase attack state against the blue aircraft through scissor maneuvers and gains an advantage in the combat situation. Figure 4 and Figure 5 The altitude and speed curves of the red and blue aircraft are given respectively. During the confrontation, their altitude and speed show an alternating trend. Figure 6 The angle of attack and sideslip angle curves of the red engine are given, and they vary within a defined permissible range. Figure 7The attitude angle curves of the red aircraft are given, showing that the red aircraft performed a large number of circling maneuvers during the maneuver in order to gain situational advantage. Figure 8 The three-axis angular velocity curves of the red machine are presented, which show that a large amount of angular motion accompanied the confrontation. Figure 9 and Figure 10 The aerodynamic control surface deflection curves and throttle control curves of the Red aircraft in close-range air combat are given respectively, with the Red aircraft's engine always at full throttle. Figure 11 The reward-reward curve in the air combat simulation is given. It can be seen that in the entire air combat confrontation, the red aircraft and the blue aircraft take turns to seize the advantage of the confrontation situation. The red aircraft is initially at a disadvantage, but it gains the advantage of the confrontation situation through maneuvering. This shows the effectiveness of the six-degree-of-freedom end-to-end intelligent decision-making model designed in this patent.

[0272] This specific embodiment is merely an explanation of the present invention and is not intended to limit the invention. After reading this specification, those skilled in the art can make modifications to this embodiment without contributing any inventive step, but such modifications are protected by patent law as long as they are within the scope of the claims of the present invention.

Claims

1. A design method for a six-degree-of-freedom, incompletely observable, air combat maneuvering intelligent decision-making model, characterized in that, It includes a red aircraft and a blue aircraft. The red aircraft is an aircraft that requires the establishment of a six-degree-of-freedom end-to-end maneuvering intelligent decision-making model; the blue aircraft is the enemy aircraft being countered. The specific steps are as follows: S1: Establish the motion dynamics model of the red machine, establish the motion dynamics model of the blue machine, and design the reward model; S2: Reinforcement learning training of the decision model; The method can also be applied to the design of decision models where the opponent's attitude angle information is known. That is, in the design of a six-degree-of-freedom end-to-end intelligent decision model, the blue aircraft's attitude angle is used as the input information of the neural network decision model, and the blue aircraft's motion is abstracted into three-degree-of-freedom motion. The three angles of the three-degree-of-freedom trajectory motion, including the trajectory azimuth angle, are used to determine the motion. Track inclination With speed and roll angle In conjunction with the angle of attack With sideslip angle The attitude angles of the blue aircraft, including the yaw angle, were determined by reverse engineering. Pitch angle With roll angle The specific implementation is as follows: L1: First, by using the azimuth angle of the flight path Track inclination With speed and roll angle Determine the direction cosine matrix from the ground coordinate system to the airflow coordinate system: （19）； L2: via angle of attack With sideslip angle The direction cosine matrix from the airflow coordinate system to the body coordinate system is obtained: （20）； L3: Based on and We obtain the direction cosine matrix from the ground coordinate system to the body coordinate system: （21）； L4: Based on yaw angle Pitch angle With roll angle Establish a direction cosine array The relationship is as follows: （22）； L5: The attitude angles of the blue machine were calculated as follows: （23）； （24）； （25）； The attitude angle information of the blue aircraft is obtained through conversion, and then applied to the design of the maneuver decision model.

2. The design method for a six-degree-of-freedom, incompletely observable air combat maneuver intelligent decision-making model according to claim 1, characterized in that: The steps for establishing the motion dynamics model of the red machine described in step S1 are as follows: S101: Define the relevant coordinate systems that describe the motion of the aircraft, including the aircraft body coordinate system, the airflow coordinate system, and the ground coordinate system; S102: Construct the dynamic equations of the red aircraft's trajectory, as follows: （1）； （2）； in, These are the aircraft's position coordinates in the ground coordinate system. The velocity vector in the aircraft system; The aerodynamic forces of the aircraft system in the x, y, and z directions can be represented by drag in the airflow system. Lateral force and lift The conversion yielded the result. For engine throttle; This is the engine's maximum thrust; It is the acceleration due to gravity; For the direction cosine array from the aircraft system to the ground system; S103: The equations of motion for the attitude of the red machine are as follows: （3）； in, For attitude quaternions; This represents the angular velocity vector in the aircraft system. For the aircraft inertia tensor; , and These are the roll, pitch, and yaw aerodynamic moments acting on the aircraft, respectively. S104: The aerodynamic model of the Red Engine is constructed as follows: （5）； （6）； in, These are the aircraft's drag coefficient, lateral force coefficient, and lift coefficient, respectively. These are the rolling moment coefficient, pitching moment coefficient, and yaw moment coefficient under the aircraft system, respectively. For dynamic pressure; For reference area; , These are the aircraft's lateral reference length and longitudinal reference length, respectively. The models for the aircraft's drag coefficient, side force coefficient, and lift coefficient, as well as the models for the aircraft system's roll moment coefficient, pitch moment coefficient, and yaw moment coefficient, are as follows: （7）； （8）； in, Angle of attack; Sideslip angle; Indicates the static aerodynamic moment coefficient; , , For the coefficient of the dynamic derivative; , and These are the elevator, ailerons, and rudder; S105: Determine the state variables in the six-degree-of-freedom motion of the red machine using the model established by S102-S104. The state quantity X includes a position vector. Velocity vector Posture Quaternions With angular velocity vector The corresponding control quantity is .

3. The design method for a six-degree-of-freedom, incompletely observable air combat maneuver intelligent decision-making model according to claim 1, characterized in that: The steps for establishing the blue aircraft's motion dynamics model in step S1 are as follows: Based on the coordinate system describing the aircraft's motion defined in S101, the motion dynamics equations of the blue aircraft are constructed. S106: Construct the dynamic equations of the blue aircraft's trajectory. The equations are as follows: （9）； （10）； in, These are the aircraft's position coordinates in the ground coordinate system. This refers to the velocity amplitude. The azimuth angle of the flight path; The inclination angle of the flight path; Angle of attack; The roll angle around the velocity vector; For engine throttle; For engine thrust; This is the engine's maximum thrust; As resistance; For lift; For aircraft mass; It is the acceleration due to gravity; S107: The aerodynamic model for constructing the Blue Machine is as follows: （11）； in, For dynamic pressure; Atmospheric density; For reference area; and Let be the drag coefficient and lift coefficient of the aircraft, respectively. Ignoring the effects of control surface deflection and sideslip angle, these are functions of the angle of attack and can be written as: and ; S108: The model established by S106-S107 determines the state variable X in the three-degree-of-freedom motion of the blue machine, wherein the state variable X is... The corresponding control quantity is .

4. The design method for a six-degree-of-freedom, incompletely observable air combat maneuver intelligent decision-making model according to claim 1, characterized in that: The reward function in S1 has the following specific form: （12）； in, , , , , and These are the angle advantage reward function, speed advantage reward function, height advantage reward function, distance advantage reward function, angle of attack penalty function, and sideslip angle penalty function; , , , , and These are the weight parameters corresponding to the angle advantage reward function, speed advantage reward function, height advantage reward function, distance advantage reward function, angle of attack penalty function, and sideslip angle penalty function.

5. The design method for a six-degree-of-freedom, incompletely observable air combat maneuver intelligent decision-making model according to claim 4, characterized in that: The angle advantage reward function is: （13）； in, The angle of attack for the red aircraft; The escape angle for the blue aircraft; The speed advantage reward function is: （14）； in, For the red machine speed; For the speed of the blue machine; For optimal air combat speed; The height advantage reward function is: （15）； in, The height of the red machine; The height of the blue machine; The optimal altitude for air combat; The distance advantage reward function is: （16）； in, It is the distance between the red and blue aircraft; This is the optimal air combat distance that the Red plane expects; It is a constant parameter; It is an absolute value function; The angle of attack penalty function is: （17）； in, To allow for upper boundary values ​​of the angle of attack; To allow for the lower boundary value of the angle of attack; The larger the value of the parameter used to adjust the abruptness of the cup-shaped function, the more severe the cup-shaped function penalty triggering. A positive parameter close to 0; The sideslip angle penalty function is: （18）； in, To allow for the magnitude of the sideslip angle.

6. The design method for a six-degree-of-freedom, incompletely observable air combat maneuver intelligent decision-making model according to claim 1, characterized in that: In step S2, during the reinforcement learning training of the decision-making model, close-range air combat maneuver decisions are abstracted into a Markov decision process, which is described as the agent in... Observe the enemy and friend status at all times According to the strategy Select Action Environmental feedback rewards the intelligent agent Then the agent enters a new state. ; Corresponding to the elements of the Markov decision process, in the 6-DOF close-range air combat maneuver intelligent decision-making model, the input of the neural network decision-making model is... It includes enemy and friendly situation information and the aircraft's own status information. The enemy and friendly situation information includes relative distance, elevation angle, line of sight azimuth angle, and relative speed. The enemy and friendly situation information is calculated from the position and speed information of both the enemy and friendly parties. The aircraft's own state information includes altitude, speed, attitude, attitude angular velocity information, and angle of attack and sideslip angle information that are closely related to the aircraft's stability; action output. Throttle for aircraft engines Aerodynamic control surfaces, including elevators ,aileron With rudder .

7. The design method for a six-degree-of-freedom, incompletely observable air combat maneuver intelligent decision-making model according to claim 1, characterized in that: The reinforcement learning training method for the decision model in S2 is to use a multi-agent reinforcement learning algorithm based on the MADDPG or MAPPO framework for "multi-dimensional continuous state input and multi-dimensional continuous action output". The decision model is trained through a learning framework of centralized training and distributed execution.

8. The design method for a six-degree-of-freedom, incompletely observable air combat maneuver intelligent decision-making model according to claim 1, characterized in that: The reinforcement learning training method for the decision model in S2 can be trained by using a single-agent deep reinforcement learning algorithm for "multi-dimensional continuous state input and multi-dimensional continuous action output".

9. The design method for a six-degree-of-freedom, incompletely observable air combat maneuver intelligent decision-making model according to claim 1, characterized in that: The intelligent decision-making model design method can also be applied to the design of a six-degree-of-freedom end-to-end intelligent decision-making model that considers manipulation and control. The manipulation and control input devices include a joystick, a foot pedal, and a throttle lever, and the control input is the displacement of these devices.

Images