A method and system for improving the quality of flight special situation mission training based on reinforcement learning

Abstract

The present invention discloses a method and system for improving the quality of flight special situation mission training based on reinforcement learning, and the method includes the following steps: S1: Based on the special situation mission of the target aircraft, a flight dynamics model and a special situation mission model are constructed, and a reinforcement learning environment is established. A reward function is established based on the flight safety operation requirements. S2: A hybrid Actor-Critic network is established, and an Actor-Separated Proximal Policy Optimization (ASPPO) algorithm is used to train a special situation handling decision network to obtain an optimal control agent. S3: Based on the optimal control agent, multiple groups of action sequence sets are generated, and the maximum frequent action sequence in the sequence set is analyzed to identify problems such as delayed operation actions, redundant actions, missing actions, and wrong action sequences.

Description

Technical Field

[0001] The present invention belongs to the field of flight safety and flight training, and specifically relates to a method and system for improving the quality of flight special situation mission training based on reinforcement learning. Background Art

[0002] The occurrence rate of flight special situations caused by bad weather, mechanical failure, and serious operational errors of pilots is extremely low during civil aviation transport flights, but once they occur, they may lead to serious accident signs or even flight accidents. In this regard, pilots generally need to undergo recurrent training, proficiency inspection and other training on full-mission flight simulators to maintain their ability to handle flight special situations. In simulation training, instructors focus on pilots' observation, reaction, decision-making and other skills by setting up special situation mission subjects such as encountering bad weather and mechanical failures. In the current simulator training process, instructors mainly rely on qualitative analysis and experience explanation of pilots' flight processes, and lack quantitative analysis methods for operating actions. If the pilots do not have a deep understanding of the operating actions, it is easy to have problems such as delayed operation, redundant actions, missed actions, and wrong sequence of actions. Delayed action means that the pilot fails to complete a certain flight operation link in a timely manner according to the prescribed requirements, and the action is too early or too late; redundant action means that the pilot fails to perform redundant actions according to the prescribed requirements; missed action means that the pilot fails to perform necessary actions according to the prescribed requirements; wrong sequence of actions means that the pilot fails to perform actions in the order of sequence according to the prescribed requirements. These problems will reduce the quality of flight training. During actual flight, incorrect flight operations may cause the aircraft's flight status to exceed the envelope, the use of onboard equipment to exceed the limit, and other risks, thereby inducing various unsafe incidents and threatening flight safety.

[0003] Reinforcement learning uses a deep decision network to interact in a complex environment with uncertainty, learns strategies in a trial-and-error manner, optimizes the decision network parameters by maximizing the reward function, obtains the optimal control agent, and further gives the optimal control action sequence based on the state. The elements of reinforcement learning include an agent composed of a deep decision network, an environment that interacts with the decision network, and a reward function designed according to the target state. The agent obtains the feedback state and reward from the environment, and outputs the decision action to the environment based on the feedback state. Reinforcement learning has been applied in UAV flight control, but it is not practical to control civil transport aircraft based on reinforcement learning. In addition, traditional reinforcement learning algorithms can only handle discrete or continuous actions in a single way. In the process of handling special situations, pilots include both continuous operations such as throttle levers and rudder controls, as well as discrete operations such as flaps and landing gear. Therefore, to construct an optimal control agent through a reinforcement learning algorithm, the agent needs to give discrete or continuous mixed actions.

[0004] If the optimal control agent based on reinforcement learning can give the optimal action sequence for handling special situations, it can guide the pilot's flight mission training. However, in theory, this agent can only provide an operation action sequence for a high-level pilot, and still lacks quantitative technical guidance. If the action sequence of the agent can be further studied, quantitative technical guidance can be provided to pilots to help them quickly master the skills of handling special situations and improve the quality of flight training. Summary of the invention

[0005] In view of the shortcomings of existing special situation flight training, the present invention proposes a method and system for improving the quality of flight special situation task training based on reinforcement learning, and generates an action sequence through the optimal control agent obtained by reinforcement learning. The action sequence is analyzed and compared with the actual operation action sequence of the pilot to detect problems such as operation action bias, redundant action, action omission, and action misorder. In view of the problem that traditional reinforcement learning algorithms cannot be applied to continuous and discrete mixed action spaces, a hybrid Actor-Critic network is constructed, that is, two branch networks are introduced after the first three layers of the Actor network, and discrete and continuous actions are output respectively. And based on the proximal policy optimization (PPO) algorithm commonly used in engineering applications, an action separation proximal policy optimization (ASPPO) algorithm is constructed. Based on the action sequence data generated by the optimal control agent, the flight action sequence under the optimal strategy is obtained by frequent sequence pattern mining. The time interval of the frequent action sequence pattern is further statistically analyzed to give the optimal time interval between the operation actions.

[0006] To achieve the above object, the present invention provides the following solutions:

[0007] A method for improving the quality of flight special situation mission training based on reinforcement learning, comprising the following steps:

[0008] S1: Build a reinforcement learning environment based on the special situation training task of the target aircraft; establish a reward function based on the flight safety operation requirements;

[0009] S2: Establish a hybrid Actor-Critic network as a special situation handling decision network, based on a reinforcement learning environment and reward function, and use the policy separation proximal policy optimization algorithm ASPPO to train the special situation handling decision network to obtain the optimal control agent;

[0010] S3: Based on the optimal control agent, multiple sets of action sequence sets are generated, and the most frequent action sequence in the action sequence set is analyzed to identify problems such as delayed operation actions, redundant actions, missed actions, and wrong action sequences.

[0011] Preferably, in S1, the method for constructing a reinforcement learning environment based on the target aircraft special situation training task includes:

[0012] Construct the flight dynamics model of the target aircraft and the dynamics model of the special situation mission. The aircraft flight dynamics model will receive the output of the dynamics model of the special situation mission, that is, combine the dynamics model of the special situation mission with the flight dynamics model of the target aircraft to form a reinforcement learning environment.

[0013] Preferably, in S2, a hybrid Actor-Critic network is established, the first three layers of which are multi-layer perception networks MLP, and the subsequent network introduces two branches, discrete action network discrete-Actor and continuous action network continuous-Actor, to output discrete actions and continuous actions respectively, and the multi-layer perception network MLP is connected to the discrete action network discrete-Actor and the continuous action network continuous-Actor through a fully connected network.

[0014] Preferably, in S2, based on the reinforcement learning environment and the reward function, and using the policy separation proximal policy optimization algorithm ASPPO to train the special situation handling decision network, the method for obtaining the optimal control agent includes:

[0015] After initializing the network parameters, we first collect the trained state, action, and reward data sets {s t ,a t ,R t}, where t∈[0,T st ], T st Indicates the total time step of a training;

[0016] Based on the state, action and reward data set obtained by training {s t ,a t ,R t}, use the PPO algorithm to calculate the advantage function Among them, V(s t ) indicates state s t The value function of Indicates status The value function of is output by the Critic network; R t represents the reward value at time t, γ∈[0,1] represents the discount factor;

[0017] Based on advantage function Calculate the gradient of the loss function of the two action networks and Estimates: Among them, L clip(·) represents the loss function estimator, clip represents the truncation function, and the subscripts d and c represent discrete and continuous actions, respectively. represents the expected value at time t, ε represents the clipping ratio, represents the probability ratio at time t, Indicates the previous strategy. Represents the probability value given by the previous strategy, Indicates the current strategy, Represents the probability value given by the current strategy, a t represents the action at time t, s t represents the state at time t; due to a t =(a t,d ,a t,c ), a t,d represents a discrete action, a t,c represents continuous action, and the probability ratios of continuous action and discrete action selection at time t are: in, is the probability ratio of the discrete action strategy network at time t, represents the previous discrete action strategy, represents the probability value given by the previous discrete action strategy, represents the current discrete action strategy, Represents the probability value given by the current discrete action strategy; is the probability ratio of the continuous action strategy network at time t, represents the previous continuous action strategy, represents the probability value given by the previous continuous action strategy, represents the current continuous action strategy, Represents the probability value given by the current continuous action strategy.

[0018] Based on two action network loss function gradients and The estimates of , discrete and continuous mis-action networks are updated as: in, represents the Actor network parameters, the subscripts d and c represent discrete and continuous actions respectively, The network parameters representing discrete actions, represents the network parameters of continuous actions, α represents the learning rate;

[0019] Critic network updates follow the PPO algorithm;

[0020] After reaching the maximum number of training times M max Finally, the optimal control agent is obtained.

[0021] Preferably, in S3, based on the optimal control agent, multiple action sequence sets are generated, and the maximum frequent action sequence in the action sequence set is analyzed to identify the problems of delayed operation action, redundant action, missing action, and wrong action sequence, including:

[0022] Construct an action sequence S based on the actions given by the optimal decision agent in each experiment j =<a1,a2,a3,…> , where a i ={a} is S j In the element, {a} is an action event, which indicates one action or multiple actions that are operated simultaneously;

[0023] The length of the preset action sequence is the number of elements;

[0024] The size of the preset action sequence is the total number of action events contained in all elements;

[0025] Action sequence S j Middle i The determination requires a preset minimum time interval T m , if the time interval between two or more actions is less than T m , then it is considered that two or more actions are performed simultaneously, where j = 1, 2, 3, ..., N is the experimental sequence number;

[0026] Preset minimum support p min , for S j Mining frequent sequence patterns to generate candidate frequent action sequence patterns

[0027] In generating candidate frequent action sequence patterns After that, if at least one subsequence of length κ in the candidate sequence of size κ+1 is infrequent, it will be discarded;

[0028] Based on the frequent action sequence pattern with the maximum length, the optimal action sequence and time interval in the special task training process are obtained by counting the interval time in the action sequence in multiple experimental data and calculating the mean and variance;

[0029] Based on the action sequence and time interval of the pilot's operations in special situation mission training, and compared with the optimal action sequence and time interval obtained by the optimal decision-making result analysis, the pilot's operations are identified and improved.

[0030] Preferably, the minimum support p is preset min , for S j Methods for mining frequent sequence patterns include:

[0031] The action sequence set Γ=(S1,S2,…,S j ,,SN ), calculate the frequent action sequence pattern L of size κ κ , calculate the support of all action sequences of size κ in Γ: Among them, p( <A1,A2,…,A k >) represents the sequence in the action sequence set Γ <A1,A2,…,A k >The number of occurrences is defined by the sequence size and sequence length, k≤κ, count(·) represents the count; for the sequence <A1,A2,…,A k >Any two actions If n<m, then i<l, A n , A m Indicates one or more simultaneous actions, regardless of <A1,A2,…,A k >In some sequence S j No matter how many times it appears in the , it is counted only once;

[0032] Determine p( <A1,A2,…,A k >)≥p min Is it true? If not, the algorithm ends, indicating that there is no frequent action sequence pattern L κ ; if it holds, then the frequent action sequence pattern L of size κ+1 in Γ is carried out κ+1 Calculation.

[0033] Preferably, generate candidate frequent action sequence patterns The methods include:

[0034] In the frequent action sequence mode L κ To the frequent action sequence pattern L κ+1 In the solution of , candidate frequent action sequence patterns will be generated The candidate rules are as follows:

[0035] Any two frequent action sequence patterns L κ The sequences ζ1 and ζ2 in ζ1 are merged only if the subsequence obtained by removing the first action from ζ1 is the same as the subsequence obtained by removing the last action from ζ2. The merged result is the concatenation of the last action of ζ1 and ζ2. There are two ways of concatenation: 1) If the last two actions of ζ2 belong to the same element, then the last action of ζ2 is part of the last element of ζ1 in the merged sequence; 2) If the last two actions of ζ2 belong to different elements, then the last action of ζ2 becomes a separate element connected to the tail of ζ1 in the merged sequence.

[0036] In generating candidate frequent action sequence patterns After that, if at least one subsequence of length κ in the candidate sequence of size κ+1 is infrequent, it will be discarded. If no candidate sequence of size κ+1 can be found, the maximum length sequence pattern is L κ .

[0037] The present invention also provides a flight special situation mission training quality improvement system based on reinforcement learning, comprising: a construction module, a training module and an identification module;

[0038] The building module is used to build a reinforcement learning environment based on the special situation training mission of the target aircraft; and to establish a reward function based on the flight safety operation requirements;

[0039] The training module is used to establish a hybrid Actor-Critic network as a special situation handling decision network. Based on the reinforcement learning environment and reward function, the special situation handling decision network is trained using the policy separation proximal policy optimization algorithm ASPPO to obtain the optimal control agent.

[0040] The recognition module is used to generate multiple sets of action sequence sets based on the optimal control agent, and analyze the most frequent action sequence in the action sequence set to identify problems such as delayed operation actions, redundant actions, missed actions, and wrong action sequences.

[0041] Compared with the prior art, the present invention has the following beneficial effects:

[0042] The present invention generates an action sequence by an optimal control agent obtained by reinforcement learning. The action sequence is analyzed and compared with the actual operation action sequence of the pilot to detect the problems of operation action bias, redundant action, missing action and wrong action sequence. In view of the problem that the traditional reinforcement learning algorithm cannot be applied to the continuous and discrete mixed action space, a hybrid Actor-Critic network is constructed, that is, two branch networks are introduced after the first three layers of the Actor network to output discrete and continuous actions respectively. And based on the proximal policy optimization (PPO) algorithm commonly used in engineering applications, an action separation proximal policy optimization (ASPPO) algorithm is constructed. Based on the action sequence data generated by the optimal control agent, the flight action sequence under the optimal strategy is obtained by frequent sequence pattern mining. The time interval of the frequent action sequence pattern is further statistically analyzed to give the optimal time interval between the operation actions. Based on the action sequence and action time interval operated by the pilot in the special task training, compared with the optimal action sequence and time interval obtained by the optimal decision result analysis, the problems of action bias, redundant action, missing action and wrong action sequence in the operation of the pilot are improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solution of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0044] Figure 1 Schematic diagram of the overall framework of a method for improving the quality of flight special situation mission training based on reinforcement learning according to an embodiment of the present invention;

[0045] Figure 2 Schematic diagram of a hybrid Actor-Critic network framework according to an embodiment of the present invention;

[0046] Figure 3 Schematic diagram of the ASPPO algorithm framework according to the embodiment of the present invention;

[0047] Figure 4 Schematic diagram of a frequent action sequence pattern mining algorithm according to an embodiment of the present invention;

[0048] Figure 5 Schematic diagram for comparing wind shear disengagement operations according to an embodiment of the present invention. DETAILED DESCRIPTION

[0049] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0050] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0051] Embodiment 1

[0052] like Figure 1 As shown, the present invention provides a method for improving the quality of flight special situation mission training based on reinforcement learning, comprising the following steps:

[0053] S1: Build a reinforcement learning environment based on the special situation training task of the target aircraft; establish a reward function based on the flight safety operation requirements;

[0054] S2: Establish a hybrid Actor-Critic network as a special situation handling decision network, based on a reinforcement learning environment and reward function, and use the policy separation proximal policy optimization algorithm ASPPO to train the special situation handling decision network to obtain the optimal control agent;

[0055] S3: Based on the optimal control agent, multiple sets of action sequence sets are generated, and the most frequent action sequence in the action sequence set is analyzed to identify problems such as delayed operation actions, redundant actions, missed actions, and wrong action sequences.

[0056] In this embodiment, in the pilot special situation training improvement method, in S1, based on the target aircraft special situation training task, the method of constructing a reinforcement learning environment includes:

[0057] According to the special situation training tasks, a reinforcement learning environment is established. The flight dynamics model of the target aircraft is constructed, and the aircraft center of mass motion equations, rotational motion equations, kinematic equations and navigation equations are established according to the principles of dynamics.

[0058] The center of mass motion equations in the body coordinate system are:

[0059]

[0060] Among them, V x , V y and V z is the three-axis component of the airspeed along the aircraft system, and V x , V y and V z The derivative with respect to time. W xB , W yB and W zB is the three-axis component of wind speed along the turbine system, and W xB , W yB and W zB Derivative with respect to time. X, Y and Z are the three-axis forces of the aircraft, m is the mass of the aircraft, g is the acceleration of gravity, φ is the roll angle, θ is the pitch angle, p, q and r are the three-axis angular velocities of the aircraft, and the calculation method of the angle of attack α and the sideslip angle β is:

[0061]

[0062] in,

[0063] The kinematic equations are:

[0064]

[0065] Among them, φ is the roll angle, θ is the pitch angle, and ψ is the yaw angle; is the roll angular rate, is the pitch angular rate, is the yaw angular rate.

[0066] The rotational motion equations of the machine system are:

[0067]

[0068] in, I x is the inertia moment of the body axis x-axis, I y is the moment of inertia about the y-axis of the body, I z is the moment of inertia about the z-axis of the body, I xz is the inertia product of the x-axis and z-axis of the machine body. p, q and r are the angular velocities of the three axes of the machine system. and is the time derivative of p, q and r. x is the rolling moment, M y is the pitching moment, M z is the yaw moment.

[0069] The navigation equations are:

[0070]

[0071] Among them, φ is the roll angle, θ is the pitch angle, and ψ is the yaw angle. and W is the ground speed component along the three axes of the ground system. x , W y and W z is the three-axis component of wind speed along the ground. The above five equations are the core equations of the flight dynamics model.

[0072] The dynamic model of the special situation mission is established (this depends on the specific situation, for example, the present invention cites the model of wind shear in the special situation), and the Soesman model represents the wind shear induced wind field with radial fitting function and vertical fitting function respectively. The radial fitting function is as follows:

[0073]

[0074] Among them, W R is the horizontal radial induced wind speed at any point in three-dimensional space. R is the radial wind shear intensity, D is the wind field diameter, and R represents the relative position of any point (x, y, z) in space to the center of the wind shear (x c ,y c ,z c ), that is, z = z cWhen:

[0075]

[0076] The horizontal wind speed components can be decomposed using the following formula:

[0077] W x =cosψ w W R (8)

[0078] W y = sinψ w W R (9)

[0079] Among them, W x and W y Respectively represent the induced wind speed in the x-direction and y-direction in the horizontal plane. w It is W R The angle with the x-axis.

[0080] The vertical fitting function of the Soesman model is as follows:

[0081]

[0082] Among them, f z is the vertical wind shear intensity, z c is the height. z is the vertical induced wind speed at any point in three-dimensional space.

[0083] The output of the dynamic model of the special situation mission is passed to the flight dynamics model (taking the wind shear escape special situation as an example, the wind speed data generated by the wind shear wind field at the current position of the aircraft is transmitted to the wind data interface in the aircraft flight dynamics model), forming the environment required for reinforcement learning.

[0084] The environment is used to interact with the agent, which is an Actor-Critic network, and the reinforcement learning decision network is an Actor network.

[0085] In this embodiment, in S1, the method of establishing a reward function based on flight safety operation requirements includes: establishing a reward function with reference to safety operation requirements in special situation flights. Considering risk prevention measures in special situation missions, establishing a reward function, and guiding the aircraft to complete the flight mission with excellent flight quality.

[0086] The reward function before leaving is: ||·||1 represents the vector 1-norm, which is the sum of the absolute values ​​of all elements in the vector; represents the attitude reward, ξ represents the sideslip angle β, pitch angle θ, roll angle φ, yaw angle ψ, roll angular velocity p, yaw angular velocity r parameters, ξ target Indicates the target state that the parameter needs to reach;

[0087] The reward function after leaving is: in, Indicates attitude reward, represents the speed limit reward, Indicates command height limit reward.

[0088] In this embodiment, in the above pilot special situation training improvement method, step S2 includes:

[0089] The flight states and related operations that need to be paid attention to in a specific training task are selected as the input and output of the reinforcement learning decision network. A hybrid Actor-Critic network is further constructed as a special situation handling decision network. The first three layers of the network are multi-layer perceptrons (MLP). The subsequent network introduces two branches, discrete-Actor network and continuous-Actor network, to output discrete actions and continuous actions respectively. MLP is connected to discrete-Actor and continuous-Actor through a fully connected network, such as Figure 2 As shown. And use the ASPPO reinforcement learning algorithm to update the network parameters, as shown Figure 3 shown.

[0090] In the ASPPO algorithm framework, the Critic network parameter update is consistent with the PPO algorithm. The Actor network is divided into two branches for update. After initializing the network parameters, first collect the state, action and reward data sets obtained by training. t ,a t ,R t}, where t∈[0,T st ], T st Indicates the total time step of a training. Advantage function is calculated based on PPO algorithm

[0091]

[0092] Among them, V(s t ) indicates state s t The value function of Indicates status The value function of is output by the Critic network. t represents the reward value at time t, and γ∈[0,1] represents the discount factor.

[0093] Further calculate the gradient of the loss function of the two action networks and Estimates:

[0094]

[0095] Among them, L clip (·) represents the loss function estimator, clip represents the truncation function to avoid too large a policy update; subscripts d and c represent discrete and continuous actions, respectively. represents the expected value at time t, and ε represents the clipping ratio; represents the probability ratio at time t, Indicates the previous strategy. Represents the probability value given by the previous strategy, Indicates the current strategy, Indicates the probability value given by the current strategy. t represents the action at time t, s t represents the state at time t. t =(a t,d ,a t,c ), a t,d represents a discrete action, a t,c Represents a continuous action. The probability ratios of continuous action and discrete action selection at time t are:

[0096]

[0097] Among them, in the first formula of formula (3), is the probability ratio of the discrete action strategy network at time t, represents the previous discrete action strategy, represents the probability value given by the previous discrete action strategy, represents the current discrete action strategy, Represents the probability value given by the current discrete action strategy. is the probability ratio of the continuous action strategy network at time t, represents the previous continuous action strategy, represents the probability value given by the previous continuous action strategy, represents the current continuous action strategy, represents the probability value given by the current continuous action strategy. Then the discrete and continuous misaction network updates are:

[0098]

[0099] in, Represents the Actor network parameters. The subscripts d and c represent discrete and continuous actions, respectively. The network parameters representing discrete actions, represents the network parameters of continuous actions, α represents the learning rate. The critic network update follows the PPO algorithm.

[0100] After reaching the maximum number of training times M max Finally, the optimal control agent is obtained.

[0101] In this embodiment, in the above pilot special situation training improvement method, step S3 includes:

[0102] The optimal control agent generates N data sets and mines frequent action sequence patterns. First, the action sequence S is constructed based on the actions given by the optimal decision agent in each experiment. j =<a1,a2,a3,…> , where a i ={a} is S j In the element, {a} is an action event, which represents one action or multiple actions that are performed simultaneously.

[0103] Define the length of the sequence to be the number of elements.

[0104] The size of the sequence is defined as the total number of action events contained in all elements.

[0105] Action sequence S j Middle i The determination requires the definition of the minimum time interval T m , if the time interval between two or more actions is less than T m , then it is considered that two or more actions are performed simultaneously. j = 1, 2, 3, ..., N is the experimental sequence number.

[0106] Define the minimum support p min , for S j Mining frequent sequence patterns:

[0107] The action sequence set Γ=(S1,S2,…,S j ,…,S N ), calculate the frequent action sequence pattern L of size κ κ , calculate the support of all action sequences of size κ in Γ:

[0108]

[0109] Among them, p( <A1,A2,…,A k >) represents the sequence in the action sequence set Γ <A1,A2,…,A k >The number of occurrences, according to the definition of sequence size and sequence length, k≤κ. count(·) represents the count. <A1,A2,…,A k >Any two actions If n<m, then i<l. n , A m Indicates one or more simultaneous actions, regardless of <A1,A2,…,A k >In some sequence S j No matter how many times it appears in the , it is counted only once. <A1,A2,…,A k >)≥p min Is it true? If not, the algorithm ends, indicating that there is no frequent action sequence pattern L κ ; if it holds, then the frequent action sequence pattern L of size κ+1 in Γ is carried out κ+1 Calculation of Figure 4 shown.

[0110] In the frequent action sequence mode L κ To the frequent action sequence pattern L κ+1 In the solution of , candidate frequent action sequence patterns will be generated The candidate rules are as follows:

[0111] Any two frequent action sequence patterns L κ The sequences ζ1 and ζ2 in ζ1 are merged only if the subsequence obtained by removing the first action from ζ1 is the same as the subsequence obtained by removing the last action from ζ2. The merged result is the concatenation of the last actions of ζ1 and ζ2. There are two ways of concatenation: 1) If the last two actions of ζ2 belong to the same element, then the last action of ζ2 is part of the last element of ζ1 in the merged sequence; 2) If the last two actions of ζ2 belong to different elements, then the last action of ζ2 becomes a separate element connected to the tail of ζ1 in the merged sequence.

[0112] In generating candidate frequent action sequence patterns After that, if at least one subsequence of length κ in the candidate sequence of size κ+1 is infrequent, it will be discarded. If no candidate sequence of size κ+1 can be found, the maximum length sequence pattern is L κ .

[0113] Based on the frequent action sequence pattern with the maximum length, the optimal operation time interval of the action sequence in the action sequence in multiple experimental data is obtained by calculating the mean and variance, and the optimal operation time interval of the action sequence in the special task training process is obtained to assist pilots to better determine the operation time.

[0114] Based on the action sequence and action time interval of the pilots in the special situation mission training, the optimal action sequence and time interval obtained by the optimal decision result analysis are compared. By comparing the operations on the timeline, such as Figure 5As shown, it is possible to identify problems such as delayed operation, redundant operation, missed operation, wrong operation sequence, etc. during special situation training, thereby guiding the pilots to improve their operations during special situation training.

[0115] Embodiment 2

[0116] The present invention also provides a flight special situation mission training quality improvement system based on reinforcement learning, comprising: a construction module, a training module and an identification module;

[0117] The building module is used to build a reinforcement learning environment based on the special situation training mission of the target aircraft; and to establish a reward function based on the flight safety operation requirements;

[0118] The training module is used to establish a hybrid Actor-Critic network as a special situation handling decision network. Based on the reinforcement learning environment and reward function, the special situation handling decision network is trained using the policy separation proximal policy optimization algorithm ASPPO to obtain the optimal control agent.

[0119] The recognition module is used to generate multiple sets of action sequence sets based on the optimal control agent, and analyze the most frequent action sequence in the action sequence set to identify problems such as delayed operation actions, redundant actions, missed actions, and wrong action sequences.

[0120] In this embodiment, in S1, the method for constructing a reinforcement learning environment based on the target aircraft special situation training task includes:

[0121] Construct the flight dynamics model of the target aircraft and the dynamics model of the special situation mission. The aircraft flight dynamics model will receive the output of the dynamics model of the special situation mission, that is, combine the dynamics model of the special situation mission with the flight dynamics model of the target aircraft to form a reinforcement learning environment.

[0122] In this embodiment, in S1, based on the flight safety operation requirements, the method of establishing the reward function includes:

[0123] The reward function before leaving is: ||·||1 represents the vector 1-norm, which is the sum of the absolute values ​​of all elements in the vector; represents the attitude reward, ξ represents the sideslip angle β, pitch angle θ, roll angle φ, yaw angle ψ, roll angular velocity p, yaw angular velocity r parameters, ξ target Indicates the target state that the parameter needs to reach;

[0124] The reward function after leaving is: in, Indicates attitude reward, represents the speed limit reward, Indicates command height limit reward.

[0125] In this embodiment, in S2, a hybrid Actor-Critic network is established, the first three layers of the network are multi-layer perception networks MLP, and the subsequent network introduces two branches, discrete action network discrete-Actor and continuous action network continuous-Actor, to output discrete actions and continuous actions respectively, and the multi-layer perception network MLP is connected to the discrete action network discrete-Actor and the continuous action network continuous-Actor through a fully connected network.

[0126] In this embodiment, in S2, based on the reinforcement learning environment and the reward function, and using the policy separation proximal policy optimization algorithm ASPPO to train the special situation handling decision network, the method for obtaining the optimal control agent includes:

[0127] After initializing the network parameters, we first collect the trained state, action, and reward data sets {s t ,a t ,R t}, where t∈[0,T st ], T st Indicates the total time step of a training;

[0128] Based on the state, action and reward data set obtained by training {s t ,a t ,R t}, use the PPO algorithm to calculate the advantage function Among them, V(s t ) indicates state s t The value function of Indicates status The value function of is output by the Critic network; R t represents the reward value at time t, γ∈[0,1] represents the discount factor;

[0129] Based on advantage function Calculate the gradient of the loss function of the two action networks and Estimates: Among them, L clip (·) represents the loss function estimator, clip represents the truncation function, and the subscripts d and c represent discrete and continuous actions, respectively. represents the expected value at time t, ε represents the clipping ratio, represents the probability ratio at time t, Indicates the previous strategy. Represents the probability value given by the previous strategy, Indicates the current strategy, Represents the probability value given by the current strategy, a t represents the action at time t, s t represents the state at time t; due to a t =(a t,d ,a t,c ), a t,d represents a discrete action, a t,c represents continuous action, and the probability ratios of continuous action and discrete action selection at time t are: in, is the probability ratio of the discrete action strategy network at time t, represents the previous discrete action strategy, represents the probability value given by the previous discrete action strategy, represents the current discrete action strategy, Represents the probability value given by the current discrete action strategy; is the probability ratio of the continuous action strategy network at time t, represents the previous continuous action strategy, represents the probability value given by the previous continuous action strategy, represents the current continuous action strategy, Represents the probability value given by the current continuous action strategy.

[0130] Based on two action network loss function gradients and The estimates of , discrete and continuous mis-action networks are updated as: in, represents the Actor network parameters, the subscripts d and c represent discrete and continuous actions respectively, The network parameters representing discrete actions, represents the network parameters of continuous actions, α represents the learning rate;

[0131] Critic network updates follow the PPO algorithm;

[0132] After reaching the maximum number of training times M max Finally, the optimal control agent is obtained.

[0133] In this embodiment, in S3, based on the optimal control agent, multiple action sequence sets are generated, and the maximum frequent action sequence in the action sequence set is analyzed to identify the operation error action. The method includes:

[0134] Construct an action sequence S based on the actions given by the optimal decision agent in each experiment j =<a1,a2,a3,…> , where a i ={a} is S jIn the element, {a} is an action event, which indicates one action or multiple actions that are operated simultaneously;

[0135] The length of the preset action sequence is the number of elements;

[0136] The size of the preset action sequence is the total number of action events contained in all elements;

[0137] Action sequence S j Middle i The determination requires a preset minimum time interval T m , if the time interval between two or more actions is less than T m , then it is considered that two or more actions are performed simultaneously, where j = 1, 2, 3, ..., N is the experimental sequence number;

[0138] Preset minimum support p min , for S j Mining frequent sequence patterns to generate candidate frequent action sequence patterns

[0139] In generating candidate frequent action sequence patterns After that, if at least one subsequence of length κ in the candidate sequence of size κ+1 is infrequent, it will be discarded;

[0140] Based on the frequent action sequence pattern with the maximum length, the optimal action sequence and time interval in the special task training process are obtained by counting the interval time in the action sequence in multiple experimental data and calculating the mean and variance;

[0141] Based on the action sequence and time interval of the pilot's operations in special situation mission training, and compared with the optimal action sequence and time interval obtained by the optimal decision-making result analysis, the pilot's operations are identified and improved.

[0142] In this embodiment, the minimum support p is preset min , for S j Methods for mining frequent sequence patterns include:

[0143] The action sequence set Γ=(S1,S2,…,S j ,…,S N ), calculate the frequent action sequence pattern L of size κ κ , calculate the support of all action sequences of size κ in Γ: Among them, p( <A1,A2,…,A k >) represents the sequence in the action sequence set Γ <A1,A2,…,A k>The number of occurrences is defined by the sequence size and sequence length, k≤κ, count(·) represents the count; for the sequence <A1,A2,…,A k >Any two actions If n<m, then i<l, A n , A m Indicates one or more simultaneous actions, regardless of <A1,A2,…,A k >In some sequence S j No matter how many times it appears in the , it is counted only once;

[0144] Determine p( <A1,A2,…,A k >)≥p min Is it true? If not, the algorithm ends, indicating that there is no frequent action sequence pattern L κ ; if it holds, then the frequent action sequence pattern L of size κ+1 in Γ is carried out κ+1 Calculation.

[0145] In this embodiment, candidate frequent action sequence patterns are generated The methods include:

[0146] In the frequent action sequence mode L κ To the frequent action sequence pattern L κ+1 In the solution of , candidate frequent action sequence patterns will be generated The candidate rules are as follows:

[0147] Any two frequent action sequence patterns L κ The sequences ζ1 and ζ2 in ζ1 are merged only if the subsequence obtained by removing the first action from ζ1 is the same as the subsequence obtained by removing the last action from ζ2. The merged result is the concatenation of the last action of ζ1 and ζ2. There are two ways of concatenation: 1) If the last two actions of ζ2 belong to the same element, then the last action of ζ2 is part of the last element of ζ1 in the merged sequence; 2) If the last two actions of ζ2 belong to different elements, then the last action of ζ2 becomes a separate element connected to the tail of ζ1 in the merged sequence.

[0148] In generating candidate frequent action sequence patterns After that, if at least one subsequence of length κ in the candidate sequence of size κ+1 is infrequent, it will be discarded. If no candidate sequence of size κ+1 can be found, the maximum length sequence pattern is L κ .

[0149] Embodiment 3

[0150] This embodiment takes the pilot's training for escaping wind shear special situation during approach and landing on a Boeing B737 simulator as an example. Based on the wind shear escape special situation training, an intelligent agent is obtained through reinforcement learning training. The action data generated by the intelligent agent is analyzed and compared with the operation action data obtained by the pilot in the simulator. The problems of delayed action, redundant action, missed action and wrong action sequence in the pilot's operation are obtained.

[0151] S1.1: During the approach to wind shear, the aircraft's airspeed, descent rate, and attitude must be closely monitored for safe operation. Therefore, the altitude h, true airspeed V TAS The eight flight state quantities, namely, sideslip angle β, pitch angle θ, roll angle φ, yaw angle ψ, roll angular velocity p, and yaw angular velocity r, constitute the state space S = [h, V TAS ,β,θ,φ,ψ,p,r] T , the superscript T indicates transposition. According to the safety operation requirements, the throttle must be pushed to the maximum, while maintaining a stable target attitude and retracting the speed brake. Disengage and retract flaps 15, retract the landing gear, adjust the throttle to the go-around thrust (90% N1), and then start to retract flaps 5 above the minimum acceleration altitude. Therefore, the throttle lever δ is selected. T and aileron δ a 、Elevator δ e 、Rudder δ r 、Landing gear L 、Flap δ f and speed brake δ d As the action space A = [δ T ,δ e ,δ a ,δ r ,δ L ,δ f ,δ d ] T Among them, the action is input to the environment by the Actor network, and the environment gives the state and reward at the next moment.

[0152] S1.2: According to flight safety operation requirements, during wind shear disengagement, the pitch angle is controlled at 15° to ensure that the attitude and throttle output changes are not too large and the aircraft maintains stable flight. The reward at this time is:

[0153]

[0154] Among them, ξ represents the parameters of β,θ,φ,ψ,p,r, ξ target Indicates the target state that the parameter needs to reach. According to the above control requirements, the control targets of each posture parameter are as follows:

[0155]

[0156] Before leaving, the reward function is: ||·||1 represents the vector 1-norm, which is the sum of the absolute values ​​of all elements in the vector.

[0157] After the wind shear is released, the aircraft must not only maintain stable and level flight without deviation, but also must not exceed the airspeed limit for flaps and landing gear, and must not exceed the control command altitude. At this time, the reward function is:

[0158]

[0159] in, is the reward function of the speed limit, V TAS represents true airspeed, C1, C2 and C3 are constants, V limit1 V is the airspeed limit when the landing gear is extended. limit2 The airspeed limit for flap 5, V limit3 The airspeed limit is flap 15. The command altitude limit bonus is:

[0160]

[0161] Where C is a constant, h limit Indicates the height limit.

[0162] Therefore, the reward function after leaving is:

[0163]

[0164] S2: Construct the same wind shear environment model and flight dynamics model as those for wind shear escape training, and use the ASPPO algorithm for training. Use the agent to conduct N = 100 simulated wind shear escape experiments to obtain 100 action sequences.

[0165] For the action sequence S in the frequent action sequence pattern solution method j =<a1,a2,a3,…> In this example, the pilots are asked to perform 100 wind shear special situation mission simulation trainings, and the initial state and wind field are different each time. At the same time, the same initial state and wind field are applied to the agent 100 times, and the action sequences given by the pilots and the agent are obtained respectively. Figure 5 As shown, the comparison revealed problems with the pilot's operating actions.

[0166] Furthermore, the time intervals between action sequences in the most frequent action sequence mode were statistically analyzed and compared with the time intervals of pilot operations. It was found that the pilots had timing bias during operations, that is, the pilots operated too early or too late.

[0167] Through the above comparative analysis, we can find out what problems the pilots have in their operations and correct and improve their operations.

[0168] The above embodiments are only descriptions of the preferred embodiments of the present invention, and are not intended to limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary technicians in this field should fall within the protection scope determined by the claims of the present invention.

Claims

1. A method for improving the quality of flight special situation mission training based on reinforcement learning, characterized in that: The following steps are involved: S1: Construct a reinforcement learning environment based on the special situation training task of the target aircraft; Establish reward functions based on flight safety operation requirements; S2: Establish a hybrid Actor-Critic network as a special situation handling decision network, based on a reinforcement learning environment and reward function, and use the policy separation proximal policy optimization algorithm ASPPO to train the special situation handling decision network to obtain the optimal control agent; S3: Based on the optimal control agent, multiple action sequence sets are generated, and the most frequent action sequence in the action sequence set is analyzed to identify problems such as delayed operation, redundant actions, missed actions, and wrong action sequence; In S3, based on the optimal control agent, multiple action sequence sets are generated, and the maximum frequent action sequence in the action sequence set is analyzed to identify the problems of delayed operation action, redundant action, missed action, and wrong action sequence. The methods include: Construct an action sequence S based on the actions given by the optimal decision agent in each experiment j =<a1,a2,a3,…> , where a i ={a} is S j In the element, {a} is an action event, which indicates an action or multiple actions that are operated simultaneously; The length of the preset action sequence is the number of elements; The size of the preset action sequence is the total number of action events contained in all elements; Action sequence S j Middle i The determination requires a preset minimum time interval T m , if the time interval between two or more actions is less than T m , then it is considered that two or more actions are performed simultaneously, where j = 1, 2, 3, ..., N is the experimental sequence number; Preset minimum support p min , for S j Mining frequent sequence patterns to generate candidate frequent action sequence patterns In generating candidate frequent action sequence patterns After that, if at least one subsequence of length κ in the candidate sequence of size κ+1 is infrequent, it will be discarded; if no candidate sequence of size κ+1 can be found, the maximum length sequence pattern is L κ ; Based on the frequent action sequence pattern with the maximum length, the optimal action sequence and time interval in the special task training process are obtained by counting the interval time in the action sequence in multiple experimental data and calculating the mean and variance; Based on the action sequence and action time interval of the pilot's operation in the special situation mission training, the optimal action sequence and time interval obtained by the optimal decision-making result analysis are compared to identify and improve the pilot's operation; Preset minimum support p min , for S j Methods for mining frequent sequence patterns include: The action sequence set Γ=(S1,S2,…,S j ,…,S N ), calculate the frequent action sequence pattern L of size κ κ , calculate the support of all action sequences of size κ in Γ: Among them, p( <A1,A2,…,A k >) represents the sequence in the action sequence set Γ <A1,A2,…,A k >The number of occurrences is defined by the sequence size and sequence length, k≤κ, count(·) represents the count; for the sequence <A1,A2,…,A k >Any two actions If n<m, then i<l, A n , A m Indicates one or more simultaneous actions, regardless of <A1,A2,…,A k In a sequence S j No matter how many times it appears in the , it is counted only once; Determine p( <A1,A2,…,A k >)≥p min Is it true? If not, the algorithm ends, indicating that there is no frequent action sequence pattern L κ ; if it holds, then the frequent action sequence pattern L of size κ+1 in Γ is carried out κ+1 Calculation.

2. The method for improving the quality of flight special situation mission training based on reinforcement learning according to claim 1, characterized in that: In S1, based on the special situation training task of the target aircraft, the method of constructing a reinforcement learning environment includes: Construct the flight dynamics model of the target aircraft and the dynamics model of the special situation mission. The aircraft flight dynamics model will receive the output of the dynamics model of the special situation mission, that is, combine the dynamics model of the special situation mission with the flight dynamics model of the target aircraft to form a reinforcement learning environment.

3. The method for improving the quality of flight special situation mission training based on reinforcement learning according to claim 1, characterized in that: In S2, a hybrid Actor-Critic network is established. The first three layers of the network are multi-layer perception networks (MLP). The subsequent network introduces two branches, discrete-Actor and continuous-Actor, to output discrete actions and continuous actions respectively. The multi-layer perception network (MLP) connects the discrete-Actor and continuous-Actor networks through a fully connected network.

4. The method for improving the quality of flight special situation mission training based on reinforcement learning according to claim 3 is characterized in that: In S2, based on the reinforcement learning environment and reward function, the strategy separation proximal policy optimization algorithm ASPPO is used to train the special situation handling decision network to obtain the optimal control agent method, including: After initializing the network parameters, we first collect the trained state, action, and reward data sets {s t ,a t ,R t }, where t∈[0,T st ], T st Indicates the total time step of a training; Based on the state, action and reward data set obtained by training {s t ,a t ,R t }, use the PPO algorithm to calculate the advantage function Among them, V(s t ) indicates state s t The value function of Indicates status The value function of is output by the Critic network; R t represents the reward value at time t, γ∈[0,1] represents the discount factor; Based on advantage function Calculate the gradient of the loss function of the two action networks and Estimates: Among them, L clip (·) represents the loss function estimator, clip represents the truncation function, and the subscripts d and c represent discrete and continuous actions, respectively. represents the expected value at time t, ε represents the clipping ratio, θ represents the Actor network parameter, represents the probability ratio at time t, represents the previous step strategy, π θ (a t |s t ) represents the probability value given by the previous strategy, π θ Indicates the current strategy, Represents the probability value given by the current strategy, a t represents the action at time t, s t represents the state at time t; due to a t =(a t,d ,a t,c ), a t,d represents a discrete action, a t,c represents continuous action, and the probability ratios of continuous action and discrete action selection at time t are: Among them, r t,d (θ d ) is the probability ratio of the discrete action strategy network at time t, represents the previous discrete action strategy, represents the probability value given by the previous discrete action strategy, represents the current discrete action strategy, Represents the probability value given by the current discrete action strategy; r t,c (θ c ) is the probability ratio of the continuous action strategy network at time t, represents the previous continuous action strategy, represents the probability value given by the previous continuous action strategy, represents the current continuous action strategy, Represents the probability value given by the current continuous action strategy; Based on two action network loss function gradients and The estimates of , discrete and continuous mis-action networks are updated as: where the subscripts d and c denote discrete and continuous actions, respectively, and θ d The network parameters representing discrete actions, θ c represents the network parameters of continuous actions, α represents the learning rate; Critic network updates follow the PPO algorithm; After reaching the maximum number of training times M max Finally, the optimal control agent is obtained.

5. The method for improving the quality of flight special situation mission training based on reinforcement learning according to claim 1, characterized in that: Generate candidate frequent action sequence patterns The methods include: In the frequent action sequence mode L κ To the frequent action sequence pattern L κ+1 In the solution of , candidate frequent action sequence patterns will be generated The candidate rules are as follows: Any two frequent action sequence patterns L κ The sequences ζ1 and ζ2 in ζ1 are merged only if the subsequence obtained by removing the first action from ζ1 is the same as the subsequence obtained by removing the last action from ζ2. The merged result is the concatenation of the last action of ζ1 and ζ2. There are two ways of concatenation: 1) If the last two actions of ζ2 belong to the same element, then the last action of ζ2 is part of the last element of ζ1 in the merged sequence; 2) If the last two actions of ζ2 belong to different elements, then the last action of ζ2 becomes a separate element connected to the tail of ζ1 in the merged sequence. In generating candidate frequent action sequence patterns After that, if at least one subsequence of length κ in the candidate sequence of size κ+1 is infrequent, it will be discarded; if no candidate sequence of size κ+1 can be found, the maximum length sequence pattern is L κ .

6. A system for improving the quality of flight special situation mission training based on reinforcement learning, the system is used to implement the method described in any one of claims 1 to 5, characterized in that: include: Building module, training module and recognition module; The building module is used to build a reinforcement learning environment based on the target aircraft special situation training task; Establish reward functions based on flight safety operation requirements; The training module is used to establish a hybrid Actor-Critic network as a special situation handling decision network. Based on the reinforcement learning environment and reward function, the special situation handling decision network is trained using the policy separation proximal policy optimization algorithm ASPPO to obtain the optimal control agent. The recognition module is used to generate multiple sets of action sequence sets based on the optimal control agent, and analyze the most frequent action sequence in the action sequence set to identify problems such as delayed operation actions, redundant actions, missed actions, and wrong action sequences.

Images