Artificial intelligence driven emergency pilot assistance system and method

Abstract

The present disclosure relates to artificial intelligence driven emergency pilot assistance systems and methods. The system can include an artificial neural network configured to calculate a reward (Q) value based on a state-action vector associated with an aircraft. The state-action vector can include state data associated with the aircraft and action data associated with the aircraft. The system can further include a user output device configured to provide an indication of an action to a user, wherein the action corresponds to an agent action having a highest reward Q value calculated by the artificial neural network.

Description

TECHNICAL FIELD

[0001] The present disclosure relates generally to the field of emergency pilot assistance, and in particular to artificial intelligence driven emergency pilot assistance. BACKGROUND

[0002] Emergencies in an aircraft can place stress on a pilot and can put the pilot under pressure conditions. While there are checklists and protocols for assisting a pilot during an emergency, the checklists and protocols are typically not associated with a current location and system state of the aircraft. The pilot makes decisions about actions to take and, in extreme cases, determines a safe landing zone.

[0003] While pilots are typically well trained to handle emergencies, real-time cross-verification of the actions of the pilot are not typically performed. Analysis is typically performed after the emergency is resolved through debriefing. Furthermore, predictions or visualizations of an action space are not typically performed. As a result, the pilot can be unsure about the results that can result from their actions. SUMMARY

[0004] An artificial intelligence based autonomous pilot assistance agent is described herein. The pilot assistance agent (also referred to herein as an artificial intelligence driven emergency pilot assistance system) can be trained based on scenarios run in a flight simulator. The system can calculate a speed, an altitude, and a heading of an aircraft from a given origin and destination without human intervention. Based on the calculation, the pilot assistance agent can provide advisory assistance and guidance to a pilot by translating the calculated speed, altitude, and heading into control actions that can be performed by the pilot during an emergency. During normal flight conditions, the system can act as a performance evaluation system. In either case, the pilot can maintain control of the aircraft.

[0005] In an example, an emergency pilot assistance system includes an artificial neural network configured to calculate a reward (Q) value based on a state-action vector associated with an aircraft, wherein the state-action vector includes state data associated with the aircraft and action data associated with the aircraft. The system can further include a user output device 310 configured to provide an indication of an action to a user, wherein the action corresponds to an agent action having a highest reward Q value calculated by the artificial neural network.

[0006] In some examples, the highest reward Q value is associated with landing the aircraft at a predetermined destination or an emergency destination calculated in response to an emergency. In some examples, the state data includes a data matrix associated with the aircraft, indicating heading values, position values, system state values, environmental condition values, feedback values, pilot action values, system availability values, roll values, pitch values, yaw values, the rate of change of roll values, the rate of change of pitch values, the rate of change of yaw values, longitude values, latitude values, the rate of change of position values, the rate of change of speed values, or any combination thereof. In some examples, the action data corresponds to changes in heading, speed, roll, pitch, yaw, the rate of change of roll, the rate of change of pitch, the rate of change of yaw, the rate of change of position, the rate of change of speed, or any combination thereof. In some examples, an inverse aircraft model is used to translate agent actions into aircraft surface control actions.

[0007] In some examples, proxy actions are taken from the flight envelope that includes aircraft flight constraints, which include acceleration and deceleration maps, rate of climb, rate of descent, speed thresholds, roll rate of change thresholds, pitch rate of change thresholds, yaw rate of change thresholds, roll thresholds, pitch thresholds, and yaw thresholds.

[0008] In some examples, the artificial neural network includes a deep Q-network. In some examples, the user output device is integrated into the aircraft cockpit. In some examples, the indication of user actions includes visual indications, audio indications, written indications, or any combination thereof. In some examples, the artificial neural network is implemented at one or more processors, wherein the one or more processors are configured to determine state data based on one or more aircraft systems; determine availability data associated with one or more aircraft systems; determine a safe landing zone based on the state data and the availability data; determine action data based on the safe landing zone, availability data, state data, and stored constraint data; and generate a state-action vector based on the state data and action data. In some examples, one or more processors are configured to determine heading and velocity data associated with the highest reward Q-value and perform one or more inverse dynamic operations to translate the heading and velocity data into surrogate actions. In some examples, one or more processors are configured to compare user input with the indicated actions and generate a performance rating.

[0009] In some examples, the user output device is configured to alert the user when the user input differs from the action. In some examples, one or more processors are configured to generate an updated state-action vector associated with the aircraft based on updated state data and updated action data, and to calculate an additional reward Q-value based on the updated state-action vector, wherein the user output device is configured to provide the user with additional instructions for additional actions, wherein the additional actions correspond to updated proxy actions with the highest reward Q-value calculated by an artificial neural network.

[0010] In some examples, a method for training an artificial neural network for an emergency pilot assistance system includes: generating training data for a deep Q-network by receiving state data associated with the aircraft and its environment from a simulator while a user operates the simulator; receiving action data associated with the user's actions from the simulator; generating a set of state-action vectors based on the state data and action data; and determining a reward Q-value associated with the set of state-action vectors. The method also includes training the deep Q-network based on the training data.

[0011] In some examples, the method includes: generating additional training data for a deep Q-network by receiving automated state data associated with an aircraft from a memory, the automated state data corresponding to an automated scenario; receiving automated action data associated with the automated scenario from a memory; generating a set of additional state-action vectors based on the automated state data and the automated action data; and determining an additional reward Q-value associated with the set of additional state-action vectors. The method also includes training the deep Q-network based on the additional training data.

[0012] In the example, the emergency pilot assistance method includes: calculating a reward Q-value using a deep Q-network, wherein the reward value is based on a state-action vector associated with the aircraft, and wherein the state-action vector includes state data associated with the aircraft and action data associated with the aircraft. The method further includes providing a user with an action instruction at a user output device, wherein the action corresponds to an agent action having the highest reward Q-value calculated by the deep Q-network. In some examples, the highest reward Q-value is associated with landing the aircraft at a predetermined destination or an emergency destination calculated in response to an emergency. Attached Figure Description

[0013] Figure 1 This is a block diagram depicting an example of a system used to train an artificial neural network in the first training phase.

[0014] Figure 2 This is a block diagram depicting an example of a system used to train an artificial neural network in the second training phase.

[0015] Figure 3This is a block diagram depicting an example of the emergency pilot assistance system in the first iteration.

[0016] Figure 4 This is a block diagram depicting an example of the emergency pilot assistance system in the second iteration.

[0017] Figure 5 This is a block diagram depicting an example of state data.

[0018] Figure 6 This is a block diagram describing an example of motion data.

[0019] Figure 7 This is a block diagram illustrating an example of a flight envelope.

[0020] Figure 8 This is a flowchart depicting an example of a method used to train an artificial neural network.

[0021] Figure 9 This is a flowchart illustrating an example of a method for emergency pilot assistance.

[0022] While this disclosure is susceptible to various modifications and alternatives, specific examples have been shown by way of example in the accompanying drawings and will be described in detail herein. However, it should be understood that this disclosure is not intended to be limited to the particular forms disclosed. Rather, it is intended to cover all modifications, equivalents, and substitutions that fall within the scope of this disclosure. Detailed Implementation

[0023] This paper describes a reinforcement learning-based autonomous pilot assistance agent (also referred to herein as an AI-driven emergency pilot assistance system), which can be trained using an aircraft simulator and can perform tasks such as calculating the aircraft's speed, altitude, and heading from a given origin and destination without human intervention. The pilot assistance agent can be used to assist and guide pilots during emergency situations. For example, the calculated speed, altitude, and heading can be translated into control actions that the pilot can perform to guide the aircraft to a safe landing zone.

[0024] The system described in this paper can rely on deep Q-networks to achieve model-free deep Q-learning to obtain a complete reward-based mapping. This mapping can be used to determine action sequences during emergency situations. As a brief overview of deep Q-learning, when applied in this paper, during an emergency, the system can identify candidate targets (which may include, for example, determining a safe landing location). The system also has access to a user policy, which may be based on aircraft flight constraints, flight envelope, maps of acceleration and deceleration, rate of climb, and rate of descent. The user policy effectively describes the possible actions that can be taken at any given time within the aircraft. Based on these parameters, the system can iteratively map possible sequences of actions that will bring the aircraft to the candidate targets. If a sequence successfully brings the aircraft to the candidate target (i.e., if the sequence will result in a safe landing at the safe landing location), a high reward Q-value (e.g., 1.0) can be assigned. If the sequence is unsuccessful, a low reward Q-value (e.g., 0.0) can be assigned. Since each sequence can branch at each iteration, the reward Q-value can increase or decrease throughout the iterations, depending on the probability of a safe landing at any given point in the action sequence.

[0025] The system can interact with the aircraft environment and the pilot to select actions in a way that maximizes future rewards. During system calculations, because future states cannot be perfectly determined, a standard assumption can be made that future rewards can be discounted at each time step by a set factor. The future discounted reward R can be calculated as follows. t :

[0026]

[0027] Where T is the flight duration, t′ is the current time step, t is the next time step in the iteration, γ is the discount factor, and r t′ This is the current discount return. For the example described in this article, γ is set to 0.99. However, other values ​​are possible.

[0028] The expected action-value function Q*(s, a) can be defined as the optimal expected reward achievable by following a policy based on sequence s and action a. Q*(s, a) can be derived from the Bellman equation, which is known relative to deep Q-learning. For the purposes of this disclosure, this relationship can be described as follows: if the optimal value Q*(s, a) of the sequence at the next time step is known for all possible actions, then the optimization policy is to select the action that maximizes the expected value of r + ΥQ*(s', a'), where r is the discounted reward and Υ is the discount factor.

[0029] The reinforcement learning algorithm described above can be used to estimate the action-value function by using the Bellman equation as an iterative update. If executed fully, the algorithm will converge to the optimal action-value function. However, in practice, this approach can be impractical because the action-value function would be estimated separately for each sequence without any generalization. Therefore, the computation would scale exponentially, potentially requiring more processing power than is available. Instead, a function approximator can be used to estimate the action-value function Q(s, a; θ) < Q(s, a). In reinforcement learning, this is typically a linear function approximator. By relying on the training data received during simulation, deep Q-networks can be developed to approximate the optimal action to achieve the maximum probability of a successful outcome.

[0030] Figure 1 and Figure 2 A system for training an artificial neural network for use with an emergency pilot assistance system is described. Training the artificial neural network can occur in two phases. The first phase may include training the model based on input from a pilot in a simulator and determining the success or failure of the outcome during an emergency training scenario. During the second phase, the model can be trained based on an automated scenario without the presence of a pilot.

[0031] During the first phase, the training of the artificial neural network can be performed concurrently with the training of the pilot in a training simulator. The system can learn an end-to-end mapping of the aircraft's flight path (e.g., speed, altitude, and heading) based on environmental observations and user input, along with mission rewards (e.g., safe landing) as supervision. Rewards can be calculated based on ensuring a safe landing at or near the desired location. From the perspective of the trained system, the pilot's actions can be incorporated into a policy that also includes constraints such as the flight envelope, acceleration and deceleration maps, climb rate, descent rate, and other policy data for safe flight. From the pilot's perspective, the system can behave as an adaptive interface that learns personalized mappings from the pilot's commands, environment, target space, and flight constraint policy to the flight path actions and its other parameters.

[0032] refer to Figure 1 An example of a system 100 for training an artificial neural network in a first training phase is depicted. System 100 may include a simulator 110 and a deep Q-network 140. Those skilled in the art who will benefit from this disclosure will understand that the deep Q-network 140 can be implemented as described in reference... Figure 3 This is part of a broader artificial neural network, further described below. Simulator 110 can simulate aircraft 114 and its environment 112 during pilot training for user 116.

[0033] During operation, as user 116 performs training exercises in simulator 110, state data 120 associated with aircraft 114 and its environment 112 can be collected from simulator 110. State data 120 can indicate the current state of aircraft 114 and environment 112. A portion of state data 120 can also be based on system availability 122 of aircraft 114. For example, during an emergency, one or more systems of aircraft 114 may be inoperable or otherwise unavailable. These factors can be considered when generating state data 120. State data 120 can also be based on aircraft performance operational constraints 124, which can represent limitations on what a particular aircraft can do in a specific scenario running at simulator 110.

[0034] Motion data 126 can also be collected from simulator 110. Motion data 126 can be derived from actions 115 taken by user 116 during flight training. Motion data 126 can also be based on flight envelope 131, which represents actions that can be taken for a specific aircraft.

[0035] Training data 130 can be compiled based on state data 120 and action data 126. Training data 130 may include a set of state-action vectors 132 formed by combining state data 120 and action data 126 in incremental steps during simulation. A reward Q-value 134 can be determined based on the results associated with this set of state-action vectors 132 and based on the discount return function described herein. Training data 130 may also include the reward Q-value 134 and can be used as training data for a deep Q-network 140.

[0036] A challenge typically associated with training emergency assistance systems is adapting standard deep reinforcement learning techniques that utilize continuous inputs from action 115 and adjust the inputs based on the results of feedback associated with action 115. As described herein, when user 116 actively uses simulator 110, system 100 can learn an approximate state-action value function by using human deep Q-learning in a loop. This function computes the expected future return value without having to compute every possible path in the state-action vector 132 for each action given current environmental observations and the pilot's control inputs. Deep Q-network 140 can be trained to determine the high-value action closest to the user's input, rather than finding the highest-value action. This approach balances taking the optimal action with preserving the pilot's feedback control loop. This approach also allows user 116 to directly adjust the level of assistance via parameter α ∈ [0, 1], which can set a threshold for tolerance to suboptimal actions.

[0037] Standard deep reinforcement learning algorithms can include a large number of interactions over very long periods to provide sufficient training. Simulator training alone may be insufficient, as obtaining enough data may be impractical. During the second phase of training, pilot control inputs can be replaced with automated scenario files containing fixed control inputs from different origins to different destinations. During these scenarios, the automated scenario files can cover a wider range of aircraft operating conditions. This automated training approach can also be used to cover extreme emergency situations that may be difficult to simulate with a pilot. In some cases, this training will enable the system to determine safe maneuver routes more reliably than a pilot by learning based on the full spectrum of inputs from each scenario and on scenarios not yet anticipated by the pilot.

[0038] The remainder of the second phase of training can be referenced. Figure 1 The description is identical. Deep Q-learning can be used to learn an approximate state-action value function, which computes the expected future return of an action given current environmental observations, policy constraints, and inputs from an automated scenario. Equipped with this value function, the system can execute a high-value action that best approximates the control inputs of the scenario. The reward function for the agent can be a combination of known items computed for each state, plus the terminal reward provided by the user for a successful and safe landing of the aircraft.

[0039] Reference Figure 2 This paper depicts an example of a system 200 for training an artificial neural network in a second training phase. System 200 may include a memory 210 and a deep Q-network 140. The memory 210 may store automated scenarios 212 associated with the aircraft 114. In fact, many automated scenarios will be stored in the memory 210.

[0040] Memory 210 may include memory devices such as random access memory (RAM), read-only memory (ROM), disk storage, optical disk storage, flash memory, another type of memory capable of storing data and processor instructions, or combinations thereof. Furthermore, memory may be part of a processing device (not shown) (such as a computing device).

[0041] During operation, automation status data 220 associated with aircraft 114 and automation scenario 212 can be collected. In some examples, the collection may take the form of multiple automation scenario files. Automation status data 220 can indicate the current status of aircraft 114 during automation scenario 212. (See reference...) Figure 1As described, a portion of the automated state data 220 may also be based on the system availability 122 and aircraft performance operational constraints 124 of the aircraft 114. Automated action data 226 may also be derived from the automated scenario 212 and flight envelope 131, representing actions that can be taken for the aircraft 114.

[0042] Based on the automated state data 220 and automated action data 226, additional training data 230 can be compiled. The additional training data 230 may include a set of additional state-action vectors 232 formed by combining the automated state data 220 and the automated action data 226. An additional reward Q-value 234 can be determined based on the results associated with this set of additional state-action vectors 232 and based on the discount return function described herein. The additional training data 230 may include the additional reward Q-value 234 and can be used to train a deep Q-network 140.

[0043] Although Figure 2 This is described relative to a single automated scenario 212, but in practice, many scenarios can be stored in memory 210 and used to generate additional training data 230. Because the additional training data 230 is not compiled based on real-time scenarios, it can be generated more quickly, thus enabling the generation of sufficient training data to fully train the deep Q-network 140. By using data generated by the pilot simulator and automated scenario data, the deep Q-network 140 can be trained to learn actual pilot responses for a complete set of emergency scenarios.

[0044] refer to Figure 3 and Figure 4 A deep Q-network 140 can be implemented in the example of the assistance system 300 to assist the pilot during an emergency. As a brief overview, during an emergency, system 300 can determine the state of aircraft 302. The state can involve factors such as whether the aircraft is landing, approaching, or climbing. Other possible states may exist. Similarly, system 300 can analyze the availability of the onboard systems of aircraft 302 to determine availability data 342. System availability can involve potential engine failure, ground control failure, fuel availability, and structural integrity. Based on system availability and aircraft situational conditions, system 300 can determine a safe landing zone and guide the pilot to perform maneuvers. Based on the current system state and current aircraft feedback, system 300 can estimate a near-optimal trajectory to the safe landing destination. System 300 can continuously assess the situation to guide the pilot to take necessary actions.

[0045] System 300 may include or otherwise be implemented at aircraft 302. The system may also include one or more processors 330, which may be implemented at aircraft 302 or, in some examples, may be distributed in a distributed manner. System 300 may also include an artificial neural network 338. Parts of system 300 may be implemented at one or more processors 330. However, for clarity, different functional aspects of system 300 may be depicted as separate from processors 330.

[0046] Aircraft 302 may include aircraft systems 304 and a cockpit 308. Aircraft systems 304 may include mechanical systems, electrical systems, sensors, actuators, etc. At least some of the aircraft systems 304 are capable of determining the presence of an emergency 306. Cockpit 308 may include user output devices 310. User output devices 310 may include visual output systems, audio output systems, text output systems, etc. Aircraft 302 may include additional systems that perform functions typically associated with the aircraft; however, for clarity, these systems are not described here. Figure 3 Omitted in .

[0047] One or more processors 330 may include a microcontroller, a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a peripheral interface controller (PIC), another type of microprocessor, and / or combinations thereof. Furthermore, one or more processors 330 may be implemented as integrated circuits, complementary metal-oxide-semiconductor (CMOS) field-effect transistor (MOSFET) circuits, very large-scale integrated circuits (VLSI) circuits, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), combinations of logic gates, other types of digital or analog electrical design components, or combinations thereof.

[0048] Artificial neural network 338 may include deep Q-network 140 and may be trained as described herein. Specifically, the artificial neural network may be trained to perform an approximation function to determine a reward Q-value associated with the state and possible actions of the aircraft 302. Those skilled in the art who benefit from this disclosure will understand that artificial neural network 338 may be a broader network, wherein deep Q-network 140 may be a part of artificial neural network 338.

[0049] During operation, an emergency 306 may be caused by or detected by one or more aircraft systems 304. In response to an emergency 306, one or more processors 330 may determine state data 334 and action data 336 based on the aircraft systems 304. For example, state data 334 may include a matrix of aircraft heading, position and speed, current state, environmental conditions, feedback, pilot actions, aircraft system availability, such as current values ​​of roll, pitch, and yaw, rates of change of roll, pitch, and yaw, longitude and latitude, rates of change of position and speed, other state parameters or combinations thereof associated with the aircraft 302. Action data 336 may be based on heading and speed, such as values ​​of roll, pitch, and yaw, rates of change of roll, pitch, and yaw, rates of change of position and speed. A state-action vector 332 may be generated based on state data 334 and action data 336.

[0050] Processor 330 can determine and / or compile availability data related to aircraft system 304. For example, in an emergency 306, some systems may be unavailable. A safe landing zone 344 can be determined based on status data 334 and availability data 342. The safe landing zone 344 can be a predetermined destination 346, or in some cases, an emergency destination 348 can be determined based on the location of aircraft 302 and availability data 342 associated with aircraft 302 and stored constraint data 358. Action data 336 can depend on the safe landing zone 344, availability data 342, status data 334, and stored constraint data 358.

[0051] Artificial neural network 338 can be used to determine heading and velocity data 350 that can be associated with a calculated reward Q-value 352. The reward Q-value 352 can be determined based on state-action vector 332 and can be associated with enabling aircraft 302 to land safely at safe landing zone 344. For example, the higher the reward Q-value 352, the more likely a safe landing is. Based on heading and velocity data 350, heading and velocity data 354 can be associated with the highest reward Q-value 356 determined by artificial neural network 338.

[0052] One or more inverse dynamic operations 360 can be performed to translate heading and speed data 354 into proxy actions 366. Furthermore, in some examples, additional data from the heading and speed data 350 can be translated into proxy actions 362. Each proxy action 362 can be associated with a reward Q value 364 that corresponds to a reward Q value 352. Proxy actions 366 can be associated with a highest reward Q value 368 that corresponds to the highest reward Q value 356 of the heading and speed data 354. An inverse aircraft model 367 can be used to translate proxy actions 366 into surface control actions 369 that can be used as instructions from user 324 to guide aircraft 302.

[0053] Within cockpit 308, user output device 310 can provide user 324 with instructions 312 for action 314. Action 314 may correspond to proxy action 366 and may also be derived from surface control action 369. Instruction 312 for action 314 may include visual instructions 316, audio instructions 318, written instructions 320, or any combination thereof. If user 324 does not perform action 314, user output device 310 may generate warning 322. User can perform actions using user input 326, which may include flight controls and / or other controls associated with the aircraft cockpit. In the absence of an emergency, system 300 can still generate a flight-related performance rating 370 based on a comparison of proxy action 362 generated by artificial neural network 338 with user input 326.

[0054] It should be noted that the process described regarding system 300 is iterative and can be executed continuously during flight and / or during in-flight emergencies. Therefore, as the state-action vector 332 changes, agent actions can be continuously fed to the user output device 310. (Refer to...) Figure 4 This describes the continuous updating. When aircraft 302 changes its vector and when availability data 342 changes based on aircraft system 304 during emergency 306, an updated state-action vector 432 can be generated. The updated state-action vector 432 may include updated state data 434 and updated action data 436.

[0055] Artificial neural network 338 can be used to generate updated heading and velocity data 450, which can be associated with an additional reward Q value 452. Updated heading and velocity data 454, associated with the highest additional reward Q value 456, can be determined to safely guide user 324 to land at a safe landing zone 344. Based on the updated heading and velocity data 450, updated agent actions 462 can be generated and associated with an additional reward Q value 464, which can be correlated with the additional reward Q value of the updated heading and velocity data 450. Updated agent actions 466 can be associated with the highest additional reward Q value 468, which can be correlated with the highest additional reward Q value 456 of the updated heading and velocity data 454. Updated agent actions 466 can be used to generate updated surface control actions 469.

[0056] User output device 310 can be configured to provide user 324 with additional instructions 412 for additional actions 414. Additional instructions 412 may include additional visual instructions 416, additional audio instructions 418, additional written instructions 420, or any combination thereof. If user 324 does not perform the additional action 414, an additional warning 422 may be generated. As previously described, an updated performance rating 470 may be generated based on a comparison of user input 326 with the updated agent action 462.

[0057] By providing instructions to the pilot to take actions that will ensure a safe landing in a safe landing zone, System 300 can reduce pilot workload in emergency situations. Furthermore, System 300 can warn the pilot when actions could lead to catastrophic failures. Moreover, even in the absence of an emergency, System 300 can still assess pilot performance for training purposes. Other advantages may also exist.

[0058] refer to Figure 5 An example of state data 500 is depicted. State data 500 may include a data matrix 502 associated with an aircraft (such as aircraft 302), which may indicate heading values ​​504, position values ​​506, system state values ​​508, environmental condition values ​​510, feedback values ​​512, pilot action values ​​514, system availability values ​​516, roll values ​​518, pitch values ​​520, yaw values ​​522, rate of change of roll values ​​524, rate of change of pitch values ​​526, rate of change of yaw values ​​528, longitude values ​​530, latitude values ​​532, rate of change of position values ​​534, rate of change of speed values ​​536, or any combination thereof. State data 500 may correspond to state data 120, automation state data 220, state data 334, and / or updated state data 434.

[0059] refer to Figure 6 An example of motion data 600 is depicted. Motion data 600 may include changes in heading 602, changes in speed 604, changes in roll 606, changes in pitch 608, changes in yaw 610, changes in the rate of change of roll 612, changes in the rate of change of pitch 614, changes in the rate of change of yaw 616, changes in the rate of change of position 618, changes in the rate of change of speed 620, or any combination thereof. Motion data 600 may correspond to motion data 126, automated motion data 226, motion data 336, and / or updated motion data 436.

[0060] refer to Figure 7An example of a flight envelope 700 is depicted. Motion data 336 may be based at least in part on the flight envelope 700. The flight envelope 700 may include aircraft flight constraints 702. Aircraft flight constraints 702 may include acceleration and deceleration maps 704, rate of climb 706, rate of descent 708, speed threshold 710, roll rate of change threshold 712, pitch rate of change threshold 714, yaw rate of change threshold 716, roll threshold 718, pitch threshold 720, and yaw threshold 722.

[0061] refer to Figure 8 The diagram depicts a flowchart illustrating an example of a method 800 for training an artificial neural network. This method 800 may include generating training data for a deep Q-network at point 802. For example, training data 130 for a deep Q-network 140 may be generated by a simulator 110.

[0062] Generating training data may include receiving state data associated with the aircraft and its environment from the simulator at point 804 while the user is operating the simulator. For example, state data 120 may be received from simulator 110 while user 116 is operating simulator 110.

[0063] Generating training data may also include receiving motion data associated with the user's actions from the simulator at point 806. For example, motion data 126 can be received from simulator 110.

[0064] Generating training data may also include, at point 808, generating a set of state-action vectors based on the state data and action data. For example, a set of state-action vectors 132 can be generated based on state data 120 and action data 126.

[0065] Generating training data may include, at 810, determining a reward Q-value 134 associated with the set of state-action vectors 132. For example, the reward Q-value 134 may be determined by system 100 and may be associated with the set of state-action vectors 132.

[0066] Method 800 may further include, at 812, training a deep Q-network based on training data. For example, a deep Q-network 140 may be trained based on training data 130.

[0067] Method 800 may also include, at 814, generating additional training data for the deep Q-network. For example, additional training data 230 may be generated based on the automated scene 212 and additional automated scenes during additional iterations.

[0068] Generating additional training data may include receiving, at point 816, automation state data associated with the aircraft from memory, which corresponds to an automation scenario. For example, automation state data 220 may be received from memory 210.

[0069] Generating additional training data may also include receiving automated action data from memory at point 818, which is associated with the automated scenario. For example, automated action data 226 may be received from memory 210.

[0070] Generating additional training data may also include generating an additional set of state-action vectors based on the automated state data and automated action data at point 820. For example, an additional set of state-action vectors 232 may be generated based on automated state data 220 and automated action data 226.

[0071] Generating additional training data could include, at point 822, determining an additional reward Q-value associated with this set of additional state-action vectors. For example, an additional reward Q-value 234 could be generated and associated with this set of additional state-action vectors 232.

[0072] Method 800 may include, at 824, training a deep Q-network based on additional training data. For example, a deep Q-network 140 may be trained based on additional training data 230.

[0073] refer to Figure 9 The diagram depicts a flowchart illustrating an example of a method 900 for emergency pilot assistance. Method 900 may include, at 902, calculating a reward Q-value using a deep Q-network, wherein the reward value is based on a state-action vector associated with the aircraft, and wherein the state-action vector includes state data associated with the aircraft and action data associated with the aircraft. For example, the reward Q-value 352 may be calculated using a deep Q-network 140 and may be based on a state-action vector 332 associated with aircraft 302.

[0074] Method 900 may further include, at 904, providing a user with an instruction for an action at a user output device, wherein the action corresponds to an agent action having the highest reward Q value calculated by the deep Q-network. For example, an instruction 312 for action 314 may be provided to user 324 at user output device 310.

[0075] Further illustrative and non-exclusive examples according to this disclosure are described in the following paragraphs:

[0076] In an example according to this disclosure, an emergency pilot assistance system (300) includes: an artificial neural network (338) configured to calculate a reward (Q) value (352) based on a state-action vector (332) associated with an aircraft (302), wherein the state-action vector (332) includes state data (334, 500) associated with the aircraft (302) and action data (336, 600) associated with the aircraft (302); and a user output device (310) configured to provide a user (324) with an instruction (312) of an action (314), wherein the action (314) corresponds to a proxy action (366) having the highest reward Q value (368) calculated by the artificial neural network (338).

[0077] Optionally, in the system (300) of the preceding paragraph, the highest reward Q value (356) is associated with enabling the aircraft (302) to land at the predetermined destination (346) or an emergency destination (348) calculated in response to an emergency (306).

[0078] Optionally, in the system (300) of one of the preceding paragraphs, the status data (500) includes a data matrix (502) associated with the aircraft (302), which indicates heading values ​​(504), position values ​​(506), system status values ​​(508), environmental condition values ​​(510), feedback values ​​(512), pilot action values ​​(514), system availability values ​​(516), roll values ​​(518), pitch values ​​(520), yaw values ​​(522), rate of change of roll values ​​(524), rate of change of pitch values ​​(526), ​​rate of change of yaw values ​​(528), longitude values ​​(530), latitude values ​​(532), rate of change of position values ​​(534), rate of change of speed values ​​(536), or any combination thereof.

[0079] Optionally, in the system (300) of one of the preceding paragraphs, the motion data (600) corresponds to changes in heading (602), changes in speed (604), changes in roll (606), changes in pitch (608), changes in yaw (610), changes in the rate of change of roll (612), changes in the rate of change of pitch (614), changes in the rate of change of yaw (616), changes in the rate of change of position (618), changes in the rate of change of speed (620), or any combination thereof.

[0080] Optionally, in one of the preceding paragraphs of the system (300), the agent action (366) is converted into an aircraft surface control action (369) using an inverse aircraft model (367).

[0081] Optionally, in one of the preceding paragraphs of the system (300), an agent action (366) is taken from the flight envelope (700) including aircraft flight constraints (702), wherein the aircraft flight constraints (702) include acceleration and deceleration maps (704), rate of climb (706), rate of descent (708), speed threshold (710), roll rate threshold (712), pitch rate threshold (714), yaw rate threshold (716), roll threshold (718), pitch threshold (720), and yaw threshold (722).

[0082] Optionally, in one of the preceding paragraphs, the artificial neural network (338) includes a deep Q-network (140).

[0083] Optionally, in one of the preceding systems (300), a user output device (310) is integrated into the cockpit (308) of the aircraft (302), and wherein the instruction (312) of the action (314) includes visual instructions (316), audio instructions (318), written instructions (320), or any combination thereof.

[0084] Optionally, in the system (300) of one of the preceding paragraphs, the artificial neural network (338) is implemented at one or more processors (330), and wherein the one or more processors (330) are further configured to: determine state data (334) based on one or more aircraft systems (304); determine availability data (342) associated with one or more aircraft systems (304); determine a safe landing zone (344) based on the state data (334) and the availability data (342); determine action data (336) based on the safe landing zone (344), the availability data (342), the state data (334), and the stored constraint data (358); and generate a state-action vector (332) based on the state data (334) and the action data (336).

[0085] Optionally, in the system (300) of one of the preceding paragraphs, the artificial neural network (338) is implemented at one or more processors (330), wherein the one or more processors (330) are further configured to: determine heading and velocity data (354) associated with the highest reward Q value (356); and perform one or more inverse dynamic operations (360) to convert the heading and velocity data (354) into agent actions (366).

[0086] Optionally, in one of the preceding paragraphs of the system (300), the artificial neural network (338) is implemented at one or more processors (330), and wherein the one or more processors (330) are further configured to: compare user input (326) with actions (314) and generate performance ratings (370).

[0087] Optionally, in one of the preceding paragraphs of the system (300), the user output device (310) is further configured to warn the user (324) when the user input (326) is different from the action (314).

[0088] Optionally, in the system (300) of one of the preceding paragraphs, the artificial neural network (338) is implemented at one or more processors (330), and wherein the one or more processors (330) are further configured to: generate an updated state-action vector (432) associated with the aircraft (302) based on updated state data (434) and updated action data (436); and calculate an additional reward Q value (452) based on the updated state-action vector (432), wherein the user output device (310) is configured to provide the user (324) with an additional instruction (412) for an additional action (414), wherein the additional action (414) corresponds to an updated proxy action (466) having an updated highest reward Q value (468) calculated by the artificial neural network (338).

[0089] In another embodiment of the present disclosure, a method (800) for training an artificial neural network (140) for an emergency pilot assistance system (300) includes generating (802) training data (130) for a deep Q-network (140) by receiving (804) state numbers associated with the aircraft (114) and the environment (112) of the aircraft (114) from the simulator (110) while a user (116) is operating the simulator (110). According to (120, 500); receive (806) motion data (126, 600) associated with the action (115) of the user (116) from the simulator (110); generate (808) a set of state-action vectors (132) based on the state data (120) and the motion data (126); determine (810) the reward Q value (134) associated with the set of state-action vectors (132); and train (812) a deep Q network (140) based on the training data (130).

[0090] Optionally, the method (800) in the preceding paragraph further includes generating (814) additional training data (230) for the deep Q-network (140) by: receiving (816) automated state data (220) associated with the aircraft (114) from memory (210), the automated state data (220) corresponding to an automated scene (212); receiving (818) automated action data (226) associated with the automated scene (212) from memory (210); generating (820) a set of additional state-action vectors (232) based on the automated state data (220) and the automated action data (226); determining (822) an additional reward Q value (234) associated with the set of additional state-action vectors (232); and training (824) the deep Q-network (140) based on the additional training data (230).

[0091] Optionally, in one of the methods (800) in the preceding paragraphs, the state data (500) includes a data matrix (502) associated with the aircraft (114), which indicates heading values ​​(504), position values ​​(506), system state values ​​(508), environmental condition values ​​(510), feedback values ​​(512), pilot action values ​​(514), system availability values ​​(516), roll values ​​(518), pitch values ​​(520), yaw values ​​(522), rate of change of roll values ​​(524), rate of change of pitch values ​​(526), ​​rate of change of yaw values ​​(528), longitude values ​​(530), latitude values ​​(532), rate of change of position values ​​(534), rate of change of speed values ​​(536), or any combination thereof.

[0092] Optionally, in one of the methods (800) in the preceding paragraphs, the motion data (126, 600) correspond to changes in heading (602), changes in speed (604), changes in roll (606), changes in pitch (608), changes in yaw (610), changes in the rate of change of roll (612), changes in the rate of change of pitch (614), changes in the rate of change of yaw (616), changes in the rate of change of position (618), changes in the rate of change of speed (620), or any combination thereof.

[0093] Optionally, in one of the methods (800) in the preceding paragraphs, the motion data (126, 600) are based on a flight envelope (131, 700) including aircraft flight constraints (702), wherein the aircraft flight constraints (702) include acceleration and deceleration maps (704), rate of climb (706), rate of descent (708), speed threshold (710), roll rate threshold (712), pitch rate threshold (714), yaw rate threshold (716), roll threshold (718), pitch threshold (720), and yaw threshold (722).

[0094] In another embodiment of the present disclosure, an emergency pilot assistance method (900) includes: calculating (902) a reward (Q) value (352) using a deep Q-network (140), wherein the reward value (352) is based on a state-action vector (332) associated with an aircraft (302), and wherein the state-action vector (332) includes state data (334) associated with the aircraft (302) and action data (336) associated with the aircraft (302); and providing an instruction (312) for an action (314) to a user (324) at a user output device (310), wherein the action (314) corresponds to a proxy action (366) having the highest reward Q value (368) calculated by the deep Q-network (140).

[0095] Optionally, in the method (900) in the preceding paragraph, the highest reward Q value (356) is associated with landing the aircraft (302) at the predetermined destination (346) or at an emergency destination (348) calculated in response to an emergency (306).

[0096] Although different examples have been shown and described, this disclosure is not limited thereto and will be understood to include all such modifications and variations that are obvious to those skilled in the art.

Claims

1. An emergency pilot assistance system, comprising: An artificial neural network is configured to calculate a reward Q-value based on a state-action vector associated with an aircraft, wherein the state-action vector includes state data associated with the aircraft and action data associated with the aircraft; and A user output device is configured to provide a user with an instruction for an action, wherein the action corresponds to an agent action having the highest reward Q-value calculated by the artificial neural network. The artificial neural network is trained in two stages. The first phase includes training the artificial neural network based on input from pilots in a simulator and determining the success of the results during emergency training scenarios; and The second phase involves training the artificial neural network based on automated scenarios in the absence of a pilot.

2. The system according to claim 1, wherein, The highest reward Q value is associated with landing the aircraft at a predetermined destination or an emergency destination calculated in response to an emergency.

3. The system according to claim 1, wherein, The status data includes a data matrix associated with the aircraft, which indicates heading values, position values, system status values, environmental condition values, feedback values, pilot action values, system availability values, roll values, pitch values, yaw values, rate of change of roll values, rate of change of pitch values, rate of change of yaw values, longitude values, latitude values, rate of change of position values, rate of change of speed values, or any combination of the above values.

4. The system according to claim 1, wherein, The motion data corresponds to changes in heading, speed, roll, pitch, yaw, rate of change of roll, rate of change of pitch, rate of change of yaw, rate of change of position, rate of change of speed, or any combination of the above values.

5. The system according to claim 4, wherein, The agent actions are converted into aircraft surface control actions using an inverse aircraft model.

6. The system according to claim 1, wherein, The agent action is taken from the flight envelope that includes aircraft flight constraints, wherein the aircraft flight constraints include acceleration and deceleration maps, rate of climb, rate of descent, speed threshold, roll rate of change threshold, pitch rate of change threshold, yaw rate of change threshold, roll threshold, pitch threshold, and yaw threshold.

7. The system according to claim 1, wherein, The artificial neural network includes a deep Q-network.

8. The system according to claim 1, wherein, The user output device is integrated into the cockpit of the aircraft, and the instructions for the action include visual instructions, audio instructions, written instructions, or any combination of the above.

9. The system according to claim 1, wherein, The artificial neural network is implemented at one or more processors, and wherein the one or more processors are further configured to: The status data is determined based on one or more aircraft systems; Determine availability data associated with the one or more aircraft systems; A safe landing zone is determined based on the status data and the availability data; The action data is determined based on the safe landing zone, the availability data, the status data, and the stored constraint data; and The state-action vector is generated based on the state data and the action data.

10. The system according to claim 1, wherein, The artificial neural network is implemented at one or more processors, and wherein the one or more processors are further configured to: Determine the heading and speed data associated with the highest reward Q value; and Perform one or more inverse dynamic operations to transform the heading and speed data into the agent action.

11. The system according to claim 1, wherein, The artificial neural network is implemented at one or more processors, and wherein the one or more processors are further configured to: The user input is compared with the action, and a performance rating is generated.

12. The system according to claim 1, wherein, The user output device is further configured to warn the user when the user inputs something different from the action.

13. The system according to claim 1, wherein, The artificial neural network is implemented at one or more processors, and wherein the one or more processors are further configured to: An updated state-action vector associated with the aircraft is generated based on the updated state data and updated action data; and An additional reward Q-value is calculated based on the updated state-action vector, wherein the user output device is configured to provide the user with additional instructions for additional actions, wherein the additional actions correspond to updated agent actions with updated highest reward Q-values ​​calculated by the artificial neural network.

14. An emergency pilot assistance method using the emergency pilot assistance system according to any of the preceding claims, comprising: A reward Q-value is calculated using a deep Q-network, wherein the reward Q-value is based on a state-action vector associated with the aircraft, and wherein the state-action vector includes state data associated with the aircraft and action data associated with the aircraft; and The user is given instructions for an action at the user output device, wherein the action corresponds to an agent action with the highest reward Q value calculated by the deep Q network.

15. The method according to claim 14, wherein, The highest reward Q value is associated with landing the aircraft at a predetermined destination or an emergency destination calculated in response to an emergency.

Images