A neural network-based system for flight situation analysis and communications

Abstract

A pilot support system comprises: a processing circuit; and a memory storing instructions, which, when executed by the processing circuit, cause the processing circuit to: receive input data regarding a current state of a vehicle; encode the input data to generate encoded input data; supply the encoded input data to a trained statistical model; calculate a current situation of the vehicle based on the input data using the trained statistical model; calculate one or more pilot feedback indications based on the current situation using the trained statistical model; and provide the one or more pilot feedback indications to a cockpit of the vehicle.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to and the benefit of U.S. Provisional Patent Application No. 63 / 025,825, filed in the U.S. Patent and Trademark Office on May 15, 2020, the entire disclosure of which is incorporated herein by reference. Technical Field

[0003] Aspects of embodiments of the present disclosure relate to systems and methods for analyzing flight conditions using neural networks and user interfaces for communicating flight analysis information to a pilot. Background Art

[0004] When an aircraft is flying, a human pilot integrates information from a variety of different standard sensors or instruments to maintain situational awareness. Different measurements from sensors and instruments can indicate different operational scenarios to the pilot, which can correspond to different flight phases, such as takeoff, ascent, cruise, descent, and landing, and can also include other scenarios, such as low visibility conditions, storm conditions, turbulence, and partial failure of aircraft systems. Information from different instruments can have different levels of relevance or importance in different operational scenarios. In addition, the pilot can respond to specific conditions by actuating various aircraft controls, such as steering to bypass turbulence. Summary of the invention

[0005] Aspects of embodiments of the present disclosure relate to systems and methods for analyzing flight conditions using neural networks and user interfaces for transmitting flight analysis information to pilots. For example, in some embodiments, a neural network or other statistical model receives sensor input to classify a current flight condition into one of a plurality of different flight scenarios. In some embodiments, the neural network or other statistical model also predicts or estimates the value of missing sensor input based on the detected current flight scenario. In some embodiments, the neural network or other statistical model reports its analysis results through aircraft instruments. In various embodiments, the analysis results include indications of instruments that are particularly important in the current flight scenario, semantic tags about the current scenario inferred based on sensor input, and indications of control recommendations that respond to the current flight condition.

[0006] According to one embodiment of the present disclosure, a pilot support system includes: a processing circuit; and a memory storing instructions, which, when executed by the processing circuit, cause the processing circuit to: receive input data about a current state of a vehicle; encode the input data to generate encoded input data; supply the encoded input data to a trained statistical model; calculate a current situation of the vehicle based on the input data using the trained statistical model; calculate one or more pilot feedback indications based on the current situation using the trained statistical model; and provide the one or more pilot feedback indications to a cockpit of the vehicle.

[0007] The trained statistical model may include a neural network.

[0008] Neural networks may include Restricted Boltzmann Machines.

[0009] The input data may include cockpit sensor data and cockpit control data.

[0010] The vehicle may be an aircraft, the cockpit sensor data may include: aircraft pitch angle and aircraft yaw angle, and the cockpit control data may include: flight joystick position.

[0011] The trained statistical model may be trained using training data corresponding to operations of an expert pilot, the training data comprising: training sensor data; training cockpit control data; and semantic labels associated with different scenarios.

[0012] Instructions for encoding input data may include instructions that, when executed by the processing circuit, cause the processing circuit to perform the following operations: divide the input data into multiple time windows; and generate a summary value for each time window, the summary value including: a mean value; a maximum value; and a minimum value.

[0013] The pilot feedback indications may include instruments relevant to the current context, and the instructions to provide one or more pilot feedback indications may include instructions that, when executed by the processing circuitry, cause the processing circuitry to illuminate lights associated with the instruments relevant to the current context.

[0014] The pilot feedback indication may include a recommended control actuation, and the instructions to provide one or more pilot feedback indications may include instructions that, when executed by the processing circuit, cause the processing circuit to display a recommended direction in which to actuate a cockpit control.

[0015] The pilot feedback indication may include a semantic indicia, and the instructions to provide one or more pilot feedback indications may include instructions that, when executed by the processing circuit, cause the processing circuit to display the semantic indicia on a display device in the cockpit.

[0016] According to one embodiment of the present invention, a method for providing feedback to a pilot includes: receiving input data about the current state of a vehicle operated by the pilot through a processing circuit; encoding the input data to generate encoded input data through the processing circuit; supplying the encoded input data to a trained statistical model through the processing circuit; calculating the current situation of the vehicle based on the input data through the processing circuit using the trained statistical model; calculating one or more pilot feedback indications based on the current situation through the processing circuit using the trained statistical model; and providing the one or more pilot feedback indications to a cockpit of the vehicle through the processing circuit.

[0017] The trained statistical model may include a neural network.

[0018] Neural networks can include restricted Boltzmann machines.

[0019] The input data may include cockpit sensor data and cockpit control data.

[0020] The vehicle may be an aircraft, the cockpit sensor data may include: aircraft pitch angle and aircraft yaw angle, and the cockpit control data may include: flight joystick position.

[0021] The trained statistical model may be trained using training data corresponding to operations of an expert pilot, the training data comprising: training sensor data; training cockpit control data; and semantic labels associated with different scenarios.

[0022] Encoding the input data may include: dividing the input data into a plurality of time windows; and generating a summary value for each time window, the summary value including: a mean value; a maximum value; and a minimum value.

[0023] The pilot feedback indications may include instruments relevant to the current situation, and providing one or more pilot feedback indications may include illuminating lights associated with the instruments relevant to the current situation.

[0024] The pilot feedback indication may include a recommended control actuation, and providing one or more pilot feedback indications may include displaying a recommended direction in which to actuate the cockpit control.

[0025] The pilot feedback indication may include a semantic indicia, and providing one or more pilot feedback indications may include displaying the semantic indicia on a display device in the cockpit. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The drawings, together with the specification, illustrate exemplary embodiments of the present invention, and, together with the description, serve to explain the principles of the present invention.

[0027] Figure 1 It is a schematic block diagram illustrating the function of a pilot support system according to one embodiment of the present disclosure.

[0028] Figure 2 A schematic block diagram is provided for illustrating a process for training a statistical model, and more specifically, a neural network, as a component of a pilot support system according to one embodiment of the present disclosure.

[0029] Figure 3 Flowchart of a method for training a statistical model, such as a neural network, as a component of a pilot support system according to one embodiment of the present disclosure.

[0030] Figure 4A and 4B is a schematic representation of the flow of information through a trained statistical model when the trained statistical model is used to report situational conditions during operation, such as while an aircraft is in flight.

[0031] Figure 5A and 5B Schematic illustration of information flow from vehicle sensors and cockpit sensors to a neural network for inferring current context and information flow from the neural network to cockpit indicators and controls, respectively, according to one embodiment of the present disclosure.

[0032] Figure 6 The present invention is a flowchart of a method for detecting situations and generating pilot support information using a pilot support system according to one embodiment of the present disclosure.

[0033] Figure 7 is a block diagram of a computer system that can be used in conjunction with embodiments of the present invention.

[0034] Figure 8 A depiction of some form of non-volatile storage media. DETAILED DESCRIPTION

[0035] In the following detailed description, only certain exemplary embodiments of the present invention are shown and described by means of illustrations. Those skilled in the art will recognize that the present invention can be embodied in many different forms and should not be construed as being limited to the embodiments described herein. Throughout this specification, similar reference numerals represent similar elements.

[0036] Aspects of embodiments of the present invention relate to pilot support, including systems and methods for continuously detecting flight conditions and transmitting situational context information to a pilot operating an aircraft. Although various embodiments of the present disclosure are described herein with respect to supporting a pilot in flying an aircraft, embodiments of the present disclosure are not limited thereto and may also be applied to other systems for assisting a pilot in operating a vehicle, such as other types of aircraft (e.g., gliders and helicopters), land vehicles (e.g., cars and trucks), and watercraft (e.g., boats, ships, and submarines). In addition, embodiments of the present disclosure may be suitable for the operation of other machines, such as machine tools, woodworking tools, and other equipment, where sensors are used to provide feedback to a user controlling the machine.

[0037] Various vehicles may include a variety of sensors and other instruments to provide the pilot with information about the current state of the vehicle. A land vehicle may include a speedometer indicating current land speed and a tachometer indicating current engine speed. An aircraft may include flight instruments such as an altimeter, an airspeed indicator, a vertical speed indicator, a heading indicator (e.g., a compass), a gyroscopic pitch indicator (e.g., an artificial horizon), and the like. A submarine may include instruments indicating current depth, vertical and horizontal speed, buoyancy, and the like.

[0038] Many vehicles include various levels of autonomous operation, such as the autopilot functionality of aircraft and advanced driver assistance systems (ADAS) and various levels of automation in automobiles. However, some tasks are extremely difficult to automate, but many other tasks can be handled much more effectively by automated systems than by humans. This poses a challenge in how to effectively distribute the task load between humans and machines.

[0039] When operating a vehicle, a human pilot is exposed to a large amount of information, much of which is irrelevant to the majority of situations the aircraft will experience. Through training and experience, a pilot may learn to determine the current condition of the vehicle being flown based on the output of various combinations of different instruments, which may have very different normal operating values ​​depending on the situation (e.g., airspeed during cruise versus when landing or taxiing), and what information is relevant to the current situation. However, effective presentation of relevant information can help the pilot understand the changing situation and improve response time.

[0040] Fast and intuitive communication between human pilots and auxiliary machinery improves control of aircraft and other vehicles, thereby improving the safety and performance of vehicle operations. Embodiments of the present disclosure detect flight conditions and states based on combining information from a variety of different inputs about the current state of the vehicle, the information including sensor data, current control data, and data from external sources, thereby supporting a wide range of human-machine collaborative operations. Embodiments of the present disclosure are also capable of identifying abnormal flight conditions before humans can, thereby improving aircraft safety. Therefore, aspects of embodiments of the present disclosure relate to systems and methods for improving communication between pilots, vehicles, and the automatic operation functionality of the vehicle.

[0041] Figure 1 FIG. 2 is a schematic block diagram illustrating the function of a pilot support system according to an embodiment of the present disclosure. Figure 1 As shown, a vehicle 1, such as an aircraft, may include a collection of sensors 10, a cockpit 20 for a pilot, and controls 30 in the cockpit 20. Various instruments 21 in the cockpit 20 may report information collected by the sensors 10 (e.g., an artificial horizon may provide information from a gyro sensor). In addition, various controls 30 in the cockpit 20 allow the pilot to control the vehicle, such as a throttle to control engine power (in the case of an aircraft, this may also include a joystick or flight control stick to control pitch and yaw angles and pedals for controlling the rudder).

[0042] More generally, a pilot support system 100 according to an embodiment of the present disclosure receives current information streams from a variety of different sources, such as: sensors 10 embedded in the vehicle (e.g., the aircraft's current pitch angle, yaw angle, altitude, airspeed, vertical speed, etc.); internal data sources (e.g., static data loaded from a computer system on the vehicle, such as a map); data from external sources (e.g., dynamic and / or static data received via a communication network, such as global positioning system data and weather information); cockpit control information (e.g., the current position and settings of various controls 30 such as throttle, flight stick, and rudder); and other information about the current state of the vehicle, such as semantic tags of conditions. In some embodiments, the correlation of various instruments is generated by the pilot support system (see FIG. 1 ) based on specific context detection. Figure 5B) calculated and the nominal or average values ​​of these outputs to identify instruments currently displaying abnormal values ​​(e.g., outliers), which may be different from the nominal value (e.g., statistically significantly different). In addition, in some embodiments of the present disclosure, the pilot support system may access information about the pilot's actions through cockpit control information, as well as other sensors that potentially track the pilot's activities (e.g., camera systems that determine the pilot's current physical posture, such as which direction the pilot is facing; eye tracking to determine where the pilot is looking; pressure sensors that determine what controls the pilot is touching, before or when actuating those controls).

[0043] However, many important conditions may not be readily apparent through any single sensor input. A neural network (or other statistical model 200) trained on these information streams may be used to identify these complex conditions or combinations of conditions based on multiple inputs about the current conditions to establish a "context." Once identified, the context will change which information is of most direct value to the human pilot and the autopilot system, such as weather-based contexts, such as "stormy weather" or "ice clouds," and vehicle-based contexts, such as "engine failure," "nacelle fire," or "depressurization." The system will then highlight the most directly relevant instruments (e.g., gauges) and controls.

[0044] Some aspects of embodiments of the present disclosure relate to a training phase, wherein a sensor data stream from a vehicle (e.g., an aircraft) is combined with control data and semantic tags of a situation. These data are used as input for training a statistical model (e.g., a neural network), which hierarchically clusters the data into different scenarios (e.g., using unsupervised learning). Some additional aspects of embodiments of the present disclosure relate to using a trained statistical model (e.g., a neural network) during an operational phase or a deployment phase, wherein a pilot support system outputs prompts, such as visual focus areas and semantic tags, which draw attention to situation-specific information. In some embodiments of the present disclosure, feedback of the pilot's actions to the control system allows two-way learning, because the pilot support system feeds back learning to the pilot, and the pilot also learns to cooperate with the pilot support system. Some aspects of embodiments of the present disclosure further relate to switching between operating modes of the integrated pilot support system.

[0045] In more detail, aspects of embodiments of the present invention include dividing input data into sensory, control, and semantic types, and using these same divisions to report flight conditions. Rather than monitoring individual systems, a pilot support system according to an embodiment of the present disclosure monitors a large number of sensors, determines the relative importance ("importance value") of each sensor, and directs the pilot's attention to these relevant instruments. In some embodiments, a relative importance value or importance value is determined based on whether the current output value is different from a nominal value (e.g., an average or typical value) and how much the difference is. In some embodiments, by comparing Figure 5B The relative importance is determined from the output values ​​and the average values ​​shown in (e.g., by subtracting the average or mean from the output values ​​to calculate a difference representing an importance value). The resulting difference is then correlated with the relative importance. In some embodiments, typical statistics of the outputs are also known, and the importance values ​​are calculated in units of standard deviation of the mean. This is advantageous in situations where a human pilot must quickly adapt to the context of the situation, such as when switching between autopilot and manual operating modes. In this case, embodiments of the present disclosure can reduce the time to recover from the performance degradation when the switch occurs.

[0046] In some embodiments, the content and format of the status output from the pilot support system 100 depends on the content and format of the input to the neural network, for example, if the input includes readings from a series of flight instruments, the output will report the flight condition or flight situation by directing attention to the flight instruments related to the condition. As another example, if the input includes information about the location of pilot controls 30 in the cockpit 20, the output can include a suggested action to be taken by highlighting individual controls (e.g., flashing a light on the control) and indicating the direction in which the individual control can be manipulated (e.g., flipping a switch or moving the throttle in a particular direction). As a third example, if the input includes semantic information or a tag about the presence of a particular condition (e.g., turbulence), the output can include a similar semantic tag that can be displayed to the pilot, such as on a display in the cockpit 20 (e.g., on a dashboard of a land vehicle or on a multifunction display of an instrument panel of an aircraft).

[0047] The pilot support system according to various embodiments of the present disclosure can be implemented and integrated into the vehicle in various different ways. In various embodiments of the present disclosure, the pilot support system 100 is implemented using a computer system including an electronic processing circuit. The processing circuit may include one or more processors and one or more memory modules storing instructions, which, when executed by one or more processors, cause the one or more processors to perform operations to implement the system and method of the embodiments of the present disclosure. The one or more processors may include a combination of various types of different processing circuits, such as a central processing unit (CPU), a graphics processing unit (GPU), a field programmable gate array (FPGA), a tensor processor, an artificial intelligence (AI) accelerator (e.g., a processing circuit designed to perform operations common in artificial intelligence algorithms such as forward propagation in a neural network, such as dot products, activation functions, etc.), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), etc. The computer system implementing the pilot support system can be integrated with other existing computing systems of the vehicle, such as a flight computer, an existing autopilot, an ADAS system of the vehicle, etc., or can be implemented using separate hardware that communicates with the existing computing system of the vehicle through a communication bus such as a controller area network (CAN) bus.

[0048] Aspects of embodiments of the present disclosure may operate in two phases, including a training phase during which input training data is processed and encoded into a format for use by a statistical model, wherein the statistical model is trained based on the training data. For example, where the statistical model includes a neural network, in a supervised learning process, this may include supplying the encoded input data to the neural network in a forward propagation or inference mode, followed by comparing the network output calculated based on a set of parameters with the expected output (e.g., based on labeled training data), and updating the parameters to reduce the error between the calculated output and the expected output (e.g., using a backpropagation algorithm and gradient descent), or automatically clustering the training data into a plurality of different clusters using an unsupervised learning process. Details of an example method of training a statistical model according to embodiments of the present disclosure are presented in more detail below.

[0049] During operation, given current inputs to the statistical model 200 , the trained statistical model 200 identifies the current operating context, and the pilot support system 100 reports the current operating context to the pilot via one or more interfaces customized to the category of input.

[0050] Figure 2 A schematic block diagram is provided for illustrating a process for training a statistical model, and more specifically, a neural network, as a component of a pilot support system, according to one embodiment of the present disclosure. Figure 3 Flowchart of a method for training a statistical model, such as a neural network, as a component of a pilot support system according to one embodiment of the present disclosure.

[0051] like Figure 2 As shown, the input training data 210 of the statistical model 200 is classified into three categories: cockpit sensors 211, cockpit controls 213, and semantic tags 215. During the training phase, the complete vector of data corresponding to the time point is used as input for generating a hierarchical situational representation using the statistical model 200. In some embodiments of the present disclosure, the complete vector of these input training data 210 is obtained from the flight records of expert pilots. During the recording of this training data, the values ​​of the cockpit sensors 211 (such as the current pitch angle, yaw angle and altitude of the aircraft, load, position of flaps and slats, cabin pressure, temperature, humidity, brake torque sensor, health and usage monitoring system (HUMS), angle of attack (AOA), air speed, engine speed (RPM) and various status indicator lights) are continuously sampled and saved together with the state of the controls 213 manipulated by the expert pilot (such as the current position of the throttle, rudder, flight control stick, brake, flap position, navigation and autopilot settings, and switches such as de-icing, lights and power) and semantic tags 215 of the flight conditions. These semantic tags 215 may be generated by an expert pilot during recording, or as a post-processing step when reviewing the data. In some embodiments, the data is captured or sampled at a rate of 10 samples / second (or 10 Hz), but embodiments of the present disclosure are not limited thereto, and higher sample rates or lower sample rates may be used. In some embodiments of the present disclosure, the sampling rate is high enough to capture substantially real-time information (e.g., no more than a few seconds between samples).

[0052] According to some embodiments of the present disclosure, the statistical model 200 includes a neural network to establish a hierarchical representation of the current vehicle context. The lowest level 220 of the hierarchy of representation may include common clusters of sensor inputs, such as steering modes or co-existing sensor values, while the highest level 240 of the hierarchy may cover broad categories of conditions, such as spoilers or mechanical failures. Examples of neural network architectures suitable for hierarchical clustering include: deep restricted Boltzmann machines (e.g., see Salakhutdinov, R. and Hinton, G. (April 2009), Deep Boltzmann Machines, Artificial Intelligence and Statistics (pp. 448-455); deep autoencoders (e.g., see Hinton, G.E. and Salakhutdinov, R.R. (2006), Dimensionality Reduction of Data Using Neural Networks, Science, 313(5786), 504-507) and / or generative adversarial networks (e.g., see See, for example, Radford, A., Metz, L., and Chintala, S. (2015), Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks, arXiv preprint arXiv:1511.06434). Each cluster identified by training the statistical model may represent a different operational scenario for the vehicle. Some of these operational scenarios may require a response from the pilot. For example, a turbulence condition may require the pilot to steer the vehicle away from the turbulence zone. As another example, a traffic light condition may require the driver to slow down the car when approaching an intersection.

[0053] refer to Figure 3 , training data from recorded sensor data and cockpit control data captured when an expert pilot is operating a vehicle is supplied as an input to the training process. As described above, data may be captured at a sampling rate of, for example, 10 Hz, where data from different sources are time aligned, and where each data source corresponds to one or more time series data streams. Figure 3The operations shown in the figure may be performed by a training system such as a computer system, which may include an electronic processing circuit. The processing circuit may include one or more processors and one or more memory modules storing instructions, which, when executed by the one or more processors, cause the one or more processors to perform operations to implement the system and method of the embodiments of the present disclosure. The one or more processors may include a combination of various types of different processing circuits, such as a central processing unit (CPU), a graphics processing unit (GPU), a field programmable gate array (FPGA), a tensor processor, an artificial intelligence (AI) accelerator (e.g., a processing circuit designed to perform operations common in artificial intelligence algorithms such as forward propagation in a neural network, such as dot products, activation functions, etc.), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), etc. In general, the process for training a statistical model is more computationally intensive than operating a statistical model (e.g., using a statistical model to perform inference or prediction). Therefore, a statistical model according to an embodiment of the present invention can be trained "offline", for example, using hardware different from that used to execute the model as a component of the pilot support system 100. For example, the training system may include one or more computer servers and may operate in a cloud environment (e.g., in a data center).

[0054] exist Figure 3 In the embodiment shown in FIG. 2 , the training data 210 includes training sensor data 211, training cockpit control data 213, and training semantic tags 215. Figure 3 As shown, in operation 310, training sensor data 211 is encoded into a form suitable for use with statistical model 200, in operation 320, training control data 213 is encoded into a form suitable for use with statistical model 200, and in operation 330, training semantic tags 215 are encoded into a form for use with statistical model 200. For example, in some embodiments of the present disclosure, continuous values ​​such as speed, altitude, control position, etc. can be encoded into floating point values ​​within a specific range (e.g., within a continuous range of 0 to 1 or within a range of 0 to 255). In some embodiments of the present disclosure, values ​​can be converted between different representation spaces (e.g., non-linear values ​​can be converted to linear spaces, such as by applying a square root function to values ​​in a quadratic space). In some embodiments of the present disclosure, discrete values ​​are encoded using discrete values. For example, semantic tags 215 can be encoded using binary values ​​such as 0 and 1, indicating the absence and presence of these tags, respectively. As another example, semantic tags that decrease along the progression of degree can be labeled using sequential values ​​(e.g., the level of a storm indicated by a weather report).

[0055] In addition, in some embodiments of the present disclosure, the data streams are grouped into time windows, wherein each time window may include multiple samples for each data stream (e.g., for each sensor input and for each cockpit control input). Therefore, the encoded data may include an aggregate or summary value for each data stream calculated based on the value in a given time window. These aggregate or summary values ​​may include: mean, maximum, minimum, variance or standard deviation, norm, etc. For example, sensor data may be collected from each sensor at a sampling rate of 10 Hz and grouped into a time window of 1 second so that each time window includes 10 samples. The summary values ​​described above are then calculated using the 10 samples collected from a given sensor in a 1 second time window to generate a vector of input features of the sensor for a given time window. More specifically, the input sensor data may include pitch angle, yaw angle, and altitude. For a given time window, there may be 10 pitch angle values, 10 yaw angle values, and 10 altitude values. A summary value may be calculated for each of these sensors. For example, for a yaw angle sensor, the vector of the time window may include: mean yaw angle; maximum yaw angle; minimum yaw angle; standard deviation of yaw angle; norm of yaw angle, etc. Similarly, for a pitch angle sensor, the vector of the same time window may include: mean pitch angle; maximum pitch angle; minimum pitch angle; standard deviation of pitch angle; norm of pitch angle, etc.

[0056] As described above, in some embodiments of the present disclosure, the statistical model 200 includes a restricted Boltzmann machine, which is a statistical model capable of hierarchical clustering and bidirectional information flow for use by the pilot support system 100 according to embodiments of the present disclosure. In its simplest form, the restricted Boltzmann machine includes two layers: a visible layer and a hidden layer, represented as vectors v and h, respectively. However, embodiments of the present disclosure are not limited in this regard and may include deep neural networks (e.g., restricted Boltzmann machines having more than two layers). Therefore, in such embodiments, the encoding of the training sensor data 211, the training control data 213, and the training semantic tags 215 performed in operations 310, 320, and 330 encodes these data into respective training vectors v (e.g., v sensor 、v controls and v labels In operation 340, the training system merges the data from different sources to generate a merged training vector (v training ), where data from different sources are aligned according to their timestamps (e.g., where the “data point” of each training vector corresponds to the value of a sensor, control, and semantic tag at a specific point in time or during a specific time window).

[0057] The weights connecting the visible layer and the hidden layer are represented as a matrix W. (In embodiments with a deep neural network with more than two layers, e.g. Figure 2In the embodiment shown in , there is a separate matrix W representing the connection between each pair of layers. ) The energy level of the network state is defined as shown in equation (1):

[0058] E(v, h; W) = -v T Wh (1)

[0059] In some embodiments of the present disclosure, individual units in v and h are probabilistic binary units with values ​​1 or 0, where the input data of the Boltzmann network can be set in the vector v of the visible layer. In the absence of external input, the model can assign probabilities to the vector v according to equations (2) and (3):

[0060]

[0061]

[0062] where p*(v;W) is the denormalized probability and Z(W) is the partition function. j ) or visible unit i (e.g., the i-th visible unit v i ) is expressed by equations (4) and (5):

[0063]

[0064]

[0065] Where σ(x) = 1 / (1 + exp(x)), D is the length of v, and P is the length of h. The learning rule of the weight matrix W is represented as shown in equations (6), (7) and (8):

[0066]

[0067] P data (h,v;W)=P(h|v;W)P data (v) (7)

[0068]

[0069] where α is the learning rate. The learning rule (e.g., as shown in Equations (6), (7), and (8)) expresses the data-dependent expectation and model expectations According to equation (2), P data is the data-dependent probability distribution of network activity when v remains at a constant input value in the dataset, and P modelis the probability distribution of the default network activity. In operation 350, the training system uses the merged training vector to train the model to learn the operating context of the vehicle. Training on the complete data set includes applying the learning rule to each data point in the training set (e.g., each training vector v at each time point or time window). training , where the data points may be supplied in a random order), and a determination is made in operation 360 whether the model has converged. If the model has not yet converged, training continues in operation 350, for example by continuing to apply the learning rule to update the weights W of the network until convergence. Convergence is reached when the energy level E(v,h;W) of the network (see equation (1)) no longer changes significantly when the learning rule is applied to the data points in the data set. Figure 3 As shown, the result of the training process is a trained statistical model 200 .

[0070] As a specific example of a training process for a statistical model according to an embodiment of the present disclosure, data is collected from a flight simulator software system, wherein data streams from cockpit sensors and semantic tags are used to train a statistical model 200 according to an embodiment of the present disclosure to detect clear weather versus stormy weather.

[0071] In order to collect training data, the weather customization feature of the flight simulator software system is used to generate two flight scenarios of clear weather and stormy weather. For each of these two scenarios, about 30 minutes of data are recorded from the simulated sensor when flying at a fixed altitude. For the purpose of this example, the pitch angle and roll angle values ​​of the data are collected from the simulator in real time at about 10Hz. The data is then divided into blocks of 1 second time windows. Subsequently, the data of each block is pre-processed by calculating the number of changes in positive and negative signs, the number of times the mean is traversed, the mean, standard deviation, norm, maximum value and minimum value, so that 7 summary values ​​are calculated for each original input (roll angle and pitch angle), thereby generating 14 data inputs. These values ​​are saved for the situational awareness of the training neural network 200. In addition, two empty semantic inputs are included during training, one of which is set to 1 (clear situation) during the clear situation, and the other is set to 1 (storm situation) during the storm situation. Combined with the semantic input, v has a size of 16 and h is set to a size of 2. The neural network 200 trains 1,000 epochs on the entire data set.

[0072] Operational Phase

[0073] Information Flow

[0074] As described above, during the training phase, information flows forward from the input sensors to the neural network for context representation. On the other hand, during the operation phase, information first flows forward through the trained statistical model 200 to identify the current context of the operation (e.g., cruise, spoiler, landing, taxiing, etc.). The information then flows in reverse, providing a reconstruction of the input data that was not initially provided. The reconstructed input data further flows in reverse to an interface designed to intuitively convey information to the pilot.

[0075] Figure 4A and 4B is a schematic representation of the flow of information through the trained statistical model 200 when the trained statistical model 200 is used to report detected situations during operation, such as while an aircraft is in flight. Figure 5A and 5B Schematic illustration of information flow from vehicle sensors and cockpit sensors to a neural network for inferring current context and information flow from the neural network to cockpit indicators and controls, respectively, according to one embodiment of the present disclosure. In 200, shaded values ​​indicate nodes with high values, thereby forming an input context indicator. The shaded values ​​in 210 indicate input sensor values ​​initially provided during the first stage of the information flow, while the shaded values ​​in 510 indicate all values ​​available to the pilot during the second stage of the information flow. Elements 211, 213, and 215 represent raw input value types, while elements 511, 513, and 515 form a physical reporting interface through which information from 510 is reported to the pilot.

[0076] During flight, situations are identified by the neural network and reported to the pilot. Figure 4A and 5A As shown in , in order to recognize the situation, an input set representing the current state of the aircraft is input to the neural network 200. Figure 4A As shown, to detect a situation, the input v is held at a fixed data point (as indicated by the bar in the input node v) representing the state of the vehicle (e.g., aircraft). The set of sensor data may be partial and have missing inputs. For example, Figure 5A As shown, the sensor input may contain only shadow sensors (e.g., sensor 1 and sensor N), and all other inputs may be empty or non-existent. Information flows from the input to the output h, such that the activation of each cell in h represents the detection strength of the context. Thus, the statistical model 200 matches the input to the closest known context level.

[0077] When reporting detected or inferred contexts, information from the neural network 200 flows in reverse, reconstructing plausible values ​​for any missing inputs. Figure 4BAs shown, to reconstruct the sensor information based on the current context, h is maintained at its new value and information flows back to v. This reconstructs the input most relevant to the active context and can be used to report the context back to the cockpit. As a more specific example, Figure 5B As shown, Figure 5A Data values ​​missing from the scenario (e.g., sensor 2, control 1 to control N, and tag 1 to tag N) are reconstructed from the scenario. Using the reconstructed inputs allows the known current input values ​​from the existing sensors to correct errors (e.g., detect a malfunctioning sensor that reports information different from the predicted value), filter noise, and fill in information gaps that exist in the original data (e.g., due to a malfunctioning sensor or the absence of a particular sensor in a particular vehicle). Once the reconstructed values ​​are obtained for a particular scenario, the reconstructed mean of all scenario detections can be subtracted from it, providing a measure of the relevance of each data value. These relevance scores can be used to prioritize the information that is prominently presented to the pilot.

[0078] Figure 6 FIG. 6 is a flow chart of a method for detecting situations and generating pilot support information using a pilot support system according to an embodiment of the present disclosure. Embodiments of the present disclosure may be used in an operation phase during real-world or simulated operations (e.g., real-world flight or flight in a flight simulator for training). In operation 610, the pilot support system 100 loads a trained statistical model (e.g., based on the trained statistical model). Figure 3 The pilot support system 100 enters a loop where, in operation 620, the pilot support system 100 determines whether the vehicle is currently in operation (e.g., whether the aircraft is in flight). If not, the process ends and the pilot support system 100 terminates operation.

[0079] If the vehicle is currently operating, then data is read from the sensors in operation 630. The data from the vehicle sensors is collected in the same format used to train the context detection model (e.g., similar to the format described above with respect to Figure 3 6). In operation 640, the pilot support system 100 detects a context by supplying the encoded input to the trained statistical model 200. The model generates controls, sensor values, and labels related to the context it detects, and accordingly subsequently reports the context to the human pilot by recommending control actions to handle the context in operation 650, visually highlighting sensors related to the context in operation 660, and displaying semantic labels for better human understanding in operation 670. The pilot support system 100 proceeds with operation 620 to continue the loop during the remaining operating period (e.g., during flight).

[0080] Reporting interface

[0081] As described above, in various embodiments of the present disclosure, information generated by the pilot support system 100 is communicated back to the pilot based on the initial category of the input data, such as recommended control actions, visual prominence of sensors relevant to the situation, and semantic labeling. For cockpit sensor-based input, information is communicated visually or audibly to draw attention to the cockpit sensor aspect. For example, an indicator light placed adjacent to or in a line with a particular instrument may be illuminated or set to flash to draw the pilot's attention to the instrument. In some embodiments where a digital display panel (e.g., a computer display panel) is used to display instrument data, the pilot assistance system 100 may highlight specific controls by increasing the prominence of these instruments in the display, such as by reducing the brightness of other instruments, increasing the brightness of related instruments, increasing the relative physical size of related instruments on the display panel, or generating a customized dashboard that displays the sensor data that is currently most relevant to the current situation.

[0082] For information based on controls 30, the suggested manipulation associated with the situation may be communicated to the pilot by moving arrows. For example, various controls 30 may include indicator arrows that light up or flash in the direction suggested by the pilot support system 100. As another example, a digital display in the cockpit may display a visual representation of the control to be actuated and show an animation of the recommended actuation direction of the control. As an additional example, tactile feedback such as vibration of the control or other physical actuation of the control may be provided to the pilot to indicate that the pilot support system 100 suggests that the control be moved in a particular direction.

[0083] In some embodiments, for information based on semantic tags associated with the context, text-based prompts are displayed on screens within panels in the cockpit. In some embodiments, text-based prompts are superimposed as part of a heads-up display for the pilot. In some embodiments, symbols are used to represent various contexts instead of using text-based semantic tags.

[0084] Furthermore, in other embodiments of the present disclosure, additional information may be provided to the pilot via other user interface elements within cockpit 20, such as auditory cues for particular controls, as well as other types of controlled illuminated cues, such as spotlights to highlight particular controls.

[0085] Continuing with the specific example introduced above, the trained statistical model or neural network 200 is used during the operational phase, where real-time sensor data from the simulation is supplied to the neural network 200 through the pilot support system 100. In response, when the neural network determines that the sensed data is from a storm condition, the pilot support system 100 reports the spoiler information to the pilot in real-time by displaying the word "spoiler" in text on the screen.

[0086] The simulator or software on the aircraft sends information about the controls and the aircraft position data to the pilot support system 100, which prepares the data by encoding the input for the neural network 200, such as by calculating the mean, maximum, minimum, and norm of the data within a given time window (e.g., the past 1 second of data). In some embodiments of the present disclosure, the input to the pilot support system 100 is independent of any actual actions taken by the pilot, and the report is based only on the current state of the sensor measurements and / or control position measurements.

[0087] The neural network classifies the current state of the vehicle from the available information to detect the relevant vehicle scenarios that the neural network has been trained on. In this example, the neural network was trained on stormy weather versus clear weather, but this approach is generally sufficient to function in a substantially similar manner for other scenarios, and can occur simultaneously in superimposed scenarios (e.g., engine failure and turbulence are separate scenarios).

[0088] Thus, example embodiments of the present invention are able to report flight situations to pilots in substantially real time.During storm situations, the neural network correctly detected the situation with over 90% accuracy and reported the detected situation by displaying semantic data on the screen via text.

[0089] Thus, aspects of embodiments of the present disclosure relate to systems and methods for providing pilot support by automatically taking into account data from multiple different vehicle sensors and multiple different cockpit sensors regarding the current state of controls in the cockpit. In more detail, embodiments of the present disclosure use training data captured during the operation of an aircraft by an expert pilot (either in simulation or in real flight) to train a statistical model, such as a neural network, to detect scenarios and organize the scenarios into hierarchical clusters. The various clusters represent different types of environments experienced by the vehicle during operation, and different information therein may have specific relevance to the pilot during these time periods. Thus, relevant instrumentation and suggested control actuations may be automatically reported to the pilot, thereby improving pilot performance.

[0090] Computer Systems

[0091] Figure 7An exemplary computer system 1200 according to an embodiment is shown in . For example, a computer system similar to those described herein can be used to implement the model training system described above and / or the training models in pre-deployment and deployment, but embodiments of the present invention are not limited thereto. The exemplary computer system 1200 is configured to perform calculations, processes, operations, and / or functions associated with a program or algorithm. In one embodiment, certain processes and steps discussed herein are implemented as a series of instructions (e.g., software programs) that reside in a computer-readable memory unit and are executed by one or more processors of the exemplary computer system 1200. When executed, the instructions cause the exemplary computer system 1200 to perform specific actions and exhibit specific characteristics, such as described herein.

[0092] The exemplary computer system 1200 may include an address / data bus 1210 configured to transmit information. In addition, one or more data processing units, such as a processor 1220, are coupled to the address / data bus 1210. The processor 1220 is configured to process information and instructions. In an embodiment, the processor 1220 is a microprocessor. Alternatively, the processor 1220 may be a different type of processor, such as a parallel processor or a field programmable gate array.

[0093] The exemplary computer system 1200 is configured to utilize one or more data storage units. The exemplary computer system 1200 may include a volatile memory unit 1230 (e.g., random access memory (“RAM”), static RAM, dynamic RAM, etc.) coupled to the address / data bus 1210, wherein the volatile memory unit 1230 is configured to store information and instructions for the processor 1220. The exemplary computer system 1200 may also include a non-volatile memory unit 1240 (e.g., read-only memory (“ROM”), programmable ROM (“PROM”), erasable programmable ROM (“EPROM”), electrically erasable programmable ROM (“EEPROM”), flash memory, etc.) coupled to the address / data bus 1210, wherein the non-volatile memory unit 1240 is configured to store static information and instructions for the processor 1220. Alternatively, the exemplary computer system 1200 may execute instructions retrieved from an online data storage unit, such as in “cloud” computing. In an embodiment, the exemplary computer system 1200 may also include one or more interfaces, such as interface 1250, coupled to the address / data bus 1210. The one or more interfaces are configured to enable the exemplary computer system 1200 to interface with other electronic devices and computer systems. The communication interface implemented by the one or more interfaces may include wired (e.g., serial cable, modem, network adapter, etc.) and / or wireless (e.g., wireless modem, wireless network adapter, etc.) communication technology.

[0094] In one embodiment, the exemplary computer system 1200 may include an input device 1260 coupled to the address / data bus 1210, wherein the input device 1260 is configured to communicate information and command selections to the processor 1220. According to one embodiment, the input device 1260 is an alphanumeric input device, such as a keyboard, which may include alphanumeric and / or function keys. Alternatively, the input device 1260 may be an input device other than an alphanumeric input device. In an embodiment, the exemplary computer system 1200 may include a cursor control device 1270 coupled to the address / data bus 1210, wherein the cursor control device 1270 is configured to communicate user input information and / or command selections to the processor 1220. In an embodiment, the cursor control device 1270 is implemented using a device such as a mouse, a trackball, a trackpad, an optical tracking device, or a touch screen. Despite the above description, in an embodiment, the cursor control device 1270 is directed and / or activated via input from the input device 1260, such as in response to the use of special keys and key sequence commands associated with the input device 1260. In an alternative embodiment, cursor control device 1270 is configured to be directed or indicated by voice commands.

[0095] In an embodiment, the exemplary computer system 1200 may also include one or more optional computer usable data storage devices, such as a storage device 1280, coupled to the address / data bus 1210. The storage device 1280 is configured to store information and / or computer executable instructions. In an embodiment, as Figure 8 As shown in , storage device 1280 is a storage device such as a magnetic or optical disk drive (e.g., a hard disk drive ("HDD"), a floppy disk 1282, a compact disk read-only memory ("CD-ROM") 1284, a digital versatile disk ("DVD")), or a flash memory (e.g., NAND flash in the form of a USB drive) 1286. According to one embodiment, a display device 1290 is coupled to address / data bus 1210, wherein display device 1290 is configured to display video and / or graphics. In embodiments, display device 1290 may include a cathode ray tube ("CRT"), a liquid crystal display ("LCD"), a field emission display ("FED"), a plasma display, or any other display device suitable for displaying video and / or graphic images and user-recognizable alphanumeric characters.

[0096] An exemplary computer system 1200 is presented herein as an exemplary computing environment according to an embodiment. However, the exemplary computer system 1200 is not strictly limited to a computer system. For example, the embodiment specifies that the exemplary computer system 1200 represents a class of data processing analysis that can be used according to the various embodiments described herein. In addition, other computing systems can also be implemented. In fact, the spirit and scope of the technology of the present invention are not limited to any single data processing environment. Therefore, in an embodiment, computer executable instructions such as program modules executed by a computer are used to control or implement one or more operations of various embodiments of the technology of the present invention. In an exemplary embodiment, such program modules include routines, programs, objects, components and / or data structures configured to perform specific tasks or implement specific abstract data types. In addition, the embodiment specifies that one or more aspects of the technology of the present invention are implemented by utilizing one or more distributed computing environments, such as in the distributed computing environment, tasks are performed by remote processing devices linked by a communication network, or, for example, in the distributed computing environment, various program modules are located in local and remote computer storage media containing memory storage devices.

[0097] While the invention has been described with respect to certain exemplary embodiments, it should be understood that the invention is not limited to the disclosed embodiments, but on the contrary is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims and their equivalents.

Claims

1. A pilot support system, characterized in that: include: Processing circuit; as well as a memory storing instructions that, when executed by the processing circuit, cause the processing circuit to: receiving input data regarding the current state of the vehicle; Encoding the input data to produce encoded input data including a plurality of vectors of summary values ​​calculated from the input data, comprising: grouping the input data into time windows, each time window comprising a plurality of data samples of the input data; and calculating the plurality of vectors of summary values, each vector of the plurality of vectors of summary values ​​being calculated based on the plurality of data samples grouped into a corresponding one of the time windows; supplying the plurality of vectors of summary values ​​of the encoded input data to a trained statistical model, the trained statistical model configured to cluster data corresponding to a current operating state of the vehicle into different contexts of the vehicle; calculating a current context of the vehicle based on the plurality of vectors of aggregated values ​​of the encoded input data using the trained statistical model; calculating one or more pilot feedback indicators based on the current context using the trained statistical model; and The one or more pilot feedback indications are provided to a cockpit of the vehicle.

2. The pilot support system according to claim 1, characterized in that: The trained statistical model comprises a neural network.

3. The pilot support system according to claim 2, characterized in that: The neural network comprises a restricted Boltzmann machine.

4. The pilot support system according to claim 1, 2 or 3, characterized in that: The input data includes cockpit sensor data and cockpit control data.

5. The pilot support system according to claim 4, characterized in that: The vehicle is an aircraft, The cockpit sensor data includes: aircraft pitch angle and aircraft yaw angle, and The cockpit control data includes: flight control stick position.

6. The pilot support system according to claim 1, characterized in that: The trained statistical model is trained using training data corresponding to the pilot's operation of the vehicle, the training data comprising: Training sensor data; Training cockpit controls data; and Semantic tags associated with different contexts.

7. The pilot support system according to claim 1, characterized in that: The vector of summary values ​​includes: Mean; Maximum value; and Minimum value.

8. The pilot support system according to claim 1, characterized in that: The pilot feedback indication includes instruments relevant to the current situation, and Wherein the instructions for providing the one or more pilot feedback indications include instructions that, when executed by the processing circuit, cause the processing circuit to illuminate a light associated with the instrument relevant to the current situation.

9. The pilot support system according to claim 1 or 8, characterized in that: The pilot feedback indication includes a recommended control actuation, and Wherein the instructions for providing the one or more pilot feedback indications include instructions that, when executed by the processing circuit, cause the processing circuit to display a recommended direction in which to actuate a cockpit control.

10. The pilot support system according to claim 1, characterized in that: The pilot feedback indication includes semantic labeling, and Wherein the instructions for providing the one or more pilot feedback indications include instructions that, when executed by the processing circuit, cause the processing circuit to display the semantic indicia on a display device in the cockpit.

11. A method for providing feedback to a pilot, characterized in that: The method comprises: receiving, via processing circuitry, input data regarding a current state of a vehicle being operated by the pilot; Encoding, by the processing circuit, the input data to generate encoded input data including a plurality of vectors of summary values ​​calculated from the input data, generating the encoded input data comprises: grouping the input data into time windows, each time window comprising a plurality of data samples of the input data; and calculating the plurality of vectors of summary values, each vector of the plurality of vectors of summary values ​​being calculated based on the plurality of data samples grouped into a corresponding one of the time windows; supplying, by the processing circuitry, the plurality of vectors of summary values ​​of the encoded input data to a trained statistical model, the trained statistical model configured to cluster data corresponding to a current operating state of the vehicle into different contexts of the vehicle; calculating, by the processing circuit, a current context of the vehicle based on the plurality of vectors of summary values ​​of the encoded input data using the trained statistical model; calculating, by the processing circuitry, one or more pilot feedback indicators based on the current context using the trained statistical model; and The one or more pilot feedback indications are provided to a cockpit of the vehicle by the processing circuit.

12. The method according to claim 11, characterized in that: The trained statistical model comprises a neural network.

13. The method according to claim 12, characterized in that: The neural network comprises a restricted Boltzmann machine.

14. The method according to claim 11, 12 or 13, characterized in that: The input data includes cockpit sensor data and cockpit control data.

15. The method according to claim 14, characterized in that: The vehicle is an aircraft, The cockpit sensor data includes: aircraft pitch angle and aircraft yaw angle, and The cockpit control data includes: flight control stick position.

16. The method according to claim 11, characterized in that: The trained statistical model is trained using training data corresponding to the pilot's operation of the vehicle, the training data comprising: Training sensor data; Training cockpit controls data; and Semantic tags associated with different contexts.

17. The method according to claim 11, characterized in that: The vector of summary values ​​includes: Mean; Maximum value; and Minimum value.

18. The method according to claim 11, characterized in that: The pilot feedback indication includes instruments relevant to the current situation, and Wherein providing the one or more pilot feedback indications includes illuminating a light associated with the instrument relevant to the current situation.

19. The method according to claim 11 or 18, characterized in that: The pilot feedback indication includes a recommended control actuation, and Wherein providing the one or more pilot feedback indications comprises displaying a recommended direction in which to actuate a cockpit control.

20. The method according to claim 11, characterized in that: The pilot feedback indication includes semantic labeling, and Wherein providing the one or more pilot feedback indications comprises displaying the semantic indicia on a display device in the cockpit.

Images