Pilot Cognitive Reasoning System and Methods

Abstract

Pilot Cognitive Reasoning System and Method. A system for determining a pilot's cognitive state includes one or more processors coupled to a memory, the processors being configured to receive physiological data and aircraft status data; determine the pilot's mental workload and mental fatigue based on the physiological data; determine the pilot's available attentional resources based on the mental workload and mental fatigue; determine the pilot's attention allocation based on the available attentional resources and a gaze pattern derived from the physiological data; determine the pilot's situational awareness based on the attention allocation and the aircraft status data; and generate visualizations of the pilot's available attentional resources, the pilot's attention allocation, the pilot's situational awareness, and the aircraft status data.

Description

Technical Field

[0001] This disclosure generally relates to cognitive reasoning systems and methods for pilots. Background Technology

[0002] Pilots may experience various cognitive challenges during flight, including pattern confusion and spatial disorientation, which can lead to serious errors in aircraft operation. Pattern confusion occurs when a pilot's mental model of the aircraft's automated state differs from reality, often due to the increasing complexity of cockpit automation. This can result in inappropriate control inputs or an inability to respond to changing flight conditions. Similarly, spatial disorientation can occur when a pilot's perception of the aircraft's position, attitude, or motion differs from reality. These cognitive discrepancies can stem from a variety of factors, including mental fatigue, high workload, inattention, and the limitations of human information processing in dynamic, multi-tasking environments.

[0003] Some previous approaches to addressing these problems relied on standardized training and procedural measures without incorporating real-time physiological data or personalized cognitive assessments. These types of approaches do not take into account the unique cognitive and physiological characteristics of each pilot. This lack of personalized data limits the ability to accurately detect and address cognitive challenges, as they arise during flight operations.

[0004] This potential for cognitive dissonance poses a significant risk to flight safety, particularly in situations requiring rapid decision-making or when pilots are operating with reduced crew complements. Developing effective and implementable interventions to mitigate these risks is challenging given the inability to account for individual differences among pilots. Summary of the Invention

[0005] According to one embodiment of this disclosure, a system for determining a pilot's cognitive state includes one or more processors coupled to a memory, the one or more processors being configured to receive physiological data and aircraft status data, determine the pilot's mental workload and mental fatigue based on the physiological data, determine the pilot's available attention resources based on the mental workload and mental fatigue, determine the pilot's attention allocation based on the available attention resources and a gaze pattern derived from the physiological data, determine the pilot's situational awareness based on the pilot's attention allocation and the aircraft status data, and generate visualizations of the pilot's available attention resources, the pilot's attention allocation, the pilot's situational awareness, and the aircraft status data.

[0006] According to another embodiment of this disclosure, a method for determining a pilot's cognitive state includes receiving physiological data from a pilot using multiple sensors. The method also includes receiving aircraft state data from an aircraft state data generator. The method further includes determining the pilot's mental workload and mental fatigue based on the physiological data. The method also includes determining the pilot's available attentional resources based on mental workload and mental fatigue. The method further includes determining the pilot's attention allocation based on available attentional resources and gaze patterns derived from the physiological data. The method further includes determining the pilot's situational awareness based on attention allocation and aircraft state data. The method also includes generating visualizations of the pilot's available attentional resources, the pilot's attention allocation, the pilot's situational awareness, and the aircraft state data.

[0007] On the other hand, determining the mental workload and mental fatigue includes: using multiple Kalman filters (112, 406) to track different hypotheses about the pilot's cognitive state to generate one or more outputs; combining the one or more outputs of these Kalman filters using a Gaussian mixture model, i.e., GMM (112, 310, 408, 410, 414), wherein the output of each Kalman filter is represented as a Gaussian component; and dynamically adjusting one or more weights of the Gaussian components based on the physiological data and aircraft state data.

[0008] On the other hand, it also includes using a probabilistic graphical modeling algorithm, namely the PGM algorithm (112, 310, 414), to track the pilot's current knowledge of flight variables.

[0009] On the other hand, the PGM algorithm described therein involves modeling the decisions of the pilot's visual inspection instruments as events in a fully observable recursive Markov chain.

[0010] On the other hand, generating the visualization includes generating a real-time graphical representation of the pilot's available attention resources, the pilot's attention allocation, the pilot's situational awareness, and relevant aircraft parameters associated with the aircraft state data.

[0011] In another aspect, determining the situational awareness includes comparing the pilot's attention allocation with critical flight information.

[0012] A non-transitory computer-readable medium storing instructions, which, when executed by one or more processors (108, 920), cause the one or more processors to: determine (128, 208, 308, 606) a pilot's mental workload and mental fatigue based on physiological data; determine (128, 208, 308, 608) the pilot's available attention resources based on the mental workload and the mental fatigue; determine (128, 208, 308, 610) the pilot's attention allocation based on the available attention resources and a gaze pattern derived from the physiological data; determine (128, 208, 308, 612) the pilot's situational awareness based on the attention allocation and flight data; and generate (132, 614) visualizations of the pilot's available attention resources, the pilot's attention allocation, the pilot's situational awareness, and aircraft status data.

[0013] In another aspect, the one or more processors are configured to: estimate the pilot's mental workload and mental fatigue using at least one Kalman filter (112, 406); combine the outputs of the at least one Kalman filter using a Gaussian mixture model, i.e., GMM (112, 310, 408, 410, 414), wherein the outputs represent weighted Gaussian components; and adjust the weights of the Gaussian components in real time based on the physiological data and aircraft status data.

[0014] In another aspect, the one or more processors are configured to implement a probabilistic graphical modeling algorithm, namely PGM (112, 310, 414), to track the pilot’s current knowledge of flight variables.

[0015] In another embodiment, the one or more processors are configured to provide a real-time graphical representation of the pilot's available attention resources, the pilot's attention allocation, the pilot's situational awareness, and relevant aircraft parameters associated with the aircraft state data.

[0016] In another aspect, the one or more processors are configured to determine the situational awareness, which includes comparing the pilot’s attention allocation with critical flight information.

[0017] According to another implementation of this disclosure, a non-transitory computer-readable medium stores instructions that, when executed by one or more processors, cause the one or more processors to determine a pilot's mental workload and mental fatigue based on physiological data. The instructions also cause the one or more processors to determine the pilot's available attentional resources based on mental workload and mental fatigue. Furthermore, the instructions cause the one or more processors to determine the pilot's attention allocation based on available attentional resources and gaze patterns derived from the physiological data. The instructions further cause the one or more processors to determine the pilot's situational awareness based on attention allocation and flight data. Finally, the instructions cause the one or more processors to generate visualizations of the pilot's available attentional resources, the pilot's attention allocation, the pilot's situational awareness, and aircraft status data.

[0018] The features, functions and advantages described herein may be implemented independently in various embodiments or in combination in other embodiments, further details of which can be found in the following description and figures. Attached Figure Description

[0019] Figure 1 This is a diagram illustrating the cognitive reasoning system of a pilot.

[0020] Figure 2 This illustrates a specific implementation of a pilot's cognitive reasoning system.

[0021] Figure 3 This illustrates a specific implementation of an estimation algorithm using a multimodal cognitive state estimation framework.

[0022] Figure 4 This is a diagram illustrating a specific implementation of the estimation algorithm using a Gaussian mixture model (GMM).

[0023] Figure 5 This diagram illustrates a specific implementation of a pilot cognitive reasoning system that focuses on processing eye-tracking data and integrating it with aircraft status data.

[0024] Figure 6 This is a flowchart illustrating how to use the pilot monitoring system.

[0025] Figure 7 This is a flowchart illustrating an example of the lifecycle of an aircraft, including a pilot monitoring system.

[0026] Figure 8 This is a block diagram illustrating various aspects of an illustrative aircraft, including a pilot monitoring system.

[0027] Figure 9 This is a diagram of the electronic components of the pilot monitoring system. Detailed Implementation

[0028] This article discloses systems, devices, and methods for monitoring pilots to help prevent accidents caused by cockpit confusion or misunderstanding. The system uses various sensors to measure things such as the pilot's eye movements, heart rate, and skin reactions. It also collects data on the aircraft's current status, such as its altitude, speed, and orientation.

[0029] All of this information is processed by the pilot monitoring system, which analyzes data to determine the pilot's mental state. This system considers factors such as how tired or stressed the pilot may be, the amount of mental workload they are experiencing, and their level of focus on critical information. The system uses this information to detect if the pilot may be experiencing mode confusion. Mode confusion occurs when a pilot misunderstands what the aircraft's systems are doing. For example, a pilot might believe the autopilot is engaged when it isn't. If the system detects that the pilot may be experiencing mode confusion, it can alert the pilot and / or activate one or more controls (such as the autopilot). The system can also create a visual display of the pilot's mental state and any detected confusion, which may be used for training or review purposes.

[0030] The system uses one or more algorithms to determine the pilot's cognitive state, which is used to detect pattern confusion. For example, one or more algorithms can be configured to track what information the pilot may have seen based on where the pilot has been looking. Another algorithm can estimate the phase of flight (e.g., takeoff, cruise, or landing) based on aircraft data. These algorithms work together to create a comprehensive picture of the pilot's perception and current situation.

[0031] By using the technologies and systems described herein, air travel safety is improved because the pilot monitoring system assists pilots in maintaining better situational awareness, ultimately reducing the risk of accidents caused by human error or misunderstanding of complex aircraft systems. Furthermore, the data and insights collected from the system can be used to enhance and refine the underlying avionics systems, including improving autopilot functionality and the user interface. This iterative process of monitoring, analyzing, and improving can lead to more intuitive and error-resistant aircraft systems, further reducing the likelihood of pattern confusion and enhancing overall flight safety. The accompanying drawings and the following description illustrate specific exemplary embodiments. It should be understood that those skilled in the art will be able to design various arrangements, which, although not explicitly described or shown herein, embody the principles set forth herein and are included within the scope of the claims accompanying this specification. Furthermore, any examples described herein are intended to aid in understanding the principles of this disclosure and should be construed as not limiting. Therefore, this disclosure is not limited to the specific embodiments or examples described below, but is limited by the claims and their equivalents.

[0032] Specific embodiments are described herein with reference to the accompanying drawings. In the specification, common features are identified by common reference numerals throughout the drawings. In some figures, multiple instances of a particular type of feature are used. Although these features are physically and / or logically different, the same reference numerals are used for each feature, and different instances are distinguished by adding letters to the reference numerals. When the feature is referred to herein as a group or type (e.g., when no specific feature is referenced), reference numerals are used without distinguishing letters. However, when the reference herein refers to a specific feature among multiple features of the same type, reference numerals are used together with distinguishing letters. For example, referencing… Figure 1 Multiple sensors 102 are shown and associated with reference numerals 102A, 102B, 102C, 102D, and 102E. When referring to a specific system within these deployable data logger systems (e.g., eye-tracking sensor 102A), the distinguishing letter "A" is used. However, when referring to any of these sensors, reference numeral 102 is used without the distinguishing letter.

[0033] As used herein, various terms are used only for the purpose of describing particular embodiments and are not intended to be limiting. For example, the singular forms “a,” “an,” and “the” are also intended to include the plural forms unless the context clearly indicates otherwise. Furthermore, some features described herein are singular in some embodiments and plural in others. For illustration, Figure 9 It describes a system that includes one or more processors ( Figure 9 The term "processor" (920) refers to a computing device 910, indicating that in some embodiments, the computing device 910 includes a single processor 920, and in other embodiments, the computing device 910 includes multiple processors 920. For ease of reference herein, these features are generally introduced as "one or more" features, followed by a singular or optional plural (as generally indicated by "(s)"), unless an aspect relating to multiple features is described.

[0034] The terms “comprise,” “comprises,” and “comprising” are used interchangeably with “include,” “includes,” or “including.” Furthermore, the term “wherein” is used interchangeably with the term “where.” As used herein, “exemplary” indicates an example, implementation, and / or aspect and should not be construed as limiting or indicating a preference or preferred implementation. As used herein, ordinal numbers (such as “first,” “second,” “third,” etc.) used to modify elements (such as structures, components, operations, etc.) do not themselves indicate any priority or order of that element relative to another element, but merely distinguish that element from another element with the same name (but using an ordinal number). As used herein, the term “set” refers to a grouping of one or more elements, and the term “multiple” refers to multiple elements.

[0035] As used herein, unless the context otherwise indicates, “generating,” “calculating,” “using,” “selecting,” “accessing,” and “determining” are interchangeable. For example, “generating,” “calculating,” or “determining” a parameter (or signal) can refer to actively generating, calculating, or determining the parameter (or signal), or it can refer to using, selecting, or accessing a parameter (or signal) that has already been generated (e.g., by another component or device). As used herein, “coupling” can include “communication coupling,” “electrical coupling,” or “physical coupling,” and may also (or alternatively) include any combination thereof. Two devices (or components) may be coupled directly or indirectly (e.g., communication coupling, electrical coupling, or physical coupling) via one or more other devices, components, wires, buses, networks (e.g., wired networks, wireless networks, or combinations thereof). As an illustrative, non-limiting example, two electrically coupled devices (or components) may be in the same or different devices and may be connected via electronics, one or more connectors, or inductive coupling. In some implementations, two communication-coupled devices (or components) (e.g., in electrical communications) can directly or indirectly send and receive electrical signals (digital or analog signals) (e.g., via one or more wires, buses, networks, etc.). As used herein, "direct coupling" is used to describe two devices coupled (e.g., communication coupling, electrical coupling, or physical coupling) without intermediate components.

[0036] Figure 1This is a diagram illustrating an example pilot cognitive reasoning system 100. The cognitive reasoning system 100 includes one or more sensors 102, a device 104 coupled to the sensors 102, and a display device 134. Device 104 includes a processor 108 coupled to a memory 110. Processor 108 includes an algorithm estimator 112, an aircraft state data generator 114, and a data visualization generator 118. While system 100 can be implemented on a computing device located within an aircraft, or on a distributed computing system with components both on and off the aircraft, it can also be implemented on various vehicles, such as cars, ships, helicopters, trains, etc.

[0037] System 100 can be configured to use procedures to improve its accuracy and reliability in determining the pilot's cognitive state. For example, system 100 can be configured to use physiological baselines and calibrations based on environmental factors.

[0038] A physiological baseline can be used to reduce the influence of confounding variables and account for physiological differences among pilots. This approach helps mitigate the effects of stress or pressure caused by participating in data collection events by allowing pilots to acclimatize to the testing environment for a short period. Physiological data can be acquired for a short time while pilots are at rest; this allows System 100 to derive additional variables representing the differences in physiological data from the baseline during data collection. For example, as will be explained in more detail below, System 100 can be configured to utilize variables such as “heart_rate_difference,” which represents the difference between instantaneous and average heart rates during baseline data collection. By using a physiological baseline, this helps minimize inter-pilot and pilot-individual variations in the algorithm when predicting mental fatigue and cognitive workload by considering how pilots' current physiology deviates from their resting physiology.

[0039] In addition to a physiological baseline, System 100 can also be configured to employ a calibration procedure to account for environmental factors, particularly the effect of lighting conditions on pupil diameter. Individual differences, such as age, can affect pupil response to changes in ambient lighting. To account for this, System 100 can be configured to measure a subject's pupil diameter while the subject observes a screen with varying brightness levels. The resulting pairwise data, which correlates each brightness value with its corresponding pupil diameter measurement, can be used to fit a pupil response curve specific to that individual pilot. As described below in more detail regarding the processing of data associated with eye-tracking data (e.g., data 122A), this curve can be used during mission execution to estimate the effect of dynamic ambient lighting conditions on the pilot's pupil diameter. This allows System 100 to distinguish between pupil size variations caused by cognitive factors, such as mental workload, and pupil size variations caused by ambient lighting conditions, thereby improving the robustness of System 100's algorithmic estimator 112.

[0040] Sensor 102 may include an eye tracker sensor 102A, an ambient light sensor 102B, a heart rate sensor 102C, a skin conductance sensor 102D, a microphone sensor 102E, or a combination thereof. Although Figure 1 Five sensors are depicted, but in other embodiments, a different number (e.g., two, three, four, or some other number) of sensors 102 may be used.

[0041] Eye tracker sensor 102A can be configured to monitor various aspects of a pilot's visual behavior. Eye tracker sensor 102A can be configured to measure gaze direction in three-dimensional space, enabling device 104 to determine where the pilot is looking at any given moment. This allows device 104 to determine which instruments or displays the pilot has been observing. Eye tracker sensor 102A can also be configured to measure pupil diameter, which can be an indicator of cognitive workload or emotional state. In some aspects, eye tracker sensor 102A can be configured to track head position and orientation, thereby providing information about the pilot's posture and general direction of attention. Eye tracker sensor 102A can be configured to detect and analyze saccades (rapid eye movements), fixation (periods of relative eye rest), and blinks, all of which can provide insights into the pilot's attention patterns and fatigue levels.

[0042] Eye-tracking sensor 102A can be configured to send data 122A (e.g., eye-tracking data) to processor 108 of device 104. For example, eye-tracking sensor 102A can be configured to send data 122A to eye-tracking processor 108A to analyze the pilot's visual attention patterns and determine various cognitive states. In some embodiments, eye-tracking processor 108A can be configured to calculate metrics such as gaze duration on cockpit instruments, scan rate, and scan pattern. These metrics help determine where the pilot is focusing their attention and how efficiently they are collecting visual information. For example, a longer gaze duration on a particular instrument may indicate increased cognitive processing of that information, while frequent scans between instruments may indicate high situational awareness or potential information overload.

[0043] Sensor 102 may also include one or more ambient light sensors 102B. The one or more ambient light sensors 102B may be configured to measure the illuminance of the cockpit environment. This information enables device 104 to determine the effect of lighting conditions on pupil dilation, thereby allowing for a more accurate interpretation of data 122A by distinguishing between pupil changes caused by cognitive factors and pupil changes caused by fluctuations in ambient lighting. The one or more ambient light sensors 102B may be configured to send data 122B (e.g., ambient light data) to processor 108. For example, the one or more ambient light sensors 102B may be configured to send data 122B to ambient light processor 108B. In some aspects, system 100 may be configured to measure the illuminance of the digital screen directly from software. For example, system 100 may access screen brightness settings or pixel intensity values ​​from display software, thereby providing an additional source for data 122B.

[0044] The heart rate sensor 102C can be configured to monitor a pilot's heart rate and heart rate variability. The heart rate sensor 102C can be configured to monitor the pilot continuously, at periodic intervals, or a combination thereof. These physiological signals can provide insights into the pilot's stress level, workload, and overall physiological state. For example, changes in heart rate patterns may indicate increased cognitive load or fatigue episodes, both of which can be factors in maintaining safe flight operations.

[0045] In some implementations, the heart rate sensor 102C may be a wearable device, such as a wristband, ring, watch, etc. The heart rate sensor 102C may be configured to send data 122C (e.g., heart rate data) to a processor 108. For example, the heart rate sensor 102C may be configured to send data 122C to a heart rate processor 108C, which processes the data 122C to assess the pilot's stress level and overall physiological arousal. In some aspects, the heart rate processor 108C analyzes heart rate variability (HRV) measures, such as the standard deviation of normal-to-normal (NN) intervals (SDNN) and the root mean square of the difference in consecutive R-wave intervals (RMSSD). A lower HRV may indicate increased stress or mental workload, while changes in HRV patterns over time may signal the onset of fatigue. For example, a sustained decrease in SDNN during complex flight maneuvers may indicate increased cognitive load, while a gradual decrease in RMSSD during long-haul flights may indicate increased fatigue.

[0046] The electrical activity of the skin (EDA) sensor 102D can be configured to measure changes in the electrical properties of a pilot's skin. Specifically, the EDA sensor 102D can be configured to track skin conductance, which tends to increase during periods of stress or heightened cognitive activity.

[0047] Device 104 uses data 122D (e.g., EDA data) to measure the pilot's physiological arousal and stress levels. In processing this data 122D, device 104 can be configured to identify skin conductance response (SCR) and analyze its frequency and amplitude. Increased SCR activity may indicate elevated stress or cognitive load (especially when related to a specific event or task during flight). For example, a sudden increase in SCR frequency and amplitude during unexpected weather changes may indicate elevated stress levels, while sustained high SCR activity during complex approach procedures may indicate high cognitive load.

[0048] EDA sensor 102D can be configured to send data 122D (e.g., EDA data) to sensor processor 108. For example, EDA sensor 102D can be configured to send data 122D to EDA processor 108D, which performs these detailed analyses on the SCR to help make a holistic assessment of the pilot’s physiological arousal and stress levels, as described in more detail herein.

[0049] The microphone sensor 102E can be configured to capture data 122E (e.g., audio data) (specifically, the pilot's voice) from the cockpit. The microphone sensor 102E can be configured to capture various aspects of the sound pattern (including frequency, communication interval, and reaction time).

[0050] Device 104 can be configured to process data 122E to perform detailed analysis of the pilot's speech patterns. Device 104 can be configured to examine characteristics such as speech rate, pitch variations, and vocal tension. Changes in these parameters may indicate increased stress or cognitive load. For example, higher pitch and faster speech rate may indicate an increased stress level, while longer reaction times or increased pauses may indicate higher cognitive load or fatigue.

[0051] The microphone sensor 102E can be configured to send data 122E (e.g., audio data) to the sensor processor 108. For example, the microphone sensor 102E can be configured to send data 122E to the microphone processor 108E, which performs these detailed analyses of the voice patterns to help make a holistic assessment of the pilot’s cognitive state and stress level, as described in more detail herein.

[0052] In some embodiments, device 104 includes interfaces for each sensor 102, which preprocess the data 122 before sending it to their respective processor 108. These interfaces and initial preprocessing will... Figure 2 This will be discussed in more detail later.

[0053] The eye tracker processor 108A can be configured to analyze data 122A from the eye tracker sensor 102A. The eye tracker processor 108A can be configured to determine gaze duration and frequency, detect saccade patterns, and measure pupil diameter changes. The eye tracker processor 108A can be configured to calculate head position and orientation. In some aspects, the eye tracker processor 108A can be configured to calculate metrics such as PERCLOS (percentage of eyelid closure) for fatigue detection and gaze entropy for assessing situational awareness.

[0054] The ambient light processor 108B can be configured to normalize and calibrate light measurement results from the data 122B, thereby converting sensor readings into standardized illuminance units. The ambient light processor 108B can also be configured to detect changes in lighting conditions that may affect pupil dilation and apply a smoothing algorithm to reduce noise in the data 122B.

[0055] The heart rate processor 108C can be configured to analyze data 122C, including cardiac signals from the heart rate sensor 102C. The heart rate processor 108C can be configured to calculate heart rate and heart rate variability (HRV). The heart rate processor 108C can be configured to calculate metrics such as the standard deviation of the NN interval (SDNN) and the root mean square of the difference in consecutive R-wave intervals (RMSSD), which provide insights into the pilot's stress levels and autonomic nervous system activity.

[0056] The electrical activity of the skin (EDA) processor 108D can be configured to identify significant skin conductance responses (SCRs) from the data 122D. The EDA processor 108D can be configured to calculate the amplitude and frequency of these responses and derive an overall measurement of sympathetic arousal. The EDA processor 108D can be configured to separate the tone (baseline) and phase (rapid change) components of the data 122D, thereby providing a nuanced view of the pilot's physiological arousal state.

[0057] Microphone processor 108E can be configured to analyze various aspects of data 122E from microphone sensor 102E. Microphone processor 108E can be configured to measure the fundamental frequency (tone), analyze spectral characteristics, and potentially apply speech recognition algorithms to data 122E. Microphone processor 108E can be configured to calculate metrics in the sound (such as jitter and flicker), which can be indicators of stress or fatigue. Microphone processor 108E can be configured to perform cepstral analysis to obtain features such as cepstral peak salience, which can be used to determine sound fatigue.

[0058] After data 122 has been processed by one or more individual processors 108, the resulting processed data 126A is then sent via ZeroMQ (ZMQ) processor 106 to one or more algorithm estimators 112 and memory 110. This ensures that the algorithm estimator 112 has access to the latest processed data 126A for real-time analysis, while also retaining the processed data 126A for later review, analysis, or potential reprocessing using improved algorithms.

[0059] The aircraft state data generator 114 can be configured to collect and compile various data points representing the current state and performance of the aircraft to generate data 124. This includes, but is not limited to, altitude data, roll data, pitch data, yaw data, and airspeed data, or some combination thereof. The aircraft state data generator 114 continuously monitors these parameters to collect real-time information about the aircraft's position, orientation, and movement in three-dimensional space. In some embodiments, the aircraft state data generator 114 can be configured to collect data about engine performance, fuel levels, and other systems critical to flight operations.

[0060] In some implementations, the aircraft state data generator 114 may be configured to interface with various airborne systems and sensors to collect the data 124. The aircraft state data generator 114 may be configured to collect information from the aircraft's inertial measurement unit (IMU), altimeter, airspeed indicator, and other avionics systems. The aircraft state data generator 114 may be configured to process the different data formats and sampling rates from these various sources, merging them into a coherent data stream (e.g., data 124). The aircraft state data generator 114 may be configured to transmit data 124 to the aircraft state data processor 108F.

[0061] In some implementations, device 104 includes a dedicated interface for data 124, such as Figure 2 As described in more detail below. The aircraft state data processor 108F can be configured to receive data 124 from the aircraft state data generator 114. The aircraft state data processor 108F can be configured to filter and smooth the data 124 to reduce noise and eliminate spurious readings. The aircraft state data processor 108F can be configured to use various signal processing techniques, such as moving averages, Kalman filters, or other algorithms, to achieve this.

[0062] In some implementations, the aircraft state data processor 108F can be configured to calculate acquired metrics that provide insights into the aircraft's behavior. For example, the aircraft state data processor 108F can calculate climb or descent rates, turn rates, or accelerations on various axes. These acquired metrics can provide information about the aircraft's dynamics. The aircraft state data processor 108F can be configured to detect significant changes or anomalies in the aircraft's state. For example, it can identify sudden changes in altitude, unusual attitude angles, or unexpected changes in airspeed.

[0063] In some implementations, the aircraft state data processor 108F can be configured to contextualize the data 124 within the current flight phase. It can be used in conjunction with a Flight Forward Tracking Phase (PFFT) algorithm or another algorithm to interpret the aircraft state data in the context of whether the aircraft is in takeoff, climb, cruise, descent, or landing phase. This contextualization helps to understand whether the current aircraft state is normal or abnormal for a given flight phase.

[0064] Once the aircraft status data has been processed, the resulting processed data 126B is sent via the ZMQ processor 106 to both the algorithm estimator 112 and the memory 110. This ensures that the processed data 126B is available for real-time analysis by the algorithm estimator 112, while also being stored in the storage device for later review, analysis, or potential reprocessing. This dual-path approach allows for both immediate use of the processed data 126B in pilot monitoring and long-term storage for post-flight analysis or system 100 improvements.

[0065] Algorithm estimator 112 can be configured to perform multiple functions once it receives processed data 126. Algorithm estimator 112 can be configured to integrate different data streams to build a comprehensive picture of the pilot's cognitive state. For example, if data 122A shows a rapid scan between instruments, data 122C indicates an increased stress level, and data 122E indicates increased tension / stress, then algorithm estimator 112 can determine that the pilot is experiencing a high mental workload and is potentially approaching cognitive overload.

[0066] In some implementations, the algorithm estimator 112 includes multiple Kalman filters and a Gaussian mixture model (GMM) algorithm. Each Kalman filter represents a different hypothesis about the pilot's cognitive state (taking into account individual variations in the pilot's response). The GMM algorithm can be configured to combine the outputs from these multiple Kalman filters, thereby allowing a probabilistic representation of the pilot's cognitive state that captures both the most probable state and the uncertainty in the estimation.

[0067] The GMM algorithm can be configured to combine the weighted outputs of multiple Kalman filters. The output of each filter can be represented as a Gaussian component in a mixture, and the weights assigned to these components can be dynamically adjusted based on data 122, 124 (e.g., physiological and aircraft state data). This dynamic weighting mechanism enables device 104 to adapt its estimates to the individual characteristics of the monitored pilot and varying flight conditions.

[0068] In some respects, data 122 and 124 (e.g., physiological and aircraft state data) can be used to update the individual Kalman filters and their corresponding weights in the GMM algorithm. This adaptive approach enables device 104 to provide a more accurate and nuanced estimate of the pilot's cognitive state over time, enhancing its ability to detect potential pattern confusion or other cognitive problems that may affect flight safety.

[0069] The algorithm estimator 112 can be configured to include a composite data fusion scheme. This scheme enables the algorithm estimator 112 to incorporate multiple interdependent estimation algorithms within the context of a probabilistic graphical model (PGM) algorithm. This approach allows the algorithm estimator 112 to leverage the strengths of different analytical techniques while maintaining a coherent probabilistic framework.

[0070] Within the multimodal cognitive state estimation framework, various machine learning models can be employed to process different aspects of the processed data 126. For example, neural networks can be used to classify eye movement patterns, while Bayesian inference model algorithms can estimate fatigue levels based on processed data 126 (e.g., physiological data). The outputs from each algorithm can be weighted according to their interpretability of a given human state and fused into a single probability estimate. The outputs of these individual models can then be integrated into a PGM algorithm, which represents the relationships between different cognitive states and observable data as a graphical structure.

[0071] Algorithm estimator 112 can be configured to include a Flight Forward Tracking Phase (PFFT) algorithm. The PFFT algorithm uses a Hidden Markov Model to compute the probability of discrete flight phase states based on processed data 126. It leverages a learned probability model and known flight dynamics phases to produce both a maximized posterior estimate and a classification uncertainty value for the current flight phase. By understanding the current flight phase, algorithm estimator 112 can be configured to contextualize pilot actions and attention requirements.

[0072] In some aspects, the algorithm estimator 112 can be configured to include a PGM algorithm to track the pilot's current awareness of various flight variables represented on instruments and meters in the flight deck. The PGM algorithm can combine gaze tracking data (e.g., from processed data 126) with a three-dimensional (3D) model of the environment to proactively record the time since a given instrument was viewed and any changes that have occurred during that time. The PGM algorithm can include fully observable recurrent Markov chains to model the pilot's visual inspection of instruments, where state progression represents the time since information internalization, and resets occur at random intervals based on gaze duration. This enables the algorithm estimator 112 to estimate the probability that the pilot knows the current state of each instrument based on the time since the pilot last viewed each instrument and how the instrument readings have changed since then.

[0073] Algorithm estimator 112 can be configured to include a probabilistic perception estimation algorithm. This method leverages the sequential nature of gaze data (e.g., processed data 126A) to create probabilistic perception estimates of discrete gaze events with quantified uncertainty. It can use a weighted aggregation (by saccade subdivision) of the raw gaze measurements over a time window to remove cognitive uncertainty and produce a 3D probabilistic view cone. This cone can then be projected onto a two-dimensional (2D) surface along with world model objects to compute probabilistic object intersections.

[0074] Algorithm estimator 112 can be configured to include a multi-model cognitive state estimator algorithm. The multi-model cognitive state estimator algorithm can provide cognitive state estimates for the monitored pilot. The algorithm can start with a set of pre-trained cognitive estimation models, each learned from historical data of various pilots using an expectation-maximization model approach. These models can represent different patterns of how physiological signals relate to cognitive state. Each pre-trained model can be implemented as a measurement likelihood function in a separate Kalman filter (using a near-constant position dynamics model that assumes cognitive state changes slowly over time unless disturbed by new observations).

[0075] In some implementations, when the multi-model cognitive state estimator algorithm receives new data from the current pilot (e.g., processed data 126), it can run these multiple Kalman filters in parallel. The outputs of these parallel filters can then be combined using a dynamic weighting scheme, where weights can be computed based on how well each model's prediction matches the input data from the current pilot (e.g., processed data 126). This enables the algorithm estimator 112 to adapt its estimates to the individual characteristics of the monitored pilot. The set of weighted filter outputs is represented by the GMM algorithm, providing a probabilistic estimate of the pilot's cognitive state that captures both the most probable state and uncertainty in the estimate.

[0076] Throughout the flight, as more data is collected from the pilot (e.g., processed data 126), the weights of the different models can be continuously updated. This continuous refinement allows the algorithmic estimator 112 to adjust its estimates over time to adapt to the specific patterns exhibited by the current pilot.

[0077] In some respects, the algorithm estimator 112 can be configured to compare the pilot's attention allocation with critical flight information. This comparison involves analyzing where the pilot is focusing their attention relative to the most important instruments or displays for the current phase of flight. For example, if the aircraft is approaching landing, the system 100 can be configured to assess whether the pilot is appropriately allocating their attention among the altimeter, airspeed indicator, and visual references outside the cockpit. Furthermore, the algorithm estimator 112 can be configured to determine altitude information available from multiple sources, such as the primary flight display, backup altimeter, and radio altimeter. If the pilot has recently viewed any of these instruments, the algorithm estimator 112 can be configured to determine the level of altitude awareness, even without directly observing the primary altimeter. This analysis helps identify potential gaps in situational awareness that could affect flight safety.

[0078] Algorithm estimator 112 can process data 122, 124 in a modular and flexible manner by combining these algorithms. Depending on the specific implementation and / or requirements, algorithm estimator 112 may include and use all these algorithms in combination, or select a subset thereof. For example, the GMM algorithm may be configured to combine the outputs from multiple Kalman filters, while the PFFT algorithm may be configured to provide context about the flight phase, which can inform the interpretations of other algorithms. The PGM algorithm may integrate the outputs from various other algorithms to build a comprehensive model of the pilot's perception. This modular approach allows system 100 to adapt to different scenarios and requirements.

[0079] After processing data 126 using this flexible algorithm combination, algorithm estimator 112 can be configured to generate data 128 and send data 128 to data visualization generator 118. Data visualization generator 118 can be configured to receive data 128 from algorithm estimator 112 and generate output data 132, which includes dynamic, real-time visualizations that provide intuitive insights into the pilot's cognitive state and situational awareness.

[0080] The data visualization generator 118 can be configured to create dynamic graphical representations of the pilot's cognitive state and aircraft parameters. These visualizations can include color-coded instruments, trend lines, or other intuitive formats that display the pilot's available attention resources, attention allocation, and situational awareness, as well as relevant aircraft status data. Output data 132 can be displayed via a display device 134 in these various readable formats, allowing for rapid interpretation of the pilot's current cognitive conditions relative to the flight situation.

[0081] The data visualization generator 118 can be configured to generate output data 132 to include visualizations of the probability outputs of the multi-model cognitive state estimator (GMM) algorithm. The visualizations can represent the GMM algorithm as a probability distribution curve or a confidence interval around a point estimate, thus providing a visualization of both the estimated cognitive state and the associated uncertainty.

[0082] The data visualization generator 118 can be configured to generate output data 132 to include visualizations related to the pilot's gaze behavior and situational awareness. The visualizations may include heat maps overlaid on the cockpit schematic to show where the pilot has been looking or which instruments have recently been checked and which instruments may require attention (output based on a probabilistic graphical model algorithm).

[0083] The data visualization generator 118 can be configured to generate output data 132 to include a composite display integrating multiple data streams from data 128. For example, the data visualization generator 118 can combine cognitive state estimates with gaze data (e.g., data 122A) and aircraft state information (e.g., data 124) to provide the pilot with a comprehensive view of the current conditions and perceptions relative to the flight situation.

[0084] The data visualization generator 118 can be configured to send output data 132 to a display device 134 configured to display the output data 132. The display device 134 may include a screen in the cockpit, a tablet computer used by a flight instructor, or any other suitable visual interface that allows real-time monitoring of the pilot's cognitive state and situational awareness.

[0085] In some implementations, the data visualization generator 118 can be configured to send output data 132 to the memory 110. This allows the output data 132 to be archived for post-flight analysis, training purposes, or long-term studies of pilot performance and cognitive patterns. By storing the output data 132, system 100 enables more comprehensive retrospective analysis and continuous improvement of pilot training and support systems.

[0086] The ZeroMQ (ZMQ) processor 106 can be configured to serve as a central communication hub for the entire system 100, thereby facilitating data flow between the sensor 102, various components within the device 104, and external systems.

[0087] When system 100 initializes, ZMQ processor 106 establishes a series of message queues corresponding to different data types and processing stages. ZMQ processor 106 can be configured to create separate queues for data 122, processed data 126, aircraft status data 124, data 128, output data 132, or combinations thereof.

[0088] The ZMQ processor 106 can be configured to interface directly with sensors of each sensor type in the sensor array 102 (e.g., ...). Figure 2 The sensor interface 202 described herein is used for interface docking. When the sensor 102 collects data 122, its corresponding interface publishes the data 122 to the appropriate queue. The ZMQ processor 106 can be configured to make the data 122 available to the corresponding processor 108.

[0089] Within device 104, ZMQ processor 106 can be configured to manage data flows 126, 128, 132, or combinations thereof between processor 108, aircraft status data processor 108F, algorithm estimator 112, data visualization generator 118, or combinations thereof. ZMQ processor 106 enables these components to subscribe to relevant data streams and publish their output without needing to know the details of the overall system architecture.

[0090] For example, when the eye tracker processor 108A finishes processing the data 122A, it publishes the processed data 126A to a specific queue. The ZMQ processor 106 then ensures that the processed data 126A can be used both for recording in the memory 110 and for further analysis by the algorithm estimator 112.

[0091] ZMQ processor 106 is also configured to implement a logging system that subscribes to all data streams, allowing for comprehensive data capture and subsequent playback. ZMQ processor 106 can also be configured to provide error handling and system status monitoring. For example, if any component encounters an error or failure, ZMQ processor 106 can publish this information to a dedicated error queue, enabling system 100 to react appropriately. While system 100 is shown including ZMQ processor 106, other messaging protocols, such as Message Queuing Telemetry Transport (MQTT) or Advanced Message Queuing Protocol (AMQP), can be used.

[0092] By centralizing communication between all components via the ZMQ processor 106, system 100 achieves a high degree of modularity. This design allows for the easy addition or modification of components without altering the entire system 100, provided that the new component adheres to the established messaging protocol.

[0093] During operation, sensor 102 continuously measures and collects data 122 (e.g., physiological data) from the pilot, including eye movements, heart rate, skin responses, audio, or combinations thereof. Simultaneously, aircraft state data generator 114 collects data 124 (e.g., flight data) regarding various flight parameters such as altitude, speed, and orientation. These data streams (e.g., data 122 and data 124) are then sent to their respective processors 108A-108F. Processor 108 cleans the data 122 (e.g., physiological data) and extracts relevant features from it, while aircraft state data processor 108F processes and contextualizes the data 124 (e.g., flight data).

[0094] The processed data 126A and 126B are then sent via ZMQ processor 106 to both memory 110 for archiving and to algorithm estimator 112. Algorithm estimator 112 (including one or more algorithms for human state estimation, pilot information tracking, and flight phase tracking) analyzes the processed data 126 (e.g., the incoming data stream). Algorithm estimator 112 generates estimates of the pilot's cognitive state (including mental workload, fatigue, attention allocation, and situational awareness).

[0095] Algorithm estimator 112 generates data 128, which includes mental workload, fatigue, attention allocation, and situational awareness, or combinations thereof. Data 128 is then sent to data visualization generator 118. Data visualization generator 118 acquires data 128 and transforms it into a meaningful visual representation. Data visualization generator 118 creates dynamic, real-time visualizations that provide intuitive insights into the pilot's cognitive state and situational awareness, thereby generating output data 132.

[0096] Output data 132 is sent to display device 134, which presents visualizations in a format easily interpretable by cockpit personnel, researchers, ground station operators, flight commands, or a combination thereof. This may include color-coded instruments, trend lines, or other readable formats that allow for rapid interpretation of the pilot's current cognitive condition and perception of the aircraft's status.

[0097] The technological advantages of using System 100 include providing a comprehensive, real-time assessment of the pilot's cognitive state and situational awareness, which was previously difficult to obtain non-intrusively in operational settings. This can significantly improve flight safety by detecting potential problems before they become serious. Another technological advantage includes the modular and flexible architecture of System 100, which allows for easy integration of new sensors or algorithms, enabling System 100 to adapt to future technological advancements or specific research needs. The use of the ZMQ processor 106 enables efficient, decoupled communication between components, which enhances the system's reliability and scalability.

[0098] Another technological advantage includes the system's ability to process multiple data streams simultaneously and fuse them into meaningful insights. By combining physiological data with aircraft status information, System 100 provides a more comprehensive view of pilot performance and perception than traditional monitoring methods.

[0099] Another technological advantage includes real-time visualization capabilities, which make complex data easily interpretable, enabling flight crews or researchers to make rapid decisions. This is particularly valuable in identifying and mitigating pattern confusion or other cognitive problems that may otherwise go unnoticed.

[0100] Another technological advantage includes the system's 100% data logging, playback, and reprocessing capabilities, which provide valuable tools for post-event analysis, training, and system improvement. This feature allows for detailed examination of pilot performance and system behavior, which can inform future training protocols and system enhancements. The reprocessing capability allows raw data to be fed back through the system using new or updated models to generate higher-quality output. This means that as algorithms and models are refined over time, historical data can be reanalyzed to generate new insights or improve accuracy, maximizing the value of the collected data and enabling continuous improvement of system performance.

[0101] Another technical advantage includes that the Gaussian Mixture Model (GMM) enables device 104 to represent complex multimodal probability distributions, which can capture the nuances of different cognitive states. By using multiple Gaussian components, the GMM can simultaneously represent multiple hypotheses about the pilot's state, where the weights of these components reflect the relative probability of each hypothesis. This approach is particularly useful in situations where the pilot's cognitive state may be ambiguous or rapidly changing. The GMM also provides a way to incorporate uncertainty into the estimation, which is crucial for making robust decisions based on these cognitive state assessments.

[0102] Figure 2 This diagram illustrates another pilot cognitive reasoning system 200. The system includes sensors 102, device 104, a ZMQ processor 106, memory 110, an algorithm estimator 112, an aircraft state data generator 114, a data visualization generator 118, a display device 134, and a sensor interface 202. While system 100 can be implemented on a computing device located within an aircraft, or on a distributed computing system with components both on and off the aircraft, it can also be implemented on various vehicles, such as cars, ships, helicopters, trains, etc.

[0103] like Figure 2As shown, device 104 includes a set of sensor interfaces 202 that interface with various sensors 102. Specifically, device 104 includes an eye tracker sensor interface 202A, an ambient light sensor interface 202B, a heart rate sensor interface 202C, a skin conductance sensor interface 202D, a microphone sensor interface 202E, or combinations thereof. These interfaces 202 are configured to preprocess data 122 (e.g., raw data) from the respective sensors 102 before passing it to processor 108. For example, sensor interfaces 202 may be configured to perform one or more of the following: remove NAN (non-digital) values, apply bandpass filtering to remove noise outside the frequency range of interest, remove outliers that may distort the analysis, perform linear interpolation on missing values, or combinations thereof.

[0104] The eye-tracking sensor 102A can be configured to monitor various aspects of a pilot's visual behavior. For example... Figure 1 The device can be configured to measure gaze position, pupil diameter, head position, and orientation in three-dimensional space, and to detect saccades, fixation, and blinks. The eye-tracking sensor 102A can be configured to send data 122A to the eye-tracking sensor interface 202A. The eye-tracking sensor interface 202A can be configured to receive data 122A and perform initial processing. The eye-tracking sensor interface 202A can be configured to handle tasks such as noise reduction, blink detection, and converting raw sensor output into meaningful eye-tracking parameters.

[0105] One or more ambient light sensors 102B can be configured to measure the illuminance of the cockpit environment, such as Figure 1 One or more ambient light sensors 102B can be configured to send data 122B to an ambient light sensor interface 202B. The ambient light sensor interface 202B can be configured to process raw light intensity readings, potentially converting them to standardized illuminance units. The ambient light sensor interface 202B is also configured to apply a calibration factor or smoothing algorithm to ensure consistent and accurate ambient light measurements.

[0106] The heart rate sensor 102C can be configured to continuously monitor the pilot's heart rate and heart rate variability, such as Figure 1 As shown in detail, the heart rate sensor 102C can be configured to send data 122C to the heart rate sensor interface 202C. The heart rate sensor interface 202C can be configured to process the raw cardiac signal. The heart rate sensor interface 202C can be configured to perform tasks such as R-peak detection, heart rate calculation, and initial heart rate variability calculation. The heart rate sensor interface 202C is also configured to handle any necessary signal filtering or artifact removal.

[0107] like Figure 1As shown, the skin conductance sensor 102D can be configured to measure changes in the electrical properties of a pilot's skin. The skin conductance sensor 102D can be configured to send data 122D to the skin conductance sensor interface 202D. The skin conductance sensor interface 202D can be configured to process the raw skin conductance signal. The skin conductance sensor interface 202D can be configured to perform initial feature extraction, such as identifying skin conductance responses or calculating the tone and phase components of the EDA signal.

[0108] The microphone sensor 102E can be configured to capture audio data (specifically, the pilot's voice) from the cockpit, such as... Figure 1 The microphone sensor 102E can be configured to send data 122E to the microphone sensor interface 202E. The microphone sensor interface 202E can be configured to perform initial audio processing tasks. The microphone sensor interface 202E can be configured to process noise reduction, sound activity detection, audio feature extraction, or a combination thereof. The microphone sensor interface 202E is also configured to manage any necessary audio format conversions or sampling rate adjustments.

[0109] In addition to these sensor interfaces 202, device 104 may also include an aircraft status interface 204. The aircraft status interface 204 may be configured to receive and preprocess data 124 from the aircraft status data generator 114, thereby ensuring that the aircraft status information is in a suitable format for further processing.

[0110] ZMQ processor 106 can be configured to manage the data flow between sensor interface 202 and processor 108. ZMQ processor 106 can be configured to establish a publish-subscribe messaging pattern, where each sensor interface 202 acts as a publisher and the corresponding processor 108 acts as a subscriber. When sensor interface 202 has data ready to send, ZMQ processor 106 can be configured to publish the data to a specific topic. ZMQ processor 106 is then configured to route these messages to the subscribed processors 108. This publish-subscribe system allows for decoupled asynchronous communication. If sensor 202 temporarily generates data 122 faster than it can process, ZMQ processor 106 can be configured to buffer messages. ZMQ processor 106 is also configured to allow the easy addition of new data consumers (e.g., memory 110) that can subscribe to all data topics to record raw data without affecting the main processing pipeline. While system 200 is shown including ZMQ processor 106, other messaging protocols, such as Message Queuing Telemetry Transport (MQTT) or Advanced Message Queuing Protocol (AMQP), can be used.

[0111] This architecture (with sensor interface 202, ZMQ processor 106 and processor 108) enables system 200 to effectively manage data streams from multiple different sensors, each with its own data format and timing characteristics.

[0112] The eye tracker processor 108A can be configured to receive pre-processed data (e.g., data 122A processed by the eye tracker sensor interface 202A) and execute one or more algorithms to detect and classify different types of eye movements (e.g., smooth tracking, microsaccades, and tremors). The eye tracker processor 108A can be configured to correlate eye movements with a 3D model of the cockpit environment, allowing it to accurately determine which instruments or displays the pilot is looking at at a given moment. The eye tracker processor 108A can be configured to calculate complex metrics (e.g., scan path, dwell time in a specific region of interest, and transitions between different cockpit areas).

[0113] The ambient light processor 108B can be configured to receive pre-processed data (e.g., data 122B processed by the ambient light sensor interface 202B) and execute one or more algorithms to track changes in illumination conditions over time, thereby potentially identifying patterns associated with different phases of flight or environmental conditions. The ambient light processor 108B can be configured to correlate light measurements with other data 122 (e.g., pupil dilation from the eye tracker sensor 102A) to provide context for interpreting physiological responses.

[0114] The heart rate processor 108C can be configured to receive data (e.g., processed data 122C by the heart rate sensor interface 202C) and perform time-domain, frequency-domain, and nonlinear analyses of heart rate variability. The heart rate processor 108C can be configured to calculate metrics such as power spectral density in different frequency bands, which can provide insights into the balance between sympathetic and parasympathetic nervous system activity.

[0115] EDA processor 108D can be configured to receive data (e.g., processed data 122D by EDA interface 202D) and perform signal processing techniques to separate the tone and phase components of the EDA signal. EDA processor 108D can be configured to perform temporal analysis of SCR, potentially identifying patterns or rhythms of sympathetic nervous system activation. EDA processor 108D can be configured to correlate EDA responses with specific events or stimuli in the cockpit environment, thereby providing a more contextualized understanding of the pilot's physiological arousal.

[0116] Microphone processor 108E can be configured to receive data (e.g., data processed by microphone sensor interface 202E 122E) and perform speech recognition to transcribe the pilot's speech. Microphone processor 108E can be configured to perform emotion analysis on the transcribed speech, potentially identifying emotional states from vocal cues. Microphone processor 108E can be configured to analyze non-vocalizations, such as sighs or throat clearing, which can provide additional insights into the pilot's cognitive state.

[0117] The aircraft status data processor 108F is configurable to receive data (e.g., processed data 124 from the aircraft status interface 204), process, and track a wide variety of flight parameters, including but not limited to altitude, airspeed, vertical speed, heading, pitch, roll, yaw, engine performance metrics, and system status indicators, such as... Figure 1 The aircraft state data processor 108F can be configured to detect significant changes or anomalies in these parameters, potentially identifying abnormal flight conditions or system malfunctions. The aircraft state data processor 108F can be configured to correlate processed data received from the aircraft state interface 204 with pilot actions and physiological responses. The aircraft state data processor 108F can be configured to identify patterns that indicate how different flight conditions affect the pilot's cognitive state. The aircraft state data processor 108F can be configured to implement predictive algorithms to predict future aircraft state based on current trends and pilot input. Such predictions may be crucial for the early detection of potential problems or conflicts.

[0118] Processor 108 can be configured to send the corresponding processed data 206 to ZMQ processor 106. ZMQ processor 106 can then be configured to route the processed data 206 to both algorithm estimator 112 and memory 110. This ensures that algorithm estimator 112 has access to the latest, analyzed data for real-time cognitive state estimation, while also retaining the processed data 206 for later review, analysis, or potential reprocessing using improved algorithms. Algorithm estimator 112 uses the processed data 206 to determine pilot mental workload and mental fatigue based on physiological data. For example, algorithm estimator 112 can analyze the processed data 206 (e.g., physiological data such as heart rate variability, eye movement patterns, and even underlying vocal characteristics) to gauge the pilot's mental workload and fatigue level. Algorithm estimator 112 can employ machine learning models (such as... Figure 3 and Figure 4 The above) is used to identify patterns in the processed data 206 that are associated with these cognitive states.

[0119] Algorithm estimator 112 can be configured to determine a pilot's available attentional resources based on mental workload and mental fatigue. For example, based on the determined mental workload and fatigue, algorithm estimator 112 can calculate the pilot's available attentional resources. This represents the pilot's cognitive ability to process information and react to events at a given moment. Algorithm estimator 112 may include a combination of rule-based logical and probabilistic models to infer this ability from workload and fatigue estimates.

[0120] Algorithm estimator 112 can be configured to determine the pilot's attention allocation based on available attentional resources and gaze patterns derived from physiological data. For example, algorithm estimator 112 can combine gaze patterns (e.g., from processed data 206) with available attentional resources to determine the pilot's attention allocation. This process may involve evaluating how the pilot allocates their attention across various instruments, displays, and the external environment. Algorithm estimator 112 can employ methods such as probabilistic gaze intersection analysis (e.g., Figure 5 The technology described above is used to track the pilot's visual focus and infer their attention priorities.

[0121] Algorithm estimator 112 can be configured to determine the pilot's situational awareness based on attention allocation and aircraft state data. For example, algorithm estimator 112 can be configured to integrate the pilot's attention allocation with real-time aircraft state data (e.g., data 124) to assess the pilot's situational awareness. This process may involve assessing whether the pilot is handling the most critical information and instruments (given the current flight phase and aircraft conditions). Algorithm estimator 112 can utilize probabilistic models and contextual information about the flight to infer the pilot's understanding of the overall situation.

[0122] The data visualization generator 118 receives data 208 from the algorithm estimator 112 and converts it into one or more visual representations, such as... Figure 1 The data visualization generator 118 creates dynamic visualizations that provide intuitive insights into the pilot's cognitive state and situational awareness, generating output data 210. This output data 210 may include color-coded instruments, trend lines, or other readable formats that allow for rapid interpretation of the pilot's current cognitive condition and perception of the aircraft's status. The data visualization generator 118 then sends the output data 210 to a display device 134, which presents these visualizations in a format easily interpretable by cockpit personnel or researchers.

[0123] The technological advantages of using System 200 can include providing a comprehensive, multi-layered architecture for real-time processing and analysis of diverse sensor data to infer pilot cognitive states. The use of dedicated sensor interfaces (202A-202E) for each sensor type allows for customized, optimized data acquisition and initial preprocessing based on the unique characteristics of each sensor. This approach improves data quality and reliability from the outset. The ZMQ processor 106, implemented as a central communication hub, provides a flexible, scalable, and efficient means of data distribution throughout the system. This publish-subscribe model allows for easy integration of new components and ensures that all parts of the system have access to the latest information. Dedicated processors (108A-108E) and the flight data processor 108F are capable of background awareness analysis on each data stream. This allows for the extraction of complex, high-level features from sensor data 122, enabling System 200 to perform accurate cognitive state estimations.

[0124] Another technical advantage of using System 200 can include its ability to correlate data from multiple sensors with aircraft states, as processed by these components, thereby enabling a more comprehensive understanding of the pilot's cognitive state relative to the flight environment and ongoing operations. Furthermore, System 200 supports both real-time processing for immediate cognitive state estimation and data storage for post-event analysis and system improvement. This dual-purpose design enhances the utility of System 200 for both immediate safety applications and long-term research and development efforts.

[0125] Figure 3 This is a specific implementation 300 illustrating an example of an algorithmic estimator 310 including a multimodal cognitive state estimation framework. The algorithmic estimator 310 may include, for example... Figure 1 and Figure 2 The algorithm estimator 112 described herein.

[0126] The algorithm estimator 310 can be configured to take one or more data points 302 as input. The data points 302 may include data 302A, data 302B, and data 302C, each of which can represent... Figure 1 and Figure 2 Various combinations of the data types described herein. These data types include data 122A-122E, aircraft status data 124, processed physiological data 126, processed sensor data 206, or any combination thereof. Each data input (302A, 302B, 302C) can contain different combinations of these data types, thereby allowing for flexible and comprehensive analysis by the algorithm estimator 310.

[0127] Each data input 302 is processed by a corresponding machine learning model 304. The first model (machine learning model 304A) can employ Bayesian inference techniques. Bayesian inference is a statistical method that updates the probability of a hypothesis as more evidence becomes available. In this context, it can be used to estimate the likelihood of various pilot states or conditions based on data. For example, machine learning model 304A can be configured to calculate the probability of pilot fatigue given observed physiological signals, flight duration, and time of day. Machine learning model 304A using Bayesian inference methods offers the technical advantage of handling uncertainty and incorporating prior knowledge about typical pilot behavior or physiological responses.

[0128] Machine learning model 304B can be configured to utilize a neural network architecture. Neural networks are inspired by the human brain and consist of interconnected nodes organized in a hierarchical manner. Machine learning model 304B can be configured to identify complex patterns in data 302. For example, machine learning model 304B can be trained to recognize patterns in pilot movements, eye movements, or physiological data that indicate certain cognitive states or levels of situational awareness.

[0129] Machine Learning Model 304C can be configured to use one or more regression models. Regression models can be configured to understand the relationships between variables and make predictions. For example, the regression models used by Machine Learning Model 304C can be used to predict continuous variables (such as stress levels, reaction time, or performance metrics) based on various input factors. For example, Machine Learning Model 304C can be configured to estimate a pilot's current mental workload level based on factors such as flight phase, weather conditions, and recent aircraft system alerts. Machine Learning Model 304C can be configured to help quantify the impact of different factors on pilot performance and cognitive state.

[0130] The outputs from these machine learning models 304 can then be combined via data fusion 306 and output as data 308. Data 308 can then be visualized by a data visualization generator 118 and displayed on a display device 134, such as... Figure 1 As stated above.

[0131] Using the integrated data 308, the algorithm estimator 310 (e.g., a multimodal cognitive state estimation framework) enables the device (e.g., Figures 1-2Device 104 is capable of determining the pilot's available attentional resources based on mental workload and mental fatigue. This assessment can be configured to combine the outputs from machine learning model 304 to estimate the pilot's current cognitive capacity. Algorithm estimator 310 can then be configured to determine the pilot's attention allocation. This process takes into account the previously calculated available attentional resources and incorporates gaze patterns derived from data 302. Algorithm estimator 310 can be configured to assess the pilot's situational awareness by combining attention allocation data with aircraft state data.

[0132] This multi-machine learning model 304 approach allows the algorithm estimator 310 to provide a comprehensive assessment of the pilot's cognitive state and performance. By integrating different data sources (e.g., data 302A-302C) and employing multiple machine learning models 304A-304C, the algorithm estimator 310 can be configured to provide insights that contribute to improving the safety and efficiency of aviation operations.

[0133] In some implementations, machine learning models 304A-304C can be configured to use various types of algorithms. For example, machine learning models 304A-304C can employ decision trees, random forests, support vector machines, gradient boosting machines, or deep learning architectures (such as convolutional neural networks (CNNs) or recurrent neural networks (RNNs)). The model selection for each input can be based on the characteristics of the data and the specific aspect of the pilot's cognitive state being estimated. In some aspects, machine learning models 304A-304C can be of the same type (if desired), such as all being neural networks or all being regression models. Different models (even those of the same type) can learn in different ways and provide different outputs due to differences in their architecture, training data, or hyperparameters. This flexibility allows system 300 to be optimized for different scenarios or data input types.

[0134] Figure 4 Figure 400 illustrates a specific implementation of an algorithm estimator 414, including a Gaussian Mixture Model (GMM) algorithm. The algorithm estimator 414 can be configured to process multiple data streams to estimate various cognitive states of the pilot. The algorithm estimator 414 may include, for example... Figure 1 and Figure 2 The algorithm estimator 112 described in the document, such as Figure 3 The algorithm estimator 310 or a combination thereof is described in the document.

[0135] The algorithm estimator 414 can be configured to take one or more data 402 as input. Data 402 may include data 402A, data 402B, and data 402C, each of which can represent... Figure 1 and Figure 2Various combinations of the data types described herein. These data types include data 122A-122E, aircraft status data 124, processed physiological data 126, processed sensor data 206, or any combination thereof. Each data input (402A, 402B, 402C) can contain different combinations of these data types, thereby allowing for flexible and comprehensive analysis by the algorithm estimator 414.

[0136] Each data point 402 is processed by a separate machine learning model 404. For example, data 402A is processed by machine learning model 404A, data 402B by machine learning model 404B, and data 402C by machine learning model 404C. Each of these machine learning models 404 can be configured to implement an expectation-maximization (EM) algorithm. In some implementations, based on the specific characteristics of the data 402 and the desired output, the machine learning model 404 can be configured to use a neural network, support vector machine, random forest, gradient boosting machine, hidden Markov model, or a combination thereof.

[0137] In some implementations, the machine learning model 404 can be configured to be trained using the EM algorithm. The EM algorithm can be an iterative method that enables the machine learning model 404 to learn the relationship between data 402 (e.g., physiological signals, eye movements) and corresponding cognitive states (e.g., workload, fatigue, attention). In some aspects, the EM algorithm can estimate the mean and covariance of a Gaussian distribution, which represents the probability of a particular cognitive state given data 402. The EM algorithm can iteratively refine these estimates by calculating the expected value of the log-likelihood function and maximizing it relative to the mean and covariance. The EM algorithm can continue this process until the model 404 converges to the maximum likelihood estimate of the mean and covariance, thereby improving the accuracy and reliability of the cognitive state estimation. The outputs of these machine learning models 404 can then be fed to filters 406A-406C.

[0138] In some implementations, machine learning model 404 may be configured to determine patterns and relationships in data 402. Each machine learning model 404 may be configured to identify key features and map these features to cognitive state estimates. For example, machine learning model 404A may be configured to estimate mental workload based on heart rate variability and eye movement patterns, while another machine learning model 404B may be configured to estimate fatigue based on blink rate and voice characteristics.

[0139] The output of the machine learning model 404 can then be fed into filters 406A-406C. These filters 406 can be configured to include Kalman filters. In some embodiments, filters 406 can be configured to include particle filters, unscented Kalman filters, extended Kalman filters, H-infinity filters, or combinations thereof.

[0140] Each filter 406 can be configured to refine and smooth the estimates generated by the machine learning model 404. Filters 406 can be configured to take into account the temporal aspect of the data, reduce noise, and provide more stable estimates over time. For example, filter 406 can be configured to smooth rapid fluctuations in the estimation workload that might be due to measurement noise rather than actual changes in cognitive state.

[0141] The outputs of these individual filters 406 can then be combined at the summing node 408. The summing node 408 can be configured to aggregate estimates from different data streams, potentially applying weights to prioritize some estimates over others based on their reliability or relevance.

[0142] The aggregated estimate generated by summing node 408 can be processed by filter 410. Filter 410 can be configured to include a Gaussian Mixture Model (GMM) algorithm. Filter 410 can be configured to determine the uncertainty in the estimate and potentially represent multiple assumptions about the pilot's cognitive state.

[0143] The output of filter 410 can be represented as data 412, which may contain a final estimate of the pilot's cognitive state. Data 412 can then be visualized by data visualization generator 118 and displayed on display device 134, such as... Figure 1 As stated above.

[0144] During operation, algorithm estimator 414 may include a Gaussian mixture model (GMM) filtering algorithm, which can be configured to determine the pilot's available attentional resources based on mental workload and mental fatigue estimates. By combining workload and fatigue estimates from various data 402, algorithm estimator 414 can infer how much attentional resources the pilot has available at any given time. Algorithm estimator 414 can be configured to determine the pilot's attention allocation based on available attentional resources and gaze patterns derived from physiological data (e.g., data 402). By analyzing where the pilot is looking in the context of their available attentional resources (from data 402), algorithm estimator 414 can estimate how the pilot allocates their attention across different instruments and areas of the cockpit. Algorithm estimator 414 can be configured to determine the pilot's situational awareness based on attention allocation and aircraft state data (e.g., data 402). By combining information about where the pilot is allocating their attention (e.g., at summation node 408) with data about the current state of the aircraft (e.g., altitude, speed, and system status) (e.g., data 402), algorithm estimator 414 can estimate how the pilot perceives the current situation.

[0145] Figure 5 This diagram illustrates a specific implementation of a pilot cognitive reasoning system 500, focusing on the processing of eye-tracking data (e.g., data 122) and its integration with aircraft status data (e.g., data 124). System 500 may include an eye-tracking sensor 102A, which can be configured to capture raw gaze data (e.g., data 122) from the pilot. Figure 1 and Figure 2 The eye tracker sensor 102A can be configured to measure gaze position, pupil diameter, eyelid opening, head position and orientation in three-dimensional space, and detect saccades, fixation, and blinking. Data 122 is sent to device 104 for processing.

[0146] Device 104 can be configured to include an eye-tracker sensor interface 202A, which can be configured for data 122. The eye-tracker sensor interface 202A may include a gaze refinement unit 502, which can be configured to perform initial preprocessing on the data 122. The gaze refinement unit 502 can be configured to filter noise, calibrate gaze coordinates, and perform other low-level processing tasks. The gaze refinement unit 502 can be configured to divide the data 122 into windows of predetermined lengths and further subdivide it by saccade events. The gaze refinement unit 502 can be configured to apply a weighted average to the gaze origin and direction using gaze quality as a weight. The gaze refinement unit 502 generates refined data 504A, which includes a weighted mean of the gaze origin and direction, and a 3D uncertainty ellipsoid.

[0147] The refined data 504A is then sent to the eye tracker processor 108A. The eye tracker processor 108A can be configured to include an object detector 506, which can be configured to analyze the refined data 504A and determine which objects or areas of interest the pilot is looking at in the cockpit. The object detector 506 can be configured to implement a probabilistic method in which a 3D probabilistic visual conic projection is onto a 2D surface perpendicular to the gaze vector. The object detector 506 can be configured to allow the user to use either a size-based weighting scheme or an object importance-based weighting scheme. This flexibility allows the object detector 506 to adapt to different analysis priorities or cockpit configurations. The object detector 506 can be configured to calculate the overlap between the projected ellipse and cockpit objects using the selected weighting scheme to determine the probability of the gaze intersecting with each object. The object detector generates data 508A that includes probabilistic gaze intersection information.

[0148] System 500 may include an aircraft status data generator 114, which can be configured to collect various flight parameters and system states, such as Figure 1 and Figure 2 Various flight parameters and system states are included in data 124, which is sent to the aircraft status interface 204. The aircraft status interface 204 preprocesses the data 124 (e.g., ...). Figure 2 The refined data 504B is generated and sent to the aircraft status data processor 108F. The aircraft status data processor 108F processes the refined data 508B (e.g., as described above) to generate detailed data 504B, and sends the detailed data 504B to the aircraft status data processor 108F. Figure 1 The processed data described in reference 126B and as follows Figure 2(Refer to data 206), and then send data 508B to the algorithm estimator 112.

[0149] Data 508A and data 508B can then be fed into algorithm estimator 112. Algorithm estimator 112 can be configured to include information extractor module 510. Information extractor module 510 can be configured to combine data 508A with data 508B to determine information about the pilot's perception and situational awareness. Information extractor module 510 can be configured to track the time since the last time the instruments were viewed and how their values ​​have changed since then. Information extractor module 510 can be configured to generate data 512, which represents a comprehensive probabilistic assessment of the pilot's Level 1 situational awareness—their perception of key elements in the environment. Data 512 may include information about what the pilot is currently looking at, as well as a time-based record of what information they have recently acquired and how that information may have changed.

[0150] Then, data 512 from algorithm estimator 112 can be sent to both memory 110 and data visualization generator 118. Memory 110 can be configured to archive data 512 for later analysis or review. Data visualization generator 118 can be configured to generate output data 514, which may include a visual representation of data 512, which is then displayed on display device 134. Throughout the process, ZMQ processor 106 can be configured to manage data flow between different components of the system, such as... Figure 1 and Figure 2 As stated above.

[0151] The technological advantages of using System 500 include probabilistic analysis of pilot gaze behavior within the context of the current flight situation. By combining detailed eye-tracking data with the latest aircraft status information, System 500 can provide a nuanced understanding of the uncertainties in a pilot's perception at a given moment.

[0152] Figure 6 This is a flowchart of method 600 using a pilot monitoring system. Method 600 includes receiving physiological data from multiple sensors from a pilot at block 602. For example, Figure 1 The system 100 can be configured to receive various types of data 122 from the sensor 102. The data 122 may include eye movement and pupil dilation data from the eye tracker sensor 102A, ambient light measurements from the ambient light sensor 102B, heart rate and heart rate variability data from the heart rate sensor 102C, skin conductance responses from the skin conductance sensor 102D, and sound patterns and characteristics from the microphone sensor 102E.

[0153] Method 600 includes receiving aircraft state data from an aircraft state data generator in block 604. For example, Figure 1 System 100 can be configured to receive various types of data 124 from aircraft status data generator 114. Data 124 (cab) includes information about the aircraft's position (altitude), orientation (roll, pitch, yaw), and motion (airspeed) in three-dimensional space. System 100 can also receive data 124, which includes engine performance, fuel levels, and other critical flight systems.

[0154] Method 600 includes determining a pilot's mental workload and mental fatigue based on physiological data in block 606. For example, Figure 1 The algorithm estimator 112 can be configured to determine a pilot's mental workload and mental fatigue based on physiological data. The algorithm estimator 112 can be configured to integrate different data streams to build a comprehensive picture of the pilot's cognitive state. For example, if data 122A shows a rapid scan between instruments, data 122C indicates an increased stress level, and data 122E indicates increased tension, then the algorithm estimator 112 can determine that the pilot is experiencing high mental workload and is potentially approaching cognitive overload. The algorithm estimator 112 can be configured to use a multimodal cognitive state estimation framework. The multimodal cognitive state estimation framework enables the algorithm estimator 112 to use a composite data fusion scheme to estimate human cognitive states such as mental workload, fatigue, and attention. The multimodal cognitive state estimation framework combines multiple interdependent estimation algorithms within the context of a probabilistic graphical model (PGM) algorithm. This approach allows the algorithm estimator 112 to leverage the strengths of different analytical techniques while maintaining a coherent probabilistic framework.

[0155] Method 600 includes determining a pilot's available attentional resources based on mental workload and mental fatigue in block 608. For example, Figure 1 The algorithm estimator 112 processes data 122 received from sensor 102 to determine the pilot's mental workload and fatigue. Using these determinations, the algorithm estimator can estimate the pilot's available attentional resources, thereby providing an assessment of the pilot's current cognitive capabilities.

[0156] Method 600 includes determining the pilot's attention allocation in block 610 based on available attentional resources and gaze patterns derived from physiological data. For example, Figure 1 The algorithm estimator 112 can use previously calculated available attention resources and gaze patterns derived from data 122A to determine how the pilots are allocating their attention.

[0157] Method 600 includes determining the pilot's situational awareness based on attention allocation and aircraft state data in block 612. For example, Figure 1 The algorithm estimator 112 can be configured to assess the pilot’s situational awareness by combining attention allocation data with aircraft state data.

[0158] Method 600 includes generating visualizations of the pilot's available attention resources, pilot attention allocation, pilot situational awareness, and aircraft state data in block 614. For example, Figure 1 The data visualization generator 118 can be configured to receive data 128 from the algorithm estimator 112 and generate output data 132, which includes dynamic, real-time visualizations that provide intuitive insights into the pilot's cognitive state and situational awareness. The data visualization generator 118 can be configured to create dynamic, real-time graphical representations of the pilot's cognitive state and aircraft parameters. These visualizations can include color-coded instruments, trend lines, or other intuitive formats that display the pilot's available attention resources, attention allocation, and situational awareness (alongside relevant aircraft state data). The output data 132 can be displayed via a display device 134 in these various readable formats, which allow for rapid interpretation of the pilot's current cognitive conditions relative to the flight situation.

[0159] Figure 7 It shows including Figure 1 and Figure 2 The flowchart illustrates an example 700 of the lifecycle of the aircraft for the pilot monitoring systems 100 and 200. During pre-production, the exemplary method 700 includes the specifications and design of the aircraft in block 702. During the specifications and design phase, method 700 may include the specifications and design of device 104 and the orientation where device 104 will be placed. In block 704, method 700 includes material procurement, which may include procuring materials for device 104 or procuring pre-assembled device 104.

[0160] During production, method 700 includes the manufacture of components and sub-assemblies in block 706 and the system integration of the aircraft in block 708. For example, method 700 may include the manufacture of components and sub-assemblies of device 104, the system integration of device 104 with the aircraft, or both. In block 710, method 700 includes the certification and delivery of the aircraft, and in block 712, putting the aircraft into service. Certification and delivery may include the certification of device 104 to put device 104 into service. During customer service, the aircraft may be scheduled for routine maintenance and servicing (which may also include modification, reconfiguration, refurbishment, etc.). In block 714, method 700 includes performing maintenance and servicing on the aircraft, which may include performing maintenance and servicing on device 104. For example, maintenance and servicing may include updating one or more algorithms used by the estimation algorithm, replacing one or more sensor interfaces 202, replacing one or more processors 108, or combinations thereof.

[0161] Each process of Method 700 may be performed or implemented by a systems integrator, a third party, and / or an operator (e.g., a customer). For the purposes of this specification, a systems integrator may include, but is not limited to, any number of aircraft manufacturers and major systems subcontractors; a third party may include, but is not limited to, any number of suppliers, subcontractors, and vendors; and an operator may be an airline, leasing company, military entity, service organization, etc.

[0162] The various aspects of this disclosure can be found in, for example Figure 8 The description is presented in the context of an example of an aircraft 800. Figure 8 In the example, aircraft 800 includes a fuselage 802 having multiple systems 804 and an interior 806. Examples of the multiple systems 804 include one or more of a propulsion system 808, an electrical system 810, an environmental system 812, a hydraulic system 814, and equipment 104. Any number of other systems may be included. Figure 8 In the example, aircraft 800 includes device 104 according to one or more aspects of this disclosure, such as Figures 1-7 The device 104 is partially included in the housing 802 and the interior 806. Furthermore, the device 104 utilizes a portion of the electrical system 810. For example, the device 104 may be powered by the electrical system 810.

[0163] Figure 9 This is a block diagram of a computing environment 900, which includes a computing device 910 configured to support aspects of computer-implemented methods and computer-executable program instructions (or code) according to the present disclosure. For example, the computing device 910 or a portion thereof may be configured to execute instructions to initiate, execute, or control references. Figures 1-8 One or more operations described.

[0164] The computing device 910 includes one or more processors 920. In some aspects, the processor 920 includes a processor 108, such as... Figures 1-8 The processor 920 is configured to communicate with system memory 930, one or more storage devices 940, one or more input / output interfaces 950, one or more communication interfaces 960, or any combination thereof. System memory 930 includes volatile memory devices (e.g., random access memory (RAM) devices), non-volatile memory devices (e.g., read-only memory (ROM) devices, programmable read-only memory, and flash memory) or both. System memory 930 stores operating system 932, which may include a basic input / output system for booting computing device 910 and a complete operating system enabling computing device 910 to interact with users, other programs, and other devices. System memory 930 stores system (program) data 936, such as data 122, data 124, data 128, data 206, data 302, data 308, data 402, data 412, data 508, data 512, processed data 126, processed data 208, refined data 504, output data 132, output data 210, output data 514, or combinations thereof.

[0165] System memory 930 includes one or more operating systems 932 and / or one or more application programs 934 (e.g., instruction sets) executable by processor 920. For example, one or more application programs 934 include instructions executable by processor 920 to start, control, or execute references. Figures 1-8 The description includes one or more operations, such as determining a pilot's mental workload and mental fatigue based on physiological data, determining a pilot's available attentional resources based on mental workload and mental fatigue, determining a pilot's attention allocation based on available attentional resources and gaze patterns derived from physiological data, determining a pilot's situational awareness based on attention allocation and flight data, and generating visualizations of the pilot's available attentional resources, the pilot's attention allocation, the pilot's situational awareness, and aircraft status data.

[0166] In a particular embodiment, system memory 930 includes a non-transitory computer-readable medium storing instructions that, when executed by processor 920, cause processor 920 to initiate, perform, or control operations to aid in the design of an object. These operations include determining pilot mental workload and fatigue based on physiological data; determining pilot available attentional resources based on mental workload and fatigue; determining pilot attention allocation based on available attentional resources and gaze patterns derived from physiological data; determining pilot situational awareness based on attention allocation and flight data; and generating visualizations of pilot available attentional resources, pilot attention allocation, pilot situational awareness, and aircraft status data.

[0167] One or more storage devices 940 include non-volatile storage devices, such as disks, optical disks, or flash memory devices. In a particular example, storage device 940 includes both removable and non-removable memory devices. Storage device 940 is configured to store an operating system, an image of the operating system, applications (e.g., one or more applications 934), and program data (e.g., program data 936). In a particular aspect, system memory 930, storage device 940, or both include tangible computer-readable media. In a particular aspect, one or more storage devices 940 are located external to computing device 910.

[0168] One or more input / output interfaces 950 enable computing device 910 to communicate with one or more input / output devices 970 to facilitate user interaction. For example, one or more input / output interfaces 950 may include sensor interface 202, display interface, input interface, or both. For example, input / output interface 950 is adapted to receive input from a user, input from another computing device, or a combination thereof. In some embodiments, input / output interface 950 conforms to one or more standard interface protocols, including serial interfaces (e.g., Universal Serial Bus (USB) interfaces or Institute of Electrical and Electronics Engineers (IEEE) interface standards), parallel interfaces, display adapters, audio adapters, or custom interfaces (“IEEE” is a registered trademark of Institute of Electrical and Electronics Engineers, Inc., Piscataway, New Jersey). In some embodiments, input / output device 970 includes one or more user interface devices and displays, including some combination of buttons, keyboards, pointing devices, displays, speakers, microphones, touchscreens, and other devices.

[0169] The processor 920 is configured to communicate with the device or controller 980 via one or more communication interfaces 960. For example, the one or more communication interfaces 960 may include a network interface. In another example, the one or more devices or controllers 980 include a sensor 102.

[0170] In some implementations, a non-transitory computer-readable medium stores instructions that, when executed by one or more processors, cause one or more processors to initiate, execute, or control operations to perform some or all of the functions described above. For example, the instructions may execute to implement Figures 1-6 One or more of the operations or methods. In some implementations, Figures 1-6One or more operations or methods may be implemented, in whole or in part, by one or more processors executing instructions (e.g., one or more central processing units (CPUs), one or more graphics processing units (GPUs), one or more digital signal processors (DSPs)), dedicated hardware circuitry systems, or any combination thereof).

[0171] Specific aspects of this disclosure are described below in the form of a group of interconnected examples: According to Example 1, a system for determining a pilot's cognitive state includes one or more processors coupled to a memory, the processors being configured to receive physiological data and aircraft status data; determine the pilot's mental workload and mental fatigue based on the physiological data; determine the pilot's available attention resources based on the mental workload and mental fatigue; determine the pilot's attention allocation based on the available attention resources and a gaze pattern derived from the physiological data; determine the pilot's situational awareness based on the attention allocation and the aircraft status data; and generate visualizations of the pilot's available attention resources, the pilot's attention allocation, the pilot's situational awareness, and the aircraft status data.

[0172] Example 2 includes the system according to Example 1, and further includes a plurality of sensors configured to collect the physiological data from the pilot, wherein the plurality of sensors include one or more of an eye tracker, an ambient light sensor, or a heart rate monitor.

[0173] Example 3 includes a system according to Example 1 or Example 2, wherein the determination of the mental workload and the mental fatigue is achieved by combining the outputs of multiple Kalman filters using a Gaussian mixture model (GMM), each filter representing a different hypothesis about the pilot's cognitive state.

[0174] Example 4 includes the system according to Example 3, wherein the GMM is configured to combine the weighted outputs of multiple Kalman filters, wherein the weights of each filter are dynamically adjusted based on the physiological data and the aircraft state data.

[0175] Example 5 includes a system according to any one of Examples 1 to 5, wherein the one or more processors are further configured to use a probabilistic graphical model (PGM) algorithm to track the pilot’s current knowledge of flight variables.

[0176] Example 6 includes the system according to Example 5, wherein the PGM algorithm models the decisions of the pilot's visual inspection instrument as events in a fully observable recursive Markov chain.

[0177] Example 7 includes a system according to any one of Examples 1 to 6, wherein the visualization includes the pilot's mental workload and mental fatigue, the pilot's available attention resources, the pilot's attention allocation, the pilot's situational awareness, and real-time updates of relevant aircraft parameters associated with the aircraft state data.

[0178] Example 8 includes a system according to any one of Examples 1 to 7, wherein one or more processors are configured to determine the situational awareness, which includes comparing the pilot’s attention allocation with critical flight information.

[0179] According to Example 9, a method for determining a pilot's cognitive state includes receiving physiological data from a pilot from multiple sensors; receiving aircraft state data from an aircraft state data generator; determining the pilot's mental workload and mental fatigue based on the physiological data; determining the pilot's available attention resources based on the mental workload and mental fatigue; determining the pilot's attention allocation based on the available attention resources derived from the physiological data and a gaze pattern; determining the pilot's situational awareness based on the attention allocation and the aircraft state data; and generating visualizations of the pilot's available attention resources, the pilot's attention allocation, the pilot's situational awareness, and the aircraft state data.

[0180] Example 10 includes the method according to Example 9, further comprising preprocessing the physiological data by performing one or more of the following: removing NAN values, applying bandpass filtering, rejecting outliers, or performing linear interpolation on missing values.

[0181] Example 11 includes the method according to Example 9 or Example 10, wherein determining the mental workload and mental fatigue includes: using multiple Kalman filters to track different hypotheses about the pilot's cognitive state to generate one or more outputs; combining the one or more outputs of these Kalman filters using a Gaussian mixture model (GMM), wherein the output of each Kalman filter is represented as a Gaussian component; and dynamically adjusting one or more weights of the Gaussian components based on the physiological data and aircraft state data.

[0182] Example 12 includes the method according to any one of Examples 9 to 11, and further includes using a probabilistic graphical model (PGM) algorithm to track the pilot’s current knowledge of flight variables.

[0183] Example 13 includes the method according to Example 12, wherein the PGM algorithm includes modeling the decisions of the pilot's visual inspection instrument as events in a fully observable recursive Markov chain.

[0184] Example 14 includes the method according to any one of Examples 9 to 13, wherein generating the visualization includes generating a real-time graphical representation of the pilot's available attention resources, the pilot's attention allocation, the pilot's situational awareness, and related aircraft parameters associated with the aircraft state data.

[0185] Example 15 includes the method according to any one of Examples 9 to 14, wherein determining the situational awareness includes comparing the pilot's attention allocation with critical flight information.

[0186] According to Example 16, a non-transitory computer-readable medium stores instructions that, when executed by one or more processors, cause the processors to determine a pilot's mental workload and mental fatigue based on physiological data; determine the pilot's available attentional resources based on the mental workload and mental fatigue; determine the pilot's attentional allocation based on the available attentional resources and a gaze pattern derived from the physiological data; determine the pilot's situational awareness based on the attentional allocation and flight data; and generate visualizations of the pilot's available attentional resources, the pilot's attentional allocation, the pilot's situational awareness, and aircraft status data.

[0187] Example 17 includes a non-transitory computer-readable medium according to Example 16, wherein one or more processors are configured to use at least one Kalman filter to estimate the pilot’s mental workload and mental fatigue; combine the outputs of the at least one Kalman filter using a Gaussian mixture model (GMM), wherein the outputs represent weighted Gaussian components; and adjust the weights of the Gaussian components in real time based on the physiological data and aircraft state data.

[0188] Example 18 includes a non-transitory computer-readable medium according to Example 16 or Example 17, wherein one or more processors are configured to implement a probabilistic graphical model (PGM) algorithm to track the pilot’s current knowledge of flight variables.

[0189] Example 19 includes a non-transitory computer-readable medium according to any one of Examples 16 to 18, wherein one or more processors are configured to provide a real-time graphical representation of the pilot's available attention resources, the pilot's attention allocation, the pilot's situational awareness, and related aircraft parameters associated with the aircraft state data.

[0190] Example 20 includes a non-transitory computer-readable medium according to any one of Examples 16 to 19, wherein one or more processors are configured to determine the situational awareness, which includes comparing the pilot’s attention allocation with critical flight information.

[0191] The illustrations described herein are intended to provide a general understanding of the structure of various embodiments. These illustrations are not intended as a complete description of all elements and features of apparatuses and systems utilizing the structures or methods described herein. Many other embodiments may become apparent to those skilled in the art upon review of this disclosure. Other embodiments may be utilized and derived from this disclosure, allowing for structural and logical substitutions and changes without departing from the scope of this disclosure. For example, method operations may be performed in a different order than those shown in the drawings, or one or more method operations may be omitted. Therefore, this disclosure and the drawings should be considered illustrative rather than restrictive.

[0192] Furthermore, although specific examples have been shown and described herein, it should be understood that any subsequent arrangements designed to achieve the same or similar results may replace the specific embodiments shown. This disclosure is intended to cover any and all subsequent adaptations or variations of the various embodiments. Combinations of the above embodiments and other embodiments not specifically described herein will be apparent to those skilled in the art upon review of this specification.

[0193] The abstract of this disclosure is intended to be understood as such that it is not to be construed as limiting the scope or meaning of the claims. Furthermore, in the specific embodiments described above, various features may be combined together or described in a single embodiment for the purpose of simplification. The examples above are illustrative but not limiting of this disclosure. It should also be understood that many modifications and variations are possible based on the principles of this disclosure. As reflected in the following claims, the claimed subject matter may apply to fewer than all features of any of the disclosed examples. Therefore, the scope of this disclosure is defined by the appended claims and their equivalents.

Claims

1. A system (100, 200) for determining the cognitive state of a pilot, comprising: One or more processors (108, 920) coupled to a memory (110, 930), said one or more processors (108, 920) being configured to: Receive physiological data (122) and spacecraft status data (124); Based on the aforementioned physiological data, the mental workload and mental fatigue of the pilots were determined (128, 208, 308, 606). Based on the aforementioned mental workload and mental fatigue, determine (128, 208, 308, 608) the pilot's available attention resources; The pilot's attention allocation is determined (128, 208, 308, 610) based on the available attention resources and the gaze patterns derived from the physiological data. The pilot's situational awareness is determined (128, 208, 308, 612) based on the attention allocation and the aircraft status data. and Generate (132, 614) visualizations of the pilot's available attention resources, the pilot's attention allocation, the pilot's situational awareness, and the aircraft state data.

2. The system of claim 1 further includes a plurality of sensors (102) configured to collect the physiological data from the pilot, wherein the plurality of sensors include one or more of the following: Eye tracker (102A). Ambient light sensor (102B), or Heart rate monitor (102C).

3. The system according to claim 1, wherein, The determination of the mental workload and the mental fatigue uses a Gaussian mixture model, or GMM (112, 310, 414), to combine the outputs from multiple Kalman filters, each representing a different hypothesis about the pilot's cognitive state.

4. The system of claim 3, wherein the GMM is configured to combine a weighted output of a plurality of Kalman filters (406), wherein the weight of each filter is dynamically adjusted based on the physiological data and the aircraft state data.

5. The system according to claim 1, wherein, The one or more processors are also configured to use a probabilistic graphical modeling algorithm, namely the PGM algorithm (112, 310, 414), to track the pilot’s current knowledge of flight variables.

6. The system of claim 5, wherein the PGM algorithm models the decisions of the pilot visual inspection instrument as events in a fully observable recursive Markov chain.

7. The system of claim 1, wherein the visualization includes the pilot's mental workload and mental fatigue, the pilot's available attention resources, the pilot's attention allocation, the pilot's situational awareness, and real-time updates of relevant aircraft parameters associated with the aircraft status data.

8. The system of claim 1, wherein the one or more processors are configured to determine the situational awareness, including comparing the pilot’s attention allocation with critical flight information.

9. A method for determining a pilot's cognitive state, comprising: Physiological data (122) from the pilot are received (602) from multiple sensors (102); Receive (604) aircraft status data (124) from the aircraft status data generator (114); Based on the physiological data, the mental workload and mental fatigue of the pilots were determined (128, 208, 308, 606). The available attentional resources of the pilots are determined (128, 208, 308, 608) based on the mental workload and mental fatigue. The pilot's attention allocation (128, 208, 308, 610) is determined based on the available attention resources and the gaze patterns derived from the physiological data. The pilot's situational awareness is determined (128, 208, 308, 612) based on the attention allocation and the aircraft status data. and Generate (132, 614) visualizations of the pilot's available attention resources, the pilot's attention allocation, the pilot's situational awareness, and the aircraft state data.

10. The method of claim 9 further comprises preprocessing (202) the physiological data by performing one or more of the following: removing NAN values, applying bandpass filtering, removing outliers, or performing linear interpolation of missing values.

Images