Pilot attention assessment system and method based on multi-modal physiological signal fusion

Abstract

The invention discloses a pilot attention assessment system and method based on multi-mode physiological signal fusion. The system collects EEG, EOG and simulator data, and microsecond-level synchronization is achieved through a precise time protocol; iCA, 0.5-45 Hz filtering and baseline correction are carried out on the EEG, frequency band energy, sample entropy and LZ complexity are extracted, fixation / glancing segmentation and DBSCAN clustering are carried out on the EOG to form a region of interest, and stability features are calculated. And the double-flow time sequence network introduces cross attention between the second fusion layer and the fourth fusion layer to realize cross-modal dynamic registration, and outputs five-level attention and the operation risk in the next three seconds. ATI indexes are constructed, adaptive weighting is carried out according to flight phases, and a gaze thermodynamic diagram is generated by combining kernel density estimation and used for prompting and training feedback. According to the scheme, the state of the pilot is accurately evaluated in real time, and the method has good robustness and generalization ability.

Description

Technical Field

[0001] This invention belongs to the field of assessment system technology, and in particular relates to a pilot attention assessment system and method based on multimodal physiological signal fusion. Background Technology

[0002] Pilot attention is directly related to flight safety. Existing assessments often rely on single modalities (such as EEG or EOG only) or external indicators like behavioral / control inputs, making them susceptible to noise, artifacts, and individual differences, resulting in insufficient stability and transferability. Existing multimodal methods often employ rule-based thresholding or simple early / late stage splicing and fusion, lacking fine-grained registration across modal time series and channel context, making it difficult to generate continuous and reliable real-time quantification results.

[0003] In flight simulation training scenarios, physiological signals and simulator parameters often have different sampling frequencies and inconsistent clocks, leading to sample-level alignment errors. Insufficient EEG preprocessing (only filtering / power spectrum) and coarse EOG region of interest division result in inadequate feature representation. Evaluation metrics also lack adaptive weights and visual feedback for stages such as "takeoff / landing, cruise, and emergency handling." The scenario complexity and adversarial disturbances of the training data are not adequately considered, resulting in poor robustness and generalization ability of the model in complex situations. Summary of the Invention

[0004] In view of this, the present invention aims to propose a pilot attention assessment system and method based on multimodal physiological signal fusion, so as to at least solve one of the problems in the background art.

[0005] To achieve the above objectives, the technical solution of the present invention is implemented as follows: A pilot attention assessment system based on multimodal physiological signal fusion includes: a multi-source data acquisition module for simultaneously acquiring the pilot's EEG, EOG, and flight simulator control inputs and instrument readings, wherein the EEG is an electroencephalogram signal and the EOG is an eye movement signal; The time synchronization module is used to perform microsecond-level time synchronization of the EEG, EOG and simulator data using a precision time protocol; The preprocessing and feature extraction module is used to perform independent component analysis to remove artifacts, bandpass filtering and baseline correction on EEG and extract frequency band energy, complexity and entropy features; and to perform gaze / saccade segmentation and outlier removal on EOG and generate a set of regions of interest. The multimodal fusion evaluation module adopts a dual-stream temporal network that includes EEG and EOG branches, and introduces cross-attention with dynamic weights between preset fusion layers to achieve cross-modal feature registration and fusion, and outputs the pilot's attention level and short-term operational risk probability. The indicator calculation and visualization module is used to calculate comprehensive evaluation indicators based on EEG concentration, EOG stability and operational responsiveness, and to map the gaze point to the cockpit projection plane to generate a heat map for prompts and training feedback.

[0006] Furthermore, the EEG acquisition in the multi-source data acquisition module adopts a 32-lead dry electrode array with a sampling rate of not less than 1000Hz; the preprocessing and feature extraction module performs 0.5-45Hz bandpass filtering and resting baseline correction on the EEG, and extracts the energy proportions of Delta, Theta, Alpha, Beta and Gamma frequency bands by wavelet packet decomposition, and forms EEG feature vectors by combining sample entropy and Lempel-Ziv complexity.

[0007] Furthermore, the preprocessing and feature extraction module uses a density-based clustering method to generate the region of interest set of EOGs and merges clusters with a center distance less than a threshold; the gaze stability index is calculated based on the covariance matrix of coordinate points during the gaze period and Mahalanobis distance, and combined with the standard deviation of saccade velocity to form the EOG stability feature.

[0008] Furthermore, the EEG branch of the multimodal fusion evaluation module includes a structure of multi-scale parallel temporal convolution and pooling, and the EOG branch includes spatial attention and temporal attention concatenated and combined with a bidirectional recurrent network; the cross attention is set between the second and fourth fusion layers, and the channels and temporal context of the two branches are dynamically weighted to achieve cross-modal feature registration.

[0009] Furthermore, the output of the multimodal fusion evaluation module includes five levels of attention and the probability of operational risk in the next m seconds.

[0010] Furthermore, the indicator calculation and visualization module constructs a comprehensive evaluation index ATI, which is obtained by weighting EEG concentration, EOG stability and operational responsiveness. The weights are adaptively adjusted according to flight phases such as takeoff / landing, cruise and emergency handling. Kernel density estimation is used to generate a gaze heatmap and the weights are enhanced by combining the operational sequence.

[0011] Furthermore, it also includes a model training module, which builds a benchmark library using data from experienced pilots in typical scenarios, gradually increases the complexity of scenarios through course learning, and introduces adversarial examples for training to improve the robustness and generalization ability of the model.

[0012] Furthermore, this scheme discloses a pilot attention assessment method based on multimodal physiological signal fusion, including: S1. Synchronously collect EEG, EOG and flight simulator data, and complete time synchronization through a precision time protocol; S2. Perform independent component analysis on the EEG to remove artifacts, bandpass filtering and baseline correction, and extract frequency band energy, complexity and entropy features; S3. Perform gaze / saccade segmentation and density clustering on the EOG to generate a set of regions of interest, and calculate the standard deviation of saccade velocity and gaze stability index; S4. Input EEG and EOG features into a dual-stream temporal network, and achieve cross-modal feature registration and dynamic weighted fusion through cross attention between preset fusion layers to obtain attention level and operational risk probability in the next 3 seconds. S5. Calculate ATI based on EEG concentration, EOG stability and operational responsiveness, and adaptively adjust the weights according to the flight phase. At the same time, map the gaze point to the cockpit projection plane to generate a heat map for prompting and training feedback.

[0013] Furthermore, the set of regions of interest in S3 is generated using the DBSCAN algorithm, and clusters with a center distance less than a preset threshold are merged; the EEG branch in S4 adopts a multi-scale parallel temporal convolutional structure, the EOG branch adopts a bidirectional recurrent network structure combining spatial attention and temporal attention, and cross attention is set between the second and fourth fusion layers to achieve dynamically weighted cross-modal feature registration in the channel and time dimensions.

[0014] Furthermore, this solution discloses a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, is used to implement the method steps.

[0015] Compared with existing technologies, the pilot attention assessment system and method based on multimodal physiological signal fusion described in this invention have the following advantages: (1) The pilot attention assessment system based on multimodal physiological signal fusion described in this invention significantly improves the accuracy, real-time performance and noise resistance of attention assessment by achieving microsecond-level synchronization of EEG, EOG and flight parameters, as well as robust features and dual-stream cross-attention fusion. (2) The pilot attention assessment system based on multimodal physiological signal fusion described in this invention outputs five levels of attention and risk in the next 3 seconds. ATI adaptively weights the data according to the flight phase and combines it with gaze heatmap feedback. Combined with course and adversarial training, it enhances robustness and generalization. Attached Figure Description

[0016] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an undue limitation of the invention. In the drawings: Figure 1 This is a schematic diagram of the system architecture described in an embodiment of the present invention; Figure 2This is a schematic diagram of the dual-stream neural network structure described in an embodiment of the present invention. Detailed Implementation

[0017] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other.

[0018] In the description of this invention, it should be understood that the terms "center," "longitudinal," "lateral," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer," etc., indicating orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, are only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the invention. Furthermore, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined with "first," "second," etc., may explicitly or implicitly include one or more of that feature. In the description of this invention, unless otherwise stated, "a plurality of" means two or more.

[0019] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art will understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0020] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0021] This embodiment presents the overall structure and deployment of a pilot attention assessment system based on multimodal physiological signal fusion, which can operate in a flight training environment. The system deploys typical and special training scenarios (such as instrument malfunction, sudden weather changes, and emergency avoidance) within a Level D full-motion simulator. It synchronously acquires pilot EEG, EOG, and flight simulator inputs and instrument readings via a multi-source data acquisition module. The EEG uses a 32-lead dry electrode array (sampling rate ≥1000Hz), while the EOG acquires three parameters: pupil diameter, fixation point coordinates, and saccade velocity. The flight parameter interface records real-time data on stick inputs and instrument deviations. To ensure cross-modal data alignment, the time synchronization module uses Precision Time Protocol (PTP) to achieve microsecond-level time synchronization and timestamps the data at the acquisition side for subsequent fusion processing. All data streams are accessed and processed in real-time by the system's feature processing and fusion assessment link, providing basic input for training feedback and risk alerts.

[0022] This solution employs a three-tiered evaluation system. The first layer is a multi-source data acquisition layer, including wearable EEG acquisition, eye tracking, and interfaces for flight simulator manipulation and instrument readings, achieving microsecond-level time synchronization through a precise time protocol. The second layer is a feature processing layer. On the EEG side, it performs 1Hz high-pass preprocessing, independent component analysis for artifact removal, zero-phase bandpass filtering, and resting baseline correction; on the eye tracking side, it performs fixation and saccade segmentation, outlier removal, and density-based region of interest (AOI) generation. The third layer is a deep fusion evaluation and visualization layer. A dual-stream temporal network outputs attention levels and short-term risk probabilities, with heatmaps and alarm information overlaid in real-time on the interface to support training intervention and review.

[0023] For EEG signals, the preprocessing and feature extraction module first performs independent component analysis (ICA) for artifact removal, bandpass filtering, and resting baseline correction: after a 1Hz high-pass preprocessing of the original EEG, FastICA decomposition is performed to identify and remove artifact components such as EOG / EMG / ECG, and reconstruct a clean EEG; then, a 0.5–45Hz zero-phase Butterworth bandpass filter is used to suppress low-frequency drift and high-frequency noise, and the baseline is corrected using the resting segment with eyes closed before the experiment to form a stable time-series waveform.

[0024] Based on this, the module uses wavelet packet decomposition (WPD, db4, 5 layers) to calculate the energy proportion of each frequency band (Delta / Theta / Alpha / Beta / Gamma). It further extracts nonlinear dynamic features such as sample entropy (SE, m=2, r=0.2·std) and Lempel-Ziv complexity (LZC, binarized and normalized by median threshold) to jointly construct EEG feature vectors for subsequent time series modeling and staged attention measurement.

[0025] For EOG signals, the preprocessing and feature extraction module first performs gaze / saccade segmentation and outlier removal (blink and coordinate abrupt change removal, velocity threshold suppression). Then, DBSCAN density clustering is used to generate a set of regions of interest (AOIs), and nearest neighbor clusters with a center distance less than the threshold are merged to stabilize the AOI boundaries. In terms of dynamic features, the standard deviation of saccade velocity is calculated to characterize the fluctuation of visual search load, and the gaze stability index (FSI) is constructed based on the covariance matrix of coordinate points during the gaze period and Mahalanobis distance to characterize the micro-motion amplitude during a single gaze. The two together constitute the EOG stability feature. The system also provides the normal range and warning threshold of the standard deviation of saccade velocity and FSI, which facilitates anomaly detection and prompting during the evaluation stage.

[0026] The multimodal fusion evaluation module adopts a dual-stream temporal network structure including EEG and EOG branches: the EEG branch is a cascaded Inception-Time module of multi-scale parallel temporal convolution and pooling, with 32-channel EEG waveforms as input; the EOG branch concatenates spatial attention and temporal attention and is combined with a bidirectional recurrent network (Bi-LSTM) to jointly model the correlation between eye-tracking spatial thermodynamics and temporal series. For cross-modal alignment, this embodiment introduces a dynamic weighted cross-attention mechanism between the second and fourth fusion layers, with the weight calculation formula as follows: in, This represents the correlation strength between the features of the i-th EEG / EOG time step and the features of the j-th EOG / EEG time step.

[0027] This represents a feature query derived from EEG / EOG signals. It originates from the output feature map of the EEG / EOG branch at a specific fusion layer.

[0028] This represents an index of all key information derived from eye-tracking / electroencephalography (EEG) signals. It originates from the output features of the EOG / EEG branches at the corresponding fusion layer.

[0029] Forward fusion (second layer): The fused features are fed into the next layer of the EEG branch network. The aim is to use eye-tracking information to enhance and supplement the contextual awareness of EEG features.

[0030] Backward fusion (fourth layer): The fused features are fed into the next layer of the EOG branch. The purpose is to enable eye-tracking features to dynamically reference the state of brain activity during evolution.

[0031] By adaptively weighting the channels and temporal context of the two branches, fine-grained registration of low-frequency EEG and high-frequency EOG behaviors is achieved. The model output provides five levels of attention (e.g., hyperfocus / normal / distracted / fatigued / abnormal) and the probability of operational risk in the next 3 seconds, which can be used for short-term risk warning and instructor intervention reference.

[0032] In terms of indicator calculation and visualization, the system constructs a comprehensive evaluation indicator, ATI, which is obtained by weighting EEG concentration, EOG stability, and operational responsiveness. in, Weights for EEG concentration; Calculation based on the energy percentage of Beta waves (13-30Hz) in EEG signals: ; β is the weight for EOG stability; Combined gaze stability index (FSI) and AOI switching frequency: Weights for operational responsiveness; Calculate the time deviation between the operation delay and the standard action ( ) and accuracy (error rate) ): The dynamic weight adjustment strategy is as follows: 1. Based on the flight phase: 2. Based on real-time risk If the risk prediction value is greater than 0.7, it automatically switches to emergency weight (0.2, 0.1, 0.7) and uses a fuzzy logic controller to dynamically fine-tune the weight based on multi-sensor input.

[0033] ATI's grading threshold: ATI range Rating Training suggestions 0.8-1.0 excellent Maintain the current state and try complex scenarios. 0.6-0.8 qualified Strengthen attention allocation training ＜0.6 Intervention is needed Immediately pause training and conduct cognitive load analysis. The weights can be adaptively adjusted based on flight phases (takeoff / landing, cruise, emergency handling) and real-time risk predictions; EEG concentration can be dynamically standardized by the energy proportion of Beta waves (13–30Hz) combined with individual baselines; EOG stability is jointly characterized by the switching frequency of FSI and AOI; and operational responsiveness is given by the standard action time deviation and accuracy. The visualization module maps the gaze point to the cockpit projection plane and uses kernel density estimation (KDE) to generate a gaze heatmap; when the time deviation between the current operation and the standard action is less than a threshold, the relevant areas are weighted and enhanced, while high-risk areas predicted by the model are overlaid for prominent indication. The interface provides components such as a main window, a real-time data panel, a historical comparison module, and alarm prompt boxes. When ATI or sub-indicators are abnormal, voice / text alarms are triggered for immediate training intervention and review.

[0034] To enhance the model's robustness and generalization ability, this embodiment sets up a model training module to construct a benchmark library of "experienced pilots - typical scenarios." Multimodal data from over 100 experienced pilots in typical scenarios such as instrument malfunctions, sudden weather changes, and emergency evasive maneuvers are collected. After standardized preprocessing and feature extraction, a knowledge graph mapping "scenario-attention pattern" is established. During training, the complexity of the scenarios is gradually increased through a course-based learning approach, and adversarial examples and real data are introduced for mixed training, ensuring the evaluation model maintains stable output under complex and unexpected situations. At the deployment level, the system utilizes a high-performance embedded platform for algorithm acceleration and code optimization, keeping the overall inference and feature processing latency below 50ms. It can also integrate a biofeedback module with a VR training system for real-time closed-loop training, helping trainees adjust their state and improve training effectiveness.

[0035] The corresponding implementation process is as follows: First, before the simulator training task begins, the sensors are put online and the PTP clock is calibrated. After entering the scene, EEG, EOG, and flight parameters are collected synchronously (S1). Then, ICA artifact removal, 0.5–45Hz bandpass filtering, and resting baseline correction are performed on the EEG, and features such as frequency band energy proportion, sample entropy, and LZ complexity are extracted (S2). In parallel, gaze / saccade segmentation and outlier removal are performed on the EOG. AOI sets and cluster merging are generated using DBSCAN, and the standard deviation of saccade velocity and gaze stability are calculated. Stability features such as the stability index (S3); EEG and EOG features are input into a dual-stream temporal network, and cross-modal feature registration and dynamic weighted fusion are achieved through cross-attention in the second and fourth fusion layers, outputting five levels of attention and the probability of operational risk in the next 3 seconds (S4); finally, ATI is calculated based on EEG concentration, EOG stability and operational responsiveness, and the weights are adaptively adjusted according to the flight phase. At the same time, the gaze point is mapped to the cockpit projection plane to generate a heat map using KDE, so as to realize intuitive prompts and training feedback on attention distribution and risk areas (S5).

[0036] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A pilot attention assessment system based on multimodal physiological signal fusion, characterized in that, include: The multi-source data acquisition module is used to simultaneously acquire the pilot's EEG, EOG, and the flight simulator's control inputs and instrument readings. The EEG is an electroencephalogram (EEG) signal, and the EOG is an eye movement signal. The time synchronization module is used to perform microsecond-level time synchronization of the EEG, EOG and simulator data using a precision time protocol; The preprocessing and feature extraction module is used to perform independent component analysis to remove artifacts, bandpass filtering and baseline correction on EEG and extract frequency band energy, complexity and entropy features; and to perform gaze / saccade segmentation and outlier removal on EOG and generate a set of regions of interest. The multimodal fusion evaluation module adopts a dual-stream temporal network that includes EEG and EOG branches, and introduces cross-attention with dynamic weights between preset fusion layers to achieve cross-modal feature registration and fusion, and outputs the pilot's attention level and short-term operational risk probability. The indicator calculation and visualization module is used to calculate comprehensive evaluation indicators based on EEG concentration, EOG stability and operational responsiveness, and to map the gaze point to the cockpit projection plane to generate a heat map for prompts and training feedback.

2. The system according to claim 1, characterized in that, The EEG acquisition in the multi-source data acquisition module uses a 32-lead dry electrode array with a sampling rate of not less than 1000Hz; the preprocessing and feature extraction module performs 0.5-45Hz bandpass filtering and resting baseline correction on the EEG, and extracts the energy proportions of Delta, Theta, Alpha, Beta and Gamma frequency bands by wavelet packet decomposition, and forms EEG feature vectors by combining sample entropy and Lempel-Ziv complexity.

3. The system according to claim 1, characterized in that, The preprocessing and feature extraction module uses a density-based clustering method to generate the region of interest set of EOG and merges clusters with a center distance less than a threshold. The gaze stability index is calculated based on the covariance matrix of coordinate points during the gaze period and Mahalanobis distance, and combined with the standard deviation of saccade velocity to form the EOG stability feature.

4. The system according to claim 1, characterized in that, The EEG branch of the multimodal fusion evaluation module includes a structure of multi-scale parallel temporal convolution and pooling, while the EOG branch includes a combination of spatial attention and temporal attention in a bidirectional recurrent network. The cross attention is set between the second and fourth fusion layers to dynamically weight the channels and temporal context of the two branches to achieve cross-modal feature registration.

5. The system according to claim 1, characterized in that, The output of the multimodal fusion evaluation module includes five levels of attention and the probability of operational risk in the next m seconds.

6. The system according to claim 1, characterized in that, The indicator calculation and visualization module constructs a comprehensive evaluation index ATI, which is obtained by weighting EEG concentration, EOG stability and operational responsiveness. The weights are adaptively adjusted according to flight phases such as takeoff / landing, cruise and emergency handling. Kernel density estimation is used to generate a gaze heatmap and the weights are enhanced by combining the operational sequence.

7. The system according to claim 1, characterized in that, It also includes a model training module, which builds a benchmark library using data from experienced pilots in typical scenarios, gradually increases the complexity of scenarios through course learning, and introduces adversarial examples for training to improve the robustness and generalization ability of the model.

8. A pilot attention assessment method based on multimodal physiological signal fusion, characterized in that, include: S1. Synchronously collect EEG, EOG and flight simulator data, and complete time synchronization through a precision time protocol; S2. Perform independent component analysis on the EEG to remove artifacts, bandpass filtering and baseline correction, and extract frequency band energy, complexity and entropy features; S3. Perform gaze / saccade segmentation and density clustering on the EOG to generate a set of regions of interest, and calculate the standard deviation of saccade velocity and gaze stability index; S4. Input EEG and EOG features into a dual-stream temporal network, and achieve cross-modal feature registration and dynamic weighted fusion through cross attention between preset fusion layers to obtain attention level and operational risk probability in the next 3 seconds. S5. Calculate ATI based on EEG concentration, EOG stability and operational responsiveness, and adaptively adjust the weights according to the flight phase. At the same time, map the gaze point to the cockpit projection plane to generate a heat map for prompting and training feedback.

9. The method according to claim 8, characterized in that, The region of interest set in S3 is generated using the DBSCAN algorithm, and clusters with a center distance less than a preset threshold are merged. The EEG branch in S4 adopts a multi-scale parallel temporal convolutional structure, and the EOG branch adopts a bidirectional recurrent network structure combining spatial attention and temporal attention. Cross attention is set between the second and fourth fusion layers to achieve dynamic weighted cross-modal feature registration in the channel and time dimensions.

10. A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is used to implement the steps of the method of claim 8 or 9.

Images