Multimodal Intelligent Monitoring and Decision Support System and Method for Flight Training

Abstract

This invention relates to the fields of aviation flight training, human factors engineering, and intelligent decision support. Specifically, it discloses a multimodal intelligent monitoring and decision support system and method for flight training, comprising: a multimodal physiological data acquisition module for non-invasive real-time acquisition of pilot physiological signals; a flight operation and situation data synchronization module for acquiring aircraft status, control inputs, and external environmental data; a data fusion and feature extraction module for synchronously processing data and extracting fused feature vectors; an intelligent assessment and decision support model library for outputting status assessments, operational trend predictions, and decision support information based on machine learning algorithms; and a personalized decision support and feedback module for providing real-time feedback or decision suggestions to pilot and instructor terminals. This invention enables objective, quantitative, and real-time monitoring of pilots' physiological and psychological states, provides intelligent decision support, and improves training efficiency and flight safety.

Description

Technical Field

[0001] This invention relates to the fields of aviation flight training, human factors engineering, and intelligent auxiliary decision-making technology, specifically to a multimodal intelligent monitoring and auxiliary decision-making system and method for flight training. Background Technology

[0002] Traditional flight training relies heavily on the subjective observation and experience of flight instructors. Instructors assess a pilot's skill level, condition, and decision-making quality through their external performance, operational results, and verbal communication. This approach has the following inherent drawbacks: It is highly subjective and lacks objective quantitative data: it is difficult to accurately assess the pilot's cognitive load, attention allocation, situational awareness and other internal psychological states.

[0003] Delayed feedback: Debriefing is usually conducted after the flight, making it difficult for pilots to accurately recall their instantaneous psychological state and decision-making details during the flight.

[0004] Special situation training is costly and risky: simulation training for high-difficulty faults or emergencies has high requirements for equipment and safety assurance, and it is difficult to dynamically adjust according to the individual conditions of trainees.

[0005] Insufficient monitoring of fatigue and condition: mainly relying on pilot self-reporting and instructor observation, lacking an effective physiological indicator early warning mechanism.

[0006] Currently, although some studies have introduced single physiological signals (such as heart rate) into training assessment, there are problems such as limited information dimensions, susceptibility to interference, and weak correlation with specific flight missions, which cannot comprehensively and accurately reflect the multidimensional state of pilots in complex flight environments.

[0007] In summary, existing aviation flight training, human factors engineering, and intelligent auxiliary decision-making technologies suffer from problems such as limited information dimensions, susceptibility to interference, and weak correlation with specific flight missions. There is an urgent need for a comprehensive, high-precision, and real-time intelligent monitoring and auxiliary decision-making system, which aims to achieve objective and refined assessment and intelligent intervention of pilot training status. Summary of the Invention

[0008] To address the aforementioned problems in the prior art, this invention provides a multimodal intelligent monitoring and auxiliary decision-making system and method for flight training, which can comprehensively and accurately reflect the multidimensional state of pilots in complex flight environments.

[0009] To achieve the above objectives, this invention proposes a multimodal intelligent monitoring and auxiliary decision-making system for flight training, comprising: A multimodal physiological data acquisition module is used for non-invasive real-time acquisition of pilots' physiological signals, including electroencephalogram (EEG) signals, electrocardiogram (ECG) signals, respiratory signals, skin conductance signals, eye movement signals, and speech signals. The flight operation and situation data synchronization module is used to acquire aircraft status parameters, pilot control inputs, and external environment data in real time from the flight simulator or airborne bus. The external environment data includes meteorological environment data, airspace environment data, site environment data, and special situation environment data directly related to flight training. Meteorological environment data includes visibility, runway visual range, cloud height, wind speed, wind direction, precipitation type and intensity. Airspace environment data includes flight airspace type, number and location of aircraft in the airspace, and air traffic control instructions. Site environment data includes runway status, airport airspace clearance conditions, and ground navigation facility operating status. Special situation environment data includes airflow disturbances under special situation simulation and external environment change parameters such as navigation environment missing conditions. The data fusion and feature extraction module is used to receive and synchronously process the data output by the multimodal physiological data acquisition module and the flight operation and situation data synchronization module, and extract a fusion feature vector for evaluating the pilot's state. The fusion feature vector is a 64-dimensional situation-physiological-operation fusion feature vector, which is constructed by dividing the data into 16-dimensional flight situation features, 32-dimensional physiological features and 16-dimensional operation features. The intelligent assessment and auxiliary decision-making model library is used to process the fused feature vector based on machine learning algorithms to obtain pilot status assessment results, operational trend prediction results, and auxiliary decision-making information. The intelligent assessment and auxiliary decision-making model library includes a multi-dimensional pilot status assessment model, an operational deviation prediction model, and a special situation handling decision tree generation model. The multi-dimensional pilot status assessment model is a deep learning network combining a spatiotemporal convolutional network and an attention mechanism. The operational deviation prediction model is a fusion prediction model of temporal LSTM and an attention mechanism. The personalized decision support and feedback module is used to generate and provide real-time feedback or decision suggestions to the pilot and instructor terminals based on the output of the intelligent assessment and decision support model library. The decision quality assessment and optimization closed loop is used to obtain decision effectiveness data through real-time process evaluation and long-term result evaluation, and to iteratively optimize the intelligent assessment and auxiliary decision-making model library based on this data.

[0010] Preferably, the multimodal physiological data acquisition module includes a dry electrode EEG sensor array integrated into a flight helmet or head-mounted device, an ECG and respiration sensor integrated into clothing or a seat, a skin conductance sensor worn on the wrist or integrated into the control stick, a fixed non-contact eye-tracking device, and an in-cabin voice acquisition device; the fixed non-contact eye-tracking device consists of 2-3 miniature high frame rate infrared cameras and a near-infrared light source, and is integrated and installed on the upper shell of the instrument panel, windshield frame, or around the main display screen of the flight simulator cockpit, and captures the pilot's eye features through the collaborative work of multiple camera arrays.

[0011] Preferably, the operations performed by the data fusion and feature extraction module include: High-precision time node synchronization of multi-source data streams is achieved by establishing a unified time reference based on the PPS second pulse signal of the flight data bus, with a synchronization accuracy of ≤10ms. Adaptive filtering and wavelet transform methods are used to denoise and remove artifacts from the original physiological signals. Extract multidimensional features including EEG power spectrum entropy, the ratio of low-frequency to high-frequency components of heart rate variability, pupil diameter change rate, dwell time on a specific instrument, and the spectral characteristics of lever force input; Based on temporal correlation, the extracted physiological features are spatiotemporally aligned and fused with specific flight phases and control actions, and a 64-dimensional context-physiology-operation fusion feature vector is constructed according to the block rules of flight context features, physiological features, and operation features. By inputting the fused feature vectors into the intelligent assessment and decision-making model library, the pilot's psychological state assessment results and auxiliary decision-making information are obtained.

[0012] Preferably, the 16-dimensional flight situation features include one-hot encoding of the flight phase, normalized aircraft state parameters, external environmental data, and one-hot encoding of special situation types; the 32-dimensional physiological features include 8-dimensional EEG features, 8-dimensional ECG features, 8-dimensional eye-tracking features, and 8-dimensional comprehensive physiological features, with the EEG features including EEG power spectral entropy and prefrontal cortex... The frequency band power, electrocardiogram features including heart rate variability (LF / HF ratio), SDNN, and RMSSD, eye movement features including pupil diameter change rate and fixation duration, and comprehensive physiological features including skin conductance, respiratory rate, and average speech rate; the 16-dimensional operational features include the real-time values ​​and change rates of pilot control input parameters, control smoothness, stick force input spectrum features, and operation response delay.

[0013] Preferably, the special situation handling decision tree generation model is constructed based on a reinforcement learning framework; the decision generation of the intelligent assessment and auxiliary decision model library is based on a hierarchical reasoning model, including a safety rule engine based on aviation safety theory and standard operating procedures, and a cognitive performance optimization engine based on cognitive load theory and human factors engineering.

[0014] Preferably, the output of the pilot multidimensional state assessment model is a real-time quantitative assessment result of at least one of the pilot's current cognitive load level, fatigue state, and situational awareness level. Cognitive load is divided into three levels: low, medium, and high, with a comprehensive index of 0.0 ≤ low load < 0.3, 0.3 ≤ medium load < 0.7, and 0.7 ≤ high load ≤ 1.0. Situational awareness is divided into three levels: good, average, and insufficient, with a score above 80 being good, 60-80 being average, and below 60 being insufficient. Fatigue level is divided into three levels: alert, mild fatigue, and significant fatigue. A comprehensive characteristic deviation from the alertness baseline of less than 20% indicates alertness, 20% ≤ deviation from the alertness baseline of ≤ 50% indicates mild fatigue, and greater than 50% indicates significant fatigue.

[0015] Preferably, the feedback forms implemented by the personalized auxiliary decision-making and feedback module include hierarchical visual alarms, context-aware prompts, adaptive training adjustments, and post-analysis report generation.

[0016] Preferably, the graded visual alarm is indicated by green, yellow and red markings for optimal, high load and overload status levels via a head-up display or helmet display. The head-up display only displays the pilot status quantification index, graded visual alarm indicator ring, flight status assessment information and flight risk assessment information.

[0017] Preferably, the operation deviation prediction model takes flight status, pilot multidimensional state assessment results, and operation sequence of the past 30 seconds as inputs, and outputs the probability and risk level of flight operation deviation within the next 10 seconds. The flight status is a 1×12-dimensional time-series vector formed by normalized aircraft state parameters at the current moment and the previous 5 seconds. The pilot multidimensional state assessment results are a 1×9-dimensional time-series vector formed by cognitive load index, situational awareness score, and fatigue characteristic deviation degree at the current moment and the previous 5 seconds. The operation sequence of the past 30 seconds is a 1×32-dimensional time-series vector formed by pilot control input parameters and operation characteristics of the previous 30 seconds. The three types of input data are concatenated according to the time dimension into a 30×53 time-series input tensor, which is then input into the operation deviation prediction model.

[0018] A multimodal intelligent monitoring and decision support method for flight training is also proposed, including the following steps: S1. Non-invasive real-time acquisition of pilots' electroencephalogram (EEG), electrocardiogram (ECG), respiration, skin conductance, eye movement, and speech physiological signals through a multimodal physiological data acquisition module; S2. The flight operation and situation data synchronization module acquires the aircraft status parameters, pilot control inputs and external environment data in real time from the flight simulator or airborne bus. The external environment data includes meteorological environment data, airspace environment data, site environment data and special situation environment data. S3. The data obtained in steps S1 and S2 are received and processed synchronously through the data fusion and feature extraction module. After time synchronization and signal preprocessing of the multi-source data stream, multi-dimensional features are extracted and a 64-dimensional situation-physiology-operation fusion feature vector is constructed according to the block rule of 16-dimensional flight situation features, 32-dimensional physiological features, and 16-dimensional operation features. S4. Using the intelligent assessment and auxiliary decision-making model library, the fused feature vector is processed based on machine learning algorithms. After the 64-dimensional situation-physiology-operation fused feature vector is input into the pilot's multi-dimensional state assessment model, the model extracts the spatial correlation information of the features through the spatiotemporal convolutional layer, and combines the attention layer to give key physiological features and key situation features a weight. The quantitative assessment result is output through the fully connected layer. The probability of operation deviation and the risk level of operation deviation in the next 10 seconds are obtained by using the operation deviation prediction model with a 30×53 time-series input tensor as input. The auxiliary decision-making information is obtained by combining the special situation handling decision tree generation model. S5. Through the personalized auxiliary decision-making and feedback module, based on the output of the intelligent assessment and auxiliary decision-making model library, generate hierarchical visual alarms and context-aware prompts in real time and push them to the pilot terminal, and generate status monitoring and decision-making suggestion information and push them to the instructor terminal. S6. Decision-making effectiveness data is obtained through real-time process evaluation based on physiological compliance rate, state improvement rate, and deviation correction rate, as well as long-term result evaluation based on training task completion efficiency and learning curve acceleration, through decision quality assessment and optimization closed loop. The intelligent assessment and auxiliary decision-making model library is iteratively optimized based on this data. The long-term result evaluation also includes the assessment of pilot state stability and stress resistance.

[0019] Therefore, this invention proposes a multimodal intelligent monitoring and auxiliary decision-making system and method for flight training, the beneficial effects of which are as follows: (1) To achieve objective and quantitative real-time monitoring of the pilot’s physiological and psychological state, through deep spatiotemporal fusion of multimodal physiological data and flight data, to accurately capture multidimensional states such as cognitive load and situational awareness, breaking through the limitations of traditional subjective assessment.

[0020] (2) Construct an intelligent auxiliary decision-making system for in-process intervention, generate personalized operation prompts, status alarms and training adjustment suggestions based on hierarchical reasoning models, accurately push through multi-channel feedback, and continuously improve decision-making efficiency through closed-loop optimization mechanism, thereby significantly improving training efficiency and flight safety.

[0021] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0022] Figure 1This is the overall architecture diagram of the multimodal intelligent monitoring and auxiliary decision-making system and method for flight training based on the present invention; Figure 2 This is a flowchart of the electrocardiogram signal processing and heart rate variability analysis of the intelligent monitoring and auxiliary decision-making system and method for multimodal flight training based on the present invention; Figure 3 This is a schematic diagram of the working mechanism of the intelligent evaluation and auxiliary decision-making model library of the intelligent monitoring and auxiliary decision-making system and method for flight training based on multimodality of the present invention; Figure 4 This is a schematic diagram of the head-up display feedback example interface of the multimodal intelligent monitoring and auxiliary decision-making system and method for flight training based on the present invention. Detailed Implementation

[0023] To make the technical solutions, advantages, and objectives of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below. The described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the described embodiments of the present invention without creative effort are within the protection scope of this application.

[0024] Unless otherwise defined, the technical or scientific terms used in this invention shall have the ordinary meaning as understood by one of ordinary skill in the art to which this invention pertains.

[0025] like Figures 1-4 As shown, the multimodal-based intelligent monitoring and auxiliary decision-making system for flight training provided by this invention includes: The multimodal physiological data acquisition module is used for non-invasive real-time acquisition of pilots' physiological signals, including electroencephalogram (EEG) signals, electrocardiogram (ECG) signals, respiratory signals, skin conductance signals, eye movement signals, and speech signals. The multimodal physiological data acquisition module includes a dry electrode EEG sensor array integrated into a flight helmet or head-mounted device, ECG and respiration sensors integrated into clothing or seats, skin conductance sensors worn on the wrist or integrated into the control stick, a fixed non-contact eye tracking device, and an in-cabin voice acquisition device. The fixed non-contact eye-tracking device consists of 2-3 miniature high-frame-rate infrared cameras and a near-infrared light source. It is integrated and installed on the upper shell of the instrument panel, windshield frame or around the main display screen of the flight simulator cockpit, and captures the pilot's eye features through the collaborative work of multiple camera arrays.

[0026] The flight operation and situation data synchronization module is used to acquire aircraft status parameters, pilot control inputs and external environment data in real time from flight simulators or airborne buses; External environment data includes meteorological environment data, airspace environment data, site environment data, and special situation environment data that are directly related to flight training. Meteorological environment data includes visibility, runway visual range, cloud height, wind speed, wind direction, precipitation type and intensity. Airspace environment data includes flight airspace type, number and location of aircraft in the airspace, and air traffic control instructions. Site environment data includes runway status, airport airspace clearance conditions, and the working status of ground navigation facilities. Special situation environment data includes airflow disturbances under special situation simulation and external environment change parameters such as missing navigation environment. The data fusion and feature extraction module receives and synchronously processes data output from the multimodal physiological data acquisition module and the flight operation and situation data synchronization module. It extracts a fusion feature vector for assessing the pilot's state. The fusion feature vector is a 64-dimensional context-physiological-operational fusion feature vector, which is constructed from 16-dimensional flight context features, 32-dimensional physiological features, and 16-dimensional operational features. The 64-dimensional context-physiological-operational fusion feature vector is an ordered feature vector, in which the first 16 dimensions are flight context features, the middle 32 dimensions are physiological features, and the last 16 dimensions are operational features. The three types of features are precisely timestamped based on the PPS second pulse signal of the flight data bus to ensure the spatiotemporal correlation between features.

[0027] The operations performed by the data fusion and feature extraction module include: High-precision time node synchronization of multi-source data streams is achieved by establishing a unified time reference based on the PPS second pulse signal of the flight data bus, with a synchronization accuracy of ≤10ms. Adaptive filtering and wavelet transform methods are used to denoise and remove artifacts from the original physiological signals. Extract multidimensional features including EEG power spectrum entropy, the ratio of low-frequency to high-frequency components of heart rate variability, pupil diameter change rate, dwell time on a specific instrument, and the spectral characteristics of lever force input; Based on temporal correlation, the extracted physiological features are spatiotemporally aligned and fused with specific flight phases and control actions, and a 64-dimensional context-physiology-operation fusion feature vector is constructed according to the block rules of flight context features, physiological features, and operation features. By inputting the fused feature vectors into the intelligent assessment and decision-making model library, the pilot's psychological state assessment results and auxiliary decision-making information are obtained.

[0028] The 16-dimensional flight context features include one-hot encoding of the flight phase, normalized aircraft state parameters, external environmental data, and one-hot encoding of special situation types; the 32-dimensional physiological features include 8-dimensional EEG features, 8-dimensional ECG features, 8-dimensional eye-tracking features, and 8-dimensional comprehensive physiological features. The EEG features include EEG power spectral entropy and prefrontal cortex... Frequency band power, ECG features including heart rate variability (LF / HF ratio), SDNN, RMSSD, eye movement features including pupil diameter change rate and fixation duration, comprehensive physiological features including skin conductance, respiratory rate, and average speech rate; 16-dimensional operational features include real-time values ​​and change rates of pilot control input parameters, control smoothness, stick force input spectrum features, and operation response delay.

[0029] The intelligent assessment and decision support model library is used to process fused feature vectors based on machine learning algorithms to obtain pilot status assessment results, operational trend prediction results, and decision support information. The intelligent assessment and decision support model library includes a multi-dimensional pilot status assessment model, an operational deviation prediction model, and a decision tree generation model for handling special situations. The pilot multidimensional state assessment model is a deep learning network that combines a spatiotemporal convolutional network with an attention mechanism. The processing logic of the pilot multidimensional state assessment model for the 64-dimensional fused feature vector is as follows: First, the spatiotemporal convolutional layer extracts features of spatial and temporal dimensions from the fused features. Then, the attention layer assigns higher weights to core features such as EEG features, eye movement features, special situation type features, and flight phase features. Finally, the fully connected layer maps the extracted features into quantitative values ​​of cognitive load index, situational awareness score, and fatigue feature deviation. The output of the pilot's multidimensional state assessment model is a real-time quantitative assessment of at least one of the pilot's current cognitive load level, fatigue state, and situational awareness level. Cognitive load is divided into three levels: low, medium, and high, with a comprehensive index of 0.0 ≤ low load < 0.3, 0.3 ≤ medium load < 0.7, and 0.7 ≤ high load ≤ 1.0. Situational awareness is divided into three levels: good, average, and insufficient, with a score above 80 indicating good, 60-80 indicating average, and below 60 indicating insufficient. Fatigue level is divided into three levels: alert, mild fatigue, and significant fatigue. A comprehensive characteristic deviation from the alertness baseline of less than 20% indicates alertness, 20% ≤ deviation from the alertness baseline of ≤ 50% indicates mild fatigue, and greater than 50% indicates significant fatigue.

[0030] The operational deviation prediction model is a fusion prediction model of temporal LSTM and attention mechanism. The core processing logic of the model is as follows: the 30×53 temporal input tensor formed by the flight situation, the pilot's multi-dimensional state assessment results and the operation sequence of the past 30 seconds is weighted by the attention layer with a weight of 3:4:3, the temporal correlation information of the temporal features is extracted by the LSTM layer, and the probability of different flight operational deviations occurring in the next 10 seconds is output by the Softmax layer. The risk level is classified in combination with aviation safety standards. The operational deviation prediction model takes flight status, pilot multidimensional status assessment results, and the operational sequence of the past 30 seconds as inputs, and outputs the probability and risk level of flight operational deviations within the next 10 seconds. The flight status is a 1×12-dimensional time-series vector formed by the normalized aircraft state parameters at the current moment and the previous 5 seconds. The pilot's multidimensional state assessment results are a 1×9-dimensional time-series vector formed by the cognitive load index, situational awareness score, and fatigue characteristic deviation at the current moment and the previous 5 seconds. The operation sequence of the past 30 seconds is a 1×32-dimensional time-series vector formed by the pilot's control input parameters and operation characteristics in the previous 30 seconds. The three types of input data are concatenated along the time dimension to form a 30×53 time-series input tensor, which is then input into the operation deviation prediction model.

[0031] After inputting into the operation deviation prediction model of the fusion prediction model of temporal LSTM + attention mechanism, the model assigns a 3:4:3 weight to the three types of input features through the attention layer. Combined with the LSTM layer's ability to extract temporal features, it outputs the probability of occurrence and risk level of different types of flight operation deviations within the next 10 seconds.

[0032] The personalized decision support and feedback module is used to generate and provide real-time feedback or decision suggestions to pilots and instructors based on the output of the intelligent assessment and decision support model library. The personalized decision support and feedback module provides feedback in the form of tiered visual alerts, context-aware prompts, adaptive training adjustments, and post-event analysis report generation.

[0033] The graded visual alarms are indicated by green, yellow, and red on the head-up display or helmet display, representing the optimal, high load, and overload status levels, respectively. The head-up display only shows the pilot's status quantification indicators, the graded visual alarm indicator ring, flight status assessment information, and flight risk assessment information.

[0034] The emergency response decision tree generation model is built on a reinforcement learning framework; the decision generation of the intelligent assessment and auxiliary decision model library is based on a hierarchical reasoning model, including a safety rule engine based on aviation safety theory and standard operating procedures, and a cognitive performance optimization engine based on cognitive load theory and human factors engineering.

[0035] The decision quality assessment and optimization closed loop is used to obtain decision effectiveness data through real-time process evaluation and long-term result evaluation, and to iteratively optimize the intelligent assessment and auxiliary decision-making model library based on this data.

[0036] A multimodal intelligent monitoring and decision support method for flight training is also proposed, including the following steps: S1. Non-invasive real-time acquisition of pilots' electroencephalogram (EEG), electrocardiogram (ECG), respiration, skin conductance, eye movement, and speech physiological signals through a multimodal physiological data acquisition module; S2. The flight operation and situation data synchronization module acquires the aircraft status parameters, pilot control inputs and external environment data in real time from the flight simulator or airborne bus. The external environment data includes meteorological environment data, airspace environment data, site environment data and special situation environment data. S3. The data obtained in steps S1 and S2 are received and processed synchronously through the data fusion and feature extraction module. After time synchronization and signal preprocessing of the multi-source data stream, multi-dimensional features are extracted and a 64-dimensional situation-physiology-operation fusion feature vector is constructed according to the block rule of 16-dimensional flight situation features, 32-dimensional physiological features, and 16-dimensional operation features. S4. Through the intelligent assessment and auxiliary decision-making model library, the fused feature vector is processed based on machine learning algorithms. After the 64-dimensional situation-physiology-operation fused feature vector is input into the pilot's multi-dimensional state assessment model, the model extracts the spatial correlation information of the features through the spatiotemporal convolutional layer, and combines the attention layer to give key physiological features and key situation features a weight. The quantitative assessment result is output through the fully connected layer. The probability of operation deviation and the risk level of operation deviation in the next 10 seconds are obtained through the operation deviation prediction model with a 30×53 time-series input tensor. The auxiliary decision-making information is obtained by combining the special situation handling decision tree generation model. S5. Through the personalized decision support and feedback module, based on the output of the intelligent assessment and decision support model library, generate hierarchical visual alarms and context-aware prompts in real time and push them to the pilot terminal, and generate status monitoring and decision suggestion information and push them to the instructor terminal. S6. Decision-making effectiveness data is obtained through real-time process evaluation based on physiological compliance rate, state improvement rate, and deviation correction rate, as well as long-term result evaluation based on training task completion efficiency and learning curve acceleration, through decision quality assessment and optimization closed loop. The intelligent assessment and auxiliary decision-making model library is iteratively optimized based on this data. The long-term result evaluation also includes the assessment of pilot state stability and stress resistance.

[0037] This invention uses civil aviation pilot simulator approach and landing training as an application scenario to verify the effectiveness of the method. The specific implementation process is as follows: I. System Deployment and Initialization: (a) Deployment Module: Multimodal physiological data acquisition module: The pilot wears a flight helmet with an integrated 8-channel dry electrode EEG sensor, with the frontal lobe and motor cortex corresponding to the core sensor acquisition area; chest patch ECG and respiration sensors are attached to the apex of the heart and abdomen; the inner side of the control stick grip integrates a skin conductance sensor; one miniature infrared camera (3 in total) is installed on the upper shell of the cockpit instrument panel and on each side of the windshield frame, which, together with a near-infrared light source, form an eye-tracking device; an omnidirectional voice acquisition microphone is arranged in the center of the top of the cockpit.

[0038] Flight Operation and Situation Data Synchronization Module: Directly connected to the simulator bus via the ARINC429 protocol data bus interface or a protocol data bus of equivalent performance, it acquires more than 20 flight parameters such as altitude, airspeed, and heading in real time, with a synchronization accuracy of ≤10ms.

[0039] Data fusion and feature extraction module: Using NVIDIA Jetson Xavier NX edge computing units or edge computing units of equivalent performance, deployed in the cockpit equipment cabinet, responsible for parallel processing of multi-source data, ensuring processing latency <50ms.

[0040] Intelligent evaluation and decision support model library: Based on Intel Xeon Gold 6330 server (32GB memory) or server with equivalent performance, deployed in ground data center, communicating with simulator via 5G network, model inference latency <20ms.

[0041] Personalized decision support and feedback module: The pilot is equipped with an AR-HUD (15°×8° field of view, 1920×1080 resolution) installed on the inside of the windshield and wears bone conduction headphones; the instructor's room is equipped with a 27-inch dual-screen workstation as a monitoring terminal; the simulator joystick is replaced with a dedicated joystick with force feedback function (force feedback accuracy ±0.1N).

[0042] (II) System Initialization Process: System initialization will begin 30 minutes before training starts: After the pilot puts on all the sensing devices, the system performs a self-check to ensure that the signals from the EEG, ECG, and skin conductance sensors are transmitted normally, and that the eye-tracking device covers the head movement range of ±30° (horizontal / vertical).

[0043] Complete time synchronization calibration by aligning the timestamps of all sensor data with the timestamps of flight parameters using the PPS (pulse per second) signal of the flight data bus as a reference, ensuring synchronization accuracy ≤10ms.

[0044] Establishing a personalized physiological baseline: The pilot remained at rest for 5 minutes, and the system collected the electroencephalogram (EEG) of the prefrontal cortex during this period. Frequency band, occipital region Data on frequency band power, electrocardiogram (resting heart rate, HRV-related indicators), eye movement (baseline pupil diameter, blink frequency), and skin conductance (baseline skin conductance level) were used as benchmarks for subsequent condition assessment.

[0045] II. Data Acquisition and Processing Implementation: (a) Synchronous acquisition of multi-source data: During the training process (taking the five-way approach and landing training as an example, lasting approximately 15 minutes), the system collects two types of data in parallel: Physiological data: EEG signal sampling rate 250Hz, ECG signal sampling rate 500Hz, skin conductance signal sampling rate 100Hz, eye movement data frame rate 120fps, speech signal sampling rate 16kHz, all data are stamped with a unified timestamp in real time.

[0046] Flight data: Flight parameters are updated at a frequency of 50Hz, including altitude, airspeed, pitch angle, roll angle, heading, landing gear status, throttle position, etc.; at the same time, the pilot's control inputs (stick force, rudder angle, throttle control) and flight phase labels (takeoff, climb, cruise, approach, landing) are recorded.

[0047] (II) Data Preprocessing and Feature Extraction: Signal preprocessing: EEG signals were filtered to remove noise and independent component analysis (ICA) was used to remove electrooculography artifacts.

[0048] After the electrocardiogram signal is bandpass filtered from 0.5 to 35 Hz, the peak value of the R wave is identified by the Pan-Tompkins algorithm, the RR interval sequence is generated, outliers exceeding the average RRI ± 20% are removed, and spline interpolation is used to fill the gap.

[0049] Eye-tracking data were thresholded to remove blink artifacts, pupil diameter measurements were calibrated, and normalized to a percentage change relative to baseline.

[0050] The skin conductance signal was filtered using a moving average to remove high-frequency noise while preserving the skin conductance level (SCL) and nonspecific skin conductance response (NS-SCR) characteristics.

[0051] Feature extraction: The system segments the data stream into "context frames," with each context frame corresponding to a specific flight mission unit (e.g., "approach and runway alignment phase" or "10 seconds before landing touchdown"). Two types of features are extracted within each frame: Reactive characteristics: such as delayed pupil dilation (≤0.5 seconds) and dilation amplitude (≥8%) after engine malfunction alarm, and heart rate acceleration slope (≥5 beats / minute·second).

[0052] Pattern characteristics: EEG calculations of the prefrontal cortex Bandwidth power, pincushion region Frequency band desynchronization degree and cognitive load index; ECG calculation of average heart rate, SDNN, RMSSD, and LF / HF ratio; eye movement calculation of pupil diameter change rate, fixation dwell time, fixation entropy, and saccade speed; operation calculation of lever force input spectral features and manipulation smoothness; speech analysis of average speech rate and speech energy fluctuation.

[0053] Feature vector fusion construction: Based on temporal correlation, the extracted physiological characteristics are spatiotemporally aligned with flight phases and control actions. For example, "approach phase (altitude 500 feet) - pitch angle adjustment (+2°) - frontal lobe" is aligned with these phases. Information such as "power increase of 30% - pupil dilation of 10%" is fused and constructed into a 64-dimensional "context-physiology-operation" fusion feature vector according to the block rule of the first 16 dimensions of flight context features, the middle 32 dimensions of physiological features, and the last 16 dimensions of operational features.

[0054] III. Implementation of Intelligent Assessment and Decision Support: (a) Pilot multidimensional status assessment: After the 64-dimensional fused feature vector is input into the pilot's multidimensional state assessment model, the model extracts the spatiotemporal correlation features of "approach phase + prefrontal power increase + pitch angle adjustment" through a spatiotemporal convolutional layer. The attention layer assigns 65% of the core weights to prefrontal power, pupil dilation amplitude, and approach phase labels, and outputs quantitative assessment results in real time. Cognitive load: Comprehensive index 0.58, judged as moderate load (optimal working range), based on prefrontal cortex... Power increased by 38% compared to baseline, top pillow The power exhibits task-related fluctuations, which align with the cognitive load assessment criteria.

[0055] Situational awareness: Score 75, judged as average, based on a comprehensive assessment of features such as eye-tracking gaze entropy of 1.8 (the instrument scanning is regular but there are some omissions) and voice communication response delay ≤1 second.

[0056] Fatigue level: The overall characteristics deviated from the baseline of wakefulness by 15%, which was judged as a wakefulness state. No fatigue characteristics such as increased delta slow waves or prolonged eyelid closure time were observed.

[0057] (II) Operational Deviation Prediction and Decision Generation: Operational Deviation Prediction: The operational deviation prediction model concatenates the flight status (altitude 500 feet, airspeed 140 knots, pitch angle +1.5°), multidimensional state assessment results, and the operation sequence of the past 30 seconds into a 30×53 temporal input tensor. After the attention layer weights the three types of inputs with a weight of 3:4:3, the LSTM layer extracts the temporal features of continuously low altitude and insufficient pitch angle adjustment in the past 5 seconds. It predicts that the probability of "altitude deviation from the target glide path" in the next 10 seconds is 62%, with a medium risk level.

[0058] Decision Generation Assistance: Generating Decision Suggestions Based on a Hierarchical Reasoning Model Security rule engine: If the security red line rule is not triggered, the decision priority is lower than the optimization engine output.

[0059] Cognitive Efficacy Optimization Engine (Reinforcement Learning Model): Combining the current state of "moderate cognitive load + general situational awareness + moderate risk of high deviation", it generates real-time operation assistance decisions - "highlighting prompts in the AR-HUD altimeter area and simultaneously issuing voice prompts through bone conduction headphones: 'Check altitude, adjust pitch angle to +2°'". This decision has been verified through offline training and can improve the altitude deviation correction rate by more than 70%.

[0060] (III) Example of decision-making in special situation handling scenarios: When a "left engine failure" situation is introduced during training (approach phase altitude 800 feet): Status assessment: The system detected a rapid increase in the pilot's skin conductance signal (60% increase from baseline), a heart rate increase of 30 beats / minute from baseline, and a significant increase in the LF / HF ratio. After weighting the skin conductance signal, heart rate changes, and special situation type characteristics, the model output a cognitive load index of 0.82 (high load), which was determined to be a high stress state.

[0061] Decision Generation: The special situation handling decision tree generation model is launched, dynamically generating personalized handling steps based on a reinforcement learning framework. Step 1 (0-3 seconds): The AR-HUD highlights the "Left Engine Failure" warning icon, and the voice prompt "Left Engine Failure, Maintain Attitude" is given. At this time, the pilot's stress level is high, and the prompt is concise and clear.

[0062] Step 2 (3-8 seconds): The HUD will display "Reduce left throttle to slow down" and "Adjust course to runway center" in steps. At the same time, the instructor terminal will push a prompt "The trainee is in a high stress state. It is recommended not to add any additional tasks for the time being."

[0063] Step 3 (8-15 seconds): Based on the pilot's response (left engine throttle reduced, heading deviation ≤2°), display "Check remaining engine thrust" and "Maintain airspeed 130 knots", gradually guiding the pilot through the initial emergency response.

[0064] IV. Feedback Implementation and Closed-Loop Optimization: (a) Implementation through multiple channels of feedback: Feedback for pilots: Visual feedback: A red pulsating halo appears in the AR-HUD altimeter area (highlighted), simultaneously displaying the numerical indication of "pitch angle +2°"; the comprehensive status indicator ring is yellow (high load). This AR-HUD interface only includes the pilot's cognitive load index, situational awareness score, graded alarm indicator ring, flight status parameters such as altitude / airspeed, and altitude deviation risk level, without any other irrelevant content.

[0065] Auditory feedback: The bone conduction headphones emit a spatial voice prompt "Check height, adjust the pitch angle to +2°", with a voice delay of <5ms.

[0066] Tactile feedback: When the pilot does not respond in time, the joystick vibrates slightly (frequency 2Hz) to guide the pilot to adjust the control action.

[0067] Feedback for instructors: The instructor's terminal displays the trainee's status curve (cognitive load, situational awareness, and stress level changes over time) in real time, and marks the warning information as "high deviation from moderate risk".

[0068] The push notification with the decision suggestion text reads: "The student's current cognitive load is moderate, but their situational awareness is average. A high-level adjustment prompt has been triggered. It is recommended to continue monitoring the response to subsequent operations."

[0069] (II) Decision-making quality assessment and closed-loop optimization: Real-time process evaluation: Physiological compliance rate: If the pilot’s gaze moves to the altimeter and stays there for 1.5 seconds within 2 seconds after the voice prompt is given, it is considered as valid physiological compliance. The physiological compliance rate for this type of decision is 82%.

[0070] Improvement in condition: Within 10 seconds of decision execution, the cognitive load index dropped to 0.52, the altitude deviation converged from -30 feet to -8 feet, and the improvement in condition reached 73%.

[0071] Deviation correction rate: (30-8) / 30≈73%, which meets the effective decision-making criteria (deviation correction rate ≥60%).

[0072] Long-term optimization of the closed loop: Experience storage: The complete experience tuple (S_t, A_t, R_t, S_{t+1}) of "approach phase - medium cognitive load - high deviation risk - highlight + voice prompt - deviation correction" in this training is stored in the system experience pool.

[0073] Model retraining: After the experience pool accumulates 1,000 new experiences, the system uses the PPO algorithm to retrain the cognitive efficiency optimization engine offline, update the decision strategy weights, and improve the decision efficiency by 5%-8% in similar scenarios.

[0074] Rule base evolution: After 100 similar training tests, the decision-making mode of "triggering highlight + voice prompt when the risk of high deviation is greater than 60%" has an effectiveness rate of 85%, and it has been solidified as an expert rule and added to the security rule engine.

[0075] (III) Generation of post-event analysis report: After training, the system automatically generates a multimodal data fusion analysis report, the core contents of which include: Timeline visualization: Using time as the axis, it synchronously displays the curves of flight parameter changes, physiological index fluctuations, and the correspondence between operation sequences and decision trigger nodes.

[0076] Key event analysis: Highlight multimodal data of key nodes such as emergency handling and deviation warning, such as "3 seconds after left motor failure, skin conductance increased by 60%, heart rate increased by 30 beats / minute, and the first operation response was delayed by 1.2 seconds".

[0077] Training effectiveness evaluation: A total of 18 decisions were triggered during this training, of which 15 (83%) were effective; the trainees' cognitive load range accounted for 65% and high load range accounted for 20%, with a training effectiveness score of 82 points.

[0078] Personalized training recommendations: "Trainees are proficient in handling emergency procedures under high stress, but the initial response delay is slightly long. It is recommended that the next training session include training on a combined emergency situation of 'engine failure + communication interference' to enhance their ability to make rapid decisions under stress."

[0079] Therefore, this invention provides a multimodal intelligent monitoring and auxiliary decision-making system and method for flight training. The system non-invasively collects physiological signals such as EEG and ECG through a multimodal physiological data acquisition module, acquires flight-related data through a flight operation and situational data synchronization module, and constructs fused feature vectors through data fusion and feature extraction modules. Based on an intelligent assessment and auxiliary decision-making model library, it outputs state assessments, deviation predictions, and decision information. Finally, a personalized auxiliary decision-making and feedback module provides pilots and instructors with tiered alarms and operational prompts. Based on aviation safety theory and cognitive load theory, the system constructs a hierarchical decision-making architecture and a "monitoring-decision-feedback-optimization" closed loop, achieving objective, quantitative, and real-time monitoring of pilots' physiological and psychological states. This shifts the focus from "post-training debriefing" to "in-process intervention," improving training efficiency and flight safety margins.

[0080] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the technical solutions of the present invention, and these modifications or equivalent substitutions cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. A multimodal intelligent monitoring and auxiliary decision-making system for flight training, characterized in that, include: A multimodal physiological data acquisition module is used for non-invasive real-time acquisition of pilots' physiological signals, including electroencephalogram (EEG) signals, electrocardiogram (ECG) signals, respiratory signals, skin conductance signals, eye movement signals, and speech signals. The flight operation and situation data synchronization module is used to acquire aircraft status parameters, pilot control inputs, and external environment data in real time from the flight simulator or airborne bus. The external environment data includes meteorological environment data, airspace environment data, site environment data, and special situation environment data directly related to flight training. Meteorological environment data includes visibility, runway visual range, cloud height, wind speed, wind direction, precipitation type and intensity. Airspace environment data includes flight airspace type, number and location of aircraft in the airspace, and air traffic control instructions. Site environment data includes runway status, airport airspace clearance conditions, and ground navigation facility operating status. Special situation environment data includes airflow disturbances under special situation simulation and external environment change parameters such as navigation environment missing conditions. The data fusion and feature extraction module is used to receive and synchronously process the data output by the multimodal physiological data acquisition module and the flight operation and situation data synchronization module, and extract a fusion feature vector for evaluating the pilot's state. The fusion feature vector is a 64-dimensional situation-physiological-operation fusion feature vector, which is constructed by dividing the data into 16-dimensional flight situation features, 32-dimensional physiological features and 16-dimensional operation features. The intelligent assessment and auxiliary decision-making model library is used to process the fused feature vector based on machine learning algorithms to obtain pilot status assessment results, operational trend prediction results, and auxiliary decision-making information. The intelligent assessment and auxiliary decision-making model library includes a multi-dimensional pilot status assessment model, an operational deviation prediction model, and a special situation handling decision tree generation model. The multi-dimensional pilot status assessment model is a deep learning network combining a spatiotemporal convolutional network and an attention mechanism. The operational deviation prediction model is a fusion prediction model of temporal LSTM and an attention mechanism. The personalized decision support and feedback module is used to generate and provide real-time feedback or decision suggestions to the pilot and instructor terminals based on the output of the intelligent assessment and decision support model library. The decision quality assessment and optimization closed loop is used to obtain decision effectiveness data through real-time process evaluation and long-term result evaluation, and to iteratively optimize the intelligent assessment and auxiliary decision-making model library based on this data.

2. The multimodal-based intelligent monitoring and auxiliary decision-making system for flight training according to claim 1, characterized in that, The multimodal physiological data acquisition module includes a dry electrode EEG sensor array integrated into a flight helmet or head-mounted device, an ECG and respiration sensor integrated into clothing or a seat, a skin conductance sensor worn on the wrist or integrated into the control stick, a fixed non-contact eye-tracking device, and an in-cabin voice acquisition device. The fixed non-contact eye-tracking device consists of 2-3 miniature high frame rate infrared cameras and a near-infrared light source, which are integrated and installed on the upper shell of the instrument panel, the windshield frame, or around the main display screen of the flight simulator cockpit. It captures the pilot's eye features through the collaborative work of multiple camera arrays.

3. The multimodal-based intelligent monitoring and auxiliary decision-making system for flight training according to claim 1, characterized in that, The operations performed by the data fusion and feature extraction module include: High-precision time node synchronization of multi-source data streams is achieved by establishing a unified time reference based on the PPS second pulse signal of the flight data bus, with a synchronization accuracy of ≤10ms. Adaptive filtering and wavelet transform methods are used to denoise and remove artifacts from the original physiological signals. Extract multidimensional features including EEG power spectrum entropy, the ratio of low-frequency to high-frequency components of heart rate variability, pupil diameter change rate, dwell time on a specific instrument, and the spectral characteristics of lever force input; Based on temporal correlation, the extracted physiological features are spatiotemporally aligned and fused with specific flight phases and control actions, and a 64-dimensional context-physiology-operation fusion feature vector is constructed according to the block rules of flight context features, physiological features, and operation features. By inputting the fused feature vectors into the intelligent assessment and decision-making model library, the pilot's psychological state assessment results and auxiliary decision-making information are obtained.

4. The multimodal-based intelligent monitoring and auxiliary decision-making system for flight training according to claim 3, characterized in that, The 16-dimensional flight scenario features include one-hot encoding of flight phases, normalized aircraft state parameters, external environmental data, and one-hot encoding of special situation types; the 32-dimensional physiological features include 8-dimensional EEG features, 8-dimensional ECG features, 8-dimensional eye-tracking features, and 8-dimensional comprehensive physiological features. The EEG features include EEG power spectral entropy and prefrontal cortex... The frequency band power, electrocardiogram features including heart rate variability (LF / HF ratio), SDNN, and RMSSD, eye movement features including pupil diameter change rate and fixation duration, and comprehensive physiological features including skin conductance, respiratory rate, and average speech rate; the 16-dimensional operational features include the real-time values ​​and change rates of pilot control input parameters, control smoothness, stick force input spectrum features, and operation response delay.

5. The multimodal-based intelligent monitoring and auxiliary decision-making system for flight training according to claim 1, characterized in that, The special situation handling decision tree generation model is built on a reinforcement learning framework; the decision generation of the intelligent assessment and auxiliary decision model library is based on a hierarchical reasoning model, including a safety rule engine based on aviation safety theory and standard operating procedures, and a cognitive performance optimization engine based on cognitive load theory and human factors engineering.

6. The multimodal-based intelligent monitoring and auxiliary decision-making system for flight training according to claim 1, characterized in that, The output of the pilot multidimensional state assessment model is a real-time quantitative assessment of at least one of the pilot's current cognitive load level, fatigue state, and situational awareness level. Cognitive load is divided into three levels: low, medium, and high, with a comprehensive index of 0.0 ≤ low load < 0.3, 0.3 ≤ medium load < 0.7, and 0.7 ≤ high load ≤ 1.

0. Situational awareness is divided into three levels: good, average, and insufficient, with a score above 80 indicating good, 60-80 indicating average, and below 60 indicating insufficient. Fatigue level is divided into three levels: alert, mild fatigue, and significant fatigue. A comprehensive characteristic deviation from the alertness baseline of less than 20% indicates alertness, 20% ≤ deviation from the alertness baseline of ≤ 50% indicates mild fatigue, and greater than 50% indicates significant fatigue.

7. The multimodal-based intelligent monitoring and auxiliary decision-making system for flight training according to claim 1, characterized in that, The personalized decision support and feedback module provides feedback in the form of tiered visual alerts, context-aware prompts, adaptive training adjustments, and post-analysis report generation.

8. The multimodal-based intelligent monitoring and auxiliary decision-making system for flight training according to claim 7, characterized in that, The graded visual alarms are indicated by green, yellow, and red markings for optimal, high load, and overload status levels, respectively, via a head-up display or helmet display. The head-up display only shows the pilot's status quantification indicators, graded visual alarm indicator rings, flight status assessment information, and flight risk assessment information.

9. The multimodal-based intelligent monitoring and auxiliary decision-making system for flight training according to claim 1, characterized in that, The operational deviation prediction model takes flight status, pilot multidimensional state assessment results, and the operational sequence of the past 30 seconds as inputs, and outputs the probability and risk level of flight operational deviation within the next 10 seconds. The flight status is a 1×12-dimensional time-series vector formed by the normalized aircraft state parameters of the current moment and the previous 5 seconds. The pilot multidimensional state assessment results are a 1×9-dimensional time-series vector formed by the cognitive load index, situational awareness score, and fatigue characteristic deviation of the current moment and the previous 5 seconds. The operational sequence of the past 30 seconds is a 1×32-dimensional time-series vector formed by the pilot's control input parameters and operational characteristics of the previous 30 seconds. The three types of input data are concatenated along the time dimension into a 30×53 time-series input tensor, which is then input into the operational deviation prediction model.

10. A multimodal intelligent monitoring and auxiliary decision-making method for flight training, characterized in that, The multimodal-based intelligent monitoring and auxiliary decision-making system for flight training as described in any one of claims 1-9 includes the following steps: S1. Non-invasive real-time acquisition of pilots' electroencephalogram (EEG), electrocardiogram (ECG), respiration, skin conductance, eye movement, and speech physiological signals through a multimodal physiological data acquisition module; S2. The flight operation and situation data synchronization module acquires the aircraft status parameters, pilot control inputs and external environment data in real time from the flight simulator or airborne bus. The external environment data includes meteorological environment data, airspace environment data, site environment data and special situation environment data. S3. The data obtained in steps S1 and S2 are received and processed synchronously through the data fusion and feature extraction module. After time synchronization and signal preprocessing of the multi-source data stream, multi-dimensional features are extracted and a 64-dimensional situation-physiology-operation fusion feature vector is constructed according to the block rule of 16-dimensional flight situation features, 32-dimensional physiological features, and 16-dimensional operation features. S4. Using the intelligent assessment and auxiliary decision-making model library, the fused feature vector is processed based on machine learning algorithms. After the 64-dimensional situation-physiology-operation fused feature vector is input into the pilot's multi-dimensional state assessment model, the model extracts the spatial correlation information of the features through the spatiotemporal convolutional layer, and combines the attention layer to give key physiological features and key situation features a weight. The quantitative assessment result is output through the fully connected layer. The probability of operation deviation and the risk level of operation deviation in the next 10 seconds are obtained by using the operation deviation prediction model with a 30×53 time-series input tensor as input. The auxiliary decision-making information is obtained by combining the special situation handling decision tree generation model. S5. Through the personalized auxiliary decision-making and feedback module, based on the output of the intelligent assessment and auxiliary decision-making model library, generate hierarchical visual alarms and context-aware prompts in real time and push them to the pilot terminal, and generate status monitoring and decision-making suggestion information and push them to the instructor terminal. S6. Decision-making effectiveness data is obtained through real-time process evaluation based on physiological compliance rate, state improvement rate, and deviation correction rate, as well as long-term result evaluation based on training task completion efficiency and learning curve acceleration, through decision quality assessment and optimization closed loop. The intelligent assessment and auxiliary decision-making model library is iteratively optimized based on this data. The long-term result evaluation also includes the assessment of pilot state stability and stress resistance.

Images