Multimodal detection and stress identification method and system for pilot emergency response capabilities

Abstract

This invention discloses a method and system for multimodal detection and stress identification of pilots' emergency response capabilities, relating to the fields of aviation safety and intelligent assessment technology. The method includes: synchronously collecting multimodal state data of pilots and preprocessing it to generate standardized feature vectors; performing cross-modal fusion analysis on the feature vectors to identify multi-dimensional states of the pilot's emergency response capabilities; constructing a capability index system based on the identified multi-dimensional states for quantitative evaluation, generating a comprehensive emergency response capability score and emergency response capability assessment level for the pilot; and performing real-time monitoring and anomaly diagnosis based on the score and level, automatically triggering dynamic feedback adapted to individual characteristics and generating a feedback report when performance risks or operational anomalies are detected. This invention improves the accuracy of pilots' emergency response and training effectiveness through synchronous multimodal data collection, cross-modal feature fusion, deep learning model state identification, comprehensive capability quantitative evaluation, and real-time monitoring and personalized feedback.

Description

Technical Field

[0001] This invention relates to the interdisciplinary field of aviation safety and intelligent assessment, specifically to a method and system for multimodal detection and stress identification of pilots' emergency response capabilities. Background Technology

[0002] Pilots' emergency response capabilities and real-time stress levels are core elements for ensuring civil aviation flight safety and improving airline operational efficiency. They are directly related to the safety of crew members and passengers, as well as the effectiveness of the national civil aviation safety system.

[0003] Currently, technological development in this field still faces numerous challenges. Data collection often relies on single dimensions such as physiological signals or situational questionnaires, making it difficult to comprehensively and objectively depict the pilot's overall condition. At the state identification level, existing methods can typically only distinguish superficial psychological states such as tension and fatigue, lacking effective quantification tools for deeper psychological traits that determine the quality of emergency decision-making, such as emotional regulation ability and stress resilience, leading to difficulties in predicting hidden risks. Operational assessments tend to focus on basic indicators such as reaction time and mission accuracy, lacking targeted verification of dimensions directly related to safety hazards, such as operational compliance and attention allocation.

[0004] Furthermore, existing models generally suffer from insufficient cross-scenario generalization ability. Data from stable simulator environments is poorly adapted to the complex scenarios of real flights, resulting in unstable accuracy in stress identification and capability assessment. More critically, existing evaluation systems are often disconnected from the training phase, typically only outputting a "pass / fail" conclusion without generating personalized improvement plans based on individual weaknesses. Ultimately, this leads to a mismatch between airlines' retraining investment and effectiveness improvement, resulting in low efficiency in emergency response capability development. Summary of the Invention

[0005] This invention provides a multimodal detection and stress identification method for pilots' emergency response capabilities, which solves the technical problems of existing pilot capability assessment methods, such as single data dimension, superficial psychological state identification, basic operational assessment indicators, weak model generalization ability across scenarios, and disconnect between assessment and training systems.

[0006] This invention is achieved through the following technical solution:

[0007] Firstly, this application provides a multimodal detection and stress identification method for pilot emergency response capabilities, comprising the following steps:

[0008] Simultaneously collect multimodal state data of the pilot and preprocess it to generate standardized feature vectors;

[0009] Cross-modal fusion analysis of feature vectors is performed to identify the multi-dimensional state of a pilot's emergency response capability;

[0010] Based on the identified multi-dimensional states, a capability index system is constructed for quantitative evaluation, generating a pilot's comprehensive emergency response capability score and emergency response capability assessment level.

[0011] Real-time monitoring and anomaly diagnosis are conducted based on the comprehensive emergency response capability score and emergency response capability assessment level. When performance risks or operational anomalies are detected, dynamic feedback adapted to individual characteristics is automatically triggered and a feedback report is generated.

[0012] A further optimization scheme is that the multimodal state data includes psychological questionnaire data, physiological signal data, eye-tracking data, and flight performance data.

[0013] A further optimized solution is that the cross-modal fusion analysis includes:

[0014] Feature sequences from different modalities are input into corresponding recurrent neural networks to obtain their respective dynamic context encodings;

[0015] Calculate cross-modal attention weights between different dynamic context encodings;

[0016] The cross-modal attention weights are used to perform weighted fusion of the corresponding feature codes to generate a deep fusion feature vector.

[0017] A further optimized solution is that the multi-dimensional state includes the pilot's real-time stress state, deep psychological traits, control intentions, and emergency operation compliance.

[0018] A further optimized solution is that the identification of compliance in emergency operations includes:

[0019] A pre-configured rule base based on flight safety priority rules;

[0020] Based on the pilot's actual operational sequence and the required sequences in the rule base, a compliance score is calculated using a time decay weighted average. The calculation formula is:

[0021] )

[0022] in, This refers to the actual operation of the pilot's emergency response at time t. This refers to the compliant operation required by the priority rules at time t. =0.9 is the time decay coefficient. The indicator function is T, which represents the total number of time steps from the start of the operation to the end of the current evaluation.

[0023] A further optimized solution is that the dynamic feedback intervention is based on real-time calculated stress risk value, attention deviation degree, and operational risk index, and is triggered through one or more of the following methods: voice prompts, visual prompts, or information pop-ups.

[0024] A further optimization is that the method further includes incrementally updating the model used for state recognition using a federated learning paradigm, wherein the incremental update includes:

[0025] The central server distributes the global model to each local server;

[0026] Each local server uses local data to incrementally update the global model, generates encrypted model update parameters, and uploads them to the central server.

[0027] The central server securely aggregates the collected encrypted model update parameters to generate a new generation of global model.

[0028] Secondly, this application provides a multimodal detection and stress identification system for pilot emergency response capabilities, used to implement the multimodal detection and stress identification method for pilot emergency response capabilities as described above; the system includes:

[0029] The data acquisition and preprocessing module is used to simultaneously acquire multimodal state data of the pilot and preprocess it to generate standardized feature vectors.

[0030] The feature fusion and state recognition module is communicatively connected to the data acquisition and preprocessing module. It is used to perform cross-modal fusion analysis on the standardized feature vectors and identify multi-dimensional states that characterize the pilot's emergency response capabilities.

[0031] The emergency response capability comprehensive evaluation module is communicatively connected to the feature fusion and state recognition module. It is used to perform quantitative evaluation based on the identified multi-dimensional states and generate the pilot's comprehensive emergency response capability score and emergency response capability assessment level.

[0032] The terminal response and personalized feedback module is communicatively connected to the emergency response capability comprehensive evaluation module. It is used to perform real-time monitoring and anomaly diagnosis based on the emergency comprehensive capability score and emergency capability judgment level, and to trigger dynamic feedback intervention and generate capability improvement reports.

[0033] A further optimization involves the terminal response and personalized feedback module communicating with the cockpit display system and voice system of a flight simulator or real aircraft to provide visual and voice prompts.

[0034] Thirdly, this application provides a computer-readable storage medium storing a pilot emergency response capability multimodal detection and stress identification program, wherein when the pilot emergency response capability multimodal detection and stress identification program is executed by a processor, it implements the steps of the pilot emergency response capability multimodal detection and stress identification method as described above.

[0035] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0036] By synchronously collecting multimodal state data from pilots and performing standardized preprocessing, data consistency and integrity are ensured. Then, through cross-modal fusion analysis, the pilots' real-time stress state, deep psychological traits, and other multidimensional states are deeply mined and accurately identified. Based on this, a capability index system is constructed to achieve an objective quantitative assessment of emergency response capabilities, generating pilots' comprehensive emergency response capability scores and emergency response capability assessment levels. Finally, through real-time monitoring and anomaly diagnosis, dynamic feedback tailored to individual characteristics is automatically triggered, and targeted capability improvement reports are generated, thereby significantly improving the accuracy of pilots' emergency response and training efficiency, ultimately ensuring aviation safety and operational effectiveness. Attached Figure Description

[0037] To more clearly illustrate the technical solutions of the exemplary embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly described below. It should be understood that the following drawings only show some embodiments of the present invention and should not be considered as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort. In the drawings:

[0038] Figure 1 A flowchart illustrating the multimodal detection and stress identification method for pilot emergency response capabilities provided in this application embodiment;

[0039] Figure 2 This is a functional block diagram of the multimodal detection and stress identification system for pilot emergency response capabilities provided in an embodiment of this application. Detailed Implementation

[0040] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the embodiments and accompanying drawings. The illustrative embodiments and descriptions of the present invention are only used to explain the present invention and are not intended to limit the present invention.

[0041] First, some of the technical terms used in this application will be explained to help those skilled in the art understand this application.

[0042] AE: Acoustic Emission;

[0043] HR: Heart Rate;

[0044] EDA: Electrodermal Activity;

[0045] HRV: Heart Rate Variability;

[0046] EEG: Electroencephalogram (EEG).

[0047] FCN-DAE: Fully Convolutional Network-Denoising Autoencoder;

[0048] AVNN: Average of all Normal to Normal intervals;

[0049] SDNN: Standard Deviation of all Normal to Normal Intervals;

[0050] LF: Low Frequency;

[0051] HF: High Frequency;

[0052] ROI: Region of Interest;

[0053] PFD: Primary Flight Display;

[0054] PCA: Principal Component Analysis;

[0055] GRU: Gated Recurrent Unit;

[0056] CNN: Convolutional Neural Network;

[0057] LSTM: Long Short-Term Memory;

[0058] TTS: Text-To-Speech;

[0059] HUD: Head-Up Display;

[0060] AOC: Airline Operations Center;

[0061] SimCLR: Simple Framework for Contrastive Learning of Visual Representations;

[0062] ResNet: Residual Network;

[0063] CRITIC: Criteria Importance Through Intercriteria Correlation, a standard importance method based on the correlation of indicators;

[0064] ICAO: International Civil Aviation Organization

[0065] Firstly, such as Figure 1 As shown, this application provides a multimodal detection and stress identification method for pilot emergency response capabilities, including the following steps:

[0066] Step S1: Synchronously collect multimodal state data of the pilot and preprocess it to generate standardized feature vectors;

[0067] Step S2: Perform cross-modal fusion analysis on the feature vectors to identify the multi-dimensional state of the pilot's emergency response capability;

[0068] Step S3: Based on the identified multi-dimensional states, construct a capability index system for quantitative evaluation, and generate the pilot's comprehensive emergency response capability score and emergency response capability assessment level;

[0069] Step S4: Based on the comprehensive emergency response capability score and emergency response capability assessment level, conduct real-time monitoring and anomaly diagnosis. When performance risks or operational anomalies are detected, automatically trigger dynamic feedback adapted to individual characteristics and generate a feedback report.

[0070] This embodiment achieves comprehensive perception of pilot status by synchronously collecting multimodal state data of pilots and eliminating data heterogeneity through standardized preprocessing. Then, cross-modal fusion analysis is used to deeply explore the correlation between surface stress and deep psychological traits, accurately identifying the multidimensional psychological states of pilots. Based on this, a capability index system is constructed to generate a pilot's comprehensive emergency response capability score and emergency response capability assessment level through weighted quantitative evaluation, objectively reflecting the pilot's emergency handling level. Finally, through real-time monitoring and anomaly diagnosis mechanisms, dynamic feedback adapted to individual characteristics is automatically triggered when performance risks are detected, and targeted capability improvement reports are generated, thereby improving the accuracy of pilot emergency response, training efficiency, and aviation safety management capabilities.

[0071] In one embodiment, step S1: synchronously acquiring multimodal state data of the pilot and preprocessing it to generate standardized feature vectors, specifically includes the following steps:

[0072] Step S11: Simultaneously acquire multimodal state data of the pilot, including psychological questionnaire data, physiological signal data, eye movement data, and flight performance data; the detailed plan for the multimodal state data acquisition of the pilot is shown in Table 1.

[0073] Table 1. Detailed List of Pilot Multimodal Status Data Acquisition Scheme

[0074] Step S12: Perform the following standardization processing on the collected multimodal state data to generate standardized feature vectors, including the following psychological feature values. Physiological feature vectors Eye movement feature vector and flight performance feature vector :

[0075] A. By using an improved multinomial statistical analysis model, quantitative analysis is performed on the questionnaire data to derive psychological characteristic values. The formula is:

[0076] Equation (1)

[0077] Where 1≤i≤j≤3, n=3, x i and x j The input features for the model are the various indicators collected from the pilot questionnaires. , , , and These are parameters based on expert experience; specifically, i and j are integer indices from 1 to 3, representing the numbers of the three indicators "psychological resilience", "self-efficacy" and "cognitive level" in the psychological questionnaire, respectively, and are used to calculate the interaction terms of the polynomial.

[0078] B. A fully convolutional autoencoder (FCN-DAE) is used to process physiological signal noise and extract physiological feature values. Specific HRV features include time-domain features (average AVNN and standard deviation SDNN of all normal sinus intervals) and frequency-domain features (low-frequency power LF, high-frequency power HF, and their ratio LF / HF). EEG features extract the theta and alpha wave power values, and the EEG feature vector is calculated using short-time Fourier transform. , A value greater than 1.2 indicates a state of tension; among which, , They are respectively Wave power, Wave power. In addition, EDA respiratory characteristics include mean skin conductivity rate. With respiratory rate As shown in the following formula, a unified physiological feature vector is finally generated by fusing the features of various physiological indicators. :

[0079] Equation (2)

[0080] C. Percentage of time spent looking at regions of interest The calculation formula is:

[0081] Equation (3)

[0082] ROI stands for Region of Interest, referring to a specific area on the interface that is crucial to the current task, such as flight attitude indicators, airspeed indicators, and engine parameter instruments; T ROI It is the total time spent on a specific ROI; It is the sum of the pilot's gaze time in all areas (including ROI and non-ROI).

[0083] Calculate gaze switching frequency :

[0084] Equation (4)

[0085] In the formula, K represents the number of switching operations. This parameter represents the average number of times the gaze point switches between different regions per unit of time, representing the total scene duration.

[0086] Generate eye-tracking feature vectors :

[0087] Equation (5)

[0088] in, This refers to the percentage of time spent looking at a specific region of interest (ROI), such as the primary flight display (PFD).

[0089] And calculate ROI compliance metrics. The formula is used to assess whether the pilot's total gaze time at all key ROIs throughout the emergency response process meets procedural expectations, as shown below:

[0090] = Equation (6)

[0091] D. The normalized flight performance characteristic data indicators are normalized using the min-max normalization method, and their calculation formula is as follows:

[0092] Equation (7)

[0093] The specific flight performance characteristic data indicators are as follows:

[0094] ①Reaction time: (The interval from emergency triggering to the first operation should be ≤1.5s).

[0095] ② Operational error rate: Operational error rate E = Number of errors / Total number of operations;

[0096] ③ Emergency procedure execution time: (Standard duration) (Based on ICAO program definition)

[0097] The flight performance feature vector obtained after normalization is:

[0098] Equation (8)

[0099] In the formula, and This is the normalized value.

[0100] In one embodiment, the preprocessed pilot feature data exhibits significant heterogeneity, with vastly different dimensions and information densities. Direct fusion would lead to a model biased towards high-dimensional features and neglect of low-value interference. First, Principal Component Analysis (PCA) is used to reduce the dimensionality of high-dimensional features (such as EEG power spectra) to eliminate redundancy and unify dimensions. Simultaneously, a cross-modal attention fusion mechanism is used to deeply correlate pilots' eye-tracking behavior and physiological stress responses during emergency response. Step S2 involves performing cross-modal fusion analysis on the feature vectors to identify the multi-dimensional states of pilots' emergency response capabilities, specifically including the following steps:

[0101] Step S21: Principal Component Analysis (PCA) is used to perform high-dimensional feature reduction on the preprocessed pilot feature data, and then the data is fused based on a cross-modal attention mechanism to obtain a fused feature vector, i.e., a deep fused feature vector; specifically, the following steps are included:

[0102] Step S211: The fixation duration sequence S eye =[T ROI1 ,T ROI2 ,...,T ROIk Input the gated recurrent unit (GRU) to obtain the eye-tracking sequence encoding H of its attention allocation. eye ;

[0103] Step S212: Physiological characteristic sequence S within the synchronization time window phys =[LF / HF,P θ / P α Input another GRU to obtain the physiological sequence encoding H. phys ;

[0104] Step S213: Calculate the eye-tracking sequence encoding H eye Encoding H of physiological sequences phys attention weights :

[0105] Equation (9)

[0106] in, This is a learnable weight matrix; the weights reflect the degree to which the pilot's attention allocation affects their physiological state at a specific moment.

[0107] Step S214: Using A cross Weighted fusion of physiological features yields enhanced physiological features strongly correlated with eye-movement behavior. ;

[0108] Ultimately, H eye and questionnaire features F q Flight performance characteristics The features are concatenated to form a deep fusion feature vector F. fusion−deep .

[0109] Equation (10)

[0110] In the formula, SE stands for Squeeze-and-Excitation, which is an attention mechanism module used to enhance the weights of key feature channels;

[0111] Step S22: Stress Status and Deep Psychological Recognition. Surface stress status recognition focuses on the pilot's real-time stress level in emergency scenarios, primarily manifested as "immediate fluctuations" in physiological signals such as heart rate (HR), electrical conductance of the skin (EDA), heart rate variability (HRV), and electroencephalography (EEG). Deep psychological trait assessment uncovers the pilot's long-term stable psychological foundation (such as psychological resilience, self-efficacy, and stress response tendency), specifically including:

[0112] The global model is maintained by a central server (such as the regulatory authority or certification center) and distributed to the local servers of each participating airline.

[0113] Each airline uses locally anonymized data (within its local network) to incrementally update the global model, generating local model update parameters ΔW. local .

[0114] The central server collects encrypted model update parameters from each airline and generates a new generation of global models using secure aggregation algorithms (such as weighted averaging). :

[0115] Equation (11)

[0116] in, Let N be the data volume of the k-th airline, and N be the total data volume. The polymerization rate is denoted as .

[0117] Model Iteration: A new generation of global models is distributed to begin a new round of iterations. This process only exchanges model parameters, not raw data, fundamentally protecting the data privacy of each airline, while leveraging distributed data to achieve a collective improvement in model performance.

[0118] Based on the aforementioned CNN-LSTM fusion model, the pilot's physiological characteristics are input. (HR / EDA / HRV / EEG):

[0119] CNN layer: Extracts EEG frequency domain features;

[0120] LSTM layer: Captures HR / EDA timing fluctuations;

[0121] Classification layer: Softmax outputs the pilot's surface pressure level, which is the pressure risk value mentioned later. (0 = no stress, 1 = mild, 2 = moderate, 3 = severe)

[0122] (2) In-depth psychological trait assessment: based on fusion characteristics A regression model of psychological indicators was constructed to calculate "emotion regulation ability". "Compressive toughness" ":

[0123] Equation (12)

[0124] Equation (13)

[0125] in , , , The model parameters can be obtained through training on "psychological traits-features" samples annotated by aviation experts. The evaluation levels are divided into S≥85 (excellent), 70≤S<85 (qualified), and S<70 (needs improvement).

[0126] Step S23: Determine the rationality of the manipulation intent and emergency operation, including:

[0127] A: A self-supervised pre-trained model (SimCLR) is used to process eye movement and performance features, and the operational intent is identified by comparing loss functions;

[0128] B: Calculate the emergency operation compliance score based on the safety priority rule base. :

[0129] Equation (14)

[0130] in, This refers to the actual operation of the pilot's emergency response at time t. This refers to the compliant operation required by the priority rules at time t. =0.9 is the time decay coefficient. For indicator functions (1 if true, 0 if false).

[0131] In one embodiment, step S3: Based on the identified multi-dimensional states, constructing a capability index system for quantitative evaluation, and generating a pilot's comprehensive emergency response capability score and emergency response capability assessment level, specifically includes the following steps:

[0132] Step S31: Construct an assessment index system, using the CRITIC method and expert scoring to determine weights, including psychological adaptability (weight 0.25), stress coping ability (weight 0.20), attention allocation ability (weight 0.20), and emergency response ability (weight 0.35), as shown in the table below:

[0133] Table 2. Assessment Table for Calculating the Comprehensive Competency Score of Pilots and Other Professionals

[0134] Step S32: Linearly normalize each secondary indicator to [0, 100] to obtain the normalized secondary indicators;

[0135] Equation (15)

[0136] The normalized secondary indicators are weighted and summed to obtain the pilot's comprehensive score for emergency response capability. :

[0137] Equation (16)

[0138] In the formula, , , , These are the weights of the normalized secondary indicators A, B, C, and D, respectively.

[0139] And based on the pilot's comprehensive emergency response capabilities, a score will be awarded. In addition, the overall capability level is used to determine the emergency response capability level, and the specific standards are as follows:

[0140] Level I (Excellent): ;

[0141] Level II (Qualified): ;

[0142] Level III (To be upgraded): .

[0143] In one embodiment, step S4: Real-time monitoring and anomaly diagnosis are performed based on the comprehensive emergency response capability score and emergency response capability assessment level. When performance risks or operational anomalies are detected, dynamic feedback adapted to individual characteristics is automatically triggered and a feedback report is generated. Specifically, this includes the following steps:

[0144] Step S41: Perform anomaly detection and response; specifically, the terminal verifies every 30 seconds, and an anomaly is triggered when any of the following conditions are met. Response measures include pop-up notifications, push alarms to the AOC command center, and calling the qualified data stored on the blockchain to restore guidance: push anomaly logs in the following format:

[0145] { , , , , Equation (17)

[0146] in, For terminal identifier, For timestamps, This is an operation error type. As a feature of fusion, This is a score based on overall ability.

[0147] If severe pressure persists for more than 10 seconds and the operation error rate E > 20%, the pressure condition is determined to be abnormal.

[0148] If compliance score The operation was deemed to be non-compliant.

[0149] If the missing rate of multimodal data is >30%, the data integrity is considered abnormal.

[0150] Step S42: Based on real-time calculated pressure risk value Attention deviation and operational risk index When the risk value exceeds the threshold, personalized feedback is dynamically generated through voice prompts, visual prompts, or pop-up messages, and intervention logs are recorded for optimization.

[0151] The system initiates a high-frequency (e.g., 1Hz) lightweight real-time analysis thread to analyze physiological characteristic data within a sliding time window (e.g., 5 seconds). The analysis focuses on key stress indicators, particularly the heart rate variability ratio (LF / HF) and brainwave power ratio (Pθ / Pα). Then, a pre-trained lightweight classifier (such as logistic regression or a small neural network) rapidly calculates these features, outputting a probability value representing the risk of the current stress level deviating from the normal range. ;

[0152] The system is based on real-time acquired eye-tracking features The system focuses on examining the pilot's fixation duration (Ri) and fixation switching frequency (F) for the current emergency procedure's key areas of interest (ROI). This real-time eye-tracking data is compared to the standard attention allocation pattern expected in the current emergency phase from the built-in "attention rule base." By calculating the difference between the real-time data and the expected pattern (such as cosine similarity or percentage difference), the system quantifies the degree to which the pilot's attention allocation deviates from the procedure requirements. This quantified value is the... ;

[0153] The system continuously monitors the pilot's real-time operational sequence (part of flight performance data). This is then compared with the "Flight Safety Priority Rule Base." This differs from ex post-event scoring of operational compliance (Scomp). It focuses on forward-looking prediction. By analyzing recent operational context, the system assesses the likelihood of a pilot violating safety rules in their next action, thereby calculating a probability index that predicts an impending operational error. .

[0154] The following uses a simulator engine failure emergency scenario as an example to explain in detail the implementation process of the multimodal detection and stress identification method for pilot emergency response capabilities provided in this application:

[0155] 1. Implementation of Multimodal Data Acquisition and Preprocessing

[0156] (1) Data collection

[0157] The simulator constructed a "left engine failure during takeoff" scenario, lasting 120 seconds, with 10 pilots participating in the test;

[0158] Equipment deployment:

[0159] Questionnaire data collection: Questionnaires were distributed electronically before the test;

[0160] Physiological data acquisition: HR (700Hz) and EDA (100Hz) were acquired using a flexible ECG patch; data were also acquired using a head-mounted EEG sensor. Wave (250Hz);

[0161] Eye-tracking data acquisition: The helmet integrates an eye tracker, dividing the ROI into "engine instrument panel ROI", "throttle lever ROI", and "fire alarm light ROI";

[0162] Flight performance data acquisition: The simulator flight control system collects "reaction time", "operational error rate" and "emergency procedure execution time".

[0163] (2) Preprocessing results

[0164] Questionnaire characteristics: Psychological resilience score =85;

[0165] Physiological characteristics: HRV index extracted after FCN-DAE denoising. =850ms , (Moderate stress) ,

[0166] (HR / SDNN / LF / HF / EDA / Respiratory Rate);

[0167] Eye movement characteristics:

[0168] ;

[0169] Flight performance: Reaction time = 1.2 seconds ( ), Operational error rate Emergency procedure execution time = 45 seconds ( ).

[0170] , , =45s ( After normalization .

[0171] 2. Implementation of Feature Fusion and State Recognition

[0172] (1) Feature fusion

[0173] PCA dimensionality reduction: The EEG power spectrum (originally 128 dimensions) is reduced to 32 dimensions.

[0174] Sequence Construction and Encoding: For the "engine failure" emergency scenario, a 30-second time window is extracted from the start of the failure alarm. Eye-tracking data (the fixation duration sequence Seye, focusing on the "engine instrument ROI" and "throttle lever ROI") at 1-second intervals within this window, along with synchronized physiological data (LF / HF ratio sequence Sphys), are input into two independent gated recurrent unit (GRU) networks to obtain their dynamic context encodings Heye and Hphys.

[0175] Attention Calculation and Feature Enhancement: Calculating cross-modal attention weights (Across). Example calculations show that the correlation weight between the pilot's gaze behavior toward the "engine instrument ROI" and the fluctuation of its LF / HF ratio reaches its peak (Across=0.72) 3-5 seconds after the warning, indicating that attention allocation during this phase is highly synchronized with physiological stress response.

[0176] The physiological feature encoding Hphys is weighted using Across to obtain an enhanced physiological feature vector Hphys′ weighted by eye-tracking behavior. This vector is then concatenated to form a deep fusion feature vector Ffusion−deep. The fusion feature dimension is 192 (questionnaire 3 + enhanced physiological 8 + eye-tracking 4 + performance 5).

[0177] (2) Status recognition results

[0178] Stress State: The CNN-LSTM model outputs "Moderate stress (S=2)", compared to... match;

[0179] Deep Psychology: Emotional Regulation Ability (Qualified), compressive toughness =82 (Excellent);

[0180] Manipulation intent: The SimCLR model identifies "prioritizing throttle adjustment → checking engine parameters → requesting forced landing," which aligns with compliance intent. Figure 1 To;

[0181] Operational compliance: =88 (Complies with the priority rule of "engine failure → throttle control → attitude adjustment").

[0182] 3. Implementation of Comprehensive Assessment of Emergency Response Capabilities

[0183] (1) Calculation of indicators

[0184] Psychological adaptability (Weight 0.25);

[0185] Stress coping ability (Weight 0.20);

[0186] Attention allocation ability (Weight 0.20);

[0187] Emergency operation capability (Weight 0.35).

[0188] (2) Overall score and grade

[0189] →Level II (Pass), the main weakness is "attention allocation (C=45.8)".

[0190] 4. Implementation of Terminal Response and Feedback

[0191] Anomaly Detection: No anomalies found (pressure not reached severe level, operation compliant);

[0192] Personalized feedback:

[0193] At the 38th second of the simulation training (standard procedure requires pilots to check engine parameters at this point), the system's real-time monitoring thread detected an increase in real-time attention deviation (Dattention) (the percentage of the pilot's gaze at the engine instrument ROI, Ri, was below the threshold of 40% for 5 seconds). Simultaneously, the real-time stress risk value, Rstress, also increased (the LF / HF ratio rose to 2.3).

[0194] The system determined that the pilot had a risk of "distraction" and, combined with the pilot's assessment report showing a weakness in "attention allocation ability C=45.8," immediately triggered dynamic feedback. A prompt was issued via TTS: "Please check N1 RPM and EGT temperature." Upon receiving the prompt, the pilot quickly returned their gaze to the engine instrument panel and completed the check within 2 seconds, improving subsequent operational compliance. The type of intervention event, the trigger time, and the pilot's response were recorded by the system for optimizing the pilot's personalized feedback model.

[0195] Secondly, such as Figure 2As shown, this application provides a multimodal detection and stress identification system for pilot emergency response capabilities, used to implement the multimodal detection and stress identification method for pilot emergency response capabilities as described above; the system includes a data acquisition and preprocessing module 100, a feature fusion and state identification module 200, an emergency response capability comprehensive evaluation module 300, and a terminal response and personalized feedback module 400;

[0196] The data acquisition and preprocessing module 100 is used to synchronously acquire multimodal state data of the pilot and preprocess it to generate standardized feature vectors;

[0197] The feature fusion and state recognition module 200 is communicatively connected to the data acquisition and preprocessing module 100, and is used to perform cross-modal fusion analysis on the standardized feature vectors and identify multi-dimensional states that characterize the pilot's emergency response capabilities.

[0198] The emergency response capability comprehensive evaluation module 300 is communicatively connected to the feature fusion and state recognition module 200, and is used to perform quantitative evaluation based on the identified multi-dimensional states to generate the pilot's comprehensive emergency response capability score and emergency response capability assessment level.

[0199] The terminal response and personalized feedback module 400 is communicatively connected to the emergency response capability comprehensive evaluation module 300, and is used to perform real-time monitoring and anomaly diagnosis based on the emergency comprehensive capability score and emergency capability judgment level, and trigger dynamic feedback intervention and generate capability improvement reports.

[0200] In one embodiment, the terminal response and personalized feedback module is communicatively connected to the cockpit display system and voice system of a flight simulator or a real aircraft, and is used to perform visual and voice prompts.

[0201] The functions of each module in the above-mentioned multimodal detection and stress identification system for pilot emergency response capabilities correspond to the steps in the above-mentioned embodiment of the multimodal detection and stress identification method for pilot emergency response capabilities. Their functions and implementation processes will not be described in detail here.

[0202] Thirdly, embodiments of this application provide a multimodal detection and stress identification device for pilot emergency response capabilities. This device can be a personal computer (PC), laptop computer, server, or other device with data processing capabilities.

[0203] In this embodiment of the application, the multimodal detection and stress identification device for pilot emergency response capabilities may include a processor, a memory, a communication interface, and a communication bus.

[0204] The communication bus can be of any type and is used to interconnect the processor, memory, and communication interface.

[0205] The communication interface includes input / output (I / O) interfaces, physical interfaces, and logical interfaces. These interfaces enable interconnection of internal components within the multimodal detection and stress identification equipment for pilot emergency response capabilities, as well as interfaces for interconnection between the equipment and other devices (such as other computing devices or user equipment). Physical interfaces can be Ethernet interfaces, fiber optic interfaces, ATM interfaces, etc.; user equipment can be displays, keyboards, etc.

[0206] Memory can be various types of storage media, such as random access memory (RAM), read-only memory (ROM), non-volatile RAM (NVRAM), flash memory, optical storage, hard disk, programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), etc.

[0207] The processor can be a general-purpose processor, which can call the multimodal detection and stress identification program for pilot emergency response capabilities stored in memory and execute the multimodal detection and stress identification method for pilot emergency response capabilities provided in the embodiments of this application. For example, the general-purpose processor can be a central processing unit (CPU). The method executed when the multimodal detection and stress identification program for pilot emergency response capabilities is called can refer to the various embodiments of the multimodal detection and stress identification method for pilot emergency response capabilities of this application, and will not be repeated here.

[0208] Fourthly, embodiments of this application also provide a readable storage medium.

[0209] The present application stores a pilot emergency response capability multimodal detection and stress identification program on a readable storage medium, wherein when the pilot emergency response capability multimodal detection and stress identification program is executed by a processor, it implements the steps of the pilot emergency response capability multimodal detection and stress identification method as described above.

[0210] The method implemented when the multimodal detection and stress identification procedure for pilot emergency response capability is executed can refer to the various embodiments of the multimodal detection and stress identification method for pilot emergency response capability of this application, and will not be repeated here.

[0211] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A multimodal detection and stress identification method for pilots' emergency response capabilities, characterized in that, Includes the following steps: Simultaneously collect multimodal state data of the pilot and preprocess it to generate standardized feature vectors; Cross-modal fusion analysis of feature vectors is performed to identify the multi-dimensional state of a pilot's emergency response capability; Based on the identified multi-dimensional states, a capability index system is constructed for quantitative evaluation, generating a pilot's comprehensive emergency response capability score and emergency response capability assessment level. Real-time monitoring and anomaly diagnosis are conducted based on the comprehensive emergency response capability score and emergency response capability assessment level. When performance risks or operational anomalies are detected, dynamic feedback adapted to individual characteristics is automatically triggered and a feedback report is generated.

2. The multimodal detection and stress identification method for pilot emergency response capabilities as described in claim 1, characterized in that, The multimodal state data includes psychological questionnaire data, physiological signal data, eye-tracking data, and flight performance data.

3. The multimodal detection and stress identification method for pilot emergency response capabilities as described in claim 2, characterized in that, The cross-modal fusion analysis includes: Feature sequences from different modalities are input into corresponding recurrent neural networks to obtain their respective dynamic context encodings; Calculate cross-modal attention weights between different dynamic context encodings; The cross-modal attention weights are used to perform weighted fusion of the corresponding feature codes to generate a deep fusion feature vector.

4. The multimodal detection and stress identification method for pilot emergency response capabilities as described in claim 1, characterized in that, The multi-dimensional state includes the pilot's real-time stress state, deep psychological traits, control intentions, and compliance with emergency operations.

5. The multimodal detection and stress identification method for pilot emergency response capabilities as described in claim 4, characterized in that, The identification of emergency operation compliance includes: A pre-configured rule base based on flight safety priority rules; Based on the pilot's actual operational sequence and the required sequences in the rule base, a compliance score is calculated using a time decay weighted average. The calculation formula is: )； in, This refers to the actual operation of the pilot's emergency response at time t. This refers to the compliant operation required by the priority rules at time t. The time decay coefficient, The indicator function is T, which represents the total number of time steps from the start of the operation to the end of the current evaluation.

6. The multimodal detection and stress identification method for pilot emergency response capabilities as described in claim 1, characterized in that, The dynamic feedback intervention is based on real-time calculated stress risk value, attention deviation degree and operational risk index, and is triggered through one or more of the following methods: voice prompts, visual prompts or information pop-ups.

7. The multimodal detection and stress identification method for pilot emergency response capability as described in claim 1, characterized in that, The method further includes incrementally updating the model used for state recognition using a federated learning paradigm, wherein the incremental update includes: The central server distributes the global model to each local server; Each local server uses local data to incrementally update the global model, generates encrypted model update parameters, and uploads them to the central server. The central server securely aggregates the collected encrypted model update parameters to generate a new generation of global model.

8. A multimodal detection and stress identification system for pilot emergency response capabilities, characterized in that, A method for multimodal detection and stress identification of pilot emergency response capabilities as described in any one of claims 1 to 7; the system comprises: The data acquisition and preprocessing module is used to simultaneously acquire multimodal state data of the pilot and preprocess it to generate standardized feature vectors. The feature fusion and state recognition module is communicatively connected to the data acquisition and preprocessing module. It is used to perform cross-modal fusion analysis on the standardized feature vectors and identify multi-dimensional states that characterize the pilot's emergency response capabilities. The emergency response capability comprehensive evaluation module is communicatively connected to the feature fusion and state recognition module. It is used to perform quantitative evaluation based on the identified multi-dimensional states and generate the pilot's comprehensive emergency response capability score and emergency response capability assessment level. The terminal response and personalized feedback module is communicatively connected to the emergency response capability comprehensive evaluation module. It is used to perform real-time monitoring and anomaly diagnosis based on the emergency comprehensive capability score and emergency capability judgment level, and to trigger dynamic feedback intervention and generate capability improvement reports.

9. The multimodal detection and stress identification system for pilot emergency response capabilities as described in claim 8, characterized in that, The terminal response and personalized feedback module is connected to the cockpit display system and voice system of the flight simulator or real aircraft to provide visual and voice prompts.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a multimodal detection and stress identification program for pilot emergency response capabilities, wherein when the multimodal detection and stress identification program for pilot emergency response capabilities is executed by a processor, it implements the steps of the multimodal detection and stress identification method for pilot emergency response capabilities as described in any one of claims 1 to 7.

Images