A method for predicting pilot stress and emotional state based on video, audio, and QAR data

Abstract

This invention discloses a method for inferring pilot stress and emotional states based on video, audio, and QAR data, belonging to the field of intelligent perception technology for multimodal pilot psychological states. The method includes temporal alignment, noise reduction, and data cleaning of the acquired data; unified modeling of multimodal features; outputting results and generating interpretive visualization reports. This invention improves the usability and deployment convenience of the system in real flight or simulated training environments; enhances the accuracy and robustness of psychological state recognition; strengthens the model's cross-human generalization ability; and meets the stringent real-time and reliability requirements of aviation scenarios.

Description

Technical Field

[0001] This invention relates to the field of intelligent perception technology for multimodal pilot psychological states, and in particular to a method for inferring pilot stress and emotional states based on video, audio, and QAR data. Background Technology

[0002] With the rapid development of the air transport industry and the increasing complexity of flight missions, pilots are experiencing increasingly severe psychological burdens and emotional stress during training and actual flight operations. Scenarios such as high-density airspace, extreme weather responses, and emergency malfunction handling place extremely high demands on pilots' cognitive abilities, decision-making efficiency, and emotional stability. At the same time, the civil aviation safety regulatory system is increasingly emphasizing the crucial role of human factors in the flight accident chain, urgently requiring a technological means to monitor pilots' psychological state in real-time, objectively, and non-invasively, in order to achieve early warning, intervention, optimized training, and improved human-machine collaboration efficiency.

[0003] However, existing technologies still have significant shortcomings in assessing pilots' psychological state. Traditional methods mostly rely on subjective questionnaires or physiological sensors, such as electrocardiograms and electrodermal data, for stress assessment. The former suffers from poor timeliness and is susceptible to recall bias, while the latter requires wearing specialized equipment, which not only interferes with normal operations but is also difficult to deploy on a large scale in real flight environments. Although some studies have attempted to use single modalities, such as video or voice alone, for emotion recognition, they are limited by issues such as incomplete modal information, significant environmental noise interference, and uncalibrated individual differences, resulting in low accuracy, weak generalization ability, and a lack of interpretability. Furthermore, existing systems generally lack modeling of the correlation between flight operation behavior data and psychological state, making it impossible to achieve temporal alignment and collaborative reasoning of multi-source heterogeneous information, and thus failing to meet the comprehensive requirements of reliability, real-time performance, and individual adaptability in aviation scenarios. Summary of the Invention

[0004] This invention addresses the problems existing in current methods for inferring pilot stress and emotional state based on video, audio, and QAR data.

[0005] Therefore, the problem to be solved by this invention is: how to achieve real-time, accurate, individualized, and interpretable non-contact monitoring of pilot stress and emotional state by integrating video, audio, and flight operation data without relying on wearable devices.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0007] In a first aspect, embodiments of the present invention provide a method for inferring pilot stress and emotional state based on video, audio and QAR data, which includes acquiring video, audio and flight operation data, and performing time-series alignment, noise reduction and data cleaning on the acquired data;

[0008] Extract visual, acoustic, and operational parameter features, and complete unified modeling of multimodal features through Transformer networks or attention mechanisms;

[0009] Based on the fusion features, the stress index and emotional state are calculated, an individualized benchmark is constructed, individual differences are eliminated, and the results are output and an interpretive visualization report is generated.

[0010] As a preferred embodiment of the pilot stress and emotional state inference method based on video, audio and QAR data described in this invention, the step of performing time-series alignment, noise reduction and data cleaning on the acquired data includes using multimodal data acquisition and time synchronization, and installing high-definition cameras, directional microphone arrays and QAR data interfaces in flight simulators or real cockpits.

[0011] The high-definition camera includes those located above the instrument panel and to the left front, which capture images of the pilot's face and upper body.

[0012] The directional microphone array is used to collect pilot voice and suppress background wind noise and cabin broadcast interference. The QAR data interface is used to read flight parameters, including pitch angle, roll angle, heading angle, control input, flight altitude, speed and acceleration.

[0013] The acquired data is sampled synchronously with a unified clock, video frame timestamps are corrected using PTS, audio signals are resampled according to the sampling rate, and time step interpolation is performed using the QAR data interface.

[0014] A synchronization module is used to maintain a global buffer and output trimodal frame groups according to time windows.

[0015] As a preferred embodiment of the pilot stress and emotional state inference method based on video, audio and QAR data described in this invention, the extraction of visual, acoustic and operational parameter features includes video feature extraction, audio feature extraction and QAR feature extraction;

[0016] The video feature extraction includes using MTCNN-based face detection and KLT-based point tracking algorithms to extract facial features, eye indicators, head pose, and micro-expression detection;

[0017] The facial expression features include facial action unit analysis, such as AU1, AU4, and AU12, and their dynamic intensity is calculated.

[0018] The eye indicators include blink frequency, eyelid closure time, and gaze stability, used to assess attention and stress.

[0019] The head posture includes three-dimensional posture angles that reflect alertness or distraction.

[0020] The micro-expression detection includes using an I3D network to detect micro-expression changes lasting less than 500ms, in order to capture instantaneous emotional fluctuations.

[0021] As a preferred embodiment of the pilot stress and emotional state inference method based on video, audio and QAR data described in this invention, the audio feature extraction includes extracting low-level acoustic features and high-level speech features after the audio signal has been processed by speech activity detection and beamforming noise reduction.

[0022] The low-level acoustic features include fundamental frequency F0, formants, energy, spectral centroid, MFCC, ΔMFCC, rate of change of intensity, jitter, and shimmer.

[0023] The high-level speech features include extracting contextual speech embeddings based on the Wav2Vec2 pre-trained model and analyzing semantic sentiment using the BERT model.

[0024] As a preferred embodiment of the pilot stress and emotional state inference method based on video, audio and QAR data described in this invention, the QAR feature extraction includes QAR data being processed through a sliding window to calculate operational stability features, flight attitude fluctuations, flight load parameters and control feedback indicators.

[0025] The operational stability characteristic refers to the joystick input volatility; the flight attitude volatility refers to the pitch rate.

[0026] The flight load parameter refers to the rate of change of vertical acceleration; the control feedback index refers to the redundant control input.

[0027] As a preferred embodiment of the pilot stress and emotional state inference method based on video, audio and QAR data described in this invention, the unified modeling of multimodal features through Transformer network or attention mechanism includes using a cross-modal Transformer network, inputting a multi-head attention layer through the attention mechanism to model the correlation between modalities, completing temporal dependency modeling through a temporal convolutional network or Bi-LSTM, and extracting the emotional change trend.

[0028] The overall loss function is obtained by connecting the stress index prediction head and the emotion dimension prediction head; the stress index prediction head refers to the output of Stress Index St∈[0,1] through a fully connected sigmoid function; the emotion dimension prediction head refers to the output of the values ​​of the emotion of pleasure (Valence) and activation (Arousal).

[0029] The formula for the overall loss function is:

[0030] ;

[0031] in, Represents the total loss function. , and Indicates the weighting coefficient. This indicates the loss from the stress prediction task. This indicates the loss in the emotion prediction task. This represents the loss of temporal consistency.

[0032] The cross-modal Transformer network includes the following inputs for computation time step t:

[0033] ;

[0034] in, This represents the fused feature vector at time step t. This represents the visual feature vector at time step t. This represents the audio feature vector at time step t. This represents the QAR feature vector at time step t;

[0035] The formula for the correlation between the multi-head attention layer modeling modalities is:

[0036] ;

[0037] in, This represents the attention mechanism function. Represents the query matrix. Represents the key matrix. Represents a value matrix, This represents the multimodal fusion feature vector at time step t.

[0038] As a preferred embodiment of the pilot stress and emotional state inference method based on video, audio and QAR data described in this invention, the construction of individualized benchmarks includes recording the average feature vector of multimodal signals of the pilot in a relaxed state during the baseline modeling phase, and calculating the relative deviation during detection.

[0039] The formula for the average eigenvector of the multimodal signal is:

[0040] ;

[0041] in, This represents the baseline vector of the pilot's visual features when in a relaxed state. This represents the baseline vector of audio features when the pilot is in a relaxed state. This represents the baseline vector of the QAR operational features of the pilot in a relaxed state;

[0042] The formula for calculating the relative deviation during detection is:

[0043] ;

[0044] in, This represents the visual feature bias vector at time step t. This represents the audio feature deviation vector at time step t. This represents the QAR operation feature bias vector at time step t. Representing speech features, Indicates flight operation parameters;

[0045] The output results, which generate interpretive visualization reports, include output stress index and emotional state.

[0046] Secondly, embodiments of the present invention provide a pilot stress and emotional state prediction system based on video, audio and QAR data, which includes: a multimodal data acquisition module, used to acquire video, audio and flight operation data;

[0047] The signal preprocessing and time synchronization module is used to perform timing alignment, noise reduction, and data cleaning.

[0048] The multimodal feature extraction module is used to extract visual, acoustic, and operational parameter features;

[0049] The cross-modal temporal fusion module is used to achieve unified modeling of multimodal features through Transformer networks or attention mechanisms;

[0050] The emotion and stress inference module is used to calculate the stress index and emotional state based on fused features.

[0051] The Individual Baseline Modeling and Adaptation module is used to build individualized baselines and eliminate individual differences.

[0052] The feedback and visualization module is used to output results and generate explanatory visualization reports.

[0053] Thirdly, embodiments of the present invention provide a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement any step of the above-described method for inferring pilot stress and emotional state based on video, audio, and QAR data.

[0054] Fourthly, embodiments of the present invention provide a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program is executed by a processor, it implements any step of the above-described method for inferring pilot stress and emotional state based on video, audio, and QAR data.

[0055] The beneficial effects of this invention are as follows: By integrating video, audio, and QAR flight operation data, this invention constructs a non-contact, multimodal collaborative real-time inference method for pilot stress and emotional state, which has significant advantages: First, it eliminates the need for any physiological sensors, avoiding interference with the pilot's normal operation and improving the system's usability and deployment convenience in real flight or simulated training environments; second, through high-precision time synchronization and cross-modal attention mechanisms, it effectively integrates three complementary types of information—visual expression, vocal emotion, and operational behavior—significantly improving the accuracy and robustness of psychological state recognition; third, by introducing individual baseline modeling and differential feature calculation mechanisms, it can automatically correct for individual differences among pilots in facial expression habits, tone styles, and operational preferences, significantly enhancing the model's cross-human generalization ability; furthermore, the system output not only includes quantitative results for stress index and emotional dimension but also provides interpretable information such as feature contribution and confidence intervals, facilitating retrospective analysis and intervention decisions by instructors or safety systems; finally, the overall architecture supports embedded edge deployment with an inference latency of less than 500 milliseconds, meeting the stringent requirements of aviation scenarios for real-time performance and reliability. Attached Figure Description

[0056] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein:

[0057] Figure 1 A flowchart illustrating a method for inferring pilot stress and emotional state based on video, audio, and QAR data, provided in an embodiment of the present invention.

[0058] Figure 2 This is a system schematic diagram of a method for inferring pilot stress and emotional state based on video, audio, and QAR data, provided in an embodiment of the present invention. Detailed Implementation

[0059] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of the present invention.

[0060] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0061] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0062] This invention is described in detail with reference to the schematic diagrams. When detailing the embodiments of this invention, for ease of explanation, the cross-sectional views illustrating the device structure may be partially enlarged, not adhering to the usual scale. Furthermore, the schematic diagrams are merely examples and should not be construed as limiting the scope of protection of this invention. In actual fabrication, the three-dimensional spatial dimensions of length, width, and depth should be included.

[0063] Furthermore, in the description of this invention, it should be noted that the terms "upper," "lower," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. These terms are used solely for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. In addition, the terms "first," "second," or "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0064] Unless otherwise explicitly specified and limited, the terms "installation," "connection," and "joining" in this invention should be interpreted broadly. For example, they can refer to fixed connections, detachable connections, or integral connections; similarly, they can refer to mechanical connections, electrical connections, or direct connections, or indirect connections through an intermediate medium, or internal connections between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0065] Example

[0066] Reference Figure 1 and Figure 2 This is the first embodiment of the present invention, which provides a method for inferring pilot stress and emotional state based on video, audio, and QAR data, including:

[0067] S1: Acquire video, audio, and flight operation data, and perform time-series alignment, noise reduction, and data cleaning on the acquired data.

[0068] Among them, the time alignment, noise reduction and data cleaning of the acquired data include the use of multimodal data acquisition and time synchronization, and the installation of high-definition cameras, directional microphone arrays and QAR data interfaces in flight simulators or real cockpits;

[0069] High-definition cameras, including those located above the instrument panel and to the left front, capture images of the pilot's face and upper body.

[0070] The directional microphone array is used to collect pilot voice and suppress background wind noise and cabin broadcast interference. The QAR data interface is used to read flight parameters, including pitch angle, roll angle, yaw angle, control input, flight altitude, speed and acceleration.

[0071] The acquired data is sampled synchronously with a unified clock, video frame timestamps are corrected using PTS, audio signals are resampled according to the sampling rate, and time step interpolation is performed using the QAR data interface.

[0072] A synchronization module is used to maintain a global buffer and output trimodal frame groups according to time windows.

[0073] Furthermore, the acquired data undergoes time-series alignment, noise reduction, and data cleaning, including the use of multimodal data acquisition and time synchronization, and the installation of high-definition cameras, directional microphone arrays, and QAR data interfaces in flight simulators or real cockpits.

[0074] High-definition cameras, including those located above the instrument panel and to the left front, capture images of the pilot's face and upper body (at 30fps or higher).

[0075] The directional microphone array is used to collect pilot voice and suppress background wind noise and cabin broadcast interference. The QAR data interface is used to read flight parameters, including pitch angle, roll angle, heading angle, control inputs (throttle, joystick, rudder), flight altitude, speed and acceleration.

[0076] The acquired data is sampled synchronously with a unified clock, the video frame timestamps are corrected using PTS (Presentation Time Stamp), the audio signal is resampled according to the sampling rate, and the QAR data interface is interpolated in 100ms time steps.

[0077] A synchronization module is used to maintain the global buffer, and three-modal frame groups are output according to the time window Δt=2 seconds.

[0078] S2: Extract visual, acoustic, and operational parameter features, and complete the unified modeling of multimodal features through Transformer networks or attention mechanisms.

[0079] Extracting visual, acoustic, and operational parameter features includes video feature extraction, audio feature extraction, and QAR feature extraction;

[0080] Video feature extraction includes using MTCNN-based face detection and KLT-based point tracking algorithms to extract facial features, eye metrics, head pose, and micro-expression detection;

[0081] Facial expression features include analysis based on facial action units, such as AU1, AU4, and AU12, and calculation of their dynamic intensity;

[0082] Eye indicators, including blink frequency, eyelid closure time, and gaze stability, are used to assess attention and stress.

[0083] Head posture, including three-dimensional posture angles, reflects alertness or distraction.

[0084] Micro-expression detection includes using the I3D network to detect micro-expression changes lasting less than 500ms, in order to capture instantaneous emotional fluctuations.

[0085] Audio feature extraction includes extracting low-level acoustic features and high-level speech features from audio signals after speech activity detection and beamforming noise reduction processing;

[0086] Low-level acoustic features include fundamental frequency F0, formants, energy, spectral centroid, MFCC, ΔMFCC, rate of change of intensity, jitter, and shimmer.

[0087] High-level speech features include extracting contextual speech embeddings using a Wav2Vec2 pre-trained model and analyzing semantic sentiment using a BERT model.

[0088] Audio feature extraction includes extracting low-level acoustic features and high-level speech features from audio signals after speech activity detection and beamforming noise reduction processing;

[0089] Low-level acoustic features include fundamental frequency F0, formants, energy, spectral centroid, MFCC (13 dimensions), ΔMFCC, rate of change of intensity, jitter, and shimmer.

[0090] High-level speech features include extracting contextual speech embeddings based on the Wav2Vec2 pre-trained model and analyzing semantic sentiment (positive / negative tone, hesitation, command, anxiety, etc.) using the BERT model.

[0091] The audio features are averaged and pooled over time to generate a 256-dimensional audio embedding vector At.

[0092] QAR feature extraction involves processing QAR data through a sliding window to calculate operational stability features, flight attitude fluctuations, flight load parameters, and control feedback indicators.

[0093] Operational stability characteristics refer to the joystick input fluctuation rate; flight attitude fluctuation refers to the pitch rate.

[0094] Flight load parameters refer to the rate of change of vertical acceleration; control feedback parameters refer to redundant control inputs.

[0095] QAR feature extraction involves processing QAR data through a sliding window to calculate operational stability features, flight attitude fluctuations, flight load parameters, and control feedback indicators.

[0096] Operational stability characteristics refer to the joystick input fluctuation rate; flight attitude fluctuation refers to the pitch rate.

[0097] Flight load parameters refer to the rate of change of vertical acceleration; control feedback indicators refer to redundant control parameters. These features are normalized to form a 64-dimensional QAR embedding vector Qt.

[0098] S2: Extract visual, acoustic, and operational parameter features, and complete the unified modeling of multimodal features through Transformer networks or attention mechanisms.

[0099] Among them, the unified modeling of multimodal features through Transformer networks or attention mechanisms includes using cross-modal Transformer networks, inputting multi-head attention layers through attention mechanisms to model the correlation between modalities, completing temporal dependency modeling through temporal convolutional networks or Bi-LSTM, and extracting the trend of emotion changes.

[0100] The overall loss function is obtained by connecting the stress index prediction head and the emotion dimension prediction head; the stress index prediction head is to output the Stress Index St∈[0,1] through a fully connected sigmoid function; the emotion dimension prediction head is to output the values ​​of the emotion of pleasure (Valence) and activation (Arousal).

[0101] The formula for the overall loss function is:

[0102] ;

[0103] in, Represents the total loss function. , and Indicates the weighting coefficient. This indicates the loss from the stress prediction task. This indicates the loss in the emotion prediction task. This represents the loss of temporal consistency.

[0104] The cross-modal Transformer network includes the following inputs for computing time step t:

[0105] ;

[0106] in, This represents the fused feature vector at time step t. This represents the visual feature vector at time step t. This represents the audio feature vector at time step t. This represents the QAR feature vector at time step t;

[0107] The formula for the correlation between modalities in multi-head attention layer modeling is:

[0108] ;

[0109] in, This represents the attention mechanism function. Represents the query matrix. Represents the key matrix. Represents a value matrix, This represents the multimodal fusion feature vector at time step t.

[0110] Unified modeling of multimodal features through Transformer networks or attention mechanisms includes using cross-modal Transformer networks, inputting multi-head attention layers through attention mechanisms to model intermodal correlations, and using temporal convolutional networks or Bi-LSTM to complete temporal dependency modeling, extracting emotional change trends within a 2–5 second time range.

[0111] The overall loss function is obtained by connecting the stress index prediction head and the emotion dimension prediction head; the stress index prediction head is to output the Stress Index St∈[0,1] through a fully connected sigmoid function; the emotion dimension prediction head is to output the values ​​of the emotion of pleasure (Valence) and activation (Arousal).

[0112] The formula for the overall loss function is:

[0113] ;

[0114] in, Represents the total loss function. , and Indicates the weighting coefficient. This indicates the loss from the stress prediction task. This indicates the loss in the emotion prediction task. This represents the loss of temporal consistency.

[0115] The cross-modal Transformer network includes the following inputs for computing time step t:

[0116] ;

[0117] in, This represents the fused feature vector at time step t. This represents the visual feature vector at time step t. This represents the audio feature vector at time step t. This represents the QAR feature vector at time step t;

[0118] The formula for the correlation between modalities in multi-head attention layer modeling is:

[0119] ;

[0120] in, This represents the attention mechanism function. Represents the query matrix. Represents the key matrix. Represents a value matrix, This represents the multimodal fusion feature vector at time step t.

[0121] S3: Calculates stress index and emotional state based on fusion features, constructs individualized benchmarks, eliminates individual differences, outputs results and generates interpretive visualization reports.

[0122] Among them, building an individualized benchmark includes recording the average feature vector of multimodal signals of the pilot in a relaxed state during the baseline modeling phase, and calculating the relative deviation during detection;

[0123] The formula for the average eigenvector of a multimodal signal is:

[0124] ;

[0125] in, This represents the baseline vector of the pilot's visual features when in a relaxed state. This represents the baseline vector of audio features when the pilot is in a relaxed state. This represents the baseline vector of the QAR operational features of the pilot in a relaxed state;

[0126] The formula for calculating the relative deviation during testing is:

[0127] ;

[0128] in, This represents the visual feature bias vector at time step t. This represents the audio feature deviation vector at time step t. This represents the QAR operation feature bias vector at time step t. Representing speech features, Indicates flight operation parameters;

[0129] Output results and generate interpretive visualization reports, including output stress index and emotional state.

[0130] Furthermore, regarding the output and interpretation of stress and emotional states, the model outputs two metrics in real time:

[0131] Stress Index (SI): a continuous value of 0–1, with a value ≥0.7 considered as a high-pressure condition;

[0132] Emotional State (ES): Represented in two-dimensional coordinates (valence-arousal plane).

[0133] The system uses color-coded visual intervals (green for normal, yellow for medium pressure, and red for high pressure); when the SI exceeds the threshold for more than 5 consecutive seconds, the system triggers an alarm and generates a report.

[0134] The report includes:

[0135] Time series plot (SI curve + Arousal curve); description of feature contribution (e.g., high-frequency speech jitter + facial tension + manipulation jitter are the main signal sources); confidence interval (e.g., 0.87±0.05).

[0136] System Implementation and Deployment

[0137] Hardware implementation: Front end: Embedded camera module + dual microphone array + QAR interface; Processing end: NVIDIA Jetson Orin or equivalent GPU computing node; Communication: Real-time transmission via Ethernet or CAN bus.

[0138] Software architecture:

[0139] Data acquisition thread: Implemented using a hybrid Python + C++ architecture with multi-threaded asynchronous caching; Model inference service: Accelerated using PyTorch → ONNX → TensorRT; Interface display: Web frontend + backend API, with real-time system latency controlled to less than 500ms to ensure immediate response capability.

[0140] In a preferred embodiment, a pilot stress and emotional state inference system based on video, audio, and QAR data includes: a multimodal data acquisition module for acquiring video, audio, and flight operation data; a signal preprocessing and time synchronization module for performing time alignment, noise reduction, and data cleaning; a multimodal feature extraction module for extracting visual, acoustic, and operational parameter features; a cross-modal temporal fusion module for unified modeling of multimodal features using Transformer networks or attention mechanisms; an emotion and stress inference module for calculating stress index and emotional state based on fused features; an individual baseline modeling and adaptation module for constructing individualized benchmarks and eliminating individual differences; and a feedback and visualization module for outputting results and generating explanatory visualization reports.

[0141] The above-mentioned unit modules can be embedded in the processor of the computer device in hardware form or independent of it, or they can be stored in the memory of the computer device in software form, so that the processor can call and execute the corresponding operations of the above modules.

[0142] In one embodiment, a computer device is provided, which may be a terminal. The computer device includes a processor, memory, a communication interface, a display screen, and an input device connected via a system bus. The processor of the computer device provides computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage medium. The communication interface of the computer device is used for wired or wireless communication with external terminals. Wireless communication can be achieved through Wi-Fi, carrier networks, NFC (Near Field Communication), or other technologies. The display screen of the computer device may be an LCD screen or an e-ink display screen. The input device of the computer device may be a touch layer covering the display screen, or buttons, a trackball, or a touchpad located on the casing of the computer device, or an external keyboard, touchpad, or mouse, etc.

[0143] In summary, this invention constructs a non-contact, multimodal collaborative real-time inference method for pilot stress and emotional state by integrating video, audio, and QAR flight operation data. This method offers significant advantages: First, it eliminates the need for any physiological sensors, avoiding interference with normal pilot operations and improving the system's usability and deployment convenience in real flight or simulated training environments. Second, through high-precision time synchronization and cross-modal attention mechanisms, it effectively integrates three complementary types of information: visual expression, vocal emotion, and operational behavior, significantly improving the accuracy and robustness of psychological state recognition. Third, the introduction of individual baseline modeling and differential feature computation mechanisms automatically corrects for individual differences among pilots in facial expression habits, tone styles, and operational preferences, significantly enhancing the model's cross-personal generalization ability. Furthermore, the system output not only includes quantitative results for stress index and emotional dimensions but also provides interpretable information such as feature contribution and confidence intervals, facilitating retrospective analysis and intervention decisions by instructors or safety systems. Finally, the overall architecture supports embedded edge deployment with an inference latency of less than 500 milliseconds, meeting the stringent real-time and reliability requirements of aviation scenarios.

[0144] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for inferring pilot stress and emotional state based on video, audio, and QAR data, characterized in that: include, Acquire video, audio, and flight operation data, and perform time-series alignment, noise reduction, and data cleaning on the acquired data; Extract visual, acoustic, and operational parameter features, and complete unified modeling of multimodal features through Transformer networks or attention mechanisms; Based on the fusion features, the stress index and emotional state are calculated, an individualized benchmark is constructed, individual differences are eliminated, and the results are output and an interpretive visualization report is generated.

2. The method for inferring pilot stress and emotional state based on video, audio, and QAR data as described in claim 1, characterized in that: The time-series alignment, noise reduction, and data cleaning of the acquired data include using multimodal data acquisition and time synchronization, and installing high-definition cameras, directional microphone arrays, and QAR data interfaces in flight simulators or real cockpits; The high-definition camera includes those located above the instrument panel and to the left front, which capture images of the pilot's face and upper body. The directional microphone array is used to collect pilot voice and suppress background wind noise and cabin broadcast interference. The QAR data interface is used to read flight parameters, including pitch angle, roll angle, heading angle, control input, flight altitude, speed and acceleration. The acquired data is sampled synchronously with a unified clock, video frame timestamps are corrected using PTS, audio signals are resampled according to the sampling rate, and time step interpolation is performed using the QAR data interface. A synchronization module is used to maintain a global buffer and output trimodal frame groups according to time windows.

3. The method for inferring pilot stress and emotional state based on video, audio, and QAR data as described in claim 2, characterized in that: The extraction of visual, acoustic, and operational parameter features includes video feature extraction, audio feature extraction, and QAR feature extraction. The video feature extraction includes using MTCNN-based face detection and KLT-based point tracking algorithms to extract facial features, eye indicators, head pose, and micro-expression detection; The facial expression features include facial action unit analysis, such as AU1, AU4, and AU12, and their dynamic intensity is calculated. The eye indicators include blink frequency, eyelid closure time, and gaze stability, used to assess attention and stress. The head posture includes three-dimensional posture angles that reflect alertness or distraction. The micro-expression detection includes using an I3D network to detect micro-expression changes lasting less than 500ms, in order to capture instantaneous emotional fluctuations.

4. The method for inferring pilot stress and emotional state based on video, audio, and QAR data as described in claim 3, characterized in that: The audio feature extraction includes extracting low-level acoustic features and high-level speech features after the audio signal has undergone speech activity detection and beamforming noise reduction processing. The low-level acoustic features include fundamental frequency F0, formants, energy, spectral centroid, MFCC, ΔMFCC, rate of change of intensity, jitter, and shimmer. The high-level speech features include extracting contextual speech embeddings based on the Wav2Vec2 pre-trained model and analyzing semantic sentiment using the BERT model.

5. The method for inferring pilot stress and emotional state based on video, audio, and QAR data as described in claim 4, characterized in that: The QAR feature extraction includes processing QAR data through a sliding window to calculate operational stability features, flight attitude fluctuations, flight load parameters, and control feedback indicators. The operational stability characteristic refers to the joystick input volatility; the flight attitude volatility refers to the pitch rate. The flight load parameter refers to the rate of change of vertical acceleration; the control feedback index refers to the redundant control input.

6. The method for inferring pilot stress and emotional state based on video, audio, and QAR data as described in claim 5, characterized in that: The unified modeling of multimodal features through Transformer networks or attention mechanisms includes using a cross-modal Transformer network, inputting a multi-head attention layer through an attention mechanism to model the correlation between modalities, and then using a temporal convolutional network or Bi-LSTM to complete temporal dependency modeling and extract the trend of emotion changes. The overall loss function is obtained by connecting the stress index prediction head and the emotion dimension prediction head; the stress index prediction head refers to the output of Stress Index St∈[0,1] through a fully connected sigmoid function; the emotion dimension prediction head refers to the output of the values ​​of the emotion of pleasure (Valence) and activation (Arousal). The formula for the overall loss function is: ； in, Represents the total loss function. , and Indicates the weighting coefficient. This represents the loss from the stress prediction task. This indicates the loss in the emotion prediction task. This represents the loss of timing consistency. The cross-modal Transformer network includes the following inputs for computation time step t: ； in, This represents the fused feature vector at time step t. This represents the visual feature vector at time step t. This represents the audio feature vector at time step t. This represents the QAR feature vector at time step t; The formula for the correlation between the multi-head attention layer modeling modalities is: ； in, This represents the attention mechanism function. Represents the query matrix. Represents the key matrix, Represents a value matrix, This represents the multimodal fusion feature vector at time step t.

7. The method for inferring pilot stress and emotional state based on video, audio, and QAR data as described in claim 6, characterized in that: The construction of individualized benchmarks includes recording the average feature vector of multimodal signals of the pilot in a relaxed state during the baseline modeling phase, and calculating the relative deviation during detection; The formula for the average eigenvector of the multimodal signal is: ； in, This represents the baseline vector of the pilot's visual features when in a relaxed state. This represents the baseline vector of audio features when the pilot is in a relaxed state. This represents the baseline vector of the QAR operational features of the pilot in a relaxed state; The formula for calculating the relative deviation during detection is: ； in, The visual feature bias vector represents time step t. This represents the audio feature deviation vector at time step t. This represents the QAR operation feature bias vector at time step t. Representing speech features, Indicates flight operation parameters; The output results, which generate interpretive visualization reports, include output stress index and emotional state.

8. A pilot stress and emotional state estimation system based on video, audio, and QAR data, based on the pilot stress and emotional state estimation method based on video, audio, and QAR data as described in any one of claims 1 to 7, characterized in that: include, The multimodal data acquisition module is responsible for acquiring video, audio, and flight operation data; The signal preprocessing and time synchronization module is used to perform timing alignment, noise reduction, and data cleaning. The multimodal feature extraction module is used to extract visual, acoustic, and operational parameter features; The cross-modal temporal fusion module is used to achieve unified modeling of multimodal features through Transformer networks or attention mechanisms; The emotion and stress inference module is used to calculate the stress index and emotional state based on fused features. The Individual Baseline Modeling and Adaptation module is used to build individualized baselines and eliminate individual differences. The feedback and visualization module is used to output results and generate explanatory visualization reports.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, it implements the steps of the pilot stress and emotional state inference method based on video, audio, and QAR data as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, it implements the steps of the pilot stress and emotional state inference method based on video, audio and QAR data as described in any one of claims 1 to 7.

Images