Pilot expression and physiological index-based unsafe behavior recognition method and system

Abstract

The application provides a pilot expression and physiological index-based unsafe behavior identification method and system, relates to the technical field of modern aviation safety management, and comprises multi-dimensional data collection, pilot facial expression data collection through a camera and expression calculation software, pilot physiological index data collection through a wearable device, flight parameter data collection through a flight quality monitoring system and generation of a pilot behavior mode label, and three types of data, namely, pilot facial expression data, pilot physiological index data and flight parameter data, are summarized to form an original data set; the application effectively solves the problems of poor timeliness and lagging recognition window in the prior art through integrated multi-modal real-time data collection and intelligent processing mechanism; specifically, the system realizes non-invasive real-time monitoring of facial expressions and physiological indexes by using a camera and a wearable device, and synchronously generates a behavior mode label through a flight quality monitoring system.

Description

Technical Field

[0001] This invention relates to the field of modern aviation safety management technology, specifically to a method and system for identifying unsafe behaviors based on pilot facial expressions and physiological indicators. Background Technology

[0002] With the continuous development of aviation safety management, Flight Quality Monitoring (QAR) systems have become one of the core technologies of modern aviation. This system effectively characterizes pilot operational behavior patterns by recording and analyzing various parameters during flight in real time, such as temporal characteristics of altitude, speed, and attitude changes. Currently, professional flight quality analysis algorithms are widely used in practice, enabling the labeling and evaluation of pilot behavior patterns based on QAR parameter characteristics after a flight, supporting airlines in safety management and training optimization.

[0003] However, existing pilot behavior pattern recognition technologies face the following key technical challenges in practical applications:

[0004] Due to poor timeliness and delayed identification window, the existing QAR-based monitoring mode mainly relies on post-event data processing and cannot achieve pre-event or in-event early warning, resulting in the inability to intervene in potential unsafe behaviors in a timely manner.

[0005] The multimodal data association is insufficient. Although the methods for collecting facial expressions and physiological indicators (such as heart rate and blood oxygen) are relatively mature, there is a lack of effective data processing and analysis mechanisms, which makes it impossible to reliably associate these behavioral precursors with flight patterns.

[0006] Due to limited capacity for processing imbalanced data and the scarcity of unsafe pattern samples in real flight scenarios, existing algorithms struggle to effectively address class imbalance, resulting in low accuracy in minority class pattern recognition.

[0007] Therefore, a method and system for identifying unsafe behaviors based on pilot facial expressions and physiological indicators is needed to solve the above problems. Summary of the Invention

[0008] Technical problems to be solved

[0009] To address the shortcomings of existing technologies, this invention provides a method and system for identifying unsafe behaviors based on pilot facial expressions and physiological indicators, thus solving the problems of existing technologies.

[0010] Technical solution

[0011] To achieve the above objectives, the present invention provides the following technical solution: a method and system for identifying unsafe behaviors based on pilot facial expressions and physiological indicators, comprising the following steps:

[0012] Sp1: Multidimensional data acquisition involves collecting pilot facial expression data via cameras and facial expression calculation software, collecting pilot physiological indicator data via wearable devices, and collecting flight parameter data through a flight quality monitoring system to generate pilot behavior pattern labels. These three types of data—facial expression data, physiological indicator data, and flight parameter data—are aggregated to form the raw dataset. The facial expression data includes time-series data of valence and arousal, while the physiological indicator data includes time-domain features of heart rate variability, heart rate, blood oxygen saturation, and body temperature. The cameras, facial expression calculation software, and wearable devices all interact with the flight quality monitoring system to ensure the time synchronization of the collected data.

[0013] Sp2: Raw data preprocessing involves filtering, quality screening, and reorganization of the raw dataset. A multi-level intelligent imputation strategy is employed to handle missing values, and features are standardized. The data is reorganized into a variable-time series sample dataset, and a hierarchical cross-validation method is used to divide the training and test sets. The multi-level intelligent imputation strategy is based on individual and population baselines, and combines the physiological meaning of features with correlation inference, multivariate derivation, baseline quantile sampling, and a physiological constraint fallback scheme to imput missing features. Sp2 receives the raw dataset summarized in Sp1, and the processed data is transmitted to Sp3.

[0014] Sp3: Feature optimization employs a two-stage progressive optimization architecture. The first stage expands and generates a candidate feature pool through frequency domain feature extraction, statistical feature extraction, and physiological feature extraction. The second stage performs intelligent feature selection based on a multi-dimensional feature importance evaluation system to obtain the optimal feature subset. The multi-dimensional feature importance evaluation system integrates RandomForest importance evaluation, correlation analysis, variance analysis, statistical significance test, and physiological prior knowledge dimensions. Sp3 interacts with Sp2, receiving preprocessed data and outputting the optimal feature subset to Sp4.

[0015] Sp4: Differentiated category balancing enhancement. Based on the degree of category imbalance of each flight mode label, it intelligently selects a balanced category enhancer or an extreme category enhancer to augment the training set. The balanced category enhancer adopts a mild balancing strategy, integrating time warp, amplitude warp, and Gaussian noise injection techniques and applying physiological constraints. The extreme category enhancer adopts an aggressive balancing strategy, integrating synthetic sample generation and frequency domain shift techniques, combined with a negative sample downsampling mechanism. Sp4 receives the optimal feature subset output by Sp3, and the enhanced training set is transmitted to Sp5.

[0016] Sp5: Model training. An algorithm pool containing traditional machine learning algorithms and deep learning algorithms is constructed. Candidate algorithm combinations are dynamically selected based on the distribution characteristics of the enhanced data, and a parallel competitive training mechanism is used to complete model training. The deep learning algorithm includes the Enhanced_GRU model, which integrates FocalLoss and SMOTE technologies and supports variable time series processing. Sp5 interacts with Sp4, receives the enhanced training set, and the trained model is transferred to Sp6.

[0017] Sp6: Intelligent model selection employs a four-stage selection strategy to determine the optimal model, enabling the identification and early warning of unsafe pilot behavior patterns. The four-stage selection strategy consists of a quality screening stage, an anomaly detection stage, a service scoring stage, and a final selection stage. The service scoring stage uses a preset weight configuration to calculate a comprehensive service score. Sp6 receives the trained model output from Sp5, selects it, and outputs the optimal model for unsafe behavior pattern identification and early warning.

[0018] Preferably, the screening of the original dataset described in Sp2 includes flight phase screening and operator screening, retaining data records marked as landing phase and actually performed by pilots; the evaluation criteria for the quality screening are that each feature has at least 3 valid values, and at least 50% of the features are in a valid state.

[0019] Preferably, the process of reorganizing the data into a variable-length time series sample dataset as described in Sp2 adopts a variable-length sequence storage method to maintain the original time length of each flight sample; the hierarchical cross-validation adopts a five-fold multi-label hierarchical K-fold algorithm to ensure that the proportion of various label combinations in each fold is consistent with the overall dataset.

[0020] Preferably, in Sp3, the frequency domain feature extraction in the first stage uses the Welch power spectral density estimation method for heart rate variability signals to extract power values ​​and derived indices in the very low frequency band, low frequency band, and high frequency band, and actively excludes the total power frequency band to reduce redundancy; the physiological feature extraction in the first stage calculates the proportion of abnormal values ​​of physiological features relative to the normal range and the range utilization rate.

[0021] Preferably, the class imbalance described in Sp4 is divided into four levels: very minority class, high minority class, moderate minority class, and extremely minority class; the class balancer is configured with different enhancement factors and technology combinations for the very minority class, high minority class, and moderate minority class respectively; the extreme class balancer is configured with aggressive enhancement parameters for the extremely minority class; and the synthetic sample generation is achieved by mixing two real samples of the same class and injecting noise.

[0022] Preferably, the process of dynamically selecting candidate algorithm combinations based on the distribution characteristics of the enhanced data as described in Sp5 involves generating a data distribution profile, constructing an algorithm characteristic profile, implementing a hierarchical algorithm selection strategy based on data balance state, data complexity, and enhancement factor, and calculating the comprehensive priority of the algorithms through a multi-dimensional scoring function.

[0023] Preferably, in Sp6, the quality screening stage of the four-stage selection strategy sets a performance threshold to adapt to imbalanced data; the anomaly detection stage of the four-stage selection strategy identifies the over-prediction mode of the algorithm; the business scoring stage of the four-stage selection strategy is configured with the following weights: precision weight 40%, recall weight 35%, F1 score weight 15%, and efficiency weight 10%; the final selection stage of the four-stage selection strategy selects the model with the highest comprehensive score, and the alternative selection mechanism is activated when the candidate model pool is empty.

[0024] Preferably, the system includes a data acquisition module, a data preprocessing module, a feature optimization engineering module, a data augmentation module, a model training module, and an intelligent selection module, and the system is used to execute the method;

[0025] The data acquisition module is used to collect pilots' facial expression data, physiological index data, and flight parameter data, and generate behavioral pattern labels, which are then aggregated to form a raw dataset. The data acquisition module includes a camera and facial expression calculation software, a wearable device, and a flight quality monitoring submodule. The camera and facial expression calculation software and the wearable device all interact with the flight quality monitoring submodule to ensure the time synchronization of the collected data. The data acquisition module is communicatively connected to the data preprocessing module to transmit the raw dataset to the data preprocessing module.

[0026] The data preprocessing module is used to filter, quality screen and reorganize the original dataset, handle missing values, standardize features, reorganize data and divide it into training and test sets; the data preprocessing module has a built-in multi-level intelligent imputation unit, which constructs individual baselines and group baselines, and selects the corresponding imputation scheme based on the physiological meaning of features; the data preprocessing module interacts with the feature optimization engineering module, and transmits the processed data to the feature optimization engineering module;

[0027] The feature optimization engineering module is used to implement feature expansion and intelligent feature selection to obtain the optimal feature subset. The feature optimization engineering module includes an intelligent feature engineer and an intelligent feature selector. The intelligent feature engineer integrates frequency domain, statistical and physiological feature extraction techniques, and the intelligent feature selector adopts a multi-dimensional feature importance evaluation system. The feature optimization engineering module is communicatively connected to the data augmentation module and transmits the optimal feature subset to the data augmentation module.

[0028] The data augmentation module is used to solve the class imbalance problem of training data, and includes a balanced class augmenter and an extreme class balancer. The balanced class augmenter adopts a mild balancing strategy and applies physiological constraints, while the extreme class balancer adopts an aggressive balancing strategy and supports synthetic sample generation and negative sample downsampling. The data augmentation module interacts with the model training module, transferring the augmented training set to the model training module.

[0029] The model training module is used to implement parallel competitive training of multiple algorithms, and includes an algorithm pool, a dynamic algorithm selector, and an algorithm factory. The algorithm pool includes traditional machine learning algorithms and deep learning algorithms. The dynamic algorithm selector selects candidate algorithm combinations based on data distribution characteristics. The algorithm factory provides a unified interface for creating and calling algorithm instances. The model training module communicates with the intelligent selection module and transmits the trained model to the intelligent selection module.

[0030] The intelligent selection module is used to determine the optimal model through a four-stage selection strategy, thereby enabling the identification and early warning of unsafe behavior patterns of pilots. The intelligent selection module includes a quality screening unit, an anomaly detection unit, a service scoring unit, and a final selection unit. The intelligent selection module maintains instruction transmission with the data acquisition module and can adjust the data acquisition parameters based on the feedback of the identification results.

[0031] Preferably, the wearable device included in the data acquisition module is a smartwatch, which supports the acquisition of heart rate time interval, mean heart rate, body temperature and blood oxygen value data based on PPG; the facial expression calculation software included in the data acquisition module is a facial expression analysis system, which can analyze seven basic emotions, valence and arousal indicators.

[0032] Preferably, the deep learning algorithms in the algorithm pool included in the model training module include Enhanced_GRU, GRU, LSTM_Attention, and CNN-LSTM, while the traditional machine learning algorithms include RandomForest, XGBoost, LightGBM, SVM, LogisticRegression, and GradientBoosting. The Enhanced_GRU model integrates FocalLoss and SMOTE techniques and includes GRU stacked layers, global average pooling layers, and a fully connected classification head.

[0033] Beneficial effects

[0034] This invention provides a method and system for identifying unsafe behaviors based on pilot facial expressions and physiological indicators. It offers the following advantages:

[0035] 1. This invention effectively solves the problems of poor timeliness and delayed recognition window in existing technologies by integrating multimodal real-time data acquisition and intelligent processing mechanisms. Specifically, the system utilizes cameras and wearable devices to achieve non-invasive real-time monitoring of facial expressions and physiological indicators, and synchronously generates behavioral pattern tags through the flight quality monitoring system. End-to-end processing from data acquisition to early warning is completed in a continuous process from Sp1 to Sp6, enabling proactive early warning. This design significantly improves the response speed of flight safety monitoring, reduces the risk of potential unsafe behaviors, supports pilots in making immediate adjustments to operations during simulation training or actual flights, and improves the overall level of aviation safety.

[0036] 2. This invention overcomes the problem of insufficient multimodal data correlation in existing technologies by using multimodal feature fusion and dynamic algorithm selection. Specifically, in the Sp3 feature optimization step, a two-stage architecture is adopted to expand and select time-domain, frequency-domain, and physiological features to achieve effective fusion of facial expression data (such as valence and arousal) and physiological indicators (such as heart rate variability). In the Sp5 model training, an algorithm pool is constructed and an adaptive model is dynamically selected to further strengthen the correlation between behavioral precursors and flight patterns. This innovation improves the accuracy and interpretability of recognition, enabling the system to reliably transform physiological signals into behavioral warning evidence, thus optimizing risk management and decision support for airlines.

[0037] 3. This invention addresses the limitation of imbalanced data processing capabilities in existing technologies through differentiated class balancing enhancement and a four-stage intelligent model selection. Specifically, in step Sp4, a balanced or extreme enhancer is intelligently selected, and different imbalance levels are processed hierarchically (e.g., 8x enhancement is used for the very minority class), combined with physiological constraints to ensure sample authenticity. In Sp6, a four-stage strategy (e.g., anomaly detection to exclude overprediction) optimizes model performance. This mechanism significantly improves the recall and precision of minority class patterns (e.g., extremely rare unsafe behaviors), reduces the risk of false alarms, and enhances the robustness and applicability of the system in real flight scenarios, demonstrating broad market prospects and social benefits. Attached Figure Description

[0038] Figure 1 This is a system framework diagram of the present invention;

[0039] Figure 2 This is the core flowchart of the system of the present invention;

[0040] Figure 3 This is a flowchart of the model training module of the present invention;

[0041] Figure 4 This is a structural diagram of the Enhanced_GRU model of the present invention;

[0042] Figure 5This is a flowchart of the intelligent model selection process of the present invention. Detailed Implementation

[0043] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. Specific Implementation Example 1:

[0045] like Figures 1 to 5 As shown, this embodiment provides a comprehensive and detailed explanation of the "method and system for identifying unsafe behaviors based on pilot facial expressions and physiological indicators". This method achieves accurate identification and early warning of unsafe behavior patterns of pilots through the coordinated execution of six core steps, Sp1 to Sp6. The specific implementation process, technical parameters and operation details of each step are as follows.

[0046] I. Sp1: Detailed Implementation of Multidimensional Data Acquisition

[0047] The core objective of this step is to achieve the synchronous collection and aggregation of pilot facial expression data, physiological index data, and flight parameter data, providing a complete and time-consistent raw dataset for subsequent processing. Specific implementation details are as follows:

[0048] 1. Data Acquisition Equipment and Tool Configuration: A high-definition industrial camera (1920×1080 resolution, 30fps) is installed directly in front of the cockpit to ensure complete capture of the pilot's facial area; the facial expression calculation software uses an authoritative certified facial expression analysis system (FaceReader), which supports the recognition of seven basic emotions (happiness, sadness, anger, surprise, fear, disgust, and neutrality) with an average accuracy of over 96%, and can output time-series data of core emotional quantitative indicators such as valence and arousal; the wearable device is a smartwatch supporting PPG technology, with a sampling frequency set to 100Hz, and has the function of real-time acquisition of data such as heart rate time interval, average heart rate, body temperature (accuracy 0.1℃), and blood oxygen value (accuracy 1%); flight parameter data is collected through the aircraft's built-in Flight Quality Assurance (FOQA) system, which can record more than 20 flight parameters such as control stick actions, flight path, altitude, and speed in real time, and has a built-in pattern recognition submodule for generating behavioral pattern labels.

[0049] 2. Data Acquisition Process: Before the pilot performs the flight mission, camera calibration, smartwatch binding (associated with the pilot's unique ID), and FOQA system data interface configuration are completed to ensure time synchronization among the three (based on FOQA system time, the time error between the camera and smartwatch is controlled within ±50ms). During flight, the camera captures one frame of facial image every second and transmits it to the expression calculation software. The software analyzes and outputs quantitative values ​​of valence and arousal (range 0-100) in real time, forming time series data. The smartwatch collects the average heart rate every 5 seconds, body temperature every 10 seconds, and blood oxygen value every minute, while continuously collecting and storing heart rate time interval data. The FOQA system collects flight parameter data at a frequency of 10Hz and generates behavioral pattern labels (including 6 modes such as high slope and gradually changing slope, encoded as integers from 0 to 1) based on flight parameters (such as descent rate, airspeed, and angle of attack). During the data collection process, the camera, facial expression calculation software, and wearable devices all interact with the FOQA system via Ethernet, synchronizing timestamps every 30 seconds to ensure the temporal consistency of the three types of data.

[0050] 3. Construction of the Original Dataset: The collected three types of data were associated and matched according to "pilot's unique ID + flight number + timestamp" to form the original dataset. This dataset contains five categories of information: 1) Index identification information (pilot ID, flight number, time window identifier); 2) Facial expression data (valence, time series of arousal); 3) Physiological indicator data (time-domain features of heart rate variability, heart rate statistical features, blood oxygen saturation features, body temperature features); 4) Flight metadata (departure time, landing time, landing phase markers, operator markers); 5) Behavioral pattern labels (0-1 codes for 6 flight modes). The original dataset is stored in CSV format, with each record containing 32 fields, covering all the above information dimensions.

[0051] II. Sp2: Detailed Implementation of Raw Data Preprocessing

[0052] This step involves filtering, quality screening, missing value imputation, standardization, and dataset partitioning of the original dataset to ensure that the data quality meets the requirements for subsequent model training. Specific implementation details are as follows:

[0053] 1. Data Filtering: A two-layer filtering logic is employed. The first layer is flight phase filtering. Since this invention focuses on identifying unsafe behaviors during the landing phase, only data records marked as "landing phase" are retained by filtering the "landing phase marker" field. The second layer is operator filtering. Based on the "operator marker" field, data of pilots who actually performed the landing operation is filtered out (excluding monitoring personnel data) to avoid interference from non-target data. After filtering, the data volume is approximately 35%-40% of the original data (due to differences in landing duration between different flights).

[0054] 2. Quality Screening and Reorganization: First, data columns were reorganized, deleting columns that did not contribute to model training (such as departure time, aircraft number, etc.), and retaining the identifier columns, core facial expression and physiological characteristic columns (23 columns in total), and flight mode label columns. Then, a data quality assessment was performed, with the following criteria: ① Each feature has at least 3 valid values ​​(to ensure statistical reliability); ② At least 50% of the features are valid (to avoid excessive data sparsity). Pilot data that passed the assessment was retained, while data that failed (such as a pilot with a 60% missing blood oxygen data rate) was directly removed. The final data quality pass rate was approximately 85%.

[0055] 3. Multi-level intelligent imputation strategy implementation: For missing values ​​in the filtered data, a differentiated imputation scheme is adopted based on individual and population baselines, combined with the physiological meaning of the characteristics: ① Individual variability indicators (such as heart rate variability): Prioritize imputation using the historical median of this indicator of the pilot (at least 3 valid values ​​are required), if insufficient, use the population median; ② Time-domain indicators with mathematical correlation (such as mean heart rate and median): Imputation is derived by using the correlation between indicators. For example, when the mean heart rate is missing, it is estimated by median × 0.98, when the median is missing, it is estimated by mean × 1.02, and the maximum value is estimated by mean + 2σ estimation: The minimum value is estimated using the mean minus 2σ. If the correlation imputation fails, it falls back to the individual mean or the physiological default value; ③ Blood oxygen saturation: Based on the physiological characteristics of pilots' stable blood oxygen levels, values ​​are randomly selected from the 95th to 99th quantiles of the population statistics for imputation; ④ Body temperature: Integrating individual mean, seasonal adjustment, diurnal adjustment, and random noise imputation, 0.2℃ is subtracted in winter (December-February), 0.1℃ is added in summer (June-August), 0.1℃ is added during the day (6-2 pm), and 0.2℃ is subtracted at night (10-6 am). Finally, normal distributed noise with a mean of 0 and a standard deviation of 0.1 is added. The missing data rate after imputation is controlled within 5%.

[0056] 4. Feature Standardization: Due to the significant differences in the units of measurement for different physiological features (e.g., heart rate variability is measured in milliseconds, body temperature in °C), Z-score standardization is employed. All time steps of all samples are vertically stacked to form a large matrix. The mean and standard deviation are calculated independently for each feature, and standardized using the formula (x - mean) / standard deviation. This ensures that all feature values ​​are mapped to the same order of magnitude (mean 0, standard deviation 1), preventing the model from becoming overly sensitive to features with large numerical values.

[0057] 5. Construction and partitioning of the variable-length time series dataset: Using "pilot's unique ID + flight number" as the grouping key, all time windows of the same landing are aggregated and arranged in chronological order to form a continuous time series. Variable-length sequence storage is used (retaining the original time length without padding or truncation). A five-fold multi-label hierarchical K-fold algorithm is used to partition the training and test sets. First, the label combination of each sample is calculated, and then stratified sampling is performed to ensure that the proportion of each label combination in each fold is consistent with the overall dataset to avoid data leakage. Finally, the training set accounts for 70% and the test set accounts for 30%.

[0058] III. Sp3: Detailed Implementation of Feature Optimization

[0059] This step employs a two-stage progressive optimization architecture to achieve feature expansion and intelligent filtering, obtaining the optimal feature subset. Specific implementation details are as follows:

[0060] 1. First Stage: Candidate Feature Pool Expansion (Core Feature Extraction): Integrating three major feature extraction techniques—frequency domain, statistical, and physiological—to expand the original 23-dimensional features into a 251-dimensional candidate feature pool: ① Frequency Domain Feature Extraction: For heart rate variability signals, the Welch power spectral density estimation method is used to divide the signal into three frequency bands: extremely low frequency (0.0033-0.04Hz), low frequency (0.04-0.15Hz), and high frequency (0.15-0.4Hz). Power values ​​and derived indices (such as the low-frequency / high-frequency ratio) are extracted for each band. Total power frequency bands are actively excluded to reduce redundancy, resulting in a total of 11 candidate feature pools. ① 4-dimensional frequency domain features; ② Statistical feature extraction: Adaptive sliding window strategy is adopted (the window size is the minimum of 1 / 5 and 10 of the sequence length, not less than 3) to extract four core statistical features: mean stability, standard deviation variability, trend strength, and volatility, generating a total of 76 statistical features; ③ Physiological feature extraction: The normal range of each physiological indicator is predefined (heart rate 40-200 beats / minute, blood oxygen 85%-100%, etc.), and the proportion of outliers (out-of-range time step ratio) and range utilization rate (signal span / normal range span) of each feature are calculated, generating a total of 38 physiological features.

[0061] 2. Second Stage: Intelligent Feature Selection (Optimal Feature Subset Acquisition): The optimal feature subset is selected based on a multi-dimensional feature importance evaluation system. This system includes five dimensions: ① RandomForest Importance Evaluation: A RandomForest classifier with 100 decision trees is constructed (maximum depth 10, minimum leaf node sample count 5), and the Gini importance score for each feature is obtained; ② Correlation Analysis: The Pearson correlation coefficient matrix between features is calculated, a threshold of 0.95 is set, and highly correlated redundant features are labeled; ③ Analysis of Variance: The variance and coefficient of variation of each feature are calculated, and low-variance features with variances below the first percentile of all feature variances are removed; ④ Statistical Significance Test: The F-test (P<0.05 is significant) and mutual information methods are used to evaluate the association strength between features and labels; ⑤ Physiological Prior Knowledge: Weights are assigned to different types of features (original physiological features 3.0, heart rate variability features 2.5, etc.). After normalizing the scores of the five dimensions, the scores were summed by weight (RandomForest 40%, Variance 20%, Mutual Information 20%, Physiological Prior 20%). The top 100 features were selected as the optimal feature subset based on the score ranking, resulting in a feature reduction of 40%.

[0062] IV. Detailed Implementation of Sp4: Enhanced Differentiation of Category Balance

[0063] This step intelligently selects the enhancer based on the degree of class imbalance to solve the problem of class imbalance in the training data. The specific implementation details are as follows:

[0064] 1. Classification of Imbalance: By statistically analyzing the proportion of positive samples for each flight mode label, the imbalance is classified into four levels: extremely minority (<5%), highly minority (5%-10%), moderately minority (10%-25%), and extremely minority (<5% and sample size <100). Statistical analysis shows that in the target application scenario, the positive sample proportion for the "grounded micro-stable" mode is only 3.2% (extremely minority), the "high slope" mode is 7.4% (highly minority), and the "gradually changing slope" mode is 20.3% (moderately minority).

[0065] 2. Balanced Category Augmenter Implementation (Primary): A mild balancing strategy is employed, with differentiated parameters configured for different imbalance levels: ① Very Minority / High Minority: A 5-8x amplification factor for positive samples is used, with a target positive sample ratio of 25%, enabling three techniques: time warp (scaling factor 0.8-1.2), amplitude warp (scaling factor 0.7-1.3), and Gaussian noise injection (5% of standard deviation); ② Moderate Minority: A 2.5x amplification factor is used, with a target ratio of 35%, enabling only time warp and amplitude warp techniques. Physiological constraints are applied during augmentation to ensure that the enhanced feature values ​​are within a reasonable range (e.g., heart rate variability ≤30%, blood oxygen variability ≤7%). Implementation employs multi-round iterative augmentation (maximum 5 rounds), with each round randomly selecting base samples to generate new samples until the target sample size is reached.

[0066] 3. Extreme Class Balancer Implementation (Supplementary): An aggressive balancing strategy is adopted for the extremely rare minority class, configuring a 20x positive sample enhancement factor and a 0.8x negative sample downsampling factor, with a target positive sample ratio of 40%. Five techniques are enabled: time warp, amplitude warp, Gaussian noise, frequency domain shift (frequency shift -0.1 to 0.1), and synthetic sample generation. Synthetic sample generation is achieved by mixing two real samples of the same class: after unifying the sample length, a mixing coefficient of 0.3-0.7 is randomly selected for weighted combination, and then differential noise (8% standard deviation of heart rate variability noise and 2% body temperature noise) is injected according to the feature group to ensure the authenticity of the samples.

[0067] 4. Enhancement effect verification: After enhancement, the proportion of positive samples in the "grounded micro-stable" mode increased to 38.5%, the "high slope" mode increased to 24.5%, and the "gradually changing slope" mode increased to 33.3%, significantly improving the class imbalance problem; after physiological rationality test, the characteristic values ​​of the enhanced samples were all within the normal range, and no false data were generated.

[0068] V. Sp5: Detailed Implementation of Model Training

[0069] This step involves building an algorithm pool to enable parallel competitive training of multiple algorithms. Specific implementation details are as follows:

[0070] 1. Algorithm Pool Construction: This includes two main categories: traditional machine learning algorithms and deep learning algorithms, totaling 10 algorithms: ① Traditional Machine Learning Algorithms: RandomForest (ensemble of 100 decision trees), XGBoost (learning rate 0.1, maximum depth 6), LightGBM (31 leaves, learning rate 0.05), SVM (Gaussian kernel), LogisticRegression (regularization strength 0.1), GradientBoosting (learning rate 0.1); ② Deep Learning Algorithms: Enhanced_GRU (core innovative model), GRU (hidden layer dimension 64), LSTM_Attention (attention heads 4), CNN-LSTM (convolutional kernel size 3×3, LSTM hidden layer dimension 64). A feature profile is created for each algorithm, recording its class imbalance handling capability, training speed, memory usage, and other information.

[0071] 2. Dynamic Algorithm Selection: Generate a distribution profile of the augmented data, including three dimensions: balance state (proportion of positive samples), data complexity (number of samples, number of features), and augmentation factor. Based on the profile and algorithm characteristic profile matching, divide the data into three levels: core algorithms (3-4 optimally adapted algorithms), exploratory algorithms (1-4 potential algorithms), and backup algorithms (emergency algorithms). For example, for extremely rare minority class data (poor balance state, high complexity), the core algorithms are Enhanced_GRU, XGBoost, and LSTM_Attention, while the exploratory algorithms are LightGBM and CNN-LSTM; for moderately rare minority class data, the core algorithms are RandomForest, XGBoost, and LightGBM. Priorities are calculated using a multi-dimensional scoring function (40% for imbalance handling capability, 30% for training speed, and 30% for memory usage) to determine the training order.

[0072] 3. Parallel Competitive Training Mechanism: A hybrid strategy of "traditional machine learning parallelism + deep learning sequential training" is adopted: ① Traditional machine learning algorithms: Concurrent training is conducted through a thread pool (size ≤ 4). Each thread independently creates algorithm instances, executes training, and calculates evaluation metrics, avoiding resource contention; ② Deep learning algorithms: Training is performed sequentially according to priority (avoiding GPU memory overflow), GPU acceleration is enabled (CUDA supported), and an early stopping mechanism (patience value 10) and checkpoint saving function are integrated. During training, metrics such as loss, precision, and recall are monitored in real time, and training history is recorded.

[0073] 4. Enhanced_GRU Model Training: As the core innovative model, its training process is as follows: ① Input layer: Receives enhanced variable-length time sequence data (shape: batch×time×100), and processes the variable-length sequence through a mask mechanism; ② Network backbone: 1-2 stacked GRU layers (hidden layer dimension 64) extract temporal dependencies, global average pooling layer obtains fixed-length vectors (reduces overfitting), and fully connected classification head (activation function Sigmoid) maps to the pattern probability space; ③ Loss function: FocalLoss (γ=2, α=0.25) is used to focus on hard-to-classify samples and alleviate class imbalance; ④ Optimizer: AdamW (learning rate 0.001, weight decay 0.01) is used, combined with Dropout (probability 0.2) regularization, with 50 training epochs, and an early stopping mechanism to monitor the validation set F1 score.

[0074] VI. Sp6: Detailed Implementation of Intelligent Model Selection

[0075] This step employs a four-stage selection strategy to determine the optimal model, enabling the identification and early warning of unsafe behavior patterns. Specific implementation details are as follows:

[0076] 1. First Stage: Quality Screening Stage: Set performance thresholds for adapting to imbalanced data: Precision ≥ 0.25, Recall ≥ 0.3, F1 score ≥ 0.3, Precision-Recall Gap ≤ 0.5. All trained models undergo threshold testing; only models passing all thresholds advance to the next stage. For example, a LogisticRegression model with a recall of 0.28 (below the threshold of 0.3) is directly eliminated; six models, including Enhanced_GRU and XGBoost, pass the screening.

[0077] 2. Second Stage: Anomaly Detection Stage: This stage focuses on identifying overprediction patterns, categorized into two levels: ① Severe overprediction (precision ≤ 0.5 and recall ≥ 0.95), these models completely lose their discriminative ability and are directly excluded; ② Moderate overprediction (precision < 0.6, recall > 0.85 and discrepancy > 0.3), which are also excluded. Upon testing, a certain LSTM_Attention model with a precision of 0.48 and a recall of 0.96 (severe overprediction) was identified as an anomaly and excluded.

[0078] 3. Third Stage: Business Scoring Stage: The total business score is calculated based on the weighted configuration of flight safety scenarios. The weights are: Precision 40%, Recall 35%, F1 Score 15%, and Training Efficiency 10%. The scoring calculation method is as follows: ① Precision Score = Precision × 40%; ② Recall Score = Recall × 35%; ③ F1 Score = F1 × 15%; ④ Training Efficiency Score = (1 - Current Model Training Time / Maximum Training Time) × 10%. For example, the Enhanced_GRU model has a precision of 0.72, a recall of 0.68, an F1 score of 0.70, and a training time of 80 minutes (maximum training time 120 minutes). Its total business score is: 0.72 × 0.4 + 0.68 × 0.35 + 0.70 × 0.15 + (1 - 80 / 120) × 0.1 = 0.288 + 0.238 + 0.105 + 0.033 = 0.664. A model fit ranking is generated by sorting the models according to their total business scores.

[0079] 4. Fourth Stage: Final Selection Stage: Under normal circumstances, the model with the highest overall business score is selected as the optimal model, and a detailed selection reasoning explanation is generated (including score details, performance indicators, selection advantages, etc.). For example, the Enhanced_GRU model has an overall business score of 0.664 (ranked first), and its selection advantages are "high accuracy in adapting to flight safety scenarios, support for imbalanced data processing, and strong time series modeling capabilities." If the candidate model pool is empty (all models have failed the first two stages), an alternative mechanism is activated, selecting the model with the highest F1 score as the emergency solution, and a warning message is added (manual review is recommended). After the optimal model is output, it is used for real-time identification and early warning of unsafe pilot behavior patterns, with an identification delay of ≤2 seconds. Specific Implementation Example 2:

[0081] like Figures 1 to 5 As shown, this embodiment provides a comprehensive and detailed description of the "Pilot Unsafe Behavior Pattern Recognition System Based on Facial Expressions and Physiological Indicators". The system includes six core modules, each of which independently performs a specific function and works in concert to ensure the efficient execution of the recognition method.

[0082] The data acquisition module is responsible for multi-source synchronous acquisition and preliminary aggregation, including a camera submodule, a smartwatch submodule, and a flight quality monitoring submodule. This module uses an embedded processor (such as an ARM Cortex-A53) as the core control unit to ensure low latency and high reliability during the acquisition process. The camera submodule is equipped with a high-definition industrial-grade camera (resolution at least 1920×1080, frame rate above 30fps), mounted directly in front of the cockpit to cover the pilot's entire face, capturing video in real time and inputting it into facial expression analysis software. This software calculates the probability distribution of seven basic emotions (happiness, sadness, anger, surprise, fear, disgust, and neutrality) based on a deep learning model (average accuracy above 96%), and simultaneously outputs time-series data on valence (emotional intensity, range 0-100), arousal (activation level, range 0-100), and emotional attitude indicators (such as interest, confusion, and boredom), updated every second. The smartwatch submodule uses commercial-grade equipment supporting PPG technology, with a sampling frequency of over 100Hz. It can continuously collect heart rate time intervals (for calculating variability indicators such as SDNN and RMSSD), average heart rate every 5 seconds (range 40-200 beats / min), body temperature every 10 seconds (accuracy 0.1°C, normal range 36.0-38.0°C), and blood oxygen saturation every minute (accuracy 1%, normal range 85%-100%). The data is uploaded to the module buffer via Bluetooth or Wi-Fi. The flight quality monitoring submodule is integrated with the aircraft FOQA system, collecting more than 20 flight parameters (such as altitude change rate, speed, attitude, and ground load), recording once every 10Hz, and using a built-in pattern recognition algorithm to generate six unsafe mode labels (high slope descent, gradual slope change, anomaly within a specific altitude range, multi-level altitude change, insufficient stability after grounding, and ground micro-stability anomaly). The labels are output in 0-1 integer encoding format. The three sub-modules ensure multimodal data alignment through a timestamp synchronization interaction mechanism (based on FOQA time, with an error controlled within ±50ms). For example, camera data and physiological data are calibrated every 30 seconds to avoid time sequence offset. The aggregated raw dataset is stored in a structured format (such as JSON or CSV), containing pilot identification, flight metadata, facial expression sequences, physiological sequences, and label information, and is directly transmitted to the input queue of the data preprocessing module to achieve zero-latency connection.

[0083] The data preprocessing module, serving as the system's data entry point, incorporates multi-level intelligent filling and standardization units. It utilizes multi-core CPUs (such as Intel i7 series) to handle high-concurrency data streams, ensuring a processing speed of over 1000 records per second. This module first performs data filtering and quality screening: the filtering logic includes flight phase filtering (retaining only data marked during the landing phase, approximately 35%-40% of the original data) and operator filtering (extracting actual pilot records based on markers, excluding monitor data); the quality screening removes irrelevant columns (such as takeoff time and aircraft number) and evaluates each pilot dataset, requiring at least three valid values ​​for each feature and at least 50% of the features to be valid. Unqualified data is automatically discarded (with a pass rate of approximately 85%). The multi-level intelligent filling unit constructs a filling strategy based on individual baselines (calculating the mean, median, standard deviation, and quartiles of each pilot) and population baselines (summarizing global statistics of all pilots). It selects correlation derivation, multivariate sampling, or constraint fallback for different physiological characteristics: for example, heart rate variability is filled first using the individual median, and if insufficient, it is backed up to the population median; heart rate statistics are derived using mathematical correlation (e.g., the maximum value is estimated as the mean plus 2 times the standard deviation); blood oxygen saturation is randomly sampled from the 95%-99% quartile of the population; body temperature is adjusted by adding seasonal adjustment (0.2°C in winter, 0.1°C in summer, and 0.1°C in autumn) and diurnal adjustment (0.1°C during the day and 0.2°C at night) on the basis of individual mean, and normal noise with a mean of 0 and a standard deviation of 0.1 is added to simulate natural fluctuations. After filling, the missing rate is controlled within 5%. The standardization unit employs the Z-score global method, stacking all sample time steps into a matrix and then independently calculating the mean and standard deviation for each feature to normalize it, ensuring dimensional consistency. The module also includes a time series reconstruction unit, aggregating data using pilot-flight combinations as the key to form variable-length sequence samples (preserving the original length without truncation or padding), with labels taken from the end of the sequence. The partitioning unit uses a five-fold multi-label hierarchical K-fold algorithm to ensure consistent label proportions in each fold, avoiding data leakage. After processing, the clean dataset is transferred to the feature optimization engineering module's buffer in NumPy array format, supporting batch parallel input.

[0084] The feature optimization engineering module includes an intelligent feature engineer and an intelligent feature selector. It utilizes GPU acceleration (such as NVIDIA RTX series) to handle high-dimensional computations, ensuring that expansion and filtering time does not exceed 30 seconds. The module takes a preprocessed sequence dataset as input. The intelligent feature engineer integrates time-domain, frequency-domain, and nonlinear extraction techniques to expand the initial 23-dimensional features into a 251-dimensional candidate feature pool. In the frequency domain, Welch power spectral density estimation is used for heart rate variability, dividing the frequency into three segments: very low frequency (0.0033-0.04Hz), low frequency (0.04-0.15Hz), and high frequency (0.15-0.4Hz). Power values ​​and derived indices (such as the low-frequency / high-frequency ratio and normalized power) are calculated, and total power is actively excluded to reduce redundancy, resulting in 114 dimensions. The statistical part uses an adaptive sliding window (size...). The sequence length is between 1 / 5 and 10 (minimum value, not less than 3), and the following are extracted: mean stability (rolling mean standard deviation), standard deviation variability (rolling standard deviation), trend strength (absolute value of Pearson correlation between sequence and time index), and volatility (first difference standard deviation), totaling 76 dimensions. The physiological part calculates the proportion of outliers (the proportion of time steps outside the normal range) and range utilization (the ratio of signal span to the normal range). The normal range includes heart rate 40-200 beats / min, blood oxygen 85%-100%, body temperature 36.0-38.0°C, and heart rate variability 20-200ms, totaling 38 dimensions. The intelligent feature selector filters the optimal subset based on a multi-dimensional evaluation system: integrating RandomForest Gini importance (constructing 100 trees, depth 10, minimum leaf size 5), correlation analysis (marking redundant pairs with a threshold of 0.95, identifying intra-group duplication through hierarchical clustering), analysis of variance (calculating the coefficient of variation, removing low-information features below the 1st percentile), statistical significance (F-test, P < 0.05 for significance, mutual information quantification dependency), and physiological prior weights (original physiology 3.0, heart rate variability 2.5, frequency domain 1.8, etc.); after normalization, a weighted sum is calculated (RandomForest 40%, variance 20%, mutual information 20%, physiological prior 20%), and the top 100 features are selected by score, reducing the feature set by 40%, and validating the physiological range (e.g., heart rate feature 40-200 beats / min). The output feature subset is transferred to the data augmentation module via memory mapping for efficient integration.

[0085] The data augmentation module includes a balanced class augmenter and an extreme class balancer, supporting CUDA-accelerated computation with an augmentation speed of up to 100,000 samples per minute. The module takes an optimal feature subset as input, first assessing the degree of class imbalance (graded by positive sample proportion: extremely small minority <5%, very small minority 5%-10%, moderately small minority 10%-25%, extremely small minority <5% and very few samples), and then intelligently switching augmenters based on the severity. The balanced class enhancer serves as the primary component, responsible for a mild strategy: integrating time warp (scaling factor 0.8-1.2, adjusting sequence length via linear interpolation), amplitude warp (random scaling 0.7-1.3x to simulate individual differences), and Gaussian noise injection (5% of standard deviation, dynamically adjusting noise intensity); tiered configuration parameters, such as 8x enhancement to the target proportion of the very minority class to 25%, 5x enhancement to 25% for the high minority class, and 2.5x enhancement to 35% for the moderate minority class, through up to 5 iterations (each round randomly sampling the base sample, applying 1-2 techniques to generate new samples, with a conservative coefficient of 0.9 to control the number), and imposing physiological constraints (heart rate variability <30%, blood oxygen <7%, body temperature <3%) to ensure realism. An extreme category balancer is used as an auxiliary tool, employing an aggressive strategy for ultra-extreme scenarios: 20x positive sample augmentation + 0.8x negative sample downsampling to 40% of the target, integrating ADASYN adaptive sampling, boundary sample enhancement, frequency domain shift (shift parameter -0.1-0.1, sinusoidal modulation depth <5%), and synthetic sample generation (randomly selecting two similar samples, uniformly mixing them with a weighted coefficient of 0.3-0.7, and then injecting noise into groups: heart rate variability 80% probability 8% standard deviation, heart rate 70% probability 5%, etc.); up to 10 rounds of multi-generation iterations (early large-scale generation, later fine-tuning). The augmented training set is transmitted to the model training module in tensor format, and the module monitors the augmentation completion status through an event-triggered mechanism.

[0086] The model training module integrates an algorithm pool, a dynamic algorithm selector, and an algorithm factory. Consumer-grade graphics cards (such as RTX 5090) can meet the computational requirements, ensuring that the training cycle does not exceed 2 hours. The input to this module is the augmented training set. The dynamic algorithm selector first generates a data distribution profile (four levels of balance, three levels of complexity, and augmentation factor), loads the algorithm characteristic profile (5-dimensional features of 10 algorithms: type, imbalance handling rating, speed rating, memory rating, and applicable state), calculates the matching score, and sorts them through multi-dimensional scores (40% imbalance, 30% speed, 30% memory) to form a core (3-4), exploration (1-4), and backup algorithm group.

[0087] The algorithm factory provides a unified interface: a registry maintains algorithm metadata (implementation class, type, default parameters), intelligently creates instances (merging user-defined parameters, supporting GPU acceleration flags), and exposes training (input features / labels, returns a history dictionary), prediction (input test features, returns a category), and probabilistic prediction methods. Traditional algorithms are wrapped with the scikit-learn interface, and deep learning is wrapped with PyTorch logic. Training employs a hybrid strategy: traditional machine learning uses multi-threaded parallelism (thread pool size min(candidate number, 4)), while deep learning executes sequentially to avoid resource conflicts, integrating performance monitoring, early stopping (patience 10), and checkpoint saving. The core Enhanced_GRU model implements SMOTE pre-processing (generating new samples to balance batches), GRU stacking (1-2 layers, hiding 64 dimensions to extract temporal dependencies), global average pooling (fixed-length vectors to reduce overfitting), a fully connected head (Sigmoid output probability), Focal Loss optimization (focusing on hard samples), and AdamW (learning rate 0.001, weight decay 0.01, Dropout 0.2). After training, the model and metrics are serialized and transferred to the intelligent selection module.

[0088] The intelligent selection module comprises four units: quality screening, anomaly detection, business scoring, and final selection. It employs a decision tree engine to process multi-model evaluation, ensuring a selection time of less than 10 seconds. The module takes a pool of training models as input. First, quality screening sets thresholds (precision ≥ 0.25, recall ≥ 0.3, F1 ≥ 0.3, gap ≤ 0.5) to filter out unqualified models. Anomaly detection identifies over-predictions (severe: precision ≤ 0.5 and recall ≥ 0.95; moderate: precision < 0.6 and recall > 0.85 and gap > 0.3), excluding models with false positive risks. Business scoring calculates a total score based on flight safety weights (precision 40%, recall 35%, F1 15%, efficiency 10%, efficiency being one minus relative training time), and generates a leaderboard by score. Finally, the highest-scoring model is selected, generating an inference report (method identifier, detailed dictionary, performance dictionary, advantage list, health / normal identifier). If the pool is empty, the model with the highest F1 score is selected, accompanied by a warning. This module also sends instructions to the data acquisition module, dynamically adjusting parameters based on the recognition results (such as increasing the camera frame rate to 60fps during high-risk phases, or reducing low-priority physiological sampling to optimize power consumption), thereby achieving system adaptive closed-loop and improving overall robustness. Specific Implementation Example 3:

[0090] like Figures 1 to 5As shown, this embodiment provides a detailed description of the input-output relationships and data transmission paths between system modules. It also elaborates on the core algorithm, including its input data, output results, calculation process, and specific applications within the system. This system forms a complete data processing chain through close collaboration between modules, avoiding functional isolation and ensuring efficient end-to-end operation from initial data acquisition to final alerts.

[0091] The system's data transmission path primarily follows a linear flow, supplemented by a feedback mechanism to achieve closed-loop optimization. The data acquisition module outputs the raw dataset, including facial expression sequences, physiological indicator sequences, and flight parameter labels. This dataset is directly transmitted as input to the data preprocessing module via an internal communication interface in real time, ensuring no data loss. The data preprocessing module receives the raw dataset and performs filtering, filling, and standardization. Its output is a clean, variable-length sequence training and test set, which is transmitted as an array to the feature optimization engineering module to avoid intermediate storage overhead. The feature optimization engineering module takes the preprocessed sequence dataset as input and outputs a 100-dimensional optimal feature subset, which is directly injected into the input buffer of the data augmentation module via a memory-sharing mechanism. The data augmentation module receives the optimal feature subset as input and generates an augmented balanced training set, which is transmitted in tensor format to the algorithm pool entry point of the model training module. The model training module outputs multiple trained model instances and performance metrics. These model files are transmitted serially to the candidate pool of the intelligent selection module. The intelligent selection module's final output is the optimal model and its reasoning explanation. This model is deployed in the system's early warning engine, and the recognition results are simultaneously sent back to the data acquisition module via a feedback channel. This allows for dynamic adjustment of acquisition parameters, such as increasing the sampling rate in high-risk modes or reducing the sampling accuracy of physiological indicators in low-imbalance scenarios to save resources. This process ensures that the delay from data acquisition to early warning does not exceed 5 seconds, supporting real-time flight monitoring.

[0092] In terms of algorithms, the core algorithms of the system include the dynamic algorithm selection algorithm, the Enhanced_GRU model, the RandomForest algorithm, the XGBoost algorithm, the LightGBM algorithm, the SVM algorithm, the Logistic Regression algorithm, the Gradient Boosting algorithm, the GRU algorithm, the LSTM_Attention algorithm, and the CNN-LSTM algorithm. These algorithms are distributed in the algorithm pool of the model training module and are used to handle unsafe behavior pattern recognition under different data distributions. The following provides a detailed description of the input data, output results, calculation process, and specific applications of each algorithm in the system.

[0093] The dynamic algorithm selects the distribution profile of the augmented data as input, including the class balance state (determined by dividing the number of positive samples by the total number of samples, and divided into four levels: balanced, slightly imbalanced, moderately imbalanced, and severely imbalanced), data complexity (evaluated by multiplying the sample size by the feature dimension, and divided into three levels: low, medium, and high), and augmentation factor (obtained by dividing the number of augmented positive samples by the original number of positive samples). The calculation process first loads an algorithm profile matrix, which records the type of each algorithm, its imbalance handling capability rating (excellent, good, average), training speed rating (very fast, fast, medium, slow, very slow), memory usage rating (low, medium, high, very high), and applicable balance state. Then, a matching score is calculated for each algorithm, including a balance matching score (points are added if the algorithm's applicable state covers the input's balance state, otherwise points are subtracted) and a complexity matching score (high-complexity data favors deep learning algorithms, with weighted adjustments). Subsequently, a multi-dimensional scoring function is applied to assign weights to the imbalance handling capability, training speed, and memory usage, and the results are summed to obtain a comprehensive priority score for each algorithm. Finally, the scores are sorted in descending order to form a candidate list, including a core algorithm group (selecting the top 3-4 highest-scoring algorithms), an exploration algorithm group (selecting the middle 1-4 potential algorithms), and a backup algorithm group (selecting the remaining fast and interpretable algorithms). The output is a sorted list of candidate algorithms. In the system, it is used to optimize the execution order of the model training module. For example, in the case of severely imbalanced data, Enhanced_GRU is placed first to ensure that algorithms that adapt to imbalance are trained first, thereby reducing invalid computations and improving training efficiency by more than 20%.

[0094] The Enhanced_GRU model takes enhanced variable-length sequence data as input. First, it addresses class imbalance through a pre-processing SMOTE algorithm, calculating the nearest neighbors among minority class samples to generate new samples and balance the class distribution, while ensuring reasonable feature values ​​under physiological constraints. The computation process consists of network forward propagation and optimization: the input sequence enters the GRU stacked layer, where an update gate controls the incorporation of new information, a reset gate determines the forgetting of historical states, and candidate hidden states fuse current input and historical information to generate a hidden state sequence. A global average pooling layer averages all hidden states along the time dimension to obtain a fixed-length vector to reduce overfitting risk. A fully connected classification head maps the vector to the pattern probability space through linear transformation and activation functions. The loss function focuses on hard-to-classify samples using a weighted approach, reducing the influence of easy-to-classify samples and mitigating class imbalance. The optimization process uses the AdamW optimizer to progressively adjust model parameters, combined with Dropout to randomly discard some units in the hidden layers to prevent overfitting, and monitors validation set performance to achieve early stopping. The output is the probability of unsafe modes for each input sequence. In the system, it is used as a core component of the early warning engine. It receives current flight data in real time, calculates probability thresholds to trigger alarms, such as activating voice prompts when the probability exceeds 0.5, to help pilots correct their behavior in time.

[0095] The RandomForest algorithm takes tabular feature data as input, including the optimal feature subset and behavioral pattern labels. The computation process first constructs multiple decision trees, each randomly sampling a subset of data and features, recursively splitting nodes until purity is maximized, selecting the optimal split point using the Gini impurity index. Then, the results from all trees are aggregated through majority voting to generate the final classification decision. The output is the pattern prediction probability and feature importance ranking. In systems, it is the preferred algorithm for balancing data scenarios, used for quickly identifying low-complexity patterns, such as evaluating feature contributions in the training module to help optimize subsequent iterations.

[0096] The XGBoost algorithm takes a feature matrix and a label vector as input. The computation process iteratively builds a decision tree using a gradient boosting framework. In each round, the residual is calculated as the target for the new tree. The learning rate is applied to scale the tree contribution, and a built-in regularization term controls the tree complexity. It also supports built-in imbalance handling, such as weight adjustment. The output is the boosted prediction score. In the system, it is used to handle moderately imbalanced data, providing high-accuracy predictions. For example, in early warning systems, it can be combined with physiological indicators to quickly detect grounding stability anomalies.

[0097] The LightGBM algorithm has similar inputs to XGBoost, but its computation process uses histogram approximation to accelerate splitting and searching, a leaf-first growth strategy to reduce computational overhead, and supports class weight imbalance adjustment. The output is a highly efficient prediction result, making it an alternative for memory-constrained scenarios, providing fast training for real-time feedback and adjustment.

[0098] The SVM algorithm takes a high-dimensional feature space as input. The computation process maps the data to a high dimension using a kernel function, finds the maximum margin hyperplane, and optimizes the support vectors to minimize classification error. It supports various kernels, such as the Gaussian kernel, to handle non-linearity. The output is the classification boundary decision. Its application in the system is high-dimensional physiological data classification, ensuring stable predictions such as in the detection of abnormal blood oxygenation.

[0099] The Logistic Regression algorithm takes a linear combination of features as input. The computation process maps the linear combination to a probability using a logistic function, optimizes the coefficients using gradient descent, and adds regularization to prevent overfitting. The output is a probability estimate. In systems, it serves as a highly interpretable benchmark model, used to validate the performance of other algorithms in simpler modes.

[0100] The Gradient Boosting algorithm takes features and residual sequences as input. The computation process is similar to XGBoost, but with a greater emphasis on robustness, correcting previous errors by sequentially adding weak learners. The output is a cumulative prediction. Its application in systems is the progressive optimization of complex temporal patterns, such as recognizing multi-level height variations.

[0101] The input to the GRU algorithm is a time sequence. The computation process updates the hidden state through a gating mechanism, simplifying LSTM to improve efficiency. The output is a time sequence code. Its application in the system is basic time series modeling, assisting Enhanced_GRU in pre-training on slightly imbalanced data.

[0102] The LSTM Attention algorithm takes long sequences of data as input. The computation process combines LSTM memory units and an attention mechanism, assigning weights to key time steps. The output is an encoding of the focus of attention. Its application in systems is capturing long-dependent patterns, such as detecting gradual slope changes.

[0103] The CNN-LSTM algorithm takes a spatiotemporal sequence as input. The computation process first uses convolution to extract spatial features, then uses LSTM to process temporal dependencies. The output is a mixed-mode prediction. In this system, it is used to process physiological flight data that combines spatial and temporal aspects, improving the accuracy of post-landing stability recognition. Specific Implementation Example 4:

[0105] like Figures 1 to 5 As shown, the following are specific use cases:

[0106] 1. Application in pilot training simulators:

[0107] The system is deployed in the simulator cockpit of an aviation training center and interfaces with the simulator software. During training, a new pilot performs a landing simulation mission. The system captures facial expressions through a cockpit camera, collects heart rate variability, body temperature, and blood oxygen data through a smartwatch, and provides virtual flight parameters such as altitude changes and speed through the simulator. The data acquisition module aggregates the raw dataset in real time, the preprocessing module filters the landing phase data and fills in missing heart rate values ​​(using the individual baseline median), and the feature optimization module extracts 100-dimensional features such as frequency domain power and the proportion of physiological abnormalities. The model training module dynamically selects XGBoost and Enhanced_GRU as the core algorithms for parallel prediction; the intelligent selection module selects the Enhanced_GRU model (with a total business score of 0.68) through a four-stage evaluation. In the simulation, the system identifies an abnormally high level of facial arousal combined with fluctuations in heart rate variability, predicting a "high-slope descent" probability of 0.75, immediately triggering simulator warning lights and a voice prompt to "adjust the descent rate." After the training, the system generates a report showing a recall rate of 65%, helping instructors analyze behavioral pattern deviations under pilot stress, optimize training plans, and improve the pass rate of new pilots by 15%.

[0108] 2. Real-time monitoring during long-haul commercial flights:

[0109] This system is integrated into the cockpit electronics of a passenger aircraft and networked with the onboard computer. After a transoceanic flight takes off, the system continuously monitors the facial expressions and physiological indicators of the two pilots, while simultaneously collecting parameters from the flight quality monitoring system, such as attitude and grounding stability. The data acquisition module ensures synchronization of the three types of data, and the preprocessing module reassembles them into variable-length sequences and standardizes features (e.g., global Z-score processing for blood oxygen saturation). The feature optimization module expands statistical features to detect trend intensity anomalies. The model training module prioritizes LightGBM and LSTM_Attention based on data profiling. After prediction, the intelligent selection module eliminates over-predictive models and selects LSTM_Attention (precision weighted, total score 0.71). As the flight approaches landing, the system analyzes a pilot's decreased valence and increased heart rate standard deviation, and combined with flight parameters, predicts a "grounding micro-stability anomaly" probability of 0.62, sending a real-time alert to the head-mounted device, prompting "check grounding attitude." The pilot then makes minor adjustments to avoid the potential risk of grounding overload; the entire process has a delay of only 1.5 seconds. Post-flight logs showed that the system reduced false alarm fatigue by 5%, improved crew alertness, and ensured safe landing of the flight.

[0110] 3. High-intensity early warning during military flight missions:

[0111] Installed in the cockpit of a fighter jet, this system is adapted to high G-forces and complex environments. During missions, the system collects facial data via a shock-resistant camera embedded in the helmet, monitors physiological indicators using a military smartwatch, and provides real-time flight parameters such as multi-level altitude changes via the mission computer. The data acquisition module aggregates the dataset, and the preprocessing module applies physiological constraints to fill in missing body temperature values ​​(with day-night adjustments). The feature optimization module focuses on extracting high-frequency power features to capture stress responses. The model training module intelligently selects Enhanced_GRU and CNN-LSTM as cores, performing sequential predictions to manage resources; the intelligent selection module excludes models with excessive recall through anomaly detection, selecting CNN-LSTM (dominated by F1 score, total score 0.65). During missions, the system detects pilot anger combined with decreased blood oxygen levels, predicting an "anomaly within a specific altitude range" probability of 0.81, activating vibration alarms and displaying "stable altitude" on the HUD. The pilot corrects in time, avoiding potential collisions. This application improves mission success rates and reduces losses caused by human error.

[0112] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising a reference structure" does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.

[0113] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for identifying unsafe behaviors based on pilot facial expressions and physiological indicators, characterized in that, Includes the following steps: Sp1: Multidimensional data acquisition involves collecting pilot facial expression data via cameras and facial expression calculation software, collecting pilot physiological indicator data via wearable devices, and collecting flight parameter data through a flight quality monitoring system to generate pilot behavior pattern labels. These three types of data—facial expression data, physiological indicator data, and flight parameter data—are aggregated to form the raw dataset. The facial expression data includes time-series data of valence and arousal, while the physiological indicator data includes time-domain features of heart rate variability, heart rate, blood oxygen saturation, and body temperature. The cameras, facial expression calculation software, and wearable devices all interact with the flight quality monitoring system to ensure the time synchronization of the collected data. Sp2: Raw data preprocessing involves filtering, quality screening, and reorganization of the raw dataset. A multi-level intelligent imputation strategy is employed to handle missing values, and features are standardized. The data is reorganized into a variable-time series sample dataset, and a hierarchical cross-validation method is used to divide the training and test sets. The multi-level intelligent imputation strategy is based on individual and population baselines, and combines the physiological meaning of features with correlation inference, multivariate derivation, baseline quantile sampling, and a physiological constraint fallback scheme to imput missing features. Sp2 receives the raw dataset summarized in Sp1, and the processed data is transmitted to Sp3. Sp3: Feature optimization employs a two-stage progressive optimization architecture. The first stage expands and generates a candidate feature pool through frequency domain feature extraction, statistical feature extraction, and physiological feature extraction. The second stage performs intelligent feature selection based on a multi-dimensional feature importance evaluation system to obtain the optimal feature subset. The multi-dimensional feature importance evaluation system integrates RandomForest importance evaluation, correlation analysis, analysis of variance, statistical significance test, and physiological prior knowledge dimensions. Sp3 interacts with Sp2, receiving preprocessed data and outputting the optimal feature subset to Sp4. Sp4: Differentiated category balancing enhancement. Based on the degree of category imbalance of each flight mode label, it intelligently selects a balanced category enhancer or an extreme category enhancer to augment the training set. The balanced category enhancer adopts a mild balancing strategy, integrating time warp, amplitude warp, and Gaussian noise injection techniques and applying physiological constraints. The extreme category enhancer adopts an aggressive balancing strategy, integrating synthetic sample generation and frequency domain shift techniques, combined with a negative sample downsampling mechanism. Sp4 receives the optimal feature subset output by Sp3, and the enhanced training set is transmitted to Sp5. Sp5: Model training. An algorithm pool containing traditional machine learning algorithms and deep learning algorithms is constructed. Candidate algorithm combinations are dynamically selected based on the distribution characteristics of the enhanced data, and a parallel competitive training mechanism is used to complete model training. The deep learning algorithm includes the Enhanced_GRU model, which integrates FocalLoss and SMOTE technologies and supports variable time series processing. Sp5 interacts with Sp4, receives the enhanced training set, and the trained model is transferred to Sp6. Sp6: Intelligent model selection employs a four-stage selection strategy to determine the optimal model, enabling the identification and early warning of unsafe pilot behavior patterns. The four-stage selection strategy consists of a quality screening stage, an anomaly detection stage, a service scoring stage, and a final selection stage. The service scoring stage uses a preset weight configuration to calculate a comprehensive service score. Sp6 receives the trained model output from Sp5, selects it, and outputs the optimal model for unsafe behavior pattern identification and early warning.

2. The method for identifying unsafe behaviors based on pilot facial expressions and physiological indicators according to claim 1, characterized in that, The screening of the original dataset described in Sp2 includes flight phase screening and operator screening, retaining data records marked as landing phase and actually performed by pilots; the evaluation criteria for quality screening are that each feature has at least 3 valid values ​​and at least 50% of the features are in a valid state.

3. The method for identifying unsafe behaviors based on pilot facial expressions and physiological indicators according to claim 1, characterized in that, The process of reorganizing the data into a variable-length time series sample dataset as described in Sp2 adopts a variable-length sequence storage method to maintain the original time length of each flight sample; the hierarchical cross-validation adopts a five-fold multi-label hierarchical K-fold algorithm to ensure that the proportion of various label combinations in each fold is consistent with the overall dataset.

4. The method for identifying unsafe behaviors based on pilot facial expressions and physiological indicators according to claim 1, characterized in that, In the first stage of SP3, the frequency domain feature extraction uses the Welch power spectral density estimation method for heart rate variability signals to extract power values ​​and derived indices in the very low frequency band, low frequency band, and high frequency band, and actively excludes the total power frequency band to reduce redundancy; the physiological feature extraction in the first stage calculates the proportion of abnormal values ​​of physiological features relative to the normal range and the range utilization rate.

5. The method for identifying unsafe behaviors based on pilot facial expressions and physiological indicators according to claim 1, characterized in that, The class imbalance described in Sp4 is divided into four levels: very minority class, high minority class, moderate minority class, and extremely minority class. The class balancer is configured with different enhancement factors and technology combinations for the very minority class, high minority class, and moderate minority class respectively. The extreme class balancer is configured with aggressive enhancement parameters for the extremely minority class. The synthetic sample generation is achieved by mixing two real samples of the same class and injecting noise.

6. The method for identifying unsafe behaviors based on pilot facial expressions and physiological indicators according to claim 1, characterized in that, The process described in Sp5 for dynamically selecting candidate algorithm combinations based on the distribution characteristics of enhanced data involves generating a data distribution profile, constructing an algorithm characteristic profile, implementing a hierarchical algorithm selection strategy based on data balance, data complexity, and enhancement factor, and calculating the comprehensive priority of the algorithms through a multi-dimensional scoring function.

7. The method for identifying unsafe behaviors based on pilot facial expressions and physiological indicators according to claim 1, characterized in that, In SP6, the quality screening stage of the four-stage selection strategy sets a performance threshold to adapt to imbalanced data; the anomaly detection stage of the four-stage selection strategy identifies over-prediction patterns in the algorithm; the business scoring stage of the four-stage selection strategy is weighted as follows: precision 40%, recall 35%, F1 score 15%, and efficiency 10%; the final selection stage of the four-stage selection strategy selects the model with the highest comprehensive score, and the alternative selection mechanism is activated when the candidate model pool is empty.

8. A system based on the unsafe behavior recognition method based on pilot facial expressions and physiological indicators as described in any one of claims 1-7, characterized in that, The system includes a data acquisition module, a data preprocessing module, a feature optimization engineering module, a data augmentation module, a model training module, and an intelligent selection module, and is used to execute the method. The data acquisition module is used to collect pilots' facial expression data, physiological index data, and flight parameter data, and generate behavioral pattern labels, which are then aggregated to form a raw dataset. The data acquisition module includes a camera and facial expression calculation software, a wearable device, and a flight quality monitoring submodule. The camera and facial expression calculation software and the wearable device all interact with the flight quality monitoring submodule to ensure the time synchronization of the collected data. The data acquisition module is communicatively connected to the data preprocessing module to transmit the raw dataset to the data preprocessing module. The data preprocessing module is used to filter, quality screen and reorganize the original dataset, handle missing values, standardize features, reorganize data and divide it into training set and test set; the data preprocessing module has a built-in multi-level intelligent imputation unit, which constructs individual baseline and group baseline, and selects the corresponding imputation scheme in combination with the physiological meaning of features. The data preprocessing module interacts with the feature optimization engineering module, transmitting the processed data to the feature optimization engineering module. The feature optimization engineering module is used to implement feature expansion and intelligent feature selection to obtain the optimal feature subset. The feature optimization engineering module includes an intelligent feature engineer and an intelligent feature selector. The intelligent feature engineer integrates frequency domain, statistical and physiological feature extraction techniques, and the intelligent feature selector adopts a multi-dimensional feature importance evaluation system. The feature optimization engineering module communicates with the data augmentation module to transmit the optimal feature subset to the data augmentation module; The data augmentation module is used to solve the class imbalance problem of training data, and includes a balanced class augmenter and an extreme class balancer. The balanced class augmenter adopts a mild balancing strategy and applies physiological constraints, while the extreme class balancer adopts an aggressive balancing strategy and supports synthetic sample generation and negative sample downsampling. The data augmentation module interacts with the model training module, transferring the augmented training set to the model training module. The model training module is used to implement parallel competitive training of multiple algorithms, and includes an algorithm pool, a dynamic algorithm selector, and an algorithm factory. The algorithm pool includes traditional machine learning algorithms and deep learning algorithms. The dynamic algorithm selector selects candidate algorithm combinations based on data distribution characteristics. The algorithm factory provides a unified interface for creating and calling algorithm instances. The model training module communicates with the intelligent selection module and transmits the trained model to the intelligent selection module. The intelligent selection module is used to determine the optimal model through a four-stage selection strategy, thereby enabling the identification and early warning of unsafe behavior patterns of pilots. The intelligent selection module includes a quality screening unit, an anomaly detection unit, a service scoring unit, and a final selection unit. The intelligent selection module maintains instruction transmission with the data acquisition module and can adjust the data acquisition parameters based on the feedback of the identification results.

9. The system corresponding to the unsafe behavior recognition method based on pilot facial expressions and physiological indicators according to claim 8, characterized in that, The data acquisition module includes a wearable device, a smartwatch, which supports the collection of PPG-based heart rate time intervals, mean heart rate, body temperature, and blood oxygen levels. The data acquisition module also includes facial expression calculation software, a facial expression analysis system, which can analyze seven basic emotions, valence, and arousal indicators.

10. The system corresponding to the unsafe behavior recognition method based on pilot facial expressions and physiological indicators according to claim 8, characterized in that, The model training module includes deep learning algorithms such as Enhanced_GRU, GRU, LSTM_Attention, and CNN-LSTM, and traditional machine learning algorithms such as RandomForest, XGBoost, LightGBM, SVM, LogisticRegression, and GradientBoosting. The Enhanced_GRU model integrates FocalLoss and SMOTE techniques and includes GRU stacked layers, global average pooling layers, and a fully connected classification head.

Images