A method for pilot workload assessment

Abstract

The application discloses a kind of pilot workload evaluation methods, comprising: data acquisition and data pre-processing step;Collect the original electroencephalogram signal of pilot and original electromyogram signal and pre-process;Data enhancement step;Position coding is added in time sequence feature sequence, obtain multimodal time sequence feature;From time sequence feature sequence and EMG time sequence feature sequence, extract out the fusion feature highly relevant with workload state;Total loss function L construction step;Determine total loss function L by workload classification loss and antagonistic loss function;Electroencephalogram signal and electromyogram signal are input into multimodal fusion classification model, and the workload evaluation result of pilot is output.The pilot workload evaluation method of the application has the advantages that low load, medium compliance and high load three kinds of workload state can be automatically distinguished, the accuracy and stability of pilot workload evaluation are improved, etc.

Description

Technical Field

[0001] This invention relates to a physiological signal processing technique, and more particularly to a method for assessing pilot workload. Background Technology

[0002] Pilot workload assessment is a core issue in aviation human factors engineering and flight safety. Flight missions are typically characterized by strong time-varying and high-risk features. Both excessively low and excessively high workloads can lead to decreased operational performance and even trigger flight accidents. Existing research on pilot workload assessment largely relies on indicators such as behavioral performance, operational error rate, mission completion time, and subjective scales (e.g., NASA-TLX). While this assessment method can reflect workload levels to some extent, it suffers from drawbacks such as strong subjectivity and limited timeliness, making it difficult to continuously and objectively monitor the pilot's instantaneous state in complex flight scenarios.

[0003] With the development of physiological signal acquisition and analysis technologies, an increasing number of studies are incorporating physiological indicators such as electroencephalography (EEG), electrocardiography (ECG), and electromyography (EMG) to assist in assessing the physiological and psychological workload of pilots. Among these, EEG signals can reflect changes in cognitive activity in the cerebral cortex, making them suitable for characterizing mental workload; forearm EMG signals are directly related to flight control behaviors such as joystick movements, reflecting muscle exertion and operational tension. Multimodal physiological signals can theoretically provide a more comprehensive representation of workload, but existing methods mostly employ a single modality (e.g., using only EEG) or perform simple feature splicing in multimodal scenarios, failing to fully consider the distribution differences and complementary information between different modalities, resulting in insufficient fusion efficiency and robustness.

[0004] In recent years, deep learning technology has been widely applied to physiological signal analysis and workload identification tasks. Models such as Convolutional Neural Networks (CNN), Recurrent Neural Networks (RNN), and Long Short-Term Memory Networks (LSTM) can automatically extract temporal and local spatial features; compared with traditional hand-crafted features and shallow classifiers, their recognition performance has been improved to some extent. However, these models have limited ability to model long-range temporal dependencies and global context, making it difficult to fully capture the complex dynamic changes across phases and time slices in flight missions. At the same time, most works are still based on single-modal signals, or do not explicitly model the alignment relationship between modalities in multimodal scenarios, making them susceptible to cross-modal distribution shifts and noise interference.

[0005] In multimodal learning, some studies have attempted to use multiple physiological signals as network inputs to identify workload or fatigue states through early or mid-stage feature fusion. However, these methods typically only perform simple concatenation or weighted averaging at the feature layer, lacking adaptive alignment mechanisms for different modal feature distributions. When EEG and EMG differ significantly in statistical distribution, signal-to-noise ratio, and temporal structure, direct fusion can easily lead to one modality dominating the training process, weakening the complementary advantages of multimodal learning and limiting the model's generalization ability and stability in real-world flight environments.

[0006] Therefore, there is an urgent need for a multimodal deep learning method for pilot workload assessment that can solve the above-mentioned problems in multimodal signal fusion. Summary of the Invention

[0007] To avoid the shortcomings of the existing technologies, this invention provides a method for assessing pilot workload. This method can fully model multi-scale temporal relationships based on the combined use of EEG and EMG signals, and alleviate cross-modal distribution differences through an effective modal alignment mechanism. This enables high accuracy and robustness in identifying different workload levels, such as low, medium, and high workloads, in complex flight mission scenarios.

[0008] The present invention adopts the following technical solution to solve the technical problem.

[0009] A method for assessing pilot workload according to the present invention includes the following steps: Step 1: Data Acquisition and Preprocessing; Acquiring the pilot's raw EEG signals X e and the original electromyographic signal X m and the original EEG signal X e and the original electromyographic signal X m Preprocessing was performed to obtain the processed EEG signal time segment X. e,t and electromyographic signal time fragment X m,t ; Step 2: Data augmentation step; X-ray of EEG signal time segment e,t and electromyographic signal time fragment X m,t Data augmentation was performed, and then the augmented EEG signal time segment was X-rayed. e,t and electromyographic signal time fragment X m,t Mapped to a unified feature dimension D; Step 3: In the time-series feature sequence Positional encoding is added to the data, and then multimodal temporal features Z are obtained through a multimodal feature extraction module. {e,m} ; Step 4: Process the multimodal temporal features Z using the multimodal feature fusion module F. {e,m}EEG time series feature sequence Z e and EMG time series feature sequence Z m Processing is performed to align the distributions of the EEG feature space and the EMG feature space; Step 5: Constructing the total loss function L; classifying the loss L based on workload. cls and adversarial loss function L adv To determine the total loss function L; Step 6: Transfer the EEG signal X e and electromyographic signal X m Input temporal relationship modeling and multi-channel adversarial alignment network, output pilot workload assessment results.

[0010] The structural features of the pilot workload assessment method of the present invention also lie in: Furthermore, in step 1, the preprocessing includes bandpass filtering, power frequency interference suppression, and removal of obvious artifacts.

[0011] Furthermore, in step 1, after preprocessing, the original EEG signal X is processed for a fixed duration. e and the original electromyographic signal X m Segmentation and time alignment are performed to obtain time segment X of the EEG signal. e,t and electromyographic signal time fragment X m,t .

[0012] Furthermore, in step 2, data augmentation is performed using random flipping and additive Gaussian noise.

[0013] Furthermore, in step 2, a one-dimensional temporal convolution method is used to convert the time segment X of the EEG signal. e,t Feature dimensions and electromyographic signal time segments X m,t The feature dimension is unified to D-dimensional.

[0014] Furthermore, step 3 includes the following steps: Step 31: In the time-series feature sequence By adding positional encoding, a feature sequence O with positional encoding is obtained. {e,m} ; Step 32: Extract the feature sequence O with positional encoding. {e,m} In the input multi-head self-attention structure, the multi-head self-attention output features are obtained. ; Step 33: [The text appears to be incomplete and contains several grammatical errors. A more accurate translation would require the The multimodal temporal features Z are obtained by processing the input into the multimodal feature extraction module F. {e,m} .

[0015] Furthermore, in step 33, the multimodal feature extraction module E includes a position feedforward network and a layer normalization submodule LayerNorm.

[0016] Furthermore, step 4 includes the following steps: Step 401: The multimodal feature fusion module F extracts data from the EEG time-series feature sequence Z. e and EMG time series feature sequence Z m Extract the fusion feature F(Z) that is highly correlated with the workload state. {e,m} ); Step 402: Modality Classifier M cls For the fusion feature F(Z) {e,m} Modal discrimination is performed using workload classifier C. cls The merged features are classified into three workload categories; Step 403: Based on the modal classification results Define adversarial loss L adv .

[0017] Furthermore, step 5 includes the following steps: Step 501: Workload Classification Loss L cls Calculation steps; Step 502: Calculation steps for the total loss function L; Step 503: Calculation steps for workload prediction label c; Step 504: Calculation steps for accuracy (Acc) and average F1 score (F1) across multiple classes.

[0018] Furthermore, in step 6, the workload evaluation results of the temporal relationship modeling and multi-channel adversarial alignment network include three categories: low workload, medium workload, and high workload.

[0019] Compared with existing technologies, the beneficial effects of this invention are reflected in: This invention discloses a method for assessing pilot workload, comprising: data acquisition and data preprocessing steps; acquiring the pilot's raw electroencephalogram (EEG) signals X. e and the original electromyographic signal X m And preprocessing; data augmentation steps; in the time-series feature sequence By incorporating positional encoding, multimodal temporal features O are obtained. {e,m} From the EEG time series feature sequence Z e and EMG time series feature sequence Z m Extract the fusion feature F(Z) that is highly correlated with the workload state. {e,m} ); Steps for constructing the total loss function L; Loss L is classified by workload clsand adversarial loss function L adv Determine the total loss function L; use the EEG signal X e and electromyographic signal X m Input a multimodal fusion classification model and output the pilot's workload assessment results.

[0020] This invention aims to address the problems in pilot workload assessment, such as reliance on subjective scales and single physiological indicators leading to highly subjective results, insufficient real-time performance, and inadequate accuracy. It also addresses the shortcomings of existing multimodal methods, which often rely on simple feature concatenation, struggle to effectively align EEG and EMG feature distributions, and exhibit poor robustness. By constructing a pilot workload assessment method based on Transformer temporal relationship modeling and a multimodal adversarial alignment network, this invention jointly utilizes EEG and EMG signals from flight missions for feature extraction and fusion. This allows for automatic differentiation of three workload states: low, moderate, and high, thereby improving the accuracy and stability of pilot workload assessment.

[0021] The pilot workload assessment method of the present invention has the advantages of being able to automatically distinguish between three types of workload states: low workload, medium workload, and high workload, and improving the accuracy and stability of pilot workload assessment. Attached Figure Description

[0022] Figure 1 This is a schematic diagram of the overall process of the pilot workload assessment method of the present invention; Figure 2 This is a system framework diagram of the pilot workload assessment method of the present invention;

[0023] Figure 3 This is a schematic diagram of the experimental design for pilot workload assessment according to the present invention;

[0024] Figure 4 This is a performance comparison chart of the method of the present invention and two existing comparative methods in pilot workload prediction tasks;

[0025] Figure 5 A schematic diagram of the confusion matrix for the workload assessment results of all subjects;

[0026] Figure 6 Typical EEG topography maps at different workload levels (high, medium, and low) obtained using the method of this invention;

[0027] The present invention will be further described below through specific embodiments and in conjunction with the accompanying drawings. Detailed Implementation

[0028] See Figures 1-6 The present invention provides a method for assessing pilot workload, comprising the following steps: Step 1: Data Acquisition and Preprocessing; Acquiring the pilot's raw EEG signals X e and the original electromyographic signal X m and the original EEG signal X e and the original electromyographic signal X m Preprocessing was performed to obtain the processed EEG signal time segment X. e,t and electromyographic signal time fragment X m,t ; Acquiring raw EEG signals X from pilots during flight simulation missions e and the original electromyographic signal X m Furthermore, multimodal data with workload labels was constructed to provide basic input for subsequent temporal relationship modeling and training of multi-channel adversarial alignment networks. The multimodal data in this invention includes, but is limited to, EEG and EMG modalities. In a standard flight simulator environment, several pilot subjects were recruited to participate in flight mission experiments. The entire flight mission was divided into several typical phases, including but not limited to taxiing, takeoff, cruise, approach, and landing. During the experiment, EEG electrodes were placed on the subjects' scalps, and surface electromyography (EMG) electrode arrays were placed in the forearm extensor muscle region to synchronously collect data throughout the flight mission.

[0029] Step 2: Data augmentation step; X-ray of EEG signal time segment e,t and electromyographic signal time fragment X m,t Data augmentation was performed, and then the augmented EEG signal time segment was X-rayed. e,t and electromyographic signal time fragment X m,t Mapped to a unified feature dimension D; Step 3: In the time-series feature sequence Positional encoding is added to the data, and then multimodal temporal features Z are obtained through a multimodal feature extraction module. {e,m} ; The time-series feature sequence obtained in step 2 Based on this, positional encoding is introduced and the multi-head self-attention mechanism of Transformer is used to model long-range temporal dependencies, resulting in the multimodal temporal features Z output by the multimodal feature extraction module E. {e,m} .

[0030] Step 4: Process the multimodal temporal features Z using the multimodal feature fusion module F. {e,m} EEG time series feature sequence Z e and EMG time series feature sequence Z m Processing is performed to align the distributions of the EEG feature space and the EMG feature space; The multimodal temporal features Z obtained in step 3 {e,m}Based on this, a multimodal feature fusion module F is used to extract discriminative features that are highly correlated with the workload, and a modality classifier M is introduced. cls With the adversarial training mechanism, the distribution of EEG and EMG feature spaces is aligned, reducing the distribution offset between different modalities.

[0031] Step 5: Constructing the total loss function L; classifying the loss L based on workload. cls and adversarial loss function L adv To determine the total loss function L; Based on the multimodal fusion features obtained in step 4, the workload classifier C is used. cls Pilot workload is categorized into three classes: low workload, medium workload, and high workload, and this is combined with adversarial alignment. adv Loss L related to workload classification cls Construct the total loss function L, and provide the decision rules and performance evaluation metrics for the testing phase.

[0032] Step 6: Transfer the EEG signal X e and electromyographic signal X m Input temporal relationship modeling and multi-channel adversarial alignment network, output pilot workload assessment results.

[0033] Using the temporal relationship modeling and multi-channel adversarial alignment network trained in step 5, the input EEG signal X is processed. e and electromyographic signal X m The system performs feature extraction and fusion classification reasoning to output three types of workload assessment results: low workload, medium workload, and high workload, thus completing the pilot workload assessment task.

[0034] like Figure 1This invention discloses a method for assessing pilot workload, comprising steps including data acquisition and preprocessing, data augmentation, one-dimensional convolution, positional encoding, Transformer multi-head self-attention, multimodal feature fusion, modal adversarial alignment, and output workload classification. First, during flight simulation missions, the pilot's electroencephalogram (EEG) and forearm electromyography (EMG) signals are simultaneously acquired. The raw signals are filtered for noise reduction, artifact removal, and segmented into fixed time windows. Based on the flight mission stage and subjective scale labeling, each time segment is assigned to three workload categories: normal, mild fatigue, and severe fatigue, thus completing data acquisition and preprocessing. Second, each pair of EEG and EMG segments undergoes data augmentation operations such as random flipping and additive Gaussian noise to expand the effective sample and improve the model's robustness to noise and individual differences. The augmented multimodal data is then mapped to a unified feature dimension through a one-dimensional convolutional layer to extract local temporal structure features. Subsequently, positional encoding based on sine and cosine functions is added to the convolutional features and input into the Transformer multi-head self-attention module to model the long-range dependency between EEG and EMG signals on a global time scale, obtaining multimodal temporal features containing temporal context information. Next, the temporal features from the two modalities are fed into a multimodal feature fusion module for joint modeling. A one-dimensional fusion representation suitable for classification is obtained through cascaded convolution, pooling, and flattening operations. Simultaneously, a modality classifier and gradient inversion mechanism are introduced to form a modality adversarial alignment module, which allows the fused features to benefit workload classification during training while reducing the distributional differences between EEG and EMG in the feature space. Finally, the aligned fusion features are input into a workload classifier, outputting three workload prediction results: normal, mild fatigue, and severe fatigue, thus completing the automatic assessment of the pilot's workload status. The above steps together constitute... Figure 1 The overall flow of the pilot workload assessment method of the present invention is shown.

[0035] In specific implementation, step 1 includes preprocessing such as bandpass filtering, power frequency interference suppression, and removal of obvious artifacts.

[0036] In specific implementation, in step 1, after preprocessing, the original EEG signal X is processed for a fixed duration. e and the original electromyographic signal X m Segmentation and time alignment are performed to obtain time segment X of the EEG signal. e,t and electromyographic signal time fragment X m,t .

[0037] Let X be the raw EEG signal collected during each flight mission. e And the original electromyographic signal is X m ;X e For multi-channel EEG time series, X mThis is a multichannel electromyography time series. For X... e and X m Standard data preprocessing was performed separately. Data preprocessing included bandpass filtering, power line interference suppression, and removal of obvious artifacts. To adapt to the input format of subsequent temporal relationship modeling and multi-channel adversarial alignment networks, the preprocessed EEG and EMG signals were segmented into fixed-duration segments with a window length of 1 second and no overlap between windows. The preprocessed EEG signals were divided into a series of time segments, denoted as X. e,t The preprocessed electromyographic signal was segmented into segments corresponding to the X signal. e,t A series of time-aligned segments, denoted as X. m,t Where t represents the t-th time window in this task. Time alignment ensures that X corresponding to the same t is... e,t With X m,t Being in the same time interval, thus forming a pair of synchronized multimodal input samples: X e,t With X m,t .

[0038] After completing the flight mission, each participant's subjective workload level was scored according to the NASA-TLX workload scale for each flight phase. Based on pre-defined grouping rules, the scores for each phase were divided into three categories: "low workload," "medium workload," and "high workload," and discrete label values ​​were assigned to each of these three workload categories. For example, using y... t The category label represents the workload in the t-th time window of a certain flight phase, taking a value of 1, 2, or 3, corresponding to low, medium, and high workloads respectively; for example, y1 uses the category label to represent "low workload". All time segments X within this flight phase... e,t With X m,t All inherit the category label y corresponding to this stage. t .

[0039] In specific implementation, step 2 uses random flipping and additive Gaussian noise for data augmentation.

[0040] The raw EEG signal X obtained in step S1 e and the original electromyographic signal X m Data augmentation is performed, and the two modalities are mapped to a unified feature dimension through one-dimensional convolution, preparing for subsequent temporal modeling. For each pair of multimodal inputs (X... e X m Random flipping and additive Gaussian noise enhancement are performed. This paper addresses two modes, namely the one-to-many modal input (X... e X m This is uniformly denoted as "original sequence X". {e,m} ", i.e. X{e,m} Including X e and X m The markers {e,m} in the following text indicate that they include data related to both EEG and EMG modes, and are consistent with X here. {e,m} The expression is similar; the original sequence X is expressed by the following formula (1). {e,m} Enhanced samples are generated after processing.

[0041] (1); In formula (1), Indicates the original sequence The flipped sequence obtained after performing a random flip operation on the timeline; Indicates in the original sequence Superimposed Gaussian noise N(0, The superimposed sequence obtained after ) ; N(0, () indicates that the mean is 0 and the variance is 0. Gaussian noise; This represents the enhanced multimodal augmentation sequence.

[0042] In specific implementation, step 2 uses one-dimensional temporal convolution to process the time segments of the EEG signal. Feature dimensions and electromyographic signal time segments The feature dimension is unified to D-dimensional.

[0043] To project data from different modalities onto the same feature dimension D, the enhanced multimodal augmentation sequence is... One-dimensional temporal convolution Conv1D processing is performed to extract local temporal features in the time dimension, and a preliminary temporal feature representation is obtained, as shown in the following formula (2). (2); In formula (2), the right side of the equal sign This represents the multimodal enhancement sequence obtained through formula (1); the left side of the equals sign represents... This indicates that the temporal feature sequence obtained after one-dimensional convolution has its feature dimension unified to D dimensions; in this invention, the left and right sides of the equal sign... The physical meanings are the same; both represent the feature sequence of the modality at the current stage. The only difference is that the right side represents the input before the one-dimensional convolution, and the left side represents the output after the one-dimensional convolution. For the sake of simplification, the notation is unified using... express; (Right now and The values ​​represent the one-dimensional convolution kernel sizes for the electroencephalogram (EEG) and electromyogram (EMG) channels, respectively. Step 2 completes the processing of the raw EEG signals. and raw electromyographic signals Data augmentation and one-dimensional convolutional feature mapping are used to obtain temporal feature sequences for subsequent temporal modeling. .

[0044] In practice, step 3 includes the following steps: Step 31: In the time-series feature sequence By adding positional encoding, a feature sequence with positional encoding is obtained. ; First, traditional sine and cosine positional encodings are added to the convolutional sequence to introduce temporal positional information, resulting in a feature sequence with positional information. See formula (3) below; (3); In formula (3), For convolutional sequences The matrix obtained by position encoding represents the relationship between... A sine-cosine position coding matrix of the same size is used to incorporate time position information; The element definition is shown in the following formula (4); (4); In formula (4), This represents the element value at position index pos and feature channel index 2k in the position encoding matrix PEX; This represents the element value at position index pos and feature channel index 2k+1 in the position encoding matrix PEX; D represents the total dimension of the convolution output features (i.e., the feature dimension mentioned in Formula 2 is unified to D dimensions). Convolutional sequence The position index is L, where L is the convolution sequence. The sequence length, Indexed by feature dimensions.

[0045] Step 32: Input the feature sequence O{e,m} with positional encoding into the multi-head self-attention structure to obtain the multi-head self-attention output features. ; Then, the feature sequence with position encoding Input is fed into a multi-head self-attention structure. First, three learnable matrices are used to... The mapping is as follows: query (Q), key (K) and value (V), as shown in formula (5) below; (5) In formula (5), Q, K, and V represent the input feature sequences, respectively. The query matrix, key matrix, and value matrix obtained by linear transformation These are the learnable weight parameters for the query matrix Q, the key matrix K, and the value matrix V, respectively.

[0046] Based on the above (Q, K, V), the output of a single self-attention head This can be expressed as the following formula (6); (6); In formula (6), This represents the output feature sequence of the i-th self-attention head in the EEG or EMG mode. This represents the transpose of the key matrix K. Softmax is the normalization function, which normalizes the attention score at each time point so that the sum of the weights is 1.

[0047] Based on this, the outputs of multiple attention heads are concatenated to obtain the multi-head self-attention layer output. See formula (7) below; (7); In formula (7), This represents the output of all attention heads. The multi-head self-attention output features are obtained by concatenating them along the feature dimension, where n is the number of attention heads, 1≤i≤n. The CONCAT function is a general-purpose function for concatenating multiple strings or arrays, used to perform concatenation operations along the feature dimension.

[0048] Step 33: [The text appears to be incomplete and contains several grammatical errors. A more accurate translation would require the The input is processed in the multimodal feature extraction module F to obtain multimodal temporal features. .

[0049] In specific implementation, step 33 includes a multimodal feature extraction module E comprising a position feedforward network and a layer normalization submodule LayerNorm.

[0050] The result obtained in step 32 The input location feedforward network, combined with residual connections and layer normalization (LayerNorm) operations, yields the final multimodal temporal features of the multimodal feature extraction module E. See formula (8) below; (8); In formula (8), FeedForward represents the location feedforward network (feedforward fully connected sub-network) of the multimodal feature extraction module E, which is usually composed of two fully connected layers and a nonlinear activation function; LayerNorm represents the layer normalization operation performed on the input features along the feature dimension. In this way, multimodal temporal features... EEG time series feature sequences and EMG time series feature sequences It can represent the temporal feature representations extracted from the EEG and EMG modalities respectively. Through step 3, long-range temporal modeling of EEG and EMG signals is realized in a unified feature space, providing high-quality input for subsequent multimodal feature fusion and adversarial modality alignment.

[0051] In practice, step 4 includes the following steps: Step 401: The multimodal feature fusion module F extracts features from the EEG time series feature sequence. Fusion features highly correlated with workload status were extracted from the EMG time series feature sequence Zm. ; The EEG time series feature sequence output in step 3 and EMG time series feature sequences The multimodal feature fusion module F is input separately. The multimodal feature fusion module F consists of four consecutive convolutional blocks, one max pooling layer, and one flattening layer; the four consecutive convolutional blocks are the first convolutional block B1, the second convolutional block B2, the third convolutional block B3, and the fourth convolutional block B3. (1) The first convolutional block in the four consecutive convolutional blocks contains a convolutional layer with a kernel size of 3, a batch normalization (BN) layer, and a ReLU activation function; (2) The conventional structure of the ConvNeXt network includes an initialization module, residual blocks, depthwise convolutions, downsampling, normalization, fully connected layers, and a global average pooling module. The initialization module processes the input features through large-size convolutions to extract coarse-grained spatial features; the residual blocks combine convolutions with residual connections to enhance feature extraction capabilities and prevent the gradient vanishing problem; the depthwise convolution module reduces computational cost while maintaining feature representation capabilities; the downsampling module reduces the resolution of the feature map and increases the number of channels to improve the receptive field of the model; the normalization layer accelerates training and improves stability; the fully connected layer is used for the final classification task; and the global average pooling module reduces the number of parameters and generates a global feature representation.

[0052] In this invention, the subsequent three blocks of the four consecutive convolutional blocks adopt the structure of the first three blocks of the conventional ConvNeXt network (i.e., the initial module, residual block, and depthwise convolution), which optimizes the performance and efficiency of traditional convolutional neural networks. Each convolutional block consists of a convolutional layer with a kernel size of 7, a layer normalization (LayerNorm) layer, and a Gelu activation function; a max-pooling layer with a 2×2 kernel size and a stride of 2 is set between the first and second blocks; finally, a flattening layer converts the three-dimensional representation into a one-dimensional feature vector, obtaining a one-dimensional representation that can be used for classification. Through this process, the multimodal feature fusion module F obtains... and Extract fusion features that are highly correlated with workload status. and These represent the temporal characteristics of the EEG and EMG modes, respectively.

[0053] Step 402: Modality Classifier Fusion features Modal discrimination and workload classifier The merged features are classified into three workload categories; Based on this, a modality classifier is used. The result after processing by the multimodal feature fusion module F Modality discrimination is performed to construct adversarial alignment constraints. The multimodal feature fusion module F and the modality classifier are also included. and workload classifier The output relationship is shown in the following formula (9); (9); In formula (9), This represents a modality classifier, used to determine which modality (EEG or EMG) the input feature belongs to. This represents a workload classifier used to classify the fused features into three workload categories. This indicates the modality classification result (prediction of modality ID). This represents the workload classification results based on the fusion features of EEG and EMG (predicted probability distributions of the three workload states). Indicates the fusion of features The input multimodal feature fusion module F is used to fuse the features into a single fused feature, including... and .

[0054] Step 403: Based on the modal classification results Define adversarial loss .

[0055] During the training phase, adversarial alignment of multimodal features is achieved. It is necessary to base the results on the modal classification. Define the adversarial loss as shown in the following formula (10); (10); In formula (10), This represents the adversarial loss function used for modal adversarial alignment. The true label for the j-th modality; Represents the number of modes, 1≤j≤ In this invention, take =2 (meaning it only includes the EEG and EMG modalities). During adversarial training, the modality classifier M minimizes... To distinguish different modalities as accurately as possible; while the multimodal feature fusion module F maximizes This is used to "confuse" the modality classifier M, making it difficult for the modality classifier M to distinguish the feature source modality, thereby prompting the EEG features and EMG features to align in the distribution within the fusion space.

[0056] To facilitate end-to-end optimization of the temporal relation modeling and multimodal adversarial alignment network (TRM-MAAN) in this invention, a gradient reversal operation is introduced between F and M during training. The gradient generated by the adversarial loss is inverted in sign when propagated to F, thereby achieving the aforementioned... , The goal of optimization is to counteract the opposition.

[0057] In step 4, the multimodal feature fusion module F and the modality classifier... Working collaboratively under an adversarial training mechanism, this approach extracts fusion features highly correlated with workload discrimination on one hand, and reduces the distributional differences between EEG and EMG features in the fusion space on the other, thus providing a basis for subsequent training based on... This lays the foundation for workload classification and overall loss function design.

[0058] In practice, step 5 includes the following steps: Step 501: Workload Classification Loss Calculation steps; The EEG fusion features output from step 4 Features of EMG Fusion The features are concatenated along the feature dimension and then input into the workload classifier. The predicted probability distributions of the three types of workload states are denoted as follows: ,in Each dimension corresponds to the probability of one of three types of workload: low workload, medium workload, and high workload.

[0059] During the training phase, according to The difference between the label and the actual workload is defined as the workload classification loss. In this invention, workload is categorized into losses. It is expressed in the form of cross-entropy, as shown in the following formula (11); (11); In formula (11), The classification loss function represents the classification of pilot workload. This represents the true label for the p-th workload state, where N is the number of workload state categories, 1 ≤ p ≤ N. In this invention, N = 3, corresponding to low workload, medium workload, and high workload.

[0060] Step 502: Calculation steps for the total loss function L; Combined with the adversarial loss defined in step 4 based on the modality classifier M By adding the two together to construct the overall loss function L, the joint optimization of workload classification performance and modal distribution alignment can be achieved, as shown in the following formula (12). (12).

[0061] In the specific training process, the learning rate is used. The entire temporal relationship modeling and multi-channel adversarial alignment network are trained end-to-end, and gradients with absolute values ​​greater than 10 are pruned to stabilize the training process.

[0062] Step 503: Calculation steps for workload prediction label c; During the testing phase, EEG test data is provided for testing. and EMG test data ,Will and Inputting the previously trained temporal relationship modeling and multi-channel adversarial alignment network yields the corresponding workload prediction probability distribution. .because Let c represent a probability distribution. The final workload prediction label c is determined by selecting the position with the highest probability, as shown in the following formula (13). (13); In formula (13), Let represent the predicted probability of the p-th workload state. This means taking the value on all class indexes p such that The index at which the maximum value is obtained.

[0063] Step 504: Accuracy The steps for calculating the average F1 score across multiple classes.

[0064] To quantitatively evaluate the classification performance of this method, accuracy (Acc) and the average F1 score of the multiple classes are used as the main performance indicators. The calculation formulas for Acc and F1 in this invention are shown in formulas (14) and (15) below. (14); (15); In formulas (14) and (15), TP (True Positive) represents the number of samples correctly classified as a positive class (target workload state), TN (True Negative) represents the number of samples correctly classified as not being positive, FP (False Positive) represents the number of samples incorrectly classified as being positive, and FN (False Negative) represents the number of samples incorrectly classified as not being positive. These indicators comprehensively evaluate the overall recognition accuracy and positive sample recognition ability of this method in pilot workload classification tasks.

[0065] In specific implementation, in step 6, the workload evaluation results of the temporal relationship modeling and multi-channel adversarial alignment network include three categories: low workload, medium workload, and high workload.

[0066] This invention focuses on multimodal EEG-EMG data collected from pilots during flight simulation missions. The specific process is as follows: First, according to Figure 3 The experimental design shown involves collecting EEG and forearm EMG signals from the pilots in a flight simulator, and labeling the data as three types of workload: normal, mild fatigue, and severe fatigue, based on the flight phase and NASA-TLX score. Then refer to Figure 1 The multimodal signal is filtered and denoised, segmented into 1-second non-overlapping segments, and augmented. After one-dimensional convolution and position encoding, it is input into the signal. Figure 2 The temporal relationship modeling and multi-channel adversarial alignment network proposed in this invention extracts temporal features through the Transformer multi-head self-attention and multi-modal feature fusion module, and obtains the aligned fused representation under the modal adversarial alignment mechanism. Finally, the workload classifier outputs three types of workload evaluation results.

[0067] As shown in Figure 2, the right side represents the training phase, and the left side represents the testing phase. (1) During training, the brain signals are... and electromyographic signals The fused features are obtained by sequentially inputting the one-dimensional temporal convolution module, the multimodal feature extraction module E, and the multimodal feature fusion module F. On the one hand, the data is fed into the workload state classifier C to calculate the classification loss, and on the other hand, it is fed into the modality classifier M through the gradient inversion layer to calculate the adversarial loss. The two are used together to optimize the network parameters. (2) The testing phase and the training phase share the same set of pre-trained parameters, but only the parameters from the training phase are retained. The forward path to classifier C is used to output the pilot workload assessment results. Figure 2 The "Num" annotation inside the multimodal feature extraction module E indicates the number of stacked layers of the Transformer encoder block, i.e., the number of times the multi-head self-attention + feedforward network structure is repeated. The workload state (N) is a three-class label (normal, mild fatigue, severe fatigue) used to supervise the workload classifier; the modality identifier (Nm) is a modality ID label (EEG or EMG) used to supervise the modality classifier M to distinguish different modalities during adversarial training, thereby promoting better alignment of the fused features in the modality dimension.

[0068] After training, the performance of the method of this invention is compared with that of the comparative methods MHyEEG and Husformer on the test set. Figure 4 As shown, the temporal relationship modeling and multi-channel adversarial alignment network in the method of this invention outperform the comparison model in terms of accuracy and F1 score. A confusion matrix was plotted for the three-class classification results of all subjects, as follows: Figure 5 As shown, all types of workloads achieved high recognition rates; At the same time, representative EEG topography maps under different workloads were selected, such as... Figure 6 As shown, the brain region activity patterns change systematically with increasing workload, and the experimental results verify the effectiveness and interpretability of the method of the present invention in assessing pilot workload.

[0069] The pilot workload assessment method of the present invention has the following technical features: (1) The Transformer-based temporal relationship modeling module is used to perform end-to-end feature extraction on the raw physiological signals of pilots' EEG and EMG. This can capture long-term temporal dependencies and high-dimensional complex features, reducing reliance on manual feature engineering and prior experience, thereby significantly improving the recognition accuracy of workload states. In the classification task of three workload states (low workload, medium workload, and high workload) of eight pilots, the average recognition accuracy of the method of this invention was 91.90%±1.72%, and the average F1 value was 91.86%±1.75%, with stable overall performance.

[0070] (2) An adversarial alignment-based multimodal feature fusion mechanism is introduced. Through adversarial training between the modality classifier and the multimodal feature fusion module, the difference in EEG and EMG feature distributions is effectively reduced, achieving alignment of the multimodal feature space and significantly improving the robustness and generalization ability of the model. Ablation experiments show that when the adversarial alignment mechanism is removed, the average classification accuracy decreases from 91.90% to 82.14%, a decrease of about 9.76%; when the Transformer is removed, the average classification accuracy decreases to 86.15%, a decrease of about 5.75%, verifying the necessity and effectiveness of the above key modules in improving the overall performance.

[0071] (3) While ensuring high-precision identification, the present invention takes into account the real-time and resource consumption requirements in practical applications, and is suitable for deployment in flight simulation training and flight mission in-loop monitoring systems. It achieves lower latency and smaller computational overhead while ensuring higher classification accuracy.

[0072] (4) It has good robustness to individual differences and experimental interference factors. The results of eight pilot subjects showed that the classification accuracy of the method of the present invention was more than 89% on all subjects, with a mean accuracy standard deviation of 1.72%, which was significantly lower than some comparative methods. Moreover, the confusion matrix showed that the classification accuracy of most subjects exceeded 80% in the three workload states. It can still maintain stable and reliable recognition performance under the conditions of electrode contact fluctuation, sweating and other interference, as well as individual physiological differences.

[0073] (5) The workload assessment framework based on EEG and EMG multimodal physiological signals of this invention can objectively and continuously reflect the cognitive load and physiological state of pilots, providing technical support for early identification of overload or underload risks and reducing flight accidents caused by human factors. At the same time, this assessment framework can be used as part of a clinical monitoring system and can be widely applied to scenarios such as fatigue monitoring, mental state assessment and monitoring of cognitive psychological disorders, which has good engineering application value and clinical translation prospects.

[0074] This invention discloses a method for assessing pilot workload, comprising: a data acquisition and preprocessing module that simultaneously acquires, filters, denoises, and segments EEG and forearm EMG signals from flight simulation tasks, and labels each time segment with three workload categories: low workload, moderate workload, and high workload; a data augmentation module that randomly flips and adds noise to the multimodal segments to expand the training samples; a one-dimensional convolution module that maps EEG and EMG to a unified feature dimension; a positional encoding and Transformer multi-head self-attention module that jointly models the temporal dependencies of multimodal signals to obtain temporal features containing long-range information; a multimodal feature fusion module that jointly models the two modal features; a modality adversarial alignment module that aligns the distribution of EEG and EMG in the feature space using a modality classifier and gradient inversion mechanism; and finally, a workload classification module that outputs three types of workload assessment results. Compared with existing methods that rely solely on a single physiological signal or simple feature splicing, this invention can simultaneously utilize complementary spatiotemporal information from EEG and EMG, alleviating overfitting problems caused by cross-modal distribution differences and limited sample size, significantly improving the accuracy and robustness of pilot workload assessment, and can be applied to scenarios such as flight training safety monitoring, pilot fatigue management, and aviation human factors engineering.

[0075] This invention utilizes EEG and EMG signals collected during flight simulation missions for joint modeling to achieve automatic discrimination of pilots' three workload states: low, medium, and high. This overcomes the problem of insufficient recognition accuracy and robustness of traditional single-modal or manual feature methods in complex flight mission scenarios.

[0076] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

[0077] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. A method for assessing pilot workload, characterized in that, Includes the following steps: Step 1: Data Acquisition and Preprocessing; Acquiring the pilot's raw EEG signals X e and the original electromyographic signal X m and the original EEG signal X e and the original electromyographic signal X m Preprocessing was performed to obtain the processed EEG signal time segment X. e,t and electromyographic signal time fragment X m,t ; Step 2: Data augmentation step; X-ray of EEG signal time segment e,t and electromyographic signal time fragment X m,t Data augmentation was performed, and then the augmented EEG signal time segment was X-rayed. e,t and electromyographic signal time fragment X m,t Mapped to a unified feature dimension D; Step 3: In the time-series feature sequence Positional encoding is added to the data, and then multimodal temporal features Z are obtained through a multimodal feature extraction module. {e,m} ; Step 4: Process the multimodal temporal features Z using the multimodal feature fusion module F. {e,m} EEG time series feature sequence Z e and EMG time series feature sequence Z m Processing is performed to align the distributions of the EEG feature space and the EMG feature space; Step 5: Constructing the total loss function L; classifying the loss L based on workload. cls and adversarial loss function L adv To determine the total loss function L; Step 6: Transfer the EEG signal X e and electromyographic signal X m Input temporal relationship modeling and multi-channel adversarial alignment network, output pilot workload assessment results.

2. The pilot workload assessment method according to claim 1, characterized in that, In step 1, the preprocessing includes bandpass filtering, power frequency interference suppression, and removal of obvious artifacts.

3. The pilot workload assessment method according to claim 2, characterized in that, In step 1, after preprocessing, the original EEG signal X is processed for a fixed duration. e and the original electromyographic signal X m Segmentation and time alignment are performed to obtain time segment X of the EEG signal. e,t and electromyographic signal time fragment X m,t .

4. The pilot workload assessment method according to claim 1, characterized in that, In step 2, data augmentation is performed using random flipping and additive Gaussian noise.

5. The pilot workload assessment method according to claim 1, characterized in that, In step 2, one-dimensional temporal convolution is used to convert the EEG signal time segment X. e,t Feature dimensions and electromyographic signal time segments X m,t The feature dimension is unified to D-dimensional.

6. The pilot workload assessment method according to claim 1, characterized in that, Step 3 includes the following steps: Step 31: In the time-series feature sequence By adding positional encoding, a feature sequence O with positional encoding is obtained. {e,m} ; Step 32: Extract the feature sequence O with positional encoding. {e,m} In the input multi-head self-attention structure, the multi-head self-attention output features are obtained. ; Step 33: [The text appears to be incomplete and contains several grammatical errors. A more accurate translation would require the The multimodal temporal features Z are obtained by processing the input into the multimodal feature extraction module F. {e,m} .

7. The pilot workload assessment method according to claim 6, characterized in that, In step 33, the multimodal feature extraction module E includes a position feedforward network and a layer normalization submodule LayerNorm.

8. The pilot workload assessment method according to claim 1, characterized in that, Step 4 includes the following steps: Step 401: The multimodal feature fusion module F extracts data from the EEG time-series feature sequence Z. e and EMG time series feature sequence Z m Extract the fusion feature F(Z) that is highly correlated with the workload state. {e,m} ); Step 402: Modality Classifier M cls For the fusion feature F(Z) {e,m} Modal discrimination is performed using workload classifier C. cls The merged features are classified into three workload categories; Step 403: Based on the modal classification results Define adversarial loss L adv .

9. The pilot workload assessment method according to claim 1, characterized in that, Step 5 includes the following steps: Step 501: Workload Classification Loss L cls Calculation steps; Step 502: Calculation steps for the total loss function L; Step 503: Calculation steps for workload prediction label c; Step 504: Calculation steps for accuracy (Acc) and average F1 score (F1) across multiple classes.

10. A method for assessing pilot workload according to claim 1, characterized in that, In step 6, the workload evaluation results of the temporal relationship modeling and multi-channel adversarial alignment network include three categories: low workload, medium workload, and high workload.

Images