Method for recognizing workload under electroencephalogram mode

Abstract

This invention discloses a method for workload identification in the absence of EEG modalities, belonging to the field of physiological signal recognition. The invention first performs simple preprocessing on the original multimodal data containing EEG signals. Then, it extracts time-domain and frequency-domain features from different physiological signal modalities, making the features more comprehensive. Next, to enable the acquisition of EEG features during inference, an EEG feature generation module is established, which increases the mutual information between EEG features and other features to reduce the generation difficulty, thus allowing the model to possess rich features even when EEG modalities are missing. To reduce redundant features in the fused features, an autoencoder is used to reduce the dimensionality of the features. Addressing the difficulty of acquiring EEG signals, this invention helps deploy workload identification models developed based on EEG in practical applications, ensuring the practical application of workload identification models.

Description

Technical Field

[0001] This invention relates to the field of physiological signal recognition, and more particularly to a method for identifying workload in the absence of EEG modalities. Background Technology

[0002] Workload (also known as cognitive load or mental workload) refers to the utilization rate of mental resources by the human body during work. Different levels of workload will result in different cognitive abilities. Excessive workload will reduce overall work efficiency, increase the possibility of errors, and may even lead to serious accidents such as sudden death. Insufficient workload may lead to low work efficiency, lack of concentration, and reduced alertness. Therefore, in complex and safety-critical environments such as civil aviation and aerospace, it is crucial to effectively identify operator workload and provide timely feedback. This can, to some extent, prevent operational errors caused by excessive workload for personnel in special professions, and is of great significance to ensuring the normal operation of society and the safety of people's lives and property.

[0003] In recent years, with the development of wearable sensors, more and more dimensional (modal) physiological data can be used for workload identification. Using multi-dimensional (modal) sensor data for workload identification has many advantages: (1) The information between multiple dimensions of sensor data can complement each other. For example, electroencephalogram (EEG) signals can characterize brain activity, and electrocardiogram (ECG) signals can characterize some physiological activities of the heart. Fusing these two signals can cover a more comprehensive range of physiological information of the subject; (2) Using multi-modal signals to identify workload is more robust to noise. If the data of one dimension of sensor data is contaminated by noise, data from other dimensions can still be used. Therefore, some existing workload identification methods have begun to try to use multi-modal data to identify workload.

[0004] However, due to convenience and other reasons, EEG is difficult to acquire in real-life conditions. EEG acquisition devices are divided into dry electrodes and wet electrodes, each with its own problems in practical use. Dry electrodes are in direct contact with the scalp and do not require additional electrolytes, but they are susceptible to motion artifacts in real-world situations, and prolonged wear can cause headaches. Wet electrodes require the injection of electrolytes between the scalp and the sensor, resulting in longer wearing times and inconvenience in practical use. Worse still, both dry and wet electrodes require the placement of an amplifier (which is often bulky due to the weak nature of EEG signals), further hindering the practical application of EEG. Therefore, the challenge in workload recognition under real-world conditions is that physiological signals other than EEG are insufficient to characterize the subject's workload, and EEG is difficult to deploy in everyday use scenarios. Thus, the lack of EEG modalities during reasoning is a problem that urgently needs to be addressed.

[0005] The simplest way to solve the above problems is to use late fusion, where a separate network is used for feature extraction between each modality, and the predictions from models without missing modalities are fused. However, this approach does not utilize any model information from missing data and often fails to achieve good results. Building on the basic idea of ​​late fusion, some researchers have attempted to further improve the utilization of multimodal data in the training set through multitasking. Another approach attempts to transfer knowledge from the teacher model (which takes a complete multimodal model as input) to the student model (which takes a model missing one modality as input), based on the idea of ​​knowledge distillation, so that the student model can achieve better results without having to input all modalities. Finally, there is a type of algorithm based on generative models. This type of method completes the missing modalities by translating other available modalities into unavailable ones, thus achieving better results during testing. Existing generative models often directly generate missing EEG features, which is often difficult to optimize, resulting in poor results. Summary of the Invention

[0006] This invention addresses the shortcomings of existing modality-missing algorithms based on generative models by proposing a workload identification method for missing EEG modalities. This method optimizes the mutual information between features of other modalities and EEG modal features, reduces the redundancy of fused features, and uses an autoencoder to retain key features of the fused features, thereby further improving the identification accuracy of the workload identification model.

[0007] The technical solution adopted in this invention is as follows:

[0008] A method for workload identification in the absence of EEG modalities, the method comprising:

[0009] Step 1: Set up the workload identification model, which includes a physiological signal feature extraction network, an EEG feature generation module, a feature fusion module, and a classifier;

[0010] Among them, the physiological signal feature extraction network is used to extract signal features of physiological signals of different modalities;

[0011] The input to the EEG feature generation module is the signal features of physiological signals that are not EEG signals. It acquires the interaction information features between different modalities, and then fuses all the interaction information features to obtain its output features, namely the generated EEG signal features.

[0012] The feature fusion module is used to perform redundancy-free feature fusion on the signal features of non-EEG signals output by the physiological signal feature extraction network and the generated EEG signal features to obtain multimodal redundancy-free features for input to the classifier;

[0013] The classifier is used to output the classification probability of the workload category;

[0014] Step 2: Train the workload identification model based on the collected training dataset to obtain a trained workload identification model;

[0015] During training, the total loss function is a weighted sum of the loss functions of the EEG feature generation module, the feature fusion module, and the classification loss; among which, the loss function of the EEG feature generation module includes reconstruction loss and interaction information loss.

[0016] Furthermore, the physiological signal feature extraction network extracts signal features from different modalities of physiological signals, specifically including:

[0017] The physiological signal of the sample is segmented based on a preset time window size, and the physiological signal is divided into several signal segments. Each segment is input into a convolutional neural network for dimensionality reduction to obtain the encoded representation of each segment. Then, the physiological signal encoding composed of the encoded representations of all segments is input into a Transformer encoder for feature extraction to obtain the temporal features of the sample.

[0018] Frequency domain features of the sample are extracted based on the attention mechanism.

[0019] By splicing and fusing time-domain and frequency-domain features, signal features of physiological signals of different modalities of the sample are obtained.

[0020] Furthermore, the EEG feature generation module includes: a modality feature conversion module and a Transformer encoder;

[0021] The modality feature transformation module is used to transform the signal features of each input modality of physiological signal, and then input the transformed features into the Transformer encoder;

[0022] The Transformer encoder is used to calculate the interactive information features between signal features of any two modes;

[0023] The output of the EEG feature generation module is obtained by summing all the interactive information features output by the Transformer encoder.

[0024] The modality feature transformation module includes a cascaded first linear layer, a ReLU activation function layer, a batch normalization layer, and a second linear layer.

[0025] Furthermore, the interaction information loss of the EEG feature generation module is the sum of the opposite numbers of the mutual information between modalities; the reconstruction loss of the EEG feature generation module is the sum of the squares of the differences between the EEG signal features extracted by the physiological signal feature extraction network of the sample and the generated EEG signal features.

[0026] Furthermore, the feature fusion module includes an encoder and a decoder;

[0027] The encoder's input includes: signal features of each physiological signal input to the EEG feature generation module and generated EEG signal features output by the EEG feature generation module, i.e., multimodal signal features f. i Where, subscript i is the sample number;

[0028] After concatenating the multimodal signal features from the input encoder, the low-dimensional feature h after redundancy removal is obtained through an autoencoder. i ;

[0029] Use a decoder on low-dimensional features h i Reconstruction is performed to restore the multimodal signal characteristics of the input encoder.

[0030] According to the formula Calculate the loss function of the feature fusion module, where N represents the number of samples.

[0031] The technical solution provided by this invention brings at least the following beneficial effects:

[0032] This invention first performs simple preprocessing on the raw multimodal (including EEG) data, then extracts time-domain and frequency-domain features from different physiological signal modalities, making the features more comprehensive. To enable the acquisition of EEG features during inference, this invention designs an EEG feature generation module, which reduces the generation difficulty by increasing the mutual information between EEG features and other features, thus allowing the model to have rich features even when EEG modalities are missing. To reduce redundant features in the fused features, an autoencoder is used to reduce the dimensionality of the features. Due to the inconvenience of EEG devices, this invention can help deploy workload recognition models developed based on EEG in practical applications, ensuring the practical application of workload recognition models. Attached Figure Description

[0033] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0034] Figure 1 A schematic diagram illustrating the implementation process of the workload identification method under missing EEG modalities provided in this embodiment of the invention;

[0035] Figure 2 This is a flowchart illustrating the EEG generation module in an embodiment of the present invention.

[0036] Figure 3 This is a schematic diagram of the AutoEncoder structure in the feature fusion module of this invention. Detailed Implementation

[0037] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be described in detail and completely below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Generally, the components of the embodiments of the present invention described and shown in the accompanying drawings can be arranged and designed using different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of the present invention.

[0038] The workload identification method under missing EEG modalities provided in this invention is for workload identification scenarios under missing EEG conditions. See [link to relevant documentation]. Figure 1The main workload identification model includes: a (physiological signal) feature extraction network, an EEG feature generation module, a feature fusion module, and a classifier; its implementation process specifically includes:

[0039] Step S1: Preprocess EEG data and other physiological signals; The main types of noise in EEG include: (1) baseline drift (2) power frequency interference (3) electromyographic interference. Baseline drift is a type of noise with a frequency lower than 1 Hz caused by the relative displacement between the electrodes of the acquisition device and the skin due to the human body's own breathing, movement, or acquisition device. Therefore, a high-pass filter based on 1 Hz can basically remove baseline drift noise. Power frequency interference mainly comes from interference from other electrical appliances, with the main frequency concentrated around 50 Hz, so it can be removed by a 48 Hz-52 Hz concave filter. Electromyographic interference is often concentrated at higher frequencies, while the main frequencies of EEG are basically less than 40 Hz, so a 40 Hz low-pass filter is used to eliminate high-frequency interference such as electromyographic interference. In order to facilitate model training and performance evaluation, this embodiment of the invention segments the EEG signal with a 1-second time window, and the time windows of two samples on the time axis do not overlap. Then, the entire dataset is normalized.

[0040] The Transformer class has demonstrated significant advantages in processing time-series data. However, directly inputting physiological signals into the Transformer Encoder for feature extraction can lead to overfitting. To address this issue, a series of processing steps are required for the physiological signals. The input physiological signals are divided into multiple time windows, each of which undergoes dimensionality reduction via a convolutional neural network. Finally, the output encodes the signal within that window. This processing reduces the number of parameters in the overall model, thereby reducing overfitting.

[0041] Let the size of the physiological signal segmentation window (time window) be W, and the step size be S. This time window can divide a physiological signal into several segments. Let K represent the number of segments. Then, the segmented physiological signal can be represented as:

[0042]

[0043] Among them, S i This represents the segmentation result of sample i, that is, the set of segmented physiological signals. Let j represent the j-th segment, where j = 1, 2, ..., K.

[0044] Then each segment Input to coding layer E m This yields the encoded representation of each segment: Thus, the encoded representation based on K segments Obtain the encoding of this physiological signal

[0045] The encoded physiological signal is then input into the Transformer Encoder for feature extraction, as shown in the formula:

[0046]

[0047] Where En(·) is the Transformer Encoder. The obtained time-domain features.

[0048] The entire feature extraction process of a time-domain signal can be expressed by the following formula:

[0049]

[0050] in, This represents the entire time-domain feature extraction process, including the division of physiological signal time windows and feature extraction based on En and En.

[0051] Different physiological signals have different dominant frequencies. For example, the dominant frequency of an electrocardiogram (ECG) signal is 0.5-45 Hz, while that of an electroencephalogram (EEG) signal is 0.5-80 Hz. Therefore, noise outside the dominant frequency should be discarded, and the model should focus more on the dominant frequency. Furthermore, the importance of the dominant frequencies of physiological signals is not the same. For example, workload requires more attention to the alpha band (8-12 Hz) and theta band (4-8 Hz) of an EEG signal. Based on this, this embodiment of the invention uses an attention mechanism to make the model focus on those frequency bands that are more important to the workload, thereby improving the generalization ability of the network model. The entire feature extraction process of the frequency domain signal is represented by the following formula:

[0052]

[0053] in This represents a frequency domain feature extractor. This represents the extracted frequency domain features.

[0054] The time-domain feature extractor and the frequency-domain feature extractor constitute the time-frequency feature extractor (i.e., the feature extraction network in this embodiment). After extracting the time-domain and frequency-domain features, the two types of features are fused using a concatenation method, as shown in the following formula:

[0055]

[0056] Step S2: Train the model. During the training process, EEG is used to enrich the feature information.

[0057] In this embodiment of the invention, the goal of the EEG feature generation module is to train a deep neural network capable of generating EEG features. Here, EEG features refer to the features obtained after inputting EEG data into the feature extraction network. This allows the subsequent feature fusion module to obtain EEG features, providing a more comprehensive representation of the workload and improving overall model performance. See also... Figure 2 In this embodiment of the invention, the processing procedure of the EEG feature generation module is as follows:

[0058] Assume the EEG feature generation module is represented as The input and output of the EEG feature generation module can then be expressed by the following formula:

[0059]

[0060] in, f represents the EEG features generated by the EEG feature generation module. ecg f ppg Features representing other physiological signal modalities (obtained by a feature extraction network). The specific physiological signals used depend on the dataset. For example, the MOCAS dataset includes two physiological signals besides EEG: pulsed blood volume (PPG) and gastric skin signal (GSR); while the WAUC dataset includes four physiological signals besides EEG: electrocardiogram (ECG), pulsed blood volume (PPG), gastric skin signal (GSR), and respiratory rate (BR). In this embodiment of the invention, other physiological signals include: electrocardiogram signal f ecg Blood volume pulse signal f ppg and skin electrical signals f gsr .

[0061] Given the difficulty of directly generating corresponding EEG features from physiological signal features of other modalities, this invention increases the mutual information between EEG features and other physiological signal features to reduce the learning difficulty of the model. Mutual information between two features represents the degree of similarity between them. The greater the mutual information between the features corresponding to each modality and the features of the EEG signal, the higher the similarity of the features, thus making subsequent EEG signal feature generation easier. Simultaneously increasing the mutual information between other modal physiological signals and EEG signals can also align the features learned by the two feature extraction networks, further improving the network's generalization performance.

[0062] The formula for mutual information is as follows:

[0063]

[0064] Among them, I x,yLet p(x) represent the mutual information between random variables X = x and Y = y, p(x) represent the probability of X = x occurring, p(y) represent the probability of Y = y occurring, and p(x,y) represent the probability of both X = x and Y = y occurring simultaneously. Therefore, this embodiment of the invention uses mutual information loss, as shown in the following formula:

[0065]

[0066] Among them, I eeg,ecg f eeg and f ecg Mutual information between them, I eeg,ppg f eeg and f ppg Mutual information between them, I eeg,gsr f eeg and f gsr Mutual information between the models. By using mutual information loss, the generalization performance of the entire model can be effectively improved, and it can also help in the subsequent generation of EEG features.

[0067] For each modality of data, further feature transformation is performed to obtain the corresponding transformed features. The feature transformation network contains four layers: (1) Linear: Linear layer (2) ReLU layer (3) Batch Normalization (BN): Batch Normalization layer (4) Linear: Linear layer. Then, the transformed physiological signal features other than EEG modalities are input into the Transformer Encoder. Information interaction is performed on these features, and finally, multiple outputs are added together to obtain the final output. That is, the characteristics of the generated electroencephalogram (EEG) signals. Using reconstruction loss... The entire network is supervised to ensure that the final generated features are as similar as possible to those obtained by the EEG feature extraction network. The reconstruction loss is expressed by the following formula:

[0068]

[0069] Therefore, the loss function of the entire EEG feature generation module can be expressed as:

[0070]

[0071] During inference, the EEG feature generation module can generate missing EEG signal features. This module uses two losses to ensure that the generated features are similar to the actual features. Mutual information loss effectively reduces the difficulty of the entire learning process, while reconstruction loss provides supervision for the entire learning process.

[0072] The feature fusion module's role is to fuse multiple physiological signal features. Since the EEG feature generation module increases the mutual information between EEG features and other physiological signal features, these other features and EEG features may contain redundant information. Therefore, one goal of the feature fusion module is to reduce the overall redundancy of the model. The Autoencoder can extract key information from redundant inputs and ignore redundant features; see [link to Autoencoder]. Figure 3 It includes an encoder and a decoder, through which the encoder converts redundant input ( The encoder maps the signal features of both non-EEG signals to low-dimensional features. The decoder then restores the low-dimensional features output by the encoder back to the encoder's initial input, ensuring that the information contained in the low-dimensional features is complete. Therefore, this embodiment of the invention uses an AutoEncoder to remove redundancy from the input multimodal physiological signal features.

[0073] Assumption For Encoder, If we are a decoder, then the encoder mapping redundant input to low-dimensional features can be represented as:

[0074] h = F e (cat(f eeg f ecg ,f ppg ,f gsr ,...))

[0075] Where h represents the low-dimensional feature after redundancy removal, and cat(·) represents concatenating multiple features.

[0076] The reconstruction of features using a decoder can be expressed by the following formula:

[0077]

[0078] in, This represents the low-dimensional features of different modalities after redundancy removal.

[0079] Loss during training of AutoEncoder It can be represented as:

[0080]

[0081] Where N represents the number of modes of physiological signal features, f i This represents the characteristic of the i-th physiological signal. This represents the physiological signal feature reconstructed by the decoder for the i-th type.

[0082] In summary, the overall loss of the workload identification model provided in this embodiment of the invention during training can be expressed by the formula:

[0083]

[0084] It can also be set to The weighted sum of the losses is represented by the weight coefficients of each loss, which are hyperparameters.

[0085] Step S3: Since EEG signals are unavailable during inference, this differs somewhat from the forward propagation process in training. EEG features generated using the EEG feature generation module are used. EEG characteristics during alternative training f eeg Complete the subsequent reasoning. Since the EEG generation module has been trained well during the training process, the EEG features generated during reasoning are... It is relatively realistic and can achieve good results.

[0086] It should be noted that the type of workload identification results can be set based on actual application needs, and then the corresponding dataset can be selected. For example, the STEW dataset divides workload levels into two categories, while the CONTEST dataset will stimulate three different levels of workload in the subjects.

[0087] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

[0088] The above descriptions are merely some embodiments of the present invention. Those skilled in the art can make various modifications and improvements without departing from the inventive concept of the present invention, and these all fall within the scope of protection of the present invention.

Claims

1. A method for workload identification under missing EEG modalities, characterized in that, Includes the following steps: Step 1: Set up the workload identification model, which includes a physiological signal feature extraction network, an EEG feature generation module, a feature fusion module, and a classifier; Among them, the physiological signal feature extraction network is used to extract signal features of physiological signals of different modalities; The input to the EEG feature generation module is the signal features of physiological signals that are not EEG signals. It acquires the interaction information features between different modalities, and then fuses all the interaction information features to obtain its output features, namely the generated EEG signal features. The feature fusion module is used to perform redundancy-free feature fusion on the signal features of non-EEG signals output by the physiological signal feature extraction network and the generated EEG signal features to obtain multimodal redundancy-free features for input to the classifier; The classifier is used to output the classification probability of the workload category; Step 2: Train the workload identification model based on the collected training dataset to obtain a trained workload identification model; During training, the total loss function is a weighted sum of the loss functions of the EEG feature generation module, the feature fusion module, and the classification loss; the loss function of the EEG feature generation module includes reconstruction loss and interaction information loss; the interaction information loss of the EEG feature generation module is the sum of the negative numbers of the mutual information between the signal features of EEG signals and non-EEG signals. The EEG feature generation module includes: a modality feature conversion module and a Transformer encoder; The modality feature transformation module is used to transform the signal features of each input modality of physiological signal, and then input the transformed features into the Transformer encoder; The Transformer encoder is used to calculate the interactive information features between signal features of any two modes; The output of the EEG feature generation module is obtained by summing all the interactive information features output by the Transformer encoder. The modality feature transformation module includes a cascaded first linear layer, a ReLU activation function layer, a batch normalization layer, and a second linear layer.

2. The method as described in claim 1, characterized in that, Physiological signal feature extraction networks extract signal features from physiological signals of different modalities, specifically including: The physiological signal of the sample is segmented based on a preset time window size, and the physiological signal is divided into several signal segments. Each segment is input into a convolutional neural network for dimensionality reduction to obtain the encoded representation of each segment. Then, the physiological signal encoding composed of the encoded representations of all segments is input into a Transformer encoder for feature extraction to obtain the temporal features of the sample. Frequency domain features of the sample are extracted based on the attention mechanism. By splicing and fusing time-domain and frequency-domain features, signal features of physiological signals of different modalities of the sample are obtained.

3. The method as described in claim 2, characterized in that, The time window is 1 second long.

4. The method as described in claim 3, characterized in that, The reconstruction loss of the EEG feature generation module is the sum of squares of the differences between the EEG signal features extracted by the physiological signal feature extraction network of the sample and the generated EEG signal features.

5. The method as described in claim 1, characterized in that, The feature fusion module includes an encoder and a decoder; The encoder's input includes: signal features of each physiological signal input to the EEG feature generation module and generated EEG signal features output by the EEG feature generation module, i.e., multimodal signal features. Where, subscript i is the sample number; After concatenating the multimodal signal features from the input encoder, the signals are then passed through an autoencoder to obtain the redundancy-free, low-dimensional features. ; Use a decoder for low-dimensional features Reconstruction is performed to restore the multimodal signal characteristics of the input encoder. ; According to the formula Calculate the loss function of the feature fusion module, where N represents the number of samples.

6. The method as described in claim 1, characterized in that, Physiological signals other than electroencephalogram (EEG) signals include: electrocardiogram (ECG) signals. Blood volume pulse signal and skin electrical signals .

Images