Workload identification method in absence of electroencephalogram mode
By setting up a physiological signal feature extraction network and EEG feature generation module in workload recognition, combined with the feature fusion of the autoencoder, the problem of difficult to obtain EEG mode is solved, the recognition accuracy and application of the model are improved, and the Identification of effective workloads in complex environments.
Patent Information
- Application Number
- CN202410944261.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-15
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2044-07-15
AI Technical Summary
When identifying workloads, EEG modes are difficult to obtain and deploy, especially in complex environments, resulting in poor recognition of workloads under missing EEG modes, and difficult to optimize existing generative models, resulting in recognition accuracy. Lower.
Using the workload recognition method under the missing EEG mode, by setting up physiological signal feature extraction network, EEG feature generation module, feature fusion module and classifier, the autoencoder reduces redundant features and increases EEG and other modes The mutual information between features improves recognition accuracy.
It realizes the recognition accuracy of the workload recognition model under the missing EEG mode, can be deployed in practical applications, ensure the implementation of the workload recognition model, reduces the redundancy of features, and improves the generalization performance of the model. .
Smart Images

Figure CN118924296A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of physiological signal recognition, and in particular to a method for identifying workload in the absence of an electroencephalogram (EEG) modality. Background Art
[0002] Workload (also called cognitive load or mental load) refers to the occupancy rate of the human brain resources when the human body is working. Different levels of workload on the human body will cause it to exhibit different cognitive abilities. Too high a workload will reduce overall work efficiency, increase the possibility of errors, and may even lead to major accidents such as sudden death. Too low a workload may lead to problems such as low work efficiency, lack of concentration, and reduced alertness of staff. Therefore, it is crucial to effectively identify the workload of operators and provide timely feedback in complex environments with high safety requirements such as civil aviation flights and aerospace. This can avoid operational errors caused by excessive workload for personnel engaged in special occupations to a certain extent, which is of great significance to ensuring the normal operation of society and the safety of life and property of the people.
[0003] With the development of wearable sensors in recent years, more and more dimensional (modal) physiological data can be used for workload identification. Using multi-dimensional (modal) sensor data for workload identification has many benefits: (1) The information between sensor data of multiple dimensions can complement each other. For example, electroencephalogram (EEG) signals can represent brain activities, and electrocardiogram (ECG) signals can represent some physiological activities of the heart. The fusion of these two signals can cover a more comprehensive range of physiological information of the subject; (2) Using multi-modal signals to identify workload is more robust to noise. After the data of one dimension of sensor data is contaminated by noise, data of other dimensions can still be used. Therefore, some existing workload identification methods have begun to try to use multi-modal data to identify workload.
[0004] However, due to reasons such as convenience, EEG is difficult to obtain under real-life conditions. EEG acquisition equipment is divided into dry electrodes and wet electrodes, and both dry electrodes and wet electrodes have their own problems in actual use. Dry electrodes are in direct contact with the scalp and do not require other electrolytes, but in actual situations, they are easily interfered by motion artifacts, and the head is prone to pain after wearing for a long time. Wet electrodes require electrolytes to be injected between the scalp and the sensor, which results in a long time of wearing wet electrode equipment and is not convenient for actual use. What's worse is that both dry electrodes and wet electrodes require amplifiers to be placed aside (because the EEG signal is relatively weak, the amplifier is often large in size), which is also a factor that hinders the actual implementation of EEG. Therefore, the problem faced by workload identification under actual conditions is that it is difficult to characterize the workload of the subject with physiological signals other than EEG, and EEG is difficult to deploy in daily use scenarios. Therefore, the problem of missing EEG modalities during reasoning is an urgent problem to be solved.
[0005] The simplest way to solve the above problem is to use the late fusion method, using a network for feature extraction between each modality, and fusing the prediction results of the model without missing modalities. However, this approach does not utilize any model information of missing data and often cannot achieve good results. Based on the basic idea of late fusion, some researchers try to further improve the utilization of multimodal data in the training set through multi-task. Another method attempts to transfer the knowledge in the teacher model (the input is a complete multimodal model) to the student model (the input is a model with a missing modality) based on the idea of knowledge distillation, so that the student model does not have to input all modalities to obtain better results. Finally, there is another type of algorithm based on the generative model. This type of method translates other test-available modalities into unavailable modalities and completes the missing modalities, so that better results can be obtained during testing. Existing generative models often directly generate missing EEG features, which are often difficult to optimize, resulting in poor results. Summary of the invention
[0006] In view of the shortcomings of the existing modality missing algorithm based on generative model, the present invention proposes a workload identification method under the missing EEG modality, so as to optimize the mutual information between the features of other modalities and the EEG modality features, reduce the redundancy of fused features, use autoencoder to retain the key features of the fused features, and further improve the recognition accuracy of the workload identification model.
[0007] The technical solution adopted by the present invention is:
[0008] A method for identifying workload in the absence of EEG modality, the method comprising:
[0009] Step 1, setting a workload identification model, which includes a physiological signal feature extraction network, an EEG feature generation module, a feature fusion module and a classifier;
[0010] Among them, the physiological signal feature extraction network is used for signal features of physiological signals of different modalities;
[0011] The input of the EEG feature generation module is the signal features of the physiological signals other than EEG signals, and the interactive information features between different modalities are obtained. Then, all the interactive information features are integrated to obtain the output features, i.e., the generated EEG signal features.
[0012] The feature fusion module is used to perform de-redundant feature fusion on the signal features of the non-EEG signal output by the physiological signal feature extraction network and the generated EEG signal features, so as to obtain multi-modal de-redundant features for input into the classifier;
[0013] The classifier is used to output the classification probability of the workload category;
[0014] Step 2: training the workload recognition model based on the collected training data set to obtain a trained workload recognition model;
[0015] During training, the total loss function is the weighted sum of the loss function of the EEG feature generation module, the loss function of the feature fusion module and the classification loss; among them, the loss function of the EEG feature generation module includes reconstruction loss and mutual information loss.
[0016] Furthermore, the physiological signal feature extraction network extracts signal features of physiological signals of different modalities, specifically including:
[0017] The physiological signal of the sample is cut and processed based on the preset time window size, and the physiological signal is cut into several signal segments; each segment is input into the convolutional neural network for dimension reduction to obtain the encoding representation of each segment; the physiological signal encoding composed of the encoding representation of all segments is then input into the Transformer encoder for feature extraction to obtain the time domain features of the sample;
[0018] Extract frequency domain features of the physiological signals of the samples based on the attention mechanism to obtain the frequency domain features of the samples;
[0019] The time domain features and frequency domain features are spliced and fused to obtain the signal characteristics of physiological signals of different modalities of the samples.
[0020] Furthermore, the EEG feature generation module includes: a modality feature conversion module and a Transformer encoder;
[0021] The modality feature conversion module is used to transform the signal features of each modality of the input physiological signal, and then input the transformed features into the Transformer encoder;
[0022] The Transformer encoder is used to calculate the mutual information features between the signal features of any two modes;
[0023] Add all the interactive information features output by the Transformer encoder to get the output of the EEG feature generation module;
[0024] Among them, the modal feature conversion module includes a cascaded first linear layer, a ReLU activation function layer, a batch normalization layer and a second linear layer.
[0025] Furthermore, the mutual information loss of the EEG feature generation module is: the sum of the inverses of the mutual information between the modalities; the reconstruction loss of the EEG feature generation module is: the sum of the squares of the differences between the EEG signal features extracted by the sample's physiological signal feature extraction network and the generated EEG signal features.
[0026] Furthermore, the feature fusion module includes: an encoder and a decoder;
[0027] The input of the encoder includes: the signal features of each physiological signal input to the EEG feature generation module and the EEG signal features generated by the EEG feature generation module, that is, the multi-modal signal features f i ; Wherein, subscript i is the sample number;
[0028] After concatenating the multi-modal signal features of the input encoder, the redundant low-dimensional features h are obtained after passing through the autoencoder. i ;
[0029] Use the decoder to decode the low-dimensional features h i Reconstruction is performed to reconstruct and restore the multi-modal signal characteristics of the input encoder
[0030] According to the formula Calculate the loss function of the feature fusion module, where N represents the number of samples.
[0031] The technical solution provided by the present invention brings at least the following beneficial effects:
[0032] The present invention first performs simple preprocessing on the original multimodal (including EEG) data, and then uses the extraction of time domain features and frequency domain features of different physiological signal modalities to make the features contained in the features more comprehensive; in order to obtain EEG features in the reasoning process, the present invention designs an EEG feature generation module, which makes the generation easier by increasing the mutual information between EEG features and other features, so that the model can have rich features when the EEG modality is missing; in order to reduce the redundant features in the fusion features, the autoencoder is used to reduce the dimension of the features. Due to the inconvenient characteristics of EEG devices, the present invention can help the workload identification model developed based on EEG to be deployed in practical applications, and ensure the application of the workload identification model. BRIEF DESCRIPTION OF THE DRAWINGS
[0033] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0034] Figure 1 A schematic diagram of the implementation process of the workload identification method under missing EEG modality provided by an embodiment of the present invention;
[0035] Figure 2 is a flow chart of the EEG generation module in an embodiment of the present invention;
[0036] Figure 3 Schematic diagram of the AutoEncoder structure in the feature fusion module in an embodiment of the present invention. DETAILED DESCRIPTION
[0037] To make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be described in detail and completely in conjunction with the drawings in the implementation of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings can be arranged and designed using different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the present application for protection, but merely represents selected embodiments of the present invention.
[0038] The workload identification method under the absence of EEG modality provided by the embodiment of the present invention is used for workload identification scenarios under the absence of EEG modality. Figure 1,The main workload recognition model includes (physiological signal) feature extraction network, EEG feature generation module, feature fusion module and ,classifier; its implementation process specifically includes:
[0039] Step S1: Preprocess EEG data and other physiological signals; the main types of noise contained in EEG include: (1) baseline drift (2) power frequency interference (3) electromyographic interference. Baseline drift is a noise with a frequency lower than 1 Hz caused by human breathing, movement or relative displacement between the electrode and skin of the acquisition device. Therefore, a high-pass filter based on 1 Hz can basically remove the baseline drift noise. Power frequency interference mainly comes from interference from other electrical appliances, and the main frequency is concentrated around 50 Hz, so it can be removed by a 48 Hz-52 Hz notch filter. Electromyographic interference is often concentrated at higher frequencies, and the main frequency of EEG is basically less than 40 Hz, so a 40 Hz low-pass filter is performed to eliminate high-frequency interference such as electromyography. In order to facilitate model training and performance evaluation, the embodiment of the present invention divides the EEG signal into a 1-second time window, and there is no overlap between the time windows of the two samples before and after the time axis. Then the entire data set is normalized.
[0040] Transformer has shown great advantages in processing time series data. However, directly inputting physiological signals into Transformer Encoder for feature extraction will cause overfitting problems. In order to solve this problem, a series of processing of physiological signals is required. The input physiological signals are divided into multiple time windows, each time window is reduced in dimension by a convolutional neural network, and finally the encoding of the signal in this window is output. This processing can reduce the parameters of the entire model, thereby reducing the overfitting of the model.
[0041] Set the size of the physiological signal cutting window (time window) to W and the step size to S. Through this time window, a physiological signal can be cut into several segments. Define K to represent the number of segments. The cut physiological signal can be expressed as:
[0042]
[0043] Among them, S i represents the cutting result of sample i, that is, the set of cutting segments of a physiological signal, represents the j-th fragment, j=1,2,…,K.
[0044] Then each fragment Input to the encoding layer E m , get the encoding representation of each fragment: Thus, the encoding representation based on K fragments Get the encoding of this physiological signal
[0045] Then the obtained physiological signal encoding is input into Transformer Encoder for feature extraction. The formula is expressed as:
[0046]
[0047] Where En(·) is the Transformer Encoder, is the obtained time domain feature.
[0048] The entire feature extraction process of the time domain signal can be expressed as:
[0049]
[0050] in, Represents the entire time domain feature extraction process, including the division of physiological signal time windows and feature extraction based on En and En.
[0051] Different physiological signals have different main frequencies. For example, the main frequency of the ECG signal is 0.5-45HZ, and the main frequency of the EEG is 0.5-80HZ. Therefore, the noise outside the main frequency should be discarded, and the model should focus more on the main frequency. In addition, the importance of the main frequencies of physiological signals is not the same. For example, the workload focuses more on the α band (8-12HZ) and θ band (4-8HZ) of the EEG. Based on this, the embodiment of the present invention uses an attention mechanism for the frequency domain feature extraction network to make the model focus on those frequency bands that are more important to the workload, thereby improving the generalization ability of the network model. The entire feature extraction process of the entire frequency domain signal is expressed using the following formula:
[0052]
[0053] in represents the frequency domain feature extractor, Represents the extracted frequency domain features.
[0054] The time domain feature extractor and the frequency domain feature extractor constitute a time-frequency feature extractor (i.e., the feature extraction network of the embodiment of the present invention). After extracting the time domain features and the frequency domain features, the two features are fused by splicing, and the formula is as follows:
[0055]
[0056] Step S2: Train the model. During the training process, EEG is used to enrich the feature information.
[0057] In the embodiment of the present invention, the goal of the EEG feature generation module is to train a deep neural network that can generate EEG features. The EEG features refer to the features obtained after the EEG is input into the feature extraction network. This enables the subsequent feature fusion module to obtain EEG features, which can provide a more comprehensive representation of the workload and improve the overall model performance. Figure 2 In the embodiment of the present invention, the processing process of the EEG feature generation module is specifically as follows:
[0058] Assume that the EEG feature generation module is expressed as Then the input and output of the EEG feature generation module can be expressed as:
[0059]
[0060] in, represents the EEG features generated by the EEG feature generation module, f ecg , f ppg Represents the features corresponding to other physiological signal modalities (obtained by the feature extraction network). The specific physiological signals used are determined by different data sets. For example, in addition to EEG, the MOCAS data set also includes two physiological signals: blood volume pulse signal (PPG) and skin electrode signal (GSR); while in addition to EEG, the WAUC data set also includes four physiological signals: electrocardiogram (ECG), blood volume pulse signal (PPG), skin electrode signal (GSR), and respiratory rate signal (BR). In the embodiment of the present invention, other physiological signals include: electrocardiogram signal f ecg , blood volume pulse signal f ppg and skin electrical signals gsr .
[0061] In view of the fact that it is difficult to directly generate corresponding EEG features from physiological signal features of other modalities, in order to reduce the difficulty of model learning, the embodiment of the present invention increases the mutual information (MutualInformation) between EEG features and other physiological signal features. The mutual information between two features is expressed as the degree of similarity between the two features. The greater the mutual information of the features corresponding to each modality for the features of the EEG signal, the higher the degree of similarity of the features, making it easier to generate subsequent EEG signal features. At the same time, increasing the mutual information of physiological signals of other modalities and EEG signals can also align the features learned by the two feature extraction networks, so that the generalization performance of the network is further improved.
[0062] The mutual information formula is as follows:
[0063]
[0064] Among them, I x,yrepresents the mutual information of random variable X=x= and random variable Y=y, p(x) represents the probability of X=x appearing, p(y) represents the probability of Y=y appearing, and p(x,y) represents the probability of X=x and Y=y appearing at the same time. Therefore, the embodiment of the present invention uses mutual information loss, and the formula is as follows:
[0065]
[0066] Among them, I eeg,ecg represents f eeg and f ecg The mutual information between eeg,ppg represents f eeg and f ppg The mutual information between eeg,gsr represents f eeg and f gsr Through mutual information loss, the generalization performance of the entire model can be effectively improved, and it can also help generate subsequent EEG features.
[0067] The data of each modality is further transformed to obtain the corresponding transformed features, where the feature transformation network contains 4 layers: (1) Linear: linear layer (2) ReLU layer (3) BN: batch normalization layer (4) Linear: linear layer. Then, the transformed physiological signal features except the EEG modality are input into the Transformer Encoder, and these features are interacted with each other. Finally, multiple outputs are added to obtain the final output. That is, the generated EEG signal features. Using reconstruction loss Provide supervision to the entire network so that the final generated features are as similar as possible to the features obtained by the EEG feature extraction network. The reconstruction loss is expressed as follows:
[0068]
[0069] Therefore, the loss function of the entire EEG feature generation module can be expressed as:
[0070]
[0071] During inference, the missing EEG signal features can be generated through the EEG feature generation module. The EEG feature generation module uses two losses to make the final generated features more similar to the real features. Mutual information loss can effectively reduce the difficulty of the entire learning process, while reconstruction loss provides supervision for the entire learning process.
[0072] The function of the feature fusion module is to fuse multiple physiological signal features. Since the EEG feature generation module increases the mutual information between EEG features and other physiological signal features, other physiological signal features and EEG features may have redundant information. Therefore, one goal of the feature fusion module is to reduce the overall redundancy of the model. AutoEncoder can extract key information from redundant inputs and ignore those redundant features, see Figure 3 , which includes an encoder and a decoder, through which the redundant input ( The decoder maps the low-dimensional features output by the encoder to the initial input of the encoder, so that the information contained in the low-dimensional features is complete. Therefore, the embodiment of the present invention removes redundancy of the input multimodal physiological signal features based on AutoEncoder.
[0073] Assumptions For Encoder, For Decoder, the encoder maps redundant input to low-dimensional features and can be expressed as:
[0074] h=F e (cat(f eeg f ecg ,f ppg ,f gsr ,...))
[0075] Among them, h represents the low-dimensional features after removing redundancy, and cat(·) represents the concatenation of multiple features.
[0076] Using the decoder to reconstruct the features can be expressed as:
[0077]
[0078] in, Represents the low-dimensional features of different modalities after redundancy removal.
[0079] The loss of training AuotoEncoder It can be expressed as:
[0080]
[0081] Where N represents the number of modalities of physiological signal features, f i represents the i-th physiological signal feature, Represents the i-th physiological signal feature reconstructed by the decoder.
[0082] In summary, the overall loss of the workload identification model provided by the embodiment of the present invention during training can be expressed as follows:
[0083]
[0084] You can also set it to The weighted sum of , the weight coefficient of each loss is a hyperparameter.
[0085] Step S3: Since EEG signals are not available during inference, the forward propagation process is somewhat different from that during training. EEG characteristics during alternative training eeg Complete the subsequent reasoning. Since the training of the EEG generation module has been completed well during the training process, the EEG features generated during the reasoning process It is more realistic and can achieve better results.
[0086] It should be noted that the type setting of the workload identification result can be set based on the actual application requirements, and then the corresponding data set can be selected. For example, the workload levels are divided into two categories in the STEW data set, while three different levels of workloads will be stimulated in the subjects in the CONTEST data set.
[0087] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.
[0088] The above are only some embodiments of the present invention. For those skilled in the art, several modifications and improvements can be made without departing from the creative concept of the present invention, which all belong to the protection scope of the present invention.
Claims
1. A method for identifying workload in the absence of EEG modality, characterized in that: The following steps are involved: Step 1, setting a workload identification model, which includes a physiological signal feature extraction network, an EEG feature generation module, a feature fusion module and a classifier; Among them, the physiological signal feature extraction network is used for signal features of physiological signals of different modalities; The input of the EEG feature generation module is the signal features of the physiological signals other than EEG signals, and the interactive information features between different modalities are obtained. Then, all the interactive information features are integrated to obtain the output features, i.e., the generated EEG signal features. The feature fusion module is used to perform de-redundant feature fusion on the signal features of the non-EEG signal output by the physiological signal feature extraction network and the generated EEG signal features, so as to obtain multi-modal de-redundant features for input into the classifier; The classifier is used to output the classification probability of the workload category; Step 2: training the workload recognition model based on the collected training data set to obtain a trained workload recognition model; During training, the total loss function is the weighted sum of the loss function of the EEG feature generation module, the loss function of the feature fusion module and the classification loss; among them, the loss function of the EEG feature generation module includes reconstruction loss and mutual information loss.
2. The method according to claim 1, characterized in that The physiological signal feature extraction network extracts the signal features of physiological signals of different modalities, specifically including: The physiological signal of the sample is cut and processed based on the preset time window size, and the physiological signal is cut into several signal segments; each segment is input into the convolutional neural network for dimension reduction to obtain the encoding representation of each segment; the physiological signal encoding composed of the encoding representation of all segments is then input into the Transformer encoder for feature extraction to obtain the time domain features of the sample; Extract frequency domain features of the physiological signals of the samples based on the attention mechanism to obtain the frequency domain features of the samples; The time domain features and frequency domain features are spliced and fused to obtain the signal characteristics of physiological signals of different modalities of the samples.
3. The method according to claim 1, characterized in that The time window length is 1 second.
4. The method according to claim 1, characterized in that The EEG feature generation module includes: modality feature conversion module and Transformer encoder; The modality feature conversion module is used to transform the signal features of each modality of the input physiological signal, and then input the transformed features into the Transformer encoder; The Transformer encoder is used to calculate the mutual information features between the signal features of any two modes; Add all the interactive information features output by the Transformer encoder to get the output of the EEG feature generation module; Among them, the modal feature conversion module includes a cascaded first linear layer, a ReLU activation function layer, a batch normalization layer and a second linear layer.
5. The method according to claim 3, characterized in that The mutual information loss of the EEG feature generation module is: the sum of the inverses of the mutual information between the modalities; the reconstruction loss of the EEG feature generation module is: the sum of the squares of the differences between the EEG signal features extracted by the sample's physiological signal feature extraction network and the generated EEG signal features.
6. The method according to claim 1, characterized in that The feature fusion module includes: encoder and decoder; The input of the encoder includes: the signal features of each physiological signal input to the EEG feature generation module and the EEG signal features generated by the EEG feature generation module, that is, the multi-modal signal features f i ; Wherein, subscript i is the sample number; After concatenating the multi-modal signal features of the input encoder, the redundant low-dimensional features h are obtained after passing through the autoencoder. i ; Use the decoder to decode the low-dimensional features h i Reconstruction is performed to reconstruct and restore the multi-modal signal characteristics of the input encoder According to the formula Calculate the loss function of the feature fusion module, where N represents the number of samples.
7. The method according to claim 1, characterized in that Non-EEG physiological signals include: ECG signals f ecg , blood volume pulse signal f ppg and skin electrical signals gsr .
Citation Information
Patent Citations
Information interaction system and information interaction method for combining brain electricity and eye electricity
CN103338265A
Form factors for the multi-modal physiological assessment of brain health
CN105592788A
Mental workload assessment method, device, equipment and medium
CN113576481A
Method for generating electroencephalogram emotion classification model based on mutual information driving
CN115444431A
Mental workload detection and model training method and device, equipment and storage medium
CN116304643A
Cited By
Multi-modal fusion alertness assessment method and system based on cross attention
CN120124011A
