A lightweight method for assessing mental workload based on multimodal physiological parameters

Abstract

This invention relates to a method for assessing mental workload based on multimodal physiological parameters. To achieve a lightweight and highly accurate method for assessing mental workload, the technical solution adopted in this invention includes the following steps: S1. Constructing a workload test for multimodal data acquisition; S2. Adding noise to the data, data preprocessing, feature extraction, and dataset construction; S3. Constructing multiple classification models; S4. Constructing an ensemble model based on a show-of-hands voting method; S5. Using the ensemble model to classify the subjects' EEG and ECG data. This invention collects and processes single-lead EEG and ECG data from subjects in a subtraction calculation task, adds noise to the data, extracts features, constructs multiple classification models, and uses a show-of-hands voting method to integrate the models. Results show that the ensemble model proposed in this method improves classification accuracy compared to the original feature set model.

Description

Technical Field

[0001] This application relates to the design of a lightweight mental workload assessment method based on multimodal physiological parameters, belonging to the field of mental workload recognition technology. Background Technology

[0002] In various fields, rapid technological advancements have placed higher demands on relevant personnel. This is mainly reflected in higher technical complexity, more refined technical solutions, and more diverse operational requirements, which in turn increases the mental workload of these personnel. High mental workload under intense work conditions can affect work efficiency, and in severe cases, lead to task failure and endanger personal safety. Therefore, it is crucial to research a method for objectively assessing the mental workload of relevant personnel and subsequently providing reasonable work arrangements to reduce errors caused by high mental workload.

[0003] Mental workload is essentially a multidimensional concept, distinct from work workload and task load. Mental workload places greater emphasis on an individual's subjective feelings and the pressure brought about by task demands when performing a task. Most people define mental workload by describing it using various characteristics, such as loss of motivation at work under high mental workload, discomfort caused by decreased work efficiency, and loss or weakening of the ability to respond to stimuli.

[0004] Mental workload assessment methods can utilize numerous physiological parameters as data acquisition objects, with electroencephalogram (EEG) signals being the mainstream physiological parameter. For example, patent publication number CN112773365A discloses a mental workload monitoring system for submariners during underwater operations. This system involves a mental workload assessment method that collects EEG data from submariners during underwater operations, performs filtering and feature extraction on the EEG data, and then constructs a mental workload classification model based on the support vector machine algorithm to assess mental workload. While this method is simple to operate, it only uses single-modality data, and there is room for improvement in the accuracy of mental workload assessment.

[0005] Currently, although some methods for assessing mental workload have been reported, their data collection processes are cumbersome, requiring high-precision multi-lead acquisition equipment, and the high cost makes practical implementation difficult. For example, patent CN111839506A discloses a method for assessing mental workload that uses a 16-channel medical-grade EEG cap to collect EEG data and trains a mental workload classification model based on deep learning. This model can then be used to classify the mental workload of subjects. However, this method is costly to implement and requires medical-grade equipment for data collection, severely limiting its widespread adoption. Therefore, there is an urgent need for a low-cost, lightweight method for assessing mental workload that allows for convenient evaluation.

[0006] Domestic and international researchers have conducted extensive research on methods for assessing mental fatigue. Mental workload assessment methods are generally divided into subjective and objective types based on data acquisition. Subjective mental workload assessment requires subjects to fill out a mental workload scale during or after performing mental workload tasks. This method requires almost no data acquisition equipment, using only paper and pen to measure mental workload. However, the stability of mental workload measurement is not high, and it is too subjective. Objective mental workload assessment relies on the collection of data such as physiological electrical signals during the subject's performance of fatigue tasks. This method is generally non-invasive and non-surgical, thus offering better subject comfort. This invention uses more objective single-lead multimodal physiological electrical signals as training data. In the network structure part, most scholars use a single model for classification. This invention uses multiple models to form an ensemble model through a voting process. This model can integrate the advantages of sub-models, combining several weak models with slightly better classification results into a strong model with better classification results. Summary of the Invention

[0007] This invention addresses the challenges of data acquisition in existing mental workload assessment methods, such as the difficulty of data collection (requiring bulky multi-lead acquisition equipment, susceptibility to interference during acquisition), and the low accuracy of current methods. Based on single-lead multimodal electrophysiological signals, and utilizing deep learning and ensemble learning theories, a new mental workload assessment method is proposed. This method extracts features from raw EEG and ECG data, as well as noisy EEG and ECG data, to obtain different feature sets. Different classification models are then constructed using these feature sets based on a BiLSTM-attention mechanism-GRU composite network. The results are then integrated using a show of hands voting to obtain the classification result of the ensemble model. This method offers advantages such as ease of operation and high assessment accuracy. It can assess the mental workload of subjects and assign them reasonable and appropriate workloads based on the assessment results, reducing errors caused by a mismatch between workload difficulty and current mental workload, thereby improving work efficiency and safety. The method used in this invention is suitable for situations with relatively poor signal quality.

[0008] This invention is achieved through the following technical solution:

[0009] This invention relates to a lightweight method for assessing mental workload based on multimodal physiological parameters, comprising the following steps:

[0010] S1. Design a mental load induced experiment based on subtraction arithmetic and collect EEG and ECG physiological data of subjects in both computational and resting states during the experiment.

[0011] S2. Add Gaussian noise of different sizes to the physiological data, and preprocess the physiological data and the physiological data after adding noise to obtain samples. Extract features from the obtained samples, extract the time domain features and frequency domain features of the signals, and construct multiple feature datasets.

[0012] S3. Use the multiple feature datasets to construct different classification models based on deep learning networks.

[0013] S4. Construct an ensemble model based on the show-of-hands voting method using the different classification models.

[0014] S5. Classify the electroencephalogram (EEG) and electrocardiogram (ECG) physiological data using the integrated model.

[0015] The mental load induced experiment involving subtraction arithmetic refers to the following: Experimental software is run on a computer, guiding subjects into different mental load states for subsequent EEG and ECG data acquisition. These states include a resting state and a calculation state. In the resting state, the software does not present any calculations, and subjects do not perform any calculations. Subjects should relax to obtain high-quality resting EEG and ECG data. The resting state lasts for one minute. In the calculation state, the software presents a four-digit subtraction of a two-digit number every few seconds. Subjects must mentally calculate the equation and input the result into the software using a mouse and keyboard. Subjects should engage in normal mental activity to obtain high-quality calculation-state EEG and ECG data. The calculation state lasts for five minutes. One resting state and one calculation state constitute one acquisition cycle. Each complete data acquisition session consists of two acquisition cycles, totaling twelve minutes.

[0016] The aforementioned data collection based on the computational and resting states of the subjects' electroencephalogram (EEG) and electrocardiogram (ECG) physiological data refers to the following: During the experiment, EEG and ECG data of the subjects are collected. A single-lead EEG cap is used to collect frontal EEG data, and a single-lead ECG compression garment is used to collect chest ECG data. Data is collected from the subjects undergoing the experiment using acquisition equipment to obtain their EEG and ECG data during the mental exertion experiment.

[0017] The addition of Gaussian noise of different magnitudes refers to adding noise with a mean of 0 and different variances to the collected EEG and ECG physiological data, creating multiple noisy datasets. These noisy datasets are used to build subsequent noisy classification models, which are ultimately used for model ensemble.

[0018] The preprocessing refers to: using a third-order Butterworth high-pass filter with a cutoff frequency of 0.65Hz to filter out low-frequency noise in the EEG and ECG signals, and using third-order Butterworth band-pass filters with cutoff frequencies of 48Hz and 52Hz to filter out power frequency noise in the EEG and ECG signals. The filtered EEG and ECG data are then segmented into samples with a length of 1 second and an 80% overlap rate, and corresponding sample labels are obtained based on the mental workload state of each sample.

[0019] The extraction of time-domain and frequency-domain features of the signal refers to the extraction of the following features from the preprocessed EEG and ECG: mean, variance, and standard deviation in the time domain; and power spectrum, amplitude, and 8th-order autoregressive model coefficients in the frequency domain. Specifically, Fourier transform is used to extract the spectral amplitude features of five frequency bands: delta (2-4Hz), theta (4-8Hz), alpha (8-12Hz), beta (12-30Hz), and gamma (30-50Hz). The 50Hz frequency band was divided into 25 segments, each 2Hz, and 25 frequency band features were extracted. Power spectrum features of five frequency bands, delta (2-4Hz), theta (4-8Hz), alpha (8-12Hz), beta (12-30Hz), and gamma (30-50Hz), were extracted using the periodogram method. The 50Hz frequency band of the signal was also divided into 25 segments, each 2Hz, and 25 frequency band features were extracted. The coefficient features of the 8th order autoregressive model (autoreg, AR) of the signal were extracted.

[0020] The construction of multiple feature datasets refers to the following: the feature set obtained after extracting features from the physiological data is combined with the corresponding label to form the original feature dataset; and the corresponding feature sets obtained from multiple noisy datasets are combined with the label to form multiple noisy feature datasets.

[0021] The deep learning network refers to the following: The EEG feature set is processed through a bidirectional long short-term memory network to further extract features, then through a batch normalization layer (which accelerates network convergence and avoids gradient explosion or vanishing during training), followed by an attention mechanism layer to enable the model to automatically learn important parts of the data, and finally a dropout regularization layer (which randomly deletes neurons to reduce overfitting). This part constitutes the EEG feature subnetwork. Simultaneously, the ECG features are processed through a bidirectional long short-term memory network, then through a batch normalization layer, followed by an attention mechanism layer, and finally a dropout regularization layer. This part constitutes the ECG feature subnetwork. The outputs of the EEG and ECG feature subnetworks are fused in a fusion layer to obtain fused features, which are then further extracted through a gated recurrent neural network. Finally, a tensor flattening layer extends the features to one dimension, and the final result is output after two fully connected layers.

[0022] The construction of different classification models refers to: using the physiological data feature set to establish a mental workload classification sub-model through a deep learning network, and at the same time, multiple noisy feature datasets are also used to establish multiple mental workload classification sub-models through a deep learning network.

[0023] The aforementioned show-of-hands voting method for constructing an ensemble model refers to the process of integrating the results of multiple classification sub-models through a show-of-hands voting method. Specifically, the show-of-hands voting method involves multiple models outputting multiple results, and the final output of the ensemble model is determined based on the label with the most occurrences in the results. Attached Figure Description

[0024] Figure 1 This is a flowchart of the method provided in the embodiments of the present invention;

[0025] Figure 2 This is a schematic diagram of data acquisition in an embodiment of the present invention.

[0026] Figure 3 This is a model framework diagram of an embodiment of the present invention.

[0027] Figure 4 This is a schematic diagram of the integrated model of an embodiment of the present invention. Detailed Implementation

[0028] This embodiment relates to a lightweight mental workload assessment method based on multimodal physiological parameters, including the following steps:

[0029] S1. Design a mental load induced experiment based on subtraction arithmetic and collect EEG and ECG physiological data of subjects in both computational and resting states during the experiment.

[0030] S2. Add Gaussian noise of different sizes to the physiological data, and preprocess the physiological data and the noisy physiological data. Extract the features of the sample obtained after preprocessing, extract the time domain features and frequency domain features of the signal, and construct multiple feature datasets.

[0031] S3. Use the multiple feature datasets to construct different classification models based on deep learning networks.

[0032] S4. Construct an ensemble model based on the show-of-hands voting method using the different classification models.

[0033] S5. Classify the EEG and cardiac workload data using the integrated model.

[0034] This embodiment first designs an experiment to obtain mental load data from the subjects. The program used in the experiment is written in Python and encapsulated in a user-friendly graphical user interface using the PYQT library. The experimental program mainly consists of a signal display module, an equation display module, a result input module, and a time display module. The signal display module can display the collected electrocardiogram and electroencephalogram signals in real time. The equation display module can display the equations that the subjects need to calculate in real time. The equation format is four digits minus two digits. The result input module allows the subjects to input the calculation results after thinking. The time display module can display the remaining time and the number of equations remaining in the current stage (resting state stage or calculation state stage) in real time.

[0035] This embodiment recruited 7 participants to conduct a mental workload experiment to collect mental workload data. Participants were fully familiar with the process before the start of this embodiment. Wearing a single-lead wearable EEG cap and a single-lead wearable ECG suit, participants sat in front of a computer running the mental workload induction experiment program, maintaining their attention. At the beginning of the program, a blank screen appeared, only indicating that the participant was in a resting state. The participant did not perform any operation. The wearable device collected the participant's EEG and ECG data and displayed them in real time on the program; this phase lasted one minute. Afterward, the program entered a calculation phase, where different formulas were presented before and after a certain time to induce a high mental workload state in the participant; this phase lasted five minutes. The resting state phase and the calculation phase were repeated twice as one data collection cycle for this embodiment. A total of 29 mental workload data points were collected. Data with problems and test accuracy below 80% were filtered out, leaving 16 data points.

[0036] Gaussian noise with a mean of 0 and a variance of different values ​​(ranging from 1e-03 to 1e-06) was added to the original mental workload data, resulting in four noise datasets. Combined with the original dataset, there are now a total of five datasets.

[0037] The seven datasets were preprocessed. First, the data were filtered to remove low-frequency noise using a third-order Butterworth high-pass filter with a truncation frequency of 0.65Hz. Then, the data were filtered to remove power frequency noise using third-order Butterworth band-pass filters with truncation frequencies of 48Hz and 52Hz. Finally, the data were segmented into samples with a length of 1 second and an 80% overlap rate, and corresponding sample labels were obtained based on the mental workload state of the samples.

[0038] The next step is feature extraction. For each sample in each dataset, its spectral features and 8th-order autoregressive model coefficients in the frequency domain are extracted. The spectral feature extraction process is as follows: First, a Fourier transform is performed on the sample to obtain the signal's spectrum. The spectral amplitude features of five frequency bands are extracted: delta (2-4Hz), theta (4-8Hz), alpha (8-12Hz), beta (12-30Hz), and gamma (30-50Hz). Simultaneously, the 0-50Hz frequency band of the signal is divided into 25 segments of 2Hz each, and the spectral features of these 25 segments are extracted. The 8th-order autoregressive model coefficient features of the sample are also extracted. A total of 39 features are extracted for each sample in each dataset.

[0039] To perform five-fold cross-validation, the five feature sets are divided into training and test sets as follows: First, the original feature set is shuffled and divided into five parts. One part is selected each time as the validation set, and the remaining four parts are used as the training set. This process is repeated for each part. Second, the noisy feature set is shuffled in the same way as the original feature set, and then the training and test sets are divided in the same manner. Simultaneously, the validation set at this point is replaced with the validation set elements from the original feature set at the same positions. Five-fold cross-validation avoids the specificities and limitations of traditional data partitioning methods and can more comprehensively reflect model performance.

[0040] After dividing the five feature sets into training and validation sets each time, a brain load classification model was constructed based on a deep learning network, with a total of 25 training iterations and tests. The deep learning network consists of a bidirectional long short-term memory network, a batch normalization layer, an attention mechanism layer, a dropout regularization layer, a fusion layer, a gated recurrent neural network, a tensor flattening layer, and a fully connected layer. EEG and ECG features were first further extracted using the bidirectional long short-term memory network, then normalized using the batch normalization layer, followed by the attention mechanism layer to allow the model to automatically learn the important parts of the data, and then the dropout regularization layer randomly discarded a certain number of neurons. The further processed EEG and ECG features were then fused in the fusion layer, further extracted using the gated recurrent neural network, and finally output through two fully connected layers. The results are shown in the table below, with an average accuracy of approximately 90% on the test set.

[0041] experiment Fold 1 2 folds 3% off 4% off 50% off average 0 noise 91.39% 87.09% 93.16% 90.62% 86.81% 89.82% 0.000001 noise 91.52% 89.07% 92.69% 90.84% 93.56% 91.54% 0.00001 noise 90.22% 90.88% 91.22% 93.07% 90.81% 91.24% 0.0001 noise 85.66% 80.18% 80.61% 85.29% 93.58% 85.06% 0.001 noise 86.38% 86.27% 87.19% 87.72% 84.70% 86.45%

[0042] The ensemble model also underwent five-fold cross-validation. Following the five-fold division order in the table above, at each fold, the classification model trained on the original feature set and the models trained on different noise feature sets were integrated based on a show of hands. The experiment selected three of the five sub-models for integration, with 0 representing the zero-noise model and 1 representing the 0.000001 noise model, and the naming convention followed similarly thereafter. Based on the three classification sub-models, each validation sample had three classification results. If the resting-state predictions were the majority, the ensemble model's prediction for that sample was considered resting-state; otherwise, it was considered computational. The results of each fold were calculated, and the results are shown in the table below.

[0043] experiment Fold 1 2 folds 3% off 4% off 50% off average 012 95.01% 94.37% 95.35% 95.65% 94.65% 95.00% 013 93.09% 90.75% 94.10% 94.50% 96.46% 93.78% 014 94.57% 93.39% 95.16% 94.88% 94.69% 94.54% 123 94.10% 92.22% 94.03% 95.31% 96.57% 94.45% 124 93.93% 94.71% 94.31% 94.46% 94.61% 94.40% 234 92.43% 92.35% 92.80% 94.40% 95.08% 93.41%

[0044] As shown in the table, the ensemble model has the highest accuracy in assessing the mental workload of the subjects. Compared with the assessment accuracy of individual models, the ensemble model shows an increase in the accuracy of mental workload assessment.

[0045] The above embodiments are merely preferred embodiments. Based on these specific embodiments, those skilled in the art can make various improvements and partial adjustments without departing from the principles and spirit of this invention, and such improvements and adjustments should be considered within the scope of protection of this invention within the legal constraints.

Claims

1. A method for mental workload assessment based on multi-modal physiological parameters, characterized in that, The method comprises the following steps: S1. Design a brain load inducing experiment, and obtain electroencephalogram and electrocardiogram physiological data of subjects under different loads in the experiment; S2. Add Gaussian noise of different sizes to the physiological data, and pre-process the physiological data and the physiological data after adding noise to obtain samples, extract features from the obtained samples, extract time domain features and frequency domain features, and construct multiple feature data sets; S3. Use the multiple feature data sets to construct different classification models based on a deep learning network, wherein the classification model uses the feature data set constructed from the physiological data to establish a brain load classification sub-model through the deep learning network, and the feature data set constructed from the multiple physiological data after adding noise also establishes multiple brain load classification sub-models through the deep learning network; S4. Construct an integrated model based on the hand-raising voting method using the different classification models; S5. Classify and evaluate the electroencephalogram and electrocardiogram physiological data of the subjects using the integrated model.

2. The brain load inducing test according to claim 1, characterized in that, The brain load state is induced by mental calculation of multi-digit subtraction arithmetic, wherein a resting state subject does not perform any action, and only the subject data is collected; the calculation state is subtraction calculation, and the subject calculates in the brain according to the subtraction formula appearing on the computer, and inputs the result on the computer using the keyboard; The electroencephalogram is collected from the forehead single-channel electroencephalogram signal of the subject, and the electrocardiogram is collected from the chest single-channel electrocardiogram signal of the subject, wherein the length of the resting state is 1 minute, the length of the calculation state is 5 minutes, one resting state and one calculation state constitute one collection cycle, and one experiment has two collection cycles.

3. The method of claim 1, wherein, The adding of Gaussian noise of different sizes refers to adding Gaussian noise with a mean of 0 and a variance of n, wherein n takes a value of 1e-03 to 1e-06 with an interval of 1 order of magnitude, and finally 5 data sets are obtained.

4. The method of claim 1, wherein, The pre-processing refers to filtering the physiological data using a 3rd-order Butterworth high-pass filter with a cut-off frequency of 0.65 Hz and a 3rd-order Butterworth band-pass filter with a cut-off frequency of 48 Hz and 52 Hz to filter out low-frequency noise and power frequency noise; segmenting the filtered physiological data into samples according to a length of 1 second and an overlap rate of 80%, and obtaining corresponding sample labels according to the brain load state to which the obtained samples belong.

5. The method of claim 1, wherein, The feature extraction refers to extracting mean and variance features in the time domain of the samples, extracting power spectrum features and amplitude features in the frequency domain of the samples, and then combining the sample features and sample labels to form multiple feature data sets.

6. The method of claim 1 wherein, The deep learning network refers to a multi-modal BiLSTM brain load classification network model composed of a BiLSTM module, an attention mechanism module, and a GRU module, wherein the electroencephalogram feature set and the electrocardiogram feature set of the original data and the noise-added data are respectively subjected to the BiLSTM module and the attention mechanism module, then feature fusion is performed, and finally the GRU module is used to output a classification result.

7. The method of claim 1, wherein, The integrated model refers to classifying the electroencephalogram and electrocardiogram physiological data using the different classification models, voting the classification results of different models, and using the voting results as the classification result of the integrated model.

Images