Electroencephalogram-based fatigue detection model system, electroencephalogram-based fatigue detection method, electronic device, and storage medium

Abstract

The application discloses an electroencephalogram-based fatigue detection model system, method, electronic equipment and storage medium, wherein the electroencephalogram-based fatigue detection model system comprises a modal data acquisition unit configured to acquire different modal data; a multi-view embedding module comprising a dedicated embedding layer for the different modal data and configured to extract embedding sequences of the different modal data by using linear and nonlinear methods; a modal mixing transformer configured to perform multi-modal feature interaction on the embedding sequences to obtain a multi-modal mixing result; and an adversarial domain generalization module configured to perform transfer learning on the multi-modal mixing result, so that the model can learn more generalizable feature representations.

Description

TECHNICAL FIELD

[0001] The embodiment of the present application relates to the technical field of fatigue detection based on electroencephalogram, and particularly relates to a fatigue detection model based on electroencephalogram, a fatigue detection method based on electroencephalogram, an electronic device and a storage medium. BACKGROUND

[0002] In the related art, fatigue driving is still a major cause of traffic accidents. The traditional method of fatigue detection is to quantify the percentage of eyelid closure over the pupil over time (PERCLOS, the percentage of eyelid closure over the pupil over time), which is a method of measuring the percentage of closing eyes. Previous studies have emphasized the potential of electroencephalogram (EEG, electroencephalogram) and electrooculogram (EOG, electrooculogram) in estimating vigilance by predicting PERCLOS from eye tracking glasses.

[0003] Due to the large individual differences in physiological data, it is inevitable to emphasize the influence of individual differences. Domain generalization is a transfer learning method aimed at solving the case with multiple labeled source domains. The domain generalization method does not require target domain data during training, but learns domain-invariant features in multiple source domains to improve the generalization ability to unknown domains.

[0004] Multi-modal model is a component of machine learning and deep learning paradigm, which has the ability to process different data types or information sources. Fusing information from multiple modalities is beneficial to enhance the robustness of the model and extract complex feature representations. The significance of fusing electroencephalogram and electrooculogram lies in combining the information of brain activity and eye movement to provide more comprehensive physiological features. The fusion of these patterns is expected to more accurately improve the performance of fatigue state estimation. SUMMARY

[0005] The embodiment of the present application provides a fatigue detection model based on electroencephalogram, a method, an electronic device and a storage medium, which are used to at least solve one of the above technical problems.

[0006] In a first aspect, the embodiment of the present application provides a fatigue detection device based on electroencephalogram, comprising: a modal data acquisition unit, configured to acquire different modal data; a multi-view embedding module, comprising a dedicated embedding layer for the different modal data, configured to extract embedding sequences of the different modal data through linear and nonlinear methods; a modal mixing transformer, configured to perform multi-modal feature interaction on the embedding sequences to obtain a multi-modal mixing result; and an adversarial domain generalization module, configured to perform transfer learning on the multi-modal mixing result, so that the model can learn more generalizable feature representations.

[0007] In a second aspect, an embodiment of the present application provides a fatigue detection method based on electroencephalogram. The method uses the fatigue detection model based on electroencephalogram according to the first aspect. The method includes: acquiring different modal data of a person to be detected by using the modal data acquisition unit; extracting an embedding sequence of the different modal data by using the multi-view embedding module; performing single-modal feature extraction on the embedding sequence by using the modal isolation transformer to obtain at least one single-modal result; performing multi-modal feature interaction on the at least one single-modal result by using the modal mixing transformer to obtain a multi-modal mixed result; performing transfer learning on the multi-modal mixed result by using the adversarial domain generalization module so that the model can learn more generalizable feature representations; and calculating fatigue degree by using the alertness regressor on the multi-modal mixed result to obtain a current fatigue degree prediction value.

[0008] In a third aspect, an embodiment of the present application provides an electronic device, which includes at least one processor and a memory connected to the at least one processor in communication. The memory stores instructions executable by the at least one processor. The instructions are executed by the at least one processor to enable the at least one processor to perform any of the fatigue detection methods based on electroencephalogram described above.

[0009] In a fourth aspect, an embodiment of the present application provides a storage medium, which stores one or more programs including execution instructions. The execution instructions can be read and executed by an electronic device (including but not limited to a computer, a server, or a network device, etc.) to perform any of the fatigue detection methods based on electroencephalogram described above.

[0010] In a fifth aspect, an embodiment of the present application further provides a computer program product, which includes a computer program stored on a storage medium. The computer program includes program instructions, which, when executed by a computer, cause the computer to perform any of the fatigue detection methods based on electroencephalogram described above.

[0011] The scheme of the embodiments of the present application can more effectively process multi-modal data, fully utilize the interaction information between different modalities, improve the feature extraction capability and prediction performance of the model on physiological data, have good domain generalization capability, and be suitable for different domain data sets. BRIEF DESCRIPTION OF DRAWINGS

[0012] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed in the embodiment description. Obviously, the drawings in the following description are some embodiments of the present application, and those skilled in the art can also obtain other drawings according to these drawings without creative labor.

[0013] Figure 1 A structural block diagram of a fatigue detection model system based on electroencephalogram according to an embodiment of the present application is provided.

[0014] Figure 2 A flowchart of a fatigue detection method based on electroencephalogram according to an embodiment of the present application is provided.

[0015] Figure 3 A flowchart of a specific example of a fatigue detection method based on electroencephalogram according to an embodiment of the present application is provided.

[0016] Figure 4 A comparison of the method and the baseline of the present application in the dependent subject and cross-subject settings in terms of RMSE (Root Mean Square Error), PCC (Pearson’s correlation coefficient) and CCC (Concordance Correlation Coefficient) according to an embodiment of the present application is provided.

[0017] Figure 5 A comparison of the method and the baseline of the present application in the cross-subject settings in terms of RMSE and PCC according to an embodiment of the present application is provided.

[0018] Figure 6 A prediction comparison of different methods according to an embodiment of the present application is provided.

[0019] Figure 7 A structural schematic diagram of an electronic device according to an embodiment of the present application is provided. DETAILED DESCRIPTION

[0020] In order to make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0021] Reference is made to Figure 1 which shows a structural block diagram of a fatigue detection model system based on electroencephalogram according to an embodiment of the present application.

[0022] As Figure 1As shown, a fatigue monitoring model based on electroencephalogram includes: a modal data acquisition unit for acquiring different modal data; a multi-view embedding module including a dedicated embedding layer for the different modal data, for extracting embedding sequences of different modal data through linear and nonlinear methods; a modal mixing transformer for multi-modal feature interaction on the embedding sequences to obtain a multi-modal mixed result; and an adversarial domain generalization module for transfer learning on the multi-modal mixed result, so that the model can learn more generalizable feature representations.

[0023] The model system of the embodiments of the present application can more effectively process multi-modal data, fully utilize the interaction information between different modalities, improve the feature extraction capability and prediction performance of the model on physiological data, and has good domain generalization capability, and is suitable for data sets in different fields.

[0024] In some optional embodiments, the above-mentioned model further includes: a modal isolation transformer for single-modal feature extraction on the embedding sequences to obtain at least one single-modal result, wherein the at least one single-modal result is used as input to the modal mixing transformer.

[0025] In some optional embodiments, the multi-view embedding module further includes a modal encoding and position information of each modal data in the embedding sequence.

[0026] In some optional embodiments, the above-mentioned model further includes: an alertness regressor for calculating fatigue degree on the multi-modal mixed result.

[0027] In some optional embodiments, the adversarial domain generalization module includes a gradient reversal layer.

[0028] Please refer to Figure 2 which shows a flowchart of a fatigue detection method based on electroencephalogram provided by an embodiment of the present application, using the above-mentioned fatigue detection model system based on electroencephalogram.

[0029] As shown in the figure, Figure 2 In step 201, the modal data acquisition unit is used to acquire different modal data of a person to be detected;

[0030] In step 202, the multi-view embedding module is used to extract embedding sequences of the different modal data;

[0031] In step 203, the modal isolation transformer is used to extract single-modal features from the embedding sequences to obtain at least one single-modal result;

[0032] In step 204, the multi-modal feature interaction of the at least one single-modal result is performed using the modal mixing transformer to obtain a multi-modal mixed result;

[0033] In step 205, the multi-modal mixed result is subjected to transfer learning using the adversarial domain generalization module to enable the model to learn more generalizable feature representations;

[0034] In step 206, the multi-modal mixed result is subjected to fatigue degree calculation using the alertness regressor to obtain a current fatigue degree prediction value.

[0035] The method of the embodiments of the present application can more effectively process multi-modal data, fully utilize the interaction information between different modalities, improve the feature extraction capability and prediction performance of the model on physiological data, and has good domain generalization capability, and is suitable for data sets in different domains.

[0036] In some optional embodiments, the different modal data includes electroencephalogram data and electrooculogram data, and the modal data acquisition unit includes an electroencephalogram data acquisition unit and an electrooculogram data acquisition unit.

[0037] In order to meet the needs of fatigue detection and take advantage of the latest technological advances, the embodiments of the present application introduce a multi-modal adversarial regressive transformer (MART, Multi-modal Adversarial Regressive Transformer) model. The MART encodes, concatenates and transmits the obtained sequence through a transformer equipped with a multi-head attention mechanism. The obtained sequence is then used for regression. It is worth noting that the model of the embodiments of the present application has an optional function of adversarial domain generalization, allowing transfer learning to be applied in a cross-subject setting. In addition, the model is also scalable and can handle any number of input modes, thus highlighting its adaptability in multi-modal fatigue monitoring.

[0038] In the related art, electroencephalogram data or electroencephalogram data combined with eye movement data are used as input data, and neural networks or convolutional neural networks based on fully connected layers are used to predict PERCLOS values and then infer the fatigue level of the user. These technologies may use traditional machine learning methods or deep learning methods for feature extraction and prediction. First, the physiological data input into these technologies needs to be selected. Generally, these input data are electroencephalogram features or electrooculogram features or simple splicing of the two features. Second, for the input features, these prior art technologies usually use fully connected layers for feature transformation, i.e. the input feature vector is transformed through a matrix, and some prior art technologies also use convolution and other methods for feature extraction. Finally, these prior art technologies return the PERCLOS value corresponding to the current input features.

[0039] The inventors found that the related art may have one or more of the following defects. Firstly, the limitation of feature extraction. Similar technologies usually use fully connected layers or convolutional layers for feature extraction, which may not be able to fully capture the complex features and patterns in physiological data, resulting in poor performance in fatigue level prediction. Secondly, the deficiency in processing multi-modal data. Most similar technologies treat electroencephalogram data or eye movement data as single modal data processing, simply concatenating or processing them in series. This processing method may not fully utilize the interaction information between different modalities, resulting in limited understanding and prediction ability of the model for data. Thirdly, the lack of generalization ability. In some similar technologies, the model may have the problem of overfitting, that is, it performs well on the training set, but has poor generalization ability on novel or different domain data. This may limit the applicability and reliability of the model in practical applications. Fourthly, the limitation of model structure. Some similar technologies use traditional model structures such as fully connected layers and convolutional layers. These structures may not be well adapted to complex physiological data features, affecting the performance and effectiveness of the model. Fifthly, the lack of consideration of domain generalization. Most similar technologies are less concerned about the characteristics and differences of data in different domains when designed, resulting in poor performance in domain generalization and limiting the application ability of the model in different environments.

[0040] The inventors found that the defects in the related art may be caused by the following reasons. Firstly, from the structure of the model, it is the outdated model structure of the prior art, such as fully connected layers and convolutional layer structure, that causes the model to be unable to well extract the fatigue information contained in the physiological data, resulting in a low prediction accuracy of fatigue. Secondly, since the multi-modal model is a relatively novel concept, the prior art did not fully consider the processing and analysis of multi-modal data when it was proposed, resulting in the processing of multi-modal data being simply concatenated and treated as single modal. At present, models using transformers and multi-modal data processing are gradually being promoted to electroencephalogram data analysis. For example, in the problem of emotion recognition, such methods have been promoted. However, there is still no prior art research on fatigue monitoring through electroencephalogram signals.

[0041] If the above problems are to be solved, first of all, those skilled in the art will improve the feature extraction method. They may try to design a more complex feature extraction method, including using a deeper convolutional neural network or a recurrent neural network, to better capture the features and patterns in the data. Secondly, those skilled in the art will optimize the model structure. They may design new model structures, such as introducing attention mechanisms, graph neural networks, etc., to improve the performance and generalization ability of the model. Thirdly, those skilled in the art will adopt multi-modal data fusion methods. They may try more advanced multi-modal data fusion methods, such as joint training, interactive learning, etc., to better utilize the associated information between different modal data. Finally, those skilled in the art will adopt domain generalization techniques. They may study domain generalization techniques, including domain adaptation, adversarial training, etc., to improve the generalization ability of the model in different domains.

[0042] In the embodiments of the present application, through in-depth analysis and evaluation of the prior art, the inventors have found several defects in the prior art. Specifically, the inventors have noticed that the prior art mainly uses traditional structures such as fully connected layers or convolutional layers for feature extraction, which may not be the optimal choice when processing physiological characteristics such as electroencephalogram data. In addition, the prior art often simply concatenates multi-modal data and treats it as single-modal data for processing, resulting in the model being unable to fully utilize the interactive information between modalities.

[0043] With reference to Figure 1 , a structural block diagram of an embodiment of the present application is shown.

[0044] As Figure 1 shown, first, a multi-view embedding module: the embodiments of the present application use dedicated embedding layers for different modal data, extracting features through linear and nonlinear methods to obtain embedding sequences with more information.

[0045] Secondly, a transformer module: the model of the embodiments of the present application combines modal isolation transformers and modal mixing transformers, which can capture the feature interaction between modalities, and improve the feature extraction effect compared with traditional fully connected layers or convolutional layers.

[0046] Finally, an adversarial domain generalization module: an adversarial domain generalization module is introduced, which enables the model to learn more generalizable feature representations through gradient reversal layers and other techniques, solving the limitations of the prior art on domain generalization.

[0047] The specific steps of the method of the embodiments of the present application include: performing dedicated embedding processing on multi-modal data to obtain embedding sequences. The embedding sequences are input into a transformer module, and modal isolation transformers and modal mixing transformers are used for feature extraction and interaction. An adversarial domain generalization module is introduced, and more generalizable feature representations are obtained through adversarial training.

[0048] With reference to both Figs. 1 and 2, Fig. 1 shows a flowchart of a specific example of the EEG-based fatigue detection provided by an embodiment of the present application. As shown in Fig. 1, first, the EEG collection device, the EOG collection device, etc. are worn, and data is collected; second, the collected data is preprocessed; then, the preprocessed data is input into the method described in the embodiment of the present application, and an intermediate result is output through the multi-view embedding module; next, the intermediate result is further operated through the modal isolation transformer to generate a new intermediate result; then, the intermediate result is further operated through the modal mixing transformer to generate a new intermediate result; after that, the intermediate result is simultaneously migrated through the domain generalization module, and the fatigue degree is calculated through the vigilance regressor; finally, the current fatigue degree is returned. Figure 3 Figure 3 As shown in Fig. 1, first, the EEG collection device, the EOG collection device, etc. are worn, and data is collected; second, the collected data is preprocessed; then, the preprocessed data is input into the method described in the embodiment of the present application, and an intermediate result is output through the multi-view embedding module; next, the intermediate result is further operated through the modal isolation transformer to generate a new intermediate result; then, the intermediate result is further operated through the modal mixing transformer to generate a new intermediate result; after that, the intermediate result is simultaneously migrated through the domain generalization module, and the fatigue degree is calculated through the vigilance regressor; finally, the current fatigue degree is returned.

[0049] The multi-modal data fusion method adopted in the embodiment of the present application can improve the prediction accuracy of the model. Specifically, by using an advanced multi-modal data fusion method, the input data is sequenced and modal information and position information are introduced through the multi-view embedding module, which can better utilize the associated information between different modal data, thereby improving the prediction accuracy of the model for indicators such as fatigue degree.

[0050] The transformer architecture adopted in the embodiment of the present application can better capture the characteristics of multi-modal data. Specifically, since the multi-modal mixing transformer architecture is adopted in the embodiment of the present application, which includes the modal isolation transformer and the modal mixing transformer module, the features of different modal data can be extracted and interacted, thereby more comprehensively capturing the feature information of multi-modal data and improving the understanding ability of the model for multi-modal data.

[0051] The adversarial domain generalization module adopted in the embodiment of the present application can improve the generalization ability of the model. Specifically, by introducing the adversarial domain generalization module and using techniques such as gradient reversal layer, the model can learn to represent features with good generalization ability for different domains. This makes the model of the embodiment of the present application not only perform well on training data, but also achieve good prediction results in unknown domains or new environments, thereby improving the practicality and reliability of the model.

[0052] In summary, the scheme of the embodiment of the present application, through in-depth analysis and improvement of the defects of the prior art, proposes a more advantageous multi-modal mixing transformer neural network, which has high innovation and practicality in the field of physiological data analysis.

[0053] During the development process, the inventors have considered using only a single transformer to simply concatenate the input features for fatigue detection. The inventors have also tried to consider only the information between multi-modal data without separately considering the feature information of single modal data, i.e., without the previously mentioned modal isolation transformer, and only retaining the modal mixing transformer.​

[0054] There is a beta version in this process, that is, a version without modal isolation transformer, only with modal mixing transformer. In this version, the model may only consider the features between multiple modalities, ignoring the feature extraction of a single modality. In this version, since the model is simpler, its volume and operation speed are higher. However, since it does not isolate a single modality, part of the information will be lost in the feature extraction process, resulting in poorer performance than the final proposed version.

[0055] The embodiment of the application provides a multi-modal mixed transformer architecture, introduces a transformer-based neural network architecture, combines the features of multi-modal data, and realizes effective modeling and feature interaction of multi-modal data through modal isolation and modal mixing transformer modules. The embodiment of the application also introduces an adversarial domain generalization module, which enables the neural network to learn a feature representation with good generalization ability for different domains through techniques such as gradient reversal layer, thereby improving the applicability and robustness of the model.

[0056] The implementation details of the embodiments of the application are described in detail below through specific experiments and reasoning processes, so that those skilled in the art can better understand the application.

[0057] In the experiment, the inventors applied MART to the SEED-VIG dataset together with other comparative baselines. The dataset consists of 23 experimental data, each containing 885 samples, including forehead electroencephalogram (f-EEG), temporal and posterior electroencephalogram (tp-EEG), and EOG features (electrooculogram features). The electroencephalogram data obtained by placing 4 electrodes on the forehead is classified as f-EEG, while the remaining data obtained by placing 17 electrodes on the temporal and posterior parts is classified as tp-EEG. The continuous label assigned to each sample varies from 0 to 1, representing PERCLOS indicating a state from wakefulness to drowsiness. The f-EEG and tp-EEG features are differential entropy features processed by a linear dynamic system, divided into 5 frequency bands: δ (1Hz-4Hz), θ (4Hz-8Hz), α (8Hz-14Hz), β (14Hz-31Hz), and γ (31Hz-50Hz). The original vertical EOG and horizontal EOG are obtained through independent component analysis and difference method to obtain EOG features.

[0058] Figure 1An overview of the MART method proposed by the embodiments of the present application is shown. The MART consists of three modules: a multi-view embedding module, a transformer module, and an adversarial domain generalization module. The black solid line represents the forward propagation path, the dashed line represents the partial back propagation path, and the red dashed line represents the back propagation path through the gradient back layer.

[0059] The model MART structure designed by the embodiments of the present application is shown in Figure 1 The model consists of three modules: a multi-view embedding module, a transformer module, and an adversarial domain generalization module.

[0060] 1) Multi-view embedding module: The multi-view embedding module consists of multiple mode-specific multi-view embedding layers. For the i-th modality, the input x i is projected through a linear and nonlinear method by a sigmoid activation function. Two intermediate features are obtained, and their element-wise Hadamard product is calculated to form an embedding sequence E i .

[0061] Before inputting the sequence E i into the transformer, a cls token is added in front of each sequence, and then all the sequences are concatenated into one sequence. In order to add position and modality information, the embodiments of the present application add a position embedding symbol E pos and a modality embedding symbol E mod to the concatenated sequence.

[0062] 2) Transformer module: The transformer module includes two types of transformers: modality isolation transformer and modality mixing transformer, which are connected in turn. The basic components of these two transformers are multi-head self-attention modules.

[0063] The salient feature of the modality isolation transformer and the modality mixing transformer lies in the design of the feed-forward network (FFN). In the modality isolation transformer, each feed-forward network only processes one modality to capture the features of the specific modality. In contrast, in the modality mixing transformer, the structure of the feed-forward network can capture the features and interactions between multiple modalities. The embodiments of the present application integrate L i modality isolation transformer and L m modality mixing transformer into the architecture.

[0064] 3) Adversarial domain generalization module: The adversarial domain generalization module aims to obtain domain-invariant features in different domains. This module adopts an adversarial strategy to cause confusion in the source attribution of each feature, thereby requiring the definition of a domain loss, denoted as L domGradient Reverse Layer (GRL) is a key element in this adversarial domain generalization module. GRL changes the gradient direction in backpropagation, thereby blurring the discrimination of feature source domains, forcing the model to learn domain-irrelevant representations. The loss function of MART is shown in equation (1):

[0065]

[0066] where CE(-,-) denotes the cross-entropy function, and RMSE(-,-) denotes the regression loss function. D i and D^ i denote the real domain label and the predicted domain label. The hyperparameter λ dom plays an important role in weighting the domain loss in the overall model.

[0067] Experiments and results

[0068] The inventors compared the method proposed in the embodiments of the present application with other baseline models in subject-dependent and cross-subject settings. In the subject-dependent setting, the embodiments of the present application used Support Vector Regression (SVR), Multi-Layer Perceptron (MLP) and VIGNet (a deep convolutional neural network for EEG-based driver vigilance estimation). Each model used a five-fold cross-validation method to train and evaluate each subject individually. In the cross-subject setting, the evaluation baselines include SVR, MLP, VIGNet, domain generalization with domain adversarial neural network (DG-DANN) and multi-modal feature space alignment network (MFSAN). The evaluation scheme used a leave-one-out cross-validation approach.

[0069] The inventors used RMSE as the main criterion to evaluate the performance of each model. In addition to RMSE, the inventors introduced PCC and CCC as auxiliary evaluation criteria. PCC is used to quantify the linear correlation between two variables or sequences, with a value ranging from -1 to 1, but PCC can ignore the offset error between two sequences. The inventors also added CCC to consider both correlation and offset between predicted PERCLOS and true PERCLOS. The relationship between CCC and PCC is shown in equation (2):

[0070]

[0071] where σ Y and σ Y^ are the standard deviations of the two sequences, and μ Y and μ Y^ are the mean values of the two sequences.

[0072] During the hyperparameter fine-tuning process, the embedding dimension is selected from {16 x i | i = 2, 3,..., 8}, the depth of the modal isolation transformer L i is fixed at 2, the depth of the modal mixing transformer L m is selected from {1, 2,..., 6}, the number of attention heads is selected from {20, 21,..., 24}, and the learning rate is selected from the interval [10-5, 10-1]. When the activation domain is generalized, the weight of the domain loss L dom ranges from [102, 10-2]. The fine-tuning method is based on the Tree-structured Parzen Estimator (TPE). In each training, when the RMSE does not decrease for 50 consecutive periods, the training is stopped.

[0073] Figure 4 The experimental results are shown, comparing the proposed method and the baseline in terms of RMSE, PCC, and CCC in both subject-dependent and cross-subject settings. The inventors selected the minimum RMSE of each training stage, as well as the corresponding PCC and CCC, and calculated the mean and standard deviation of all training stages. To study the effect of the number of modalities on performance, the inventors recorded the results of the model of the present application and other baseline models under three combinations of learning-based methods.

[0074] In the cross-subject setting, the inventors compared the performance of MART with several existing methods. The inventors evaluated the mentioned existing works using all three modalities. The results of the comparative study are shown in Figure 5 , Figure 5The comparison of the method of the present embodiment with the baselines in RMSE and PCC across subject setups is shown. It is worth noting that CCC is not included in this comparison because existing studies do not use it as an evaluation metric. Existing studies usually use RMSE and PCC as evaluation metrics. The MART method proposed in the present embodiment has superior performance, with an average RMSE of 0.1249, which exceeds existing methods. This finding highlights the effectiveness of the MART model in cross-subject fatigue monitoring.

[0075] Figure 6 The prediction comparison of different methods is shown. As shown in Figure 6 , the black solid line represents the true value of PERCLOS over time, the blue solid line is the prediction value of the method proposed in the present embodiment, and the remaining solid lines are the prediction values of other baselines.

[0076] To visually evaluate the effect of regression, the inventors designated a subject as the test subject in the leave-one-out cross-subject setup. The inventors showed the true PERCLOS of the subject and compared it with the prediction results of the method proposed in the present embodiment, MART, and other baseline models. The prediction results of the method of the present embodiment and other baseline models were generated using the respective optimal hyperparameter set. The comparison of the prediction results of these models with the PERCLOS label is shown in Figure 6 . It can be noted that the RMSE of the method MART proposed in the present embodiment is the lowest compared to the baseline models, so the prediction result is very close to the PERCLOS label.

[0077] The present embodiment introduces a new method of driving fatigue monitoring MART using f-EEG, tp-EEG, and EOG. MART uses a specific modal transformer and an adversarial domain generalization module, which is superior to the baselines in both subject-dependent and cross-subject settings. In the subject-dependent setting, the RMSE of MART is 0.1015, which shows its effectiveness in capturing features. In the cross-subject setting, the RMSE of MART is 0.1556, which exceeds non-transfer learning models. In terms of domain generalization, the RMSE of MART is 0.1249, which exceeds transfer learning methods. When analyzing the impact of including multi-modal data, the inventors observed that the RMSE decreased from 0.1382 to 0.1249 as the number of modalities increased. Visualization of prediction performance supports the superior prediction performance of MART. The method of the present embodiment proves the effectiveness of cross-subject fatigue monitoring.

[0078] In some embodiments, the present application also provides a non-transitory computer readable storage medium storing computer executable instructions, which, when executed by a computer, cause the computer to perform the method of any of the above-mentioned embodiments of the method for detecting fatigue based on electroencephalogram.

[0079] As an implementation form, the non-transitory computer readable storage medium of the present application stores computer executable instructions, which are configured to:

[0080] acquire different modal data of a to-be-detected person using the modal data acquisition unit;

[0081] extract an embedding sequence of the different modal data using the multi-view embedding module;

[0082] perform single-modal feature extraction on the embedding sequence using the modal isolation transformer to obtain at least one single-modal result;

[0083] perform multi-modal feature interaction on the at least one single-modal result using the modal mixing transformer to obtain a multi-modal mixed result;

[0084] perform transfer learning on the multi-modal mixed result using the adversarial domain generalization module so that the model can learn more generalizable feature representations;

[0085] perform fatigue degree calculation on the multi-modal mixed result using the alertness regressor to obtain a current fatigue degree prediction value.

[0086] The non-transitory computer readable storage medium can include a program storage area and a data storage area, wherein the program storage area can store an operating system and at least one application required by a function; the data storage area can store data created according to the use of the device for detecting fatigue based on electroencephalogram, etc. In addition, the non-transitory computer readable storage medium can include a high-speed random access memory, and can also include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some embodiments, the non-transitory computer readable storage medium can optionally include a memory remotely arranged relative to the processor, and these remote memories can be connected to the device for detecting fatigue based on electroencephalogram through a network. Examples of the above-mentioned network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0087] The present application also provides a computer program product, which includes a computer program stored on a non-transitory computer readable storage medium, and the computer program includes program instructions, which, when executed by a computer, cause the computer to perform any of the above-mentioned methods for detecting fatigue based on electroencephalogram.

[0088] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention, such as... Figure 7 As shown, the device includes: one or more processors 710 and memory 720. Figure 7 Taking a processor 710 as an example, the device for fatigue detection based on electroencephalography (EEG) may further include an input device 730 and an output device 740. The processor 710, memory 720, input device 730, and output device 740 can be connected via a bus or other means. Figure 7 Taking a bus connection as an example, the memory 720 is the aforementioned non-volatile computer-readable storage medium. The processor 710 executes various server functions and data processing by running non-volatile software programs, instructions, and modules stored in the memory 720, thereby implementing the brainwave-based fatigue detection method described in the above embodiment. The input device 730 can receive input digital or character information and generate key signal inputs related to user settings and function control of the communication compensation device. The output device 740 may include a display screen or other display device.

[0089] The above-described product can execute the method provided in the embodiments of the present invention, and has the corresponding functional modules and beneficial effects for executing the method. Technical details not described in detail in this embodiment can be found in the method provided in the embodiments of the present invention.

[0090] In one embodiment, the above-described electronic device is applied in a device for EEG-based fatigue detection, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to:

[0091] The modal data acquisition unit is used to acquire different modal data of the person to be tested.

[0092] The multi-view embedding module is used to extract the embedding sequences of the different modal data;

[0093] The modality isolation transformer is used to extract single-modality features from the embedded sequence to obtain at least one single-modality result;

[0094] The modality mixing transformer is used to perform multimodal feature interaction on the at least one single modality result to obtain a multimodal mixing result;

[0095] Using the adversarial domain generalization module to perform transfer learning on the multimodal mixture results enables the model to learn more generalizable feature representations;

[0096] The current fatigue prediction value is obtained by using the alertness regressor to calculate the fatigue level of the multimodal mixture results.

[0097] The electronic device of the embodiments of the present application exists in various forms, including but not limited to:

[0098] (1) Mobile communication device: This kind of device is characterized by having mobile communication function, and providing voice and data communication as the main target. This kind of terminal includes: smart phone (such as iPhone), multimedia phone, functional phone, and low-end phone, etc.

[0099] (2) Ultra mobile personal computer device: This kind of device belongs to the category of personal computer, has computing and processing function, and generally has mobile Internet feature. This kind of terminal includes: PDA, MID and UMPC device, etc., such as iPad.

[0100] (3) Portable entertainment device: This kind of device can display and play multimedia content. This kind of device includes: audio and video player (such as iPod), handheld game console, electronic book, and smart toy and portable car navigation device.

[0101] (4) Server: A device providing computing service, the server is composed of processor, hard disk, memory, system bus, etc., the server is similar to general computer architecture, but since it needs to provide high reliable service, therefore, it has high requirements in processing capability, stability, reliability, security, scalability, manageability, etc.

[0102] (5) Other electronic devices with data interaction function.

[0103] The device embodiments described above are only schematic, wherein the units shown as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, i.e., can be located in one place or distributed on a plurality of network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the embodiments. Those skilled in the art can understand and implement without creative labor.

[0104] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be realized by means of software and necessary general hardware platform, and of course, it can also be realized by hardware. Based on such understanding, the above technical solutions essentially or say the part of the prior art which makes contribution can be embodied in the form of software product, which can be stored in a computer readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., including a plurality of instructions to make a computer device (which can be a personal computer, server, or network device, etc.) execute the method of each embodiment or some part of the embodiment.

[0105] It should be pointed out finally that the above embodiments are only used to illustrate the technical solutions of the present application, but not to limit the same; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that the technical solutions recorded in the foregoing embodiments can still be modified, or some technical features therein can be replaced equivalently; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A fatigue detection model system based on electroencephalography (EEG), comprising: Modal data acquisition unit, used to acquire data of different modalities; The multi-view embedding module includes a dedicated embedding layer for the different modal data, used to extract the embedding sequences of the different modal data using linear and nonlinear methods; A modal isolation transformer is used to extract single-modal features from the embedded sequence to obtain at least one single-modal result, wherein the at least one single-modal result is used as input to a modal mixing transformer; A modal mixing transformer is used to perform multimodal feature interaction on the embedded sequence to obtain a multimodal mixing result; an adversarial domain generalization module is used to perform transfer learning on the multimodal mixing result, so that the model can learn more generalizable feature representations; An alertness regressor is used to calculate fatigue level on the multimodal mixture results. The modal isolation transformer and the modal hybrid transformer are sequentially connected to each other. The basic components of both the modal isolation transformer and the modal hybrid transformer are multi-head self-attention modules. The significant feature of the modal isolation transformer and the modal hybrid transformer lies in the design of the feedforward network. In the modal isolation transformer, each feedforward network processes only one mode and captures the features of a specific mode. In the modal hybrid transformer, the structure of the feedforward network can capture the features and interactions between multiple modes.

2. The system according to claim 1, wherein, The multi-view embedding module also includes modal coding and position information of each modal data in the embedding sequence.

3. The system according to claim 1, wherein, The adversarial domain generalization module includes a gradient inversion layer.

4. A fatigue detection method based on electroencephalography (EEG), using the EEG-based fatigue detection model system according to any one of claims 1-3, the method comprising: The modal data acquisition unit is used to acquire different modal data of the person to be tested. The multi-view embedding module is used to extract the embedding sequences of the different modal data; The modality isolation transformer is used to extract single-modality features from the embedded sequence to obtain at least one single-modality result; The modality mixing transformer is used to perform multimodal feature interaction on the at least one single modality result to obtain a multimodal mixing result; Using the adversarial domain generalization module to perform transfer learning on the multimodal mixture results enables the model to learn more generalizable feature representations; The current fatigue prediction value is obtained by using the alertness regressor to calculate the fatigue level of the multimodal mixture results.

5. The method according to claim 4, wherein, The different modal data include electroencephalogram (EEG) data and electrooculogram (EOG) data, and the modal data acquisition unit includes an EEG data acquisition unit and an EOG data acquisition unit.

6. An electronic device comprising: At least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the steps of the method of claim 4 or 5.

7. A storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the method of claim 4 or 5.

8. A computer program product comprising a computer program stored on a storage medium, the computer program including program instructions that, when executed by a computer, enable the computer to perform the steps of the method of claim 4 or 5.

Images