Fatigue Detection Method Based on Feature Fusion of Electroencephalogram Signals and Electrooculogram Signals

Abstract

The present invention discloses a fatigue detection method based on the feature fusion of electroencephalogram (EEG) signals and electrooculogram (EOG) signals. The EEG signals and EOG signals of each subject in the normal state and the fatigued state are respectively collected to generate training samples. A fatigue detection model is constructed, which includes an EEG signal data dimensionality reduction processing module, a spatial attention feature extraction module, an EEG signal channel and time feature extraction module, an EEG signal feature dimensionality reduction module, an EOG signal time feature extraction module, an EOG signal channel and time feature extraction module, a feature splicing module, and a classification module, and is trained using the training samples. When it is necessary to detect the fatigue of a certain person, the EEG signals and EOG signals are obtained, and the EEG signals are preprocessed by the same method and then input into the trained fatigue detection model together with the EOG signals to obtain the fatigue detection result. The present invention can better extract and retain spatio-temporal features and improve the accuracy of fatigue detection.

Description

Technical Field

[0001] The present invention belongs to the technical field of electroencephalogram signals. More specifically, it relates to a fatigue detection method based on the feature fusion of electroencephalogram signals and electrooculogram signals. Background Art

[0002] In traditional research, fatigue is a state of physical and mental dysfunction and discomfort caused by overuse of the body. This uncomfortable state can affect people's various behaviors in daily life, especially in some operations that require high concentration and sensitive physical reactions, such as driving, flying, and safety activities. According to data from the National Highway Traffic Safety Administration of the United States, there are approximately 100,000 traffic accidents reported by the police each year involving fatigue driving or drowsiness, resulting in more than 1,550 deaths and 71,000 injuries. The risks brought by fatigue driving cannot be ignored, so a large amount of research is needed for the accurate detection of the fatigue state. Currently, there are mainly three mass-produced fatigue driving detection methods: 1) Detect and analyze the operating characteristics of the steering wheel; 2) Analyze the facial features of the driver using a camera; 3) Check the trajectory characteristics of the vehicle. These methods all have good detection effects, but when any one of these algorithms detects the fatigue state of the driver, the driver is already in a relatively serious fatigue state, even when detecting the steering wheel and vehicle trajectory. By the time this is detected, the driver has already caused misoperations that may lead to more serious consequences and cannot predict the driver's fatigue state to avoid danger. Electroencephalogram signals can make up for this part of the omission.

[0003] In recent years, great progress has been made in the research on using electroencephalogram signals to detect people's mental and physical states. Especially in fatigue driving, electroencephalogram signals are called the "gold standard" for fatigue detection. It is a direct, accurate expression of the human fatigue state, more accurate, real-time, and reliable. In 2018, Garnelo et al. proposed a model called neural processes. The neural process network model combines the advantages of neural networks and Gaussian processes. It can effectively learn some features like a neural network, optimize its parameters, make full use of data like a Gaussian process, infer the distribution of data, and effectively analyze data. In 2022, Fang et al. converted electroencephalogram signals into images through methods such as feature frequency band selection, energy calculation, and spatial channel construction, and classified the images through a CNN + LSTM network, achieving good results. The advantage of this method is that it can display the brain activity area through images, more intuitively reflect the changes in brain movement, and is also easier to interpret.

[0004] However, there are also problems such as fatigue judgment criteria when using electroencephalogram (EEG) signals for detecting driver fatigue. Currently, there are two types of fatigue assessments: subjective and objective. Subjective fatigue state is judged by filling out a scale (such as the Karolinska Sleepiness Scale), which can comprehensively analyze a person's state and determine whether the subject is fatigued. However, the judgment data is easily affected by the subject's subjective concept and has a relatively large deviation. To objectively judge the fatigue level of a driver, physiological data such as heart rate, blood pressure, and facial information must be monitored. These information are more objective, but there are still many differences in the threshold division for judging fatigue at present, and in-depth analysis and experiments are needed to provide a more objective, effective, and accurate fatigue detection method. Summary of the Invention

[0005] The object of the present invention is to overcome the deficiencies of the prior art and provide a fatigue detection method based on the feature fusion of electroencephalogram (EEG) signals and electrooculogram (EOG) signals. By adopting multi-stage feature fusion, it can better extract and retain spatio-temporal features and improve the accuracy of fatigue detection.

[0006] To achieve the above object of the invention, the fatigue detection method based on the feature fusion of electroencephalogram (EEG) signals and electrooculogram (EOG) signals of the present invention includes the following steps:

[0007] S1: Determine the test scenario and the subjects according to actual needs, and collect the electroencephalogram (EEG) signals of each subject with a duration of T in the normal state and the fatigued state and electrooculogram (EOG) signals;

[0008] S2: Use a preset preprocessing method to preprocess the electroencephalogram (EEG) signals to obtain electroencephalogram (EEG) signals;

[0009] S3: Take the corresponding electroencephalogram (EEG) signals and electrooculogram (EOG) signals as a group of input signals, and label each group of input signals with a corresponding label label. label = 1 indicates that the subject is in a fatigued state, and label = 0 indicates that the subject is in a normal state, thereby obtaining training samples;

[0010] S4: Construct a fatigue detection model, including an electroencephalogram (EEG) signal data dimensionality reduction processing module, a spatial attention feature extraction module, an electroencephalogram (EEG) signal channel and time feature extraction module, an electroencephalogram (EEG) signal feature dimensionality reduction module, an electrooculogram (EOG) signal time feature extraction module, an electrooculogram (EOG) signal channel and time feature extraction module, a feature splicing module, and a classification module, where:

[0011] The electroencephalogram (EEG) signal data dimensionality reduction processing module is used to perform data dimensionality reduction on the electroencephalogram (EEG) signals, and send the dimensionality-reduced electroencephalogram (EEG) signals 1 to the spatial attention feature extraction module;

[0012] The spatial attention feature extraction module is used to adopt a spatial attention mechanism for the dimensionality-reduced electroencephalogram (EEG) signals1 Perform spatial feature extraction, and send the obtained spatial features t of the EEG signals EEG to the EEG signal channel and time feature extraction module;

[0013] The EEG signal channel and time feature extraction module is used to further extract features from the received spatial features t of the EEG signals EEG and send the obtained EEG signal features f EEG to the EEG signal feature dimensionality reduction module; The EEG signal channel and time feature extraction module includes a first convolutional layer, a second convolutional layer, an average pooling layer, and an activation function layer, where:

[0014] The first convolutional layer is used to perform a convolutional operation on the spatial features t of the EEG signals EEG and then send the obtained features to the second convolutional layer;

[0015] The second convolutional layer is used to perform a convolutional operation on the received features and then send the obtained features to the average pooling layer;

[0016] The average pooling layer is used to perform average pooling on the received features and then send the obtained features to the activation function layer;

[0017] The activation function layer is used to process the received features using the Relu activation function and then output the obtained features as the EEG signal features f EEG for output;

[0018] The EEG signal feature dimensionality reduction module is used to reduce the dimensionality of the received EEG signal features f EEG to obtain the dimensionality-reduced EEG signal features F EEG such that the size of the EEG signal features F EEG is the same as that of the electrooculogram (EOG) signal features F EOG and then send the EEG signal features F EEG to the feature concatenation module;

[0019] The EOG signal time feature extraction module is used to extract time features from the EOG signals and send the obtained EOG signal time features t EOG to the EOG signal channel and time feature extraction module;

[0020] The EOG signal channel and time feature extraction module is used to further extract features from the received EOG signal time features t EOG and send the obtained EOG signal features F EOG to the feature concatenation module; The EOG signal channel and time feature extraction module includes a convolutional layer, an average pooling layer, and an activation function layer, where:

[0021] The convolutional layer is used to perform a convolutional operation on the received EOG signal time features tEOG Perform a convolution operation and send the resulting features to the average pooling layer;

[0022] The average pooling layer is used to perform average pooling on the received features and then send the resulting features to the activation function layer;

[0023] The activation function layer is used to process the received features using the Relu activation function and then use the resulting features as the electrooculogram signal feature F EOG for output;

[0024] The feature splicing module is used to splice the electroencephalogram signal feature F EEG and the electrooculogram signal feature F EOG perform splicing, and send the spliced feature F con to the classification module;

[0025] The classification module is used to classify according to the spliced feature F con to obtain the detection result of whether the person corresponding to the input signal is fatigued;

[0026] S5: Use the training samples in step S3 to train the fatigue detection model to obtain a trained fatigue detection model;

[0027] S6: When it is necessary to detect the fatigue of a certain person, obtain the electroencephalogram signal with a duration of T and the electrooculogram signal EOG′, preprocess the electroencephalogram signal using the same method in step S2 to obtain the electroencephalogram signal EEG′, and then input the electroencephalogram signal EEG′ and the electrooculogram signal EOG′ into the trained fatigue detection model to obtain the fatigue detection result.

[0028] The fatigue detection method based on the fusion of electroencephalogram signal and electrooculogram signal features of the present invention determines the test scenario and the test subject according to actual needs, respectively collects the electroencephalogram signals and electrooculogram signals of each test subject in the normal state and the fatigued state, preprocesses the electroencephalogram signals, and uses the electroencephalogram signals and the corresponding electrooculogram signals as input signals, and labels the corresponding labels, so as to obtain training samples, construct a fatigue detection model including an electroencephalogram signal data dimensionality reduction processing module, a spatial attention feature extraction module, an electroencephalogram signal channel and time feature extraction module, an electroencephalogram signal feature dimensionality reduction module, an electrooculogram signal time feature extraction module, an electrooculogram signal channel and time feature extraction module, a feature splicing module and a classification module, and use the training samples for training. When it is necessary to detect the fatigue of a certain person, obtain the electroencephalogram signal and the electrooculogram signal, preprocess the electroencephalogram signal using the same method and input it into the trained fatigue detection model together with the electrooculogram signal to obtain the fatigue detection result.

[0029] The present invention has the following beneficial effects:

[0030] 1) The present invention comprehensively utilizes the high accuracy of electroencephalogram (EEG) signals and the good cross-subject recognition accuracy of electrooculogram (EOG) signals to detect driver fatigue through a multi-modal method, thereby improving the accuracy of fatigue detection.

[0031] 2) The present invention uses different feature extraction networks for EEG signals and EOG signals to extract features, enabling the extracted features to better reflect the characteristics of EEG signals and EOG signals, thereby further improving the accuracy of fatigue detection. Brief Description of the Drawings

[0032] Figure 1 is a flowchart of the specific implementation of the fatigue detection method based on the feature fusion of EEG signals and EOG signals of the present invention;

[0033] Figure 2 is a structural diagram of the fatigue detection model of the present invention;

[0034] Figure 3 is a schematic diagram of the specific sampling electrode positions of the SEED-VIG dataset;

[0035] Figure 4 is a specific structural diagram of the fatigue detection model in this embodiment;

[0036] Figure 5 is a cross-subject recognition accuracy curve graph of the present invention and two comparison methods in this embodiment. Detailed Description of the Preferred Embodiments

[0037] The following describes the specific implementation of the present invention in conjunction with the drawings, so that those skilled in the art can better understand the present invention. It should be particularly noted that in the following description, when the detailed description of known functions and designs may dilute the main content of the present invention, these descriptions will be omitted here.

[0038] Embodiment

[0039] Figure 1 is a flowchart of the specific implementation of the fatigue detection method based on the feature fusion of EEG signals and EOG signals of the present invention. As Figure 1 shown, the specific steps of the fatigue detection method based on the feature fusion of EEG signals and EOG signals of the present invention include:

[0040] S101: Collect EEG signals and EOG signals:

[0041] Determine the test scenario and the test subjects according to actual needs, and collect EEG signals with a duration of T for each subject in the normal state and the fatigued state, and

[0042] In practical applications, both electroencephalogram (EEG) signals and electrooculogram (EOG) signals are collected using multiple electrodes. The arrangement of the electrodes (especially for EEG signals) has a significant impact on the final fatigue detection effect, and the electrode positions can be determined according to actual needs. Through research, the present invention has found that, for EEG signals, the regions that have a greater impact on the fatigue recognition result are the left rear and right front of the head. The characteristics of the recognition signals of these channels have a great influence on the recognition effect. Therefore, EEG electrodes should be preferentially arranged at the left rear and right front of the head.

[0043] S102: Preprocessing of EEG signals:

[0044] In practical applications, the collection of EEG signals usually adopts non-invasive collection methods. The collected EEG signals usually have a relatively low signal-to-noise ratio and are affected by noises such as power frequency and electromyogram noise. Therefore, it is necessary to use a preset preprocessing method to preprocess the EEG signals to obtain the EEG signals, so as to highlight their characteristics. In this embodiment, the preprocessing method of the EEG signals includes:

[0045] 1) Filter the EEG signals including band-pass filtering and power frequency filtering, and the specific filtering parameters can be set according to actual needs.

[0046] 2) Select the frequency band of the EEG signals according to the actual situation. The human brain mainly has five different brain waves: alpha wave, beta wave, gamma wave, delta wave, and theta wave. In this embodiment, through comparison, the alpha wave, delta wave, and theta wave in the filtered EEG signals are selected, so as to obtain the screened EEG signals.

[0047] 3) Extract differential entropy features:

[0048] Differential entropy (DE) is a generalized form of Shannon information entropy for continuous variables, and its calculation formula is:

[0049]

[0050] where p(x) represents the probability density function of the continuous information x, and [a, b] represents the interval of information values.

[0051] For a specific length of EEG signals that approximately follow a Gaussian distribution the calculation formula of its differential entropy can be expressed as follows:

[0052]

[0053] Calculate the differential entropy of the screened EEG signals, and use the obtained signals as the final EEG signals for use.

[0054] S103: Generate training samples:

[0055] Use the corresponding electroencephalogram (EEG) signal and electrooculogram (EOG) signal as a set of input signals, and label each set of input signals with a label. label = 1 indicates that the subject is in a fatigued state, and label = 0 indicates that the subject is in a normal state, thereby obtaining training samples.

[0056] S104: Construct a fatigue detection model:

[0057] To achieve accurate fatigue detection, when constructing the fatigue detection model of the present invention, a spatial attention architecture is introduced to further extract the EEG signal features of different channels in the spatial signal. At the same time, the architecture idea of DenseNet is also absorbed to fuse multi-stage features for result prediction. Figure 2 is the structural diagram of the fatigue detection model in the present invention. As Figure 2 shown, the fatigue detection model in the present invention includes an EEG signal data dimensionality reduction processing module, a spatial attention feature extraction module, an EEG signal channel and time feature extraction module, an EEG signal feature dimensionality reduction module, an EOG signal time feature extraction module, an EOG signal channel and time feature extraction module, a feature splicing module, and a classification module, where:

[0058] The EEG signal data dimensionality reduction processing module is used to perform data dimensionality reduction on the EEG signal, and send the dimensionality-reduced EEG signal 1 to the spatial attention feature extraction module. As Figure 2 shown, in this embodiment, the EEG signal data dimensionality reduction module includes a convolutional layer, an average pooling layer, and an activation function layer, where:

[0059] The convolutional layer is used to perform convolutional processing on the EEG signal of each channel respectively, and send the obtained feature signals to the average pooling layer.

[0060] The average pooling layer is used to perform average pooling processing on the received feature signals of each channel, and send the obtained feature signals to the activation function layer.

[0061] The activation function layer is used to process the received feature signals using the Relu activation function, and output the obtained feature signals as the dimensionality-reduced EEG signal.

[0062] The spatial attention feature extraction module is used to extract spatial features from the dimensionality-reduced EEG signal using the spatial attention mechanism 1 and send the obtained EEG signal spatial features t EEG to the EEG signal channel and time feature extraction module. As Figure 2As shown in the figure, in this embodiment, the spatial attention feature extraction module includes a max pooling layer, an average pooling layer, and a feature concatenation layer, where:

[0063] The max pooling layer is used to perform max pooling on the downsampled EEG signals, and then send the obtained max pooling features to the feature concatenation layer.

[0064] The average pooling layer is used to average the downsampled EEG signals, and then send the obtained average pooling features to the feature concatenation layer.

[0065] The feature concatenation layer is used to concatenate the max pooling features and the average pooling features, and use the concatenated features as the spatial feature t EEG for output.

[0066] The EEG signal channel and time feature extraction module is used to further extract features from the received EEG signal spatial feature t EEG and send the obtained EEG signal features f EEG to the EEG signal feature downsampling module. As Figure 2 shown in the figure, in this embodiment, the EEG signal channel and time feature extraction module includes a first convolutional layer, a second convolutional layer, an average pooling layer, and an activation function layer, where:

[0067] The first convolutional layer is used to perform a convolutional operation on the EEG signal spatial feature t EEG and then send the obtained features to the second convolutional layer.

[0068] The second convolutional layer is used to perform a convolutional operation on the received features, and then send the obtained features to the average pooling layer.

[0069] The average pooling layer is used to perform average pooling on the received features, and then send the obtained features to the activation function layer.

[0070] The activation function layer is used to process the received features using the Relu activation function, and then use the obtained features as the EEG signal features f EEG for output.

[0071] The EEG signal feature downsampling module is used to downsample the received EEG signal features f EEG to obtain the downsampled EEG signal features F EEG so that the size of the EEG signal features F EEG is the same as that of the electrooculogram signal features F EOG , and then send the EEG signal features F EEG to the feature concatenation module. In this embodiment, the EEG signal feature downsampling module includes a first convolutional layer, a second convolutional layer, an average pooling layer, and an activation function layer, where:

[0072] The first convolutional layer is used to perform a convolution operation on the EEG signal feature f EEG and then send the obtained feature to the second convolutional layer.

[0073] The second convolutional layer is used to perform a convolution operation on the received feature and then send the obtained feature to the average pooling layer.

[0074] The average pooling layer is used to perform average pooling on the received feature and then send the obtained feature to the activation function layer.

[0075] The activation function layer is used to process the received feature using the Relu activation function and then use the obtained feature as the EEG signal feature F after dimensionality reduction EEG for output.

[0076] The EOG signal time feature extraction module is used to extract the time feature of the EOG signal, and send the obtained EOG signal time feature t EOG to the EOG signal channel and time feature extraction module. As Figure 2 shown, in this embodiment, the EOG signal time feature extraction module includes K channel convolutional layers, an average pooling layer, and an activation function layer, where K represents the number of EOG signal channels, and:

[0077] Each channel convolutional layer respectively performs a convolution operation on each channel signal of the EOG signal EOG, and sends the obtained channel feature to the average pooling layer.

[0078] The average pooling layer is used to perform average pooling on the received K channel features and then send the obtained feature to the activation function layer.

[0079] The activation function layer is used to process the received feature using the Relu activation function and then use the obtained feature as the EOG signal time feature t EOG for output.

[0080] The EOG signal channel and time feature extraction module is used to further extract the feature of the received EOG signal time feature t EOG and send the obtained EOG signal feature F EOG to the feature splicing module. As Figure 2 shown, in this embodiment, the EOG signal channel and time feature extraction module includes a convolutional layer, an average pooling layer, and an activation function layer, where:

[0081] The convolutional layer is used to perform a convolution operation on the received EOG signal time feature t EOG and send the obtained feature to the average pooling layer.

[0082] The average pooling layer is used to perform average pooling on the received feature and then send the obtained feature to the activation function layer.

[0083] The activation function layer is used to process the received features using the Relu activation function, and then take the obtained features as the electrooculogram signal features F EOG for output.

[0084] The feature splicing module is used to splice the electroencephalogram signal features F EEG and the electrooculogram signal features F EOG perform splicing, and send the spliced features F con to the classification module.

[0085] The classification module is used to classify according to the spliced features F con to obtain the detection result of whether the person corresponding to the input signal is fatigued. As Figure 2 shown, in this embodiment, the classification module is implemented by using two cascaded fully connected layers and a softmax layer.

[0086] S105: Train the fatigue detection model:

[0087] Use the training samples in step S103 to train the fatigue detection model to obtain a trained fatigue detection model.

[0088] In the process of training the fatigue detection model in this embodiment, the cross-entropy loss function is used for training to determine the closeness between the actual output and the expected output of the fatigue detection. The calculation formula of the cross-entropy loss H(p,q) is:

[0089] H(p,q)=-∑ x (p(x)logq(x))+(1 + p(x)log(1 - q(x)))

[0090] where p(x) represents the expected output of sample x, and q(x) represents the actual output of sample x.

[0091] The Adam optimizer has an extremely fast convergence speed when training neural networks and multi-layer neural networks. Therefore, the Adam optimizer is selected to train the model in this embodiment. The learning rate is 0.001, and the batch size is 30. In addition, since more convolutional layers are used and the number of model layers is large, the dropout rate is set to 0.25 to avoid overfitting.

[0092] S106: Fatigue detection:

[0093] When it is necessary to detect the fatigue of a certain person, an electroencephalogram signal with a duration of T is obtained For the electrooculogram signal EOG′, the electroencephalogram signal is preprocessed using the same method as in step S102 to obtain the electroencephalogram signal EEG′, and then the electroencephalogram signal EEG′ and the electrooculogram signal EOG′ are input into the trained fatigue detection model to obtain the fatigue detection result.

[0094] To better demonstrate the technical effects of the present invention, a specific example is used to experimentally verify the present invention. In this embodiment, the publicly available dataset SEED-VIG released by Shanghai Jiao Tong University in 2017 is used as the test dataset. The data of this dataset was collected through driving simulation, and the collection process was to collect electroencephalogram signals and electroencephalogram signals through a simulation driving system. During the collection process, a screen was placed in front of the subject, showing a four-lane highway scene without unnecessary engines or other components. The subject controlled the vehicle in the software through the steering wheel and pedals, and the scene was synchronously updated according to the subject's operations. In the scene design, a relatively straightforward and monotonous scene was adopted to better induce fatigue driving.

[0095] During the collection process of the SEED-VIG dataset, a neuroscan system was used for data collection. A total of 23 subjects were tested and data were collected. Most of the experiments were carried out in the early afternoon, which is usually the best time for the subjects' lunch breaks because they are likely to induce biological rhythm-related drowsy habits. The entire experiment lasted for two hours. The electroencephalogram signal sampling electrodes were selected from 18 sampling electrode channels such as CP1, CP2, and CPZ in the international 10-20 standard. The electroencephalogram signal and the electrooculogram signal were sampled at 1000 Hz and 200 Hz respectively. Figure 3 It is a schematic diagram of the specific sampling electrode positions of the SEED-VIG dataset.

[0096] The SEED-VIG dataset includes the PERCLOS index synchronously recorded with the eye tracker during the experiment. The PERCLOS index is a value that can effectively characterize a person's drowsiness level obtained through repeated experimental investigations and research by the Carnegie Mellon Institute. Its calculation formula is:

[0097]

[0098] Among them, eye_close_time represents the closed-eye time, and total_time represents the total duration, which is the sum of the blink time, closed-eye time, and saccade time.

[0099] According to the PERCLOS index, the status label of the subject is determined. In this embodiment, the fatigue threshold is set to 0.35. Therefore, the data label can be determined using the following formula:

[0100]

[0101] Thus, a dataset containing electroencephalogram (EEG) signals, electrooculogram (EOG) signals, and data label PERCLOS_label can be obtained.

[0102] Figure 4 is the specific structural diagram of the fatigue detection model in this embodiment. Table 1 is Figure 3 the parameter table of the shown fatigue detection model.

[0103]

[0104]

[0105] Table 1

[0106] In this embodiment, when training the fatigue detection model, the learning rate is set to 0.001, the batch size is set to 30, and the dropout rate is set to 0.25. The experimental device is tested using NVIDIA A100. The Python version is 3.8, the Pytorch version is 1.13.1, and the Cuda version is 11.3.

[0107] To better demonstrate the performance of the present invention, the training results of the fatigue detection model of the present invention are compared with the convolutional neural network (CNN), linear discriminant analysis (LDA), and EEGNet model. Among them, CNN, LDA, and EEGNet are all trained and tested using only the electroencephalogram (EEG) signal alone, while the fatigue detection model of the present invention uses electroencephalogram-oculogram signal fusion, that is, uses two signals, the electroencephalogram (EEG) signal and the electrooculogram (EOG) signal, for training and testing. LDA uses the common spatial pattern algorithm (CSP) to extract features, and then uses these features to train a linear classifier to classify the data. Accuracy and recall are selected to evaluate the performance of the proposed model. Table 2 is the comparison table of the training results of the present invention and the three comparison models in this embodiment.

[0108] Model Intra-class accuracy Intra-class recall Inter-class accuracy CNN 0.87 0.80 0.62 CSP+LDA 0.72 0.72 0.60 EEGNet 0.93 0.87 0.82 The present invention 0.97 0.96 0.87

[0109] Table 2

[0110] As shown in Table 2, the electroencephalogram-oculogram signal fusion model of the present invention is superior to the traditional electroencephalogram-based fatigue detection method in terms of recognition accuracy and cross-subject recognition accuracy. In addition, the decrease in the cross-object recognition accuracy of the electroencephalogram-oculogram fusion model is significantly lower than that of other algorithms, which can prove that the model has strong cross-disciplinary generalization ability.

[0111] Next, ablation experiments are designed for each module in the electroencephalogram-oculogram signal integration model of the present invention to verify the effects of the electroencephalogram signal feature extraction module, the electrooculogram signal feature extraction module, and the spatial attention feature extraction module.

[0112] Table 3 is a performance comparison table of the ablation experiment in this embodiment.

[0113]

[0114] Table 3

[0115] As can be seen from Table 3, the feature extraction modules of EEG signals and EOG signals have greatly improved the fatigue detection accuracy of the fusion model, and the addition of spatial attention has further improved the feature extraction ability of EEG signals. In addition, since the spatial attention feature extraction module effectively improves the effective features of EEG signals between different channels, which provides convenience for the recognition of the final model, the depthwise separable convolution used in the network improves the recognition accuracy.

[0116] To verify the cross-subject recognition ability of the model of the present invention, this embodiment refers to the experimental design of the literature "D. Gao, K. Wang, M. Wang, J. Zhou and Y. Zhang, "SFT-Net: A Network for Detecting Fatigue From EEG Signals by Combining 4D Feature Flow and Attention Mechanism," in IEEE Journal of Biomedical and Health Informatics, doi: 10.1109 / JBHI.2023.3285268." Select the data of one subject sample as the training set, and test the recognition accuracy of the experimental data of other subject samples. This strategy selection method ensures that the test data set is a strictly new data set, making the test process more credible. At the same time, CNN and EEGNet are selected as comparison models in this cross-subject experiment. Figure 5 is a cross-subject recognition accuracy rate curve graph of the present invention and two comparisons in this embodiment. As Figure 5 shown, the average correct rate of cross-subject recognition of the present invention is 87%, and the recognition correct rates of subjects 1, 3, 8, 9, 16, 19, 20, and 23 are all above 90%.

[0117] Although the above describes the illustrative specific embodiments of the present invention for the convenience of those skilled in the art to understand the present invention, it should be clear that the present invention is not limited to the scope of the specific embodiments. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention defined and determined by the appended claims, these changes are obvious, and all inventions made using the concept of the present invention are within the scope of protection.

Claims

1. A fatigue detection method based on the fusion of EEG signal and EOG signal features, characterized in that: The following steps are involved: S1: Determine the test scene and subjects according to actual needs, and collect EEG signals of each subject in normal state and fatigue state for a duration of T and electrooculogram (EOG); S2: Use the preset preprocessing method to process the EEG signal Perform preprocessing to obtain EEG signals; S3: Take the corresponding EEG and EOG signals as a group of input signals, and annotate the label corresponding to each group of input signals, where label = 1 indicates that the subject is in a fatigue state, and label = 0 indicates that the subject is in a normal state, thereby obtaining training samples; S4: construct a fatigue detection model, including an EEG signal data dimensionality reduction processing module, a spatial attention feature extraction module, an EEG signal channel and time feature extraction module, an EEG signal feature dimensionality reduction module, an electrooculogram signal time feature extraction module, an electrooculogram signal channel and time feature extraction module, a feature splicing module and a classification module, wherein: The EEG signal data dimension reduction processing module is used to perform data dimension reduction on the EEG signal, and send the EEG signal after dimension reduction to the spatial attention feature extraction module; The spatial attention feature extraction module is used to extract the spatial features of the EEG signal EEG1 after dimensionality reduction using the spatial attention mechanism, and the obtained EEG signal spatial features t EEG Sent to the EEG signal channel and time feature extraction module; the spatial attention feature extraction module includes the maximum pooling layer, the average pooling layer and the feature concatenation layer, where: The maximum pooling layer is used to perform maximum pooling on the EEG signal after dimensionality reduction, and then send the obtained maximum pooling features to the feature concatenation layer; The average pooling layer is used to average the EEG signals after dimensionality reduction, and then the obtained average pooling features are sent to the feature concatenation layer; The feature concatenation layer is used to concatenate the maximum pooling feature and the average pooling feature, and the concatenated feature is used as the spatial feature t EEG Output; The EEG signal channel and time feature extraction module is used to extract the spatial features of the received EEG signal. EEG Further feature extraction is performed to obtain the EEG signal feature f EEG The EEG signal channel and time feature extraction module includes the first convolution layer, the second convolution layer, the mean pooling layer and the activation function layer, where: The first convolutional layer is used to transform the spatial features of EEG signals t EEG Perform convolution operation and then send the obtained features to the second convolution layer; The second convolutional layer is used to perform convolution operations on the received features, and then send the obtained features to the mean pooling layer; The mean pooling layer is used to perform mean pooling on the received features and then send the obtained features to the activation function layer; The activation function layer is used to process the received features using the Relu activation function, and then the obtained features are used as the EEG signal features f EEG Output; The EEG signal feature dimensionality reduction module is used to receive the EEG signal feature f EEG Perform dimensionality reduction to obtain the reduced dimensionality EEG signal feature F EEG , so that the EEG signal feature F EEG The size and electrooculogram signal characteristics F EOG The same, then the EEG signal feature F EEG Output to the feature stitching module; The electrooculogram signal time feature extraction module is used to extract the time feature of the electrooculogram signal EOG. EOG The signal is sent to the electrooculogram signal channel and time feature extraction module; the electrooculogram signal time feature extraction module includes K channel convolution layers, mean pooling layers and activation function layers, where K represents the number of electrooculogram signal channels, where: Each channel convolution layer performs convolution operation on each channel signal of the electrooculogram signal EOG, and sends the obtained channel features to the mean pooling layer; The mean pooling layer is used to perform mean pooling on the received K channel features, and then send the obtained features to the activation function layer; The activation function layer is used to process the received features using the Relu activation function, and then the obtained features are used as the time features of the electrooculogram signal t EOG Output; The electrooculogram signal channel and time feature extraction module is used to extract the time feature t of the received electrooculogram signal. EOG Further feature extraction is performed to obtain the electrooculogram signal feature F EOG Send to the feature splicing module; the electrooculogram signal channel and time feature extraction module includes a convolution layer, a mean pooling layer and an activation function layer, where: The convolutional layer is used to receive the temporal features of the electrooculogram signal t EOG Perform convolution operation and send the obtained features to the mean pooling layer; The mean pooling layer is used to perform mean pooling on the received features and then send the obtained features to the activation function layer; The activation function layer is used to process the received features using the Relu activation function, and then the obtained features are used as the electrooculogram signal features F EOG Output; The feature concatenation module is used to convert the EEG signal feature F EEG and electrooculogram signal feature F EOG To splice, the splicing feature F con Send to the classification module; The classification module is used to classify the splicing features F con Classify and obtain the detection result of whether the person corresponding to the input signal is fatigued; S5: training the fatigue detection model using the training samples in step S3 to obtain a trained fatigue detection model; S6: When fatigue detection is required for a person, an EEG signal with a duration of T is obtained. The EEG signal is preprocessed with the same method as in step S2 to obtain the EEG signal EEG′, and then the EEG signal EEG′ and the EEG signal EOG′ are input into the trained fatigue detection model to obtain the fatigue detection result.

2. The fatigue detection method according to claim 1, characterized in that: In step S1, the EEG signal electrodes are arranged at the left rear and right front of the head.

3. The fatigue detection method according to claim 1, characterized in that: The method for preprocessing the EEG signal in step S2 comprises the following steps: S2.1: EEG signals Perform filtering, including bandpass filtering and power frequency filtering; S2.2: selecting α wave, δ wave and θ wave in the filtered EEG signal, thereby obtaining the filtered EEG signal; S2.3: Calculate the differential entropy of the screened EEG signal, and use the resulting signal as the final EEG signal.

4. The fatigue detection method according to claim 1, characterized in that: The EEG signal data dimensionality reduction module in step S3 includes a convolution layer, a mean pooling layer and an activation function layer, wherein: The convolution layer is used to perform convolution processing on the EEG signal of each channel respectively, and send the obtained feature signal to the mean pooling layer; The mean pooling layer is used to perform mean pooling processing on the feature signals of each channel received, and send the obtained feature signals to the activation function layer; The activation function layer is used to process the received feature signal using the Relu activation function, and output the obtained feature signal as the EEG signal after dimensionality reduction.

5. The fatigue detection method according to claim 1, characterized in that: The EEG signal feature dimensionality reduction module in step S3 includes a first convolution layer, a second convolution layer, a mean pooling layer and an activation function layer, wherein: The first convolutional layer is used to transform the EEG signal features f EEG Perform convolution operation and then send the obtained features to the second convolution layer; The second convolutional layer is used to perform convolution operations on the received features, and then send the obtained features to the mean pooling layer; The mean pooling layer is used to perform mean pooling on the received features and then send the obtained features to the activation function layer; The activation function layer is used to process the received features using the Relu activation function, and then the obtained features are used as the EEG signal features after dimensionality reduction F EEG Output.

Images