Fatigue detection method based on electroencephalogram and electrooculogram bimodal time-frequency-space fusion features

Abstract

The invention discloses a fatigue detection method based on electroencephalogram and electro-oculogram bimodal time-frequency-space fusion features, and belongs to the technical field of fatigue detection, and the method comprises the following steps: S1, collecting an EEG signal and an EOG signal of a user, and generating artificial features; s2, inputting the artificial features into a network framework to obtain a final feature result; and S3, processing the final feature result by using an activation function, and outputting a fatigue level. According to the method, the characteristics of electroencephalogram and electro-oculogram are effectively fused, compared with a current most advanced model, the prediction performance and noise robustness are remarkably improved, and a new view angle is provided for multi-mode alertness evaluation.

Description

Technical Field

[0001] The present invention belongs to the technical field of fatigue detection, and in particular relates to a fatigue detection method based on the dual-modal time-frequency-space fusion characteristics of electroencephalogram and electrooculogram. Background Art

[0002] In recent years, the incidence of traffic accidents has continued to increase, and fatigue driving is one of the main causes of traffic accidents, which poses a huge threat to road traffic safety. Driver alertness assessment usually involves monitoring indicators such as the driver's reaction time, attention allocation, and cognitive ability. These indicators can effectively reflect the driver's alertness. When the driver is fatigued, these alertness indicators usually drop significantly, thereby increasing the risk of accidents. Alertness is an indicator of the sensitivity of the human brain to external stimuli. Accurately evaluating driver alertness can effectively reduce the occurrence of road safety accidents.

[0003] Currently, several types of methods have been used to assess alertness, including subjective scales, facial expressions, and physiological signals such as electroencephalogram (EEG) signals and electrooculogram (EOG) signals. However, non-biological factors such as subjective scales and facial expressions can have large inter-individual differences due to different cultural backgrounds and personal habits, which may cause unreliability in alertness assessment. Signals such as EEG and EOG are widely used in driver alertness assessment because they are less affected by subjective factors and can record information about alertness.

[0004] According to prior knowledge of neurophysiology, EEG signals can directly reflect the activity of the human brain and capture changes in brain waves caused by fatigue or sleepiness. As the gold standard for fatigue detection, EEG has excellent stability and reliability in diagnosing and predicting driver fatigue, making it one of the most widely used direct measurement tools for evaluating driver alertness. Similarly, EOG is also a widely used physiological signal, which is collected from electrodes around the human eye and contains information about alertness and eye movements (such as blinking and saccades). Compared with EEG, EOG has a relatively high amplitude and is therefore more robust to noise. Many studies have shown that the features extracted from EOG are significantly correlated with fatigue status. At present, a large number of studies have used EEG or EOG signals to evaluate driver alertness, but because fatigue is a complex state, it is difficult to establish an effective and robust driving fatigue detection system based on only one physiological signal. Previous studies have shown that combining multiple electrophysiological signals to evaluate driver alertness is a more effective method.

[0005] EEG and EOG represent internal cognitive states and external subconscious behaviors, respectively. The information reflected by these two modalities is closely related and complementary. Relevant studies have shown that in the task of vigilance assessment, the multimodal features of EEG and EOG have significant advantages over using either one alone. At present, some fusion strategies have been proposed to fuse EEG and EOG, such as feature fusion (FLF) and decision fusion (DLF). These fusion strategies have performed well in the task of vigilance assessment using EEG and EOG as multimodal. However, there are still challenges in using multimodal signals for vigilance assessment: 1) Like other physiological signals, EEG and EOG signals are easily disturbed by the environment and carry noise, and are also affected by artifacts caused by muscle activity, and their fusion performance may be greatly reduced. However, most existing methods do not consider how to develop effective fusion mechanisms to improve the accuracy and robustness of vigilance assessment. 2) Most studies on multimodal vigilance assessment use frequency domain features, time domain features of EEG signals and statistical features of EOG signals. Although these features have been shown to have excellent fatigue characterization performance in vigilance assessment tasks, they ignore the dynamic changes in the relationship between EEG signal channels on a time scale and the time dependence of EOG signals, as well as the deep complementary relationship between the two in fatigue characterization. 3) Currently, most studies use a single feature of multimodal biological signals for vigilance assessment, which cannot capture multidimensional and three-dimensional vigilance characterization information, and can easily cause unstable monitoring performance. Summary of the invention

[0006] In order to solve the above problems, the present invention proposes a fatigue detection method based on the dual-modal time-frequency-space fusion characteristics of EEG and EOG.

[0007] The technical solution of the present invention is: a fatigue detection method based on the dual-modal time-frequency-space fusion characteristics of EEG and EOG includes the following steps:

[0008] S1, collect the user's EEG signal and EOG signal, and generate artificial features;

[0009] S2, input the artificial features into the network framework to obtain the final feature results;

[0010] S3. Use the activation function to process the final feature results and output the fatigue level.

[0011] Furthermore, S1 includes the following sub-steps:

[0012] S11, collecting EEG signals and EOG signals of the user;

[0013] S12, collect power spectrum density and differential entropy of EEG signals;

[0014] S13, extracting a vertical electrooculogram (VEOG) signal and a horizontal electrooculogram (HEOG) signal from the EOG signal;

[0015] S14, calculating continuous wavelet coefficients according to the vertical electrooculogram (VEOG) signal and the horizontal electrooculogram (HEOG) signal;

[0016] S15. The power spectral density and differential entropy of the EEG signal and the continuous wavelet coefficients of the EOG signal are used as artificial features.

[0017] Furthermore, in S12, the power spectral density of the EEG signal The calculation formula is:

[0018] ;

[0019] In the formula, represents the autocorrelation function of the signal, represents the index, Indicates frequency, express, express;

[0020] In S12, the differential entropy of the EEG signal The calculation formula is:

[0021] ;

[0022] In the formula, Represents the standard deviation of the power spectral density of the EEG signal.

[0023] Furthermore, in S14, the continuous wavelet coefficients The calculation formula is:

[0024] ;

[0025] In the formula, represents the standard deviation of the power spectral density of the EEG signal, Indicates time, Represents an exponent.

[0026] Furthermore, S2 includes the following sub-steps:

[0027] S21, using the power spectrum density and differential entropy of the EEG signal to form a joint feature map, inputting the joint feature map into the DSC module to obtain an eight-band feature map;

[0028] S22, using the BAM unit and the SAM unit in the BSAM module to process the eight-band feature map to obtain the output result of the BASM module;

[0029] S23, input the output result of the BASM module into the Cov-LSTM Module module to obtain the output result of the Cov-LSTM Module module;

[0030] S24, input the output result of the Cov-LSTM Module module into the EOG-ConvLSTM Module module to obtain the output result of the EOG-ConvLSTM Module module;

[0031] S25. Input the output result of the EOG-ConvLSTM Module into the EM-MLP Module to obtain the final feature result.

[0032] Furthermore, in S21, the power spectral density and differential entropy of the EEG signal are used to form a joint feature map, the joint feature map is input into the DSC module, and the four frequency band features in the joint feature map are convolved separately using the four convolution kernels in the DSC module to obtain a feature matrix group; the feature matrix group is convolved using the eight convolution kernels in the DSC module to obtain an eight-band feature map.

[0033] Further, S22 includes the following sub-steps:

[0034] S221, input the eight-band feature maps into the MaxPooling layer and AvgPooling layer of the BAM unit respectively;

[0035] S222, input the output results of the MaxPooling layer and the output results of the AvgPooling layer into the ShareMLP layer;

[0036] S223, using a sigmoid activation function to activate the output result of the ShareMLP layer to obtain a first activation result and a second activation result;

[0037] S224, adding the first activation result and the second activation result as the output result of the BAM unit;

[0038] S225. Input the output result of the BAM unit into the SAM unit for processing, and obtain the output of the SAM unit as the output result of the BSAM module.

[0039] Further, in S224, the output result of the BAM unit The expression is:

[0040] ;

[0041] In the formula, represents the activation function, represents the eight-band feature map, represents the average pooling layer, represents the maximum pooling layer, Represents a multi-layer perceptron.

[0042] Furthermore, in S225, the output of the SAM unit The expression is:

[0043] ;

[0044] In the formula, represents the activation function, represents the eight-band feature map, represents the average pooling layer, represents the maximum pooling layer, Represents a convolution operation with a convolution kernel size of 7×7.

[0045] The beneficial effects of the present invention are:

[0046] (1) The present invention designs four different sub-networks to extract multi-domain features from EEG and EOG signals respectively. These sub-networks not only realize feature extraction of EEG and EOG, but also map these features to dimensions that can represent the level of alertness through dimensionality reduction and fully connected layers;

[0047] (2) The present invention uses the LSTM model in conjunction with the frequency band-spatial attention module to capture the dynamic changes in the time scale of the EEG signal and the key features of the frequency domain and spatial distribution; the present invention proposes an LSTM module to effectively capture the time dependency and eye behavior characteristics of the EOG signal;

[0048] (3) The present invention effectively integrates the features of EEG and EOG, and has significant improvements in prediction performance and noise robustness compared to the current most advanced models, providing a new perspective for multimodal alertness assessment. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 This is a flow chart of a fatigue detection method based on the dual-modal time-frequency-space fusion features of EEG and EOG;

[0050] Figure 2 Diagram of the construction process of EEG frequency band features;

[0051] Figure 3 It is the process diagram of depth-wise separable convolution;

[0052] Figure 4 Flowchart for EEG signal covariance matrix calculation. DETAILED DESCRIPTION

[0053] The embodiments of the present invention will be further described below in conjunction with the accompanying drawings.

[0054] like Figure 1 As shown, the present invention provides a fatigue detection method based on the dual-modal time-frequency-space fusion features of EEG and EOG, comprising the following steps:

[0055] S1, collect the user's EEG signal and EOG signal, and generate artificial features;

[0056] S2, input the artificial features into the network framework to obtain the final feature results;

[0057] S3. Use the activation function to process the final feature results and output the fatigue level.

[0058] Firstly, in order to address the inherent challenges of multimodal EEG-EOG alertness assessment in existing methods, namely, the lack of consideration of the dynamic relationship between EEG channels and the difficulty in capturing multi-dimensional and three-dimensional alertness representation information, the present invention adds four sub-networks to the feature coding layer, which are used to learn the multi-feature information of EEG and EOG respectively; finally, the present invention proposes a feature weight fusion mechanism for the EEG and EOG features output by the feature coding layer, and outputs the final alertness level after processing by a multi-layer perceptron.

[0059] In this embodiment of the present invention, S1 includes the following sub-steps:

[0060] S11, collecting EEG signals and EOG signals of the user;

[0061] S12, collect power spectrum density and differential entropy of EEG signals;

[0062] S13, extracting a vertical electrooculogram (VEOG) signal and a horizontal electrooculogram (HEOG) signal from the EOG signal;

[0063] S14, calculating continuous wavelet coefficients according to the vertical electrooculogram (VEOG) signal and the horizontal electrooculogram (HEOG) signal;

[0064] S15. The power spectral density and differential entropy of the EEG signal and the continuous wavelet coefficients of the EOG signal are used as artificial features.

[0065] In the embodiment of the present invention, in S12, the power spectral density of the EEG signal The calculation formula is:

[0066] ;

[0067] In the formula, represents the autocorrelation function of the signal, represents the index, Indicates frequency, express, express;

[0068] In S12, the differential entropy of the EEG signal The calculation formula is:

[0069] ;

[0070] In the formula, Represents the standard deviation of the power spectral density of the EEG signal.

[0071] In the embodiment of the present invention, in S14, the continuous wavelet coefficients The calculation formula is:

[0072] ;

[0073] In the formula, represents the standard deviation of the power spectral density of the EEG signal, Indicates time, Represents an exponent.

[0074] In this embodiment of the present invention, S2 includes the following sub-steps:

[0075] S21, using the power spectrum density and differential entropy of the EEG signal to form a joint feature map, inputting the joint feature map into the DSC module to obtain an eight-band feature map;

[0076] S22, using the BAM unit and the SAM unit in the BSAM module to process the eight-band feature map to obtain the output result of the BASM module;

[0077] S23, input the output result of the BASM module into the Cov-LSTM Module module to obtain the output result of the Cov-LSTM Module module;

[0078] S24, input the output result of the Cov-LSTM Module module into the EOG-ConvLSTM Module module to obtain the output result of the EOG-ConvLSTM Module module;

[0079] S25. Input the output result of the EOG-ConvLSTM Module into the EM-MLP Module to obtain the final feature result.

[0080] In the embodiment of the present invention, Figure 2As shown in the figure, the EEG and EOG signals are divided into 25 sub-bands with a bandwidth of 2 Hz in the whole frequency band. Feature smoothing is implemented on the divided frequency signals based on moving average (MA) and linear dynamic system (LDS), and features with non-repeating 8S windows are extracted, namely PSD-MA, PSD-LDS, DE-MA and DE-LDS.

[0081] In an embodiment of the present invention, in S21, the power spectral density and differential entropy of the EEG signal are used to form a joint feature map, the joint feature map is input into the DSC module, and the four frequency band features in the joint feature map are convolved separately using the four convolution kernels in the DSC module to obtain a feature matrix group; the feature matrix group is convolved using the eight convolution kernels in the DSC module to obtain an eight-band feature map.

[0082] The DSC module is used to increase the dimension of the joint feature map in order to extract more complex high-dimensional feature information. The DSC module consists of two stages: depth-wise convolution and point-wise convolution. In the depth-wise convolution stage, the input PSD and DE joint features are grouped and convolved to keep the number of frequency band features of the feature map unchanged. Four 3x3 convolution kernels are used to process the four frequency band features respectively. Then, the point-wise convolution stage is introduced, and eight 1x1 convolution kernels are used to convolve the feature matrix group. Each convolution kernel outputs a 17x25 feature map, and finally an 8-band feature map is obtained. The depth-wise separable convolution process is as follows: Figure 3 shown.

[0083] In this embodiment of the present invention, S22 includes the following sub-steps:

[0084] S221, input the eight-band feature maps into the MaxPooling layer and AvgPooling layer of the BAM unit respectively;

[0085] S222, input the output results of the MaxPooling layer and the output results of the AvgPooling layer into the ShareMLP layer;

[0086] S223, using a sigmoid activation function to activate the output result of the ShareMLP layer to obtain a first activation result and a second activation result;

[0087] S224, adding the first activation result and the second activation result as the output result of the BAM unit;

[0088] S225. Input the output result of the BAM unit into the SAM unit for processing, and obtain the output of the SAM unit as the output result of the BSAM module.

[0089] The BSAM module is used to further capture the key information between different frequency band features and different spatial distributions of the feature map after dimensionality increase. The BSAM module also contains two sub-modules: the Band Attention Module (BAM) and the Spatial Attention Module (SAM). For BAM, the frequency band dimension of the original feature map is maintained unchanged and its spatial dimension is compressed. This module focuses on the meaningful information in the input feature map. The input feature map passes through the parallel MaxPooling layer and AvgPooling layer, and then passes through the Share MLP module, in which it first compresses the number of frequency band features to 1 / 8 of the original number, and then expands it to the original number of frequency bands. After the sigmoid activation function, two activated results are obtained. The two output results are added element by element to comprehensively utilize the information they carry. This can enhance the expressive power of the joint features of PSD and DE under 25 frequency bands, and help to better capture the relevant features of PSD and DE in multiple sub-bands. Then, a sigmoid activation function is used to obtain the output result of Band Attention, which is multiplied by the original feature map to give the original feature map frequency band attention weight information.

[0090] In the embodiment of the present invention, in S224, the output result of the BAM unit The expression is:

[0091] ;

[0092] In the formula, represents the activation function, represents the eight-band feature map, represents the average pooling layer, represents the maximum pooling layer, Represents a multi-layer perceptron.

[0093] In the embodiment of the present invention, in S225, the output of the SAM unit The expression is:

[0094] ;

[0095] In the formula, represents the activation function, represents the eight-band feature map, represents the average pooling layer, represents the maximum pooling layer, Represents a convolution operation with a convolution kernel size of 7×7.

[0096] In the embodiment of the present invention, the second module is a Cov-LSTM Module. The covariance matrix of the original EEG signal is calculated. The original EEG signal has 17 channels. Therefore, the calculated covariance matrix is ​​a 17*17 matrix, which represents the relationship between the 17 channels. The calculation formula of each element in the covariance matrix is ​​defined as:

[0097] ;

[0098] in, or represents the element of column i or column j, or Represents the expected value of the i-th column or the j-th column. The covariance matrix is ​​a symmetric matrix. In order to reduce the computational burden, its lower triangular matrix is ​​taken and converted into a one-dimensional feature vector. An 8-second sample is divided into 8 consecutive time steps through a 1-second non-overlapping time window, and their covariance matrices are calculated separately to obtain 8 one-dimensional feature vectors. These 8 one-dimensional feature vectors are continuous in time, and a layer of LSTM network is used to capture their temporal dependencies to reveal the dynamic differences in the time scale of inter-channel relationships. The LSTM network of this module contains 8 memory cells, which can update the current cell state through the historical state. By using a common cell state, the cells in the LSTM sequence are able to maintain long-term dependencies. The LSTM cell state is defined as:

[0099] ;

[0100] ;

[0101] ;

[0102] ;

[0103] ;

[0104] in, and Represents the current cell state and the historical cell state, is the cell input, , , , represents the weight, , , , is the deviation, and its calculation process is as follows Figure 4 shown.

[0105] The third module is the EOG-ConvLSTM Module. This module captures the time-dependent characteristics of EOG signals by effectively fusing convolution with LSTM. EOG signals have a high temporal resolution, and their dynamic differences in time scale reflect the pattern changes of eye movements at different levels of alertness. LSTM networks are used to explore the potential information related to alertness. Unlike conventional LSTM networks, ConvLSTM introduces convolution operations at each time step, and the convolution kernels are shared at each time step, which helps to capture the spatial information contained in the time series data and extract similar features. The two-dimensional data composed of vertical and horizontal electrooculogram signals is divided into 8-second samples in the same way as the EEG covariance matrix, and the resulting 8 sub-samples are input into ConvLSTM as continuous time windows to capture spatiotemporal features.

[0106] The fourth module is the EM (Eye Movement)-MLP Module. This module converges the three sets of features calculated from the eye movement features into a value that can reflect the alertness level through the MLP model of the fully connected structure. The MLP consists of an input layer, a hidden layer, and an output layer. The input layer inputs a feature map composed of three sets of eye movement features, the hidden layer represents the features through learning, and the output layer generates feature weights that can ultimately characterize the alertness level. Each neuron in the hidden layer and the output layer has an activation function for introducing nonlinear mapping. Here, the eye movement features are first converted into a one-dimensional vector with 108 elements as the input layer, and the hidden layer is set to one layer. The one-dimensional vector of the input layer is directly linearly mapped to the output layer. This can reduce the amount of calculation while converging the feature map to the feature weight.

[0107] In the entire network framework, different sub-network modules are developed to capture the differentiated characteristics of the signal. Finally, the feature vector calculated by each module is converged to a value representing the alertness through a linear layer, and finally mapped by a sigmoid function, with the value limited to 0-1.

[0108] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the principles of the present invention, and should be understood that the protection scope of the present invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific variations and combinations that do not deviate from the essence of the present invention based on the technical revelations disclosed by the present invention, and these variations and combinations are still within the protection scope of the present invention.

Claims

1. A fatigue detection method based on the dual-modal time-frequency-space fusion features of EEG and EOG, characterized in that: The following steps are involved: S1, collect the user's EEG signal and EOG signal, and generate artificial features; S2, input the artificial features into the network framework to obtain the final feature results; S3. Use the activation function to process the final feature results and output the fatigue level.

2. The fatigue detection method based on the dual-modal time-frequency-space fusion features of EEG and EOG according to claim 1 is characterized in that: The S1 comprises the following sub-steps: S11, collecting EEG signals and EOG signals of the user; S12, collect power spectrum density and differential entropy of EEG signals; S13, extracting a vertical electrooculogram (VEOG) signal and a horizontal electrooculogram (HEOG) signal from the EOG signal; S14, calculating continuous wavelet coefficients according to the vertical electrooculogram (VEOG) signal and the horizontal electrooculogram (HEOG) signal; S15. The power spectral density and differential entropy of the EEG signal and the continuous wavelet coefficients of the EOG signal are used as artificial features.

3. The fatigue detection method based on the dual-modal time-frequency-space fusion features of EEG and EOG according to claim 2 is characterized in that: In S12, the power spectral density of the EEG signal The calculation formula is: ； In the formula, represents the autocorrelation function of the signal, represents the index, Indicates frequency, express, express; In S12, the differential entropy of the EEG signal The calculation formula is: ； In the formula, Represents the standard deviation of the power spectral density of the EEG signal.

4. The fatigue detection method based on the dual-modal time-frequency-space fusion features of EEG and EOG according to claim 2 is characterized in that: In S14, the continuous wavelet coefficients The calculation formula is: ； In the formula, represents the standard deviation of the power spectral density of the EEG signal, Indicates time, Represents an exponent.

5. The fatigue detection method based on the dual-modal time-frequency-space fusion features of EEG and EOG according to claim 1 is characterized in that: The S2 comprises the following sub-steps: S21, using the power spectrum density and differential entropy of the EEG signal to form a joint feature map, inputting the joint feature map into the DSC module to obtain an eight-band feature map; S22, using the BAM unit and the SAM unit in the BSAM module to process the eight-band feature map to obtain the output result of the BASM module; S23, input the output result of the BASM module into the Cov-LSTM Module module to obtain the output result of the Cov-LSTM Module module; S24, input the output result of the Cov-LSTM Module module into the EOG-ConvLSTM Module module to obtain the output result of the EOG-ConvLSTM Module module; S25. Input the output result of the EOG-ConvLSTM Module into the EM-MLP Module to obtain the final feature result.

6. The fatigue detection method based on the dual-modal time-frequency-space fusion features of EEG and EOG according to claim 5 is characterized in that: In the S21, the power spectral density and differential entropy of the EEG signal are used to form a joint feature map, the joint feature map is input into the DSC module, and the four frequency band features in the joint feature map are convolved respectively using the four convolution kernels in the DSC module to obtain a feature matrix group; the feature matrix group is convolved using the eight convolution kernels in the DSC module to obtain an eight-band feature map.

7. The fatigue detection method based on the dual-modal time-frequency-space fusion features of EEG and EOG according to claim 5 is characterized in that: The S22 comprises the following sub-steps: S221, input the eight-band feature maps into the MaxPooling layer and AvgPooling layer of the BAM unit respectively; S222, input the output results of the MaxPooling layer and the output results of the AvgPooling layer into the ShareMLP layer; S223, using a sigmoid activation function to activate the output result of the ShareMLP layer to obtain a first activation result and a second activation result; S224, adding the first activation result and the second activation result as the output result of the BAM unit; S225. Input the output result of the BAM unit into the SAM unit for processing, and obtain the output of the SAM unit as the output result of the BSAM module.

8. The fatigue detection method based on the dual-modal time-frequency-space fusion features of EEG and EOG according to claim 7 is characterized in that: In S224, the output result of the BAM unit The expression is: ； In the formula, represents the activation function, represents the eight-band feature map, represents the average pooling layer, represents the maximum pooling layer, Represents a multilayer perceptron.

9. The fatigue detection method based on the dual-modal time-frequency-space fusion features of EEG and EOG according to claim 7 is characterized in that: In S225, the output of the SAM unit The expression is: ； In the formula, represents the activation function, represents the eight-band feature map, represents the average pooling layer, represents the maximum pooling layer, Represents a convolution operation with a convolution kernel size of 7×7.

Images