Fatigue classification method based on four-dimensional attention convolutional recurrent neural network

Abstract

The present invention discloses a fatigue classification method based on a four-dimensional attention convolutional recurrent neural network, comprising the following steps: S1, collecting EEG signals and inputting them into a four-dimensional feature extraction module to extract the four-dimensional features of the EEG signals; S2, inputting the extracted four-dimensional features into the attention module to obtain features with space-channel attention; S3, inputting the features with space-channel attention into the convolutional recurrent neural network module to perform fatigue classification. The present invention solves the problem of poor interpretability of neural networks based on EEG signals. It not only improves the accuracy of classification, but also can be used to visualize from the perspective of space and frequency band, thereby improving the interpretability of the network. Using a deep separable convolutional layer, compared with an ordinary convolutional layer, the model size is reduced by about 70%, but the accuracy is increased by 1.44%, and a dual-branch deep separable convolution is proposed, which integrates two scale receptive fields in the processing of spatial information, further improving the accuracy by 0.45%.

Description

Technical Field

[0001] The invention relates to a fatigue classification method based on a four-dimensional attention convolutional recurrent neural network. Background Art

[0002] Driving fatigue is usually caused by excessive activity and lack of rest. It will weaken the driver's ability to control the vehicle and has become one of the main causes of traffic accidents. According to statistics, about 20%-30% of traffic accidents are caused by fatigue driving, and about 60% of people admit to having experienced fatigue driving. According to the data of the National Highway Traffic Safety Administration (NHTSA) of the United States, there are about 100,000 traffic accidents caused by fatigue driving every year. Therefore, reliable fatigue detection has positive significance for traffic safety.

[0003] According to previous studies, there are three main methods for monitoring fatigue driving. The first is a psychology-based method, which usually relies on psychometric questionnaires to assess individual fatigue levels; the second is a video-based method, which usually monitors the driver's operating behavior and physical state, such as facial condition, head position, reaction time, vehicle operation, lane departure, etc.; the third is a physiological-based method, such as the driver's heart rate, breathing, EEG, and respiratory rate. After considering the influence of several uncertain factors such as personal habits, lighting, image acquisition angle, and vehicle condition, physiological signals have great advantages in detecting driver fatigue. Studies have shown that EEG is an effective method for detecting driver fatigue. The advantages of EEG are relatively low acquisition cost, high temporal resolution, and objectivity. In addition, with the rapid development of wearable EEG devices and dry electrode technology, many algorithms based on EEG signals have been proposed and used to analyze and detect driver fatigue.

[0004] Studies have shown that the signal-to-noise ratio of EEG signals is low, so in research, they are usually preprocessed first, such as baseline calibration, bandpass filtering, ICA artifact filtering, manual artifact removal, etc. Since the brain network is a non-stationary system and the causal relationship is nonlinear, many methods have been proposed to extract useful information from EEG signals, such as Fourier transform in frequency domain analysis, wavelet transform in time-frequency analysis, and entropy feature extraction in nonlinear analysis. Finally, the extracted features and fatigue labels are used through machine learning classification models, such as SVM (Support Vector Machine), to perform fatigue classification and obtain classification results.

[0005] In recent years, a variety of deep learning methods have shown high research value in many fields. In particular, in the field of computer vision, the ability of automatic feature learning has achieved great success. Compared with shallow algorithms, deep learning algorithms learn high-level features from raw data by stacking multiple layers. In recent years, some EEG studies have adopted deep learning algorithms. Jingwei et al. applied convolutional neural networks (CNN) to motor imagery tasks. Hajinoroozi et al. tried to predict the cognitive state (drowsiness or alertness) of drivers using wide channel convolutional neural networks (CCNN). Lu et al. used restricted Boltzmann machines (RBMs) for motor imagery tasks. Researchers use the power of deep learning to explore the differences in brain dynamics when performing specific tasks or experiencing changes in cognitive states. However, how to better utilize the knowledge of EEG signals from different models has become a problem. In previous studies, the frequency bands of EEG signals reflect the psychological state. Dissanayaka's research showed that the energy of the alpha band increases when participants are in a sleepy state. Temporal information is also essential in EEG analysis, and the connection of adjacent time frames represents the trend of the brain's dynamic state. The spatial information of EEG channels has been a hot topic in recent years. Studies have shown that cognitive load is closely related to the frontal and parietal lobes. In Hooi's study, the EEG signals were converted into a topographic map, which is a method of incorporating electrode locations into EEG readings. Previous studies have shown a variety of methods for processing multidimensional data for cognitive load assessment. Motion features embedded in spatial and temporal domains are effectively extracted. The performance of 3-D CNN is promising in many applications, but neural networks still have problems in EEG-based fatigue detection, such as insufficient dimensionality of the input feature domain, unreasonable number of model parameters, and poor interpretability. Summary of the invention

[0006] The purpose of the present invention is to overcome the shortcomings of the prior art and provide a fatigue classification method based on a four-dimensional attention convolutional recurrent neural network using a depthwise separable convolutional layer, which reduces the model size by about 70% compared to ordinary convolutional layers and improves the accuracy by 1.44%.

[0007] The objective of the present invention is achieved through the following technical solution: A fatigue classification method based on a four-dimensional attention convolutional recurrent neural network comprises the following steps:

[0008] S1, collecting EEG signals and inputting them into a four-dimensional feature extraction module to extract four-dimensional features of the EEG signals;

[0009] S2, input the extracted four-dimensional features into the attention module to obtain features with spatial-channel attention;

[0010] S3. Input the features with spatial-channel attention into the convolutional recurrent neural network module for fatigue classification.

[0011] Furthermore, the specific implementation method of step S1 is:

[0012] S11, using multiple electrodes to collect EEG signals, and dividing the EEG signals collected by each channel into multiple non-overlapping EEG signal segments;

[0013] S12, divide each EEG signal segment into a window of 0.5 seconds, and then decompose each 0.5 second signal into five frequency bands of δ, θ, α, β and γ using a Butterworth filter, wherein the frequency of the δ band is 0.5-3 Hz, the frequency of the θ band is 4-7 Hz, the frequency of the α band is 8-13 Hz, the frequency of the β band is 14-30 Hz, and the frequency of the γ band is 30-50 Hz;

[0014] S13. Extract differential entropy features from the signals of the five frequency bands respectively:

[0015]

[0016] Among them, σ i represents the standard deviation of the i-th 0.5 second signal;

[0017] For each EEG signal segment, the differential entropy of each frequency band is calculated with a window size of 0.5 seconds, and then the differential entropy of all channels in each frequency band within 0.5 seconds is converted into a two-dimensional matrix to obtain 5 two-dimensional matrices. For each EEG signal segment, a total of 2T five two-dimensional matrices with a window size of 0.5 seconds are obtained; then the signals of all windows in an EEG signal segment are connected in time order to obtain the four-dimensional features of the EEG signal segment, expressed as X n ∈R h×w×d×2T , n = 1, 2, ..., N; where N represents the total number of samples, h and w represent the height and width of the converted two-dimensional matrix, d is the number of frequency bands, and T represents the sampling time.

[0018] Furthermore, the attention module consists of two parts: spatial attention and channel attention. The spatial attention part is first compressed on the channel, and then excited on the spatial part, and finally obtains the feature with spatial attention; the channel attention part first passes through a global pooling layer, two linear layers and an activation function, and then excites on the channel part, and finally obtains the feature with channel attention; finally, the feature with spatial attention and the feature with channel attention are combined to obtain the feature with spatial-channel attention.

[0019] Furthermore, the convolutional recurrent neural network module is composed of a convolutional neural network, a recurrent neural network and a linear classification layer;

[0020] The convolutional neural network includes three depth-wise separable convolutional layers, a dual-branch convolutional layer, two depth-wise separable convolutional layers, an average pooling layer and a linear layer in sequence;

[0021] The dual-branch convolution layer consists of two branches. Each branch performs depth-wise separable convolution on the signals of half the channels. The first branch uses one depth-wise separable convolution layer, and the second branch uses two depth-wise separable convolution layers. Finally, the features output by the two branches are fused and output.

[0022] The recurrent neural network adopts two layers of LSTM;

[0023] The linear classification layer is composed of two linear layers. The two linear layers are used to perform the final output of the features output by the recurrent neural network through linear transformation. The final output is a predicted value. The predicted value is compared with the label threshold to obtain the predicted classification result:

[0024]

[0025] When the predicted value y' is greater than or equal to the preset threshold value P, it is considered as fatigue; otherwise it is considered as wakefulness.

[0026] The beneficial effect of the present invention is that the present invention explicitly integrates the spatial, frequency and temporal information in the EEG signal into a four-dimensional feature information stream, which not only integrates more comprehensive dimensional information, but also provides more intuitive interpretability for the network model. Then, the frequency and spatial information of the four-dimensional feature information stream are further integrated through the attention module, which solves the problem of poor interpretability of the neural network based on EEG signals to a certain extent. It not only improves the accuracy of classification, but also can be visualized from the perspective of space and frequency band, thereby improving the interpretability of the network. In order to solve the problem that the number of parameters and accuracy that usually occur in deep learning cannot be taken into account at the same time, the convolutional recurrent neural network module in this network model uses a deep separable convolution layer. Compared with the ordinary convolution layer, the model size is reduced by about 70%, but the accuracy is increased by 1.44%; and a dual-branch deep separable convolution is proposed, which integrates two scale receptive fields in the processing of spatial information, further improving the accuracy by 0.45%. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 It is a flow chart of the fatigue classification method based on four-dimensional attention convolutional recurrent neural network of the present invention;

[0028] Figure 2 It is a schematic diagram of the four-dimensional feature combination of the present invention;

[0029] Figure 3 is a compact two-dimensional matrix of electrode locations of the present invention;

[0030] Figure 4 Schematic diagram of the working of the attention module of the present invention;

[0031] Figure 5 This is a schematic diagram of the convolution cycle module structure of the present invention;

[0032] Figure 6 Schematic diagram of the depth-separable convolution operation of the present invention;

[0033] Figure 7 Schematic diagram of the structure of the depth-separable convolutional layer of the present invention;

[0034] Figure 8 Schematic diagram of the network structure of the dual-branch convolution module of the present invention;

[0035] Fig. 9 Graphs showing experimental results for different numbers of convolutional layers in this embodiment;

[0036] Fig.10 This is a radar chart of the experimental results of using two convolutional layers in this embodiment;

[0037] Fig.11 A radar chart for comparison of models in this embodiment;

[0038] Fig.12 Spatial attention heatmap under fatigue label;

[0039] Fig.13 Heatmap of spatial attention under the awake label;

[0040] Fig.14 The difference between fatigue and wakefulness is the attention heat map;

[0041] Fig.15 is the channel attention heat map;

[0042] Fig.16 Radar chart of experimental results in five frequency bands. DETAILED DESCRIPTION

[0043] In order to solve the problems of insufficient input feature domain dimension, unreasonable model parameter quantity and poor interpretability of neural network in fatigue detection based on EEG signals, the present invention proposes a new EEG-based four-dimensional attention convolutional recurrent neural network (4D-EACRNN). First, the network uses EEG signals to construct a four-dimensional feature information flow. The four-dimensional information flow explicitly integrates time, space and frequency domain information. The sufficient input dimension information flow makes the network extract features more effectively. Then, the attention module is used to perform attention fusion on the channel and space of the four-dimensional feature information flow respectively. After attention fusion, the four-dimensional information flow has better interpretability. Then, features are extracted through the convolutional recurrent neural network module, in which the convolutional neural network extracts spatial and frequency domain information features, and the recurrent neural network extracts time information features. Finally, fatigue classification is performed through a linear layer.

[0044] The technical solution of the present invention is further described below in conjunction with the accompanying drawings.

[0045] like Figure 1 As shown, a fatigue classification method based on a four-dimensional attention convolutional recurrent neural network (4D-ACRNN) of the present invention comprises the following steps:

[0046] S1, collecting EEG signals and inputting them into a four-dimensional feature extraction module to extract four-dimensional features of the EEG signals;

[0047] In order to simultaneously integrate the frequency, space and time information of EEG signals, the present invention constructs a four-dimensional information flow structure containing these three types of information, such as Figure 2 The specific implementation method is:

[0048] S11. Use multiple electrodes to collect EEG signals from multiple channels. In order to increase the amount of training data, the EEG signals collected by each channel are divided into multiple non-overlapping EEG signal segments, and a label is assigned to each EEG signal segment. The label is the ratio of the eye closure time in the EEG signal segment to the total time, that is, a value from 0 to 1, which can be regarded as the fatigue level of this EEG signal, thereby achieving data + label training.

[0049] S12. Divide each EEG signal segment into a window of 0.5 seconds, and then decompose each 0.5-second signal into five frequency bands of δ, θ, α, β and γ using a Butterworth filter, wherein the frequency of the δ band is 0.5-3 Hz, the frequency of the θ band is 4-7 Hz, the frequency of the α band is 8-13 Hz, the frequency of the β band is 14-30 Hz, and the frequency of the γ band is 30-50 Hz; when the human body performs different activities, the signals of these five frequency bands will show different trends of strengthening or weakening;

[0050] S13. Extract differential entropy features from the signals of the five frequency bands respectively. Differential entropy (DE) has been proved to be the most stable and effective entropy feature in fatigue detection. Differential entropy is Shannon information entropy-∑ x The generalized form of p(x)log(p(x)) for continuous variables is:

[0051]

[0052] Among them, p(x) represents the probability density function of continuous information; and for a certain length, it approximately follows the Gaussian distribution N(μ,σ i 2 ) of the EEG signal, its differential entropy is

[0053]

[0054] Among them, σ i represents the standard deviation of the i-th 0.5 second signal; for example, if the sampling frequency is 200 Hz, 100 signals can be collected in 0.5 seconds, then σ i is the standard deviation of these 100 signals.

[0055] For each EEG signal segment, the differential entropy of each frequency band is calculated with a window size of 0.5 seconds, and then the differential entropy of all channels in each frequency band within 0.5 seconds is converted into a two-dimensional matrix to obtain 5 two-dimensional matrices. For each EEG signal segment, a total of 2T five two-dimensional matrices with a window size of 0.5 seconds are obtained; then the signals of all windows in an EEG signal segment are connected in time order to obtain the four-dimensional features of the EEG signal segment, expressed as X n ∈R h×w×d×2T , n=1,2,…,N where N represents the total number of samples, h and w represent the height and width of the converted two-dimensional matrix respectively, d is the number of frequency bands, and T represents the sampling time; in this embodiment, h is 6, w is 9, and d is 5.

[0056] In order to preserve the spatial information about the electrode locations, Figure 3 The left picture shows the electrode placement of the EEG signal acquisition device. The electrodes in the 6 circles on both sides are located at the temporal part, and the electrodes in the 11 circles in the middle are located at the top. Figure 3The right figure shows the converted two-dimensional matrix, in which the unused electrode positions are represented by 0, and the values ​​of the used electrodes are the calculated differential entropy. According to the converted two-dimensional matrix positions, the two-dimensional matrices of the five frequency bands are converted, and finally superimposed to obtain a three-dimensional matrix. Finally, the three-dimensional matrix of the continuous fragments constitutes the four-dimensional feature representation we want. Among them, the CPZ EEG signal generally selects one channel as the reference electrode (that is, zero potential). Although the signal of the CPZ electrode is collected, it is used as the reference electrode. According to the FT7 row as the first row of the two-dimensional conversion matrix, the O1 row as the last row of the two-dimensional conversion matrix, the FT7 column as the first column of the two-dimensional conversion matrix, and the FT8 column as the last column of the two-dimensional conversion matrix, it constitutes 6 rows and 9 columns, which just accommodates the positions of 17 valid channels, and the empty spaces in the matrix are filled with 0. For specific conversion methods, please refer to the methods in the two articles "Shen F, Dai G, Lin G, et al. EEG-based emotion recognition using 4D convolutional recurrent neural network [J]. Cognitive Neurodynamics, 2020: 1-14" or "Lin CT, Chuang CH, Hung YC, et al. A Driving Performance Forecasting System Based on Brain Dynamic State Analysis Using 4-D Convolutional Neural Networks [J]. IEEE Transactions on Cybernetics, 2020, PP (99): 1-9".

[0057] This four-dimensional feature contains the differential entropy characteristics of the five frequency bands of the EEG signal. A three-dimensional matrix is ​​constructed through the two-dimensional matrix corresponding to the electrode position. Finally, the three-dimensional matrix of each continuous time window constitutes a four-dimensional feature, which contains information in the three domains of time, frequency and space.

[0058] S2, input the extracted four-dimensional features into the attention module to obtain features with spatial-channel attention;

[0059] The attention module was first proposed in the field of image segmentation. It was inspired by the SE module and consists of two parts: spatial attention and channel attention. Figure 4 As shown in Figure 2, this paper uses the attention module to extract features of space and frequency bands at the same time. For EEG signals, the five frequency bands correspond to five channels.

[0060] The attention module consists of two parts: spatial attention (sSE) and channel attention (cSE). The spatial attention part is first compressed on the channel, and then excited on the spatial part, and finally obtains the feature with spatial attention; the channel attention part first passes through a global pooling layer, two linear layers and an activation function, and then excites on the channel part, and finally obtains the feature with channel attention; finally, the feature with spatial attention and the feature with channel attention are combined to obtain the feature with spatial-channel attention, which can further integrate the spatial and frequency band information from the constructed four-dimensional features.

[0061] S3, input the features with spatial-channel attention into the convolutional recurrent neural network module for fatigue classification;

[0062] The convolutional recurrent neural network module consists of a convolutional neural network, a recurrent neural network and a linear classification layer. Figure 5 As shown;

[0063] Convolutional neural networks are quite effective in processing two-dimensional data such as images. Effective high-dimensional features can be extracted by stacking ordinary convolutional layers, which is usually done in the convolutional neural network research of EEG signals. However, as the current EEG signal research pursues real-time performance, and lightweight convolutional layers such as deep separable convolutions emerge, compared to ordinary convolutional layers, they not only have fewer parameters, but also have a certain degree of accuracy increase. In the present invention, the convolutional layers in the convolutional neural network module all use deep separable convolutions.

[0064] The core of the convolutional neural network module is the depthwise separable convolution. The convolutional neural network includes three depthwise separable convolutional layers, one dual-branch convolutional layer, two depthwise separable convolutional layers, one average pooling layer and one linear layer in sequence. The parameter configuration of each layer is shown in Table 1.

[0065] Table 1

[0066] Convolutional Layer Convolution kernel dimension Input size Depthwise Separable Convolutional Layer 1 3x3x5x32 32x5x6x9 Depthwise Separable Convolutional Layer 2 3x3x32x64 32x32x6x9 Depthwise Separable Convolutional Layer 3 3x3x64x128 32x64x6x9 Dual-branch convolutional layer 3x3x128x128 32x128x6x9 Depthwise Separable Convolutional Layer 4 3x3x128x64 32x128x6x9 Depthwise Separable Convolutional Layer 5 3x3x64x32 32x64x6x9 Average Pooling Layer Pooling 2x2 32x32x6x9

[0067] Depthwise separable convolution is a lightweight convolution structure proposed by mobilenet. Depthwise separable convolution replaces a standard convolution with a depthwise convolution and a pointwise convolution. Figure 6As shown, depth-wise convolution is performed first, considering regional information. Then point-wise convolution is performed, considering channel information. Using depth-wise separable convolution to replace ordinary convolution can greatly reduce the number of parameters. From the perspective of neural network, compared with ordinary convolution, although this will sacrifice certain information between channels, the spatial and channel information has been integrated in the attention module. In the present invention, the depth-wise separable convolution layer is composed of sequentially connected depth-wise convolution with a convolution kernel of 3x3, point-wise convolution with a convolution kernel of 1x1, two-dimensional normalization and Relu activation function, as shown in FIG. Figure 7 As shown;

[0068] The dual-branch convolutional layer was first proposed in the field of face recognition and is used to build context to increase the target range of detection. The dual-branch convolutional layer of the present invention consists of two branches, such as Figure 8 As shown in the figure, each branch performs depthwise separable convolution on half of the channels. The first branch uses one depthwise separable convolution layer, and the second branch uses two depthwise separable convolution layers. Finally, the features output by the two branches are fused and output. This structure can fully integrate the spatial information in the four-dimensional features, avoid the isolation of information that is far apart in space, and after passing through two receptive fields of different scales, the information perception effect between different positions in the global space is better.

[0069] The recurrent neural network is a neural network that can learn the time correlation of input data, and can extract the time features between time slices in the four-dimensional feature information stream. The recurrent neural network uses two layers of LSTM, each layer of LSTM includes 2T LSTM modules, and the vertical 2T LSTM modules are one layer of LSTM, a total of two layers of LSTM, and LSTM is used to extract time features. Since EEG signals contain rich time information, the changes between the time slices of the four-dimensional feature structure we constructed may hide additional information, which helps to detect fatigue more accurately. Therefore, after the convolutional neural network extracts the spatial and frequency features of the four-dimensional feature information stream, the two layers of recurrent neural networks with LSTM cells are used to further extract its time features.

[0070] The linear classification layer is composed of two linear layers. The two linear layers are used to perform the final output of the features output by the recurrent neural network through linear transformation. The final output is a predicted value. The predicted value is compared with the label threshold to obtain the predicted classification result:

[0071]

[0072] When the predicted value y' is greater than or equal to the preset threshold P (generally set to 0.35), it is considered fatigue; when it is less than 0.35, it is considered awake.

[0073] The fatigue classification effect of the present invention is further verified by experiments below.

[0074] 1. Dataset introduction: SEED (SJTU Emotion EEG Dataset) is an open EEG dataset provided by the laboratory of the Brain-like Computing and Machine Intelligence Research Center of Shanghai Jiao Tong University. SEED-VIG (SJTU Emotion EEG Dataset-Vigilance Estimation) is a sub-dataset of SEED. It develops a simulated driving environment with straight and monotonous roads, which are more likely to induce fatigue in subjects, and uses a neural scanning system to record EEG signals. There were 23 subjects in the experiment, most of whom were tested after lunch and lasted about 2 hours. The driver's psychological fatigue state is recorded by the percentage of eye closure time within 8 seconds (PERCLOS), which is divided into three types: wakefulness, fatigue and drowsiness, with thresholds of 0.35 and 0.7 respectively.

[0075] The data set contains EEG signals and EOG signals. The present invention selects the original data of EEG signals, with a sampling rate of 200 Hz and a duration of 118 minutes. A total of 17 channel electrode signals are collected. The original EEG signal data format of each subject is sampling number * channel (1416000*17).

[0076] In this experiment, in order to be more in line with the actual situation and better compare with other models, the current EEG fatigue signal research, the classification model is mainly binary classification, so only the threshold of 0.35 is used to divide into awake and fatigue. In order to better evaluate the performance of the model, all experiments are carried out using 5-fold cross validation to reduce the randomness of the experimental results to a certain extent.

[0077] Evaluation indicators: The evaluation indicators use five commonly used classification indicators, namely accuracy, recall, precision, F1 score and Kappa. Accuracy refers to the ratio of the total number of correctly classified samples to the total number of samples; Recall refers to the ratio of the number of correctly classified positive samples to the total number of actual positive samples; Precision refers to the ratio of the number of correctly classified positive samples to the total number of samples predicted as positive samples; F1 score refers to the harmonic mean of recall and precision; Kappa is an indicator of consistency in statistics, and is used to measure whether the model prediction results are consistent with the actual classification results in classification problems.

[0078] Implementation details: The popular pytorch framework is used to implement 4D-ACRNN. ​​The initial learning rate is 2e-3, and the Adamw optimizer is used for training. The decay weight is 2e-2. Each training is performed for 200 epochs, and the round with the highest accuracy of the test set is recorded. Each group is cross-validated with a five-fold cross validation, and the random number seed is fixed to ensure the reliability of the experimental results. However, in the parameter selection experiment, in order to save time, only 100 epochs are trained. All experiments in this article are run on an NVIDIA GeForce GTX 1660SUPER GPU.

[0079] Parameter selection experiment: The number of convolutional layers is an important parameter of convolutional neural networks. To ensure the effectiveness of convolutional recurrent neural networks, experiments were conducted on convolutional neural networks with four different numbers of layers: 4, 5, 6, and 7. The experimental settings are shown in Table 2.

[0080] Table 2

[0081]

[0082] The experimental results are as follows Fig. 9 As shown in the figure, it can be seen from the accuracy, Kappa, and F1 scores of the evaluation indicators that compared with the other three results, the effect of using 6 convolutional layers is relatively good. Although the accuracy is not the highest, the recall rate and precision rate are actually a pair of contradictory metrics. Usually, the two will not be high at the same time. The two indicators cannot be unilaterally viewed to indicate the quality of model classification. The F1 score is the harmonic mean of the two, so the F1 score can be observed instead of observing the recall rate and precision rate separately.

[0083] 2. Ablation experiment: In order to verify the effectiveness of the attention module and dual-branch convolution module in the network model for fatigue classification results, ablation experiments were performed on them respectively.

[0084] Table 3 shows the results of ablation experiments on the model of the present invention, removing the attention module, and removing the dual-branch convolution module.

[0085] Table 3

[0086] Model of the present invention Remove the attention module Remove the double-branch convolution module Accuracy 0.8713 0.8697 0.8668 Recall 0.9215 0.9187 0.9203 Kappa 0.7171 0.7138 0.7066 F1 Score 0.9011 0.8997 0.8978 Accuracy 0.8817 0.8815 0.8765

[0087] In order to verify the effect of the depthwise separable convolutional layer used in the convolutional neural network part of the model, experiments were conducted using ordinary convolutional layers and depthwise separable convolutional layers. The experimental results are shown in Table 4. The radar chart of the evaluation index is shown in Fig.10 shown.

[0088] Table 4

[0089]

[0090] from Fig.10 As can be seen from Table 4, compared with the use of ordinary convolutional layers, the number of model parameters is reduced from 0.462M to 0.14M, and the number of model parameters is reduced by about 70%. On the contrary, the accuracy, F1 score, Kappa and other indicators are further improved. It can be seen that the deep separable convolutional layer is suitable for this model, which can reduce the number of model parameters and improve the performance of the classification model.

[0091] 3. Model comparison experiment: In order to demonstrate the effectiveness of the model of the present invention, this paper compares the four-dimensional attention convolutional recurrent neural network model with four popular models under the SEED-VIG dataset (these four models have no public code, but have specific network structures. We have reproduced these four models). The radar chart of the model comparison is as follows: Fig.11 The experimental results are shown in Table 5.

[0092] Below is a brief introduction to the other four models.

[0093] 1. EEG_Conv: A convolutional neural network model based on EEG signals proposed by Zeng et al.

[0094] 2. EEG_Conv_R: Zeng et al. proposed a new model based on the EEG_Conv model and combined it with the residual structure.

[0095] 3.EEGNET: Lawhern V et al. proposed a compact convolutional neural network for BCI.

[0096] 4. ESTCNN: A spatiotemporal convolutional neural network model based on EEG signals proposed by Gao et al.

[0097] Table 5

[0098] 4D-ACRNN EEG_Conv EEG_Conv_R EEGNet ESTCNN Accuracy 0.8713 0.7948 0.7684 0.7266 0.6407 Recall 0.9215 0.8957 0.8722 0.8792 0.9186 Kappa 0.7171 0.5364 0.4784 0.3654 0.0832 F1 Score 0.9011 0.8475 0.8273 0.8038 0.7639 Accuracy 0.8817 0.8045 0.7874 0.7404 0.6573

[0099] like Fig.11 As shown in Table 5, on the SEED-VIG dataset, our 4D-EACRNN has a more outstanding performance among the five models. It outperforms the other four models in terms of accuracy, recall, Kappa, F1 score and precision, and is 7.65% higher than the accuracy of EEG-Conv.

[0100] This is thanks to our model, which integrates information from three domains: time, space, and frequency, constructs a four-dimensional feature information flow, and further integrates the four-dimensional features through the attention module. After the convolutional neural network is composed of a deep separable convolutional layer, the spatial and frequency features are extracted, and then the time features between the four-dimensional feature time slices are extracted through the recurrent neural network, and finally classified through the linear layer.

[0101] 4. Model interpretability: In the current deep learning research on EEG signals, the interpretability of deep learning has always been a gap that has not been paid attention to. Thanks to the addition of the attention module, our model 4D-EARCNN not only has good performance on the SEED-VIG dataset, but also has a certain degree of interpretability.

[0102] 1. Attention module analysis: The spatial attention in the attention module refers to the 17 electrode positions of the EEG signal, while the channel attention refers to the 5 frequency bands in the EEG signal.

[0103] Spatial attention: By setting the output of the neural network, the spatial attention output of the attention module in the trained model is output, and the spatial attention labeled as fatigue and wakefulness is separated. Finally, the average spatial attention in the two states is obtained and visualized.

[0104] From the perspective of the neural network, the darker the color corresponding to the spatial position, the greater the influence of the EEG signal at that electrode position on fatigue classification. Fig.12 and Fig.13 It can be observed that the left and right temporal electrodes of EEG signals in both fatigue and awake states have a great influence on the classification results. However, in order to better observe the difference between the two, we obtain the difference in spatial attention under the fatigue and awake labels for visualization. Fig.12 As shown, the spatial attention heat map under the awake label is as follows Fig.13 As shown in the figure, the difference between fatigue and wakefulness is shown in the heat map of attention. Fig.14 shown.

[0105] Depend on Fig.14 It can be seen that the EEG signals corresponding to the T7 and T8 electrode positions have a certain influence on the fatigue classification results.

[0106] Channel attention: From the trained model, we can extract the weight value of the channel attention in the attention module, use bilinear interpolation and visualize it as a heat map for easy observation, and correspond the five frequency bands to it. The channel attention heat map is as follows: Fig.15 As shown in the figure, from top to bottom are five frequency bands: δ, θ, α, β and γ. From the perspective of the neural network, the darker the color corresponding to the frequency band, the greater the influence of the frequency band on fatigue classification. It can be seen that as the frequencies of the five frequency bands increase, the influence on fatigue classification also increases. In order to verify the visualization results of the attention module, experiments are conducted on the influence of the five frequency bands on the model classification.

[0107] 2. Channel attention verification experiment: The five frequency bands of δ, θ, α, β and γ in the four-dimensional feature information flow are constructed separately and input into the model. Finally, the evaluation index is used to observe whether it is consistent with the conclusion of channel attention visualization. The experimental results are shown in Table 6 and the radar chart of the experimental results is shown in Fig.16 shown.

[0108] Table 6

[0109] Five frequency bands Delta Band Theta Band Alpha band Beta band Gamma Band Accuracy 0.8713 0.7889 0.8247 0.8331 0.8500 0.8530 Recall 0.9215 0.8619 0.8941 0.9097 0.9122 0.9087 Kappa 0.7171 0.5342 0.6119 0.6257 0.6684 0.6697 F1 Score 0.9011 0.8386 0.8665 0.8737 0.8855 0.8847 Accuracy 0.8817 0.8168 0.8406 0.8405 0.8605 0.8625

[0110] From Table 6 and Fig.16 It can be seen that as the frequency increases, from δ, θ, α, β and γ, the fatigue classification performance is better, which verifies the visualization results in the attention module and proves that the γ frequency band is indeed more important for the fatigue classification results.

[0111] The attention module in the 4D-EACRNN proposed in this paper can not only improve the classification effect of the model, but also make the model interpretable. In this data set, as the frequency of the EEG signal increases, the γ frequency band has a significant impact on the fatigue classification results, and the changes in the EEG signals at the T7 and T8 electrode positions may be important electrode positions for distinguishing fatigue and wakefulness. And the results of classification using five frequency bands are significantly better than using a single frequency band.

[0112] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the principles of the present invention, and should be understood that the protection scope of the present invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific variations and combinations that do not deviate from the essence of the present invention based on the technical revelations disclosed by the present invention, and these variations and combinations are still within the protection scope of the present invention.

Claims

1. A fatigue classification method based on a four-dimensional attention convolutional recurrent neural network, characterized in that: The following steps are involved: S1, collecting EEG signals and inputting them into a four-dimensional feature extraction module to extract four-dimensional features of the EEG signals; S2, input the extracted four-dimensional features into the attention module to obtain features with spatial-channel attention; S3, inputting the features with spatial-channel attention into a convolutional recurrent neural network module for fatigue classification; the convolutional recurrent neural network module is composed of a convolutional neural network, a recurrent neural network and a linear classification layer; The convolutional neural network includes three depth-wise separable convolutional layers, a dual-branch convolutional layer, two depth-wise separable convolutional layers, an average pooling layer and a linear layer in sequence; The dual-branch convolution layer consists of two branches. Each branch performs depth-wise separable convolution on the signals of half the channels. The first branch uses one depth-wise separable convolution layer, and the second branch uses two depth-wise separable convolution layers. Finally, the features output by the two branches are fused and output. The recurrent neural network adopts two layers of LSTM; The linear classification layer is composed of two linear layers. The two linear layers are used to perform the final output of the features output by the recurrent neural network through linear transformation. The final output is a predicted value. The predicted value is compared with the label threshold to obtain the predicted classification result: When the predicted value y' is greater than or equal to the preset threshold value P, it is considered as fatigue; otherwise it is considered as wakefulness.

2. The fatigue classification method based on four-dimensional attention convolutional recurrent neural network according to claim 1 is characterized in that: The specific implementation method of step S1 is: S11, using multiple electrodes to collect EEG signals, and dividing the EEG signals collected by each channel into multiple non-overlapping EEG signal segments; S12, divide each EEG signal segment into a window of 0.5 seconds, and then decompose each 0.5 second signal into five frequency bands of δ, θ, α, β and γ using a Butterworth filter, wherein the frequency of the δ band is 0.5-3 Hz, the frequency of the θ band is 4-7 Hz, the frequency of the α band is 8-13 Hz, the frequency of the β band is 14-30 Hz, and the frequency of the γ band is 30-50 Hz; S13. Extract differential entropy features from the signals of the five frequency bands respectively: Among them, σ i represents the standard deviation of the i-th 0.5 second signal; For each EEG signal segment, the differential entropy of each frequency band is calculated with a window size of 0.5 seconds, and then the differential entropy of all channels in each frequency band within 0.5 seconds is converted into a two-dimensional matrix to obtain 5 two-dimensional matrices. For each EEG signal segment, a total of 2T five two-dimensional matrices with a window size of 0.5 seconds are obtained; then the signals of all windows in an EEG signal segment are connected in time order to obtain the four-dimensional features of the EEG signal segment, expressed as X n ∈R h×w×d×2T , n = 1, 2, ..., N; where N represents the total number of samples, h and w represent the height and width of the converted two-dimensional matrix respectively, d is the number of frequency bands, and T represents the sampling time.

3. The fatigue classification method based on four-dimensional attention convolutional recurrent neural network according to claim 1 is characterized in that: The attention module consists of two parts: spatial attention and channel attention. The spatial attention part is first compressed on the channel, then stimulated on the spatial part, and finally obtains the feature with spatial attention; The channel attention part first passes through a global pooling layer, two linear layers and an activation function, and then is stimulated on the channel part to finally obtain features with channel attention; finally, the features with spatial attention and the features with channel attention are combined to obtain features with spatial-channel attention.

Images