Electroencephalogram fatigue recognition method based on multi-scale convolution and double attention mechanism

Abstract

The invention discloses an electroencephalogram fatigue recognition method based on multi-scale convolution and a double attention mechanism, and belongs to the technical field of electroencephalogram signal analysis and fatigue detection. The method comprises the following steps: carrying out preprocessing and data integration on multi-channel electroencephalogram signals, extracting sub-band energy by adopting wavelet packet decomposition, and mapping the sub-band energy to four frequency bands of delta, theta, alpha and beta; constructing a data matrix of channel * frequency band * time step, and inputting the data matrix into a multi-scale convolutional neural network to extract time domain features; a space attention mechanism is introduced, and key channel response is enhanced; time sequence characteristics are modeled through an LSTM network, a frequency attention mechanism is added, and a fatigue related frequency band is highlighted. According to the method provided by the invention, the modeling capability of time dependence and spatial frequency band correlation of the electroencephalogram signals is remarkably enhanced, and deep mining of fatigue state characteristics is realized. The model can realize efficient and accurate fatigue recognition without priori knowledge, and is suitable for an electroencephalogram intelligent monitoring scene in a complex environment.

Description

Technical Field

[0001] The present invention relates to the technical field of electroencephalogram (EEG) signal analysis and fatigue detection, and in particular to an EEG fatigue recognition method based on multi-scale convolution and dual attention mechanisms. Background Art

[0002] Human fatigue is a significant factor affecting cognitive function, work efficiency, and safe operation. This is particularly true when working for long hours, experiencing mental overload, or experiencing sleep deprivation. It can easily lead to decreased attention and delayed reactions, leading to safety accidents and health damage. Statistics show that fatigue causes numerous accidents annually in industrial production, transportation, and high-risk operations, resulting in high injury and mortality rates. Therefore, accurately identifying and warning of human fatigue is crucial for ensuring personal safety and improving productivity.

[0003] Traditional fatigue detection methods are primarily categorized into subjective evaluation and objective measurement. Subjective evaluation includes driver self-assessment questionnaires and response feedback; objective measurement methods include those based on multiple sources of information, including physiological signals, driving behavior, vehicle dynamics, and visual images. For example, visual feature-based methods use cameras to monitor the driver's facial expressions and eye closure (e.g., the PERCLOS indicator) to determine fatigue, but these methods are significantly affected by lighting and occlusion. Driving state-based methods monitor vehicle trajectory and steering wheel rotation, but are also susceptible to interference from the road and driving style. Physiological signal detection uses parameters such as heart rate, electrodermal conduction, and electroencephalogram (EEG) to reflect fatigue, offering the advantage of objectivity. However, EEG instruments are complex to use and susceptible to motion artifacts. Existing research has employed a variety of classification algorithms, such as support vector machines (SVMs), artificial neural networks (ANNs), and Bayesian classifiers, to classify fatigue states. However, these methods rely on manual feature extraction and have limited generalization capabilities.

[0004] Compared with the above methods, fatigue detection based on electroencephalogram (EEG) signals can directly measure the brain's neural electrical activity, providing a more direct and objective fatigue indicator. EEG signals have high temporal resolution and rich time-frequency characteristics, and can detect subtle changes in cerebral cortical activity before the driver shows obvious external signs of fatigue. However, due to the low signal-to-noise ratio of EEG signals themselves and large individual differences, how to effectively extract their features has always been a research difficulty. In recent years, deep learning technology has been widely used in the field of EEG signal processing. Models such as convolutional neural networks (CNN) and long short-term memory networks (LSTM) can automatically learn multi-scale spatiotemporal features, effectively improving the accuracy and robustness of EEG fatigue recognition. Therefore, a deep learning model that combines multi-scale convolution and attention mechanism is designed to accurately identify EEG fatigue. Summary of the Invention

[0005] (1) Technical problems solved

[0006] In response to the problems in the existing technology of low accuracy in EEG fatigue recognition and poor feature extraction effect, the present invention proposes an EEG fatigue recognition method based on multi-scale convolution and dual attention mechanism, which solves the problems raised in the above background technology.

[0007] (2) Technical solution

[0008] In order to achieve the above-mentioned purpose, the present invention specifically adopts the following technical solutions:

[0009] A method for identifying EEG fatigue based on multi-scale convolution and dual attention mechanism, characterized by comprising the following steps:

[0010] S1: EEG data preprocessing;

[0011] S2: Extract EEG signal data features;

[0012] S3: Set the model structure and training parameters;

[0013] S4: Train the model and output the results.

[0014] Furthermore, the EEG data preprocessing in step S1 includes: downsampling, filtering, and artifact removal. The sampling frequency of the EEG device is too high, so the sampling frequency of all channels is first downsampled to 128Hz; by reasonably filtering and baseline correction of the signal, power supply interference, low-frequency drift and other clutter are eliminated; independent component analysis (ICA) is used to remove blinking and electromyographic artifacts to obtain a clean EEG signal.

[0015] Furthermore, the step S2 of extracting EEG signal data features includes:

[0016] Wavelet packet decomposition and energy feature extraction: The preprocessed EEG signal is decomposed into 32 sub-bands using Daubechies 4 (db4) wavelet. The energy feature of each sub-band is calculated, and the energy of the i-th sub-band is defined as:

[0017]

[0018] Among them, E i For its energy characteristics, C i，j For each sub-band decomposition coefficient, i represents the sub-band index, j represents the jth wavelet coefficient, and N i It represents the total number of wavelet coefficients in the ith sub-band, and reflects the energy distribution of EEG signals in each sub-band through energy characteristics;

[0019] Frequency band fusion and data matrix construction: According to the physiological significance of each EEG frequency band, the 32 sub-bands are combined into the classic four frequency bands (δ, θ, α, β). Specifically, the sub-band coefficients belonging to the same frequency band are merged to reconstruct the four frequency band signals. The frequency band signals of each of the eight EEG channels are cropped or interpolated to obtain a time series of length 768. Finally, the data is organized into a three-dimensional matrix M of 8×4×768, where 8 is the number of channels, 4 is the number of frequency bands, and 768 is the time step, providing multi-channel time-frequency features for subsequent deep network input.

[0020] Furthermore, the model structure constructed in step S3 includes:

[0021] Multi-scale convolution feature extraction: A parallel multi-scale convolutional network structure is designed to extract time-domain features from the matrix M. Three one-dimensional convolution kernels of different sizes are set, such as lengths of 3, 5, and 7, corresponding to different receptive fields to capture time-domain information of different granularities. Each convolution branch outputs several channels, such as 16 output channels. After batch normalization and activation function processing, the output feature maps of the parallel branches are spliced ​​in the channel direction to obtain multi-scale fusion feature representations. After the multi-scale convolution, a spatial attention mechanism is introduced to weight the channel features of the convolution output. A method similar to channel attention is used to calculate the weight coefficient of each feature channel through operations such as global pooling, and the feature maps after multi-scale convolution are weighted by channel to generate feature maps with enhanced channel dependence, so that the network pays more attention to the spatial channel information related to EEG fatigue.

[0022] The feature map after spatial attention weighting is expanded into a sequence according to the time step and input into the long short-term memory network LSTM. A two-layer LSTM network structure is adopted, with 128 hidden units in each layer. The feature vector of each time step is processed to capture the temporal change law. Through the LSTM network, the dynamic change characteristics of the EEG signal can be mined from the time dimension. After the LSTM output, the frequency attention module is introduced to weight the features of different frequency bands.

[0023] After dividing the features of the LSTM output by frequency band, a fully connected network is used to generate attention weights for the four frequency bands. The frequency band features of the original feature map are weighted by multiplication to highlight the frequency band information that is more strongly associated with the fatigue state and suppress the noise of other frequency bands. The features processed by the dual attention mechanism are input into the fully connected classification layer, and finally the Softmax function is used to output the classification results. For the two-category fatigue / awake classification problem, when the output vectors of the fully connected layer are z1 and z2, Softmax is used to calculate the probability of each class:

[0024]

[0025] Among them, P iThat is, the probability of predicting the i-th category is used to judge the driver's fatigue state based on the output probability.

[0026] Furthermore, the model is trained and the results are output in step S4, including: inputting the extracted feature data into the constructed neural network model to realize the discrimination learning of fatigue state; performing segmentation processing on the EEG signal according to the above steps, constructing the corresponding multi-input model structure, and using the five-fold cross-validation method to train and evaluate the model, while selecting Cross Entropy as the loss function to measure the prediction error of the model during the training process, and using the adaptive moment estimation optimization algorithm Adam Optimizer to iteratively optimize the model parameters.

[0027] (3) Beneficial effects

[0028] Compared with the existing technology, the present invention provides a method for EEG fatigue recognition based on multi-scale convolution and dual attention mechanism, which has the following beneficial effects:

[0029] The present invention uses the design of multi-scale convolution structure and dual attention mechanism to enable the model to capture the fatigue characteristics of EEG signals from multiple angles: multi-scale convolution supplements the shortcomings of ordinary single-scale convolution and improves the feature extraction capability; spatial attention and frequency attention respectively enhance the attention to key channels and key frequency bands, thereby effectively improving the accuracy and robustness of fatigue recognition. Compared with the existing technology, the method of the present invention can extract EEG fatigue-related features more comprehensively and accurately, significantly improving the detection performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 It is a schematic diagram of the overall process of the present invention;

[0031] Figure 2 is the EEG channel diagram of the present invention;

[0032] Figure 3 Schematic diagram of wavelet packet decomposition of the present invention. DETAILED DESCRIPTION

[0033] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0034] Example

[0035] like Figure 1The figure shows the overall process diagram of the present invention, which shows the complete processing path from EEG signal input to fatigue state output. The overall process includes five main modules: EEG signal acquisition, preprocessing, feature extraction, model setting, and training and result output. Among them, the EEG signal acquisition module is used to obtain original multi-channel EEG data; the preprocessing module includes downsampling, bandpass filtering and artifact removal steps, which are respectively used to reduce data redundancy, retain effective frequency components (0.5-45Hz) and remove interference signals such as eye movement and electromyography; the feature extraction module includes wavelet packet decomposition and frequency band fusion, first decomposing the signal into 32 sub-bands, and then mapping and merging them into four classic frequency band features of δ, θ, α, and β to construct a channel-frequency band-time three-dimensional data matrix; the model setting module includes a multi-scale convolutional neural network, a spatial attention module, an LSTM module and a frequency band attention module, which extract multi-scale features, strengthen key channel and frequency band information, and model temporal features in turn; the training and output module performs deep learning training on the above features and completes the final recognition and classification of fatigue state.

[0036] A method for identifying EEG fatigue based on multi-scale convolution and dual attention mechanism, specifically comprising the following steps:

[0037] S1: EEG data preprocessing;

[0038] like Figure 2 The EEG channel diagram of the present invention is shown, using eight electrodes from the 10-20 standard lead set by the International Electroencephalography Society: P1, PZ, P2, PO2, PO3, PO4, O1, and O2. This paper analyzes fatigue detection on these eight channels of EEG data. First, each channel signal is downsampled to 128Hz to reduce computational complexity. A sixth-order Butterworth bandpass filter with a passband of 0.5–45Hz is then used, along with bidirectional digital filtering to achieve zero-phase response. Power supply interference and myoelectric noise are filtered out, and baseline drift is removed after filtering. Independent component analysis (ICA) is then applied to decompose the filtered signal. Artifacts such as blinks and eye movements are identified and removed from the resulting independent components, ultimately reconstructing a clean EEG signal.

[0039] S2: Extract EEG signal data features;

[0040] The pre-processed EEG signal is decomposed by db4 wavelet packet. The decomposition process is as follows: Figure 3 As shown, starting from the frequency range of 0 to 64 Hz, the decomposition layer is 5 layers, and 2 5 =32 sub-bands. The energy of the ith sub-band is defined as

[0041]

[0042] Among them, E i For its energy characteristics, C i,j is the decomposition coefficient of each sub-band (i represents the sub-band index, j represents the jth wavelet coefficient), N i Represents the total number of wavelet coefficients in the ith sub-band. After calculating the energy of all 8 channels and 32 sub-bands, these sub-bands can be divided into the four classic frequency bands of EEG analysis according to the frequency range: δ (0.5-4Hz), θ (4-8Hz), α (8-13Hz), β (13-30Hz), etc. The sub-band coefficients belonging to the same frequency band are merged to restore the time domain signal of the corresponding frequency band. The sub-band energy of each frequency band is directly averaged.

[0043]

[0044] in is the sub-band number set corresponding to the frequency band x∈{δ,θ,α,β}, n x is the number of sub-bands contained in the frequency band, E x For each channel, 768 sampling points are retained in each of the four frequency bands, forming a three-dimensional data matrix M of size 8×4×768, where the dimension 8 is the number of channels, 4 is the number of frequency bands, and 768 is the time points.

[0045] S3: Set the model structure and training parameters;

[0046] The data matrix M is input into the multi-scale convolutional network. Specifically, this embodiment sets three parallel branches, using one-dimensional convolution kernels of length 3, 5, and 7 respectively (convolution is performed along the time dimension), and each convolution branch outputs a feature map of 16 channels. After each convolution layer, batch normalization and ReLU activation are performed in sequence, and then the outputs of the three branches are spliced ​​in the channel dimension to obtain a multi-scale fusion feature map containing 16×3=48 channels, maintaining the size of 8×4×768. The expression of each convolution branch is as follows:

[0047] F (i) =ReLU(BN(W i *X+b i )),i=1,2,3

[0048] Among them, W i is the convolution kernel of the i-th branch, b i is the bias term, * represents a one-dimensional convolution operation, BN is the batch normalization function, and ReLU is the activation function. The three branches are output and then channel-joined:

[0049] F concat =Concat(F (1) ,F (2) ,F(3) )

[0050] Finally, the fusion feature map is obtained

[0051] The spatial attention mechanism is introduced after the multi-scale convolution output. The specific implementation is: the feature map of the multi-scale convolution output First, global average pooling and maximum pooling are performed to obtain two 48×1×1 description vectors, which are mapped and added through a shared fully connected layer, and then a spatial attention weight matrix is ​​generated after Sigmoid activation. Then F and A s Channel-by-channel multiplication:

[0052]

[0053] in Denotes the channel-by-channel element multiplication, and the weighted feature map F' is obtained. This step strengthens the key channel features and suppresses redundant information.

[0054] The attention-weighted feature map F' is then expanded into a sequence by time step, with each time step containing 48 features. This is then fed into a two-layer LSTM network (with 128 hidden units per layer). The LSTM models the feature sequence in the temporal dimension, outputting new hidden state features at each time step. By learning long-term and short-term dependencies, the LSTM is able to capture the temporal dynamics of EEG signals.

[0055] The output of LSTM is then reorganized into 4 frequency bands per channel, and attention weighting is applied to the 4 frequency bands. The specific method is: the fully connected sub-network outputs 4 weight coefficients And perform Softmax normalization:

[0056]

[0057] where a k is the attention score of the kth frequency band, α k is the normalized attention weight. The normalized weight is applied to the corresponding frequency band feature vector to complete the frequency band weighted fusion.

[0058] The features weighted by the dual attention are flattened and input into the fully connected layer for classification. The last layer uses Softmax activation to output the probability of each category. If a binary classification (fatigue and wakefulness) is used, when the output vector of the fully connected layer is [z1,z2], the Softmax probability calculation formula is

[0059]

[0060] Among them, P iThat is the probability of predicting the i-th category.

[0061] S4: Train the model and output the results;

[0062] The extracted feature data is input into the constructed neural network model to achieve discrimination learning of fatigue status. Specifically, the EEG signal is segmented according to the above steps, the corresponding multi-input model structure is constructed, and the model is trained and evaluated using a five-fold cross-validation method. The five-fold cross-validation includes the following steps: first, the complete data set is randomly divided into five subsets of equal size, ensuring that each subset is consistent with the overall data set in terms of sample size and category distribution; then, in each round of training, one of the subsets is selected as the test set, and the remaining four subsets are selected as the training set, and all five validations are completed in turn. During the training process, to improve the model's learning effect and classification performance, the following training hyperparameters were set: the learning rate was set to 0.001, the number of training epochs was 50, and the number of samples processed per batch (batch size) was 64. At the same time, cross entropy was selected as the loss function to measure the model's prediction error during training, and the adaptive moment estimation optimization algorithm (Adam Optimizer) was used to iteratively optimize the model parameters to achieve accurate classification of fatigue and non-fatigue states.

[0063] In summary, the present invention provides a complete EEG fatigue recognition process based on multi-scale convolution and dual attention mechanisms. Each step, from signal preprocessing and spectral feature extraction to deep learning classification, has been meticulously designed: wavelet packet feature extraction provides rich time-frequency information, a multi-scale convolutional network automatically learns multi-resolution time-domain features, and spatial and frequency attention complement each other to enhance key channel and frequency band information, significantly improving recognition performance. These steps work together to achieve an end-to-end process from raw EEG data to fatigue classification results.

[0064] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art will be able to modify the technical solutions described in the aforementioned embodiments or substitute equivalents for some of the technical features. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the present invention.

Claims

1. A method for EEG fatigue recognition based on multi-scale convolution and dual attention mechanism, characterized in that: The specific steps include: S1: EEG data preprocessing; S2: Extract EEG signal data features; S3: Set the model structure and training parameters; S4: Train the model and output the results.

2. The method for identifying EEG fatigue based on multi-scale convolution and dual attention mechanism according to claim 1, characterized in that: The EEG data preprocessing in step S1 includes: downsampling, filtering, and artifact removal. If the sampling frequency of the EEG device is too high, the sampling frequency of all channels is first downsampled to 128 Hz; by reasonably filtering the signal and performing baseline correction, power supply interference, low-frequency drift and other clutter are eliminated; independent component analysis (ICA) is used to remove blinking and electromyographic artifacts to obtain a clean EEG signal.

3. The method for identifying EEG fatigue based on multi-scale convolution and dual attention mechanism according to claim 1, characterized in that: The step S2 of extracting EEG signal data features includes: Wavelet packet decomposition and energy feature extraction: The preprocessed EEG signal is decomposed into 32 sub-bands using Daubechies 4 (db4) wavelet. The energy feature of each sub-band is calculated, and the energy of the i-th sub-band is defined as: Among them, E i For its energy characteristics, C i，j For each sub-band decomposition coefficient, i represents the sub-band index, j represents the jth wavelet coefficient, and N i It represents the total number of wavelet coefficients in the ith sub-band, and reflects the energy distribution of EEG signals in each sub-band through energy characteristics; Frequency band fusion and data matrix construction: According to the physiological significance of each EEG frequency band, the 32 sub-bands are combined into the classic four frequency bands (δ, θ, α, β). Specifically, the sub-band coefficients belonging to the same frequency band are merged to reconstruct the four frequency band signals. The frequency band signals of each of the eight EEG channels are cropped or interpolated to obtain a time series of length 768. Finally, the data is organized into a three-dimensional matrix M of 8×4×768, where 8 is the number of channels, 4 is the number of frequency bands, and 768 is the time step, providing multi-channel time-frequency features for subsequent deep network input.

4. The method for identifying EEG fatigue based on multi-scale convolution and dual attention mechanism according to claim 1, characterized in that: The model structure constructed in step S3 includes: Multi-scale convolution feature extraction: A parallel multi-scale convolutional network structure is designed to extract time-domain features from the matrix M. Three one-dimensional convolution kernels of different sizes are set, such as lengths of 3, 5, and 7, corresponding to different receptive fields to capture time-domain information of different granularities. Each convolution branch outputs several channels, such as 16 output channels. After batch normalization and activation function processing, the output feature maps of the parallel branches are spliced ​​in the channel direction to obtain multi-scale fusion feature representations. After the multi-scale convolution, a spatial attention mechanism is introduced to weight the channel features of the convolution output. A method similar to channel attention is used to calculate the weight coefficient of each feature channel through operations such as global pooling, and the feature maps after multi-scale convolution are weighted by channel to generate feature maps with enhanced channel dependence, so that the network pays more attention to the spatial channel information related to EEG fatigue. The feature map after spatial attention weighting is expanded into a sequence according to the time step and input into the long short-term memory network LSTM. A two-layer LSTM network structure is adopted, with 128 hidden units in each layer. The feature vector of each time step is processed to capture the temporal change law. Through the LSTM network, the dynamic change characteristics of the EEG signal can be mined from the time dimension. After the LSTM output, the frequency attention module is introduced to weight the features of different frequency bands. After dividing the features of the LSTM output by frequency band, a fully connected network is used to generate attention weights for the four frequency bands. The frequency band features of the original feature map are weighted by multiplication to highlight the frequency band information that is more strongly associated with the fatigue state and suppress the noise of other frequency bands. The features processed by the dual attention mechanism are input into the fully connected classification layer, and finally the Softmax function is used to output the classification results. For the two-category fatigue / awake classification problem, when the output vectors of the fully connected layer are z1 and z2, Softmax is used to calculate the probability of each class: Among them, P i That is, the probability of predicting the i-th category is used to judge the driver's fatigue state based on the output probability.

5. The method for identifying EEG fatigue based on multi-scale convolution and dual attention mechanism according to claim 1, characterized in that: Training the model and outputting the results in step S4 includes: inputting the extracted feature data into the constructed neural network model to achieve fatigue state discrimination learning; segmenting the EEG signal according to the above steps, constructing a corresponding multi-input model structure, and using a five-fold cross-validation method to train and evaluate the model. At the same time, cross entropy is selected as the loss function to measure the prediction error of the model during the training process, and an adaptive moment estimation optimization algorithm Adam Optimizer is used to iteratively optimize the model parameters.

Images