Rapid fatigue detection method and device based on electroencephalogram characteristics

Abstract

The invention belongs to the technical field of electroencephalogram analysis, and discloses a rapid fatigue detection method and device based on electroencephalogram characteristics. Fatigue induction is carried out on the testee based on simulated driving, electroencephalogram signals are collected based on an electroencephalogram signal collector, and eye movement information is recorded in combination with an eye movement recorder; fatigue state judgment is carried out through the eye movement signals; preprocessing the electroencephalogram signal to obtain a corresponding electroencephalogram data set; and taking the electroencephalogram data set and the fatigue state as training data, training a deep convolutional neural network to learn electroencephalogram fatigue related features, and performing rapid fatigue detection on the electroencephalogram features by the trained deep convolutional neural network. Development of the fatigue detection field can be effectively promoted; and for the fatigue state of the user, a more accurate fatigue detection mode can be provided, and the user is assisted to evaluate the own state in a more detailed manner.

Description

Technical Field

[0001] The present invention relates to the technical field of electroencephalogram analysis, and particularly to a rapid fatigue detection method and device based on electroencephalogram features. Background Art

[0002] In today's society, due to factors such as high social competition pressure, long-term engagement in single jobs, and poor sleep quality, mental fatigue is extremely likely to be induced, seriously affecting people's normal production and life. Mental fatigue refers to the situation where people, in high-pressure, high-load, or repetitive and monotonous work, experience emotional disorders, psychological fatigue, and exhaustion, resulting in problems such as decreased cognitive and reaction abilities, inattention, and reduced work efficiency in work and life. With the increase in modern work intensity and the complexity of human-computer interaction scenarios, safety accidents caused by mental fatigue have become a global problem. Driving fatigue is a typical case of mental fatigue, and fatigue driving has become one of the main causes endangering traffic safety. When driving fatigued, the driver's brain alertness level decreases, working memory weakens, executive function is impaired, and it is difficult to precisely control the vehicle and perceive the surrounding environment, greatly increasing the accident risk. In the field of transportation, it is equally difficult to avoid the occurrence of fatigue driving among personnel in special driving industries such as airline pilots and high-speed rail drivers. This fully shows that mental fatigue has become a physiological problem commonly faced by many drivers. In fact, not only in special driving industries, but also in special work fields such as clinical medicine and high-altitude operations, workers often experience brain fatigue after a long period of mental labor. Once brain fatigue occurs, it is likely to trigger production accidents, often causing irreparable serious harm.

[0003] Currently, the evaluation of brain fatigue status is mainly divided into subjective evaluation methods and objective evaluation methods. The subjective evaluation method uses a psychological measurement questionnaire to let individuals or others evaluate their own fatigue status. Although it can reflect individual subjective feelings, does not require complex equipment, and the results have both individuality and universality, the evaluation accuracy is not high and it is mostly used as an auxiliary reference. The objective evaluation method is further divided into methods based on individual behavioral characteristics and physiological characteristics. In the method based on individual behavioral characteristics, the universality of fatigue behavior characteristics is not strong and it is operable, which is easy to interfere with the evaluation results. The method based on individual physiological characteristics is to evaluate the fatigue status through physiological signals such as electroencephalogram, electrocardiogram, and electrooculogram. Electroencephalogram (EEG) signals, as the direct electrophysiological representation of the central nervous system, can reflect changes in the brain's cognitive state, and electroencephalogram signals have always been the key signals for detecting biological states such as emotions and fatigue in the biomedical field.

[0004] Research shows that when an individual is in a fatigued state, characteristic changes will occur in their electroencephalogram (EEG) signals. In a fatigued state, low-frequency activity increases, which means the brain enters a more relaxed or near-sleep state. At the same time, high-frequency activity decreases, indicating a reduced attention concentration and weakened cognitive processing ability. Based on the characteristic changes in EEG signals, many scholars have proposed their own methods for fatigue detection based on EEG characteristics.

[0005] The Chinese patent "CN115721310A A Driving Fatigue Detection and Mitigation Method and System Based on Electroencephalogram" uses the sample entropy of the Theta wave in a single electrode signal as a characteristic index, and based on the sample entropy index, it reflects the fatigue state, and mitigates driving fatigue through methods such as music and electrical stimulation. However, it only uses the sample entropy of the Theta wave in a single electrode signal as an index, with too little information used, and the judgment of fatigue characteristics is not comprehensive. At the same time, it does not take into account the differences in EEG data among different people, and it is difficult to adapt to the differences in EEG characteristics of different people.

[0006] The Chinese patent "CN114403897A A Human Fatigue Detection Method and System Based on Electroencephalogram Signals" preprocesses the noisy EEG signals to obtain clean EEG signals, performs wavelet packet transform decomposition on the clean EEG signals to obtain section filtering. By calculating the section filtering energy ratio, it is used as training data and input into an artificial neural network to train the model to obtain the probabilities of fatigue and non-fatigue. However, this method relies on the power spectrum characteristics of a single frequency band, is easily affected by external electromagnetic interference and human physiological activities, and at the same time lacks the joint analysis of multi-band coupling characteristics and non-linear characteristics, resulting in an incomplete representation of the fatigue state. At the same time, the training network is not evaluated among different individuals, reducing the universality of the method.

[0007] The Chinese patent "CN114782933B A Driver Fatigue Detection System Based on a Multi-modal Transformer Network" provides a data acquisition module for obtaining the EEG signals and facial expression image blocks of a driver, and performs multi-head self-attention analysis on the EEG signals and image blocks through a Transformer encoder for result prediction, and can obtain the driving state of the driver based on the driver's EEG and facial expressions. This method provides a multi-modal fatigue prediction solution. Although it can improve the accuracy, the number of model parameters is large, the computing power requirements for the device are high, it is difficult to achieve low-latency edge computing, and it cannot meet the high timeliness requirements of fatigue detection. Summary of the Invention

[0008] In the current field of fatigue detection based on electroencephalogram (EEG) analysis, traditional methods face many problems that urgently need to be solved. Most of the existing fatigue detection methods only rely on features such as differential entropy and power in frequency bands for analysis and judgment. However, EEG signals contain rich information, and multi-band coupling characteristics (such as the θ / β wave ratio) and non-linear characteristics also play a key role in characterizing the fatigue state. Ignoring these important information will inevitably lead to an incomplete and inaccurate characterization of the fatigue state. More prominently, there are significant individual differences in EEG signals, and the EEG signal characteristics of different individuals vary greatly. The static models used in traditional methods, such as fixed-threshold classification, lack flexibility and cannot adapt to this individual difference. Their generalization ability in cross-user scenarios is extremely poor, which greatly limits their application scope and effect. At the same time, in the current research direction of deep learning for EEG, most models are too complex and have too many parameters, making it difficult to provide real-time feedback results.

[0009] In view of the above problems, the core objective of the present invention is to propose a fast fatigue detection method and device based on EEG characteristics. First, a lightweight network structure is carefully designed, and a lightweight feature extraction algorithm is adopted. This algorithm can achieve the rapid fusion of multi-dimensional features, greatly improving the efficiency of feature representation. The constructed lightweight network structure can better extract the common features in EEG signals, enabling this method to achieve rapid adaptation in cross-user scenarios and meet the requirements of low-power deployment at the same time. While ensuring high efficiency, this method also takes into account the high accuracy of prediction, providing a more advantageous solution for the field of fatigue detection.

[0010] The technical solution of the present invention is as follows: A fast fatigue detection method based on EEG characteristics induces fatigue in the tester based on simulated driving, collects EEG signals through an EEG signal collector, and records eye movement information in combination with an eye movement recorder; judges the fatigue state through the eye movement signal; preprocesses the EEG signals to obtain the corresponding EEG data set; uses the EEG data set and the fatigue state as training data to train a deep convolutional neural network to learn EEG fatigue-related features, and the trained deep convolutional neural network performs fast fatigue detection of EEG features.

[0011] The deep convolutional neural network is a lightweight network structure, including an ECANet attention module and a deep convolutional module, which are connected in sequence front and back;

[0012] The deep convolutional module includes nine network layers; pointwise convolution, batch normalization layer, and feature enhancement layer are the first three network layers. Then, feature convolution is performed through the deep convolutional layer, followed by a ReLU activation layer, a Dropout layer, a batch normalization layer, a global average pooling layer, a dense layer, and a Softmax activation layer, and finally outputs the probabilities of the predicted data being in the awake state and the fatigue state.

[0013] The first network layer uses N1 A pointwise convolution node to generate N 1 new channels of signals with the same length; for a given input sample X (m,n) , m is the number of channels, n is the length of a single signal; the i-th node has m weight parameters and a bias parameter 1 ≤ i ≤ N 1 ; the output of the first network layer is:

[0014]

[0015] where i = 1, 2, 3, …, N 1 and x p,j is the j-th sampling point of the p-th channel of the input sample; the output result of the first network layer is normalized to obtain the output result of the second network layer;

[0016]

[0017] In the third network layer, a 1×1 convolutional layer is used to expand the dimension of the data, and at the same time, it can effectively convert N 1 dimensional information into F-dimensional data, and the specific output result is as follows;

[0018]

[0019] The fourth network layer is a depth convolutional layer for extracting features from the F-dimensional data; each new signal in the F-dimensional data is convolved with a depth convolutional node in the fourth network layer; since there are F channels in the F-dimensional data output from the third network layer, there are a total of F depth convolutional nodes; the i-th node has l weights and 1 bias parameter, 1 ≤ i ≤ F; the output result is as follows:

[0020]

[0021] The fifth network layer uses the ReLU activation function, the probability of Dropout in the sixth network layer is 0.5, and the seventh network layer is a normalization layer;

[0022]

[0023]

[0024]

[0025] The eighth layer is global average pooling, and the specific output formula is as follows;

[0026]

[0027] The last two fully connected layers and the Softmax activation layer are used to map the obtained features to a two-dimensional space to represent the probabilities of being awake and fatigued. The formula is as follows;

[0028]

[0029]

[0030] During the training process of the deep convolutional neural network, cross-entropy is used as the loss function to represent the difference between the true label and the predicted label. y i represents the sample label, and p i represents the sample prediction result. m represents the number of EEG samples used for training;

[0031]

[0032] The fatigue state is judged as follows:

[0033] Based on the eye movement signal, PERCLOS is calculated as the fatigue data standard. The fatigue state is divided into two states: fatigue and wakefulness. When PERCLOS is greater than 0.35, it is considered to be in a fatigued state; otherwise, it is in a wakeful state.

[0034] The formula for calculating PERCLOS is as follows: Blink represents the blink time, Clos represents the closed-eye time, Fixation represents the fixation time, and Saccade is the saccade time;

[0035]

[0036] The preprocessing of the EEG data is as follows:

[0037] Step 1: Use a notch filter to remove 50Hz electromagnetic noise, and use a band-pass filter to limit the EEG signal range to 1 - 50Hz;

[0038] Step 2: Reduce the sampling frequency of the filtered EEG signal data to 128Hz through downsampling;

[0039] Step 3: Remove the bad segments of the EEG signal with noise;

[0040] Step 4: Remove the eye movement signal based on the independent component analysis method.

[0041] A fast fatigue detection device based on EEG features includes:

[0042] A data acquisition module for collecting EEG signals and eye movement information;

[0043] A data processing module for processing EEG signals to obtain corresponding EEG datasets and for fatigue state judgment;

[0044] A training module for storing and training a deep convolutional neural network to learn EEG fatigue-related features, and the trained deep convolutional neural network for rapid fatigue detection of EEG features.

[0045] The deep convolutional neural network in the training module includes an ECANet attention module and a deep convolutional module, which are connected in sequence front and back;

[0046] The deep convolutional module includes nine network layers; pointwise convolution, batch normalization layer, and feature enhancement layer are the first three network layers, and then feature convolution is performed through the deep convolutional layer, followed by a ReLU activation layer, a Dropout layer, a batch normalization layer, a global average pooling layer, a dense layer, and a Softmax activation layer, and finally the probabilities of the predicted data being in a wake state and a fatigue state are output.

[0047] Advantages of the present invention: The present invention introduces a new method for fatigue detection, which can effectively promote the development of the fatigue detection field; for the fatigue state of users, it can give a more accurate fatigue detection method to assist users in making a more detailed assessment of their own state. Description of the Drawings

[0048] Figure 1 It is a distribution diagram of EEG electrode positions;

[0049] Figure 2 It is an ECA attention structure;

[0050] Figure 3 It is a flowchart for fatigue state detection. Detailed Embodiment

[0052] The present invention induces fatigue in testers based on simulated driving, simultaneously collects EEG signals based on an EEG signal collector, and combines an eye movement recorder to record eye movement information. The fatigue state is judged through the eye movement signal, and then the EEG is preprocessed to obtain the corresponding EEG dataset. The EEG dataset and the corresponding fatigue state are used as training data to let the deep convolutional neural network learn EEG fatigue-related features, and the probabilities of being in a wake state and a fatigue state are obtained to achieve rapid fatigue detection based on EEG features.

[0053] 1. Construction of the EEG dataset;

[0054] In the process of constructing the EEG dataset of the present invention, the distribution of EEG electrode positions collected based on the EEG signal instrument is as Figure 1As shown in the figure. Calculate PERCLOS (Percentage of Eye Closure) based on the eye movement signal as the fatigue data standard, and divide the fatigue state into two states: fatigue and wakefulness. The calculation formula is as follows. Where Blink represents the blink time, Clos represents the graduation time, Fixation represents the fixation time, and Saccade is the saccade time.

[0055]

[0056] Preprocessing the EEG data can effectively remove the noise in the EEG data acquisition process and improve the prediction accuracy of the deep convolutional neural network. The specific steps are as follows:

[0057] Step 1: Use Python to call mne, use a notch filter to remove 50Hz electromagnetic noise, and use a band-pass filter to limit the EEG signal range to 1 - 50Hz.

[0058] Step 2: Reduce the EEG signal data acquisition frequency to 128Hz through downsampling, reduce the data sampling frequency, and reduce the model parameters.

[0059] Step 3: Based on the EEGLAB toolbox in MATLAB, remove the bad segments of the EEG signal with noise.

[0060] Step 4: Remove the eye movement signal based on the ICA (Independent Component Analysis) method to ensure that the EEG data is not interfered by eye movement.

[0061] 2. Training of the deep convolutional neural network

[0062] For the EEG dataset, the present invention designs a lightweight network structure based on the depthwise separable convolutional neural network. The network structure mainly includes an ECANet attention module and a depth convolutional module. This deep convolutional neural network structure reduces the number of model parameters and can quickly perform fatigue detection while ensuring the model accuracy.

[0063] ECANet (Efficient Channel Attention Network) is a lightweight channel attention mechanism designed to enhance the feature representation ability of the convolutional neural network in an efficient way. The core idea of ECANet is to generate channel attention weights through global average pooling (GAP) and one-dimensional convolution, so as to enhance the features of important channels. The specific structure is as Figure 2 shown.

[0064] The deep convolutional module consists of nine network layers, and its main task is to extract EEG fatigue features and classify fatigue states. The pointwise convolutional layer, batch normalization layer, and feature enhancement layer are implemented in the first three layers. Then, feature convolution is performed through the deep convolutional layer, followed by the ReLU activation layer, Dropout layer, batch normalization layer, global average pooling layer, dense layer, and Softmax activation layer. Finally, the probabilities of the predicted data being in the awake state and the fatigued state are output.

[0065] In the first network layer, we use N 1 pointwise convolutional nodes to generate N 1 new signal channels of the same length. For a given input sample X (m,n) , m is 30, representing 30 channels, and n is 382, representing the length of a single signal. The i-th (1 ≤ i ≤ N 1 ) node has m weight parameters and one bias parameter . The output of the first network layer is:

[0066]

[0067] where i = 1, 2, 3, …, N 1 and x p,j is the j-th sampling point of the p-th channel of the input sample. The superscripts of the output formula and network parameters indicate which network layer they belong to. Here, N 1 is set to 32 to reduce redundancy and encourage the convergence of the network. After normalizing the output result of the first network layer, the output result of the second network layer is obtained.

[0068]

[0069] In the third network layer, mainly a 1×1 convolutional layer is used to expand the dimension of the data, and at the same time, it can effectively convert N 1 -dimensional information into F-dimensional data, where F is equal to 64, which can effectively increase the feature complexity. The specific output result is as follows.

[0070]

[0071] The fourth network layer is the deep convolutional layer, which is used to extract features from the F-dimensional data, that is, the F obtained signals. Specifically, each new signal is convolved with a deep convolutional node in the fourth network layer. Since there are F channels outputting F-dimensional data from the third network layer, there are a total of F deep convolutional nodes. Assuming the length of the kernel is l, the i-th (1 ≤ i ≤ F) node has l weights and 1 bias parameter. The output result is as follows:

[0072]

[0073] The fifth, sixth, and seventh network layers mainly process and transform the input data to prevent the model from overfitting. The fifth network layer uses the ReLU activation function, the probability of Dropout in the sixth network layer is 0.5, and the seventh network layer is a normalization layer.

[0074]

[0075]

[0076]

[0077] The eighth network layer is the Global Average Pooling (GAP). Compared with the widely used fully connected layer, the Global Average Pooling (GAP) layer significantly reduces the parameters, so it can effectively prevent overfitting, enhance the generalization ability of the model, and extract the common features of different targets. Here, the parameter l = 64, and the specific output formula is as follows.

[0078]

[0079] The last two layers of the model are the fully connected layer and the Softmax activation layer, which are used to map the obtained features to a two-dimensional space to represent the probability of being awake and the probability of being fatigued. The formula is as follows. When the output probability of being awake is greater than 0.5, it is considered awake.

[0080]

[0081] During the training process of the deep convolutional neural network, cross-entropy is used as the loss function, which represents the difference between the true label and the predicted label. y i represents the sample label, and p i represents the sample prediction result, and m represents the number of EEG samples used for training.

[0082]

[0083] For the above-designed deep convolutional neural network, the publicly available dataset SEED-VIG data is used for model training and testing. To verify the generalization ability of the model, the leave-one-subject-out (LOSO) method is adopted for model training and testing. The leave-one-subject-out method is a form of cross-validation method. Each time, only the data of one subject is left as the tester, and the data of other testers is used as the training set and the validation set, which can effectively study the performance of the model among different individuals. At the same time, the results of the model are compared with the results of models such as InterpretableCNN, EEGConform, EEGDeformer, and EEGNet to verify the performance of the model. Here, the accuracy index is selected for model performance evaluation. The results are shown in Table 1. From the average accuracy of the prediction, the designed network model is better than other existing models.

[0084] Table 1 Results of model comparison experiments

[0085]

[0086] The present invention proposes a deep convolutional neural network, which includes an ECANet attention module and a deep convolutional module, effectively improving the generalization ability of the model. At the same time, the performance of the model is verified using the publicly available dataset SEED-VIG, improving the prediction accuracy in cross-target electroencephalogram signal fatigue detection.

[0087] In terms of feature extraction, the deep convolutional neural network proposed by the present invention realizes fast multi-dimensional feature fusion, comprehensively captures the fatigue-related information in the electroencephalogram signal, overcomes the limitation of the traditional method relying on single-band features, improves the feature representation efficiency, and makes the fatigue state representation more accurate. The deep convolutional neural network effectively extracts the common features of the electroencephalogram signal, enhances the generalization ability in cross-user scenarios, reduces the computing power requirements for embedded devices, meets the low-power deployment requirements, solves the problems of complex traditional models, many parameters, and difficult real-time feedback, and can quickly and accurately detect the fatigue state.

Claims

1. A rapid fatigue detection method based on EEG features, characterized in that: The test subjects were fatigued by simulated driving, EEG signals were collected by EEG signal collector, and eye movement information was recorded by eye movement recorder. Judging fatigue status through eye movement signals; The EEG signal is preprocessed to obtain the corresponding EEG data set; the EEG data set and fatigue status are used as training data to train a deep convolutional neural network to learn EEG fatigue-related features, and the trained deep convolutional neural network performs rapid fatigue detection of EEG features.

2. The rapid fatigue detection method based on EEG characteristics according to claim 1 is characterized in that: The deep convolutional neural network is a lightweight network structure, including an ECANet attention module and a deep convolution module, which are connected front and back; The deep convolution module includes nine network layers; point-by-point convolution, batch normalization layer, and feature enhancement layer are the first three network layers, followed by feature convolution through the deep convolution layer, followed by the ReLU activation layer, Dropout layer, batch normalization layer, global average pooling layer, dense layer and Softmax activation layer, and finally outputs the probability of predicting the data being in the awake state and the fatigue state.

3. The rapid fatigue detection method based on EEG features according to claim 2 is characterized in that: The first network layer uses N1 point-by-point convolution nodes to generate N1 new signal channels of the same length; for a given input sample X (m,n) , m is the number of channels, n is the length of a single signal; the i-th node has m weight parameters and a bias parameter The output of the first network layer is: where i = 1, 2, 3, ..., N1 and x p,j is the jth sampling point of the pth channel of the input sample; The output result of the first network layer is normalized to obtain the output result of the second network layer; In the third network layer, a 1×1 convolutional layer is used to expand the data dimension, and it can effectively convert N1-dimensional information into F-dimensional data. The specific output results are as follows; The fourth network layer is a deep convolutional layer, which is used to extract features from F-dimensional data; each new signal in the F-dimensional data is convolved with a deep convolution node in the fourth network layer; since there are F channels that output F-dimensional data from the third network layer, there are a total of F deep convolution nodes; the i-th node has l weights and 1 bias Parameters, 1≤i≤F; the output results are as follows: The fifth network layer uses the ReLU activation function, the sixth network layer has a Dropout probability of 0.5, and the seventh network layer is a normalization layer; The eighth network layer is the global average pooling, and the specific output formula is as follows; The last two fully connected layers and the Softmax activation layer are used to map the obtained features into a two-dimensional space to represent the probability of wakefulness and fatigue. The formula is as follows; 4. The rapid fatigue detection method based on EEG features according to claim 2 is characterized in that: In the training process of the deep convolutional neural network, cross entropy is used as the loss function to represent the difference between the true label and the predicted label, y i represents the sample label, p i It represents the sample prediction result, and m represents the number of EEG samples used for training; 5. The rapid fatigue detection method based on EEG characteristics according to claim 1 is characterized in that: The fatigue state is determined as follows: PERCLOS is calculated based on eye movement signals as the fatigue data standard, and the fatigue state is divided into two states: fatigue and wakefulness. When PERCLOS is greater than 0.35, it is considered to be in a fatigue state, otherwise it is in an awake state.

6. The rapid fatigue detection method based on EEG characteristics according to claim 5 is characterized in that: The PERCLOS calculation formula is as follows: Blink represents blinking time, Clos represents closing time, Fixation represents fixation time, Saccade represents saccade time, and when PERCLOS is greater than 0.35, it is considered to be in a fatigue state; 7. The rapid fatigue detection method based on EEG characteristics according to claim 1 is characterized in that: The EEG data preprocessing includes the following specific steps: Step 1: Use a notch filter to remove 50Hz electromagnetic noise and a bandpass filter to limit the EEG signal range to 1-50Hz; Step 2: Reduce the frequency of the filtered EEG data acquisition to 128 Hz by downsampling; Step 3: Remove bad EEG signal segments with noise; Step 4: Remove eye movement signals based on independent component analysis method.

8. A rapid fatigue detection device based on EEG characteristics, characterized in that: include: Data acquisition module, used to collect EEG signals and eye movement information; A data processing module is used to process EEG signals to obtain corresponding EEG data sets and to judge fatigue status; The training module is used to store and train the deep convolutional neural network to learn EEG fatigue-related features, and the trained deep convolutional neural network performs rapid fatigue detection of EEG features.

9. The rapid fatigue detection device based on EEG characteristics according to claim 8 is characterized in that: The deep convolutional neural network includes an ECANet attention module and a deep convolution module, which are connected front and back; The deep convolution module includes nine network layers; point-by-point convolution, batch normalization layer, and feature enhancement layer are the first three network layers, followed by feature convolution through the deep convolution layer, followed by the ReLU activation layer, Dropout layer, batch normalization layer, global average pooling layer, dense layer and Softmax activation layer, and finally outputs the probability of predicting the data being in the awake state and the fatigue state.

Images