Electroencephalogram-based fatigue detection device and detection method

By designing a fatigue detection device based on EEG, using gel electrodes and improved deep learning models, the efficiency and accuracy of fatigue detection in the prior art are solved, and convenient and high-precision fatigue state evaluation and real-time monitoring are achieved. Adapt to the differences between different individuals.

CN118975805APending Publication Date: 2024-11-19XIAN ZHONGKE MICRO INTELLIGENT ELECTRONICS CO LTD
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202411079340.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2024-06-05
Filing Date
2024-08-07
Publication Date
2024-11-19

AI Technical Summary

Technical Problem

The existing technology cannot provide efficient, objective and portable fatigue state detection methods. Deep learning models face limitations in data quality and computing resources in EEG fatigue detection, and individual differences have a great impact, making it difficult to achieve real-time and accurate fatigue state assessment. .

Method used

A fatigue detection device based on EEG is designed, including EEG acquisition module, processing module and analysis module. The gel electrode is used to collect EEG signals, and the features are extracted through filtering and multi-scale fuzzy entropy, combined with an improved deep learning model. , including YOLOv8's C2f module, SENet and LANet, perform classification and feedback, and use transfer learning to improve model adaptability.

Benefits of technology

Convenient and high-precision fatigue detection is realized, which can monitor and predict fatigue status in real time, reduce accident risk, improve work efficiency, and enhance the robustness and adaptability of the model through multimodal feature extraction and attention mechanism.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118975805A_ABST
    Figure CN118975805A_ABST
Patent Text Reader

Abstract

The invention relates to an electroencephalogram-based fatigue detection device, the detection device comprises an electroencephalogram acquisition module, an electroencephalogram processing module, an electroencephalogram analysis module and a fatigue feedback module, the electroencephalogram acquisition module is used for acquiring an original electroencephalogram signal of a user by using a gel electrode, the electroencephalogram processing module is used for extracting a blink signal of notch filtering, and the fatigue feedback module is used for feedback of the electroencephalogram. The collected electroencephalogram signals are filtered, the number of times of blink signals is found out, the rhythm of the blink signals is extracted, the original electroencephalogram rhythm is extracted through multi-scale fuzzy entropy, and the rhythm characteristics of the electroencephalogram during fatigue are found out; the electroencephalogram analysis module is used for classifying the integrated data by using an improved deep learning model, the model is deployed on an end-side chip for reasoning, and the fatigue feedback module is used for feeding a fatigue judgment result obtained by classification back to a mobile phone or a computer of a user by using a Bluetooth module. According to the scheme, convenient, high-precision and accurate fatigue detection equipment is provided for a user, and in addition, a fatigue detection method based on electroencephalogram is provided.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention relates to a fatigue detection device and a detection method based on electroencephalogram, belonging to the technical field of fatigue detection. Background Art

[0002] With the fast pace of life and increasingly fierce competition in modern society, fatigue classification has become a critical issue in many industries and application fields. Traditional fatigue detection methods are often limited by subjectivity or non-portability, such as questionnaires or biological parameter monitoring. These methods cannot provide real-time and objective fatigue status assessment, nor can they adapt to the efficient work and life needs of modern society. Therefore, there is an urgent need for an efficient, objective and portable method to accurately detect and identify fatigue status in order to improve work efficiency and protect people's health.

[0003] EEG fatigue state detection technology is expected to achieve major breakthroughs in the future and be widely used in medical, industrial, transportation and other fields. Future development trends include multimodal fusion and personalized models. Multimodal fusion combines EEG signals with other physiological signals (such as heart rate, eye movement, skin electrical activity, etc.) to improve the accuracy and reliability of fatigue state detection and provide a more comprehensive basis for prevention and intervention measures. The personalized model will be dynamically adjusted according to the characteristics of the individual, and technologies such as transfer learning and incremental learning will be used to improve the adaptability and predictive ability for different populations. In addition, real-time monitoring and feedback are also important directions for future development. A real-time monitoring and feedback system will be established in combination with the Internet of Things technology to promptly detect and respond to the fatigue state of individuals, reduce the risk of accidents, and improve work efficiency.

[0004] Deep learning models play a key role in EEG fatigue state detection. In particular, convolutional neural networks (CNN) and Transformer models perform well in processing complex time series data. CNN can effectively extract the spatial features of EEG signals, while Transformer is good at capturing the temporal dependencies in the signal. Combining the two and applying them to EEG signal processing can significantly improve the accuracy of fatigue state detection. As the latest generation of target detection models, the YOLOv8 model introduces a C2f module with a multi-branch flow design that can enhance the ability of feature extraction. Therefore, applying the YOLOv8 model to EEG fatigue detection, combined with multimodal feature extraction and transfer learning technology, can achieve high-precision and real-time fatigue state detection. However, the application of deep learning models also faces challenges. First, the quality and individual differences of EEG signal data have a great impact on the performance of the model, which needs to be considered in the model design and training process. Secondly, the Transformer-based model has a high complexity and requires a lot of computing resources for training and reasoning, which may bring certain challenges in practical applications. Therefore, in the research and application of deep learning models, it is necessary to comprehensively consider data quality, individual differences, and computing resource limitations to improve the practicality and universality of the model. Summary of the invention

[0005] The present invention is aimed at the technical problems existing in the prior art and provides a fatigue detection device based on EEG. On the one hand, the technical solution provides a convenient EEG fatigue detection device, thereby providing users with a convenient, high-precision and accurate fatigue detection equipment, and on the other hand, provides a fatigue detection method based on EEG.

[0006] In order to achieve the above object, the technical solution of the present invention is as follows: a fatigue detection device based on EEG, the detection device includes an EEG acquisition module, an EEG processing module, an EEG analysis module and a fatigue feedback module.

[0007] in,

[0008] EEG acquisition module: gel electrodes are used to collect the user's original EEG signals. The EEG acquisition chip uses ADS1299. As shown in the block diagram, magnetic beads are used at the acquisition end to remove high-frequency noise interference and filter out high-frequency noise in the EEG part. The frontal lobe, parietal lobe, and occipital lobe are used in the brain area, which correspond to the brain's decision-making function, imagination function, and visual function respectively. The selected points are Fp1, Fp2, C3, C4, O1, and O2. The EEG processing module, on the one hand, extracts the blinking signal based on the notch filter, filters the collected EEG signal, finds the number of blinking signals, and extracts the blinking signal rhythm. On the other hand, the original EEG rhythm is extracted through multi-scale fuzzy entropy to find out the rhythmic characteristics of EEG during fatigue.

[0009] The EEG analysis module classifies the integrated data using an improved deep learning model, which is deployed on the end-side chip for inference. The fatigue feedback module uses the Bluetooth module to feed back the fatigue judgment results obtained by classification to the user's mobile phone or computer.

[0010] A fatigue detection method based on electroencephalogram, the method comprising the following steps:

[0011] Step 1: EEG signal acquisition,

[0012] Step 2: Extract features from the signal.

[0013] Step 3: Transmit the EEG features to the EEG processing module in real time to achieve five-category fatigue detection for the user.

[0014] Step 4: Save the received classification results to the memory, and send the results to the monitoring device through the wireless transmission device according to the fixed protocol. Wireless protocols and hardware such as Bluetooth, WiFi, 4G, monitoring devices such as computers, mobile phones or some specific IoT devices.

[0015] Among them, step 1: EEG signal acquisition is performed by the EEG acquisition module, as follows:

[0016] The composition of the EEG acquisition module consists of a series of neural sensing electrodes. These electrodes are gel electrodes that can provide a good signal-to-noise ratio, thereby improving the quality of the signal. The gel electrodes can reduce impedance and improve signal clarity by forming a layer of medium between the electrode and the skin.

[0017] The selected EEG collection points include Fp1, Fp2 (frontal area), C3, C4 (central area), O1, O2 (occipital area). These points are usually related to cognitive function, attention and visual processing. The signals at these points can provide a more comprehensive understanding of the brain's activity status.

[0018] Reference electrodes are usually placed on the left and right earlobes. This is because the electrical activity of the earlobes is relatively low and can serve as a stable reference point, helping to improve the accuracy of the signal.

[0019] Filtering: The EEG acquisition module contains a low-pass filter, which is used to filter out high-frequency noise above 100Hz. High-frequency noise may be caused by myoelectric interference, power supply interference, etc. Removing these noises helps improve the clarity and reliability of the signal. Signal acquisition frequency: The frequency of EEG acquisition is 1000Hz, which means that 1000 data points are collected per second. The high sampling frequency helps capture rapidly changing EEG activities and provide more detailed data for subsequent analysis.

[0020] Real-time transmission: The collected EEG features need to be transmitted to the EEG processing module in real time. Real-time performance is very important for some application scenarios (such as driving fatigue monitoring, medical monitoring, etc.) because they require a quick response to take corresponding measures.

[0021] Among them, step 2: extracting features from the signal, as follows:

[0022] Bandpass filtering: First, the raw EEG data is preprocessed using a 0.5-30 Hz bandpass filter to retain frequency components related to brain activity while removing irrelevant noise. This includes removing 50 Hz power frequency interference, which is usually generated by power lines and may affect the quality of EEG signals.

[0023] EEG feature extraction 1: Feature extraction is performed on the filtered EEG signal to capture features related to fatigue. Fuzzy entropy: This is an indicator of signal uncertainty and complexity, by calculating entropy at different time scales. The steps for calculating fuzzy entropy are as follows: For a time series of length N

[0024] {x(i), i=1,2,3,...,n}

[0025] Reconstruct the sequence and solve the vector relative distance to obtain Dt ij

[0026] Dt ij =max(|x(i+k)-x(j+k)|), k∈[0, m-1]

[0027] Fuzzy membership function A(x)

[0028]

[0029] For i=1, 2, .., N-m+1 and j=1, 2, .., N-m+1, and j≠i. For multiple vectors in this range, calculate the mean of A(x)

[0030]

[0031] then

[0032]

[0033] The fuzzy entropy of the original time series is

[0034]

[0035] m can be called the pattern dimension or embedding dimension, r represents the width of the fuzzy function boundary. N determines the slope of the similarity tolerance boundary. The larger N is, the larger the slope is. m = 2, r = 0.2 * the standard deviation of the sequence, N = 2.

[0036] EEG feature extraction 2: STFT spectrum and eye movement features of Fp1 channel: In order to extract features related to visual fatigue and attention state, the EEG signal is bandpass filtered at 5-15Hz to ensure that the blinking feature is more obvious. Then, the following indicators are calculated from the filtered signal:

[0037] STFT calculation formula: where x[k] is the time signal and ω[k] is the chosen window.

[0038] The spectrum of STFT is obtained, and the spectrum is calculated every 5000 sampling points, and the feature matrix of the obtained graph is 630*480*3. Among them, step 3: transmit the EEG features to the EEG processing module in real time to realize the five-category fatigue detection of the user, as follows:

[0039] The 6*6 EEG spatiotemporal feature matrix composed of the above six features is input into the improved deep learning module to realize the five-category fatigue detection of users.

[0040] The improved deep learning modules include YOLOv8's C2f module (Faster Implementation of CSPBottleneck with 2convolutions), SENet module (Squeeze-and-Excitation Networks), LANet module (local attention network), convolutional layer, pooling layer, fully connected layer, Transformer encoding layer (TransformerEncoder), and output layer.

[0041] First, the 6*6 EEG spatiotemporal feature matrix x is regarded as a single-channel image data and input into the convolution layer. A small-sized convolution kernel is used to extract local features, and batch normalization and activation function ReLU are used to process the convolutional feature matrix.

[0042] y = ReLU(x*W+b)

[0043] Where x is the input matrix, W is the convolution kernel, b is the bias term, and ReLU is the activation function.

[0044] The pooling layer uses the maximum pooling layer, and the pooling window size is 2x2, which is used to reduce the feature dimension to obtain the new matrix x1. The C2f module of YOLOv8 performs preliminary feature extraction. The SENet module and the LANet module perform channel attention enhancement and spatial attention enhancement respectively. The Transformer Encoder layer consists of multiple encoders, each of which includes a sub-attention mechanism and a feedforward neural network. The dimension of each encoding layer is 6, the number of attention heads is 3, and the hidden layer dimension of the feedforward network is 128. After the Transformer Encoder layer, the dimension of the feature matrix remains unchanged, recorded as x2.

[0045] y=Attention(x1)

[0046] The EEG spatiotemporal feature matrix x2 processed by the Transformer Encoder layer needs to be flattened before being input into the fully connected layer for classification. The flattening layer transforms its dimension into a 1×n one-dimensional vector x3, and then inputs x3 into the fully connected layer for classification. Finally, the softmax function is used for multi-classification. The number of output neurons is equal to the number of categories, that is, 5 categories. The output value is the predicted probability of each fatigue state.

[0047]

[0048] Where Z i is the output value of the th node, and C is the number of output nodes, that is, the number of categories. The Softmax function can be used to convert the output values ​​of multiple categories into a probability distribution ranging from [0, 1] and 1.

[0049] In order to solve the problem of cross-subject EEG data differences, the present invention adopts a transfer learning method. First, a CNN-Transformer model is pre-trained on a large-scale EEG dataset to extract common EEG features. Then, the feature extraction layer of the pre-trained model is frozen, and a new fully connected layer is added on this basis, and a small amount of new subject data is used to fine-tune the newly added layer. In this way, the adaptability and generalization ability of the model between different individuals can be effectively improved.

[0050] Compared with the prior art, the present invention has the following advantages: 1. The technical solution proposes an innovative fatigue state detection method based on EEG signals, aiming to achieve accurate quantitative assessment of the fatigue degree of the subject. The method cleverly solves the challenge of effectively extracting and perceiving EEG signals in neural networks, and innovatively designs a multi-feature two-dimensional matrix for the training process of the model. 2. The core technical highlights of the solution cover the feature extraction of blinking signals and the feature analysis of EEG rhythms. The EEG data is analyzed and trained using an improved deep learning model, and the trained model is combined with an FPGA portable device to achieve real-time detection and prediction of EEG fatigue state. The C2f module of YOLOv8 is a deep neural network module that uses a multi-branch structure and feature fusion technology. It can retain more original information, enhance gradient flow, and improve the diversity of expression. Due to its multi-branch structure, it can also improve the computational efficiency of the model. SENet enables the network to automatically enhance its attention to important features and suppress irrelevant or unimportant features by learning the importance weight of each channel. The LANet module is an attention mechanism module that focuses on local feature enhancement, aiming to enhance the network's attention to local important features. The C2f module is used as the basic feature extraction module, and the SENet module and LANet module are used for channel attention enhancement and spatial attention enhancement. The transformer is used to capture the global dependencies in the entire time series data matrix through the self-attention mechanism, which is very important for understanding the overall pattern of fatigue state. The model has better robustness. 3. In terms of feature extraction, this scheme chooses to fuse the frontal EEG signal with the occipital EEG signal. The frontal EEG signal is not blocked by hair and has a good signal-to-noise ratio. It can detect obvious blinking signals. The occipital area is a concentrated area of ​​optic nerves, which can effectively monitor visual fatigue. At the same time, it can detect user fatigue even during the use of SSVEP. 4. This scheme extracts multimodal features from EEG signals, and the model has better stability. 5. This scheme adopts the method of transfer learning to achieve high-level adaptation and accurate classification of EEG data across subjects. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 It is a convenient device for detecting EEG fatigue;

[0052] Figure 2 This is a schematic diagram of the EEG acquisition module;

[0053] Figure 3 It is a schematic diagram of the data processing module;

[0054] Figure 4 Schematic diagram of the EEG analysis module. DETAILED DESCRIPTION

[0055] In order to deepen the understanding of the present invention, the present embodiment is described in detail below with reference to the accompanying drawings.

[0056] Example 1: See Figure 1 , a fatigue detection device based on EEG, the detection device includes an EEG acquisition module, an EEG processing module, an EEG analysis module and a fatigue feedback module,

[0057] in,

[0058] EEG acquisition module: Use gel electrodes to collect the user's original EEG signals. The EEG acquisition chip uses ADS1299, as shown in the box. Figure 1 The acquisition end shown uses magnetic beads to remove high-frequency noise interference and filter out high-frequency noise in the EEG part. The brain regions used are the frontal lobe, parietal lobe, and occipital lobe, which correspond to the brain's decision-making function, imagination function, and visual function, respectively. The selected points are Fp1, Fp2, C3, C4, O1, and O2. The EEG processing module, on the one hand, extracts the blink signal based on the notch filter, filters the collected EEG signal, finds the number of blink signals, and extracts the blink signal rhythm. On the other hand, it extracts the original EEG rhythm through multi-scale fuzzy entropy to find out the rhythmic characteristics of EEG during fatigue.

[0059] The EEG analysis module classifies the integrated data using an improved neural network model, which is deployed on the end-side chip for inference. The fatigue feedback module uses the Bluetooth module to feed back the fatigue judgment results obtained by classification to the user's mobile phone or computer.

[0060] Example 2: See Figure 1 , a fatigue detection method based on electroencephalogram, the method comprising the following steps:

[0061] Step 1: EEG signal acquisition,

[0062] Among them, step 1: EEG signal acquisition is performed by the EEG acquisition module, as follows:

[0063] The composition of the EEG acquisition module consists of a series of neural sensing electrodes. These electrodes are gel electrodes that can provide a good signal-to-noise ratio, thereby improving the quality of the signal. The gel electrodes can reduce impedance and improve signal clarity by forming a layer of medium between the electrode and the skin.

[0064] The selected EEG collection points include Fp1, Fp2 (frontal area), C3, C4 (central area), O1, O2 (occipital area). These points are usually related to cognitive function, attention and visual processing. The signals at these points can provide a more comprehensive understanding of the brain's activity status.

[0065] Reference electrodes are usually placed on the left and right earlobes. This is because the electrical activity of the earlobes is relatively low and can serve as a stable reference point, helping to improve the accuracy of the signal.

[0066] Filtering: The EEG acquisition module contains a low-pass filter, which is used to filter out high-frequency noise above 100Hz. High-frequency noise may be caused by myoelectric interference, power supply interference, etc. Removing these noises helps improve the clarity and reliability of the signal. Signal acquisition frequency: The frequency of EEG acquisition is 1000Hz, which means that 1000 data points are collected per second. The high sampling frequency helps capture rapidly changing EEG activities and provide more detailed data for subsequent analysis.

[0067] Real-time transmission: The collected EEG features need to be transmitted to the EEG processing module in real time. Real-time performance is very important for some application scenarios (such as driving fatigue monitoring, medical monitoring, etc.) because they require a quick response to take corresponding measures.

[0068] Among them, step 2: extracting features from the signal, as follows:

[0069] Bandpass filtering: First, the raw EEG data is preprocessed using a 0.5-30 Hz bandpass filter to retain frequency components related to brain activity while removing irrelevant noise. This includes removing 50 Hz power frequency interference, which is usually generated by power lines and may affect the quality of EEG signals.

[0070] EEG feature extraction 1: Feature extraction is performed on the filtered EEG signal to capture features related to fatigue. Fuzzy entropy: This is an indicator of signal uncertainty and complexity, by calculating entropy at different time scales. The steps for calculating fuzzy entropy are as follows: For a time series of length N

[0071] {x(i), i=1,2,3,...,n}

[0072] Reconstruct the sequence and solve the vector relative distance to obtain Dt ii Dt ij =max(|x(i+k)-x(j+k)|), k∈[0, m-1]

[0073] Fuzzy membership function A(x)

[0074]

[0075] For i=1, 2, .., N-m+1 and j=1, 2, .., N-m+1, and j≠i. For multiple vectors in this range, calculate the mean of A(x)

[0076]

[0077] then

[0078]

[0079] The fuzzy entropy of the original time series is

[0080]

[0081] m can be called the pattern dimension or embedding dimension, r represents the width of the fuzzy function boundary. N determines the slope of the similarity tolerance boundary. The larger N is, the larger the slope is. m = 2, r = 0.2 * the standard deviation of the sequence, N = 2.

[0082] EEG feature extraction 2: STFT spectrum and eye movement features of Fp1 channel: In order to extract features related to visual fatigue and attention state, the EEG signal is bandpass filtered at 5-15Hz to ensure that the blinking feature is more obvious. Then, the following indicators are calculated from the filtered signal:

[0083] STFT calculation formula: where x[k] is the time signal and ω[k] is the chosen window.

[0084] The spectrum of STFT is obtained. The spectrum is calculated every 5000 sampling points, and the feature matrix of the obtained graph is 630*480*3. Through these steps, we can construct a multimodal feature that contains rich information and can be used to analyze and predict the fatigue state of an individual. This analysis has important application value in understanding brain activity, evaluating cognitive function, and monitoring fatigue levels.

[0085] Among them, step 3: transmit the EEG features to the EEG processing module in real time to realize the five-category fatigue detection of the user, as follows:

[0086] The 6*6 EEG spatiotemporal feature matrix composed of the above six features is input into the improved neural network module to realize the five-category fatigue detection of users.

[0087] The improved deep learning modules include YOLOv8's C2f module (Faster Implementation of CSPBottleneckwith 2convolutions), SENet module (Squeeze-and-Excitation Networks), LANet module (localattention network), convolutional layer (Convolutional Layer), pooling layer (PoolingLayer), fully connected layer (FullyConnected Layer), Transformer encoding layer (TransformerEncoder), and output layer.

[0088] First, the 6*6 EEG spatiotemporal feature matrix x is regarded as a single-channel image data and input into the convolution layer. A small-sized convolution kernel is used to extract local features, and batch normalization and activation function ReLU are used to process the convolutional feature matrix.

[0089] y = ReLU(x*W+b)

[0090] Where x is the input matrix, W is the convolution kernel, b is the bias term, and ReLU is the activation function.

[0091] The pooling layer uses the maximum pooling layer with a pooling window size of 2x2 to reduce the feature dimension to obtain the new matrix x1. The C2f module of YOLOv8 is used for preliminary feature extraction. The SENet module and LANet module perform channel attention enhancement and spatial attention enhancement respectively. The Transformer Encoder layer consists of multiple encoders, each of which includes a sub-attention mechanism and a feedforward neural network. The dimension of each encoding layer is 6, the number of attention heads is 3, and the hidden layer dimension of the feedforward network is 128. After the Transformer Encoder layer, the dimension of the feature matrix remains unchanged, denoted as x2.

[0092] y=Attention(x1)

[0093] The EEG spatiotemporal feature matrix x2 processed by the Transformer Encoder layer needs to be flattened before being input into the fully connected layer for classification. The flattening layer transforms its dimension into a 1×n one-dimensional vector x3, and then inputs x3 into the fully connected layer for classification. Finally, the softmax function is used for multi-classification. The number of output neurons is equal to the number of categories, that is, 5 categories. The output value is the predicted probability of each fatigue state.

[0094]

[0095] Where Z i is the output value of the th node, and C is the number of output nodes, that is, the number of categories. The Softmax function can be used to convert the output values ​​of multiple categories into a probability distribution ranging from [0, 1] and 1.

[0096] In order to solve the problem of cross-subject EEG data differences, the present invention adopts a transfer learning method. First, a CNN-Transformer model is pre-trained on a large-scale EEG dataset to extract common EEG features. Then, the feature extraction layer of the pre-trained model is frozen, and a new fully connected layer is added on this basis, and a small amount of new subject data is used to fine-tune the newly added layer. In this way, the adaptability and generalization ability of the model between different individuals can be effectively improved.

[0097] Step 4: Save the received classification results to the memory, and send the results to the monitoring device through the wireless transmission device according to the fixed protocol. Wireless protocols and hardware such as Bluetooth, WiFi, 4G, monitoring devices such as computers, mobile phones or some specific IoT devices.

[0098] Embodiment 3: A fatigue detection method based on electroencephalogram, the method comprising the following steps:

[0099] Step 1: EEG signal acquisition: The driver wears an EEG acquisition device equipped with an ADS1299 chip, which contacts the scalp through gel electrodes to ensure a good signal-to-noise ratio.

[0100] Electrodes were placed in the frontal (Fp1, Fp2), parietal (C3, C4), and occipital (O1, O2) regions, which are closely related to cognitive function, attention, and visual processing.

[0101] Use magnetic beads for low-pass filtering to remove high-frequency noise above 100 Hz.

[0102] Step 2: Feature extraction, using a 0.5-30 Hz bandpass filter to further filter out noise and retain frequency components related to brain activity.

[0103] Fuzzy entropy was calculated to capture the underlying activity level, volatility, and complexity of EEG signals.

[0104] Perform 8-15Hz bandpass filtering on the blink signal, extract the STFT spectrum of Fp1, and construct the features:

[0105] Step 3: Transmit the EEG features to the EEG processing module in real time. The module is deployed on the end-side chip FPGA to achieve fast response.

[0106] Improved Neural Network Model Analysis:

[0107] The EEG spatiotemporal feature matrix is ​​input into the improved neural network module, which includes the C2f module of YOLOv8, the SENet module, the LANet module, the convolution layer, the pooling layer, the Transformer, the flattening layer, the fully connected layer and the output layer.

[0108] Batch Normalization and ReLU activation functions are used to process the convolutional features, and the feature dimension is reduced through the maximum pooling layer.

[0109] The C2f module of YOLOV8 performs preliminary feature extraction to improve the diversity of feature expression.

[0110] The SENet module learns the importance weight of each channel and performs channel attention enhancement.

[0111] The LANet module considers the contextual information of the neighborhood in the feature map, learns the importance of each position in the feature map, suppresses background or unwanted noise, and performs spatial attention enhancement.

[0112] The Transformer encoding layer processes sequential data and captures global dependencies.

[0113] Fatigue status classification:

[0114] The fully connected layer classifies the features and uses the softmax function to output the predicted probabilities of the five fatigue states.

[0115] Step 4: The classification results are fed back to the driver’s mobile phone or vehicle system in real time via the Bluetooth module.

[0116] If the system detects that the driver is in a high fatigue state, it will automatically trigger a warning signal and advise the driver to take a rest or take other safety measures. The classification results of the fatigue state are saved in the memory and can be sent to the monitoring device, such as the medical monitoring center or the security monitoring system, through the wireless transmission device.

[0117] It should be noted that the above embodiments are not intended to limit the protection scope of the present invention, and equivalent changes or substitutions made on the basis of the above technical solutions all fall within the protection scope of the claims of the present invention.

Claims

1. A fatigue detection device based on electroencephalogram, characterized in that: The detection device includes an EEG acquisition module, an EEG processing module, an EEG analysis module and a fatigue feedback module. in, EEG acquisition module: gel electrodes are used to collect the user's original EEG signals. ADS1299 is used as the EEG acquisition chip. Magnetic beads are used at the acquisition end to remove high-frequency noise interference and filter out high-frequency noise in the EEG part. The frontal lobe, parietal lobe, and occipital lobe are used in the brain area, which correspond to the brain's decision-making function, imagination function, and visual function respectively. The EEG processing module, on the one hand, extracts blink signals based on notch filtering, filters the collected EEG signals, finds the number of blink signals, and extracts the blink signal rhythm. On the other hand, it extracts the original EEG rhythm through multi-scale fuzzy entropy to find out the rhythmic characteristics of EEG during fatigue. The EEG analysis module classifies the integrated data using an improved deep learning model, which is deployed on the end-side chip for inference. Fatigue feedback module: The fatigue judgment results obtained by classification are fed back to the user's mobile phone or computer using the Bluetooth module.

2. A fatigue detection method based on electroencephalogram, characterized in that: Using the fatigue detection device according to claim 1, the method comprises the following steps: Step 1: EEG signal acquisition, Step 2: Extract features from the signal. Step 3: Transmit the EEG features to the EEG processing module in real time to achieve five-category fatigue detection for the user. Step 4: Save the received classification results into the memory, and send the results to the monitoring device through the wireless transmission device according to the fixed protocol.

3. The EEG-based fatigue detection method according to claim 2, characterized in that: Step 1: EEG signal acquisition is performed by the EEG acquisition module, as follows: The EEG acquisition module consists of a series of neural sensing electrodes, where the electrodes are gel electrodes that form a layer of medium between the electrodes and the skin. The EEG acquisition points include the frontal pole, central area, and occipital area. These points are usually related to cognitive function, attention, and visual processing. Reference electrodes, which are placed on the left and right earlobes; Filtering processing, the EEG acquisition module contains a low-pass filter, which is used to filter out high-frequency noise above 100Hz. Signal acquisition frequency, the frequency of EEG acquisition is 1000Hz, Real-time transmission, the collected EEG features need to be transmitted to the EEG processing module in real time.

4. The fatigue detection method based on electroencephalogram according to claim 3, characterized in that: Step 2: Extract features from the signal, as follows: Bandpass filtering: First, the raw EEG data were preprocessed using a 0.5-30 Hz second-order Butterworth bandpass filter to retain the frequency components related to brain activity while removing irrelevant noise. EEG feature extraction 1: Feature extraction is performed on the filtered EEG signal to capture features related to fatigue. Fuzzy entropy: This is an indicator to measure the uncertainty and complexity of the signal. By calculating the entropy at different time scales, the steps for calculating the fuzzy entropy are as follows: for a time series of length N {x(i), i=1,2,3,...,n} Reconstruct the sequence and solve the vector relative distance to obtain D ij Dt ij =max(|x(i+k)-x(j+k)|),k∈[0,m-1] Fuzzy membership function A(x) For i=1, 2, .., N-m+1 and j=1, 2, .., N-m+1, and j≠i, calculate the mean of A(x) for multiple vectors in this range then The fuzzy entropy of the original time series is m is called the pattern dimension or embedding dimension, r represents the width of the fuzzy function boundary, and N determines the slope of the similarity tolerance boundary. The larger N is, the larger the slope is. m = 2, r = 0.2 * the standard deviation of the sequence, N = 2, EEG feature extraction 2: STFT spectrum and eye movement features of Fp1 channel: In order to extract features related to visual fatigue and attention state, the EEG signal is bandpass filtered at 5-15Hz to ensure that the blinking feature is more obvious. Then, the following indicators are calculated from the filtered signal: STFT calculation formula: where x[k] is the time signal, ω[k] is the selected window, The spectrum of STFT is obtained. The spectrum is calculated every 5000 sampling points, and the feature matrix of the obtained spectrum is 630*480*3.

5. The fatigue detection method based on electroencephalogram according to claim 4, characterized in that: Step 3: Transmit the EEG features to the EEG processing module in real time to implement five-category fatigue detection of the user, as follows: The above two features are input into the improved deep learning module to realize the five-category fatigue detection of users. The improved deep learning modules include YOLOv8's C2f module (Faster Implementation of CSPBottleneck with 2convolutions), SENet module (Squeeze-and-Excitation Networks), LANet module (local attention network), convolutional layer, pooling layer, fully connected layer, Transformer encoding layer (TransformerEncoder), and output layer. First, the 630*480*3 STFT spectrum map is reduced to 224*240*3 pixels, and then input into the convolution layer 1. A 3*3 convolution kernel is used to extract local features, with a step size of 1 and a padding of 1. Batch normalization is performed after convolution to stabilize the training process and accelerate convergence, and then input into the activation function 1ReLU to process the feature matrix after convolution. y = ReLU(x*W+b) Where x is the input matrix, WW is the convolution kernel, b is the bias term, and ReLU is the activation function. The pooling layer uses the maximum pooling layer with a pooling window size of 2x2 to reduce the feature dimension to obtain a new matrix X0. First, x1 is input into the convolution layer 2, and a 3*3 convolution kernel is used to extract local features, with a step size of 1 and a padding of 1. Batch normalization is performed after convolution to stabilize the training process and accelerate convergence, and then input into the activation function 2ReLU to process the feature matrix after convolution. y = ReLU(x*W+b) Where x is the input matrix, WW is the convolution kernel, b is the bias term, and ReLU is the activation function. The pooling layer uses the maximum pooling layer, and the pooling window size is 2x2, which is used to reduce the feature dimension to obtain the new matrix x1. The size of X1 is 56*60*32. After passing through the C2f module of YOLOv8, a 50*60*96 matrix is ​​obtained. After passing through the fully connected layer, it is flattened to a 630*1 matrix, which is converted into a 63*10 two-dimensional matrix and combined with feature one. Since the STFT spectrum map is one for every 5000 sampling points, the fuzzy entropy is one for every 500 sampling points, and all six channels need to be calculated, it is 6*10 features. The feature matrix of the fuzzy entropy is directly spliced ​​under the 63*10 matrix of the STFT to form a 69*10 feature matrix, which is input into the SENet and LANet modules for channel attention enhancement and spatial attention enhancement, and then input into the Transformer Encoder. The Transformer Encoder layer consists of multiple encoders. Each layer includes a sub-attention mechanism and a feedforward neural network. The dimension of each encoding layer is 6, the number of attention heads is 3, and the hidden layer dimension of the feedforward network is 128. After the Transformer Encoder layer, the dimension of the feature matrix remains unchanged and is recorded as x2. y=Attention(x1) The EEG spatiotemporal feature matrix x2 processed by the Transformer Encoder layer needs to be flattened before being input into the fully connected layer for classification. The flattening layer transforms its dimension into a 1×n one-dimensional vector x3, and then inputs x3 into the fully connected layer for classification. Finally, the softmax function is used for multi-classification. The number of output neurons is equal to the number of categories, that is, 5 categories. The output value is the predicted probability of each fatigue state. Where Z i is the output value of the th node, C is the number of output nodes, that is, the number of classification categories. The Softmax function can be used to convert the output value of multiple classifications into a probability distribution in the range of [0, 1] and 1.

Citation Information

Cited By

  • Brain fatigue detection method, system and device based on electroencephalogram signals and storage medium

    CN120837099A