Electroencephalogram fatigue detection method and system based on space-time convolution recurrent neural network

Abstract

The application belongs to the technical field of artificial intelligence, and particularly relates to an electroencephalogram fatigue detection method and system based on a space-time convolution recurrent neural network, which comprises the following steps: acquiring a multi-lead electroencephalogram signal data segment; inputting the multi-lead electroencephalogram signal data segment into a pre-trained fatigue detection model to obtain a fatigue detection result; the fatigue detection model comprises a space-time convolution recurrent neural network, a differential entropy feature extraction module and a classifier; the space-time convolution recurrent neural network comprises: a multi-stage residual time series convolution module, which adopts a multi-stage processing mode to extract local time sequence features of the multi-lead electroencephalogram signal data segment; at least one layer of a bidirectional long short-term memory network, which processes the local time sequence features to obtain global context time sequence features; the differential entropy feature extraction module extracts differential entropy features of the global context time sequence features; and the classifier processes the differential entropy features to obtain the fatigue detection result. The application improves the fatigue detection accuracy and precision.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method and system for detecting brain fatigue based on spatiotemporal convolutional recurrent neural networks. Background Technology

[0002] Fatigue detection is of great significance in traffic safety, high-risk operation monitoring, and health management. Traditional fatigue detection mainly relies on subjective questionnaires or behavioral observations (such as blinking frequency and posture changes), which suffer from drawbacks such as strong response lag, significant environmental interference, and difficulty in quantification. Therefore, fatigue detection methods based on physiological signals have emerged, such as those based on electrocardiogram (ECG), electromyography (EMG), and electroencephalography (EEG). However, the sensing methods of ECG and EMG are susceptible to motion artifacts, and the functional near-infrared spectroscopy (fNIRS) equipment used in these methods is expensive and has limited spatial resolution. In contrast, EEG, as a direct electrophysiological representation of the central nervous system, can better reflect the brain's fatigue state in real time, hence the emergence of EEG-based fatigue detection methods.

[0003] In fatigue detection technologies based on electroencephalogram (EEG) signals, feature extraction relies on prior knowledge and requires manual design of frequency band power (such as alpha / beta / theta waves) or nonlinear entropy features (such as sample entropy and permutation entropy). The extraction process demands specialized domain knowledge and suffers from poor feature generalization ability. While using short-time segment classification, the limited information contained in these segments leads to poor robustness in fatigue detection methods. Furthermore, the deep learning models used (such as simple convolutional neural networks (CNNs) or simple recurrent neural networks (RNNs) cannot learn complex EEG signal transformation functions and struggle to simultaneously capture the local and global temporal dynamics of EEG signals. This results in insufficient temporal feature coupling, limited feature representation capabilities, and low fatigue detection accuracy. Summary of the Invention

[0004] This application aims to at least solve the technical problems existing in the prior art and provide a method and system for detecting brain fatigue based on spatiotemporal convolutional recurrent neural networks.

[0005] In a first aspect, this application provides a method for detecting brainwave fatigue based on a spatiotemporal convolutional recurrent neural network, the method comprising:

[0006] Acquire multi-lead EEG signal data segments of a preset time length;

[0007] Inputting multi-lead EEG signal data segments into a pre-trained fatigue detection model yields fatigue detection results for the multi-lead EEG signal data segments. The fatigue detection results include probability distributions of various fatigue states. The fatigue detection model includes a spatiotemporal convolutional recurrent neural network, a differential entropy feature extraction module, and a classifier.

[0008] The spatiotemporal convolutional recurrent neural network includes:

[0009] The multi-stage residual temporal convolution module uses a multi-stage processing approach to extract local temporal features from multi-lead EEG signal data segments;

[0010] At least one layer of bidirectional long short-term memory network is used to process local temporal features to obtain global contextual temporal features;

[0011] The differential entropy feature extraction module extracts the differential entropy features of the global context temporal features;

[0012] The classifier classifies the differential entropy features to obtain fatigue detection results.

[0013] Secondly, this application provides a brainwave fatigue detection system based on a spatiotemporal convolutional recurrent neural network, used to implement the brainwave fatigue detection method based on a spatiotemporal convolutional recurrent neural network described in the first aspect of this application, the system comprising:

[0014] The data acquisition module acquires multi-lead EEG signal data segments of a preset time length;

[0015] The detection module inputs multi-lead EEG signal data segments into a pre-trained fatigue detection model to obtain fatigue detection results for the multi-lead EEG signal data segments. The fatigue detection results include probability distributions of various fatigue states. The fatigue detection model includes a spatiotemporal convolutional recurrent neural network, a differential entropy feature extraction module, and a classifier.

[0016] The spatiotemporal convolutional recurrent neural network includes:

[0017] The multi-stage residual temporal convolution module uses a multi-stage processing approach to extract local temporal features from multi-lead EEG signal data segments;

[0018] At least one layer of bidirectional long short-term memory network is used to process local temporal features to obtain global contextual temporal features;

[0019] The differential entropy feature extraction module extracts the differential entropy features of the global context temporal features;

[0020] The classifier classifies the differential entropy features to obtain fatigue detection results.

[0021] Thirdly, this application provides a computer program product, including a computer program, characterized in that, when executed by a processor, the computer program implements the steps of the brainwave fatigue detection method based on a spatiotemporal convolutional recurrent neural network described in the first aspect of this application.

[0022] The beneficial technical effects of this application are as follows:

[0023] 1. The fatigue detection model is based on the analysis and processing of multi-lead EEG signal data segments of a preset time length, which is generally set to about 30 seconds. Compared with the short segments in related technologies, it can capture more useful information.

[0024] 2. In the fatigue detection model, the spatiotemporal convolutional recurrent neural network utilizes a multi-stage residual temporal convolution module to extract local temporal features from multi-lead EEG signal data segments using a multi-stage processing approach. Each stage of processing includes channel number enhancement, dimensionality compression, and temporal feature extraction. By increasing the number of channels and compressing the temporal dimension, the local temporal features of the multi-lead EEG signal data can be extracted more fully. Then, at least one layer of bidirectional long short-term memory network is used to capture the long-term dependencies of the multi-lead EEG signal data and obtain global contextual temporal features. This achieves simultaneous capture of the local and global temporal features of EEG signal data, making the global contextual temporal features more significant and helping to improve the accuracy of fatigue state detection.

[0025] 3. By utilizing the differential entropy feature extraction module to capture the nonlinear dynamic characteristics of EEG signals from global context temporal features, differential entropy features are obtained, which can more deeply reflect EEG features, enabling subsequent classifiers to achieve better classification results. This further improves the accuracy and precision of fatigue state detection, achieving refined fatigue state classification. Compared with related technologies (traditional convolutional neural networks or traditional recurrent neural networks), the fatigue classification accuracy is increased to over 84%, and the error rate is reduced to below 16%.

[0026] 4. Based on data-driven principles, it does not require additional prior information, thus improving the generalization ability and robustness of the fatigue detection model. Attached Figure Description

[0027] Figure 1 This is a flowchart illustrating a preferred embodiment of the EEG fatigue detection method based on a spatiotemporal convolutional recurrent neural network according to the present invention.

[0028] Figure 2 This is a schematic diagram of the fatigue detection model processing procedure in a preferred embodiment of the present invention;

[0029] Figure 3 This is a schematic diagram of the spatiotemporal convolutional recurrent neural network in a preferred embodiment of the present invention;

[0030] Figure 4 This is a schematic diagram of the structure of a first-stage residual temporal convolution unit, a second-stage residual temporal convolution unit, or a third-stage residual temporal convolution unit in a preferred embodiment of the present invention.

[0031] Figure 5 This is a schematic diagram of the structure of a transformed convolution block in a preferred embodiment of the present invention;

[0032] Figure 6 This is a schematic diagram of the structure of a first temporal convolution block or a second temporal convolution block in a preferred embodiment of the present invention;

[0033] Figure 7 This is a schematic diagram of the confusion matrix for the fatigue detection model of the present invention on the test set;

[0034] Figure 8 This is a schematic diagram of the classifier in a preferred embodiment of the present invention. Detailed Implementation

[0035] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0036] The execution entity of the EEG fatigue detection method based on spatiotemporal convolutional recurrent neural network provided by this invention includes, but is not limited to, at least one of the following electronic devices that can be configured to execute the method provided in the embodiments of this application: a server, a terminal, etc. In other words, the EEG fatigue detection method based on spatiotemporal convolutional recurrent neural network can be executed by software or hardware installed on a terminal device or a server device. The server includes, but is not limited to: a single server, a server cluster, a cloud server, or a cloud server cluster. The server can be an independent server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0037] This invention discloses a method for detecting brainwave fatigue based on a spatiotemporal convolutional recurrent neural network. In a preferred embodiment, please see... Figure 1 The method includes:

[0038] Step S1: Acquire multi-lead EEG signal data segments of a preset time length;

[0039] Step S2: Input multi-lead EEG signal data segments into a pre-trained fatigue detection model to obtain fatigue detection results of the multi-lead EEG signal data segments. The fatigue detection results include the probability distribution of various fatigue states. The fatigue detection model includes a spatiotemporal convolutional recurrent neural network, a differential entropy feature extraction module, and a classifier.

[0040] In this embodiment, the executing entity is not limited to reading multi-lead EEG signal data segments from memory, or reading multi-lead EEG signal data segments from other devices (such as multi-lead EEG acquisition devices) using a communication interface (such as a serial interface, parallel interface, or wireless communication interface). The preset time length is 20 to 40 seconds, preferably 30 seconds. Multi-lead EEG signal data segments with a preset time length can retain more useful information. The data from a single lead in the multi-lead EEG signal data segment is one-dimensional data, including the current subject's neural activity information and frequency domain features related to fatigue state.

[0041] In this embodiment, preferably, step S1, acquiring a multi-lead EEG signal data segment of a preset time length, includes:

[0042] Step S11: Acquire multi-lead EEG signal data of the subject, remove reference electrode EEG signal data from the multi-lead EEG signal data, and obtain multi-lead test EEG signal data.

[0043] For example, the multi-lead EEG signal data includes 64 channels of EEG signal data. The mean of the EEG signal data of the reference electrode channel is subtracted from the EEG signal data of all channels, and the reference electrode channel is removed, finally obtaining 63 channels of multi-channel test lead EEG signal data.

[0044] Step S12: Extract raw multi-lead EEG signal data segments of a preset time length from the multi-channel test lead EEG signal data.

[0045] For example, the EEG signal data from multiple test leads is segmented according to a preset time length of 30 seconds to obtain one or more original multi-lead EEG signal data segments of a preset time length. The original multi-lead EEG signal data segments are then downsampled, with the EEG signal data sampling rate of 5000Hz downsampled to 250Hz.

[0046] Step S13: Perform bandpass filtering, power frequency notch filtering, principal component analysis, and dimensional transformation on the original multi-lead EEG signal data segments to obtain multi-lead EEG signal data segments.

[0047] In this embodiment, the frequency range of the bandpass filtering is 4Hz to 45Hz; power frequency notch filtering is performed with a notch center frequency of 50Hz; principal component analysis is used to remove electrooculography and electromyography artifacts, and dimensional transformation processing is performed to adapt to the input requirements of the one-dimensional convolutional layer at the input end of the spatiotemporal convolutional recurrent neural network.

[0048] In this embodiment, the bandpass filtering process for the EEG signal data of each lead in the original multi-lead EEG signal data segment can be expressed as:

[0049]

[0050] Power frequency notch filtering can be represented as:

[0051]

[0052] in, This represents a bandpass filter function, where s is a complex frequency variable; The center angular frequency of the bandpass. , For the bandpass center frequency, For low cutoff frequency and high cutoff frequency The mean; For quality factor, ; This represents the notch filter function, where z is a variable in the Z-transform domain; The center angular frequency of the notch filter. Notch filter center frequency , The sampling frequency of the EEG signal. This is the notch depth control factor.

[0053] In this embodiment, the fatigue detection results include the probability distribution of three fatigue states, and the sum of the probabilities of the three fatigue states equals 1. For example, the three fatigue states are labeled as mental, normal, and fatigued, respectively, according to the degree of fatigue from low to high.

[0054] In this embodiment, please see Figure 2 Spatiotemporal convolutional recurrent neural networks include:

[0055] The multi-stage residual temporal convolution module uses a multi-stage processing approach to extract local temporal features from multi-lead EEG signal data segments. The multi-stage processing approach means extracting local temporal features stage by stage. Each stage of processing includes channel number enhancement, dimensionality compression, and temporal feature extraction.

[0056] At least one layer of bidirectional long short-term memory network is used to process local temporal features to obtain global contextual temporal features; preferably, a two-layer bidirectional long short-term memory network is used.

[0057] Among them, the differential entropy feature extraction module extracts the differential entropy features of the global context temporal features. Specifically, it extracts the differential entropy sub-features of each channel in the global context temporal features, and then integrates the differential entropy sub-features of all channels in the global context temporal features to obtain the differential entropy features.

[0058] The classifier classifies the differential entropy features to obtain fatigue detection results.

[0059] In this embodiment, the classifier is not limited to including one or more cascaded fully connected layers and normalized exponential function (softmax) units.

[0060] In information theory, differential entropy is often used to quantify the uncertainty of random variables. This implementation utilizes a differential entropy feature extraction module to extract the differential entropy features of the global context temporal features, fulfilling a requirement for high-order feature representation: while raw EEG signals contain rich physiological information, they also contain a large amount of noise and nonlinear interference. This application first performs deep temporal feature extraction on multi-lead EEG signal data segments, where: a multi-stage residual temporal convolution module is responsible for extracting local temporal features; a bidirectional long short-term memory network further captures global contextual dependencies. After the above processing, the global context temporal features are no longer the raw EEG signals, but rather highly abstracted and denoised high-order temporal representations. Calculating differential entropy on this basis can more effectively capture the nonlinear dynamic characteristics related to fatigue states, rather than simply the frequency domain energy distribution. This implementation calculates differential entropy on the high-order features output by the temporal convolutional recurrent neural network, essentially quantifying the distribution stability of these features, which helps improve the classification accuracy of fatigue states.

[0061] In this embodiment, the multi-stage residual temporal convolution module increases the number of channels while compressing the number of temporal dimensions in stages to extract more useful local information and obtain local temporal features, thereby increasing the receptive field represented by each point and helping to improve the accuracy of fatigue detection results.

[0062] In a preferred embodiment, please see Figure 3 The multi-stage residual temporal convolution module includes a cascaded first-stage residual temporal convolution unit, a second-stage residual temporal convolution unit, and a third-stage residual temporal convolution unit; the number of channels of the output features of the first-stage residual temporal convolution unit, the second-stage residual temporal convolution unit, and the third-stage residual temporal convolution unit increases sequentially, while the number of temporal dimensions decreases sequentially.

[0063] In this embodiment, a three-stage processing approach is employed to extract local temporal features from multi-lead EEG signal data segments. This approach progressively increases the number of feature channels and compresses the temporal dimension of the features, while simultaneously extracting temporal features at multiple levels. During the process of increasing the number of channels, more channels are obtained by transforming existing channels. This fully captures the local temporal features of the multi-lead EEG signals and effectively balances processing time with the salience of local temporal features. This improves processing efficiency while ensuring high accuracy in fatigue detection results.

[0064] For example, the first-stage residual temporal convolutional unit uses a 64-channel convolutional kernel with a kernel size of 3 to extract local temporal features from the multi-lead EEG signal data segment, obtaining the first local temporal feature. The second-stage residual temporal convolutional unit increases the number of channels of the first local temporal feature to 128 and compresses the temporal dimension using a convolutional kernel with a stride S of 2, extracting the second local temporal feature. The third-stage residual temporal convolutional unit further expands the number of channels of the second local temporal feature to 256 and compresses the temporal dimension again, continuing to extract local temporal features to obtain the local temporal features of the multi-lead EEG signal data segment.

[0065] In this embodiment, preferably, please see Figure 4 At least one of the first-stage residual temporal convolutional unit, the second-stage residual temporal convolutional unit, and the third-stage residual temporal convolutional unit includes:

[0066] Transform the convolutional blocks to increase the number of channels and compress the temporal dimension of the input features of the first-stage residual temporal convolutional unit, the second-stage residual temporal convolutional unit, or the third-stage residual temporal convolutional unit;

[0067] The first temporal convolutional block processes the output features of the transformed convolutional block to obtain the first temporal features;

[0068] The second temporal convolutional block processes the first temporal features to obtain local temporal features.

[0069] In this embodiment, the first-stage residual temporal convolution unit, the second-stage residual temporal convolution unit, or the third-stage residual temporal convolution unit adopts... Figure 4 The stacked structure of one transform convolutional block and two temporal convolutional blocks (first temporal convolutional block and second temporal convolutional block) shown can realize the processing flow of first transforming the number of channels and temporal dimension, and then extracting the local temporal features of EEG data within this dimension, ensuring that the feature extraction is sufficient.

[0070] In this embodiment, preferably, the convolution stride of the transform convolution block of the first-stage residual temporal convolution unit is set to 1, in order to ensure that the dimension remains unchanged in the early stage of feature extraction so as to fully extract early EEG data features.

[0071] In this embodiment, preferably, please see Figure 5 As shown, the transform convolutional block includes:

[0072] The transformation of the main branch includes the sequentially connected first one-dimensional convolution, the first batch of normalized layers, and...

[0073] First activation function unit, first dropout unit, second one-dimensional convolution, second batch normalization layer, first residual unit and second activation function unit;

[0074] The transformed residual branch includes a third one-dimensional convolution and a third batch normalization layer connected in sequence. The input of the third one-dimensional convolution is connected to the input of the first one-dimensional convolution, and the output of the third batch normalization layer is connected to the residual connection of the first residual unit.

[0075] In this embodiment, preferably, the first activation function unit and the second activation function unit adopt the PReLU (Parameterized Corrected Linear Unit) activation function. Preferably, the stride S of the first one-dimensional convolution is 2, and the kernel size is 3; the stride S of the second one-dimensional convolution is 1, and the kernel size is 3; the stride S of the third one-dimensional convolution is 2, and the kernel size is 1. The dropout probability in the first dropout unit is 20%, and dropout regularization is performed. Since the transform convolution block increases the number of channels and reduces the temporal dimension during processing, a third one-dimensional convolution and a third batch normalization layer are set in the transform residual branch to achieve residual adaptation with the first residual unit. The first residual unit is used to perform element-wise addition of the output features of the third batch normalization layer and the output features of the second batch normalization layer.

[0076] In this embodiment, preferably, please see Figure 6 As shown, the first temporal convolutional block or the second temporal convolutional block includes:

[0077] The temporal main branch includes a fourth one-dimensional convolution, a fourth batch normalization layer, a third activation function unit, a second dropout unit, a fifth one-dimensional convolution, a fifth batch normalization layer, a second residual unit, and a fourth activation function unit connected in sequence.

[0078] The temporal residual branch is used to connect the input of the fourth one-dimensional convolution and the residual connection of the second residual unit.

[0079] In this embodiment, the kernel size k of the fourth and fifth one-dimensional convolutions is 3, and the stride S is 1. The third and fourth activation function units use the PReLU (Parameterized Corrected Linear Unit) activation function. The second dropout unit has a dropout probability of 20% and performs dropout regularization. The second residual unit performs element-wise addition of the input features of the first or second temporal convolutional block with the output features of the fifth batch normalization layer.

[0080] Through the detailed structural descriptions of the transformed convolutional block, the first temporal convolutional block, and the second temporal convolutional block, and combined with... Figure 5 and Figure 6 The transformed convolutional block, the first temporal convolutional block, and the second temporal convolutional block each consist of two one-dimensional convolutions in the main branch. After each one-dimensional convolution, batch normalization and activation function processing are performed. An additional dropout regularization with a 20% probability is added after the first one-dimensional convolution. Residual branches are set up for each block to stabilize gradient propagation.

[0081] In this embodiment, in the transformed convolution block of the second-stage residual temporal convolution unit or the third-stage residual temporal convolution unit, the convolution stride S of the third one-dimensional convolution is 2 to ensure that the features after compressing the temporal dimension can be correctly connected with the first residual unit.

[0082] In this embodiment, the two-layer bidirectional long short-term memory network achieves the same number of input channels as the number of output channels in the convolutional layer, with the number of output channels remaining unchanged, and the dropout probability is also set to 20%. Compared to one-dimensional convolution, the two-layer bidirectional long short-term memory network is better at capturing long-term dependencies in sequences by recursively updating memory units, thereby obtaining global contextual temporal features; while the one-dimensional convolution in the multi-stage residual temporal convolution module tends to extract fixed-size contextual information within its receptive field, i.e., obtaining local temporal features. Thus, the two-layer bidirectional long short-term memory network and the multi-stage residual temporal convolution module are completely different yet complementary. Since the computational cost of the two-layer bidirectional long short-term memory network is higher than that of one-dimensional convolution, it would be too time-consuming to directly process multi-lead EEG signal data segments. Therefore, this application first extracts and compresses the local invariant features of the EEG data through the one-dimensional convolution of the multi-stage residual temporal convolution module, and then uses the two-layer bidirectional long short-term memory network to model long-term dependencies. This not only improves the saliency of global contextual temporal features and reduces training time, but also avoids larger memory usage and more model parameters.

[0083] In a preferred embodiment, the differential entropy feature extraction module performs:

[0084] Calculate the variance of the features of each channel in the global context temporal features;

[0085] The differential entropy sub-feature of each channel is calculated based on the variance of each channel;

[0086] Differential entropy features are obtained by integrating the differential entropy sub-features of all channels in the global context temporal features. Specifically, the integration method is not limited to combining the differential entropy sub-features of all channels in the global context temporal features into a single vector.

[0087] In this embodiment, the variance of the features of a certain channel in the global context temporal features is set to Then the differential entropy sub-feature of this channel The calculation formula is:

[0088]

[0089] This represents the natural constant. Differential entropy feature extraction is performed based on the two-layer global context temporal features output after processing by a spatiotemporal convolutional recurrent neural network, which can more deeply reflect EEG characteristics and achieve better fatigue detection results.

[0090] Preferably, such as Figure 8 As shown, the classifier consists of a cascaded first fully connected layer, a second fully connected layer, and a normalized exponential function unit, where the normalized exponential function is the softmax function.

[0091] For example, the shape of the global context temporal feature output by the spatiotemporal convolutional recurrent neural network is (Batch, 6, 256). Batch represents the batch size, which indicates the number of samples processed at one time, and is not limited to 32; 6 represents the number of temporal features per channel of the global context temporal feature; 256 represents the number of channels of the global context temporal feature.

[0092] In the differential entropy feature extraction module, the variance of the six temporal features for each channel of the global context temporal features needs to be calculated first. Then follow the formula The differential entropy sub-feature DE of each channel is calculated. Thus, the differential entropy feature shape output by the differential entropy feature extraction module is (Batch, 256), where 256 represents the differential entropy sub-feature DE of 256 channels.

[0093] The first fully connected layer in the classifier compresses the single feature vector of length 256 for each sample and outputs the first feature with shape (Batch, 32), which is equivalent to compressing the number of channels.

[0094] The second fully connected layer in the classifier further compresses the first feature output by the first fully connected layer to obtain a feature of shape (Batch, 3). For each sample, a prediction score vector of length 3 is obtained. The three elements of the prediction score vector correspond to the prediction scores of three fatigue states: mental, normal, and fatigued.

[0095] The normalization exponential function unit in the classifier normalizes the predicted scores of the three fatigue states (mental, general, and fatigued) in the predicted score vector of each sample to obtain the probability values ​​of the three fatigue states of the sample. The sum of the probability values ​​of the three fatigue states is 1.

[0096] In a preferred embodiment, the training process of the fatigue detection model includes:

[0097] Step 1: Construct a sample set of multi-lead EEG signal data segments and assign a fatigue state label to each multi-lead EEG signal data segment sample. Specifically, this includes:

[0098] (1) Using multi-lead EEG acquisition equipment, multiple raw multi-lead EEG signal data segments of subjects are collected when multiple subjects perform psychomotor alertness task (PVT), or when performing psychomotor alertness task (PVT) and second-order back task (2-back), as well as the reaction time of the subject performing the task corresponding to each raw multi-lead EEG signal data segment.

[0099] (2) For each original multi-lead EEG signal data segment, bandpass filtering, power frequency notch filtering, principal component analysis and dimensional transformation are performed sequentially to obtain a corresponding multi-lead EEG signal data segment sample.

[0100] (3) Three non-overlapping reaction time intervals for fatigue states are pre-defined, corresponding to the three fatigue state labels: mental, normal, and fatigued. A fatigue state label is assigned to the multi-lead EEG data segment sample corresponding to each original multi-lead EEG signal data segment based on the reaction time interval into which the subject's reaction time falls. For example, if the reaction time of a subject corresponding to a certain original multi-lead EEG signal data segment falls within the reaction time interval corresponding to mental, then the fatigue state label of the multi-lead EEG signal data segment sample corresponding to that original multi-lead EEG signal data segment is set to mental.

[0101] The fatigue state labels are converted into one-hot vectors in the order of [mental general fatigue]. For example, the one-hot vector of the mental fatigue state label is represented as [1 0 0].

[0102] Step 2: Divide the multi-lead EEG signal data fragment sample set into a training set, a test set, and a validation set according to a preset ratio, which is not limited to 8:1:1.

[0103] Step 3: Construct a fatigue detection model network, which includes a spatiotemporal convolutional recurrent neural network, a differential entropy feature extraction module, and a classifier.

[0104] Step 4: Iteratively train the fatigue detection model network using the training set until the training stopping condition is met. The training stopping condition is not limited to reaching the preset maximum number of training iterations, or the loss function... The value is lower than the preset loss threshold. The loss function is calculated during training. And according to the loss function The network parameters of the fatigue detection model network are updated using the gradient descent method.

[0105] Preferably, the loss function for:

[0106]

[0107] in, Index representing fatigue state; The one-hot vector representing the fatigue state label of the sample. The component values ​​corresponding to each fatigue state; This represents the fatigue detection result of the sample output by the fatigue detection model. The probability value of a fatigue state.

[0108] Step 5: Test and validate the fatigue detection model network and its parameters trained in Step 4 using the test set and validation set respectively. If the test and validation are successful, obtain the fatigue detection model by combining the fatigue detection model network and its parameters trained in Step 4. If the test or validation fails, adjust the training parameters (such as the learning rate) and return to Step 4 and Step 5.

[0109] In this embodiment, for the test set, the predicted values ​​in the fatigue detection results of the fatigue detection model are compared with the actual values ​​of the fatigue state labels. Figure 7 The confusion matrix of the fatigue detection model obtained from training is shown on the test set.

[0110] Accuracy verification experiments were conducted on a test set using traditional convolutional neural networks, traditional recurrent neural networks, and the fatigue detection model of this application. The comparison results shown in Table 1 below indicate that the fatigue detection model of this application has the highest accuracy.

[0111] Table 1 Accuracy Comparison Results

[0112]

[0113] This invention also discloses a brainwave fatigue detection system based on a spatiotemporal convolutional recurrent neural network, used to implement the aforementioned brainwave fatigue detection method based on a spatiotemporal convolutional recurrent neural network. In a preferred embodiment, the system includes:

[0114] The data acquisition module acquires multi-lead EEG signal data segments of a preset time length;

[0115] The detection module takes multi-lead EEG signal data segments as input to a pre-trained fatigue detection model and obtains fatigue detection results for the multi-lead EEG signal data segments. The fatigue detection results include the probability distribution of various fatigue states. The fatigue detection model includes a spatiotemporal convolutional recurrent neural network, a differential entropy feature extraction module, and a classifier.

[0116] Spatiotemporal convolutional recurrent neural networks include:

[0117] The multi-stage residual temporal convolution module uses a multi-stage processing approach to extract local temporal features from multi-lead EEG signal data segments. Each stage of processing includes channel number enhancement, dimensionality compression, and temporal feature extraction.

[0118] At least one layer of bidirectional long short-term memory network is used to process local temporal features to obtain global contextual temporal features;

[0119] Among them, the differential entropy feature extraction module extracts the differential entropy features of the global context temporal features;

[0120] The classifier classifies the differential entropy features to obtain fatigue detection results.

[0121] In this embodiment, the data acquisition module and the detection module correspond one-to-one with steps S1 and S2 of the above-described EEG fatigue detection method based on spatiotemporal convolutional recurrent neural networks, and will not be described again here.

[0122] The present invention also discloses a computer program product, including a computer program that, when executed by a processor, implements the steps of the above-mentioned brainwave fatigue detection method based on spatiotemporal convolutional recurrent neural network provided by the present invention. The computer program product should be understood as a software product that mainly implements its solution through a computer program, such as a program product integrated in the cloud or a software library.

[0123] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

Claims

1. A brain fatigue detection method based on spatiotemporal convolutional recurrent neural networks, characterized in that, The method includes: Acquire multi-lead EEG signal data segments of a preset time length; Inputting multi-lead EEG signal data segments into a pre-trained fatigue detection model yields fatigue detection results for the multi-lead EEG signal data segments. The fatigue detection results include probability distributions of various fatigue states. The fatigue detection model includes a spatiotemporal convolutional recurrent neural network, a differential entropy feature extraction module, and a classifier. The spatiotemporal convolutional recurrent neural network includes: The multi-stage residual temporal convolution module uses a multi-stage processing approach to extract local temporal features from multi-lead EEG signal data segments; At least one layer of bidirectional long short-term memory network is used to process local temporal features to obtain global contextual temporal features; The differential entropy feature extraction module extracts the differential entropy features of the global context temporal features; The classifier classifies the differential entropy features to obtain fatigue detection results. The multi-stage residual temporal convolution module includes a cascaded first-stage residual temporal convolution unit, a second-stage residual temporal convolution unit, and a third-stage residual temporal convolution unit; the number of channels of the output features of the first-stage residual temporal convolution unit, the second-stage residual temporal convolution unit, and the third-stage residual temporal convolution unit increases sequentially, while the number of temporal dimensions decreases sequentially.

2. The brain fatigue detection method based on spatiotemporal convolutional recurrent neural network according to claim 1, characterized in that, At least one of the first-stage residual temporal convolutional unit, the second-stage residual temporal convolutional unit, and the third-stage residual temporal convolutional unit includes: Transform the convolutional blocks to increase the number of channels and compress the temporal dimension of the input features of the first-stage residual temporal convolutional unit, the second-stage residual temporal convolutional unit, or the third-stage residual temporal convolutional unit; The first temporal convolutional block processes the output features of the transformed convolutional block to obtain the first temporal features; The second temporal convolutional block processes the first temporal features to obtain local temporal features.

3. The EEG fatigue detection method based on spatiotemporal convolutional recurrent neural network according to claim 2, characterized in that, Transformed convolutional blocks include: The transformation of the main branch includes the sequentially connected first one-dimensional convolution, the first batch of normalized layers, and... First activation function unit, first dropout unit, second one-dimensional convolution, second batch normalization layer, first residual unit and second activation function unit; The transformed residual branch includes a third one-dimensional convolution and a third batch normalization layer connected in sequence. The input of the third one-dimensional convolution is connected to the input of the first one-dimensional convolution, and the output of the third batch normalization layer is connected to the residual connection of the first residual unit.

4. The EEG fatigue detection method based on spatiotemporal convolutional recurrent neural network according to claim 2, characterized in that, The first temporal convolutional block or the second temporal convolutional block includes: The temporal main branch includes a fourth one-dimensional convolution, a fourth batch normalization layer, a third activation function unit, a second dropout unit, a fifth one-dimensional convolution, a fifth batch normalization layer, a second residual unit, and a fourth activation function unit connected in sequence. The temporal residual branch is used to connect the input of the fourth one-dimensional convolution and the residual connection of the second residual unit.

5. The brain fatigue detection method based on spatiotemporal convolutional recurrent neural network according to any one of claims 1-4, characterized in that, The acquisition of multi-lead EEG signal data segments of a preset time length includes: Multi-lead EEG signal data of subjects were collected, and reference electrode EEG signal data were removed from the multi-lead EEG signal data to obtain multi-lead test EEG signal data. Extract raw multi-lead EEG signal data segments of a preset time length from multi-channel test lead EEG signal data; The original multi-lead EEG signal data segments were processed by bandpass filtering, power frequency notch filtering, principal component analysis, and dimensional transformation to obtain multi-lead EEG signal data segments.

6. The EEG fatigue detection method based on spatiotemporal convolutional recurrent neural networks according to any one of claims 1-4, characterized in that, During the training process of the fatigue detection model, the loss function for: in, Index representing fatigue state; The one-hot vector representing the fatigue state label of the sample. The component values ​​corresponding to each fatigue state; This represents the fatigue detection result of the sample output by the fatigue detection model. The probability value of a fatigue state.

7. The brain fatigue detection method based on spatiotemporal convolutional recurrent neural network according to any one of claims 1-4, characterized in that, The differential entropy feature extraction module executes as follows: Calculate the variance of the features of each channel in the global context temporal features; The differential entropy sub-feature of each channel is calculated based on the variance of each channel; Differential entropy features are obtained by integrating the differential entropy sub-features of all channels in the global context temporal features.

8. A brainwave fatigue detection system based on a spatiotemporal convolutional recurrent neural network, used to implement the brainwave fatigue detection method based on a spatiotemporal convolutional recurrent neural network as described in any one of claims 1-7, characterized in that, The system includes: The data acquisition module acquires multi-lead EEG signal data segments of a preset time length; The detection module inputs multi-lead EEG signal data segments into a pre-trained fatigue detection model to obtain fatigue detection results for the multi-lead EEG signal data segments. The fatigue detection results include probability distributions of various fatigue states. The fatigue detection model includes a spatiotemporal convolutional recurrent neural network, a differential entropy feature extraction module, and a classifier. The spatiotemporal convolutional recurrent neural network includes: The multi-stage residual temporal convolution module uses a multi-stage processing approach to extract local temporal features from multi-lead EEG signal data segments; At least one layer of bidirectional long short-term memory network is used to process local temporal features to obtain global contextual temporal features; The differential entropy feature extraction module extracts the differential entropy features of the global context temporal features; The classifier classifies the differential entropy features to obtain fatigue detection results.

9. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the steps of the brain fatigue detection method based on a spatiotemporal convolutional recurrent neural network as described in any one of claims 1-7.

Images