A brain fatigue detection method, system, device and storage medium based on electroencephalogram signals

Abstract

The present application relates to the technical field of artificial intelligence, and relates to a brain fatigue detection method, system, device and storage medium based on electroencephalogram signals; wherein the brain fatigue detection method based on electroencephalogram signals comprises: acquiring an electroencephalogram signal sequence of a subject; inputting the electroencephalogram signal sequence into a feature extraction network, the feature extraction network extracting spatial features and time features of each electroencephalogram signal in the electroencephalogram signal sequence respectively, and fusing the spatial features and the time features to obtain spatio-temporal fusion features of the electroencephalogram signal; calculating differential entropy features of the electroencephalogram signal according to the spatio-temporal fusion features; and determining a brain fatigue state detection result of the electroencephalogram signal of the subject according to the differential entropy features. The present application can make the recognized brain fatigue state detection result more accurate.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to a brain fatigue detection method, system, device and storage medium based on electroencephalogram signals. BACKGROUND

[0002] At present, the fatigue state detection technology based on electroencephalogram (EEG for short) is widely used in driving safety, medical rehabilitation and other fields. The fatigue state detection technology based on electroencephalogram analyzes the characteristics of brain electrical activity to quantify the degree of brain fatigue. The process of brain fatigue detection method includes signal acquisition, feature extraction and fatigue classification modeling (such as classifying fatigue state by machine learning or deep learning), and the brain fatigue detection result is obtained.

[0003] At present, the fatigue state detection technology based on electroencephalogram has many technical defects, and the specific research situation is as follows:

[0004] The patent application with the patent name EEG-based fatigue detection and the publication number WO2004028362A1 uses the traditional frequency domain analysis method to identify the fatigue state by the power change of frequency band, but this method has a high error rate, and needs additional electrodes to monitor multiple physiological signals, so the system complexity is high, and it is difficult to popularize in practical application.

[0005] The patent application with the invention name EEG signal power spectrum and R value evaluation method for discomfort caused by watching 3D image and the publication number CN102657526A proposes an electroencephalogram evaluation method based on power spectrum analysis, but it is only suitable for specific visual fatigue scenarios, lacks effective recognition ability for general fatigue state, and the feature extraction method is single, which cannot capture the complex nonlinear characteristics of electroencephalogram signals.

[0006] The patent application with the invention name EEG signal unsupervised feature learning method based on convolutional network and auto-encoding and the publication number CN108256629B tries to introduce deep learning technology, and uses convolutional auto-encoder for feature learning, but only uses a single unsupervised learning architecture, lacks effective learning of discriminative features between different fatigue states, and does not consider the individual differences between subjects, resulting in serious lack of cross-individual generalization ability.

[0007] In addition, the patent application with the disclosure number CN118303883A and the invention name of fatigue detection method based on EEG signal and EOG signal feature fusion proposes a method of fusing EEG (Electroencephalogram) and EOG (Electrooculogram) signals, performing fatigue detection through multi-level feature extraction and feature splicing such as spatial convolution and time convolution. Although this method has made progress in feature extraction, it relies on the synchronous collection of multiple physiological signals, and the system structure is complex. In addition, this patent application mainly realizes two classification or coarse classification, and it is difficult to meet the needs of fine and multi-level recognition of fatigue state in actual application.

[0008] The above-mentioned existing technologies generally have the following key problems: (1) feature extraction is severely dependent on traditional signal processing methods or simple convolution structures, and cannot adaptively learn complex EEG patterns, with limited feature expression ability; (2) there is no effective processing mechanism for individual differences of subjects, resulting in large performance differences between models in different individuals; (3) the classification accuracy is generally low and unstable, making it difficult to achieve accurate recognition of multi-level fatigue state, and unable to meet the needs of actual application. SUMMARY

[0009] The present application aims to at least solve the technical problems existing in the prior art, and provides a brain fatigue detection method, system, device and storage medium based on EEG signals.

[0010] In a first aspect, the present application provides a brain fatigue detection method based on EEG signals, which comprises:

[0011] obtaining an EEG signal sequence of a subject;

[0012] inputting the EEG signal sequence into a feature extraction network, the feature extraction network extracting spatial features and time features of each EEG signal in the EEG signal sequence respectively, and fusing the spatial features and the time features to obtain spatio-temporal fusion features of the EEG signal;

[0013] calculating differential entropy features of the EEG signal according to the spatio-temporal fusion features;

[0014] determining a brain fatigue state detection result of the EEG signal of the subject according to the differential entropy features.

[0015] By adopting the technical solution, the fatigue state recognition is only performed by using the scalp electroencephalogram (EEG), without the need of synchronously collecting other physiological signals such as the electrooculogram, the system structure is greatly simplified, the hardware cost is reduced, and the actual promotion and large-scale application are facilitated, which is different from a complex system relying on multi-modal signals; the multi-level features (spatial features and time features) of the electroencephalogram are extracted, and the space-time fusion features are used for calculation of the differential entropy features. The differential entropy features of the space-time fusion features of the electroencephalogram are calculated to obtain enhanced feature representation, the high-order, nonlinear, and space-time coupling features automatically learned by the deep network are fully mined, the discriminability of the features is further enhanced through the statistical measurement of the differential entropy, and the brain fatigue state detection result recognized is more accurate.

[0016] Optionally, the method further comprises:

[0017] The method further comprises:

[0018] The method further comprises:

[0019] The method further comprises:

[0020] By adopting the technical solution, the pseudo-traces in the electroencephalogram are effectively removed through the preprocessing of the electroencephalogram raw data stream, and the signal-to-noise ratio of the electroencephalogram is improved.

[0021] Optionally, the method further comprises:

[0022] The method further comprises:

[0023] The method further comprises:

[0024] The method further comprises:

[0025] In each training, the feature extraction network extracts the space-time features of the positive samples and the space-time features of the negative samples, respectively, determines the normalized temperature scaling cross-entropy loss of the feature extraction network according to the space-time features of the positive samples and the space-time features of the negative samples, and performs self-supervised contrastive learning pre-training according to the normalized temperature scaling cross-entropy loss to optimize the network parameters of the feature extraction network, thereby obtaining the final feature extraction network.

[0026] By adopting the technical solution, the contrast learning technology is adopted based on the training data set, a space-time convolutional neural network architecture is built for contrast learning training, and the feature extraction network learns the neural activity pattern and state difference feature related to fatigue; after the training is completed, the to-be-tested electroencephalogram signal is input into the pre-trained feature extraction network to extract features, and the differential entropy and multilayer perception classification are combined to realize accurate recognition of the cross-subject brain fatigue state.

[0027] Optionally, for the negative samples being electroencephalogram signals of different subjects, after the space-time features of the negative samples are extracted, the space-time feature distribution of the negative samples is subjected to hierarchical normalization processing to obtain standardized negative sample space-time features.

[0028] The normalized temperature scaling cross-entropy loss is used for self-supervised contrast learning pre-training based on the space-time features of the positive samples and the standardized negative sample space-time features, and the parameters of the feature extraction network are optimized.

[0029] By adopting the technical solution, the hierarchical normalization strategy is used to standardize the features, which can ensure the training stability, significantly improve the generalization ability and robustness of the model under the cross-subject and cross-scene conditions, and enable the method to adapt to the changes of electroencephalogram signals in different individuals and various practical application environments, thereby effectively reducing the performance fluctuations caused by individual differences and environmental noise.

[0030] Optionally, the calculation formula of the hierarchical normalization processing is as follows:

[0031]

[0032] wherein, is the original feature, is the mean value of the subject electroencephalogram signal sequence data, is the standard deviation of the subject electroencephalogram signal sequence data, is a small constant used to prevent division by zero.

[0033] By adopting the technical solution, the specific processing steps of the hierarchical normalization processing are defined.

[0034] Optionally, the loss function expression of the normalized temperature scaling cross-entropy loss is as follows:

[0035] ;

[0036] wherein, , and respectively represent different subjects, and respectively represent different stimuli, is a temperature parameter, is a cosine similarity function, representing a subject a feature vector of an electroencephalogram signal generated under the first stimulation, representing a subject a feature vector of an electroencephalogram signal generated under the first stimulation, representing a subject a feature vector of an electroencephalogram signal generated under the first stimulation, representing the number of stimulations received by the subject within a batch; representing an indicator function, taking a value of 1 when , and a value of 0 when .

[0037] By adopting the technical solution described above, the specific calculation manner of the loss function is clarified.

[0038] Optionally, the brain fatigue detection result includes at least two fatigue state levels, and a probability distribution corresponding to each fatigue state level.

[0039] In a second aspect, the present application provides a brain fatigue detection system based on electroencephalogram signals, the system comprising:

[0040] an acquisition module configured to acquire an electroencephalogram signal sequence of a subject;

[0041] a feature extraction module configured to input the electroencephalogram signal sequence into a feature extraction network, the feature extraction network being configured to extract spatial features and temporal features of each electroencephalogram signal in the electroencephalogram signal sequence, and to fuse the spatial features and the temporal features to obtain spatio-temporal fusion features of the electroencephalogram signal;

[0042] a differential entropy feature calculation module configured to calculate differential entropy features of the electroencephalogram signal based on the spatio-temporal fusion features;

[0043] an output module configured to determine a brain fatigue state detection result of the electroencephalogram signal of the subject based on the differential entropy features.

[0044] In a third aspect, the present application provides an electronic device, the electronic device comprising:

[0045] at least one processor; and

[0046] a memory communicatively connected to the at least one processor; wherein

[0047] the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to execute the brain fatigue detection method based on electroencephalogram signals described above.

[0048] In a fourth aspect, the present application also provides a computer readable storage medium, wherein at least one computer program is stored in the computer readable storage medium, and the at least one computer program is executed by a processor in an electronic device to implement the above-mentioned brain fatigue detection method based on electroencephalogram signals.

[0049] In summary, the present application has the following beneficial technical effects:

[0050] The present application only utilizes scalp electroencephalogram signals (EEG) for fatigue state recognition, without the need for synchronous acquisition of eye signals and other physiological signals, greatly simplifying the system structure, reducing the hardware cost, facilitating actual promotion and large-scale application, and being different from complex systems relying on multi-modal signals.

[0051] The multi-level features (spatial features and temporal features) of the electroencephalogram signals are extracted, and the spatio-temporal fusion features are used for calculation of differential entropy features. By calculating the differential entropy features of the spatio-temporal fusion features of the electroencephalogram signals, enhanced feature representations are obtained, which can fully excavate the high-order, nonlinear, spatio-temporal coupling features automatically learned by the deep network, further enhance the discriminability of the features through statistical measurement of the differential entropy, and make the detection result of the recognized brain fatigue state more accurate.

[0052] Based on the training data set, a contrast learning technology is adopted to build a spatio-temporal convolutional neural network architecture for contrast learning training, so that the feature extraction network learns the fatigue-related neural activity patterns and state difference features. After training, the electroencephalogram signals to be detected are input into the pre-trained feature extraction network to extract features, and the differential entropy and multi-layer perceptron classification are combined to realize accurate recognition of the brain fatigue state of different subjects. BRIEF DESCRIPTION OF DRAWINGS

[0053] Figure 1 A flowchart of the brain fatigue detection method based on electroencephalogram signals provided by an embodiment of the present application is shown;

[0054] Figure 2 A raw data stream waveform graph of the electroencephalogram signals of a subject is shown;

[0055] Figure 3 A waveform graph of the electroencephalogram signal sequence obtained after preprocessing is shown;

[0056] Figure 4 A network system structure diagram of the brain fatigue state recognition method based on contrast learning and scalp electroencephalogram signals provided by an embodiment of the present application is shown;

[0057] Figure 5 A differential entropy feature visualization result extracted by the encoder trained by contrast learning of the present application is shown;

[0058] Figure 6 A classification confusion matrix result graph is shown;

[0059] Figure 7 Fig. 1 shows a structural schematic diagram of an electronic device for implementing the brain fatigue detection method based on electroencephalogram signals according to an embodiment of the present application.

[0060] Fig. 1 shows a structural schematic diagram of an electronic device for implementing the brain fatigue detection method based on electroencephalogram signals according to an embodiment of the present application.

[0061] The implementation, functional features and advantages of the present application will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0062] The embodiments of the present application will be described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar notations represent the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by referring to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be understood as a limitation of the present application.

[0063] In the description of the present application, it should be understood that the terms "longitudinal", "transverse", "upper", "lower", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer" and the like indicate the orientation or positional relationship shown in the drawings based on the orientation or positional relationship shown in the drawings, and are only used to facilitate the description of the present application and simplify the description, and therefore cannot be understood as indicating or implying that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation of the present application.

[0064] In the description of the present application, unless otherwise specified and limited, it should be noted that the terms "mounting", "connecting", "connecting" should be understood broadly, for example, it can be a mechanical connection or an electrical connection, or a communication between two elements, or a direct connection, or an indirect connection through an intermediate medium, and the specific meaning of the above terms can be understood by a person of ordinary skill in the art according to the specific circumstances.

[0065] Referring to Figure 1 Fig. 1 shows a structural schematic diagram of an electronic device for implementing the brain fatigue detection method based on electroencephalogram signals according to an embodiment of the present application.

[0066] S1, obtaining an electroencephalogram signal sequence of a subject.

[0067] Specifically, the subject is an individual participating in an experiment, and the subject provides scalp electroencephalogram signals by wearing an electroencephalogram acquisition device (such as an electrode cap); before the electroencephalogram acquisition device is used to collect the electroencephalogram signals of the subject, the subject needs to clean the hair near the scalp to ensure the quality of the collected signals. The electroencephalogram signals include the neural activity information of the current subject and the frequency domain features related to the fatigue state.

[0068] The neural activity information and the frequency domain features related to the fatigue state are features corresponding to emotion-related brain regions, that is, the activation intensity and distribution pattern of different brain regions (such as the temporal lobe, the frontal lobe, and the occipital lobe) under emotional stimulation. By analyzing the neural activity information and the frequency domain features related to the fatigue state of the previous subject, the fatigue state of the subject's brain can be determined.

[0069] In this embodiment, the step of obtaining the electroencephalogram signal sequence of the subject includes:

[0070] S11, collecting an electroencephalogram signal raw data stream of the subject.

[0071] S12, sequentially performing band-pass filtering, power frequency notch filtering, and independent component analysis processing on the electroencephalogram signal raw data stream to obtain a preprocessed data stream.

[0072] In one example of this embodiment, the process of preprocessing the electroencephalogram signal raw data stream is as follows: first, the electroencephalogram signal raw data stream of the current subject is stably and accurately collected through a programming interface; first, the electroencephalogram signal raw data stream is passed through a band-pass filter, and the band-pass filtering frequency range of the band-pass filter is 0.5-100 Hz; then, the electroencephalogram signal data stream output from the band-pass filter is subjected to power frequency notch filtering, and the power frequency notch filtering frequency is 50 Hz; finally, the electroencephalogram signal data stream subjected to power frequency notch filtering is imported into an independent component analysis tool (such as EEGLAB, Electroencephalogram Laboratory), and a channel position information template (such as the standard-10-5-cap385 template of the international 10-5 system) matching the recorded data is loaded to determine the spatial distribution of the electrodes; the independent component analysis tool decomposes the multi-channel electroencephalogram signal into statistically independent components, each component contains a time process and a topographic distribution, and artifacts such as eye blinking, eye drift, and power frequency interference are separated out, thereby effectively removing artifacts in the electroencephalogram signal and improving the signal-to-noise ratio of the electroencephalogram signal.

[0073] S13, dividing the preprocessed data stream into a plurality of electroencephalogram data segments using a time window to obtain an electroencephalogram signal sequence.

[0074] In one example of this embodiment, the length of the time window is 2.5 seconds, corresponding to 1250 electroencephalogram sampling points, and the electroencephalogram signal data segments are divided with a 50% overlap rate.Figure 2 and Figure 3 , Figure 2 is a raw data stream waveform diagram of the brain electrical signal of the subject, Figure 3 is a sequence waveform diagram of the brain electrical signal after preprocessing, and preprocessing of the raw data stream of the brain electrical signal can effectively remove artifacts in the brain electrical signal and improve the signal-to-noise ratio of the brain electrical signal.

[0075] The application only uses scalp electroencephalogram (EEG) for fatigue state recognition, without the need for synchronous collection of eye signals and other physiological signals, greatly simplifying the system structure, reducing the hardware cost, facilitating actual promotion and large-scale application, and being different from a complex system relying on multi-modal signals.

[0076] S2, input the brain electrical signal sequence into a feature extraction network, the feature extraction network extracts spatial features and temporal features of each brain electrical signal in the brain electrical signal sequence respectively, and fuses the spatial features and the temporal features to obtain spatiotemporal fusion features of the brain electrical signal.

[0077] Reference Figure 4 The feature extraction network is a deep neural network, and the application adopts an end-to-end deep neural network structure to automatically complete feature learning and classification, and the model structure is compact and efficient, suitable for real-time deployment on terminals such as wearable devices, and convenient for continuous monitoring and feedback in actual scenes.

[0078] The feature extraction network includes a spatial convolution layer and a temporal convolution layer connected in sequence, the preprocessed brain electrical signal data is input into the spatial convolution layer, the spatial convolution layer uses 16 63x1 convolution kernels to extract spatial distribution features of the brain electrical signal sequence to obtain spatial features; the spatial convolution result output by the spatial convolution layer is input into the temporal convolution layer, the temporal convolution layer uses 16 1x60 convolution kernels to dynamically extract time sequence features of the brain electrical signal to obtain temporal features, and fuses the spatial features and the temporal features to output a temporal convolution result; the temporal convolution result output by the temporal convolution layer is subjected to an average pooling operation, the pooling window size is 1x30, and an activation function is used to impose a nonlinear transformation on the output of the neuron; finally, a 2-fold channel expansion factor is used to flatten the feature map after the average pooling operation and the activation function processing to output a 9216-dimensional feature vector, thereby obtaining the spatiotemporal fusion features of the brain electrical signal.

[0079] In this embodiment, the calculation formula of the spatial convolution operation is:

[0080]

[0081] wherein, represents the output of the i-th spatial filter, i represents the index of the spatial filter, represents an activation function, represents a spatial convolution kernel weight, represents the electroencephalogram signal of the jth electrode, j represents the index of the electrode of the electroencephalogram signal acquisition device collecting the electroencephalogram signal, represents the number of electrode channels, represents the bias term of the ith spatial filter.

[0082] The calculation formula of the time convolution operation is:

[0083]

[0084] wherein, represents the output of the jth time filter, represents the index of the spatial filter, represents the time convolution kernel weight, represents the index of the time step, represents the time offset, represents the bias term of the jth time filter. S3, calculating the differential entropy feature of the electroencephalogram signal according to the spatio-temporal fusion feature.

[0085] The calculation formula of the differential entropy feature of the electroencephalogram signal is:

[0086]

[0087]

[0088] wherein, represents the spatio-temporal fusion feature of the electroencephalogram signal, represents the spatio-temporal fusion feature of the electroencephalogram signal corresponding differential entropy feature, is the variance of the spatio-temporal fusion feature of the electroencephalogram signal in the time dimension, and e is a natural constant.

[0089] The feature extraction network automatically extracts multi-level features of the electroencephalogram signal by using spatial convolution layers and time convolution layers, and uses the features of the intermediate layers of the deep network (i.e. the extracted spatio-temporal fusion features) for the calculation of the differential entropy feature. This method is different from the traditional differential entropy feature extraction method, which can fully exploit the high-order, nonlinear, spatio-temporal fusion features learned by the deep network, and further enhance the discriminability of the features through the statistical measurement of the differential entropy, so that the detected brain fatigue state detection result is more accurate.

[0090] Referring to Figure 5 , the present application fuses the deep neural network with the differential entropy and other statistical features, which can fully exploit the high-order nonlinear and spatio-temporal fusion features of the electroencephalogram signal, improve the accuracy and expressiveness of the fatigue state recognition, and overcome the dependence on artificial feature design and the limited expressiveness of the traditional method. ​​

[0091] S4, determining a brain fatigue state detection result of the subject's electroencephalogram according to the differential entropy feature.

[0092] The brain fatigue state detection result includes at least two fatigue state grades, and a probability distribution corresponding to each fatigue state grade. In this embodiment, the brain fatigue state grade is four, and the four brain fatigue state grades are mild fatigue, moderate fatigue, severe fatigue and extreme fatigue. In this embodiment, the differential entropy feature is input into the classifier for reasoning, and the probability distribution of the four fatigue states is output by the classifier.

[0093] Referring to Figure 5 , the present application meets the actual application requirements by four-level fine classification of mild fatigue, moderate fatigue, severe fatigue and extreme fatigue, breaks through the limitation that the prior art cannot accurately identify multiple fatigue states, and has overall performance significantly better than the prior art, meeting the actual application requirements for fatigue classification.

[0094] In addition, in the preferred embodiment of the present embodiment, the present application adopts a compact classifier structure, which avoids the overfitting risk brought by a complex network structure, fully adapts to the statistical characteristics of the electroencephalogram differential entropy feature, has good generalization ability and calculation efficiency, and is convenient for deployment and application in actual scenarios.

[0095] The differential entropy feature is input into the classifier for reasoning; and a probability distribution including four fatigue states is output to complete automatic recognition of fatigue states across subjects.

[0096] The complete classification pipeline can be represented as:

[0097]

[0098] Among them, is the original electroencephalogram signal, Preprocess(.) represents a preprocessing operation, CNN(.) represents a spatiotemporal convolution feature extraction, DE(.) represents a differential entropy feature calculation, a softmax function is used to convert an input vector into a probability distribution, an argmax function is used to obtain the index of the maximum element in the vector, and is commonly used in classification tasks to determine the final class; p represents a fatigue state grade index, is the final fatigue state prediction result.

[0099] In this embodiment, in order to verify the effectiveness of the proposed method, detailed experimental verification was performed on a multi-subject electroencephalogram fatigue dataset. 63 effective electrodes (reference electrode removed) were used, the sampling frequency was 500 Hz, and about 2 hours of continuous electroencephalogram recording was recorded for each subject, covering four fatigue states: mild fatigue, moderate fatigue, severe fatigue and extreme fatigue. The detailed performance of each fatigue state is shown in Table 1 and Table 2. Figure 6As shown, Table 1 shows the classification performance of various fatigue states, Figure 6 The confusion matrix result map is used for classification, which shows the comparison between the model prediction results and the real labels in detail.

[0100] Table 1 Classification performance of various fatigue states

[0101]

[0102] Compared with the high error rate of the traditional frequency domain analysis method, the feature extraction network of the present application automatically extracts multi-level and space-time coupled deep features, and combines with differential entropy calculation, which can fully capture the high-order nonlinear dynamic characteristics of the electroencephalogram signal, and improve the fatigue state classification accuracy.

[0103] In another embodiment of the present application, the brain fatigue detection method based on electroencephalogram signal further includes a step of training the feature extraction network, and the training step is as follows:

[0104] S21, obtaining a training data set.

[0105] Specifically, the training data set can use the currently commonly used scalp electroencephalogram signal data set, such as the clinical application data set (Temple University Hospital EEG Corpus Database, TUH EEG data set) and the epilepsy research data set (CHB-MIT Scalp EEG data set, the English full name of CHB-MIT Scalp EEG data set is Children's Hospital Boston-Massachusetts Institute of Technology Scalp EEG Database). The TUH EEG data set contains tens of thousands of clinical records, covering diseases such as epilepsy and sleep disorders. The data is accompanied by patient basic information and diagnosis information, which is suitable for neuroscience research and machine learning model training; the CHB-MIT Scalp EEG data set contains long-term scalp electroencephalogram records (sampling rate 256Hz) of 23 pediatric epilepsy patients, covering the stages of seizure, interictal and preictal; the scalp electroencephalogram signals of different subjects can also be collected by using the electroencephalogram acquisition device as the training data set, and the present embodiment does not make any limitation.

[0106] In the process of collecting scalp EEG signals of different subjects as the training data set by using the brain electrical acquisition device, the subjects of the collected EEG signals in the training data set include healthy volunteers of different ages, genders, and handedness (dominant hand) habits, as well as patients with specific diseases (such as stroke and amyotrophic lateral sclerosis patients), and the EEG signals collected by the subjects in different physical conditions or fatigue states can also be selected as the training sample data, and the embodiment is not limited.

[0107] The training data set includes positive samples and negative samples, the positive samples are the EEG signals of the same subject in the same fatigue state, and the negative samples are the EEG signals of the same subject in different fatigue states and / or the EEG signals of different subjects.

[0108] In the preferred embodiment of the present embodiment, after obtaining the EEG signal raw data streams of the same subject in different fatigue states and / or different subjects, the EEG signal raw data streams are subjected to pretreatment operations such as filtering, noise reduction, and removal of eye movement; then the preprocessed EEG signal data is used to construct a training data set containing positive samples and negative samples, the positive samples are EEG signal segments of the same subject in similar fatigue states, and the negative samples are EEG signal segments of different fatigue states or different subjects (such as signals in the state of wakefulness and fatigue).

[0109] S22, construct the network structure of the feature extraction network.

[0110] In the present embodiment, the network structure of the feature extraction network includes spatial convolution layers, time convolution layers and processing modules connected in sequence, the spatial convolution layers are used to extract the spatial features of the EEG signals, the time convolution layers are used to extract the time features of the EEG signals, the processing modules are used to fuse the time features and the spatial features, and the space-time fused features are subjected to average pooling processing and activation function processing.

[0111] S23, iteratively train the feature extraction network using the training data set until the training termination condition is reached.

[0112] Based on the training data set, a contrast learning technology is used to build a space-time convolutional neural network architecture for contrast learning training, so that the feature extraction network learns the fatigue-related neural activity patterns and state difference features; after the training is completed, the to-be-tested EEG signals are input into the pre-trained feature extraction network to extract features, and the differential entropy and multilayer perception classification are combined to realize accurate recognition of the cross-subject brain fatigue state.

[0113] Specifically, in each training, the feature extraction network extracts the spatio-temporal features of the positive samples and the spatio-temporal features of the negative samples respectively, determines the normalized temperature scaling cross-entropy loss of the feature extraction network according to the spatio-temporal features of the positive samples and the spatio-temporal features of the negative samples, and performs self-supervised contrastive learning pre-training according to the normalized temperature scaling cross-entropy loss, to optimize the network parameters of the encoder in the feature extraction network, so that the feature extraction network learns to distinguish discriminative feature representations of different fatigue states. The model is trained by using an optimization algorithm, and different fatigue state features are distinguished in the feature space; finally, the to-be-detected electroencephalogram signal is extracted by the pre-trained model, combined with differential entropy, and classified by a multilayer perception machine, to realize accurate identification of cross-subject fatigue states.

[0114] Figure 5 The differential entropy feature visualization result extracted by the encoder trained by contrastive learning of the present application is shown. The brain fatigue detection method based on electroencephalogram signals in the embodiment effectively solves the generalization problem of cross-subject electroencephalogram fatigue state recognition by enhancing the feature discriminativeness through contrastive learning and combining the biological interpretability of differential entropy.

[0115] When collecting the scalp electroencephalogram signals of different subjects as the training data set by using the electroencephalogram acquisition device, the collected electroencephalogram signal raw data stream of the subject is preprocessed to obtain an electroencephalogram signal sequence, and then two different enhanced views are generated based on each electroencephalogram data segment in the electroencephalogram signal sequence. The means for generating the enhanced views include data enhancement on the original electroencephalogram signal, including time shift, noise addition and amplitude scaling, wherein the time shift range is ±300 sampling points, the noise level of Gaussian noise addition is 0.25 times the data standard deviation, and the amplitude scaling range is 0.3-1.7 times. The application probability of each enhancement operation is 0.8, and the enhancement intensity parameter is set to 2.5 to control the intensity of the enhancement effect.

[0116] Through contrastive learning training, the subject invariance feature is learned by maximizing the similarity of electroencephalogram representations of different subjects to the same emotional stimulus and minimizing the similarity of representations of different stimuli. Through contrastive learning, the representation learned by the model satisfies: the cross-subject representation distance of the same stimulus is minimized, and the distance of different stimuli is maximized.

[0117] In addition, unlike the traditional convolutional network for extracting features, our model architecture is composed of two key stages: a contrastive learning stage and a prediction stage.

[0118] In the contrastive learning stage: the encoder is used to convert the original EEG data into a cross-subject aligned spatio-temporal representation, and then the non-linear projector is used to map it to the latent space, so as to calculate the similarity between samples.

[0119] In this embodiment, two different augmented views are generated for each EEG data segment, with a time shift range of ±300 sampling points, a noise level of 0.25 times the data standard deviation for Gaussian noise addition, an amplitude scaling range of 0.3-1.7 times, an application probability of 0.8 for each augmentation operation, and an augmentation intensity parameter set to 2.5 to control the intensity of the augmentation effect. The hierarchical normalization strategy is used for feature standardization to ensure training stability.

[0120] For negative samples being EEG signals of different subjects, after extracting the spatio-temporal features of the negative samples, the hierarchical normalization processing is performed on the spatio-temporal feature distribution of the negative samples to obtain the standardized negative sample spatio-temporal features. According to the spatio-temporal features of the positive samples and the standardized negative sample spatio-temporal features, the normalized temperature scaling cross-entropy loss is used for self-supervised contrastive learning pre-training to optimize the parameters of the feature extraction network, so that the feature extraction network can automatically adjust the scale and distribution of the features when facing EEG signals of different individuals.

[0121] The calculation formula of hierarchical normalization processing is:

[0122]

[0123] wherein, is the original feature, is the mean of the subject's EEG signal sequence data, is the standard deviation of the subject's EEG signal sequence data, is a small constant used to prevent division by zero.

[0124] The hierarchical normalization strategy is used for feature standardization to ensure training stability. Through the above diversified data augmentation strategy and hierarchical normalization processing, the generalization ability and robustness of the model under cross-subject and cross-scene conditions are significantly improved, so that the method can adapt to the changes of EEG signals in different individuals and various practical application environments, effectively reducing the performance fluctuations caused by individual differences and environmental noise.

[0125] In addition, we innovatively use adaptive feature normalization for differential entropy features in the prediction stage, dynamically updating the mean and variance of the test data (based on the initial values of the training data and real-time weighting of the test data), without accessing new subject data to quickly adapt and improve the practical application feasibility. The specific formula is:

[0126]

[0127] wherein, denotes the new mean obtained after dynamic updating, denotes the initial mean of the training data, denotes the real-time mean of the test data, represents a weight coefficient; part of the electroencephalogram signal sequence of the training data set is taken as training data, and another part of the electroencephalogram signal sequence in the training data set is taken as test data. The training data is a data set used for training the model, and the model learns features, patterns and parameters by analyzing these data; the test data is a data set used to evaluate the generalization ability of the model.

[0128] In the preferred embodiment of the present embodiment, the reference Figure 4 is processed by a fully connected classifier. Specifically, the spatio-temporal fusion feature vector is input into a three-layer fully connected projection head, the first layer has a dimension of 9216 to 256, the second layer has a dimension of 256 to 256, and the third layer has a dimension of 256 to 128; the Adam optimizer and the cosine annealing scheduler are used to learn the discriminative feature representation that distinguishes different fatigue states by minimizing the normalized temperature scaling cross-entropy loss.

[0129] The fully connected classifier design includes: first, layer normalization processing is performed on the input differential entropy features to reduce the difference between subjects; then the features are mapped to a 256-dimensional space through an input projection layer; a two-layer fully connected network (256→128 dimensions) is used for feature extraction, and each layer uses linear transformation, batch normalization, ReLU activation function and Dropout regularization technology in turn; finally, the logits values of four fatigue states are output through a classification head containing a 64-dimensional intermediate layer, and a cross-entropy loss function is used for supervised learning.

[0130] The homologous stimulus pair (positive example) and the heterologous stimulus pair (negative example) are defined , and the loss function is constructed by temperature τ scaling similarity Softmax normalization. In the present embodiment, the loss function expression of the normalized temperature scaling cross-entropy loss is:

[0131] ;

[0132] wherein , and represent different subjects, and represent different stimuli, in electroencephalogram signal processing, the stimulus is a standardized external input that can induce the subject to produce a specific labelable psychological state (such as emotion, fatigue, etc.); when collecting the scalp electroencephalogram signal of the subject, the subject is shown the stimulus material, and the subject's brain neural activity is intervened through the stimulus material; in the present embodiment, the stimulus material can be pictures of several different scenes, or a piece of image material with changing scenes, and the scalp electroencephalogram signal will change when the subject watches the stimulus material.

[0133] ​ temperature parameter, cosine similarity function, representing a subject a feature vector of the electroencephalogram signal generated under the first a feature vector of the electroencephalogram signal generated under the first representing a subject a feature vector of the electroencephalogram signal generated under the first a feature vector of the electroencephalogram signal generated under the first representing a subject a feature vector of the electroencephalogram signal generated under the first a feature vector of the electroencephalogram signal generated under the first representing the number of stimuli received by the subject within a batch representing an indicator function, taking a value of 1 when and a value of 0 when .

[0134] In some examples of the embodiment, the temperature parameter τ is set to 0.07, the learning rate is 0.0005, the weight decay is 0.0001, the cosine annealing scheduler is used to decay the learning rate from the initial value to the minimum learning rate and the model is trained for 1000 rounds to learn discriminative feature representations for distinguishing different fatigue states.

[0135] For each pair of subjects, a positive sample pair (electroencephalogram signal segments of the same stimulus) and a negative sample pair (electroencephalogram signal segments of different stimuli) are constructed, the sample size increases in a square order of the number of subjects, and the robustness of the model is enhanced. Cross-subject representation alignment is achieved by minimizing the total loss. The application of normalized temperature scaling cross-entropy loss to electroencephalogram signal fatigue recognition improves the generalization ability of the model to cross-subject scenarios and effectively learns similar electroencephalogram signal data of different subjects.

[0136] Based on the same inventive concept, an embodiment of the present application provides a brain fatigue detection system based on electroencephalogram signals.

[0137] The brain fatigue detection system can be loaded in an electronic device. According to the functions implemented, the brain fatigue detection system comprises:

[0138] An acquisition module is configured to acquire an electroencephalogram signal sequence of a subject.

[0139] A feature extraction module is configured to input the electroencephalogram signal sequence into a feature extraction network, the feature extraction network extracts spatial features and temporal features of each electroencephalogram signal in the electroencephalogram signal sequence respectively, and fuses the spatial features and the temporal features to obtain spatiotemporal fusion features of the electroencephalogram signal.

[0140] A differential entropy feature calculation module is configured to calculate differential entropy features of the electroencephalogram signal according to the spatiotemporal fusion features.

[0141] An output module is configured to determine a brain fatigue state detection result of the subject's electroencephalogram according to the differential entropy feature.

[0142] The module can also be referred to as a unit, which refers to a series of computer program segments capable of being executed by an electronic device processor and capable of completing a fixed function, and stored in the memory of the electronic device.

[0143] The various changes and specific examples of the brain fatigue detection method based on the electroencephalogram signal provided by the above embodiments are also applicable to the brain fatigue detection system of the present embodiment. Through the foregoing detailed description of the brain fatigue detection method based on the electroencephalogram signal, those skilled in the art can clearly know the implementation method of the brain fatigue detection system in the present embodiment. For the sake of brevity of the specification, the implementation method of the brain fatigue detection system in the present embodiment will not be described in detail here.

[0144] The present application also discloses an electronic device, such as Figure 7 As shown in the figure, it is a structural schematic diagram of an electronic device provided by the brain fatigue detection method based on the electroencephalogram signal. The electronic device can include at least one processor 10, a memory 11 in communication with the at least one processor, a communication bus 12, and a communication interface 13, and can also include a computer program stored in the memory 11 and executable on the processor 10, such as a program for the brain fatigue detection method based on the electroencephalogram signal.

[0145] In some embodiments, the processor 10 can be composed of an integrated circuit, for example, it can be composed of a single packaged integrated circuit, or it can be composed of multiple packaged integrated circuits with the same function or different functions, including one or more central processing units (CPU), microprocessors, digital processing chips, graphics processors, and combinations of various control chips, etc. The processor 10 is the control core (Control Unit) of the electronic device, which connects all components of the electronic device through various interfaces and lines, and executes or runs the programs or modules stored in the memory 11 (such as the brain fatigue detection method, etc.), and calls the data stored in the memory 11, to execute various functions and process data of the electronic device.

[0146] The memory 11 includes at least one type of readable storage medium, including a flash memory, a mobile hard disk, a multimedia card, a card-type memory (e.g., an SD or DX memory, etc.), a magnetic memory, a disk, an optical disk, etc. The memory 11 can be an internal storage unit of the electronic device in some embodiments, such as a mobile hard disk of the electronic device. The memory 11 can also be an external storage device of the electronic device in other embodiments, such as a plug-in mobile hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device. Further, the memory 11 can include both the internal storage unit and the external storage device of the electronic device. The memory 11 can be used not only to store application software and various data installed in the electronic device, such as the code of the method program for detecting brain fatigue, but also to temporarily store data that has been output or will be output.

[0147] The communication bus 12 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. The bus is provided to enable connection and communication between the memory 11, the at least one processor 10, etc.

[0148] The communication interface 13 is used for communication between the above-mentioned electronic device and other devices, including a network interface and a user interface. Optionally, the network interface can include a wired interface and / or a wireless interface (such as a WI-FI interface, a Bluetooth interface, etc.), which is usually used to establish a communication connection between the electronic device and other electronic devices. The user interface can be a display, an input unit (such as a keyboard), and optionally, the user interface can also be a standard wired interface, a wireless interface. Optionally, in some embodiments, the display can be an LED display, a liquid crystal display, a touch liquid crystal display, and an OLED (Organic Light-Emitting Diode) touch, etc. Among them, the display can also be appropriately referred to as a display screen or a display unit, which is used to display information processed in the electronic device and to display a visualized user interface.

[0149] Figure 7 Only the electronic device with components is shown, and those skilled in the art can understand that, Figure 7The illustrated structure does not constitute a limitation on the electronic device, and can include fewer or more components than shown, or combine certain components, or arrange the components differently.

[0150] For example, although not shown, the electronic device can further include a power supply (such as a battery) to supply power to the various components. Preferably, the power supply can be logically connected to the at least one processor 10 through a power management device, so that the power management device implements functions such as charge management, discharge management, and power consumption management. The power supply can also include one or more direct current or alternating current power supplies, a recharging device, a power supply failure detection circuit, a power supply converter or inverter, a power supply status indicator, and the like. The electronic device can also include various sensors, a Bluetooth module, a Wi-Fi module, and the like, which are not described here.

[0151] It should be understood that the embodiments are for illustration only and are not limited in scope by the structure described.

[0152] Further, the modules / units integrated in the electronic device, if implemented in the form of software function units and sold or used as independent products, can be stored in a computer readable storage medium. The computer readable storage medium can be volatile or non-volatile.

[0153] The embodiment of the present application provides a computer readable storage medium, for example, any entity or device capable of carrying the computer program code, a recording medium, a U disk, a mobile hard disk, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM). The computer readable storage medium stores a computer program capable of being loaded by a processor and executing the brain fatigue detection method based on electroencephalogram signals of the above-mentioned embodiment.

[0154] In the description of the present specification, the description of the terms "one embodiment", "some embodiments", "example", "specific example", "one implementation", "one preferred implementation" or "some examples" and the like means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In the present specification, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0155] Although the embodiments of the present application have been shown and described, those of ordinary skill in the art can understand that various changes, modifications, replacements and variations can be made to the embodiments without departing from the principles and spirit of the present application, and the scope of the present application is defined by the claims and their equivalents.

Claims

1. A brain fatigue detection method based on electroencephalogram signals, characterized in that, The method comprises: obtaining an electroencephalogram sequence of a subject; inputting the electroencephalogram sequence into a feature extraction network, the feature extraction network extracting spatial features and temporal features of each electroencephalogram in the electroencephalogram sequence respectively, and fusing the spatial features and the temporal features to obtain spatio-temporal fusion features of the electroencephalogram; calculating differential entropy features of the electroencephalogram according to the spatio-temporal fusion features; determining a brain fatigue state detection result of the electroencephalogram of the subject according to the differential entropy features; The method further comprises: obtaining a training data set; the training data set includes positive samples and negative samples, the positive samples are electroencephalograms of the same subject in the same fatigue state, and the negative samples are electroencephalograms of the same subject in different fatigue states and / or electroencephalograms of different subjects; constructing a network structure of the feature extraction network; iteratively training the feature extraction network using the training data set until a training termination condition is reached; in each training, the feature extraction network extracts spatio-temporal features of the positive samples and spatio-temporal features of the negative samples respectively, determines a normalized temperature scaled cross-entropy loss according to the spatio-temporal features of the positive samples and the spatio-temporal features of the negative samples, and performs self-supervised contrastive learning pre-training according to the normalized temperature scaled cross-entropy loss to optimize network parameters of the feature extraction network, thereby obtaining a final feature extraction network; the loss function expression of the normalized temperature scaled cross-entropy loss is: ； wherein, , , denote different subjects, , denote different stimuli, is a temperature parameter, is a cosine similarity function, denotes a subject a feature vector of the electroencephalogram signal produced by the subject under the i-th stimulus, denotes a subject a feature vector of the electroencephalogram signal produced by the subject under the i-th stimulus, denotes a subject a feature vector of the electroencephalogram signal produced by the subject under the i-th stimulus, denotes the number of stimuli received by a subject within a batch; denotes an indicator function that takes the value 1 when and the value 0 when .

2. The brain fatigue detection method based on electroencephalogram signals according to claim 1, wherein, The method comprises: obtaining an electroencephalogram sequence of a subject; collecting raw electroencephalogram data streams of the subject; sequentially performing band-pass filtering, power trap filtering and independent component analysis processing on the raw electroencephalogram data streams to obtain preprocessed data streams; 3. The brain fatigue detection method based on electroencephalogram signals according to claim 1, wherein, segmenting the preprocessed data streams into a plurality of electroencephalogram data segments using a time window to obtain the electroencephalogram sequence. For the negative samples being electroencephalograms of different subjects, after extracting the spatio-temporal features of the negative samples, hierarchical normalization processing is performed on the spatio-temporal feature distribution of the negative samples to obtain standardized negative sample spatio-temporal features; 4. The brain fatigue detection method based on electroencephalogram signals according to claim 3, wherein, performing self-supervised contrastive learning pre-training on the spatio-temporal features of the positive samples and the standardized negative sample spatio-temporal features through the normalized temperature scaled cross-entropy loss to optimize the parameters of the feature extraction network. ； wherein, is the original feature, is the mean of the subject's electroencephalogram sequence data, is the standard deviation of the subject's electroencephalogram sequence data, is a small constant used to prevent division by zero.

5. The brain fatigue detection method based on electroencephalogram signals according to claim 1 or 2 or 3 or 4, characterized in that, The calculation formula of the hierarchical normalization processing is:

6. A brain fatigue detection system based on electroencephalogram signals, for implementing the brain fatigue detection method based on electroencephalogram signals according to any one of claims 1 to 5, characterized in that, The brain fatigue state detection result includes at least two fatigue state levels and a probability distribution corresponding to each fatigue state level. comprises: an acquisition module configured to obtain an electroencephalogram sequence of a subject; a feature extraction module configured to input the electroencephalogram sequence into a feature extraction network, the feature extraction network extracting spatial features and temporal features of each electroencephalogram in the electroencephalogram sequence respectively, and fusing the spatial features and the temporal features to obtain spatio-temporal fusion features of the electroencephalogram; a differential entropy feature calculation module configured to calculate differential entropy features of the electroencephalogram according to the spatio-temporal fusion features; 7. An electronic device, comprising: an output module configured to determine a brain fatigue state detection result of the electroencephalogram of the subject according to the differential entropy features. The electronic device comprises: at least one processor (10); and a memory (11) connected in communication with the at least one processor (10); The memory (11) stores a computer program executable by the at least one processor (10), and the computer program is executed by the at least one processor (10) to enable the at least one processor (10) to perform the brain fatigue detection method based on electroencephalogram signals according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the brain fatigue detection method based on electroencephalogram signals according to any one of claims 1 to 5.

Images