A fatigue state objectivization recognition method based on electroencephalogram classifier assisted labeling
By synchronously acquiring multi-dimensional physiological signals and using a closed-loop optimization paradigm, and by employing an EEG classifier to assist in labeling, the problems of cross-modal data labeling and device comfort were solved, achieving high-precision automated recognition of fatigue states.
Patent Information
- Application Number
- CN202610465109.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-04-09
- Publication Date
- 2026-07-10
AI Technical Summary
In existing technologies, subjective fatigue assessment methods lack objectivity and consistency, making it difficult to reflect the dynamic changes in fatigue state. Cross-modal EEG data annotation suffers from noise amplification, and the comfort of EEG devices is poor, limiting their application in real-world scenarios.
By employing multi-dimensional synchronous acquisition of physiological signals, combined with non-smooth non-negative matrix factorization, improved Unet pupil detection, and decision-level multimodal fusion technology, a closed-loop optimization paradigm is constructed. Through EEG classifier-assisted annotation, high-quality annotation and automated recognition of cross-modal data are achieved.
It significantly improves the accuracy of cross-modal data annotation and pupil detection precision, achieves high-precision fatigue state recognition, avoids the inconvenience of wearing the device, and builds an efficient fatigue monitoring system.
Smart Images
Figure CN122350731A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of human-computer interaction and biosignal processing technology, specifically to a method for objectively identifying fatigue states based on EEG classifier-assisted annotation, which is particularly suitable for monitoring and identifying operator fatigue states in safety-critical fields such as aviation, rail transportation, and long-distance driving. Background Technology
[0002] In safety-critical sectors such as aviation, rail transportation, and long-distance driving, operator fatigue is a major contributing factor to human error and major accidents. Data from the International Civil Aviation Organization (ICAO) shows that 15%–20% of human-caused aviation accidents worldwide are due to operator fatigue, and this figure reaches as high as 60% in the rail sector. Accurately and objectively identifying fatigue levels is crucial for ensuring public safety.
[0003] Currently, the assessment of operator fatigue primarily relies on subjective fatigue scales. Among them, the Fatigue Scale 14 (FS14), developed in 1992 by Trudie Chalder et al. at the Psychological Medicine Research Laboratory of King's College Hospital in the UK, is one of the most widely used self-report fatigue scales internationally. This scale consists of 14 items. Subjects answer "yes" or "no" based on their own experience. Items 10, 13, and 14 are negatively scored, while the rest are positively scored. Physical fatigue scores (items 1-8) range from 0 to 8, mental fatigue scores (items 9-14) range from 0 to 6, and the total score ranges from 0 to 14. Higher scores indicate more severe fatigue.
[0004] Fatigue Scale-14 (FS14) Items and Scoring Rules However, these self-report-based fatigue assessment methods have significant limitations: First, the assessment results are highly dependent on individual subjective feelings, easily influenced by emotions, cognitive biases, and cultural background, lacking objectivity and consistency; second, the scales are usually used for a single assessment, making it difficult to reflect the dynamic and continuous changes in fatigue status; third, significant individual differences make it difficult to capture subtle changes in fatigue. These problems make subjective scales unreliable as a basis for high-precision fatigue monitoring systems.
[0005] On the other hand, objective fatigue identification methods based on physiological signals, especially electroencephalography (EEG), are considered the "gold standard" for fatigue identification because they directly reflect brain neural activity. However, these methods face a key bottleneck in practical applications: building a high-precision EEG fatigue identification model requires a large amount of high-quality datasets with accurate fatigue state labels. Traditional data annotation methods either rely on subjective scales (which have poor reliability) or require significant manpower and time costs for expert interpretation, making them difficult to scale. Furthermore, EEG acquisition devices typically require wearing electrode caps, which are uncomfortable and limit their widespread adoption in real-world scenarios.
[0006] Existing technologies include semi-supervised learning methods that use classifiers to pseudo-label unlabeled data. However, these methods are typically applied to single-modal data (e.g., EEG-to-EEG) and do not consider the heterogeneity issues present in cross-modal labeling. When the output of an EEG classifier is used as a label for different modalities such as EMG and eye movement, direct transfer leads to amplified label noise due to fundamental differences in acquisition principles and signal characteristics, severely impacting the reliability of subsequent model training. Therefore, developing an objective labeling method that can suppress noise propagation and achieve closed-loop optimization for cross-modal scenarios is a pressing technical challenge in this field. Summary of the Invention
[0007] The purpose of this invention is to address the problems existing in the prior art by providing an objective identification method for fatigue state based on EEG classifier-assisted annotation. This method constructs a closed-loop research paradigm of "scale-guided experiment → data model construction → model data refinement" and introduces non-smooth non-negative matrix factorization channel weighting, improved Unet pupil detection, and decision-level multimodal fusion technology. This solves the bottleneck problem of obtaining high-quality cross-modal labels in fatigue research and realizes the transformation from subjective assessment to objective and automated identification.
[0008] The objective of this invention can be achieved through the following technical measures: A method for objectively identifying fatigue states based on EEG classifier-assisted annotation, characterized by the following steps: Step 1: Synchronous acquisition of multi-dimensional physiological signals Design and execute a high-cognitive-load simulation manipulation experiment, and simultaneously collect multi-dimensional physiological signals of the subjects during the task execution process. The multi-dimensional physiological signals include at least electroencephalogram (EEG) signals, electromyogram (EMG) signals, and eye-tracking video signals. During the experiment, the subjects also fill out a fatigue scale 14 to record their subjective fatigue perception.
[0009] Step 2: EEG channel weighting and feature extraction based on non-smooth non-negative matrix factorization The acquired EEG signals were preprocessed to extract four EEG rhythm signals: delta waves (0.5-4Hz), theta waves (4-8Hz), alpha waves (8-13Hz), and beta waves (13-30Hz). A non-smooth, non-negative matrix factorization algorithm was used to calculate the contribution weight of each electrode channel, specifically including: The EEG data of each subject were represented as a matrix. V Its dimensions are c×n Where c is the number of channels and n is the number of sample points; Calculate the covariance matrix of each channel. Rcov This is used as the input matrix for the nsNMF algorithm; Will Rcov Decomposed into basis matrices W Encoding matrix H Non-smooth constraint matrix S The product of: R cov ≈W.S.H ; For the encoding matrix H Normalization is performed to obtain the weight values of each electrode channel; The obtained weight values are multiplied by the corresponding EEG channels to generate weighted EEG features; Frequency domain features, including the relative powers of delta waves, theta waves, alpha waves, and beta waves, are extracted from the weighted EEG signals, and combined features are constructed as follows: (θ + α) / β , θ / β wait.
[0010] Step 3: Build and train the EEG fatigue classifier Based on the weighted EEG features extracted in Step 2, an initial EEG fatigue classifier was constructed using a lightweight MobileNetV2 convolutional neural network. The MobileNetV2 network adopted an inverted residual structure, with the bottleneck layer using the ReLU6 activation function. The network terminal used the Softmax function to output the probability distribution of fatigue and wakefulness. The classifier was initially trained using some labeled EEG data.
[0011] Step 4: Preprocessing of EMG and Eye-tracking Data for Cross-Modal Annotation The acquired electromyographic signals were preprocessed, and the integrated electromyographic values and average power frequency were extracted as time-frequency domain features. For the acquired eye-tracking video signals, an improved Unet pupil detection algorithm is used to extract eye-tracking features. The improved Unet pupil detection algorithm includes: • A convolutional attention mechanism is introduced into the bottleneck layer of the Unet network. The feature maps output by the encoder and decoder are concatenated and then weighted by the channel attention module and the spatial attention module, respectively. • Add an ellipse fitting error loss term to the loss function to constrain the pupil segmentation result to conform to the elliptical geometry. The total loss function is: L = L BCE + λ.L EFE , where λ is a preset weighting coefficient; • Based on the extracted pupil position information, the peak velocity of saccades and the curvature of the saccade trajectory are calculated as eye movement features.
[0012] Step 5: Objective cross-modal data annotation based on EEG classifier The original signal segments corresponding to the eye movement features extracted in Step 4 are input into the initial EEG fatigue classifier constructed in Step 3. The classifier automatically generates the fatigue state probability distribution for each data sample and assigns an objective label of "fatigue" or "awake" accordingly. For ambiguous samples whose EEG classifier output probability is within a preset threshold range (such as 0.4 to 0.6), they are excluded from the labeled dataset to ensure the labeling quality. This forms an objective multimodal fatigue dataset.
[0013] Step 6: Sub-classifier training and decision-level fusion based on closed-loop optimization Using the objective multimodal fatigue dataset generated in Step 5, an eye-tracking sub-classifier was trained; the eye-tracking sub-classifier used the K-nearest neighbor algorithm (K=2). The DS evidence theory is used to fuse the outputs of the EEG classifier and the eye-tracking sub-classifier at the decision level to construct a comprehensive fatigue recognition model. The DS evidence theory fusion process includes: • Build a recognition framework Θ = {A1, A2} , representing fatigue and wakefulness respectively; • Each subclassifier generates a basic probability assignment through a Softmax layer; • Calculate the conflict coefficient K to determine the degree of conflict between sources of evidence; ·like K If the value is less than the preset conflict threshold, the probability allocation after fusion is calculated according to the Dempster combination rule; if K is greater than or equal to the preset conflict threshold, the weighted average strategy is used instead of the Dempster combination rule to avoid the paradox caused by the fusion of high-conflict evidence. • Select the state with the higher probability as the final classification result; The fused recognition results are used as feedback to optimize the network parameters of the EEG classifier in Step 3, forming a closed-loop optimization paradigm of "scale-guided experiment → data-based model construction → model-refined data".
[0014] Step 7: Fatigue State Identification and Application The eye-tracking video signal of the individual to be identified is input into the comprehensive fatigue recognition model constructed in Step 6, and the model outputs the fatigue state recognition result of the individual.
[0015] The method of this invention is characterized by: the parameters of the non-smooth non-negative matrix factorization algorithm in Step 2. θ ∈[0,1], by adjusting the value of θ, the smoothness of the constraint matrix S is controlled, thereby affecting the sparsity of the basis matrix W and the encoding matrix H.
[0016] The method of the present invention is characterized in that: the preset threshold range in Step 5 can be dynamically adjusted according to the classification confidence of the EEG classifier, and is initially set to 0.4 to 0.6.
[0017] The method of the present invention is characterized in that: the eye-tracking sub-classifier in Step 6 is the K-nearest neighbor algorithm, where K=2.
[0018] The method of the present invention is characterized in that: the multidimensional physiological signal also includes electromyography (EMG) signal; Step 4 further includes preprocessing the acquired EMG signal to extract the integral EMG value and average power frequency as EMG features; Step 6 further includes training the EMG classifier using an objective multimodal fatigue dataset and fusing the outputs of the EEG classifier, EMG classifier and eye-tracking subclassifier at the decision layer.
[0019] The beneficial effects of this invention are as follows: 1. Overcoming the Bottleneck of Cross-Modal Data Labeling: This invention addresses the difficulty of obtaining high-quality labels for electromyography (EMG) and eye movement (EMG) data in multimodal fatigue monitoring. It innovatively utilizes an EEG classifier as the labeler and introduces a fuzzy sample exclusion mechanism, significantly reducing label noise and providing an objective and automated solution for cross-modal data labeling. Compared to existing semi-supervised labeling methods, this invention significantly improves labeling quality; experimental data shows that the labeling accuracy can reach over 85%.
[0020] 2. Improved Eye Tracking Data Detection Accuracy: This invention addresses the technical bottlenecks of high noise interference and insufficient pupil detection accuracy in eye tracking monitoring by innovatively improving the Unet model. By introducing a convolutional attention mechanism and ellipse fitting error constraints, the pupil localization accuracy in complex human-computer interaction scenarios is significantly improved, with the pupil detection IoU index improved by approximately 12%.
[0021] 3. Achieving high-quality fusion of multimodal data: This invention uses DS evidence theory to fuse the outputs of three sub-classifiers—EEG, EMG, and eye movement—at the decision level, and introduces a conflict evidence handling mechanism, which effectively solves the information conflict problem in heterogeneous data fusion. The constructed comprehensive fatigue recognition model achieves a classification accuracy of over 90%, and avoids the inconvenience of wearing EEG devices.
[0022] 4. Forming a closed-loop optimization paradigm: This invention constructs a closed-loop research paradigm of "scale-guided experiment → data-based model construction → model-refined data". By comprehensively identifying feedback from the model to optimize the EEG classifier, the continuous iterative upgrade of the model is realized, providing technical support for the engineering application of fatigue monitoring systems. Attached Figure Description
[0023] Figure 1 This is an overall flowchart of the fatigue state objectification identification method based on EEG classifier-assisted annotation described in this invention.
[0024] Figure 2 This is a schematic diagram of the brainwave channel weighting method based on nsNMF in this invention.
[0025] Figure 3 This is a network structure diagram of the improved Unet pupil detection algorithm in this invention.
[0026] Figure 4 This is a schematic diagram of the decision-level fusion process of DS evidence theory in this invention. Detailed Implementation
[0027] The present invention will be further described below with reference to specific embodiments and accompanying drawings.
[0028] 1. Synchronous acquisition of multi-dimensional physiological signals To construct a high-quality dataset, this invention designs a simulated flight control experiment. Several healthy volunteers aged 20 to 30 years are recruited as subjects. The experiment is conducted at a fixed time (e.g., 11:30-14:00) to control for the influence of circadian rhythms on physiological signals. Before the experiment, subjects must complete an informed consent form and pass a medical questionnaire screening to ensure they have no history of neurological diseases and have not ingested caffeine or other neuroactive substances within 72 hours.
[0029] During the experiment, multi-dimensional physiological signals of the subjects were collected simultaneously: • EEG signal: An EEG signal acquisition device with a sampling rate of 256Hz is used to capture bioelectric signals generated by the activity of neurons in the cerebral cortex. The electrodes are placed in accordance with the international 10-20 system standard.
[0030] • Electromyographic signals: An electromyographic signal acquisition device with a sampling rate of 1024Hz was used to capture bioelectrical signals during skeletal muscle contraction.
[0031] • Eye-tracking video signal: A Tobii eye-tracking device with a sampling rate of 90Hz was used, along with a head-mounted near-infrared camera with a sampling rate of 30Hz for pupil localization algorithm research.
[0032] Subjects completed the Fatigue Scale-14 before, during, and after the experiment to record their subjective perception of fatigue.
[0033] 2. EEG Channel Weighting and Feature Extraction Based on nsNMF like Figure 2 As shown, the acquired EEG signals were preprocessed, including bandpass filtering (0.5-50Hz) to remove baseline drift and high-frequency noise, and independent component analysis to remove electrooculogram artifacts.
[0034] Four EEG rhythm signals were extracted: delta wave (0.5-4Hz), theta wave (4-8Hz), alpha wave (8-13Hz), and beta wave (13-30Hz).
[0035] The contribution weights of each electrode channel are calculated using a non-smooth, non-negative matrix factorization algorithm: • Represent the EEG data of each subject as a matrix V, with dimensions of . c×n , where c is the number of channels (e.g., 32 channels) and n is the number of samples; • Calculate the covariance matrix of each channel. Rcov This is used as the input matrix for the nsNMF algorithm; • Set the decomposition order r=2 ,Will Rcov Decomposed into R cov ≈W.S.H ; • Normalize the encoding matrix H: weight = (H − μH) / δH To obtain the weight values of each electrode channel; The obtained weight values are multiplied by the corresponding EEG channels to generate weighted EEG features.
[0036] Frequency domain features were extracted from the weighted EEG signals. The relative powers of delta, theta, alpha, and beta waves were calculated using Fast Fourier Transform, and combined features were constructed as follows: (θ + α) / β , θ / α A total of 64-dimensional EEG features were obtained.
[0037] 3. Construct and train an EEG fatigue classifier. Based on the extracted weighted EEG features, a lightweight MobileNetV2 convolutional neural network was constructed as the initial EEG fatigue classifier. This network is formed by stacking multiple bottleneck modules, each containing a 1×1 convolution, a 3×3 depthwise separable convolution, and a ReLU6 activation function. Finally, after global average pooling and fully connected layers, the network outputs the probability distribution of fatigue and wakefulness using the Softmax function.
[0038] The classifier was initially trained using partially labeled EEG data collected in the experiment (corresponding to the time periods when the fatigue scale score exceeded the threshold). The training set and the test set were divided in an 8:2 ratio. The cross-entropy loss function and the Adam optimizer were used, and the learning rate was set to 0.001.
[0039] 4. Preprocessing of EMG and Eye-tracking Data for Cross-Modal Annotation The acquired electromyographic (EMG) signals underwent preprocessing, including bandpass filtering (10-500Hz), 50Hz power frequency notch filtering, and full-wave rectification. A sliding window technique was used to extract features, with a window length of 1 second and an overlap rate of 50%. The integrated EMG values and average power frequency were extracted as time-frequency domain features.
[0040] For the acquired eye-tracking video signals, an improved Unet pupil detection algorithm was used to extract eye-tracking features. For example... Figure 3 As shown, the improved Unet includes: • Introduce a convolutional attention mechanism at the skip connections between the encoder and decoder to weight the feature maps by channel and spatial dimensions; • After concatenating the feature maps of the encoder output and decoder input at the bottleneck layer, they are passed through the channel attention module and the spatial attention module, respectively. The loss function is a weighted sum of binary cross-entropy and elliptic fitting error: L = L BCE + λ ,in λ = 0.1; • Based on the extracted pupil position, the peak velocity of saccades and the curvature of the saccade trajectory are calculated as eye movement features.
[0041] 5. Objective cross-modal data annotation based on EEG classifier The extracted eye-tracking features and their corresponding raw signal segments are input into a trained EEG fatigue classifier. The classifier outputs the fatigue probability for each data sample. P fatigue Set the threshold for excluding fuzzy samples: when 0.4 < P fatigue When the value is less than 0.6, the sample is classified as a fuzzy sample and excluded from the labeled dataset; when... P fatigue ≥ At 0.6, it is labeled "fatigue"; when P fatigue ≤ At a value of 0.4, the "awake" label was assigned. This resulted in an objective multimodal fatigue dataset containing approximately 50,000 valid samples.
[0042] 6. Subclassifier training and decision-level fusion based on closed-loop optimization An eye-tracking sub-classifier was trained using the generated objective dataset, employing the K-nearest neighbor algorithm (K=2) based on Euclidean distance. Figure 4 As shown, the DS evidence theory is used for fusion at the decision-making level: Building a recognition framework Θ = {A 1 (fatigue), A 2 (awake)} ; Each subclassifier generates a basic probability assignment using Softmax. m 1 、m 2 、m 3 ; · Calculate the conflict coefficient K, if K <0.9, fuse according to Dempster's combinatorial rules; if k≥ 0.9, using a weighted average strategy: m(A) = [m 1 (A) + m 2 (A) + m 3 （A）] / 3 ; • Select the state with the higher probability as the final classification result.
[0043] The fused recognition results are compared with the subjective scale labels to calculate the classification error. The network parameters of the EEG classifier are then optimized through backpropagation to form a closed-loop iteration.
[0044] 7. Fatigue State Identification and Application The eye-tracking video signal of the individual to be identified is input into a comprehensive fatigue recognition model, which outputs the individual's fatigue state. This system can be applied to scenarios such as cockpits and control rooms, and can obtain high-precision fatigue monitoring results by wearing comfortable electromyography and eye-tracking sensors.
Claims
1. A method for objectively identifying fatigue states based on EEG classifier-assisted annotation, characterized in that, Includes the following steps: Step 1: Design and execute a high-cognitive-load simulation manipulation experiment, and simultaneously collect multi-dimensional physiological signals of the subjects during the task execution process. The multi-dimensional physiological signals include at least electroencephalogram (EEG) signals, electromyogram (EMG) signals, and eye-tracking video signals. Step 2: Preprocess the acquired EEG signals to extract four EEG rhythm signals: delta wave, theta wave, alpha wave, and beta wave; use a non-smooth non-negative matrix factorization algorithm to calculate the contribution weight of each electrode channel, and multiply the obtained weight value by the corresponding EEG channel to generate weighted EEG features. Extracting frequency domain features from weighted EEG signals; Step 3: Based on the weighted EEG features extracted in Step 2, an initial EEG fatigue classifier is constructed using a MobileNetV2 lightweight convolutional neural network, and the classifier is initially trained using some labeled EEG data. Step 4: For the acquired eye-tracking video signals, an improved Unet pupil detection algorithm is used to extract eye-tracking features. The improved Unet pupil detection algorithm includes introducing a convolutional attention mechanism into the bottleneck layer of the Unet network and adding an ellipse fitting error loss term to the loss function; eye-tracking features are calculated based on the extracted pupil position information. Step 5: Input the original signal segments corresponding to the eye movement features extracted in Step 4 into the initial EEG fatigue classifier constructed in Step 3. The classifier will automatically generate the fatigue state probability distribution for each data sample and assign an objective label accordingly. For ambiguous samples whose output probability of the EEG classifier is within the preset threshold range, they will be excluded from the labeled dataset. Step 6: Using the objective multimodal fatigue dataset generated in Step 5, train the eye-tracking sub-classifier; use DS evidence theory to fuse the outputs of the EEG classifier and the eye-tracking sub-classifier at the decision level to construct a comprehensive fatigue recognition model; use the fused recognition results as feedback to optimize the network parameters of the EEG classifier in Step 3, forming a closed-loop optimization paradigm. Step 7: Input the eye-tracking video signal of the individual to be identified into the comprehensive fatigue recognition model constructed in Step 6. The model outputs the fatigue state recognition result of the individual.
2. The method according to claim 1, characterized in that, The specific steps for calculating the contribution weights of each electrode channel using the non-smooth, non-negative matrix factorization algorithm described in Step 2 include: The EEG data of each subject are represented as matrix V, with dimensions of... c×n Where c is the number of channels and n is the number of samples; Calculate the covariance matrix of each channel. R cov This is used as the input matrix for the nsNMF algorithm; Will R cov Decomposed into the product of the basis matrix W, the encoding matrix H, and the non-smooth constraint matrix S: R cov ≈WSH ; The encoding matrix H is normalized to obtain the weight values of each electrode channel.
3. The method according to claim 1, characterized in that, The loss function of the improved Unet pupil detection algorithm described in Step 4 is a weighted sum of binary cross-entropy and ellipse fitting error: L=L BCE +λ.L EFE ,That λ These are preset weighting coefficients.
4. The method according to claim 1, characterized in that, The preset threshold range mentioned in Step 5 can be dynamically adjusted according to the classification confidence of the EEG classifier, and is initially set to 0.4 to 0.
6.
5. The method according to claim 1, characterized in that, The fusion process of DS evidence theory described in Step 6 includes: when the conflict coefficient K is greater than or equal to the preset conflict threshold, the weighted average strategy is used to replace the Dempster combination rule in order to avoid the paradox caused by the fusion of high-conflict evidence.
6. The method according to claim 1, characterized in that, The eye-tracking sub-classifier described in Step 6 is the K-nearest neighbor algorithm, where K=2.
7. The method according to claim 1, characterized in that, The multidimensional physiological signals also include electromyography (EMG) signals. Step 4 further includes preprocessing the acquired EMG signals to extract the integrated EMG value and average power frequency as EMG features. Step 6 also includes training a myoelectronic classifier using an objective multimodal fatigue dataset and fusing the outputs of the EEG classifier, myoelectronic classifier, and eye-tracking sub-classifier at the decision layer.