Method for establishing personalized cognitive fatigue model

Abstract

The invention discloses a method for establishing a personalized cognitive fatigue model. According to the method, an N-back task is used for performing task testing on a subject, physiological feature data are collected at the same time, a k-means algorithm is used for clustering N-back task accuracy and reaction time, fatigue labeling of original data is obtained, and therefore labeling of the physiological feature data collected in the whole process is achieved. And then personalized model establishment is carried out on different individuals by using a deep learning classification model. According to the method, the N-back task performance is used as a cognitive fatigue mark, the personalized fatigue prediction model is established for different individuals for the first time, data of different batches are used as a training set and a test set for training and testing of the model, and the fatigue prediction effect superior to a subjective evaluation index is achieved.

Description

Technical Field

[0001] The present invention belongs to the technical field of cognitive fatigue, and particularly relates to a method for establishing a personalized cognitive fatigue model. Background Art

[0002] Cognitive fatigue can directly affect work efficiency and accuracy. When an individual is in a state of cognitive fatigue, their thinking becomes dull, reaction speed slows down, and they are prone to making mistakes. The most significant meaning of cognitive fatigue state assessment is to prevent serious accidents caused by the decline of job performance of staff in important positions. Therefore, the decline of job performance is the gold standard for cognitive fatigue. The N-back task is a classic cognitive test task, commonly used to study and train cognitive abilities such as working memory and attention, which are also important abilities required for positions such as large equipment operators, warehouse in-and-out managers, and drug distributors. Once there is a significant decrease in these abilities, it will bring great potential safety hazards. Different difficulty levels of the N-back task (i.e., different values of n) can provide different levels of tests for subjects with different abilities, thereby increasing its scope of application. Currently, the performance evaluation indicators of the N-back task include reaction time, accuracy rate, continuous memory quantity, etc. These indicators can only relatively evaluate the degree of cognitive fatigue and cannot perform data annotation. The annotation methods of cognitive states can be divided into three categories: based on subjective judgment, based on physiological and biochemical index measurement, and based on task performance. Based on subjective scales or individual subjective feelings cannot truly reflect the decline of job ability. On the one hand, after the feeling of fatigue occurs, operators will reallocate cognitive resources through subjective efforts and can often maintain sufficient job ability. Evaluating cognitive fatigue based on physiological and biochemical index measurement lacks authoritative cognitive fatigue markers. Based on task performance is considered the gold standard for cognitive state assessment, but achieving real-time assessment is also a huge challenge it faces. As mentioned above, subjects can overcome the decline of job efficiency caused by the fatigue state to a certain extent through subjective efforts, and the influencing factors are complex. Therefore, job efficiency does not decline linearly with the increase of job duration, but intermittently appears randomly in discrete time periods. Therefore, only by realizing real-time cognitive fatigue monitoring can a meaningful cognitive fatigue state - job performance decline be accurately annotated.

[0003] General models are usually trained on a large amount of diverse data, aiming to capture as many common features and patterns as possible to achieve good generalization ability for unknown data. This design philosophy enables general models to provide basic functions and services in a wide range of fields. However, when it comes to highly specialized or personalized application scenarios, the limitations of general models become apparent. Since they are not customized for specific tasks or users, they may not fully consider the individual differences in user behavior habits, resulting in poor performance. Summary of the Invention

[0004] The purpose of the present invention is to provide a method for establishing a personalized cognitive fatigue model. The N-back task is used to test the subjects, and the k-means algorithm is used to label the original data. The labeled data is used to perform personalized training and testing of the classification model for different people by using deep learning methods.

[0005] A method for establishing a personalized cognitive fatigue model is carried out according to the following steps:

[0006] 1) Recruit subjects to perform the N-back test task twice. Conduct a formal test on the subjects for 1-3 hours and record whether the answer to each question is correct and the answering time during the test. Calculate the accuracy rate of the answering results of the subjects, and use the clustering algorithm to cluster the subject data mixed with the accuracy rate and reaction time. Select the classification labels with shorter reaction duration and lower correct rate, and mark them as negative samples (0); Mark the labels with shorter reaction time and higher correct rate as positive samples (1);

[0007] 2) Perform data preprocessing on the collected data of different modalities, and label the preprocessed feature data with the labels of N-back clustering;

[0008] 3) Use the labeled feature data of the first N-back test task as the training set and the validation set, and use the classification algorithm to train the classification model;

[0009] 4) Use the trained model to test on the test set, perform smoothing processing on the prediction results, and screen out the optimal model.

[0010] For the step 2), the preprocessing methods of data of different modalities are different, and the time windows divided for the data of different modalities after processing are also different.

[0011] For the step 3), the classification algorithm is a deep learning model, which is carried out by calling the tsai toolbox.

[0012] For the step 3), the first batch of feature data is divided into the training set and the validation set according to the ratio of 9:1.

[0013] The specific parameter settings of the classification model for the step 3) are: tfms = [None, TSClassification()], batch_tfms = TSStandardize(by_sample = True), device = torch.device('cuda' if torch.cuda.is_available() else 'cpu'), path ='models', metrics = accuracy, epoch = 100, lr = 1e-2.

[0014] The specific parameters of the smoothing method in step 4) include window_size, thred_01, and thred_10; window_size is the size of the smoothing window, and thred_01 and thred_10 are the judgment thresholds after state conversion.

[0015] Advantages of the present invention: The present invention uses N-back task performance (accuracy and reaction time) as cognitive fatigue annotation, and uses deep learning method as a classification model, and for the first time proposes a method for establishing personalized fatigue models for different individuals, realizing real-time evaluation of personalized cognitive fatigue state based on physiological indicators. Description of the Drawings

[0016] Figure 1 Performance test of the real-time prediction model of the personalized cognitive load of the subjects for intra- and cross-batch data. Detailed Embodiments

[0017] To facilitate the understanding of the present invention, the present invention will be described more comprehensively below. However, the present invention can be implemented in many different forms and is not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present invention more thorough and comprehensive.

[0018] Embodiment 1 Preprocessing of Eye Movement Data

[0019] Fixation Feature Extraction

[0020] I Remove the values where the fixation positions of the left and right eyes are -1, that is, the data where Validity Left and Validity Right are -1;

[0021] II Remove the values where the pupil diameter is not between 10 - 35 px;

[0022] III Remove the values where the absolute value of the difference in the X coordinate of the fixation positions of the left and right pupils is greater than or equal to 120, and the absolute value of the difference in the Y coordinate of the fixation positions of the left and right pupils is greater than or equal to 100;

[0023] IV Remove the values where the absolute value of the difference in the pupil diameters of the left and right eyes is greater than or equal to 3; if the left eye is empty, fill it with the data of the right eye; the same applies to the right eye; if neither is available, fill it with the average of the non-empty values above and below; perform one up-filling and down-filling on the entire table to fill the non-empty values without upper or lower values.

[0024] Blink Feature Extraction

[0025] I Calculate the difference between the maximum and minimum values of the Blink Index within a 30s time window as the blink frequency;

[0026] II Calculate the proportion of non-empty lines of Blink Index within a 30s window, and multiply it by 3000 to obtain the blink duration in ms;

[0027] Saccade feature extraction

[0028] I Take the maximum value of "Saccade Single Velocity [px / ms]" within 1s as the saccade speed;

[0029] II Take the maximum value of "Saccade Duration [ms]" as the saccade duration;

[0030] Eye movement feature extraction

[0031] Average the fixation features, saccade features, and blink features by second, and concatenate the three features together along the time axis. Each row after concatenation constitutes the eye movement features per second.

[0032] Eye movement data and task performance mapping rules

[0033] Use the eye movement features within the time interval (5.5s) between every two subtasks in the N-back task as the unit input, and map them to the task performance at the time of the subsequent subtask. The task performance of each subtask is defined as the reaction time for executing the task, as well as the execution accuracy of this subtask and the previous nine subtasks. Establish the mapping relationship between eye movement data and task performance through the above principles, so as to classify and annotate the eye movement data within each task interval according to task performance, and achieve the annotation of the cognitive load level of eye movement data. Since the time interval between N-back subtasks is 5.5s, there will be cases where sometimes 5 rows and sometimes 6 rows are intercepted. Uniformly limit all features to 5 rows to ensure consistent feature dimensions.

[0034] Example 2: Facial expression data preprocessing

[0035] Remove irrelevant feature columns such as timestamps, faceID, and screen brightness. Take the average of the remaining features by second as the features corresponding to each second.

[0036] Facial expression data and task performance mapping rules.

[0037] Using the facial expression features within the time interval (5.5 s) between every two sub-tasks in the N-back task as the unit input, map to the task performance at the time corresponding to the subsequent sub-task. The task performance of each sub-task is defined as the reaction time for executing the task, as well as the execution accuracy of this sub-task and the nine sub-tasks before it. Establish the mapping relationship between the facial expression data and the task performance through the above principles, so as to classify and annotate the facial expression data within each task interval according to the task performance, and realize the annotation of the cognitive load level of the facial expression data. Since the time interval between N-back sub-tasks is 5.5 s, there will be cases where sometimes 5 rows are intercepted and sometimes 6 rows are intercepted. All features are uniformly restricted to 5 rows to ensure consistent feature dimensions.

[0038] Example 3 Electrocardiogram Data Preprocessing

[0039] First, use a band-pass filter to remove noise and unnecessary frequency components, and retain the useful ECG signal. The low-frequency cut-off frequency of the filter is set to 0.5 Hz, the high-frequency cut-off frequency is set to 50 Hz, the sampling frequency is set to 1000 Hz, and the filter order is set to 4.

[0040] Then, segment the filtered ECG data into multiple fixed-length segments, and the length of each segment is set to 5000 sampling points. In this way, the data can be processed segment by segment to extract time-domain and frequency-domain features.

[0041] The extraction of time-domain features is to calculate the mean, standard deviation, maximum value, minimum value, and root mean square value of each segment; the extraction of frequency-domain features is to calculate the total power, main frequency, average frequency, and bandwidth of each segment. The main frequency refers to the frequency corresponding to the maximum value of the power spectral density (PSD), the average frequency is the weighted average frequency, and the bandwidth is the frequency range containing 5% to 95% of the total energy.

[0042] Each row of the preprocessed features represents the features extracted from a 5000-sampling-point time window, that is, a 5-s time window.

[0043] Mapping rule between electrocardiogram data and task performance: Using the electrocardiogram features of a time window with a window length of 50 and a step size of 1 (that is, 250 s of feature data, moving 5 s each time) as the input, map to the task performance corresponding to the feature at the last time point within the time window. The task performance at the last time point is defined as the reaction time of the first N-back task after this time point, as well as the execution accuracy of this sub-task and the nine sub-tasks before it. Establish the mapping relationship between the electrocardiogram data and the task performance through the above principles, so as to classify and annotate the preprocessed electrocardiogram data according to the task performance, and realize the annotation of the cognitive load level of the electrocardiogram data.

[0044] Example 4 Establishment of a General Cognitive Load Prediction Model

[0045] Using the FCNPlus, RNNPlus, XceptionTimePlus, RNN_FCNPlus, TSTPlus, XResNet1dPlus, TSiT, RNNAttention, RNNAttentionPlus, TransformerRNNPlus, HydraPlus, HydraMultiRocketPlus, ConvTranPlus models and the LSTM model in the tsai toolkit of python, the average performance of the general models based on different algorithms was calculated using the leave-one-out cross-validation method, and the running results are shown in Table 1.

[0046] Table 1 Performance of the General Model for Real-Time Prediction of Cognitive Load Based on Different Algorithms

[0047]

[0048]

[0049] The above results show that the sensitivity of the general cognitive load prediction model is very low. It is speculated that the large differences in the characteristic behavior data between people during the operation process may be the main factor causing the poor sensitivity of the model.

[0050] Example 5 Establishment of a Personalized Cognitive Load Prediction Model

[0051] Using the FCNPlus, RNNPlus, XceptionTimePlus, RNN_FCNPlus, TSTPlus, XResNet1dPlus, TSiT, RNNAttention, RNNAttentionPlus, TransformerRNNPlus, HydraPlus, HydraMultiRocketPlus, ConvTranPlus, LSTM models in the tsai toolkit of python, the personalized model of the subject was trained using the data in the first half of the time in a certain batch of the subject's dataset, and the performance of the model was verified using the data in the second half of the time. The optimal performance of the model for each person based on different algorithms was calculated, and the running results are shown in Table 2.

[0052] Table 2 Prediction Performance Test of the Subject's Personalized Cognitive Load Real-Time Prediction Model for Data in the Same Batch

[0053]

[0054]

[0055] It can be seen from the table that the prediction performance of the model can reach a very high level under the same batch.

[0056] Using the FCNPlus, RNNPlus, XceptionTimePlus, RNN_FCNPlus, TSTPlus, XResNet1dPlus, TSiT, RNNAttention, RNNAttentionPlus, TransformerRNNPlus, HydraPlus, HydraMultiRocketPlus, ConvTranPlus, and LSTM models in the tsai toolkit of Python, with the data of the first batch used as the training and validation set and the data of the second batch used as the test set, the optimal performance of each person's model based on different algorithms was calculated, and the running results are shown in Table 3.

[0057] Table 3 Prediction performance test of the personalized cognitive load real-time prediction model for cross-batch data

[0058]

[0059]

[0060] The above results show that, in the case of avoiding the cross-person difference effect, although the cross-batch effect will reduce the model performance to a certain extent, the overall prediction performance is still greatly improved compared with the general model of all subjects, and it is comprehensively better than the index of the subjective mental workload perception of the subjects.

[0061] The present invention first proposes to establish a personalized cognitive fatigue model for different individuals. At present, the determination of cognitive fatigue mainly uses subjective threshold setting, such as artificially setting the duration of the cognitive fatigue induction task, or setting the threshold of fatigue annotation measurement indicators (such as subjective scale scores or objective physiological and biochemical index measurement values), etc. We use a mathematical standard to effectively identify the objective cognitive fatigue state, and through this marking method, mark the physiological characteristic data, and realize the establishment of a personalized cognitive fatigue model for different individuals.

[0062] Specifically, relying on the established high mental workload state induction and evaluation platform, the selected standardized high mental workload state induction technology for network attack and defense positions personnel is adopted to cluster the business performance (accuracy and reaction time) of the test data for subjects performing the N-back task for 1.5 hours, 3 hours, and a mixture of the two. The physiological feature data collected simultaneously is labeled using the labeled values obtained from the clustering. After obtaining the labeled data, the data of the first batch is used as the training and validation set, and the data of the second batch is used as the test set. The model in tsai is used for training and testing, and the optimal model performance of each person is calculated. From the prediction performance test tables of the general model and the personalized model, it can be seen that the effect of the personalized model is significantly better than that of the general model, indicating that individual differences do indeed affect the prediction performance of the general model to a great extent. The effect of the individualized model for the same batch is better than that of the cross-batch, indicating that there are also certain differences between cross-batches, but the cross-batch effect is still better than that of the general model.

[0063] The above-described embodiments merely represent several implementation manners of the present invention. The description is relatively specific and detailed, but it should not be construed as a limitation on the scope of the invention patent. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present invention, several modifications and improvements can still be made, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the present invention patent shall be subject to the appended claims.

Claims

1. A method for establishing a personalized cognitive fatigue model, characterized in that: Follow these steps: 1) Recruit subjects to perform two N-back test tasks, conduct a formal test of 1-3 hours on the subjects and record whether the answer to each question is correct and the time taken to answer the questions during the test. Calculate the accuracy of the subjects' answers, use a clustering algorithm to cluster the test data of the mixed accuracy and reaction time, select the classification labels with long reaction time and low accuracy, and mark them as negative samples (0); mark the labels with shorter reaction time and higher accuracy as positive samples (1); 2) Preprocess the collected data of different modalities and label the preprocessed feature data through N-back clustering labels; 3) Using the labeled feature data of the first N-back test task as the training set and validation set, the classification model is trained using the classification algorithm; 4) Use the trained model to test on the test set, smooth the prediction results, and select the optimal model.

2. The method for establishing a personalized cognitive fatigue model according to claim 1, characterized in that: In the step 2), different modal data are preprocessed in different ways, and the time windows for dividing the processed data into different modal data are also different.

3. The method for establishing a personalized cognitive fatigue model according to claim 1, characterized in that: The classification algorithm in step 3) is a deep learning model, which is performed by calling the tsai toolkit.

4. The method for establishing a personalized cognitive fatigue model according to claim 1, characterized in that: In step 3), the first batch of feature data is divided into a training set and a validation set in a ratio of 9:

1.

5. The method for establishing a personalized cognitive fatigue model according to claim 1, characterized in that: The specific parameters of the classification model in step 3) are set as follows: tfms = [None, TSClassification()], batch_tfms = TSStandardize(by_sample = True), device = torch.device('cuda' if torch.cuda.is_available() else 'cpu'), path = 'models', metrics = accuracy, epoch = 100, lr = 1e-2.

6. The method for establishing a personalized cognitive fatigue model according to claim 1, characterized in that: The specific parameters of the smoothing method in step 4) include window_size, thred_01, and thred_10; window_size is the smoothing window size, and thred_01 and thred_10 are the judgment thresholds after state conversion.

Images