Method for evaluating cognitive fatigue data annotation by using work memory normal form performance indicators

Abstract

The invention discloses a data labeling method for evaluating cognitive fatigue by using a work memory normal form performance indicator. According to the method, an N-back task is used for carrying out task testing on a subject, a k-means algorithm is used for clustering the accuracy rate and the reaction time of the subject, BIC, a contour coefficient, a Clinski-Harabasz index and a Davies-Bouldin index are evaluated, meanwhile, a normalization processing method is used for improving the accuracy and quality of data, a segmentation threshold value for marking original data is obtained, and the accuracy and the quality of the data are improved. Therefore, the test data acquired in the whole process can be labeled. According to the method, the N-back task performance is used as the cognitive fatigue label, the cognitive fatigue classification and the corresponding threshold are objectively judged through unsupervised learning on the task performance for the first time, and the cognitive fatigue classification and the corresponding threshold are used as the cognitive fatigue label standard to guide the corresponding physiological index data classification training, so that the cognitive fatigue state real-time evaluation based on the physiological indexes is realized.

Description

Technical Field

[0001] The present invention belongs to the technical field of cognitive fatigue, and specifically relates to a data labeling method for evaluating cognitive fatigue using working memory paradigm performance indicators. Background Art

[0002] Cognitive fatigue directly affects work efficiency and accuracy. When an individual is in a state of cognitive fatigue, his or her thinking becomes dull, the reaction speed slows down, and he or she is prone to making mistakes. The greatest significance of cognitive fatigue status assessment is to prevent serious accidents caused by the decline in work performance of key personnel. Therefore, the decline in work performance is the gold standard for cognitive fatigue. The N-back task is a classic cognitive test task, which is often used to study and train cognitive abilities such as working memory and attention. These abilities are also important abilities required for positions such as large equipment operation, warehouse management personnel, and drug distribution personnel. Once these abilities are significantly reduced, they will bring great safety hazards. The different difficulty levels of the N-back task (i.e., different values ​​of n) can provide different levels of testing for subjects with different abilities, thereby increasing its scope of application. At present, the performance evaluation indicators of the N-back task include reaction time, accuracy, number of continuous memories, etc. These indicators can only make a relative evaluation of the degree of cognitive fatigue, and cannot perform data annotation. The annotation methods of cognitive status can be divided into three categories: based on subjective judgment, based on physiological and biochemical index measurement, and based on task performance. Based on subjective scales or individual subjective feelings, it cannot truly reflect the decline in work ability. On the one hand, after the feeling of fatigue occurs, the operator will redeploy cognitive resources through subjective efforts, and can often maintain sufficient work capacity. There is a lack of authoritative cognitive fatigue markers for the evaluation of cognitive fatigue based on physiological and biochemical indicators. Task performance is considered to be the gold standard for cognitive state assessment, but achieving real-time evaluation is also a huge challenge. As mentioned earlier, the subjects can overcome the decline in work efficiency caused by fatigue to a certain extent through subjective efforts, and the influencing factors are complex. Therefore, work efficiency does not decrease linearly with the increase in work time, but appears intermittently and randomly in discrete time periods. Therefore, only by realizing real-time cognitive fatigue monitoring can we accurately mark the meaningful cognitive fatigue state-decline in work performance.

[0003] Data annotation plays a vital role in data processing, machine learning and artificial intelligence. The significance of accurate data annotation is mainly reflected in improving model accuracy, enhancing generalization ability, reducing model training time, and reducing computing resource consumption. In terms of annotation of cognitive fatigue, a large number of similar studies at home and abroad use independent cognitive fatigue judgment tasks before and after work to annotate cognitive fatigue status. This strategy can only evaluate cognitive fatigue status at two time points, the beginning and the end, and cannot achieve real-time evaluation throughout the process, making it difficult to obtain accurate and sufficient annotated data for training. Summary of the invention

[0004] The purpose of the present invention is to provide a data annotation method for evaluating cognitive fatigue using working memory paradigm performance indicators. The N-back task is used to test the subjects, and the k-means algorithm is used to cluster their accuracy and reaction time and evaluate BIC, silhouette coefficient, Calinski-Harabasz index and Davies-Bouldin index. At the same time, the z-score normalization method is used to improve the accuracy and quality of the data, and the segmentation threshold for annotating the original data is obtained, thereby realizing the annotation of the test data collected throughout the process.

[0005] A data annotation method for evaluating cognitive fatigue using working memory paradigm performance indicators is carried out in the following steps:

[0006] 1) Recruit subjects to perform N-back test tasks and conduct practice test phase, which lasts for 10-20 minutes. Set the N value according to the accuracy of the subjects to ensure that the accuracy of the answers is higher than 70%;

[0007] 2) Conduct a formal test of 1-3 hours on the subjects and record whether the answer to each question is correct and the time taken to answer the questions;

[0008] 3) Calculate the accuracy of the subjects’ answers in a 10-20 minute time window with a step length of 1-10 seconds, and normalize the calculated task performance;

[0009] 4) Use clustering algorithm to cluster the test data of both accuracy and reaction time;

[0010] 5) Statistically cluster the data into different categories. First, sort them in ascending order using reaction time as the first sorting criterion and accuracy as the second sorting criterion. Select the classification labels with long reaction time and low accuracy and mark them as negative samples (0); mark the labels with shorter reaction time and higher accuracy as positive samples (1); after completing data labeling, use it as a training set for subsequent deep learning.

[0011] The task performance in step 3) is accuracy and reaction time.

[0012] The step 3) uses a z-score normalization method to perform normalization processing, and the processed data obtained establish an initial sample matrix.

[0013] The clustering algorithm in step 4) is a k-means method, which is performed by calling the KMeans tool of the sklearn.cluster toolkit.

[0014] The parameters calculated by the k-means algorithm should meet the following conditions: BIC < -30000, silhouette coefficient > 0.45, Calinski-Harabasz index > 48000 and Davies-Bouldin index < 1.05, then the results are credible.

[0015] The specific parameters of step 3) are set as follows: n_clusters=2, random_state=42, init='random', batch_size=32.

[0016] Beneficial effects of the present invention: The present invention adopts N-back task performance (accuracy and reaction time) as cognitive fatigue annotation, and for the first time proposes to objectively determine cognitive fatigue classification and corresponding thresholds through unsupervised learning of task performance, and uses this as a cognitive fatigue annotation standard to guide the corresponding physiological indicator data classification training, thereby realizing real-time evaluation of cognitive fatigue status based on physiological indicators. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 Clustering results of k-means, FCM and GMM algorithms.

[0018] Figure 2 The cognitive fatigue thresholds obtained by optimal clustering of the test data for 1.5 hours, 3 hours, and a mixture of the two. DETAILED DESCRIPTION

[0019] In order to facilitate the understanding of the present invention, the present invention will be described more fully below. However, the present invention can be implemented in many different forms and is not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present invention more thorough and comprehensive.

[0020] Example 1 N-back task test data collection and accuracy and reaction time recording

[0021] Turn on the eye tracker, calibrate it, click "Record" to start recording eye movement data, and click "Stop Recording" after the data collection is completed;

[0022] Open the N-back task software, set the parameters (N=3), click start, and let the subjects perform the 10-min N-back task as required. Adjust the N value according to the accuracy rate (reduce the N value by 1 if the accuracy rate is low, and increase the N value by 1 if the accuracy rate is high) and restart the experiment until the accuracy rate is >70%, and then start the formal 90-min task.

[0023] The software automatically records the accuracy and reaction time, as shown in Table 1:

[0024] Table 1

[0025]

[0026] Example 2 Clustering Algorithm Selection

[0027] KMeans, FCM and GMM of sklearn.cluster toolkit of Python were used to cluster the reaction time and accuracy data in the result table of Example 1 respectively, and the BIC, silhouette coefficient, Calinski-Harabasz index and Davies-Bouldin index of various clustering methods were compared. The parameter settings are shown in Table 2:

[0028] Table 2

[0029]

[0030] The operation results are shown in Table 3-6:

[0031] Table 3 BIC values ​​of different clustering results of three unsupervised learning methods

[0032]

[0033] Table 4 Silhouette coefficients of different clustering results of three unsupervised learning methods

[0034]

[0035] Table 5 Calinski-Harabasz index of different clustering results of three unsupervised learning methods

[0036]

[0037]

[0038] Table 6 Davies-Bouldin index of different clustering results of three unsupervised learning methods

[0039]

[0040] According to the above results, k-means is selected as the optimal clustering method, and the clustering parameters are n_clusters=2, random_state=42, init='random', batch_size=32.

[0041] Example 3: Using the results of clustering algorithm to label data

[0042] The collected eye movement and ECG data are labeled using the labels obtained by clustering. The labeling method is as follows: The clustering results calculated in Example 2 are expressed in the following form:

[0043] Table 7

[0044]

[0045] According to the above results, reaction time is used as the first sorting basis and accuracy rate is used as the second sorting basis to sort in ascending order. The classification labels with long reaction time and low accuracy rate are selected and marked as negative samples (0); the labels with shorter reaction time and higher accuracy rate are marked as positive samples (1). The original data is labeled with the set classification labels (the cluster label 1 in the above table is marked as 1, and the cluster label 2 is marked as 0), and subsequent deep learning training is carried out.

[0046] The present invention proposes for the first time to objectively determine cognitive fatigue classification and corresponding thresholds through unsupervised learning of task performance. At present, the judgment of cognitive fatigue mainly adopts subjective threshold setting, such as artificially setting the duration of cognitive fatigue-inducing tasks, or setting thresholds of fatigue labeling measurement indicators (such as subjective scale scores or objective physiological and biochemical indicators). We innovatively propose to achieve effective identification of objective cognitive fatigue states based on mathematical standards. Specifically, the silhouette coefficient, Calinski-Harabasz index and Davies-Bouldin index objective indicators are used to select the optimal unsupervised learning (clustering) method, and the Bayesian information criterion BIC value is used to evaluate the optimal number of cognitive fatigue state classifications, and the objective threshold of fatigue state classification is obtained by clustering the optimal unsupervised learning method.

[0047] Specifically, relying on the established high mental workload state induction and evaluation platform, the selected standardized high mental workload state induction technology for network attack and defense personnel was used to cluster the business performance (accuracy and reaction time) of the test data of subjects who performed the N-back task for 1.5 hours, 3 hours, and a mixture of the two. In order to ensure the comparability of business performance between subjects, on the one hand, an appropriate N value was set according to the accuracy of the subjects, and on the other hand, the task performance (accuracy and reaction time) of the subjects was normalized. On this basis, three representative clustering methods, k-means, FCM, and GMM, were used to attempt the best clustering. The calculation results of BIC, silhouette coefficient, Calinski-Harabasz index, and Davies-Bouldin index showed that the k-means method had the best clustering performance when clustering into two categories. The scatter plot of the task performance clustering showed that only the two states clustered by this clustering method and FCM were highly consistent with the decrease in N-back task performance (decreased accuracy and increased reaction time). At the same time, the above-mentioned multiple mathematical indicators show that the k-means method has the best discrimination when clustered into two categories, suggesting that it represents two naturally different physiological states. Two potential naturally different physiological states were obtained through this clustering method, and they are highly consistent with the reduction in task performance, suggesting that this clustering can accurately label the cognitive fatigue state. At the same time, the test data of 1.5 hours, 3 hours and a mixture of the two all obtained highly similar thresholds through this clustering method, further suggesting that this threshold represents the objective cognitive fatigue state, and suggesting that our fatigue-inducing model has consistency in the fatigue state threshold induced by different durations. In other words, this fatigue state threshold can be used to label the fatigue state of tasks of different durations.

[0048] The above-mentioned embodiments only express several implementation methods of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. A data annotation method for evaluating cognitive fatigue using working memory paradigm performance indicators, characterized in that: Follow these steps: 1) Recruit subjects to perform N-back test tasks and conduct practice test phase, which lasts for 10-20 minutes. Set the N value according to the accuracy of the subjects to ensure that the accuracy of the answers is higher than 70%; 2) Conduct a formal test of 1-3 hours on the subjects and record whether the answer to each question is correct and the time taken to answer the questions; 3) Calculate the accuracy of the subjects’ answers in a 10-20 minute time window with a step length of 1-10 seconds, and normalize the calculated task performance; 4) Use clustering algorithm to cluster the test data of both accuracy and reaction time; 5) Statistically cluster the data into different categories. First, sort them in ascending order using reaction time as the first sorting criterion and accuracy as the second sorting criterion. Select the classification labels with long reaction time and low accuracy and mark them as negative samples (0); mark the labels with shorter reaction time and higher accuracy as positive samples (1); after completing data labeling, use it as a training set for subsequent deep learning.

2. The data annotation method for evaluating cognitive fatigue using working memory paradigm performance indicators according to claim 1, characterized in that: Step 3) The task performance is accuracy and reaction time.

3. The data annotation method for evaluating cognitive fatigue using working memory paradigm performance indicators according to claim 1, characterized in that: The step 3) uses a z-score normalization method to perform normalization processing, and the processed data obtained establish an initial sample matrix.

4. The data annotation method for evaluating cognitive fatigue using working memory paradigm performance indicators according to claim 1, characterized in that: The clustering algorithm in step 4) is a k-means method, which is performed by calling the KMeans tool of the sklearn.cluster toolkit.

5. The data annotation method for evaluating cognitive fatigue using working memory paradigm performance indicators according to claim 4 is characterized in that: The parameters calculated by the k-means algorithm should meet the following conditions: BIC < -30000, silhouette coefficient > 0.45, Calinski-Harabasz index > 48000 and Davies-Bouldin index < 1.05, then the results are credible.

6. The data annotation method for evaluating cognitive fatigue using working memory paradigm performance indicators according to claim 1, characterized in that: The specific parameters of step 3) are set as follows: n_clusters=2, random_state=42, init='random', batch_size=32.

Images