A method and system for evaluating mental workload in a real environment based on eye movement data

Abstract

The present invention relates to a method and system for evaluating mental workload in a real environment based on eye movement data. The method comprises the following steps: respectively collecting eye movement data under different mental workload states, and preprocessing the eye movement data to construct an eye movement feature set; the eye movement feature set includes eye movement type statistical features, saccade speed statistical features, pupil diameter statistical features, and eye movement type transition features; establishing a mental workload evaluation model, training the mental workload evaluation model based on the eye movement feature set to obtain a trained mental workload evaluation model; collecting eye movement data of a user to be evaluated, preprocessing the eye movement data to construct an eye movement feature set of the user to be evaluated, and inputting the eye movement feature set into the trained mental workload evaluation model to obtain a mental workload evaluation result of the user to be evaluated.

Description

Technical Field

[0001] The present invention relates to the technical field of mental workload assessment, and particularly to a method and system for assessing real - environment mental workload based on eye movement data. Background Art

[0002] Workload has been increasingly emphasized in the development of complex human - machine interaction systems because workload has an important impact on the performance of operators in tasks such as visual search, decision - making, and operation. In ergonomics, existing workload measurement methods mainly include three types: behavioral measurement, subjective measurement, and neurophysiological measurement. Behavioral measurement: For example, using the performance of a specific task as an indicator of workload. Subjective measurement: Using some subjective questionnaires to collect subjective workload values during or after a task. Neurophysiological measurement: Such as electroencephalogram and electrocardiogram, as well as eye movement data used in the present invention, to quantitatively reflect the value of workload.

[0003] Existing measurement methods have the following problems:

[0004] Confidence problem: The performance indicators used in behavioral measurement are highly related to factors such as the operator's personal ability and proficiency; in subjective measurement, users may have selection biases due to psychological hints when answering questionnaires; electromyogram data, heart rate data, etc. are related to variables such as the operator's exercise intensity. Therefore, existing measurement methods may lead to unconfident measurement data.

[0005] Problem of data sparsity: Due to their measurement characteristics, behavioral measurement and subjective measurement do not have the advantage of being able to measure in real - time during a task like neurophysiological measurement methods. Therefore, the data obtained from the measurement has a long time interval, and each measurement interrupts the normal task process. Using the load data of subjective measurement and behavioral measurement as a baseline cannot avoid the problem of data sparsity. The sampling frequency of neurophysiological measurement (for example, 100 data points are collected per second) is often much higher than the load data of subjective measurement and behavioral measurement (for example, data is obtained about every 10 minutes). Therefore, learning algorithms may be difficult to detect short - term high workloads.

[0006] Practical application problem: Some neurophysiological measurement methods, such as electromyogram data and electroencephalogram data, require complex cables to be connected to the operator in contact, which will have a certain impact on the operator itself; and some electroencephalogram devices are bulky and cannot be applied to actual task processes. Summary of the Invention

[0007] In view of the above - mentioned analysis, embodiments of the present invention aim to provide a method and system for assessing real - environment mental workload based on eye movement data to solve the problems of low confidence, data sparsity, and high implementation difficulty in existing mental workload assessment.

[0008] On the one hand, an embodiment of the present invention provides a method for evaluating mental workload in a real environment based on eye movement data, including the following steps:

[0009] Collect eye movement data under different mental workload states respectively, and preprocess the eye movement data to construct an eye movement feature set; the eye movement feature set includes eye movement type statistical features, saccade speed statistical features, pupil diameter statistical features, and eye movement type transition features;

[0010] Establish a mental workload evaluation model, train the mental workload evaluation model based on the eye movement feature set, and obtain a trained mental workload evaluation model;

[0011] Collect the eye movement data of the user to be evaluated, preprocess the eye movement data to construct an eye movement feature set of the user to be evaluated, and input the eye movement feature set into the trained mental workload evaluation model to obtain the mental workload evaluation result of the user to be evaluated.

[0012] Based on a further improvement of the above technical solution, the eye movement types include fixation, saccade, unclassified, and no eye found; preprocessing the eye movement data to obtain an eye movement feature set includes:

[0013] Determine the eye movement type of the unclassified sequence according to the eye movement types of the preamble and subsequent eye movement type sequences of the unclassified sequence;

[0014] Replace the no eye found sequence with a blink when the sequence length is within the first threshold range according to the sampling frequency;

[0015] Calculate the count value, average value, variance, upper and lower quartiles, median, and maximum value of each eye movement type within each unit time respectively to construct eye movement type statistical features;

[0016] Calculate the average value, variance, upper and lower quartiles, median, and maximum value of the saccade speed, pupil diameter, and pupil diameter change rate within each unit time respectively to construct saccade speed statistical features and pupil diameter statistical features;

[0017] Calculate the Markov transition matrix of the eye movement types within each unit time respectively, and construct eye movement type transition features based on the Markov transition matrix.

[0018] Further, the following method is used to calculate the Markov transition matrix of the eye movement types within unit time:

[0019] According to the formula Calculate the transition probability P j from eye movement type C k to eye movement type C jk , and construct the Markov transition matrix P,

[0020]

[0021] Among them, P(x i = C j ) represents the probability that the eye movement type of the i-th data within a unit time is C j ; P(x i+1 = C k ; x i = C j ) represents the probability that the eye movement type of the i-th data within a unit time is C j , and the eye movement type of the (i + 1)-th data is C k , where k, j = 0, 1, 2, 3.

[0022] Furthermore, constructing eye movement type transition features based on the Markov transition matrix includes:

[0023] Taking the element values of the Markov transition matrix and the eigenvalues of the Markov transition matrix as eye movement type transition features.

[0024] Furthermore, constructing eye movement type transition features based on the Markov transition matrix includes:

[0025] Extracting the elements in the Markov transition matrix that are not related to the non-found eye type and the elements in the eigenvalues of the Markov transition matrix that are not related to the non-found eye type as eye movement type transition features.

[0026] Furthermore, determining the eye movement type of the unclassified sequence according to the eye movement types of the pre-order and post-order eye movement type sequences of the unclassified sequence includes:

[0027] If the eye movement types of the pre-order and post-order eye movement type sequences of the unclassified sequence are the same, replacing all of the unclassified sequence with the eye movement types of the pre-order and post-order eye movement type sequences;

[0028] If the eye movement types of the pre-order and post-order eye movement type sequences of the unclassified sequence are different, then replacing the first L u / 2 unclassified data of the unclassified sequence with the eye movement type of the pre-order eye movement type sequence, and replacing the last L u - L u / 2 unclassified data of the unclassified sequence with the eye movement type of the post-order eye movement type sequence, where L u represents the length of the current unclassified sequence.

[0029] Furthermore, using a feature selection algorithm to select features from the eye movement feature set whose feature weights are greater than a second threshold, and training the mental workload assessment model based on the selected features to obtain a trained mental workload assessment model.

[0030] Further, the eye movement feature set further includes: the difference in saccade speed between the left and right eyes, the difference in pupil diameter between the left and right eyes, and the difference in pupil diameter change rate between the left and right eyes.

[0031] Further, the eye movement feature set further includes head movement features;

[0032] The head movement features are obtained in the following manner:

[0033] The head movement acceleration data and angular velocity data in different mental workload states are collected respectively, and the average head movement acceleration and average angular velocity in each coordinate direction per unit time are calculated respectively to construct the head movement features.

[0034] On the other hand, an embodiment of the present invention provides a real - environment mental workload assessment system based on eye movement data, and the system includes the following modules:

[0035] An eye movement feature set construction module, configured to collect eye movement data in different mental workload states respectively, and pre - process the eye movement data to construct an eye movement feature set; the eye movement feature set includes eye movement type statistical features, saccade speed statistical features, pupil diameter statistical features, and eye movement type transition features;

[0036] A model training module, configured to establish a mental workload assessment model, and train the mental workload assessment model based on the eye movement feature set to obtain a trained mental workload assessment model;

[0037] A mental workload assessment module, configured to collect the eye movement data of the user to be evaluated, pre - process the eye movement data to construct an eye movement feature set of the user to be evaluated, and input the eye movement feature set into the trained mental workload assessment model to obtain the mental workload assessment result of the user to be evaluated.

[0038] Compared with the prior art, the present invention collects eye movement data in different mental workload states by using an eye tracker, thereby realizing non - contact collection, which can be measured during the task, avoiding the confidence problem and data sparsity problem, thus improving the measurement accuracy and making the evaluation more accurate and objective. And the device is lightweight, with the least impact on the operator, further improving the evaluation accuracy while being easy to implement.

[0039] In the present invention, the above - mentioned technical solutions can also be combined with each other to achieve more preferred combination schemes. Other features and advantages of the present invention will be described in the subsequent description, and some advantages can be made obvious from the description, or understood by implementing the present invention. The objectives and other advantages of the present invention can be realized and obtained through the content specifically pointed out in the description and the drawings. Description of the Drawings

[0040] The accompanying drawings are only for the purpose of showing specific embodiments and are not considered as limiting the present invention. Throughout the drawings, the same reference signs denote the same components.

[0041] Figure 1 This is a flowchart of the method for evaluating mental workload in a real environment based on eye movement data according to an embodiment of the present invention;

[0042] Figure 2 This is a block diagram of the system for evaluating mental workload in a real environment based on eye movement data according to an embodiment of the present invention;

[0043] Figure 3 This is a schematic diagram of the variance test of the eye movement type transfer characteristics according to an embodiment of the present invention;

[0044] Figure 4 This is a schematic diagram of the ranking of feature importance according to an embodiment of the present invention;

[0045] Figure 5 This is a schematic diagram of the five-fold cross-validation result of the evaluation model according to an embodiment of the present invention;

[0046] Figure 6 This is a schematic diagram of the classification ROC curve of the evaluation model according to an embodiment of the present invention. Detailed Embodiments

[0047] The preferred embodiments of the present invention will be specifically described below with reference to the accompanying drawings. The accompanying drawings form a part of this application and are used together with the embodiments of the present invention to explain the principles of the present invention, rather than to limit the scope of the present invention.

[0048] Workload has received increasing attention in the development of complex human-machine interaction systems because workload has an important impact on the performance of operators in tasks such as visual search, decision-making, and operation. In ergonomics, the existing workload measurement methods mainly include three types: behavioral measurement, subjective measurement, and neurophysiological measurement. Behavioral measurement: For example, taking the performance of a specific task as an indicator of workload. Subjective measurement: Using some subjective questionnaires to collect subjective workload values during or after a task. Neurophysiological measurement: Such as electroencephalogram and electrocardiogram, as well as eye movement data used in the present invention, to quantitatively reflect the value of workload.

[0049] The existing measurement methods have the following problems:

[0050] Confidence problem: The performance indicators used in behavioral measurement are highly related to factors such as the operator's personal ability and proficiency; in subjective measurement, users may have selection biases due to psychological cues when answering questionnaires; electromyography data, heart rate data, etc. are related to variables such as the operator's exercise intensity, so the existing measurement methods may lead to unconfident measurement data.

[0051] Problem of data sparsity: Due to their measurement characteristics, behavioral measurement and subjective measurement cannot have the advantage of real-time measurement during tasks like neurophysiological measurement methods. As a result, the data obtained from the measurement has a long time interval, and each measurement will interrupt the normal task process. Using the load data of subjective measurement and behavioral measurement as the baseline cannot avoid the problem of data sparsity. The sampling frequency of neurophysiological measurement (for example, 100 pieces of data are collected per second) is often much higher than that of the load data of subjective measurement and behavioral measurement (for example, data is obtained about every 10 minutes). Therefore, the learning algorithm may be difficult to detect short-term high loads.

[0052] Practical application problems: Some neurophysiological measurement methods, such as electromyogram data and electroencephalogram data, need to connect complex cables in contact with the operator, which will have a certain impact on the operator itself; and some electroencephalogram devices are huge and cannot be applied to the actual task process.

[0053] To solve the problems of low confidence, data sparsity, and great implementation difficulty in existing mental workload assessment, a specific embodiment of the present invention discloses a method for assessing mental workload in a real environment based on eye movement data, as Figure 1 shown, including the following steps:

[0054] S1. Respectively collect eye movement data in different mental workload states, and preprocess the eye movement data to construct an eye movement feature set; the eye movement feature set includes eye movement type statistical features, saccade speed statistical features, pupil diameter statistical features, and eye movement type transition features;

[0055] S2. Establish a mental workload assessment model, train the mental workload assessment model based on the eye movement feature set, and obtain a trained mental workload assessment model;

[0056] S3. Collect the eye movement data of the user to be evaluated, preprocess the eye movement data to construct an eye movement feature set of the user to be evaluated, and input the eye movement feature set into the trained mental workload assessment model to obtain the mental workload assessment result of the user to be evaluated.

[0057] During implementation, an eye tracker is used to collect eye movement data in different mental workload states. For example, the Tobii Glasses Pro2 eye tracker can be used.

[0058] By using an eye tracker to collect eye movement data in different mental workload states, non-contact collection can be achieved, and measurement can be carried out during tasks, avoiding the confidence problem and data sparsity problem, thereby improving the measurement accuracy and making the assessment more accurate and objective. And the device is lightweight, with the least impact on the operator, further improving the assessment accuracy while being easy to implement.

[0059] Specifically, the data collection process is as follows:

[0060] An eye tracker is used to collect the eye movement data of the subjects in a waking and task-free state as the eye movement data in the state of no mental load, and the corresponding data label is "no load".

[0061] The subjects perform test tasks of different difficulties. Each task has an observation and thinking time period and a response time period, and there is a rest time period after each task. The experiments of each difficulty level are repeated multiple times. For example, test tasks of three difficulty levels can be carried out. Each task has 10 s for observation and thinking, 2 s for response, and 3 s for rest. The tasks of each difficulty level can be repeated five times. The corresponding load labels, such as low load, medium load, and high load, are added to the eye movement data collected for each difficulty level task. To ensure that the collected data is under load conditions, only the eye movement data collected during the observation and thinking time period is retained.

[0062] In the experiment, to avoid the influence of the subjects' fatigue on the eye movement data, an experimental process with a relatively short time (such as 10 s) is adopted, and there is a rest time period after each experimental task. The overall experimental time is also relatively short, so as to ensure to the greatest extent that the subjects are not in a state of fatigue, and the changes in eye movement indicators all come from the load intensity objectively given in the experiment, thereby realizing accurate mental load assessment.

[0063] One piece of sampled data from the eye tracker includes eye movement type, left and right eye pupil diameters, and left and right eye fixation coordinates. Among them, the eye movement types include fixation, saccade, unclassified, and no eye found. The eye tracker cannot directly collect blink data, but some existing studies have shown that the indicators related to blinking have a strong correlation with the load. The existing method usually extracts the blink type from "no eye found" and "unclassified". The blink features obtained in this way are generally binary classification data, that is, blink or no blink. Therefore, the features extracted in this way only contain the information of whether there is a blink in the current piece of eye movement data, and the information content is less. The sampling frequency of the eye tracker is usually about 100 Hz, that is, there are 100 pieces of sampled data in one second. Directly classifying the collected data will cause data redundancy, and the actual load level detection does not require such a high detection frequency; compared with the method of denoising by smoothing filtering, the denoising effect of the averaged value is better and it is less affected by missing values; many existing eye movement indicators cannot be calculated in a single piece of data and can only be statistically analyzed after a period of recording. Therefore, this is also the reason why it is difficult to apply to classification algorithms. Therefore, it is necessary to preprocess the collected data, extract eye movement features, and provide a data basis for accurately evaluating mental load.

[0064] Specifically, in step S1, the preprocessing of the eye movement data to obtain an eye movement feature set includes:

[0065] S11. Determine the eye movement type of the unclassified sequence according to the eye movement types of the pre-order and post-order eye movement type sequences of the unclassified sequence;

[0066] Specifically, determining the eye movement type of the unclassified sequence according to the eye movement types of the pre-order and post-order eye movement type sequences of the unclassified sequence includes:

[0067] If the eye movement types of the pre-order and post-order eye movement type sequences of the unclassified sequence are the same, replace all of the unclassified sequence with the eye movement type of the pre-order and post-order eye movement type sequences;

[0068] If the eye movement types of the pre-order and post-order eye movement type sequences of the unclassified sequence are different, then replace the first L u / 2 unclassified data of the unclassified sequence with the eye movement type of the pre-order eye movement type sequence, and replace the last L u -L u / 2 unclassified data of the unclassified sequence with the eye movement type of the post-order eye movement type sequence, where L u represents the length of the current unclassified sequence.

[0069] That is, if the front and back of the unclassified sequence are of the same eye movement type, for example: …, A, A, A, unclassified, …, unclassified, A, A, …, and the eye movement types before and after are both A, then replace the unclassified in it with A;

[0070] If the front and back are of different eye movement types, for example: …, A, A, A, unclassified, …, unclassified, B, B, …, then replace the first half (L u / 2) of the unclassified sequence with type A, and the second half (L u -L u / 2) with type B.

[0071] S12. Replace the eye-not-found sequence with a blink according to the sampling frequency when the sequence length is within the first threshold range;

[0072] The inventor found through research that the natural blinking behavior of humans is generally 0.15 - 0.6 seconds. Therefore, if the length of the eye-not-found sequence is within the first threshold range, there must be blink data. Among them, the first threshold is calculated according to the sampling frequency, that is, the first threshold range = [sampling frequency × 0.15, sampling frequency × 0.6]. For example, if the sampling frequency is 100 Hz, then the first threshold range = [15, 60], and replace the eye-not-found sequence (EyesNotFound) with a blink (Blink) when the sequence length is within the range of 15 - 60.

[0073] Existing blink acquisitions are basically carried out through desktop eye trackers, or through the acquisition of image information and identification by means of computer vision (CV) - related methods, which are more costly and difficult to apply in actual task scenarios. The method of this application can conveniently and quickly extract blink classifications.

[0074] S13. Calculate the count value, average value, variance, upper and lower quartiles, median, and maximum value of each eye movement type within each unit time respectively to construct eye movement type statistical features;

[0075] During implementation, the unit time can be set to 1 second. If the sampling rate is 100 Hz, that is, 100 sampling data are sequentially extracted to calculate the count value, average value, variance, upper and lower quartiles, median, and maximum value of each eye movement type within this period of time.

[0076] Take the count value, average value, variance, upper and lower quartiles, median, and maximum value of each eye movement type as eye movement type statistical features and add them to the eye movement feature set.

[0077] S14. Calculate the average value, variance, upper and lower quartiles, median, and maximum value of saccade speed, pupil diameter, and pupil diameter change rate within each unit time respectively to construct saccade speed statistical features and pupil diameter statistical features;

[0078] During implementation, similarly, the unit time can be set to 1 second. If the sampling rate is 100 Hz, that is, 100 sampling data are sequentially extracted to calculate the count value, average value, variance, upper and lower quartiles, median, and maximum value of saccade speed within this period of time as saccade speed statistical features and add them to the eye movement feature set. Calculate the count value, average value, variance, upper and lower quartiles, median, and maximum value of pupil diameter and pupil diameter change rate within this period of time as pupil diameter statistical features and add them to the eye movement feature set.

[0079] During implementation, calculate the included angle between the fixation coordinates at two adjacent sampling moments within the unit time according to the following formula:

[0080]

[0081] (a i-1 , b i-1 , c i-1 ) represents the fixation coordinates at the (i - 1) sampling moment, and (a i , b i , c i ) represents the fixation coordinates at the i sampling moment;

[0082] Calculate the saccade speed between adjacent sampling moments according to the following formula:

[0083] V i =α i ×freq

[0084] Among them, freq represents the sampling frequency, and Vi represents the i-th saccade speed per unit time.

[0085] During implementation, the pupil change rate is calculated by dividing the forward difference of the pupil diameter data at two adjacent sampling moments by the time difference between the two sampling moments.

[0086] By adding the statistical features of eye movement types, saccade speeds, and pupil diameters, the features are less affected by missing values, the classification features are more abundant, which is beneficial to training a more accurate evaluation model, thereby achieving a more accurate mental workload assessment.

[0087] S15. Calculate the Markov transition matrix of the eye movement types within each unit time respectively, and construct the eye movement type transition features based on the Markov transition matrix.

[0088] Specifically, the following method is used to calculate the Markov transition matrix of the eye movement types within the unit time and construct the eye movement type transition features based on the Markov transition matrix:

[0089] According to the formula calculate the transition probability P j from the eye movement type C k to the eye movement type C jk within the unit time, and construct the Markov transition matrix P,

[0090]

[0091] Take the element values of the Markov transition matrix and the eigenvalue of the Markov transition matrix as the eye movement type transition features;

[0092] Among them, P(x i =C j ) represents the probability that the eye movement type of the i-th data within the unit time is C j , and P(x i+1 =C k ; x i =C j ) represents the probability that the eye movement type of the i-th data within the unit time is C j , and the eye movement type of the (i + 1)-th data is C k , where k, j = 0, 1, 2, 3.

[0093] After being processed through steps S11 to S12, the eye movement types in the eye movement data include fixation, saccade, blink, and no eye found. The four types of states, namely fixation, saccade, blink, and no eye found, are labeled as: 1, 2, 3, 0. Within each unit of time, that is, within every 100 sampled data, the transition probabilities between these four types of states are calculated according to the above formula to construct a Markov transition matrix P. The element values of the Markov transition matrix and the eigenvalues of the Markov transition matrix are used as the eye movement type transition features and added to the eye movement feature set. Since the no eye found type is meaningless for mental workload assessment, for the sake of simplifying the calculation, the elements in the Markov transition matrix that are related to the no eye found type (the elements in the first row and the first column of matrix P) are removed, and the other matrix elements are added to the eye movement feature set. At the same time, the eigenvalues related to the no eye found type (i.e., the first eigenvalue of the calculated matrix P) are removed from the eigenvalues, and the other eigenvalues are added to the eye movement feature set. That is, the elements in the Markov transition matrix that are not related to the no eye found type and the eigenvalues in the eigenvalues of the Markov transition matrix that are not related to the no eye found type are extracted as the eye movement type transition features.

[0094] Traditional eye movement type features are discrete data features. The current data row only contains the current state information. The Markov transition matrix calculates the state transition probability over a period of time, converting the sparse discrete states into continuous state transition probabilities. The data information it contains is not only the current state but also the state change information in a period of time before and after the current time point. Therefore, compared with traditional state features, it contains more feature information, making the feature information richer and facilitating accurate mental workload assessment.

[0095] Such as Figure 3 shows the variance test comparison results between the traditional blink count index and the Markov state transition probability index. From Figure 3 (a), it can be seen that when there is a workload, the distributions of blink counts are similar (the bar charts corresponding to the first, second, and third categories are almost the same), while in 3(b), the relevant features of the Markov transition matrix clearly show that the distributions of each category are different.

[0096] The research found that the pupil difference between the left and right eyes is related to cognitive load and task complexity. During implementation, in order to further improve the classification accuracy, the difference in pupil diameter between the left and right eyes and the difference in the change rate of pupil diameter between the left and right eyes are added to the eye movement feature set as part of the pupil diameter statistical features. The difference in saccade speed between the left and right eyes is added to the eye movement feature set as part of the saccade speed statistical features.

[0097] In addition, research shows that the task difficulty is significantly correlated with the head movement of the subjects. To further improve the evaluation accuracy, head movement information is added to the eye movement feature set. The Tobii Glasses Pro2 eye tracker is equipped with sensors and gyroscopes for collecting head movement acceleration and angle information. During implementation, while the eye tracker collects eye movement data under different mental workload states, it also collects head movement acceleration data and angular velocity data. The average acceleration in three coordinate directions and the average angular velocity in three coordinate directions per unit time are used as head movement features and added to the eye movement feature set.

[0098] After the above processing steps, the eye movement feature set includes the statistical features of eye movement types per second, saccade speed statistical features, pupil diameter statistical features, pupil diameter change rate statistical features, and head movement features. The feature data and the corresponding workload type labels (no load, low load, medium load, high load) constitute the training sample data.

[0099] During implementation, since there are many features in the eye movement feature set, to improve the calculation efficiency, a feature selection algorithm is used to select features with feature weights greater than the second threshold from the eye movement feature set, and the mental workload evaluation model is trained based on the selected features to obtain a trained mental workload evaluation model.

[0100] During implementation, the ReliefF algorithm can be used to calculate the feature weights and select features with weights greater than the second threshold. During implementation, the second threshold can be set to 0, that is, the features showing positive performance are used as the input features for model training. Figure 4 Shows the weight ranking of each feature, and features with weight values greater than the second threshold can be selected for model training.

[0101] During implementation, the established mental workload evaluation model can be an SVM model, or other models can also be used. The mental workload evaluation SVM model constructed based on the selected features is trained according to the ratio of 8:2 for the training set and the test set to obtain a trained mental workload evaluation model.

[0102] The classification effect of the finally trained model is as Figure 5 and Figure 6 shown. Figure 5 The result of model verification using the five-fold cross-validation method shows that the overall classification accuracy is 77.5%, and the classification accuracy for no load and high load both reaches over 80% (no load 83.8%, high load 80.7%). Figure 6 This is the ROC curve (Receiver Operating Characteristic Curve) of the model classification. Figure 6 (a) Positive class: no load, negative classes: low load, medium load, high load; Figure 6(b) Positive class: low load, negative classes: no load, medium load, high load; Figure 6 (c) Positive class: medium load, negative classes: no load, low load, high load; 6 (d) Positive class: high load, negative classes: no load, low load, medium load. Generally, the area under the ROC curve (AUC) is used as a performance metric to measure the quality of a learner. The closer the AUC is to 1, the better the performance of the learner. As can be seen from Figure 6 it, all four AUC values are above 0.9, indicating that the classifier has excellent performance.

[0103] After obtaining the trained mental workload assessment model, for the user to be evaluated, first collect their eye movement data, preprocess the collected eye movement data according to the preprocessing process in step S1 to obtain the eye movement feature data of the user to be evaluated, and input the feature data into the trained mental workload assessment model to obtain the mental workload assessment result of the user to be evaluated.

[0104] A specific embodiment of the present invention discloses a real - environment mental workload assessment system based on eye movement data, as Figure 2 shown. The system includes the following modules:

[0105] An eye movement feature set construction module, which is used to collect eye movement data in different mental workload states respectively, preprocess the eye movement data to construct an eye movement feature set; the eye movement feature set includes eye movement type statistical features, saccade speed statistical features, pupil diameter statistical features, and eye movement type transition features;

[0106] A model training module, which is used to establish a mental workload assessment model, train the mental workload assessment model based on the eye movement feature set, and obtain a trained mental workload assessment model;

[0107] A mental workload assessment module, which is used to collect the eye movement data of the user to be evaluated, preprocess the eye movement data to construct an eye movement feature set of the user to be evaluated, and input the eye movement feature set into the trained mental workload assessment model to obtain the mental workload assessment result of the user to be evaluated.

[0108] The above - mentioned method embodiments and system embodiments are based on the same principle, and their related parts can be mutually referred to and can achieve the same technical effects. For the specific implementation process, refer to the foregoing embodiments, and details are not described herein again.

[0109] Those skilled in the art can understand that all or part of the processes for implementing the above - mentioned method embodiments can be completed by instructing relevant hardware through a computer program, and the program can be stored in a computer - readable storage medium. Among them, the computer - readable storage medium is a magnetic disk, an optical disk, a read - only memory, or a random access memory, etc.

[0110] The above are only the preferred specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by those skilled in the art within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention.

Claims

1. A method for evaluating mental workload in a real environment based on eye movement data, characterized in that, It includes the following steps: Collect eye movement data under different mental workload states respectively, and preprocess the eye movement data to construct an eye movement feature set; the eye movement feature set includes eye movement type statistical features, saccade speed statistical features, pupil diameter statistical features, and eye movement type transition features; Establish a mental workload assessment model, train the mental workload assessment model based on the eye movement feature set, and obtain a trained mental workload assessment model; Collect the eye movement data of the user to be evaluated, preprocess the eye movement data to construct an eye movement feature set of the user to be evaluated, and input the eye movement feature set into the trained mental workload assessment model to obtain the mental workload assessment result of the user to be evaluated; The eye movement types include fixation, saccade, unclassified, and no eye found; Preprocessing the eye movement data to obtain an eye movement feature set includes: Determine the eye movement type of the unclassified sequence according to the eye movement types of the previous and subsequent eye movement type sequences of the unclassified sequence; Replace the no eye found sequence with a blink if its sequence length is within the first threshold range according to the sampling frequency; Calculate the count value, average value, variance, upper and lower quartiles, median, and maximum value of each eye movement type within each unit time respectively to construct eye movement type statistical features; Calculate the average value, variance, upper and lower quartiles, median, and maximum value of the saccade speed, pupil diameter, and pupil diameter change rate within each unit time respectively to construct saccade speed statistical features and pupil diameter statistical features; Calculate the Markov transition matrix of the eye movement types within each unit time respectively, and construct eye movement type transition features based on the Markov transition matrix; Calculate the Markov transition matrix of the eye movement types within each unit time in the following way: According to the formula calculate the transition probability P j from eye movement type C k to eye movement type C jk per unit time, and construct the Markov transition matrix P Among them, P(x i = C j ) represents the probability that the eye movement type of the i-th data within a unit time is C j , P(x i+1 = C k ; x i = C j ) represents the probability that the eye movement type of the i-th data within a unit time is C j , and the eye movement type of the (i + 1)-th data is C k , where k, j = 0, 1, 2, 3; Determine the eye movement type of the unclassified sequence according to the eye movement types of the previous and subsequent eye movement type sequences of the unclassified sequence, including: If the eye movement types of the previous and subsequent eye movement type sequences of the unclassified sequence are the same, replace all the unclassified sequences with the eye movement types of the previous and subsequent eye movement type sequences; If the eye movement types of the preamble and the subsequent eye movement type sequences of the unclassified sequence are different, then the first L u / 2 unclassified data of the unclassified sequence are replaced with the eye movement type of the preamble eye movement type sequence, and the last L u -L u / 2 unclassified data of the unclassified sequence are replaced with the eye movement type of the subsequent eye movement type sequence, where L u represents the length of the current unclassified sequence; The data collection process includes: Use an eye tracker to collect the eye movement data of the subject in a waking and task-free state as the eye movement data in the no mental workload state; The subject performs test tasks of different difficulties. Each task has an observation and thinking time and a response time, and there is a rest time after the task. Collect the eye movement data during the observation and thinking period to obtain the eye movement data corresponding to different mental workloads.

2. The method for evaluating the mental workload in a real environment based on eye movement data according to claim 1, wherein Construct eye movement type transition features based on the Markov transition matrix, including: Use the element values of the Markov transition matrix and the eigenvalue of the Markov transition matrix as eye movement type transition features.

3. The real environment mental workload assessment method based on eye movement data according to claim 1, wherein, Construct eye movement type transition features based on the Markov transition matrix, including: Extract the elements in the Markov transition matrix that are not related to the no eye found type and the elements in the eigenvalue of the Markov transition matrix that are not related to the no eye found type as eye movement type transition features.

4. The method for evaluating the mental workload in a real environment based on eye movement data according to claim 1, characterized in that, Use a feature selection algorithm to select the features with feature weights greater than the second threshold from the eye movement feature set, and train the mental workload assessment model based on the selected features to obtain a trained mental workload assessment model.

5. The method for evaluating the mental workload in a real environment based on eye movement data according to claim 1, wherein The eye movement feature set also includes: the difference in saccade speed between the left and right eyes, the difference in pupil diameter between the left and right eyes, and the difference in pupil diameter change rate between the left and right eyes.

6. The method for evaluating the mental workload in a real environment based on eye movement data according to claim 1, wherein The eye movement feature set also includes head movement features; The head movement features are obtained in the following way: Collect the head movement acceleration data and angular velocity data under different mental load states respectively, and calculate the average head movement acceleration and average angular velocity in each coordinate direction per unit time to construct the head movement features.

7. A real - environment mental workload assessment system based on eye movement data, characterized in that, The system includes the following modules: An eye movement feature set construction module, which is used to collect eye movement data under different mental load states respectively, and preprocess the eye movement data to construct an eye movement feature set; the eye movement feature set includes eye movement type statistical features, saccade speed statistical features, pupil diameter statistical features, and eye movement type transition features; A model training module, which is used to establish a mental load assessment model, train the mental load assessment model based on the eye movement feature set, and obtain a trained mental load assessment model; A mental load assessment module, which is used to collect the eye movement data of the user to be evaluated, preprocess the eye movement data to construct the eye movement feature set of the user to be evaluated, and input the eye movement feature set into the trained mental load assessment model to obtain the mental load assessment result of the user to be evaluated; The eye movement types include fixation, saccade, unclassified, and no eye found; Preprocessing the eye movement data to obtain an eye movement feature set includes: Determine the eye movement type of the unclassified sequence according to the eye movement types of the preceding and subsequent eye movement type sequences of the unclassified sequence; Replace the no eye found sequence with a blink if the sequence length is within the first threshold range according to the sampling frequency; Calculate the count value, average value, variance, upper and lower quartiles, median, and maximum value of each eye movement type per unit time respectively to construct the eye movement type statistical features; Calculate the average value, variance, upper and lower quartiles, median, and maximum value of the saccade speed, pupil diameter, and pupil diameter change rate per unit time respectively to construct the saccade speed statistical features and pupil diameter statistical features; Calculate the Markov transition matrix of the eye movement types per unit time respectively, and construct the eye movement type transition features based on the Markov transition matrix; Calculate the Markov transition matrix of the eye movement types per unit time in the following way: According to the formula calculate the transition probability P j from eye movement type C k to eye movement type C jk per unit time, and construct the Markov transition matrix P where P(x i = C j ) represents the probability that the eye movement type of the i-th data within a unit time is C j , P(x i+1 = C k ; x i = C j ) represents the probability that the eye movement type of the i-th data within a unit time is C j , the eye movement type of the (i + 1)-th data is C k , k, j = 0, 1, 2, 3; Determine the eye movement type of the unclassified sequence according to the eye movement types of the preceding and subsequent eye movement type sequences of the unclassified sequence, including: If the eye movement types of the preceding and subsequent eye movement type sequences of the unclassified sequence are the same, replace all the unclassified sequences with the eye movement types of the preceding and subsequent eye movement type sequences; If the eye movement types of the pre-order and post-order eye movement type sequences of the unclassified sequence are different, then the first L u / 2 unclassified data of the unclassified sequence are replaced with the eye movement type of the pre-order eye movement type sequence, and the last L u -L u / 2 unclassified data of the unclassified sequence are replaced with the eye movement type of the post-order eye movement type sequence, where L u represents the length of the current unclassified sequence; The data collection process includes: Use an eye tracker to collect the eye movement data of the subject in the awake and task-free state as the eye movement data in the no mental load state; The subject performs test tasks of different difficulties. Each task has an observation and thinking time and an answering time, and there is a rest time after the task. Collect the eye movement data during the observation and thinking period to obtain the eye movement data corresponding to different mental loads.

Images