A cognitive load detection method based on eye movement data

Abstract

The application discloses a cognitive load detection method based on eye movement data and belongs to the technical field of physiological signal processing. The method comprises the following steps: acquiring eye movement data with a cognitive load label, constructing eye movement time sequence samples and corresponding blink labels; constructing pupil feature vectors of the eye movement time sequence samples based on the eye movement time sequence samples; performing event division on the eye movement time sequence samples by using a speed threshold value recognition algorithm to obtain event feature vectors of the eye movement time sequence samples; training a cognitive load deep learning model by using the event feature vectors and the pupil feature vectors; confirming key features of cognitive load calculation by using a SHAP method based on the trained cognitive load deep learning model; and obtaining a cognitive load result by using the trained cognitive load deep learning model based on target eye movement data corresponding to the key features. The application effectively reduces the model overfitting risk, extracts key features and improves the cognitive load detection efficiency.

Description

Technical Field

[0001] This invention belongs to the field of deep learning and physiological signal processing technology, specifically relating to a cognitive load detection method based on eye-tracking data. Background Technology

[0002] Cognitive load reflects the intensity of brain activity during task execution. Real-time and accurate detection of cognitive load is crucial for optimizing human-computer interaction and improving the operational safety of complex systems. Traditional cognitive load assessment methods, such as subjective self-assessment scales and task performance indicators, have inherent limitations, including requiring interruption of the main task, exhibiting lag, and being susceptible to subjective interference, making it difficult to achieve continuous, real-time dynamic detection.

[0003] Eye movement (EM) signals, due to their objective, non-invasive, and continuous acquisition characteristics, have become an ideal physiological indicator for cognitive workload detection. Existing research indicates that eye movement parameters such as pupil diameter, fixation duration, and saccade amplitude are correlated with cognitive workload.

[0004] However, current technologies still face the following challenges: raw eye-tracking data contains a large amount of noise; different studies employ significantly different interpolation methods and filtering strategies, resulting in a lack of comparability between studies; deep learning models suffer from overfitting and insufficient generalization in cognitive load classification; and the excessively high dimensionality of eye-tracking features affects real-time detection efficiency. Existing methods often extract more than ten dimensions of features, which contain a large amount of redundant information, increasing computational overhead and hindering the deployment of real-time cognitive load detection systems. Summary of the Invention

[0005] The purpose of this invention is to provide a cognitive load detection method based on eye-tracking data. By analyzing the trained cognitive load deep learning model using the SHAP method, the input features of the cognitive load deep learning model are simplified, thereby improving the efficiency of cognitive load detection.

[0006] To achieve the above objectives, the present invention is implemented using the following technical solution.

[0007] This invention provides a cognitive load detection method based on eye-tracking data, comprising:

[0008] Acquire eye-tracking data of the target object;

[0009] The eye-tracking data of the target to be tested is input into the trained cognitive load deep learning model to obtain the cognitive load detection result;

[0010] The training method for the cognitive load deep learning model includes:

[0011] Based on the acquired eye-tracking data with cognitive load labels, eye-tracking time-series samples with cognitive load labels are constructed through standardization processing and a sliding window of preset length, and blink labels are constructed for the eye-tracking time-series samples.

[0012] Construct pupil feature vectors for the eye movement time series samples based on the aforementioned eye movement time series samples;

[0013] An eye-tracking time series sample was segmented into events using a velocity threshold recognition algorithm to obtain event labels for the eye-tracking time series sample.

[0014] Based on blink labels and event labels of eye-tracking time-series samples, an event feature vector of eye-tracking time-series samples is constructed;

[0015] Based on the event feature vector and pupil feature vector of eye movement time sequence samples, a cognitive load deep learning model is trained through a regularization strategy.

[0016] The method for acquiring the eye-tracking data of the target to be inspected includes:

[0017] Based on the trained cognitive load deep learning model, the key features of the cognitive load deep learning model for cognitive load calculation were identified by the SHAP method.

[0018] Obtain the eye movement data of the target to be inspected corresponding to the key features.

[0019] Optionally, the eye-tracking data includes single-frame validity data, single-frame pupil diameter data, gaze origin position, gaze direction, and binocular position;

[0020] The process involves constructing eye-tracking time-series samples with cognitive load labels based on the acquired eye-tracking data, through standardization and a sliding window of preset length, and constructing blink labels for the eye-tracking time-series samples, including:

[0021] Based on the single-frame validity data, it is determined whether the preset blink duration constraint is met. If it is met, the corresponding eye movement data is marked as a blinking action; otherwise, the corresponding eye movement data is marked as missing data.

[0022] The eye-tracking data marked as missing data were filled using linear interpolation to obtain complete eye-tracking data.

[0023] Based on the completed eye-tracking data, an eye-tracking temporal sequence is obtained through standardization processing;

[0024] Based on the eye-tracking time sequence, eye-tracking time samples are constructed using a sliding window of a preset length;

[0025] Blink labels for eye movement time series samples are constructed based on eye movement time series samples and eye movement data labeled as blinking actions.

[0026] Optionally, the eye-tracking time-series sample includes multiple sets of single-frame validity data and multiple sets of single-frame pupil diameter data;

[0027] The construction of the pupil feature vector of the eye movement time series sample based on the eye movement time series sample includes:

[0028] Based on the eye-tracking time-series samples, the fused pupil diameter is obtained by averaging the single-frame pupil diameter data corresponding to the single-frame valid data.

[0029] The average pupil diameter of the eye movement time series samples is obtained by summing and averaging the fused pupil diameters.

[0030] The standard deviation of the pupil diameter is calculated based on the mean pupil diameter and the fused pupil diameter.

[0031] The dynamic change rate of adjacent valid frames is calculated based on the fused pupil diameter;

[0032] Based on the dynamic rate of change, the pupil change rate is calculated using the arithmetic mean.

[0033] A pupil feature vector is constructed based on the mean pupil diameter, standard deviation of pupil diameter, and pupil change rate.

[0034] Optionally, the expression for calculating the mean pupil diameter of the eye movement time series samples is as follows:

[0035] ;

[0036] in, The average pupil diameter. Let t be the fused pupil diameter, N be the number of valid sampling points within the window, and W be the time set of the data within the window;

[0037] The standard deviation of the pupil diameter is calculated as follows:

[0038] ;

[0039] in, The standard deviation of pupil diameter;

[0040] The expression for calculating the pupil change rate is:

[0041] ;

[0042] ;

[0043] in, for The rate of dynamic change at any given moment. for The fused pupil diameter at any given moment for The fused pupil diameter at time step, where R is the set of real numbers representing the number of valid sampling points within the window. This represents the pupillary change rate.

[0044] Optionally, the step of using a velocity threshold recognition algorithm to segment eye-tracking time-series samples into events to obtain event labels for the eye-tracking time-series samples includes:

[0045] Based on the aforementioned eye-tracking time-series samples, event labels for the eye-tracking time-series samples are constructed by filtering through fixation constraints and saccade constraints.

[0046] Both the gaze constraint and the saccade constraint include an angular velocity threshold and a duration threshold.

[0047] Optionally, the event feature vector includes saccade feature vector, gaze feature vector, and blink feature vector;

[0048] The saccade feature vector includes the average saccade amplitude, the average saccade speed, and the number of saccades.

[0049] The gaze feature vector includes average gaze duration, number of gazes, and gaze dispersion.

[0050] The blink feature vector includes blink frequency.

[0051] Optionally, the loss function during the training of the cognitive load deep learning model is the classification cross-entropy function;

[0052] The event feature vector and pupil feature vector based on eye movement time-series samples are used to train a cognitive load deep learning model through a regularization strategy, including:

[0053] An eye movement time series sample set is constructed based on the event feature vector and pupil feature vector of the aforementioned eye movement time series samples;

[0054] The learning parameters of the cognitive load deep learning model are updated by fitting the eye-tracking time-series sample set until the classification cross-entropy function result satisfies the preset iteration termination condition or the preset number of iterations, thus obtaining the trained cognitive load deep learning model.

[0055] Optionally, the structure of the cognitive load deep learning model includes an input layer, a bidirectional LSTM layer, a batch normalization layer, a fully connected layer, a Dropout layer, and an output layer; the bidirectional LSTM layer includes a first LSTM layer and a second LSTM layer, and the batch normalization layer includes a first batch normalization layer and a second batch normalization layer.

[0056] When updating the learning parameters of the cognitive load deep learning model using the eye movement time series sample set, the event feature vector and pupil feature vector are input into the first LSTM layer in time series using the input layer;

[0057] Based on the event feature vector and pupil feature vector, the temporal dependency is captured through the first LSTM layer to obtain the time step features;

[0058] Based on the time step features, the time step features are standardized by the first batch of normalization layers.

[0059] Based on the standardized time step features, feature compression is performed through the second LSTM layer to obtain the temporal feature vector;

[0060] The time series feature vectors are standardized using the second batch of normalization layers to obtain stable time series feature vectors.

[0061] Based on the stable temporal feature vector, a nonlinear mapping is performed through a fully connected layer, and a Dropout layer is used for generalization to obtain the temporal mapping features.

[0062] The cognitive load level is obtained by mapping based on the temporal mapping features using the output layer.

[0063] Optionally, a background pool of eye-tracking time-series samples is constructed based on the event feature vector and pupil feature vector of the eye-tracking time-series samples, and the cognitive load features of the input cognitive load deep learning model are confirmed.

[0064] The background pool of the eye-tracking time-series samples includes multiple sets of cognitive load features;

[0065] Based on the background pool, the cognitive load level is predicted by the trained cognitive load deep learning model to obtain the baseline cognitive load prediction value.

[0066] The trained cognitive load deep learning model, through the SHAP method, identifies key characteristics of cognitive load calculation using this model, including:

[0067] Step S1: Randomly select a cognitive load feature as the cognitive load feature to be tested;

[0068] Step S2: Construct a test feature dataset based on the acquired background pool features of the cognitive load to be tested;

[0069] Step S3: Randomly replace the cognitive load features to be tested in the background pool with data from the dataset of features to be tested to obtain the background pool of cognitive load features to be tested.

[0070] Step S4: Based on the background pool of the cognitive load features to be tested, the cognitive load level is predicted by the trained cognitive load deep learning model to obtain the predicted value of the cognitive load to be tested.

[0071] Step S5: Calculate the contribution value of the current cognitive load to be measured based on the predicted value of the cognitive load to be measured and the baseline predicted value of the cognitive load.

[0072] Repeat steps S1 to S5 until all cognitive load features have corresponding contribution values; sort and normalize the cognitive load features according to the contribution values, and confirm the key features for cognitive load calculation by the cognitive load deep learning model through preset key feature constraints.

[0073] Optionally, the calculation expression for the contribution value of the current cognitive load to be measured, calculated based on the predicted value of the cognitive load to be measured and the baseline predicted value of the cognitive load, is as follows:

[0074]

[0075] in, Let F be the contribution value of cognitive load feature i, and F be the background pool. This represents the background pool of cognitive load features obtained by randomly replacing the cognitive load features in the background pool using the dataset of features to be tested. The background pool for the cognitive load characteristics to be measured. The predicted value of the cognitive load to be measured. The baseline cognitive load forecast value, To represent factorial, This indicates the number of features in the current set.

[0076] Compared with the prior art, the beneficial effects achieved by the present invention are as follows:

[0077] This invention improves the accuracy of eye-tracking time-series sample labeling by using a velocity threshold recognition algorithm, thereby enhancing the prediction accuracy of the subsequent cognitive load deep learning model optimized based on these event labels. The use of a regularization strategy to train the cognitive load deep learning model effectively improves its generalization ability. Furthermore, the SHAP method identifies key features of the cognitive load deep learning model, accurately recognizing crucial characteristics affecting cognitive load calculation and simplifying the input feature requirements. This invention reduces the cost of cognitive load detection and enables real-time cognitive load detection. Attached Figure Description

[0078] Figure 1 This is a schematic diagram of the bidirectional LSTM model structure constructed in this invention;

[0079] Figure 2 This is a comparison chart of the training results of the LSTM model of this invention with other models;

[0080] Figure 3 This is a ranking diagram of the importance of SHAP features in this invention. Detailed Implementation

[0081] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the embodiments of the present invention and the specific features in the embodiments are detailed descriptions of the technical solution of the present invention, rather than limitations thereof. In the absence of conflict, the embodiments of the present invention and the technical features in the embodiments can be combined with each other.

[0082] The term "and / or" simply describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. Additionally, the character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0083] Example 1

[0084] This embodiment introduces a cognitive load detection method based on eye-tracking data, including:

[0085] Based on eye-tracking data labeled with cognitive load, a time-series eye-tracking sequence was obtained through standardization. The cognitive load-labeled eye-tracking data was obtained from an open-source dataset containing eye-tracking data from 100 healthy subjects performing tasks of three difficulty levels in a VR environment. Low load consisted of single-task visual tracking, medium load included mental arithmetic, and high load added an auditory alertness task. Each subject completed nine trials (three repetitions for each of the three difficulty levels), verbally reporting their mental state after each trial. Eye-tracking data was acquired using an eye tracker at 120Hz, including left and right pupil diameters, three-dimensional coordinates of the fixation point, and cognitive load labels.

[0086] Based on the eye movement time sequence, eye movement time sequence samples with cognitive load labels are constructed by dividing the data through a sliding window of preset length; based on the eye movement time sequence samples, blink labels of the eye movement time sequence samples are constructed by identifying and analyzing the data.

[0087] Based on eye-tracking time-series samples, pupil feature vectors of the eye-tracking time-series samples are constructed by calculating the pupil diameters of the left and right eyes;

[0088] Based on the preset gaze constraints and saccade constraints, the eye movement time series sample is divided into events using a velocity threshold recognition algorithm to obtain event labels for each time period within the eye movement time series sample.

[0089] Based on blink and event labels of eye-tracking time-series samples, an event feature vector is constructed for the eye-tracking time-series samples. The event feature vector includes saccade feature vector, fixation feature vector, and blink feature vector. The saccade feature vector includes average saccade amplitude, average saccade velocity, and number of saccades. The fixation feature vector includes average fixation duration, number of fixations, and fixation dispersion. The blink feature vector includes blink frequency.

[0090] Based on the event feature vector and pupil feature vector of eye movement time sequence samples, a cognitive load deep learning model is trained through a regularization strategy to reduce the probability of overfitting and insufficient generalization caused by model overtraining.

[0091] Based on the trained cognitive load deep learning model, the key features for cognitive load calculation are identified using the Shapley Additive exPlanations (SHAP) algorithm. By pre-setting constraints on these key features, cognitive load features with minimal impact on the results are eliminated, reducing the computational burden of feature construction and model building. By reducing the number of input features required for cognitive load calculation by the cognitive load deep learning model, the efficiency of input feature acquisition is improved, ensuring the immediacy of cognitive load detection and reducing the difficulty and cost of acquiring eye-tracking data of the target object.

[0092] Obtain the eye movement data of the target object corresponding to the key features; reverse the calculation of the key eye movement data required for cognitive load by using the key features, which not only reduces the difficulty of obtaining eye movement data, but also improves the efficiency of constructing key features based on the eye movement data.

[0093] The eye-tracking data of the target to be examined is input into the trained cognitive load deep learning model to obtain the cognitive load result.

[0094] Example 2

[0095] Based on the same inventive concept as Embodiment 1, this embodiment introduces a cognitive load detection method based on eye-tracking data, specifically including:

[0096] Based on the acquisition of eye-tracking data with cognitive load labels, the eye-tracking data includes validity data, pupil diameter data, gaze origin position, gaze direction, and binocular position. Validity data is used to describe whether the eye-tracking data acquired in a single frame is valid, which is called single-frame validity data. Pupil diameter data is used to describe the pupil diameter parameter acquired in a single frame, which is called single-frame pupil diameter data.

[0097] Based on the validity data, it is determined whether the preset blink duration constraint is met. If it is met, the corresponding eye movement data is marked as a blink action; otherwise, the corresponding eye movement data is marked as missing data.

[0098] Linear interpolation was used to fill in the eye movement data marked as missing data to obtain complete eye movement data. The linear interpolation not only supplemented the valid data, but also a set of pupil diameter data, fixation origin position, fixation direction and binocular position corresponding to the valid data. Because eye movement is a smooth process without jumps, the linear interpolation method can maximize the accuracy of the data, maintain local continuity and reduce the risk of overfitting.

[0099] Eye movement time sequence is obtained by standardizing the complete eye movement data; data standardization ensures the efficiency of subsequent data processing.

[0100] Based on eye-tracking time sequences, eye-tracking time sequence samples are constructed using a sliding window of preset length. The size of the eye-tracking time sequence samples can be freely adjusted by setting the length of the sliding window. The eye-tracking time sequence samples include multiple sets of single-frame valid data and multiple sets of single-frame pupil diameter data.

[0101] Based on eye movement time-series samples and eye movement data labeled as blinking actions, blink labels are constructed for eye movement time-series samples. Specifically, based on the threshold set in the code, it is determined whether continuous valid data within the time-series sample is marked as invalid, and hard classification is performed to mark values ​​within the threshold as blinks and values ​​outside the threshold as noise. The duration of a normal spontaneous blink is about 50~120ms. In this implementation, the range is widened to 20~150ms to cover individual differences. At the same time, invalid segments caused by instantaneous sensor jitter that are too short or invalid segments caused by device detachment and eye closure that are too long are removed.

[0102] Construct pupil feature vectors based on eye-tracking time-series samples, including:

[0103] Based on eye-tracking time-series samples, the fused pupil diameter is obtained by averaging the single-frame pupil diameter data corresponding to the single-frame valid data.

[0104] The mean pupil diameter of the eye-tracking time series samples is obtained by summing and averaging the fused pupil diameters. The specific calculation expression is as follows:

[0105] ;

[0106] in, The average pupil diameter. Let t be the fused pupil diameter, N be the number of valid sampling points within the window, and W be the time set of the data within the window;

[0107] The standard deviation of pupil diameter is calculated based on the mean pupil diameter and the fused pupil diameter. The specific calculation expression is as follows:

[0108] ;

[0109] in, The standard deviation of pupil diameter;

[0110] The dynamic change rate between adjacent valid frames is calculated based on the fused pupil diameter, and the calculation expression is as follows:

[0111] ;

[0112] Based on the dynamic rate of change, the pupillary change rate is calculated using the arithmetic mean. The calculation expression is as follows:

[0113] ;

[0114] in, for The rate of dynamic change at any given moment. for The fused pupil diameter at any given moment for The fused pupil diameter at time step, where R is the set of real numbers representing the number of valid sampling points within the window. This represents the pupillary change rate.

[0115] A pupil feature vector is constructed based on the mean pupil diameter, standard deviation of pupil diameter, and pupil change rate.

[0116] An eye-tracking time-series sample is segmented using a velocity threshold recognition algorithm to obtain event labels for the sample, including:

[0117] Based on eye-tracking time-series samples, event labels for the samples are constructed by filtering them through gaze constraints and saccade constraints.

[0118] The gaze constraint and saccade constraint include angular velocity threshold and duration threshold.

[0119] In this embodiment, angular velocity is calculated based on the angle difference and time difference between adjacent sampling points. A threshold of 30° / s is used to distinguish between fixation and saccades: ≤30° / s indicates fixation, ≥40° / s indicates saccades, and the intermediate state is undefined. The event duration is verified: fixation 100-600ms, saccades 20-100ms; any deviations are marked as noise.

[0120] Based on blink labels and event labels of eye-tracking time-series samples, an event feature vector of eye-tracking time-series samples is constructed;

[0121] Based on the event feature vector and pupil feature vector of eye movement time series samples, a cognitive load deep learning model is trained through a regularization strategy, including: constructing an eye movement time series sample set based on the event feature vector and pupil feature vector of eye movement time series samples;

[0122] The learning parameters of the cognitive load deep learning model are updated by fitting the eye-tracking time-series sample set until the classification cross-entropy function result meets the preset iteration termination condition or preset number of iterations, thus obtaining the trained cognitive load deep learning model.

[0123] The structure of the cognitive load deep learning model includes an input layer, a bidirectional LSTM layer, a batch normalization layer, a fully connected layer, a Dropout layer, and an output layer; the bidirectional LSTM layer includes a first LSTM layer and a second LSTM layer, and the batch normalization layer includes a first batch normalization layer and a second batch normalization layer;

[0124] When updating the learning parameters of the cognitive load deep learning model using the eye movement time series sample set, the event feature vector and pupil feature vector are input into the first LSTM layer in time series using the input layer;

[0125] Based on the event feature vector and pupil feature vector, the temporal dependency is captured through the first LSTM layer to obtain the time step features;

[0126] Based on the time step features, the time step features are standardized by the first batch of normalization layers.

[0127] Based on the standardized time step features, feature compression is performed through the second LSTM layer to obtain the temporal feature vector;

[0128] The time series feature vectors are standardized using the second batch of normalization layers to obtain stable time series feature vectors.

[0129] Based on temporal feature vectors, nonlinear mapping is performed through fully connected layers, and generalization is performed using Dropout layers to obtain temporal mapping features.

[0130] The cognitive load level is obtained by mapping based on the temporal mapping features using the output layer.

[0131] The specific bidirectional LSTM model structure in this embodiment is as follows: Figure 1 As shown, where, For the current time step The input vector, For time steps The hidden state of the output. For time steps The updated status is similar. and For time steps And the cell state after 2t update, For time steps The activated state variables are similar. and For time step 1 And the state variables after 2t activation, The sigmoid activation function has an output range of [0,1] and controls the proportion of information that passes through. The tanh layer is a hyperbolic tangent activation function layer with an output range of [-1,1], which generates candidate information. For element-wise multiplication, the ReLU layer is the activation layer; forget gate and Used to determine the cell state at the previous moment. How much information needs to be retained and how much needs to be discarded?

[0132] The bidirectional LSTM model structure specifically includes a forget gate, an input gate, a state update gate, and an output gate;

[0133] like Figure 1 As shown, the specific process includes: according to the time step Hidden state of output and input vector After concatenation, the data is multiplied element-wise through ReLU and tanh layers. Specifically, a sigmoid activation function is used to determine high-value state information. Then, based on the high-value state information, candidate updated state content is generated through the tanh layer. Status content Through the Gate of Oblivion and Addition and renewal of cell state yields time step Updated status ;Use the output gate according to the time step Updated status Decision time step Hidden state This refers to the time step feature.

[0134] In this embodiment, the first LSTM layer of the cognitive load deep learning model is set as Bi-LSTM layer 1, the second LSTM layer is set as Bi-LSTM layer 2, the first batch normalization layer is labeled as batch normalization layer 1, the second batch normalization layer is labeled as batch normalization layer 2, and the fully connected layer and Dropout layer are set as fully connected layer 1, Dropout layer 1, fully connected layer 2, and Dropout layer 2.

[0135] Input layer: Receives features (including pupil, gaze, saccade, blink, etc. feature vectors) with a length of 100 time steps and 18 dimensions for each time step as temporal input;

[0136] Bi-LSTM layer 1: Sets 64 hidden units and adopts a bidirectional structure (forward and backward LSTM) to capture the temporal dependencies of eye-tracking sequences from front to back and from back to front; returns the complete time step output, with each time step outputting 128 dimensions. Because of the bidirectional concatenation, it is used for subsequent layers to continue extracting temporal features.

[0137] Batch Normalization Layer 1: Standardizes the features at each time step output of Bi-LSTM Layer 1 to zero mean and unit variance, accelerating training convergence and improving network stability.

[0138] Bi-LSTM layer 2: Sets 32 hidden units, also a bidirectional structure, but only returns the output of the last time step, with a dimension of 64, further refining high-level temporal features and compressing sequence information into a fixed-length vector;

[0139] Batch Normalization Layer 2: Normalizes the 64-dimensional features output by Bi-LSTM Layer 2 to stabilize the distribution of hidden states and reduce internal covariate shift.

[0140] Fully connected layer 1: Contains 32 neurons, uses the ReLU activation function to perform non-linear mapping on the batch-normalized features, and further extracts discriminative patterns related to cognitive load;

[0141] Dropout layer 1: The dropout probability is set to 0.4, randomly dropping 40% of the neuron connections during training to prevent overfitting and enhance the model's generalization ability;

[0142] Fully connected layer 2: Contains 16 neurons and uses the ReLU activation function to further compress the feature dimension and extract a more compact representation;

[0143] Dropout layer 2: Also uses a dropout probability of 0.4 to further enhance generalization and reduce dependence on specific feature combinations;

[0144] Output layer: Contains 3 neurons, using the Softmax activation function to map the features output from fully connected layer 2 to probability distributions of three cognitive loads (low, medium, and high), ultimately outputting the classification result. Specific parameters are shown in Table 1.

[0145] Table 1. Parameter Configuration of Cognitive Load Deep Learning Model

[0146]

[0147] The model training process uses the Adam optimizer with an initial learning rate of 0.001 and the loss function is the classification cross-entropy function. Regularization strategies include: early stopping (terminating the model if the validation loss shows no improvement after 15 consecutive rounds and restoring the optimal weights); and learning rate decay (halving the learning rate if there is no improvement after 10 consecutive rounds, down to a minimum of 1×10⁻⁻⁶).6 L2 weight decay, coefficient 0.01; Dropout layer, dropout probability 0.4 after fully connected layer; batch normalization. Batch size 32 during training, maximum training epochs 100.

[0148] The feature vector acquisition method in this embodiment shows significant performance advantages over existing technologies for training deep learning models with cognitive load. As shown in Table 2, Bi-LSTM achieves an accuracy of 66.49% and an AUC-ROC of 91.93%, significantly outperforming traditional models. The models compared include Linear Kernel Support Vector Machine (SVM-linear), Gaussian Kernel Support Vector Machine (SVM-rbf), Polynomial Kernel Support Vector Machine (SVM-poly), Random Forest (RF), Extreme Gradient Boosting (XGBoost), LSTM-64 with 64 hidden units, and the baseline LSTM model without regularization optimization. XGB-200 represents a decision tree count of 200; similarly, XGB-300 and XGB-100 represent decision tree counts of 300 and 100, respectively. RF-100, RF-200, and RF-300 represent random forest models with 100, 200, and 300 decision trees, respectively. Figure 2 The image shows a comparison of the training results of the LSTM model in this embodiment with other models. It is clear that the accuracy, precision, and recall of the LSTM model in this embodiment are superior to other existing models.

[0149] Table 2 Comparison of the feature vector acquisition method in this embodiment with existing technologies

[0150]

[0151] AUC-ROC is the area under the receiver operating characteristic curve.

[0152] Based on the trained cognitive load deep learning model, the SHAP method was used to identify the key characteristics of the cognitive load deep learning model for cognitive load calculation, including...

[0153] The background pool of eye-tracking time-series samples is constructed based on the event feature vector and pupil feature vector of the eye-tracking time-series samples, and the cognitive load features of the input cognitive load deep learning model are confirmed.

[0154] The background pool of eye-tracking time-series samples includes multiple sets of cognitive load features;

[0155] Based on the background pool, the cognitive load level is predicted by the trained cognitive load deep learning model to obtain the baseline cognitive load prediction value.

[0156] Step S1: Randomly select one cognitive load feature as the cognitive load feature to be tested; or multiple cognitive load features can be randomly selected together for contribution value calculation. If the contribution value has too large an impact, the contribution value of multiple cognitive load features can be calculated separately to reduce the amount of calculation.

[0157] Step S2: Construct a test feature dataset based on the acquired background pool features of the cognitive load to be tested;

[0158] Step S3: Randomly replace the cognitive load features to be tested in the background pool with data from the dataset of features to be tested to obtain the background pool of cognitive load features to be tested.

[0159] Step S4: Based on the background pool of cognitive load features to be tested, the cognitive load level is predicted by the trained cognitive load deep learning model to obtain the predicted value of the cognitive load to be tested.

[0160] Step S5: Calculate the contribution value of the current cognitive load to be measured based on the predicted value of the cognitive load to be measured and the baseline predicted value of the cognitive load.

[0161] Repeat steps S1 to S5 until all cognitive load features have corresponding contribution values; sort and normalize the cognitive load features according to the contribution values, and confirm the key features for cognitive load calculation by the cognitive load deep learning model through preset key feature constraints.

[0162] The calculation expression for the contribution value of the current cognitive load, which is calculated based on the predicted probability of the cognitive load level to be measured and the predicted probability of the baseline cognitive load level, is as follows:

[0163]

[0164] in, Let F be the contribution value of cognitive load feature i, and F be the background pool. S is the background pool of cognitive load features obtained by randomly replacing the cognitive load features in the background pool using the dataset of features to be tested. The predicted value of the cognitive load to be measured. The baseline cognitive load forecast value, For factorial, This indicates the number of features in the current set.

[0165] The specific implementation method in this embodiment is as follows:

[0166] A small subset of samples is randomly selected from the training set, with 100 cognitive load events each for low, medium, and high cognitive load events. Here, a cognitive load event refers to a complete small experiment, with low, medium, and high representing the difficulty of the experiment, forming the background pool. When we need to remove a feature, we don't actually delete the data; instead, we fill it with the value of that feature from the background dataset. Corresponding to the calculation premises of f(S) and f(S∪{i}), the sampling basis for marginalization is defined as "what value a feature takes when it doesn't exist."

[0167] For the specific feature x to be explained, multiple binary mask vectors are randomly generated to represent the number of cognitive load features to be measured. In this embodiment, the number of cognitive load features is 18, so the length of the binary mask vector is equal to 18. For example, the mask [1, 0, 1, ...] indicates that the first feature exists, the second feature is removed (i.e., filled with background values), and the third feature exists. Similarly, if it is 0, it means the feature is removed; if it is 1, it means the corresponding cognitive load feature is used normally. Traversing multiple masks means using random sampling to repeatedly calculate the contribution value to determine which cognitive load features have higher contribution values ​​and will affect the cognitive load calculation.

[0168] Based on the number of 1s in the mask Total characteristic number Calculate a weight. This corresponds to the fractional coefficient at the beginning of the formula. .

[0169] Based on the difference between the predicted cognitive load under each mask and the baseline predicted cognitive load, and the weights, the contribution values ​​corresponding to the 18 cognitive load features are obtained by using weighted least squares and calculating the global average.

[0170] Cognitive load characteristics are sorted and normalized based on their contribution values. The values ​​are sorted from largest to smallest and divided by the largest value, mapping to a range of 0-1. For example... Figure 3 As shown, this represents the contribution values ​​(i.e., importance) of cognitive load features input into the cognitive load deep learning model in this embodiment. Among the calculated results, the mean fixation duration scored the highest, demonstrating a significant positive contribution to high cognitive load prediction; blink frequency ranked second. Furthermore, features such as scan path length and mean pupil diameter also exhibited strong discriminative capabilities. By calculating the importance results multiple times through the trained cognitive load deep learning model, relatively accurate key features can be determined, ensuring the stability and reliability of the determined key features. This embodiment can identify which features influence cognitive load calculation by setting a preset key feature constraint.

[0171] The process involves acquiring eye-tracking data of the target object corresponding to key features; then, by inversely inferring the key eye-tracking data required for cognitive load calculation using these key features, the difficulty of acquiring eye-tracking data is reduced, and the efficiency of constructing key features based on the eye-tracking data is improved. Finally, the eye-tracking data of the target object is input into a trained cognitive load deep learning model to obtain the cognitive load result.

[0172] In summary, the cognitive load detection method based on eye-tracking data proposed in this embodiment establishes a standardized data cleaning and eye-tracking event classification process. It employs a velocity threshold recognition algorithm to distinguish between fixation and saccades, and constructs time-series samples based on a sliding window. Statistical features covering four dimensions—pupil, fixation, saccades, and blinking—are extracted. A bidirectional LSTM model is constructed, and strategies such as early stopping, learning rate decay, L2 regularization, and Dropout are comprehensively used to effectively suppress overfitting. SHAP interpretability analysis identifies the mean pupil diameter and fixation dispersion as the core discriminant features for cognitive load classification. The results show that compared to traditional machine learning models such as support vector machines and random forests, the bidirectional LSTM model proposed in this embodiment has higher classification accuracy and AUC-ROC value, providing reliable technical support and theoretical basis for real-time cognitive load detection and the design of adaptive human-computer interaction systems.

[0173] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0174] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0175] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0176] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0177] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims. All of these forms are within the protection scope of the present invention.

Claims

1. A method for detecting cognitive load based on eye-tracking data, characterized in that, include: Acquire eye-tracking data of the target object; The eye-tracking data of the target to be tested is input into the trained cognitive load deep learning model to obtain the cognitive load detection result; The training method for the cognitive load deep learning model includes: Based on the acquired eye-tracking data with cognitive load labels, eye-tracking time-series samples with cognitive load labels are constructed through standardization processing and a sliding window of preset length, and blink labels are constructed for the eye-tracking time-series samples. Construct pupil feature vectors for the eye movement time series samples based on the aforementioned eye movement time series samples; An eye-tracking time series sample was segmented into events using a velocity threshold recognition algorithm to obtain event labels for the eye-tracking time series sample. Based on blink labels and event labels of eye-tracking time-series samples, an event feature vector of eye-tracking time-series samples is constructed; Based on the event feature vector and pupil feature vector of eye movement time sequence samples, a cognitive load deep learning model is trained through a regularization strategy. The method for acquiring the eye-tracking data of the target to be inspected includes: Based on the trained cognitive load deep learning model, the key features of the cognitive load deep learning model for cognitive load calculation were identified by the SHAP method. Obtain the eye movement data of the target to be inspected corresponding to the key features.

2. The cognitive load detection method based on eye-tracking data according to claim 1, characterized in that, The eye-tracking data includes single-frame validity data, single-frame pupil diameter data, gaze origin position, gaze direction, and binocular position; The process involves constructing eye-tracking time-series samples with cognitive load labels based on the acquired eye-tracking data, through standardization and a sliding window of preset length, and constructing blink labels for the eye-tracking time-series samples, including: Based on the single-frame validity data, it is determined whether the preset blink duration constraint is met. If it is met, the corresponding eye movement data is marked as a blinking action; otherwise, the corresponding eye movement data is marked as missing data. The eye-tracking data marked as missing data were filled using linear interpolation to obtain complete eye-tracking data. Based on the completed eye-tracking data, an eye-tracking temporal sequence is obtained through standardization processing; Based on the eye-tracking time sequence, eye-tracking time samples are constructed using a sliding window of a preset length; Blink labels for eye movement time series samples are constructed based on eye movement time series samples and eye movement data labeled as blinking actions.

3. The cognitive load detection method based on eye-tracking data according to claim 1, characterized in that, The eye-tracking time-series samples include multiple sets of single-frame validity data and multiple sets of single-frame pupil diameter data; The construction of the pupil feature vector of the eye movement time series sample based on the eye movement time series sample includes: Based on the eye-tracking time-series samples, the fused pupil diameter is obtained by averaging the single-frame pupil diameter data corresponding to the single-frame valid data. The average pupil diameter of the eye movement time series samples is obtained by summing and averaging the fused pupil diameters. The standard deviation of the pupil diameter is calculated based on the mean pupil diameter and the fused pupil diameter. The dynamic change rate of adjacent valid frames is calculated based on the fused pupil diameter; Based on the dynamic rate of change, the pupil change rate is calculated using the arithmetic mean. A pupil feature vector is constructed based on the mean pupil diameter, standard deviation of pupil diameter, and pupil change rate.

4. The cognitive load detection method based on eye-tracking data according to claim 3, characterized in that, The formula for calculating the mean pupil diameter of the eye movement time series samples is as follows: ； in, The average pupil diameter. Let t be the fused pupil diameter, N be the number of valid sampling points within the window, and W be the time set of the data within the window; The standard deviation of the pupil diameter is calculated as follows: ； in, The standard deviation of pupil diameter; The expression for calculating the pupil change rate is: ； ； in, for The rate of dynamic change at any given moment. for The fused pupil diameter at any given moment for The fused pupil diameter at time step, where R is the set of real numbers representing the number of valid sampling points within the window. This represents the pupillary change rate.

5. The cognitive load detection method based on eye-tracking data according to claim 1, characterized in that, The process of using a velocity threshold recognition algorithm to segment eye-tracking time-series samples to obtain event labels for the eye-tracking time-series samples includes: Based on the aforementioned eye-tracking time-series samples, event labels for the eye-tracking time-series samples are constructed by filtering through fixation constraints and saccade constraints. Both the gaze constraint and the saccade constraint include an angular velocity threshold and a duration threshold.

6. The cognitive load detection method based on eye-tracking data according to claim 1, characterized in that, The event feature vector includes saccade feature vector, gaze feature vector, and blink feature vector; The saccade feature vector includes the average saccade amplitude, the average saccade speed, and the number of saccades. The gaze feature vector includes average gaze duration, number of gazes, and gaze dispersion. The blink feature vector includes blink frequency.

7. The cognitive load detection method based on eye-tracking data according to claim 1, characterized in that, The loss function during the training of the cognitive load deep learning model is the classification cross-entropy function; The event feature vector and pupil feature vector based on eye movement time-series samples are used to train a cognitive load deep learning model through a regularization strategy, including: An eye movement time series sample set is constructed based on the event feature vector and pupil feature vector of the aforementioned eye movement time series samples; The learning parameters of the cognitive load deep learning model are updated by fitting the eye-tracking time-series sample set until the classification cross-entropy function result satisfies the preset iteration termination condition or the preset number of iterations, thus obtaining the trained cognitive load deep learning model.

8. The cognitive load detection method based on eye-tracking data according to claim 7, characterized in that, The structure of the cognitive load deep learning model includes an input layer, a bidirectional LSTM layer, a batch normalization layer, a fully connected layer, a Dropout layer, and an output layer; the bidirectional LSTM layer includes a first LSTM layer and a second LSTM layer, and the batch normalization layer includes a first batch normalization layer and a second batch normalization layer; When updating the learning parameters of the cognitive load deep learning model using the eye movement time series sample set, the event feature vector and pupil feature vector are input into the first LSTM layer in time series using the input layer; Based on the event feature vector and pupil feature vector, the temporal dependency is captured through the first LSTM layer to obtain the time step features; Based on the time step features, the time step features are standardized by the first batch of normalization layers. Based on the standardized time step features, feature compression is performed through the second LSTM layer to obtain the temporal feature vector; The time series feature vectors are standardized using the second batch of normalization layers to obtain stable time series feature vectors. Based on the stable temporal feature vector, a nonlinear mapping is performed through a fully connected layer, and a Dropout layer is used for generalization to obtain the temporal mapping features. The cognitive load level is obtained by mapping based on the temporal mapping features using the output layer.

9. The cognitive load detection method based on eye-tracking data according to claim 1, characterized in that, The background pool of eye-tracking time-series samples is constructed based on the event feature vector and pupil feature vector of the eye-tracking time-series samples, and the cognitive load features of the input cognitive load deep learning model are confirmed. The background pool of the eye-tracking time-series samples includes multiple sets of cognitive load features; Based on the background pool, the cognitive load level is predicted by the trained cognitive load deep learning model to obtain the baseline cognitive load prediction value. The trained cognitive load deep learning model, through the SHAP method, identifies key characteristics of cognitive load calculation using this model, including: Step S1: Randomly select a cognitive load feature as the cognitive load feature to be tested; Step S2: Construct a test feature dataset based on the acquired background pool features of the cognitive load to be tested; Step S3: Randomly replace the cognitive load features to be tested in the background pool with data from the dataset of features to be tested to obtain the background pool of cognitive load features to be tested. Step S4: Based on the background pool of the cognitive load features to be tested, the cognitive load level is predicted by the trained cognitive load deep learning model to obtain the predicted value of the cognitive load to be tested. Step S5: Calculate the contribution value of the current cognitive load to be measured based on the predicted value of the cognitive load to be measured and the baseline predicted value of the cognitive load. Repeat steps S1 to S5 until all cognitive load features have corresponding contribution values; sort and normalize the cognitive load features according to the contribution values, and confirm the key features for cognitive load calculation by the cognitive load deep learning model through preset key feature constraints.

10. The cognitive load detection method based on eye-tracking data according to claim 9, characterized in that, The calculation expression for the contribution value of the current cognitive load to be measured, calculated based on the predicted value of the cognitive load to be measured and the baseline predicted value of the cognitive load, is as follows: in, Let F be the contribution value of cognitive load feature i, and F be the background pool. This represents the background pool of cognitive load features obtained by randomly replacing the cognitive load features in the background pool using the dataset of features to be tested. The background pool for the cognitive load characteristics to be measured. The predicted value of the cognitive load to be measured. The baseline cognitive load forecast value, To represent factorial, This indicates the number of features in the current set.

Images