Method and apparatus for measuring cognitive load of interaction task

Abstract

This application provides a method and apparatus for measuring cognitive load in interactive tasks, relating to the field of cognitive load prediction technology. The method for measuring cognitive load in interactive tasks includes: extracting features from the collected raw interactive data using a sliding window approach to determine pupil features and behavioral manipulation complexity features, wherein pupil features are vectors representing visual cognitive load, and behavioral manipulation complexity features are vectors reflecting the complexity of manipulation behavior; and using a pre-trained cognitive load prediction model to determine the predicted cognitive load value based on the pupil features and behavioral manipulation complexity features. This application combines pupil features and manipulation complexity indicators to characterize cognitive load from both visual and motor dimensions. This indicator comprehensively quantifies the degree of action intersection using factors such as frequency and entropy value, effectively distinguishing the impact of different operations. By using a prediction model to fuse features, it significantly improves the comprehensiveness and accuracy of the assessment, thereby enhancing the measurement effect of cognitive load.

Description

Technical Field

[0001] This application relates to the field of cognitive load prediction technology, and for example to a method and apparatus for measuring cognitive load in an interactive task. Background Technology

[0002] Accurate measurement of cognitive load is crucial for optimizing human-computer interaction design, improving operational performance, and ensuring job safety. In manipulation-intensive human-computer interaction scenarios, such as e-sports, aircraft piloting, and complex industrial equipment control, operators not only need to continuously acquire information in a highly dynamic visual environment but also must complete diverse and frequent manual manipulation actions within a very short time. The cognitive resource demands of such tasks far exceed those of conventional visual viewing or simple operational tasks. Therefore, how to objectively and in real-time measure cognitive load in manipulation-intensive interaction tasks has become a research hotspot in the field of ergonomics. Currently, cognitive load measurement methods based on physiological signals have received widespread attention due to their objectivity and continuity. Among them, pupil size, as a sensitive indicator reflecting central nervous system activity, has been extensively studied and confirmed to be positively correlated with cognitive load within a certain range. However, most existing patented technologies are limited to the application of a single pupil indicator, and their applicable scenarios are mainly tasks involving visual information processing. These technical solutions are commonly used in scenarios such as online learning, video viewing, and driving simulation. Although the above scenarios involve visual cognitive processing, the operator does not need to perform high-density, high-complexity manual manipulation. Their cognitive load mainly stems from the visual input and comprehension process, rather than the execution and coordination of manipulative behaviors. Therefore, relying solely on pupillary indicators is insufficient to fully reflect the cognitive resource consumption caused by diverse manual operations in manipulation-intensive tasks.

[0003] To overcome the limitations of single physiological indicators, related technologies attempt to combine pupil data with other indicators to improve measurement accuracy. For example, physiological parameters such as heart rate and blood oxygen saturation, blink frequency, and behavioral performance indicators have been introduced. While these technologies enrich the dimensions of cognitive load assessment to some extent, they are generally not suitable for interactive scenarios such as esports and complex equipment operation, which combine high-intensity visual tracking with high-frequency manual responses. This results in poor measurement performance of current technologies in scenarios involving intensive interactive operations. Summary of the Invention

[0004] This application aims to provide a method and apparatus for measuring cognitive load in interactive tasks.

[0005] According to one aspect of this application, a method for measuring cognitive load of interactive tasks is proposed, comprising: extracting features from the collected raw interactive data using a sliding window approach to determine pupil features and behavioral manipulation complexity features, wherein the pupil features are vectors used to characterize visual cognitive load, and the behavioral manipulation complexity features are vectors used to reflect the complexity of manipulation behavior; and determining the predicted value of cognitive load based on the pupil features and behavioral manipulation complexity features using a pre-trained cognitive load prediction model.

[0006] According to one aspect of this application, a device for measuring the cognitive load of an interactive task is proposed, comprising: The data extraction module is used to extract features from the collected raw interaction data based on a sliding window method to determine pupil features and behavioral manipulation complexity features. The pupil features are vectors used to characterize visual cognitive load, and the behavioral manipulation complexity features are vectors used to reflect the complexity of manipulation behavior. The prediction module is used to determine the predicted value of cognitive load based on pupil features and behavioral manipulation complexity features using a pre-trained cognitive load prediction model.

[0007] According to one aspect of this application, an electronic device is provided, comprising: a processor; and a memory storing a computer program that, when executed by the processor, causes the processor to perform the method described above.

[0008] According to one aspect of this application, a non-transitory computer-readable medium is proposed, on which readable instructions are stored, which, when executed by a processor, cause the processor to perform the method described above.

[0009] It should be understood that the above general description and the following detailed description are merely exemplary and do not limit this application.

[0010] Beneficial effects: The embodiments provided in this application combine pupil features with manipulation complexity features to characterize cognitive load from both visual processing and action execution dimensions. This overcomes the limitations of related technologies that rely solely on a single visual indicator or fail to quantify the complexity of the manipulation behavior itself, filling the gap in measurement dimensions for manipulation-intensive scenarios. Addressing the characteristics of high-frequency, diverse concurrent manipulation, this application also innovatively proposes a manipulation complexity index. This index comprehensively considers operation frequency, number of types, entropy value, and number of switching operations to quantify the degree of cross-combination of manipulation actions per unit time, effectively distinguishing the different impacts of high-frequency repetition and complex interleaving on cognitive load, thus overcoming the deficiency of traditional performance indicators that only focus on the correctness of results. Through a pre-trained cognitive load prediction model, the predicted value of cognitive load is determined based on pupil features and behavioral manipulation complexity features, significantly improving the comprehensiveness and accuracy of cognitive load assessment. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings, without exceeding the scope of protection claimed by this application.

[0012] Figure 1 A flowchart illustrating a method for measuring the cognitive load of an interactive task provided in an embodiment of this application; Figure 2 Timing diagram of the training phase provided in the embodiments of this application; Figure 3 A timing diagram of the real-time measurement stage provided in the embodiments of this application; Figure 4 A block diagram of a device for measuring the cognitive load of an interactive task provided in an embodiment of this application; Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0013] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the embodiments set forth herein; rather, they are provided so that this application will be thorough and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted.

[0014] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a thorough understanding of embodiments of this application. However, those skilled in the art will recognize that the technical solutions of this application can be practiced without one or more of the specific details, or other methods, components, apparatuses, steps, etc., can be employed. In other instances, well-known methods, apparatuses, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of this application.

[0015] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0016] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0017] It should be understood that although the terms first, second, third, etc., may be used herein to describe various components, these components should not be limited by these terms. These terms are used to distinguish one component from another. Therefore, the first component discussed below may be referred to as the second component without departing from the teachings of this application. As used herein, the term "and / or" includes all combinations of any one and more of the associated listed items.

[0018] For specific implementation details, please refer to the following examples.

[0019] Figure 1 A flowchart illustrating a method for measuring the cognitive load of an interactive task provided in an embodiment of this application. Figure 1 As shown, the method includes steps S10, S11, S12 and S13.

[0020] In step S10, feature extraction is performed on the collected raw interaction data based on a sliding window method to determine pupil features and behavioral manipulation complexity features. The pupil features are vectors used to characterize visual cognitive load, and the behavioral manipulation complexity features are vectors used to reflect the complexity of manipulation behavior.

[0021] In this application, a data acquisition device can be set on the interactive device to collect the raw interaction data corresponding to the user's human-computer interaction operations on the interactive device. Since the time from the start of the interaction to its completion may be relatively long, the interaction process can be divided into multiple execution segments (corresponding to subtasks) using a sliding window approach. That is, the execution process of the task can be divided into multiple execution segments (corresponding to subtasks). The execution process of these execution segments is continuous. The task corresponding to each execution segment may be a manipulation-intensive subtask or a simple interaction task. However, in the interaction task targeted by this application, there is at least one manipulation-intensive subtask.

[0022] In some implementations, feature extraction methods can be pre-defined, such as pre-setting the requirements and characteristics of the features to be extracted, thereby extracting pupil features and behavioral manipulation complexity features from the original interaction data. Pupil features are used to represent visual cognitive load and are presented in vector data form, while behavioral manipulation complexity features are used to reflect the complexity of manipulation behavior and are also presented in vector data form.

[0023] In step S11, a pre-trained cognitive load prediction model is used to determine the predicted cognitive load value based on pupil features and behavioral manipulation complexity features.

[0024] In this application, a cognitive load prediction model can be pre-trained, which can employ a multi-layer fully connected neural network. In some implementations, the network structure of this application may include an input layer, hidden layers, and an output layer. The loss function in this application can use mean squared error to measure the difference between the predicted value and the true score, and uses Adam as the optimizer with an exponential decay strategy.

[0025] The pupil features and behavioral manipulation complexity features obtained in step S20 are input into the trained cognitive load prediction model, and the cognitive load prediction value for the current time window is output.

[0026] This application combines pupil features with manipulation complexity features to characterize cognitive load from both visual processing and action execution dimensions. This overcomes the limitations of related technologies that rely solely on single visual indicators or fail to quantify the complexity of the manipulation behavior itself, filling the gap in measurement dimensions for manipulation-intensive scenarios. Addressing the characteristics of high-frequency, diverse concurrent manipulation, this application also innovatively proposes a manipulation complexity index. This index comprehensively considers operation frequency, number of types, entropy value, and number of switching operations to quantify the degree of cross-combination of manipulation actions per unit time, effectively distinguishing the different impacts of high-frequency repetition and complex interleaving on cognitive load, thus overcoming the deficiency of traditional performance indicators that only focus on the correctness of results. Through a pre-trained cognitive load prediction model, the predicted value of cognitive load is determined based on pupil features and behavioral manipulation complexity features, significantly improving the comprehensiveness and accuracy of cognitive load assessment.

[0027] According to some embodiments, pupil data collected and sent by an eye tracker and control behavior data input through an input device can be received; original interaction data can be determined based on pupil data, control behavior data and a pre-set pupil diameter threshold; the original interaction data can be sliced ​​based on a sliding window method to extract pupil features corresponding to the pupil data and behavior control complexity features corresponding to the control behavior data.

[0028] In this application, a desktop or head-mounted eye tracker can be used to collect pupil data, with a sampling frequency of no less than 120Hz. Before the formal task begins, pupil diameter data of the subject (user) in a resting state is first collected. In some implementations, the subject gazes at a blank screen or a fixed crosshair, and data is collected continuously for 30 seconds. The average pupil diameter during this period is calculated and named the pupil diameter threshold. This value is used as the individual baseline. During task execution, the eye tracker synchronously collects the pupil diameter of both eyes of the subject in real time as pupil data. The input devices in this application may include keyboards, mice, etc. The interactive device may have a self-defined control event recording software pre-installed to capture input events from the input device. In some implementations, the underlying interface of the operating system can be used to record events such as keyboard key presses, mouse clicks, and mouse drags. Each record includes a timestamp, device type, operation type, and operation value. This data serves as control behavior data.

[0029] In some implementations, the average value of the left and right eyes can be taken from the pupil data as the pupil diameter at the current moment. And calculate the pupil difference. To eliminate interference from factors such as lighting and distance, the eye tracker must be used under constant ambient lighting and with the head in a relatively fixed position. Pupil data, pupil difference, and manipulation behavior data can be used together as raw interaction data.

[0030] For pupil data and manipulation behavior data, feature extraction methods can be pre-set, and then targeted extraction can be performed to obtain pupil features corresponding to pupil data and manipulation complexity features corresponding to manipulation behavior data.

[0031] This application effectively integrates and filters heterogeneous data from visual cognition and manual manipulation by receiving pupil data collected by an eye tracker and manipulation behavior data collected by manipulation event recording software, combined with a preset pupil diameter threshold. This avoids interference from invalid or low-quality data and improves the accuracy and robustness of feature extraction. By slicing the original interaction data using a sliding window approach, fine-grained segmentation of the continuous interaction process in the time dimension is achieved, enabling dynamic and real-time feature extraction and effectively capturing the temporal characteristics of cognitive load changes with task progress. By extracting pupil features and behavioral manipulation complexity features separately, this application simultaneously characterizes the cognitive resource consumption of visual processing and action execution within a unified time window, achieving collaborative perception of multi-dimensional cognitive load. This overcomes the shortcomings of traditional methods that rely solely on single-modal data or ignore the dynamic complexity of manipulation behavior, providing more refined and realistic data support for cognitive state assessment under manipulation-intensive tasks.

[0032] According to some embodiments, for a fixed time window corresponding to the sliding window method, the mean, maximum, standard deviation, and minimum pupil difference values ​​can be determined based on pupil data and pupil diameter thresholds. Pupil characteristics can then be determined based on these values. Furthermore, based on manipulation behavior data, the operation frequency, number of operation types, and operation variability can be determined for the corresponding time window. Finally, the behavioral manipulation complexity characteristics can be determined based on these factors.

[0033] In this application, regarding pupil features, the pupil difference sequence within the window is statistically analyzed, and the mean of the differences is extracted. Standard deviation Indices such as maximum and minimum values ​​constitute a feature vector representing visual cognitive load. Among them, pupil difference... These parameters form the pupil feature vector. In some implementations, more parameters may be involved than just the mean pupil difference, maximum pupil difference, standard deviation of pupil difference, and minimum pupil difference.

[0034] Manipulation complexity aims to quantify the degree of cross-combination of multiple manipulation actions per unit of time, rather than simply the frequency of operations. Specifically, it can be extracted from three dimensions: first, operation frequency, i.e., the total number of manipulation events within the window; second, operation diversity, i.e., the number of different operation types appearing within the window; and third, operation variability, used to distinguish between two different states: "high-frequency repetition of the same operation" and "frequent interleaving of multiple operations," which can be represented by entropy.

[0035] This application constructs pupil features by calculating the mean, maximum, standard deviation, and minimum values ​​of pupil difference within a fixed time window. This accurately captures the dynamic fluctuations in pupil diameter relative to the baseline, effectively reflecting the instantaneous load changes and fluctuation intensity during visual cognitive processing. Compared to simply using absolute pupil diameter, it significantly reduces the interference of external factors such as ambient light, improving the robustness and sensitivity of visual load indicators. Simultaneously, this application constructs behavioral manipulation complexity features by extracting operation frequency, number of operation types, and operation variability, achieving multi-level quantification of manipulation behavior from quantity to type to randomness. The combination of these two aspects allows the model to synchronously and precisely represent the dual cognitive load of visual perception and action execution at a unified temporal granularity, providing highly discriminative feature inputs for cognitive state assessment in complex interactive scenarios.

[0036] According to some embodiments, the total number of manipulation events under a corresponding time window can be determined from the manipulation behavior data to determine the operation frequency; the default operation type obtained by fine-grained segmentation based on task characteristics can be obtained; the number of default operation types under a corresponding time window can be determined from the manipulation behavior data to determine the number of operation types; the operation frequency of different operation types corresponding to the number of operation types can be determined from the manipulation behavior data, and based on the operation frequency, the entropy value of the operation sequence corresponding to different operation types can be determined to determine the entropy value as the operation variability; and the behavior manipulation complexity characteristics can be determined based on the operation frequency, the number of operation types, and the operation variability.

[0037] In this application, the total number of manipulation events is first detected from the manipulation behavior data, such as the total number of left and right mouse clicks, drags, etc., as the operation frequency. Based on task characteristics, different manipulation events can be finely categorized. For example, a mouse may include left, right, and middle buttons, and clicking can be either double-clicking or single-clicking. Therefore, mouse manipulation events can be categorized into more specific types such as left-click, left-double-click, right-click, and right-double-click, as the default operation type. Different keyboard keys and dragging are considered independent types. By combining these dimensions, a set of feature vectors reflecting the complexity of the manipulation behavior can be generated.

[0038] For the default operation types identified above, labels are created, and then different default operation types are determined from the control behavior data. The number of corresponding operation types and the operation frequency of each operation type among all operation types are recorded.

[0039] To avoid misclassifying high complexity as repeated key presses, the entropy H of the operation sequence is introduced, which can be measured using Shannon entropy. ,in For the first The frequency of class operations occurring within the window. Using entropy as the operation variability. and operation frequency Number of operation types We jointly construct the characteristics of behavioral manipulation complexity.

[0040] This application constructs multi-dimensional behavioral manipulation complexity features by extracting operation frequency, number of operation types, and operation variability from manipulation behavior data, achieving a refined and structured quantification of the complexity of manipulation behavior. Specifically, operation frequency is determined by the total number of manipulation events within a statistical time window, reflecting the activity level of user operations; the number of actual triggered types is statistically analyzed by using default operation types with fine-grained classification based on task characteristics, reflecting the diversity of operational behavior; furthermore, by calculating the frequency distribution of different operation types and obtaining the entropy value of their operation sequence as operation variability, the randomness and unpredictability of operational behavior are effectively characterized, and the differences in cognitive resource consumption between regular repetitive operations and highly complex interspersed operations can be distinguished. Compared with traditional methods that only rely on macro-level performance indicators such as the number of operations or task completion time, this application comprehensively models manipulation complexity from three levels: quantity (frequency), class (number of types), and order (variability / entropy value), more comprehensively reflecting the dynamic scheduling process of users' cognitive resources in manipulation-intensive tasks. This feature construction method not only improves the granularity and sensitivity of cognitive load assessment, but also enhances the model's ability to identify complex interaction patterns, providing reliable technical support for accurately predicting cognitive load in scenarios with multiple concurrent tasks and high-frequency switching.

[0041] According to some embodiments, the number of times adjacent operation types change can be determined based on manipulation behavior data; based on operation frequency, number of operation types, operation variability and number of times, behavioral manipulation complexity features are constituted.

[0042] In this application, the number of times adjacent operation types change in operation behavior data can be detected, denoted as... , with operational variability and operation frequency Number of operation types Jointly construct behavioral manipulation complexity features .

[0043] This application, building upon existing metrics such as operation frequency, number of operation types, and operation variability (entropy), further introduces the number of changes between adjacent operation types (i.e., operation switching frequency) as a key dimension, constructing a more comprehensive feature of behavioral manipulation complexity. It can accurately capture the cost of mental leaps and attention shifts during multitasking, effectively distinguishing between two distinct interaction modes: "maintaining a single operation for a long time" and "frequent switching between different operations." Even if the total number of operations and entropy values ​​are the same, the resulting cognitive load often differs significantly. By integrating the number of operation switching, this application fills the gap where statistical distribution characteristics alone cannot fully characterize the dynamic changes in operation sequences. It achieves a comprehensive portrayal of the complexity of manipulation behavior from three levels: total volume, distribution, and dynamic switching, significantly improving the interpretability and accuracy of cognitive load prediction models in complex, high-frequency interaction scenarios.

[0044] According to some embodiments, historical pupil features and historical behavioral manipulation complexity features corresponding to multiple time windows can also be obtained; the historical pupil features and historical behavioral manipulation complexity features are concatenated to generate a fused feature vector; the subjective rating labels of users for each of the multiple time windows are obtained, and the subjective rating labels are associated with the corresponding fused feature vectors to construct a training dataset; the initial prediction model is trained based on the training dataset to generate a pre-trained cognitive load prediction model, wherein the cognitive load prediction model adopts a multi-layer fully connected neural network.

[0045] In this application, historical pupil features and historical behavioral manipulation complexity features corresponding to multiple time windows can be obtained. The historical pupil features and historical behavioral manipulation complexity features of the same window are concatenated to form a fused feature vector, which is then aligned with the subjective rating label corresponding to that window to construct a training dataset. The subjective labeling of cognitive load is performed during task execution. This application can design manipulation-intensive interactive tasks (such as real-time strategy games or flight simulators), requiring participants to perform the task continuously. After each task segment is completed (e.g., every 2 minutes), the task automatically pauses, and a cognitive load self-assessment scale pops up on the screen, using a 10-level rating scale from 0 to 9 (0 representing completely easy, 9 representing extremely difficult). Participants have 3 seconds to input their rating using numeric keys; this rating serves as the subjective rating label for the cognitive load of the current segment. If no rating is input, the data for that time window is displayed.

[0046] The initial prediction model was refined using the training dataset to obtain the cognitive load prediction model.

[0047] In some implementations, due to significant differences in the dimensions of features such as pupil difference, operation frequency, and entropy, normalization is required. In some implementations, Z-score standardization can be used: for each feature dimension, the mean of that feature on the training set is calculated. and standard deviation The normalized eigenvalues ​​are The normalization parameters need to be saved, and the same transformation should be applied to the input features during real-time measurement. It should be noted that the normalization method is not limited to the one provided in this application.

[0048] This application generates a fused feature vector containing both visual and behavioral information by concatenating historical pupil features corresponding to multiple time windows with historical behavioral manipulation complexity features. This fused feature vector is then associated with the user's subjective rating labels to construct a training dataset, achieving a leap from single-modal features to multi-modal fused features, providing the model with more comprehensive and richer cognitive state representation information. A multi-layer fully connected neural network is used to train the initial prediction model, effectively learning the nonlinear mapping relationship between features of each dimension in the fused feature vector and cognitive load, automatically extracting deep feature combination patterns, thereby generating a high-precision cognitive load prediction model. This training method not only overcomes the prediction bias problem caused by traditional methods relying on manually set weights or simple linear regression, but also achieves deep fusion and collaborative evaluation of visual processing load and action execution load through end-to-end learning, significantly improving the model's prediction accuracy and generalization ability for dynamic changes in cognitive load in complex interaction scenarios.

[0049] According to some embodiments, the training dataset can be divided into a training set, a test set, and a validation set according to a preset ratio; the initial prediction model can be initially trained using an early stopping method, according to a preset batch size and a preset number of iterations; after the initial training is completed, the model that has been initially trained can be evaluated using a test set to generate performance indicators; if the performance indicators are greater than a preset threshold, the performance is determined to be up to standard, so as to generate a pre-trained cognitive load prediction model.

[0050] In this application, the collected dataset can be divided into a training set, a validation set, and a test set in a preset ratio of 8:1:1. The training set is used for model training, with a batch size of 32 and 100 iterations. Early stopping is employed, meaning training stops when the validation set loss fails to decrease for 10 consecutive iterations to prevent overfitting. After training, model performance is evaluated on the test set, using metrics including root mean square error (RMSE), mean absolute error, and coefficient of determination. If the performance meets the requirements, save the model structure and weight files.

[0051] In some implementations, the trained model can be encapsulated as a real-time cognitive load measurement engine and integrated into the target human-computer interaction system. The real-time operation process is as follows: (1) Continuously collect pupil data streams from the eye tracker, and calculate the pupil difference statistics in real time using a sliding window (e.g., width 2 seconds, step size 0.5 seconds). (2) Synchronously collect the control event stream, and calculate the control complexity features in real time using the same sliding window. (3) Normalize the fusion features of the current window to be consistent with the training set. (4) Input the normalized features into the neural network model, and obtain the cognitive load prediction value (a floating-point number between 0 and 9) through forward propagation. (5) Output the prediction value for subsequent detection or system adaptive adjustment.

[0052] The timing diagram for the training phase can be referenced. Figure 2 The timing diagram for the real-time measurement phase can be referenced. Figure 3 . This application establishes a standardized model training and evaluation process by strictly dividing the training dataset into training, test, and validation sets according to a preset ratio. This effectively avoids data leakage and evaluation bias, ensuring the objectivity and reliability of model performance evaluation. Early stopping, combined with preset batch size and iteration rounds, is used to initially train the prediction model. This allows for dynamic monitoring of the model's performance on the validation set during training, automatically terminating training when performance no longer improves. This effectively prevents overfitting on the training set and significantly improves the model's generalization ability and training efficiency. After initial training, the model is independently evaluated using the test set, generating performance metrics. The model is only considered to have met the performance standards and the final prediction model is output when the performance metrics exceed a preset threshold. This establishes a strict model admission mechanism, ensuring that the generated cognitive load prediction model possesses sufficient prediction accuracy and stability. This application not only automates and standardizes the model training process but also ensures the model's robustness in practical applications through multiple verification and screening mechanisms, providing reliable technical support for real-time cognitive load prediction in manipulation-intensive task scenarios.

[0053] The following describes an apparatus embodiment of this application, which can be used to perform the method embodiment of this application. For details not disclosed in the apparatus embodiment of this application, please refer to the method embodiment of this application.

[0054] Figure 4 A block diagram of a device for measuring the cognitive load of an interactive task provided in an embodiment of this application. Figure 4 As shown, the cognitive load measurement device 400 for interactive tasks includes a data extraction module 401 and a prediction module 402.

[0055] The data extraction module 401 is used to extract features from the collected raw interaction data based on a sliding window method to determine pupil features and behavioral manipulation complexity features. The pupil features are vectors used to characterize visual cognitive load, and the behavioral manipulation complexity features are vectors used to reflect the complexity of manipulation behavior. The prediction module 402 is used to determine the predicted value of cognitive load for the current time window based on pupil features and behavioral manipulation complexity features using a pre-trained cognitive load prediction model.

[0056] Optionally, the data extraction module 401 is specifically used for: Receive pupil data collected and transmitted by the eye tracker and control behavior data input through the input device; The original interaction data is determined based on pupil data, manipulation behavior data, and a pre-set pupil diameter threshold. The original interactive data is sliced ​​using a sliding window approach to extract pupil features corresponding to pupil data and behavioral control complexity features corresponding to control behavior data.

[0057] Optionally, when the data extraction module 401 performs slicing processing on the original interaction data based on a sliding window method to extract pupil features corresponding to pupil data and behavioral control complexity features corresponding to control behavior data, it is specifically used for: For a fixed time window corresponding to the sliding window method, based on pupil data and pupil diameter threshold, the mean, maximum, standard deviation, and minimum pupil difference values ​​under the corresponding time window are determined, and pupil characteristics are determined based on the mean, maximum, standard deviation, and minimum pupil difference values. Based on the manipulation behavior data, determine the operation frequency, number of operation types, and operation variability under the corresponding time window, and determine the behavioral manipulation complexity characteristics based on the operation frequency, number of operation types, and operation variability.

[0058] Optionally, the data extraction module 401, after determining the operation frequency, number of operation types, and operation variability within the corresponding time window based on the control behavior data, and determining the behavioral control complexity characteristics based on the operation frequency, number of operation types, and operation variability, is specifically used for: The total number of manipulation events within the corresponding time window is determined from the manipulation behavior data to determine the operation frequency; Retrieve the default operation type obtained by fine-grained division based on task characteristics; The number of default operation types within the corresponding time window is determined from the manipulation behavior data, thereby determining the total number of operation types. The frequency of operation for different operation types corresponding to the number of operation types is determined from the manipulation behavior data, and the entropy value of the operation sequence corresponding to different operation types is determined based on the operation frequency, so that the entropy value is determined as the operation variability. The characteristics of behavioral manipulation complexity are determined based on the frequency of operation, the number of operation types, and the degree of operation variability.

[0059] Optionally, when the data extraction module 401 determines the behavioral manipulation complexity characteristics based on operation frequency, number of operation types, and operation variability, it is specifically used for: Based on manipulation behavior data, determine the number of times adjacent operation types change; The complexity of behavioral manipulation is characterized by the frequency of operations, the number of operation types, the degree of operation variability, and the number of times operations are performed.

[0060] Optionally, the cognitive load measurement device 400 for the interactive task also includes a model training module 403 for: Obtain historical pupil features and historical behavior manipulation complexity features corresponding to multiple time windows; Historical pupil features and historical behavioral manipulation complexity features are concatenated to generate a fused feature vector; Obtain the subjective rating labels of users for each of the multiple time windows, and associate the subjective rating labels with the corresponding fused feature vectors to construct a training dataset; The initial prediction model is trained based on the training dataset to generate a pre-trained cognitive load prediction model, which employs a multi-layer fully connected neural network.

[0061] Optionally, when the model training module 403 trains the initial prediction model based on the training dataset to generate a pre-trained cognitive load prediction model, it is specifically used for: The training dataset is divided into a training set, a test set, and a validation set according to a preset ratio. Using the early stopping method, the initial prediction model is initially trained according to the preset batch size and preset iteration rounds; After the initial training is completed, the model is evaluated using a test set to generate performance metrics. If the performance index exceeds the preset threshold, the performance is deemed satisfactory, and a pre-trained cognitive load prediction model is generated.

[0062] The device performs functions similar to those described above; other functions are described in the preceding text and will not be repeated here.

[0063] Figure 5 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application, such as... Figure 5As shown, the electronic device 500 of this embodiment may include a memory 501 and a processor 502.

[0064] The memory 501 stores a computer program, which, when executed by the processor 502, causes the processor 502 to perform the method described in the above embodiments.

[0065] The processor 502 and the memory 501 are connected, for example, via a bus.

[0066] Optionally, the electronic device 500 may also include a transceiver. It should be noted that in practical applications, the transceiver is not limited to one, and the structure of the electronic device 500 does not constitute a limitation on the embodiments of this application.

[0067] Processor 502 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 502 may also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0068] A bus can include a pathway for transmitting information between the aforementioned components. The bus can be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, only one thick line is used in the diagram, but this does not imply that there is only one bus or one type of bus.

[0069] The memory 501 can be ROM (Read Only Memory) or other types of static storage devices capable of storing static information and instructions, RAM (Random Access Memory) or other types of dynamic storage devices capable of storing information and instructions, or it can be EEPROM (Electrically Erasable Programmable Read Only Memory), CD. ROM (Compact Disc Read Only Memory) or other optical disc storage, optical disc storage (including compressed discs, laser discs, optical discs, digital universal discs, Blu-ray discs, etc.), disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto.

[0070] The memory 501 is used to store application code that executes the solution of this application, and its execution is controlled by the processor 502. The processor 502 is used to execute the application code stored in the memory 501 to implement the content shown in the foregoing method embodiments.

[0071] Electronic devices include, but are not limited to: mobile terminals such as mobile phones, laptops, digital radio receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), and in-vehicle terminals (such as in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Servers can also be included. Figure 5 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0072] The electronic device in this embodiment can be used to execute the method of any of the above embodiments, and its implementation principle and technical effect are similar, so they will not be described again here.

[0073] This application also provides a non-transitory computer-readable storage medium storing computer-readable instructions thereon, which, when executed by a processor, cause the processor to perform the method as described in the above embodiments.

[0074] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a non-transitory computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0075] The embodiments of this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only for the purpose of helping to understand the method and core ideas of this application. Furthermore, any changes or modifications made by those skilled in the art based on the ideas of this application, and on the specific implementation methods and application scope of this application, are all within the scope of protection of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for measuring the cognitive load of an interactive task, characterized in that, include: Feature extraction is performed on the collected raw interaction data using a sliding window method to determine pupil features and behavioral manipulation complexity features. The pupil features are vectors used to characterize visual cognitive load, and the behavioral manipulation complexity features are vectors used to reflect the complexity of manipulation behavior. Using a pre-trained cognitive load prediction model, the predicted cognitive load value is determined based on the pupil features and behavioral manipulation complexity features.

2. The method of claim 1, wherein, The feature extraction of the collected raw interaction data based on the sliding window method to determine pupil features and behavioral manipulation complexity features includes: Receive pupil data collected and sent by the eye tracker and control behavior data input through the input device; The original interaction data is determined based on the pupil data, the control behavior data, and a preset pupil diameter threshold. The original interactive data is sliced ​​using a sliding window approach to extract pupil features corresponding to the pupil data and behavioral control complexity features corresponding to the control behavior data.

3. The method of claim 2, wherein, The step of slicing the original interaction data using a sliding window approach to extract pupil features corresponding to the pupil data and behavioral control complexity features corresponding to the control behavior data includes: For a fixed time window corresponding to the sliding window method, based on the pupil data and the pupil diameter threshold, the mean pupil difference, maximum pupil difference, standard deviation of pupil difference, and minimum pupil difference are determined for the corresponding time window, and the pupil feature is determined based on the mean pupil difference, maximum pupil difference, standard deviation of pupil difference, and minimum pupil difference. Based on the control behavior data, the frequency of operation, the number of operation types, and the variability of operation are determined under the corresponding time window, and the complexity characteristics of the control behavior are determined based on the frequency of operation, the number of operation types, and the variability of operation.

4. The method of claim 3, wherein, The step of determining the operation frequency, number of operation types, and operation variability within a corresponding time window based on the control behavior data, and determining the behavior control complexity characteristics based on the operation frequency, number of operation types, and operation variability, includes: The total number of control events within the corresponding time window is determined from the control behavior data to determine the operation frequency; Retrieve the default operation type obtained by fine-grained division based on task characteristics; The number of the default operation types under the corresponding time window is determined from the control behavior data to determine the number of operation types; The operation frequency of different operation types corresponding to the number of operation types is determined from the operation behavior data, and the entropy value of the operation sequence corresponding to the different operation types is determined based on the operation frequency, so as to determine the entropy value as the operation variability. The behavioral manipulation complexity features are determined based on the operation frequency, the number of operation types, and the operation variability.

5. The method according to claim 3 or 4, characterized in that, The step of determining the behavioral manipulation complexity feature based on the operation frequency, the number of operation types, and the operation variability includes: Based on the aforementioned manipulation behavior data, determine the number of times adjacent operation types change; The complexity features of the behavior manipulation are constituted based on the operation frequency, the number of operation types, the operation variability, and the number of times.

6. The method of claim 1, wherein, Also includes: Obtain historical pupil features and historical behavior manipulation complexity features corresponding to multiple time windows; The historical pupil features and the historical behavior manipulation complexity features are concatenated to generate a fused feature vector; Obtain the subjective rating labels of users for each of the multiple time windows, and associate the subjective rating labels with the corresponding fused feature vectors to construct a training dataset; The initial prediction model is trained based on the training dataset to generate the pre-trained cognitive load prediction model, wherein the cognitive load prediction model employs a multi-layer fully connected neural network.

7. The method of claim 6, wherein, The step of training the initial prediction model based on the training dataset to generate the pre-trained cognitive load prediction model includes: The training dataset is divided into a training set, a test set, and a validation set according to a preset ratio. Using the early stopping method, the initial prediction model is initially trained according to a preset batch size and a preset number of iterations; After the initial training is completed, the model that has completed the initial training is evaluated using the test set to generate performance metrics; If the performance index is greater than the preset threshold, the performance is deemed satisfactory, and the pre-trained cognitive load prediction model is generated.

8. A device for measuring cognitive load of an interactive task, characterized by, include: The data extraction module is used to extract features from the collected raw interaction data based on a sliding window method to determine pupil features and behavioral manipulation complexity features, wherein the pupil features are vectors used to characterize visual cognitive load, and the behavioral manipulation complexity features are vectors used to reflect the complexity of manipulation behavior. The prediction module is used to determine the predicted value of cognitive load based on the pupil features and behavioral manipulation complexity features using a pre-trained cognitive load prediction model.

9. An electronic device, comprising: include: processor; A memory storing a computer program that, when executed by the processor, causes the processor to perform the method as described in any one of claims 1-7.

10. A non-transitory computer-readable storage medium, comprising: It stores computer-readable instructions that, when executed by a processor, cause the processor to perform the method as described in any one of claims 1-7.

Images