A method, application method, and apparatus for determining a cognitive load prediction model.

Abstract

This application discloses a method, application method, and apparatus for determining a cognitive load prediction model, relating to the field of human-computer interaction technology. The method includes acquiring EEG signals, task difficulty of the main task, working memory capacity of the subjects, and performance data of secondary tasks from several subjects; preprocessing the EEG signals and extracting multi-dimensional EEG features; combining the secondary task performance data with task difficulty and working memory capacity, using a residual-mixed effects model to generate a comprehensive cognitive load score, and mapping it to a comprehensive cognitive load score level using a percentage method; constructing a time-series dataset based on the multi-dimensional EEG features and corresponding levels; using the multi-dimensional EEG features as input and the corresponding levels as labels, training a CNN-Bi-LSTM-Attention network using a cross-entropy loss function to obtain a trained cognitive load prediction model. This application can use this model to predict cognitive load levels.

Description

Technical Field

[0001] This application relates to the field of human-computer interaction technology, and in particular to a method, application method and apparatus for determining a cognitive load prediction model. Background Technology

[0002] In recent years, human-machine remote collaborative operation and maintenance has developed rapidly and is often applied in space stations and high-risk industrial scenarios. These scenarios are characterized by high risk, high complexity and high timeliness, which puts extremely high demands on the cognitive ability of operators. Cognitive load, as a core indicator reflecting the consumption of psychological resources, can lead to an increase in the rate of operational errors when it is unbalanced (overload or underload), seriously threatening mission safety.

[0003] Existing methods for assessing cognitive load have significant limitations: subjective scales (such as NASA-TLX) rely on retrospective reports, resulting in time lags and an inability to reflect dynamic changes in real time; behavioral performance indicators lack sensitivity and struggle to capture subtle cognitive fluctuations; among physiological signals, electroencephalogram (EEG) signals offer advantages such as high temporal resolution and non-invasiveness, but current research primarily focuses on immediate assessments, lacking forward-looking predictions of future states, and failing to address the issue of quantitative comparability caused by differences in cross-task difficulty and individual abilities. Furthermore, research on dynamic cognitive load balancing strategies for human-machine collaborative operation and maintenance scenarios in different locations is scarce.

[0004] Therefore, there is an urgent need for a cognitive load prediction method that can predict cognitive load in real time and adapt to complex scenarios. Summary of the Invention

[0005] The purpose of this application is to provide a method, application method, and apparatus for determining a cognitive load prediction model, which can predict the cognitive load level at the next time step.

[0006] To achieve the above objectives, this application provides the following solution: In a first aspect, this application provides a method for determining a cognitive load prediction model, comprising: The study acquired EEG signals, main task difficulty, working memory capacity, and sub-task performance data from several subjects. The task difficulty was defined by dividing the main task into different levels based on parameters such as the translation distance, directional deviation, and safety distance of the target point. The working memory capacity was obtained through the Corsi block-tapping test. The task performance data was obtained from a human-machine remote collaborative operation and maintenance dual-task experimental platform in a virtual reality scenario.

[0007] The EEG signal is preprocessed to obtain the preprocessed EEG signal.

[0008] Feature extraction is performed on the preprocessed EEG signal to obtain multi-dimensional EEG features.

[0009] The subtask performance data is combined with the task difficulty and working memory capacity. A residual-mixed effects model is used to generate a comprehensive cognitive load score that crosses task difficulty and individual ability. The comprehensive cognitive load score is then mapped to a grade using a percentage method to generate a comprehensive cognitive load score grade.

[0010] A time-series dataset was constructed based on the aforementioned multidimensional EEG features and the corresponding comprehensive cognitive load score levels.

[0011] Based on the aforementioned time-series dataset, using the multi-dimensional EEG features as input and the corresponding comprehensive cognitive load score level as the label, the CNN-Bi-LSTM-Attention network is trained using the cross-entropy loss function as the optimization objective to obtain a trained cognitive load prediction model.

[0012] Secondly, this application provides a method for applying a cognitive load prediction model, including: The EEG signals of the subjects to be tested are acquired, the EEG signals are preprocessed, and multidimensional EEG features of the preprocessed EEG signals are extracted.

[0013] The multidimensional EEG features are input into the cognitive load prediction model to obtain the comprehensive cognitive load score level of the subject at the next time step; the cognitive load prediction model is a model obtained based on the method for determining the cognitive load prediction model described above.

[0014] Thirdly, this application provides a computer device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method for determining the cognitive load prediction model or the method for applying the cognitive load prediction model described above.

[0015] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method for determining the cognitive load prediction model or the method for applying the cognitive load prediction model described above.

[0016] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the method for determining the cognitive load prediction model or the method for applying the cognitive load prediction model described above.

[0017] According to the specific embodiments provided in this application, this application has the following technical effects: This application provides a method, application method, and apparatus for determining a cognitive load prediction model. By acquiring EEG signals, main task difficulty, working memory capacity, and sub-task performance data from multiple subjects, it solves the problem of incomplete representation of cognitive load by single-modality data, achieving complete and unified acquisition of multi-source heterogeneous experimental data, and providing a heterogeneous and complementary information foundation for subsequent modeling. Preprocessing the EEG signals solves the problem of the raw EEG signals being unusable directly, significantly improving signal quality and providing a reliable data source for feature extraction. Feature extraction from the preprocessed EEG signals yields multi-dimensional EEG features, addressing the issue of preprocessed EEG signals being unusable for modeling, and achieving effective extraction of multi-dimensional EEG features. By combining the sub-task performance data with the task difficulty and working memory capacity, a residual-mixed effects model is used to generate a comprehensive cognitive load score that crosses task difficulty and individual ability. A percentage-based mapping method is used to rank the comprehensive cognitive load score, generating a comprehensive cognitive load. The scoring system addresses the quantification of cross-task difficulty and individual differences, enabling the calculation of comprehensive cognitive load scores and standardized level mapping, effectively improving the comparability of cognitive load assessment results for different subjects under different task conditions. By constructing a time-series dataset based on the multi-dimensional EEG features and corresponding comprehensive cognitive load scoring levels, the system solves the temporal alignment problem of samples and labels during model training, achieving standardized construction of a time-series, structured dataset. Using the time-series dataset, multi-dimensional EEG features as input, and corresponding comprehensive cognitive load scoring levels as labels, a CNN-Bi-LSTM-Attention network is trained using the cross-entropy loss function as the optimization objective, resulting in a trained cognitive load prediction model. This solves the modeling problem of the complex mapping from high-dimensional EEG features to cognitive load levels, enabling the construction of a cognitive load prediction model. Furthermore, this model, trained on data from several subjects, can cover individual differences in task difficulty and working memory capacity among different subjects, thus possessing good potential for cross-individual application. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is an application environment diagram illustrating a method for determining and applying a cognitive load prediction model according to an embodiment of this application. Figure 2A flowchart illustrating a method for determining a cognitive load prediction model according to an embodiment of this application; Figure 3 A simplified flowchart illustrating a method for determining a cognitive load prediction model, provided in an embodiment of this application; Figure 4 A schematic diagram of the timing flow of a dual-task experiment in a method for determining a cognitive load prediction model provided in another embodiment of this application; Figure 5 A schematic diagram of 39 features significantly related to cognitive load in a method for determining a cognitive load prediction model provided in an embodiment of this application; Figure 6 A schematic diagram of a CNN-Bi-LSTM-Attention network in a method for determining a cognitive load prediction model provided in an embodiment of this application; Figure 7 A schematic diagram illustrating the training parameter settings in a method for determining a cognitive load prediction model according to an embodiment of this application; Figure 8 A flowchart illustrating an application method for a cognitive load prediction model provided in an embodiment of this application; Figure 9 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0020] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0021] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0022] The method for determining a cognitive load prediction model provided in this application embodiment can be applied to, for example, Figure 1In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be set up independently, integrated into server 104, or placed in the cloud or on other servers. Terminal 102 can send the EEG signals of several subjects, the task difficulty of the main task, the working memory capacity of the subjects, and the performance data of the secondary tasks to server 104. The task difficulty is defined by dividing the main task into parameters such as the translation distance, directional deviation, and safety distance of the target point; the working memory capacity is data obtained through the Corsi block-tapping test; and the task performance data is data obtained from a human-machine remote collaborative operation and maintenance dual-task experimental platform based on a virtual reality scenario. Server 104 receives the EEG signals of several subjects, the task difficulty of the main task, the working memory capacity of the subjects, and the performance data of the secondary tasks. For the EEG signals of several subjects, the task difficulty of the main task, the working memory capacity of the subjects, and the performance data of the secondary tasks, server 104 processes the EEG signals... Preprocessing is performed to obtain preprocessed EEG signals; feature extraction is performed on the preprocessed EEG signals to obtain multi-dimensional EEG features; the sub-task performance data is combined with the task difficulty and working memory capacity, and a residual-mixed effects model is used to generate a comprehensive cognitive load score that crosses task difficulty and individual ability; the comprehensive cognitive load score is mapped to a level using a percentage method to generate a comprehensive cognitive load score level; a time-series dataset is constructed based on the multi-dimensional EEG features and the corresponding comprehensive cognitive load score level; based on the time-series dataset, using the multi-dimensional EEG features as input and the corresponding comprehensive cognitive load score level as label, a CNN-Bi-LSTM-Attention network is trained using the cross-entropy loss function as the optimization objective to obtain a trained cognitive load prediction model. The server 104 can feed back the trained cognitive load prediction model to the terminal 102. Furthermore, in some embodiments, the method for determining the cognitive load prediction model can also be implemented separately by the server 104 or the terminal 102. For example, the terminal 102 can directly train the cognitive load prediction model based on the acquired EEG signals of several subjects, the task difficulty of the main task, the working memory capacity of the subjects, and the performance data of the secondary task. Alternatively, the server 104 can acquire the EEG signals of several subjects, the task difficulty of the main task, the working memory capacity of the subjects, and the performance data of the secondary task from the data storage system, and train the cognitive load prediction model based on the acquired EEG signals of several subjects, the task difficulty of the main task, the working memory capacity of the subjects, and the performance data of the secondary task.

[0023] The application method of the cognitive load prediction model provided in this application embodiment can be applied to, for example... Figure 1In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be set up independently, integrated into server 104, or placed in the cloud or on another server. Terminal 102 can send the acquired EEG signals of the test subject to server 104. Server 104 receives the EEG signals of the test subject, preprocesses them, and extracts multi-dimensional EEG features from the preprocessed EEG signals. These multi-dimensional EEG features are then input into a cognitive load prediction model to obtain the comprehensive cognitive load score level of the test subject at the next time step. Server 104 can then feed back the obtained comprehensive cognitive load score level of the test subject at the next time step to terminal 102. Furthermore, in some embodiments, the application method of the cognitive load prediction model can also be implemented separately by the server 104 or the terminal 102. For example, the terminal 102 can directly predict the comprehensive cognitive load score level of the subject at the next time step based on the cognitive load prediction model using the acquired EEG signal of the subject. Alternatively, the server 104 can acquire the EEG signal of the subject from the data storage system and predict the comprehensive cognitive load score level of the subject at the next time step based on the cognitive load prediction model.

[0024] The terminal 102 can be, but is not limited to, various desktop computers, laptops, smartphones, tablets, etc. The server 104 can be implemented using a standalone server or a server cluster composed of multiple servers, or it can be a cloud server.

[0025] In one exemplary embodiment, such as Figure 2 As shown, a method for determining a cognitive load prediction model is provided. This method is executed by a computer device, specifically by a terminal or server alone, or by both a terminal and a server. In this embodiment, the method is applied to... Figure 1 Taking server 104 as an example, the explanation includes the following steps S201 to S206.

[0026] S201: Acquire EEG signals, main task difficulty, working memory capacity, and sub-task performance data of several subjects; the task difficulty is defined by dividing the main task into difficulty levels based on parameters such as translation distance, directional deviation, and safety distance of the target point; the working memory capacity is data obtained through the Corsi block-tapping test; the task performance data is data obtained from a human-machine remote collaborative operation and maintenance dual-task experimental platform in a virtual reality scenario.

[0027] S202: Preprocess the EEG signal to obtain a preprocessed EEG signal.

[0028] S203: Extract features from the preprocessed EEG signal to obtain multi-dimensional EEG features.

[0029] S204: Combining the subtask performance data with the task difficulty and the working memory capacity, a residual-mixed effects model is used to generate a comprehensive cognitive load score that crosses task difficulty and individual ability. The comprehensive cognitive load score is then mapped to a grade using a percentage method to generate a comprehensive cognitive load score grade.

[0030] S205: Construct a time-series dataset based on the multi-dimensional EEG features and the corresponding comprehensive cognitive load score levels.

[0031] S206: Based on the time-series dataset, using the multi-dimensional EEG features as input and the corresponding comprehensive cognitive load score level as label, the CNN-Bi-LSTM-Attention network is trained using the cross-entropy loss function as the optimization objective to obtain a trained cognitive load prediction model.

[0032] By implementing steps S201 to S206 above, this application achieves accurate and time-sequential intelligent prediction of cognitive load by combining individual differences and task difficulty, effectively improving the objectivity, accuracy and generalization ability of cognitive load assessment and prediction in operation and maintenance scenarios.

[0033] As an optional implementation method, the EEG signals, main task difficulty, working memory capacity, and sub-task performance data of several subjects are acquired. The task difficulty is defined by dividing the main task into difficulty levels based on parameters such as the translation distance, directional deviation, and safety distance of the target point. The working memory capacity is obtained through the Corsi block-tapping test. The task performance data is obtained from a human-machine remote collaborative operation and maintenance dual-task experimental platform in a virtual reality scenario, specifically including the following: A human-machine remote collaborative operation and maintenance dual-task experimental platform in a virtual reality scenario is constructed. The dual tasks include a primary task and a secondary task, wherein the primary task is the space station robotic arm gripping task, and the secondary task is the selective auditory monitoring task.

[0034] The main task is divided into 5 levels of difficulty by setting parameters such as the translation distance, directional deviation, and safety distance of the target point. The translation distance ranges from 1.5 to 5 meters, the directional deviation ranges from 3° to 20°, and the safety distance ranges from 0.4 to 1.2 meters. The sub-task generates 5 speech letter stimuli through a pseudo-random sequence. The subject needs to repeat the preset target letter combination verbally. Non-target letters have an inhibitory response. The target letter combination is updated every 4 minutes.

[0035] The experimental platform synchronously collects the subjects' EEG signals and subtask performance data; the Corsi block-tapping test is used to collect the individual working memory capacity of the subjects; the subtask performance data includes the subtask score.

[0036] It should be noted that after the experiment, the NASA-TLX subjective scale was also used to collect subjective cognitive scores from six dimensions: mental and physical demands, time demands, self-performance, effort level, and frustration. The NASA-TLX scale data was obtained by statistical analysis based on the NASA-TLX scale data to verify that there are significant differences in subjective cognitive load scores under different task difficulties, thereby verifying the validity of the experiment.

[0037] In another exemplary embodiment of this application, step S202; preprocessing the EEG signal to obtain a preprocessed EEG signal, is replaced by steps S2021 to S2025: S2021: Using an average reference, EEG signals with abnormal impedance or signal distortion are removed to obtain the first EEG signal.

[0038] S2022: Calculate the average value of the first EEG signal time series, and eliminate the common global noise of the first EEG signal based on the average value to obtain the second EEG signal.

[0039] S2023: By using high-pass filtering and low-pass filtering, the frequency bands related to brain activity in the second EEG signal are extracted to obtain the third EEG signal; the high-pass filtering is used to remove useless low-frequency drift; the low-pass filtering is used to remove useless high-frequency noise.

[0040] S2024: For EEG signals with abnormal impedance or signal distortion, spatial weighted interpolation is performed based on several third EEG signals surrounding the EEG signal with abnormal impedance or signal distortion to estimate and recover the EEG signal with abnormal impedance or signal distortion, thereby obtaining a fourth EEG signal; wherein, the weights of the spatial weighted interpolation are determined by the spherical Laplace equation.

[0041] S2025: Independent component analysis (ICA) was used to remove ocular, electromyographic, and electrocardiographic artifacts from the third and fourth EEG signals to obtain preprocessed EEG signals.

[0042] Specifically, the EEG signal preprocessing includes the following steps: whole-brain average rereference, 0.1Hz high-pass filtering and 50Hz low-pass filtering, bad lead interpolation, and ICA independent component analysis to remove EEG and EMG artifacts; the multi-dimensional EEG features include time-domain features (mean, variance, differential entropy), frequency-domain features (… , , , , Band power spectral density, Hjorth complexity and activity, autoregressive coefficients) and global characteristics (prefrontal cortex) Lateralization index, Spectral slope).

[0043] In another exemplary embodiment of this application, step S204 above, which combines the subtask performance data with the task difficulty and the working memory capacity, uses a residual-mixed effects model to generate a comprehensive cognitive load score that crosses task difficulty and individual ability, and uses a percentage method to perform a grade mapping on the comprehensive cognitive load score to generate a comprehensive cognitive load score grade, is replaced by the following steps S2041 to S2044: S2041: Perform conditional interval scaling and normalization on the sub-task scores in the sub-task performance data while retaining anomaly indicators to obtain standardized sub-task scores.

[0044] S2042: Perform binarization on the task difficulty to obtain the binarized task difficulty; S2043: Perform Min-Max normalization on the working memory capacity to obtain the normalized working memory capacity.

[0045] S2044: Using the standardized sub-task score as the dependent variable, the binarized task difficulty and the normalized working memory capacity as fixed effects, and individual differences among subjects as random effects, a comprehensive cognitive load score is generated through a residual-mixed effects model. The comprehensive cognitive load score is then mapped to a grade using a percentage method to generate a comprehensive cognitive load score grade.

[0046] Specifically, the subtask performance data are combined with task difficulty and the subject's working memory capacity, and a residual-mixed effects model is used to generate a comprehensive cognitive load score that crosses task difficulty and individual ability.

[0047] The construction of the residual-mixed effects model includes: performing conditional interval scaling and normalization on sub-task scores while retaining anomaly indicators; binarizing task difficulty; and performing Min-Max normalization on working memory capacity. Using standardized sub-task scores as the dependent variable, task difficulty and working memory capacity as fixed effects, and individual differences among subjects as random effects, a comprehensive cognitive load score is generated by combining model residuals with a moderation term, and then divided into four levels of cognitive load using the percentile method.

[0048] As an optional implementation, the expression for the residual-mixed effects model is: ; ; ; in, For the first i The subject was on the 1st j The predicted score of the standardized sub-task based on the difficulty level of the primary task; This is the global intercept; The compensation effect coefficient for personnel capabilities; The inhibition effect coefficient represents the difficulty of the task. For the first i The random intercept of each individual subject; For the first i Normalized working memory capacity of each subject; Codition The task difficulty after binarization; For the first i The subject was on the 1st j Standardized sub-task scores for tasks of varying difficulty levels; For the first i The subject was on the 1st j The residual of the task difficulty level; ; ; ; For the first i The subject was on the 1st j A comprehensive cognitive load score for task difficulty.

[0049] As an optional implementation, step S205: Constructing a time-series dataset based on the multi-dimensional EEG features and the corresponding comprehensive cognitive load score levels, specifically including the following: A time series dataset was constructed using the sliding window method, with a window length of 4 time steps (60 seconds) and a prediction step length of 1 time step (15 seconds). The dataset was divided into a training set and a validation set in an 8:2 ratio. The training set was augmented by injecting Gaussian noise (mean 0, standard deviation 0.02).

[0050] As an optional implementation, the CNN-Bi-LSTM-Attention network specifically includes the following: The layers consist of an input layer, a CNN layer, a Bi-LSTM layer, an Attention layer, a fully connected layer, and an output layer.

[0051] The input layer is used to input multi-dimensional EEG temporal features.

[0052] The CNN layer uses 1D convolution and max pooling to extract the local dependency features of the multi-dimensional EEG features in the spatial dimension and output the feature sequence.

[0053] The Bi-LSTM layer captures long-short-term time series dependencies through a bidirectional LSTM network, fuses forward and backward time series features, and outputs the hidden state sequence at each time step.

[0054] The Attention layer receives the hidden state sequence output by the Bi-LSTM layer, calculates the attention weights at each time step and sums them up, focuses on the features of key time steps, and outputs a weighted temporal feature vector.

[0055] The fully connected layer receives the vector output by the Attention layer, performs non-linear mapping and dimensional transformation, and outputs a feature vector with a dimension equal to the number of categories.

[0056] The output layer receives the feature vectors output by the fully connected layer, performs classification using the Softmax function, and outputs the comprehensive cognitive load score level for the next time step.

[0057] Specifically, the CNN layer uses 1D convolution (64 channels, kernel size 3) and max pooling (stride 2) to extract local EEG spatial features; the Bi-LSTM layer uses a 2-layer bidirectional LSTM network (64 hidden dimensions) to capture the long-term and short-term temporal dependencies of cognitive load; the Attention layer calculates the weights of features at each time step and sums them up to focus on key time steps and key brain region features; the output layer outputs the 4-level cognitive load prediction results through a fully connected layer and a Softmax function.

[0058] As an optional implementation, the expression for the cross-entropy loss function is: ; in, L This represents the cross-entropy loss value. N Number of categories; This is a real label; The model predicts the first Class probability.

[0059] In practical applications, such as Figure 3 As shown, the method for determining the cognitive load prediction model disclosed in this application utilizes virtual reality technology to build a human-machine remote collaborative operation and maintenance experiment in the context of a space station, and collects and processes the data in the experiment; builds a dataset based on the experimental data, constructs a CNN-Bi-LSTM-Attention model based on EEG data and trains it using a training set, and finally obtains a cognitive load prediction method for human-machine remote collaborative operation and maintenance scenarios.

[0060] Step 1: Build a human-machine remote collaborative operation and maintenance experimental platform in a virtual reality environment, design a dual-task experimental paradigm, and comprehensively reflect the cognitive load of the main task through the performance of the secondary task and subjective scales.

[0061] Step 1.1: Main Task Experiment Design: The main task, referencing tasks such as module transfer in real-world human-machine collaborative operation and maintenance scenarios on a space station, is designed as a robotic arm gripping task. In the initial stage of the experiment, the robotic arm automatically returns to the preset spatial coordinate origin (…). X =0, Y =0, Z =0). The subjects controlled the end effector of the robotic arm to complete a series of tasks through dynamic hand gesture commands: (1) visually identify a specific yellow target container (size 30×30×50cm); (2) perform multi-dimensional spatial gripping actions (six degrees of freedom control); (3) implement high-precision material delivery to a designated area (circular projection diameter 20cm). The experimental parameter settings included orthogonal combinations of path curvature radius (1.5m and 3.0m dual levels) and delivery accuracy requirements (target area diameter 20cm and 40cm dual levels).

[0062] Step 1.2: Sub-task Experiment Design: The sub-task is designed as a selective auditory monitoring task, simulating a typical scenario of parallel processing of multi-source information during space station operations. Speech letters ( / a / , / b / , / c / , / n / , / o / ) are presented at fixed 3-second intervals. Participants are required to quickly repeat the preset target letters, and the allocation of cognitive resources is quantified through reaction time and accuracy.

[0063] like Figure 4 The diagram illustrates the timeline of a dual-task experiment, employing a parallel timeline structure to present the experimental progress of the main and secondary tasks. The main task (top) is conducted in 20-minute trial units, with a 5-minute break and questionnaire completion period between trials. The entire process is repeated twice. The secondary task (bottom) uses an auditory letter recognition paradigm. The task switches the target letter ("a" or "c") every 4 minutes, presenting letter stimuli at a rate of 3 seconds per letter. Each group completes a 15-second interval, and secondary task performance is recorded simultaneously. The secondary task is conducted concurrently with the main task, also including 5-minute breaks and questionnaire completion periods at key points in the process. The entire process is repeated twice.

[0064] Step 2: Experimental data acquisition and preprocessing.

[0065] Step 2.1: EEG signal acquisition: Electrodes were laid out according to the international 10-20 system using a 32-channel EEG device, covering cognitive-related brain regions such as the frontal and parietal lobes, with a sampling rate of 128Hz, and signals were recorded synchronously.

[0066] Step 2.2: Preprocess the EEG signals acquired in Step 2.1 using EEGlab in Matlab. The specific method is as follows: Re-reference: Using an average reference, electrodes with abnormal impedance or signal distortion are removed; for the remaining m effective electrodes, the average value of the time series is calculated, and the common global noise of the EEG signals measured by the remaining m effective electrodes is eliminated based on the average value.

[0067] Signal filtering: High-pass filtering at 0.1 Hz and low-pass filtering at 50 Hz were used to extract the frequency bands related to brain activity in the EEG signal, while removing useless low-frequency drift and high-frequency noise.

[0068] Interpolated bad conduction: For the differential voltage at the bad conduction point, the weight of the normal electrodes is determined by the spherical Laplace equation based on the spatial weighting of the surrounding n normal electrodes.

[0069] ICA artifact removal: ICA independent component analysis is used to remove artifacts from electrooculography (EOG), electromyography (EMG), and electrocardiography (ECG).

[0070] Step 2.3: EEG Feature Extraction: Extracting temporal features (mean, variance, differential entropy), frequency domain features (Hjorth complexity, activity), and global features (prefrontal cortex). Lateralization index, Spectral slope), selected through Pearson correlation analysis, such as Figure 5 The 39 features shown are significantly associated with cognitive load.

[0071] Step 2.4: Sub-task performance and subjective cognitive load rating: Record sub-task performance to obtain sub-task performance data.

[0072] Following the experiment, the NASA-TLX subjective scale was used to collect subjective cognitive scores from six dimensions: mental and physical demands, time demands, self-performance, effort level, and frustration. Data analysis was then conducted based on the NASA-TLX data to verify that there were significant differences in subjective cognitive load scores under different task difficulties, thereby validating the effectiveness of the experiment.

[0073] Step 2.5: Personnel Capacity Collection: The working memory capacity of individual subjects was collected through the Corsi block-tapping test.

[0074] Step 3: Model dataset processing.

[0075] Step 3.1: Normalization Processing: Perform segmented normalization on the subtask scores (retain 0-point outlier markers, linear scaling for 1-5 points, and non-linear transformation for 6-12 points); perform 0-1 binarization on the task difficulty; and perform Min-Max normalization on the working memory capacity.

[0076] Step 3.2: Based on the residual-mixed effects model, combine the sub-task score, task difficulty and working memory capacity to separate fixed effects and random effects, generate a comprehensive cognitive load score and classify it into 0-3 levels using the percentage method.

[0077] The Residual Mixed Effects Model (RMM) is an extension of the Generalized Linear Mixture Model (GLMM). Its core lies in achieving refined modeling of complex data structures by decomposing systematic and random variations in observations. Compared to traditional mixture models, this model, by explicitly extracting residual terms and assigning interpretability, can separate individual-specific fluctuations (such as attention drift and strategy adjustment) not explained by fixed effects (such as personnel ability and task difficulty). In this application, this model was chosen primarily based on the following advantages: (1) Hierarchical data modeling: The dual-task experimental data has a nested structure (repeated measurement of the same subject, cross-design of tasks with different difficulty levels), and the random effect item (subject) can capture stable trait differences among subjects.

[0078] (2) Analysis of variance components: Fixed effects (Ability normalization, Condition normalization) explain systematic variation, and residual terms capture random fluctuations at trial levels.

[0079] (3) Scalability: Through the residual re-integration mechanism, a mathematical basis is provided for constructing a comprehensive score.

[0080] This step constructs a residual-mixed effects model to generate predicted scores for each subject's task at each time step. The specific formula is as follows: ; ; ; in, For the first i The subject was on the 1st j The predicted score of the standardized sub-task based on the difficulty level of the primary task; This is the global intercept; The compensation effect coefficient for personnel capabilities; The inhibition effect coefficient represents the difficulty of the task. For the first i The random intercept of each individual subject; For the first i Normalized working memory capacity of each subject; Codition The task difficulty after binarization; For the first i The subject was on the 1st jStandardized sub-task scores for tasks of varying difficulty levels; For the first i The subject was on the 1st j The residual of the task difficulty level; ; ; ; For the first i The subject was on the 1st j A comprehensive cognitive load score for task difficulty.

[0081] Step 3.3: Based on the EEG features from Step 2.3 and the cognitive scores from Step 3.2, construct a dataset, divide the training set and test set into an 8:2 ratio, construct a time-series dataset using a sliding window (15-second time step), and add a normal distribution random perturbation with a mean of 0 and a standard deviation of 0.02 to each feature dimension of the training set. Then, merge the training set with the original training set through vertical concatenation to form an enhanced training set of double the size.

[0082] Step 4: Cognitive load prediction model training. The multi-dimensional EEG features are used as input, and the comprehensive cognitive load score is used as a label to train the model to predict the cognitive load level in the next 15 seconds.

[0083] Step 4.1: Construct the CNN-Bi-LSTM-Attention network.

[0084] In the scenario of human-machine collaborative operation and maintenance on the space station, the dynamic nature and spatiotemporal coupling of cognitive load pose unique challenges to predictive models. Traditional models face three limitations in this scenario: First, the spatial topological characteristics of EEG signals (such as cross-brain region functional connectivity and local electrode cluster patterns) are difficult to be effectively represented by shallow networks, leading to the loss of key features in the spatial dimension; second, the temporal evolution of cognitive load has long-range dependence and bidirectional correlation (such as the impact of previous task accumulation on the current state), and conventional time-series models are unable to capture dynamic correlations across time steps; third, in multi-task concurrent scenarios, there are dynamic differences in the feature contributions of key brain regions and key time periods, but traditional static weight allocation mechanisms cannot achieve adaptive focusing.

[0085] To address the aforementioned issues, this application proposes a deep learning framework that integrates convolutional neural networks (CNN), bidirectional long short-term memory networks (Bi-LSTM), and attention mechanisms: First, a convolutional neural network is used to model the local dependencies of pre-selected EEG features in the spatial dimension. A hierarchical abstraction mechanism refines the feature representation, providing high-quality input features for subsequent processing. Then, a bidirectional long short-term memory network is used to capture the long-term temporal dependencies of EEG sequences, modeling the inherent dynamic evolution of the signals. Finally, an attention mechanism is used to selectively focus on key information dimensions and suppress interference from irrelevant features through an adaptive feature weighting strategy. By organically integrating the Bi-LSTM network and the attention mechanism, synergistic optimization of temporal dependency modeling and feature importance assessment is achieved. This fully leverages the complementary advantages of each module, enabling the system to simultaneously capture local spatial correlations, model long-term temporal relationships, and strengthen the expression of key features, thereby significantly improving the prediction accuracy of cognitive load states.

[0086] Step 4.1.1: Convolutional Neural Network (CNN): (1) One-dimensional convolutional layer: The number of filter banks is K Each filter has a size of d =3 (odd kernel) ensures capture of local feature correlations between adjacent time steps. A padding=1 strategy maintains the temporal length of the features after convolution. A BatchNorm layer is connected after convolution for batch normalization, and then a ReLU activation function is used to achieve non-linear transformation, enhancing feature discriminativeness. The convolution operation formula is as follows: ; in, For the first Each convolutional kernel at time step The output feature value; Input the number of channels for EEG; d The kernel size; This is the temporal offset index within the convolution kernel; For the k-th convolutional kernel in the channel c Intranuclear shift Weight at each location; To indicate the input data in the channel Time Index The value; The bias term is K; the number of convolution kernels is K. T This represents the timing length.

[0087] Pooling layer: Use max pooling with a kernel size of 2 and a stride of s=2. Downsampling is used to compress the feature time step to... It retains a significant activation pattern.

[0088] Step 4.1.2: Bidirectional Long Short-Term Memory Network (Bidirectional LSTM Network): Captures the bidirectional temporal dependence of cognitive load, integrates forward and backward temporal features to enhance modeling capabilities, and performs temporal processing on the optimized high-dimensional representation provided by the CNN module.

[0089] Step 4.1.3: Attention mechanism: Dynamically weight the bidirectional temporal features of the bidirectional long short-term memory network output, focusing on key features related to cognitive load.

[0090] Step 4.1.4: Fully connected layer: Perform non-linear mapping and dimensional transformation, and output a feature vector with a dimension equal to the number of categories.

[0091] Step 4.1.5: Output layer: Softmax classification is used to output 4 levels of cognitive load.

[0092] Specifically, such as Figure 6 As shown, the input layer first samples the raw multi-dimensional EEG time-series data at the bottom of the diagram using a sliding window with a window size of 4 and a stride of 1, forming the model's input feature sequence. Then, the CNN layer, corresponding to the "local feature association" module in the diagram, extracts the local dependencies of the multi-dimensional EEG features in the spatial dimension through 1D convolution and max pooling operations, outputting the feature sequence to provide a foundation for subsequent time-series modeling. Next, the Bi-LSTM layer, corresponding to the "time series modeling" module in the diagram, uses a forward LSTM (LSTM... f ) and backward LSTM (LSTM) b A bidirectional LSTM network is constructed to capture the long-term and short-term temporal dependencies of EEG signals, fusing forward and backward temporal features and outputting the hidden state sequence at each time step. Then, the Attention layer, corresponding to the "dynamic weighting of time steps" module in the diagram, receives the hidden state sequence output by the Bi-LSTM layer, calculates the attention weights for each time step, and sums them to focus on key time step features, outputting a weighted temporal feature vector. This process corresponds to the weight calculation and feature aggregation process in the upper left corner of the diagram. Next, the fully connected layers, corresponding to Linear Layer 1 and Linear Layer 2 in the diagram, receive the vector output by the Attention layer and perform non-linear mapping and dimensional transformation, outputting a feature vector with a dimension equal to the number of categories. Finally, the output layer receives the feature vector output by the fully connected layers, uses the Softmax function for classification, outputs the comprehensive cognitive load score level for the next time step, and completes the final prediction.

[0093] Step 4.2: Training parameter settings. See below for specific training parameters. Figure 7 .

[0094] In one exemplary embodiment, such as Figure 8As shown, a method for applying a cognitive load prediction model is provided, including the following steps S301 to S302: S301: Acquire the EEG signal of the subject to be tested, preprocess the EEG signal, and extract the multi-dimensional EEG features of the preprocessed EEG signal.

[0095] S302: Input the multidimensional EEG features into the cognitive load prediction model to obtain the comprehensive cognitive load score level of the subject at the next time step (e.g., the next 15 seconds).

[0096] Table 1 shows the classification performance of the CNN-Bi-LSTM-Attention model across four categories (low, medium, high, and very high CL). The model achieved the highest precision (0.9623) for the "low CL" category, the best recall (0.9403) and F1 score (0.9065) for the "very high CL" category, and the lowest precision (0.8182) for the "medium CL" category. Overall, the model achieved an accuracy of 0.8789, with macro-average and weighted average F1 scores of 0.8783 and 0.8790, respectively, indicating a relatively balanced performance across different categories and good overall classification results.

[0097] Table 1. Model performance of CNN-Bi-LSTM-Attention

[0098] It should be noted that alternative methods for determining and applying cognitive load prediction models mainly include the following categories: At the prediction model level, traditional machine learning algorithms (such as random forests, support vector machines, logistic regression, extreme gradient boosting, and adaptive reinforcement) and basic deep learning architectures (such as recurrent neural networks, unidirectional long short-term memory networks, convolutional neural network-long short-term memory network hybrid models, bidirectional long short-term memory networks, long short-term memory network-attention mechanism models, and bidirectional long short-term memory network-attention mechanism models) can be used. Although these models can achieve cognitive load-related assessments, their performance in spatial local feature extraction, long-term temporal dependency capture, and key pattern adaptive focusing is inferior to the CNN-Bi- proposed in this application. The LSTM-Attention hybrid architecture is used for cognitive load assessment. While methods such as single subjective scales (e.g., the NASA-TLX scale used independently), simple behavioral performance analysis (judging load solely based on task completion), and ECG signal (heart rate variability) assessment can be employed, these methods may suffer from time lag, emotional interference, or insufficient temporal resolution, making it difficult to meet the real-time and accuracy requirements of remote human-machine collaborative operation and maintenance scenarios on the space station. At the data processing and prediction logic level, traditional time series analysis, regression analysis, and expert evaluation methods can be used. However, these methods are insufficient to effectively characterize the nonlinear and time-varying characteristics of cognitive load and cannot address the issue of quantifying and comparing cognitive load across task difficulty and individual capabilities.

[0099] In summary, compared with the prior art, this application has the following advantages: (1) The residual-mixed effects model solves the problem of quantifying cross-task and individual differences, and improves the comparability of results.

[0100] (2) By capturing temporal dynamics bidirectionally using Bi-LSTM, the problem of gradient vanishing in long sequences can be effectively solved.

[0101] (3) The CNN-Bi-LSTM-Attention network model is used to predict the cognitive load in the next 15 seconds. The prediction accuracy is 88.31%, the recall rate is 87.89%, and the F1-Score is 87.90%, which can provide early warning of cognitive abnormalities.

[0102] In one exemplary embodiment, a computer device is provided, which may be a server or a terminal, and its internal structure diagram may be as follows. Figure 9As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media. The database stores data during the training and application of the cognitive load prediction model. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements a method for determining and applying a cognitive load prediction model.

[0103] Figure 9 The structures shown are merely block diagrams of some structures related to the present application and do not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than shown in the figures, or combine certain components, or have different component arrangements. In an exemplary embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0104] In one exemplary embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0105] In one exemplary embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0106] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with relevant regulations and be authorized by the owner of the corresponding device.

[0107] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).

[0108] The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0109] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0110] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for determining a cognitive load prediction model, characterized in that, The method for determining the cognitive load prediction model includes: The study acquired EEG signals, main task difficulty, working memory capacity, and sub-task performance data from several subjects. The task difficulty was defined by setting parameters such as the translation distance, directional deviation, and safety distance of the target point in the main task. The working memory capacity was obtained through the Corsi block-tapping test. The task performance data was obtained from a human-machine remote collaborative operation and maintenance dual-task experimental platform in a virtual reality scenario. The EEG signal is preprocessed to obtain the preprocessed EEG signal; Feature extraction is performed on the preprocessed EEG signals to obtain multi-dimensional EEG features; The subtask performance data is combined with the task difficulty and the working memory capacity. A residual-mixed effects model is used to generate a comprehensive cognitive load score that crosses task difficulty and individual ability. The comprehensive cognitive load score is then mapped to a grade using a percentage method to generate a comprehensive cognitive load score grade. A time-series dataset was constructed based on the aforementioned multi-dimensional EEG features and the corresponding comprehensive cognitive load score levels. Based on the aforementioned time-series dataset, using the multi-dimensional EEG features as input and the corresponding comprehensive cognitive load score level as the label, the CNN-Bi-LSTM-Attention network is trained using the cross-entropy loss function as the optimization objective to obtain a trained cognitive load prediction model. 2.The method of claim 1, wherein, The EEG signal is preprocessed to obtain a preprocessed EEG signal, specifically including: Using an average reference, EEG signals with abnormal impedance or signal distortion are removed to obtain the first EEG signal; Calculate the average value of the first EEG signal time series, and eliminate the common global noise of the first EEG signal based on the average value to obtain the second EEG signal; The third EEG signal is obtained by extracting the frequency bands related to brain activity from the second EEG signal through high-pass filtering and low-pass filtering; the high-pass filtering is used to remove useless low-frequency drift; the low-pass filtering is used to remove useless high-frequency noise. For EEG signals with abnormal impedance or signal distortion, spatial weighted interpolation is performed based on several third EEG signals surrounding the EEG signal with abnormal impedance or signal distortion to estimate and recover the EEG signal with abnormal impedance or signal distortion, thus obtaining a fourth EEG signal; wherein, the weights of the spatial weighted interpolation are determined by the spherical Laplace equation. Independent component analysis (ICA) was used to remove artifacts of electrooculography (EOG), electromyography (EMG), and electrocardiography (ECG) from the third and fourth EEG signals, resulting in preprocessed EEG signals. 3.The method of claim 1, wherein, Combining the subtask performance data with the task difficulty and working memory capacity, a residual-mixed effects model is used to generate a comprehensive cognitive load score that integrates task difficulty and individual ability. A percentage-based grading method is then used to map the comprehensive cognitive load score to generate a comprehensive cognitive load score level, specifically including: The sub-task scores in the sub-task performance data are scaled and normalized by retaining anomaly indicators to obtain standardized sub-task scores. The task difficulty is binarized to obtain the binarized task difficulty. The working memory capacity is normalized using Min-Max to obtain the normalized working memory capacity; Using standardized sub-task scores as the dependent variable, the binarized task difficulty and the normalized working memory capacity as fixed effects, and individual differences among subjects as random effects, a comprehensive cognitive load score is generated through a residual-mixed effects model. The comprehensive cognitive load score is then mapped to a grade using a percentage method to generate a comprehensive cognitive load score level.

4. The method of claim 3, wherein the cognitive load prediction model is determined by: The expression for the residual-mixed effects model is: ； ； ； wherein, is the predicted value of the standardized subtask score of the nth subject at the mth level of task difficulty; i j is the predicted value of the standardized subtask score of the nth subject at the mth level of task difficulty; is the global intercept; is the compensation effect coefficient of the person ability; is the suppression effect coefficient of the task difficulty; is the random intercept of the nth subject; i is the normalized working memory capacity of the nth subject; is the binary task difficulty; i is the normalized working memory capacity of the nth subject; Codition is the binary task difficulty; is the observed value of the standardized subtask score of the nth subject at the mth level of task difficulty; i is the observed value of the standardized subtask score of the nth subject at the mth level of task difficulty; j is the residual of the nth subject at the mth level of task difficulty; is the residual of the nth subject at the mth level of task difficulty; i is the residual of the nth subject at the mth level of task difficulty; j is the residual of the nth subject at the mth level of task difficulty; ; ; ; is the comprehensive cognitive load score of the nth subject at the mth level of task difficulty; i is the comprehensive cognitive load score of the nth subject at the mth level of task difficulty; j is the comprehensive cognitive load score of the nth subject at the mth level of task difficulty.​ 5. The method of claim 1, wherein the cognitive load prediction model is determined by: The CNN-Bi-LSTM-Attention network specifically includes: Input layer, CNN layer, Bi-LSTM layer, Attention layer, fully connected layer, and output layer; The input layer is used to input multi-dimensional EEG temporal features; The CNN layer uses 1D convolution and max pooling to extract the local dependency features of the multi-dimensional EEG features in the spatial dimension and output the feature sequence. The Bi-LSTM layer captures long-short-term time dependencies through a bidirectional LSTM network, fuses forward and backward time features, and outputs the hidden state sequence at each time step. The Attention layer receives the hidden state sequence output by the Bi-LSTM layer, calculates the attention weights at each time step and sums them up, focuses on the features of key time steps, and outputs a weighted temporal feature vector. The fully connected layer receives the vector output by the Attention layer, performs non-linear mapping and dimensional transformation, and outputs a feature vector with a dimension equal to the number of categories. The output layer receives the feature vectors output by the fully connected layer, performs classification using the Softmax function, and outputs the comprehensive cognitive load score level for the next time step.

6. The method of claim 1, wherein the cognitive load prediction model is determined based on a plurality of cognitive load prediction models. The expression for the cross-entropy loss function is: ； in, L This represents the cross-entropy loss value. N Number of categories; This is a real label; The model predicts the first Class probability.

7. A method for applying a cognitive load prediction model, characterized in that, The application methods of the cognitive load prediction model include: The brainwave signals of the subjects to be tested are acquired, the brainwave signals are preprocessed, and the multidimensional brainwave features of the preprocessed brainwave signals are extracted. The multidimensional EEG features are input into the cognitive load prediction model to obtain the comprehensive cognitive load score level of the subject at the next time step; the cognitive load prediction model is a model obtained based on the method for determining the cognitive load prediction model according to any one of claims 1-6.

8. A computer device, comprising: A memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that the processor executes the computer program to implement the method for determining the cognitive load prediction model according to any one of claims 1-6 or the method for applying the cognitive load prediction model according to claim 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the method for determining the cognitive load prediction model according to any one of claims 1-6 or the method for applying the cognitive load prediction model according to claim 7.

10. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the method for determining the cognitive load prediction model according to any one of claims 1-6 or the method for applying the cognitive load prediction model according to claim 7.

Images