Cognitive load assessment method, system, device and medium for human-computer system

Abstract

The present application provides a method, system, device and medium for cognitive load assessment of human-computer systems, which relates to the field of human-computer systems; the method includes: conducting a cognitive ability test on a first target group to divide the first target group into a high cognitive ability group and a low cognitive ability group, and constructing a first classifier and a second classifier; collecting first multimodal data and performing preprocessing and feature extraction, and fusing the obtained key features into a target feature vector; based on the target feature vector, introducing a balance factor to train the first classifier and the second classifier respectively, and obtaining a first target evaluation model and a second target evaluation model; and fusing to obtain a comprehensive evaluation model. The present application has the characteristics of being cross-participant and cross-task, which improves the scope of application and practicality of the assessment, while enhancing the versatility and ease of use of the model; it realizes the cognitive load assessment of human-computer systems with multi-feature fusion, and provides users with a comprehensive, accurate and easy-to-operate assessment tool.

Description

Technical Field

[0001] The present application relates to the technical field of human-machine systems, and in particular to a method, system, device and medium for evaluating cognitive load of human-machine systems. Background Art

[0002] Cognitive load is a multidimensional structure that represents the load imposed on learners' cognitive systems when processing specific tasks. This multidimensional structure consists of the causal dimension that reflects the interaction between tasks and learners' characteristics and the evaluation dimension that reflects measurable concepts such as mental load, mental effort and performance.

[0003] In related technologies, cognitive load assessment in human-computer systems mainly includes the following methods:

[0004] (1) Currently, most studies on cognitive load assessment are limited to specific tasks or individuals. Many existing methods often produce good results only in specific tasks or individuals. It is difficult to conduct intelligent assessment across tasks or participants, and ignores the impact of individual differences on cognitive load assessment, such as differences in each person's baseline and physiological response. It is difficult to achieve intelligent assessment of cognitive load across participants and tasks in human-computer systems.

[0005] (2) Cognitive load assessment is carried out through subjective assessment methods, task performance methods, etc., which are complex and highly subjective. At the same time, both subjective assessment methods and task performance methods require results to be obtained after the task is executed, which cannot meet the needs of real-time assessment of cognitive load during the task state in human-computer systems.

[0006] (3) Collect basic data based on wearable devices such as EEG, and use machine learning methods to explore the correlation between cognitive load and EEG data, and construct a mapping relationship between EEG data and cognitive load. However, wearable data collection devices such as EEG have disadvantages such as being difficult to deploy, difficult to wear in human-machine systems, and causing interference to individuals. Summary of the invention

[0007] In response to the deficiencies in the prior art, the present application provides a method, system, device and medium for cognitive load assessment in human-computer systems, which solves the problems of large limitations, strong subjectivity and difficulty in deploying and testing cognitive load assessment in current human-computer systems.

[0008] To achieve the above objectives, this application is implemented through the following technical solutions:

[0009] In a first aspect, an embodiment of the present application provides a cognitive load assessment method for a human-computer system, the cognitive load assessment method comprising: performing a cognitive ability test on a preset first target group, and dividing the first target group into a high cognitive ability group and a low cognitive ability group according to the obtained cognitive test results and a preset classification threshold; constructing a first classifier and a second classifier corresponding to the high cognitive ability group and the low cognitive ability group; collecting first multimodal data of the first target group during the execution of an N-back task through a preset acquisition device, and performing preprocessing and feature extraction, and fusing the obtained key features into a target feature vector; based on the target feature vector, introducing a balancing factor through a preset BDA balancing domain adaptation method, and using a K-fold cross-validation method to train the first classifier and the second classifier respectively, to obtain a first target assessment model and a second target assessment model;

[0010] The first target evaluation model and the second target evaluation model are integrated to obtain a comprehensive evaluation model for cognitive load evaluation.

[0011] According to the first aspect of the embodiment of the present application, the above-mentioned fusion of the first target evaluation model and the second target evaluation model to obtain a comprehensive evaluation model for cognitive load evaluation includes the following steps: according to the cognitive test results of each participant in the first target group, a clustering method is used to measure the similarity of different individuals to obtain a similarity measurement result; based on the similarity measurement result, different weights are assigned to the first target evaluation model and the second target evaluation model, and normalization is performed to obtain a comprehensive evaluation model for cognitive load evaluation.

[0012] According to the first aspect of the embodiment of the present application, after the first target evaluation model and the second target evaluation model are fused to obtain a comprehensive evaluation model for cognitive load evaluation, the cognitive load evaluation method for human-computer systems may also include the following steps: non-invasively collecting second multimodal data of a preset second target group during the execution of the N-back task through a preset acquisition device; preprocessing and feature extracting the second multimodal data, and inputting the extracted key features into the comprehensive evaluation model to perform performance evaluation on the comprehensive evaluation model.

[0013] According to the first aspect of the embodiment of the present application, after the first target evaluation model and the second target evaluation model are fused to obtain a comprehensive evaluation model for cognitive load evaluation, the cognitive load evaluation method for human-computer systems may also include the following steps: predicting data in a new data set based on the comprehensive evaluation model to obtain data between 0 and 1 to characterize the probability of cognitive load; converting the obtained data between 0 and 1 into a binary classification result of the cognitive load evaluation of participants in the human-computer system to make decisions and feedback in practical applications.

[0014] According to a first aspect of an embodiment of the present application, the aforementioned preset acquisition device includes an E4 wristband and a visible light camera, and the aforementioned first multimodal data includes a PPG signal, an EDA signal, a skin temperature signal, and visible light video information related to cognitive load.

[0015] According to the first aspect of the embodiment of the present application, the aforementioned method of collecting the first multimodal data of the first target group during the execution of the N-back task through a preset acquisition device, and performing preprocessing and feature extraction, and fusing the obtained key features into a target feature vector, may include the following steps: denoising the PPG signal through a preset second-order Butterworth filter low-pass filtering to obtain a target PPG signal; denoising the EDA signal using a fourth-order Butterworth filter low-pass filtering to obtain a target EDA signal; standardizing the target PPG signal, target EDA signal, skin temperature signal, and visible light video information to align time and eliminate dimensional differences to obtain a target skin temperature signal and target visible light video information; based on the target PPG signal, target EDA signal, target skin temperature signal, and target visible light video information, extracting multiple key features of the cognitive load of participants in the task state; performing high-dimensional feature fusion on the extracted multiple key features to generate a target feature vector.

[0016] According to a first aspect of an embodiment of the present application, the target PPG signal characteristics include: pulse period, pulse amplitude, pulse width, pulse rise time, heart rate, signal skewness, signal kurtosis, total power, frequency bandwidth and heart rate variability HRV.

[0017] According to a first aspect of an embodiment of the present application, target EDA signal features include: the number of galvanic skin response peaks, the average amplitude of galvanic skin response peaks, the standard deviation of galvanic skin tension, and galvanic skin autocorrelation.

[0018] According to a first aspect of an embodiment of the present application, the target skin temperature signal includes: skin temperature mean, skin temperature median, skin temperature maximum, skin temperature minimum, skin temperature standard deviation, skin temperature variance, skin temperature skewness and skin temperature kurtosis.

[0019] According to a first aspect of an embodiment of the present application, the target visible light video information includes facial behavior and eye movement characteristics, and the eye movement characteristics include scanning characteristics, gaze characteristics and blinking frequency; wherein the scanning characteristics correspond to a peak scanning speed.

[0020] According to a first aspect of an embodiment of the present application, a peak scanning speed calculation process includes: analyzing each frame of the target visible light video information, extracting the inner and outer corners of the eye position information and the pupil center coordinates; extracting the normalized scanning amplitude based on the inner and outer corners of the eye position information and the pupil center coordinates; subtracting the normalized scanning amplitudes between two consecutive frames, and determining the maximum difference as the peak scanning speed;

[0021] Among them, the calculation process of the normalized scanning amplitude satisfies the expression:

[0022]

[0023]

[0024]

[0025] Where NSA is the normalized scan amplitude, S is the initial scan amplitude, and D is the distance between the inner and outer corners of the eye. , ) is the coordinate of pupil center, ( , ) is the coordinate of the outer corner of the eye, ( , ) is the inner eye corner coordinate.

[0026] In a second aspect, an embodiment of the present application provides a cognitive load assessment system for a human-computer system, the cognitive load assessment system comprising: a test division module, a construction module, an acquisition and extraction module, a training module and a fusion module; wherein the test division module is used to perform a cognitive ability test on a preset first target group, and divide the first target group into a high cognitive ability group and a low cognitive ability group according to the obtained cognitive test results and a preset classification threshold; the construction module is used to construct a first classifier and a second classifier corresponding to the high cognitive ability group and the low cognitive ability group; the acquisition and extraction module is used to collect the first multimodal data of the first target group during the execution of the N-back task through a preset acquisition device, and perform preprocessing and feature extraction, and fuse the obtained key features into a target feature vector; the training module is used to introduce a balancing factor based on the target feature vector through a preset BDA balancing domain adaptation method, and adopt a K-fold cross-validation method to train the first classifier and the second classifier respectively, to obtain a first target evaluation model and a second target evaluation model; the fusion module is used to fuse the first target evaluation model and the second target evaluation model to obtain a comprehensive evaluation model for cognitive load assessment.

[0027] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a processor, a memory, and a program stored in the memory and executable on the processor, wherein when the program is executed by the processor, the method for assessing cognitive load for human-computer systems in the first aspect is implemented.

[0028] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium storing a program or instruction. When the program or instruction is executed by a processor, the method for assessing cognitive load for human-computer systems in the first aspect is implemented.

[0029] This application provides a method, system, device and medium for evaluating cognitive load of human-computer systems. Compared with the prior art, it has the following beneficial effects:

[0030] The present application first divides the preset first target group into a high cognitive ability group and a low cognitive ability group, and the first classifier and the second classifier correspond to the characteristics of the high cognitive ability group and the low cognitive ability group respectively. By collecting the first multimodal data of the first target group during the execution of the N-back task, and performing preprocessing and feature extraction, the obtained key features are fused into a target feature vector; the first multimodal data integrates a variety of physiological indicators to achieve a comprehensive and accurate evaluation of cognitive load; by introducing the balanced domain adaptation technology, the model can adapt to the needs of different individuals and tasks, and reduce dependence on specific data sets; the first classifier and the second classifier are trained respectively according to the characteristics of the high cognitive ability group and the low cognitive ability group, and the first target evaluation model and the second target evaluation model are obtained and fused; this method has the characteristics of cross-participant and cross-task, which significantly improves the scope of application and practicality of the evaluation, and at the same time enhances the versatility and ease of use of the model; it realizes the cognitive load evaluation of the human-computer system with multi-feature fusion, and provides users with a comprehensive, accurate and easy-to-operate evaluation tool. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0032] Figure 1 It is a flowchart of a cognitive load assessment method for a human-computer system provided in an embodiment of the present application;

[0033] Figure 2 It is a flowchart of another method for evaluating cognitive load of a human-computer system provided in an embodiment of the present application;

[0034] Figure 3 yes Figure 1 An exemplary flow chart of S130;

[0035] Figure 4 It is a schematic diagram of extracting feature dimensions of the comprehensive evaluation model provided in the embodiment of the present application;

[0036] Figure 5 is an exemplary experimental flow chart of comprehensive evaluation model data collection provided in the embodiments of the present application;

[0037] Figure 6 It is a structural diagram of a cognitive load assessment system for a human-computer system provided in an embodiment of the present application;

[0038] Figure 7 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0039] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention are clearly and completely described. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0040] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0041] The embodiments of the present application provide a method, system, device and medium for cognitive load assessment in human-machine systems, thereby solving the problems of large limitations, strong subjectivity and difficulty in deploying and testing cognitive load assessment in current human-machine systems.

[0042] The technical solution in the embodiment of the present application is to solve the above technical problems, and the overall idea is as follows:

[0043] The human-machine system refers to an overall system composed of two interacting and interdependent subsystems, human and machine, in order to achieve a certain predetermined purpose; the human-machine system is composed of humans and machines and relies on the interaction between humans and machines to complete certain functions. In modern production management and engineering technology design, it is a very important issue to rationally design the human-machine system so that it can function reliably and efficiently.

[0044] Cognitive load is a multidimensional structure that represents the load imposed on learners' cognitive systems when processing specific tasks. This multidimensional structure consists of the causal dimension that reflects the interaction between tasks and learners' characteristics and the evaluation dimension that reflects measurable concepts such as mental load, mental effort and performance.

[0045] In related technologies, cognitive load assessment in human-computer systems mainly includes the following methods:

[0046] (1) Currently, most studies on cognitive load assessment are limited to specific tasks or individuals. Many existing methods often produce good results only in specific tasks or individuals. It is difficult to conduct intelligent assessment across tasks or participants, and ignores the impact of individual differences on cognitive load assessment, such as differences in each person's baseline and physiological response. It is difficult to achieve intelligent assessment of cognitive load across participants and tasks in human-computer systems.

[0047] (2) Cognitive load assessment is carried out through subjective assessment methods, task performance methods, etc., which are complex and highly subjective. At the same time, both subjective assessment methods and task performance methods require results to be obtained after the task is executed, which cannot meet the needs of real-time assessment of cognitive load during the task state in human-computer systems.

[0048] (3) Collect basic data based on wearable devices such as EEG, and use machine learning methods to explore the correlation between cognitive load and EEG data, and construct a mapping relationship between EEG data and cognitive load. However, wearable data collection devices such as EEG have disadvantages such as being difficult to deploy, difficult to wear in human-machine systems, and causing interference to individuals. They have a certain impact on the normal operation of controllers and may affect the accuracy of the results. In addition, the experimental data is complex and requires professional analysis. In addition, changes in cognitive load will inevitably cause fluctuations in multiple physiological indexes. A single physiological technology or indicator is not accurate and has large errors. The established model is also relatively simple, especially for the lack of consideration of individual differences.

[0049] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0050] The following first introduces a cognitive load assessment method for a human-computer system provided in an embodiment of the present application.

[0051] The present invention provides a flowchart of a method for evaluating cognitive load of a human-computer system. Figure 1 and Figure 2 The cognitive load assessment method for human-computer systems may include the following steps S110-S160.

[0052] S110: Perform a cognitive ability test on a preset first target group, and divide the first target group into a high cognitive ability group and a low cognitive ability group according to the obtained cognitive test results and a preset classification threshold.

[0053] S120, constructing a first classifier and a second classifier corresponding to the high cognitive ability group and the low cognitive ability group.

[0054] S130, collecting first multimodal data of the first target group during the N-back task through a preset collection device, performing preprocessing and feature extraction, and fusing the obtained key features into a target feature vector.

[0055] S140. Based on the target feature vector, a balancing factor is introduced through a preset BDA balancing domain adaptation method, and a K-fold cross-validation method is used to train the first classifier and the second classifier respectively to obtain a first target evaluation model and a second target evaluation model.

[0056] S150, integrating the first target evaluation model and the second target evaluation model to obtain a comprehensive evaluation model for cognitive load evaluation.

[0057] In the embodiment of the present application, it should be noted that the first multimodal data involves two aspects: physiological data and behavioral data; the activity of the autonomic nervous system will affect the change of cognitive load, resulting in changes in the activities of cardiovascular and respiratory systems, and its electrocardiogram, electrodermal and facial motor unit characteristics are important features of cognitive load assessment, which is conducive to the realization of real-time and objective assessment of cognitive load by multi-feature fusion.

[0058] It can be understood that the present application first divides the preset first target group into a high cognitive ability group and a low cognitive ability group. The first classifier and the second classifier constructed by the present application correspond to the characteristics of the high cognitive ability group and the low cognitive ability group respectively, and collects the first multimodal data of the first target group during the execution of the N-back task. The first multimodal data integrates multiple physiological indicators; the present application preprocesses and extracts features of the first multimodal data, and fuses the extracted key features into a target feature vector.

[0059] Furthermore, this application adjusts the feature distribution of the source domain and the target domain by introducing balanced domain adaptation technology. BDA balances the distribution difference between the source domain and the target domain by introducing a balancing factor, controls the model's attention to the source domain and target domain data during training, enables the model to better generalize to new data, and improves the model's adaptability to new individual data; enables the model to adapt to the needs of different individuals and tasks, and reduces dependence on specific data sets.

[0060] This application trains the first classifier and the second classifier respectively according to the characteristics of the high cognitive ability group and the low cognitive ability group, obtains the first target evaluation model and the second target evaluation model and fuses them, and uses k-fold cross-validation and multi-task learning strategies to reduce the risk of overfitting and promote the model's learning of common features, further broadening the scope of application of the model.

[0061] It should also be noted that this application achieves accurate assessment of different cognitive levels by dividing the preset first target group into high and low categories according to the participants' cognitive abilities, and independently training models for each category. It has cross-participant and cross-task characteristics, which significantly improves the scope of application and practicality of the assessment, while enhancing the versatility and ease of use of the model; it realizes the cognitive load assessment of the human-computer system with multi-feature fusion, and provides users with a comprehensive, accurate and easy-to-operate assessment tool.

[0062] In some embodiments, regarding the aforementioned target feature vector, a balance factor is introduced through a preset BDA balance domain adaptation method, and a K-fold cross validation method is used to train the first classifier and the second classifier, respectively, to obtain a first target evaluation model and a second target evaluation model; that is, in the aforementioned S140, it can be understood that the present application conducted cognitive ability tests on multiple participants in the first target group, and divided the first target group into a high cognitive ability group and a low cognitive ability group according to the test results corresponding to each participant; further, the N-back task is performed on the high cognitive ability group and the low cognitive ability group included in the first target group, and the first multimodal data corresponding to the high cognitive ability group and the low cognitive ability group are collected and processed to obtain the target feature vector. The first target evaluation model and the second target evaluation model trained by the present application correspond to the characteristics of the high cognitive ability group and the low cognitive ability group, respectively. The present application conducts independent training for each type of data set to ensure that the model can accurately evaluate people with different cognitive levels.

[0063] It should be noted that the balance factor μ is a key parameter used to control the model's attention to the source domain and target domain data during training; the value of μ can be adjusted according to the similarity of the source domain and target domain data; when μ is close to 0, the model pays more attention to the source domain data; when μ is close to 1, the model pays more attention to the target domain data; in the process of training the model based on the target feature vector and the balance factor μ, the model learns how to predict the cognitive load state of the target domain data based on the feature distribution of the source domain data.

[0064] In one example, the first target evaluation model and the second target evaluation model are integrated to obtain a comprehensive evaluation model to perform cognitive load evaluation, that is, the aforementioned S160 may specifically include the following steps:

[0065] S210. Based on the cognitive test results of each participant in the first target group, a clustering method is used to perform similarity measurement on different individuals to obtain a similarity measurement result.

[0066] S220. Based on the similarity measurement result, different weights are assigned to the first target evaluation model and the second target evaluation model, and normalization is performed to obtain a comprehensive evaluation model for cognitive load evaluation.

[0067] It can be understood that the present application adopts an integrated learning strategy to combine the first target evaluation model and the second target evaluation model for high cognitive load levels and low cognitive load levels respectively to obtain a more comprehensive comprehensive evaluation model, which can effectively evaluate different cognitive load levels.

[0068] In some embodiments, after the first target evaluation model and the second target evaluation model are merged to obtain a comprehensive evaluation model for cognitive load evaluation, that is, after S160, the cognitive load evaluation method for a human-computer system may further include:

[0069] S310: non-invasively collect second multimodal data of a preset second target group during the N-back task by using a preset collection device.

[0070] S320, preprocessing and feature extraction are performed on the second multimodal data, and the extracted key features are input into the comprehensive evaluation model to perform performance evaluation on the comprehensive evaluation model.

[0071] In the embodiments of the present application, it can be understood that after obtaining the comprehensive evaluation model, the present application non-invasively collects second multimodal data of each participant in the second target group during the execution of the task in order to perform performance evaluation on the comprehensive evaluation model, and the second multimodal data is consistent with the aforementioned first multimodal data in data type; the second multimodal data is new participant data in the target domain, and the comprehensive evaluation model is tested and evaluated based on the second multimodal data. During the model evaluation process, indicators such as the model's accuracy, recall rate, and F1 score are obtained to judge the performance of the comprehensive evaluation model.

[0072] In some embodiments, the aforementioned preset acquisition device is capable of non-invasively acquiring the first multimodal data, the preset acquisition device includes an E4 wristband and a visible light camera, and the first multimodal data includes PPG signals, EDA signals, skin temperature signals, and visible light video information related to cognitive load.

[0073] In one example, after the first target evaluation model and the second target evaluation model are integrated to obtain a comprehensive evaluation model for cognitive load evaluation, that is, after S160, the cognitive load evaluation method for a human-computer system may further include:

[0074] S410, predicting data in the new data set based on the comprehensive evaluation model to obtain data between 0 and 1 to represent the probability of cognitive load.

[0075] S420, converting the obtained data between 0 and 1 into a binary classification result of the cognitive load assessment of the participant in the human-computer system, so as to make decisions and provide feedback in practical applications.

[0076] In the embodiments of this application, please refer to Figure 4 and Figure 5 It can be understood that before predicting the data in the new data set based on the comprehensive evaluation model, features such as facial AUs, eye behavior, PPG signals and skin temperature signals can be collected through the aforementioned preset collection device, and cognitive load-related features can be extracted; a new data set is obtained, and the new data set is input into the comprehensive evaluation model for cognitive load prediction, and a binary classification result is output.

[0077] In some embodiments, please refer to Figure 3 , the aforementioned method collects the first multimodal data of the first target group during the N-back task through a preset collection device, performs preprocessing and feature extraction, and fuses the obtained key features into a target feature vector, that is, the aforementioned S130 may specifically include the following steps:

[0078] S510 , performing noise reduction on the PPG signal by low-pass filtering using a preset second-order Butterworth filter to obtain a target PPG signal.

[0079] S520, using a low-pass filter of a fourth-order Butterworth filter to reduce noise on the EDA signal to obtain a target EDA signal.

[0080] S530, standardize the target PPG signal, the target EDA signal, the skin temperature signal, and the visible light video information to align the time and eliminate the dimension difference, so as to obtain the target skin temperature signal and the target visible light video information.

[0081] S540, extracting multiple key features of the participant's cognitive load in the task state based on the target PPG signal, the target EDA signal, the target skin temperature signal and the target visible light video information.

[0082] S550, performing high-dimensional feature fusion on the extracted multiple key features to generate a target feature vector.

[0083] In the embodiments of the present application, it can be understood that by standardizing the signal, the interference of individual differences can be reduced. The design goal of the Butterworth filter is to achieve a ripple-free pass band and a smooth attenuation curve in the frequency response; the target feature vector obtained by fusion integrates multiple information sources such as physiological signals and behavioral data. The model provides comprehensive data support, enhances its robustness in diverse tasks and individuals, and can comprehensively characterize the changing state of cognitive load.

[0084] In one example, in the process of standardizing the data, a Z-score standardization method is used to subtract the mean of each feature and divide the value by the standard deviation so that all features have the same mean and standard deviation.

[0085] In one example, the present application directly concatenates the standardized feature vectors into a long vector. The low cognitive ability group extracts 24 features from the target PPG signal, 2 features from the target EDA signal, 4 features from the target skin temperature signal, 17 eye movement features and 47 AU features from the target visible light video information, and the concatenated feature vector of the low cognitive ability group is a 94-dimensional vector.

[0086] The high cognitive ability group extracted 25 features from the target PPG signal, 3 features from the target EDA signal, 6 features from the target skin temperature signal, 18 eye movement features and 52 AU features from the target visible light video information. The concatenated feature vector of the high cognitive ability group is a 104-dimensional vector.

[0087] In one example, the transfer function H(s) of an n-th order Butterworth filter satisfies the expression:

[0088]

[0089] in, is the angular frequency of the signal, is the cutoff frequency of the filter and n is the order of the filter.

[0090] In some embodiments, in order to achieve continuous optimization and prediction of the model, the present application introduces an incremental learning mechanism so that the comprehensive evaluation model can be continuously updated and optimized according to new data to maintain the timeliness and accuracy of the model.

[0091] In one example, in the process of extracting key features from target PPG signals, target EDA signals, target skin temperature signals, and target visible light video information, a data-driven feature selection method can be used to automatically extract the most representative key features from multiple signal information through a machine learning algorithm, which can abandon the reliance on specific tasks or individual prior knowledge and enhance the model's adaptability to diverse scenarios. Among them, PPG stands for photoplethysmography, and EDA signals are used to characterize skin electrical activity.

[0092] In some embodiments, the target PPG signal features include: pulse period, pulse amplitude, pulse width, pulse rise time, heart rate, signal skewness, signal kurtosis, total power, frequency bandwidth and heart rate variability HRV. It can be understood that the target PPG signal is a standardized PPG signal; the frequency bandwidth refers to the effective frequency range of the signal, which is usually defined as the frequency range of more than 90% of the energy in the power spectrum.

[0093] In some embodiments, extracting key features from the target PPG signal may specifically include:

[0094] Detect the peak values ​​of all pulse waves in the target PPG signal and calculate the time difference between two adjacent peak values ​​to obtain the pulse period;

[0095] Find the peak value and the previous trough of each pulse wave, and calculate the difference between the peak value and the trough value to obtain the pulse amplitude;

[0096] Use interpolation and other methods to determine the 50% peak point of each pulse wave in the target PPG signal, and calculate the time difference between the 50% point of the rising edge and the 50% point of the falling edge of the pulse wave to obtain the pulse width;

[0097] Determine the starting point and peak point of the pulse wave, and calculate the time difference between the trough and the peak to obtain the pulse rise time; where the starting point corresponds to the trough of the signal;

[0098] The heartbeat peak is detected from the pulse wave in the target PPG signal, the heartbeat peak detected within a period of time is calculated, and the heartbeat peak is converted into the number of heartbeats per minute to obtain the heart rate; it can be understood that the process of a heartbeat is also the process of a peak, and the heart rate can be obtained by simply calculating the number of heartbeat peaks detected within a period of time and then converting the peak number into the number of heartbeats per minute;

[0099] Calculate signal skewness and signal kurtosis using target PPG signal data points;

[0100] The power spectrum density is obtained by fast Fourier transform, the power spectrum is obtained by power spectrum density calculation, and then the power spectrum is integrated over the entire frequency range to obtain the total power;

[0101] Calculate the frequency range covered by 90% of the total power to obtain the frequency bandwidth.

[0102] In one example, the total power calculation process satisfies the expression:

[0103]

[0104] in, is the frequency domain representation of the target PPG signal, is the total power of the target PPG signal.

[0105] In one example, the pulse period calculation process satisfies the expression:

[0106]

[0107] in, is the time point of the i+1th pulse wave peak, is the time point of the peak value of the i-th pulse wave, is the pulse period;

[0108] The calculation process of the pulse amplitude satisfies the expression:

[0109]

[0110] Where A is the pulse amplitude, is the peak amplitude of the pulse wave, is the amplitude of the trough corresponding to the pulse wave;

[0111] The calculation process of pulse width satisfies the expression:

[0112]

[0113] in, and They are the time points when the falling and rising edges of the pulse wave reach 50% of their peak values. is the pulse width;

[0114] The calculation process of pulse rise time satisfies the expression:

[0115]

[0116] in, is the time point of the pulse wave peak. It is the time of trough. is the pulse rise time;

[0117] The calculation process of heart rate satisfies the expression:

[0118]

[0119] in, is the time interval The number of pulse wave peaks within is the heart rate and the unit is: beats / minute;

[0120] The calculation process of signal skewness satisfies the expression:

[0121]

[0122] in, is the ith data point, is the mean of the signal, is the standard deviation of the signal, is the total number of data points of the signal, S is the signal skewness;

[0123] The calculation process of signal kurtosis satisfies the expression:

[0124]

[0125] in, is the ith data point, is the mean of the signal, is the standard deviation of the signal, is the total number of data points of the signal, is the signal kurtosis.

[0126] In some embodiments, the target EDA signal features include: the number of galvanic skin response peaks, the average amplitude of galvanic skin response peaks, the standard deviation of galvanic skin tension, and galvanic skin autocorrelation; wherein the galvanic skin autocorrelation is used to quantify the similarity of galvanic skin activity signals over time; the number of galvanic skin response peaks refers to the number of galvanic skin response peaks detected in the galvanic skin activity signal over a period of time, and the peaks are found by the derivative method, all peaks that meet the conditions are recorded, and the total number of peaks is calculated. The calculation process of the galvanic skin autocorrelation satisfies the expression:

[0127]

[0128] in, It is a time lag The autocorrelation coefficient of is the tth data point in the time series, is the first data points, is the mean of the time series, is the standard deviation of the time series, and N is the length of the time series.

[0129] It should be noted that the average peak amplitude of the galvanic skin response refers to the average amplitude of all detected galvanic skin response peaks within a period of time; to calculate the number of galvanic skin response peaks, it is necessary to first find the peak through the derivative method, and then calculate the average amplitude based on each detected galvanic skin response peak.

[0130] It should also be noted that the standard deviation of skin electrical tension refers to the standard deviation of the tension component of the skin electrical activity signal within a certain period of time; skin electrical tension represents the slowly changing part of skin conductance to characterize the basic skin conductance level, and is not directly affected by short-term stimulation; the standard deviation of skin electrical tension is calculated to meet the expression:

[0131]

[0132] Where N is the number of data points in the signal, is the value of the ith data point, is the mean of these data points, is the standard deviation of the skin electrical tension.

[0133] In some embodiments, the target skin temperature signal includes: skin temperature mean, skin temperature median, skin temperature maximum, skin temperature minimum, skin temperature standard deviation, skin temperature variance, skin temperature skewness and skin temperature kurtosis; it can be understood that the skin temperature signal is a statistical feature of skin temperature data related to cognitive load.

[0134] In some embodiments, it is understood that when cognitive load changes, a person's facial movement units and eye movements will change. The aforementioned target visible light video information includes facial behavior and eye movement features, and the eye movement features include glance features, gaze features, and blinking frequency; wherein the glance features correspond to a peak glance velocity.

[0135] In one example, the calculation process of the aforementioned peak scanning velocity includes:

[0136] S610: Analyze each frame of the target visible light video information to extract the inner and outer corner position information of the eye and the coordinates of the pupil center point.

[0137] S620: extracting a normalized scanning amplitude based on the inner and outer eye corner position information and the pupil center coordinates.

[0138] S630, subtract the normalized scanning amplitudes between two consecutive frames, and determine the maximum difference as the peak scanning speed.

[0139] Among them, the calculation process of the normalized scanning amplitude satisfies the expression:

[0140]

[0141]

[0142]

[0143] Where NSA is the normalized scan amplitude, S is the initial scan amplitude, and D is the distance between the inner and outer corners of the eye. , ) is the coordinate of pupil center, ( , ) is the coordinate of the outer corner of the eye, ( , ) is the inner eye corner coordinate.

[0144] In one example, in the process of extracting the facial AUs sequence and the eye gaze direction coordinate sequence, a preset OpenFace facial detection method is used to extract 17 AUs intensities and 12 eye gaze direction coordinates of each frame of a face image in a facial video. The 17 AUs intensities include: AU01_r, AU02_r, AU04_r, AU05_r, AU06_r,AU07_r, AU09_r, AU10_r, AU12_r, AU14_r, AU15_r, AU17_r, AU20_r, AU23_r, AU25_r, AU26_r, AU45_r; the 12 eye gaze direction coordinates include: gaze_0_x, gaze_0_y, gaze_0_z, gaze_1_x, gaze_1_y, gaze_1_z, gaze_h0_x, gaze_h0_y, gaze_h0_z, gaze_h1_x, gaze_h1_y, gaze_h1_z.

[0145] Among them, AUs are facial movement units, AU01_r represents the medial eyebrow levator muscle and inner eyebrow lifting; AU02_r represents the lateral eyebrow levator muscle and outer eyebrow lifting; AU04_r represents the contraction of the glabella and frowning of the brows; AU05_r represents the lifting of the upper eyelid; AU06_r represents the contraction of the zygomatic major muscle and the tightening of the eye socket / smiling; AU07_r represents the tightening of the lower eyelid; AU09_r represents the lifting of the nose; AU10_r represents the lifting of the upper lip; AU12_r represents the contraction of the zygomatic major muscle and the lifting of the lip corner / smiling; AU14_r represents the tightening of the lip corner; AU15_r represents the pulling down of the lip corner; AU17_r represents the pushing forward of the mandible; AU20_r represents the tightening of the lips; AU23_r represents the closing of the lips; AU25_r represents the opening of the lips; AU26_r represents the pulling down of the mandible; AU45_r represents the closing of the eyelid and blinking; gaze_0_x, gaze_0_y, gaze_0_z represents the components corresponding to the X-axis, Y-axis, and Z-axis of the left eye's gaze direction; gaze_1_x, gaze_1_y, gaze_1_z represent the components corresponding to the X-axis, Y-axis, and Z-axis of the right eye's gaze direction; gaze_h0_x, gaze_h0_y, gaze_h0_z represent the components corresponding to the X-axis, Y-axis, and Z-axis of the sight vector from the left eye pupil to the center of the head; gaze_h1_x, gaze_h1_y, gaze_h1_z represent the components corresponding to the X-axis, Y-axis, and Z-axis of the sight vector from the right eye pupil to the center of the head.

[0146] It should be emphasized that in order to verify the effectiveness of the cognitive load assessment method for human-computer systems proposed in this application, this application conducted a human-computer system cognitive load assessment research experiment in a laboratory environment. With the approval of the relevant ethics committee, facial videos, cognitive ability tests and contact physiological data of 22 subjects were collected; during the data processing, it was found that the data signals collected by the E4 wristbands of 5 subjects were interrupted, and finally the comprehensive data of 17 subjects were selected for analysis. In order to comprehensively consider the impact of the complexity of the task on the cognitive load level of the subjects, the entire experiment was divided into two stages: the baseline stage and the task stage. The E4 wristband physiological data and the video data based on the visible light camera were collected from the tasks in two different stages.

[0147] During the baseline phase, the subjects' personal characteristics are collected through scales, and the experimental process and rules are introduced to the subjects. At the same time, in order to eliminate the practice effect of cognitive ability tests, the subjects are also given cognitive ability test exercises to ensure the effectiveness of the formal experimental process.

[0148] During the task phase, cognitive load was assessed using a standardized working memory task, the N-back task, which can induce a controllable level of cognitive load. During the test, the subjects were asked to sit quietly, fill out a scale, and complete three tasks of different difficulty levels, namely 1-back, 2-back, and 3-back. The subjects' physical and psychological data were collected during the task, and the results of each task performance were recorded. The collection environment is as follows: Figure 1 As shown: artificial intelligence host, visible light camera, thermal infrared camera, triple screen, E4 wristband and paper quality meter. The collected data of the N-back task accuracy and reaction time of the subjects are shown in Table 1.

[0149] Table 1: N-back experimental data

[0150]

[0151] Combined with the data shown in Table 1, during the experiment, not only the classification results under different task conditions were analyzed, but also the performance of the model under cross-task and cross-participant conditions was evaluated. In the classification between 1-back and 2-back tasks, the average accuracy of the model reached 74%; the classification accuracy in 2-back and 3-back tasks was 75%; and between 1-back and 3-back tasks, the classification accuracy of the model further increased to 79%. These results show that the model can achieve relatively stable classification performance under different task conditions, especially under conditions with large differences in task complexity, such as 1-back and 3-back, the model's prediction performance has improved. In addition, cross-participant analysis shows that despite cognitive differences between different individuals, the model can still generalize well, indicating that it has strong robustness and adaptability.

[0152] In some embodiments, the present application provides a cognitive load assessment system 700 for a human-computer system. Figure 6 As shown, the cognitive load evaluation system 700 may include the following modules:

[0153] A test division module 710 is used to perform a cognitive ability test on a preset first target group, and divide the first target group into a high cognitive ability group and a low cognitive ability group according to the obtained cognitive test results and a preset classification threshold;

[0154] A construction module 720, for constructing a first classifier and a second classifier corresponding to a high cognitive ability group and a low cognitive ability group;

[0155] The collection and extraction module 730 is used to collect the first multimodal data of the first target group during the N-back task through a preset collection device, perform preprocessing and feature extraction, and fuse the obtained key features into a target feature vector;

[0156] A training module 740 is used to introduce a balancing factor based on the target feature vector by a preset BDA balancing domain adaptation method, and to train the first classifier and the second classifier respectively by a K-fold cross validation method to obtain a first target evaluation model and a second target evaluation model;

[0157] The fusion module 750 is used to fuse the first target evaluation model and the second target evaluation model to obtain a comprehensive evaluation model for cognitive load evaluation.

[0158] According to an embodiment of the present application, any multiple modules among the test partition module 710, the construction module 720, the acquisition and extraction module 730, the training module 740 and the fusion module 750 can be combined into one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module.

[0159] In some embodiments, the fusion module 750 may be specifically used to:

[0160] According to the cognitive test results of each participant in the first target group, the clustering method is used to measure the similarity of different individuals to obtain the similarity measurement results;

[0161] Based on the similarity measurement results, different weights are assigned to the first target evaluation model and the second target evaluation model, and normalized to obtain a comprehensive evaluation model for cognitive load evaluation.

[0162] In some embodiments, the collection and extraction module 730 may be specifically used to:

[0163] The PPG signal is denoised by low-pass filtering using a preset second-order Butterworth filter to obtain a target PPG signal;

[0164] Use low-pass filtering of a fourth-order Butterworth filter to reduce the noise of the EDA signal and obtain the target EDA signal;

[0165] The target PPG signal, target EDA signal, skin temperature signal and visible light video information are standardized to align the time and eliminate the dimension difference, so as to obtain the target skin temperature signal and target visible light video information;

[0166] Based on the target PPG signal, target EDA signal, target skin temperature signal and target visible light video information, multiple key features of the participants' cognitive load under the task state are extracted;

[0167] The extracted multiple key features are fused into high-dimensional features to generate the target feature vector.

[0168] Figure 6 The cognitive load assessment system 700 for a human-machine system shown has the corresponding effect of implementing the aforementioned cognitive load assessment method for a human-machine system, and for the sake of brevity, it will not be described in detail here.

[0169] In some embodiments, the present application provides an electronic device, the structure diagram of the electronic device is as follows Figure 7 shown.

[0170] The electronic device may include a processor 810 and a memory 820 storing computer program instructions.

[0171] Specifically, the processor 810 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0172] The memory 820 may include a large capacity memory for data or instructions. By way of example and not limitation, the memory 820 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive or a combination of two or more of these. In appropriate cases, the memory 820 may include a removable or non-removable (or fixed) medium. In appropriate cases, the memory 820 may be inside or outside the integrated gateway disaster recovery device. In a specific embodiment, the memory 820 is a non-volatile solid-state memory.

[0173] The memory 820 may include a read-only memory (ROM), a random access memory (RAM), a disk storage medium device, an optical storage medium device, a flash memory device, an electrical, optical or other physical / tangible memory storage device. Therefore, generally, the memory 820 includes one or more tangible (non-transitory) computer-readable storage media (e.g., a memory device) encoded with software including computer executable instructions, and when the software is executed (e.g., by one or more processors), it can perform the operations described in any one of the above-mentioned embodiments of the cognitive load assessment method for human-computer systems.

[0174] The processor 810 implements any one of the cognitive load assessment methods for human-machine systems in the above embodiments by reading and executing computer program instructions stored in the memory 820 .

[0175] In one example, the electronic device may further include a communication interface 830 and a bus 800. Figure 7As shown, the processor 810, the memory 820, and the communication interface 830 are connected via a bus 800 and communicate with each other.

[0176] The communication interface 830 is mainly used to implement communication between various modules, devices, units and / or equipment in the embodiments of the present application.

[0177] Bus 800 includes hardware, software or both, and the parts of online data flow billing equipment are coupled to each other. For example, but not limitation, bus may include accelerated graphics port (AGP) or other graphics bus, enhanced industrial standard architecture (EISA) bus, front-end bus (FSB), hypertransport (HT) interconnection, industrial standard architecture (ISA) bus, infinite bandwidth interconnection, low pin count (LPC) bus, memory bus, micro channel architecture (MCA) bus, peripheral component interconnection (PCI) bus, PCI-Express (PCI-X) bus, serial advanced technology attachment (SATA) bus, video electronics standard association local (VLB) bus or other suitable bus or two or more of these combinations. In appropriate cases, bus 800 may include one or more buses. Although the present application embodiment describes and shows a specific bus, the application considers any suitable bus or interconnection.

[0178] In addition, in combination with the cognitive load assessment method for a human-machine system in the above embodiments, the present application embodiment may provide a computer storage medium for implementation. The computer storage medium stores computer program instructions; when the computer program instructions are executed by a processor, any one of the cognitive load assessment methods for a human-machine system in the above embodiments is implemented.

[0179] It should be clear that the present application is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present application is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present application.

[0180] The functional blocks shown in the above block diagram can be implemented as hardware, software, firmware or a combination thereof. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present application are programs or code segments that are used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link by a data signal carried in a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, optical fiber media, radio frequency (RF) links, etc. The code segment can be downloaded via a computer network such as the Internet, an intranet, etc.

[0181] It should also be noted that the exemplary embodiments mentioned in this application describe some methods or systems based on a series of steps or devices. However, this application is not limited to the order of the above steps, that is, the steps can be performed in the order mentioned in the embodiment, or in a different order from the embodiment, or several steps can be performed simultaneously.

[0182] Aspects of the present disclosure are described above with reference to the flowchart and / or block diagram of the method, device (system) and computer program product according to the embodiment of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device to produce a machine so that these instructions executed by the processor of the computer or other programmable data processing device enable the implementation of the function / action specified in one or more boxes of the flowchart and / or block diagram. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field programmable logic circuit. It can also be understood that each box in the block diagram and / or flowchart and the combination of boxes in the block diagram and / or flowchart can also be implemented by dedicated hardware that performs a specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions.

[0183] In summary, compared with the prior art, this application has the following beneficial effects:

[0184] 1. The present invention has achieved significant technical innovation and effect improvement in the field of cognitive load assessment of human-computer systems. First, the invention proposes an assessment method based on multi-feature fusion, which not only spans different participants and tasks, but also extracts facial action units and eye behavior features through facial videos, combines the PPG signal, EDA signal and skin temperature signal collected by the E4 wristband, and applies balanced domain adaptation technology. The constructed cognitive load intelligent assessment model has achieved good results across different tasks, with an accuracy rate of 74% between 1-back and 2-back tasks; 75% between 2-back and 3-back tasks; and 79% between 1-back and 3-back tasks. Compared with traditional assessment technologies that rely on a single data source, this method demonstrates the advantages of low cost, low interference and convenience, making the assessment of cognitive load more efficient and practical.

[0185] 2. The present invention adopts an innovative BDA method in model construction. By dividing the data set into high and low categories according to the participants' cognitive abilities and training models independently for each category, accurate assessment of different cognitive levels is achieved. The cross-participant and cross-task characteristics of this method significantly improve the scope of application and practicality of the assessment, while enhancing the versatility and ease of use of the model.

[0186] 3. In terms of system device construction, the present invention has created a comprehensive cognitive load assessment system through system integration. The system integrates multiple modules such as information collection, signal processing analysis, feature extraction and index fusion, cognitive load rating and interaction, and realizes the cognitive load assessment of the human-machine system with multi-feature fusion, providing users with a comprehensive, accurate and easy-to-operate assessment tool.

[0187] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A cognitive load assessment method for human-computer systems, characterized in that: include: Conducting a cognitive ability test on a preset first target group, and dividing the first target group into a high cognitive ability group and a low cognitive ability group according to the obtained cognitive test results and a preset classification threshold; Constructing a first classifier and a second classifier corresponding to the high cognitive ability group and the low cognitive ability group; The first multimodal data of the first target group during the N-back task is collected by a preset collection device, and preprocessed and feature extracted, and the obtained key features are fused into a target feature vector; Based on the target feature vector, a balancing factor is introduced by a preset BDA balancing domain adaptation method, and the first classifier and the second classifier are trained respectively by a K-fold cross validation method to obtain a first target evaluation model and a second target evaluation model; fusing the first target evaluation model and the second target evaluation model to obtain a comprehensive evaluation model for cognitive load evaluation; The first multimodal data includes a PPG signal, an EDA signal, a skin temperature signal, and visible light video information related to cognitive load; The first multimodal data of the first target group during the N-back task is collected by a preset collection device, and preprocessed and feature extracted, and the obtained key features are fused into a target feature vector, including: The PPG signal is subjected to noise reduction by low-pass filtering using a preset second-order Butterworth filter to obtain a target PPG signal; Using a low-pass filter of a fourth-order Butterworth filter to reduce noise on the EDA signal to obtain a target EDA signal; Standardizing the target PPG signal, the target EDA signal, the skin temperature signal, and the visible light video information to align time and eliminate dimensional differences, thereby obtaining a target skin temperature signal and target visible light video information; Extracting multiple key features of the participant's cognitive load in the task state based on the target PPG signal, the target EDA signal, the target skin temperature signal, and the target visible light video information; Performing high-dimensional feature fusion on the extracted multiple key features to generate a target feature vector; The target EDA signal features include: the number of peaks of the galvanic skin response, the average amplitude of the galvanic skin response peaks, the standard deviation of the galvanic skin tension, and the galvanic skin autocorrelation; The target skin temperature signal includes: skin temperature mean, skin temperature median, skin temperature maximum, skin temperature minimum, skin temperature standard deviation, skin temperature variance, skin temperature skewness and skin temperature kurtosis; The target visible light video information includes facial behavior and eye movement characteristics, and the eye movement characteristics include a glance feature, a gaze feature, and a blinking frequency; wherein the glance feature corresponds to a peak glance velocity; The calculation process of the peak scanning velocity includes: Analyze each frame of the target visible light video information to extract the inner and outer corners of the eye and the coordinates of the pupil center; Extracting a normalized scanning amplitude based on the inner and outer eye corner position information and the pupil center coordinates; Subtracting the normalized scanning amplitudes between two consecutive frames to determine the maximum difference as the peak scanning velocity; The calculation process of the normalized sweep amplitude satisfies the expression: Where, NSA is the normalized sweep amplitude, S is the initial sweep amplitude, and D is the distance between the inner and outer corners of the eye. , ) is the coordinate of pupil center, ( , ) is the coordinate of the outer corner of the eye, ( , ) is the inner eye corner coordinate.

2. The method for evaluating cognitive load of a human-computer system according to claim 1, wherein: The step of fusing the first target evaluation model and the second target evaluation model to obtain a comprehensive evaluation model for cognitive load evaluation includes: Based on the cognitive test results of each participant in the first target group, a clustering method is used to measure the similarity of different individuals to obtain a similarity measurement result; Based on the similarity measurement result, different weights are assigned to the first target evaluation model and the second target evaluation model, and normalization is performed to obtain a comprehensive evaluation model for cognitive load evaluation.

3. The method for evaluating cognitive load of a human-computer system according to claim 1, wherein: After fusing the first target evaluation model and the second target evaluation model to obtain a comprehensive evaluation model for cognitive load evaluation, the human-machine system-oriented cognitive load evaluation method further includes: non-invasively collecting second multimodal data of a preset second target group during the N-back task by using a preset collection device; The second multimodal data is preprocessed and features are extracted, and the extracted key features are input into the comprehensive evaluation model to perform performance evaluation on the comprehensive evaluation model.

4. The method for evaluating cognitive load of a human-computer system according to claim 1, wherein: After fusing the first target evaluation model and the second target evaluation model to obtain a comprehensive evaluation model for cognitive load evaluation, the human-machine system-oriented cognitive load evaluation method further includes: Predicting data in a new data set based on the comprehensive evaluation model to obtain data between 0 and 1 to represent the probability of cognitive load; The obtained data between 0 and 1 is converted into a binary classification result of the cognitive load assessment of the participant in the human-computer system to make decisions and provide feedback in practical applications.

5. The method for evaluating cognitive load of a human-computer system according to claim 1, wherein: The preset collection equipment includes an E4 wristband and a visible light camera.

6. The method for evaluating cognitive load of a human-computer system according to claim 5, characterized in that: The target PPG signal characteristics include: pulse period, pulse amplitude, pulse width, pulse rise time, heart rate, signal skewness, signal kurtosis, total power, frequency bandwidth and heart rate variability HRV.

7. A cognitive load assessment system for human-computer systems, characterized in that: include: A test division module, used to perform a cognitive ability test on a preset first target group, and divide the first target group into a high cognitive ability group and a low cognitive ability group according to the obtained cognitive test results and a preset classification threshold; A construction module, used to construct a first classifier and a second classifier corresponding to the high cognitive ability group and the low cognitive ability group; A collection and extraction module, used to collect the first multimodal data of the first target group during the N-back task through a preset collection device, perform preprocessing and feature extraction, and fuse the obtained key features into a target feature vector; A training module, used to introduce a balancing factor based on the target feature vector through a preset BDA balancing domain adaptation method, and adopt a K-fold cross-validation method to train the first classifier and the second classifier respectively to obtain a first target evaluation model and a second target evaluation model; A fusion module, used for fusing the first target evaluation model and the second target evaluation model to obtain a comprehensive evaluation model for cognitive load evaluation; The first multimodal data includes a PPG signal, an EDA signal, a skin temperature signal, and visible light video information related to cognitive load; The first multimodal data of the first target group during the N-back task is collected by a preset collection device, and preprocessed and feature extracted, and the obtained key features are fused into a target feature vector, including: The PPG signal is subjected to noise reduction by low-pass filtering using a preset second-order Butterworth filter to obtain a target PPG signal; Using a low-pass filter of a fourth-order Butterworth filter to reduce noise on the EDA signal to obtain a target EDA signal; Standardizing the target PPG signal, the target EDA signal, the skin temperature signal, and the visible light video information to align time and eliminate dimensional differences, thereby obtaining a target skin temperature signal and target visible light video information; Extracting multiple key features of the participant's cognitive load in the task state based on the target PPG signal, the target EDA signal, the target skin temperature signal, and the target visible light video information; Performing high-dimensional feature fusion on the extracted multiple key features to generate a target feature vector; The target EDA signal features include: the number of peaks of the galvanic skin response, the average amplitude of the galvanic skin response peaks, the standard deviation of the galvanic skin tension, and the galvanic skin autocorrelation; The target skin temperature signal includes: skin temperature mean, skin temperature median, skin temperature maximum, skin temperature minimum, skin temperature standard deviation, skin temperature variance, skin temperature skewness and skin temperature kurtosis; The target visible light video information includes facial behavior and eye movement characteristics, and the eye movement characteristics include a glance feature, a gaze feature, and a blinking frequency; wherein the glance feature corresponds to a peak glance velocity; The calculation process of the peak scanning velocity includes: Analyze each frame of the target visible light video information to extract the inner and outer corners of the eye and the coordinates of the pupil center; Extracting a normalized scanning amplitude based on the inner and outer eye corner position information and the pupil center coordinates; Subtracting the normalized scanning amplitudes between two consecutive frames to determine the maximum difference as the peak scanning velocity; The calculation process of the normalized sweep amplitude satisfies the expression: Where, NSA is the normalized sweep amplitude, S is the initial sweep amplitude, and D is the distance between the inner and outer corners of the eye. , ) is the coordinate of pupil center, ( , ) is the coordinate of the outer corner of the eye, ( , ) is the inner eye corner coordinate.

8. An electronic device, characterized in that: include: A processor, a memory, and a program stored in the memory and executable on the processor, wherein when the program is executed by the processor, the method for evaluating cognitive load for a human-machine system according to any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a program or instruction, and when the program or instruction is executed by a processor, the method for assessing cognitive load for a human-computer system as claimed in any one of claims 1 to 6 is implemented.

Images