Method for predicting man-machine interaction task load based on multi-modal signal

Through the regression analysis of multimodal signal data, a task load prediction model was constructed, which solved the problem that the existing technology could not capture the task load in real time and accurately, achieved high accuracy prediction of the task load level, and improved human-computer interaction System efficiency and user experience.

CN118940223APending Publication Date: 2024-11-12NANJING UNIV OF SCI & TECH
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202410929389.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-11
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

Existing human-computer interaction task load assessment methods cannot capture the operator's subtle physiological and emotional changes in complex interaction environments in real time and accurately, especially single physiological signals such as EEG or eye movement data cannot fully express the user's cognitive status and Load level.

Method used

Using multimodal signal data, the EEG, EEG and IEM equipment are integrated through the experimental operation platform, combined with regression analysis to construct the cumulative probability prediction formula and independent probability prediction formula of the task load level, and the multi-classified logistic regression model is used to construct the task load prediction. Model to realize real-time prediction of task load level.

Benefits of technology

It realizes high accuracy prediction of task load level, exceeds the limitation of single signal data, provides a more comprehensive and scientific basis for load management and optimization, and improves the efficiency and user experience of human-computer interaction system.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118940223A_ABST
    Figure CN118940223A_ABST
Patent Text Reader

Abstract

The invention discloses a man-machine interaction task load prediction method based on a multi-modal signal, and the method comprises the steps: (1) carrying out the load grade division of corresponding task difficulty, and obtaining a specific task load grade of a corresponding task through scale scoring after an experiment operation; (2) building an experiment operation platform; constructing an experimental operation platform fused with multi-modal physiological evaluation; (3) accurately extracting physiological indexes: extracting electroencephalogram, electro-oculogram and myoelectricity indexes related to task load according to the multi-modal physiological signal data collected by the experimental operation platform; (4) constructing a prediction model through regression analysis: performing regression analysis by taking the task load as a dependent variable and the physiological index as an independent variable, and constructing an accumulated probability prediction formula and an independent probability prediction formula of the task load level; and (5) verifying the prediction accuracy of the prediction model. According to the method, the load levels of the user in different task execution processes can be more accurately predicted, so that the efficiency of a man-machine interaction system and the user experience are improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of human-computer interaction, and in particular to a method for predicting human-computer interaction task load based on multimodal signals. Background Art

[0002] In most studies on HCI task load evaluation indicators, there are three basic methods: measures of task performance in primary and / or secondary tasks, subjective reports, and physiological indicators. Traditional HCI evaluation methods, such as questionnaires or simple behavioral observations, still have some methodological problems in subjective workload measurement: the intrusiveness of online or real-time scoring will distract the experimenter's cognitive resources, and the post-task scoring will be biased by the subject's memory recall. Although they provide some insights, they often fail to capture the subtle physiological and emotional changes that operators experience during fast and complex interactions. At present, physiological analysis can directly reflect the operator's physiological functions and monitor and evaluate the operator's human cognitive state in real time. Researchers use psychophysiological feedback devices such as skin conductance, electroencephalography, and electrocardiography to detect the user's cognitive state during task execution. However, in some cases, only EEG or eye movement data are used. It may not be possible to fully capture and explain the user's cognitive state and load level in a complex interactive environment. For example, EEG signals can reflect changes in brain activity, but may not provide information about user attention allocation or visual fatigue alone. Similarly, eye movement data can reflect the focus and frequency of visual attention but may not fully convey changes in load. Summary of the invention

[0003] The object of the present invention is to provide a method for predicting human-computer interaction task load based on multimodal signals.

[0004] To achieve the above object, the present invention provides the following technical solution: a method for predicting human-computer interaction task load based on multimodal signals, comprising the following steps:

[0005] (1) The load level of the corresponding task difficulty is divided into different levels. The specific task load level of the corresponding task is obtained through the scale score after the experimental operation;

[0006] (2) Experimental operation platform construction: construct an experimental operation platform integrating multimodal physiological assessment, which includes a test machine and a main test machine;

[0007] (3) Accurate extraction of physiological indicators: Extract EEG, EOG, and EMG indicators related to task load based on the multimodal physiological signal data collected by the experimental operation platform;

[0008] (4) Constructing a prediction model through regression analysis: Based on the EEG, EOG and EMG physiological signal data extracted in step (3), the task load is used as the dependent variable and the physiological index is used as the independent variable for regression analysis to construct a cumulative probability prediction formula and an independent probability prediction formula for the task load level;

[0009] (5) Verify the prediction accuracy of the prediction model.

[0010] Furthermore, step (1) specifically includes:

[0011] Corresponding to the load level division of task difficulty, a task difficulty test was conducted on multiple people. The subjects performed n difficult tasks respectively. After completing each difficult task, they filled in the National Aeronautics and Space Administration-Task Load Index (NASA-TLX). The obtained scale score data were averaged and divided into n load levels according to the scores to ensure that each load level had tasks assigned to it, n≥3.

[0012] Furthermore, step (2) specifically includes:

[0013] Step (2-1): Integrate and configure hardware in the test machine, integrate EEG, EOG, and EMG equipment, and ensure that the equipment can work together and capture multi-modal physiological signal data;

[0014] Step (2-2): Integrate and configure software in the main test machine, integrate a software system that can support human-computer interaction task simulation, which can allow researchers to change task parameters and settings, while achieving synchronous recording and visualization of physiological signal data, and can process and analyze the physiological signal data.

[0015] Furthermore, step (3) specifically includes:

[0016] Step (3-1): Execute the experimental task. In addition, multiple people perform the task operation on the experimental operation platform built in step (2). The subjects perform n difficult tasks respectively, and collect multimodal physiological signal data of the subjects during the task execution process;

[0017] Step (3-2): signal preprocessing, preprocessing the collected original multimodal physiological signal data, including filtering, denoising, and baseline correction to improve data quality;

[0018] Step (3-3): Feature extraction, extracting key physiological index data from the preprocessed data. For EEG data, extract the power spectrum density of different frequency bands, including Delta, Theta, Alpha, Low Beta, and High Beta; for electrooculogram data, calculate the electrooculogram response rate (EBR); for electromyography data, extract the root mean square (RMS) and rectified electromyography (IEMG) values;

[0019] Step (3-4): significance analysis, using SPSS software to perform significance tests and select indicators that show significant differences between different task load levels.

[0020] Furthermore, step (4) specifically includes:

[0021] Step (4-1): Import the extracted EEG, EOG and EMG physiological index data into SPSS software, define the physiological index as the independent variable and the task load level as the dependent variable in SPSS software;

[0022] Step (4-2): Perform descriptive statistical analysis on the physiological index values ​​to understand the data distribution and remove potential outliers to achieve data screening;

[0023] Step (4-3): Use the multi-classification logistic regression model and select multivariate logistic analysis in SPSS software. The model fitting information table and parameter estimation table are obtained in the SPSS software data analysis result report. If the significance P<0.05 in the model fitting information table, the model is significantly established;

[0024] Step (4-4): Probability formula establishment: Based on the parameter estimation table in step (4-3), the cumulative probability prediction formula (2) and independent probability prediction formula (3) of the task load level are constructed, where P is the cumulative probability, X is the task load level, and B is the probability of the task load level. i , B1, B2…B n The model parameters obtained from the parameter estimation table in step (4-3), X1, X2…X n Take the corresponding physiological index data and obtain the probability of the corresponding physiological index data under n task load levels under task execution according to the independent probability prediction formula (3);

[0025] The cumulative probability prediction formula is as follows:

[0026] logit(Py>i|x))=-B i +B1X1+B2X2+…+B n X n (1)

[0027]

[0028] The independent probability prediction formula is as follows:

[0029] P(y=i|x)=P(y≤i|x)-(y≤i-1|x) (3)

[0030] i=1,2,…,n;x is the independent variable X1,X2,X3…X n ; y is the multi-classification dependent variable.

[0031] Furthermore, step (5) specifically includes:

[0032] Substitute the physiological signal data collected during the task execution of step (3-1) into the specific independent probability prediction formula obtained in step (4-4) to obtain the predicted task load level, and compare it with the actual task load level to obtain the prediction accuracy of the model.

[0033] Furthermore, in step (1), n=3, a score less than 40 points is task load level 1, a score greater than 40 points and less than 55 points is task load level 2, and a score greater than 55 points is task load level 3.

[0034] Furthermore, the number of people in step (1) and step (3-1) is more than 20.

[0035] Compared with the prior art, the present invention has the following beneficial effects:

[0036] The present invention arranges task operation steps of different loads, systematically collects and processes multimodal physiological information of operators under different task loads, extracts multiple physiological indicators related to task load, including multiple frequency bands (Delta, Theta, Alpha, LowBeta, HighBeta, P300, P200) of EEG frequency band power spectral density (PSD), electrooculogram (EBR), and root mean square (RMS) and integrated electromyography (IEMG) of electromyographic response, performs regression analysis on the data, constructs a cumulative probability prediction formula and an independent probability formula for the task load level, clarifies the influence relationship between the physiological indicators and the task load, and transcends the limitation of relying on a single signal. By using the constructed regression model, the physiological indicators collected in real time can be input into the model to realize the prediction of the task load with high prediction accuracy, thereby providing a scientific basis for load management and optimization in human-computer interaction tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 It is a schematic diagram of a method for predicting human-computer interaction task load based on multimodal signals of the present invention.

[0038] Figure 2 It is a task flow chart after dividing the task load of the present invention.

[0039] Figure 3 2 is a diagram of a multimodal experimental operation platform in an embodiment of the present invention. DETAILED DESCRIPTION

[0040] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0041] like Figure 1 As shown, a method for predicting human-computer interaction task load based on multimodal signals includes the following steps:

[0042] (1) The load level of the corresponding task difficulty is divided into different levels. The specific task load level of the corresponding task is obtained through the scale score after the experimental operation;

[0043] (2) Experimental operation platform construction: construct an experimental operation platform integrating multimodal physiological assessment, which includes a test machine and a main test machine;

[0044] (3) Accurate extraction of physiological indicators: Extract EEG, EOG, and EMG indicators related to task load based on the multimodal physiological signal data collected by the experimental operation platform;

[0045] (4) Constructing a prediction model through regression analysis: Based on the EEG, EOG and EMG physiological signal data extracted in step (3), the task load is used as the dependent variable and the physiological index is used as the independent variable for regression analysis to construct a cumulative probability prediction formula and an independent probability prediction formula for the task load level;

[0046] (5) Verify the prediction accuracy of the prediction model.

[0047] like Figure 2As shown, in step (1), the task difficulty was manipulated according to the working memory load, and the task was designed to be similar to the auditory n-back task. The operator asked to count the number of short-circuited plates displayed in a specific color (red or blue, balanced between participants), and to press the correct count of four possibilities (2, 3, 4) with one hand. Participants started the trial by pressing the space bar. In addition, participants were required to complete each of the four stages of the working memory load task (four levels: no n-back, 0-back, 1-back, 2-back) once. After listening to a series of numbers (randomized between 2 and 50) at intervals of 2 seconds, they were instructed to repeat the nth to last (n-back) number in the sequence. The subjects underwent 8 sets of trials, each with 30 trials, for a total of 240 trials. 18 people were invited to take a task difficulty test and fill out the National Aeronautics and Space Administration-Task Load Index (NASA-TLX) scale. Scores of 0-40 were task load level 1, scores of 40-55 were task load level 2, and scores of 55-100 were task load level 3.

[0048] like Figure 3 It is a multimodal experimental operation platform in the present invention. The test machine is mainly responsible for simulating the human-computer interaction task, and the main test machine is mainly responsible for the collection and monitoring of physiological data. For step (2), it specifically includes:

[0049] Step (2-1): Integrate and configure hardware in the test machine, integrate Brain Products EEG, EOG acquisition system, Trigno Ecosystem EMG acquisition system and other equipment, and ensure that they can work together to capture multimodal physiological signals at the same time.

[0050] Step (2-2): Integrate and configure software in the main test machine, integrate software systems that can support complex human-computer interaction task simulations, such as E-prime3 and BrainVision Recorder and EMGworksAcquisition systems for simultaneous recording and visualization of physiological data. This system allows researchers to change task parameters and settings, while achieving simultaneous recording and visualization of physiological signal data, and can process and analyze the physiological signal data.

[0051] Step (3) specifically includes:

[0052] Step (3-1): Execution of the experimental task. Invite another 20 people to perform the task on the experimental operation platform integrating multimodal physiological assessment built in step (2). The subjects perform four difficult tasks respectively, and collect the physiological data of the operators during the task execution.

[0053] Step (3-2): Signal preprocessing, preprocess the collected raw data, and EEG, ERP and EOG analysis are processed using MATLAB R2019 software and EEGLAB 14.1.1b toolkit. First, preprocess the raw EEG signal. First, the electrode positioning is performed, the brain average value is used as the heavy reference value, and a 0.5-50Hz finite impulse response bandpass filter is used to run an independent component analysis on the data, and the independent components are classified and labeled to eliminate the main non-brain source signals such as electrooculography, electromyography, and electrocardiography. For vertical EOG data, blinks within each block are identified by an automatic program and then checked by visual inspection. EBR is defined as the ratio between the total number of blinks in a stage and the duration of the stage (in minutes). The raw sEMG signal is filtered with a sixth-order bandpass Butterworth filter (cutoff frequency 20-150Hz) to attenuate noise.

[0054] Step (3-3): Feature extraction, extract key physiological features from the preprocessed data. For EEG data, extract the power spectrum density of different frequency bands (such as Delta, Theta, Alpha, Low Beta, High Beta); for electrooculogram data, calculate EBR; for electromyography data, extract RMS and IEMG values.

[0055] Step (3-4): significance analysis, using SPSS software to perform significance tests and select indicators that show significant differences between different task load levels.

[0056] Step (4) specifically includes:

[0057] Step (4-1): Import the extracted data of physiological indicators such as EEG, EOG and EMG into SPSS software. Define the independent variables (physiological indicators) and dependent variables (task load level) in SPSS software.

[0058] Step (4-2): Perform descriptive statistical analysis on the indicator values ​​to understand the data distribution and remove potential outliers.

[0059] Step (4-3): Use the multi-classification logistic regression model and select multivariate logistic analysis in SPSS software. The model fitting information table and parameter estimation table can be obtained in the SPSS data analysis result report. If the significance P<0.05 in the model fitting information table, the model is significantly established;

[0060] Table 1 Parameter estimation table

[0061]

[0062] Step (4-4): Probability formula establishment: Based on the parameter estimation table in step (4-3), the cumulative probability prediction formula (2) and independent probability prediction formula (3) of the task load level are constructed, where P is the cumulative probability, X is the task load level, and B is the probability of the task load level. i , B1, B2…B n are the estimated values ​​in the parameter estimation table in step (4-3), X1, X2…X n Take the corresponding physiological index data and obtain the probability of the corresponding physiological index data under task load levels 1, 2, and 3 according to the independent probability prediction formula (3); (i = 1, 2, ..., n; x is the independent variable X1, X2, X3 ... X n ; y is a multi-classification dependent variable;)

[0063] The cumulative probability prediction formula is as follows:

[0064] logit(Py>i|x))=-B i +B1X1+B2X2+…+B n X n (1)

[0065]

[0066] The independent probability prediction formula is as follows:

[0067] P(y=i|x)=P(y≤i|x)-(y≤i-1|x) (3)

[0068] Substitute the estimated values ​​of the multi-mode signal model parameters into formula (3) to form a specific independent probability formula. For the convenience of subsequent formula expression, let a = Delta band PSD, b = Theta band PSD, c = Alpha band PSD, and d = EBR.

[0069] The cumulative probability prediction formula is as follows:

[0070] logit(P(y≤1))=ln(P(y≤1) / (1-P(y≤1))=24.663-2.117·a+4.610·b-6.264·c-11.276·d (4)

[0071] logit(P(y≤2))=ln(P(y≤2) / (1-P(y≤2))=28.725-2.117·a+4.610·b-6.264·c-11.276·d (5)

[0072] logit(P(y≤3))=ln(P(y≤3) / (1-P(y≤3))=30.720-2.117·a+4.610·b-6.264·c-11.276·d (6)

[0073] From the above cumulative probability formula, the independent probability prediction formula for different task load levels can be obtained as follows, where i is 1, 2, 3, B1=24.663, B2=28.725, B3=30.720.

[0074] logit(Py>i|x))=-B i +2.117·a-4.610·b+6.264·c+11.276·d (7)

[0075]

[0076] Step (5) specifically includes:

[0077] Substitute the physiological data collected during the task execution in step (3-1) into the specific independent probability prediction formula (9) obtained in step (4-4) to obtain the predicted task load level. Compared with the actual task load level, it is found that the prediction accuracy of the model is 77.78%.

[0078] The multimodal signal prediction model of the present invention aims to provide a more comprehensive and integrated task load assessment method by combining multiple physiological signals such as EEG, EOG and EMG. This method can more accurately predict the user's load level during the execution of different tasks, thereby improving the efficiency of the human-computer interaction system and the user experience.

[0079] The above shows and describes the basic principles, main features and advantages of the present invention. It should be understood by those skilled in the art that the present invention is not limited to the above embodiments. The above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention to be protected. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. A method for predicting human-computer interaction task load based on multimodal signals, characterized in that: The following steps are involved: (1) The load level classification corresponding to the task difficulty is obtained by scoring the scale after the experimental operation to obtain the specific task load level of the corresponding task; (2) Experimental operation platform construction: construct an experimental operation platform integrating multimodal physiological assessment, which includes a test machine and a main test machine; (3) Accurate extraction of physiological indicators: Extract EEG, EOG, and EMG indicators related to task load based on the multimodal physiological signal data collected by the experimental operation platform; (4) Constructing a prediction model through regression analysis: Based on the EEG, EOG and EMG physiological signal data extracted in step (3), the task load is used as the dependent variable and the physiological index is used as the independent variable for regression analysis to construct a cumulative probability prediction formula and an independent probability prediction formula for the task load level; (5) Verify the prediction accuracy of the prediction model.

2. A method for predicting human-computer interaction task load based on multimodal signals as claimed in claim 1, characterized in that: Step (1) specifically includes: Corresponding to the load level division of task difficulty, a task difficulty test was conducted on multiple people. The subjects performed n difficult tasks respectively. After completing each difficult task, they filled in the National Aeronautics and Space Administration-Task Load Index (NASA-TLX). The obtained scale score data were averaged and divided into n load levels according to the scores to ensure that each load level had tasks assigned to it, n≥3.

3. A method for predicting human-computer interaction task load based on multimodal signals as claimed in claim 2, characterized in that: Step (2) specifically includes: Step (2-1): Integrate and configure hardware in the test machine, integrate EEG, EOG, and EMG equipment, and ensure that the equipment can work together and capture multi-modal physiological signal data; Step (2-2): Integrate and configure software in the main test machine, integrate a software system that can support human-computer interaction task simulation, which can allow researchers to change task parameters and settings, while achieving synchronous recording and visualization of physiological signal data, and can process and analyze the physiological signal data.

4. A method for predicting human-computer interaction task load based on multimodal signals as claimed in claim 3, characterized in that: Step (3) specifically includes: Step (3-1): Execute the experimental task. In addition, multiple people perform the task operation on the experimental operation platform built in step (2). The subjects perform n difficult tasks respectively, and collect multimodal physiological signal data of the subjects during the task execution process; Step (3-2): signal preprocessing, preprocessing the collected original multimodal physiological signal data, including filtering, denoising, and baseline correction to improve data quality; Step (3-3): Feature extraction, extracting key physiological index data from the preprocessed data. For EEG data, extract the power spectrum density of different frequency bands, including Delta, Theta, Alpha, Low Beta, and High Beta; for electrooculogram data, calculate the electrooculogram response rate (EBR); for electromyography data, extract the root mean square (RMS) and rectified electromyography (IEMG) values; Step (3-4): significance analysis, using SPSS software to perform significance tests and select indicators that show significant differences between different task load levels.

5. A method for predicting human-computer interaction task load based on multimodal signals as claimed in claim 4, characterized in that: Step (4) specifically includes: Step (4-1): Import the extracted EEG, EOG and EMG physiological index data into SPSS software, define the physiological index as the independent variable and the task load level as the dependent variable in SPSS software; Step (4-2): Perform descriptive statistical analysis on the physiological index values ​​to understand the data distribution and remove potential outliers to achieve data screening; Step (4-3): Use the multi-classification logistic regression model and select multivariate logistic analysis in SPSS software. The model fitting information table and parameter estimation table are obtained in the SPSS software data analysis result report. If the significance P<0.05 in the model fitting information table, the model is significantly established; Step (4-4): Probability formula establishment: Based on the parameter estimation table in step (4-3), the cumulative probability prediction formula (2) and independent probability prediction formula (3) of the task load level are constructed, where P is the cumulative probability, X is the task load level, and B i , B1, B2…B n The model parameters obtained from the parameter estimation table in step (4-3), X1, X2…X n Take the corresponding physiological index data and obtain the probability of the corresponding physiological index data under n task load levels under task execution according to the independent probability prediction formula (3); The cumulative probability prediction formula is as follows: logit(P(y>i|x))=-B i +B1X1+B2X2+…+B n X n (1) The independent probability prediction formula is as follows: P(y=i|x)=P(y≤i|x)-(y≤i-1|x) (3) i=1,2,…,n;x is the independent variable X1,X2,X3…X n ; y is the multi-classification dependent variable.

6. A method for predicting human-computer interaction task load based on multimodal signals as claimed in claim 5, characterized in that: Step (5) specifically includes: Substitute the physiological signal data collected during the task execution of step (3-1) into the specific independent probability prediction formula obtained in step (4-4) to obtain the predicted task load level, and compare it with the actual task load level to obtain the prediction accuracy of the model.

7. A method for predicting human-computer interaction task load based on multimodal signals as claimed in claim 2, characterized in that: In step (1), n=3, a score less than 40 points is task load level 1, a score greater than 40 points and less than 55 points is task load level 2, and a score greater than 55 points is task load level 3.

8. The method for predicting human-computer interaction task load based on multimodal signals as claimed in claim 2, characterized in that: The number of people in step (1) and step (3-1) is 20 or more.

Citation Information

Patent Citations

  • VR system cognitive load quantification evaluation method

    CN109976530A

  • Multi-physiological-parameter mental load prediction method and device

    CN114305441A

  • N-Back task electroencephalogram test method based on multi-complexity speed reducer assembly scene

    CN115804601A

  • Human body balance evaluation method and system based on differentiated scenes

    CN118121187A