Multi-mode cognitive load detection method
By independently encoding and dynamically reconstructing multimodal data, and combining modal attention and sensitive encoding, the limitations of multimodal data representation are overcome, and the performance of cognitive load detection and individual perception ability are improved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- TSINGHUA UNIVERSITY
- Filing Date
- 2026-01-20
- Publication Date
- 2026-05-12
AI Technical Summary
Existing multimodal cognitive load detection methods have limitations in multimodal data representation, limited model performance, and insufficient multimodal fusion strategies.
By independently encoding multimodal data, modal weights are dynamically learned using a channel attention mechanism, and feature reconstruction is achieved by combining broadcast Hadamard product. The final features are generated through load-task-sensitive and individual-sensitive encoders, and finally, cognitive load levels are classified using a multilayer perceptron.
It improves the model's ability to represent multimodal data, enhances the performance of cognitive load detection tasks, and strengthens the perception of load tasks and individuals.
Smart Images
Figure CN122020459A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method for detecting multimodal cognitive load. Background Technology
[0002] Existing methods for detecting cognitive load mainly include subjective and objective measurement methods. Subjective measurement methods collect individuals' cognitive load experiences through scales. While these methods are low-cost and easy to implement, they lack immediate feedback and are significantly influenced by subjective factors. Objective measurement primarily relies on physiological and behavioral signals, using machine learning methods to map input signals to cognitive load levels, thereby predicting cognitive load levels. In recent years, with the development and widespread adoption of wearable devices, research on objective measurement methods for cognitive load has received extensive attention.
[0003] Existing methods for objective measurement of cognitive load can be divided into single-modal methods and multimodal methods according to the number of supported data modalities. Single-modal methods are based on single-modal data, such as brain signal EEG[2], photoplethysmography (PPG), electrocardiogram (ECG), and electrodermal response (EDA / GSR). Although single-modal methods are easier to implement and have made some progress, different modalities can provide useful clues for cognitive load detection, and single-modal methods cannot utilize the complementary information of different modalities in the cognitive load detection task. Multimodal cognitive load detection has received more attention in recent years, but existing multimodal cognitive load detection methods generally adopt simple feature splicing strategies for multimodal fusion, and the model's ability to represent multimodal data is limited. Achieving effective representation of multimodal data by the model and improving the performance of multimodal-based objective measurement methods for cognitive load remains a challenge. Summary of the Invention
[0004] The main objective of this invention is to provide a multimodal cognitive load detection method that addresses the limitations of existing technologies in multimodal data representation and the performance constraints of models in cognitive load detection tasks.
[0005] Another objective of this invention is to provide a multimodal cognitive load detection device.
[0006] The third objective of this invention is to provide an electronic device.
[0007] To achieve the above objectives, a first aspect of the present invention provides a method for detecting multimodal cognitive load, comprising: The target multimodal data is independently encoded to obtain different modal features after preliminary encoding; Based on the different modal features after initial encoding, the weights of each modality are dynamically learned using the channel attention mechanism, and the dynamic reconstruction of multimodal features is achieved by broadcasting the Hadamard product. The reconstructed feature vectors are input into the load-sensitive encoder and the individual-sensitive encoder respectively to generate task-sensitive features and individual-sensitive features, and finally generate the final features. Based on the final features, cognitive load levels are classified using a multilayer perceptron.
[0008] Optionally, before independently encoding the target multimodal data, the following steps are also included: The modal data in the CL-Drive dataset are divided into non-overlapping samples according to a 10-second window length; the corresponding load level labels are constructed using the subject's subjective load score of the corresponding modal signal based on the binary and tri-classification task categories of the load level. The segmented modal data were screened, and modal data with continuous missing values exceeding 2 seconds were excluded to obtain valid samples; All valid samples were divided using 10-fold cross-validation, with each portion serving as the test set and the remaining 9 portions forming the training set. The training set is used as the target multimodal data for independent encoding processing.
[0009] Optionally, the step of independently encoding the target multimodal data to obtain different modal features after preliminary encoding includes: The target multimodal data is encoded by a modal encoding module, which includes a modal encoder for initial encoding of each modality. The modal encoder consists of two parts: modal attention and dynamic fusion. It consists of three convolutional blocks with the same structure and a global average pooling layer. The layers in the convolutional block are as follows: a one-dimensional convolutional layer, a batch normalization layer, a linear rectified unit, a one-dimensional convolutional layer, a batch normalization layer, a linear rectified unit, and a max pooling layer. The encoding process is represented as follows:
[0010] in, This represents the input data corresponding to mode M. For the encoder corresponding to mode M, For the corresponding input The encoded features.
[0011] Optionally, based on the initially encoded modal features, the weights of each modality are dynamically learned using a channel attention mechanism, and the dynamic reconstruction of multimodal features is achieved through broadcasting the Hadamard product, including: Different modal features after initial encoding Each channel constitutes a feature. , is represented as:
[0012] Stack() is a stacking function used to combine the input as different channels into a multi-channel data set. These are the encoded features of EEG, ECG, EDA, and Gaze modal data, respectively. This indicates the four channels after stacking, each corresponding to... , This represents the feature resulting from the stacking of four modal features; Features The input is fed into the modal attention module to achieve dynamic modality reconstruction. The modal attention module includes dynamic weight learning and feature reconstruction based on dynamic weights. The specific process is as follows:
[0013]
[0014]
[0015] Where N is the dimension of the features after modal data encoding, ⊙ represents the broadcast Hadamard product, and Flatten() represents feature flattening. LN() represents the Sigmoid activation function, and LN() represents a linear layer. To modify the activation function of the linear unit, express The i-th component (i=1,2,3,4), express Mean (i=1,2,3,4), To learn the weights corresponding to the four modalities, This is the reconstructed feature vector.
[0016] Optionally, the reconstructed feature vectors are input into the load-sensitive encoder and the individual-sensitive encoder, respectively, to generate task-sensitive features and individual-sensitive features, and finally, the features are generated, including: The reconstructed feature vector Through load task sensitive encoder and individual sensitive encoder Separate encoding is used to decompose the input features into workload-sensitive and individual-sensitive features. The workload-sensitive encoder... and individual sensitive encoder For a perceptron with one hidden layer, the encoding process is represented as:
[0017] in, This represents the features encoded in the previous stage. and These are the task and the subject's sensitivity characteristics, respectively. The final features are generated based on task-sensitive features and subject-sensitive features. , is represented as:
[0018] in, This is the final feature.
[0019] Optionally, based on the final features, the cognitive load level is classified using a multilayer perceptron, including: Will eventually This will be used as input to the cognitive load level classification module, which will then classify the load levels. The cognitive load level classification module is a multilayer perceptron with two hidden layers, separated by a Dropout layer and a ReLU activation layer. The specific calculation process for the cognitive load level categories is as follows:
[0020] in, This is the classification result.
[0021] Optionally, the loss function for the multimodal cognitive load detection method is as follows:
[0022] in, For the category loss item, and These are respectively load-task-sensitive loss and individual-sensitive loss terms. To rebuild the losses, These are the weighting coefficients.
[0023] Optional, also includes: The classification loss term is the cross-entropy loss function, which is used to optimize the model's classification of cognitive load levels; The load-level task-sensitive loss term and the individual-sensitive loss term are used to achieve the optimization objective of making the sample distance between the corresponding load-level induced task and individual label as close as possible, and vice versa, thereby improving the model's perception of the load-level induced task and the individual. Among them, the load-level induced task label uses the load-level category label corresponding to the sample, and the individual label corresponds to different subject numbers.
[0024] Load task sensitive loss The definition of is:
[0025]
[0026]
[0027] in, For similarity measurement functions, For temperature parameters, For indicator functions, The vector of the k-th sample after being encoded by the load-sensitive encoder. This represents the task label induced by the load level corresponding to the k-th sample; Individual sensitive loss items Defined as:
[0028]
[0029]
[0030] Reconstruction losses Used to implement input features The decomposed load-task-sensitive coding and individual-sensitive coding features Reconstructing input features Reconstruction loss items Defined as mean square error function :
[0031] The reconstructed feature vector is obtained through a perceptron containing a linear layer. ,Right now .
[0032] To achieve the above objectives, a second aspect of the present invention provides a multimodal cognitive load detection device, comprising: The first module is used to independently encode the target multimodal data to obtain different modal features after preliminary encoding; The second module is used to dynamically learn the weights of each modality based on the initially encoded modal features using a channel attention mechanism, and to achieve dynamic reconstruction of multimodal features by broadcasting the Hadamard product. The third module is used to input the reconstructed feature vectors into the load-sensitive encoder and the individual-sensitive encoder respectively, generate task-sensitive features and individual-sensitive features, and generate the final features; The fourth module is used to classify cognitive load levels using a multilayer perceptron based on the final features.
[0033] To achieve the above objectives, a third aspect of this application provides an electronic device, including a processor and a memory; wherein the processor runs a program corresponding to the executable program code stored in the memory to implement the method described in the first aspect.
[0034] To achieve the above objectives, a fourth aspect of this application provides a non-transitory computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the method described in the first aspect.
[0035] The embodiments of the present invention have the following beneficial effects: (1) Multimodal cognitive load detection based on modal attention and load task / person sensitivity: The method of this invention realizes modal fusion based on modal reconstruction through modal attention, and improves the model's ability to represent multimodal data by adding load task / person sensitivity encoding and optimization, thereby improving the model's performance on cognitive load detection tasks.
[0036] (2) Dynamic reconstruction of multimodal features based on attention mechanism: Dynamic reconstruction of multimodal features based on attention mechanism formalizes the fusion of multimodal features into a multimodal feature reconstruction based on channel attention. It dynamically learns modal weights from the initial encoded features of different modalities to achieve modal association learning and modal importance perception. (3) Introduce sensitive learning strategies at the load task and individual levels to enhance the model’s ability to perceive load tasks and personnel: Introducing sensitive learning strategies at the load task and individual levels can help the model to better perceive load tasks and personnel, further enhance the model’s ability to represent multimodal data, and improve the model’s performance on load detection tasks. Attached Figure Description
[0037] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein: Figure 1 A flowchart of a multimodal cognitive load detection method provided in an embodiment of the present invention; Figure 2 This is a network architecture diagram of a multimodal cognitive load detection method provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of a modal encoder provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of the modal attention module provided in an embodiment of the present invention. Detailed Implementation
[0038] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.
[0039] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.
[0040] The following describes a multimodal cognitive load detection method and apparatus according to embodiments of the present invention with reference to the accompanying drawings. The present invention proposes a multimodal cognitive load detection method, which includes three main innovations: (1) a multimodal cognitive load detection model based on modal attention and load task / person sensitivity, realizing cognitive load detection based on effective representation of multimodal data; (2) dynamic reconstruction of multimodal features based on attention mechanism, improving the learning of intermodal correlation and modal fusion based on modal importance; (3) introducing sensitive constraints at the load task and individual levels, improving the model's perception of load tasks and personnel, and enhancing the model's representation ability of multimodal data.
[0041] like Figure 1 As shown, the method includes the following steps: S1, independently encode the target multimodal data to obtain different modal features after preliminary encoding.
[0042] In this embodiment of the invention, before independently encoding the target multimodal data, it is first necessary to determine the target multimodal data. In this application, the CL-Drive dataset is divided into training data and test data. The training set is used for model training, and the test set is used for performance evaluation of the trained model. Subsequently, the training set is used as the input information for the target multimodal data to be independently encoded. Figure 2 In the model shown, feedforward calculations are performed. (Refer to...) Figure 2 The method proposed in this invention mainly consists of three parts: a modal coding module, a workload / individual perception coding module, and a cognitive workload level classification module.
[0043] Specifically, the process of partitioning the CL-Drive dataset includes the following steps: First, the modal data in the CL-Drive dataset are divided into non-overlapping samples according to a 10-second window. The corresponding load level labels are constructed based on the binary and tri-class load level task categories, using the subject's subjective load score (1-9) of the corresponding modal signal (the construction method for binary task labels is: low:1-4, high:5-9, and the construction method for tri-class task labels is: low:1-3, medium:4-6, high:7-9).
[0044] Next, the segmented modal data were screened, and modal data with continuous missing values exceeding 2 seconds were excluded to obtain valid samples.
[0045] In one possible implementation, 2781 valid samples were ultimately obtained.
[0046] Finally, a 10-fold cross-validation method was used to divide all the valid samples, with each sample serving as the test set and the remaining 9 samples forming the training set.
[0047] After obtaining the target multimodal data, in step S1, the modal encoding module encodes the target multimodal data. In this embodiment of the invention, the modal encoding module includes a modal encoder for preliminary encoding of each modality. (Each modality uses an independent encoder) and modality attention dynamic fusion are two parts.
[0048] Specifically, the encoder used for preliminary modal encoding can employ a common data encoding network structure; in particular, this invention uses a VGG-style network (…). Figure 3 This invention uses a modal encoder ( ) as an example to illustrate the structure of a modal encoder. Figure 3 It consists of three convolutional blocks with the same structure and a global average pooling layer. The layers in the convolutional block are as follows: a one-dimensional convolutional layer, a batch normalization layer, a linear rectified unit (ReLU activation layer), a one-dimensional convolutional layer, a batch normalization layer, a linear rectified unit, and a max pooling layer.
[0049] In this embodiment of the invention, the encoding process is represented as follows:
[0050] in, This represents the input data corresponding to mode M. For the encoder corresponding to mode M, For the corresponding input The encoded features.
[0051] S2, based on the different modal features after initial encoding, uses the channel attention mechanism to dynamically learn the weights of each modality, and realizes the dynamic reconstruction of multimodal features through broadcast Hadamard product.
[0052] In this embodiment of the invention, the different modal features after preliminary encoding are... Each channel constitutes a feature. , is represented as:
[0053] Stack() is a stacking function used to combine the input as different channels into a multi-channel data set. These are the encoded features of EEG, ECG, EDA, and Gaze modal data, respectively. This indicates the four channels after stacking, each corresponding to... , This represents the feature resulting from the stacking of four modal features.
[0054] Then in S2, the features The input is fed into the modal attention module to achieve dynamic modality reconstruction. The modal attention module includes dynamic weight learning and feature reconstruction based on dynamic weights. The specific process is as follows:
[0055]
[0056]
[0057] Where N is the dimension of the features after modal data encoding, ⊙ represents the broadcast Hadamard product, and Flatten() represents feature flattening. LN() represents the Sigmoid activation function, and LN() represents a linear layer. To modify the activation function of the linear unit, express The i-th component (i=1,2,3,4), express Mean (i=1,2,3,4), To learn the weights corresponding to the four modalities, This is the reconstructed feature vector.
[0058] In one embodiment of the present invention, a schematic diagram of the modal attention module is shown below. Figure 4 As shown.
[0059] S3. Input the reconstructed feature vectors into the load-sensitive encoder and the individual-sensitive encoder respectively to generate task-sensitive features and individual-sensitive features, and generate the final features.
[0060] In this embodiment of the invention, the reconstructed feature vector Through load task sensitive encoder and individual sensitive encoder Separate encoding is used to decompose the input features into workload-sensitive and individual-sensitive features. It should be noted that the workload-sensitive encoder in this application... and individual sensitive encoder For a perceptron with one hidden layer, the encoding process is represented as:
[0061] in, This represents the features encoded in the previous stage. and These are the task and the subject's sensitivity characteristics, respectively.
[0062] Finally, based on task-sensitive features and subject-sensitive features, the final features are generated. , is represented as:
[0063] in, This is the final feature.
[0064] S4. Based on the final features, the cognitive load level is classified using a multilayer perceptron.
[0065] In this embodiment of the application, the final This will be used as input to the cognitive load level classification module, which will then classify the load levels. The cognitive load level classification module is a multilayer perceptron with two hidden layers, separated by a Dropout layer and a ReLU activation layer. The specific calculation process for the cognitive load level categories is as follows:
[0066] in, This is the classification result.
[0067] Furthermore, it should be noted that, in order to achieve model training, the embodiments of this application define the model's optimization objective function (loss function), specifically as follows:
[0068] in, For the category loss item, and These are respectively load-task-sensitive loss and individual-sensitive loss terms. To rebuild the losses, These are the weighting coefficients.
[0069] Specifically, the classification loss term is the cross-entropy loss function, which is used to optimize the model's classification of cognitive load levels.
[0070] The load-level task-sensitive loss term and the individual-sensitive loss term are used to achieve the optimization objective of making the sample distance between the corresponding load-level induced task and individual label as close as possible, and vice versa, thereby improving the model's perception of the load-level induced task and the individual. Among them, the load-level induced task label uses the load-level category label corresponding to the sample, and the individual label corresponds to different subject numbers.
[0071] Load task sensitive loss The definition of is:
[0072]
[0073]
[0074] in, For similarity measurement functions, This is the temperature parameter (set to 0.05 in the implementation). This is an indicator function (1 if the condition is true, 0 otherwise). For the first k The vectors obtained after each sample is encoded by the workload-sensitive encoder Indicates the first k The load level induced the task label for each sample.
[0075] Individual sensitive loss items Defined as:
[0076]
[0077]
[0078] Reconstruction losses Used to implement input features The decomposed load-task-sensitive coding and individual-sensitive coding features Reconstructing input features Reconstruction loss items Defined as mean square error function :
[0079] The reconstructed feature vector is obtained through a perceptron containing a linear layer. ,Right now .
[0080] In one embodiment of the present invention, during model training, an example of weight coefficients selected based on experience is... .
[0081] Experiments show that the method of this invention improves the model's ability to represent multimodal data and enhances its performance on the cognitive load detection task. Tables 1 and 2 show the accuracy and F1 score of binary and tri-class classification of cognitive load levels on the CL-Drive dataset, including the method of this invention and different comparative methods. The results listed in the tables verify the effectiveness of this invention on the cognitive load detection task.
[0082] Table 1. Load-level binary classification accuracy (F1) results (%) of different methods on the CL-Drive dataset
[0083] Table 2. Load-level tri-class classification accuracy (F1) results (%) of different methods on the CL-Drive dataset
[0084] This invention also provides a multimodal cognitive load detection device, which includes: The first module is used to independently encode the target multimodal data to obtain different modal features after preliminary encoding; The second module is used to dynamically learn the weights of each modality based on the initially encoded modal features using a channel attention mechanism, and to achieve dynamic reconstruction of multimodal features by broadcasting the Hadamard product. The third module is used to input the reconstructed feature vectors into the load-sensitive encoder and the individual-sensitive encoder respectively, generate task-sensitive features and individual-sensitive features, and generate the final features; The fourth module is used to classify cognitive load levels using a multilayer perceptron based on the final features.
[0085] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.
[0086] To implement the methods of the above embodiments, the present invention also provides an electronic device, which includes a memory and a processor; wherein the processor reads executable program code stored in the memory to run a program corresponding to the executable program code, so as to implement the various steps of the methods described above.
[0087] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.
[0088] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.
[0089] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.
Claims
1. A method for detecting multimodal cognitive load, characterized in that, include: The target multimodal data is independently encoded to obtain different modal features after preliminary encoding; Based on the different modal features after initial encoding, the weights of each modality are dynamically learned using the channel attention mechanism, and the dynamic reconstruction of multimodal features is achieved by broadcasting the Hadamard product. The reconstructed feature vectors are input into the load-sensitive encoder and the individual-sensitive encoder respectively to generate task-sensitive features and individual-sensitive features, and finally generate the final features. Based on the final features, cognitive load levels are classified using a multilayer perceptron.
2. The method according to claim 1, characterized in that, Before independently encoding the target multimodal data, the following steps are also included: The modal data in the CL-Drive dataset are divided into non-overlapping samples according to a 10-second window length; the corresponding load level labels are constructed using the subject's subjective load score of the corresponding modal signal based on the binary and tri-classification task categories of the load level. The segmented modal data were screened, and modal data with continuous missing values exceeding 2 seconds were excluded to obtain valid samples; All valid samples were divided using 10-fold cross-validation, with each portion serving as the test set and the remaining 9 portions forming the training set. The training set is used as the target multimodal data for independent encoding processing.
3. The method according to claim 2, characterized in that, The process of independently encoding the target multimodal data to obtain pre-encoded modal features includes: The target multimodal data is encoded by a modal encoding module, which includes a modal encoder for initial encoding of each modality. The modal encoder consists of two parts: modal attention and dynamic fusion. It consists of three convolutional blocks with the same structure and a global average pooling layer. The layers included in the convolutional block are, in order: a one-dimensional convolutional layer, a batch normalization layer, a linear rectified unit, a one-dimensional convolutional layer, a batch normalization layer, a linear rectified unit, and a max pooling layer. The encoding process is represented as follows: in, This represents the input data corresponding to mode M. For the encoder corresponding to mode M, For the corresponding input The encoded features.
4. The method according to claim 3, characterized in that, Based on the initially encoded modal features, the weights of each modality are dynamically learned using a channel attention mechanism, and the dynamic reconstruction of multimodal features is achieved through broadcasting the Hadamard product, including: Different modal features after initial encoding Each channel constitutes a feature. , is represented as: Stack() is a stacking function used to combine the input as different channels into a multi-channel data set. These are the encoded features of EEG, ECG, EDA, and Gaze modal data, respectively. This indicates the four channels after stacking, each corresponding to... , This represents the feature resulting from the stacking of four modal features; Features The input is fed into the modal attention module to achieve dynamic modality reconstruction. The modal attention module includes dynamic weight learning and feature reconstruction based on dynamic weights. The specific process is as follows: Where N is the dimension of the features after modal data encoding, ⊙ represents the broadcast Hadamard product, and Flatten() represents feature flattening. LN() represents the Sigmoid activation function, and LN() represents a linear layer. To modify the activation function of the linear unit, express The i-th component (i=1,2,3,4), express Mean (i=1,2,3,4), To learn the weights corresponding to the four modalities, This is the reconstructed feature vector.
5. The method according to claim 4, characterized in that, The reconstructed feature vectors are input into the workload-sensitive encoder and the individual-sensitive encoder, respectively, to generate task-sensitive features and individual-sensitive features, and finally, the features are generated, including: The reconstructed feature vector Through load task sensitive encoder and individual sensitive encoder Separate encoding is used to decompose the input features into workload-sensitive and individual-sensitive features. The workload-sensitive encoder... and individual sensitive encoder For a perceptron with one hidden layer, the encoding process is represented as: in, This represents the features encoded in the previous stage. and These are the task and the subject's sensitivity characteristics, respectively. The final features are generated based on task-sensitive features and subject-sensitive features. , is represented as: in, This is the final feature.
6. The method according to claim 5, characterized in that, Based on the final features, the cognitive load level is classified using a multilayer perceptron, including: Ultimately This will be used as input to the cognitive load level classification module, which will then classify the load levels. The cognitive load level classification module is a multilayer perceptron with two hidden layers, separated by a Dropout layer and a ReLU activation layer. The specific calculation process for the cognitive load level categories is as follows: in, This is the classification result.
7. The method according to claim 6, characterized in that, The loss function of the multimodal cognitive load detection method is as follows: in, For the category loss item, and These are respectively load-task-sensitive loss and individual-sensitive loss terms. To rebuild the losses, These are the weighting coefficients.
8. The method according to claim 7, characterized in that, Also includes: The classification loss term is the cross-entropy loss function, which is used to optimize the model's classification of cognitive load levels; The load-level task-sensitive loss term and the individual-sensitive loss term are used to achieve the optimization objective of making the sample distance between the corresponding load-level induced task and individual label as close as possible, and vice versa, thereby improving the model's perception of the load-level induced task and the individual. Among them, the load-level induced task label uses the load-level category label corresponding to the sample, and the individual label corresponds to different subject numbers. Load task sensitive loss The definition of is: in, For similarity measurement functions, For temperature parameters, For indicator functions, The vector of the k-th sample after being encoded by the load-sensitive encoder. This represents the task label induced by the load level corresponding to the k-th sample; Individual sensitive loss items Defined as: Reconstruction losses Used to implement input features The decomposed load-task-sensitive coding and individual-sensitive coding features Reconstructing input features Reconstruction loss items Defined as mean square error function : The reconstructed feature vector is obtained through a perceptron containing a linear layer. ,Right now MSE() represents the mean squared error function.
9. A multimodal cognitive load detection device, characterized in that, include: The first module is used to independently encode the target multimodal data to obtain different modal features after preliminary encoding; The second module is used to dynamically learn the weights of each modality based on the initially encoded modal features using a channel attention mechanism, and to achieve dynamic reconstruction of multimodal features by broadcasting the Hadamard product. The third module is used to input the reconstructed feature vectors into the load-sensitive encoder and the individual-sensitive encoder respectively, generate task-sensitive features and individual-sensitive features, and generate the final features; The fourth module is used to classify cognitive load levels using a multilayer perceptron based on the final features.
10. An electronic device, characterized in that, Including processor and memory; The processor reads executable program code stored in the memory to run a program corresponding to the executable program code, so as to implement the method as described in any one of claims 1-8.