Multi-modal representation supported learner element cognitive competence evaluation method and multi-modal representation supported learner element cognitive competence evaluation system

Abstract

The invention discloses a multi-modal representation supported learner element cognitive ability evaluation method and system, and belongs to the field of model evaluation and prediction. The method comprises the following steps: firstly, constructing a learner meta-cognitive ability evaluation model consisting of a feature extraction module, an enhanced representation module, a multi-modal fusion module and a prediction module; and then, in combination with a fusion auxiliary task of multi-level cross-modal comparison predictive coding, training the learner meta-cognitive ability evaluation model according to a framework of combining semi-supervised learning and active learning, and performing learner meta-cognitive ability evaluation by using the trained learner meta-cognitive ability evaluation model. According to the method, multi-modal unified meta-cognitive quantitative representation can be effectively utilized, the dynamic dependence and long-term incidence relation among multiple modals can be accurately captured based on a cavity cross-modal attention mechanism, the model training efficiency is improved through a framework combining semi-supervised learning and active learning, and meta-cognitive ability evaluation is achieved.

Description

Technical Field

[0001] The present invention belongs to the field of model evaluation and prediction, and specifically relates to a learner metacognitive ability evaluation method and system supported by multimodal representation. Background Art

[0002] Metacognition is a key factor in learners' autonomous learning and development, and directly affects learning outcomes and knowledge transfer capabilities. In a modern educational environment, learners not only need to master knowledge content, but also need to effectively plan, monitor and regulate their own learning process. This understanding and control of one's own cognitive process, namely metacognitive ability, has a decisive influence on learning outcomes. However, existing research and practice still face the following key challenges in the assessment of metacognitive ability:

[0003] 1. Insufficient comprehensive utilization of multimodal information: Traditional assessments mainly rely on a single modality (such as questionnaires, texts, or behavioral records), which makes it difficult to fully capture the multidimensional metacognitive characteristics of learners in actual learning scenarios. In particular, metacognitive information at the neurophysiological level (such as EEG signals) has not been fully utilized.

[0004] 2. Limited dynamic representation capabilities: Existing evaluation methods are often based on analysis of static snapshots or single time points, ignoring the dynamic evolution of metacognitive processes and unable to effectively capture the temporal dependencies between learners’ behaviors, language, and EEG signals at different cognitive stages.

[0005] 3. Lack of a unified modal representation framework: Data of different modalities differ significantly in temporal resolution, feature space, and data structure. How to build a unified multimodal quantitative representation framework to support cross-modal analysis remains an unresolved problem. Summary of the invention

[0006] The purpose of the present invention is to solve the problems existing in the prior art and to provide a method and system for evaluating learners' metacognitive ability supported by multimodal representation. Through the collaborative modeling and unified representation of multimodal data, a metacognitive ability evaluation model is trained, which provides a new technical path for the quantitative evaluation of metacognitive ability.

[0007] The specific technical solutions adopted by the present invention are as follows:

[0008] In a first aspect, the present invention provides a method for evaluating learners' metacognitive ability supported by multimodal representation, comprising:

[0009] S1. Construct a learner metacognitive ability assessment model consisting of a feature extraction module, an enhanced representation module, a multimodal fusion module and a prediction module;

[0010] The feature extraction module is used to obtain multimodal data generated by learners during online learning, wherein the multimodal data includes EEG data recording EEG signal sequences, behavioral data recording learning behaviors, and text data recording online interactive discussion content; and perform time slice feature extraction on the three modes in the multimodal data respectively to obtain an EEG feature matrix, a behavioral feature matrix, and a text feature matrix composed of multimodal feature vectors of each time slice;

[0011] The enhanced representation module is used to respectively perform hidden state time series modeling on the EEG feature matrix, the behavior feature matrix and the text feature matrix through position encoding and then input into the recurrent neural network to obtain enhanced time series representations of the three modalities;

[0012] The multimodal fusion module is used to use the text modality and the behavioral modality as query modalities, and the EEG modality as the key modality and value modality, respectively, to perform multimodal fusion on the enhanced time series representations of different modalities based on the hollow cross-modal differential attention, to obtain the first modal fusion feature of the EEG modality and the text modality and the second modal fusion feature of the EEG modality and the behavioral modality; and then to obtain the final multimodal fusion result by splicing the first modal fusion feature and the second modal fusion feature through a multilayer perceptron;

[0013] The prediction module is used to map the multimodal fusion result into a learner's metacognitive ability evaluation label;

[0014] S2. Combined with the fusion auxiliary task of multi-level cross-modal contrast prediction coding, the learner metacognitive ability evaluation model is trained according to the framework of combining semi-supervised learning with active learning, and the learner metacognitive ability evaluation model is used to evaluate the learner's metacognitive ability; the total loss function used in training includes three parts, the first part is the negative log-likelihood loss generated by mutual estimation between modalities in the auxiliary task, the second part is the contrast prediction loss generated by reconstructing the enhanced time series representation of a single modality through the multimodal fusion result, and the third part is the cross entropy loss generated by the prediction module in the main classification task.

[0015] As a preferred embodiment of the above-mentioned first aspect, in the enhanced representation module, for the feature matrix of each modality, a position coding matrix is ​​first generated for each feature matrix through one-dimensional relative position coding, and the original feature matrix is ​​added to the position coding matrix to form a coding enhancement matrix, and then a recurrent neural network is used to model the temporal changes of hidden states on different time slices in the coding enhancement matrix, and the result of the recurrent neural network output that integrates spatial context and temporal dynamics is used as the enhanced time series representation.

[0016] As a preferred embodiment of the first aspect, in the multimodal fusion module, the text modality and the behavior modality are used as query modalities respectively, and the method for multimodal fusion based on hollow cross-modal differential attention is:

[0017] For the current query modality, the enhanced time series representation of the current query modality is augmented and mapped through the first projection matrix, and the mapping result is decomposed into two query matrices of the same dimension;

[0018] The enhanced time series representation of the EEG modality is augmented and mapped through the second projection matrix, and the mapping result is decomposed into two key matrices of the same dimension;

[0019] The enhanced time series representation of the EEG modality is mapped to equal dimensions through the third projection matrix, and the mapping result is used as the value matrix;

[0020] The two query matrices and the two key matrices are combined with the value matrix respectively, and are used as two different sets of queries, keys and values ​​for multimodal fusion by multi-head sliding window void attention. The two fusion results are subjected to a differential operation with adjustable weights to obtain the modal fusion features of the fused EEG modality and the current query modality.

[0021] As a preferred embodiment of the first aspect, in the prediction module, the multimodal fusion result is mapped into a learner's metacognitive ability evaluation label through a multilayer perceptron.

[0022] As a preferred embodiment of the above-mentioned first aspect, two tasks are constructed in the fusion auxiliary task of the multi-level cross-modal contrast prediction coding. The first task is to predict the enhanced time series representation of another modality using the enhanced time series representation of one modality through a mapping network; the second task is to reconstruct the enhanced time series representation of a single modality through the multimodal fusion results.

[0023] As a preferred embodiment of the first aspect, in S2, the method for training the learner metacognitive ability assessment model according to a framework combining semi-supervised learning and active learning is as follows:

[0024] S21, obtaining a labeled subset randomly labeled by an expert from the training sample set, and adding the remaining unlabeled samples to the unlabeled subset; using the labeled subset with annotations to perform preliminary training on the learner metacognitive ability assessment model;

[0025] S22. Use the learner metacognitive ability assessment model obtained through the latest training to predict the unlabeled samples in the unlabeled subset, calculate the cross entropy according to the predicted probability distribution output for each unlabeled sample, and perform a weighted summation of the cross entropy and the local sample density of the unlabeled sample in the EEG feature space as the recommended probability value of the unlabeled sample;

[0026] S23, extracting some unlabeled samples with the highest recommendation probability value from the unlabeled subset for expert labeling, adding the labeled samples to the labeled subset, and using the updated labeled subset to train the learner metacognitive ability assessment model;

[0027] S24, iteratively executing S22 and S23 until the trained learner metacognitive ability assessment model reaches a preset accuracy rate.

[0028] As a preferred embodiment of the above-mentioned first aspect, in S2, when the learner's metacognitive ability is evaluated using the trained learner's metacognitive ability evaluation model, the gradient of the attention matrix in the multimodal fusion module is calculated, the impact of different modalities on the learner's metacognitive ability is analyzed, and the analysis results for realizing attribution analysis are obtained.

[0029] In a second aspect, the present invention provides a learner metacognitive ability assessment model training system supported by multimodal representation, which comprises:

[0030] A training sample acquisition module is used to collect multimodal data generated by learners during online learning from the online learning platform and construct it into a training sample set; the multimodal data includes EEG data recording EEG signal sequences, behavioral data recording learning behaviors, and text data recording online interactive discussion content;

[0031] The model building module is used to build a learner metacognitive ability assessment model composed of a feature extraction module, an enhanced representation module, a multimodal fusion module and a prediction module; the feature extraction module is used to obtain the multimodal data generated by the learner during the online learning process, and perform time slice feature extraction on the three modes in the multimodal data respectively, to obtain an EEG feature matrix, a behavioral feature matrix and a text feature matrix composed of multimodal feature vectors of each time slice; the enhanced representation module is used to respectively input the EEG feature matrix, the behavioral feature matrix and the text feature matrix into a recurrent neural network for hidden state time series modeling after position encoding, to obtain the three The multimodal fusion module is used to use the text modality and the behavioral modality as the query modality, and the EEG modality as the key modality and the value modality, respectively, to perform multimodal fusion on the enhanced time series representations of different modalities based on the hollow cross-modal differential attention, to obtain the first modal fusion feature of the EEG modality and the text modality and the second modal fusion feature of the EEG modality and the behavioral modality; and then the first modal fusion feature and the second modal fusion feature are spliced ​​and then the final multimodal fusion result is obtained through a multilayer perceptron; the prediction module is used to map the multimodal fusion result into a metacognitive ability evaluation label of the learner;

[0032] A training module is used to combine the fusion auxiliary task of multi-level cross-modal contrast prediction coding, train the learner metacognitive ability evaluation model according to the framework of combining semi-supervised learning and active learning, and obtain a trained learner metacognitive ability evaluation model; wherein the total loss function used in training includes three parts, the first part is the negative log-likelihood loss generated by mutual estimation between modalities in the auxiliary task, the second part is the contrast prediction loss generated by reconstructing the enhanced time series representation of a single modality through the multimodal fusion result, and the third part is the cross entropy loss generated by the prediction module in the main classification task.

[0033] In a third aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method for assessing learners' metacognitive ability supported by multimodal representation as described in any one of the first aspects above is implemented.

[0034] In a third aspect, the present invention provides a computer electronic device comprising a memory and a processor;

[0035] The memory is used to store computer programs;

[0036] The processor is used to implement the learner metacognitive ability assessment method supported by multimodal representation as described in any one of the first aspects above when executing the computer program.

[0037] Compared with the prior art, the present invention has the following beneficial effects:

[0038] 1. This paper introduces EEG signals into metacognitive assessment for the first time, extracts neurophysiological characteristics of key frequency bands, combines behavioral and textual modalities, and constructs a multimodal unified metacognitive quantitative representation framework.

[0039] 2. The present invention proposes a void cross-modal attention mechanism that can accurately capture the dynamic dependencies and long-term correlations between multiple modalities.

[0040] 3. The present invention maximizes the mutual information between modalities and improves the fusion quality by designing multi-level contrast prediction coding. At the same time, it realizes the interpretability and attribution diagnosis of metacognitive evaluation through the attention gradient significance method.

[0041] 4. The present invention designs a framework that combines semi-supervised learning with active learning. By introducing active learning and label propagation technology, it can significantly reduce the cost of multimodal data annotation and improve model training efficiency.

[0042] 5. The training framework of the present invention is universal and can be extended to fields such as sentiment analysis and user behavior modeling, and can achieve efficient processing of long sequence data through hardware-friendly design. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 A schematic diagram of the steps of a method for assessing learners' metacognitive abilities supported by multimodal representations;

[0044] Figure 2 This is a schematic diagram of the module composition of the learner metacognitive ability assessment model;

[0045] Figure 3 A schematic diagram of the module composition of a learner metacognitive ability assessment model training system supported by multimodal representation;

[0046] Figure 4 A schematic diagram of the composition of a computer electronic device;

[0047] Figure 5 Flow chart of 6 step modules in an embodiment of the present invention;

[0048] Figure 6 This is a data interaction relationship diagram between the six step modules in an embodiment of the present invention. DETAILED DESCRIPTION

[0049] In order to make the above-mentioned purpose, features and advantages of the present invention more obvious and easy to understand, the specific implementation mode of the present invention is described in detail below in conjunction with the accompanying drawings. In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below. The technical features in each embodiment of the present invention can be combined accordingly without conflicting with each other.

[0050] In the description of the present invention, it should be understood that the terms "first" and "second" are only used for the purpose of distinguishing descriptions, and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features.

[0051] like Figure 1 As shown, in a preferred embodiment of the present invention, a method for assessing learners' metacognitive ability supported by multimodal representation is provided, which includes two steps S1 and S2.

[0052] S1. Construct a learner metacognitive ability evaluation model consisting of a feature extraction module, an enhanced representation module, a multimodal fusion module, and a prediction module. The structure of the learner metacognitive ability evaluation model is as follows: Figure 2 As shown in the figure, the specific functions of each module are as follows:

[0053] 1) The feature extraction module is used to obtain the multimodal data generated by learners during the online learning process, wherein the multimodal data includes EEG data recording EEG signal sequences, behavioral data recording learning behaviors, and text data recording online interactive discussion content; and perform time slice feature extraction on the three modalities in the multimodal data respectively to obtain an EEG feature matrix, a behavioral feature matrix, and a text feature matrix composed of multimodal feature vectors of each time slice.

[0054] It should be noted that data of different modalities have their own data specificity, so appropriate preprocessing and feature extraction methods can be selected for each data modality. EEG, behavior, and text are all mature data modalities. You can refer to the preprocessing and feature extraction practices of these modalities in the prior art and convert them into EEG feature matrices, behavior feature matrices, and text feature matrices according to the preset time slice step. Each vector in the matrix corresponds to a feature vector of a time slice.

[0055] 2) Enhanced representation module, which is used to perform hidden state time series modeling on the EEG feature matrix, behavioral feature matrix and text feature matrix respectively through position encoding and input into the recurrent neural network to obtain enhanced time series representation of each of the three modalities.

[0056] It should be noted that both positional encoding and recurrent neural networks (RNN) are prior art. Positional encoding is a technique that adds position information to an input sequence. It is mainly used to solve the problem that the model cannot directly process the position information of sequence data. By combining positional encoding with word embedding vectors, the model can retain the position information of word vectors, thereby improving the ability to understand the sequence. A recurrent neural network is a neural network used to process sequence data. It uses a hidden state to save information from previous time steps, thereby modeling long-term dependencies of sequence data. Each time step of the RNN receives an input, updates its hidden state, and then generates an output based on the hidden state. For the specific implementation of positional encoding and recurrent neural networks, please refer to the prior art, which will not be described in detail.

[0057] In an embodiment of the present invention, the specific process within the enhanced representation module is as follows: for the feature matrix of each modality, a position coding matrix is ​​first generated for each feature matrix through one-dimensional relative position coding, and the original feature matrix is ​​added to the position coding matrix to form a coding enhancement matrix, and then a recurrent neural network is used to model the temporal changes of hidden states on different time slices in the coding enhancement matrix, and the result of the recurrent neural network output that integrates spatial context and temporal dynamics is used as the enhanced time series representation.

[0058] 3) A multimodal fusion module is used to use the text modality and behavioral modality as query modalities, and the EEG modality as the key modality and value modality, respectively, to perform multimodal fusion on the enhanced time series representations of different modalities based on the void cross-modal differential attention, and obtain the first modal fusion features of the EEG modality and text modality and the second modal fusion features of the EEG modality and behavioral modality; then the first modal fusion features and the second modal fusion features are concatenated and the final multimodal fusion result is obtained through a multi-layer perceptron.

[0059] It should be noted that the hole cross-modal differential attention is an improved attention mechanism obtained by combining the sliding window hole attention (SWDA) and the differential operation in the present invention. Among them, the sliding window hole attention is an attention mechanism that combines the idea of ​​hole convolution in the prior art. It samples the key Key and value Value in a hole manner within the sliding window, thereby expanding the receptive field of the window attention without increasing the computational cost. This mechanism mainly utilizes the importance of local context, surrounds each query Query with a fixed-size window attention, and uses multiple stacked window attention layers to build a large receptive field. The differential operation is introduced into SWDA to further make the attention during the interaction of different modalities more sparse and precise.

[0060] In an embodiment of the present invention, in a multimodal fusion module, the text modality and the behavioral modality are used as query modalities respectively, and the method for multimodal fusion based on hollow cross-modal differential attention is:

[0061] For the current query modality, the enhanced time series representation of the current query modality is augmented and mapped through the first projection matrix, and the mapping result is decomposed into two query matrices of the same dimension;

[0062] The enhanced time series representation of the EEG modality is augmented and mapped through the second projection matrix, and the mapping result is decomposed into two key matrices of the same dimension;

[0063] The enhanced time series representation of the EEG modality is mapped to equal dimensions through the third projection matrix, and the mapping result is used as the value matrix;

[0064] The two query matrices and the two key matrices are combined with the value matrix respectively, and are used as two different sets of queries, keys and values ​​for multimodal fusion by multi-head sliding window void attention. The two fusion results are subjected to a differential operation with adjustable weights to obtain the modal fusion features of the fused EEG modality and the current query modality.

[0065] 4) A prediction module, used to map the multimodal fusion result into a metacognitive ability evaluation label of the learner. In an embodiment of the present invention, the specific mapping can be implemented by a multilayer perceptron.

[0066] S2. Combined with the fusion auxiliary task of multi-level cross-modal contrast prediction coding, the learner metacognitive ability evaluation model is trained according to the framework of combining semi-supervised learning with active learning, and the learner metacognitive ability evaluation model is used to evaluate the learner's metacognitive ability; the total loss function used in training includes three parts, the first part is the negative log-likelihood loss generated by mutual estimation between modalities in the auxiliary task, the second part is the contrast prediction loss generated by reconstructing the enhanced time series representation of a single modality through the multimodal fusion result, and the third part is the cross entropy loss generated by the prediction module in the main classification task.

[0067] It should be noted that contrastive learning belongs to the existing technology and is a learning method based on the similarities and differences between samples. The core of contrastive learning is to learn the effective representation of data by constructing positive examples (similar sample pairs) and negative examples (dissimilar sample pairs). The model is trained to maximize the similarity between positive example pairs and minimize the similarity between negative example pairs. Contrastive learning is particularly suitable for unsupervised or self-supervised learning scenarios. It trains the model by comparing the similarities and differences between positive and negative samples, thereby learning the effective representation of data.

[0068] In the present invention, the contrastive learning task is a fusion auxiliary task of multi-level cross-modal contrastive prediction coding. The specific contrastive learning task and loss function form can be designed according to the actual situation. In an embodiment of the present invention, two tasks need to be constructed in the above-mentioned fusion auxiliary task of multi-level cross-modal contrastive prediction coding. The first task is to predict the enhanced time series representation of another modality using the enhanced time series representation of one modality through a mapping network. The loss of this task is the negative log-likelihood loss generated by the mutual estimation between the modalities; the second task is to reconstruct the enhanced time series representation of a single modality through the multimodal fusion results. The loss of this task is the contrast prediction loss generated by the multimodal fusion results to reconstruct the enhanced time series representation of a single modality.

[0069] In an embodiment of the present invention, the method for training the learner metacognitive ability assessment model according to the framework combining semi-supervised learning and active learning can be implemented according to the following process:

[0070] S21, obtaining a labeled subset randomly labeled by an expert from the training sample set, and adding the remaining unlabeled samples to the unlabeled subset; using the labeled subset with annotations to perform preliminary training on the learner metacognitive ability assessment model;

[0071] S22. Use the learner metacognitive ability assessment model obtained through the latest training to predict the unlabeled samples in the unlabeled subset, calculate the cross entropy according to the predicted probability distribution output for each unlabeled sample, and perform a weighted summation of the cross entropy and the local sample density of the unlabeled sample in the EEG feature space as the recommended probability value of the unlabeled sample;

[0072] S23, extracting the part with the highest recommendation probability value from the unlabeled subset (the specific proportion can be optimized according to actual conditions) and labeling the unlabeled samples by experts, adding the labeled samples to the labeled subset, and using the updated labeled subset to train the learner metacognitive ability assessment model;

[0073] S24, iteratively executing S22 and S23 until the trained learner metacognitive ability assessment model reaches a preset accuracy rate.

[0074] It should also be noted that the learner metacognitive ability assessment method supported by multimodal representation in the above-mentioned embodiment can essentially be executed by a computer program or module. Figure 3 As shown, based on the same inventive concept, another preferred embodiment of the present invention further provides a learner metacognitive ability assessment model training system supported by multimodal representation, comprising:

[0075] The training sample acquisition module is used to collect multimodal data generated by learners during online learning from the online learning platform and the EEG device (worn on the learner's head during the learning process) and construct it into a training sample set; the multimodal data includes EEG data recording EEG signal sequences, behavioral data recording learning behaviors, and text data recording online interactive discussion content.

[0076] The model building module is used to build a learner metacognitive ability assessment model composed of a feature extraction module, an enhanced representation module, a multimodal fusion module and a prediction module; the feature extraction module is used to obtain the multimodal data generated by the learner during the online learning process, and perform time slice feature extraction on the three modes in the multimodal data respectively, to obtain an EEG feature matrix, a behavioral feature matrix and a text feature matrix composed of multimodal feature vectors of each time slice; the enhanced representation module is used to respectively input the EEG feature matrix, the behavioral feature matrix and the text feature matrix into a recurrent neural network for hidden state time series modeling after position encoding, to obtain the three The multimodal fusion module is used to respectively use the text modality and the behavioral modality as the query modality, and the EEG modality as the key modality and the value modality, and to perform multimodal fusion on the enhanced time series representations of different modalities based on the void cross-modal differential attention, so as to obtain the first modal fusion feature of the EEG modality and the text modality and the second modal fusion feature of the EEG modality and the behavioral modality; and then the first modal fusion feature and the second modal fusion feature are concatenated and the final multimodal fusion result is obtained through a multilayer perceptron; the prediction module is used to map the multimodal fusion result into a metacognitive ability evaluation label of the learner.

[0077] A training module is used to combine the fusion auxiliary task of multi-level cross-modal contrast prediction coding, train the learner metacognitive ability evaluation model according to the framework of combining semi-supervised learning and active learning, and obtain a trained learner metacognitive ability evaluation model; wherein the total loss function used in training includes three parts, the first part is the negative log-likelihood loss generated by mutual estimation between modalities in the auxiliary task, the second part is the contrast prediction loss generated by reconstructing the enhanced time series representation of a single modality through the multimodal fusion result, and the third part is the cross entropy loss generated by the prediction module in the main classification task.

[0078] It should be noted that the specific implementation of each module in the above-mentioned learner metacognitive ability assessment model training system supported by multimodal representation can refer to the learner metacognitive ability assessment method supported by multimodal representation described in S1 and S2 above.

[0079] Similarly, based on the same inventive concept, Figure 4 As shown, another preferred embodiment of the present invention further provides a computer electronic device corresponding to the learner metacognitive ability assessment method supported by multimodal representation provided in the above embodiment, which includes a memory and a processor;

[0080] The memory is used to store computer programs;

[0081] The processor is used to implement a learner metacognitive ability assessment method supported by multimodal representation in the above embodiment when executing the computer program.

[0082] In addition, the logic instructions in the above-mentioned memory can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention.

[0083] Therefore, based on the same inventive concept, another preferred embodiment of the present invention also provides a computer-readable storage medium corresponding to the learner metacognitive ability assessment method supported by a multimodal representation provided in the above embodiment, and a computer program is stored on the storage medium. When the computer program is executed by the processor, it can implement the learner metacognitive ability assessment method supported by a multimodal representation provided in the above embodiment.

[0084] It is understandable that the above storage medium may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage. The storage medium may also be a U disk, a mobile hard disk, a magnetic disk or an optical disk, etc., which can store program codes.

[0085] It is understandable that the above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0086] It should also be noted that those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working process of the system described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here. In the various embodiments provided in this application, the division of steps or modules in the system and method is only a logical function division, and there may be other division methods in actual implementation, such as multiple modules or steps can be combined or integrated together, and a module or step can also be split.

[0087] In addition, the learner metacognitive ability assessment method supported by the above-mentioned multimodal representation, in addition to assessing the learner's metacognitive ability, can also use the trained learner metacognitive ability assessment model to assess the learner's metacognitive ability, calculate the gradient of the attention matrix in the multimodal fusion module, analyze the impact of different modalities on the learner's metacognitive ability, and obtain the analysis results for implementing attribution analysis.

[0088] The following will use a specific embodiment to demonstrate the specific implementation and technical effects of the learner metacognitive ability assessment model training system supported by the multimodal representation, as well as the learner metacognitive ability assessment method supported by the multimodal representation described in S1 and S2.

[0089] Example

[0090] In this embodiment, the training process of the learner metacognitive ability assessment model supported by multimodal representation and the specific implementation process of the learner metacognitive ability assessment method supported by multimodal representation are demonstrated. Figure 5 As shown, in this embodiment, the method described in S1 and S2 is specifically implemented through 6 step modules, wherein module 1 and module 2 implement the feature extraction module and enhanced representation module in the learner metacognitive ability evaluation model, module 3 implements the multimodal fusion module in the learner metacognitive ability evaluation model, module 5 implements the prediction module in the learner metacognitive ability evaluation model, and the remaining module 4 is used to assist the training model, and module 6 is used to further implement scalable attribution analysis. The data interaction relationship of these 6 step modules is as follows: Figure 6 As shown, these six step modules are described below respectively.

[0091] Module 1: Metacognitive quantitative representation of EEG-behavior-text integration

[0092] In this module, the specific processing process is as follows:

[0093] 1.1 Multimodal Data Acquisition

[0094] The multimodal data generated by learners during the online learning process are collected by reading from the online learning platform or offline database. The multimodal data of each learner includes EEG data recording EEG signal sequences, behavioral data recording learning behaviors, and text data recording the content of online interactive discussions.

[0095] 1.2 Metacognitive Quantitative Representation by Integrating EEG, Behavior and Text Data

[0096] Aiming at the learners' metacognitive activities, this paper proposes a comprehensive quantitative representation from three aspects: EEG, behavior and interactive text. The specific methods are as follows:

[0097] (1) Quantitative representation of EEG data

[0098] EEG data reflects the learner's metacognitive processing, which mainly involves the prefrontal lobe, as well as the insula, parietal lobe, temporal lobe, and precuneus. The specific steps are as follows:

[0099] a) Channel extraction and feature extraction: Use a 32-channel EEG device to record EEG signal data. These channels correspond to the standard international 10-20 system, including the following channels: Fp1, Fp2, F3, F4, F7, F8, Fc1, Fc2, C3, C4, T7, T8, Cp1, Cp2, P3, P4, P7, P8, O1, O2, Fz, Cz, Pz, Oz, AF3, AF4, Fcz, CPz, TP7, TP8, PO3, PO4. The signal is processed in segments. The following feature types are extracted from the EEG signal of each time segment: power spectral density (PSD), frequency band energy ratio, mean, variance, peak, event-related potential (ERP), fractal dimension, and entropy value.

[0100] b) Feature integration and vectorization: Integrate multi-channel features into a low-dimensional vector of fixed length and encode it into an EEG matrix of fixed shape. The matrix shape is ,in is the number of time segments, is the feature dimension of each fragment (number of channels × number of features per channel).

[0101] (2) Quantitative representation of learning behavior data

[0102] Learning behavior data records learners’ operation behaviors on the online learning platform (such as “login”, “check-in”, “watch videos”, “complete homework”, etc.), which intuitively reflects the learning process. The specific steps are:

[0103] a) Behavior sequence coding: Collect the user's behavior logs on the platform and form a behavior sequence in chronological order. Time-series coding is performed on the behavior, and the behavior is converted into discrete identifiers of different categories (such as B1 for "login" and B2 for "watch video").

[0104] b) Behavior embedding generation: Use the Item2Vec algorithm to vectorize the behavior sequence and embed each behavior into a low-dimensional dense vector space. By operating on the entire behavior sequence, an embedding matrix of the learning behavior is generated. The final behavior embedding matrix has the shape of ,in is the length of the behavior sequence, It is the feature dimension of behavior embedding.

[0105] (3) Quantitative representation of interactive text data

[0106] The interactive text data comes from the learners’ comments in the discussion area of ​​the online learning platform (such as “this part of the concept is not clear”, “I think this algorithm can be used in practical problems”, etc.), reflecting the learners’ cognitive level and feedback information. The specific steps are:

[0107] a) Text segmentation and word embedding: Each interactive text is regarded as a "sentence", and it is segmented through BERT Tokenizer and divided into multiple "words" (Tokens).

[0108] b) Text vectorization: Use BERT to generate word vectors for each token. Construct a word embedding matrix for the text sequence. The matrix shape is ,in is the length of the text sequence, is the dimension of each word vector.

[0109] 1.3 Data alignment and unified encoding

[0110] a) Length alignment: comparison , choose the longest one as the alignment length . Fill the insufficient matrix with zeros so that all lengths are converted to .

[0111] b) Linear projection: The EEG, behavior, and text data are respectively transformed into a fixed-length matrix by multiplying them by a matrix with learnable parameters through a linear projection layer. The shape of the EEG signal quantization matrix is , the behavior sequence embedding matrix shape is , the interactive text embedding matrix shape is .in is the number of time slices, is the characteristic dimension of each fragment.

[0112] 1.4. The final output is as follows: : The quantization matrix of EEG signal, the shape is ; : The embedding matrix of the behavior sequence, with shape ; : The embedding matrix of the interactive text, the shape is ;in, Indicates the sample number.

[0113] Module 2: Single-mode time series feature extraction based on selective state space process

[0114] The input and output of this module are:

[0115] enter: , , ;

[0116] Output: Temporal feature matrix of EEG modality ( ), the temporal feature matrix of the behavioral mode ( ), the temporal feature matrix of language modality ( ), the shapes are .

[0117] Specifically, the processing process of this module is as follows:

[0118] Electroencephalogram (EEG) signals capture instantaneous changes in brain activity with millisecond-level timing accuracy. These data can not only reveal the activity state of the brain at a specific moment, but also track the dynamic changes of cognitive activities over time, thus providing valuable information for understanding the temporal characteristics of cognitive processes. Behavioral data records the performance of individuals in specific tasks and reflects the implementation and adjustment process of cognitive strategies. Through time series analysis of behavioral data, researchers can observe the cognitive dynamics of individuals during task execution and the transitions between different cognitive stages. Text modalities, including individuals' self-reports and communications, are also temporal. Text can not only reflect the content of an individual's thinking at a specific moment, but also reveal their thinking process and metacognitive monitoring capabilities. By analyzing the time series of text data, we can understand how individuals adjust their cognitive strategies and metacognitive monitoring over time, and how these adjustments affect their cognitive performance. Therefore, for these three types of unimodal data, we need to extract their temporal features in advance.

[0119] In order to enhance the perception ability of the model in the temporal dimension, the existing model not only considers the content but also the position difference. This embodiment uses one-dimensional relative position encoding to provide each sequence (EEG-behavior-text) element with its position in the sequence, so that the model can identify and distinguish elements at different positions in the sequence, and enhance the temporal capture ability, as shown in the following formula:

[0120]

[0121]

[0122] in, is the position index in the sequence, ranging from . is the dimension index, ranging from For each dimension, we use sine and cosine functions to characterize the temporal information of odd channel dimensions and even channel dimensions respectively. For each modality, the generated position encoding matrix (PE) is added to the embedded sequence of each modality data to obtain the sequence enhanced by time encoding:

[0123]

[0124] in, represents the original modal sequence, , Represents the position encoding matrix, with the same shape as same.

[0125] In order to further capture the temporal dependency characteristics of modal data, a recurrent neural network (RNN) is used to enhance the modal sequence. Perform hidden state modeling:

[0126]

[0127] in, Represents the modal sequence after temporal encoding enhancement. express The output hidden state sequence has the shape . .pass It can capture time series dependencies and effectively model the temporal features in the sequence, passing the contextual information to the hidden state to prevent the loss of modality-specific features in subsequent fusion.

[0128] After the above steps, the temporal features of each modality are enhanced, and finally the temporal feature matrix of the EEG modality is obtained ( ), the temporal feature matrix of the behavioral mode ( ), the temporal feature matrix of language modality ( ), the shapes are .

[0129] Module 3: Multimodal Fusion Method Based on Void Cross-modal Differential Attention

[0130] The input and output of this module are as follows:

[0131] Input: sequence of enhanced unimodal representations ( ), , representing the representation sequences of EEG, behavior and text modalities respectively.

[0132] Output: EEG-text fusion results : Representation after fusion of EEG and text modalities; EEG-behavior fusion results : Representation after fusion of EEG and behavioral modalities; Final fusion result : Representation after integrating EEG, text and behavioral modalities.

[0133] Specifically, the processing process of this module is as follows:

[0134] After obtaining the temporal unimodal representation of the EEG-text-behavior sequence, it is necessary to use appropriate modal fusion technology to comprehensively utilize brain activity, language and thinking content revealed by text data, and actual actions and reactions shown by behavioral sequences, so as to provide a comprehensive perspective to evaluate students' metacognitive state. However, there are great differences in the temporal resolution of EEG sequences, text sequences, and behavioral sequences. For example, the temporal resolution of EEG is extremely high, while the resolution of text and behavioral sequences is relatively low. As a result, the existing modal fusion technology has poor parameter utilization due to the poor alignment effect of different modal temporal sequences, and it is easy to capture a large number of noise features.

[0135] In order to solve the problem of large differences in temporal resolution between different modalities, this embodiment introduces a multimodal fusion method based on sliding window hole attention (SWDA). Unlike the traditional way of calculating attention for each element in the sequence, the present invention sparsely selects keys (K) and values ​​(V) in a sliding window centered on the query modality (Q), and then performs self-attention operations on these representative elements. This method explicitly satisfies the locality and sparsity properties and can effectively model long-term dependencies. Generally speaking, the query modality is a modality with lower temporal resolution, such as text and behavior sequences, while the keys and values ​​are modalities with higher temporal resolution, such as EEG data.

[0136] In order to further reduce the high noise problem when different modalities are fused, the present invention further uses differential enhancement based on the aforementioned multi-head sliding window empty attention to make the attention of different modal interactions more sparse and accurate. Differential enhancement requires first using a projection matrix to augment each modality that needs to participate in modal fusion. The augmentation mapping can be learned and The dimensions are , and After the augmented mapping Matrix and Matrix split. The dimension of the learnable matrix is Therefore, the multimodal fusion based on sliding window hole cross-modal differential attention can be expressed as follows:

[0137]

[0138]

[0139] Where: , are manually set parameters; is a learnable differential parameter used to balance the differential strength and remove the noise.

[0140] when When , the formula represents EEG ( ) and text ( ) is fused, and the output is , Representation after fusion of EEG and text modalities.

[0141] when When , the formula represents EEG ( ) and behavior ( ) is fused, and the output is , Represents the fusion of EEG and behavioral modalities.

[0142] Therefore, through the void cross-modal differential attention, the present invention finally obtains the EEG-text fusion result and the fusion results of EEG-behavior sequences .final, and After further splicing, it will be processed by the perception machine to obtain the final fusion result of the three modes. :

[0143]

[0144] Among them, [:] represents splicing.

[0145] Module 4: Fusion auxiliary task of multi-level cross-modal contrastive predictive coding

[0146] The input and output of this module are as follows:

[0147] Input: Enhanced unimodal representation ( ), , respectively represent the representation sequence of EEG, behavior and text modalities; modality fusion result ( ): The final fusion representation of EEG, text, and behavior sequences.

[0148] Output: Negative log-likelihood loss from the mutual estimation between modalities in the auxiliary task , the contrast prediction loss generated by reconstructing the enhanced time series representation of a single modality through multimodal fusion results The processing in this module is as follows:

[0149] The core of modal fusion is to perform feature mapping through inter-modal similarity. However, placing all modal interactions in the modal fusion link will put too much burden on modal fusion and fall into local optimality, and is not conducive to the retention of task-related information. It is necessary to additionally design some auxiliary tasks to the loss function to promote collaborative optimization and enhance the robustness of the model in actual teaching scenarios and its generalization ability across teaching scenarios.

[0150] For the modal representation pairs from multiple perspectives of the learner, although they appear to be independent sequences, there is actually a certain correlation between them. Therefore, the mutual information between modalities can be maximized to reduce the pressure during modal fusion and filter out modal-specific random noise that is irrelevant to the task, while retaining the invariant content of different modalities, encouraging the model to make more consistent predictions. The present invention uses contrastive prediction coding to maximize the mutual information between modalities, as shown in the following formula:

[0151]

[0152] in represents the batch size during network training, represents logarithmic calculation, is a mapping network (the present invention is implemented using an MLP network), represents the sequence of enhanced unimodal representations of the input, , represents the representation sequence of other modalities, In addition to The other two modes of . This loss function focuses on estimating the mode To Modal The predictive power of the model is optimized by negative log-likelihood, which can be used to maximize the mutual information between different modalities. By maximizing the similarity between modalities, reducing modality-specific random noise, and retaining task-related invariant information, the collaborative consistency between modalities is improved.

[0153] In addition, in order to enable the final fusion result to capture the invariant clues between the modalities and retain some of the modality-specific information of the original input, this embodiment needs to maximize the mutual information between the fusion result and the input modality, that is, to expect an opposite path to be based on the fusion representation Reconstruct the unimodal content. Therefore, this embodiment can calculate the comparative prediction coding loss as shown in the following formula:

[0154]

[0155] in, Represents the cosine similarity calculation function, It is a fusion representation and enhanced unimodal representation The similarity between . Representing modality Negative sample set, for a specific batch, modality The negative sample is the set of all samples in the current batch except the current sample. is the expected calculation function, which is used to average each sample in the current batch. By optimizing the contrastive prediction coding loss, the numerator (the similarity of positive samples) becomes larger and larger, and the denominator (the sum of the similarities of negative samples) becomes relatively smaller, thereby maximizing the mutual information between the fusion result and the single modal input.

[0156] In this embodiment, this auxiliary task is subsequently jointly trained with the label-based main task, so that the model can learn richer multimodal representations, thereby improving the performance of downstream metacognitive analysis tasks.

[0157] Module 5: Metacognitive ability assessment model training method based on active learning

[0158] The input and output of this module are as follows:

[0159] Input: Unlabeled dataset: Contains EEG ( ,Behavior( ),text( ) multimodal data; Partially labeled data: some samples randomly labeled by experts.

[0160] Output: Three-category results: Evaluate learners’ metacognitive abilities into three categories: “high”, “medium” and “low”.

[0161] The specific processing process of this module is as follows:

[0162] In the process of multimodal data annotation, both experts and teachers are faced with a huge workload. In the trimodal data (EEG, behavior, and text) involved in the present invention, the annotation process of each modality is highly specialized and abstract and complex. If it is completely dependent on experts for annotation, it will be very time-consuming and costly. In order to enable the present invention to be expanded to more modalities and a larger group of learners, the present invention adopts a framework that combines semi-supervised learning with active learning for training. Active learning selects the most informative samples for expert annotation, and propagates the labels to unlabeled samples through a label diffusion mechanism, thereby greatly reducing the total number of samples required for annotation.

[0163] Initially, all data is unlabeled. Experts randomly label a small number of samples, and the model is initially trained based on these samples. At this point, the model already has basic metacognitive ability assessment functions, but is still in an underfitting state. Subsequently, through multiple rounds of iterative training, the model is continuously optimized. Each round of iteration includes the following three steps:

[0164] First, we need to select the most uncertain samples predicted by the previous model, weight them by their density, and send them to experts for labeling. The density of the label can be estimated by the number of EEG samples within a specific distance corresponding to the label. The formula is as follows:

[0165]

[0166] in, For the model to sample The predicted probability of Represents the similarity (cross entropy) between the predicted value and the true label. Representation sample In the EEG feature space, the distance The number of samples within the range is used to estimate the sample density. Samples with higher sample density can bring greater information gain after annotation. Represents the weighted balance coefficient, which controls the weights of similarity and density in sample selection to obtain the optimal parameters. is the sample index. Represents the number of category labels. This paper takes the prediction of high, medium and low metacognitive levels as an example, that is, , represents the counting function, for distance Statistical distance is less than The number of samples can be estimated The density of. They are the EEG, behavioral sequence and text modalities. samples. Representation sample The probability of being sent to the expert for marking in the next round is calculated according to The values ​​are used to sort all samples in descending order, and the top 10% samples are selected to be sent to the next round of expert annotation. Indicates the model's response to the sample in the previous round The prediction results, and the samples with higher density can bring the greatest expected gain if they are correctly labeled by experts. EEG signals are selected as the distance statistics because they can provide the most accurate estimation of students' metacognitive state, which is in line with scientific basis.

[0167] The second step is expert annotation and label propagation. Experts follow the sample The probability distribution of the EEG data is used to score the samples with the highest probability values. After scoring, the model will propagate these data to the samples that are most similar to the EEG data through the nearest neighbor search algorithm for label diffusion. That is, each unlabeled sample finds the nearest labeled sample through the Euclidean distance in the EEG embedding space and unifies its own label with it.

[0168] The third step is to train on the data set. After label propagation, all samples in the data set will have their own labels. The model can be trained on the entire data set. After each training, the last prediction result is saved for the next round of expert labeling data selection, thus forming an iterative cycle. Continue to iterate until the model reaches a satisfactory accuracy and stop expert labeling.

[0169] The final metacognitive ability evaluation results of the model are:

[0170]

[0171] in, It represents the representation vector after the fusion of multimodal features, including the fusion features of three modalities: EEG, behavior, and text. MLP represents multi-layer perceptron. Represents the predicted value of the model (three-category results for assessing learners’ metacognitive abilities: high, medium, and low).

[0172] As mentioned above, in the process of model training according to the framework of combining semi-supervised learning with active learning, there is a fusion auxiliary task for combining multi-level cross-modal contrast prediction coding. Therefore, the total loss function used in training includes three parts. The first part is the negative log-likelihood loss generated by the mutual estimation between modalities in the auxiliary task. The second part is the contrast prediction loss generated by reconstructing the enhanced time series representation of a single modality through multimodal fusion results The third part is the cross entropy loss generated by the prediction module for the main classification task The total loss function can be expressed as:

[0173]

[0174] Therefore, this embodiment can effectively improve the annotation efficiency and model performance by intelligently selecting the most valuable data for annotation based on the above training method, while reducing manpower and time costs. This method also helps multimodal data fusion and provides an efficient, economical and accurate solution for metacognitive recognition.

[0175] After training, this embodiment can eventually obtain an evaluation model that can accurately evaluate the learner's metacognitive level.

[0176] Module 6: Metacognitive evaluation and attribution diagnosis driven by multidimensional data

[0177] In addition, the final evaluation model obtained by training can not only accurately evaluate the learner's metacognitive level, but also can be used to further explore the learner's metacognitive attribution scheme. Attribution analysis requires visualizing the attention matrix during different modal interactions and comparing the correlation with the metacognitive evaluation scale in the reasoning stage to evaluate the attribution of different metacognitions. This embodiment calculates the gradient significance of the attention layer as the attention attribution, and obtains the correct metacognitive attribution by comparing the gradients of different temporal positions of different encoders.

[0178] In attribution analysis, the chain rule is used to find the specific element , the calculation of its gradient can be expressed as follows:

[0179]

[0180] in, represents the loss function of model training, Represents the model label, Represents the layer-by-layer parameters of the neural network. Finally, a specific encoder can be obtained Corresponding to EEG encoder (RNN), text encoder (RNN), behavior sequence encoder (RNN) and EEG-text fusion module, specific elements in EEG-behavior sequence fusion module The gradient of Represents a sequence element The location of each A gradient matrix can be obtained. Each element in the matrix can approximately estimate the contribution of a specific input element to the output prediction. A larger gradient position means a greater contribution.

[0181] The attribution analysis of the model belongs to the prior art and will not be described in detail. The results of the attribution analysis include the following situations: The gradient value of each element is very large, while the gradient value of the text and behavior corresponding to the time series window has not changed significantly, which means that the student If there is physiological arousal related to metacognition, but it is not manifested in behavior or text, or there is a delay in the manifestation of this arousal, it is necessary to interview students or combine other teaching logs or classroom records for a more detailed metacognition assessment. The presence of a large gradient value means that the students' metacognition is expressed through explicit behavior. The information expressed by students in each modality can be used as an attribution exploration. With the help of text mining and sequence modeling, students' metacognitive attribution can be further explored. It is also possible that a large gradient value exists in the encoder in the modal fusion stage, while no obvious gradient value is seen in the single-modal encoder, which means that the two modalities involved in the fusion have a "resonance" at a specific time step. The two modalities can better explain the reasons for the changes in students' metacognitive levels through information complementarity. At this time, the changes in teaching elements before and after the specific time step of the two modalities should be compared to conduct a more specific attribution diagnosis.

[0182] By supplementing metacognitive assessment with attribution analysis, teachers and students on the online learning platform can be further assisted in adjusting their teaching and learning strategies.

[0183] In order to verify the effectiveness of the above-mentioned evaluation model training and evaluation method (referred to as the method of the present invention) in this embodiment, this embodiment conducts experiments on three public multimodal datasets (DEAP, MAHNOB-HCI, ASCERTAIN). The experiment uses Cognitive + Multi-SABiGRU (see Cai Y, Guo J, Yao H, et al. Evaluation model ofmetacognitive ability based on multi-channel self-attention and BiGRU[C] / / International Conference on Electronic Information Engineering and DataProcessing (EIEDP 2023). SPIE, 2023, 12700: 716-721.) and CAFNet (see Zhu Q, Zheng C, Zhang Z, et al. Dynamic Confidence-Aware Multi-Modal EmotionRecognition[J]. IEEE Transactions on Affective Computing, 2023.) as comparison methods to evaluate the effect of the method of the present invention from aspects such as metacognitive evaluation performance and modality fusion quality.

[0184] In this embodiment, the learner's metacognitive ability is classified and evaluated (divided into three categories: high, medium, and low). The evaluation indicators are classification accuracy (ACC) and weighted F1 score (F1-Score). The former is used to evaluate the overall accuracy of the model classification, and the latter is used to comprehensively evaluate the precision and recall of the classification, which is suitable for scenarios with uneven category distribution. The experimental results are shown in Table 1.

[0185] Table 1 Experimental results

[0186] Dataset method Classification Accuracy (ACC) Weighted F1-Score DEAP Cognitive + Multi-SABiGRU 87.3% 86.5% CAFNet 89.1% 88.4% Method of the present invention 91.2% 90.4% MAHNOB-HCI Cognitive + Multi-SABiGRU 83.8% 82.7% CAFNet 89.1% 88.4% Method of the present invention 90.6% 89.1% ASCERTAIN Cognitive + Multi-SABiGRU 85.5% 84.9% CAFNet 87.3% 86.8% Method of the present invention 89.6% 89.8%

[0187] It can be seen from Table 1 that the method of the present invention has obvious improvements in classification performance and modality fusion quality.

[0188] On the DEAP dataset, the classification accuracy of the proposed method reaches 91.2%, which is 3.9% higher than that of Cognitive + Multi-SABiGRU and 2.1% higher than that of CAFNet. The weighted F1 scores are also improved by 3.9% and 2.0% respectively.

[0189] On the MAHNOB-HCI dataset, the classification accuracy of the proposed method is 90.6%, which is 6.8% higher than that of Cognitive + Multi-SABiGRU and 1.5% higher than that of CAFNet.

[0190] On the ASCERTAIN dataset, the classification accuracy of the proposed method is 89.6%, which is 4.1% higher than that of Cognitive + Multi-SABiGRU and 2.3% higher than that of CAFNet.

[0191] It can be seen that the classification performance of the proposed method on all data sets is significantly better than that of the comparison methods, especially on the more difficult MAHNOB-HCI and ASCERTAIN data sets.

[0192] In addition, since the method of the present invention uses hollow cross-modal differential attention, it effectively solves the problem of temporal resolution differences between modalities and significantly improves the ability to capture inter-modal collaborative information. Compared with the cross-modal attention mechanism of CAFNet, the method of the present invention further enhances the modeling ability of long-term dependencies between modalities, making the fusion quality higher and ultimately performing better in classification performance.

[0193] Finally, the method of the present invention accurately captures the dynamic evolution characteristics of EEG, behavior and text modalities in the time dimension through selective state space modeling, effectively making up for the shortcomings of Cognitive + Multi-SABiGRU and CAFNet in dynamic time series modeling. In long time series data (such as ASCERTAIN), the method of the present invention shows a stronger ability to model time series dependence, thereby achieving higher classification performance.

[0194] The above-described embodiments are only some preferred implementations of the present invention, but are not intended to limit the present invention. A person skilled in the relevant technical field may make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, any technical solution obtained by equivalent replacement or equivalent transformation falls within the protection scope of the present invention.

Claims

1. A method for assessing learners' metacognitive ability supported by multimodal representation, characterized in that: include: S1. Construct a learner metacognitive ability assessment model consisting of a feature extraction module, an enhanced representation module, a multimodal fusion module and a prediction module; The feature extraction module is used to obtain multimodal data generated by learners during online learning, wherein the multimodal data includes EEG data recording EEG signal sequences, behavioral data recording learning behaviors, and text data recording online interactive discussion content; and perform time slice feature extraction on the three modes in the multimodal data respectively to obtain an EEG feature matrix, a behavioral feature matrix, and a text feature matrix composed of multimodal feature vectors of each time slice; The enhanced representation module is used to respectively perform hidden state time series modeling on the EEG feature matrix, the behavior feature matrix and the text feature matrix by position encoding and then inputting the matrix into the recurrent neural network to obtain enhanced time series representations of the three modalities; The multimodal fusion module is used to use the text modality and the behavioral modality as query modalities, and the EEG modality as the key modality and value modality, respectively, to perform multimodal fusion on the enhanced time series representations of different modalities based on the hollow cross-modal differential attention, to obtain the first modal fusion feature of the EEG modality and the text modality and the second modal fusion feature of the EEG modality and the behavioral modality; and then to obtain the final multimodal fusion result by splicing the first modal fusion feature and the second modal fusion feature through a multilayer perceptron; The prediction module is used to map the multimodal fusion result into a learner's metacognitive ability evaluation label; S2. Combined with the fusion auxiliary task of multi-level cross-modal contrast prediction coding, the learner metacognitive ability evaluation model is trained according to the framework of combining semi-supervised learning with active learning, and the learner metacognitive ability evaluation model is used to evaluate the learner metacognitive ability; The total loss function used in training consists of three parts: the first part is the negative log-likelihood loss generated by the mutual estimation between modalities in the auxiliary task; the second part is the contrast prediction loss generated by reconstructing the enhanced time series representation of a single modality through the multimodal fusion results; and the third part is the cross entropy loss generated by the prediction module for the main classification task.

2. The method for assessing learner metacognitive ability supported by multimodal representation as claimed in claim 1, characterized in that: In the enhanced representation module, for the feature matrix of each modality, a position coding matrix is ​​first generated for each feature matrix through one-dimensional relative position coding, and the original feature matrix is ​​added to the position coding matrix to form a coding enhancement matrix. Then, a recurrent neural network is used to model the temporal changes of hidden states on different time slices in the coding enhancement matrix, and the result of the recurrent neural network output that integrates spatial context and temporal dynamics is used as the enhanced time series representation.

3. The method for assessing learner metacognitive ability supported by multimodal representation as claimed in claim 1, characterized in that: In the multimodal fusion module, the text modality and the behavior modality are used as query modalities respectively, and the method for multimodal fusion based on hollow cross-modal differential attention is as follows: For the current query modality, the enhanced time series representation of the current query modality is augmented and mapped through the first projection matrix, and the mapping result is decomposed into two query matrices of the same dimension; The enhanced time series representation of the EEG modality is augmented and mapped through the second projection matrix, and the mapping result is decomposed into two key matrices of the same dimension; The enhanced time series representation of the EEG modality is mapped to equal dimensions through the third projection matrix, and the mapping result is used as the value matrix; The two query matrices and the two key matrices are combined with the value matrix respectively, and are used as two different sets of queries, keys and values ​​for multimodal fusion by multi-head sliding window void attention. The two fusion results are subjected to a differential operation with adjustable weights to obtain the modal fusion features of the fused EEG modality and the current query modality.

4. The method for assessing learner metacognitive ability supported by multimodal representation as claimed in claim 1, characterized in that: In the prediction module, the multimodal fusion result is mapped into a learner's metacognitive ability evaluation label through a multilayer perceptron.

5. The method for assessing learner metacognitive ability supported by multimodal representation as claimed in claim 1, characterized in that: Two tasks are constructed in the fusion auxiliary task of the multi-level cross-modal contrastive predictive coding. The first task is to predict the enhanced time series representation of another modality using the enhanced time series representation of one modality through a mapping network; the second task is to reconstruct the enhanced time series representation of a single modality through the multimodal fusion results.

6. The method for assessing learner metacognitive ability supported by multimodal representation as claimed in claim 1, characterized in that: In S2, the method for training the learner metacognitive ability assessment model according to the framework of combining semi-supervised learning with active learning is as follows: S21, obtaining a labeled subset randomly labeled by an expert from the training sample set, and adding the remaining unlabeled samples to the unlabeled subset; using the labeled subset with annotations to perform preliminary training on the learner metacognitive ability assessment model; S22. Use the learner metacognitive ability assessment model obtained through the latest training to predict the unlabeled samples in the unlabeled subset, calculate the cross entropy according to the predicted probability distribution output for each unlabeled sample, and perform a weighted summation of the cross entropy and the local sample density of the unlabeled sample in the EEG feature space as the recommended probability value of the unlabeled sample; S23, extracting some unlabeled samples with the highest recommendation probability value from the unlabeled subset for expert labeling, adding the labeled samples to the labeled subset, and using the updated labeled subset to train the learner metacognitive ability assessment model; S24, iteratively executing S22 and S23 until the trained learner metacognitive ability assessment model reaches a preset accuracy rate.

7. The method for assessing learner metacognitive ability supported by multimodal representation as claimed in claim 1, characterized in that: In S2, when the learner's metacognitive ability is evaluated using the trained learner's metacognitive ability evaluation model, the gradient of the attention matrix in the multimodal fusion module is calculated, the influence of different modalities on the learner's metacognitive ability is analyzed, and the analysis results for implementing attribution analysis are obtained.

8. A learner metacognitive ability assessment model training system supported by multimodal representation, characterized in that: include: A training sample acquisition module is used to collect multimodal data generated by learners during online learning from the online learning platform and construct it into a training sample set; the multimodal data includes EEG data recording EEG signal sequences, behavioral data recording learning behaviors, and text data recording online interactive discussion content; The model building module is used to build a learner metacognitive ability assessment model composed of a feature extraction module, an enhanced representation module, a multimodal fusion module and a prediction module; the feature extraction module is used to obtain the multimodal data generated by the learner during the online learning process, and perform time slice feature extraction on the three modes in the multimodal data respectively, to obtain an EEG feature matrix, a behavioral feature matrix and a text feature matrix composed of multimodal feature vectors of each time slice; the enhanced representation module is used to respectively input the EEG feature matrix, the behavioral feature matrix and the text feature matrix into a recurrent neural network for hidden state time series modeling after position encoding, to obtain the three The multimodal fusion module is used to use the text modality and the behavioral modality as the query modality, and the EEG modality as the key modality and the value modality, respectively, to perform multimodal fusion on the enhanced time series representations of different modalities based on the hollow cross-modal differential attention, to obtain the first modal fusion feature of the EEG modality and the text modality and the second modal fusion feature of the EEG modality and the behavioral modality; and then the first modal fusion feature and the second modal fusion feature are spliced ​​and then the final multimodal fusion result is obtained through a multilayer perceptron; the prediction module is used to map the multimodal fusion result into a metacognitive ability evaluation label of the learner; A training module is used to combine the fusion auxiliary task of multi-level cross-modal contrast prediction coding, train the learner metacognitive ability evaluation model according to the framework of combining semi-supervised learning and active learning, and obtain a trained learner metacognitive ability evaluation model; wherein the total loss function used in training includes three parts, the first part is the negative log-likelihood loss generated by mutual estimation between modalities in the auxiliary task, the second part is the contrast prediction loss generated by reconstructing the enhanced time series representation of a single modality through the multimodal fusion result, and the third part is the cross entropy loss generated by the prediction module in the main classification task.

9. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by the processor, the learner metacognitive ability assessment method supported by multimodal representation as described in any one of claims 1 to 7 is implemented.

10. A computer electronic device, characterized in that: including memory and processor; The memory is used to store computer programs; The processor is used to implement the learner metacognitive ability assessment method supported by multimodal representation as described in any one of claims 1 to 7 when executing the computer program.

Images