A cognitive diagnosis method and model based on emotional state
By extracting and deeply fusing multimodal features from teaching videos, this method addresses the limitations of existing cognitive diagnostic methods when dealing with students experiencing emotional fluctuations. It enables accurate and dynamic tracking of students' cognitive states, thereby improving the diagnostic effectiveness of personalized teaching and online learning platforms.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- GUANGDONG UNIV OF TECH
- Filing Date
- 2025-07-23
- Publication Date
- 2026-05-12
AI Technical Summary
Existing cognitive diagnostic methods are insufficient to fully reflect students' true cognitive state during the learning process. In particular, when faced with emotional fluctuations or changes in attention, the diagnostic results are one-sided. Furthermore, the heterogeneity of multimodal information is strong, and there is a lack of effective fusion technology solutions.
A cognitive diagnostic method based on affective state is adopted. By extracting multimodal features from teaching videos, performing feature alignment and modality selection to form a supermodal state, and combining self-supervised contrastive learning and Transformer encoding, a deep fusion of affective semantics is achieved, which can dynamically predict students' mastery of each knowledge point.
It improves the accuracy and applicability of cognitive diagnosis, enabling real-time perception of changes in students' emotions and cognitive states, dynamic tracking of knowledge mastery levels, and is suitable for personalized teaching and online learning platforms.
Smart Images
Figure CN121148609B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of intelligent education technology, and in particular to a cognitive diagnostic method and model based on emotional state. Background Technology
[0002] Cognitive Diagnosis Models (CDMs), as an important component of intelligent education systems, are widely used in scenarios such as assessing students' knowledge mastery levels, recommending personalized learning paths, and providing feedback on learning outcomes. Traditional cognitive diagnosis methods largely rely on students' answer behavior data, combined with the relationship between questions and knowledge points, to infer students' mastery of each knowledge point. However, existing methods generally have the following shortcomings:
[0003] On the one hand, diagnostic methods that rely solely on the results of responses are difficult to fully reflect the students' true cognitive state during the learning process. In particular, when faced with a group of students with complex emotional fluctuations or changes in attention, traditional methods often fail to accurately capture potential cognitive biases, resulting in one-sided diagnostic results.
[0004] On the other hand, although multimodal information processing technology has made some progress in recent years, and some studies have attempted to combine language, facial expression or speech data to assist teaching analysis, the heterogeneity of multimodal information is strong. There is still a lack of mature and unified technical solutions on how to effectively align, integrate and utilize information from various modalities to improve the accuracy and practicality of cognitive diagnostic models.
[0005] Furthermore, students' emotional state is closely related to their cognitive performance. Numerous educational psychology studies have shown that emotional changes directly affect students' learning engagement, comprehension ability, and knowledge absorption level. Real-time acquisition and integration of students' multimodal emotional information will help improve the comprehensiveness and accuracy of cognitive diagnosis.
[0006] Therefore, there is an urgent need for a new technical solution that can integrate multimodal information, adaptively capture changes in students' emotional and cognitive states, dynamically predict students' mastery of various knowledge points, and improve the accuracy, interpretability, and applicability of cognitive diagnostic systems. Summary of the Invention
[0007] Therefore, it is necessary to provide a cognitive diagnosis method and model based on emotional state to address the technical problem of insufficient comprehensiveness in the above-mentioned cognitive diagnosis, which has the characteristics of accuracy and strong applicability.
[0008] To achieve the above-mentioned objectives of this invention, the technical solution adopted is as follows:
[0009] A cognitive diagnostic method based on emotional state includes the following steps:
[0010] Extract emotion-related multimodal features from teaching videos and obtain their corresponding knowledge point information;
[0011] The extracted multimodal features are projected onto a unified low-dimensional space to bring the data to the same dimension, thus preparing for subsequent fusion.
[0012] Based on the information content of each modality in the multimodal features, the primary modality and auxiliary modality are adaptively selected;
[0013] Through information exchange between the primary modality and the secondary modality, a supermodal state expressing emotional semantics is recursively updated and formed.
[0014] The principal and supermodal states are deeply fused, and a unified supermodal state representation of student emotional cognitive state is generated based on cognitive diagnosis of emotional state.
[0015] A cognitive diagnostic model based on emotional state includes a cascaded multimodal information feature extraction module, a feature alignment layer, a modality selection layer, an adaptive supermodal learning layer, and a cross-modal fusion and transformation layer;
[0016] The multimodal information feature extraction module is used to extract emotion-related multimodal features from teaching videos and obtain their corresponding knowledge point information;
[0017] The feature alignment layer is used to project the extracted multimodal features into a unified low-dimensional space, so that the data are in the same dimension, in preparation for subsequent fusion.
[0018] The modality selection layer is used to adaptively select the primary and secondary modalities based on the information content of each modality in the multimodal features;
[0019] The adaptive supermodal learning layer is used to recursively update and form a supermodal state that expresses emotional semantics through information interaction between the main modality and the auxiliary modality.
[0020] The cross-modal fusion conversion layer is used to deeply fuse the main modality and supermodal states, and generate a unified supermodal state representing the student's emotional cognitive state based on the cognitive diagnosis of emotional state.
[0021] Preferably, before extracting features, the multimodal information feature extraction module introduces a self-supervised contrastive learning mechanism to pre-train and optimize the original modal data; the mechanism guides the model to learn more discriminative embedding representations within the modality by constructing positive and negative sample pairs, providing robust initial features for subsequent modeling.
[0022] Furthermore, the multimodal information feature extraction module specifically includes: a BERT pre-trained language model for extracting text semantic features, an analyzer for extracting visual behavior based on the OpenFace system, and an audio feature extractor using the Librosa tool library.
[0023] Furthermore, the feature alignment layer includes a cross-modal semantic distillation module, which uses a pre-trained multimodal model as the teacher model to generate soft labels for intermodal alignment as supervision signals; through knowledge distillation loss, it constrains the consistency of the probability distribution of each modal feature in the student model after projection onto the low-dimensional space, so as to improve the semantic consistency of feature alignment.
[0024] Furthermore, the modality selection layer calculates the emotional information entropy of each modality through a long short-term memory network (LSTM) to assess its information content and determine the dominant and auxiliary modalities.
[0025] Furthermore, the modality selection layer further allocates modality weights based on uncertainty estimation. Specifically, a random dropout layer is set for the features of each modality, and the variance of the prediction results is calculated through multiple forward inferences during the training phase to quantify the uncertainty of each modality. The modality weights are then determined according to a pre-set uncertainty threshold or a weighted formula combining information entropy.
[0026] Furthermore, the adaptive supermodal learning layer includes at least two Transformer encoding modules and multiple supermodal information fusion layers. The Transformer encoding modules are used to extract deep semantic features of the main modality information in a hierarchical manner, and the supermodal information fusion layers are used to guide the information interaction between the auxiliary modality and the main modality to form a unified supermodal representation.
[0027] Furthermore, the cross-modal fusion conversion layer introduces learnable additional labels and positional encodings between the source modal sequence and the target modal sequence, and combines a cross-attention mechanism to complete the deep fusion of multimodal information. Finally, it extracts the output of a specific position in the fused sequence as a dimensionally unified emotional cognitive state vector.
[0028] Furthermore, it also includes an emotion-state-based cognitive diagnosis module, which combines students' emotional cognitive states with knowledge point information to dynamically predict students' mastery levels of each knowledge point. The emotion-state-based cognitive diagnosis module also constructs a knowledge point relationship graph, where nodes are knowledge points and edges are prior dependencies or data-driven co-occurrence relationships between knowledge points. The state vector containing emotional information output by the bidirectional recurrent neural network is used as the initial feature of each knowledge point node. The knowledge point neighborhood information is aggregated through the graph attention network (GAT) and the node representation is updated to finally generate the mastery probability of each knowledge point.
[0029] The beneficial effects of this invention are as follows:
[0030] This invention integrates multimodal information to perceive changes in students' emotions and cognitive states during the learning process in real time, overcoming the limitations of traditional methods that suffer from single information and one-sided results. It introduces modality selection and supermodal learning mechanisms to improve the effective utilization of multimodal information and enhance system robustness and adaptability. Furthermore, it combines dynamic time series analysis to accurately model the evolution of students' mastery of knowledge points, improving the accuracy and interpretability of cognitive diagnostic results. This invention is widely applicable to intelligent education scenarios such as personalized teaching, online learning platforms, and educational intervention systems, providing reliable technical support for educational decision-making. Attached Figure Description
[0031] Figure 1 This is a flowchart illustrating a cognitive diagnostic method based on emotional states.
[0032] Figure 2 This is a schematic diagram of a cognitive diagnostic model based on emotional states;
[0033] Figure 3 This is a schematic diagram of a cognitive diagnostic module. Detailed Implementation
[0034] Example 1
[0035] like Figure 1 As shown, a cognitive diagnostic method based on emotional state includes the following steps:
[0036] Extract emotion-related multimodal features from teaching videos and obtain their corresponding knowledge point information;
[0037] By using linear mapping and embedding transformation, the extracted multimodal features are projected onto a unified low-dimensional space, so that the data are in the same dimension, which prepares for subsequent fusion.
[0038] Based on the information content of each modality in the multimodal features, the primary modality and auxiliary modality are adaptively selected;
[0039] Through information exchange between the primary modality and the secondary modality, a supermodal state expressing emotional semantics is recursively updated and formed.
[0040] The principal modality and supermodal state are deeply fused, and a unified supermodal state representing the student's emotional cognitive state is generated based on the cognitive diagnosis of emotional state.
[0041] By combining students' emotional and cognitive states with knowledge point information, a bidirectional recurrent neural network is used to dynamically predict students' mastery level of each knowledge point.
[0042] Example 2
[0043] like Figure 2 As shown, a cognitive diagnostic model based on emotional state includes a cascaded multimodal information feature extraction module, a feature alignment layer, a modality selection layer, an adaptive supermodal learning layer, a cross-modal fusion and transformation layer, and a cognitive diagnostic module based on emotional state.
[0044] The multimodal information feature extraction module is used to extract emotion-related multimodal features from teaching videos and obtain their corresponding knowledge point information;
[0045] The feature alignment layer is used to project the extracted multimodal features into a unified low-dimensional space through linear mapping and embedding transformation, so that the data are in the same dimension, preparing for subsequent fusion.
[0046] The modality selection layer is used to adaptively select the primary and secondary modalities based on the information content of each modality in the multimodal features;
[0047] The adaptive supermodal learning layer is used to recursively update and form a supermodal state that expresses emotional semantics through information interaction between the main modality and the auxiliary modality.
[0048] The cross-modal fusion conversion layer is used to deeply fuse the main modality and supermodal states, and generate a unified student emotional cognitive state representation supermodal state based on the cognitive diagnosis of emotional state.
[0049] The aforementioned cognitive diagnosis module based on emotional state is used to dynamically predict students' mastery level of each knowledge point by combining students' emotional and cognitive states with knowledge point information.
[0050] Example 3
[0051] In this embodiment, during the teaching process, multi-source data of students is collected in real time, including but not limited to:
[0052] Students' language and text information comes from classroom answers, online Q&A, quiz feedback, etc.
[0053] Students' facial expressions and behavioral data are acquired through camera devices;
[0054] The student's voice audio data, including speech rate, pitch, and tone changes.
[0055] In one specific embodiment, before extracting features, the multimodal information feature extraction module introduces a self-supervised contrastive learning mechanism to pre-train and optimize the original modal data; the mechanism guides the model to learn more discriminative embedding representations within the modality by constructing positive and negative sample pairs, providing robust initial features for subsequent modeling.
[0056] In this embodiment, specifically, for each modality, the pre-training layer first constructs a set of sample pairs with semantic similarity and differences. Among them, positive sample pairs are generated from the same semantic source under different perturbation conditions (such as different brightness enhancements of images, time-frequency perturbations of audio, scrambling of text word order, etc.), while negative sample pairs are composed of data from different students or different task scenarios.
[0057] Subsequently, a contrastive loss function, such as InfoNCE loss, is used to guide the model to bring positive sample pairs closer together and push negative sample pairs further apart in the feature space, thereby learning discriminative modality embedding representations. This process requires no manual labeling and relies solely on the structural information of the data itself to achieve self-supervised learning, making it suitable for large-scale sentiment data scenarios.
[0058] In this embodiment, text feature extraction uses the BERT pre-trained language model to obtain text feature vectors containing semantic and emotional information;
[0059] Visual feature extraction is based on the OpenFace facial behavior analysis framework, capturing facial key points, facial expressions, and movements;
[0060] Audio feature extraction uses the Librosa tool to extract features in the frequency domain, time domain, and acoustic statistics.
[0061] In this embodiment, due to the heterogeneous sources of multimodal data and their different dimensions and feature distributions, the feature alignment layer first projects all modal features to the same dimensional space through linear mapping to achieve effective fusion. Then, embedding technology is used to further compress the feature dimensions, preserving effective information while improving computational efficiency.
[0062] In one specific embodiment, the feature alignment layer includes a cross-modal semantic distillation module; this module uses a pre-trained multimodal model as the teacher model and generates soft labels for inter-modal alignment as supervision signals; through knowledge distillation loss, it constrains the consistency of the probability distribution of each modal feature in the student model after projection onto the low-dimensional space, so as to improve the semantic consistency of feature alignment.
[0063] In this embodiment, to reduce interference from redundant information, the present invention proposes a mode selection mechanism, specifically as follows:
[0064] First, sentiment scores for each of the three modalities are calculated to measure the intensity of emotional expression inherent in each modality. After normalization or absolute value processing, the sentiment scores reflect the emotional salience of each modality in the current input.
[0065] Subsequently, the emotional intensity of the three modalities is compared to determine which modality has the strongest emotional expression capability. If the determination result shows that the text modality has the strongest emotion, the current modality structure remains unchanged; otherwise, the modality with the strongest emotion exchanges information with the text modality and sets it as the dominant modality channel.
[0066] The modality with the highest information content is selected as the primary modality, and the other two are used as auxiliary modalities. The primary modality is always the one with the richest emotional information, thereby improving the downstream model's modeling effect on students' emotional states. The emotional information content of each modality is calculated based on the LSTM network.
[0067] In one specific embodiment, the modality selection layer calculates the emotional information entropy of each modality through a long short-term memory network (LSTM) to assess its information content and determine the dominant and auxiliary modalities.
[0068] In one specific embodiment, the modality selection layer further allocates modality weights based on uncertainty estimation. Specifically, a random dropout layer is set for the features of each modality, and the variance of the prediction results is calculated through multiple forward inferences during the training phase to quantify the uncertainty of each modality. The modality weights are then determined based on a preset uncertainty threshold or a weighted formula combining information entropy.
[0069] In one specific embodiment, the adaptive supermodal learning layer includes at least two Transformer encoding modules and multiple supermodal information fusion layers. The Transformer encoding modules are used to extract deep semantic features of the main modality information in a hierarchical manner, and the supermodal information fusion layers are used to guide the information interaction between the auxiliary modality and the main modality to form a unified supermodal representation.
[0070] In this embodiment, a multi-level adaptive supermodal learning structure is designed for the primary modality and auxiliary modality information, specifically including:
[0071] First, the module receives the main modal feature sequence from the previous stage, as well as the auxiliary modal feature representation, and also includes a supermodal state representation for cross-layer updates.
[0072] Within each layer, the module uses the current dominant modality features as its core and guides the information update of the dominant modality by fusing emotional cues from auxiliary modalities. This process enables the model to adaptively adjust in both the temporal and modal dimensions, strengthening the dominant modality's ability to absorb external emotional information.
[0073] Subsequently, the updated supermodal state serves as the input to the next layer, working in conjunction with the new master modality timestep features to continue cross-modal information integration. This process is repeated across multiple layers, enabling the model to progressively refine high-level, stable supermodal semantic representations.
[0074] To maintain stable information flow and prevent gradient vanishing and overfitting, a pre-normalization mechanism and residual connections are introduced in each layer to ensure that information can be effectively transmitted in the deep structure.
[0075] Finally, the module outputs a supermodal state that integrates text, audio, and visual information, containing rich emotional features and temporal dynamic information, providing a basic representation for subsequent cross-modal fusion and cognitive diagnosis.
[0076] In one specific embodiment, the cross-modal fusion transformation layer introduces learnable additional labels and positional encodings between the source modal sequence and the target modal sequence, and combines a cross-attention mechanism to complete the deep fusion of multimodal information. Finally, the output of a specific position in the fused sequence is extracted as a dimensionally unified emotional cognitive state vector.
[0077] In this embodiment, the supermodal state and the deep information of the main modality are jointly input into the cross-modal fusion conversion layer, specifically including:
[0078] The module receives two sets of inputs: one is the supermodal state sequence output from the previous stage, which serves as the source modal information; the other is the deep feature sequence of the current main modality at the corresponding time step, which serves as the target modal information.
[0079] A learnable additional token (i.e. a special token) is inserted in front of both the source sequence and the target sequence to uniformly capture the fused global semantic information and ensure that the model has centralized output capability.
[0080] To preserve the temporal structure of sequence information, positional encoding is added to both sets of inputs, enabling the model to accurately identify temporal dependencies in the input data, thereby improving the accuracy of cross-modal alignment.
[0081] The fusion module employs a dual-input Transformer encoder, alternately using the source modality as the query vector and the target modality as the key and value for multi-head attention interaction learning, thereby establishing a deep connection between the two modalities. This mechanism can effectively capture the interdependence and complementary information between the main modality and the supermodality;
[0082] After encoding, feature vectors corresponding to the additional markers are extracted from the fused sequence and used as the final fusion output. This vector integrates key information from the main modality and the supermodality, forming a unified representation of emotional cognitive state.
[0083] In this embodiment, as Figure 3 As shown, cognitive diagnosis based on emotional state is specifically as follows:
[0084] By combining students' emotional and cognitive states with knowledge point information, a bidirectional recurrent neural network (Bi-LSTM) model is constructed:
[0085] The dynamic features of emotional state changes over time are captured simultaneously by forward and backward LSTM units.
[0086] The output features are combined with the current knowledge point information and input into the prediction layer;
[0087] The final output shows the probability of students mastering each knowledge point, which is used to assess students' learning effectiveness and identify their knowledge weaknesses.
[0088] In one specific embodiment, the cognitive diagnosis module based on emotional state further constructs a knowledge point relationship graph, where nodes are knowledge points and edges are prior dependencies or data-driven co-occurrence relationships between knowledge points; the state vector containing emotional information output by the bidirectional recurrent neural network is used as the initial feature of each knowledge point node; the neighborhood information of knowledge points is aggregated through the graph attention network GAT, and the node representation is updated to finally generate the mastery probability of each knowledge point.
[0089] As can be seen, through the above steps, this invention can comprehensively and accurately model the changes in students' multimodal emotions and cognitive states during the learning process, dynamically track their knowledge mastery level, improve the credibility and practicality of cognitive diagnostic results, and is widely applicable to intelligent teaching platforms, online learning systems, learning behavior analysis, and educational intervention.
[0090] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the implementation of the present invention. Those skilled in the art can make other variations or modifications based on the above description. It is neither necessary nor possible to exhaustively describe all embodiments here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.
Claims
1. A cognitive diagnostic model based on emotional states, characterized in that, It includes a cascaded multimodal information feature extraction module, a feature alignment layer, a modality selection layer, an adaptive supermodal learning layer, and a cross-modal fusion and transformation layer; The multimodal information feature extraction module is used to extract emotion-related multimodal features from teaching videos and obtain their corresponding knowledge point information; The feature alignment layer is used to project the extracted multimodal features into a unified low-dimensional space, so that the data are in the same dimension, in preparation for subsequent fusion. The modality selection layer is used to adaptively select the primary and secondary modalities based on the information content of each modality in the multimodal features; The modality selection layer calculates the emotional information entropy of each modality through a Long Short-Term Memory (LSTM) network to assess its information content and determine the dominant and auxiliary modalities. The modality selection layer further allocates modality weights based on uncertainty estimation. Specifically, a random dropout layer is set for the features of each modality, and the variance of the prediction results is calculated through multiple forward inferences during the training phase to quantify the uncertainty of each modality. The modality weights are determined according to a pre-set uncertainty threshold or in combination with the weighted formula of information entropy. The adaptive supermodal learning layer is used to recursively update and form a supermodal state that expresses emotional semantics through information interaction between the main modality and the auxiliary modality. The cross-modal fusion conversion layer is used to deeply fuse the main modality and supermodal states, and generate a unified supermodal state representing the student's emotional cognitive state based on the cognitive diagnosis of emotional state.
2. The cognitive diagnostic model according to claim 1, characterized in that, Before extracting features, the multimodal information feature extraction module introduces a self-supervised contrastive learning mechanism to pre-train and optimize the original modal data. The mechanism guides the model to learn more discriminative embedding representations within the modality by constructing positive and negative sample pairs, providing robust initial features for subsequent modeling.
3. The cognitive diagnostic model according to claim 1, characterized in that, The multimodal information feature extraction module specifically includes: a BERT pre-trained language model for extracting text semantic features, an analyzer for extracting visual behavior based on the OpenFace system, and an audio feature extractor using the Librosa tool library.
4. The cognitive diagnostic model according to claim 1, characterized in that, The feature alignment layer includes a cross-modal semantic distillation module, which uses a pre-trained multimodal model as the teacher model and generates soft labels for intermodal alignment as supervision signals. Through knowledge distillation loss, the consistency of the probability distribution of each modality feature in the student model after being projected into the low-dimensional space is constrained, so as to improve the semantic consistency of feature alignment.
5. The cognitive diagnostic model according to claim 1, characterized in that, The adaptive supermodal learning layer includes at least two Transformer encoding modules and multiple supermodal information fusion layers. The Transformer encoding modules are used to extract deep semantic features of the main modality information in a hierarchical manner. The supermodal information fusion layers are used to guide the information interaction between the auxiliary modality and the main modality to form a unified supermodal representation.
6. The cognitive diagnostic model according to claim 1, characterized in that, The cross-modal fusion transformation layer introduces learnable additional labels and positional encodings between the source modal sequence and the target modal sequence, and combines a cross-attention mechanism to complete the deep fusion of multimodal information. Finally, it extracts the output of a specific position in the fused sequence as a dimensionally unified emotional cognitive state vector.
7. The cognitive diagnostic model according to claim 1, characterized in that, It also includes an emotion-state-based cognitive diagnosis module, which combines students' emotional cognitive states with knowledge point information to dynamically predict students' mastery level of each knowledge point; the emotion-state-based cognitive diagnosis module also constructs a knowledge point relationship graph, where nodes are knowledge points and edges are prior dependencies or data-driven co-occurrence relationships between knowledge points; the state vector containing emotional information output by the bidirectional recurrent neural network is used as the initial feature of each knowledge point node; The graph attention network (GAT) aggregates neighborhood information of knowledge points and updates node representations to ultimately generate the mastery probability of each knowledge point.
8. A cognitive diagnostic method based on emotional state, characterized in that, Based on the cognitive diagnostic model as described in any one of claims 1 to 7, the steps include: Extract emotion-related multimodal features from teaching videos and obtain their corresponding knowledge point information; The extracted multimodal features are projected onto a unified low-dimensional space to bring the data to the same dimension, thus preparing for subsequent fusion. Based on the information content of each modality in the multimodal features, the primary modality and auxiliary modality are adaptively selected; Through information exchange between the primary modality and the secondary modality, a supermodal state expressing emotional semantics is recursively updated and formed. The principal and supermodal states are deeply fused, and a unified representation of students' emotional cognitive states is generated based on cognitive diagnosis of emotional states.