Student real-time emotion recognition method based on double-branch recursive knowledge graph

Abstract

The invention belongs to the technical field of computer vision, and particularly relates to a student real-time emotion recognition method based on a double-branch recursive knowledge graph. Comprising the following steps: processing entity embedded representation by adopting a recursive knowledge graph modeling module to obtain knowledge graph features; inputting the student information into a multi-modal feature extraction module for processing to obtain self-adaptive fusion multi-modal features; calculating the total loss of the model according to the knowledge graph features and the adaptive fusion multi-modal features, and iteratively adjusting model parameters to obtain a trained student real-time emotion recognition model; obtaining a student emotion knowledge graph and carrying out embedded learning on the student emotion knowledge graph to obtain an entity embedded representation; acquiring student information and inputting the student information and the entity embedded representation into the trained model for processing to obtain a real-time emotion recognition result of the student; according to the method, multi-dimensional collaboration of knowledge, features and data is realized, the accuracy, dynamic tracking, robustness and generalization of emotion recognition are improved, and the method has a good application prospect.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, specifically relating to a real-time emotion recognition method for students based on a dual-branch recursive knowledge graph. Background Technology

[0002] In recent years, student emotion recognition technology has made significant progress in multimodal fusion and knowledge enhancement. In the field of combining knowledge graphs and emotion modeling, early research often used static knowledge graphs to assist emotion reasoning. For example, KGAT (Knowledge Graph Attention Network) uses graph attention mechanisms to mine entity relationships and improve emotion classification accuracy, but it did not design recursive structures for the dynamic evolution of emotions. While GGSNN (Gated Graph Sequence Neural Networks) can process sequence graph data and capture temporal changes in emotions, it only uses single-branch modeling, making it difficult to balance the integrity of the knowledge structure with the need for real-time emotion updates. Other research, such as EK-TextCNN (Emotion Knowledge-enhanced TextCNN), embeds emotional entities and concatenates them with text features to enhance semantic representation, but the knowledge fusion method is superficial and does not form a hierarchical knowledge system.

[0003] In the field of multimodal emotion recognition, single-modal techniques are relatively mature: in the visual modality, the Swin Transformer captures facial expression details through hierarchical attention, and the accuracy of methods based on this model in the EmotiW competition is over 85%; in the speech modality, the Audio Transformer combines Mel-spectral features with a self-attention mechanism, achieving an F1 score of over 0.8 in emotion recognition on the RAVDESS dataset; and in the text modality, it relies on pre-trained models such as BERT and DeBERTa-v3, which perform excellently in emotion analysis of student comments. In terms of multimodal fusion, MM-Transformer achieves feature interaction through cross-modal attention, and MA-Net introduces a modality adaptation module to optimize heterogeneous feature fusion. However, existing methods mostly rely on data-driven feature splicing or weighted fusion, lacking the structured constraints of knowledge graphs on the association between "multimodal features and emotional entities". Furthermore, in response to the data imbalance problem between common emotions such as "calm / focused" and rare emotions such as "confusion / anxiety" in classroom scenarios, they only use simple data augmentation (such as flipping facial expression images) or class weight adjustment, without forming a targeted optimization strategy, resulting in an abnormal emotion recognition recall rate that is generally below 60%.

[0004] In conclusion, there is an urgent need for a real-time emotion recognition method for students to improve the accuracy, generalization, and dynamic tracking capabilities of real-time emotion recognition. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention proposes a real-time student emotion recognition method based on a dual-branch recursive knowledge graph. This method includes:

[0006] The system acquires a student emotion knowledge graph and embeds it into the graph to obtain entity embedding representations. It also acquires student information and inputs the student information and entity embedding representations into a trained real-time student emotion recognition model for processing to obtain real-time student emotion recognition results. The student information includes face images, student audio, and student text.

[0007] The training process for the student real-time emotion recognition model includes:

[0008] S1: The entity embedding representation is processed using a recursive knowledge graph modeling module to obtain knowledge graph features; the recursive knowledge graph modeling module consists of a knowledge representation learning branch, a recursive state update branch, and a DiffPool layer.

[0009] S2: Input student information into the multimodal feature extraction module for processing to obtain adaptive fused multimodal features;

[0010] S3: Calculate the total loss of the model based on the knowledge graph features and the adaptive fusion of multimodal features; iteratively adjust the model parameters based on the total loss of the model to obtain a well-trained real-time emotion recognition model for students.

[0011] Preferably, the process by which the recursive knowledge graph modeling module processes entity embedding representations includes:

[0012] S11: The knowledge representation learning branch is used to process the entity embedding representation to obtain shallow features;

[0013] S12: The entity embedding representation is processed by a recursive state update branch to obtain deep features;

[0014] S13: Use the DiffPool layer to fuse shallow and deep features to obtain knowledge graph features.

[0015] Furthermore, the process of obtaining shallow features includes:

[0016] A multi-head attention mechanism is used to process the entity embedding representation, and the attention coefficients between the current entity embedding representation and its neighboring entity embedding representations are obtained under each attention head.

[0017] For each attention head, the neighbor entity embedding representations are weighted and summed according to the attention coefficients to obtain the output features of the current entity embedding representation under each attention head;

[0018] By concatenating the output features of the current entity embedding representation under all attention heads, we obtain the shallow features of the current entity embedding representation.

[0019] Furthermore, the formula for obtaining the attention coefficient is:

[0020]

[0021] in, Let represent the attention coefficients of entity embedding representations i and j under the k-th attention head. This represents the query matrix of the k-th attention head. This represents the key matrix of the k-th attention head. The entity embedding representation i, The entity embedding representation j, This represents the scaling factor for the k-th attention head. The entity embedding represents the set of neighboring entities of entity i. express Activation function This represents the sum of the exponent terms calculated for all neighboring nodes j of node i.

[0022] Furthermore, the process of obtaining deep features includes:

[0023] The current entity embedding representation is concatenated with the entity embedding representation of one of its neighbors, and the interaction features between the current entity embedding representation and the neighbor entity embedding representation are calculated using a weight matrix. The interaction features between the current entity embedding representation and all neighbor entities are stacked to form the interaction matrix of the current entity embedding representation.

[0024] The interaction matrix of the current entity embedding representation is processed using a GRU network to obtain intermediate features;

[0025] After performing a residual connection between the intermediate feature and the current entity embedding representation, layer normalization is applied to obtain the deep feature of the current entity embedding representation.

[0026] Preferably, the multimodal feature extraction module's processing of student information includes:

[0027] A dedicated encoder was used to extract features from student information, resulting in visual features, speech features, and text features.

[0028] A cross-modal attention mechanism is used to process visual features, speech features, and text features to obtain video enhancement features, speech enhancement features, and text enhancement features;

[0029] Learn the modal weights of the three features; then, based on the modal weights, perform a weighted summation of the video enhancement features, speech enhancement features, and text enhancement features to obtain adaptive fused multimodal features.

[0030] Furthermore, the video enhancement features are represented as follows:

[0031]

[0032]

[0033]

[0034] in, Indicates video enhancement features, Indicates video-to-audio weights. Indicates video-text weights, Representing video features, Representing speech features, Representing text features, Represents the video query matrix. Represents the voice query matrix. Represents a text query matrix. This represents the cross-modal attention scaling factor. express Activation function.

[0035] Preferably, the formula for calculating the total loss of the model is:

[0036]

[0037] in, This represents the total loss of the model. Represents the cross-entropy loss function. This indicates adaptive fusion of multimodal features. Representing the features of a knowledge graph. This indicates a splicing operation. Indicates the true label, Indicates instance-level loss. Indicates category-level centrality loss. Indicates the modal alignment loss. , and These represent instance weights, class weights, and modal weights, respectively.

[0038] Furthermore, the category-level centrality loss is expressed as:

[0039]

[0040] in, Indicates the number of categories. This represents the set of samples belonging to category y. This represents the adaptive fusion of multimodal features of the input sample x. The feature center of category y is represented. This represents the distance metric function.

[0041] The beneficial effects of this invention are as follows:

[0042] This invention employs a dual-branch design—"shallow TransE + multi-head attention to preserve knowledge structure, deep GGSNN + GRU to capture emotional evolution"—combined with DiffPool hierarchical fusion, to systematically model the "emotional entity-classroom context-evolutionary relationship." This results in a clearer depiction of the hierarchical relationship of emotional states, more accurate tracking of dynamic evolutionary trajectories such as "confusion → anxiety," improved accuracy of emotional reasoning in complex scenarios, and significantly superior systematicity and dynamic adaptability of knowledge representation compared to traditional solutions.

[0043] This invention accurately extracts features through a modal-specific encoder, and combines CAB cross-modal attention with DFG dynamic fusion gate to achieve deep intermodal interaction and adaptive weight allocation. It significantly improves the utilization rate of multimodal information, enhances recognition accuracy in conflict scenarios such as "smiling + negative text," strengthens modal noise suppression, and exhibits superior robustness in emotion recognition in complex classroom environments compared to traditional fusion schemes.

[0044] This invention employs a three-tiered comparative learning task—instance-level, category-level, and modality-level—combined with a dynamic difficulty weighting mechanism to optimize the feature space distribution and enhance the discriminative power of minority emotion samples. It improves the recall rate for abnormal emotion recognition by 1%-2%, resulting in more compact intra-class features, clearer inter-class boundaries, significantly alleviates model bias caused by data imbalance, and greatly enhances generalization ability in real-world classroom scenarios.

[0045] This invention achieves closed-loop optimization of "knowledge support - feature enhancement - distribution optimization" through end-to-end collaboration of "knowledge graph modeling + multimodal fusion + comparative learning optimization". It improves both emotion recognition accuracy and recall by 1%-2%, has low real-time processing latency, is suitable for dynamic monitoring scenarios in smart classrooms, and its overall performance surpasses traditional solutions that optimize only a single technology. Attached Figure Description

[0046] Figure 1 This is a flowchart of the real-time emotion recognition method for students based on a dual-branch recursive knowledge graph in this invention. Detailed Implementation

[0047] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0048] This invention proposes a real-time emotion recognition method for students based on a dual-branch recursive knowledge graph. First, a structured emotion knowledge system is constructed using a recursive knowledge graph modeling module (R-KGAT) to dynamically model the evolution of emotions. Then, a dedicated encoder extracts multimodal features from facial expressions, speech, and text. Subsequently, cross-modal attention and a dynamic fusion gate are used to achieve deep interaction of heterogeneous features. Finally, multi-granularity comparative learning optimizes the feature space distribution, improving the accuracy of emotion recognition. The modules work together to address issues of knowledge fragmentation, coarse fusion, and imbalanced data. Figure 1 As shown, the method includes the following:

[0049] The system acquires a student emotion knowledge graph and embeds it into the graph to obtain entity embedding representations. It also acquires student information and inputs the student information and entity embedding representations into a trained real-time student emotion recognition model for processing to obtain real-time student emotion recognition results. The student information includes face images, student audio, and student text.

[0050] Obtain student emotional knowledge graphs and embed them into learning, specifically:

[0051] The knowledge graph is initialized with all emotion entity sets E (such as "confused", "focused", "smile", etc.) and relation sets R (such as "evolved from", "accompanied", "caused", etc.). The embedding dimension is set to d (e.g., d=128 in this embodiment), and Xavier initialization or standard normal distribution random initialization is used to generate the initial entity embedding matrix and relation embedding matrix. At this point... All values ​​are random, and all entity vectors are normalized to satisfy... .

[0052] To enable the model to distinguish between correct and incorrect sentiment logic, for each real triple (positive sample) (h, r, t) in the knowledge graph, negative samples are generated by randomly replacing the head or tail entity. For example, for a positive sample ("confused", "evolved into", "anxious"), a negative sample ("confused", "evolved into", "happy") is generated by randomly replacing the tail entity. The set of negative samples is represented as follows:

[0053]

[0054] in, This represents a erroneous emotional logic triple after the original logic has been disrupted.

[0055] This invention uses the hinge loss function as the optimization objective, aiming to minimize the energy score of positive samples while maximizing the energy score of negative samples. The training objective function L is defined as:

[0056]

[0057] in, The score for the positive sample, i.e. , representing the distance in the vector space; The score for negative samples. This is a preset interval hyperparameter used to control the discrimination between positive and negative samples (e.g., set to 1.0). express .

[0058] The loss function described above is minimized using stochastic gradient descent (SGD) or the Adam optimizer. In each epoch, the gradients of the loss function with respect to entity vectors h, t and relation vector r are calculated, and these vectors are updated via backpropagation. After multiple iterations and convergence, the final h, r, and t are the TransE embedding parameters (including entity embedding representations) required by the R-KGAT module of this invention, which contain student sentiment semantics and structural information. These parameters are then used as input features for R-KGAT, supporting subsequent attention calculations and recursive updates.

[0059] The training process for the student real-time emotion recognition model includes:

[0060] S1: The entity embedding representation is processed using a recursive knowledge graph modeling module to obtain knowledge graph features.

[0061] The recursive knowledge graph modeling module designed in this invention consists of a knowledge representation learning branch, a recursive state update branch, and a DiffPool layer.

[0062] S11: The knowledge representation learning branch is used to process the entity embedding representation to obtain shallow features.

[0063] A multi-head attention mechanism is used to process the entity embedding representations, obtaining the attention coefficients between the current entity embedding representation and its neighboring entity embedding representations under each attention head:

[0064]

[0065] in, Let represent the attention coefficients of entity embedding representations i and j under the k-th attention head. This represents the query matrix of the k-th attention head. This represents the key matrix of the k-th attention head. The entity embedding representation i, The entity embedding representation j, This represents the scaling factor for the k-th attention head, used for normalization to prevent gradient vanishing; The entity embedding represents the set of neighboring entities of entity i. express Activation function, denominator This represents all neighboring nodes j of node i (containing in the set). (In the middle) Calculate the cumulative sum of the above exponential terms, which is used for Softmax normalization to ensure that the sum of the attention coefficients is 1.

[0066] This mechanism utilizes the semantic features provided by TransE to dynamically assign weights to different emotional evolution paths.

[0067] For each attention head, the neighbor entity embedding representations are weighted and summed according to the attention coefficients to obtain the output features of the current entity embedding representation under each attention head;

[0068] By concatenating the output features of the current entity embedding representation under all attention heads, we obtain the shallow features of the current entity embedding representation. .

[0069] S12: The entity embedding representation is processed by a recursive state update branch to obtain deep features.

[0070] The current entity embedding representation is concatenated with the entity embedding representation of one of its neighbors, and the interaction features between the current entity embedding representation and that neighbor entity embedding representation are calculated using a weight matrix. The interaction features between the current entity embedding representation and all neighbor entities are stacked to form the interaction matrix of the current entity embedding representation. The interaction features are then calculated as follows:

[0071]

[0072] in, The interaction features between entity embedding representation i and entity embedding representation j are represented. Represents the weight matrix. The entity embedding representation i, The entity embedding is represented by j.

[0073] The interaction matrix (i.e., the sequence of neighbor interaction features) of the current entity embedding representation is processed by the GRU network to obtain intermediate features containing temporal or structural context information.

[0074] Embed the intermediate features with the current entity representation After performing residual connections, layer normalization is then applied to obtain the deep features of the current entity embedding representation. .

[0075] S13: Use the DiffPool layer to fuse shallow and deep features to obtain knowledge graph features.

[0076] DiffPool hierarchical fusion generates an assignment matrix S through dynamic node clustering, fusing shallow and deep features into hierarchical knowledge graph features. This balances global knowledge structure with local dynamic evolution, as shown below:

[0077]

[0078] in, Representing the features of a knowledge graph. This represents the overall shallow feature composed of all entity embeddings. This represents the overall deep features constituted by the embedding of all entities.

[0079] S2: Input student information into the multimodal feature extraction module for processing to obtain adaptive fused multimodal features.

[0080] The multimodal feature extraction module includes a dedicated encoder and a cross-modal fusion module.

[0081] A dedicated encoder was designed for three modalities to enhance feature targeting. First, the student information used for training was input into the dedicated encoder for feature extraction. The process included:

[0082] Visual Feature Extraction (Swin Transformer): Divides the face image into 16×16 patches and maps them to vectors. It then captures facial expression details (such as furrowed brows and the curvature of the mouth) through Shifted Window multi-head self-attention, outputting visual features. .

[0083] Speech feature extraction (bidirectional Audio Transformer): The audio is converted into a Mel spectrum, time-frequency features are enhanced by depthwise separable convolution, and then speech intonation variations (such as speech rate and pitch) are encoded by a bidirectional Transformer to output speech features. .

[0084] Text Feature Extraction (DeBERTa-v3): This method utilizes a decoupled attention mechanism to separate content and location information, enhancing semantic understanding of text such as student comments and bullet screen messages (e.g., "too difficult" or "can't understand"). It then fine-tunes the output text features through sentiment adaptation. .

[0085] The cross-modal fusion module (HCT) achieves deep interaction of features through cross-modal attention (CAB) and dynamic fusion gate (DFG).

[0086] A cross-modal attention mechanism is employed to process visual, speech, and text features, resulting in video enhancement features, speech enhancement features, and text enhancement features; specifically:

[0087] Taking video enhancement features as an example, CAB uses visual features as the query and calculates attention weights using speech and text as keys respectively:

[0088]

[0089]

[0090] Generate video enhancement features based on attention weights:

[0091]

[0092] in, Indicates video enhancement features, Indicates video-to-audio weights. Indicates video-text weights, Representing video features, Representing speech features, Representing text features, Represents the video query matrix. Represents the voice query matrix. Represents a text query matrix. This represents the scaling factor for cross-modal attention, i.e., the dimension of the key vector. express Activation function.

[0093] Similarly, speech enhancement features and text enhancement features can be obtained, which will not be elaborated here.

[0094] Using DFG to learn the modal weights of three features , , (Satisfying the sum of weights of 1). Based on modal weights, video enhancement features, speech enhancement features, and text enhancement features are weighted and summed to highlight the contributions of key modalities (e.g., emphasizing visual and textual contributions when the classroom is silent), resulting in adaptively fused multimodal features. :

[0095]

[0096] S3: Calculate the total loss of the model based on the knowledge graph features and the adaptive fusion of multimodal features; iteratively adjust the model parameters based on the total loss of the model to obtain a well-trained real-time emotion recognition model for students.

[0097] A multi-granularity contrastive learning optimization module is constructed, which optimizes feature distribution through a three-layer contrastive task, thereby alleviating data imbalance and enhancing the ability to identify abnormal emotions.

[0098] 1. Instance-level comparison

[0099] Calculate the InfoNCE loss for augmented samples (such as facial expression image flipping, speech speed variation). Narrowing the distance between similar emotion samples:

[0100]

[0101] in, Indicates instance-level contrastive loss; The cosine similarity between anchor sample i and its positive sample pair j (i.e., views of the same original sample after different data augmentations) represents the similarity between the anchor sample i and its positive sample pair j. This represents the cosine similarity between anchor sample i and the k-th sample in the current training batch; This represents the temperature coefficient hyperparameter, used to control the smoothness of the probability distribution; This represents the total number of samples in the current training batch (Batch Size). This formula enhances the model's robustness to subtle perturbations in samples by maximizing the similarity between positive sample pairs and minimizing the similarity with negative samples (other samples within the batch).

[0102] 2. Category-level comparison

[0103] By optimizing intra-class compactness through categorical contrastive loss, the ability to represent scarce emotions such as "anxiety" is improved.

[0104]

[0105] in, Indicates the number of categories. This represents the set of samples belonging to category y. This means averaging the errors within that category. The advantage of doing this is that it eliminates the effects of class imbalance (even if there are many "confused" samples and few "anxious" samples, their contribution weights to the total loss are the same). This represents the adaptive fusion of multimodal features of the input sample x. The feature center of category y is represented by a vector, which depicts the standard representation (ideal center point) of the emotion "anxiety" in the feature space. This center is usually learned by the network or obtained by calculating the mean of samples in that class. This represents the distance metric function, typically the Euclidean distance (L2 norm) or cosine distance.

[0106] 3. Modal-level comparison

[0107] By maximizing the modality alignment loss through mutual information, multimodal feature semantics are aligned, modality noise interference is reduced, and classification accuracy and feature robustness are balanced.

[0108]

[0109] in, Indicates the modal alignment loss; This indicates a summation traversal over all distinct modal combinations (e.g., visual-speech, visual-text, speech-text); and These represent the same input sample in different modes. and modality High-dimensional feature vectors; The L2 norm of a vector (i.e., the magnitude of the vector); fractional terms in the formula. This represents the calculation of cosine similarity between two modal feature vectors. This loss function aims to achieve semantic alignment of multimodal features by minimizing the cosine distance between different modalities (i.e., maximizing cosine similarity).

[0110] The total loss of the model in this invention is expressed as:

[0111]

[0112] in, This represents the total loss of the model. Represents the cross-entropy loss function. This indicates adaptive fusion of multimodal features. Representing the features of a knowledge graph. This indicates a splicing operation. Indicates the true label, Indicates instance-level loss. Indicates category-level centrality loss. Indicates the modal alignment loss. , and These represent instance weights, class weights, and modal weights, respectively.

[0113] The model is trained iteratively based on its total loss. When the total loss is minimized or converged, training is stopped, the model parameters are preserved, and a well-trained real-time emotion recognition model for students is obtained.

[0114] Student information is acquired and its entity embedding representation is input into a pre-trained real-time student emotion recognition model for processing. Within the model, the features output by the multimodal fusion module are... Knowledge graph features output by the recursive knowledge graph modeling module By concatenating the data, a joint feature vector containing perceptual information and cognitive logic is constructed. Joint feature vectors The input is a fully connected classification layer, which is linearly transformed to map it to an emotion category space, yielding predicted scores corresponding to C emotion categories. The predicted scores are then normalized using a softmax activation function, and the conditional probability distribution of a sample belonging to each emotion category is calculated. Finally, the category index with the highest probability value is selected as the student's current emotion state recognition result.

[0115] Evaluation of the present invention:

[0116] The method of the present invention was verified using a public dataset of multimodal student emotion recognition and a self-collected classroom scene dataset. The specific information of the dataset is shown in Table 1.

[0117] Table 1 Dataset Details

[0118] The experimental environment was as follows: CPU: AMD Ryzen 9 5950X @ 3.4GHz; GPU: NVIDIA GeForce RTX 4090; operating system: Ubuntu 22.04; programming language: Python 3.10.6; deep learning framework: PyTorch 2.0.1. To verify the effectiveness of this invention, mainstream models and improved schemes in the field of multimodal emotion recognition were selected for comparison, and accuracy, precision, recall, and F1 score were used as evaluation metrics. The results are shown in Table 2.

[0119] Table 2 Evaluation Results of the Invention and the Comparative Method

[0120]

[0121] Table 2 shows the specific and significant advantages of this invention. Multimodal synergy advantage: The F1 score of this invention's model is 17.8% higher than the best single-modal model (text DeBERTa-v3), and 13.0% higher than simple multimodal concatenation, verifying the synergistic effect of deep multimodal fusion and knowledge graph-guided structured approach. Breakthrough in abnormal emotion recognition: For rare abnormal emotions such as "confusion" and "anxiety," the F1 score of this invention is 17.7% higher than the baseline model (MA-Net), solving the model bias problem caused by data imbalance. The synergistic value of knowledge and features: The dual-branch R-KGAT modeling of emotion evolution (e.g., the accuracy of recognizing the state transition from "confusion" to "anxiety" reaches 85.2%) makes the model significantly more robust in dynamic emotional scenarios than the multimodal model without knowledge enhancement. To clarify the contribution of each module, ablation experiments were designed as shown in Table 3.

[0122] Table 3 Module Contribution Table

[0123]

[0124] Ablation conclusion:

[0125] The R-KGAT module contributes a 5.7% improvement to the F1 score for structured modeling of emotional knowledge, and is the core solution to the problem of knowledge fragmentation; the HCT module contributes a 4.8% improvement to the F1 score for multimodal deep fusion, and is the key to leveraging the complementarity of multimodal approaches; the MCL module contributes a 4.1% improvement to the F1 score for optimizing data distribution, and is the key to breaking through the bottleneck of abnormal emotion recognition.

[0126] In summary, this invention, through dual-branch dynamic modeling combined with DiffPool hierarchical fusion, achieves more accurate capture of emotion evolution patterns, more systematic knowledge representation, and significantly improved complex emotion reasoning capabilities. The invention introduces a modality-specific encoder, CAB cross-modal attention, and DFG dynamic fusion gate, resulting in higher utilization of complementary modal information, stronger scene adaptability, and superior noise suppression and recognition accuracy compared to static fusion schemes. Furthermore, the invention employs three-level contrastive learning combined with dynamic difficulty weights, leading to stronger representation of scarce and abnormal emotions, more thorough mitigation of data imbalance, and significant improvement in model bias. Finally, this invention achieves multi-dimensional synergy of knowledge, features, and data, comprehensively improving the accuracy, dynamic tracking, robustness, and generalization of emotion recognition. Its overall performance far surpasses that of single-technology optimization schemes, demonstrating promising application prospects.

[0127] The above-described embodiments further illustrate the purpose, technical solution, and advantages of the present invention. It should be understood that the above-described embodiments are merely preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made to the present invention within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A real-time student emotion recognition method based on a dual-branch recursive knowledge graph, characterized in that, include: Obtain the student's emotional knowledge graph and embed it into learning to obtain entity embedding representations; Student information is acquired and the student information and entity embedding representation are input into a trained real-time student emotion recognition model for processing to obtain the real-time student emotion recognition result; student information includes facial images, student audio, and student text; The training process for the student real-time emotion recognition model includes: S1: The entity embedding representation is processed using a recursive knowledge graph modeling module to obtain knowledge graph features; the recursive knowledge graph modeling module consists of a knowledge representation learning branch, a recursive state update branch, and a DiffPool layer. S2: Input student information into the multimodal feature extraction module for processing to obtain adaptive fused multimodal features; S3: Calculate the total loss of the model based on the knowledge graph features and the adaptive fusion of multimodal features; iteratively adjust the model parameters based on the total loss of the model to obtain a well-trained real-time emotion recognition model for students.

2. The real-time student emotion recognition method based on a dual-branch recursive knowledge graph according to claim 1, characterized in that, The recursive knowledge graph modeling module processes entity embedding representations through the following steps: S11: The knowledge representation learning branch is used to process the entity embedding representation to obtain shallow features; S12: The entity embedding representation is processed by a recursive state update branch to obtain deep features; S13: Use the DiffPool layer to fuse shallow and deep features to obtain knowledge graph features.

3. The real-time student emotion recognition method based on a dual-branch recursive knowledge graph according to claim 2, characterized in that, The process of obtaining shallow features includes: A multi-head attention mechanism is used to process the entity embedding representation, and the attention coefficients between the current entity embedding representation and its neighboring entity embedding representations are obtained under each attention head. For each attention head, the neighbor entity embedding representations are weighted and summed according to the attention coefficients to obtain the output features of the current entity embedding representation under each attention head; By concatenating the output features of the current entity embedding representation under all attention heads, we obtain the shallow features of the current entity embedding representation.

4. The real-time student emotion recognition method based on a dual-branch recursive knowledge graph according to claim 3, characterized in that, The formula for obtaining the attention coefficient is: ； in, Let represent the attention coefficients of entity embedding representations i and j under the k-th attention head. This represents the query matrix of the k-th attention head. This represents the key matrix of the k-th attention head. The entity embedding representation i, The entity embedding representation j, This represents the scaling factor for the k-th attention head. The entity embedding represents the set of neighboring entities of entity i. express Activation function This represents the sum of the exponent terms calculated for all neighboring nodes j of node i.

5. The real-time student emotion recognition method based on a dual-branch recursive knowledge graph according to claim 2, characterized in that, The process of obtaining deep features includes: The current entity embedding representation is concatenated with the entity embedding representation of one of its neighbors, and the interaction features between the current entity embedding representation and the neighbor entity embedding representation are calculated using a weight matrix. The interaction features between the current entity embedding representation and all neighbor entities are stacked to form the interaction matrix of the current entity embedding representation. The interaction matrix of the current entity embedding representation is processed using a GRU network to obtain intermediate features; After performing a residual connection between the intermediate feature and the current entity embedding representation, layer normalization is applied to obtain the deep feature of the current entity embedding representation.

6. The real-time student emotion recognition method based on a dual-branch recursive knowledge graph according to claim 1, characterized in that, The multimodal feature extraction module's processing of student information includes: A dedicated encoder was used to extract features from student information, resulting in visual features, speech features, and text features. A cross-modal attention mechanism is used to process visual features, speech features, and text features to obtain video enhancement features, speech enhancement features, and text enhancement features; Learn the modal weights of the three features; then, based on the modal weights, perform a weighted summation of the video enhancement features, speech enhancement features, and text enhancement features to obtain adaptive fused multimodal features.

7. The real-time student emotion recognition method based on a dual-branch recursive knowledge graph according to claim 6, characterized in that, The obtained video enhancement features are represented as follows: ； ； ； in, Indicates video enhancement features, Indicates video-to-audio weights. Indicates video-text weights, Representing video features, Representing speech features, Representing text features, Represents the video query matrix. Represents the voice query matrix. Represents a text query matrix. This represents the cross-modal attention scaling factor. express Activation function.

8. The real-time student emotion recognition method based on a dual-branch recursive knowledge graph according to claim 1, characterized in that, The formula for calculating the total loss of the model is: ； in, This represents the total loss of the model. Represents the cross-entropy loss function. This indicates adaptive fusion of multimodal features. Representing the features of a knowledge graph. This indicates a splicing operation. Indicates the true label, Indicates instance-level loss. Indicates category-level centrality loss. Indicates the modal alignment loss. , and These represent instance weights, class weights, and modal weights, respectively.

9. A real-time student emotion recognition method based on a dual-branch recursive knowledge graph according to claim 8, characterized in that, Category-level centrality loss is expressed as: ； in, Indicates the number of categories. This represents the set of samples belonging to category y. This represents the adaptive fusion of multimodal features of the input sample x. The feature center of category y is represented. This represents the distance metric function.

Images