Multi-modal academic emotion recognition method and system based on label distribution learning

Abstract

The invention belongs to the technical field of emotion recognition, and particularly provides a multi-modal academic emotion recognition method and system based on label distribution learning, and the method comprises the steps: obtaining a multi-modal academic emotion recognition data set, and extracting single-modal academic emotion features; inputting the multi-modal features into a multi-modal academic emotion recognition neural network, and using an emotion prompt vector as a prompt word for cross-modal feature decoupling; performing dynamic emotion feature fusion according to academic emotion distribution; calculating an academic emotional loss function according to the regression task; and carrying out academic emotion prediction on the updated multi-modal academic emotion recognition neural network. The teaching quality can be accurately evaluated, and teachers can be helped to improve the teaching and guiding quality. Meanwhile, the method can be widely applied to academic emotion recognition, and is beneficial to monitoring the associated emotion state of the student in the teaching process, finding the emotion problem of the student in time and providing help.

Description

Technical Field

[0001] This disclosure relates to the field of emotion recognition technology, and in particular to a multimodal academic emotion recognition method and system based on label distribution learning. Background Technology

[0002] Accurately understanding students' academic emotions helps in accurately assessing teaching quality, optimizing teaching methods, and understanding students' psychological state, enabling timely psychological counseling. Traditional emotion recognition methods include psychological counseling, questionnaires, and electrocardiograms. However, due to the large number of students and limited teacher resources, as well as the often hidden nature of student emotions, these methods struggle to accurately and comprehensively capture the emotional state of each student. Currently, with the continuous development of artificial intelligence technology, multimodal emotion recognition technology offers a new direction for understanding student emotions. This technology can utilize the diversity of emotional information from different modalities to capture more accurate academic emotions.

[0003] However, multimodal sentiment data is inherently ambiguous, specifically in that: the emotions implied by different modalities in multimodal sentiment analysis are complex and may contain significant conflicts; and sentiment labels are easily influenced by the labeler's experience and cultural differences, making them subjective. Existing methods use single-point mapping to learn cross-modal fusion emotions, but this approach struggles to model the ambiguity inherent in multimodal sentiment data. Summary of the Invention

[0004] This disclosure aims to address at least one of the technical problems existing in the prior art, and proposes a multimodal academic emotion recognition method and system based on label distribution learning.

[0005] Firstly, this disclosure provides a multimodal academic emotion recognition method based on label distribution learning, including the following steps:

[0006] S1, Obtain the multimodal academic emotion recognition dataset and extract single-modal academic emotion features;

[0007] S2, Initialize the multimodal academic emotion recognition neural network and emotion cue vector;

[0008] S3, randomly select multimodal academic emotion samples, input the multimodal features into the multimodal academic emotion recognition neural network, and use the emotion cue vector as cue words for cross-modal feature decoupling;

[0009] S4, dynamic emotional feature fusion based on academic emotion distribution;

[0010] S5, Calculate the academic emotion loss function based on the regression task;

[0011] S6. Update the multimodal academic emotion recognition neural network according to the gradient of the loss function. When the stopping condition is met, continue to step S7; otherwise, return to step S3 and continue to cycle through steps S3 to S6.

[0012] S7 extracts features from the student emotion video to be predicted and inputs them into the updated multimodal academic emotion recognition neural network to predict academic emotions.

[0013] Preferably, the construction of the multimodal academic emotion recognition dataset in S1 specifically includes:

[0014] First, we use existing multimodal sentiment analysis data as the basic corpus and perform multimodal label distribution remapping to adjust the multimodal sentiment label distribution of the source data to the target academic sentiment distribution. Extreme negative, neutral, and extreme positive sentiments are mapped to extreme negative, neutral, and extreme positive sentiments in academic subjects, respectively.

[0015] Then, unimodal academic emotion annotations are added to the multimodal data to obtain a multimodal academic emotion recognition dataset.

[0016] Preferably, the extraction of single-modal academic emotion features in S1 specifically includes:

[0017] The text, image, and audio objects are separated from multimodal student videos, and then word-level text features, face-related image features, and fused audio features spliced ​​from multiple audio features are extracted from these objects.

[0018] Preferably, S4 specifically includes:

[0019] The academic emotion encoding module based on the attention mechanism encodes features of audio and image modalities, unifying the feature dimensions of each modality;

[0020] The unimodal academic emotion features and emotion cue vectors are input together into the feature decoupling layer for feature decoupling.

[0021] Preferably, the loss function in S5 is the academic emotion prediction loss and the consistency comparison loss function of the academic emotion distribution, specifically including:

[0022] For a given emotion distribution, construct a positive distribution set and a negative distribution set respectively, and use the negative bulldozer distance to measure the similarity between different academic emotion distributions;

[0023] By constructing a contrastive learning loss model based on academic emotions, the distance between positive distributions is narrowed, while the distance between negative distributions is widened. Dynamic penalty weights are then used to dynamically enhance the degree of penalty between different distributions.

[0024] Preferably, the extraction of single-modal academic emotion features in S1 specifically includes:

[0025] For the text modality, the speech-transcribed text is extracted from each student's video, and the original text at the text level is used as the primary text feature.

[0026] For image modalities, OpenFace 2.0 was used to extract the positional attributes of students' eye movements, facial key points, and facial expressions in each image frame. These attributes were then sampled using average pooling every 3 frames. All the obtained features were then concatenated according to the frame dimension to obtain a visual modal feature of size 55×709.

[0027] For audio modalities, audio is sampled and extracted from the video. The zero-crossing rate, Mel frequency cepstral coefficients, and Chroma CQT of the audio are extracted using the librosa tool. After being concatenated according to the feature dimension scale, the audio modal features of the student video in size 400×33 are obtained.

[0028] Preferably, the multimodal feature input in S3 specifically includes:

[0029] For multimodal academic emotion samples, the input includes audio modal features, text modal features, and visual modal features; and an attention-based academic emotion encoding module is used to process the audio modal features and visual modal features, and their feature dimensions are mapped to a uniform length to unify the feature cue vector with the dimensions of audio, text, and visual modal features.

[0030] Preferably, in step S3, using emotion cue vectors as cue words for cross-modal feature decoupling specifically includes:

[0031] First, the unimodal cue vector is concatenated with the text features and used as the query (Q), key (K), and value (V). Multi-head attention is then performed to obtain an audio and visual emotion cue vector that aggregates the text guidance semantics.

[0032] Then, it is used as a query (Q), and the emotional semantics are activated in their respective modal features. Based on different semantic attributes, a fixed-length emotional embedding is decomposed.

[0033] Finally, residual connections and layer normalization are used to obtain the sentiment cues and text features of the shallow decoupling layer after preliminary decoupling.

[0034] The obtained shallow cue vectors and text features are used as input to the next decoupling layer for deep activation and decoupling of single-modal features.

[0035] Preferably, S3 specifically includes:

[0036] For each emotion attribute in the decoupled features obtained from the last decoupled layer, predict its emotion intensity and obtain the emotion distribution of the single-modal features by normalization.

[0037] Meanwhile, a multi-head attention module is used to enhance the emotional attributes across modalities to extract high-level audio and visual emotional features and multimodal emotional features, which preserves high-level semantic information and reduces noise information.

[0038] This application also provides a multimodal academic emotion recognition system based on label distribution learning. The system can be used to implement the aforementioned multimodal academic emotion recognition method based on label distribution learning. The system includes:

[0039] The module acquires data to construct a multimodal academic emotion recognition dataset and extracts single-modal academic emotion features.

[0040] Initialize the module to initialize the multimodal academic emotion recognition neural network and emotion cue vectors;

[0041] The decoupling module randomly extracts multimodal academic emotion samples, inputs the multimodal features into the multimodal academic emotion recognition neural network, and uses emotion cue vectors as cue words for cross-modal feature decoupling;

[0042] The feature fusion module performs dynamic emotion feature fusion based on the distribution of academic emotions.

[0043] The loss construction module calculates the academic emotion loss function based on the regression task.

[0044] The training and update module updates the multimodal academic emotion recognition neural network according to the gradient of the loss function. When the stopping condition is met, it continues to step S7; otherwise, it returns to step S3 and continues to cycle through steps S3 to S6.

[0045] The output module extracts features from the student emotion video to be predicted and inputs them into the updated multimodal academic emotion recognition neural network to predict academic emotions.

[0046] Beneficial effects:

[0047] 1. An emotion cue vector is introduced to guide the decoupling of single-modal features into fine-grained emotion attributes, thus promoting intermodal interaction and feature fusion;

[0048] 2. A neural network model for emotion decoupling was designed. Guided by the emotion cue vector, the unimodal features were decomposed, the emotion distribution was extracted, and the single-point mapping was transformed into distribution learning, thereby reducing the influence of ambiguity in multimodal data.

[0049] 3. A distribution consistency contrast loss was designed to enhance the representation capability of multimodal features;

[0050] 4. Because the present invention uses the above-mentioned technical methods, the accuracy of multimodal student emotion recognition is improved. Attached Figure Description

[0051] Figure 1 The attached figure is a flowchart of a multimodal academic emotion recognition method based on label distribution learning according to the present invention;

[0052] Figure 2 The attached figure is a flowchart illustrating the dataset construction process of a multimodal academic emotion recognition method based on label distribution learning according to the present invention.

[0053] Figure 3 The attached figure shows an example dataset of a multimodal academic emotion recognition method based on label distribution learning according to the present invention.

[0054] Figure 4 The attached figure is a flowchart of the distribution set sampling process for a multimodal academic emotion recognition method based on label distribution learning according to the present invention. Detailed Implementation

[0055] To enable those skilled in the art to better understand the technical solutions of this disclosure, the disclosure will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0056] The following is combined with Figures 1 to 4 The specific steps of a multimodal academic emotion recognition method based on label distribution learning provided in this application are as follows:

[0057] Step 1: Construct a multimodal academic emotion recognition dataset and extract unimodal academic emotion features;

[0058] Step 2: Initialize the multimodal academic emotion recognition neural network and emotion cue vectors;

[0059] Step 3: Randomly select multimodal academic emotion samples, and input the multimodal features and emotion cue vectors into the neural network for emotion decoupling;

[0060] Step 4: Perform dynamic emotion feature fusion based on the distribution of academic emotions;

[0061] Step 5: Calculate the academic emotional loss function based on the regression task;

[0062] Step 6: Update the multimodal academic emotion recognition neural network according to the gradient of the loss function. When the stopping condition is met, continue to step 7; otherwise, return to step 3 and continue to cycle through steps 3 to 6.

[0063] Step 7: Import the multimodal academic emotion recognition neural network and emotion cue word vectors;

[0064] Step 8: Extract features from the video of the student's emotions to be predicted, and then input them into the neural network to predict academic emotions.

[0065] The multimodal academic emotion recognition method based on label distribution learning provided in this invention also has the following technical features:

[0066] In step 1, a multimodal academic emotion recognition dataset is constructed, such as... Figure 2 The flowchart of the dataset construction method for multimodal academic emotion recognition based on label distribution is shown. It includes: using existing multimodal emotion analysis data as the basic corpus and performing multimodal label distribution remapping, adjusting the multimodal emotion label distribution of the source data to the target academic emotion distribution, mapping extremely negative, neutral, and extremely positive emotions to extremely negative, neutral, and extremely positive academic emotions, respectively. Then, unimodal academic emotion annotations are added to the multimodal data, thus completing the dataset construction.

[0067] The extracted unimodal features include: word-level text features; face-related image features; and fused audio features composed of multiple audio features. Features are extracted from text, image, and audio objects separated from multimodal student videos.

[0068] The multimodal academic emotion recognition method based on label distribution learning described above in this invention may also have the following additional technical features:

[0069] In step 2, the emotion cue vector serves as a cue word for cross-modal feature decoupling.

[0070] The multimodal academic emotion recognition method based on label distribution learning provided in this invention also has the following technical features:

[0071] Step 3, which involves decoupling the multimodal academic emotion features and emotion cue vectors, includes:

[0072] The academic emotion encoding module based on the attention mechanism encodes features of audio and image modalities, unifying the feature dimensions of each modality;

[0073] The unimodal academic emotion features and emotion cue vectors are input together into the feature decoupling layer for feature decoupling.

[0074] The multimodal academic emotion recognition method based on label distribution learning described above in this invention may also have the following additional technical features:

[0075] In step 4, academic emotion characteristics are dynamically integrated based on the distribution of academic emotions.

[0076] The multimodal academic emotion recognition method based on label distribution learning described above in this invention may also have the following additional technical features:

[0077] In step 5, the loss function is the academic emotion prediction loss and the academic emotion distribution consistency comparison loss.

[0078] To achieve the above objectives, the multimodal academic emotion recognition method based on label distribution learning described in this invention further includes the extraction of single-modal academic emotion features, which includes:

[0079] For the text modality, the speech-transcribed text is extracted from each student's video, and the original text at the text level is used as the primary text feature.

[0080] For the image modality, OpenFace2.0[1] was used to extract the positional attributes of the student’s eye movements, facial key points, facial expressions and other attributes in each image frame, and the average pooling was performed every 3 frames. All the features obtained were spliced ​​together according to the frame dimension to obtain a visual modality feature of size 55×709.

[0081] For audio modalities, the audio in the video was sampled and extracted, and the zero-crossing rate, Mel frequency cepstral coefficients and Chroma CQT of the audio were extracted using the librosa[2] tool. After being spliced ​​according to the feature dimension scale, the audio modal features of the student video of size 400×33 were obtained.

[0082] Furthermore, the details of the emotion prompt vector in the multimodal academic emotion recognition method based on label distribution learning described above in this invention are as follows:

[0083] In this approach, emotion cue vectors are introduced for both audio and visual modalities. The cue vector for the audio modal is... The visual modality cue vector is ,in, Indicates the length of the prompt. To represent the dimension of a vector, use , , These represent text, image, and audio modalities, respectively.

[0084] The multimodal academic emotion recognition method based on label distribution learning described above in this invention further includes the following multimodal features:

[0085] Among them, for multimodal academic emotion samples Includes audio modal features Text modal features and visual modal features , , , express The length of the feature sequence of a mode. express Dimensions of modal features.

[0086] The multimodal academic emotion recognition method based on label distribution learning described above in this invention encodes features of audio and image modalities through an attention-based academic emotion encoding module, unifying the feature dimensions of each modality.

[0087] Among them, the academic emotion encoding module based on the attention mechanism models the relationship between the overall feature tendency and local emotion of single modality features through a multi-head self-attention mechanism, and then encodes audio modality features through a feedforward fully connected layer and a normalization layer. and visual modal features And map its feature dimensions to This unifies the dimension of feature hint vectors with those of audio, text, and visual modal features.

[0088] The feature decoupling layer in the multimodal academic emotion recognition method based on label distribution learning described above in this invention specifically includes:

[0089] A multi-head attention mechanism is used to obtain a global sentiment representation of the text modality, and important contextual features of the text modality are aggregated into the sentiment cue vector.

[0090] By leveraging emotional cues, a multi-head attention mechanism is used to activate and decouple audio and visual features separately. The decomposed multimodal features are then concatenated along the sequence dimension, and a linear transformation with residual connections is applied to obtain the final aggregated emotional features.

[0091] The decoupling layer in the multimodal academic emotion recognition method based on label distribution learning described above in this invention comprises the following specific steps:

[0092] Among them, modal cue vectors and Text features After concatenation, it becomes a query. Multi-head attention computation is performed using Key (K) and Value (V).

[0093] :

[0094]

[0095]

[0096] in, Indicates the first Layer decoupling layer.

[0097] The audio and visual emotion cues vectors, which aggregate textual guidance semantics, are used as the query Query(Q). Emotional semantics are activated within their respective modal features, and the unimodal academic emotion features are decomposed according to different semantic attributes:

[0098]

[0099]

[0100] in, , To obtain decoupling Modal characteristics.

[0101] Then, using residual connections and layer normalization, we obtain... The output of the decoupling layer.

[0102]

[0103]

[0104] in, It is a layer normalization function. , , As the first The output of the decoupling layer is then input to... The decoupling layer is used as input to further activate and decouple the single-modal features.

[0105] As described above, including Layer decoupling layer.

[0106] The multimodal academic emotion recognition method based on label distribution learning described above in this invention, which dynamically fuses emotion features, further includes the following specific steps:

[0107] For the Each emotion attribute in the decoupled features obtained from the decoupling layer is predicted to have its emotion intensity, and the emotion distribution of the single-modal features is obtained by normalization:

[0108]

[0109]

[0110] in, , It is the calculated single-modal emotion distribution.

[0111] Simultaneously, a multi-head attention module is used to enhance emotional attributes across modalities:

[0112]

[0113]

[0114]

[0115]

[0116] in, and These are the extracted audio and visual modal features, respectively. To integrate multimodal emotion features, it preserves high-level semantic information and reduces noise information.

[0117] The multimodal academic emotion recognition method based on label distribution learning described above in this invention further includes a prediction loss characterized by:

[0118] For each student's emotion sample, the prediction loss includes a unimodal prediction loss and a multimodal prediction loss:

[0119]

[0120] in, This is a unimodal sentiment prediction result. It is a real multimodal sentiment prediction label. , This is a multimodal sentiment prediction result. It is a real multimodal emotion label.

[0121] The multimodal academic emotion recognition method based on label distribution learning described above in this invention includes an academic emotion distribution consistency contrast loss, wherein, for samples... The obtained modes Emotional distribution , Construct positive distribution sets respectively and negative distribution set , Therefore, audio modality The consistency loss of academic emotion distribution is:

[0122]

[0123] in, Represents the distribution set Size, Calculate the distribution using bulldozer distance (EMD) and Similarity between them It is the dynamic penalty weight for negative samples.

[0124] Similarly, visual modal The consistency loss of academic emotion distribution is:

[0125]

[0126] Finally, the sample The consistency loss of academic emotion distribution is:

[0127]

[0128] The multimodal academic emotion recognition method based on label distribution learning described above in this invention, wherein the academic emotion distribution consistency contrast loss has a positive distribution set. and negative distribution set The construction steps are as follows: where, for anchor distribution The corresponding real unimodal emotion label is Predicting unimodal sentiment labels as From a training batch The set of positively distributed samples and negative distribution set The following conditions must be met:

[0129]

[0130]

[0131] in, , .

[0132] The multimodal academic emotion recognition method based on label distribution learning described above in this invention has the following final academic emotion loss function:

[0133]

[0134] in, It is a balanced prediction loss Distribution consistency loss The weight.

[0135] The experiment used the CMU-MOSI multimodal sentiment dataset as the basic academic data, containing 2199 opinion video clips. Each video corresponds to one multimodal label, which is mapped to a multimodal sentiment annotation ranging from -3 (extremely negative) to 3 (extremely positive). During the dataset construction process, a unimodal academic sentiment annotation was added to each sample. In the experiment, the dataset was randomly divided into a training set of 1284 samples, a validation set of 229 samples, and a test set of 686 samples. Figure 3This is an example image of a dataset sample for a multimodal academic emotion recognition method based on label distribution learning according to the present invention. Then, the video features of each viewpoint are extracted and uniformly normalized to obtain a multimodal viewpoint dataset. Using the pre-trained RoBERTa[3] model, the primary text features are encoded into word vectors with a dimension of 1024.

[0136] Furthermore, the neural network parameters were randomly initialized, and eight emotion cue vectors were initialized for the audio and visual modalities respectively, with a feature dimension of 512.

[0137] Subsequently, a batch of samples was randomly selected, and the text, audio, and image features, along with the emotion cues vector, were input into the neural network for emotion decoupling.

[0138] In this embodiment, the input unimodal data first passes through a two-layer attention-based academic emotion encoding module to uniformly map the unimodal data to... The modal features are then fed into a decoupling layer for full decoupling. An 8-head attention module is used in the decoupling layer to activate the modal features. After two decoupling layers, guided by the text modality, eight audio sentiment attributes and eight image sentiment attributes are obtained.

[0139] In the feature fusion stage, the emotion prediction layer is used to predict the emotion intensity of each emotion attribute, and the emotion features are dynamically fused based on the emotion intensity.

[0140] Figure 4 To further explain in detail the construction process of the positive and negative distribution sets during the loss calculation stage:

[0141] For each distribution obtained by the above techniques its modes The emotion label is Sentiment prediction is All samples underwent the following sampling process to construct the distribution. positive distribution set and negative distribution set The sampling process is as follows:

[0142] From training batches Medium sampling distribution This distribution mode The emotion label is Sentiment prediction is If the emotion label meets the conditions If the sentiment labels of the two distributions are similar, then the distributions will be... Add to positive distribution set If the sentiment labels of the two distributions do not satisfy the above conditions, the predicted labels do. If the sentiment labels of two distributions are dissimilar, but the sentiment predictions are similar, then the distributions will be... Add to negative distribution set If the above conditions are not met, then the distribution... Do not add it to any set. Repeat the above steps until the training batch contains no additional sets. All distributions have been examined.

[0143] The example was trained on a single NVIDIA RTX 4090 GPU device, and the relevant settings for the example are shown in Table 1.

[0144] Table 1 Parameter Settings for Example

[0145]

[0146] Experiments were conducted on the CMU-MOSI dataset and compared with existing methods, including Self-MM[4], ALMT[5], and ConFEDE[6]. The experimental results are shown in Table 2.

[0147] Table 2 Comparison of Experimental Results

[0148]

[0149] The experiments show that the method described in this invention achieves the best performance across all major evaluation metrics. Specifically, in Acc-2, it reaches 84.84%, an improvement of 0.29% compared to the ALMT method (84.55%). Regarding the F1 score, the method described in this invention reaches 84.67%, an improvement of 0.10%, higher than ALMT's 84.57%. Furthermore, the method described in this invention achieves the lowest mean absolute error (MAE) of 0.637 and the highest Corr of 0.836. Therefore, this method can effectively improve multiple metrics and has a positive effect on multimodal academic emotion recognition.

[0150] This invention helps to accurately assess teaching quality and assists teachers in improving their teaching and guidance. Simultaneously, this invention can be widely applied to academic emotion recognition and is beneficial for monitoring students' emotional states during the teaching process, promptly identifying students' emotional problems and providing assistance. As seen in the embodiments of this invention, the method uses emotion cue vectors to guide the decoupling of unimodal features, obtaining a unimodal emotion distribution, thereby enhancing the emotion representation range of unimodal features and improving the robustness to emotion ambiguity in multimodal academic emotion data.

[0151] This application also provides a multimodal academic emotion recognition system based on label distribution learning. This system can be used to implement the aforementioned multimodal academic emotion recognition method based on label distribution learning. The system includes:

[0152] The module acquires data to construct a multimodal academic emotion recognition dataset and extracts single-modal academic emotion features.

[0153] Initialize the module to initialize the multimodal academic emotion recognition neural network and emotion cue vectors;

[0154] The decoupling module randomly extracts multimodal academic emotion samples, inputs the multimodal features into the multimodal academic emotion recognition neural network, and uses emotion cue vectors as cue words for cross-modal feature decoupling;

[0155] The feature fusion module performs dynamic emotion feature fusion based on the distribution of academic emotions.

[0156] The loss construction module calculates the academic emotion loss function based on the regression task.

[0157] The training and update module updates the multimodal academic emotion recognition neural network according to the gradient of the loss function. When the stopping condition is met, it continues to step S7; otherwise, it returns to step S3 and continues to cycle through steps S3 to S6.

[0158] The output module extracts features from the student emotion video to be predicted and inputs them into the updated multimodal academic emotion recognition neural network to predict academic emotions.

[0159] It is understood that the above embodiments are merely exemplary embodiments used to illustrate the principles of this disclosure, and this disclosure is not limited thereto. For those skilled in the art, various modifications and improvements can be made without departing from the spirit and substance of this disclosure, and these modifications and improvements are also considered to be within the scope of protection of this disclosure.

Claims

1. A multimodal academic emotion recognition method based on label distribution learning, characterized in that, Includes the following steps: S1, Obtain the multimodal academic emotion recognition dataset and extract single-modal academic emotion features; S2, Initialize the multimodal academic emotion recognition neural network and emotion cue vector; S3, randomly select multimodal academic emotion samples, input the multimodal features into the multimodal academic emotion recognition neural network, and use the emotion cue vector as cue words for cross-modal feature decoupling; S4, dynamic emotional feature fusion based on academic emotion distribution; S5, Calculate the academic emotion loss function based on the regression task; S6. Update the multimodal academic emotion recognition neural network according to the gradient of the loss function. When the stopping condition is met, continue to step S7; otherwise, return to step S3 and continue to cycle through steps S3 to S6. S7 extracts features from the student emotion video to be predicted and inputs them into the updated multimodal academic emotion recognition neural network to predict academic emotions.

2. The multimodal academic emotion recognition method based on label distribution learning according to claim 1, characterized in that, The construction of the multimodal academic emotion recognition dataset in S1 specifically includes: First, we use existing multimodal sentiment analysis data as the basic corpus and perform multimodal label distribution remapping to adjust the multimodal sentiment label distribution of the source data to the target academic sentiment distribution. Extreme negative, neutral, and extreme positive sentiments are mapped to extreme negative, neutral, and extreme positive sentiments in academic subjects, respectively. Then, unimodal academic emotion annotations are added to the multimodal data to obtain a multimodal academic emotion recognition dataset.

3. The multimodal academic emotion recognition method based on label distribution learning according to claim 1, characterized in that, The extraction of unimodal academic emotion features in S1 specifically includes: The text, image, and audio objects are separated from multimodal student videos, and then word-level text features, face-related image features, and fused audio features spliced ​​from multiple audio features are extracted from these objects.

4. The multimodal academic emotion recognition method based on label distribution learning according to claim 1, characterized in that, S4 specifically includes: The academic emotion encoding module based on the attention mechanism encodes features of audio and image modalities, unifying the feature dimensions of each modality; The unimodal academic emotion features and emotion cue vectors are input together into the feature decoupling layer for feature decoupling.

5. The multimodal academic emotion recognition method based on label distribution learning according to claim 1, characterized in that, The loss function in S5 is the academic sentiment prediction loss and the consistency comparison loss function of the academic sentiment distribution, specifically including: For a given emotion distribution, construct a positive distribution set and a negative distribution set respectively, and use the negative bulldozer distance to measure the similarity between different academic emotion distributions; By constructing a contrastive learning loss model based on academic emotions, the distance between positive distributions is narrowed, while the distance between negative distributions is widened. Dynamic penalty weights are then used to dynamically enhance the degree of penalty between different distributions.

6. The multimodal academic emotion recognition method based on label distribution learning according to claim 1, characterized in that, The extraction of unimodal academic emotion features in S1 specifically includes: For the text modality, the speech-transcribed text is extracted from each student's video, and the original text at the text level is used as the primary text feature. For image modalities, OpenFace 2.0 was used to extract the positional attributes of students' eye movements, facial key points, and facial expressions in each image frame. These attributes were then sampled using average pooling every 3 frames. All the obtained features were then concatenated according to the frame dimension to obtain a visual modal feature of size 55×709. For audio modalities, audio is sampled and extracted from the video. The zero-crossing rate, Mel frequency cepstral coefficients, and Chroma CQT of the audio are extracted using the librosa tool. After being concatenated according to the feature dimension scale, the audio modal features of the student video in size 400×33 are obtained.

7. The multimodal academic emotion recognition method based on label distribution learning according to claim 1, characterized in that, The multimodal feature inputs in S3 specifically include: For multimodal academic emotion samples, the input includes audio modal features, text modal features, and visual modal features; and an attention-based academic emotion encoding module is used to process the audio modal features and visual modal features, and their feature dimensions are mapped to a uniform length to unify the feature cue vector with the dimensions of audio, text, and visual modal features.

8. The multimodal academic emotion recognition method based on label distribution learning according to claim 1, characterized in that, The use of sentiment cue vectors as cue words for cross-modal feature decoupling in S3 specifically includes: First, the unimodal cue vector is concatenated with the text features and used as the query (Q), key (K), and value (V). Multi-head attention is then performed to obtain an audio and visual emotion cue vector that aggregates the text guidance semantics. Then, it is used as a query (Q), and the emotional semantics are activated in their respective modal features. Based on different semantic attributes, a fixed-length emotional embedding is decomposed. Finally, residual connections and layer normalization are used to obtain the sentiment cues and text features of the shallow decoupling layer after preliminary decoupling. The obtained shallow cue vectors and text features are used as input to the next decoupling layer for deep activation and decoupling of single-modal features.

9. The multimodal academic emotion recognition method based on label distribution learning according to claim 8, characterized in that, S3 specifically includes: For each emotion attribute in the decoupled features obtained from the last decoupled layer, predict its emotion intensity and obtain the emotion distribution of the single-modal features by normalization. Meanwhile, a multi-head attention module is used to enhance the emotional attributes across modalities to extract high-level audio and visual emotional features and multimodal emotional features, which preserves high-level semantic information and reduces noise information.

10. A multimodal academic emotion recognition system based on label distribution learning, characterized in that, The system can be used to implement the multimodal academic emotion recognition method based on label distribution learning as described in any one of claims 1 to 9, and the system includes: The module acquires data to construct a multimodal academic emotion recognition dataset and extracts single-modal academic emotion features. Initialize the module to initialize the multimodal academic emotion recognition neural network and emotion cue vectors; The decoupling module randomly extracts multimodal academic emotion samples, inputs the multimodal features into the multimodal academic emotion recognition neural network, and uses emotion cue vectors as cue words for cross-modal feature decoupling; The feature fusion module performs dynamic emotion feature fusion based on the distribution of academic emotions. The loss construction module calculates the academic emotion loss function based on the regression task. The training and update module updates the multimodal academic emotion recognition neural network according to the gradient of the loss function. When the stopping condition is met, it continues to step S7; otherwise, it returns to step S3 and continues to cycle through steps S3 to S6. The output module extracts features from the student emotion video to be predicted and inputs them into the updated multimodal academic emotion recognition neural network to predict academic emotions.

Images