Multi-modal emotion recognition method based on dynamic EMA knowledge distillation

Abstract

The invention discloses a multi-modal emotion recognition method based on dynamic EMA knowledge distillation, and the method comprises the steps: firstly collecting the modal information of audio, video and text, obtaining multi-modal features through preprocessing, and respectively inputting the multi-modal features into a parallel teacher path and a parallel student path; and simulating a modal missing condition to respectively obtain teacher path characteristics and student path characteristics. The method comprises the steps that firstly, teacher path characteristics and student path characteristics are subjected to dynamic routing fusion, fused teacher characteristics and fused student characteristics are obtained, then the fused teacher characteristics and the fused student characteristics are subjected to self-adaptive knowledge distillation, and student path parameters and teacher path parameters are optimized. And finally, in a test stage, obtaining fusion features from the input data through a student path, and realizing emotion prediction through a full connection layer. According to the method, the model has better robustness in the face of mode missing, and the knowledge solidification problem of traditional distillation in a missing scene is solved.

Description

Technical Field

[0001] This invention belongs to the field of affective computing, and in particular relates to a multimodal affective recognition method based on dynamic EMA knowledge distillation in multimodal affective analysis. Background Technology

[0002] Multimodal emotion recognition technology aims to improve the accuracy of emotion classification by fusing information from multiple modalities, such as speech, text, and vision. Early methods mainly adopted decision-level fusion strategies, such as weighted voting or ensemble learning of single-modal classification results. Although these methods are simple to implement, they fail to fully exploit the complementary information between modalities because the interaction only occurs at the decision level, resulting in limited performance. Subsequent feature-level fusion methods, such as Tensor Fusion Networks (TFN) and Low-Rank Multimodal Fusion (LMF), attempted to perform modality alignment and fusion at the feature level. However, these methods often destroy modality specificity by forcing feature space alignment and their performance deteriorates sharply when modality is missing or there is noise interference.

[0003] One approach uses traditional knowledge distillation techniques to guide student models with a fixed teacher model. However, this introduces additional parameters, and the teacher model parameters remain static during distillation, failing to capture dynamic feature changes during student model training. Furthermore, fixed-weight knowledge transfer struggles to adapt to optimal fusion strategies across different modality combinations. Exponential Moving Average (EMA) techniques have demonstrated unique value in model optimization in recent years, dynamically fusing historical parameters with the current state through a decay coefficient to form a smooth and stable teacher model. However, existing EMA methods still have limitations in multimodal scenarios, particularly lacking adaptive mechanisms for dynamic modality loss. When the input modality combination changes, fixed EMA strategies cannot dynamically adjust the weight coefficients of knowledge transfer across modalities, leading to over-suppression or incorrect reinforcement of some modality features. Summary of the Invention

[0004] This invention addresses the issue of decreased recognition accuracy due to dynamic modality loss in the field of multimodal emotion recognition, as well as the shortcomings of traditional knowledge distillation techniques such as knowledge rigidity and modality weight instability. It proposes a multimodal emotion recognition method based on dynamic EMA knowledge distillation. This mechanism achieves robust emotion recognition in modality-loss scenarios through EMA knowledge distillation of dynamic knowledge distillation intensity.

[0005] To achieve the above-mentioned objectives of this invention, the technical solution adopted is as follows:

[0006] A multimodal emotion recognition method based on dynamic EMA knowledge distillation, the specific steps of which include:

[0007] S1: First, modal information including audio, video, and text is collected. Audio information is directly recorded via microphone during actor dialogue; video information is recorded using a camera, with some actors wearing headbands and wristbands marked with 53 reflective markers on their faces, tracked by a motion capture system; text information is obtained by manually transcribing the speech content. Feature preprocessing is performed on each modality. Pre-trained models are used to extract the raw features from audio, video, and text respectively. Then, projection layers and Dropout are used to map the modal features to a unified dimension, resulting in multimodal features of a unified dimension.

[0008] S2: Input the unified-dimensional multimodal features output from S1 into the parallel teacher path and student path respectively; the teacher path is trained using full-modal features, while the student path applies a dynamic mask to the input features to simulate the case of missing modalities, thus obtaining teacher path features and student path features respectively.

[0009] S3: Perform dynamic routing fusion on the teacher path features and student path features output from S2 to obtain fused teacher features and fused student features.

[0010] S4: Adaptive knowledge distillation is performed on the fused features obtained in S3, where the distillation intensity is automatically adjusted by the route weights generated by the dynamic routing. Student path parameters are optimized using gradient descent, while teacher path parameters are updated using exponential moving average (EMA).

[0011] S5: Test the student paths obtained in S4. The input data is used to generate the final fused features through the student paths, and sentiment prediction is achieved through a fully connected layer.

[0012] The specific methods for feature preprocessing for each modality in step one include:

[0013] For audio modalities, the original audio signal is input, and frame-level features are extracted through a multi-layer convolutional neural network of Wav2Vec. The frame-level features of the entire speech segment are aggregated to obtain an audio feature vector.

[0014] For the text modality, the original text sequence is input, the text is encoded through the DeBERTa Transformer layer, and the output of the [CLS] token is extracted as the sentence representation to obtain the text feature vector.

[0015] For the visual modality, a multi-task convolutional neural network (MTCNN) is first used to detect and align face regions from video frames. The aligned face image sequence is then input, and the visual features of each frame are extracted through a multi-scale attention network (MA-Net). The frame-level features are then averaged and pooled to obtain the visual feature vector.

[0016] Then, the audio features, video features, and text features are mapped to a unified dimension through a projection layer.

[0017] The specific methods for processing multimodal features of the same dimension in step two include:

[0018] For teacher paths, we use unmasked full-modal features for training, while for student paths, we generate dynamic masks. To maintain consistency between training and testing, we mask each batch of student paths using seven preset combinations of masking methods.

[0019] The specific methods for processing teacher path features and student path features in step three include:

[0020] The features of the teacher's path are fused by different expert encoders and then fused according to the dynamic routing weights to obtain the weighted mixed teacher features. The features of the student's path are also fused by different expert encoders according to the dynamic routing weights to obtain the weighted mixed student features.

[0021] The specific methods for distilling fused feature knowledge in step four include:

[0022] Adaptive knowledge distillation is performed using hybrid teacher and student features, the strength of which depends on the importance of the available modalities in the current batch. Student paths are updated via gradient descent, while teacher paths are updated synchronously using EMA.

[0023] The specific methods for testing student characteristics in step five include:

[0024] During testing, the input data is routed through the student path to obtain mixed student features, which are then used for final sentiment prediction via a fully connected layer.

[0025] This invention achieves breakthroughs in modal interaction modeling and robustness in missing scenarios by utilizing a multimodal emotion recognition method based on dynamic EMA knowledge distillation and introducing a dual-path architecture and adaptive distillation mechanism. Compared to existing technologies, it has the following significant advantages:

[0026] In terms of modality missing adaptability, this invention uses seven predefined mask patterns to fully cover the diverse missing conditions in real-world scenarios, achieving consistency between training and testing. Combined with a mask-aware routing strategy, it makes the model more robust when facing modality missing conditions.

[0027] In terms of knowledge distillation mechanism, this invention solves the problem of knowledge solidification in missing scenarios in traditional distillation by using a dual-path gradient isolation design (teacher path blocking gradient backpropagation / student path focusing dynamic adaptation). Its dynamically adjusted distillation intensity formula ensures that the model learns better. Attached Figure Description

[0028] Figure 1 This is a schematic diagram of the multimodal emotion recognition method based on dynamic EMA knowledge distillation described in the embodiments of the present invention;

[0029] Figure 2 This is a schematic diagram of the teacher path framework for the multimodal emotion recognition method based on dynamic EMA knowledge distillation described in this embodiment of the invention;

[0030] Figure 3 This is a schematic diagram of the student path framework of the multimodal emotion recognition method based on dynamic EMA knowledge distillation described in the embodiments of the present invention. Detailed Implementation

[0031] To make the objectives, technical solutions, and technical effects of this invention clearer, the invention will be further described in detail below with reference to the embodiments and accompanying drawings.

[0032] like Figure 1 As shown, the multimodal emotion recognition method based on dynamic EMA knowledge distillation of the present invention includes the following steps:

[0033] Step 1: For audio information, a convolutional neural network is used to extract frame-level features and encode them to obtain corresponding feature vectors. For text information, a pre-trained DeBERTa algorithm is used to transform the text information into corresponding vector representations, and finally, feature vectors are obtained on a sentence-by-sentence basis. For video information, features of each frame are extracted using a pre-trained MA-Net algorithm to obtain corresponding feature vectors. Then, projection layers and Dropout are used to map the features of each modality to a unified dimension for subsequent fusion.

[0034] The processing of speech information includes: first, performing a sampling rate consistency check on the original speech signal to confirm that the input audio meets the 16kHz sampling rate requirement; then, directly inputting the verified speech waveform into a pre-trained wav2vec-large deep neural network model; this model uses a feature encoder to perform frame-level feature extraction with a time resolution of 10 milliseconds, generating a 512-dimensional feature sequence F with a time dimension of T. a ∈R T*512 The feature sequence generated from the entire speech segment is subjected to global average pooling to fuse the temporal information of the entire speech segment, and finally outputs a fixed 512-dimensional speech feature vector f. a ∈R 512 The formula is as follows: Where F a(t,:) represents the 512-dimensional feature vector of frame t. Global average pooling can effectively suppress local noise interference and enhance feature robustness. This technique automatically learns the high-order representation of speech through deep neural networks, replacing the traditional manual feature design process. While adapting to speech inputs of different durations, it achieves robust transformation of each audio signal to a fixed-dimensional feature vector.

[0035] For text information, the preprocessing process includes: first, standardizing the original text, including filtering special characters, unifying case conversion, and removing accent marks; adding "[CLS]" tags at the beginning of the text and "[SEP]" tags at the end. The standardized text is then input into a pre-trained DeBERTa deep language model; context encoding is performed using a multi-layer Transformer architecture to extract hidden layer states; weighted fusion of hidden states at specified levels enhances semantic representation capabilities; for words composed of multiple sub-words, average pooling is used to fuse sub-word features and generate word-level embedding vectors; finally, global average pooling is performed on all word vectors in the entire sentence to aggregate global semantic information and output a fixed 1024-dimensional text feature vector f. t ∈R 1024 The multi-head attention calculation for the Transformer encoder is as follows:

[0036] MultiHead(Q,K,V)=Concat(head1,head2,...,head h )·W o

[0037]

[0038] Where h is the default number of attention heads in DeBERTa-base, and d k For each dimension of the attention head, W i Q W i k W i v W O As trainable parameters for the attention layer, relative position encoding can effectively capture long-distance semantic dependencies. This technique models long-distance semantic dependencies through an attention mechanism, replacing traditional bag-of-words or N-Gram features. While eliminating stop word interference, it achieves robust mapping from text of arbitrary length to fixed-dimensional feature vectors.

[0039] For audio information, the preprocessing procedure includes: first, locating and aligning face regions from the original video frames using the MTCNN face detection algorithm; then, uniformly scaling the detected face images to a size of 224×224 pixels and converting them into normalized tensors to obtain a standardized face image sequence I. v ∈R N*224*224*3 (N is the number of sampling frames); the preprocessed images are batch-input into the pre-trained MA-Net model, which captures local details and global facial expression features through multi-scale convolutional layers and attention modules; global average pooling is performed on the features of all video frames within the same speech segment, fusing temporal facial expression change information, and finally outputting a fixed 1024-dimensional visual feature vector f. v ∈R 1024 The formula is: This technology enhances the weight allocation of key areas such as the eyes and mouth through an attention mechanism, maintaining high discriminative power even under changes in head posture and partial occlusion, and achieving robust mapping of dynamic expressions to fixed-dimensional feature vectors.

[0040] The above three types of information are mapped to D through the projection layer and the Dropout layer. e Dimension, formula: h m =Dropout(ReLU(W m ·f m ),m∈{a,t,v}.

[0041] Step 2: Employ a feature processing architecture that parallels teacher and student paths, such as... Figure 2 and Figure 3 As shown, a dynamic masking strategy is used to simulate modality missing scenarios, thereby realizing the distillation foundation for "full-modal teachers guiding students with missing modalities".

[0042] The teacher path feature input is trained using unmasked full-modal features to ensure the teacher model learns complete multimodal complementary information. A modal mask with all values ​​of 1 is generated to preserve the preprocessed audio, text, and visual features (projected to a unified dimension in step one, denoted as "unified feature dimension") without occlusion. The input format is "[sequence length, batch size, unified feature dimension]" ("sequence length" refers to the temporal length of a single data point, and "batch size" refers to the number of samples in a single training iteration). The teacher input features are obtained by concatenating these features along the feature dimension. in, Represents element-wise multiplication, Concat(·) represents feature concatenation, h a ,h t ,h v This represents the preprocessed and projected audio, text, and visual modal features. These represent the modal masks corresponding to the teacher's path, used to control whether features are preserved or occluded. The student path feature mask generation part simulates seven modality loss combinations in a real-world scenario ({audio}, {video}, {text}, {audio, video}, {audio, text}, {video, text}, {audio, video, text}), generating dynamic masks through a "dynamic mask generation function". This function randomly generates a mask matrix that conforms to the preset seven combinations for each training batch, selectively occluding each modal feature of the student path. For example, when only the audio modality is preserved, the mask values ​​corresponding to the text and visual modalities are set to 0, realizing the simulation of modality loss. Similar to the teacher path feature concatenation logic, the unified dimension features are multiplied by the dynamic mask and then concatenated to obtain the student input features. The formula is as follows: in Represents element-wise multiplication, Concat(·) represents feature concatenation, h a ,h t ,h v This represents the modal features after preprocessing and projection. These represent the modal masks corresponding to the student paths.

[0043] Simultaneously, the modal masks for both the teacher and student paths (the default mask for the teacher path is all 1s) are formatted. The single-modal masks for audio, text, and vision are then integrated into a unified feature mask using an "input feature integration function," ensuring dimensionality matching with the input features. The mask is applied to the original features: the teacher path input features are obtained by "audio feature × teacher path audio modal mask + text feature × teacher path text modal mask + visual feature × teacher path visual modal mask," and similarly, the student path input features are calculated using the modal masks for each student path, ultimately forming a parallel input feature matrix for both paths.

[0044] Step 3: This step achieves intelligent fusion of multimodal features through a dynamic routing weight mechanism.

[0045] The expert encoder's feature processing involves inputting the features of the teacher's path and the student's path into the feature encoding module. This module models intramodal temporal dependencies through multi-head attention and outputs encoded single-modal features (including encoded audio features, encoded text features, and encoded visual features). Taking the teacher's path as an example, the encoding process formula is as follows: in These are the audio, text, and visual features of the teacher's path after passing through the expert encoder, f a ,f t ,f v The original features of each modality of the teacher path that are not masked.

[0046] The features obtained after expert encoder processing are used to calculate routing weights. This is done by using three independent multilayer perceptrons to calculate the routing weights for each modality. Taking the audio modality as an example, the audio modality weight calculation calculates the encoded audio features received by the network, outputting a 3-dimensional weight vector (corresponding to the fusion weights of the three modalities). After Softmax normalization, the audio modality routing weight matrix is ​​obtained, with dimensions "[batch size, sequence length, 3]", ensuring the weight sum is 1. The routing weight calculation logic for text and visual modalities is consistent. The encoded single-modal features are reshaped into "[batch size, sequence length, 3, unified feature dimension]" (the new dimension corresponds to the three modalities), multiplied element-wise with the expanded routing weights, and then a weighted sum is calculated by summing by "modal dimension". Taking the teacher path as an example, the formula is as follows: in To integrate teacher characteristics and student characteristics The computational logic remains the same, only the input features are replaced with student-coded features.

[0047] Step 4: Perform knowledge distillation on the student fusion features and teacher fusion features obtained in the previous step, and dynamically adjust the distillation intensity based on modal importance to solve the problems of knowledge solidification and modal weight rigidity in traditional distillation.

[0048] For calculating the distillation loss, the teacher path performs forward propagation via the `forward_with_teacher_params()` function, blocking gradient backpropagation during the process (ensuring that the teacher model parameters are not updated), thus obtaining the features. To ensure the stability of teacher pathway integration features, the formula is as follows: Where θ t M represents the teacher model parameters. t To standardize the characteristic coding for teachers, To ensure stable teacher fusion features, student paths are calculated with gradients to approximate teacher path fusion features through learning. When a student path has missing modalities (determined by the modal masks of each student path; for example, if the audio modal mask is all 0, it indicates missing audio), the route weights stored in the "recent weight storage area" are extracted, and the sum of weights for available modalities is calculated. Specifically, the sum of route weights S for available modalities is calculated, iterating through the three modalities, and only the mean of route weights for non-missing modalities is calculated, as shown in the following formula: in Let represent the global mean of the routing weights for modality m, B represent the batch size, T represent the sequence length, i is the index of the batch dimension, j is the index of the sequence dimension, k is the index of the modality fusion dimension, and miss. m For modal validity mask, dynamic coefficients The distillation loss uses mean squared error (MSE) loss to measure the difference between the student and teacher's combined features, ensuring that the student model learns the teacher's multimodal feature distribution; the task loss is selected according to the dataset type (cross-entropy loss for classification tasks, MSE loss for regression tasks), and the total loss is the weighted sum of the two, as shown in the following formula: L total =L task +λ kd ·L distill , where λ kd For the distillation loss weight, L task For mission losses, L distill This represents the loss during distillation.

[0049] This step optimizes the student model parameters using gradient descent and dynamically updates the teacher model parameters using EMA (Exponential Moving Average).

[0050] The student model parameter optimization involves the following steps: During training iterations, backpropagation is performed on the total loss to calculate the gradient. The Adam optimizer (with a default learning rate of 1e-4 and a default weight decay of 1e-5) is then used to update all trainable parameters of the student model, enabling the student model to learn from the teacher's knowledge. The parameter update formula is as follows: Where k is the number of iterations. Let η be the loss gradient for the student parameters and η be the learning rate.

[0051] Teacher model parameter EMA update: After the student model parameters are updated, a function is called to update the teacher parameters using an exponential moving average, ensuring that the teacher model smoothly tracks the optimal state of the student model. The formula is as follows: Where α = 0.999 is the EMA attenuation coefficient, and when initializing the teacher parameters, the initial student parameters need to be cloned through a function, i.e. Ensure that the starting parameters for both paths are consistent.

[0052] Step 5: During testing, the input data is processed through the student path to generate fused features, and sentiment prediction is achieved through a fully connected layer.

[0053] During testing, the input test data (preprocessed to a unified dimension in step one) is forward-propagated only through the student path. If the test data contains missing modalities, the input data consists of 7 combinations of missing modalities. The mask for the missing modal data is set to 0, and the corresponding modal mask is automatically generated (e.g., when only text is input, the audio and visual modal masks are set to 0). After dynamic routing fusion, the student path fusion features are obtained and input into the fully connected layer to output the sentiment prediction probability, as shown in the following formula: in This represents the multimodal fusion features obtained by the student model after dynamic routing fusion during the testing phase.

[0054] This invention was validated on the IEMOCAP multimodal open-source dataset, and the experimental results are shown in Table 1 below:

[0055] Table 1

[0056] Test conditions / methods IF-MMI(WA% / UA%) MRAN(WA% / UA%) This invention (WA% / UA%) {a} 55.03 / 53.20 55.44 / 57.01 67.35 / 68.38 {t} 67.02 / 68.20 65.31 / 66.42 75.36 / 75.65 {v} 51.97 / 50.41 53.23 / 49.80 55.92 / 51.17 {a,t} 74.05 / 75.44 73.00 / 74.58 77.46 / 78.33 {a,v} 65.33 / 66.52 64.70 / 64.46 66.87 / 66.72 {v,t} 72.68 / 73.62 72.11 / 72.24 74.81 / 74.14 {a,t,v} - - 76.60 / 76.99

[0057] Table 1 shows that the Invariant Feature Aware Multimodal Emotion Recognition Model (IF-MMIN) uses invariant features to compensate for missing modalities, imagining the missing modalities using available modalities to address multimodal emotion recognition tasks in the case of missing modalities. The Multimodal Relation Aware Attention Network (MRAN) method, on the other hand, reconstructs the missing modal features by introducing multimodal prior knowledge and missing labels, and forces the alignment of all modal features with the text semantic space to address multimodal emotion recognition tasks in the case of missing modalities.

[0058] The data in the table shows that the EMA knowledge distillation multimodal emotion recognition method proposed in this invention generally improves both weighted and unweighted accuracy compared to current methods. Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this invention and are not intended to limit it. Although the invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this invention.

Claims

1. A multi-modal sentiment recognition method based on dynamic EMA knowledge distillation, characterized in that, The specific steps include: S1: Collecting audio, video, and text modal information, and obtaining multi-modal features through preprocessing; S2: Inputting the preprocessed multi-modal features into the parallel teacher path and student path respectively to simulate the modal missing condition and obtain the teacher path features and student path features respectively; S3: Dynamically routing and fusing the teacher path features and student path features to obtain fused teacher features and fused student features; S4: Self-adaptive knowledge distillation of the fused teacher features and fused student features, and optimization of the student path parameters and teacher path parameters; S5: In the test stage, input data is processed through the student path to obtain fused features, and emotion prediction is realized through a fully connected layer.

2. The multi-modal sentiment recognition method based on dynamic EMA knowledge distillation according to claim 1, characterized in that, In step S1, the audio information is recorded by a microphone; the video information is recorded by a camera; and the text information is obtained by converting the speech content.

3. The multi-modal sentiment recognition method based on dynamic EMA knowledge distillation according to claim 2, characterized in that, The preprocessing is as follows: For the audio modality, the original audio signal is input, the frame-level features are extracted through the multi-layer convolutional neural network of Wav2Vec, the frame-level features of the whole speech are aggregated, and the audio feature vector is obtained; For the text modality, the original text sequence is input, the text is encoded through the Transformer layer of DeBERTa, and the text feature vector is extracted; For the visual modality, first, the multi-task convolutional neural network MTCNN is used to detect and align the face region from the video frame, the aligned face image sequence is input, then the multi-scale attention network MA-Net is used to extract the visual features of each frame, the frame-level features are averaged and pooled, and the visual feature vector is obtained; Then, the audio features, video features, and text features are mapped to the same dimension through the projection layer.

4. The multi-modal sentiment recognition method based on dynamic EMA knowledge distillation according to claim 3, characterized in that, In step S2, the teacher path is trained using full-modal features, and the student path applies dynamic masks to the input features to simulate modal missing conditions, which is implemented as follows: The teacher path feature input is trained by using full modal features without masking, and the teacher input feature is obtained by generating a modal mask with all 1 values wherein, denotes element-wise multiplication, Concat(·) denotes feature concatenation, h a ,h t ,h v denote the audio, text, and visual modal features after preprocessing and projection, respectively denote the corresponding modal masks under the teacher path. The student path feature mask generation part is used to simulate 7 kinds of modal missing combinations in a real scene, that is, the student path randomly selects modal features from three kinds of modal features for input, and generates a dynamic mask through a dynamic mask generation function. The function randomly generates a mask matrix that meets the preset 7 combinations for each training batch, selectively occludes each modal feature of the student path, and obtains the student input feature wherein denotes element-wise multiplication, Concat(·) denotes feature concatenation, h a ,h t ,h v denotes the modal feature after preprocessing and projection, respectively denote the corresponding modal mask under the student path.

5. The multi-modal sentiment recognition method based on dynamic EMA knowledge distillation according to claim 4, characterized in that, The step S3 is specifically implemented as: the features of the teacher path are fused by different expert encoders, and feature fusion is performed according to dynamic routing weights, and finally weighted fused teacher features Z are obtained t The features of the student path are also fused by different expert encoders according to the dynamic routing weights, and finally weighted fused student features Z are obtained s .

6. The multi-modal sentiment recognition method based on dynamic EMA knowledge distillation according to claim 5, characterized in that, The step S4 is implemented as follows: The self-adaptive knowledge distillation of the fused teacher features and fused student features depends on the importance of the available modalities in the current batch, and the student path parameters are optimized through gradient descent, while the teacher path parameters are updated using the exponential moving average EMA; For the calculation of distillation loss, the teacher path performs forward propagation through the loss function, blocking gradient backpropagation in the process, and obtains the feature Ensure that the teacher model parameters are not updated, and the student path normally calculates the gradient to approximate the teacher path by learning to fuse features; when the student path is missing modalities, extract the routing weights saved in the latest weight saving area, calculate the weight sum S of the available modalities, traverse the three modalities, and only count the routing weight mean of the non-missing modalities, as follows: Where represents the global mean of the routing weight of modality m, is the weight corresponding to the modality feature, i is the index of the batch dimension, j is the index of the sequence dimension, k is the index of the modality fusion dimension, miss m is the modality validity mask, and the dynamic coefficient distillation loss L distill The mean square error loss is used to measure the difference between the student and the teacher fused features; task loss L task The total loss is the weighted sum of the two according to the type of the data set: L total = L task + λ kd · L distill , where λ kd is the distillation loss weight.

7. The multi-modal sentiment recognition method based on dynamic EMA knowledge distillation according to claim 6, characterized in that, The step S5 is implemented as follows: In the test stage, the data flow is processed through the student path to obtain the mixed student features and perform the final emotion prediction through a fully connected layer.

Images