A multi-modal sentiment recognition method and system fusing bidirectional attention and self-distillation

Abstract

The application relates to a multi-modal emotion recognition method and system fusing bidirectional attention and self-distillation, first, a bidirectional cross attention mechanism is designed to realize bidirectional interaction of information flow between modes, so that text, speech and video can be mutually corrected to mine deeper complementary clues. Secondly, a self-distillation type double-strategy learning framework is introduced to realize exploration of a global optimal solution through parallel conservative fusion and aggressive fusion paths, and the adaptability and generalization performance of the model in low-labeled data, noisy labeling and cross-domain migration scenarios are enhanced. Finally, a mode contribution balance mechanism is constructed to dynamically estimate the reliability and importance of different modes in different samples, suppress the interference of low-quality modes, and enhance the contribution of high-quality modes, thereby significantly improving the robustness and stability of the model in complex real scenarios. The application improves the fusion efficiency of text, speech and video modal information and the fine-grained emotion recognition precision.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and affective computing, specifically to a multimodal affective recognition method and system that integrates bidirectional attention and self-distillation. Background Technology

[0002] Multimodal emotion recognition is a cutting-edge research direction in the field of artificial intelligence. By collecting multimodal data from individuals in natural interaction scenarios—including text, speech, video, and various sensor data—it achieves a comprehensive perception and accurate characterization of emotional states. This method, through deep fusion of information from different modalities, can significantly improve the completeness and accuracy of emotion analysis, thus providing more reliable technical support for emotion computing. In the field of human-computer interaction, multimodal emotion recognition can be used to enhance the emotion understanding capabilities of intelligent systems, enabling them to have more natural interactive feedback. In the field of education, it can help monitor students' classroom participation and emotional fluctuations, providing a basis for personalized teaching. In the field of healthcare, it can assist in the assessment of emotional states such as depression and anxiety, and provide auxiliary diagnosis for early mental health issues. In public service scenarios, it can be used for monitoring group emotions, providing data support for emergency management and public opinion risk intervention.

[0003] In recent years, with the development of advanced technologies such as deep learning, cross-modal representation learning, and large-scale models, the accuracy and robustness of multimodal emotion recognition have been continuously improved. It is expected to play a key role in a wider range of intelligent scenarios in the future, promoting the popularization and implementation of emotional intelligence technology. Currently, the mainstream method in the field of multimodal emotion recognition is based on deep learning technology using multimodal fusion. Its core idea is to obtain a more comprehensive and finer-grained emotion representation than single-modal information by fusing information from multiple data sources such as text, speech, and video. These methods typically divide the emotion recognition process into three key stages: feature encoding, modality fusion, and emotion prediction. In the feature encoding stage, the model constructs specialized encoders for different modalities. For example, it uses the BERT (Bidirectional Encoder Representations from Transformers) model to extract deep semantic embeddings from text, uses Mel-Frequency Cepstral Coefficients (MFCC) or self-supervised learning-based audio models to extract acoustic features from audio data, and uses Convolutional Neural Networks (CNN), Recurrent Neural Networks (RNN), or Transformers to extract dynamic visual features from videos. This approach encodes multi-source data separately. In the fusion stage, multimodal emotion recognition methods employ techniques such as tensor fusion, low-rank fusion, cross-modal attention mechanisms, or Transformer fusion architectures to deeply mine intermodal correlations and complementary information, and effectively suppress noise unrelated to emotion that may be carried by certain modalities. Finally, in the emotion prediction stage, the fused emotion representation is input into a classifier or regressor to achieve high-precision and robust emotion recognition. Existing multimodal fusion emotion recognition methods still have certain shortcomings: First, the models lack an adaptive modeling mechanism for the quality differences of multimodal data, making it difficult to achieve stable and robust fusion when there is significant data noise or conflicts between modalities; second, intermodal interactions are still mainly based on unidirectional or shallow associations, making it difficult to fully capture the deep complementary relationships and bidirectional semantic dependencies between modalities; third, most models rely on a large amount of supervised data for training, lacking effective cross-modal consistency constraints and structured self-supervised learning mechanisms, resulting in limited generalization ability in complex real-world scenarios. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention proposes a multimodal emotion recognition method that integrates bidirectional attention and self-distillation.

[0005] This invention integrates a bidirectional cross-attention mechanism, self-distillation technology, and a modal contribution balancing mechanism to overcome the limitations of current methods in terms of robustness, interaction depth, and generalization ability. First, by designing a bidirectional cross-attention mechanism, bidirectional interaction of information flow between modalities is achieved, enabling text, speech, and video to mutually correct each other and uncover deeper complementary cues, thereby mitigating the misjudgment of complex emotional expressions by unidirectional guidance models. Second, a self-distillation-style dual-strategy learning framework is introduced, exploring the global optimum through parallel conservative and aggressive fusion paths, enhancing the model's adaptability and generalization performance in low-label data, noisy labeling, and cross-domain transfer scenarios. Finally, a modal contribution balancing mechanism is constructed to dynamically estimate the reliability and importance of different modalities in different samples, suppressing interference from low-quality modalities while enhancing the contribution of high-quality modalities, significantly improving the model's robustness and stability in complex real-world scenarios. These three core strategies complement each other, jointly constructing a more efficient, reliable, and credible multimodal emotion recognition framework, which has significant value for the research and application of multimodal emotion recognition tasks.

[0006] This invention relates to an improved model and training method that integrates a bidirectional cross-attention mechanism, a self-distillation mechanism, and a modal contribution balancing mechanism, aiming to improve the fusion efficiency of text, speech, and video modal information and the accuracy of fine-grained emotion recognition.

[0007] The technical solution of the present invention is as follows: A multimodal emotion recognition method integrating bidirectional attention and self-distillation includes: Step 1: Multimodal data acquisition and preprocessing; Step 2: Construction of trimodal masks and generation of missing modality samples; including: constructing input masks and missing masks for the three modalities; and using the missing masks to modify the original trimodal features to form the missing version of the input used by the student path; Step 3: Trimodal Unified Embedding and Non-linguistic Modality Compression Alignment; including: mapping the three modalities to a unified embedding dimension; modeling the temporal dependencies within each modality; and performing standard multi-head attention operations and residual connections on the multimodal spliced ​​sequence to capture high-order correlations within and between sequences. Step 4: Single-modal temporal coding and bidirectional cross-attention multimodal fusion; including: performing temporal coding on each modality separately to model the temporal dependencies within the modality; using a bidirectional cross-attention multimodal fusion mechanism to explicitly model the bidirectional interaction between the three modalities on a unified time axis, and producing fused text, audio, and video representations respectively; Step 5: Adaptive hypermodal learning; including: intramodal temporal convergence to obtain global representations of the three modalities; concatenating the global representations of the three modalities on the feature dimension to construct a set of multimodal joint feature vectors, and introducing residual connections to obtain the final joint sentiment representation; performing modal contribution balancing; Step 6: Constructing the loss for teacher and student paths; Step 7: Dual-strategy joint optimization and training process; Step 8: Inference stage, realize multimodal emotion recognition.

[0008] According to a preferred embodiment of the present invention, the specific implementation process of step 1 includes: The video stream input data, i.e., the video modality, is feature-encoded using OpenFace to obtain a video feature sequence; the text data, i.e., the text modality, is feature-encoded using BERT to obtain a text feature sequence; and the input audio stream, i.e., the audio modality, is feature-encoded using Librosa to obtain an audio feature sequence. Representing video feature sequences, text feature sequences, or audio feature sequences as... ,in , It is the sequence length. is the vector dimension of each mode; where, , , These represent text modality, audio modality, and visual modality, respectively. Set emotion tags ,in C The number of categories; during training, mini-batch input is used, with a batch size of [value missing]. B The sequence lengths of each modality within the same batch are aligned to a uniform length by padding. , ,get: and the corresponding label vector: ; The complete input obtained through the above operations refers to the complete modal feature sequence. , as input for the teacher's path.

[0009] According to a preferred embodiment of the present invention, step 2 includes the following specific implementation process: Step 2.1: Construct input masks for three modes: , where: for the text modality, when the first b The first sample t When each token is a real token ,otherwise, For audio and video modalities, when the first b The first samplet When the frame is a real frame, the corresponding mask is 1; otherwise, it is 0. For each modality, based on the preset missing rate p and missing patterns, in The missing mask is obtained by upsampling at the location: 0 indicates that the position is considered missing under the current configuration, and 1 indicates that the position is considered reserved under the current configuration; Step 2.2: Modify the original trimodal features using a missing mask to form a missing version of the input used by the student path; For positions in the text modality where a word was present but was marked as 0 by a missing mask, the original token is replaced with a [UNK] vector: ; in, It's a missing mask. It is the original input mask. It is a vector of [UNK]. It is a complete text modal feature sequence, used as the teacher's path input; It is a sequence of missing text modal features, used as the student path input; For audio and video modalities, when the missing mask is 0, the entire frame feature is directly multiplied by 0: ; in, and These represent the missing masks for the audio and video modalities, respectively. and These represent the complete feature sequences for the audio and video modalities, respectively. and These represent the missing feature sequences for the audio and video modalities, respectively. The missing version input obtained through the above operations is used as the input for the student path. The missing version input refers to the modal feature sequence after missing masking. .

[0010] According to a preferred embodiment of the present invention, step 3 specifically includes the following steps: Step 3.1: Perform linear mapping on the three modes respectively, transforming the three modes from their original dimensions. Mapping to a unified embedding dimension d ; denote the mode m The mapping function is Then, the teacher path and the student path are represented in the unified embedding space as follows: ; ; in, Representing text, audio, and video modalities; For complete modal feature sequences, This is a sequence of missing modal features; For modality A linear mapping function; , These are learnable parameters; The teacher path and student path are represented by their respective paths mapped to a unified embedding space, with the teacher path and student path sharing the same set of embeddings. To ensure proper distillation alignment; Step 3.2: Employing a multi-head self-attention and feedforward network within the Transformer architecture, the temporal dependencies within each modality are modeled, and standard multi-head attention operations and residual connections are performed on the multimodal spliced ​​sequence to capture higher-order correlations within and between sequences; including: A cross-modal compression and alignment mechanism based on the text modality is employed. Using text sequences as queries and audio / video sequences as keys and values, multi-head attention operations of the Transformer are used to calculate the attention weights of the text timesteps on the audio and video timesteps. Based on this, audio and video features are weighted, thereby compressing and aligning the originally disparate audio and video sequences to the same time length as the text sequence. Non-verbal modalities are also compressed to the same length as the text sequence. Consistent expression.

[0011] Further preferably, in step 3.2, for the audio modality, the text is uniformly embedded as... Audio is uniformly embedded as Based on text input masks and audio input masks, a cross-modal compression mapping function is defined. The audio features are weighted and aggregated along the text time steps to obtain a compressed audio representation with the same length as the text. : ; in, Enter a mask for the text. Use an audio input mask; function Internally, it employs a multi-head attention structure based on text as the query and audio as the key and value, combined with masking to suppress attention weights at padding positions, thereby weighted aggregation of related audio time steps at each text time step; specifically including: Embedded with text As a query, with audio embedded As key and value pairs, multi-head attention is used to calculate the correlation score between text time steps and audio time steps. and utilize Assign values ​​to the audio padding positions in the fraction matrix To suppress its attention weights, the weights are then obtained by softmax normalization over the audio time dimension for each text time step. The audio value vectors are weighted and aggregated, and finally, a compressed audio representation with the same length as the text is obtained through multi-head concatenation and linear mapping. This allows for the alignment and aggregation of relevant audio information at each text time step.

[0012] According to a preferred embodiment of the present invention, step 4 includes the following specific implementation process: Step 4.1: Multimodal sequence concatenation and unified encoding input; By concatenating the temporal representations of the three modes in modal order on the timeline, a synthesized sequence is obtained: ; in For text embedding, This is a compressed and aligned representation of audio and video. This is the total length after splicing; A bidirectional cross-attention multimodal fusion mechanism is used to explicitly model the bidirectional interaction relationship between the three modalities on a unified time axis, and to produce fused text, audio and video representations respectively. Step 4.2: Construct various local-to-local attention branches; In the fusion layer, the following types of attention branches are constructed simultaneously within the same attention module: Trimodal global attention branch Model the global interactions between text, audio, and video across the entire concatenated sequence; Text-audio bidirectional cross-attention branch Text-video bidirectional cross-attention branch Audio-video bidirectional cross-attention branch Modeling mutual interest only between the two modes; Text self-attention branch Audio self-attention branch Video self-attention branch Modeling is only performed within the single-modal interval.

[0013] Step 4.3: Aggregate the outputs of different attention branches by modality After calculating several types of attention branches, the output of each modality is aggregated separately with its associated branch output; These outputs are concatenated along the feature dimension, denoted as: ; in, This indicates that the output subsequence at the location of the text is extracted from the corresponding attention branch. This is the temporary dimension after splicing; This refers to the joint intermediate representation of text modalities after multi-branch attention modeling; Subsequently, through linear transformation Compress back to a unified dimension The resulting fused text representation is: ; in, , These are the parameters for the text modality fusion layer; Perform the above operations on the audio and video modalities to obtain the fused audio representation. and the merged video representation .

[0014] According to a preferred embodiment of the present invention, step 5 includes the following specific implementation process: Step 5.1: Intramodal temporal convergence to obtain global representations of the three modes; First, attention convergence is performed separately in the temporal dimension for each modality, compressing the temporal features into a fixed-dimensional modality-level representation vector; for any modality m Its temporal characteristics Recorded as: ; in, Indicates the first t The hidden state at each time step; Then, an attention score is generated for each time step through a scoring network. : ; in, 、 、 For modality Attention parameters; Next, the attention score is normalized using softmax to obtain the weights in the time dimension. : ; Based on this, the modality m The modalities are obtained by weighted summation of all time steps. m global representation vector : ; Step 5.2: Concatenate the global representations of the three modalities along the feature dimension to construct a set of multimodal joint feature vectors. A feedforward network is applied to the multimodal joint feature vector. The feedforward network is a two-layer fully connected feedforward structure used for nonlinear transformation and enhanced representation of the multimodal joint feature vector; specifically, it includes: a first-layer linear transformation through a weight matrix... and bias terms Input joint feature vector Mapped to an intermediate feature space and activated by the ReLU nonlinear activation function. Introducing nonlinear expressive power; the second-level linear transformation uses a weight matrix. and bias terms intermediate features Map back to the same feature dimensions as the input, and connect the original input via residual connections. Adding the transformed features to obtain the final joint sentiment representation. ; By introducing residual connections, the final joint sentiment representation is obtained: ; ;in, For feedforward network parameters, It is a ReLU nonlinear activation function. The joint sentiment representation output by the AHL module; Both the teacher's path and the student's path obtain their joint representations through adaptive hypermodal learning. and Parameter sharing; Step 5.3: Perform modal contribution balancing; including: First, the training data is divided into samples based on sentiment categories. The training data includes trimodal inputs and their labels. For each sentiment category, the text, audio, and video modal features of the samples under that sentiment category are pooled separately to obtain the global modal representations. , and ; The average is calculated to obtain the class prototype vectors for the three modalities: Design a prototype encoder for each mode, and... Mapping to The fused feature space; let these encoding functions be: ; in, and It has matching dimensions or concatenable dimensions, and It includes several layers of fully connected networks and ReLU nonlinear activation; For each emotion category C Constructing category-conditional extended feature vectors : ; in, This represents the multimodal joint sentiment representation of the current sample. Indicate category C Typical patterns across the three modalities; For each category C Set a set of independent linear transformation and bias parameters The class conditional prediction distribution is obtained through softmax: ;in, For category C The corresponding weight matrix, For bias vectors, Indicates the category prototype C Under the given conditions, the predicted probability distribution for all category labels.

[0015] Further optimization involves introducing a category prior probability vector: ; in, For category C The frequency of occurrence in the training set, or the prior probability after smoothing / reweighting; During inference and training, all category conditional predictions are weighted according to priors to obtain the final sentiment prediction distribution: ; in, This represents the final predicted probability for each sentiment category under the current sample input conditions; The final prediction category is: ,in, To predict category labels, To predict the final distribution in categories The probability of it.

[0016] According to a preferred embodiment of the present invention, step 6 includes the following specific implementation process: Step 6.1: Let the true label vector of the current batch of samples be: ; The teacher path task loss uses cross-entropy: ; in, Batch size; Number of categories; For the first The true labels of each sample; For indicator functions; For teacher paths to categories The predicted probability; Step 6.2: In the student path, the input is the missing version of the three-modal features constructed in Step 2, and the output is the predicted probability distribution of the student path. With joint emotional expression ; Step 6.2.1: Student path task loss; Output under missing modal input Calculate the cross-entropy loss: ; in Batch size; Number of categories; For the first The true labels of each sample; For indicator functions; For student paths to categories The predicted probability; Step 6.2.2: Combined representation of distillation loss; We introduce a joint representation distillation loss based on cosine similarity, where the joint representation of teacher paths is... Student paths are jointly represented as Then for the first i Each sample is defined as follows: ; in, For Teacher Pathway i Joint representation of samples; This represents the joint representation of the student's path; This means that the gradient is not propagated back to the teacher path when calculating the gradient; it is only used as the distillation target. The distillation loss is based on cosine similarity for joint representation. The average of the entire batch is obtained. : ; Step 6.2.3: Distillation loss of attention distribution; Introducing attention distribution distillation loss, let the multimodal attention weights of the last layer of the teacher path be... A The student path corresponds to a weight of After performing softmax normalization on the bond dimension, and using KL divergence to measure the difference between the two, the attention distillation loss is defined as: ; The softmax mechanism is consistent with the masking mechanism, normalizing only at valid locations. The KL divergence is the summation of elements. This refers to attention distillation loss; Step 6.2.4: Multimodal reconstruction loss; A reconstruction branch is introduced to force the network to understand and fill in the missing information by reconstructing the trimodal features from the fused representation; For the text modality, let the reconstruction target be the encoded text features in the teacher path. The student reconstructed the output as follows Construct an effective reconstruction mask Then the text reconstruction loss for: ; in, For element-wise multiplication, To prevent small constants with a denominator of zero; For the audio modality and the video modality, the audio reconstruction loss is obtained separately. and video reconstruction loss Total reconstruction loss for: ; In summary, the total loss is constructed along the student's path. : in, These are non-negative weighted hyperparameters used to balance the contributions of various distillation and reconstruction losses to the overall training.

[0017] According to a preferred embodiment of the present invention, step 7 is specifically implemented as follows: Step 7.1: Teacher's forward and reverse paths; Use full modal input Perform forward computation according to steps 1 to 5 and step 6 to obtain... And calculate the teachers' losses Then the teacher suffered losses Perform backpropagation to update network parameters; , , These refer to complete modal sequence inputs of text, audio, and video, respectively. Step 7.2: Student path forward and backward; Using missing modal input Execute forward on the same network to obtain , ; Combining teacher pathways and Calculate the total student path loss according to step 6. ,right Perform backpropagation and update the same set of parameters again; Step 7.3: Overall Optimization Goal; The overall optimization objective of the entire training process is to minimize the weighted sum of the losses from the teacher's path and the student's path. ; in For all learnable parameters; This is a balancing factor used to balance the performance of complete modes with the robustness of missing modes.

[0018] According to a preferred embodiment of the present invention, step 8 includes the following specific implementation process: First, feature sequences of text, audio, and video are obtained from the samples to be predicted. Construct the corresponding input mask ; Next, steps 3 to 4 are executed sequentially to obtain the three-modal fusion temporal features. Perform adaptive hypermodal learning to construct a joint sentiment representation. By combining pre-statistical category prototypes and category priors, a modal contribution balancing mechanism is executed to obtain the final predicted distribution. ; Finally, according to The index of the category with the highest probability is used to output the final sentiment recognition result. .

[0019] A computer device includes a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps of the above-described multimodal emotion recognition method that integrates bidirectional attention and self-distillation.

[0020] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described multimodal emotion recognition method that integrates bidirectional attention and self-distillation.

[0021] Secondly, based on the above method, the multimodal emotion recognition system based on bidirectional attention and self-distillation provided by the present invention includes: The data acquisition and preprocessing module is configured as: multimodal data acquisition and preprocessing; The trimodal mask construction and missing modality sample generation module is configured to include: constructing input masks and missing masks for three modalities; and using the missing mask to modify the original trimodal features to form a missing version of the input used by the student path; The trimodal unified embedding and non-linguistic modality compression alignment module is configured to: map the three modalities to a unified embedding dimension; model the temporal dependencies within each modality; and perform standard multi-head attention operations and residual connections on the multimodal spliced ​​sequence to capture high-order correlations within and between sequences. The single-modal temporal coding and bidirectional cross-attention multimodal fusion module is configured to: perform temporal coding on each modality separately to model the temporal dependencies within the modality; and use a bidirectional cross-attention multimodal fusion mechanism to explicitly model the bidirectional interaction between the three modalities on a unified time axis, and produce fused text, audio, and video representations respectively. The adaptive supermodal learning module is configured to: converge intramodal temporal data to obtain global representations of the three modalities; concatenate the global representations of the three modalities on the feature dimension to construct a set of multimodal joint feature vectors, and introduce residual connections to obtain the final joint sentiment representation; and perform modal contribution balancing. The loss construction module is configured to construct losses for both teacher and student paths. The dual-policy joint optimization and training module is configured to: implement dual-policy joint optimization and training; The inference module is configured to implement multimodal emotion recognition.

[0022] The beneficial effects of this invention are as follows: 1. The bidirectional cross-attention mechanism proposed in this invention, based on unified temporal alignment, simultaneously constructs multiple attention paths within text-audio, text-video, audio-video, and each modality, achieving fine-grained, comprehensive cross-modal bidirectional interaction. Compared with traditional methods that rely on simple concatenation or unidirectional alignment, this mechanism can more deeply integrate complementary and corrective information between speech intonation, facial expressions, body movements, and text semantics, thereby significantly enhancing the discriminative power and expressive richness of multimodal emotion representation, ultimately improving overall recognition accuracy and robustness.

[0023] 2. This invention employs a self-distillation dual-strategy learning architecture, setting up two paths in parallel within the same network: a teacher path based on complete modal input, and multiple student paths simulating different modal missing configurations. Through multi-level constraints such as joint representation distillation, attention distillation, and multimodal reconstruction, the system guides the student paths to gradually learn and approximate the robust discriminative ability of the teacher path even under incomplete modal conditions. Compared to traditional models trained only on complete modalities, this method effectively addresses common data missing problems such as camera occlusion, audio packet loss, and text fragmentation in practical deployments, thereby significantly improving the system's recognition stability, robustness, and generalization performance in real-world complex scenarios.

[0024] 3. This invention proposes a modality contribution balancing mechanism. This mechanism explicitly introduces category prototypes and category prior probabilities into the output layer, and leverages category conditional features and category-specific classification heads to achieve collaborative modeling of joint sample representation, typical category patterns, and trimodal contributions. Unlike traditional unified classifiers that treat all emotion categories the same way, this invention adaptively adjusts the relative weights of text, audio, and video modalities in each category decision based on the differences in feature representation among different emotion categories. Therefore, in diverse contexts such as categories relying on text semantics, categories emphasizing speech intonation, or categories focusing on facial expressions, this invention can learn decision boundaries that better fit the data characteristics, thereby significantly improving the accuracy and robustness of fine-grained emotion classification.

[0025] 4. This invention extends the classic Transformer structure in the attention calculation part, achieving bidirectional cross-attention and modal feature compression alignment through mask design and branch splitting, without introducing too many additional complex operators. Simultaneously, the self-distillation and modal contribution balancing mechanisms can share the same backbone network parameters. Therefore, while effectively controlling computational overhead and model size, this invention significantly outperforms traditional multimodal emotion recognition methods that only employ single-path training or simple fusion strategies in terms of recognition accuracy, robustness to missing modalities, and stability in class imbalance scenarios, demonstrating significant comprehensive performance advantages. Attached Figure Description

[0026] Figure 1 This is a framework diagram of the multimodal emotion recognition method that integrates bidirectional attention and self-distillation according to the present invention; Figure 2 This is a schematic diagram of the bidirectional cross-attention method of the present invention; Figure 3 This is a schematic diagram of the self-distillation dual-strategy learning process of the present invention; Figure 4 A schematic diagram of the AHL module; Figure 5 This is a schematic diagram of the modal contribution balancing mechanism of the present invention; Figure 6 This is a schematic diagram of attention calculation. Detailed Implementation

[0027] The present invention will be further defined below with reference to the accompanying drawings and embodiments, but is not limited thereto.

[0028] Example 1 A multimodal emotion recognition method integrating bidirectional attention and self-distillation is proposed. This method accurately captures deep correlations and complementary information between text, speech, and video by constructing a bidirectional information flow modal interaction structure. A parallel self-distillation strategy is introduced to explore and constrain the global optimum, thereby enhancing the model's generalization ability in scenarios with scarce labeled data and complexities. Furthermore, a modal contribution balancing mechanism is used to dynamically model and adaptively fuse the quality and credibility of input modalities, suppressing interference from noisy modalities and strengthening the representational ability of core modalities. Figure 1 As shown, it includes: Step 1: Multimodal data acquisition and preprocessing; Step 2: Construction of trimodal masks and generation of missing modality samples; including: constructing input masks and missing masks for the three modalities; and using the missing masks to modify the original trimodal features to form the missing version of the input used by the student path; Step 3: Trimodal Unified Embedding and Non-linguistic Modality Compression Alignment; including: mapping the three modalities to a unified embedding dimension; modeling the temporal dependencies within each modality; and performing standard multi-head attention operations and residual connections on the multimodal spliced ​​sequence to capture high-order correlations within and between sequences. Step 4: Single-modal temporal coding and bidirectional cross-attention multimodal fusion; including: performing temporal coding on each modality separately, denoted as... It is used to model the temporal dependencies within a modality; through a bidirectional cross-attention multimodal fusion mechanism, it is used to explicitly model the bidirectional interaction between the three modalities (text-audio-video) on a unified time axis, and to produce fused text, audio and video representations respectively. Step 5: Adaptive Hyper-modality Learning (AHL); including: intra-modal temporal convergence to obtain global representations of the three modalities; concatenating the global representations of the three modalities on the feature dimension to construct a set of multimodal joint feature vectors, and introducing residual connections to obtain the final joint sentiment representation; and performing modal contribution balancing. Step 6: Constructing the loss for teacher and student paths; Step 7: Dual-strategy joint optimization and training process; Step 8: Inference stage, realize multimodal emotion recognition.

[0029] Example 2 The difference between the multimodal emotion recognition method that integrates bidirectional attention and self-distillation as described in Example 1 is that: The specific implementation process of step 1 includes: The video stream input data, i.e., the video modality, is feature-encoded using OpenFace. That is, the input video stream is processed by OpenFace to extract features and obtain a video feature sequence. The text data, i.e., the text modality, is feature-encoded using BERT. That is, the text is input into the BERT encoder to obtain a text feature sequence. The input audio stream, i.e., the audio modality, is feature-encoded using Librosa. That is, the audio segments are processed by Librosa to extract audio features and obtain an audio feature sequence. Representing video feature sequences, text feature sequences, or audio feature sequences as... ,in , It is the sequence length. is the vector dimension of each mode; where, , , These represent the text modality (language), audio modality (audio), and visual modality (visual), respectively. Set emotion tags ,in C Number of categories (e.g., binary classification) C =2, Seven Categories C =7); During training, mini-batch input is used, with a batch size of . B The sequence lengths of each modality within the same batch are aligned to a uniform length using padding. , ,get: and the corresponding label vector: ; The complete input obtained through the above operations refers to the complete modal feature sequence. , as input for the teacher's path.

[0030] The specific implementation process of step 2 includes: Step 2.1: Construct input masks for three modes: , where: for the text modality, when the first b The first sample t When each token is a real token ,otherwise, For audio and video modalities, when the first b The first sample t When the frame is a real frame, the corresponding mask is 1; otherwise, it is 0. This represents the self-distilled student path, i.e., the aggressive path in the subsequent dual-policy learning architecture that constructs missing modality samples. For example... Figure 3 As shown.

[0031] For each modality, based on the preset missing ratep And missing patterns (such as random feature missing, integer mode missing, time period missing, etc.), in The missing mask is obtained by upsampling at the location: 0 indicates that the position is considered missing under the current configuration, and 1 indicates that the position is considered reserved under the current configuration. For text modalities, the input is usually stored in the form of a tokenid sequence, corresponding to sentences containing special markers [CLS] and [SEP] at both ends ([CLS] represents the initial position of a sentence in BERT, and [SEP] represents the end position), and their positions are denoted as indices 0 and 1 respectively. In subsequent arbitrary missing modes, the masks at these two positions are forcibly kept at 1 to ensure that sentence-level information is always present. In this case, the input mask indicates whether there was original data, and the missing mask indicates whether the current training scenario allows the data at that position to be seen.

[0032] Step 2.2: Modify the original trimodal features using a missing mask to form a missing version of the input used by the student path; For positions in the text modality where a word was present but was marked as 0 by a missing mask, the original token is replaced with a [UNK] vector: ; in, It's a missing mask. It is the original input mask. It is a vector of [UNK]. It is a complete text modal feature sequence, used as the teacher's path input; It is a sequence of missing text modal features, used as the student path input; For audio and video modalities, when the missing mask is 0, the entire frame feature is directly multiplied by 0: ; in, and These represent the missing masks for the audio and video modalities, respectively. and These represent the complete feature sequences for the audio and video modalities, respectively. and These represent the missing feature sequences for the audio and video modalities, respectively. The missing version input obtained through the above operations is used as the input for the student path. The missing version input refers to the modal feature sequence after missing masking. .

[0033] The specific implementation process of step 3 includes: Step 3.1: Perform linear mapping on the three modalities (text modality, audio modality, and video modality, corresponding to the modal feature sequences obtained in Steps 1 and 2), transforming the three modalities (including text modality feature sequences, audio modality feature sequences, and video modality feature sequences) from their respective original dimensions. Mapping to a unified embedding dimension d ; denote the mode m The mapping function is Then, the teacher path and the student path are represented in the unified embedding space as follows: ; ; in, Representing text, audio, and video modalities; For complete modal feature sequences, This is a sequence of missing modal features; For modality A linear mapping function; , These are learnable parameters; The teacher path and student path are represented by their respective paths mapped to a unified embedding space, with the teacher path and student path sharing the same set of embeddings. To ensure proper distillation alignment; Teacher paths and student paths share the same set of mapping functions. This ensures that the two paths are aligned and distilled within the same feature space.

[0034] Step 3.2: After completing the unified embedding, multi-head self-attention and feedforward networks in the Transformer architecture are used to model the temporal dependencies within each modality. Standard multi-head attention operations and residual connections are then performed on the multimodal spliced ​​sequence to capture higher-order correlations within and between sequences; for example... Figure 6 As shown, it includes: After unified embedding is completed, the sequence representations of text, audio, and video modalities are concatenated along the sequence dimension to form a multimodal composite sequence. In the encoding layer of the Transformer structure, firstly, for Perform multi-head self-attention: obtained from linear mapping The scaling point integral is calculated, and multiple masks are constructed based on the modal interval index in the same attention operation to form a global three-modal branch, pairwise modal interaction branches, and modal self-attention branches; each branch obtains its attention weights through softmax and is then compared with the inputs of the other branches. Weighted summation yields the branch outputs; subsequently, branch outputs related to a given mode are extracted according to the modal interval and concatenated along the feature dimension. Linear compression is then applied to obtain the fused trimodal sequence representation, which is then compared with the input. The method employs residual connections and layer normalization. Then, a position-wise feedforward network (two-layer linear transformation and nonlinear activation) is applied to the residual-normalized sequence representation, followed by further residual connections and layer normalization. This model establishes temporal dependencies within each modality in the unimodal self-attention branch, and cross-modal correlations in the global and pairwise interaction branches. Furthermore, it captures higher-order correlations within and between sequences through multi-layer stacking. Building upon traditional multi-head attention, this method further utilizes a cross-modal compression and alignment mechanism based on the text modality. It uses the text sequence as the query and the audio / video sequences as the key / value pairs. Through Transformer multi-head attention operations, it calculates the attention weights of the text time step on the audio and video time steps, and weights the audio and video features accordingly. This compresses and aligns the originally disparate audio and video sequences to the same time length as the text sequence; it also compresses the non-verbal modality to the same length as the text sequence. Consistent expression.

[0035] In step 3.2, for the audio modality, the text is uniformly embedded as... Audio is uniformly embedded as Based on text input masks and audio input masks, a cross-modal compression mapping function is defined. The audio features are weighted and aggregated along the text time steps to obtain a compressed audio representation with the same length as the text. : ; in, Enter a mask for the text. Use an audio input mask; function Internally, it employs a multi-head attention structure based on text as the query and audio as the key and value, combined with masking to suppress attention weights at padding positions, thereby weighted aggregation of related audio time steps at each text time step; specifically including: Embedded with text As a query, audio is embedded As key / value pairs, multi-head attention is used to calculate the correlation score between text time steps and audio time steps. and utilize Assign values ​​to the audio padding positions in the fraction matrix To suppress its attention weights, the weights are then obtained by softmax normalization over the audio time dimension for each text time step. The audio value vectors are weighted and aggregated, and finally, a compressed audio representation with the same length as the text is obtained through multi-head concatenation and linear mapping. This allows for the alignment and aggregation of relevant audio information at each text time step.

[0036] Through the above steps, the three modalities with different original lengths are unified to a time length based on text.

[0037] The specific implementation process of step 4 includes: Step 4.1: Multimodal sequence concatenation and unified encoding input; By concatenating the temporal representations of the three modes in modal order on the timeline, a synthesized sequence is obtained: ; in For text embedding, This is a compressed and aligned representation of audio and video. This is the total length after splicing; Simultaneously, the position ranges of text, audio, and video within the concatenated sequence are recorded for subsequent construction of attention masks for different modality combinations.

[0038] A bidirectional cross-attention multimodal fusion mechanism is used to explicitly model the bidirectional interaction relationship between the three modalities (text-audio-video) on a unified time axis, and to produce fused text, audio and video representations respectively. Step 4.2: Construct various local-to-local attention branches; Within the fusion layer, the following types of attention branches are constructed simultaneously within the same attention module (illustrated in simplified form): The fusion layer employs a bidirectional cross-attention structure of "same attention operator + multiple mask branches": in the concatenated sequence Unified calculation Then, by using different binary masks, submatrices of different modal intervals are selected in the same attention scoring matrix, thereby obtaining the three-modal global interaction branch, the self-attention branches of each modality, and the bidirectional cross branches of each pair of modalities in parallel within a module; then, the branch outputs related to the modality are extracted according to the modality and spliced ​​and linearly compressed to obtain the fused text, audio, and video representations respectively.

[0039] Trimodal global attention branch Model the global interactions between text, audio, and video across the entire concatenated sequence; Text-audio bidirectional cross-attention branch Text-video bidirectional cross-attention branch Audio-video bidirectional cross-attention branch Modeling mutual interest only between the two modes; Text self-attention branch Audio self-attention branch Video self-attention branch Modeling is only performed within the single-modal interval.

[0040] Trimodal global attention branch The global multi-head self-attention branch on trimodal concatenation sequences aims to simultaneously model the overall interaction relationships between text (L), audio (A), and video (V) within the same attention space. This branch does not distinguish between modal boundaries and can directly capture the intermodal relationships between text-audio, text-video, and audio-video.

[0041] Text-audio bidirectional cross-attention branch Modeling mutual interest relationships only between the text and audio modalities; the video modal is not involved in this branch at all. Within the same attention calculation framework, attention weights involving all video regions are masked using attention masks, retaining only the attention calculation between text and audio regions. This branch specifically enhances the modeling of alignment and complementary information between language and speech, avoiding noise interference from video modalities.

[0042] Text-video bidirectional cross-attention branch This feature retains only the mutual attention relationships between the text and video modalities. All audio-related attention connections are masked. It is used to model the association between semantic information and visual cues (such as facial expressions, actions, and scenes).

[0043] Audio-video bidirectional cross-attention branch This feature retains only the mutual attention relationships between audio and video modalities. It masks all text-related attention connections. It is used to model the synergistic relationship between acoustic features and visual behavior.

[0044] Text self-attention branch Audio self-attention branch Video self-attention branch Each single-modal branch performs self-attention only within the time interval of its corresponding modality, and cross-modal information inflow is not allowed. This ensures that each modality has sufficiently strong self-representation capabilities before fusion, providing a "stable single-modal basis" for subsequent fusion.

[0045] Text-audio bidirectional cross-attention ( For example, its core idea is: when text is Q and audio is K / V, capture the direction in which the text reads information from the audio; when audio is Q and text is K / V, capture the direction in which the audio reads information from the text.

[0046] In implementation, by applying different binary masks to the unified attention scoring matrix, text-to-audio, text-to-video, audio-to-video, and sub-matrices corresponding to each modality's self-attention can be selectively retained from the same Q / K / V set, enabling various local-to-local attention computations. Specifically, traditional Transformer multi-head attention is used for scoring, scaling, and softmax normalization.

[0047] Step 4.3: Aggregate the outputs of different attention branches by modality After completing the computation of several types of attention branches, the query based on a unified construction is performed within the same multi-head attention operator. ,key ,value And its attention scoring matrix, respectively, are subjected to different binary masks to select different submatrices, thus forming: (1) Trimodal global attention branch (text / audio / video full sequence participation); (2) Text → Audio cross-attention branch (text position as query, audio position as key); (3) Audio → Text Cross-Attention Branch; (4) Cross-attention branches between text → video and video → text; (5) Self-attention branches for each modality (submatrices within the same modality). Each branch is scored, scaled, and normalized using the standard Transformer attention process to obtain the branch output. The branch outputs associated with each modality are aggregated separately; taking the text modality as an example, the outputs from the text positions in the three-modality global branches, namely the outputs from the text-audio cross branch where the text is used as a query, the outputs from the text-video cross branch where the text is used as a query, and the outputs from the text self-attention branch are concatenated.

[0048] These outputs are concatenated along the feature dimension, denoted as: ; in, This indicates that the output subsequence at the location of the text is extracted from the corresponding attention branch. This is the temporary dimension after splicing; This refers to the joint intermediate representation of text modalities after multi-branch attention modeling; Subsequently, through linear transformation Compress back to a unified dimension The resulting fused text representation is: ; in, , These are the parameters of the text modality fusion layer; the text modality fusion layer is used to perform dimensionality compression and information reorganization on the concatenated features from multiple attention branches. Specifically, a linear transformation layer is used to transform the concatenated high-dimensional intermediate representation... From the perspective of dimensions Map back to a unified embedding dimension This ensures consistency with subsequent modules in terms of feature dimensions. The linear transformation operates independently at each text time step, weighting and combining features from different attention branches to achieve effective fusion of multi-branch information.

[0049] Perform the above operations on the audio and video modalities to obtain the fused audio representation. and the merged video representation At this point, all three modalities have simultaneously included bidirectional interaction information of their own modality and other modalities at each time step.

[0050] This bidirectional cross-attention fusion mechanism is one of the core innovations of this model, enabling the model to capture modal dependencies in multiple directions such as text-audio, audio-text, and text-video on a unified time axis, rather than simply unidirectional alignment or concatenation, thereby more precisely modeling complementary and redundant information in multimodal emotional expression.

[0051] Figure 2 This is a schematic diagram of a bidirectional cross-attention module.

[0052] The specific implementation process of step 5 includes: Step 5.1: Intramodal temporal convergence to obtain global representations of the three modes; First, attention convergence is performed separately in the temporal dimension for each modality, compressing the temporal features into a fixed-dimensional modality-level representation vector; for any modality m Its temporal characteristics Recorded as: ; in, Indicates the first t The hidden state at each time step; Then, an attention score is generated for each time step through a scoring network. : ; in, 、 、 For modality Attention parameters; Next, the attention score is normalized using softmax to obtain the weights in the time dimension. : ; Based on this, the modality m The modalities are obtained by weighted summation of all time steps. m global representation vector : ; Step 5.2: Concatenate the global representations of the three modalities along the feature dimension to construct a set of multimodal joint feature vectors. To enhance feature representation capabilities, this invention applies a feedforward network layer to the multimodal joint feature vector. The feedforward network is a two-layer fully connected feedforward structure used for nonlinear transformation and enhanced representation of the multimodal joint feature vector; specifically, it includes: a first-layer linear transformation through a weight matrix... and bias terms Input joint feature vector Mapped to an intermediate feature space and activated by the ReLU nonlinear activation function. Introducing nonlinear expressive power; the second-level linear transformation uses a weight matrix. and bias terms intermediate features Map back to the same feature dimensions as the input, and connect the original input via residual connections. Adding the transformed features to obtain the final joint sentiment representation. ; By introducing residual connections, the final joint sentiment representation is obtained: ; ;in, For feedforward network parameters, It is a ReLU nonlinear activation function. The joint sentiment representation output by the AHL module; such as Figure 4 As shown, the input to the AHL module is the fused three-modal temporal representation, and its output is the joint sentiment representation. The core function of AHL is to obtain more stable cross-modal joint representations through representation learning after multimodal fusion, enabling them to simultaneously adapt to the distillation constraints of both teacher and student paths. This serves as a fundamental feature for subsequent modal contribution balancing mechanisms and distillation loss.

[0053] Both the teacher path (complete modality) and the student path (missing modality) obtain their joint representations through adaptive hypermodal learning. and Parameters are shared; the input corresponding to the teacher's path is the complete input obtained in step 1. The input corresponding to the student path is the missing version of the input obtained in step 2. .

[0054] Step 5.3: Perform modal contribution balancing; such as Figure 5 As shown, it includes: First, the training data is divided into samples based on sentiment categories. The training data is a set of training samples used for model parameter learning, including trimodal inputs and their labels. For each sentiment category, the text, audio, and video modal features of the samples under that sentiment category are pooled in the time dimension to obtain the global modal representation. , and ;in, , and These represent the overall sentiment semantic features of the current sample in text, audio, and video modalities, respectively.

[0055] The average is calculated to obtain the class prototype vectors for the three modalities: These three prototype vectors respectively characterize the typical feature patterns of different modalities within this sentiment category. Due to the original modality prototype... With joint statement The dimensions and distributions differ, therefore a prototype encoder is designed for each modality. , and These are nonlinear mapping networks designed for text, audio, and video modalities, respectively. Their structures consist of several layers of fully connected networks and ReLU nonlinear activation functions. The prototype encoder's role is to map the category prototype vectors for each modality to the joint sentiment representation. A consistent or concatenable feature space eliminates the differences in dimensionality and distribution between the original modality prototype and the joint representation. Through this encoding process, the resulting encoded prototype vector... , and It can participate in subsequent category-conditional modeling and classification decisions together with the joint sentiment representation of the current sample. Mapping to The fused feature space; let these encoding functions be:

[0056] ; in, and It has matching dimensions or concatenable dimensions, and It includes several layers of fully connected networks and ReLU nonlinear activation; For any mode Its corresponding prototype encoding function A two-layer feedforward neural network structure is adopted, specifically as follows: ; in, Indicates sentiment category In modality The original category prototype vector; This is the weight matrix of the fully connected layer; It is the bias vector; This represents the ReLU nonlinear activation function.

[0057] The result obtained through this encoding process It is similar to joint sentiment representation in both feature dimensions and distribution space. This ensures a good match, facilitating subsequent splicing and condition modeling.

[0058] Based on this, the present invention provides each emotion category C Constructing category-conditional extended feature vectors : ; in, This represents the multimodal joint sentiment representation of the current sample. Indicate category C Typical patterns in three modalities; vector By concatenating the current state of a sample with the typical pattern of the category, the subsequent classification head can learn how the three modalities should be focused on in each category.

[0059] For each category C Set a set of independent linear transformation and bias parameters The class conditional prediction distribution is obtained through softmax: ;in, For category C The corresponding weight matrix, For bias vectors, Indicates the category prototype C Under the given conditions, the predicted probability distribution for all category labels.

[0060] Furthermore, considering the differences in the proportion of different emotion categories in the training set, this invention introduces a category prior probability vector: ; in, For category CThe frequency of occurrence in the training set, or the prior probability after smoothing / reweighting; During inference and training, all category conditional predictions are weighted according to priors to obtain the final sentiment prediction distribution: ; in, This represents the final predicted probability for each sentiment category given the current sample input conditions; through... Weighting can balance the influence of different categories and alleviate the bias caused by uneven category distribution.

[0061] The final prediction category is: ,in, To predict category labels, To predict the final distribution in categories The probability of it.

[0062] The joint sentiment representation output in step 5 (From AHL) and intermediate features required to enter the modal contribution balancing mechanism; final predicted probability distribution With predictive labels After the modal contribution balancing mechanism completes the prior weighting, it is obtained, and then in step 6, based on... Calculate the loss with the real label and apply distillation constraints.

[0063] The specific implementation process of step 6 includes: Step 6.1: Under the condition of complete modality input (teacher path), through the aforementioned steps (data preprocessing → mask and missing mode construction → trimodal unified embedding and compression alignment → bidirectional cross-attention fusion → AHL module and modality contribution balancing mechanism), the final predicted probability distribution of this batch of samples can be obtained: .in, Indicates the first i The sample was predicted as the first... k The probability of each sentiment category. C This represents the total number of categories.

[0064] Let the true label vector of the current batch of samples be: ; The teacher path task loss uses cross-entropy: ; in, Batch size; Number of categories; For the first The true labels of each sample; For indicator functions; For teacher paths to categories The predicted probability; This loss is used to constrain the model's sentiment classification performance in full modality scenarios and is the most basic supervision signal during training.

[0065] Step 6.2: In the student path, the input is the missing version of the three-modal features constructed in Step 2. The subsequent processing flow is exactly the same as that of the teacher path, and the parameters are shared. The output is the predicted probability distribution of the student path. With joint emotional expression The student path input is a missing version of the three-modal features. Its subsequent network is exactly the same as the teacher's path and shares parameters: it sequentially goes through step 3 (unified embedding and compression alignment), step 4 (bidirectional cross-attention fusion), and step 5 (AHL to obtain joint sentiment representation). (and enters the modal contribution balancing mechanism), ultimately outputting the predicted probability distribution. Correspondingly, the teacher path is based on... The joint emotional expression that helps distillation With the predicted probability distribution .

[0066] Step 6.2.1: Student path task loss; Output under missing modal input Calculate the cross-entropy loss: ; in Batch size; Number of categories; For the first The true labels of each sample; For indicator functions; For student paths to categories The predicted probability; This is used to ensure that the network still has basic classification capabilities in scenarios where modalities are missing.

[0067] Step 6.2.2: Combined representation of distillation loss; To ensure that student paths approximate the discrimination strategy under missing modalities as closely as possible to those under complete modalities, this invention introduces a joint representation distillation loss based on cosine similarity. Let the joint representation of teacher paths be... Student paths are jointly represented as Then for the first i Each sample is defined as follows: ; in, For Teacher Pathway i Joint representation of samples; This represents the joint representation of the student's path; This means that the gradient is not backpropagated for the teacher path when calculating the gradient, but is only used as the distillation target; the closer the cosine similarity is to 1, the smaller the loss.

[0068] The distillation loss is based on cosine similarity for joint representation. The average of the entire batch is obtained. : ; Step 6.2.3: Distillation loss of attention distribution; To further constrain the student path to mimic the teacher path in terms of intermodal attention patterns, this invention introduces attention distribution distillation loss, assuming the multimodal attention weights of the last layer of the teacher path are... A The student path corresponds to a weight of The teacher and student paths have the same network structure and share parameters. The only difference is whether the input is a complete modality sequence or a missing version of the modality sequence. The teacher and student paths use different inputs to perform the above steps. After performing softmax normalization on the key dimension, the difference between the two is measured by KL divergence. The attention distillation loss is then defined as:

[0069] ; The softmax mechanism is consistent with the masking mechanism, normalizing only at valid locations. To calculate the KL divergence by summing element by element, we can average it over all headers, all layers, and all query positions. This refers to attention distillation loss;

[0070] This loss prompts students to emulate the teacher's modal interaction patterns as closely as possible.

[0071] Step 6.2.4: Multimodal reconstruction loss; A reconstruction branch is introduced to force the network to understand and fill in the missing information by reconstructing the trimodal features from the fused representation; For the text modality, let the reconstruction target be the encoded text features in the teacher path. The student reconstructed the output as follows Construct an effective reconstruction mask (Only at positions where the actual token is not padding, is the value 1), then the text reconstruction loss is... for: ; in, For element-wise multiplication, To prevent small constants with a denominator of zero; For the audio modality and the video modality, the audio reconstruction loss is obtained separately. and video reconstruction loss Total reconstruction loss for: ; In summary, the total loss is constructed along the student's path. : in, These are non-negative weighted hyperparameters used to balance the contributions of various distillation and reconstruction losses to the overall training.

[0072] The specific implementation process of step 7 includes: Within each training batch, the teacher and student paths share the network structure and parameters, differing only in their inputs. A dual-strategy joint optimization approach is employed during training.

[0073] Step 7.1: Teacher's forward and reverse paths; Use full modal input Perform forward computation according to steps 1 to 5 and step 6 to obtain... And calculate the teachers' losses Then the teacher suffered losses Perform backpropagation to update network parameters; , , These refer to complete modal sequence inputs of text, audio, and video, respectively. Step 7.2: Student path forward and backward; Using missing modal input Execute forward on the same network to obtain , The forward pass is performed on the same network. Specifically, it means that: first, unified embedding and attention modeling are performed on the available modalities; then, the input positions corresponding to the missing modalities are placed using a masking mechanism; then, the forward pass is completed through the same multimodal fusion layer, class conditional modeling layer and classification head, and finally the corresponding intermediate representation and prediction results are obtained.

[0074] Combining teacher pathways and Calculate the total student path loss according to step 6. ,right Perform backpropagation and update the same set of parameters again; Step 7.3: Overall Optimization Goal; The overall optimization objective of the entire training process is to minimize the weighted sum of the losses from the teacher's path and the student's path. ; in For all learnable parameters; This is a balancing factor used to balance the performance of complete modes with the robustness of missing modes.

[0075] That is, while ensuring the performance of full modality classification, the self-distillation mechanism significantly improves the robustness and generalization ability of the model to different missing modes.

[0076] The specific implementation process of step 8 includes: This invention employs a simplified single-path forward flow, retaining only the complete network structure and parameters corresponding to the teacher's path, and eliminating missing simulation, self-distillation, and reconstruction branches. The specific flow is as follows: First, feature sequences of text, audio, and video are obtained from the samples to be predicted. Construct the corresponding input mask ; Next, steps 3 to 4 are executed sequentially, including: unified embedding mapping, non-linguistic modality compression alignment (with text as the temporal basis), unimodal temporal encoding, and bidirectional cross-attention mechanism (BCAM) to obtain trimodal fused temporal features. Perform adaptive hypermodal learning to construct a joint sentiment representation. By combining pre-statistical category prototypes and category priors, a modal contribution balancing mechanism is executed to obtain the final predicted distribution. ; Finally, according to The index of the category with the highest probability is used to output the final sentiment recognition result. .

[0077] In this stage, the model only needs to perform a forward propagation once to complete multimodal emotion recognition. The computational cost is on the same order of magnitude as that of a regular multimodal Transformer model. However, because it has undergone self-distillation constraints of complete modalities and multiple missing modalities during the training phase, it has higher robustness and stability when facing modal missingness or noise in real-world scenarios.

[0078] To verify the effectiveness of the proposed multimodal fusion emotion recognition method, this embodiment selects three of the most representative benchmark datasets in the current field for experimental evaluation: CMU-MOSI, CMU-MOSEI, and CH-SIMS. These three datasets are all derived from real video corpora, widely used in multimodal emotion recognition tasks, covering text, audio, and video modalities, and equipped with sophisticated emotion annotation systems, thus possessing high research value.

[0079] The CMU-MOSI dataset: CMU Multimodal Opinion-level Sentiment Intensity (MOSI) is a multimodal video sentiment recognition dataset built by Carnegie Mellon University. It contains 2199 English video clips with opinion expressions. Each sample is manually labeled to obtain a sentiment intensity score, ranging from -3 to 3, corresponding to sentiment levels from "strongly negative" to "strongly positive." This dataset primarily uses short text videos, exhibiting high semantic density and clear emotional expression, and is recognized as one of the benchmark datasets for multimodal sentiment analysis tasks. The official dataset is divided into a training set of 1284 (63%), a validation set of 229 (11%), and a test set of 686 (26%).

[0080] The CMU-MOSEI dataset: CMU Multimodal Opinion Sentiment and Emotion Intensity (MOSEI) is a large-scale extension of MOSI, containing 22,856 video clips recorded by nearly 1,000 speakers, covering various topic domains, dialects, and speech rate differences. In addition to sentiment intensity annotation, MOSEI provides multi-label annotations for six emotion categories (Happiness, Sadness, Anger, Fear, Disgust, and Surprise), which can be used for both sentiment intensity prediction and multi-label emotion recognition tasks. Compared to MOSI, MOSEI significantly improves speaker diversity, facial expression naturalness, and emotional complexity, and has become a widely adopted comprehensive multimodal emotion dataset in recent years. The official dataset is divided into a training set of 16,326 (71%), a validation set of 1,871 (8%), and a test set of 4,659 (21%).

[0081] The CH-SIMS dataset: SIMS (The Chinese Single / Multi-modal Sentiment Dataset) is the first large-scale Chinese multimodal emotion recognition dataset, containing 2281 video clips, all from natural-context dialogues in movies and variety shows. This dataset employs a continuous sentiment annotation method consistent with MOSI / MOSEI (values ​​range from -1 to 1), and annotates sentiment polarity and its intensity level, more closely resembling modal interactions and modal conflicts in real-world language contexts. Simultaneously, SIMS provides segment-level speaker annotation, contextual annotation, and situational text information, making it an important data source for studying modal heterogeneity and modal fusion mechanisms in Chinese scenarios. The official dataset is divided into a training set of 1368 (60%), a validation set of 456 (20%), and a test set of 457 (20%).

[0082] Evaluation metrics: To comprehensively evaluate the model's classification and regression capabilities in multimodal emotion recognition tasks, this study employs four evaluation metrics to examine model performance from multiple perspectives: binary classification accuracy (wherein, the MOSI and MOSEI datasets use a labeling method that includes neutral samples 0, and when classifying positive and negative samples, neutral samples are classified as negative samples, named has0_acc_2; the SIMS dataset focuses more on continuous emotion labeling and does not emphasize neutral samples, named mult_acc2), five-class classification accuracy (Acc-5), seven-class classification accuracy (Acc-7), and Pearson correlation coefficient (Corr).

[0083] Classification accuracy (Acc) measures the model's accuracy in classifying emotions using different sentiment granularity settings, and directly reflects the model's ability to identify emotional polarity. It is calculated using the following formula:

[0084] ; in, TP (True Positive) represents the number of samples that the model correctly predicted as positive. TN (TrueNegative) represents the number of samples that the model correctly predicted as the negative class. FP (False Positive) indicates that the model incorrectly predicts a negative sample as a positive sample. FN (False Negative) indicates a situation where the model incorrectly predicts a positive sample as a negative sample. This is especially relevant when used for five-class or seven-class classification tasks. TP, TN, FP, FN The results will be aggregated by category to evaluate the model's performance on multi-category emotion recognition tasks.

[0085] The correlation coefficient (Corr) is used to assess the linear correlation between model predictions and actual sentiment intensity, and is a core metric commonly used in sentiment intensity regression tasks. It is calculated using the following formula:

[0086] ; in, For the model to the first i The predicted sentiment value for each sample As a value to indicate true sentiment, and These are the mean values ​​of the predicted and actual values, respectively. A higher correlation coefficient indicates that the model's predictions are more consistent with the actual sentiment intensity trend, and the better the regression performance.

[0087] The experimental results are as follows: The experiments selected lf_dnn, ef_lstm, tfn, lmf, mfn, graph-mfn, mult, misa, and self_mm for comparative experiments on three commonly used datasets: MOSI, SIMS, and MOSEI. This invention achieves better evaluation results for the aforementioned methods across various metrics. The various benchmark methods are described below:

[0088] lf_dnn: Encodes single-modal data through a deep neural network, concatenates and fuses the data, performs classification evaluation, and outputs prediction results.

[0089] ef_lstm: Concatenates multimodal data at the feature level and then inputs it into a bidirectional long short-term memory network to capture long-term dependencies in the data.

[0090] tfn: By performing Cartesian product operations on modal embeddings through tensor fusion layers, it explicitly models the interactions between single-modal, dual-modal, and multimodal modes.

[0091] lfn: For tfn, multimodal decomposition is performed through low-rank tensor decomposition, which reduces the number of parameters and algorithm complexity, and improves the algorithm efficiency.

[0092] mfn: Proposes a Delta-Memory Attention Network (DMAN) that summarizes multi-view gated memory to identify cross-view interactions.

[0093] graph-mfn: Based on MFN, it introduces a Dynamic Fusion Graph (DFG) module, which learns information about each modality in unaligned sequences through graph neural networks.

[0094] mult: Utilizes a bidirectional cross-modal attention mechanism to focus on the interaction between multimodal data at different time steps.

[0095] Misa transforms each modality into two distinct subspaces. It learns commonalities and reduces modality gaps through modality-invariant subspaces, captures private features through modality-specific subspaces, and finally fuses these two types of features.

[0096] self_mm: A label generation module based on a self-supervised learning strategy was designed to generate corresponding modal labels for each modality, thereby obtaining independent single-modal supervision.

[0097] tetfn: Proposes a Transformer fusion network for text enhancement, which injects emotion-related information from the linguistic modality into non-linguistic modalities such as visual and audio through text-driven multi-head attention and cross-modal mapping.

[0098] CENET proposes a cross-modal augmentation network that uses visual and acoustic features as supplementary elements and text as the core. It explicitly integrates non-linguistic context into text representation through cross-modal attention, thereby enhancing the emotional representation capabilities of the linguistic modality.

[0099] almt: Design an Adaptive Language-Guided Supermodal Learning (AHL) module to learn supermodal representations that suppress irrelevant / conflicting information from audiovisual modalities under the guidance of multi-scale language features, thereby obtaining cleaner and complementary multimodal sentiment representations.

[0100] ours: This refers to the multimodal emotion recognition method proposed in this invention.

[0101] Table 1 shows the performance metrics of each benchmark method on the MOSI dataset; Table 1 Table 2 shows the performance metrics of each benchmark method on the MOSEI dataset; Table 2

[0102] Table 3 shows the performance metrics of each benchmark method on the SIMS dataset; Table 3

[0103] Finally, the ablation experiment results of the three strategies proposed in this invention on the MOSI validation set are shown in Table 4: Table 4

[0104] Here, "Baseline" refers to the ALMT method, "BCAM" (Bidirectional Cross-Attention Mechanism), "SDM" (Self-Distillation Mechanism), and "MCBM" (Modality Contribution Balancing Mechanism). Compared to the baseline, this invention achieves a 3.2-point improvement in accuracy for seven-class classification, a 2.3-point improvement in accuracy for five-class classification, and a slight improvement in accuracy for three-class classification. This demonstrates a significant improvement in performance for high-precision classification tasks.

[0105] Experiments demonstrate that this invention outperforms existing methods in fine-grained classification accuracy on mainstream datasets (SIMS, MOSI, MOSEI). For example, on the MOSI dataset, the seven-class classification accuracy reaches 49.85% (a 3.2% improvement over the baseline), and the five-class classification accuracy reaches 54.96% (a 2.34% improvement). This method exhibits high robustness and accuracy in multimodal sentiment analysis, especially in high-precision recognition tasks, with manageable computational costs, providing a reliable solution for the practical deployment of multimodal sentiment recognition.

[0106] Example 3 A computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program described in Embodiment 1 or 2 to implement the steps of the above-described multimodal emotion recognition method that integrates bidirectional attention and self-distillation.

[0107] Example 4 A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the multimodal emotion recognition method that integrates bidirectional attention and self-distillation as described in Embodiment 1 or 2.

[0108] Example 5 A multimodal emotion recognition system based on bidirectional attention and self-distillation includes: The data acquisition and preprocessing module is configured as: multimodal data acquisition and preprocessing; The trimodal mask construction and missing modality sample generation module is configured to include: constructing input masks and missing masks for three modalities; and using the missing mask to modify the original trimodal features to form a missing version of the input used by the student path; The trimodal unified embedding and non-linguistic modality compression alignment module is configured to: map the three modalities to a unified embedding dimension; model the temporal dependencies within each modality; and perform standard multi-head attention operations and residual connections on the multimodal spliced ​​sequence to capture high-order correlations within and between sequences. The single-modal temporal coding and bidirectional cross-attention multimodal fusion module is configured to: perform temporal coding on each modality separately to model the temporal dependencies within the modality; and use a bidirectional cross-attention multimodal fusion mechanism to explicitly model the bidirectional interaction between the three modalities on a unified time axis, and produce fused text, audio, and video representations respectively. The adaptive supermodal learning module is configured to: converge intramodal temporal data to obtain global representations of the three modalities; concatenate the global representations of the three modalities on the feature dimension to construct a set of multimodal joint feature vectors, and introduce residual connections to obtain the final joint sentiment representation; and perform modal contribution balancing. The loss construction module is configured to construct losses for both teacher and student paths. The dual-policy joint optimization and training module is configured to: implement dual-policy joint optimization and training; The inference module is configured to implement multimodal emotion recognition.

Claims

1. A multimodal emotion recognition method integrating bidirectional attention and self-distillation, characterized in that, include: Step 1: Multimodal data acquisition and preprocessing; Step 2: Construction of trimodal masks and generation of missing modality samples; including: constructing input masks and missing masks for the three modalities; and using the missing masks to modify the original trimodal features to form the missing version of the input used by the student path; Step 3: Trimodal Unified Embedding and Non-linguistic Modality Compression Alignment; including: mapping the three modalities to a unified embedding dimension; modeling the temporal dependencies within each modality; and performing standard multi-head attention operations and residual connections on the multimodal spliced ​​sequence to capture high-order correlations within and between sequences. Step 4: Single-modal temporal coding and bidirectional cross-attention multimodal fusion; including: performing temporal coding on each modality separately to model the temporal dependencies within the modality; using a bidirectional cross-attention multimodal fusion mechanism to explicitly model the bidirectional interaction between the three modalities on a unified time axis, and producing fused text, audio, and video representations respectively; Step 5: Adaptive hypermodal learning; including: intramodal temporal convergence to obtain global representations of the three modalities; concatenating the global representations of the three modalities on the feature dimension to construct a set of multimodal joint feature vectors, and introducing residual connections to obtain the final joint sentiment representation; performing modal contribution balancing; Step 6: Constructing the loss for teacher and student paths; Step 7: Dual-strategy joint optimization and training process; Step 8: Inference stage, realize multimodal emotion recognition.

2. The multimodal emotion recognition method integrating bidirectional attention and self-distillation according to claim 1, characterized in that, The specific implementation process of step 1 includes: The video stream input data, i.e., the video modality, is feature-encoded using OpenFace to obtain a video feature sequence; the text data, i.e., the text modality, is feature-encoded using BERT to obtain a text feature sequence; and the input audio stream, i.e., the audio modality, is feature-encoded using Librosa to obtain an audio feature sequence. Representing video feature sequences, text feature sequences, or audio feature sequences as... ,in , It is the sequence length. is the vector dimension of each mode; where, , , These represent text modality, audio modality, and visual modality, respectively. Set emotion tags ,in C The number of categories; during training, mini-batch input is used, with a batch size of [value missing]. B The sequence lengths of each modality within the same batch are aligned to a uniform length by padding. , ,get: and the corresponding label vector: ; The complete input obtained through the above operations refers to the complete modal feature sequence. , as input for the teacher's path.

3. The multimodal emotion recognition method integrating bidirectional attention and self-distillation according to claim 1, characterized in that, The specific implementation process of step 2 includes: Step 2.1: Construct input masks for three modes: , where: for the text modality, when the first b The first sample t When each token is a real token ,otherwise, For audio and video modalities, when the first b The first sample t When the frame is a real frame, the corresponding mask is 1; otherwise, it is 0. For each modality, based on the preset missing rate p and missing patterns, in The missing mask is obtained by upsampling at the location: 0 indicates that the position is considered missing under the current configuration, and 1 indicates that the position is considered reserved under the current configuration; Step 2.2: Modify the original trimodal features using a missing mask to form a missing version of the input used by the student path; For positions in the text modality where a word was present but was marked as 0 by a missing mask, the original token is replaced with a [UNK] vector: ； in, It's a missing mask. It is the original input mask. It is a vector of [UNK]. It is a complete text modal feature sequence, used as the teacher's path input; It is a sequence of missing text modal features, used as the student path input; For audio and video modalities, when the missing mask is 0, the entire frame feature is directly multiplied by 0: ； in, and These represent the missing masks for the audio and video modalities, respectively. and These represent the complete feature sequences for the audio and video modalities, respectively. and These represent the missing feature sequences for the audio and video modalities, respectively. The missing version input obtained through the above operations is used as the input for the student path. The missing version input refers to the modal feature sequence after missing masking. .

4. The multimodal emotion recognition method integrating bidirectional attention and self-distillation according to claim 1, characterized in that, The specific implementation process of step 3 includes: Step 3.1: Perform linear mapping on the three modes respectively, transforming the three modes from their original dimensions. Mapping to a unified embedding dimension d ; denote the mode m The mapping function is Then, the teacher path and the student path are represented in the unified embedding space as follows: ； ； in, Representing text, audio, and video modalities; For complete modal feature sequences, This is a sequence of missing modal features; For modality A linear mapping function; , These are learnable parameters; The teacher path and student path are represented by their respective paths mapped to a unified embedding space, with the teacher path and student path sharing the same set of embeddings. To ensure proper distillation alignment; Step 3.2: Employing a multi-head self-attention and feedforward network within the Transformer architecture, the temporal dependencies within each modality are modeled, and standard multi-head attention operations and residual connections are performed on the multimodal spliced ​​sequence to capture higher-order correlations within and between sequences; including: A cross-modal compression and alignment mechanism based on the text modality is employed. Using text sequences as queries and audio / video sequences as keys and values, multi-head attention operations of the Transformer are used to calculate the attention weights of the text timesteps on the audio and video timesteps. Based on this, audio and video features are weighted, thereby compressing and aligning the originally disparate audio and video sequences to the same time length as the text sequence. Non-verbal modalities are also compressed to the same length as the text sequence. Consistent expression; Further preferably, in step 3.2, for the audio modality, the text is uniformly embedded as... Audio is uniformly embedded as Based on text input masks and audio input masks, a cross-modal compression mapping function is defined. The audio features are weighted and aggregated along the text time steps to obtain a compressed audio representation with the same length as the text. : ； in, Enter a mask for the text. Use an audio input mask; function Internally, it employs a multi-head attention structure based on text as the query and audio as the key and value, combined with masking to suppress attention weights at padding positions, thereby weighted aggregation of related audio time steps at each text time step; specifically including: Embedded with text As a query, with audio embedded As key and value pairs, multi-head attention is used to calculate the correlation score between text time steps and audio time steps. and utilize Assign values ​​to the audio padding positions in the fraction matrix To suppress its attention weights, the weights are then obtained by softmax normalization over the audio time dimension for each text time step. The audio value vectors are weighted and aggregated, and finally, a compressed audio representation with the same length as the text is obtained through multi-head concatenation and linear mapping. This allows for the alignment and aggregation of relevant audio information at each text time step.

5. The multimodal emotion recognition method integrating bidirectional attention and self-distillation according to claim 1, characterized in that, The specific implementation process of step 4 includes: Step 4.1: Multimodal sequence concatenation and unified encoding input; By concatenating the temporal representations of the three modes in modal order on the timeline, a synthesized sequence is obtained: ； in For text embedding, This is a compressed and aligned representation of audio and video. This is the total length after splicing; A bidirectional cross-attention multimodal fusion mechanism is used to explicitly model the bidirectional interaction relationship between the three modalities on a unified time axis, and to produce fused text, audio and video representations respectively. Step 4.2: Construct various local-to-local attention branches; In the fusion layer, the following types of attention branches are constructed simultaneously within the same attention module: Trimodal global attention branch Model the global interactions between text, audio, and video across the entire concatenated sequence; Text-audio bidirectional cross-attention branch Text-video bidirectional cross-attention branch Audio-video bidirectional cross-attention branch Modeling mutual interest only between the two modes; Text self-attention branch Audio self-attention branch Video self-attention branch Modeling is only performed within the single-modal region; Step 4.3: Aggregate the outputs of different attention branches by modality After completing the calculations for several types of attention branches, For each modality, aggregate the output of its associated branch separately; These outputs are concatenated along the feature dimension, denoted as: ； in, This indicates that the output subsequence at the location of the text is extracted from the corresponding attention branch. This is the temporary dimension after splicing; This refers to the joint intermediate representation of text modalities after multi-branch attention modeling; Subsequently, through linear transformation Compress back to a unified dimension The resulting fused text representation is: ； in, , These are the parameters for the text modality fusion layer; Perform the above operations on the audio and video modalities to obtain the fused audio representation. and the merged video representation .

6. The multimodal emotion recognition method integrating bidirectional attention and self-distillation according to claim 1, characterized in that, The specific implementation process of step 5 includes: Step 5.1: Intramodal temporal convergence to obtain global representations of the three modes; First, attention convergence is performed separately in the temporal dimension for each modality, compressing the temporal features into a fixed-dimensional modality-level representation vector; for any modality m Its temporal characteristics Recorded as: ； in, Indicates the first t The hidden state at each time step; Then, an attention score is generated for each time step through a scoring network. : ； in, 、 、 For modality Attention parameters; Next, the attention score is normalized using softmax to obtain the weights in the time dimension. : ； Based on this, the modality m The modalities are obtained by weighted summation of all time steps. m global representation vector : ； Step 5.2: Concatenate the global representations of the three modalities along the feature dimension to construct a set of multimodal joint feature vectors. A feedforward network is applied to the multimodal joint feature vector. The feedforward network is a two-layer fully connected feedforward structure used for nonlinear transformation and enhanced representation of the multimodal joint feature vector; specifically, it includes: a first-layer linear transformation through a weight matrix... and bias terms Input joint feature vector Mapped to an intermediate feature space and activated by the ReLU nonlinear activation function. Introducing nonlinear expressive power; the second-level linear transformation uses a weight matrix. and bias terms intermediate features Map back to the same feature dimensions as the input, and connect the original input via residual connections. Adding the transformed features to obtain the final joint sentiment representation. ; By introducing residual connections, the final joint sentiment representation is obtained: ; ;in, For feedforward network parameters, It is a ReLU nonlinear activation function. The joint sentiment representation output by the AHL module; Both the teacher's path and the student's path obtain their joint representations through adaptive hypermodal learning. and Parameter sharing; Step 5.3: Perform modal contribution balancing; including: First, the training data is divided into samples based on sentiment categories. The training data includes trimodal inputs and their labels. For each sentiment category, the text, audio, and video modal features of the samples under that sentiment category are pooled separately to obtain the global modal representations. , and ; The average is calculated to obtain the class prototype vectors for the three modalities: Design a prototype encoder for each mode, and... Mapping to The fused feature space; let these encoding functions be: ； in, and It has matching dimensions or concatenable dimensions, and It includes several layers of fully connected networks and ReLU nonlinear activation; For each emotion category C Constructing category-conditional extended feature vectors : ； in, This represents the multimodal joint sentiment representation of the current sample. Indicate category C Typical patterns across the three modalities; For each category C Set a set of independent linear transformation and bias parameters The class conditional prediction distribution is obtained through softmax: ;in, For category C The corresponding weight matrix, For bias vectors, Indicates the category prototype C Under the given conditions, the predicted probability distribution for all category labels; Further optimization involves introducing a category prior probability vector: ； in, For category C The frequency of occurrence in the training set, or the prior probability after smoothing / reweighting; During inference and training, all category conditional predictions are weighted according to priors to obtain the final sentiment prediction distribution: ； in, This represents the final predicted probability for each sentiment category under the current sample input conditions; The final prediction category is: ,in, To predict category labels, To predict the final distribution in categories The probability of it.

7. A multimodal emotion recognition method integrating bidirectional attention and self-distillation according to any one of claims 1-6, characterized in that, The specific implementation process of step 6 includes: Step 6.1: Let the true label vector of the current batch of samples be: ; The teacher path task loss uses cross-entropy: ； in, Batch size; Number of categories; For the first The true labels of each sample; For indicator functions; For teacher paths to categories The predicted probability; Step 6.2: In the student path, the input is the missing version of the three-modal features constructed in Step 2, and the output is the predicted probability distribution of the student path. With joint emotional expression ; Step 6.2.1: Student path task loss; Output under missing modal input Calculate the cross-entropy loss: ； in Batch size; Number of categories; For the first The true labels of each sample; For indicator functions; For student paths to categories The predicted probability; Step 6.2.2: Combined representation of distillation loss; We introduce a joint representation distillation loss based on cosine similarity, where the joint representation of teacher paths is... Student paths are jointly represented as Then for the first i Each sample is defined as follows: ； in, For Teacher Pathway i Joint representation of samples; This represents the joint representation of the student's path; This means that the gradient is not propagated back to the teacher path when calculating the gradient; it is only used as the distillation target. The distillation loss is based on cosine similarity for joint representation. The average of the entire batch is obtained. : ； Step 6.2.3: Distillation loss of attention distribution; Introducing attention distribution distillation loss, let the multimodal attention weights of the last layer of the teacher path be... A The student path corresponds to a weight of After performing softmax normalization on the bond dimension, and using KL divergence to measure the difference between the two, the attention distillation loss is defined as: ； The softmax mechanism is consistent with the masking mechanism, normalizing only at valid locations. The KL divergence is the summation of elements. This refers to attention distillation loss; Step 6.2.4: Multimodal reconstruction loss; A reconstruction branch is introduced to force the network to understand and fill in the missing information by reconstructing the trimodal features from the fused representation; For the text modality, let the reconstruction target be the encoded text features in the teacher path. The student reconstructed the output as follows Construct an effective reconstruction mask Then the text reconstruction loss for: ； in, For element-wise multiplication, To prevent small constants with a denominator of zero; For the audio modality and the video modality, the audio reconstruction loss is obtained separately. and video reconstruction loss Total reconstruction loss for: ； In summary, the total loss is constructed along the student's path. : in, These are non-negative weighted hyperparameters used to balance the contributions of each distillation and reconstruction loss to the overall training. More preferably, the specific implementation process of step 7 includes: Step 7.1: Teacher's forward and reverse paths; Use full modal input Perform forward computation according to steps 1 to 5 and step 6 to obtain... And calculate the teachers' losses Then the teacher suffered losses Perform backpropagation to update network parameters; , , These refer to complete modal sequence inputs of text, audio, and video, respectively. Step 7.2: Student path forward and backward; Using missing modal input Execute forward on the same network to obtain , ; Combining teacher pathways and Calculate the total student path loss according to step 6. ,right Perform backpropagation and update the same set of parameters again; Step 7.3: Overall Optimization Goal; The overall optimization objective of the entire training process is to minimize the weighted sum of the losses from the teacher's path and the student's path. ； in For all learnable parameters; This is a balancing factor used to balance the performance of complete modes with the robustness of missing modes; More preferably, the specific implementation process of step 8 includes: First, feature sequences of text, audio, and video are obtained from the samples to be predicted. Construct the corresponding input mask ; Next, steps 3 to 4 are executed sequentially to obtain the three-modal fusion temporal features. Perform adaptive hypermodal learning to construct a joint sentiment representation. By combining pre-statistical category prototypes and category priors, a modal contribution balancing mechanism is executed to obtain the final predicted distribution. ; Finally, according to The index of the category with the highest probability is used to output the final sentiment recognition result. .

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the multimodal emotion recognition method that integrates bidirectional attention and self-distillation as described in any one of claims 1-7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the multimodal emotion recognition method that integrates bidirectional attention and self-distillation as described in any one of claims 1-7.

10. A multimodal emotion recognition system based on bidirectional attention and self-distillation, characterized in that, include: The data acquisition and preprocessing module is configured as: multimodal data acquisition and preprocessing; The trimodal mask construction and missing modality sample generation module is configured to include: constructing input masks and missing masks for three modalities; and using the missing mask to modify the original trimodal features to form a missing version of the input used by the student path; The trimodal unified embedding and non-linguistic modality compression alignment module is configured to: map the three modalities to a unified embedding dimension; model the temporal dependencies within each modality; and perform standard multi-head attention operations and residual connections on the multimodal spliced ​​sequence to capture high-order correlations within and between sequences. The single-modal temporal coding and bidirectional cross-attention multimodal fusion module is configured to: perform temporal coding on each modality separately to model the temporal dependencies within the modality; and use a bidirectional cross-attention multimodal fusion mechanism to explicitly model the bidirectional interaction between the three modalities on a unified time axis, and produce fused text, audio, and video representations respectively. The adaptive supermodal learning module is configured to: converge intramodal temporal data to obtain global representations of the three modalities; concatenate the global representations of the three modalities on the feature dimension to construct a set of multimodal joint feature vectors, and introduce residual connections to obtain the final joint sentiment representation; and perform modal contribution balancing. The loss construction module is configured to construct losses for both teacher and student paths. The dual-policy joint optimization and training module is configured to: implement dual-policy joint optimization and training; The inference module is configured to implement multimodal emotion recognition.

Images