A multi-modal sentiment analysis method and device based on teacher guided distillation and text query selective alignment

Abstract

The application discloses a kind of multi-modal sentiment analysis method and device based on teacher guide distillation and text query selective alignment, it is related to sentiment analysis technical field, comprising: the multi-modal sentiment data is input into trained multi-modal sentiment analysis model, carries out sentiment prediction and outputs sentiment prediction value, wherein, multi-modal sentiment analysis model includes feature extraction module, position encoder, text query selective cross-modal attention module, mode coordination gate module, feature fusion module, time pooling layer and regression prediction head.Based on the above scheme, it has good accuracy and robustness in complex sentiment recognition scene.

Description

Technical Field

[0001] This invention relates to the field of sentiment analysis technology, and in particular to a multimodal sentiment analysis method and apparatus based on teacher-guided distillation and selective alignment of text queries. Background Technology

[0002] Multimodal sentiment analysis (MSA) aims to integrate information from multiple sources, such as text, audio, and visual signals, for sentiment recognition. However, in the multimodal fusion stage, the importance of each modality often changes dynamically with the context. Traditional simple splicing or fixed weight strategies cannot fully utilize the complementarity between modalities. At the same time, noise in low semantic density modalities such as audio and video can easily interfere with the stability of fusion, so it is urgent to improve the performance of sentiment recognition. Summary of the Invention

[0003] This invention provides a multimodal sentiment analysis method and apparatus based on teacher-guided distillation and selective alignment of text queries, which solves the technical problem that existing multimodal sentiment analysis methods urgently need to improve sentiment recognition performance.

[0004] The first aspect of this invention provides a multimodal sentiment analysis method based on teacher-guided distillation and selective alignment of text queries, comprising:

[0005] Input multimodal sentiment data into the trained multimodal sentiment analysis model;

[0006] The textual semantic features, audio semantic features, and video semantic features of the multimodal emotion data are determined by the feature extraction module based on teacher-guided distillation of non-textual modalities.

[0007] The text semantic features, audio semantic features, and video semantic features are encoded using a position encoder to generate text encoded features, audio encoded features, and video encoded features respectively;

[0008] Based on the text query selective cross-modal attention module, the text encoding features guide the audio encoding features and the video encoding features to perform semantic alignment, and output audio enhancement features and video enhancement features respectively;

[0009] According to the modal collaborative gating module, gating fusion is performed on the text semantic features and the audio semantic features, and on the text semantic features and the video semantic features, respectively, to determine the audio gating signal and the video gating signal;

[0010] In the feature fusion module, the audio enhancement feature is multiplied element-wise with the audio gating signal, and the video enhancement feature is multiplied element-wise with the video gating signal, and then added together to construct a comprehensive feature;

[0011] The comprehensive features are transformed through a time pooling layer and then input into a regression prediction head for sentiment prediction, outputting a sentiment prediction value.

[0012] Furthermore, the multimodal sentiment data includes text sentiment data, audio sentiment data, and video sentiment data; the feature extraction module includes a BERT-like pre-trained language model, a teacher multilayer perceptron, a student Transformer encoder, a linear layer, and a residual adapter; the step of determining the text semantic features, audio semantic features, and video semantic features of the multimodal sentiment data through teacher-guided distillation based on non-textual modalities via the feature extraction module includes:

[0013] After extracting features from the text sentiment data using a BERT-like pre-trained language model, the features are projected onto the text semantic features through a linear layer.

[0014] Based on a cascaded student Transformer encoder and linear layers, feature processing is performed on audio sentiment data and video sentiment data respectively, and corresponding student features for audio and student features for video are determined.

[0015] By using a cascaded teacher multilayer perceptron and linear layer, feature processing is performed on the audio emotion data and the video emotion data respectively, and corresponding audio distillation features and video distillation features are determined.

[0016] Using a residual adapter, the audio distillation features and the video distillation features are respectively adapted, and then residually connected with the associated audio student features and video student features to output audio semantic features and video semantic features.

[0017] The teacher multilayer perceptron and residual adapter both include cascaded linear layers, ReLU activation functions, and layer normalization layers.

[0018] Furthermore, the text query-based selective cross-modal attention module guides the audio encoding features and the video encoding features to perform semantic alignment through the text encoding features, and outputs audio enhancement features and video enhancement features, including:

[0019] Project the text encoding features into a text query matrix;

[0020] The audio encoding features are mapped into an audio key matrix and an audio value matrix;

[0021] The video coding features are mapped into a video key matrix and a video value matrix;

[0022] The audio attention weights and video attention weights are determined by performing scaling dot product operations on the text query matrix and the audio key matrix, and on the text query matrix and the video key matrix, respectively, followed by Softmax normalization.

[0023] The audio attention weights and video attention weights are multiplied element-wise to generate enhanced attention weights;

[0024] The enhanced attention weights are multiplied by the audio value matrix and the video value matrix respectively to output audio enhancement features and video enhancement features.

[0025] Furthermore, the modal collaborative gating module includes a splicing layer, a multilayer perceptron, a Sigmoid activation function, a residual gating fusion layer, an average pooling layer, an L2 normalization layer, and a linear layer; the feature processing of the modal collaborative gating module includes:

[0026] For any input non-textual semantic feature, the non-textual semantic feature and the textual semantic feature are combined through a concatenation layer to construct a collaborative feature;

[0027] The fused features are processed using a cascaded multilayer perceptron and a sigmoid activation function to generate a gating factor;

[0028] In the residual gated fusion layer, semantic gated features are constructed by multiplying the non-textual semantic features and the textual semantic features element by element based on the gated factor and then summing them. The semantic gated features are then multiplied element by element with the non-textual semantic features to determine the semantic fusion features.

[0029] The semantic fusion features are input into a cascaded average pooling layer, L2 normalization layer, and linear layer, and a gating signal is output.

[0030] Furthermore, the process of determining the teacher's multilayer perceptron includes:

[0031] For any non-text modality, obtain training non-text sentiment data and input it into the multilayer perceptron of the teacher to be trained for feature extraction, and output non-text teacher features;

[0032] After performing temporal modeling on the non-text teacher features using an LSTM encoder, a temporal pooling layer is used for temporal aggregation to determine the non-text global features.

[0033] Based on the regression multilayer perceptron, sentiment regression prediction is performed on the non-text global features, and the non-text scalar prediction value is output.

[0034] The model parameters of the teacher multilayer perceptron are optimized based on the non-text scalar prediction values ​​using the teacher regression loss function until the trained teacher multilayer perceptron is determined as the teacher multilayer perceptron for the multimodal sentiment analysis model.

[0035] Furthermore, the process of determining the trained multimodal sentiment analysis model includes:

[0036] The training multimodal sentiment data is input into the multimodal sentiment analysis model to be trained. The training audio student features, training video student features, training audio distillation features, and training video distillation features output by the feature extraction module, as well as the training sentiment prediction value output by the regression prediction head, are determined. The model parameters are then jointly iteratively optimized through sentiment master task loss and distillation loss until the trained multimodal sentiment analysis model is determined.

[0037] A second aspect of the present invention provides a multimodal sentiment analysis device based on teacher-guided distillation and selective alignment of text queries, comprising:

[0038] The input module is used to input multimodal sentiment data into the trained multimodal sentiment analysis model;

[0039] The extraction module is used to determine the text semantic features, audio semantic features, and video semantic features of the multimodal sentiment data based on teacher-guided distillation of non-textual modalities through the feature extraction module.

[0040] The encoding module is used to encode the text semantic features, the audio semantic features, and the video semantic features using a position encoder, respectively, to generate text encoded features, audio encoded features, and video encoded features;

[0041] The enhancement module is used to guide the audio encoding features and the video encoding features to perform semantic alignment based on the text query selective cross-modal attention module, and output audio enhancement features and video enhancement features respectively.

[0042] The gating module is used to perform gating fusion on the text semantic features and the audio semantic features, and the text semantic features and the video semantic features, respectively, according to the modal collaborative gating module, to determine the audio gating signal and the video gating signal;

[0043] The integration module is used in the feature fusion module to multiply the audio enhancement feature with the audio gating signal and the video enhancement feature with the video gating signal element by element and then add them together to construct a comprehensive feature;

[0044] The prediction module is used to transform the comprehensive features through a time pooling layer, input them into a regression prediction head for sentiment prediction, and output the sentiment prediction value.

[0045] A computer device provided in a third aspect of the present invention includes a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of the multimodal sentiment analysis method based on teacher-guided distillation and selective alignment of text queries as described in any of the preceding claims.

[0046] The fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed, implements the multimodal sentiment analysis method based on teacher-guided distillation and selective alignment of text queries as described in any of the preceding claims.

[0047] The fifth aspect of the present invention provides a computer program product comprising a computer program / instruction, wherein when the computer program / instruction is executed by a processor, it implements the multimodal sentiment analysis method based on teacher-guided distillation and selective alignment of text queries as described in any of the preceding claims.

[0048] As can be seen from the above technical solutions, the present invention has the following advantages:

[0049] The above-mentioned scheme of the present invention provides a multimodal sentiment analysis method based on teacher-guided distillation and text query selective alignment, comprising: inputting multimodal sentiment data into a trained multimodal sentiment analysis model; determining text semantic features, audio semantic features, and video semantic features of the multimodal sentiment data through a feature extraction module based on teacher-guided distillation of non-text modalities; encoding the text semantic features, audio semantic features, and video semantic features respectively using a position encoder to generate text encoded features, audio encoded features, and video encoded features; based on a text query selective cross-modal attention module, semantically aligning the audio encoded features and video encoded features through the text encoded features to output audio enhancement features and video enhancement features; according to a modal collaborative gating module, gating and fusing the text semantic features with the audio semantic features and the video semantic features respectively to determine the audio gating signal and the video gating signal; in the feature fusion module, multiplying the audio enhancement features with the audio gating signal and the video enhancement features with the video gating signal element by element and then adding them together to construct a comprehensive feature; transforming the comprehensive feature through a time pooling layer and inputting it into a regression prediction head for sentiment prediction, outputting the sentiment prediction value. Based on the above scheme, the ability to express the semantic features of non-text modalities is enhanced by teacher-guided distillation. Text modalities are used as reliable query vectors to explicitly filter and temporally align the features of non-text modalities. Modal collaborative gating is used to dynamically adjust the contribution of each modality to the comprehensive features in order to further eliminate redundant noise between modalities. It has shown excellent selective control ability and semantic adaptability in multimodal sentiment analysis tasks, and has good accuracy and robustness in complex sentiment recognition scenarios. Attached Figure Description

[0050] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0051] Figure 1 This is a flowchart illustrating the steps of a multimodal sentiment analysis method based on teacher-guided distillation and selective alignment of text queries, as provided in Embodiment 1 of the present invention.

[0052] Figure 2 This is a schematic diagram of the framework of the multimodal sentiment analysis model provided in Embodiment 1 of the present invention;

[0053] Figure 3 This is a schematic diagram of the structure of a teacher network provided in Embodiment 1 of the present invention;

[0054] Figure 4 This is a schematic diagram of the modal cooperative gating module provided in Embodiment 1 of the present invention;

[0055] Figure 5 This is a structural block diagram of a multimodal sentiment analysis device based on teacher-guided distillation and selective alignment of text queries, provided in Embodiment 2 of the present invention. Detailed Implementation

[0056] This invention provides a multimodal sentiment analysis method and apparatus based on teacher-guided distillation and selective alignment of text queries, which addresses the technical problem of improving sentiment recognition performance in existing multimodal sentiment analysis methods.

[0057] To make the objectives, features, and advantages of this invention more apparent and understandable, the technical solutions of the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the embodiments described below are only some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0058] Please see Figure 1 The present invention provides a multimodal sentiment analysis method based on teacher-guided distillation and selective alignment of text queries, comprising:

[0059] Step 101: Input the multimodal sentiment data into the trained multimodal sentiment analysis model.

[0060] It should be noted that, for multimodal emotion recognition tasks, this embodiment builds a platform as follows: Figure 2 The multimodal sentiment analysis model (TeQSA-G) shown includes a feature extraction module, a position encoder, a text query selective cross-modal attention (TQSA) module, a modal collaborative gating (MCG) module, a feature fusion module, a temporal pooling layer, and a regression prediction head. The model is trained by iteratively optimizing the model parameters through training the multimodal sentiment analysis model to be trained, thereby determining the trained multimodal sentiment analysis model. When the multimodal sentiment data to be analyzed is obtained, it is input into the trained multimodal sentiment analysis model for sentiment recognition processing.

[0061] Step 102: Based on teacher-guided distillation of non-textual modalities, determine the textual semantic features, audio semantic features, and video semantic features of multimodal sentiment data through the feature extraction module.

[0062] In one specific embodiment of this example, step 102 includes the following sub-steps:

[0063] S11. After extracting features from the text sentiment data using a BERT-like pre-trained language model, the features are projected onto the text semantic features through a linear layer.

[0064] S12. Based on the cascaded student Transformer encoder and linear layer, feature processing is performed on audio sentiment data and video sentiment data respectively, and corresponding student features of audio and student features of video are determined.

[0065] S13. Through cascaded teacher multilayer perceptron and linear layer, feature processing is performed on audio emotion data and video emotion data respectively, and corresponding audio distillation features and video distillation features are determined.

[0066] S14. Using a residual adapter, after performing feature adaptation on the audio distillation features and video distillation features respectively, the corresponding audio student features and video student features are residually connected to output audio semantic features and video semantic features.

[0067] It should be noted that multimodal data includes text modality (Text, t) and non-text modality. Non-text modality includes audio modality (Audio, a) and video modality (Video, v). Correspondingly, multimodal sentiment data includes text sentiment data, audio sentiment data, and video sentiment data; such as Figure 2 As shown, the feature extraction module in this implementation includes a BERT-like pre-trained language model, a teacher multilayer perceptron (Teacher MLP), a student Transformer encoder (TransformerEncoder), linear layers, and a residual adapter, specifically:

[0068] For text sentiment data: This embodiment uses a BERT-like pre-trained language model for semantic encoding. For example, the BERT-like pre-trained language model includes the RoBERTa-large model. Let the text sentiment data be represented as a sequence of tokens. , Indicates a unit index of words. Indicates the length of the text sequence. Indicates the first If there are 10 word units, then the processing procedure of the BERT-like pre-trained language model is represented as follows: , Indicates text embedding, This represents the RoBERTa-large model. This represents the output feature dimension of a BERT-like pre-trained language model. To align the features of the text modality with those of other modalities, a linear layer is applied for linear projection to achieve dimension unification. , Representing the semantic features of the text. Represents the text weight matrix. Indicates text bias. Represented as alignment feature dimension;

[0069] For both audio and video sentiment data, this embodiment introduces a teacher-guided distillation mechanism, which can enhance the representation accuracy of low semantic density modalities from the source, laying a robust foundation for subsequent fusion.

[0070] First, sequence modeling is performed using a structure-consistent student Transformer encoder to capture temporally dependent emotional cues in audio / video emotional data: audio emotional data is represented as a sequence of frame-level acoustic features. , Indicates the number of audio frames. Representing the original acoustic feature dimension, we obtain... , Indicates audio embedding. A student Transformer encoder represents the audio modality; video sentiment data is represented as a sequence of frame-level visual features. , Indicates the number of video frames. Representing the original visual feature dimension, we obtain... , Indicates video embedding, The student Transformer encoder represents the video modality;

[0071] Next, dimension alignment is performed using linear layer projection: , , Indicates audio student characteristics. Represents the audio weight matrix. Indicates the audio bias term. Indicates the characteristics of students in the video. Represents the video weight matrix. Indicates the video bias term;

[0072] Simultaneously, to provide stable and discriminative single-modal supervision signals, this embodiment introduces a teacher multilayer perceptron to inject stable single-modal prior knowledge into the audio and video modalities. This prior knowledge is used as teacher features and treated as distillation targets to guide the student Transformer encoder. Therefore, the audio teacher features / video teacher features output through the processing of audio / video sentiment data by the teacher multilayer perceptron are... , Indicates modal index, The frame index is represented, and then a fixed linear projection is applied. Align dimensions to obtain audio distillation features / video distillation features for distillation. , Indicates the batch size. Indicates the number of frames;

[0073] Finally, this embodiment introduces a lightweight residual adapter that operates at each time step (frame) to achieve flexible feature integration. The residual adapter includes cascaded linear layers, ReLU activation functions, and layer normalization layers, and its processing can be represented as follows: , , Representing modes ( The adaptation features of ) Indicates adapter, This represents the fit weight matrix. Indicates the adaptation bias term. Represents the ReLU activation function. Presentation layer normalization layer, Representing modes semantic features, Representing modes This fusion approach, which incorporates student characteristics, retains the structural features of the original modality while injecting discriminative information from the teacher network, thereby improving the quality of subsequent representations.

[0074] In one specific embodiment of this example, the process of determining the teacher's multilayer perceptron includes:

[0075] S21. For any non-text modality, obtain training non-text sentiment data and input it into the multilayer perceptron of the teacher to be trained to extract features and output non-text teacher features.

[0076] S22. After performing temporal modeling on the non-text teacher features using an LSTM encoder, a temporal pooling layer is used for temporal aggregation to determine the non-text global features.

[0077] S23. Based on the regression multilayer perceptron, perform sentiment regression prediction on non-text global features and output non-text scalar prediction values.

[0078] S24. The teacher regression loss function is used to optimize the model parameters of the teacher multilayer perceptron based on the non-text scalar prediction values ​​until the trained teacher multilayer perceptron is determined as the teacher multilayer perceptron for the multimodal sentiment analysis model.

[0079] It should be noted that the training process of the teacher multilayer perceptron is independent of the training process of the multimodal sentiment analysis model. That is, during the backpropagation phase of the multimodal sentiment analysis model training, the model parameters of the teacher multilayer perceptron remain unchanged, and it only provides forward output as a fixed prior. This design can inject stable modal prior knowledge into the main model without increasing the additional training cost, thereby enhancing the representation ability. Therefore, in this embodiment, the designed teacher network is first pre-trained on the same sentiment regression task as the main model until it converges on the sentiment regression target.

[0080] like Figure 3 As shown, the teacher network, from bottom to top, includes a teacher multilayer perceptron, an LSTM encoder, a temporal pooling layer, and a back multilayer perceptron (Back MLP), specifically:

[0081] For each mode The input is a frame-level feature sequence of non-text sentiment data. (Including audio sentiment data / video sentiment data) Indicates the frame index. Indicates the number of frames. Representing modes The Sentiment data from frames;

[0082] First, features are extracted from the input using a teacher multilayer perceptron, which consists of cascaded linear layers, ReLU activation functions, and layer normalization layers. , Representing modes The teacher's multi-layer perceptron, Representing modes The weight matrix, Representing modes The bias term, Represents the ReLU activation function. Presentation layer normalization layer, Representing modes No. The teacher characteristics of the frame;

[0083] Subsequently, teacher characteristics The sequence of hidden states is fed into an LSTM encoder to generate the hidden state sequence. ,in Representing modes No. The hidden state of the frame is then transformed into a utterance-level vector through temporal pooling, yielding the modality. global features Finally, a lightweight regression head is used to generate modality-specific scalar predictions.

[0084] Based on this scalar predicted value, the teacher regression loss value is calculated using the teacher regression loss function. For example, the mean squared error (MSE) loss function can be used: , Representing modes The teacher regression loss value, Indicates the training sample index. This represents the total number of training samples. Representing modes The One training sentiment data, Representing modes The model parameters of the teacher network, Representing modes The teacher network mapping function, Indicates the first The model parameters are iteratively optimized based on the teacher regression loss value until the teacher regression loss value converges. The trained teacher network is then determined, all its parameters are frozen, and only the teacher multilayer perceptron is reused in the multimodal sentiment analysis model to avoid semantic conflicts between high-level abstract features and the student Transformer encoder.

[0085] Step 103: Use a position encoder to encode the text semantic features, audio semantic features and video semantic features respectively to generate text encoded features, audio encoded features and video encoded features.

[0086] It should be noted that this embodiment uses a position encoder to inject position / timing information, which facilitates subsequent attention calculations.

[0087] Step 104: Based on the text query selective cross-modal attention module, the audio encoding features and video encoding features are guided to perform semantic alignment through the text encoding features, and the audio enhancement features and video enhancement features are output.

[0088] In one specific embodiment of this example, step 104 includes the following sub-steps:

[0089] S31. Project the text encoding features into a text query matrix;

[0090] S32. Map the audio coding features into an audio key matrix and an audio value matrix;

[0091] S33. Map the video coding features into a video key matrix and a video value matrix;

[0092] S34. After scaling and dot product operations using the text query matrix and audio key matrix, and the text query matrix and video key matrix respectively, Softmax normalization is performed to determine the audio attention weight and video attention weight accordingly.

[0093] S35. Multiply the audio attention weights and video attention weights element by element to generate enhanced attention weights;

[0094] S36. Multiply the enhanced attention weights with the audio value matrix and the video value matrix respectively to output the audio enhancement features and video enhancement features.

[0095] It should be noted that, in order to enhance local semantic interaction, this embodiment proposes a text query selective cross-modal attention module. By using the text modality as the guiding query and the audio-visual modality as the audio-visual modality, a dual-branch cross-modal attention network is constructed, and a gating mechanism is introduced to selectively extract information from the audio and visual modalities. This aims to explicitly achieve semantic alignment between audio-text and visual-text, thereby enhancing effective information interaction and significantly suppressing redundant noise in the modalities.

[0096] The feature processing procedure of this module can be represented as follows:

[0097] ;

[0098] in, ;

[0099] In the formula, Represents a text query matrix. Indicates text encoding features, This represents the query weight matrix. Represents the audio key matrix. Indicates audio coding characteristics, This represents the audio key weight matrix. Represents an audio value matrix. This represents the audio value weight matrix. Represents the video key matrix. Indicates video coding features, This represents the video key weight matrix. Represents the video value matrix. This represents the video value weight matrix. Indicates audio attention weights. This represents the Softmax activation function. This represents the transpose of the audio key matrix. Indicates the dimension of attention projection. Indicates video attention weights. This represents the transpose of the video key matrix. Indicates audio enhancement features, This indicates element-wise multiplication. This indicates video enhancement features.

[0100] Step 105: Based on the modal collaborative gating module, perform gating fusion on text semantic features and audio semantic features, and text semantic features and video semantic features respectively, to determine the audio gating signal and the video gating signal.

[0101] In one specific embodiment of this example, the feature processing procedure of the modal cooperative gating module includes:

[0102] S41. For any input non-textual semantic features, construct collaborative features by combining non-textual semantic features with textual semantic features through a concatenation layer;

[0103] S42. A cascaded multilayer perceptron and a sigmoid activation function are used to process the fused features and generate a gating factor.

[0104] S43. In the residual gated fusion layer, semantic gated features are constructed by multiplying non-textual semantic features and textual semantic features element by element based on the gated factor and then summing them. The semantic gated features are then multiplied element by element with the non-textual semantic features to determine the semantic fusion features.

[0105] S44. Input the semantic fusion features into the cascaded average pooling layer, L2 normalization layer and linear layer, and output the gated signal.

[0106] It should be noted that even after cross-membrane attention enhancement, the features may still contain redundant or conflicting information. This embodiment designs a modal collaborative gating module that supports trimodal modeling to perform weighted selective fusion. This module integrates a residual guidance mechanism and a normalization dynamic adjustment strategy, ensuring that the model maintains robust sentiment discrimination capabilities even when modal signals conflict or are partially missing. Figure 4As shown, the modal collaborative gating module includes a stitching layer, a multilayer perceptron, a sigmoid activation function, a residual gating fusion layer, an average pooling layer, an L2 normalization layer, and a linear layer; specifically:

[0107] The input to the modal collaborative gating module includes textual semantic features. Non-textual semantic features, including audio semantic features, are also included. and video semantic features Using any non-textual semantic feature (in The processing procedure will be explained using an example:

[0108] First, textual semantic features are used as textual priors and fused and concatenated with the non-textual semantic features of the current modality to construct the modality. synergistic features , This represents the splicing layer, where the collaborative features are processed by a multilayer perceptron (MLP) and a sigmoid activation function to generate modalities. Gating factor , This represents a multilayer perceptron. This represents the Sigmoid activation function;

[0109] Next, based on the gating factor, the text semantic features and non-text semantic features are gated and fused to determine the semantic gating features. Understandably, when At that time, the gating mechanism assigns almost all weight to the text-guided branches. At this point, the hybrid features depend more on the other modality, and the median value represents an adaptive trade-off between the two; subsequently, the semantic fusion features are obtained by superimposing the input non-textual semantic features through residual connections. To stabilize the fusion process and mitigate noise amplification effects caused by misalignment or low signal-to-noise ratio;

[0110] Furthermore, the semantic fusion features are fed into an average pooling layer to extract semantic representations. Then, L2 normalization is used to standardize the feature vectors to enhance robustness. Finally, these features are projected onto a unified representation space to obtain the gated signal. : , Representing modes The gating signal, Represents the gate weight matrix. Indicates the gating bias term. Indicates L2 normalized layer, This represents the average pooling layer.

[0111] Step 106: In the feature fusion module, the audio enhancement features and the audio gating signal, and the video enhancement features and the video gating signal are multiplied element by element and then added together to construct a comprehensive feature.

[0112] It should be noted that, as Figure 2 As shown, in the feature fusion module, enhanced features are combined with enhanced features for feature filtering to further refine the features, and a unified feature representation aligned to three modalities is obtained through an addition operation: , Indicates comprehensive characteristics, This indicates element-wise multiplication. Indicates an audio gating signal. This indicates a video gating signal.

[0113] Step 107: After transforming the comprehensive features through a time pooling layer, input them into the prediction head for sentiment prediction and output the sentiment prediction value.

[0114] It should be noted that the comprehensive features are first transformed through a temporal pooling layer, and then the prediction head uses the features to perform a sentiment regression task to calculate the sentiment prediction value: , , This indicates the pooling synthesis characteristics. Indicates a time-pooling layer. Indicates the prediction head, This represents the sentiment prediction value.

[0115] In one specific implementation of this embodiment, the process of determining the trained multimodal sentiment analysis model includes:

[0116] The training multimodal sentiment data is input into the multimodal sentiment analysis model to be trained. The training audio student features, training video student features, training audio distillation features, and training video distillation features output by the feature extraction module, as well as the training sentiment prediction value output by the prediction head, are determined. The model parameters are then jointly iteratively optimized through sentiment master task loss and distillation loss until the trained multimodal sentiment analysis model is determined.

[0117] It should be noted that the feature processing during the training of the multimodal sentiment analysis model can refer to the specific implementation process of steps 102 to 106, and will not be repeated here. When optimizing model parameters, this embodiment uses a combination of sentiment main task loss and distillation loss to construct the overall loss function. For example, the sentiment main task loss can use the mean squared error (MSE) loss function to drive the model to learn discriminative semantic information from the comprehensive features. Simultaneously, to enhance intermodal structural alignment and improve feature representation quality, this embodiment designs cross-modal distillation loss for audio and visual modalities. Specifically, the distillation loss includes MSE distillation loss and cosine similarity (COS) distillation loss. MSE distillation loss encourages the modal features extracted by the student Transformer encoder to approach the output of the teacher multilayer perceptron in Euclidean space, while cosine similarity distillation loss strengthens the directional consistency of the modal features of the student Transformer encoder, guiding it to maintain an angle consistent with the semantic distribution output of the teacher multilayer perceptron. The overall loss function includes:

[0118] ;

[0119] In the formula, Represents the overall loss function. This indicates a loss of emotional core. This represents the MSE distillation loss weighting hyperparameter. Indicates MSE distillation loss, This represents the weighted hyperparameter of the cosine similarity distillation loss. This represents the cosine similarity distillation loss. Indicates the training sample index. This represents the total number of training samples. Indicates the first A true value of emotion, Indicates the first One sentiment prediction value, Indicates audio student characteristics. Indicates audio distillation characteristics, Denotes the square of the L2 norm. Indicates the characteristics of students in the video. Indicates video distillation features, Represents the cosine similarity function;

[0120] For example, the weighting hyperparameter for distillation loss can be set as follows: and By appropriately introducing the teacher's structured prior knowledge to assist the student encoder in feature learning, while ensuring that the main task dominates the training, distillation is used as auxiliary supervision to avoid overfitting to the teacher's knowledge.

[0121] To illustrate the technical effects of this embodiment, experiments were conducted on two publicly available multimodal sentiment analysis benchmark datasets, CMU-MOSI and CMU-MOSEI. These two datasets are widely used in multimodal sentiment modeling tasks, covering text, audio, and video modalities, and are highly representative, with realistic scenes and fine-grained sentiment labels.

[0122] The model training in this embodiment was performed on an RTX 4090 GPU with 24GB of video memory, using the CUDA 12.6 and PyTorch 2.7.1 deep learning framework. To ensure high computational efficiency and experimental reproducibility, all training and testing processes were completed within the Ubuntu operating system environment. Specific hyperparameter settings are shown in Table 1.

[0123]

[0124] Since the sentiment labels in both the CMU-MOSI and CMU-MOSEI datasets are continuous sentiment scores, the prediction task is classified as a regression problem. Accordingly, this embodiment uses metrics such as mean absolute error (MAE) and Pearson correlation coefficient (Corr) for performance evaluation. In addition, to further verify the model's discriminative ability in real-world sentiment classification scenarios, this embodiment also introduces supplementary metrics such as binary classification accuracy (Acc-2), seven-class classification accuracy (Acc-7), and weighted F1 score. It can be understood that classification metrics are obtained by processing the sentiment scores output by regression. By using the above regression metrics and their derived classification metrics in combination, the performance and robustness of the proposed model can be systematically evaluated from multiple dimensions, ensuring the comprehensiveness and reliability of multimodal sentiment recognition results.

[0125] 1. Comparative experiment:

[0126] To comprehensively evaluate the effectiveness and advancement of the multimodal sentiment analysis model in this embodiment for multimodal sentiment analysis tasks, this embodiment selects a series of classic methods and cutting-edge models in the multimodal field for comparison. The experimental results are shown in Table 2:

[0127]

[0128] In Table 2, the superscript "a" indicates that the data comes from the relevant literature, and "b" indicates that it is taken from the original paper. The bolded values ​​in the table represent the best performance of each index. According to the experimental results in Table 2, the model in this embodiment outperforms all benchmark methods on the vast majority of indexes, demonstrating excellent comprehensive performance and strong generalization ability.

[0129] Comparison with classic and mainstream methods and tensor fusion methods: Compared with classic tensor fusion methods such as TFN and LMF, the model in this embodiment has achieved significant improvements in core indicators such as binary classification accuracy (Acc-2), F1 score and correlation coefficient (Corr), which indicates that the model has a stronger ability to model higher-order emotional interactions between modalities.

[0130] Compared with Transformer-class methods: Compared with MulT and MAG-BERT, the method in this embodiment retains the advantages of the attention mechanism, and effectively reduces the redundancy and noise caused by over-reliance on the underlying non-textual modal by introducing distillation supervision and collaborative fusion structure. This design achieves a good balance between performance and stability.

[0131] Compared with multi-task learning methods: Compared with recent high-performance multi-task fusion methods such as Self-MM and MMIM, the method in this embodiment achieves the best results in metrics such as F1 and Corr by simply using structured design oriented towards the main task without introducing additional auxiliary tasks. This highlights the efficiency and practicality of the single-task structured fusion strategy in multimodal emotion modeling.

[0132] Compared with the main task enhancement model: Compared with TETFN, TIEMFF and TAMATtention, the method in this embodiment further improves the accuracy of modality alignment and fusion with comparable overall architecture complexity. This proves that even without multi-task supervision, strong generalization performance can be obtained through distillation and structural optimization.

[0133] It is understandable that the feature optimization in this embodiment prioritizes semantic alignment and focuses more on the correctness of sentiment direction (positive or negative polarity) rather than the absolute accuracy of sentiment intensity. Nevertheless, the model in this embodiment consistently matches or surpasses the state-of-the-art methods in core decision metrics such as binary classification accuracy, F1 score, and Pearson correlation coefficient. This "decision-first" design is more in line with the requirements of highly accurate sentiment polarity determination in real-world scenarios and demonstrates stronger generalization robustness.

[0134] 2. Ablation experiment:

[0135] To further verify the effectiveness of each key component in the model of this embodiment, a series of ablation experiments were conducted. By gradually removing specific modules and observing the changes in model performance, the contribution of each component was analyzed. All experiments were run on the CMU-MOSI and CMU-MOSEI datasets, using the same multimodal fusion framework, and the training parameters were kept consistent to ensure the comparability of the results.

[0136] 1) Ablation analysis of key modules

[0137] The ablation components include TQSA (Text Query Selective Cross-Modal Attention), MCG (Modal Collaborative Gating), and TMLP (Teacher Multilayer Perceptron). The experimental results are shown in Tables 3 and 4, where "w / o" indicates that the model does not contain this submodule.

[0138]

[0139]

[0140] Experimental data show that:

[0141] a) Module necessity: Ablation results on two datasets demonstrate that the three components TQSA, MCG, and TMLP are necessary and complementary;

[0142] b) Performance on the MOSI dataset: The full model achieved the best performance across all four metrics. Removing TQSA caused the most significant performance drop, followed by MCG, and then TMLP. This indicates that temporal selection / alignment and modal co-gating are crucial for suppressing noise in non-linguistic modalities.

[0143] c) Performance on the MOSEI dataset: The full model consistently leads in Acc-2 and F1 metrics. Although MAE and Corr show a slight trade-off under certain ablation combinations, this reflects that the current training strategy in this embodiment is more inclined to the correctness of the decision layer than fine-grained continuous value calibration.

[0144] Experimental results verify the powerful tuning and guidance capabilities of this framework: under full multimodal input, non-verbal modalities are dynamically enhanced when information is rich and effectively suppressed when there is a lot of noise, thus achieving robust performance and better cross-modal complementarity;

[0145] 2) Modal combination ablation analysis

[0146] To further evaluate the contribution of each modality to the overall performance and verify the effectiveness of the fusion structure, modal combination ablation experiments were conducted on two datasets. The results shown in Tables 5 and 6 cover the performance comparison of single-modal input (text[t], audio[a], visual[v]), all possible bimodal combinations, and full trimodal fusion:

[0147]

[0148]

[0149] Experimental data show that:

[0150] a) Text dominance: The text modality consistently provides the most stable and significant contribution on both datasets, indicating that emotion recognition in most scenarios still highly depends on linguistic information;

[0151] b) Limitations of non-verbal modalities: In contrast, audio and visual modalities perform weaker when used independently, especially with significant drops in Corr and MAE metrics. This exposes the problems of sparse representation and noise interference in non-verbal modalities, fully demonstrating the necessity of introducing teacher-guided supervision and TQSA / MCG mechanisms into the model to dynamically improve the semantic value of non-verbal modalities.

[0152] c) Improved fusion performance: The trimodal combination consistently achieved the best results, proving that after modality guidance and dynamic fusion processing, the model effectively activated the complementary relationship between modalities. It is worth noting that the improvement brought by multimodal combination is more significant on the MOSEI dataset, mainly due to its larger scale and higher sample diversity, which allows the modality collaboration mechanism to play a full role.

[0153] Modal ablation experiments confirm that the model exhibits strong regulation and guidance capabilities when integrating non-verbal modalities, effectively enhancing cross-modal complementarity. Furthermore, the structure demonstrates robust generalization performance when handling complex samples and large-scale emotional expression scenarios.

[0154] In this embodiment of the invention, teacher-guided distillation enhances the semantic expression of non-textual modal features. Textual modal features are used as reliable query vectors to explicitly filter and temporally align the features of non-textual modal features. Modal collaborative gating is used to dynamically adjust the contribution of each modality to the comprehensive features in order to further eliminate redundant noise between modalities. This invention demonstrates excellent selective control and semantic adaptability in multimodal sentiment analysis tasks and exhibits good accuracy and robustness in complex sentiment recognition scenarios.

[0155] Please see Figure 5 The present invention provides a multimodal sentiment analysis device based on teacher-guided distillation and selective alignment of text queries, comprising:

[0156] Input module 501 is used to input multimodal sentiment data into the trained multimodal sentiment analysis model;

[0157] The extraction module 502 is used to determine the text semantic features, audio semantic features, and video semantic features of multimodal sentiment data based on teacher-guided distillation of non-textual modalities through the feature extraction module.

[0158] The encoding module 503 is used to encode the text semantic features, audio semantic features and video semantic features respectively using a position encoder to generate text encoded features, audio encoded features and video encoded features;

[0159] Enhancement module 504 is used for text query-based selective cross-modal attention module, which guides audio coding features and video coding features to perform semantic alignment through text coding features, and outputs audio enhancement features and video enhancement features;

[0160] The gating module 505 is used to perform gating fusion on text semantic features and audio semantic features, and text semantic features and video semantic features respectively, according to the modal collaborative gating module, to determine the audio gating signal and the video gating signal;

[0161] The integration module 506 is used in the feature fusion module to multiply the audio enhancement features with the audio gating signal and the video enhancement features with the video gating signal element by element and then add them together to construct a comprehensive feature.

[0162] The prediction module 507 is used to transform the comprehensive features through the time pooling layer, input them into the prediction head for sentiment prediction, and output the sentiment prediction value.

[0163] Embodiment 3 of the present invention also provides a computer device, including a memory and a processor, wherein the memory stores a computer program; when the computer program is executed by the processor, the processor performs the steps of the multimodal sentiment analysis method based on teacher-guided distillation and selective alignment of text queries as described in Embodiment 1 of the present invention.

[0164] Embodiment 4 of the present invention also provides a computer-readable storage medium storing a computer program / instruction thereon, which, when executed by a processor, implements the steps of the multimodal sentiment analysis method based on teacher-guided distillation and selective alignment of text queries as described in Embodiment 1 of the present invention.

[0165] Embodiment 5 of the present invention also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the multimodal sentiment analysis method based on teacher-guided distillation and selective alignment of text queries as described in Embodiment 1 of the present invention.

[0166] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the above-described device and module can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0167] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or modules may be electrical, mechanical, or other forms.

[0168] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0169] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0170] If the integrated module is implemented as a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0171] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multimodal sentiment analysis method based on teacher-guided distillation and selective alignment of text queries, characterized in that, include: Input multimodal sentiment data into the trained multimodal sentiment analysis model; The textual semantic features, audio semantic features, and video semantic features of the multimodal emotion data are determined by the feature extraction module based on teacher-guided distillation of non-textual modalities. The text semantic features, audio semantic features, and video semantic features are encoded using a position encoder to generate text encoded features, audio encoded features, and video encoded features respectively; Based on the text query selective cross-modal attention module, the text encoding features guide the audio encoding features and the video encoding features to perform semantic alignment, and output audio enhancement features and video enhancement features respectively; According to the modal collaborative gating module, gating fusion is performed on the text semantic features and the audio semantic features, and on the text semantic features and the video semantic features, respectively, to determine the audio gating signal and the video gating signal; In the feature fusion module, the audio enhancement feature is multiplied element-wise with the audio gating signal, and the video enhancement feature is multiplied element-wise with the video gating signal, and then added together to construct a comprehensive feature; The comprehensive features are transformed through a time pooling layer and then input into a regression prediction head for sentiment prediction, outputting a sentiment prediction value. The multimodal sentiment data includes text sentiment data, audio sentiment data, and video sentiment data; the feature extraction module includes a BERT-like pre-trained language model, a teacher multilayer perceptron, a student Transformer encoder, a linear layer, and a residual adapter; the feature extraction module determines the text semantic features, audio semantic features, and video semantic features of the multimodal sentiment data based on teacher-guided distillation of the non-text modality, including: After extracting features from the text sentiment data using a BERT-like pre-trained language model, the features are projected onto the text semantic features through a linear layer. Based on a cascaded student Transformer encoder and linear layers, feature processing is performed on audio sentiment data and video sentiment data respectively, and corresponding student features for audio and student features for video are determined. By using a cascaded teacher multilayer perceptron and linear layer, feature processing is performed on the audio emotion data and the video emotion data respectively, and corresponding audio distillation features and video distillation features are determined. Using a residual adapter, the audio distillation features and the video distillation features are respectively adapted, and then residually connected with the associated audio student features and video student features to output audio semantic features and video semantic features. Among them, the teacher multilayer perceptron and the residual adapter both include cascaded linear layers, ReLU activation functions and layer normalization layers; The text query-selective cross-modal attention module guides the audio encoding features and video encoding features to perform semantic alignment through the text encoding features, and outputs audio enhancement features and video enhancement features, including: Project the text encoding features into a text query matrix; The audio encoding features are mapped into an audio key matrix and an audio value matrix; The video coding features are mapped into a video key matrix and a video value matrix; The audio attention weight and video attention weight are determined by performing scaling dot product operations on the text query matrix and the audio key matrix, and on the text query matrix and the video key matrix, respectively, followed by Softmax normalization. The audio attention weights and video attention weights are multiplied element-wise to generate enhanced attention weights; The enhanced attention weights are multiplied by the audio value matrix and the video value matrix respectively to output audio enhancement features and video enhancement features.

2. The multimodal sentiment analysis method based on teacher-guided distillation and selective alignment of text queries according to claim 1, characterized in that, The modal collaborative gating module includes a splicing layer, a multilayer perceptron, a Sigmoid activation function, a residual gating fusion layer, an average pooling layer, an L2 normalization layer, and a linear layer; the feature processing of the modal collaborative gating module includes: For any input non-textual semantic feature, the non-textual semantic feature and the textual semantic feature are combined through a concatenation layer to construct a collaborative feature; The collaborative features are processed using a cascaded multilayer perceptron and a sigmoid activation function to generate a gating factor. In the residual gated fusion layer, semantic gated features are constructed by multiplying the non-textual semantic features and the textual semantic features element by element based on the gated factor and then summing them. The semantic gated features are then multiplied element by element with the non-textual semantic features to determine the semantic fusion features. The semantic fusion features are input into a cascaded average pooling layer, L2 normalization layer, and linear layer, and a gating signal is output.

3. The multimodal sentiment analysis method based on teacher-guided distillation and selective alignment of text queries according to claim 1, characterized in that, The process of determining the teacher's multilayer perceptron includes: For any non-text modality, obtain training non-text sentiment data and input it into the multilayer perceptron of the teacher to be trained for feature extraction, and output non-text teacher features; After performing temporal modeling on the non-text teacher features using an LSTM encoder, a temporal pooling layer is used for temporal aggregation to determine the non-text global features. Based on the regression multilayer perceptron, sentiment regression prediction is performed on the non-text global features, and the non-text scalar prediction value is output. The model parameters of the teacher multilayer perceptron are optimized based on the non-text scalar prediction values ​​using the teacher regression loss function until the trained teacher multilayer perceptron is determined as the teacher multilayer perceptron for the multimodal sentiment analysis model.

4. The multimodal sentiment analysis method based on teacher-guided distillation and selective alignment of text queries according to claim 1, characterized in that, The process of determining the trained multimodal sentiment analysis model includes: The training multimodal sentiment data is input into the multimodal sentiment analysis model to be trained. The training audio student features, training video student features, training audio distillation features, and training video distillation features output by the feature extraction module, as well as the training sentiment prediction value output by the regression prediction head, are determined. The model parameters are then jointly iteratively optimized through sentiment master task loss and distillation loss until the trained multimodal sentiment analysis model is determined.

5. A multimodal sentiment analysis device based on teacher-guided distillation and selective alignment of text queries, characterized in that, The apparatus is used to implement the multimodal sentiment analysis method based on teacher-guided distillation and selective alignment of text queries as described in any one of claims 1-4, the apparatus comprising: an input module for inputting multimodal sentiment data into a trained multimodal sentiment analysis model; The extraction module is used to determine the text semantic features, audio semantic features, and video semantic features of the multimodal sentiment data based on teacher-guided distillation of non-textual modalities through the feature extraction module. The encoding module is used to encode the text semantic features, the audio semantic features, and the video semantic features using a position encoder, respectively, to generate text encoded features, audio encoded features, and video encoded features; The enhancement module is used to guide the audio encoding features and the video encoding features to perform semantic alignment based on the text query selective cross-modal attention module, and output audio enhancement features and video enhancement features respectively. The gating module is used to perform gating fusion on the text semantic features and the audio semantic features, and the text semantic features and the video semantic features, respectively, according to the modal collaborative gating module, to determine the audio gating signal and the video gating signal; The integration module is used in the feature fusion module to multiply the audio enhancement feature with the audio gating signal and the video enhancement feature with the video gating signal element by element and then add them together to construct a comprehensive feature; The prediction module is used to transform the comprehensive features through a time pooling layer, input them into a regression prediction head for sentiment prediction, and output the sentiment prediction value.

6. A computer device, characterized in that, The system includes a memory and a processor, wherein the memory stores a computer program that, when executed by the processor, causes the processor to perform the steps of the multimodal sentiment analysis method based on teacher-guided distillation and selective alignment of text queries as described in any one of claims 1-4.

7. A computer-readable storage medium having a computer program / instructions stored thereon, characterized in that, When the computer program / instruction is executed by the processor, it implements the steps of the multimodal sentiment analysis method based on teacher-guided distillation and selective alignment of text queries as described in any one of claims 1-4.

8. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instruction is executed by the processor, it implements the steps of the multimodal sentiment analysis method based on teacher-guided distillation and selective alignment of text queries as described in any one of claims 1-4.

Images