Multimodal sentiment analysis method based on sentiment consistency distillation and information bottleneck

Abstract

The application discloses a multimodal sentiment analysis method based on sentiment consistency distillation and information bottleneck, which comprises the following steps: respectively extracting sentiment features of image, text and audio modalities, and mapping the multimodal sentiment features into multimodal semantic vectors; then, decoupling the multimodal semantic vectors into modality-specific semantic vectors and modality-common semantic vectors; extracting sentiment consistency signals between modalities through an attention mechanism, and guiding the sentiment correlation information to be fused into the modality-common semantic vectors through a knowledge distillation mechanism; removing noise information in the modality-specific semantic vectors based on the information bottleneck theory; finally, splicing the optimized modality-specific semantic vectors and the modality-common semantic vectors into unified sentiment representation, and outputting corresponding sentiment categories through a classifier. The application can effectively alleviate the alignment deviation and information pollution problems caused by noise interference in multimodal data, improve the sentiment recognition accuracy, and take into account the calculation efficiency and model stability.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, particularly the interdisciplinary technology of natural language processing and sentiment computing, and specifically to a multimodal sentiment analysis method and apparatus based on sentiment consistency distillation and information bottleneck. Background Technology

[0002] Emotions are a natural reflection of human inner psychological states, typically conveyed through various means such as tone of voice, facial expressions, and gestures. Sentiment analysis technology aims to identify the emotional states exhibited by individuals in specific situations through data modeling and intelligent algorithms, and is one of the essential foundational capabilities for achieving natural human-computer interaction. With the development of multimodal technologies such as natural language processing, computer vision, and speech recognition, sentiment analysis research has gradually evolved from single-modal analysis to multimodal sentiment analysis. Multimodal sentiment analysis integrates information from text, audio, and visual modalities, achieving a more comprehensive and accurate understanding of emotions through deep learning models. Currently, this technology has been widely applied in various scenarios such as public opinion analysis, educational feedback systems, medical emotion monitoring, and social robots, and has broad research and commercialization prospects.

[0003] Despite significant progress in multimodal sentiment analysis, numerous challenges remain in practical applications, primarily including: First, high-quality sentiment-annotated data is scarce and costly. Emotions are subjective, and different annotators may disagree on the same sentiment expression, making multimodal data annotation more difficult and inconsistent, impacting model training performance. Second, there are issues of expression differences and information redundancy between modalities. Each modality expresses emotions differently, exhibiting some distribution inconsistencies. Furthermore, redundant features exist between some modalities, increasing the model's burden and reducing the efficiency and discriminative power of feature fusion. Third, multimodal feature extraction and fusion strategies are not yet standardized. Different modalities typically rely on networks with different structures for feature extraction, leading to difficulties in modal alignment and semantic inconsistencies during the fusion stage. Effectively preserving modal advantages and avoiding information loss during deep fusion is a current research focus. Finally, the robustness of models in complex environments remains limited. In real-world scenarios, factors such as audio noise, image occlusion, and incomplete text can all degrade the quality of a particular modality, thus affecting the overall model's recognition accuracy. Therefore, enhancing the model's ability to adapt to noise and outlier data is a key aspect of improving its deployability.

[0004] Current research has attempted to use feature decoupling methods to divide the original multimodal features into modality common features and modality specific features in order to improve representation ability and information organization structure. However, two core problems remain unresolved: (1) When a modality has significant noise interference, such as image occlusion or speech distortion, it will seriously affect the alignment process between modalities, thus making it impossible for modality common features to accurately model the shared emotional information of different modalities; (2) Modality specific features generally contain a large amount of background noise or structural redundancy that is irrelevant to emotion judgment. If these are not removed, they will weaken the discriminative ability of the features, thereby affecting the final emotion classification result.

[0005] To address the problems existing in the prior art, this invention proposes a multimodal sentiment analysis method based on sentiment consistency distillation and information bottleneck, in order to effectively overcome the shortcomings of existing technical solutions. Summary of the Invention

[0006] In view of this, this invention provides a multimodal sentiment analysis method based on sentiment consistency distillation and information bottleneck, aiming to overcome the shortcomings of existing multimodal sentiment analysis techniques in terms of noise interference, feature alignment, and information redundancy. This method can stably extract effective sentiment expression information in complex environments, improve the alignment accuracy and feature discrimination ability between modalities, and achieve accurate identification of the emotional state of people in multimodal sentiment videos. At the same time, through decoupling and optimization mechanisms, this invention enables the model to have higher inference speed in the actual deployment and testing phases, significantly improving the overall system performance.

[0007] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0008] In a first aspect, the present invention provides a multimodal sentiment analysis method based on sentiment consistency distillation and information bottleneck, which mainly includes the following steps:

[0009] S1. Extract emotional features of each modality from multimodal emotional video data containing images, audio, and text to obtain multimodal emotional features;

[0010] S2. The multimodal emotion feature mapping is transformed into a multimodal semantic vector with consistent dimensions;

[0011] S3. Decouple the multimodal semantic vector into a modal common semantic vector and a modal specific semantic vector;

[0012] S4. Extract the sentiment collaboration information between modal pairs and transfer this information to the modal common semantic vector through knowledge distillation;

[0013] S5. Remove task-irrelevant interference information from modality-specific semantic vectors using information bottleneck theory;

[0014] S6. The optimized modality-specific semantic vector and modality common semantic vector are concatenated to obtain the final semantic vector. The final sentiment classification result is output using linear mapping and SoftMax.

[0015] Furthermore, in S1, the emotion feature extraction process includes:

[0016] For image modalities, feature extraction was performed using the Openface tool;

[0017] For text modalities, text information is obtained through the Whisper model and then converted into word vectors using the BERT model;

[0018] For audio modalities, acoustic features were extracted using the COVAREP toolset.

[0019] Furthermore, in S2, the method for converting multimodal features into multimodal semantic vectors is as follows:

[0020] The dimensions of each modality feature input to the one-dimensional convolutional layer are unified to achieve the aligned output of the three modalities in the semantic space, and a multimodal semantic vector with fused input is constructed.

[0021] Furthermore, in S3, the feature decoupling mapping module is used to decouple the multimodal semantic vector into a modal common semantic vector and a modal specific semantic vector; the feature decoupling mapping module includes a common encoder and multiple private encoders to extract the modal common semantic vector and the modal specific semantic vector respectively.

[0022] Furthermore, in S4, sentiment collaboration information between modal pairs is extracted through a cross-modal Transformer encoder, and this information is transferred to modal common semantic features through knowledge distillation; the specific process includes:

[0023] (1) Cross-modal collaborative feature extraction:

[0024] For modality-specific semantic vectors between different modalities, a cross-modal Transformer encoder is used to extract collaborative features between any two modalities i,j∈{L,V,A}, where L,V,A represent text, image, and audio modalities, respectively. Let the feature vector of modality i be the query, and the feature vector of modality j be the key and value. Then the output of the cross-modal Transformer encoder is:

[0025] X ij =Co-Transformer(X) i ,X j ,X j )

[0026] Among them, X ijX represents the collaborative features extracted between mode i and mode j; Co-Transformer represents the cross-modal Transformer encoder, X i ,X j Let X represent the original feature vectors of mode i and mode j, respectively, where X... i As a query, X j As keys and values;

[0027] (2) Distillation of knowledge about emotional synergistic features:

[0028] Using the modality pair collaborative features as source nodes, the modality common semantic vector... As the target node, knowledge distillation is performed; These represent the modal common semantic vectors for text, image, and audio modes, respectively; the distillation error is:

[0029] ε ij→c =||logits(X) ij )-logits(X c )||

[0030] And introduce the distillation intensity weight ω ij→c The weighted distillation loss is:

[0031]

[0032] Where logits(X) represents the logits output of feature X after mapping through a fully connected network; ω ij→c Represents the modality pair feature X ij For modal common semantic vector X c The importance of migration;

[0033] (3) Distillation edge weight learning mechanism:

[0034] By introducing an edge weight calculation function, the contribution levels of collaborative features from different modalities are dynamically learned:

[0035] ω ij→c =g([f(X) ij ,θ1),X ij ],[f(X c ,θ1),X c ],θ2)

[0036] Where f(·,θ1) represents a fully connected neural network that maps features to logits vectors; g(·,θ2) represents a weighted computation network that fuses the original features and logits; [·,·] represents the feature concatenation operation, and θ1 and θ2 are trainable parameters.

[0037] Furthermore, in S5, task-irrelevant interfering information in modality-specific semantic vectors is removed using the information bottleneck theory. The specific process includes:

[0038] (1) Constructing information bottleneck constraint objectives:

[0039] To simultaneously retain feature information relevant to sentiment prediction and compress redundant noise in the input, a loss function based on the information bottleneck theory is constructed as follows:

[0040]

[0041] Where I(Z;Y) represents the mutual information between variable Z and sentiment label Y; I(Z;X) represents the mutual information between variable Z and input feature X; β represents the coefficient that controls the trade-off between compression and preservation; and Z represents the latent sentiment expression after denoising, i.e., the latent space variable.

[0042] (2) Latent variable generation mechanism:

[0043] The design incorporates a reparameterized encoder structure, encoding modality-specific semantic vectors into latent space variables. The calculation formula is as follows:

[0044]

[0045] Where μ represents the mean, σ 2 Indicates variance; and These represent the mean and variance extraction networks, respectively. represents a modality-specific semantic vector; ∈ represents a standard normally distributed noise variable, and Z represents the denoised potential sentiment expression;

[0046] (3) Distribution constraint mechanism:

[0047] We construct the KL divergence as a regularization term to make the distribution of the latent space variable Z in the encoder output approximate a standard normal distribution. The calculation formula is as follows:

[0048]

[0049] in, Denotes the KL divergence loss function. Let p(Z) represent the posterior distribution of the encoder output, and let D represent the prior distribution of the variable Z. KL This represents the KL divergence between the posterior and prior distributions, measuring the difference between their distributions.

[0050] (4) Feature reconstruction and optimization:

[0051] Introducing decoder f dec Reconstruct variable Z and optimize the encoder. The representational ability of the output reconstructed features

[0052]

[0053] By comparing the mean squared errors between the original features and the reconstructed features, the reconstruction loss is defined as:

[0054]

[0055] Among them, ||·|| 2 Mean squared error loss:

[0056] (5) Emotional classification task guidance:

[0057] Introducing classifier f cls (·) Sentiment prediction for Z:

[0058]

[0059] Its cross-entropy loss is:

[0060]

[0061] in, y represents the sentiment prediction result; y represents the true sentiment label.

[0062] (6) Construct a composite loss function for denoising:

[0063] L dno =L rec +λ1L KL +λ2L cls

[0064] Among them, L dno L represents the final denoising loss function; rec L represents the reconstruction loss; KL L represents the KL divergence loss; cls λ represents the cross-entropy loss; λ1 and λ2 are the importance coefficients of the corresponding losses, respectively.

[0065] Furthermore, in S6, the optimized modal common semantic vector and modal specific semantic vector are concatenated to obtain the complete fused semantic vector. The calculation formula is as follows:

[0066]

[0067] Among them, X fusion Represents the fused semantic vector; X c Represents the modal common semantic vector; Represents a modality-specific semantic vector;

[0068] The concatenated fused semantic vector is input into a fully connected neural network for non-linear interaction and weighted fusion between features. The calculation formula is as follows:

[0069] X final =f(X) fusion )

[0070] Where f(·) represents a multilayer perceptron with enhanced expression capabilities;

[0071] The final sentiment classification output is calculated using the SoftMax function, whose expression is:

[0072]

[0073] in, This represents the probability distribution of the sentiment category predicted by the model; U and b are the learnable parameters of the fully connected classifier.

[0074] Secondly, the present invention also provides a multimodal sentiment analysis device based on sentiment consistency distillation and information bottleneck, which applies the above-mentioned multimodal sentiment analysis method based on sentiment consistency distillation and information bottleneck to perform multimodal sentiment analysis. The device includes:

[0075] Multimodal feature extraction module: This module receives input multimodal emotional data video and extracts features from image, audio, and text modalities to obtain multimodal emotional features.

[0076] Semantic vector construction module: used to uniformly map multimodal sentiment features into multimodal semantic vectors in a low-dimensional embedding space;

[0077] Feature decoupling mapping module: Based on a neural network structure, it decomposes multimodal semantic vectors into modality-specific semantic vectors and modality-common semantic vectors;

[0078] Modal Pair Sentiment Consistency Distillation Module: Extracts sentiment collaboration information between modal pairs through a cross-modal attention mechanism, and transfers this information to the modal common semantic vector using a knowledge distillation method;

[0079] Modality-Specific Information Optimization Module: Introducing the information bottleneck principle, the module uses an encoder-decoder structure and KL divergence regularization constraints to compress and denoise modality-specific semantic vectors;

[0080] Multimodal sentiment classification module: It concatenates the optimized modality-specific semantic vector with the modality common semantic vector to form a fused semantic representation, inputs it into the linear mapping layer and processes it through the SoftMax function, and outputs the final sentiment classification label.

[0081] Thirdly, the present invention also provides an electronic device, including a processor and a memory, wherein the memory stores machine-executable instructions that can be executed by the processor, and the processor executes the machine-executable instructions to implement the above-described multimodal sentiment analysis method based on sentiment consistency distillation and information bottleneck.

[0082] Compared with existing technologies, the present invention provides a multimodal sentiment analysis method and apparatus based on sentiment consistency distillation and information bottleneck, which has at least the following beneficial technical effects:

[0083] This invention can stably extract effective emotional expression information in complex environments, improve intermodal alignment accuracy and feature discrimination ability, and achieve accurate identification of the emotional state of characters in multimodal emotional videos. At the same time, this invention emphasizes computational efficiency, improving the actual inference speed of the model and the overall system performance through decoupling and optimization.

[0084] Other features and advantages of the invention will be set forth in the following description, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention can be realized and obtained by means of the structures particularly pointed out in the written description and the accompanying drawings.

[0085] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0086] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0087] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used together with the embodiments of the invention to explain the invention and do not constitute a limitation thereof.

[0088] Figure 1 This is a flowchart illustrating a multimodal sentiment analysis method based on sentiment consistency distillation and information bottleneck provided by the present invention.

[0089] Figure 2 This is a schematic diagram of the network model and working principle used in this invention;

[0090] Figure 3 This is a schematic diagram of the structural composition of the multimodal sentiment analysis method device provided in this embodiment of the invention;

[0091] Figure 4 This is a schematic diagram of the electronic device structure provided in an embodiment of the present invention. Detailed Implementation

[0092] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are some embodiments of the present invention, but not all embodiments.

[0093] In the description of this invention, it should be noted that some processes described in this application specification and drawings include multiple operations that appear in a specific order. However, it should be clearly understood that these operations may be performed in any order or in parallel. Furthermore, various numbers are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0094] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0095] Example 1:

[0096] See Figure 1 and Figure 2 As shown in the figure, this invention provides a multimodal sentiment analysis method based on sentiment consistency distillation and information bottleneck. The method mainly includes the following steps:

[0097] S1: Receives input multimodal emotional video data and uses deep learning neural networks to extract emotional features from image, text, and audio modalities;

[0098] In the specific implementation of this invention, multimodal emotional video data from a video acquisition device is received, and modal separation and feature extraction processing are performed on the three types of information: audio, image, and speech / text. This includes: In the text modal processing, the speech information is first transcribed into text content using the OpenAI-whisper speech recognition model, and then the text is encoded into a 768-dimensional semantic representation using the BERT model. This BERT model employs a 12-layer Transformer encoder structure and has 12 multi-head attention mechanisms. For the image modality, emotion-related facial action units (AUs) are extracted using the OpenFace toolkit, forming a set of 35-dimensional visual feature representations, which are further encoded into fixed-dimensional visual vectors. Audio modal features are extracted using the COVAREP toolset, including acoustic parameters closely related to emotional expression such as acoustic frequency, pitch, VUV, and guttural signals, constituting a total of 74-dimensional audio features.

[0099] In practice, the collected video data can be segmented and the audio separated, and then sent to the three modules mentioned above to complete the feature extraction of text, image and audio modalities, which serve as the input basis for subsequent unified semantic modeling.

[0100] S2: Input the extracted original modal features into the unified dimension mapping module to obtain multimodal semantic vectors with consistent dimensions;

[0101] In the specific implementation of this invention, the extracted original modal features are input into a unified dimension mapping module to obtain multimodal semantic vectors with consistent dimensions. This includes applying one-dimensional convolution to each of the three modal features to adjust their dimensions to a unified fixed vector form. The kernel size of the one-dimensional convolutional layer used is 5×5, the stride is 1, and the number of channels is set to 50. That is, after the three types of modal features are mapped by the one-dimensional convolutional structure, their feature dimensions are uniformly aligned, ultimately generating a set of semantic vector representations that can be uniformly processed by subsequent modules.

[0102] S3: By using a feature decoupling mapping module with a common encoder and multiple private encoders, the multimodal semantic vector is decoupled into a modal common semantic vector and a modal specific semantic vector;

[0103] In a specific implementation of this invention, a feature decoupling mapping module comprising a common encoder and multiple private encoders is used to decouple multimodal semantic vectors into modal common semantic vectors and modal specific semantic vectors, including:

[0104] (1) The public encoder has a kernel size of 1×1, a stride of 1, and 50 kernels; the private encoder has a kernel size of 1×1, a stride of 1, and 50 kernels.

[0105] The expression for decoupling is:

[0106]

[0107] Where s∈{L,V,A} represent text, visual (image), and audio modalities, respectively. For modal common features, For modality-specific features, E com It is a public encoder. It is a proprietary encoder, X s It is a multimodal feature.

[0108] (2) Autoregressive reconstruction of the decoupling features of the same mode is performed using the decoder to obtain the coupling features. The decoder is represented as D. s Its convolution kernel size is 1×1, stride is 1, and number of convolution kernels is 50.

[0109] (3) Constrain the decoupling features using a loss function:

[0110] The reconstruction loss can be formalized as follows:

[0111]

[0112] The regression loss can be formalized as follows:

[0113]

[0114] The boundary loss can be formalized as follows:

[0115]

[0116] Orthogonal loss can be formalized as:

[0117]

[0118] In the formula, α is a boundary hyperparameter used to limit the minimum separable distance between different sentiment categories. The set of triples S = {(i,j,k)|s[i]≠s[j],s[i]=s[k],c[i]=c[j],c[i]≠c[k]}, where s[i], s[j], and s[k] are the modalities of samples i, j, and k, respectively, and c[i], c[j], and c[k] are the category labels of samples i, j, and k, respectively.

[0119] S4: Extract the sentiment consistency signal between modal pairs through an attention mechanism, and transfer this information to the modal common semantic vector through a knowledge distillation mechanism;

[0120] In the specific implementation of this invention, an attention mechanism is used to extract the sentiment consistency signal (sentiment collaboration information) between modal pairs, and this information is transferred to the modal common semantic vector through a knowledge distillation mechanism, including:

[0121] (1) Cross-modal collaborative feature extraction:

[0122] For modality-specific semantic vectors across different modalities, a cross-modal Transformer encoder is used to extract collaborative features between any two modalities i,j∈{L,V,A}, where L,V,A represent text, visual, and audio modalities, respectively. Let the feature vector of modality i be the query, and the feature vector of modality j be the key and value. Then the output of the cross-modal Transformer encoder is:

[0123] X ij =Co-Transformer(X) i ,X j ,X j )

[0124] Among them, X ij This represents the collaborative features extracted between modality i and modality j, used to model the sentiment relevance between modalities. The single-head attention mechanism in Transformer is calculated as follows:

[0125]

[0126] Where Q = X i W Q K = X j W K V = X j W V , respectively, represent the linear projection of the query, key, and value.

[0127] (2) Distillation of knowledge about emotional synergistic features:

[0128] Using the above modality pair collaborative features as source nodes, the modality common semantic vector... As the target node, knowledge distillation is performed. Let the distillation error be:

[0129] ε ij→c =||logits(X) ij )-logits(X c )||

[0130] And introduce the distillation intensity weight ω ij→c The weighted distillation loss is:

[0131]

[0132] Where logits(X) represents the logits output of feature X after mapping through a fully connected network, ω ij→c Represents the modality pair feature X ij For modal common semantic vector X c The importance of transfer of (modal common features).

[0133] (3) Distillation edge weight learning mechanism:

[0134] To dynamically learn the contribution levels of collaborative features from different modalities, this embodiment introduces an edge weight calculation function:

[0135] ω ij→c =g([f(X) ij ,θ1),X ij ],[f(X c ,θ1),X c ],θ2)

[0136] Where f(·,θ1) represents a fully connected neural network that maps input features to logits vectors, g(·,θ2) represents a weighted computation network that fuses the original features and logits, [·,·] represents the feature concatenation operation, θ1 and θ2 are trainable parameters, and X ij This represents the collaborative sentiment features of modal pairs. After multiple iterations, the mechanism can construct a complete distilled edge weight matrix W, which is used to measure the contribution of collaborative information from different modalities to the common modal features.

[0137] (4) Distillation loss function:

[0138] The final distillation loss function is defined as:

[0139]

[0140] Where W is the distillation edge weight matrix, with the same dimension as the modality logarithm, E is the distillation error matrix between each modality pair and the modality common feature, ⊙ represents the element-wise multiplication of corresponding elements, and ||·|| represents the matrix norm, which is used to measure the overall distillation error.

[0141] S5: Input the modality-specific semantic vector into the modality-specific information optimization module, and remove task-irrelevant interference information through the information bottleneck theory;

[0142] In the specific implementation of this invention, the modality-specific semantic vector is input into the modality-specific information optimization module, and task-irrelevant interference information is removed through information bottleneck theory, including:

[0143] (1) Information bottleneck constraint target construction:

[0144] To simultaneously retain feature information relevant to sentiment prediction and compress redundant noise in the input, this invention constructs a loss function based on the information bottleneck theory as follows:

[0145]

[0146] Where I(Z;Y) represents the mutual information between the latent variable Z and the sentiment label Y, which is used to preserve sentiment information; I(Z;X) represents the mutual information between the latent variable and the input feature X, which reflects the degree of information compression; β is a coefficient that controls the trade-off between compression and preservation; and Z is the latent sentiment expression after denoising, i.e., the latent space variable.

[0147] (2) Latent variable generation mechanism:

[0148] To achieve the aforementioned theoretical goals, an encoder structure based on reparameterization techniques is designed to process modality-specific semantic vectors. The encoding is a latent space variable Z, specifically:

[0149]

[0150] in, and Let Z represent the mean and variance extraction networks, respectively. Let ∈ be the standard normally distributed noise variable. A differentiable sampling process is introduced, and Z represents the final latent space feature, which serves as the sentiment compression representation.

[0151] (3) Distribution constraint mechanism

[0152] To make the latent variable Z distribution of the encoder output approximate the standard normal distribution, KL divergence is constructed as a regularization term, defined as follows:

[0153]

[0154] in, The posterior distribution of the encoder output. As a prior distribution, the KL divergence is used to constrain the distribution space of Z, prevent overfitting, and suppress irrelevant information.

[0155] (4) Feature reconstruction and optimization:

[0156] To further optimize the encoder The representational capability is introduced by the decoder f. dec Reconstruct the latent variable Z and output the reconstructed features.

[0157]

[0158] By comparing the mean squared errors between the original features and the reconstructed features, the reconstruction loss is defined as follows:

[0159]

[0160] Among them, ||·|| 2 This represents the mean square error loss.

[0161] (5) Emotional classification task guidance:

[0162] To ensure that the latent variable Z can be effectively used for sentiment discrimination, a classifier f is introduced. cls (·) Sentiment prediction for Z:

[0163]

[0164] Its cross-entropy loss is:

[0165]

[0166] in, The result represents the sentiment prediction, and y represents the true sentiment label. This classification task further enhances the discriminative constraint of Z on the sentiment task.

[0167] (6) Denoising loss function:

[0168] The final denoising loss function is composed of the above three loss terms:

[0169] L dno =L rec +λ1L KL +λ2L cls

[0170] Where λ1 and λ2 are the importance coefficients for balancing the subtasks.

[0171] S6: The optimized modality-specific semantic vector and modality common semantic vector are concatenated to obtain the final semantic vector. The final sentiment classification result is output using linear mapping and SoftMax.

[0172] In the specific implementation of this invention, the final semantic vector is obtained by concatenating the optimized modality-specific semantic vector and the modality common semantic vector. The final sentiment classification result is then output using linear mapping and SoftMax, including:

[0173] The optimized modal common semantic vector X c With modality-specific semantic vectors Feature concatenation is performed to obtain the complete fused semantic vector X. fusion :

[0174]

[0175] The concatenated fusion vector is then input into a fully connected neural network to achieve non-linear interaction and weighted fusion between features.

[0176] X final =f(X) fusion )

[0177] Here, f(·) represents a multilayer perceptron structure with enhanced expression capabilities, used to improve the discriminative power of fused features. The final sentiment classification output is calculated using the SoftMax function, whose expression is:

[0178]

[0179] Where U and b are the learnable parameters of the fully connected classifier. This represents the probability distribution of the sentiment category predicted by the model, and the final sentiment label is... The category index corresponding to the highest probability.

[0180] As described in the above embodiments, those skilled in the art will understand that the present invention provides a multimodal sentiment analysis method based on sentiment consistency distillation and information bottleneck. The method includes: acquiring input multimodal sentiment video data; extracting sentiment features of image, text, and audio modalities using a deep learning neural network model, and mapping the extracted multimodal sentiment features to multimodal semantic vectors; subsequently, decoupling the multimodal semantic vectors into modality-specific semantic vectors and modality-common semantic vectors; extracting sentiment consistency signals between modality pairs through an attention mechanism, and guiding the fusion of this sentiment association information into the modality-common semantic vector through a knowledge distillation mechanism to enhance its robustness and semantic alignment effect under noisy conditions; removing noise information from the modality-specific semantic vectors based on information bottleneck theory to improve the discriminativeness of specific features; and finally concatenating the optimized modality-specific semantic vectors and modality-common semantic vectors into a unified sentiment representation, and outputting the corresponding sentiment category through a classifier. This invention effectively mitigates alignment bias and information contamination caused by noise interference in multimodal data; it can stably extract effective emotional expression information in complex environments, improve intermodal alignment accuracy and feature discrimination ability, and achieve accurate identification of emotional states of individuals in multimodal emotional videos. Simultaneously, this invention emphasizes computational efficiency; through module decoupling and optimization, it improves emotion recognition accuracy while maintaining computational efficiency and model stability, thereby enhancing the model's actual inference speed and overall system performance.

[0181] Example 2:

[0182] like Figure 3 As shown, based on the above method, the present invention also provides a multimodal sentiment analysis device based on sentiment consistency distillation and information bottleneck mechanism. This device mainly includes:

[0183] Modal feature extraction module: Receives input multimodal emotional data video, extracts features from image, text and audio modalities respectively, and obtains original multimodal emotional features;

[0184] Semantic vector construction module: maps multimodal features into multimodal semantic vectors in a low-dimensional embedding space;

[0185] Feature decoupling module: Based on a neural network structure, it decomposes multimodal semantic vectors into modality-specific semantic vectors and modality-common semantic vectors;

[0186] Modality-Specific Information Optimization Module: Introducing the information bottleneck principle, the module uses an encoder-decoder structure and KL divergence regularization constraints to compress and denoise modality-specific semantic vectors;

[0187] Modal Pair Sentiment Consistency Distillation Module: This module utilizes a cross-modal attention mechanism to extract collaborative sentiment information between modal pairs and employs knowledge distillation to transfer this information into the modal common semantic vector.

[0188] The multimodal sentiment classification output module concatenates the optimized modality-specific semantic vectors with the modality-common semantic vectors to form a fused semantic representation, which is then input into a linear mapping layer and processed by the SoftMax function to output the final sentiment classification label.

[0189] The multimodal sentiment analysis device based on sentiment consistency distillation and information bottleneck provided in this embodiment of the invention has the same implementation principle and technical effect as the aforementioned method embodiment. For the sake of brevity, any parts not mentioned in this embodiment can be referred to the corresponding content in the aforementioned method embodiment, and will not be repeated here.

[0190] Example 3:

[0191] Reference Figure 4 As shown, this embodiment of the invention also provides an electronic device, which may include a processor, a memory, a communication bus and a communication interface, and may also include a computer program stored in the memory and executable on the processor. The processor executes the computer program to implement the multimodal sentiment analysis method based on sentiment consistency distillation and information bottleneck in the above method embodiments.

[0192] In some embodiments, the processor may be composed of integrated circuits, such as a single packaged integrated circuit or multiple integrated circuits packaged with the same or different functions. This includes combinations of one or more central processing units, microprocessors, digital processing chips, graphics processors, and various control chips. The processor is the control core of the electronic device, connecting various components of the device through various interfaces and lines. It executes programs or modules stored in memory and calls data stored in the memory to perform various functions and process data within the electronic device.

[0193] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, apparatus, electronic devices, or computer program products, etc. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0194] It should be noted that the word "comprising" does not exclude the presence of components or steps not listed in the claims. The words "a" or "an" preceding a component do not exclude the presence of a plurality of such components. This invention can be implemented by means of hardware comprising several different components and by means of a suitably programmed computer.

[0195] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0196] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A multimodal sentiment analysis method based on sentiment consistency distillation and information bottleneck, characterized in that, The method includes the following steps: S1. Extract emotional features of each modality from multimodal emotional video data containing images, audio, and text to obtain multimodal emotional features; S2. The multimodal emotion feature mapping is transformed into a multimodal semantic vector with consistent dimensions; S3. Decouple the multimodal semantic vector into a modal common semantic vector and a modal specific semantic vector; S4. Extract the sentiment collaboration information between modal pairs and transfer this information to the modal common semantic vector through knowledge distillation; S5. Remove task-irrelevant interference information from modality-specific semantic vectors using information bottleneck theory; S6. The optimized modality-specific semantic vector and modality common semantic vector are concatenated to obtain the final semantic vector. The final sentiment classification result is output using linear mapping and SoftMax. In S5, task-irrelevant interfering information in modality-specific semantic vectors is removed using the information bottleneck theory. The specific process includes: (1) Constructing information bottleneck constraint objectives: To simultaneously retain feature information relevant to sentiment prediction and compress redundant noise in the input, a loss function based on the information bottleneck theory is constructed as follows: Where I(Z;Y) represents the mutual information between variable Z and sentiment label Y; I(Z;X) represents the mutual information between variable Z and input feature X; β represents the coefficient that controls the trade-off between compression and preservation; and Z represents the latent sentiment expression after denoising, i.e., the latent space variable. (2) Latent variable generation mechanism: The design incorporates a reparameterized encoder structure, encoding modality-specific semantic vectors into latent space variables. The calculation formula is as follows: Where μ represents the mean, σ 2 Indicates variance; and These represent the mean and variance extraction networks, respectively. represents a modality-specific semantic vector; ∈ represents a standard normally distributed noise variable, and Z represents the denoised potential sentiment expression; (3) Distribution constraint mechanism: We construct the KL divergence as a regularization term to make the Z-distribution of the encoder output variable approximate the standard normal distribution. The calculation formula is as follows: in, Denotes the KL divergence loss function. Let p(Z) represent the posterior distribution of the encoder output, and let D represent the prior distribution of the variable Z. KL This represents the KL divergence between the posterior and prior distributions, measuring the difference between their distributions. (4) Feature reconstruction and optimization: Introducing decoder f dec Reconstruct variable Z and optimize the encoder. The representational ability of the output reconstructed features By comparing the mean squared errors between the original features and the reconstructed features, the reconstruction loss is defined as: Among them, ||·|| 2 Mean squared error loss: (5) Emotional classification task guidance: Introducing classifier f cls (·) Sentiment prediction for Z: Its cross-entropy loss is: in, y represents the sentiment prediction result; y represents the true sentiment label. (6) Construct a composite loss function for denoising: L dno =L rec +λ1L KL +λ2L cls Among them, L dno L represents the final denoising loss function; rec L represents the reconstruction loss; KL L represents the KL divergence loss; cls λ represents the cross-entropy loss; λ1 and λ2 are the importance coefficients of the corresponding losses, respectively.

2. The multimodal sentiment analysis method based on sentiment consistency distillation and information bottleneck as described in claim 1, characterized in that, In S1, the emotional feature extraction process includes: For image modalities, feature extraction was performed using the Openface tool; For text modalities, text information is obtained through the Whisper model and then converted into word vectors using the BERT model; For audio modalities, acoustic features were extracted using the COVAREP toolset.

3. The multimodal sentiment analysis method based on sentiment consistency distillation and information bottleneck as described in claim 1, characterized in that, In S2, the method for converting multimodal features into multimodal semantic vectors is as follows: The dimensions of each modality feature input to the one-dimensional convolutional layer are unified to achieve the aligned output of the three modalities in the semantic space, and a multimodal semantic vector with fused input is constructed.

4. The multimodal sentiment analysis method based on sentiment consistency distillation and information bottleneck as described in claim 1, characterized in that, In S3, the feature decoupling mapping module is used to decouple the multimodal semantic vector into a modal common semantic vector and a modal specific semantic vector. The feature decoupling mapping module includes a common encoder and multiple private encoders to extract the modal common semantic vector and the modal specific semantic vector respectively.

5. The multimodal sentiment analysis method based on sentiment consistency distillation and information bottleneck as described in claim 1, characterized in that, In S4, sentiment collaboration information between modal pairs is extracted through a cross-modal Transformer encoder, and this information is transferred to modal common semantic features through knowledge distillation. The specific process includes: (1) Cross-modal collaborative feature extraction: For modality-specific semantic vectors between different modalities, a cross-modal Transformer encoder is used to extract collaborative features between any two modalities i,j∈{L,V,A}, where L,V,A represent text, image, and audio modalities, respectively. Let the feature vector of modality i be the query, and the feature vector of modality j be the key and value. Then the output of the cross-modal Transformer encoder is: X ij =Co-Transformer(X i ,X j ,X j ) Among them, X ij X represents the collaborative features extracted between mode i and mode j; Co-Transformer represents the cross-modal Transformer encoder, X i ,X j Let X represent the original feature vectors of mode i and mode j, respectively, where X... i As a query, X j As keys and values; (2) Distillation of knowledge about emotional synergistic features: Using the modality pair collaborative features as source nodes, the modality common semantic vector... As the target node, knowledge distillation is performed; These represent the modal common semantic vectors for text, image, and audio modes, respectively; the distillation error is: ε ij→c =||logits(X ij )-logits(X c )|| And introduce the distillation intensity weight ω ij→c The weighted distillation loss is: Where logits(X) represents the logits output of feature X after mapping through a fully connected network; ω ij→c Represents the modality pair feature X ij For modal common semantic vector X c The importance of migration; (3) Distillation edge weight learning mechanism: By introducing an edge weight calculation function, the contribution levels of collaborative features from different modalities are dynamically learned: oh ij→c =g([f(X ij ,θ1),X ij ],[f(X c ,θ1),X c ],θ2) Where f(·,θ1) represents a fully connected neural network that maps features to logits vectors; g(·,θ2) represents a weighted computation network that fuses the original features and logits; [·,·] represents the feature concatenation operation, and θ1 and θ2 are trainable parameters.

6. The multimodal sentiment analysis method based on sentiment consistency distillation and information bottleneck as described in claim 1, characterized in that, In S6, the optimized modal common semantic vector and modal specific semantic vector are concatenated to obtain the complete fused semantic vector. The calculation formula is as follows: Among them, X fusion Represents the fused semantic vector; X c Represents the modal common semantic vector; Represents a modality-specific semantic vector; The concatenated fused semantic vector is input into a fully connected neural network for non-linear interaction and weighted fusion between features. The calculation formula is as follows: X final =f(X fusion ) Where f(·) represents a multilayer perceptron with enhanced expression capabilities; The final sentiment classification output is calculated using the SoftMax function, whose expression is: in, This represents the probability distribution of the sentiment category predicted by the model; U and b are the learnable parameters of the fully connected classifier.

7. A multimodal sentiment analysis device based on sentiment consistency distillation and information bottleneck, characterized in that, When applied, the multimodal sentiment analysis method based on sentiment consistency distillation and information bottleneck as described in any one of claims 1-6 is executed, and the apparatus comprises: Multimodal feature extraction module: This module receives input multimodal emotional data video and extracts features from image, audio, and text modalities to obtain multimodal emotional features. Semantic vector construction module: used to uniformly map multimodal sentiment features into multimodal semantic vectors in a low-dimensional embedding space; Feature decoupling mapping module: Based on a neural network structure, it decomposes multimodal semantic vectors into modality-specific semantic vectors and modality-common semantic vectors; Modal Pair Sentiment Consistency Distillation Module: Extracts sentiment collaboration information between modal pairs through a cross-modal attention mechanism, and transfers this information to the modal common semantic vector using a knowledge distillation method; Modality-Specific Information Optimization Module: Introducing the information bottleneck principle, the module uses an encoder-decoder structure and KL divergence regularization constraints to compress and denoise modality-specific semantic vectors; Multimodal sentiment classification module: It concatenates the optimized modality-specific semantic vector with the modality common semantic vector to form a fused semantic representation, inputs it into the linear mapping layer and processes it through the SoftMax function, and outputs the final sentiment classification label.

8. An electronic device, characterized in that, The method includes a processor and a memory, wherein the memory stores machine-executable instructions that can be executed by the processor, and the processor executes the machine-executable instructions to implement a multimodal sentiment analysis method based on sentiment consistency distillation and information bottleneck as described in any one of claims 1-6.

Images