Robust multi-modal sentiment analysis method based on principal modal enhancement and multi-stage fusion

Abstract

This invention proposes a robust multimodal sentiment analysis method based on dominant modality enhancement and multi-stage fusion. The method includes: extracting unified features from noisy input using an embedded encoder; evaluating the integrity of text modality features within the unified features using an integrity check encoder; enhancing the unified features using a text-guided cross-modal collaborative enhancer; further enhancing the unified features using a hierarchical conditional cue generator, and adjusting weights based on text integrity scores; inputting global sentiment features into a fully connected layer and a classifier to obtain the sentiment probability distribution, and recovering complete features from the noisy input using a modality-specific reconstructor. This invention, through the complementary design of the text-guided cross-modal collaborative enhancer and the hierarchical conditional cue generator, adapts to both complete and missing dominant modalities, effectively addressing the core pain point of traditional methods' difficulty in accurately handling situations where the dominant modality itself is missing.

Description

Technical Field

[0001] This invention relates to the field of multimodal technology at the intersection of natural language processing, computer vision, and speech signal processing, and particularly to a robust multimodal sentiment analysis method based on dominant modality enhancement and multi-stage fusion. Background Technology

[0002] Multimodal sentiment analysis, by fusing heterogeneous modal information such as text, audio, and vision, achieves a deep understanding of human emotions and has been widely applied in fields such as human-computer interaction and social computing. Among them, the text modality is recognized as the dominant modality in multimodal sentiment analysis due to its high information density and accurate semantic expression. However, in real-world deployment scenarios, multimodal data often suffers from modality loss due to factors such as sensor failures, transmission errors, and limitations of speech recognition systems. In particular, the complete state change of the dominant text modality from complete to missing severely disrupts the complementarity between modalities, leading to a significant decline in the performance of traditional models.

[0003] Currently, existing multimodal sentiment analysis methods mainly suffer from the following problems: 1. Traditional models are mostly built on the ideal assumption of "modal integrity". They are not good at dealing with modal missing and noise interference, and lack systematic missing compensation and noise suppression mechanisms. They are difficult to maintain stable emotion recognition performance in scenarios with incomplete data and interference. 2. Insufficient dynamic adaptation capability of the dominant modality. Although the dominant modality-based method recognizes the importance of the text modality, it lacks an effective response mechanism when the dominant modality itself is missing. It cannot achieve full-scene coverage from complete to severely missing dominant modality and lacks a dynamic adjustment mechanism for missing modality perception. 3. Multimodal fusion is not deep enough, and deep semantic mining and cross-modal association analysis are insufficient. Existing methods either directly perform fine-grained interaction, ignoring the interference of modal heterogeneity and redundant information; or adopt single-granularity fusion, which makes it difficult to take into account both global semantics and local emotional details, and fails to explore deep emotional clues between modalities. Summary of the Invention

[0004] In view of the above situation, the main objective of this invention is to propose a robust multimodal sentiment analysis method based on main modality enhancement and multi-stage fusion to solve the above-mentioned technical problems.

[0005] This invention proposes a robust multimodal sentiment analysis method based on dominant modality enhancement and multi-stage fusion, the method comprising the following steps: Step 1: Construct a sentiment prediction model based on an embedded encoder, an integrity check encoder, a text-guided cross-modal collaborative enhancer, a hierarchical conditional cue generator, a weighted fusion unit, a coarse-to-fine cross-modal fusion module, a fully connected layer and a classifier, and a modality-specific reconstructor. Step 2: Obtain the multimodal sentiment dataset, preprocess the data of each modality in the multimodal sentiment dataset to obtain noisy input; use the embedding encoder to extract modality uniform features from the noisy input, and use the integrity check encoder to evaluate the integrity of the text modality features in the modality uniform features to obtain the text integrity score; Step 3: Use a text-guided cross-modal collaborative enhancer to enhance the modal unified features to obtain enhanced text features; use a hierarchical conditional cue generator to enhance the modal unified features and adjust the weights through text integrity scores to obtain surrogate features of the text modality. Step 4: Using a weighted fusion processor to combine the text integrity score, the enhanced text features and the proxy features of the text modality are weighted and fused to obtain the enhanced main modality features; Step 5: Input the enhanced master modality features, along with the visual and audio unified features from the modality unified features, into the coarse-to-fine cross-modal fusion module. The modules will then undergo three stages of processing: modality alignment, redundancy filtering, and sentiment refinement, to obtain the global sentiment features. Step 6: Input the global sentiment features into the fully connected layer and the classifier to obtain the sentiment probability distribution, and use the modality-specific reconstructor to recover the complete features from the noisy input; construct a loss function based on the complete features to optimize the sentiment prediction model, obtain the optimized sentiment prediction model, and use the optimized sentiment prediction model to obtain the final sentiment probability distribution.

[0006] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. This invention constructs a dual-path enhancement architecture for the main modality. Through the complementary design of a text-guided cross-modal collaborative enhancer and a hierarchical conditional prompt generator, it adapts to both complete and missing main modality scenarios, effectively solving the core pain point of traditional methods that are difficult to accurately handle when the main modality itself is missing.

[0007] 2. This invention integrates semantic, sentiment, and sentiment prior information into a hierarchical conditional cue generator, and combines auxiliary modality cues to generate semantically and sentimentally consistent proxy features. It achieves semantic compensation in the absence of the main modality in a lightweight manner, which avoids the high computational cost of complex feature generation and ensures the quality of feature representation in missing scenarios.

[0008] 3. This invention uses an integrity-aware dynamic fusion mechanism to adaptively adjust the contribution weights of the two paths based on the integrity of the text modality, thereby achieving a smooth transition of the main modality from complete to missing states, which greatly improves the model's adaptability and robustness to complex real-world scenarios.

[0009] 4. The present invention designs a coarse-to-fine cross-modal fusion strategy, which sequentially resolves heterogeneity through modal alignment, filters sentiment-irrelevant information through redundancy screening, and deepens cross-modal association through sentiment refinement. This effectively balances global semantics and local sentiment details, and significantly enhances the discriminative ability of cross-modal features.

[0010] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by means of embodiments of the invention. Attached Figure Description

[0011] Figure 1 This is a flowchart illustrating the robust multimodal sentiment analysis method based on master modality enhancement and multi-stage fusion proposed in this invention.

[0012] Figure 2 This is a diagram illustrating the overall framework of the robust multimodal sentiment analysis method based on master modality enhancement and multi-stage fusion proposed in this invention.

[0013] Figure 3 This is a structural diagram of the text-guided cross-modal collaborative enhancer and hierarchical conditional cue generator in the robust multimodal sentiment analysis method based on master modality enhancement and multi-stage fusion proposed in this invention.

[0014] Figure 4 This is a structural diagram of the coarse-to-fine cross-modal fusion strategy in the robust multimodal sentiment analysis method based on master modality enhancement and multi-stage fusion proposed in this invention. Detailed Implementation

[0015] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0016] These and other aspects of the embodiments of the present invention will become clear from the following description and accompanying drawings. In these descriptions and drawings, some specific embodiments of the present invention are specifically disclosed to provide some ways of implementing the principles of the embodiments of the present invention; however, it should be understood that the scope of the embodiments of the present invention is not limited thereto.

[0017] Please see Figure 1 This invention proposes a robust multimodal sentiment analysis method based on dominant modality enhancement and multi-stage fusion, which includes the following steps: Step 1: Construct a sentiment prediction model based on an embedded encoder, an integrity check encoder, a text-guided cross-modal collaborative enhancer, a hierarchical conditional cue generator, a weighted fusion unit, a coarse-to-fine cross-modal fusion module, a fully connected layer and a classifier, and a modality-specific reconstructor.

[0018] Step 2: Obtain the multimodal sentiment dataset, preprocess the data of each modality in the multimodal sentiment dataset to obtain noisy input; use the embedding encoder to extract modal uniform features from the noisy input, and use the integrity check encoder to evaluate the integrity of the text modal features in the modal uniform features to obtain the text integrity score.

[0019] Please see Figure 2 In step 2, a multimodal sentiment dataset is acquired, and the data of each modality in the multimodal sentiment dataset are preprocessed to obtain noisy input. An embedding encoder is used to extract modality uniform features from the noisy input, and an integrity check encoder is used to evaluate the integrity of the text modality features in the modality uniform features to obtain a text integrity score. Specifically, the steps include: A multimodal sentiment dataset was acquired. Using a pre-defined BERT model, Librosa tool, and OpenFace tool, text encoding, audio feature extraction, and visual feature extraction were performed on the dataset to obtain preprocessed text modality features, preprocessed audio modality features, and preprocessed visual modality features. ; in, Represents the sequence features of each modality. Indicates that the sequence length is The dimension of the modal feature vector is The set, Indicates the modal type index. Represents text modality, Representing visual modality, Indicates audio modality; Random deletion operations are applied to the preprocessed text modal features, preprocessed audio modal features, and preprocessed visual modal features to form noisy input; It should be noted that the random missing data operation refers to replacing the feature vectors of visual and audio modal features with zero vectors with a preset probability; for text modal features, replacing the word IDs with preset unknown tokens [UNK] or directly removing the corresponding tokens with a preset probability. By simulating data loss in real-world scenarios, this method can enhance the model's robustness to noisy inputs.

[0020] An embedded encoder is used to extract features of each modality from a noisy input, and key modal information is embedded to generate unified modal features; An integrity check encoder is used to evaluate the integrity of text features in modal uniformity features to obtain a text integrity score.

[0021] An embedded encoder is used to extract features of each modality from a noisy input, and key modal information is embedded to generate unified modal features. The relationship between these processes is as follows: ; in, This represents the modal uniformity feature after encoding by the encoder. Indicates an embedded encoder. This indicates a feature concatenation operation. This indicates the learnable start markers corresponding to each modality. Indicates a noisy input; In the step of using an integrity check encoder to evaluate the integrity of text features in modality uniformity features to obtain a text integrity score, the corresponding relationship in the process is as follows: ; in, Indicates the text integrity score. Indicates an integrity check encoder. This represents a random initialization flag used for integrity prediction. A unified feature representing text modality.

[0022] It should be noted that a random deletion operation is applied to the sequence features of each modality, randomly erasing information from 0% to 100%, ultimately forming a noisy multimodal input. Missing parts in the visual and audio modalities are filled with zero vectors, while missing positions in the text modality are filled with unknown word tags [UNK] from the BERT model. For the noisy multimodal input, an embedded encoder with two Transformer encoding layers is used to extract and unify features. An integrity check encoder is used to evaluate the integrity of the text modality features, obtaining a text integrity score, which provides a basis for subsequent weight adjustment in dual-path feature fusion. This encoder consists of a two-layer Transformer encoder and a linear regression head.

[0023] Step 3: Use a text-guided cross-modal collaborative enhancer to enhance the modal unified features to obtain enhanced text features; use a hierarchical conditional cue generator to enhance the modal unified features, and adjust the weights through text integrity scores to obtain surrogate features of the text modality.

[0024] Please see Figure 3In step 3, a text-guided cross-modal collaborative enhancer is used to enhance the modal unified features, resulting in enhanced text features. A hierarchical conditional cue generator is then used to further enhance the modal unified features, and the weights are adjusted using text integrity scores to obtain surrogate features for the text modality. Specifically, this includes the following steps: By utilizing the visual-text cross-modal mapping encoder and the audio-text cross-modal mapping encoder in the text-guided cross-modal collaborative enhancer, the visual features and audio features in the modality unification features are mapped to the text semantic space, respectively, to obtain visual alignment features and audio alignment features; Visual alignment features and audio alignment features are respectively subjected to layer normalization and random deactivation operations with the text features in the modality unification features to obtain visual mapping features and audio mapping features. The visual mapping features and audio mapping features are concatenated to obtain the concatenated features. Using the text features in the modal unified features as the query, a multi-head attention mechanism is used to perform multi-head attention interaction on the features concatenated from the visual mapping features and the audio mapping features. After residual normalization to integrate complementary information, cross-modal fusion features are obtained. The cross-modal fusion features are refined and fused through the feature refiner in the text-guided cross-modal collaborative enhancer to obtain enhanced text features; The visual features in the modality uniformity feature are processed sequentially through layer normalization, linear fully connected layer and Gaussian error linear unit activation function and random deactivation operation to obtain the enhanced visual features; The enhanced visual features are encoded using the Transformer encoder in the hierarchical conditional cue generator to obtain the encoded visual depth features. The text features in the modality unified features are processed sequentially through layer normalization, linear fully connected layer and Gaussian error linear unit activation function and random deactivation operation to obtain the enhanced intermediate text features; The enhanced intermediate text features are encoded using the Transformer encoder in the hierarchical conditional cue generator to obtain the encoded deep text features. The audio features in the modal uniformity feature are processed sequentially through layer normalization, linear fully connected layer and Gaussian error linear unit activation function and regularization operation to obtain the enhanced audio features; The enhanced audio features are encoded using the Transformer encoder in the hierarchical conditional cue generator to obtain the encoded audio deep coding features. The encoded visual depth features and the encoded audio depth features are bidirectionally complementary through a cross-modal bidirectional cross-attention encoder in a hierarchical conditional cue generator, and then averaged to generate cross-modal cues. The deep encoded features of the encoded text are subjected to layer normalization, and combined with the text integrity score, they are dynamically weighted by a multilayer perceptron with sigmoid activation in the hierarchical conditional cue generator to obtain text residual information. The text residual information is then subjected to layer normalization using preset fusion semantic cue, preset sentiment cue, and preset sentiment prior cue, and the text residual information is used to construct an initial semantic-sentiment skeleton. Using the initial semantic-sentiment skeleton as the query, cross-modal cues and textual residual information as key-value pairs, the features are processed through a multi-head attention mechanism and then normalized to stabilize them, resulting in adaptive dynamic prompts. Based on the text integrity score, the weights of the initial semantic-sentiment skeleton and the adaptive dynamic cues are adjusted to obtain the proxy features of the text modality.

[0025] It should be noted that, in order to introduce external knowledge constraints, this invention pre-defines the fusion of semantic cues, sentiment cues, and sentiment prior cues. These cue vectors are learnable parameters designed to guide the model to focus on the sentiment-related semantic space. Using a multilayer perceptron with sigmoid activation, the text integrity score is mapped to the [0,1] interval, serving as a weighting coefficient to dynamically adjust the residual connection strength, thereby achieving adaptive fusion of prior knowledge based on text integrity.

[0026] By utilizing the visual-text cross-modal mapping encoder and the audio-text cross-modal mapping encoder in the text-guided cross-modal collaborative enhancer, the visual features and audio features in the modality unification features are mapped to the text semantic space, respectively, to obtain visual alignment features and audio alignment features. The corresponding relationship in the process is as follows: ; in, This represents the visual alignment feature projected onto the text semantic space after cross-modal mapping of visual features. This represents the encoder used for visual-text cross-modal mapping. Representing visual features in modality uniformity features, This represents the audio alignment features projected onto the text semantic space after cross-modal mapping of audio features. This refers to the encoder used for audio-text cross-modal mapping. This represents the audio features in the modality uniformity feature; In the steps of fusing visual alignment features and audio alignment features with text features in modality unification features through layer normalization and random deactivation to obtain visual mapping features and audio mapping features, the corresponding relationships in the process are as follows: ; in, Represents visual mapping features, This indicates a random deactivation operation. Presentation layer normalization processing, Represents audio mapping features; In the step of concatenating visual mapping features and audio mapping features to obtain the concatenated features, the corresponding relationship in the process is as follows: ; in, This represents the feature obtained by concatenating visual mapping features and audio mapping features; In the process of using text features as the query in modal unified features, performing multi-head attention interaction on the concatenated visual and audio mapping features through a multi-head attention mechanism, and then integrating complementary information through residual normalization to obtain cross-modal fusion features, the corresponding relationship in the process is as follows: ; in, This represents the cross-modal fusion features after multi-head attention interaction and residual normalization. This represents the multi-head attention mechanism; In the feature refinement encoder of the text-guided cross-modal collaborative enhancer, the steps of refining and fusing cross-modal fusion features to obtain enhanced text features are as follows: ; in, This represents the enhanced text features. This represents the encoder for feature refinement in a text-guided cross-modal collaborative enhancer; In the process of sequentially processing the visual features in the modality-unified features through layer normalization, linear fully connected layers, Gaussian error linear unit activation functions, and random deactivation to obtain the enhanced visual features, the corresponding relationships in the process are as follows: ; in, Indicates enhanced visual features, Represents the activation function of the Gaussian error linear unit. Indicates a linear fully connected layer; In the step of encoding the enhanced visual features using the Transformer encoder in the hierarchical conditional cue generator to obtain the encoded visual depth features, the corresponding relationship in the process is as follows: ; in, This represents the enhanced visual features, after being encoded, as visual depth-coded features. This refers to the standard Transformer encoder used for single-modal feature encoding; In the step of generating cross-modal cues by bidirectionally complementing the encoded visual depth features and the encoded audio depth features through a cross-modal bidirectional cross-attention encoder in a hierarchical conditional cue generator, and then averaging the features, the corresponding relationship in the process is as follows: ; in, Indicates cross-modal cues, This represents a cross-modal bidirectional cross-attention encoder in a hierarchical conditional cue generator. This represents the depth-coded features of the encoded audio. In the steps of performing layer normalization on the deep encoded features of the encoded text and dynamically weighting them using a multilayer perceptron with sigmoid activation in a hierarchical conditional cue generator to obtain text residual information, and then using preset fused semantic cues, preset sentiment cues, and preset sentiment prior cues to perform layer normalization on the text residual information to construct the initial semantic-sentiment skeleton, the corresponding relationships in the process are as follows: ; in, Represents text residual information. This represents a multilayer perceptron with Sigmoid activation in a hierarchical conditional cue generator. This represents the deep encoding features of the encoded text. Represents the initial semantic-emotional skeleton. This indicates a pre-defined fusion semantic prompt. This indicates a pre-set emotional cue. This indicates a pre-set emotional prior cues; In the process of obtaining adaptive dynamic prompts by using the initial semantic-sentiment skeleton as the query, cross-modal cues and textual residual information as key-value pairs, processing through a multi-head attention mechanism and undergoing layer normalization to stabilize features, the corresponding relationship in the process is as follows: ; in, This indicates an adaptive dynamic suggestion; In the steps of generating proxy features for the text modality by adjusting the weights of the initial semantic-sentiment skeleton and adaptive dynamic cues based on the text integrity score, the corresponding relationship in the process is as follows: ; in, Proxy features representing text modalities.

[0027] Step 4: Using a weighted fusion processor, the enhanced text features and the proxy features of the text modality are weighted and fused together with the text integrity score to obtain the enhanced main modality features.

[0028] In step 4, a weighted fusion processor is used to combine the text integrity score with the enhanced text features and the proxy features of the text modality to obtain the enhanced main modality features. The corresponding relationship in this process is as follows: ; in, This represents the enhanced main modal features.

[0029] Step 5: Input the enhanced master modality features, along with the visual and audio unified features from the modality unified features, into the coarse-to-fine cross-modal fusion module. The modules will then undergo three stages of processing: modality alignment, redundancy filtering, and sentiment refinement, to obtain the global sentiment features.

[0030] Please see Figure 4 In step 5, the enhanced master modality features, along with the visual and audio unified features from the modality unified features, are input into the coarse-to-fine cross-modal fusion module. They undergo three stages of processing: modality alignment, redundancy filtering, and sentiment refinement, to obtain the global sentiment features. Specifically, the steps include: The enhanced master modality features, along with the visual and audio unified features from the modality unified features, are input into the coarse-to-fine cross-modal fusion module. Using the Transformer encoder, the visual and audio features from the modality unified features are aligned to the unified dominant modality space to obtain coarse-grained visual features and coarse-grained audio features. The enhanced master modality features are then subjected to layer normalization to obtain coarse-grained text features. We perform weighted summation on coarse-grained visual features, coarse-grained audio features, and coarse-grained text features to fuse the features and generate global coarse-grained cross-modal features. Using a multilayer perceptron in a coarse-to-fine cross-modal fusion module, global coarse-grained cross-modal features are compressed and reconstructed to generate reconstructed features; at the same time, redundancy is filtered through a preset temperature smoothing learnable gating to obtain bottleneck features after redundancy filtering. The bottleneck features that have been redundantly filtered are sequentially processed by global average pooling and layer normalization to obtain global features. The global features are then processed by a multilayer perceptron in the cross-modal fusion module from coarse to fine to generate sample-level dynamic weights. A residual connection is constructed between the global coarse-grained cross-modal features and the bottleneck features after redundancy screening, and dynamic weighted fusion calculation is performed by combining sample-level dynamic weights to generate medium-grained cross-modal features. Bidirectional attention computation is performed between medium-granularity cross-modal features and coarse-granular text features to obtain attention-enhanced features; the attention-enhanced features and medium-granularity cross-modal features are then enhanced by residual connections and multilayer perceptrons to obtain fine-granularity cross-modal features. The linear layers in the coarse-to-fine cross-modal fusion module are used to calculate the emotional attention weight of each token in the fine-grained cross-modal features using Softmax normalization. Then, the tokens in the fine-grained cross-modal features are weighted and summed to obtain the global emotional features.

[0031] It's important to note that a learnable temperature parameter is introduced to control the stiffness or softness of the gating function. In the early stages of training, a higher temperature value results in a smoother distribution of gating coefficients, allowing the model to explore a wide range of features. As training progresses, the temperature parameter automatically decreases, causing the gating coefficients to become more discrete, achieving hard filtering of redundant information. This smooth transition from 'soft attention' to 'hard filtering' helps the model effectively filter out heterogeneous interference between modalities while preserving key sentiment information.

[0032] Using a Transformer encoder, the visual and audio features in the modality unification feature set are aligned to the unified dominant modality space, respectively, to obtain coarse-grained visual and audio features. Layer normalization is then applied to the enhanced dominant modality features to obtain coarse-grained text features. The corresponding relationship in this process is as follows: ; in, Indicates coarse-grained visual features. Represents coarse-grained audio features. Represents coarse-grained text features; In the step of weighted summation of coarse-grained visual features, coarse-grained audio features, and coarse-grained text features to fuse features and generate global coarse-grained cross-modal features, the corresponding relationship in the process is as follows: ; in, Represents global coarse-grained cross-modal features. This represents the weighted fusion coefficient of coarse-grained multimodal features. Indicates the first Coarse-grained features corresponding to each mode; In the step of using a multilayer perceptron in a coarse-to-fine cross-modal fusion module to compress and reconstruct global coarse-grained cross-modal features to generate reconstructed features; and simultaneously using a preset temperature-smoothing learnable gating system for redundancy filtering to obtain bottleneck features after redundancy filtering, the corresponding relationship in the process is as follows: ; in, Indicates the first Reconstructed features corresponding to each modality This represents a multilayer perceptron used for feature reconstruction. A multilayer perceptron representing feature compression. Indicates the first The temperature smoothing learnable gating coefficients corresponding to each mode This represents the Sigmoid activation function. Indicates the first The learnable temperature parameters corresponding to each mode Indicates the first Bottleneck features of each modality after redundancy screening; In the process of sequentially performing global average pooling and layer normalization on the bottleneck features after redundancy screening to obtain global features, and then generating sample-level dynamic weights for the global features through a multilayer perceptron in the coarse-to-fine cross-modal fusion module, the corresponding relationship is as follows: ; in, Represents global features. This indicates a global average pooling operation. Indicates the first The sample-level dynamic weights corresponding to each modality Represents the normalized exponential function, This represents a multilayer perceptron in a coarse-to-fine cross-modal fusion module; In the steps of constructing residual connections between global coarse-grained cross-modal features and bottleneck features after redundancy screening, and performing dynamic weighted fusion calculations using sample-level dynamic weights to generate medium-grained cross-modal features, the corresponding relationships in the process are as follows: ; in, Indicates medium-granularity cross-modal characteristics; In the step of performing bidirectional attention computation between medium-granularity cross-modal features and coarse-grained text features to obtain attention-enhanced features, and then enhancing the attention-enhanced features and medium-granularity cross-modal features through residual connections and a multilayer perceptron to obtain fine-granularity cross-modal features, the corresponding relationships in the process are as follows: ; in, This indicates the enhanced attentional features after bidirectional attention. Represents fine-grained cross-modal characteristics; In the steps of calculating the sentiment attention weight of each token in the fine-grained cross-modal features through the linear layer in the coarse-to-fine cross-modal fusion module and Softmax normalization, and then performing a weighted summation of the tokens in the fine-grained cross-modal features to obtain the global sentiment features, the corresponding relationship in the process is as follows: ; in, Indicates attention weights, Indicates the index of the token. This represents the feature tensor slicing operation. Indicates global sentiment characteristics. This represents the total number of tokens.

[0033] Step 6: Input the global sentiment features into the fully connected layer and the classifier to obtain the sentiment probability distribution, and use the modality-specific reconstructor to recover the complete features from the noisy input; construct a loss function based on the complete features to optimize the sentiment prediction model, obtain the optimized sentiment prediction model, and use the optimized sentiment prediction model to obtain the final sentiment probability distribution.

[0034] In step 6, a loss function is constructed based on the complete features to optimize the sentiment prediction model, resulting in an optimized sentiment prediction model. This process includes the following steps: Obtain the true integrity label corresponding to the text integrity score; Obtain the true and complete features of each modality; Obtain the real sentiment labels corresponding to the sentiment scores; Construct an integrity check loss based on the text integrity score and the true integrity label; Construct a reconstruction loss by recovering complete features from noisy input and comparing them with the true complete features; A sentiment prediction loss is constructed based on the sentiment probability distribution and the true sentiment labels; A sentiment guidance loss is constructed based on the sentiment score predicted by the text proxy features and the real sentiment label; A single-modal information bottleneck reconstruction loss is constructed based on bottleneck features, reconstruction features, and coarse-grained features. The total information bottleneck loss is calculated using the single-modal information bottleneck reconstruction loss. An inter-modal mutual information loss is constructed based on the cosine similarity between text global features, audio global features, and visual global features. The sum of the total information bottleneck loss and the inter-modal mutual information loss is used as the exclusive constraint loss of the fusion module. By utilizing integrity check loss, reconstruction loss, sentiment prediction loss, sentiment guidance loss, and the specific constraint loss of the fusion module, the sentiment prediction model is jointly optimized to obtain an optimized sentiment prediction model.

[0035] Based on the text integrity score and the true integrity label, an integrity check loss is constructed. The corresponding relationship in the process is as follows: ; in, Indicates loss due to integrity check. This indicates the number of samples in the training set. Indicates the sample index. Indicates the first Predicted integrity score for each sample Indicates the first The integrity score of each sample and its true label. The squared L2 norm of a vector; In the step of recovering complete features from noisy input and constructing the reconstruction loss from the true complete features, the corresponding process has the following relationship: ; in, Indicates the reconstruction loss. Indicates the first The first sample The true features of a modality Indicates the first The first sample Reconstruction features of each modality; In the step of constructing the sentiment prediction loss based on the sentiment probability distribution and the true sentiment labels, the corresponding relationship in the process is as follows: ; in, This represents the final emotional score. This represents the true label of the sample. Indicates the first The final sentiment prediction results for each sample; In the step of constructing the sentiment guidance loss based on the sentiment score predicted by the text proxy features and the real sentiment label, the corresponding process has the following relationship: ; in, Indicates the predicted sentiment score, This indicates a loss of emotional guidance. Indicates the first Predicted sentiment scores for each sample; In the steps of constructing the single-modal information bottleneck reconstruction loss based on bottleneck features, reconstruction features, and coarse-grained features, and calculating the total information bottleneck loss using the single-modal information bottleneck reconstruction loss, the corresponding relationships in the process are as follows: ; in, This represents the total information bottleneck loss. This represents the mean squared error loss function. This represents the L2 regularization term. This represents the total information bottleneck loss. A parameter representing the total information bottleneck loss; In the step of constructing intermodal mutual information loss based on the cosine similarity of text global features, audio global features, and visual global features, the corresponding relationship in the process is as follows: ; in, This represents the intermodal information loss. The weighting coefficients represent the intermodal mutual information loss. The cosine similarity function represents the degree of cross-modal feature alignment.

[0036] This also includes the corresponding total loss function, and the relationship between the corresponding processes is as follows: ; in, Represents the total loss function. This represents the specific constraint loss of the fusion module.

[0037] It should be noted that the total loss function The loss is composed of a weighted sum of integrity check loss, reconstruction loss, sentiment prediction loss, sentiment guidance loss, and the specific constraint loss of the fusion module. It is used to comprehensively measure the error of the model in four dimensions: main modality integrity assessment, noise feature reconstruction, sentiment classification prediction, and cross-modal deep fusion. Based on this error, the model parameters are adjusted by backpropagation until the loss converges, thereby obtaining a robust multimodal sentiment prediction model.

[0038] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0039] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0040] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of this patent should be determined by the appended claims.

Claims

1. A robust multimodal sentiment analysis method based on dominant modality enhancement and multi-stage fusion, characterized in that, The method includes the following steps: Step 1: Construct a sentiment prediction model based on an embedded encoder, an integrity check encoder, a text-guided cross-modal collaborative enhancer, a hierarchical conditional cue generator, a weighted fusion unit, a coarse-to-fine cross-modal fusion module, a fully connected layer and a classifier, and a modality-specific reconstructor. Step 2: Obtain the multimodal sentiment dataset, preprocess the data of each modality in the multimodal sentiment dataset to obtain noisy input; use the embedding encoder to extract modality uniform features from the noisy input, and use the integrity check encoder to evaluate the integrity of the text modality features in the modality uniform features to obtain the text integrity score; Step 3: Use a text-guided cross-modal collaborative enhancer to enhance the modal unified features to obtain enhanced text features; use a hierarchical conditional cue generator to enhance the modal unified features and adjust the weights through text integrity scores to obtain surrogate features of the text modality. Step 4: Using a weighted fusion processor to combine the text integrity score, the enhanced text features and the proxy features of the text modality are weighted and fused to obtain the enhanced main modality features; Step 5: Input the enhanced master modality features, along with the visual and audio unified features from the modality unified features, into the coarse-to-fine cross-modal fusion module. The modules will then undergo three stages of processing: modality alignment, redundancy filtering, and sentiment refinement, to obtain the global sentiment features. Step 6: Input the global sentiment features into the fully connected layer and the classifier to obtain the sentiment probability distribution, and use the modality-specific reconstructor to recover the complete features from the noisy input; A loss function is constructed based on complete features to optimize the sentiment prediction model, resulting in an optimized sentiment prediction model. The optimized sentiment prediction model is then used to obtain the final sentiment probability distribution.

2. The robust multimodal sentiment analysis method based on dominant modality enhancement and multi-stage fusion according to claim 1, characterized in that, In step 2, a multimodal sentiment dataset is acquired, and the data of each modality in the multimodal sentiment dataset are preprocessed to obtain noisy input. An embedding encoder is used to extract modality uniform features from the noisy input, and an integrity check encoder is used to evaluate the integrity of the text modality features in the modality uniform features to obtain a text integrity score. Specifically, the steps include: A multimodal sentiment dataset was acquired. Using a pre-defined BERT model, Librosa tool, and OpenFace tool, text encoding, audio feature extraction, and visual feature extraction were performed on the dataset to obtain preprocessed text modality features, preprocessed audio modality features, and preprocessed visual modality features. ; in, Represents the sequence features of each modality. Indicates that the sequence length is The dimension of the modal feature vector is The set, Indicates the modal type index. Represents text modality, Representing visual modality, Indicates audio modality; Random deletion operations are applied to the preprocessed text modal features, preprocessed audio modal features, and preprocessed visual modal features to form noisy input; An embedded encoder is used to extract features of each modality from a noisy input, and key modal information is embedded to generate unified modal features; An integrity check encoder is used to evaluate the integrity of text features in modal uniformity features to obtain a text integrity score.

3. The robust multimodal sentiment analysis method based on principal modality enhancement and multi-stage fusion according to claim 2, characterized in that, An embedded encoder is used to extract features of each modality from a noisy input, and key modal information is embedded to generate unified modal features. The relationship between these processes is as follows: ； in, This represents the modal uniformity feature after encoding by the encoder. Indicates an embedded encoder. This indicates a feature concatenation operation. This indicates the learnable start markers corresponding to each modality. Indicates a noisy input; In the step of using an integrity check encoder to evaluate the integrity of text features in modality uniformity features to obtain a text integrity score, the corresponding relationship in the process is as follows: ； in, Indicates the text integrity score. Indicates an integrity check encoder. This represents a random initialization flag used for integrity prediction. A unified feature representing text modality.

4. The robust multimodal sentiment analysis method based on dominant modality enhancement and multi-stage fusion according to claim 3, characterized in that, In step 3, a text-guided cross-modal collaborative enhancer is used to enhance the modal unified features, resulting in enhanced text features. A hierarchical conditional cue generator is then used to further enhance the modal unified features, and the weights are adjusted using text integrity scores to obtain surrogate features for the text modality. Specifically, this includes the following steps: By utilizing the visual-text cross-modal mapping encoder and the audio-text cross-modal mapping encoder in the text-guided cross-modal collaborative enhancer, the visual features and audio features in the modality unification features are mapped to the text semantic space, respectively, to obtain visual alignment features and audio alignment features; Visual alignment features and audio alignment features are respectively subjected to layer normalization and random deactivation operations with the text features in the modality unification features to obtain visual mapping features and audio mapping features. The visual mapping features and audio mapping features are concatenated to obtain the concatenated features. Using the text features in the modal unified features as the query, a multi-head attention mechanism is used to perform multi-head attention interaction on the features concatenated from the visual mapping features and the audio mapping features. After residual normalization to integrate complementary information, cross-modal fusion features are obtained. The cross-modal fusion features are refined and fused through the feature refiner in the text-guided cross-modal collaborative enhancer to obtain enhanced text features; The visual features in the modality uniformity feature are processed sequentially through layer normalization, linear fully connected layer and Gaussian error linear unit activation function and random deactivation operation to obtain the enhanced visual features; The enhanced visual features are encoded using the Transformer encoder in the hierarchical conditional cue generator to obtain the encoded visual depth features. The text features in the modality unified features are processed sequentially through layer normalization, linear fully connected layer and Gaussian error linear unit activation function and random deactivation operation to obtain the enhanced intermediate text features; The enhanced intermediate text features are encoded using the Transformer encoder in the hierarchical conditional cue generator to obtain the encoded deep text features. The audio features in the modal uniformity feature are processed sequentially through layer normalization, linear fully connected layer and Gaussian error linear unit activation function and regularization operation to obtain the enhanced audio features; The enhanced audio features are encoded using the Transformer encoder in the hierarchical conditional cue generator to obtain the encoded audio deep coding features. The encoded visual depth features and the encoded audio depth features are bidirectionally complementary through a cross-modal bidirectional cross-attention encoder in a hierarchical conditional cue generator, and then averaged to generate cross-modal cues. The deep encoded features of the encoded text are subjected to layer normalization, and combined with the text integrity score, they are dynamically weighted by a multilayer perceptron with sigmoid activation in the hierarchical conditional cue generator to obtain text residual information. By utilizing preset fusion semantic prompts, preset sentiment prompts, and preset sentiment prior prompts, and performing layer normalization processing on the text residual information, an initial semantic-sentiment skeleton is constructed. Using the initial semantic-sentiment skeleton as the query, cross-modal cues and textual residual information as key-value pairs, the features are processed through a multi-head attention mechanism and then normalized to stabilize them, resulting in adaptive dynamic prompts. Based on the text integrity score, the weights of the initial semantic-sentiment skeleton and the adaptive dynamic cues are adjusted to obtain the proxy features of the text modality.

5. The robust multimodal sentiment analysis method based on principal modality enhancement and multi-stage fusion according to claim 4, characterized in that, By utilizing the visual-text cross-modal mapping encoder and the audio-text cross-modal mapping encoder in the text-guided cross-modal collaborative enhancer, the visual features and audio features in the modality unification features are mapped to the text semantic space, respectively, to obtain visual alignment features and audio alignment features. The corresponding relationship in the process is as follows: ； in, This represents the visual alignment feature projected onto the text semantic space after cross-modal mapping of visual features. This represents the encoder used for visual-text cross-modal mapping. Representing visual features in modality uniformity features, This represents the audio alignment features projected onto the text semantic space after cross-modal mapping of audio features. This refers to the encoder used for audio-text cross-modal mapping. This represents the audio features in the modality uniformity feature; In the steps of fusing visual alignment features and audio alignment features with text features in modality unification features through layer normalization and random deactivation to obtain visual mapping features and audio mapping features, the corresponding relationships in the process are as follows: ； in, Represents visual mapping features, This indicates a random deactivation operation. Presentation layer normalization processing, Represents audio mapping features; In the step of concatenating visual mapping features and audio mapping features to obtain the concatenated features, the corresponding relationship in the process is as follows: ； in, This represents the feature obtained by concatenating visual mapping features and audio mapping features; In the process of using text features as the query in modal unified features, performing multi-head attention interaction on the concatenated visual and audio mapping features through a multi-head attention mechanism, and then integrating complementary information through residual normalization to obtain cross-modal fusion features, the corresponding relationship in the process is as follows: ； in, This represents the cross-modal fusion features after multi-head attention interaction and residual normalization. This represents the multi-head attention mechanism; In the feature refinement encoder of the text-guided cross-modal collaborative enhancer, the steps of refining and fusing cross-modal fusion features to obtain enhanced text features are as follows: ； in, This represents the enhanced text features. This represents the encoder for feature refinement in a text-guided cross-modal collaborative enhancer; In the process of sequentially processing the visual features in the modality-unified features through layer normalization, linear fully connected layers, Gaussian error linear unit activation functions, and random deactivation to obtain the enhanced visual features, the corresponding relationships in the process are as follows: ； in, Indicates enhanced visual features, Represents the activation function of the Gaussian error linear unit. Indicates a linear fully connected layer; In the step of encoding the enhanced visual features using the Transformer encoder in the hierarchical conditional cue generator to obtain the encoded visual depth features, the corresponding relationship in the process is as follows: ； in, This represents the enhanced visual features, after being encoded, as visual depth-coded features. This refers to the standard Transformer encoder used for single-modal feature encoding; In the step of generating cross-modal cues by bidirectionally complementing the encoded visual depth features and the encoded audio depth features through a cross-modal bidirectional cross-attention encoder in a hierarchical conditional cue generator, and then averaging the features, the corresponding relationship in the process is as follows: ； in, Indicates cross-modal cues, This represents a cross-modal bidirectional cross-attention encoder in a hierarchical conditional cue generator. This represents the depth-coded features of the encoded audio. In the steps of performing layer normalization on the deep encoded features of the encoded text and dynamically weighting them using a multilayer perceptron with sigmoid activation in a hierarchical conditional cue generator to obtain text residual information, and then using preset fused semantic cues, preset sentiment cues, and preset sentiment prior cues to perform layer normalization on the text residual information to construct the initial semantic-sentiment skeleton, the corresponding relationships in the process are as follows: ； in, Represents text residual information. This represents a multilayer perceptron with Sigmoid activation in a hierarchical conditional cue generator. This represents the deep encoding features of the encoded text. Represents the initial semantic-emotional skeleton. This indicates a pre-defined fusion semantic prompt. This indicates a pre-set emotional cue. This indicates a pre-set emotional prior cues; In the process of obtaining adaptive dynamic prompts by using the initial semantic-sentiment skeleton as the query, cross-modal cues and textual residual information as key-value pairs, processing through a multi-head attention mechanism and undergoing layer normalization to stabilize features, the corresponding relationship in the process is as follows: ； in, This indicates an adaptive dynamic suggestion; In the step of adjusting the weights of the initial semantic-sentiment skeleton and adaptive dynamic cues based on the text integrity score to obtain the proxy features of the text modality, the corresponding relationship in the process is as follows: ； in, Proxy features representing text modalities.

6. The robust multimodal sentiment analysis method based on principal modality enhancement and multi-stage fusion according to claim 5, characterized in that, In step 4, a weighted fusion processor is used to combine the text integrity score with the enhanced text features and the proxy features of the text modality to obtain the enhanced main modality features. The corresponding relationship in this process is as follows: ； in, This represents the enhanced main modal features.

7. The robust multimodal sentiment analysis method based on dominant modality enhancement and multi-stage fusion according to claim 6, characterized in that, In step 5, the enhanced master modality features, along with the visual and audio unified features from the modality unified features, are input into the coarse-to-fine cross-modal fusion module. They undergo three stages of processing: modality alignment, redundancy filtering, and sentiment refinement, to obtain the global sentiment features. Specifically, the steps include: The enhanced master modality features, along with the visual and audio unified features from the modality unified features, are input into the coarse-to-fine cross-modal fusion module. Using the Transformer encoder, the visual and audio features from the modality unified features are aligned to the unified dominant modality space to obtain coarse-grained visual features and coarse-grained audio features. Layer normalization is performed on the enhanced main modality features to obtain coarse-grained text features; We perform weighted summation on coarse-grained visual features, coarse-grained audio features, and coarse-grained text features to fuse the features and generate global coarse-grained cross-modal features. Using a multilayer perceptron in a coarse-to-fine cross-modal fusion module, global coarse-grained cross-modal features are compressed and reconstructed to generate reconstructed features; at the same time, redundancy is filtered through a preset temperature smoothing learnable gating to obtain bottleneck features after redundancy filtering. The bottleneck features that have been redundantly filtered are sequentially processed by global average pooling and layer normalization to obtain global features. The global features are then processed by a multilayer perceptron in the cross-modal fusion module from coarse to fine to generate sample-level dynamic weights. A residual connection is constructed between the global coarse-grained cross-modal features and the bottleneck features after redundancy screening, and dynamic weighted fusion calculation is performed by combining sample-level dynamic weights to generate medium-grained cross-modal features. Bidirectional attention computation is performed between medium-granularity cross-modal features and coarse-granular text features to obtain attention-enhanced features; Attention-enhanced features and medium-granularity cross-modal features are enhanced by residual connections and multilayer perceptrons to obtain fine-granularity cross-modal features; The linear layers in the coarse-to-fine cross-modal fusion module are used to calculate the emotional attention weight of each token in the fine-grained cross-modal features using Softmax normalization. Then, the tokens in the fine-grained cross-modal features are weighted and summed to obtain the global emotional features.

8. The robust multimodal sentiment analysis method based on dominant modality enhancement and multi-stage fusion according to claim 7, characterized in that, Using the Transformer encoder, the visual and audio features in the modality unification feature are aligned to the unified dominant modality space, respectively, to obtain coarse-grained visual features and coarse-grained audio features; The enhanced main modality features are subjected to layer normalization to obtain coarse-grained text features. The corresponding relationship in this process is as follows: ； in, Indicates coarse-grained visual features. Represents coarse-grained audio features. Represents coarse-grained text features; In the step of weighted summation of coarse-grained visual features, coarse-grained audio features, and coarse-grained text features to fuse features and generate global coarse-grained cross-modal features, the corresponding relationship in the process is as follows: ； in, Represents global coarse-grained cross-modal features. This represents the weighted fusion coefficient of coarse-grained multimodal features. Indicates the first Coarse-grained features corresponding to each mode; In the step of using a multilayer perceptron in a coarse-to-fine cross-modal fusion module to compress and reconstruct global coarse-grained cross-modal features to generate reconstructed features; and simultaneously using a preset temperature-smoothing learnable gating system for redundancy filtering to obtain bottleneck features after redundancy filtering, the corresponding relationship in the process is as follows: ； in, Indicates the first Reconstructed features corresponding to each modality This represents a multilayer perceptron used for feature reconstruction. A multilayer perceptron representing feature compression. Indicates the first The temperature smoothing learnable gating coefficients corresponding to each mode This represents the Sigmoid activation function. Indicates the first The learnable temperature parameters corresponding to each mode Indicates the first Bottleneck features of each modality after redundancy screening; In the process of sequentially performing global average pooling and layer normalization on the bottleneck features after redundancy screening to obtain global features, and then generating sample-level dynamic weights for the global features through a multilayer perceptron in the coarse-to-fine cross-modal fusion module, the corresponding relationship is as follows: ； in, Represents global features. This indicates a global average pooling operation. Indicates the first The sample-level dynamic weights corresponding to each modality Represents the normalized exponential function, This represents a multilayer perceptron in a coarse-to-fine cross-modal fusion module; In the steps of constructing residual connections between global coarse-grained cross-modal features and bottleneck features after redundancy screening, and performing dynamic weighted fusion calculations using sample-level dynamic weights to generate medium-grained cross-modal features, the corresponding relationships in the process are as follows: ； in, Indicates medium-granularity cross-modal characteristics; In the step of performing bidirectional attention computation between medium-granularity cross-modal features and coarse-grained text features to obtain attention-enhanced features, and then enhancing the attention-enhanced features and medium-granularity cross-modal features through residual connections and a multilayer perceptron to obtain fine-granularity cross-modal features, the corresponding relationships in the process are as follows: ； in, This indicates the enhanced attentional features after bidirectional attention. Represents fine-grained cross-modal characteristics; In the steps of calculating the sentiment attention weight of each token in the fine-grained cross-modal features through the linear layer in the coarse-to-fine cross-modal fusion module and Softmax normalization, and then performing a weighted summation of the tokens in the fine-grained cross-modal features to obtain the global sentiment features, the corresponding relationship in the process is as follows: ； in, Indicates attention weights, Indicates the index of the token. This represents the feature tensor slicing operation. Indicates global sentiment characteristics. This represents the total number of tokens.

9. The robust multimodal sentiment analysis method based on principal modality enhancement and multi-stage fusion according to claim 8, characterized in that, In step 6, a loss function is constructed based on the complete features to optimize the sentiment prediction model, resulting in an optimized sentiment prediction model. This process includes the following steps: Obtain the true integrity label corresponding to the text integrity score; Obtain the true and complete features of each modality; Obtain the real sentiment labels corresponding to the sentiment scores; Construct an integrity check loss based on the text integrity score and the true integrity label; Construct a reconstruction loss by recovering complete features from noisy input and comparing them with the true complete features; A sentiment prediction loss is constructed based on the sentiment probability distribution and the true sentiment labels; A sentiment guidance loss is constructed based on the sentiment score predicted by the text proxy features and the real sentiment label; A single-modal information bottleneck reconstruction loss is constructed based on bottleneck features, reconstruction features, and coarse-grained features. The total information bottleneck loss is calculated using the single-modal information bottleneck reconstruction loss. An inter-modal mutual information loss is constructed based on the cosine similarity between text global features, audio global features, and visual global features. The sum of the total information bottleneck loss and the inter-modal mutual information loss is used as the exclusive constraint loss of the fusion module. By utilizing integrity check loss, reconstruction loss, sentiment prediction loss, sentiment guidance loss, and the specific constraint loss of the fusion module, the sentiment prediction model is jointly optimized to obtain an optimized sentiment prediction model.

10. The robust multimodal sentiment analysis method based on principal modality enhancement and multi-stage fusion according to claim 9, characterized in that, Based on the text integrity score and the true integrity label, an integrity check loss is constructed. The corresponding relationship in the process is as follows: ； in, Indicates loss due to integrity check. This indicates the number of samples in the training set. Indicates the sample index. Indicates the first Predicted integrity score for each sample Indicates the first The integrity score of each sample and its true label. The squared L2 norm of a vector; In the step of recovering complete features from noisy input and constructing the reconstruction loss from the true complete features, the corresponding process has the following relationship: ； in, Indicates the reconstruction loss. Indicates the first The first sample The true features of a modality Indicates the first The first sample Reconstruction features of each modality; In the step of constructing the sentiment prediction loss based on the sentiment probability distribution and the true sentiment labels, the corresponding relationship in the process is as follows: ； in, This represents the final emotional score. This represents the true label of the sample. Indicates the first The final sentiment prediction results for each sample; In the step of constructing the sentiment guidance loss based on the sentiment score predicted by the text proxy features and the real sentiment label, the corresponding process has the following relationship: ； in, Indicates the predicted sentiment score, This indicates a loss of emotional guidance. Indicates the first Predicted sentiment scores for each sample; In the steps of constructing the single-modal information bottleneck reconstruction loss based on bottleneck features, reconstruction features, and coarse-grained features, and calculating the total information bottleneck loss using the single-modal information bottleneck reconstruction loss, the corresponding relationships in the process are as follows: ； in, This represents the total information bottleneck loss. This represents the mean squared error loss function. This represents the L2 regularization term. This represents the total information bottleneck loss. A parameter representing the total information bottleneck loss; In the step of constructing intermodal mutual information loss based on the cosine similarity of text global features, audio global features, and visual global features, the corresponding relationship in the process is as follows: ； in, This represents the intermodal information loss. The weighting coefficients represent the intermodal mutual information loss. The cosine similarity function represents the degree of cross-modal feature alignment.

Images