A multi-modal sentiment analysis method for a missing modal scene

Abstract

This invention provides a multimodal sentiment analysis method for scenarios with missing modalities. First, it acquires data containing text, audio, and video, and determines whether the text modality is missing. If complete, it extracts real features; if missing, it generates pseudo-text based on audio and cue words. It utilizes CTC to align multimodal features, calculates lexical-level similarity through a cross-modal completion module, generates a mask, and corrects the text representation. A bidirectional selective state-space model, Mamba, is constructed, sharing parameters to capture long-range sentiment dynamics and enhance the text. Starting with the simulated representation as the fusion starting point, discriminative information is injected layer by layer through an attention module to obtain unified fusion features. Finally, the data is input into a sentiment prediction module, which combines sentiment regression, knowledge distillation, and representation similarity constraints for joint optimization, outputting the sentiment result. This method maintains stable discrimination even with missing or incomplete text, effectively reduces cross-modal semantic shift and noise interference, and improves the robustness and generalization ability of the analysis.

Description

Technical Field

[0001] This invention belongs to the field of natural language processing, and in particular relates to a multimodal sentiment analysis method for modality-deficient scenarios. Background Technology

[0002] Multimodal sentiment analysis aims to integrate heterogeneous information such as text, audio, and video to model and identify the intensity, polarity, or state of a user's emotions. Compared to unimodal sentiment analysis, multimodal sentiment analysis can simultaneously utilize complementary cues such as language content, voice tone, facial expressions, and behavioral actions, thus possessing significant application value in scenarios such as intelligent human-computer interaction, intelligent customer service, public opinion analysis, psychological health assessment, and educational feedback.

[0003] In real-world applications, multimodal sentiment analysis systems often struggle to consistently obtain complete and high-quality multimodal input. Factors such as automatic speech recognition errors, environmental noise, privacy constraints, unstable communication conditions, and data acquisition costs can cause text modalities to be missing, semantically incomplete, or semantically degraded; audio and video modalities may also be affected by noise, occlusion, low resolution, or unstable sampling.

[0004] Existing multimodal sentiment analysis methods typically treat text as the core semantic carrier and rely on a fixed fusion structure to integrate features from text, audio, and video. When textual information is unavailable or semantic quality degrades, the model's ability to model sentiment cues is significantly weakened, leading to unstable prediction results.

[0005] To address the problem of missing modalities, existing methods have attempted to improve robustness using strategies such as modality dropout training, modality reconstruction, or generative completion. However, these methods still have the following shortcomings: First, in the case of missing text, relying solely on audio or video to generate text semantics can easily introduce noise, leading to cross-modal semantic shifts. Second, existing completion strategies often focus on static semantic alignment, neglecting the dynamic emotional evolution patterns inherent in audio and video. Third, fixed-weight or shallow fusion methods struggle to adaptively adjust the fusion strategy based on the information quality of each modality in different samples. Fourth, in modality-missing scenarios, the lack of effective cross-modal supervision signals leads to unstable distribution of learned representations.

[0006] Therefore, there is an urgent need for a multimodal sentiment analysis method that can achieve semantic compensation, temporal enhancement, dynamic fusion, and knowledge transfer under conditions of modality loss or semantic incompleteness, so as to improve the robustness and generalization ability of the model in complex real-world environments. Summary of the Invention

[0007] Purpose of the invention: The technical problem to be solved by this invention is to address the shortcomings of existing technologies by providing a multimodal sentiment analysis method for modality-deficient scenarios, comprising the following steps:

[0008] Step 1: Obtain multimodal sentiment analysis data, which includes text modality, audio modality, and video modality, and determine the modality integrity status and text modality missing status in the input sample;

[0009] Step 2, feature extraction and text representation simulation: When the text modality is complete, extract text features, audio features and video features respectively; when the text modality is missing, generate pseudo-text representation based on audio features and preset prompt words to construct text features consistent with the semantic distribution of real text;

[0010] Step 3, perform cross-modal temporal alignment and semantic completion: Align text, audio and video features to a unified time scale, calculate the word-level correlation between audio, text and visual, and text, use hard threshold mask to filter out weakly related word pairs, and perform semantic correction on the text representation based on audio and video information;

[0011] Step 4, Constructing the simulated modal representation: Model the stable semantic representation and uncertainty of each modal feature in the latent Gaussian distribution space, generate fusion weights based on the uncertainty, and obtain the initial representation of the simulated modality;

[0012] Step 5: Construct a bidirectional selective state-space model based on audio and video. Mamba Temporal Modeling Module: The temporal modeling module uses audio and video modalities as the driving source of temporal semantics, employs a bidirectional structured state-space model to capture forward and backward temporal dynamics, and transfers the temporal evolution patterns in audio and video to text representation to obtain enhanced text features.

[0013] Step 6, construct the attention dynamic fusion module: taking the initial representation of the simulated modality as the starting point of fusion, the audio representation, enhanced text representation and video representation are mapped to key vectors and value vectors respectively, the cross-modal attention injection representation is calculated layer by layer, and weighted residual fusion is performed according to the modal weights to obtain the final multimodal fusion feature;

[0014] Step 7, perform sentiment prediction and model optimization: input the final multimodal fusion features into the sentiment prediction module, output the sentiment intensity or sentiment polarity prediction results, and use sentiment valence regression loss, simulated knowledge distillation loss and representation similarity loss to jointly train the model and generate sentiment prediction results.

[0015] Step 2 includes:

[0016] When the text modality is complete, the large language model Vicuna is used to encode the text input, the pre-trained speech model WavLM is used to encode the audio input, and the multi-scale attention network MANet is used to encode the video input, thereby obtaining text features. Audio features and video features :

[0017] ,

[0018] ,

[0019] ,

[0020] in, This indicates the original text input. Indicates the raw audio input. Indicates the original video input;

[0021] When the text modality is missing, audio features will be used. The pseudotext representation is mapped to the input space of a large language model and concatenated with the lexical representation of the preset prompt word P before being input into the large language model. :

[0022] ,

[0023] Here, LLM represents the reasoning process of a large language model based on a generative mechanism, cat() represents the feature concatenation operation, proj() represents the pre-trained projection network, and tokenizer() represents the prompt word encoding operation.

[0024] Step 3 includes:

[0025] Connectivity-Temporal Classification (CTC) is used to temporally align video, text, and audio features to obtain aligned visual features. Text features and audio features :

[0026] ,

[0027] Let the vector of the i-th audio word be represented as The vector representation of the j-th text word is as follows: After L2 normalization, the similarity between the i-th audio and the j-th text is... Represented as:

[0028] ,

[0029] Where τ is the temperature coefficient; exp is the natural exponential function. Let L represent the vector of the kth text word, L represent the length of the text word sequence, and <·,·> represent the vector dot product operation;

[0030] Based on hard threshold Generate a binary mask :

[0031] ,

[0032] By combining audio and visual modal information to perform semantic correction on text features, an enhanced text representation is obtained. :

[0033] ,

[0034] Here, ⊙ represents element-wise multiplication. Represents the audio-text similarity matrix. Represents the audio-text mask matrix. Represents the visual-text similarity matrix. Represents a visual-text mask matrix.

[0035] Step 4 includes:

[0036] The stable semantic representation of each mode is modeled based on the latent Gaussian distribution space, and the stable semantic representation of mode m is denoted as... Let the uncertainty measure of mode m be denoted as Calculate the fusion weights of mode m in the nth sample based on the uncertainty. :

[0037] ,

[0038] Where n represents the sample index, m represents the current modality, and M represents the set of modalities participating in the fusion. ;

[0039] The initial representation of the simulated modes is obtained based on the fusion weights of each mode. :

[0040] ,

[0041] Wherein, the initial representation of the simulated mode It is used to mitigate instability caused by single-modal noise, semantic degradation, or modality loss in subsequent fusion stages.

[0042] Step 5 includes:

[0043] The bidirectional selective state-space model Mamba temporal modeling module includes a forward structured state-space model and a reverse structured state-space model; for any mode m, the input features at time k... Forward state update and reverse state update are represented as follows:

[0044] ,

[0045] ,

[0046] in, This represents the input characteristics of mode m at time k. This represents the positive hidden state of mode m at time k. This represents the reverse hidden state of mode m at time k. This represents the positive output of mode m at time k. This represents the inverse output representation of mode m at time k; Indicates the positive state transition moment. Represents the reverse state transition matrix; This represents the forward input mapping matrix corresponding to mode m. This represents the inverse input mapping matrix corresponding to mode m; This represents the positive output mapping matrix corresponding to mode m. This represents the inverse output mapping matrix corresponding to mode m;

[0047] By using the output of audio and video features encoded through the bidirectional selective state-space model Mamba, the text representation is back-guided and iteratively updated to obtain enhanced text features that integrate audio and video sentiment cues. :

[0048] ,

[0049] ,

[0050] ,

[0051] Among them, the bidirectional selective state-space model This indicates a Mamba encoder that contains both forward and backward modeling paths. Indicates audio features Encoded audio timing representation; Indicates video features Encoded video timing representation; This represents the text representation enhanced with audio timing information. The text representation is enhanced by video temporal information; Mean represents the equal-weighted fusion operation performed on the enhancement results from different cross-modal methods.

[0052] Step 6 includes:

[0053] Initial representation of simulated modes As the fusion representation at layer 0, it is updated layer by layer by stacking the attention dynamic fusion module of layer R; in the... In the layer, with As a query vector, the audio representation Enhanced text representation And the video indicates They serve as their respective keys and values; for any modality t represents text, a represents audio, v represents video, and cross-modal attention injection represents... for:

[0054] ,

[0055] Where T represents transpose. Represents the dimension of the key vector. Indicates the first Layer query vector, The key vector representing mode m. The value vector representing mode m. () represents the normalized exponential function;

[0056] The cross-modal attention injection representations of each modality are concatenated and then input into a multilayer perceptron (MLP), and then processed... Normalization yields the modal weights :

[0057] ,

[0058] in, , Indicates the first Layer audio modal weights; Indicates the first Layered text modal weights; Indicates the first Layer video modal weights, and =1;

[0059] Weighted residual fusion is performed based on modal weights to obtain the first... Layer fusion representation :

[0060] ,

[0061] in, The representation layer normalization operation, finally the first Layer output As a feature of multimodal fusion.

[0062] Step 7 includes:

[0063] During the training phase, modality complete stream and text missing stream are constructed simultaneously. The modality complete stream serves as the teacher stream, and the text missing stream serves as the student stream. The modality complete stream and text missing stream share the network structure but have different input modalities.

[0064] Emotional Valence Regression Loss Represented as:

[0065] ,

[0066] in, This represents the total number of training samples, and n represents the sample index. This represents the true sentiment valence label for the nth sample; This represents the model prediction value for the nth sample;

[0067] Simulated knowledge distillation loss To ensure consistency between the simulated text representation generated from the missing text stream and the real text representation under the modal complete stream:

[0068] ,

[0069] in, This represents the time-aligned text feature of the t-th sample after the gradient stops in the modal complete flow; This represents the time-aligned text feature of the t-th sample after generation and correction in the missing text stream; This represents the temporal enhanced text feature of the t-th sample after the gradient stops in the modal complete flow; This represents the temporal augmented text feature of the t-th sample in the missing text stream. This indicates that the gradient backpropagation operation has been stopped;

[0070] Represents similarity loss To ensure consistency between the fused representations of the complete modal stream and the text-missing stream before the final regression head:

[0071] ,

[0072] in, This represents the final fused representation of the nth sample under modal integrity conditions. This represents the final fused representation of the nth sample under the condition of missing text.

[0073] Total loss function Represented as:

[0074] .

[0075] In step 7, the sentiment prediction module includes a regression head, which uses the final multimodal fusion features. The method takes the input as input and outputs the sentiment polarity category and sentiment intensity prediction accuracy. It is applicable to multimodal sentiment analysis scenarios with missing text modalities, incomplete text semantics, and degraded audio or video modal quality.

[0076] The present invention also provides an electronic device, including a processor and a memory, the memory storing program code that, when executed by the processor, causes the processor to perform the steps of the method.

[0077] The present invention also provides a storage medium storing a computer program or instructions that, when the computer program or instructions are run on a computer, execute the steps of the method described.

[0078] Compared with the prior art, the present invention has the following beneficial effects:

[0079] (1) By using text representation simulation and cross-modal completion mechanism, a stable text representation can still be constructed when the text is missing or the semantics are incomplete, thus alleviating cross-modal semantic shift.

[0080] (2) By using bidirectional Mamba temporal modeling based on audio and video, we can make full use of long-range emotional dynamics in non-textual modalities to improve the ability to recognize emotions under text-deficient conditions.

[0081] (3) Through the attention dynamic fusion mechanism, the fusion weights are adaptively allocated according to the reliability of the sample-level modality to suppress the interference of noisy modalities on the final representation.

[0082] (4) By learning from the self-distillation of the complete modality stream to the missing text stream, the difference in representation distribution under different input conditions is reduced, and the robustness of the model is improved.

[0083] (5) The method of the present invention demonstrates competitive performance on the MOSI and MOSIE datasets, verifying its effectiveness and advantages. Attached Figure Description

[0084] Figure 1 This is a flowchart of the method of the present invention.

[0085] Figure 2 This is a model framework diagram of the system of the present invention.

[0086] Figure 3 This is a flowchart of the cross-modal completion module of the present invention.

[0087] Figure 4 This is a flowchart of the timing modeling of the bidirectional selective state-space model of the present invention.

[0088] Figure 5 This is a flowchart illustrating the bidirectional selective state-space model structure of the present invention.

[0089] Figure 6 This is a flowchart of the attention dynamic fusion module of the present invention. Detailed Implementation

[0090] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments, and the advantages of the present invention in the above and / or other aspects will become clearer.

[0091] This embodiment provides a multimodal sentiment analysis method for modality-deficient scenarios, such as... Figure 1 As shown, the method of the present invention includes steps 1 to 7. The overall architecture of the system model provided in this embodiment is as follows. Figure 2 As shown, the model consists of four modules: (1) feature extraction module; (2) cross-modal completion module; (3) bidirectional Mamba temporal modeling module; and (4) attention dynamic fusion module.

[0092] The method includes the following steps:

[0093] Step 1: Obtain multimodal sentiment analysis data, which includes text modality, audio modality, and video modality, and determine the modality integrity status and text modality missing status in the input samples;

[0094] Step 2, feature extraction and text representation simulation: When the text modality is complete, extract text features, audio features and video features respectively; when the text modality is missing, generate pseudo-text representation based on audio features and preset prompt words to construct text features consistent with the semantic distribution of real text;

[0095] Step 3, perform cross-modal temporal alignment and semantic completion: Align text, audio and video features to a unified time scale, calculate the token-level correlation between audio-text and visual-text, use hard threshold mask to filter out weakly correlated token pairs, and perform semantic correction on the text representation based on audio and video information;

[0096] Step 4, Constructing the simulated modal representation: Model the stable semantic representation and uncertainty of each modal feature in the latent Gaussian distribution space, generate fusion weights based on the uncertainty, and obtain the initial representation of the simulated modality;

[0097] Step 5: Construct a bidirectional Mamba temporal modeling module based on audio and video: Using audio and video modalities as the driving source of temporal semantics, a bidirectional structured state space model is used to capture forward and backward temporal dynamics, and the temporal evolution patterns in audio and video are transferred to text representation to obtain enhanced text features;

[0098] Step 6, construct the attention dynamic fusion module: taking the initial representation of the simulated modality as the starting point of fusion, using audio, enhanced text and video features as keys and values, calculate the cross-modal attention injection representation layer by layer, and perform weighted residual fusion according to the modal weights to obtain the final multimodal fusion features;

[0099] Step 7, perform sentiment prediction and model optimization: input the final multimodal fusion features into the sentiment prediction module, output the sentiment intensity or sentiment polarity prediction results, and use sentiment valence regression loss, simulated knowledge distillation loss and representation similarity loss to jointly train the model and generate sentiment prediction results.

[0100] Step 1 includes:

[0101] In multimodal sentiment analysis research, data collection is a crucial step in model training, validation, and testing. To ensure the scientific rigor of the research and the applicability of the data, this invention selects two representative and high-quality multimodal datasets: MOSI and MOSEI. The MOSI dataset was proposed by Amir Zadeh et al. in 2016 in the journal IEEE Intelligent Systems, and described in detail in their paper "Amir Zadeh, Rowan Zellers, Eli Pincus, Louis-Philippe Morency, Multimodal sentiment intensity analysis in videos: Facial gestures and verbal messages, IEEE Intell. Syst. 31(6)(2016) 82–88". The MOSI dataset contains 93 video clips from YouTube, covering a wide range of sentiment expressions, including positive, negative, and neutral emotions. The dataset contains a total of 2199 samples, as shown in Table 1. Each sample provides three modal features: text, audio, and video. The samples are labeled within the range of [-3, +3] based on their sentiment intensity; higher values ​​indicate more positive sentiment, while lower values ​​indicate more negative sentiment. The MOSEI dataset is an extended version of the MOSI dataset, proposed by Amir Ali Bagher Zadeh et al. in their 2018 paper, "Amir Ali Bagher Zadeh, Paul Pu Liang, Soujanya Poria, Erik Cambria, Louis-Philippe Morency, Multimodal language analysis in the wild: CMU-MOSEI dataset and interpretable dynamic fusion graph," in: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, Vol. 1, 2018, pp. 2236–2246. MOSEI is currently the largest publicly available multimodal sentiment analysis dataset, containing 22,856 samples, each labeled with sentiment intensity in the range of [-3, +3].

[0102] Table 1

[0103]

[0104] Step 2 includes:

[0105] The feature extraction module aims to extract high-level semantic features from different modalities, providing a unified representation foundation for subsequent multimodal fusion and sentiment modeling. This invention employs differentiated feature extraction and representation construction strategies to address different scenarios, including whether text modalities are missing.

[0106] When the text modality is fully available, this invention employs mature and stable pre-trained models to extract features from the three modalities respectively. Specifically, the text modality... The Vicuna speech-language model, based on self-supervised learning, is used for feature encoding to obtain text representations with contextual semantic information; audio modality The large language model WavLM is then used to extract its features; visual modality Modeling is performed using the MANet network to capture temporal facial expressions and motion features related to emotion in videos. The feature extraction process described above can be formally represented as follows:

[0107] ,

[0108] ,

[0109] ,

[0110] When text modalities are missing, the model cannot directly extract semantic information from the text, significantly impacting the performance of multimodal sentiment analysis. To address this issue, this invention, based on the idea of ​​text representation simulation using large language models, utilizes audio modal information to generate pseudo-text representations consistent with the semantics of real text, thereby achieving unified representation in both cases where text modalities are present and absent. Specifically, audio features... Mapped to the input space of a large language model and matched with predefined prompt words. The data is then concatenated and used as input for Vicuna. The specific formula is as follows:

[0111] ,

[0112] in, The expression represents the reasoning process of a large language model based on a generative mechanism, where `cat` represents the feature concatenation operation and `proj` represents the pre-trained projection network. Through this text representation simulation mechanism, this invention can still construct a representation consistent with the semantic distribution of real text even when text modalities are missing, thereby effectively narrowing the semantic gap between the presence and absence of text modalities and providing a unified and stable feature input for subsequent multimodal fusion and sentiment prediction.

[0113] Step 3 includes:

[0114] In scenarios with missing text, text representations are often affected by generation errors and modal uncertainties, potentially leading to semantic shifts or instability. This uncertainty directly weakens the reliability of multimodal sentiment modeling, especially in fusion frameworks that rely on text as the dominant modality. To mitigate these issues, this invention introduces a cross-modal completion module, aiming to utilize more stable and continuous sentiment cues from audio and visual modalities to perform cross-modal semantic compensation and correction on text representations. This module, from the perspective of semantic consistency, implicitly modifies text representations through non-textual modalities, thereby improving the robustness of text features under incomplete modal conditions.

[0115] like Figure 3 As shown, firstly, to eliminate the differences in temporal scale and feature dimension among different modalities, this invention adopts a temporal alignment strategy based on Connectionist Temporal Classification (CTC) to uniformly model audio, visual, and text features. Specifically, the multimodal features obtained after alignment are represented as follows:

[0116] ,

[0117] in, These represent the time-aligned visual, textual, and audio features, respectively.

[0118] Building upon this, this invention uses audio and visual modalities as semantic reference sources, calculating their token-level relevance with the text modal to characterize the consistency of cross-modal sentiment cues. Taking audio-text as an example, let the i-th audio token be represented as... The j-th text token is represented as After L2 normalization, the similarity is defined as:

[0119] ,

[0120] in Temperature coefficient. To suppress weakly correlated or noisy token pairs, this invention introduces a hard thresholding strategy to generate a binary mask. Hard thresholding... The specific formula is as follows:

[0121] ,

[0122] Finally, by combining audio and visual modal information, and using the visual modality to enhance the text modality, a semantically corrected text representation is obtained:

[0123] ,

[0124] in, This represents element-wise multiplication. and These represent the mask matrix and similarity matrix between the visual and textual modalities, respectively. Through a cross-modal completion mechanism, the model can effectively compensate for and correct textual semantics by fully utilizing emotional cues from the audio and visual modalities when the textual modality is missing or semantically incomplete.

[0125] Step 4 includes:

[0126] To further improve the stability of multimodal representation in incomplete modal scenarios, this invention uses simulated modal representation. This method originates from modeling stable representations of various modal features in a latent Gaussian distribution space. It utilizes a variational inference framework to learn the distribution parameters of different modal features, where the mean term of the Gaussian distribution is considered the stable semantic representation of the modality, while the variance term characterizes the modality's uncertainty. Based on this modal uncertainty, the stable representations of each modality are weighted and fused to obtain a joint stable multimodal representation, i.e., a simulated modal representation.

[0127] ,

[0128] ,

[0129] in, and Let m represent the stable semantic representation of mode m and its uncertainty measure, respectively. These are the corresponding fusion weights. The simulated modal representation constructed in this way... It can alleviate the problems of noise and missing features in single-modal analysis to a certain extent, and provide a more stable and robust feature base for subsequent multimodal sentiment analysis.

[0130] Step 5 includes:

[0131] Bidirectional Mamba timing modeling. For example... Figure 4 and Figure 5 As shown, this invention employs a bidirectional Mamba structure as the backbone for long-range temporal modeling of audio and video. This method uses audio and visual modalities as the primary temporal modeling objects, simultaneously capturing the forward and backward temporal dynamics of the sequence based on a bidirectional structured state-space model (Bi-Mamba). Furthermore, through a cross-modal parameter sharing mechanism, it introduces the temporal evolution patterns from non-textual modalities into the text modeling process, thereby achieving temporal enhancement and semantic correction of the text representation.

[0132] Specifically, this invention employs a bidirectional Mamba structure as the backbone network for temporal modeling, with its core consisting of a structured state-space model (SSM) to simultaneously capture forward and backward temporal dynamics. For each mode... Bidirectional Mamba is modeled using a set of forward and backward SSM parameters, respectively. To fully incorporate the emotional temporal structure in the audio and visual modalities, this invention shares a bidirectional state transition matrix between different modalities. and This allows the text modality to inherit the shared temporal dynamics inherent in both the audio and visual modalities. Simultaneously, the input mapping matrix... Mapping matrix with output Maintain modal independence to preserve the feature representation capabilities of each modality. The state update process can be formally represented as:

[0133] ,

[0134] ,

[0135] The former is a forward SSM, and the latter is a reverse SSM. This represents the input characteristics of mode m at time k. and This represents the hidden state of mode m at time k. and This represents the output of mode m at time k. and Represents the forward and reverse state transition matrices. and This represents the forward and backward input mapping matrices related to mode m. and This represents the forward and backward output mapping matrices related to mode m.

[0136] This invention uses audio and visual modalities as the primary semantic drivers, leveraging their bidirectional Mamba output to guide and iteratively update the text representation. Specifically, the bidirectional Mamba-encoded representations of the audio and visual modalities are aggregated and used to correct the hidden state representation of the text modalities, thereby obtaining enhanced text features that fuse audio-visual emotional cues.

[0137] ,

[0138] ,

[0139] ,

[0140] in, This indicates that equal-weighted fusion is performed on results from different cross-modal augmentations. The resulting text representation is obtained through this method. It can effectively compensate for semantic biases caused by incomplete text modalities and provide a more robust text semantic foundation for subsequent multimodal sentiment modeling.

[0141] Step 6 includes:

[0142] like Figure 6 As shown, the present invention uses simulated modal initialization. As the starting point for fusion, and through stacked L-layer attention-based dynamic fusion modules, multimodal information is injected and updated layer by layer, enabling the fusion features to adaptively absorb discriminative information from audio, text, and video modalities at different fusion stages, thereby gradually enhancing their emotional expression capabilities. In the... During the layer fusion process, As a query, audio, text, and video modal features Use these as keys and values ​​respectively to construct cross-modal attention interactions. For any modality... Its corresponding cross-modal attention injection representation is defined as:

[0143] ,

[0144] Since different modalities contribute significantly differently to sentiment discrimination across different samples and fusion stages, this invention further introduces a modality weighting coefficient. This is used to adaptively adjust the injection intensity of information from each modality into the fused modality. In the weight modeling phase, the cross-modal injection representation of each modality is first... The data is then concatenated and fed into a lightweight multilayer perceptron (MLP) for nonlinear mapping to generate corresponding modal importance scores. Subsequently, through... The function normalizes the scores to obtain the modal weights:

[0145] ,

[0146] in ,and =1. This weight allocation mechanism can adaptively highlight more reliable modalities based on the multimodal consistency and discriminability of the current samples, while suppressing modal features with high noise or uncertainty. After obtaining the injected representation of each modality and its corresponding weight, this invention uses a weighted residual fusion method to update the fused features. Specifically, the first... The fusion feature of the layer is defined as:

[0147] ,

[0148] in, The representation layer normalization operation is used to mitigate feature distribution shifts and stabilize the deep fusion process. This update strategy, while preserving the original stable semantics of the surrogate modality, injects discriminative information from different modalities layer by layer, resulting in a final... It can fully integrate multimodal contexts.

[0149] Step 7 includes:

[0150] Sentiment Prediction and Joint Training: To effectively address two practical application scenarios—modal completeness and textual modality missing—this invention designs a self-distillation learning framework. Its core idea is to simultaneously construct complete modal data streams and textual missing data streams during the training phase. Utilizing multimodal semantic knowledge learned under complete modal conditions, self-supervised knowledge transfer is performed on textual missing scenarios, thereby improving the model's robustness and generalization ability under incomplete modal conditions. In this framework, the complete modal stream acts as the "teacher," and the textual missing stream acts as the "student." Both share a network structure but have different input modalities. Through the self-distillation mechanism, the model can effectively transfer the semantic information contained in the real text modality to the text representation under missing text conditions, achieving consistent modeling of cross-modal semantic knowledge.

[0151] This invention designs a set of complementary loss functions from three levels: task supervision, modal knowledge distillation, and representation consistency constraints. The total loss function is defined as: ,

[0152] Step 7.1: The sentiment analysis task is modeled as a continuous sentiment valence prediction problem; therefore, this invention uses mean squared error as the basic task loss.

[0153] ,

[0154] in, and and represent the true sentiment valence label and the model prediction value for the nth sample, respectively. This loss directly constrains the model's regression accuracy for sentiment intensity and polarity, and is the core supervision signal in the overall training objective.

[0155] Step 7.2, to improve the model's expressive ability under text modality missing conditions, this invention introduces Modal Knowledge Distillation Loss (SKD) to constrain the consistency between the simulated text representation generated in the missing text scenario and the real text representation under the complete modality condition in the semantic space. This invention treats the real text representation obtained from the complete modality stream as the distillation target and truncates its gradient when calculating the loss to ensure that the distillation process only generates backward gradients for the text missing stream. The Modal Knowledge Distillation Loss is defined as:

[0156] ,

[0157] in, and Representing text features under a complete modal flow, and This represents text features in the missing text stream. This multi-level distillation strategy enables effective knowledge transfer at both the global semantic and temporal dynamic levels.

[0158] Step 7.3: After completing the simulated knowledge distillation, it is still necessary to ensure that the model is robust to modality loss at the decision-making level. To this end, this invention further introduces representation similarity loss to constrain the fused representations of the complete modality stream and the text-missing stream to be as consistent as possible before entering the final regression head.

[0159] ,

[0160] in, This represents the final fused representation of the nth sample under modal integrity conditions. This represents the final fused representation of the nth sample under the condition of missing text.

[0161] This loss helps to reduce the differences in representation distribution under different input modalities, thereby reducing the impact of missing text on prediction performance.

[0162] Through the joint optimization described above, the model can maintain a similar representation distribution and stable sentiment prediction ability under both complete modality and missing text input conditions.

[0163] Baseline Comparison: The following models were selected as baselines for comparison of model performance in this embodiment. (1) ConKI: Specific knowledge representations are injected through external datasets, and hierarchical contrastive learning is combined to separate general knowledge and modality-specific knowledge to improve multimodal sentiment representation. (2) MInD: Multimodal inputs are decomposed into modality-invariant and modality-specific components through shared encoders and private encoders. Information optimization-driven decoupling is used to simplify fusion and improve sentiment analysis performance. (3) FDR-MSA: Sentiment features are divided into public and private parts through feature decoupling, and a dual reconstruction mechanism (single-modal and multimodal reconstruction) is introduced to reduce redundancy and retain key information. (4) KuDA: Dynamic modality attention fusion is guided by sentiment knowledge. The model adaptively selects the dominant modality and adjusts the contribution of each modality to realize a knowledge-guided dynamic attention mechanism. (5) TriDiRA: A triple untangled representation framework is constructed to separate modality-invariant, effective modality-specific, and invalid modality-specific representations from each modality to eliminate interference information and improve the quality of sentiment representation. (6) FMFN: Designs a fine-grained representation learning module to extract key sentiment information, combines token-level cross-modal contrast alignment, bridges the gap between heterogeneous modalities, and improves fine-grained sentiment understanding capabilities. (7) CMHFM: Adopts a cross-modal hierarchical fusion structure, combines multi-task learning, and simultaneously processes single-modal, bimodal, and trimodal tasks to enhance the feature representation of the final sentiment prediction. (8) P-RMF: Uses a surrogate-driven method to map single-modal data to a Gaussian distribution latent space, learns stable representations, and guides robust fusion under missing data by quantifying uncertainty. (9) AMB: A multimodal adaptation model based on BERT, adopts a layer-by-layer fusion strategy to integrate text, visual, and audio features to achieve efficient multimodal sentiment analysis.

[0164] Table 2

[0165]

[0166] Table 3

[0167]

[0168] As shown in Tables 2 and 3, the method proposed in this invention outperforms the performance of all baseline models on the two public datasets, thus verifying the effectiveness of this invention.

[0169] This invention provides a multimodal sentiment analysis method for modality-deficient scenarios. Many methods and approaches exist for implementing this technical solution; the above description is merely a preferred embodiment. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this invention, and these improvements and modifications should also be considered within the scope of protection of this invention. All components not explicitly stated in this embodiment can be implemented using existing technologies.

Claims

1. A multimodal sentiment analysis method for modality-deficient scenarios, characterized in that, Includes the following steps: Step 1: Obtain multimodal sentiment analysis data, which includes text modality, audio modality, and video modality, and determine the modality integrity status and text modality missing status in the input sample; Step 2, feature extraction and text representation simulation: When the text modality is complete, extract text features, audio features and video features respectively; When the text modality is missing, a pseudo-text representation is generated based on audio features and preset prompt words to construct text features consistent with the semantic distribution of real text; Step 3, perform cross-modal temporal alignment and semantic completion: Align text, audio and video features to a unified time scale, calculate the word-level correlation between audio, text and visual, and text, use hard threshold mask to filter out weakly related word pairs, and perform semantic correction on the text representation based on audio and video information; Step 4, Constructing the simulated modal representation: Model the stable semantic representation and uncertainty of each modal feature in the latent Gaussian distribution space, generate fusion weights based on the uncertainty, and obtain the initial representation of the simulated modality; Step 5: Construct a bidirectional selective state-space model based on audio and video. Mamba Temporal Modeling Module: The temporal modeling module uses audio and video modalities as the driving source of temporal semantics, employs a bidirectional structured state-space model to capture forward and backward temporal dynamics, and transfers the temporal evolution patterns in audio and video to text representation to obtain enhanced text features. Step 6, construct the attention dynamic fusion module: taking the initial representation of the simulated modality as the starting point of fusion, the audio representation, enhanced text representation and video representation are mapped to key vectors and value vectors respectively, the cross-modal attention injection representation is calculated layer by layer, and weighted residual fusion is performed according to the modal weights to obtain the final multimodal fusion feature; Step 7, perform sentiment prediction and model optimization: input the final multimodal fusion features into the sentiment prediction module, output the sentiment intensity or sentiment polarity prediction results, and use sentiment valence regression loss, simulated knowledge distillation loss and representation similarity loss to jointly train the model and generate sentiment prediction results.

2. The method according to claim 1, characterized in that, Step 2 includes: When the text modality is complete, the large language model Vicuna is used to encode the text input, the pre-trained speech model WavLM is used to encode the audio input, and the multi-scale attention network MANet is used to encode the video input, thereby obtaining text features. Audio features and video features : ， ， ， in, This indicates the original text input. Indicates the raw audio input. Indicates the original video input; When the text modality is missing, audio features will be used. The pseudotext representation is mapped to the input space of a large language model and concatenated with the lexical representation of the preset prompt word P before being input into the large language model. : ， Here, LLM represents the reasoning process of a large language model based on a generative mechanism, cat() represents the feature concatenation operation, proj() represents the pre-trained projection network, and tokenizer() represents the prompt word encoding operation.

3. The method according to claim 2, characterized in that, Step 3 includes: Connectivity-Temporal Classification (CTC) is used to temporally align video, text, and audio features to obtain aligned visual features. Text features and audio features : ， Let the vector of the i-th audio word be represented as The vector representation of the j-th text word is as follows: After L2 normalization, the similarity between the i-th audio and the j-th text is... Represented as: ， Where τ is the temperature coefficient; exp is the natural exponential function. Let L represent the vector of the kth text word, L represent the length of the text word sequence, and <·,·> represent the vector dot product operation; Based on hard threshold Generate a binary mask : ， By combining audio and visual modal information to perform semantic correction on text features, an enhanced text representation is obtained. : ， Here, ⊙ represents element-wise multiplication. Represents the audio-text similarity matrix. Represents the audio-text mask matrix. Represents the visual-text similarity matrix. Represents a visual-text mask matrix.

4. The method according to claim 3, characterized in that, Step 4 includes: The stable semantic representation of each mode is modeled based on the latent Gaussian distribution space, and the stable semantic representation of mode m is denoted as... Let the uncertainty measure of mode m be denoted as Calculate the fusion weights of mode m in the nth sample based on the uncertainty. : ， Where n represents the sample index, m represents the current modality, and M represents the set of modalities participating in the fusion. ; The initial representation of the simulated modes is obtained based on the fusion weights of each mode. : 。 5. The method according to claim 4, characterized in that, Step 5 includes: The bidirectional selective state-space model Mamba temporal modeling module includes a forward structured state-space model and a reverse structured state-space model; for any mode m, the input features at time k... Forward state update and reverse state update are represented as follows: ， ， in, This represents the input characteristics of mode m at time k. This represents the positive hidden state of mode m at time k. This represents the reverse hidden state of mode m at time k. This represents the positive output of mode m at time k. This represents the inverse output representation of mode m at time k; Indicates the positive state transition moment. Represents the reverse state transition matrix; This represents the forward input mapping matrix corresponding to mode m. This represents the inverse input mapping matrix corresponding to mode m; This represents the positive output mapping matrix corresponding to mode m. This represents the inverse output mapping matrix corresponding to mode m; By using the output of audio and video features encoded through the bidirectional selective state-space model Mamba, the text representation is back-guided and iteratively updated to obtain enhanced text features that integrate audio and video sentiment cues. : ， ， ， Among them, the bidirectional selective state-space model This indicates a Mamba encoder that contains both forward and backward modeling paths. Indicates audio features Encoded audio timing representation; Indicates video features Encoded video timing representation; This represents the text representation enhanced with audio timing information. The text representation is enhanced by video temporal information; Mean represents the equal-weighted fusion operation performed on the enhancement results from different cross-modal methods.

6. The method according to claim 5, characterized in that, Step 6 includes: Initial representation of simulated modes As the fusion representation at layer 0, it is updated layer by layer by stacking the attention dynamic fusion module of layer R; in the... In the layer, with As a query vector, the audio representation Enhanced text representation And the video indicates They serve as their respective keys and values; for any modality t represents text, a represents audio, v represents video, and cross-modal attention injection represents... for: ， Where T represents transpose. Represents the dimension of the key vector. Indicates the first Layer query vector, The key vector representing mode m. The value vector representing mode m. () represents the normalized exponential function; The cross-modal attention injection representations of each modality are concatenated and then input into a multilayer perceptron (MLP), and then processed... Normalization yields the modal weights : ， in, , Indicates the first Layer audio modal weights; Indicates the first Layered text modal weights; Indicates the first Layer video modal weights, and =1; Weighted residual fusion is performed based on modal weights to obtain the first... Layer fusion representation : ， in, The representation layer normalization operation, finally the first Layer output As a feature of multimodal fusion.

7. The method according to claim 6, characterized in that, Step 7 includes: During the training phase, modality complete stream and text missing stream are constructed simultaneously. The modality complete stream serves as the teacher stream, and the text missing stream serves as the student stream. The modality complete stream and text missing stream share the network structure but have different input modalities. Emotional Valence Regression Loss Represented as: ， in, This represents the total number of training samples, and n represents the sample index. This represents the true sentiment valence label for the nth sample; This represents the model prediction value for the nth sample; Simulated knowledge distillation loss To ensure consistency between the simulated text representation generated from the missing text stream and the real text representation under the modal complete stream: ， in, This represents the time-aligned text feature of the t-th sample after the gradient stops in the modal complete flow; This represents the time-aligned text feature of the t-th sample after generation and correction in the missing text stream; This represents the temporal enhanced text feature of the t-th sample after the gradient stops in the modal complete flow; This represents the temporal augmented text feature of the t-th sample in the missing text stream. This indicates that the gradient backpropagation operation has been stopped; Represents similarity loss To ensure consistency between the fused representations of the complete modal stream and the text-missing stream before the final regression head: ， in, This represents the final fused representation of the nth sample under modal integrity conditions. This represents the final fused representation of the nth sample under the condition of missing text. Total loss function Represented as: 。 8. The method according to claim 7, characterized in that, In step 7, the sentiment prediction module includes a regression head, which uses the final multimodal fusion features. The method takes the input as input and outputs the sentiment polarity category and sentiment intensity prediction accuracy. It is applicable to multimodal sentiment analysis scenarios with missing text modalities, incomplete text semantics, and degraded audio or video modal quality.

9. An electronic device, characterized in that, It includes a processor and a memory, the memory storing program code that, when executed by the processor, causes the processor to perform the steps of the method as described in any one of claims 1 to 8.

10. A storage medium, characterized in that, It stores a computer program or instructions that, when run on a computer, perform the steps of the method as described in any one of claims 1 to 8.

Images