A multi-modal sentiment analysis method and system based on uncertainty dynamic guidance

Abstract

The application discloses a kind of multi-modal sentiment analysis method and system based on uncertainty dynamic guidance, the multi-modal sentiment analysis method is first obtained and extracts feature, and the extracted feature is mapped to latent Gaussian distribution by probability coding;Then establish uncertainty guided attention mechanism;The time perception global uncertainty score of each mode is calculated, the mode with the lowest uncertainty is dynamically selected as the anchor mode, the cross-modal interaction topology is reconstructed, and other modes query information to the anchor mode;Finally, using reliability perception fusion strategy, calculate reliability weight based on posterior variance, inhibit high noise features through gate mechanism, and generate the final sentiment representation combined with modal invariant features.The application breaks the traditional static language dominant hypothesis, quantifies uncertainty as an active control signal, realizes dynamic modal guidance and anti-noise fusion, and significantly improves the sentiment recognition robustness in modal noisy or semantic fuzzy scenarios.

Description

Technical Field

[0001] This invention belongs to the field of multimodal sentiment analysis technology, specifically relating to a multimodal sentiment analysis method and system based on uncertainty dynamic guidance. Background Technology

[0002] Emotion recognition aims to identify and classify emotional states from human behavior or language. Early emotion recognition methods relied primarily on a single modality, such as using only facial expressions, speech features, or text content to identify emotions. While these methods were successful in specific contexts, the complexity and diversity of emotional expression meant that a single modality often failed to comprehensively capture an individual's emotional state. As research progressed, researchers began exploring multimodal fusion techniques, integrating information from different sensory channels (such as video, audio, and text) to improve the accuracy and robustness of emotion recognition. Multimodal fusion techniques can understand and infer emotional information from multiple angles and dimensions, thereby achieving more accurate emotion classification.

[0003] The core of multimodal fusion lies in how to efficiently integrate data from different modalities to comprehensively and accurately capture emotional information. Common fusion strategies include feature-level fusion, decision-level fusion, and model-level fusion, among which deep learning-based model-level fusion has become the mainstream in multimodal emotion recognition. However, existing methods mainly focus on the semantic relationships between modalities, often ignoring the uncertainty of the modal data itself. Although these methods can effectively model cross-modal semantic information, the lack of quantification and adaptive processing of modal reliability in the presence of noise or missing data leads to a decrease in the accuracy and robustness of emotion recognition. Some studies have attempted to adjust the modal fusion weights through uncertainty quantification, but usually fail to fully consider the semantic relationships between modalities. In emotion analysis tasks, cross-modal semantic relationships are crucial; ignoring this may limit the effectiveness of information fusion, thereby affecting the overall recognition performance.

[0004] The core of multimodal fusion lies in how to efficiently integrate data from different modalities to comprehensively and accurately capture emotional information. Common fusion strategies include feature-level fusion, decision-level fusion, and model-level fusion, among which the Transformer-based cross-modal attention architecture has become the mainstream in multimodal emotion recognition. However, existing methods mainly adopt a static modality-guided paradigm, that is, they fixate on using the language modality as the core to retrieve audiovisual information, often ignoring the ambiguity (such as irony) or noise (such as ASR errors) that may exist in the text modality itself. In addition, existing technologies are mostly based on non-biased fusion using deterministic feature vectors, assuming that all input features are completely reliable, ignoring the random uncertainties that are widespread in the data (such as background noise and image blur). Although these methods can effectively model cross-modal semantic information, when faced with low-quality or noisy data, the lack of quantification and dynamic adjustment mechanisms for modality reliability can easily lead to blind confidence and error propagation, resulting in a decrease in the accuracy and robustness of emotion recognition. Summary of the Invention

[0005] The purpose of this invention is to address the shortcomings and deficiencies of existing technologies by proposing a multimodal sentiment analysis method and system based on dynamic uncertainty guidance. Addressing the problems of attention being misled in noisy text due to the static language-dominated mechanism in existing technologies, and the blind confidence in noisy modalities due to the lack of reliability metrics in deterministic feature fusion, this invention aims to establish a probability-aware paradigm. This paradigm enables the model to proactively perceive the accidental uncertainty of data during multimodal data fusion and transform it into proactive control signals. Based on this, the guidance relationship and fusion weights between modalities are dynamically adjusted, thereby effectively improving the robustness and accuracy of sentiment recognition in complex scenarios.

[0006] In a first aspect, the present invention provides a multimodal sentiment analysis method based on uncertainty-driven dynamic guidance, the method comprising:

[0007] A sentiment analysis model is constructed, comprising a probabilistic feature encoding module, a feature interaction fusion module, and a recognition module connected in sequence. The probabilistic feature encoding module extracts features from the input feature sequences of each modality to obtain the latent representation of each modality. The feature interaction fusion module includes a common branch, a specific branch, and a reliability-aware fusion module. The common branch processes common information of each modality, and the specific branch processes unique information of each modality. The reliability-aware fusion module fuses the output features of the common branch and the specific branch, and inputs the fusion result into the recognition module to obtain the final sentiment recognition result.

[0008] An uncertainty-guided interaction module is introduced into the specific branch to perform multimodal interaction. In the uncertainty-guided interaction module, the mode with the lowest uncertainty is selected as the anchor mode. The anchor mode is self-reinforced through a self-attention mechanism, and cross-modal attention interaction is performed using the non-anchor mode as the query vector and the anchor mode as the key and value vector to obtain the attention weights of each mode. The attention weights of each mode are refined to obtain the final feature representation output by the uncertainty-guided interaction module.

[0009] Collect the multimodal data to be tested, and extract the original feature sequences of each modality data and input them into the sentiment analysis model to obtain the predicted sentiment intensity corresponding to the multimodal data to be tested.

[0010] Preferably, the probabilistic feature encoding module includes multiple probabilistic feature encoders; the number of probabilistic feature encoders is consistent with the number of modalities in the input sentiment analysis model.

[0011] In the probabilistic feature encoder, two projection layers are used to extract the mean and standard deviation of the original feature sequence, and a reparameterization layer is used to reparameterize the mean and standard deviation to obtain the latent representation.

[0012] Preferably, the anchor mode is selected based on the global uncertainty score; the method for obtaining the global uncertainty score is as follows:

[0013] The time step variance intensity of each modality is obtained based on the standard deviation of the original feature sequence; the multiple time step variance intensities of a single modality are converted into a probability distribution to obtain the time attention weight; the time step variance intensities are weighted and summed based on the time attention weight to obtain the global uncertainty score of each modality.

[0014] Preferably, in the reliability-aware fusion module, the final feature representation is linearly projected to generate a posterior mean and a posterior variance; a reliability weight is obtained based on the posterior variance; the reliability weight is inversely proportional to the posterior variance; the posterior mean is weighted element-wise according to the reliability weight, and the fusion result is concatenated with the output features of the mapped common branch to obtain a joint vector; the joint vector is gated fusion to obtain the fusion vector output by the reliability-aware fusion module.

[0015] Preferably, the refinement module includes a first sub-block and a second sub-block; the first sub-block is used to randomly deactivate the attention weights and fuse the processing result with the corresponding global uncertainty score through residual connection to obtain a first fused feature; the second sub-block processes the first fused feature through a series of layer normalization and feedforward networks, and fuses the processing result with the first fused feature through residual connection to obtain a feature sequence.

[0016] As a preferred approach, for non-anchor modalities, after refining the attention weights, feature enhancement is performed through one or more stacked self-attention layers to obtain the final feature representation.

[0017] Preferably, the common branch includes a cascaded common encoder and a self-attention module; the common encoder is composed of Transformer layers and is used to extract high-level sentiment semantics shared across modalities to obtain modality-invariant features; the self-attention module is used to process the modality-invariant features to obtain an attention score, which is used as the output feature of the common branch.

[0018] Preferably, the specific branch includes a specific encoder and an uncertainty-guided interaction module connected in series; the specific encoder is composed of a Transformer layer and is used to retain the modality-unique attributes to obtain modality-specific features; the uncertainty-guided interaction module is used to process the modality-specific features to obtain the final feature representation, which serves as the output feature of the specific branch.

[0019] Preferably, the sentiment analysis model is trained using a dataset containing multimodal data of different objects, and a hybrid loss function is constructed for end-to-end optimization during the training process; the hybrid loss function includes main task loss, reconstruction loss, KL divergence loss, supervised contrast loss, and binary cross-entropy loss.

[0020] Preferably, the multimodal data includes text data, audio data, and visual data.

[0021] Secondly, the present invention provides a multimodal sentiment analysis system based on uncertainty dynamic guidance, which is used to execute the above-mentioned multimodal sentiment analysis method; the multimodal sentiment analysis system includes a data acquisition module, a feature extraction module, a feature interaction fusion module, and a recognition module; the data acquisition module is used to acquire the multimodal data to be tested; the feature extraction module is used to perform uncertainty modeling on the multimodal features; the feature interaction fusion module is used to capture the semantic association between the distributions of each modality and to perform weighted fusion of multiple interacting features; the recognition module is used to perform sentiment recognition on the fused features.

[0022] Thirdly, the present invention provides a computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the memory stores the computer program and the processor executes the aforementioned sentiment analysis method.

[0023] Fourthly, the present invention provides a readable storage medium storing a computer program; when executed by a processor, the computer program is used to implement the aforementioned sentiment analysis method.

[0024] The beneficial effects of this invention are:

[0025] 1. This invention establishes an uncertainty-guided attention mechanism in the modal interaction stage. By calculating and comparing the time-aware global uncertainty scores of each modality in real time, it dynamically selects the clearest and most reliable modality at the current moment as the anchor point, and uses this anchor point modality as the benchmark for information retrieval to guide other modalities to conduct cross-modal attention interactions. This mechanism breaks the static language-dominated assumption of rigid fixed language as the core in traditional multimodal models, effectively avoids error propagation caused by noise or ambiguity in a single modality (such as text), significantly improves the effectiveness of multimodal interaction, and ensures that the model always corrects the noise of other modalities based on the most reliable signal.

[0026] 2. This invention introduces a probabilistic perception framework into the probabilistic coding module. The probabilistic coding module models the features as a Gaussian distribution to accurately quantify input-level uncertainty. That is, the mean of the distribution is used to represent sentiment semantics, and the variance of the distribution is used to accurately quantify the inherent random uncertainty in the data. This breaks through the limitations of the traditional deterministic point vector representation, thereby providing a reliable confidence measure for subsequent processing.

[0027] 3. This invention employs a reliability-aware fusion strategy in the feature fusion stage. It estimates the posterior uncertainty of features after interaction and calculates reliability weights inversely proportional to the posterior variance. These reliability weights are then used to adaptively filter modality-specific features through a gating mechanism, automatically suppressing the contribution of high-uncertainty features. Finally, the fused representation is generated by combining these reliability weights with modality-invariant features. This "evaluation-weighting-gating" mechanism ensures that the features ultimately used for sentiment prediction are primarily composed of high-confidence information. It automatically filters out high-noise interference and retains high-confidence information, significantly improving the model's robustness in complex scenarios.

[0028] 4. This invention significantly outperforms baseline models in emotion recognition tasks, especially in complex scenarios with high noise interference or semantic ambiguity, demonstrating stronger noise resistance, robustness and stability compared to existing technologies. Attached Figure Description

[0029] Figure 1 This is an overall flowchart of Embodiment 1 of the present invention.

[0030] Figure 2 This is a structural diagram of the sentiment analysis model in Embodiment 1 of the present invention.

[0031] Figure 3 This is a structural diagram of the uncertainty-guided interaction module in Embodiment 1 of the present invention.

[0032] Figure 4 This is a structural diagram of the reliability perception fusion module in Embodiment 1 of the present invention. Detailed Implementation

[0033] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0034] The terms "comprising" and "having," and any variations thereof, used in the embodiments of this invention are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the steps or units listed, but may optionally include other steps or units not listed, or may optionally include other steps or units inherent to such processes, methods, products, or devices.

[0035] Example 1

[0036] like Figure 1 As shown, a multimodal sentiment analysis method based on uncertainty dynamic guidance includes the following steps:

[0037] S1: Data Acquisition and Preprocessing

[0038] A dataset is constructed by acquiring multimodal data from different objects. Multimodal data includes text data. Audio data and visual data The original feature sequences of each modality are extracted using a pre-trained feature extraction model. ;in, ; The sequence length; For feature dimensions.

[0039] S2: Constructing a sentiment analysis model

[0040] like Figure 2 As shown, the sentiment analysis model comprises a probabilistic feature encoding module, a feature interaction fusion module, and a recognition module connected in sequence. The probabilistic feature encoding module is responsible for extracting features from the multimodal raw feature sequence and modeling the features as a latent Gaussian distribution, obtaining a probabilistic representation including mean and variance to accurately quantify the input-level uncertainty of the data. The feature interaction fusion module is used to fuse features from different modalities, generating a final fusion vector, which is then input into the recognition module to obtain the final sentiment recognition result.

[0041] S2-1: Probabilistic Feature Encoding Module

[0042] The probabilistic feature encoding module comprises three probabilistic feature encoders, each used to extract features from the original feature sequences of each modality, obtaining the latent representation of each modality. The probabilistic feature encoders include two parallel projection layers and one reparameterization layer. The two projection layers are used to predict the mean and log-variance of the original feature sequences, respectively. The mean primarily reflects sentiment and semantic information; the log-variance is used to quantify random uncertainties at the input level, such as background noise. The use of the predicted logarithmic form here aims to ensure the stability of numerical computation and guarantee the non-negativity of the subsequently recovered variance. Mean Sum of logarithmic variance Represented as:

[0043]

[0044]

[0045] in, and Indicates the projection layer; The standard deviation is denoted as .

[0046] In this embodiment, the projection layer and projection layer All use 1D convolutional layers.

[0047] To make the gradient differentiable during backpropagation, a reparameterization layer is used to process the output features of the two projection layers, and the reparameterization formula is used to generate the latent representation. It preserves the fluidity of the gradient; through the reparameterization formula, randomness is effectively transferred to the random noise vector, enabling the network parameters to be trained through end-to-end backpropagation; latent representation The expression is:

[0048]

[0049] in, It is a random noise vector sampled from the standard normal distribution.

[0050] S2-2: Feature Interaction Fusion Module

[0051] The feature interaction fusion module includes parallel common branches and specific branches, as well as a reliability-aware fusion module. The common branches process shared information across modalities. The specific branches process modal-specific information. The reliability-aware fusion module fuses the common and specific branches into the latent representation. The feature extraction results.

[0052] S2-2-1: Public branch road

[0053] The common branch consists of a cascaded common encoder and a self-attention module. The common encoder, composed of Transformer layers, is used to extract high-level sentiment semantics shared across modalities, and is expressed by the formula:

[0054]

[0055] in, It is a modally invariant feature; This indicates a common encoder.

[0056] The self-attention module is used for modality-invariant features. Process the data to obtain an attention score. It is represented as:

[0057]

[0058] S2-2-2: Specific branch

[0059] The specific branch includes a cascaded specific encoder and an uncertainty-guided interaction module. The specific encoder, composed of Transformer layers, is used to preserve modality-specific properties (such as intonation fluctuations in speech), which are often the main source of uncertainty. It is expressed by the formula:

[0060]

[0061] in, Modality-specific features; Indicates a specific encoder.

[0062] like Figure 3 As shown, the uncertainty-guided interaction module is used to solve the failure problem of the static language-dominated mechanism when the text is noisy, and to realize dynamic interaction. Its feature processing process is as follows:

[0063] (1) Time-aware uncertainty pooling

[0064] Considering the uneven distribution of uncertainty over time (e.g., a few seconds of speech may contain particularly loud noise), the time step variance intensity of each modality is first calculated. Then use Function computation time attention weight Finally, the global uncertainty score of this mode is obtained by weighted summation. It can be expressed by the formula:

[0065]

[0066]

[0067]

[0068] in, For feature dimensions; The variances of each modality; The temporal attention weight of mode m at time t is given a larger weight for times with high uncertainty. This represents the global uncertainty score of mode m in the current sample. The smaller the value, the more reliable the mode is.

[0069] (2) Dynamic anchor point selection

[0070] For each sample in the current batch, compare the global uncertainty scores of all modes, and dynamically select the mode with the lowest score (i.e., the most reliable) as the anchor mode. It is represented as:

[0071]

[0072] in, The index representing the selected anchor modality (e.g., if the text is noisy, audio or visual might be selected as the anchor modality). This step breaks the static assumption and ensures that the interaction is dominated by the clearest modality at any given moment.

[0073] (3) Adaptive routing interaction

[0074] Based on the selected anchor point Reconstruct the cross-modal attention topology. Project the modality-specific features of the anchor modality into a query matrix. Key matrix Sum matrix The modality-specific features of the two non-anchor modes are projected into query matrices respectively. and query matrix . Query matrix respectively and query matrix AND key matrix Sum matrix Perform cross-attention calculation and then apply the query matrix. AND key matrix Sum matrix Perform self-attention calculations to obtain attention weights for different modalities. It is represented as follows:

[0075]

[0076] in, This represents the query matrix corresponding to the non-anchor mode. ; It is a standard scaled dot product attention mechanism that ensures that the information flow always moves from "high confidence" to "low confidence".

[0077] (4) Feature refinement

[0078] Attention weights for each modality are applied using the refinement module. Processing is performed to obtain the feature sequence. For anchor-point modalities, their corresponding feature sequences are used as the final feature representations. For non-anchor-point modalities, contextual information is used to smooth the feature sequences to eliminate semantic discontinuities caused by dynamic switching of anchor-point modalities, thus obtaining the final feature representations. The final feature representations for each modality are as follows: for:

[0079]

[0080] in, It consists of three stacked self-attention layers.

[0081] The refinement module consists of a first sub-block and a second sub-block. The first sub-block is used to refine the attention weights. Random deactivation is performed, and the processing result is fused with the corresponding global uncertainty score through residual connections to obtain the first fused feature output by the first sub-block. The second sub-block processes the first fused feature through cascaded layer normalization and feedforward networks, and the processing result is fused with the first fused feature through residual connections to obtain the second fused feature output by the second sub-block, i.e., the feature sequence. .

[0082] S2-2-3: Reliability Perception Fusion Module

[0083] like Figure 4 As shown, the reliability-aware fusion module uses posterior uncertainty to weight and filter features, preventing the model from blindly fusing high-noise features. Its feature processing procedure is as follows:

[0084] (1) Posterior uncertainty estimation

[0085] The uncertainty of features changes after interaction and needs to be re-estimated. The final feature representation... Generate posterior mean by linear projection and posterior variance It is represented as:

[0086]

[0087]

[0088] in, and The projection matrix is ​​learnable; This reflects the remaining posterior uncertainty of the modal feature after interactive correction.

[0089] (2) Calculation of reliability weight

[0090] Calculate the reliability weights that are inversely proportional to the posterior variance. To prevent the model from artificially reducing variance during training to obtain high weights (i.e., "variance collapse"), the variance in the denominator is adjusted here. (Stop gradient) operation; reliability weights Represented as:

[0091]

[0092] in, To prevent the division by zero of minute values; The operation treats variance as a constant scaling factor, cuts off gradient backpropagation, and prevents the model from artificially reducing variance.

[0093] The posterior mean after Sigmoid activation is weighted element-wise using reliability weights, and then the attention score is mapped:

[0094]

[0095]

[0096] in, This represents the modality-specific features after reliability weighting. This represents the shared fusion features after projection; Use the Sigmoid activation function; and For linear projection layers; This indicates element-wise multiplication. This formula implements a soft-gating mechanism: variance. The larger the value, the higher the weight. The smaller the value, the more its contribution to the fusion is automatically suppressed.

[0097] (3) Final fusion

[0098] All weighted specific features are concatenated with modality-invariant features and then input into a gated fusion network to obtain a fusion vector. It is represented as:

[0099]

[0100]

[0101]

[0102] in, This is the concatenated joint vector; These are learnable gated vectors used to selectively enhance salient features.

[0103] S2-3: Identification Module

[0104] The recognition module uses linear layer pairs to fuse vectors. The data is processed to obtain the output of the sentiment analysis model, namely the predicted sentiment intensity value. .

[0105] S3: Train a sentiment analysis model using a dataset and construct a hybrid loss function. Perform end-to-end optimization and use a hybrid loss function. Represented as:

[0106]

[0107] in, The main task loss is used to measure prediction accuracy. The reconstruction loss is used to constrain feature decoupling; KL divergence loss is used for feature alignment; This is a supervised contrastive loss used to enhance feature discriminativeness. For binary cross-entropy loss; This is a hyperparameter used to balance the contribution weights of each loss term.

[0108] Main task loss The model's predictions are designed to approximate the true labels as closely as possible. This is achieved using the mean absolute error (MAE), calculated as follows:

[0109]

[0110] in, Indicates the size of the training batch; Indicates the first The true emotional intensity value of each sample; Indicates the first The predicted sentiment intensity value for each sample.

[0111] Reconstruction loss The aim is to ensure that the representation after feature decoupling retains key information from the original input and prevents information loss; reconstruction loss The mean squared error (MSE) is used for calculation, as shown in the following formula:

[0112]

[0113] in, This represents the mean of the original feature sequence; The decoder represents the output features of the modality-specific encoder and the modality-common encoder. This indicates a concatenation operation along the feature dimension; This represents the square of the Euclidean norm.

[0114] KL divergence loss is used to minimize the differences between the distributions of common features across different modalities, forcing the alignment of cross-modal semantics in the common space; KL divergence loss The formula is as follows:

[0115]

[0116] in, and These represent the Gaussian distributions of different modalities (such as text and audio, and text and vision) in the common feature space.

[0117] Supervised contrastive loss enhances the discriminative power of modality-specific features by narrowing the feature distance between samples with the same sentiment label and widening the feature distance between samples with different labels. The formula is as follows:

[0118]

[0119] in, This is the set of sample indexes within the current batch; Is with the sample A set of positive samples with the same label; The number of positive samples; To remove All sample sets other than; For temperature parameters; Modal-specific features The sample-level representation after pooling.

[0120] Binary cross-entropy loss This is used to help optimize the binary classification boundary of sentiment polarity and improve the accuracy of the model in determining positive and negative sentiment.

[0121] S4: The emotion recognition method provided in this embodiment will be tested on two publicly available, multimodal emotion-related datasets that are recognized in the industry. The publicly available datasets are as follows:

[0122] In emotion recognition, this invention uses the CMU-MOSI and CH-SIMS datasets to verify the effectiveness of this embodiment.

[0123] (1) CMU-MOSI: The CMU-MOSI dataset is a classic English multimodal sentiment analysis benchmark, containing 2199 video clips. These video clips are all standard monologues, and each statement is labeled with a sentiment intensity score ranging from [-3, +3], where negative values ​​represent negative sentiment and positive values ​​represent positive sentiment. The dataset is divided according to a strict standard protocol (1284 training sets, 229 validation sets, and 686 test sets) to ensure the consistency and comparability of the experiments.

[0124] (2) CH-SIMS: The CH-SIMS dataset is a challenging Chinese multimodal sentiment analysis benchmark, containing 2281 finely annotated video clips. Unlike CMU-MOSI, CH-SIMS focuses on unaligned multimodal sequences and includes more complex semantic contexts and cross-modal heterogeneity. Each statement also undergoes fine-grained sentiment annotation, reviewed by multiple experts to ensure accuracy. The dataset is divided into training, validation, and test sets in a 6:2:2 ratio.

[0125] In the experiments, a comprehensive and detailed evaluation method was employed for the CMU-MOSI dataset (regression and classification) and the CH-SIMS dataset (regression and classification) to accurately measure the model's performance in sentiment analysis tasks. Specifically, this invention calculated the mean absolute error (MAE) and Pearson correlation coefficient (Corr) for sentiment intensity prediction, as well as the accuracy of multi-class classification (Acc-7, Acc-5) and binary classification (Acc-2, F1 score) to comprehensively evaluate the model's classification accuracy and regression fitting ability. Furthermore, to comprehensively reflect the overall model performance, this invention paid particular attention to performance in fine-grained classification (such as Acc-5) and binary classification (Acc-2). This evaluation strategy not only accurately characterizes the model's classification performance at different granularities but also effectively measures the model's robustness in handling complex semantic and noisy data, thus providing a scientific basis for model optimization and improvement.

[0126] Analysis of experimental results:

[0127] The proposed method was validated on the CMU-MOSI and CH-SIMS datasets and compared with existing state-of-the-art models (such as MulT, MISA, Self-MM, DEVA, and DLF). On the CMU-MOSI dataset, the proposed method achieved superior performance across all key metrics, with a binary classification accuracy (Acc-2) of 86.59% and a MAE reduced to 0.684. Particularly noteworthy is its achievement of 55.10% in the highly challenging fine-grained sentiment classification task (Acc-5), a significant improvement of 2.77% compared to the suboptimal model (DLF), demonstrating stronger discriminative ability. On the CH-SIMS dataset, despite challenges such as modality misalignment and environmental noise, the proposed method still achieved a binary classification accuracy (Acc-2) of 81.70%, significantly outperforming the existing state-of-the-art model (DEVA) of 79.64%, representing an improvement of 2.06%. This method, through an uncertainty perception mechanism, can still accurately capture emotional features in noisy or semantically ambiguous situations, thereby improving the accuracy and robustness of emotion classification.

[0128] In robustness verification experiments, the noise intensity (σ) of the language modality is a key factor in verifying the effectiveness of the uncertainty guidance mechanism. To verify its effectiveness in handling modal uncertainty, this embodiment introduces Gaussian noise of different intensities into the language modality to simulate ASR errors. As the noise intensity increases, the F1 accuracy of the baseline model (such as DLF) drops sharply, and higher uncertainty severely weakens the quality of modal information, leading to error propagation. However, the model of this invention exhibits significant noise resistance: in low-noise environments, the model can make balanced use of information from each modality; when the noise increases (e.g., noise intensity...), the model can effectively mitigate noise. When the uncertainty of the text modality increases significantly, the uncertainty-driven attention mechanism of this invention automatically detects this change and dynamically switches the interaction anchor from text to a more reliable audio or visual modality. Experimental results show that under high noise conditions, this invention still maintains an F1 score of 65.09%, which is 7.44% higher than the baseline model. The experimental results verify the modality uncertainty-driven dynamic attention adjustment mechanism: when the uncertainty of a certain modality is high, the model can adaptively adjust the attention allocation, suppress the interference of high uncertainty modalities, enhance cross-modal semantic decoding capabilities, and thus improve the robustness of sentiment analysis tasks.

[0129] Example 2

[0130] A multimodal sentiment analysis system based on uncertainty dynamic guidance includes a data acquisition module, a probabilistic feature encoding module, a feature decoupling module, an uncertainty-guided interaction module, a reliability perception fusion module, and a sentiment prediction module.

[0131] Data acquisition module: responsible for acquiring multimodal data; the multimodal data includes text modality, audio modality, and visual modality.

[0132] The probabilistic feature encoding module is responsible for extracting features from multimodal data and modeling the features as a latent Gaussian distribution to obtain a probabilistic representation that includes the mean and variance, so as to accurately quantify the input-level uncertainty of the data.

[0133] Feature decoupling module: responsible for building specific encoders and common encoders, decoupling the latent Gaussian distribution representation into orthogonal modality-specific features and modality-invariant features, thereby separating intramodal noise dynamics from cross-modal shared semantics.

[0134] Uncertainty-guided interaction module: responsible for calculating the time-aware global uncertainty score of each modality, dynamically selecting the modality with the lowest uncertainty as the anchor point, and reconstructing the cross-modal attention topology to guide non-anchor modalities to query information from the anchor modality in order to optimize the modal interaction process.

[0135] The reliability-aware fusion module is responsible for estimating the posterior uncertainty of features after interaction, calculating reliability weights that are inversely proportional to variance, and performing weighted filtering and fusion of features through a gating mechanism to generate the final fusion vector.

[0136] The sentiment prediction module is responsible for inputting the fused feature vector into the classifier of the sentiment recognition model to obtain the final sentiment recognition result.

[0137] Finally, it should be noted that the above-described embodiments are merely specific implementations of the present invention, used to illustrate the technical solutions of the present invention, and not to limit it. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed in the present invention, or make equivalent substitutions for some of the technical features; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A multimodal sentiment analysis method based on uncertainty-driven dynamics, characterized in that: The method includes: A sentiment analysis model is constructed, comprising a probabilistic feature encoding module, a feature interaction fusion module, and a recognition module connected in sequence. The probabilistic feature encoding module extracts features from the input feature sequences of each modality to obtain the latent representation of each modality. The feature interaction fusion module includes a common branch, a specific branch, and a reliability-aware fusion module. The common branch processes common information of each modality, and the specific branch processes unique information of each modality. The reliability-aware fusion module fuses the output features of the common branch and the specific branch, and inputs the fusion result into the recognition module to obtain the final sentiment recognition result. An uncertainty-guided interaction module is introduced into the specific branch to perform multimodal interaction. In the uncertainty-guided interaction module, the mode with the lowest uncertainty is selected as the anchor mode. The anchor mode is self-reinforced through a self-attention mechanism, and cross-modal attention interaction is performed using the non-anchor mode as the query vector and the anchor mode as the key and value vector to obtain the attention weights of each mode. The attention weights of each mode are refined to obtain the final feature representation output by the uncertainty-guided interaction module. Collect the multimodal data to be tested, and extract the original feature sequences of each modality data and input them into the sentiment analysis model to obtain the predicted sentiment intensity corresponding to the multimodal data to be tested.

2. The multimodal sentiment analysis method based on uncertainty dynamic guidance according to claim 1, characterized in that: The probabilistic feature encoding module includes multiple probabilistic feature encoders; The number of probabilistic feature encoders is consistent with the number of modalities in the input sentiment analysis model; In the probabilistic feature encoder, two projection layers are used to extract the mean and standard deviation of the original feature sequence, and a reparameterization layer is used to reparameterize the mean and standard deviation to obtain the latent representation.

3. The multimodal sentiment analysis method based on uncertainty dynamic guidance according to claim 2, characterized in that: The anchor mode is selected based on the global uncertainty score; the method for obtaining the global uncertainty score is as follows: The time step variance intensity of each modality is obtained based on the standard deviation of the original feature sequence; the multiple time step variance intensities of a single modality are converted into a probability distribution to obtain the time attention weight; the time step variance intensities are weighted and summed based on the time attention weight to obtain the global uncertainty score of each modality.

4. The multimodal sentiment analysis method based on uncertainty dynamic guidance according to claim 1, characterized in that: In the reliability-aware fusion module, the final feature representation is linearly projected to generate the posterior mean and posterior variance; reliability weights are obtained based on the posterior variance; the reliability weights are inversely proportional to the posterior variance; the posterior mean is weighted element-wise according to the reliability weights, and the fusion result is concatenated with the mapped common branch output features to obtain a joint vector; the joint vector is gated fusion to obtain the fusion vector output by the reliability-aware fusion module.

5. The multimodal sentiment analysis method based on uncertainty dynamic guidance according to claim 1, characterized in that: The refinement module includes a first sub-block and a second sub-block. The first sub-block is used to randomly deactivate the attention weights and fuse the processing result with the corresponding global uncertainty score through residual connections to obtain a first fused feature. The second sub-block processes the first fused feature through a series of layer normalization and feedforward networks and fuses the processing result with the first fused feature through residual connections to obtain a feature sequence.

6. The multimodal sentiment analysis method based on uncertainty dynamic guidance according to claim 1, characterized in that: For non-anchor modalities, after refining the attention weights, feature enhancement is performed through one or more stacked self-attention layers to obtain the final feature representation.

7. The multimodal sentiment analysis method based on uncertainty dynamic guidance according to claim 1, characterized in that: The common branch includes a cascaded common encoder and a self-attention module; the common encoder is composed of Transformer layers and is used to extract high-level sentiment semantics shared across modalities to obtain modality-invariant features; the self-attention module is used to process the modality-invariant features to obtain an attention score, which is used as the output feature of the common branch.

8. The multimodal sentiment analysis method based on uncertainty dynamic guidance according to claim 1, characterized in that: The specific branch includes a specific encoder and an uncertainty-guided interaction module connected in series; the specific encoder is composed of a Transformer layer and is used to preserve the unique properties of the modality to obtain modality-specific features; the uncertainty-guided interaction module is used to process the modality-specific features to obtain the final feature representation, which serves as the output feature of the specific branch.

9. The multimodal sentiment analysis method based on uncertainty dynamic guidance according to claim 1, characterized in that: The multimodal data includes text data, audio data, and visual data.

10. A multimodal sentiment analysis system based on uncertainty-driven dynamic guidance, characterized in that: This system is used to execute a multimodal sentiment analysis method based on uncertainty dynamic guidance as described in claim 1. The multimodal sentiment analysis system includes a data acquisition module, a feature extraction module, a feature interaction fusion module, and a recognition module. The data acquisition module is used to acquire the multimodal data to be tested. The feature extraction module is used to perform uncertainty modeling on the multimodal features. The feature interaction fusion module is used to capture the semantic associations between the distributions of each modality and to perform weighted fusion of multiple interacting features. The recognition module is used to perform sentiment recognition on the fused features.

Images