A multi-modal sentiment analysis method based on multi-granularity routing and uncertainty perception fusion

Abstract

The application discloses a multi-modal sentiment analysis method based on multi-granularity routing and uncertainty perception fusion, and belongs to the technical field of artificial intelligence and multi-modal information processing. The method is aimed at the quality fluctuation problems such as occlusion, noise, blur and text recognition error of language, vision and audio modes in actual scenes. Firstly, the mode private features, mode invariant features and cross-modal features of each mode are obtained. Then, the time gating and self-attention convergence are performed on the mode invariant features to obtain the speech level shared representation. Further, the subspace routing and mode routing are constructed to realize the adaptive balance of the private and shared information in the mode and the dynamic allocation of the importance between the modes. Meanwhile, the uncertainty of each mode is predicted, and the mode routing result is calibrated through inverse variance weighting to reduce the interference of the low reliable mode on the fusion result. Finally, the calibrated multi-modal features are fused and the sentiment prediction result is output. The application can improve the robustness, stability and interpretability of multi-modal sentiment analysis in the mode quality unstable scene.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence, pattern recognition, multimodal information processing and sentiment computing, and in particular to a multimodal sentiment analysis method for scenarios with fluctuating modal quality. Background Technology

[0002] Multimodal sentiment analysis aims to jointly utilize information from multiple modalities, such as text, vision, and audio, to identify the sentiment tendency or intensity of samples. Existing methods typically use features concatenating, attention weighting, gating fusion, or cross-modal interaction to jointly model multimodal information. However, in real-world applications, the quality of different modalities varies significantly with the samples. For example, the visual modality may be affected by occlusion, changes in head pose, or motion blur; the audio modality may be affected by background noise, differences in recording equipment, or channel disturbances; and the text modality may have missing words, misspelled words, or incomplete semantics due to speech recognition errors. In such cases, if the fusion mechanism still assigns high weights to low-quality modalities, it can easily lead to unstable sentiment prediction, insufficient robustness, and increased false positive rates. Furthermore, even if the basic representation module can separate modal shared information from modal private information and generate strong cross-modal representations, relying solely on stronger feature representations cannot guarantee the final fusion stage's reliability-oriented adaptive capability. Therefore, it is necessary to explicitly estimate sample-level modal importance in the final fusion stage and dynamically calibrate it in conjunction with modal reliability. Therefore, it is necessary to propose a new multimodal sentiment analysis method that can dynamically adjust the contribution of different modalities and subspace information to the prediction results in scenarios with fluctuating modal quality, damaged modalities, missing parts of modalities, or noise interference, thereby improving the stability, robustness, and interpretability of sentiment analysis. Summary of the Invention

[0003] The purpose of this invention is to provide a multimodal sentiment analysis method based on the fusion of multi-granularity routing and uncertainty perception, in order to solve the following problems existing in the prior art: 1. Modal importance is mainly learned implicitly by the terminal task, making it difficult to explicitly reflect the true task contribution of each modality under the current sample; 2. It is difficult to dynamically calibrate the fusion weights based on modal reliability, resulting in low-quality modes having an excessive impact on the final prediction; 3. There is a lack of fine-grained coordination mechanism between modal shared information and modal private information, making it difficult to adaptively balance shared semantics and private features for different samples; 4. Existing methods are not robust enough under conditions of text occlusion, visual loss, audio loss, or modal noise.

[0004] To achieve the above objectives, the present invention adopts the following technical solution: A multimodal sentiment analysis method includes the following steps: S1. Obtain multimodal input data of the sample to be analyzed, wherein the multimodal input data includes at least two of the following: text modality, visual modality, and audio modality; S2. Input each modality into the basic feature extraction module to obtain modality-private temporal features, modality-invariant temporal features, and cross-modality summary features corresponding to each modality; S3. For the modality-invariant temporal features of each modality, perform temporal gating convergence and self-attention context convergence to obtain the discourse-level shared representation of that modality. S4. Aggregate the modality-private temporal features of each modality to obtain the discourse-level private representation of that modality; S5. Combine the discourse-level private representations, discourse-level shared representations, and cross-modal summary features of each modality to form routing input features; S6. Perform subspace routing based on the routing input features to obtain an adaptive combination result between modality private representation and modality shared representation; S7. Perform modal routing based on the routing input features to obtain the original modal gating weights for each modality; S8. Predict the uncertainty of the corresponding mode based on the cross-modal summary features of each mode, generate inverse variance weights based on the predicted uncertainty, calibrate the original mode gating weights, and obtain the reliability-aware mode gating weights. S9. Based on the reliability-aware modal gating weights, the cross-modal summary features and subspace routing results of each modality are weighted and fused, input into the prediction head, and output sentiment category or sentiment score. Beneficial effects

[0005] Compared with the prior art, the present invention has at least the following beneficial effects: 1. By modeling the subspace and modal levels separately, a fine-grained balance between shared and private information within a modality is achieved, as well as the dynamic allocation of importance among modalities; 2. By employing an uncertainty prediction and inverse variance weighting calibration mechanism, the impact of low-reliability modes in the final fusion can be significantly suppressed; 3. By using leave-one-out error increment supervision, the modality routing is made closer to the real task contribution of the current sample, thus improving the interpretability of the fusion results; 4. Enhance the stability of shared representations and improve the consistency of subsequent routing and fusion by using modality-invariant contrast alignment; 5. Under controlled perturbation conditions such as text occlusion, visual loss, and audio loss, it can maintain classification performance well and is suitable for application scenarios with significant fluctuations in modality quality. Attached Figure Description

[0006] Figure 1 This is a flowchart illustrating the overall process of a multimodal sentiment analysis method based on the fusion of multi-granularity routing and uncertainty perception according to the present invention. Figure 2 This is a schematic diagram of the modality-invariant branch timing summary module in this invention; Figure 3 This is a schematic diagram of the multi-granularity routing module in this invention; Figure 4 This is a schematic diagram of the uncertainty perception fusion module in this invention; Figure 5 This is a schematic diagram of the Leave-one-out routing supervision process during the training phase of this invention. Detailed Implementation

[0007] The present invention will be further described below with reference to the accompanying drawings and specific embodiments. It should be understood that the following embodiments are for illustrative purposes only and are not intended to limit the scope of protection of the present invention. Figure 1 As shown, the overall process of this invention includes steps such as multimodal input acquisition, basic feature extraction, temporal summarization, multi-granularity routing, uncertainty estimation, reliability-aware fusion, and sentiment output. The implementation of the basic feature extraction module is not limited to a specific neural network structure, as long as it can output modality-private representations, modality-invariant representations, and cross-modal summarization representations. In one embodiment, the system receives a text sequence, a video frame sequence, and an audio feature sequence corresponding to the utterance to be analyzed. First, the basic feature extraction module maps each modal input to a unified feature space and outputs three types of representations for each modality: modality-private temporal features. Modality-invariant time series characteristics and cross-modal summarization features .

[0008] Modal invariant branch timing summary: such as Figure 2 As shown, temporal gating convergence is first performed on modality-invariant temporal features, followed by self-attention contextual convergence, and finally the two results are combined to form a discourse-level shared representation. The gating value for the m-th modality at time position t can be expressed as: Where σ(·) represents the Sigmoid function, This represents the invariant feature of the m-th mode at time t. and These are learnable parameters. Time series summary: Simultaneously, a contextual summary is obtained by using a self-attention encoder to perform contextual encoding on modality-invariant temporal features and then aggregating them in the time dimension. The final discourse-level shared representation can be expressed as: The above design allows for the simultaneous utilization of temporal saliency information and global contextual information, thereby improving the stability of shared representations.

[0009] Modality-invariant contrastive alignment: To further stabilize the shared representation space, contrastive alignment can be performed on the shared representations at the discourse level across different modalities. For the shared representation of the i-th sample in the m-th modality, it is first mapped to the contrastive space and then normalized. Then, a contrastive loss can be constructed based on ordered modal pairs, typically in the form of InfoNCE: , This constraint allows the shared representations of the same sample obtained in different modalities to be close to each other in the common feature space, thus providing a more stable shared semantic basis for subsequent routing and fusion.

[0010] Multi-granularity routing: such as Figure 3 As shown, multi-granularity routing includes two levels: subspace routing and modal routing. First, the private temporal features of the modal routing are aggregated to obtain a private summary. This, along with the shared summary and cross-modal summary, forms the routing input: Modal routing is used to determine the importance of different modalities to the current sample sentiment prediction task. Its original modal gating can be expressed as: Modal routing is used to determine the importance of different modalities to the current sample sentiment prediction task. Its original modal gating can be expressed as: To provide explicit supervision of modal routing during the training phase, an auxiliary importance prediction head can also be set:

[0011] Leave-one-out routing supervision: such as... Figure 5 As shown, during the training phase, an additional forward process is constructed for each modality that is missing that modality. Let the prediction result for the complete modality input be... The prediction result after removing the m-th mode is If the true label is y, then the importance of this modality to the current sample can be characterized by the error increment: After normalizing the error increments for each modality, the distribution of teacher importance can be obtained: Furthermore, loss functions such as mean squared error are used to supervise the auxiliary importance prediction head: The above-mentioned supervision method only increases the computational overhead during the training phase and does not affect the ability to complete sentiment prediction in a single forward pass during the inference phase.

[0012] Uncertainty perception fusion: such as Figure 4As shown, to suppress the adverse effects of damaged modes on the final fusion result, this invention predicts the uncertainty of each mode based on cross-modal summarization features. The logarithmic variance of the m-th mode can be expressed as: Then, the uncertainty weights are obtained by normalizing the inverse variance: By combining the inverse variance weights with the original modal gating, a reliability-aware modal gating is obtained: Therefore, when the uncertainty of a certain mode increases, its corresponding inverse variance weight decreases, thus reducing its final participation in the fusion.

[0013] Reliability-aware fusion and prediction: In one embodiment, the cross-modal summary features of each modality are first subjected to linear transformation and nonlinear mapping, and then modulated by the final reliability-aware modality gating. Similarly, the subspace routing results are also modulated using the same gating: The subspace modulation results of the three modes can be further fused after being spliced ​​together: Subsequently, the gated cross-modal summaries of the three modalities are input together with the fused subspace features into the prediction head: , , The above design retains the dual modulation capability of intramodal subspace information and intermodal reliability information, enabling the final prediction branch to not only aggregate multimodal information, but also dynamically adjust the fusion result according to sample-level reliability.

[0014] Training objective: In one embodiment, the overall training objective may be composed of task loss, distillation loss, regularization term, routing supervision loss, and modality-invariant contrastive alignment loss, and its typical form can be expressed as: ,in, This indicates task loss related to sentiment prediction. This represents the distillation loss in the basic feature extraction module. This indicates regularization terms such as refactoring, cycle consistency, or orthogonality. and The weighting coefficients are used. In the above embodiments, the basic feature extraction module can be any network structure capable of outputting modality-private representations, modality-shared representations, and cross-modality summary representations, and should not be limited to a specific backbone network. The linear mappings, multilayer perceptrons, self-attention encoders, temporal pooling strategies, loss function forms, and modality types in each layer can all be adjusted or replaced without departing from the spirit of this invention.

Claims

1. A multimodal sentiment analysis method, comprising the following steps: S1. Obtaining multimodal input data of the sample to be analyzed, wherein the multimodal input data includes at least two of the following: text modality, visual modality, and audio modality; S2. Extracting modal features based on a basic feature extraction module to obtain modality-private temporal features, modality-invariant temporal features, and cross-modal summary features corresponding to each modality; S3. Performing temporal gating convergence and self-attention contextual convergence on the modality-invariant temporal features of each modality to obtain the discourse-level shared representation of that modality; S4. Performing temporal convergence on the modality-private temporal features of each modality to obtain the discourse-level private representation of that modality; S5. Combining the discourse-level private representation, discourse-level shared representation, and cross-modal summary features of each modality to form routing input features; S6. Performing subspace routing based on the routing input features to obtain an adaptive combination result of the modality-internal modality-private representation and modality-shared representation; S7. Performing modal routing based on the routing input features to obtain the original modality gating weights of each modality; S8. Predict the uncertainty of each modality based on the cross-modal summary features of each modality, generate inverse variance weights based on the predicted uncertainty, calibrate the original modal gating weights, and obtain the reliability-aware modal gating weights; S9. Based on the reliability-aware modal gating weights, perform weighted fusion of the features of each modality, and output the sentiment analysis results.

2. The method according to claim 1, wherein step S3, time-gated convergence includes: Gating coefficients are generated for each time position in the modality-invariant temporal feature sequence, and the corresponding time position features are weighted and averaged according to the gating coefficients to highlight the time positions that contribute more to the current sentiment prediction.

3. The method according to claim 1, wherein step S3, self-attention context convergence includes: Modality-invariant temporal features are input into a self-attention encoder to obtain a contextualized temporal representation, and a contextual summary is obtained by time dimension aggregation. The discourse-level shared representation is obtained by combining the time-gated convergence result and the context summary.

4. The method according to claim 1, wherein the subspace routing in step S6 includes: For each modal output, a normalized subspace weight is used to measure the relative contribution of modal private representation and modal shared representation, and the modal private representation and modal shared representation are weighted and combined according to the subspace weight.

5. The method according to claim 1, wherein the modal routing in step S7 includes: For each modality, an independent modality gating value is output to indicate the importance of that modality in the current sample sentiment prediction task.

6. In the method according to claim 1, in step S8, the uncertainty is predicted by the cross-modal summary features of each modality, and the inverse variance weight is obtained by normalizing the inverse of the prediction variance of each modality, so that the modality with greater prediction uncertainty corresponds to a smaller final fusion weight.

7. The method according to claim 1 further includes a training step: constructing an additional forward process for missing single modalities, calculating the prediction error increment before and after removing each modality, and generating a teacher importance distribution based on the prediction error increment to supervise modal routing learning.

8. The method according to claim 7, wherein the teacher importance distribution is obtained by normalizing the error increment corresponding to each modality, and the larger the error increment, the higher the contribution of the corresponding modality to the task of the current sample.

9. The method according to claim 1, further comprising: Modality-invariant contrast alignment is performed on the discourse-level shared representations of each modality so that the shared representations of the same sample in different modalities are close to each other in the common feature space.

10. In the method according to claim 1, in step S9, the cross-modal summary features and subspace routing results are first gated and modulated using reliability-aware modal gating weights, and then the modulated results are spliced ​​and fused, and the sentiment prediction results are output through a multilayer perceptron.

Images