Multi-modal sentiment analysis system and method based on dynamic routing and feature decoupling

Abstract

The application discloses a kind of multi-modal sentiment analysis system and method based on dynamic routing and feature decoupling, it is related to artificial intelligence and sentiment computing field.System includes feature decoupling module, decoupling guided routing module and dynamic attention fusion module.First, the original feature of each input mode is decomposed into modal shared feature and modal specific feature by feature decoupling module;Modal shared feature is aggregated to obtain enhanced modal shared feature;Decoupling guided routing module is based on the importance weight of decoupled mode, and adaptively generates a set of mode weights for each sample;Dynamic attention fusion module uses mode weight, dynamically weights and cross-modal attention fusion to modal specific feature, to obtain enhanced specific feature;Finally, fuse enhanced specific feature and enhanced modal shared feature to carry out sentiment prediction.The application solves the defect that existing dynamic fusion method makes decision on entangled features, realizes more accurate mode importance evaluation and fusion.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and affective computing, and more specifically to a multimodal affective analysis system and method based on dynamic routing and feature decoupling. Background Technology

[0002] With the widespread adoption of social media and video content, multimodal sentiment analysis aims to comprehensively utilize text, speech, and visual information to understand human emotions, and has significant application value in fields such as human-computer interaction and mental health monitoring. Its core challenge lies in how to effectively integrate these heterogeneous modalities.

[0003] Existing multimodal fusion methods are mainly divided into two categories: symmetric fusion strategies and text-centric fusion strategies. Symmetric fusion strategies (such as tensor fusion and cross-modal attention) treat all modalities equally but ignore the dynamic changes in the dominant sentiment modality in different scenarios. Text-centric strategies assume that the text modality is dominant and forcibly align other modalities with it; this static assumption limits the flexibility of the model. Although dynamic fusion methods have emerged, assigning different modal weights to different samples through routing networks, these methods still rely on the original, entangled features of the various modalities. Redundancy and noise in the entangled features can interfere with the routing network's judgment of the true importance of the modality, leading to inaccurate weight allocation and thus affecting the final fusion result.

[0004] Therefore, how to design a multimodal sentiment analysis fusion mechanism that can dynamically and accurately assess and integrate the importance of each modality based on the sample content is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] In view of the above problems, the present invention is proposed to provide a multimodal sentiment analysis system and method based on dynamic routing and feature decoupling to overcome or at least partially solve the above problems.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] A multimodal sentiment analysis system based on dynamic routing and feature decoupling includes a sequentially connected feature decoupling module, a decoupling-guided routing module, a dynamic attention fusion module, and a prediction module;

[0008] Feature decoupling module: Receives raw feature sequences from three modalities: text, vision, and audio. It decouples the raw feature sequences from the three modalities into a shared encoder and three modality-specific encoders, outputting modality-shared features and modality-specific features. The modality-shared features are aggregated to output enhanced modality-shared features.

[0009] Decoupling-guided routing module: Connects to the feature decoupling module, receives the decoupled modality-shared features and modality-specific features from the original feature sequences of the three modalities, concatenates the obtained modality-shared features and modality-specific features and inputs them into the routing network, outputting modality importance weights for each sample;

[0010] Dynamic Attention Fusion Module: Connects the Feature Decoupling Module and the Decoupling-Guided Routing Module. It receives modality-specific features and modality importance weights, dynamically weights and fuses the three modality-specific features through stacked dynamic attention blocks, outputs multimodal fusion features, and uses a cross-modal attention mechanism to capture the association information between the fusion state of the multimodal fusion features and the modality-specific features, outputting enhanced modality-specific features.

[0011] Prediction module: It integrates augmented modality-specific features and augmented modality-shared features to obtain the final multimodal feature representation, and predicts sentiment scores based on the final multimodal feature representation.

[0012] Preferably, the feature decoupling module includes a shared encoder and multiple modality-specific encoders. The shared encoder is used to process the input of all modalities to extract the modality-shared features common to the three modalities. The modality-specific encoders correspond to the three modalities respectively and extract the modality-specific features of each modality.

[0013] Preferably, the feature decoupling module uses a shared encoder. and modal-specific encoders Decouple multimodal features into shared information. and modality-specific information The specific formula is as follows:

[0014]

[0015] in and Represents the learnable parameters of the network. For cross-modal sharing, it is used to constrain the model to learn shared information invariant to each modality. ,Will Setting it to modal independence ensures that the model can fully capture modal-specific information. .

[0016] Preferably, the decoupled routing module includes a feature splicing unit, a routing network unit, and a weight normalization unit;

[0017] Feature splicing unit: used to splice the modality-shared features and modality-specific features of each modality to obtain a structured representation of the modality, and further splice the structured representation to form a unified routing representation;

[0018] Routing network unit: Receives the routing representation formed by the feature splicing module and outputs the initial weight distribution result;

[0019] Weight normalization unit: The initial weight distribution result is normalized using a normalization function, and the importance weight of each modality sample is output.

[0020] Preferably, the decoupled guided routing module is trained by minimizing a routing loss function, which is a weighted sum of modality equalization loss and weight similarity loss;

[0021] Modality equalization loss is calculated based on the information entropy of modality importance weights to prevent the model from over-relying on a single modality; weight similarity loss is used to measure the difference between the modality importance weights and the distribution of sentiment importance calculated based on the single-modality prediction error of each modality.

[0022] Preferably, the operations performed by the dynamic attention block in the dynamic attention fusion module include:

[0023] The three modality-specific features obtained by the feature decoupling module are linearly transformed through their respective independent projection layers, mapped to a unified fusion dimension and a unified sequence length, resulting in three aligned modality-specific features. The modality importance weights output by the decoupling-guided routing module guide the alignment of the modality-specific features to be added together, resulting in a multimodal fusion feature. The fusion feature is used as the input to a dynamic attention block, and a cross-modal attention mechanism is used to capture the correlation information between the fusion state of the fusion feature and the modality-specific features. The multimodal fusion feature is input to multiple self-attention layers, and then fed into a feedforward neural network containing two linear transformation layers and a nonlinear activation function located between the two linear transformation layers for nonlinear transformation and dimension mapping, outputting enhanced modality-specific features.

[0024] Preferably, the multimodal sentiment analysis system is trained using a joint loss function, which includes a main task loss function and a composite auxiliary loss function, and the main task loss function and the composite auxiliary loss function are used to minimize the final prediction error;

[0025] The main task loss function is used to calculate the error between the final predicted score and the actual multimodal sentiment score;

[0026] The composite auxiliary loss function is the sum of the losses from the feature decoupling module, the decoupling-guided routing module, the dynamic attention fusion module, and the deep supervision loss of the routing network; the specific formula for calculating the composite loss function is as follows:

[0027] ;

[0028] in, For composite auxiliary loss, The loss in the feature decoupling module, To decouple the loss in the routing module of the bootloader, For the loss in the dynamic attention fusion module, Losses due to in-depth monitoring.

[0029] A multimodal sentiment analysis method based on dynamic routing and feature decoupling includes the following steps:

[0030] S1. Receive the original feature sequences of three modalities: text, vision, and audio. Decouple the original feature sequences of the three modalities into a shared encoder and three modality-specific encoders. Output modality-shared features and modality-specific features. Perform an aggregation operation on the modality-shared features to output enhanced modality-shared features.

[0031] S2. Through the feature decoupling module, the modality-shared features and modality-specific features of the original feature sequences of the three modalities are decoupled. The obtained modality-shared features and modality-specific features are concatenated and input into the routing network to output the modality importance weight for each sample.

[0032] S3. The decoupled routing module receives modality-specific features and modality importance weights, and dynamically weights and fuses the three modality-specific features through stacked dynamic attention blocks to output enhanced modality-specific features.

[0033] S4. Through the dynamic attention fusion module, enhance modality-specific features and enhanced modality-shared features are fused to output enhanced modality-specific features.

[0034] S5. The modality-shared features obtained from S1 and the enhanced modality-specific features obtained from S4 are fused to obtain the final multimodal feature representation. The sentiment score is then predicted based on the final multimodal feature representation.

[0035] As can be seen from the above technical solutions, compared with the prior art, the present invention discloses a multimodal sentiment analysis system and method based on dynamic routing and feature decoupling. The beneficial effects of the above technical solutions provided by the embodiments of the present invention include at least the following:

[0036] 1. More Precise Dynamic Fusion: This invention pioneers a process of "decoupling first, then dynamic fusion." In the decoupling stage, pure shared and modality-specific features are separated through strong constraints, providing "clean" input without redundant noise for subsequent routing decisions. This enables the routing network to more accurately assess the true importance of each modality in the current sample, thereby achieving more precise weight allocation.

[0037] 2. Enhanced Model Flexibility and Generalization Ability: This invention abandons the static assumption of modal importance. Through sample-level dynamic routing and weight modulation, the model can adaptively adjust the fusion strategy according to different scenarios, thereby exhibiting stronger robustness and generalization ability when processing diverse and complex real-world data.

[0038] 3. More effective feature representation: Through feature decoupling, the model learns structured and interpretable feature representations: shared features carry cross-modal emotional commonalities, while modality-specific features retain modality-unique details. This decoupling not only facilitates dynamic fusion but is also an efficient method for feature purification and enhancement.

[0039] 4. Comprehensive Experimental Validation: This invention has been validated on multiple standard multimodal sentiment analysis and humor detection datasets, and its performance surpasses that of existing state-of-the-art methods. Ablation experiments confirm the indispensability of the three core components—feature decoupling, dynamic routing, and dynamic attention fusion—as well as the effectiveness of various loss functions. Attached Figure Description

[0040] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0041] Figure 1 This is a diagram of the dynamic fusion network architecture provided in this embodiment of the invention;

[0042] Figure 2 This is a structural diagram of the dynamic attention block provided in an embodiment of the present invention;

[0043] Figure 3 These are examples of dominant emotional modalities in different scenarios provided in the embodiments of the present invention;

[0044] Figure 4 This is the distribution of the influence of Acc-7 and Acc-5 with different numbers of dynamic attention blocks provided in the embodiments of the present invention;

[0045] Figure 5 This is the distribution of the influence of Acc-2 and F1 on different numbers of dynamic attention blocks provided in the embodiments of the present invention;

[0046] Figure 6 This is the distribution of the influence of Corr and MAE on different numbers of dynamic attention blocks provided in the embodiments of the present invention;

[0047] Figure 7This is the distribution of features on the CMU-MOSI dataset before and after modal decoupling provided in this embodiment of the invention. Detailed Implementation

[0048] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0049] like Figure 1 As shown in the figure, this invention discloses a multimodal sentiment analysis system based on dynamic routing and feature decoupling. The multimodal sentiment analysis system specifically includes a sequentially connected feature decoupling module, a decoupling-guided routing module, a dynamic attention fusion module, and a prediction module.

[0050] Feature decoupling module: Receives raw feature sequences from three modalities: text, vision, and audio. It decouples the raw feature sequences from the three modalities into a shared encoder and three modality-specific encoders, outputting modality-shared features and modality-specific features. The modality-shared features are aggregated to output enhanced modality-shared features.

[0051] Decoupling-guided routing module: Connects to the feature decoupling module, receives the decoupled modality-shared features and modality-specific features from the original feature sequences of the three modalities, concatenates the obtained modality-shared features and modality-specific features and inputs them into the routing network, outputting modality importance weights for each sample.

[0052] Dynamic Attention Fusion Module: Connects the Feature Decoupling Module and the Decoupling-Guided Routing Module. It receives modality-specific features and modality importance weights, dynamically weights and fuses the three modality-specific features through stacked dynamic attention blocks, outputs multimodal fusion features, and uses a cross-modal attention mechanism to capture the association information between the fusion state of the multimodal fusion features and the modality-specific features, outputting enhanced modality-specific features.

[0053] Prediction module: It integrates augmented modality-specific features and augmented modality-shared features to obtain the final multimodal feature representation, and predicts sentiment scores based on the final multimodal feature representation.

[0054] The feature decoupling module receives three modalities: text (T), visual (V), and audio (A). First, three independent one-dimensional convolutional networks are used to extract temporal information features, thereby obtaining low-level multimodal features: ,in , It is the sequence length of that mode, and It is the dimension of the feature vector at each time step. This standardizes the dimensions of different modalities.

[0055] Furthermore, to reduce intermodal conflicts and redundant information, a shared encoder was introduced. and three modality-specific encoders Multimodal features are decomposed into shared information. and modality-specific information :

[0056]

[0057] in and This represents the learnable parameters of the network. This is set up for cross-modal sharing to constrain the model to learn modally invariant shared information. At the same time, To ensure that the model is modally independent, it is set to capture modality-specific information. .

[0058] To achieve effective decoupling, four regularization terms are introduced to constrain these encoders.

[0059] To ensure information integrity during the decoupling process, a one-dimensional convolutional decoder is used. Decomposed shared features and modality-specific features After concatenation, the features are mapped back to the original feature space to obtain the reconstructed features. Reconstructing features With low-level features The reconstruction loss is obtained through comparison:

[0060]

[0061] To ensure modality sharing features and modality-specific features It can capture independent, non-redundant information separately, and introduces orthogonal constraint loss. Promote the orthogonality of the two modal feature spaces:

[0062]

[0063] in, This represents the calculation of cosine similarity.

[0064] Although reconstruction loss and orthogonality loss guarantee the basic conditions for decoupling, they cannot impose constraints on the intrinsic properties of the decoupled modality-specific feature subspace and shared feature subspace, thus failing to ensure the semantic effectiveness of decoupling. Therefore, for modality-specific features, the reconstructed features are re-inputted into the modality-specific encoder to obtain new modality-specific features. The difference between specific features of the original mode and specific features of the new mode is the specific loss:

[0065]

[0066]

[0067] For shared features, the core property is modality invariance. To constrain this intrinsic property of the shared feature subspace, a shared feature alignment loss is introduced. It employs a contrastive learning-based approach, directly reshaping the geometry of the shared feature subspace through an improved triplet loss function. This loss function aims to bring the anchor points closer together. Compared with positive samples The distance between them Samples from different modalities but sharing the same sentiment, while simultaneously distancing themselves from negative samples. The distance between them They originate from the same modality but have different emotions. Their formal definition is as follows:

[0068]

[0069] in, It is the set of all triples. Used to calculate the cosine distance between two eigenvectors; This represents the distance threshold.

[0070] Furthermore, the four geometric regularization losses mentioned above, which are designed to regulate the decoupling process and the decoupling subspace respectively, are integrated to form a comprehensive total decoupling loss function. These four losses work synergistically to impose strong constraints on the feature decoupling module from multiple dimensions, including information preservation, semantic orthogonality, mapping stability, and modality invariance. The total decoupling loss is defined as follows:

[0071]

[0072] in , These are the weight coefficients of each regularization term, providing a flexible adjustment mechanism that allows balancing the contributions of each regularization term, thereby more effectively calibrating the model and achieving optimal decoupling performance.

[0073] Through the above decoupling process, pure and information-independent shared features are generated for each modality. and modality-specific features After obtaining high-quality feature representations, the next key step is to adaptively evaluate and assign contribution weights to each modality based on the specific content of the samples. However, existing dynamic fusion methods typically make routing decisions directly on the raw or pre-encoded features, and the inherent inter-modal redundancy in these features can interfere with the accuracy of weight assignment. To address this, a decoupled guided routing module is proposed, which makes decisions based on refined features, thereby achieving a more accurate modality weight distribution.

[0074] Specifically, the shared features and modality-specific features obtained after decoupling are first concatenated to construct a structured modality representation. :

[0075]

[0076] in, , This represents the splicing operation. The structured representations of the three modalities are then spliced ​​and aggregated into a unified routing representation. This data is then fed into a routing network, which is a lightweight multilayer perceptron that outputs a corresponding modal importance weight for each sample. Formally:

[0077]

[0078]

[0079] in, It is a routing feature that aggregates all modal information; Represents the routing network, and is defined by parameters. The control system consists of a global average pooling layer and a multi-layer perceptron. It is the Softmax function; a temperature coefficient is also introduced. This is used to control the smoothness of the weight distribution. Ultimately, it is determined by the routing network. Generate a set of modality-related weights These weights will be used in the next stage to guide the dynamic attention module in performing targeted feature enhancement. Meanwhile, to ensure the effectiveness and balance of the routing network's learning, a corresponding loss is introduced to constrain it.

[0080] Furthermore, in multimodal learning, different modalities have varying learning difficulties and convergence speeds. Without constraints on the routing network, it may rapidly skew towards the best-performing modality early on, leading to excessive weighting of that modality and neglect of other modalities. To address this issue, modality balance loss is introduced to prevent the model from over-relying on any single modality, thereby encouraging the routing network to allocate balanced weights to all modalities. The formula is as follows:

[0081]

[0082] Where N represents the number of modes, For the weights of the corresponding modes, This represents a very small positive number. The modal equalization loss guarantees "balance," but it still needs to ensure the "effectiveness" of the weights, meaning the assigned weights should be related to the actual predictive power of that modality in the current sample. Therefore, a single-modal prediction head is added, which compares the weights generated by DGR. With a “true” importance calculated based on a single-mode prediction error The similarity between them is used to provide supervision for the routing network. The representation is as follows:

[0083]

[0084]

[0085] in, This represents the single-mode prediction value corresponding to the given mode. It's a real label. These represent the importance coefficients of the three modes. Subsequently, the weights output by the routing network are calculated. Importance score of this mode Similarity loss between them:

[0086]

[0087] in, It is a similarity function. This similarity loss helps incentivize the routing network to learn a weight allocation strategy that accurately reflects the actual predictive capabilities of each modality. Finally, it is combined with the modality equalization loss to obtain the loss of the routing network module:

[0088]

[0089] in , These are the weighting coefficients of each loss. These two losses work together to ensure that DGR generates a set of dynamic weights for the current sample that accurately reflects the relative importance of each modality. .

[0090] Furthermore, the core idea of ​​Dynamic Attention Block (DAB) is to utilize the modality weights generated by the routing network as guiding signals to guide an attention-based fusion network, enabling it to adaptively and selectively integrate specific information from each modality. First, to unify the processing of subsequent fusion modules, the three modality-specific features obtained from the decoupling module are... , , They are linearly transformed through their respective independent projection layers, mapping them to a unified fusion dimension. and sequence length .

[0091] Furthermore, since it is unclear which modality will dominate in the initial stage of fusion, the three aligned modality-specific features are added element-wise to form a preliminary, unbiased fusion feature. As the input for the first DAB:

[0092]

[0093]

[0094] in, It consists of two linear layers. Then, N DABs with identical structures are stacked to enhance modality-specific features. For the nth DAB module, the output of the previous module is... All modality-specific features after alignment , , and the modal weights calculated by the upstream routing network. As input, we obtain its output. .

[0095]

[0096] Where DAB stands for Dynamic Attention Block, and modal weights are... This is used to represent the contribution of each mode.

[0097] The three modality-specific features obtained from the feature decoupling module are linearly transformed through their respective independent projection layers, mapped to a unified fusion dimension and a unified sequence length, resulting in three aligned modality-specific features. The modality importance weights output by the decoupling-guided routing module guide the alignment of the modality-specific features to be added together, resulting in a multimodal fusion feature. The fusion feature is used as the input to a dynamic attention block, and a cross-modal attention mechanism is used to capture the correlation information between the fusion state of the fusion feature and the modality-specific features. The multimodal fusion feature is input to multiple self-attention layers, and then fed into a feedforward neural network containing two linear transformation layers and a nonlinear activation function located between the two linear transformation layers for nonlinear transformation and dimension mapping. The output of the last block is used as the final enhanced modality-specific feature, which is then output and used in subsequent MSA tasks.

[0098] like Figure 2 The diagram shows the structure of a dynamic attention block (DAB) specifically designed for modality-specific feature enhancement. Each DAB module contains an ingenious computational flow for dynamically adjusting the contributions of each modality. First, a cross-modal attention mechanism is used to capture the current fusion state. With each modality-specific feature The correlation information between them. Specifically, the multimodal fusion features. As Q, the single-modal specific features Simultaneously serving as both K and V, and stabilized through residual connections and layer normalization:

[0099]

[0100] Furthermore, modal weights generated by the routing network are introduced. To further guide the fusion process, it should be integrated with the aforementioned intermediate representation. Element-wise multiplication is performed to dynamically "scale" or "modulate" the information stream. Then, the resulting representation is compared with the input. The multimodal representations are summed to fine-tune the contributions of different modes:

[0101]

[0102]

[0103] in This represents element-wise multiplication. Finally, the summarized features... The data is first input into a standard multi-head self-attention layer and then fed into a feedforward neural network for final information integration, yielding the final output of the DAB module. After N layers of stacking, the output of the last DAB module... This serves as the final multimodal-specific feature representation after fusion, and is denoted as... And used for final fusion and prediction.

[0104] To further ensure fusion representation To fully and evenly retain specific information from each individual modality and prevent the neglect of useful cues from other modalities due to overemphasis on a dominant modality during dynamic fusion, a contrastive learning regularization term is introduced. Specifically, a contrastive predictive encoding objective is adopted and constructed within a noisy contrastive estimation framework to estimate modality-specific feature representations. Its corresponding original single-modal specific features The correlation between them. This contrastive learning loss is named... The calculation method is as follows:

[0105]

[0106] Unlike the complex dynamic enhancement process for modality-specific features, for modality-shared features... , Because they already contain common semantic information across modalities, a more direct and effective fusion strategy is adopted. For example... Figure 1 As shown, the shared features first integrate contextual information through a self-attention mechanism, then undergo nonlinear transformation and deep feature extraction through fully connected layers and projection layers, finally yielding a fused high-level shared feature representation, denoted as . .

[0107] Furthermore, the enhanced modality-specific features and shared features are concatenated to obtain the final multimodal representation. .

[0108]

[0109] The multimodal sentiment analysis system is trained end-to-end using a joint loss function, which includes a main task loss. and a composite auxiliary loss The aim is to minimize the final prediction error, while simultaneously... Ensure that each key component within the model can learn high-quality feature representations.

[0110] The core objective of the model is to accurately predict multimodal sentiment. This involves the final fused multimodal representation. A prediction head is fed in to obtain the predicted value. The mean absolute error (L1 Loss) is used to calculate the loss of the main task. :

[0111]

[0112] To provide more direct supervision signals to the intermediate layers of the model, thereby improving feature quality and model robustness, an auxiliary loss system based on the concept of deep supervision was designed, denoted as... A single-mode prediction head is introduced into the model. Since multimodal methods still rely on information from each mode, a loss function for the single mode is calculated to ensure that the single mode also has predictive capability.

[0113]

[0114]

[0115] in, This represents the prediction loss for each modality. and This represents the single-modal predicted value and the ground truth value. Furthermore, it includes the shared features after fusion. and modality-specific features The data are fed into the prediction head separately, and the deep supervision loss is calculated.

[0116]

[0117]

[0118] The composite auxiliary loss defined above Loss in the feature decoupling module Loss in the decoupled routing module Loss in the dynamic attention fusion module Deep monitoring of losses We obtain the composite auxiliary loss by performing a weighted summation:

[0119]

[0120] Ultimately, the overall learning objective of the model The format is as follows:

[0121]

[0122] like Figure 3 As shown in the figure, (a)-(c) provide examples of the dominant emotional modality in three different contexts. Figure 3 The scenario shown in (a) is saying, "In short, this movie is really great." Figure 3 The scenario presented in (b) involves saying "but there weren't that many." Figure 3 The scenario shown in (c) is that I actually shouted out loud: What... what movie am I watching? What is this?!

[0123] Currently, mainstream MSA methods mainly follow two strategies: symmetric fusion and text-centric approaches. The symmetric fusion strategy assumes that all modalities contribute equally and models the pairwise equivalence between them by designing complex interaction mechanisms. The text-centric strategy, on the other hand, assumes that the text modality carries the most core semantic information and uses it as the dominant modality to guide the alignment of visual and audio information. However, both strategies are based on a fundamental static assumption: that the importance of modalities is constant. But in complex real-world interactions, the dominant emotional cue is often dynamic; in some cases, visual, text, or audio may be the dominant modality, respectively.

[0124] Therefore, static assumptions limit the flexibility and generalization ability of models on diverse data. To overcome the limitations of static fusion, some methods consider the specific contribution of each modality to the final task and dynamically adjust the fusion. However, their dynamic mechanisms still operate on entangled features, and the inherent redundancy and noise in these features limit the accuracy of their weight allocation.

[0125] This invention also discloses a multimodal sentiment analysis method based on dynamic routing and feature decoupling, specifically including:

[0126] S1. Receive the original feature sequences of three modalities: text, vision, and audio. Decouple the original feature sequences of the three modalities into a shared encoder and three modality-specific encoders. Output modality-shared features and modality-specific features. Perform an aggregation operation on the modality-shared features to output enhanced modality-shared features.

[0127] S2. Through the feature decoupling module, the modality-shared features and modality-specific features of the original feature sequences of the three modalities are decoupled. The obtained modality-shared features and modality-specific features are concatenated and input into the routing network to output the modality importance weight for each sample.

[0128] S3. The decoupled routing module receives modality-specific features and modality importance weights, and dynamically weights and fuses the three modality-specific features through stacked dynamic attention blocks to output enhanced modality-specific features.

[0129] S4. Through the dynamic attention fusion module, enhance modality-specific features and enhanced modality-shared features are fused to output enhanced modality-specific features.

[0130] S5. The modal-shared features obtained from S1 are fused with the enhanced specific features obtained from S4 to obtain the final multimodal feature representation. The sentiment score is then predicted based on the final multimodal feature representation.

[0131] In one specific embodiment, the proposed decoupled guided dynamic fusion network (DGFN) model was compared with the state-of-the-art multimodal sentiment analysis (MSA) method. The comparison results are shown in Table 1.

[0132] Table 1 Results on the MOSI dataset

[0133]

[0134] Table 1 compares the CMU-MOSI and CMU-MOSEI datasets. Clearly, the proposed DGFN method achieves better performance than other MSA methods. In particular, it achieves a 48.4% Acc-7 score and a 79.83% relevance coefficient on the MOSI dataset, demonstrating its strong sentiment discrimination ability. Other metrics also show improvement. Compared to classic fusion methods (such as TFN and LMF) and symmetric interaction models (such as MulT) and text-centered models (such as MAG-BERT and DLF) employing static fusion strategies, DGFN exhibits significant performance advantages. This strongly suggests that by dynamically evaluating and weighting the contribution of each modality, the model can capture key sentiment cues more flexibly and accurately. Similarly, DGFN still outperforms methods employing dynamic fusion (EMOE), strongly supporting the crucial importance of decoupling and purification beforehand. Weighting on "cleaner" features allows the routing mechanism to make more accurate judgments, thereby achieving more efficient multimodal fusion.

[0135] Table 2 Results on the MOSEI dataset

[0136]

[0137] To further validate the generalization performance of DGFN, its effectiveness was evaluated on the UR_FUNNY dataset. As shown in Table 3, the proposed DGFN model outperforms the baseline model. This advantage stems from the fact that humor perception often relies on cues of dynamic changes across modalities. Therefore, the proposed DGFN, through its unique mechanism of first decoupling and then dynamically fusing, can accurately capture and utilize these crucial humor features, thus outperforming other methods when applied to MHD tasks.

[0138] Table 3 shows the results on the UR_FUNNY dataset.

[0139]

[0140] To verify the necessity of each core component in the "decoupling first, then dynamic fusion" framework, the following ablation experiments were designed. Three variant models were constructed by removing or replacing key components and compared with the complete model. Table 4 clearly shows that removing any core component leads to a comprehensive decline in model performance: removing the feature decoupling module resulted in a significant performance drop, demonstrating the crucial importance of feature purification before dynamic decision-making. Replacing the dynamic routing module with static equal weights also resulted in a performance decrease, validating the superiority of the dynamic weighting strategy. Replacing the dynamic attention module with simple concatenation resulted in a significant performance decrease, indicating that the designed dynamic fusion mechanism can learn more effective representations. In summary, these experiments demonstrate that the proposed feature decoupling, dynamic routing, and dynamic fusion—the three core components—form an organic whole, each making an indispensable contribution to the final model performance.

[0141] Table 4 Ablation results on the MOSI benchmark

[0142]

[0143] The importance of each auxiliary loss term in the composite objective function was verified by removing them one by one. When the total decoupling loss was removed... At that point, the model's performance showed a significant decline. This indicates that feature decoupling is the cornerstone of the entire framework; without effective constraints, high-quality, clean feature inputs cannot be obtained subsequently, severely impacting the final performance. Simultaneously, removing the routing loss used to constrain the routing network... This also led to a significant performance degradation, as the routing network became unable to learn meaningful modality weight allocation strategies, thereby weakening the effectiveness of the dynamic fusion mechanism. Furthermore, contrastive loss was observed. and deep supervision loss They also contribute to model performance, and removing them will also cause a performance degradation. Therefore, it is proven that each auxiliary loss term plays a positive and indispensable role.

[0144] The parameter sensitivity analysis focused on the stacking depth N of the Dynamic Attention Module (DAB), a key hyperparameter in the model, to explore its impact on model performance. This parameter determines the fusion depth of modality-specific features. Figures 4-6As shown, the performance changes of Acc-7 and Acc-5 were compared by testing N in the [1,5] interval; Acc-2 and F1 tested N in the [1,5] interval; and Corr and MAE tested N in the [1,5] interval. Experimental results show that as the number of DAB layers increases, the model performance exhibits a trend of first increasing, then stabilizing, and finally slightly decreasing. When N is small, the model may not be able to fully interact with and fuse the unique information of each modality; while when N is large, the model may face the risk of parameter redundancy and overfitting, leading to a slight decrease in performance. Finally, when N=3, the model achieves the best balance in all indicators. This result proves the rationality of the selected parameters and also shows that the model performs stably within a reasonable number of layers, exhibiting good robustness.

[0145] For visualization of decoupling features, such as Figure 7 To visually demonstrate the effectiveness of DGFN feature decoupling, the feature distributions of the three modalities on the CMU-MOSI dataset were visualized. Before decoupling, features were clustered by modality, but modality-specific features and shared features within each modality were highly entangled. After applying this module, the model successfully projected features into different subspaces. Modality-specific features formed three independent, modality-specific clusters, while all shared features were mapped into a single, overlapping, modality-independent subspace, playing a crucial role in subsequent dynamic fusion.

[0146] Finally, this work proposes a Decoupled Guided Dynamic Fusion Network (DGFN), a novel framework built upon the "decoupling first, then dynamic fusion" strategy. DGFN purifies single-modal representations to shared and specific subspaces through decoupling units, and the routing network uses decoupled features to generate modal weights to guide dynamic attention-based fusion. This breaks the inherent modal balance dilemma of static fusion strategies and solves the problem of uneven weight distribution caused by dynamic fusion mechanisms acting on entangled features, achieving strong modal feature purification and adaptive weight integration. Extensive experiments show that it improves performance compared to existing models. Furthermore, comprehensive ablation studies confirm the core assumptions of this model: feature decoupling, routing mechanisms, and dynamic fusion are all indispensable, and their synergistic effect is the main driving force behind the model's superior performance.

[0147] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0148] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A multimodal sentiment analysis system based on dynamic routing and feature decoupling, characterized in that, The multimodal sentiment analysis system includes a sequentially connected feature decoupling module, a decoupling-guided routing module, a dynamic attention fusion module, and a prediction module; Feature decoupling module: Receives raw feature sequences from three modalities: text, vision, and audio. Decouples the raw feature sequences from the three modalities into a shared encoder and three modality-specific encoders. Outputs modality-shared features and modality-specific features. The modality-shared features are aggregated to output enhanced modality-shared features. Decoupling-guided routing module: Connects to the feature decoupling module, receives the modality-shared features and modality-specific features decoupled from the original feature sequences of the three modalities, and inputs the obtained modality-shared features and modality-specific features into the routing network after concatenation, outputting modality importance weights for each sample containing the three modalities; Dynamic attention fusion module: connects the feature decoupling module and the decoupling-guided routing module, receives the modality-specific features and the modality importance weights, dynamically weights and fuses the three modality-specific features through stacked dynamic attention blocks, outputs multimodal fusion features, and uses a cross-modal attention mechanism to capture the association information between the fusion state of the multimodal fusion features and the modality-specific features, outputting enhanced modality-specific features; Prediction module: It fuses the enhanced modality-specific features with the enhanced modality-shared features to obtain the final multimodal feature representation, and predicts the sentiment score based on the final multimodal feature representation.

2. The multi-modal sentiment analysis system based on dynamic routing and feature decoupling according to claim 1, characterized in that, The feature decoupling module includes a shared encoder and multiple modality-specific encoders. The shared encoder is used to process the input of all modalities to extract the modality-shared features common to the three modalities. The modality-specific encoders correspond to the three modalities respectively and extract the modality-specific features of each modality.

3. The multi-modal sentiment analysis system based on dynamic routing and feature decoupling of claim 1, wherein, The feature decoupling module decouples the multi-modal features into shared information and modality-specific information through a shared encoder and modality-specific encoders The shared information is obtained by the shared encoder and the modality-specific information is obtained by the modality-specific encoders The specific formula is as follows: ； wherein and denote learnable parameters of the network, is shared across modalities for constraining the model to learn shared information invariant to each modality , and is modality independent, ensuring that the model can sufficiently capture modality-specific information .

4. A multimodal sentiment analysis system based on dynamic routing and feature decoupling according to claim 1, characterized in that, The decoupled routing module includes a feature splicing unit, a routing network unit, and a weight normalization unit. Feature splicing unit: used to splice the modality-shared features and modality-specific features of each modality to obtain a structured representation of the modality, and further splice the structured representation to form a unified routing representation; Routing network unit: receives the routing representation formed by the feature splicing module and outputs the initial weight distribution result; Weight normalization unit: The initial weight distribution result is normalized using a normalization function, and the importance weight of each modality sample is output.

5. A multimodal sentiment analysis system based on dynamic routing and feature decoupling according to claim 4, characterized in that, The decoupled routing module is trained by minimizing a routing loss function, which is a weighted sum of modality equalization loss and weight similarity loss. The modality equalization loss is calculated based on the information entropy of the modality importance weights to prevent the model from over-relying on a single modality; the weight similarity loss is used to measure the difference between the modality importance weights and the actual sentiment importance distribution calculated based on the single-modality prediction error of each modality, and outputs the deep supervision loss of the routing network.

6. The multi-modal sentiment analysis system based on dynamic routing and feature decoupling of claim 1, wherein, The operations performed by the dynamic attention block in the dynamic attention fusion module include: The three modality-specific features obtained by the feature decoupling module are linearly transformed through their respective independent projection layers, mapped to a unified fusion dimension and a unified sequence length, resulting in three aligned modality-specific features. The modality importance weights output by the decoupling-guided routing module guide the alignment of the modality-specific features to be added together, resulting in a multimodal fusion feature. The fusion feature is used as the input to a dynamic attention block, and a cross-modal attention mechanism is used to capture the correlation information between the fusion state of the fusion feature and the modality-specific features. The multimodal fusion feature is input to multiple self-attention layers, and then fed into a feedforward neural network containing two linear transformation layers and a nonlinear activation function located between the two linear transformation layers for nonlinear transformation and dimension mapping, outputting enhanced modality-specific features.

7. The multi-modal sentiment analysis system based on dynamic routing and feature decoupling according to claim 5, characterized in that, The multimodal sentiment analysis system is trained using a joint loss function, which includes a main task loss function and a composite auxiliary loss function. The main task loss function and the composite auxiliary loss function are used to minimize the final prediction error. The main task loss function is used to calculate the error between the final predicted score and the actual multimodal sentiment score; The composite auxiliary loss function is the sum of the loss from the feature decoupling module, the loss from the decoupling-guided routing module, the loss from the dynamic attention fusion module, and the deep supervision loss of the routing network; the specific formula for calculating the composite loss function is as follows: ； wherein, is a composite auxiliary loss, is a loss in the feature decoupling module, is a loss in the decoupled guided routing module, is a loss in the dynamic attention fusion module, is a deep supervision loss.

8. A multi-modal sentiment analysis method based on dynamic routing and feature decoupling, according to any one of claims 1-7, a multi-modal sentiment analysis system based on dynamic routing and feature decoupling, characterized in that, Includes the following steps: S1. Receive the original feature sequences of three modalities: text, vision, and audio. Decouple the original feature sequences of the three modalities into a shared encoder and three modality-specific encoders. Output modality-shared features and modality-specific features. Perform an aggregation operation on the modality-shared features to output enhanced modality-shared features. S2. Through the feature decoupling module, the modality-shared features and modality-specific features decoupled from the original feature sequences of the three modalities are received. The obtained modality-shared features and modality-specific features are concatenated and input into the routing network, and modality importance weights are output for each sample. S3. The decoupled routing module receives the modality-specific features and the modality importance weights, performs dynamic weighted fusion of the three modality-specific features through stacked dynamic attention blocks, outputs multimodal fusion features, and uses a cross-modal attention mechanism to capture the association information between the fusion state of the multimodal fusion features and the modality-specific features, outputting enhanced modality-specific features. S4. Through the dynamic attention fusion module, the enhanced modality-specific features and the enhanced modality-shared features are fused together, and the enhanced modality-specific features are output; S5. The modality-shared features obtained in S1 and the enhanced modality-specific features obtained in S4 are fused to obtain the final multimodal feature representation, and the sentiment score is predicted based on the final multimodal feature representation.

Images