Multi-modal sentiment analysis method based on knowledge guidance and modal dynamic attention fusion

Abstract

The application relates to a multi-modal sentiment analysis method based on knowledge guidance and modal dynamic attention fusion, which comprises the following steps: extracting single-modal features of input multi-modal data; obtaining knowledge enhanced representations of each mode based on the single-modal features; predicting single-modal sentiment values based on the knowledge enhanced representations, and converting the single-modal sentiment values into sentiment ratio values; performing preset dynamic attention processing on the knowledge enhanced representations and the sentiment ratio values to obtain final multi-modal representations; and predicting a sentiment score based on the final multi-modal representations. The application can improve the performance of the multi-modal sentiment analysis method and adapt it to more complex and more extensive scenarios by dynamically selecting a dominant mode and adjusting the contribution of each mode according to different samples.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of multi-modal sentiment analysis, and particularly relates to a multi-modal sentiment analysis method based on knowledge guidance and modal dynamic attention fusion. BACKGROUND

[0002] Due to the multi-modal reflection of users' emotional expression in social media, multi-modal sentiment analysis (MSA) has attracted more and more attention in recent years. It aims to mine and understand the sentiment of online videos. Recent MSA methods can be divided into two categories: triadic symmetry-based methods and text-centric methods. In triadic symmetry-based methods, previous studies mainly focus on modeling the bidirectional relationship between each modality pair and treating the contribution of each modality equally. For example, some researchers design tensor-based, LSTM-based, and MLP fusion architectures to capture the commonality between modalities simultaneously. In addition, Tsai et al. design MuLT, a transformer-based model, to find similar information between each modality pair. Moreover, some studies project each modality into two subspaces to learn the commonality and characteristics of the modalities respectively to assist the fusion process. Some researchers also introduce multi-label strategies and contrastive learning to equally improve the quality of single-modality representations. In text-centric methods, previous studies mainly focus on guiding the visual and audio modalities through the text modality to improve the quality of the fused representation. For example, Delbrouck et al. and Han et al. use transformer-based methods to integrate similar information from other modalities through the text modality. While some researchers enhance the text representation by integrating visual and audio information into pre-trained language models. In addition, to reduce potential sentiment-irrelevant and conflicting information, some studies reduce the additional noise by maximizing mutual information or adaptive representation learning. At the same time, some researchers introduce contrastive learning to learn the invariant and similar information between text and other modalities. Although MSA has achieved encouraging results, the above methods treat each modality equally or statically set the text as the dominant modality. This causes the method to be distracted by the secondary modality, hindering its dynamic adjustment to adapt to various scenarios where the dominant modality changes, thus limiting the performance of MSA.

[0003] Recent MSA methods can be divided into two categories: triadic symmetry-based methods and text-centric methods. Triadic symmetry-based methods focus on treating the contribution of each modality equally and modeling the bidirectional relationship of all modality pairs. Text-centric methods focus on using text as the dominant modality to guide the visual and audio modalities to interact with it, to appropriately adjust the contribution of different modalities. Therefore, both triadic symmetry-based methods and text-centric methods consider the importance distribution between modalities to be static and fix the dominant modality. However, as Figure 1As shown, the applicant found that in some cases, vision, text or audio can be the dominant modality, respectively. For Figure 1 The first sample in the table, the visual label is more consistent with the overall sentiment label, so it is dominant.

[0004] Based on the survey of commonly used datasets in MSA, the applicant found that these cases are not uncommon. To explore the distribution of samples with each modality as the dominant one and verify the applicant's idea, the applicant studied four MSA benchmark datasets (MOSI, MOSEI, CH-SIMS and CH-SIMSv2). The statistics and analysis results are as follows. For MOSI and MOSEI, some researchers show that the text modality has higher importance and dominance. At the same time, some ablation studies report that the binary accuracy of these two datasets decreases by about 30% after deleting the text modality (80%+ when there is text, and about 50%+ when there is no text). Therefore, most samples in MOSI and MOSEI are dominated by the text modality.

[0005] For CH-SIMS and CH-SIMSv2, the applicant uses the single-modality labels provided by the dataset to count the number of samples. As shown in Figure 2 It can be observed that the number of samples dominated by the vision, text and audio modalities accounts for 45%-60% of the total, and the distribution of each modality is relatively uniform. This shows that each modality will dominate, and this situation is not uncommon. This also shows that the dominant modality needs to be dynamically adjusted. It is worth noting that the sum of the proportions of the three modalities in the same dataset exceeds 100%, because multiple modalities may dominate in the same sample. Then, the applicant counted the number of samples containing noise modalities (the sentiment polarity of such single-modality is different from that of multi-modality), as shown in Figure 3 It can be seen that the proportion of samples containing noise modalities is about 50%, which further shows that failing to dynamically adjust the contribution of each modality will limit the performance of MSA. Therefore, when the dominant modality is not fixed, the ternary symmetric-based method cannot effectively adapt to the case where any modality dominates, because they do not consider the difference in importance between modalities. The text-centric method statically sets the text as the dominant modality, and when other modalities dominate, the attention of the model will be dispersed by the text. SUMMARY

[0006] The purpose of the present application is to propose a multi-modal sentiment analysis method based on knowledge guidance and modality dynamic attention fusion to solve the problems existing in the prior art.

[0007] To achieve the above purpose, the present application provides the following scheme:

[0008] The multi-modal sentiment analysis method based on knowledge guidance and modality dynamic attention fusion comprises:

[0009] extracting single-modal features of the input multi-modal data;

[0010] obtaining knowledge-enhanced representations of each modality based on the single-modal features;

[0011] predicting single-modal sentiment values based on the knowledge-enhanced representations, and converting the single-modal sentiment values into sentiment ratio values;

[0012] performing preset dynamic attention processing on the knowledge-enhanced representations and the sentiment ratio values to obtain final multi-modal representations;

[0013] predicting a sentiment score based on the final multi-modal representations.

[0014] Optionally, the single-modal features include semantic features of a text modality, semantic features of a visual modality, and semantic features of an audio modality.

[0015] Optionally, obtaining the knowledge-enhanced representations of each modality based on the single-modal features includes:

[0016] encoding the single-modal features into global semantic representations and knowledge sentiment representations;

[0017] combining the global semantic representations and the knowledge sentiment representations to obtain the knowledge-enhanced representations of each modality.

[0018] Optionally, the global semantic representations include global semantic representations of a text modality and global semantic representations of a visual and audio modality.

[0019] The global semantic representations of the text modality are:

[0020] H t ,O t =BERT(I t ;θ BERT ) (1)

[0021] wherein H t is the global semantic representation of the text modality, I t is the input original text sentence, θ BERT is a parameter of the BERT model, BERT(.) is used to extract initial text features by using the BERT model, and O t represents hidden states except for the last few layers.

[0022] The global semantic representations of the visual and audio modality are:

[0023]

[0024] wherein H mglobal semantic representation of visual and audio modalities, O m intermediate hidden states of visual and audio modalities, Encoder m () represents the initial features of visual and audio modalities extracted by TransformerEncoder, I m represents the original visual and audio features of the input, parameters of the TransformerEncoder of visual and audio modalities;

[0025] The knowledge sentiment representation is:

[0026]

[0027] wherein, represents the pre-training parameters of the adapter of m modalities, K m represents the knowledge sentiment representation, represents the obtained data dimension belongs to T m ×d m , T m represents the length of the knowledge sentiment representation, d m represents the dimension of the knowledge sentiment representation.

[0028] Optionally, the knowledge enhanced representation of each modality and the single modality sentiment value are respectively:

[0029]

[0030] wherein, U m represents the knowledge enhanced representation, represents the single modality sentiment value, Decoder m () represents the prediction of single modality sentiment value by using Decoder module, represents the parameters of each single modality Decoder.

[0031] Optionally, the method for converting the single modality sentiment value into a sentiment ratio value is:

[0032]

[0033] wherein, R m is a sentiment ratio value, D m represents an intermediate variable for single modality sentiment value conversion, exp() represents the nth power of e, n is the calculated value in the parentheses, k represents the slope, represents the single modality sentiment value, y represents the label value, D t represents an intermediate variable for converting the sentiment value of the text modality, D v represents an intermediate variable for converting the sentiment value of the visual modality, D aAn intermediate variable representing a converted audio modality sentiment value.

[0034] Optionally, the knowledge enhanced representation and the sentiment ratio are subjected to preset dynamic attention processing to obtain a final multi-modal representation, including:

[0035] Mapping the knowledge enhanced representation to unify the length and dimension axis of the knowledge enhanced representation;

[0036] A plurality of stacked dynamic attention modules are preset;

[0037] The mapped knowledge enhanced representation is taken as an input of a first dynamic attention module;

[0038] The output of the current dynamic attention module, the mapped knowledge enhanced representation and the sentiment ratio are taken as an input of a next dynamic attention module until the output of a last dynamic attention module is obtained as the final multi-modal representation.

[0039] Optionally, the final multi-modal representation is:

[0040]

[0041] wherein F n is the final multi-modal representation, F n-1 is the output of the current dynamic attention module, DAB is a dynamic attention block, is the mapped knowledge enhanced representation, R m is the sentiment ratio, is the dimension of the multi-modal fusion feature, T f is the length of the fusion feature, d f is the dimension of the fusion feature.

[0042] Optionally, before predicting the sentiment score, the method further includes training the final multi-modal representation by using a preset joint loss;

[0043] The preset joint loss is:

[0044]

[0045] wherein, is a multi-modal sentiment prediction value, MLP() is a multi-layer perception, Mean() is an average value of data in the length dimension, F L is the final multi-modal, i.e., the output of the last fusion layer, is a regression loss function, N is the total number of samples, y i is a label of sample i, is a multi-modal sentiment prediction value of sample i, is a regression loss function, is the correlation evaluation loss function, and a is a hyper-parameter balancing the loss contribution, is the preset joint loss.

[0046] The beneficial effects of the present application are:

[0047] The present application proposes a multi-modal sentiment analysis method based on knowledge guidance and modal dynamic attention fusion. First, single-modal features of input multi-modal data are extracted, and knowledge enhanced representations of each modal are obtained based on the single-modal features. Second, single-modal sentiment values are predicted based on the knowledge enhanced representations, and the single-modal sentiment values are converted into sentiment ratios. Then, the knowledge enhanced representations and the sentiment ratios are subjected to preset dynamic attention processing to obtain final multi-modal representations. Finally, sentiment scores are predicted based on the final multi-modal representations. The present application can improve the performance of the multi-modal sentiment analysis method and adapt it to more complex and extensive scenarios by dynamically selecting a dominant modal and adjusting the contribution of each modal according to different samples. BRIEF DESCRIPTION OF DRAWINGS

[0048] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0049] Figure 1 are schematic diagrams of three video samples in the CH-SIMS dataset, with vision, text or audio as the main modal;

[0050] Figure 2 are schematic diagrams of the distribution of each modal as the dominant modal;

[0051] Figure 3 are schematic diagrams of the distribution of samples containing noise modal;

[0052] Figure 4 is a schematic diagram of the overall architecture of KuDA in the embodiment of the present application;

[0053] Figure 5 is a schematic diagram of the architecture of the encoder, the adapter and their connection in the embodiment of the present application;

[0054] Figure 6 is a schematic diagram of the architecture of a single dynamic attention module in the embodiment of the present application;

[0055] Figure 7 is a schematic diagram of the performance visualization of CH-SIMSv2 and MOSI with respect to a in the embodiment of the present application;

[0056] Figure 8 An illustration of the use of t-SNE to visualize text, visual, audio and multi-modal features in CH-SIMS v2 for embodiments of the present application;

[0057] Figure 9 An illustration of three case studies in CH-SIMS for embodiments of the present application. DETAILED DESCRIPTION

[0058] The technical solutions in the embodiments of the present application will be apparently and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without any creative work belong to the protection scope of the present application.

[0059] In order to make the above objectives, characteristics and advantages of the present application more apparent, comprehensible and easy to understand, the present application will be further described in detail below with reference to the drawings and specific embodiments.

[0060] The present embodiment proposes a multi-modal sentiment analysis method based on knowledge guidance and modal dynamic attention fusion, including:

[0061] extracting single-modal features of input multi-modal data;

[0062] obtaining knowledge-enhanced representations of each modality based on the single-modal features;

[0063] predicting single-modal sentiment values based on the knowledge-enhanced representations, and converting the single-modal sentiment values into sentiment ratio values;

[0064] performing preset dynamic attention processing on the knowledge-enhanced representations and the sentiment ratio values to obtain final multi-modal representations;

[0065] predicting sentiment scores based on the final multi-modal representations.

[0066] Specifically, in the present embodiment, as Figure 4As shown, the overall workflow of KuDA is demonstrated. Specifically, KuDA first extracts single-modal low-level features from the raw multi-modal input. Then, the adapter and encoder extract single-modal high-level features and simultaneously learn sentiment knowledge. This embodiment utilizes the decoder to predict single-modal sentiment and convert it into sentiment ratios to guide dynamic fusion. Next, this embodiment designs a dynamic attention fusion module that selects the dominant modality according to different scenarios and dynamically adjusts the attention weights using sentiment ratios and knowledge representations. Finally, the multi-modal representation is utilized to perform the MSA task through a multi-layer perceptron (MLP) and estimate the relevance to the knowledge representation. In addition, to guide the model to adjust the attention weights through sentiment knowledge, KuDA adopts a two-stage training method.

[0067] Further, the single-modal features include: semantic features of a text modality, semantic features of a visual modality, and semantic features of an audio modality.

[0068] Specifically, in this embodiment, in the MSA task, the input data is composed of three modalities of text (t), vision (v), and audio (a). The sequence representation of the three modalities is represented as a triple (I t ,I v ,I a ), which includes and where T m , m e {t, v, a} is the sequence length, d m represents the vector dimension. The prediction is the sentiment score , i.e., the sentiment polarity (positive, neutral, or negative).

[0069] Further, based on the single-modal features, obtaining the knowledge-enhanced representation of each modality includes:

[0070] encoding the single-modal features into global semantic representations and knowledge sentiment representations;

[0071] combining the global semantic representations and the knowledge sentiment representations to obtain the knowledge-enhanced representation of each modality.

[0072] Specifically, in this embodiment, the input of each modality I m∈{t,v,a} is encoded into global semantic representations and knowledge sentiment representations

[0073] Global semantic representation: for the text modality, this embodiment uses BERT to encode the input sentence I t and extracts the hidden state of the last layer as the global semantic representation H t :

[0074] H t , O t = BERT(I t ; θ BERT ) (1)

[0075] where H t is the global semantic representation for the text modality, I t is the input raw text sentence, θ BERT is the parameter of the BERT model, BERT(.) is to extract the initial text features using the BERT model, and O t represents the hidden states except for the last few layers; for the visual and audio modalities, this embodiment uses stacked Transformer encoder layers to capture the global semantic representation H m , m e {v, a}:

[0076]

[0077] where H m represents the global semantic representation of the visual and audio modalities, O m represents the intermediate hidden states of the visual and audio modalities, Encoder m () represents the extraction of initial features of the visual and audio modalities using the TransformerEncoder, I m represents the input raw visual and audio features, represents the parameters of the TransformerEncoder of the visual and audio modalities; since the obtained O m , m e {t, v, a} mainly contains common sense, it is input into the adapter to inject emotional information.

[0078] As shown in Figure 5 , the architecture of the encoder, the adapter, and their connection is shown. The vertical gray rectangle represents the Transformer encoder layer. Down-FC and Up-FC represent the fully connected layers (FC) used to reduce and increase the dimension.

[0079] Knowledge-emotional representation: Since the adapter is usually used to enhance the knowledge cognition of the pre-trained language model, this embodiment uses it to inject single-modality emotional knowledge. In KuDA, the adapter is inserted outside the encoder and stacked with the same block, and its detailed architecture is shown in Figure 5 . For the first adapter block, this embodiment takes the single-modality feature I m∈{t,v,a} as one of the inputs. The output of the adapter is represented as the knowledge-emotional representation K m , m e {t, v, a}:

[0080]

[0081] wherein, denotes the pre-trained parameters of the adapter of the m-th modality, K m denotes the knowledge sentiment representation, denotes that the obtained data dimension belongs to T m ×d m , T m denotes the length of the knowledge sentiment representation, d m denotes the dimension of the knowledge sentiment representation.

[0082] Single-modality sentiment value: the embodiment combines the above two representations to obtain a knowledge-enhanced representation U m for each modality, m e {t, v, a}. Then, the embodiment utilizes the representation to predict a single-modality sentiment value y

[0083]

[0084] wherein, wherein, U m denotes the knowledge-enhanced representation, denotes the single-modality sentiment value, Decoder m () denotes predicting the single-modality sentiment value by utilizing the Decoder module, denotes the parameters of each single-modality Decoder, [·; ·] denotes concatenation. Since the difference between the single-modality and multi-modality sentiment scores can indicate the amount of effective information provided by the corresponding modality, the embodiment uses the sentiment score to further guide the attention weight.

[0085] Further, the knowledge-enhanced representation and the sentiment ratio are subjected to preset dynamic attention processing to obtain a final multi-modality representation, including:

[0086] mapping the knowledge-enhanced representation to unify the length and dimension axis of the knowledge-enhanced representation;

[0087] presetting a plurality of stacked dynamic attention modules;

[0088] taking the mapped knowledge-enhanced representation as the input of the first dynamic attention module;

[0089] taking the output of the current dynamic attention module, the mapped knowledge-enhanced representation and the sentiment ratio as the input of the next dynamic attention module until the output of the last dynamic attention module is obtained as the final multi-modality representation.

[0090] Specifically, in the embodiment, since the single-modality sentiment score y mThe difference from the multi-modal sentiment score y is inversely proportional to the weight of the single mode, and the embodiment selects the inverse proportional function exp(-kx) and the normalization operation, and utilizes the real label y of MSA in the training process to convert the single mode sentiment value into a sentiment ratio R m , m e {t, v, a}, to guide the subsequent dynamic fusion:

[0091]

[0092] where R m is the sentiment ratio, D m represents an intermediate variable for converting the single mode sentiment value, exp() represents the nth power of e, n is the calculation value in the parentheses, k represents the slope of the function, which can scale the sentiment ratio, represents the single mode sentiment value, y represents the label value, D t represents an intermediate variable for converting the text modal sentiment value, D v represents an intermediate variable for converting the visual modal sentiment value, and D a represents an intermediate variable for converting the audio modal sentiment value. Since the model effectively learns how to adjust the contribution between modes during training, the embodiment fixes the sentiment ratio to 1 in the test stage to highlight the ability of the model to adjust the weight.

[0093] As shown in Figure 6 , it is the architecture of a single dynamic attention module in the embodiment. In order to unify the length and dimension axes of the single mode knowledge enhanced representation of multi-modal fusion, the embodiment utilizes three mappers to update and obtain the knowledge enhanced representation of each mode. In addition, since any one or more of the text, visual and audio modes can become the dominant mode, the embodiment first adds the obtained representation as the input of the first dynamic attention block F 0 :

[0094]

[0095] where Projector m is composed of two linear layers, T f and d f represent the length and dimension of the fusion stage.

[0096] Then the embodiment stacks the dynamic attention modules to form a pipeline, as shown in Figure 4 . At the same time, the embodiment adds the output F n-1 of the previous module, the knowledge enhanced representation and the sentiment ratio R m, m e {t, v, a} as input and get its output F n :

[0097]

[0098] where F n is the final multimodal representation, F n-1 is the output of the current dynamic attention module, DAB is the dynamic attention block, is the mapped knowledge enhanced representation, R m is the sentiment ratio, is the dimension of the multimodal fusion feature, T f is the length of the fusion feature, d f is the dimension of the fusion feature. DAB is the dynamic attention block. This is because the representation has richer sentiment cues, and the sentiment ratio R m can guide the contribution of each modality. Finally, the output of the last block is taken as the final multimodal representation and used to perform the MSA task.

[0099] In the dynamic attention module, first, the input data of each single modality (referring to the knowledge enhanced representation of each modality after mapping) and the multimodal output of the previous layer (i.e. the output F n-1 of the previous module as input of the next module) are captured by the corresponding cross-modal attention mechanism, capturing the amount of similar information between them and performing layer normalization, gradually determining the dominant modality. Next, the emotion ratio is multiplied with the above obtained representation, and then a residual operation is performed to obtain three multimodal fusion features with single modality as auxiliary modality. Finally, the three multimodal fusion features with single modality as auxiliary modality are input into the multi-head attention mechanism and the feedforward neural network to obtain the output of the current dynamic attention module.

[0100] As Figure 6 shown, in order to adjust the weight of each modality, the embodiment designs a dynamic attention module. Specifically, the embodiment first introduces cross-modal attention (CAttn), which gradually determines the dominant modality by capturing the amount of similar information between the single modality representation and the multimodal representation F n-1 . Since the Q of attention is used to specify the position of attention, the embodiment takes the multimodal feature as Q, takes the single modality feature as K and V, and performs layer normalization (LN):

[0101]

[0102] Next, since the sentiment ratio R mThe dynamic fusion can be further guided, and the embodiment multiplies it with the intermediate representation Then, the embodiment adds the obtained representation to the input F n-1 Multi-modal representation to fine-tune the contribution of different modalities:

[0103]

[0104] Finally, the input is input into a multi-head attention and a feed-forward neural network to obtain the output of the dynamic attention block

[0105] To further improve the utilization rate of the dominant modalities, the embodiment estimates the correlation between the multi-modal representation F L and the single-modal representation by contrastive prediction coding, and integrates it into the noise contrast estimation framework to form a correlation estimation (CE) loss

[0106]

[0107] In the output, the embodiment inputs the representation F L into an MLP to predict the sentiment score Given the prediction and the label y, the embodiment calculates the MSA task loss by the mean absolute error Finally, the embodiment trains KuDA by the joint loss

[0108]

[0109] wherein, is the multi-modal sentiment prediction value, MLP() is a multi-layer perceptron, Mean() is the average value of data in the length dimension, F L is the final multi-modal representation, i.e., the output of the last fusion layer, is a regression loss function, N is the total number of samples, y i is the label of sample i, is the multi-modal sentiment prediction value of sample i, is a regression loss function, is a correlation evaluation loss function, and α is a hyperparameter for balancing the loss contribution, is a preset joint loss.

[0110] ​In this embodiment, a knowledge-guided dynamic modality attention fusion framework (KuDA) is proposed to improve the performance of the model and adapt it to more complex and extensive scenarios by dynamically selecting the dominant modality and adjusting the contribution of each modality according to different samples. Specifically, KuDA first uses the BERT model and two Transformer encoders to extract the semantic features of the text, visual, and audio modalities. Then, KuDA performs sentiment knowledge injection and sentiment proportion conversion through an adapter and a decoder to extract sentiment cues and guide KuDA to further select the dominant modality. Next, a dynamic attention fusion module is designed to capture similar sentiment information and gradually adjust the attention weights between modalities by interacting sentiment knowledge with multi-modal features at different levels. On the basis of evaluating the correlation between multi-modal features and single-modal features, this embodiment uses noise contrast estimation to further highlight the contribution of the dominant modality. Finally, KuDA predicts the sentiment score through a multi-layer perceptron. The main contributions of this embodiment can be summarized as follows:

[0111] This embodiment proposes KuDA, a knowledge-guided dynamic modality attention fusion framework for multi-modal sentiment analysis, which improves performance by dynamically selecting the dominant modality, enabling the model to adapt to complex and extensive scenarios.

[0112] This embodiment designs a dynamic attention fusion module that uses sentiment knowledge to guide multi-modal features at different levels and achieves dynamic fusion by adjusting the contribution of each modality.

[0113] Extensive experiments on four MSA datasets show that KuDA achieves the best performance. This embodiment further analyzes the experimental results to demonstrate the effectiveness of KuDA.

[0114] Experiments were conducted on four publicly available MSA benchmark datasets, including CH-SIMS, CH-SIMSv2, MOSI, and MOSEI. CH-SIMS is a Chinese MSA dataset containing visual, audio, and textual modalities collected from different segments of movies and TV series. Each segment has a multi-modal label and three single-modal labels, and a sentiment score from -1 (strongly negative) to 1 (strongly positive) is manually annotated. The CH-SIMSv2 dataset extends the CH-SIMS dataset. It includes 4403 labeled video segments. Like CH-SIMS, each segment contains a multi-modal and three single-modal sentiment scores from -1 to 1. MOSI is collected from 93 YouTube videos in which speakers express their opinions on movies. It contains 2199 video segments covering visual, audio, and textual modalities. Each segment is annotated with a sentiment score from -3 (strongly negative) to +3 (strongly positive). The MOSEI dataset is an upgrade of MOSI, collecting 22856 YouTube video segments from 250 different topics. Like MOSI, each segment is annotated with a sentiment score in the range [-3, 3]. The statistical details of the four datasets are shown in Table 1.

[0115] Table 1

[0116]

[0117] Following previous studies, this embodiment uses accuracy for 3 classes (Acc-3) and 5 classes (Acc-5) on CH-SIMS and CH-SIMSv2, accuracy for 7 classes (Acc-7) on MOSI and MOSEI, and accuracy for 2 classes (Acc-2), mean absolute error (MAE), Pearson correlation coefficient (Corr), and F1 score (F1) on all datasets. In addition, on MOSI and MOSEI, Acc-2 and F1 use two calculation methods: has-0 and non-0. Except for MAE, the higher the value of all indicators, the better the performance.

[0118] To verify the performance of KuDA, this embodiment is compared with multiple competitive and state-of-the-art (SOTA) ternary symmetric and text-centric based baselines.

[0119] Symmetry-based methods: TFN explores multi-modal interactions by introducing multi-dimensional tensors through computing outer products. LMF decomposes high-order tensors into low-order tensors to reduce complexity and perform efficient interactions. MFN designs a delta-attention module with multi-view gated memory networks to explore inter-modal dynamics. MulT adopts directional pairwise cross-modal attention to alleviate the restriction of rigid temporal alignment in multi-modal signals. MISA maps various single-modal features into different embedding subspaces and facilitates the fusion process. Self-MM utilizes the overall label of multi-modal objects to learn the sentiment label of single-modal objects, thereby extracting more discriminative single-modal features. CubeMLP treats multi-modal fusion as feature mixing based on the MLP framework.

[0120] Text-centric methods: BBFN focuses on dual-modal fusion and fully balances the contributions of different modal pairs. MMIM maximizes the mutual information between multi-modal fusion output and single-modal input, aiming to reduce feature noise. CENet enhances text representation by integrating visual and audio information into the language model. ALMT uses an adaptive hyper-modal learning module to remove redundant noise information from visual and audio features.

[0121] The experimental settings and training processes of the present embodiment are shown in Table 2.

[0122] Table 2

[0123]

[0124] To ensure fairness with other benchmarks, the present embodiment follows recent competitive and SOTA methods to set the proposed method. The training phase includes a single-modal phase and a multi-modal phase. In addition, the present embodiment uses BERT to extract features of the text modality, where bert-base-chinese is used for CH-SIMS and CH-SIMSv2, and bert-base-uncased is used for MOSI and MOSEI. In the visual and audio modalities, the present embodiment directly uses the features provided by the original dataset. In addition, the present embodiment uses a single NVIDIA RTX 3090 GPU to develop KuDA for all datasets. The detailed settings of the best hyperparameters can be referred to Table 2.

[0125] KuDA adopts a two-stage training method, and the specific details are shown in Algorithm 1. In the first stage, the embodiment pre-trains the knowledge-injected encoding module using external data. For CH-SIMS and CH-SIMSv2, the embodiment pre-trains using the single-modal labels of the dataset itself. For MOSI and MOSEI, considering the data scale, the embodiment translates the text of CH-SIMS and CH-SIMSv2 into English, and injects the sentiment knowledge of CH-SIMS into MOSI and the sentiment knowledge of CH-SIMSv2 into MOSEI. It is worth noting that in order to make a fair comparison with the baseline, the embodiment only pre-trains the single-modal sentiment prediction task on all datasets, without involving the MSA task. In the second stage, in order to prevent the pre-training knowledge from being covered, the embodiment freezes the adapter and decoder of each modality, and performs the MSA task based on the pre-training knowledge.

[0126] Algorithm 1

[0127]

[0128] Tables 3 and 4 show the comparison results of the baseline and the proposed method on CH-SIMS, CH-SIMSv2, MOSI and MOSEI. Since the distribution of the importance of the modalities in CH-SIMS and CH-SIMSv2 is more uniform, they are more complex than MOSI and MOSEI. As shown in Table 3, the proposed method outperforms all baselines in all indicators. It is worth noting that the method of the embodiment achieves excellent performance on the CH-SIMSv2 dataset. For example, compared with Self-MM and ALMT, the method of the embodiment achieves an improvement of 8.45% and 8.32% in Acc-5, and also achieves significant improvement in Acc-3. Therefore, the excellent performance in a more challenging scenario indicates that KuDA can adjust the distribution of the weights of the modalities to complete dynamic fusion. This also indicates that adjusting the dominant modality is crucial for MSA. In addition, although the importance distribution of the modalities in MOSI and MOSEI is not uniform, and the text modality plays an important role, as can be seen from Table 4, KuDA still achieves SOTA performance in almost all indicators. At the same time, KuDA surpasses some text-centered training methods, such as BBFN and ALMT.

[0129] Table 3

[0130]

[0131] Table 4

[0132]

[0133] Ablation studies were conducted to verify the effectiveness of each component, as shown in Table 5. By comparing "w / o DAF" and KuDA, the present embodiment observes that removing DAF severely degrades performance. This means that KuDA dynamically adjusts the attention weights between modalities to select the dominant modality for different scenarios. In addition, the performance under "w / o EKI" is degraded, which indicates that emotional knowledge can further guide dynamic fusion. The performance is degraded after removing other modules, indicating their effectiveness. Corr is improved because improving the utilization of visual and audio on MOSI introduces noise.

[0134] Table 5

[0135]

[0136] To verify the influence of each modality, the present embodiment conducts an ablation study by removing each modality, as shown in Table 6. KuDA achieves SOTA performance when all modalities are available, which indicates adaptive selection of the dominant modality to capture emotional information. When any one modality is removed, the performance is degraded, but still acceptable results can be obtained, which proves that KuDA can still focus on suboptimal modalities. Notably, by observing the performance degradation trend after removing a certain modality, it can be observed that the importance of each modality is evenly distributed in CH-SIMSv2, while the importance of the text modality is higher in MOSI.

[0137] Table 6

[0138]

[0139] To analyze the effects of different fusion techniques, the present embodiment conducts some experiments, as shown in Table 7. Obviously, using fusion methods based on ternary symmetry (TFN, LMF) or text center (BBFN, ALMT) will cause performance degradation when facing complex scenarios. This indicates that not focusing on the dominant modality or statically setting the dominant modality will limit the performance of MSA. However, using the dynamic attention fusion of the present embodiment to dynamically fuse single-modality features is the most effective.

[0140] Table 7

[0141]

[0142] As Figure 7As shown, this embodiment discusses the impact of CE loss on CH-SIMSv2 and MOSI by modifying a. This embodiment compares MAE, Acc-5 and Acc-7 as these metrics indicate the ability of the method to predict fine-grained sentiment. Compared to removing CE loss (a = 0), the model performance reaches STOA when a is 0.01. This indicates that CE loss can further highlight the contribution of the dominant modality. However, when a increases, the performance presents a downward trend, indicating that when CE loss increases, KuDA enhances the preservation of non-dominant modality features in the multi-modal representation, thus limiting its performance.

[0143] To verify that KuDA can dynamically select the dominant modality, this embodiment visualizes the text, visual, audio and multi-modal features on CH-SIMSv2 using t-SNE, as shown in Figure 8 The left plot in Figure 8 , this embodiment can see that, due to CubeMLP's equal treatment of each modality's contribution, all single-modal features are averaged around the multi-modal features. Moreover, as shown in the middle plot in Figure 8 , due to ALMT being a text-centric method, this embodiment can observe that the text features are located in the middle of audio and visual, while all single-modal features are distributed on the other side of the multi-modal features. However, as can be seen from the right plot in Figure 8 , the difference is that KuDA's multi-modal features are divided into three clusters, close to the text, audio and visual features, respectively. This indicates that KuDA dynamically selects the dominant modality, making the multi-modal features closer to it.

[0144] As shown in Figure 9 , three cases in CH-SIMS. Ground Truth consists of single-modal labels (V, T and A) and multi-modal label (M). The output shows the multi-modal predictions from KuDA. Note that the weights show the attention weight distribution of the multi-modal features with visual, text and audio modality features. In the attention weights, the left, middle and right plots of each case represent the multi-modal and visual, text and audio modality weights, respectively. The partial attention of the dominant modality is marked with a dashed box.

[0145] To better demonstrate that the method of this embodiment can dynamically adjust the contribution of different modalities, this embodiment selects three challenging cases for further analysis, as shown in Figure 9The correct predictions can be observed in case (a) even though the text and audio express strong negative sentiment. This case shows that KuDA effectively captures the speaker's expression and action information by adjusting the dominant modality, which guides the fusion of the text and audio modalities. Similar label distributions also appear in cases (b) and (c). KuDA still makes correct predictions, which shows that it captures the text semantic information in case (b) and the audio prosody information in case (c) by adjusting the attention weights. Meanwhile, from the attention weights of the text, audio, and visual modalities, it can be seen that the attention weights of the dominant modality (with denser dark blocks) are higher than those of the other modalities. This again demonstrates the importance of dynamic attention fusion for the MSA task. Figure 9

[0146] The above-described embodiments are merely intended to describe the preferred modes of the present application, and are not intended to limit the scope of the present application. Various modifications and improvements to the technical solutions of the present application made by those of ordinary skill in the art without departing from the design spirit of the present application shall fall within the scope of protection of the present application as defined by the claims.​

Claims

1. A multi-modal sentiment analysis method based on knowledge guidance and modal dynamic attention fusion, characterized in that, The method comprises the following steps: extracting single-modal features of input multi-modal data; obtaining knowledge-enhanced representations of each modality based on the single-modal features; predicting single-modal sentiment values based on the knowledge-enhanced representations and converting the single-modal sentiment values into sentiment ratio values; the knowledge-enhanced representation and the single-modal sentiment value of each modality are respectively: wherein U m represents a knowledge enhanced representation, represents a single modality sentiment value, Decoder m () represents predicting single modality sentiment values with Decoder module, represents parameters of each single modality Decoder, [·; ·] represents concatenation, H m represents global semantic representation of visual and audio modalities, K m represents a knowledge sentiment representation, represents that the obtained data dimension belongs to T m × d m , T m represents the length of the knowledge sentiment representation, d m represents the dimension of the knowledge sentiment representation; the method for converting the single-modal sentiment values into sentiment ratio values is: Wherein, R m is an emotional ratio, D m represents an intermediate variable for single-modal emotion value conversion, exp() represents e to the power of n, n is the calculated value in the parentheses, k represents the slope, y represents the label value, D t represents an intermediate variable for converting text modal emotion values, D v represents an intermediate variable for converting visual modal emotion values, D a represents an intermediate variable for converting audio modal emotion values; performing preset dynamic attention processing on the knowledge-enhanced representations and the sentiment ratio values to obtain a final multi-modal representation; predicting a sentiment score based on the final multi-modal representation.

2. The method of claim 1, wherein the knowledge-guided and modality dynamic attention fusion-based multi-modal sentiment analysis method is characterized by, The single-modal features include semantic features of a text modality, semantic features of a visual modality, and semantic features of an audio modality.

3. The method of claim 1, wherein the knowledge-guided and modality dynamic attention fusion-based multi-modal sentiment analysis method is characterized by, Obtaining the knowledge-enhanced representation of each modality based on the single-modal features comprises the following steps: encoding the single-modal features into global semantic representations and knowledge sentiment representations; combining the global semantic representations and the knowledge sentiment representations to obtain the knowledge-enhanced representation of each modality.

4. The method of claim 3, wherein the knowledge guidance and modal dynamic attention fusion-based multi-modal sentiment analysis method is characterized by, The global semantic representations include global semantic representations of a text modality and global semantic representations of a visual and audio modality. The global semantic representation of the text modality is: H t , O t = BERT(I t ; θ BERT ) (1) wherein H t is the global semantic representation of the text modality, I t is the original text sentence of the input, θ BERT is the parameter of the BERT model, BERT(.) is the initial text feature extracted by using the BERT model, O t represents the hidden state except for the last few layers; The global semantic representation of the visual and audio modality is: where H m represents the global semantic representation of visual and audio modalities, O m represents the intermediate hidden states of visual and audio modalities, Encoder m () represents the initial features of visual and audio modalities extracted by TransformerEncoder, I m represents the original visual and audio features of the input, represents the parameters of the TransformerEncoder of visual and audio modalities; The knowledge sentiment representation is: wherein, denotes pre-trained parameters of the adapter of the m-th modality, K m denotes a knowledge sentiment representation, denotes that the obtained data dimension belongs to T m × d m , T m denotes a length of the knowledge sentiment representation, d m denotes a dimension of the knowledge sentiment representation.

5. The method of claim 1, wherein, Performing preset dynamic attention processing on the knowledge-enhanced representations and the sentiment ratio values to obtain a final multi-modal representation comprises the following steps: mapping the knowledge-enhanced representations to unify the length and dimension axis of the knowledge-enhanced representations; presetting a plurality of stacked dynamic attention modules; taking the mapped knowledge-enhanced representations as the input of the first dynamic attention module; taking the output of the current dynamic attention module, the mapped knowledge-enhanced representations, and the sentiment ratio values as the input of the next dynamic attention module until the output of the last dynamic attention module is obtained as the final multi-modal representation.

6. The method of claim 5, wherein the knowledge guidance and modality dynamic attention fusion-based multi-modal sentiment analysis method is characterized by, The final multi-modal representation is: where F n is the final multimodal representation, F n-1 is the output of the current dynamic attention module, DAB is the dynamic attention block, is the mapped knowledge enhanced representation, R m is the sentiment ratio, is the dimension of the multimodal fusion feature, T f is the length of the fusion feature, d f is the dimension of the fusion feature.

7. The method of claim 1, wherein the knowledge-guided and modality dynamic attention fusion-based multi-modal sentiment analysis method is characterized by, Before predicting the sentiment score, the method further comprises the following step: training the final multi-modal representation by using a preset joint loss; The preset joint loss is: wherein, is the multimodal sentiment prediction value, MLP() is a multilayer perceptron, Mean() is the mean of data in the length dimension, F L is the final multimodal, N is the total number of samples, y i is the label of sample i, is the multimodal sentiment prediction value of sample i, is the regression loss function, is the correlation evaluation loss function, and a is a hyperparameter balancing the loss contribution, is the preset joint loss.

Images