Multi-modal sentiment analysis method based on knowledge guidance and modal dynamic attention fusion

Abstract

The invention relates to a multi-modal sentiment analysis method based on knowledge guidance and modal dynamic attention fusion. The method comprises the following steps: extracting single-modal features of input multi-modal data; based on the single-mode features, obtaining knowledge enhancement representation of each mode; predicting a single-mode emotion value based on the knowledge enhancement representation, and converting the single-mode emotion value into an emotion ratio; performing preset dynamic attention processing on the knowledge enhancement representation and the emotion ratio to obtain a final multi-modal representation; an emotion score is predicted based on the final multi-modal representation. By dynamically selecting the dominant mode and adjusting the contribution of each mode according to different samples, the performance of the multi-mode sentiment analysis method can be improved, and the multi-mode sentiment analysis method can adapt to more complex and wider scenes.

Description

Technical Field

[0001] The present invention relates to the technical field of multimodal sentiment analysis, and in particular to a multimodal sentiment analysis method based on knowledge guidance and modal dynamic attention fusion. Background Art

[0002] As users' emotional expressions are reflected in multiple modalities in social media, multimodal sentiment analysis (MSA) has attracted increasing attention in recent years. It aims to mine and understand the emotions of online videos. Recent MSA methods can be divided into two categories: ternary symmetry-based methods and text-centric methods. In the ternary symmetry-based methods, previous studies mainly focus on modeling the bidirectional relationship in each modality pair and treat the contribution of each modality equally. For example, some researchers designed tensor-based, LSTM-based, and MLP-based fusion architectures to capture the commonalities between modalities simultaneously. While Tsai et al. designed MuLT, a transformer-based model to find similar information between each modality pair. In addition, some studies projected each modality into two subspaces and learned the commonalities and features of the modalities respectively to assist the fusion process. Some researchers also introduced multi-label strategies and contrastive learning to equally improve the quality of single-modal representations. In the text-centric methods, previous studies mainly focused on guiding the visual and audio modalities through the text modality to improve the quality of the fused representation. For example, Delbrouck et al. and Han et al. used transformer-based methods to integrate similar information from other modalities through text modality. Some researchers have enhanced text representation by integrating visual and audio information into pre-trained language models. In addition, in order to reduce potential sentiment-irrelevant and conflicting information, some studies have reduced additional noise by maximizing mutual information or adaptive representation learning. At the same time, some researchers introduced contrastive learning to learn the invariant and similar information between text and other modalities. Although MSA has achieved encouraging results, the above methods treat each modality in a balanced manner or statically set the text as the dominant modality. This causes the method to be distracted by the secondary modality, hindering its dynamic adjustment to adapt to various scenarios where the dominant modality changes, thereby limiting the performance of MSA.

[0003] Recent MSA methods can be divided into two categories: triad-based methods and text-centric methods. Triad-based methods focus on treating the contribution of each modality equally and modeling the bidirectional relationship of all modal pairs. Text-centric methods focus on using text as the dominant modality to guide the visual and audio modalities to interact with it to appropriately adjust the contributions of different modalities. Therefore, both triad-based methods and text-centric methods assume that the importance distribution between modalities is static and fix the dominant modality. However, as Figure 1As shown, the applicant has found that in some cases, vision, text or audio can each become the dominant modality. Figure 1 The first sample in , dominates because its visual label is more consistent with the overall sentiment label.

[0004] Based on a survey of commonly used datasets in MSA, the applicant found that these situations are not uncommon. In order to explore the distribution of samples dominated by each modality and verify the applicant's ideas, the applicant studied four MSA benchmark datasets (MOSI, MOSEI, CH-SIMS, and CH-SIMSv2). The statistical and analysis results are as follows. For MOSI and MOSEI, some researchers have shown that the text modality is more important and dominant. At the same time, some ablation studies reported that after removing the text modality, the binary accuracy of these two datasets dropped by about 30% (80%+ when there is text and about 50%+ when there is no text). Therefore, most samples in MOSI and MOSEI have text as the dominant modality.

[0005] For CH-SIMS and CH-SIMSv2, the applicant uses the unimodal labels provided by the dataset to count the number of samples. Figure 2 As shown, it can be observed that the number of samples dominated by visual, textual and audio modalities accounts for 45%-60% of the total, and the distribution of each modality is relatively uniform. This shows that each modality will dominate, which is not uncommon. This also shows that the dominant modality needs to be adjusted dynamically. It is worth noting that the sum of the proportions of the three modalities in the same dataset exceeds 100%, because multiple modalities may dominate in the same sample. The applicant then counted the number of samples containing noise modalities (the emotional polarity of this single modality is different from that of multi-modality), as shown in Figure 3 As shown. It can be seen that the proportion of samples containing noise modes is around 50%, which further indicates that the inability to dynamically adjust the contribution of each mode will limit the performance of MSA. Therefore, when the dominant mode is not fixed, the ternary symmetry-based methods cannot effectively adapt to the situation where any mode is dominant because they do not consider the difference in importance between modes. The text-centric method statically sets the text as the dominant mode, and when other modes are dominant, the model's attention will be distracted by the text. Summary of the invention

[0006] The purpose of the present invention is to propose a multimodal sentiment analysis method based on knowledge guidance and modal dynamic attention fusion to solve the problems existing in the above-mentioned prior art.

[0007] To achieve the above object, the present invention provides the following solutions:

[0008] The multimodal sentiment analysis method based on knowledge guidance and modal dynamic attention fusion includes:

[0009] Extracting unimodal features from input multimodal data;

[0010] Based on the unimodal features, obtaining a knowledge-enhanced representation of each modality;

[0011] Predicting a unimodal sentiment value based on the knowledge enhanced representation, and converting the unimodal sentiment value into a sentiment ratio;

[0012] Performing a preset dynamic attention process on the knowledge enhancement representation and the emotion ratio to obtain a final multimodal representation;

[0013] Based on the final multimodal representation, a sentiment score is predicted.

[0014] Optionally, the unimodal features include: semantic features of text modality, semantic features of visual modality and semantic features of audio modality.

[0015] Optionally, based on the single modal features, obtaining the knowledge enhanced representation of each modality includes:

[0016] Encoding the unimodal features into global semantic representation and knowledge sentiment representation;

[0017] The global semantic representation and the knowledge sentiment representation are combined to obtain a knowledge enhanced representation of each modality.

[0018] Optionally, the global semantic representation includes: a global semantic representation of a textual modality and a global semantic representation of a visual and audio modality;

[0019] The global semantics of the text modality is expressed as:

[0020] H t ,O t =BERT(I t θ BERT ) (1)

[0021] Among them, H t is the global semantic representation of the text modality, I t is the original text sentence input, θ BERT are the parameters of the BERT model, BERT(.) is the initial text features extracted using the BERT model, t represents the hidden states except the last few layers;

[0022] The global semantics of the visual and audio modalities are expressed as:

[0023]

[0024] Among them, H mRepresents the global semantic representation of visual and audio modalities, O m Represents the intermediate hidden states of the visual and audio modalities, Encoder m () indicates the use of TransformerEncoder to extract the initial features of visual and audio modalities, I m represents the raw visual and audio features of the input, Parameters of TransformerEncoder representing visual and audio modalities;

[0025] The knowledge sentiment is expressed as:

[0026]

[0027] in, represents the pre-trained parameters of the adapter for m-mode, K m Represents knowledge emotion expression, Indicates that the obtained data dimension belongs to T m ×d m , T m represents the length of knowledge sentiment representation, d m Represents the dimension of knowledge sentiment representation.

[0028] Optionally, the knowledge enhancement representation and unimodal sentiment value of each modality are respectively:

[0029]

[0030] Among them, U m Represents knowledge enhancement representation, Represents a unimodal sentiment value, Decoder m () indicates using the Decoder module to predict the unimodal sentiment value. Indicates the parameters of each single-mode decoder.

[0031] Optionally, the method of converting the unimodal sentiment value into a sentiment ratio is:

[0032]

[0033] Among them, R m is the emotional ratio, D m It represents the intermediate variable of the unimodal sentiment value conversion, exp() represents the nth power of e, n is the calculated value in brackets, k represents the slope, represents the unimodal sentiment value, y represents the label value, and D t The intermediate variable representing the converted text modality sentiment value, D v Represents the intermediate variable for converting the visual modality sentiment value, D aRepresents the intermediate variable for converting the audio modality emotion value.

[0034] Optionally, performing a preset dynamic attention process on the knowledge enhancement representation and the emotion ratio to obtain a final multimodal representation includes:

[0035] Mapping the knowledge enhancement representation to unify the length and dimension axes of the knowledge enhancement;

[0036] Preset several stacked dynamic attention modules;

[0037] The mapped knowledge-enhanced representation is used as the input of the first dynamic attention module;

[0038] The output of the current dynamic attention module, the mapped knowledge enhancement representation and the sentiment ratio are used as the input of the next dynamic attention module until the output of the last dynamic attention module is obtained as the final multimodal representation.

[0039] Optionally, the final multimodal representation is:

[0040]

[0041] Among them, F n is the final multimodal representation, F n-1 is the output of the current dynamic attention module, DAB is the dynamic attention block, is the knowledge enhancement representation after mapping, R m is the emotional ratio, is the dimension of multimodal fusion features, T f is the length of the fusion feature, d f is the dimension of fusion features.

[0042] Optionally, before predicting the sentiment score, the method further includes: training the final multimodal representation using a preset joint loss;

[0043] The preset joint loss is:

[0044]

[0045] in, is the multimodal sentiment prediction value, MLP() is a multi-layer perceptron, Mean() is the average value of the data in the length dimension, F L is the final multimodal, that is, the output of the last fusion layer, is the regression loss function, N is the total number of samples, y i is the label of sample i, is the multimodal sentiment prediction value of sample i, is the regression loss function, is the loss function for correlation evaluation, α is the hyperparameter for balancing loss contribution, is the preset joint loss.

[0046] The beneficial effects of the present invention are:

[0047] The present invention proposes a multimodal sentiment analysis method based on knowledge guidance and modal dynamic attention fusion, firstly extracting the unimodal features of the input multimodal data, and obtaining the knowledge enhanced representation of each modality based on the unimodal features of the present embodiment; secondly predicting the unimodal sentiment value based on the knowledge enhanced representation of the present embodiment, and converting the unimodal sentiment value of the present embodiment into a sentiment ratio; then performing a preset dynamic attention processing on the knowledge enhanced representation of the present embodiment and the sentiment ratio of the present embodiment to obtain the final multimodal representation; finally predicting the sentiment score based on the final multimodal representation of the present embodiment. The present invention can improve the performance of the multimodal sentiment analysis method and adapt it to more complex and wider scenarios by dynamically selecting the dominant modality and adjusting the contribution of each modality according to different samples. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0049] Figure 1 Schematic diagram of three video samples in the CH-SIMS dataset, with vision, text or audio as the main modality;

[0050] Figure 2 Schematic diagram of the distribution dominated by each mode;

[0051] Figure 3 This is a schematic diagram of the distribution of samples containing noise modes;

[0052] Figure 4 A schematic diagram of the overall architecture of KuDA according to an embodiment of the present invention;

[0053] Figure 5 A schematic diagram of the architecture of an encoder, an adapter and their connection according to an embodiment of the present invention;

[0054] Figure 6 Schematic diagram of the architecture of a single dynamic attention module according to an embodiment of the present invention;

[0055] Figure 7 A schematic diagram showing the performance visualization of CH-SIMSv2 and MOSI as α changes according to an embodiment of the present invention;

[0056] Figure 8 A schematic diagram of visualizing text, visual, audio, and multimodal features on CH-SIMSv2 using t-SNE according to an embodiment of the present invention;

[0057] Fig. 9 Schematic diagram of three cases in CH-SIMS of an embodiment of the present invention. DETAILED DESCRIPTION

[0058] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0059] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0060] This embodiment proposes a multimodal sentiment analysis method based on knowledge guidance and modal dynamic attention fusion, including:

[0061] Extracting unimodal features from input multimodal data;

[0062] Based on the unimodal features, obtaining a knowledge-enhanced representation of each modality;

[0063] Predicting a unimodal sentiment value based on the knowledge enhanced representation, and converting the unimodal sentiment value into a sentiment ratio;

[0064] Performing a preset dynamic attention process on the knowledge enhancement representation and the emotion ratio to obtain a final multimodal representation;

[0065] Based on the final multimodal representation, a sentiment score is predicted.

[0066] Specifically, in this embodiment, Figure 4As shown in the figure, the overall workflow of KuDA is demonstrated. Specifically, KuDA first extracts unimodal low-level features from the original multimodal input. Then, the adapter and encoder extract unimodal high-level features and learn sentiment knowledge at the same time. This embodiment uses the decoder to predict unimodal sentiment and convert it into sentiment ratio to guide dynamic fusion. Next, this embodiment designs a dynamic attention fusion module, which selects the dominant modality according to different scenarios and dynamically adjusts the attention weights using sentiment ratios and knowledge representations. Finally, the multimodal representation is used to perform the MSA task through a multi-layer perceptron (MLP) and estimate the correlation with the knowledge representation. In addition, in order to guide the model to adjust the attention weights through sentiment knowledge, KuDA adopts a two-stage training method.

[0067] Furthermore, the unimodal features include: semantic features of text modality, semantic features of visual modality and semantic features of audio modality.

[0068] Specifically, in this embodiment, in the MSA task, the input data consists of three modalities: text (t), vision (v), and audio (a). The sequence of the three modalities is represented as a triple (I t ,I v ,I a ), which includes and Where T m ,m∈{t,v,a} is the sequence length, d m represents the vector dimension. The prediction is the sentiment score That is, the sentiment polarity (positive, neutral or negative).

[0069] Further, based on the single modal features, obtaining the knowledge enhanced representation of each modality includes:

[0070] Encoding the unimodal features into global semantic representation and knowledge sentiment representation;

[0071] The global semantic representation and the knowledge sentiment representation are combined to obtain a knowledge enhanced representation of each modality.

[0072] Specifically, in this embodiment, each modality I is converted into m∈{t,v,a} The input encoding is a global semantic representation and knowledge sentiment representation

[0073] Global semantic representation: For the text modality, this embodiment uses BERT to represent the input sentence I t Encode and extract the hidden state of the last layer as the global semantic representation H t :

[0074] H t , O t =BERT(I t θ BERT ) (1)

[0075] Among them, H t is the global semantic representation of the text modality, I t is the original text sentence input, θ BERT are the parameters of the BERT model, BERT(.) is the initial text features extracted using the BERT model, t represents the hidden states except the last few layers; for visual and audio modalities, this embodiment uses stacked Transformer encoder layers to capture the global semantic representation H m ,m∈{v,a}:

[0076]

[0077] Among them, H m Represents the global semantic representation of visual and audio modalities, O m Represents the intermediate hidden states of the visual and audio modalities, Encoder m () indicates the use of TransformerEncoder to extract the initial features of visual and audio modalities, I m represents the raw visual and audio features of the input, Represents the parameters of the TransformerEncoder of the visual and audio modalities; since the obtained O m ,m∈{t,v,a} mainly contains common sense, so it is input into the Adapter to inject emotional information.

[0078] like Figure 5 Figure 1 shows the architecture of the encoder, adapter, and their connections. The vertical gray rectangles represent the Transformer encoder layers. Down-FC and Up-FC represent fully connected layers (FC) for reducing and increasing dimensionality.

[0079] Knowledge and emotion representation: Since the adapter is usually used to enhance the knowledge cognition of the pre-trained language model, this embodiment uses it to inject unimodal emotion knowledge. In KuDA, the adapter is inserted outside the encoder and stacked with the same blocks. Its detailed architecture is as follows Figure 5 As shown. For the first adapter block, this embodiment converts the single-mode feature I m∈{t,v,a} As one of the inputs. The output of the adapter is represented as the knowledge sentiment representation K m ,m∈{t,v,a}:

[0080]

[0081] in, represents the pre-trained parameters of the adapter for m-mode, K m Represents knowledge emotion expression, Indicates that the obtained data dimension belongs to T m ×d m , T m represents the length of knowledge sentiment representation, d m Represents the dimension of knowledge sentiment representation.

[0082] Single-mode sentiment value: This embodiment combines the above two representations to obtain the knowledge-enhanced representation U of each modality. m ,m∈{t,v,a}. Then, this embodiment uses this representation to predict the unimodal sentiment value through a decoder composed of MLP

[0083]

[0084] Among them, U m Represents knowledge enhancement representation, Represents a unimodal sentiment value, Decoder m () indicates using the Decoder module to predict the unimodal sentiment value. = represents the parameters of each unimodal decoder, and [·;·] represents a connection. Since the difference between unimodal and multimodal sentiment scores can indicate the amount of effective information provided by the corresponding modality, this embodiment uses the sentiment score to further guide the attention weights.

[0085] Further, performing a preset dynamic attention process on the knowledge enhancement representation and the emotion ratio to obtain a final multimodal representation includes:

[0086] Mapping the knowledge enhancement representation to unify the length and dimension axes of the knowledge enhancement;

[0087] Preset several stacked dynamic attention modules;

[0088] The mapped knowledge-enhanced representation is used as the input of the first dynamic attention module;

[0089] The output of the current dynamic attention module, the mapped knowledge enhancement representation and the sentiment ratio are used as the input of the next dynamic attention module until the output of the last dynamic attention module is obtained as the final multimodal representation.

[0090] Specifically, in this embodiment, since the unimodal sentiment score y mThe difference between the multimodal sentiment score y and the unimodal weight is inversely proportional to the weight of the unimodal. This embodiment selects the inverse proportional function exp(-kx) and the normalization operation, and uses the real label y of MSA to convert the unimodal sentiment value Converted into emotional ratio R m ,m∈{t,v,a}, to guide the subsequent dynamic fusion:

[0091]

[0092] Among them, R m is the emotional ratio, D m It represents the intermediate variable of the conversion of unimodal sentiment value. exp() represents the nth power of e, n is the calculated value in brackets, and k represents the slope of the function, which can scale the sentiment ratio. represents the unimodal sentiment value, y represents the label value, and D t The intermediate variable representing the converted text modality sentiment value, D v Represents the intermediate variable for converting the visual modality sentiment value, D a Represents an intermediate variable for converting the audio modality emotion value. Since the model effectively learns how to adjust the contribution between the modalities during the training process, this embodiment fixes the emotion ratio to 1 during the test phase to highlight the model's ability to adjust weights.

[0093] like Figure 6 The following is the architecture of a single dynamic attention module in this embodiment. In order to unify the single-modal knowledge enhancement representation of multimodal fusion The length and dimension axes of , this embodiment uses three mappers to update and obtain the knowledge enhanced representation of each modality In addition, since any one or more of the text, visual, and audio modalities may become the dominant modality, this embodiment first obtains the representation Add as the first dynamic attention block F 0 Input:

[0094]

[0095] Projector m It consists of two linear layers, T f and d f Indicates the length and dimension of the fusion stage.

[0096] Then this embodiment stacks the dynamic attention modules to form a pipeline, such as Figure 4 At the same time, this embodiment converts the output F of the previous module n-1 , Knowledge Enhanced Representation and sentiment ratio R m,m∈{t,v,a} is used as the input of the next module and its output F is obtained n :

[0097]

[0098] Among them, F n is the final multimodal representation, F n-1 is the output of the current dynamic attention module, DAB is the dynamic attention block, is the knowledge enhancement representation after mapping, R m is the emotional ratio, is the dimension of multimodal fusion features, T f is the length of the fusion feature, d f is the dimension of fusion features. DAB is the dynamic attention block. This is because With richer emotional clues, the emotional ratio R m The contribution of each modality can be guided. Finally, this embodiment uses the output of the last block as the final multimodal representation And use it to perform MSA tasks.

[0099] In the dynamic attention module, first, the input data of each unimodal layer (referring to the knowledge-enhanced representation of each modality after mapping) is compared with the multimodal output of the previous layer (i.e., the output F n-1 As the input of the next module), the corresponding cross-modal attention mechanism is used to capture the similar information between them and perform layer norms to gradually determine the dominant mode. Next, the sentiment ratio is multiplied by the above-obtained representation, and then a residual operation is performed to obtain three multimodal fusion features with a single modality as the auxiliary modality. Finally, the three multimodal fusion features with a single modality as the auxiliary modality are input into the multi-head attention mechanism and the feedforward neural network to obtain the output of the dynamic attention module of this layer.

[0100] like Figure 6 As shown in Figure 2, in order to adjust the weight of each modality, this embodiment designs a dynamic attention module. Specifically, this embodiment first introduces cross-modal attention (CAttn), which captures the unimodal representation and the multimodal representation F n-1 The amount of similar information between them is used to gradually determine the dominant mode. Since the attention Q is used to specify the attention position, this embodiment takes the multimodal feature as Q, the unimodal feature as K and V, and performs layer norm (LN):

[0101]

[0102] Next, due to the sentiment ratio R mIt can further guide dynamic fusion. In this embodiment, it is combined with the intermediate representation Then, this embodiment combines the obtained representation with the input F n-1 The multimodal representations of are added together to fine-tune the contributions of different modes:

[0103]

[0104] Finally Input into the multi-head attention and feedforward neural network to get the output of the dynamic attention block

[0105] In order to further improve the utilization rate of the dominant mode, this embodiment estimates the multimodal representation F by comparing the predictive coding L With unimodal representation The correlation is integrated into the noise contrast estimation framework to form the correlation estimation (CE) loss.

[0106]

[0107] In output, this example will represent F L Input to MLP to predict sentiment score Given a prediction and label y. In this example, the MSA task loss is calculated by the mean absolute error Finally, this embodiment uses the combined loss Training KuDA:

[0108]

[0109] in, is the multimodal sentiment prediction value, MLP() is a multi-layer perceptron, Mean() is the average value of the data in the length dimension, F L The final multimodal representation is the output of the last fusion layer. is the regression loss function, N is the total number of samples, y i is the label of sample i, is the multimodal sentiment prediction value of sample i, is the regression loss function, is the loss function for correlation evaluation, α is the hyperparameter for balancing loss contribution, is the preset joint loss.

[0110] In this embodiment, a knowledge-guided dynamic modal attention fusion framework (KuDA) is proposed to improve model performance and adapt it to more complex and wider scenarios by dynamically selecting the dominant modality and adjusting the contribution of each modality according to different samples. Specifically, KuDA first uses the BERT model and two Transformer encoders to extract semantic features of text, visual, and audio modalities. Then, KuDA performs sentiment knowledge injection and sentiment ratio conversion through adapters and decoders to extract sentiment clues and guide KuDA to further select the dominant modality. Next, a dynamic attention fusion module is designed to capture similar sentiment information, and the attention weights between modalities are gradually adjusted by interacting sentiment knowledge with multimodal features at different levels. Based on the evaluation of the correlation between multimodal features and unimodal features, this embodiment uses noise contrast estimation to further highlight the contribution of the dominant modality. Finally, KuDA predicts the sentiment score through a multilayer perceptron. The main contributions of this embodiment can be summarized as follows:

[0111] This embodiment proposes KuDA, a knowledge-guided dynamic modal attention fusion framework for multimodal sentiment analysis, which improves performance by dynamically selecting the dominant modality and enables the model to adapt to complex and wide-ranging scenarios.

[0112] This embodiment designs a dynamic attention fusion module, which uses sentiment knowledge to guide multimodal features at different levels and achieves dynamic fusion by adjusting the contribution of each modality.

[0113] Extensive experiments on four MSA datasets show that KuDA achieves the best performance. This example further analyzes the experimental results to prove the effectiveness of KuDA.

[0114] This embodiment conducts experiments on four public MSA benchmark datasets, including CH-SIMS, CH-SIMSv2, MOSI, and MOSEI. CH-SIMS is a Chinese MSA dataset that contains visual, audio, and text modalities collected from different clips of movies and TV series. Each clip has a multimodal label and three unimodal labels, and is manually annotated with sentiment scores ranging from -1 (strongly negative) to 1 (strongly positive). The CH-SIMSv2 dataset extends the CH-SIMS dataset. It includes 4403 labeled video clips. Like CH-SIMS, each clip contains a multimodal and three unimodal sentiment scores ranging from -1 to 1. MOSI is collected from 93 YouTube videos in which speakers express their opinions on movies. It contains 2199 video clips covering visual, audio, and text modalities. Each clip is annotated with a sentiment score ranging from -3 (strongly negative) to +3 (strongly positive). The MOSEI dataset is an upgraded version of MOSI, which collects 22,856 YouTube video clips from 250 different topics. Like MOSI, each clip is annotated with a sentiment score in the range of [-3, 3]. The statistical details of the four datasets are shown in Table 1.

[0115] Table 1

[0116]

[0117] Following previous studies, this example uses the accuracy of 3 classes (Acc-3) and 5 classes (Acc-5) on CH-SIMS and CH-SIMSv2, the accuracy of 7 classes (Acc-7) on MOSI and MOSEI, and the accuracy, mean absolute error (MAE), Pearson correlation coefficient (Corr), and F1 score (F1) of 2 classes (Acc-2) on all datasets. In addition, on MOSI and MOSEI, Acc-2 and F1 are calculated using two methods: negative / non-negative (has-0) and negative / positive (non-0). Except for MAE, the higher the value of all indicators, the better the performance.

[0118] To verify the performance of KuDA, the present embodiment is fairly compared with several competitive and state-of-the-art (SOTA) ternary symmetry-based and text-center-based baselines.

[0119] Methods based on ternary symmetry: TFN introduces multi-dimensional tensors by calculating outer products to explore multimodal interactions. LMF decomposes high-order tensors into low-order tensors to reduce complexity and perform effective interactions. MFN designs a delta-attention module with a multi-view gated memory network to explore inter-modal dynamics. MulT adopts directed pairwise cross-modal attention to alleviate the limitation of rigid time alignment in multimodal signals. MISA maps various unimodal features into different embedding subspaces and facilitates the fusion process. Self-MM uses the overall labels of multimodal objects to learn the sentiment labels of unimodal objects, thereby extracting more discriminative unimodal features. CubeMLP regards multimodal fusion as feature mixing based on the MLP framework.

[0120] Text-centric methods: BBFN focuses on bimodal fusion and fully balances the contributions of different modal pairs. MMIM maximizes the mutual information between multimodal fusion output and unimodal input, aiming to reduce feature noise. CENet enhances text representation by integrating visual and audio information into the language model. ALMT uses an adaptive supermodal learning module to remove redundant noise information from visual and audio features.

[0121] The experimental settings and training process of this embodiment are shown in Table 2;

[0122] Table 2

[0123]

[0124] To ensure fairness with other benchmarks, this embodiment follows recent competitive and SOTA methods to set the proposed method. The training phase includes a unimodal phase and a multimodal phase. In addition, this embodiment adopts BERT to extract features of the text modality, where bert-base-chinese is used for CH-SIMS and CH-SIMSv2, and bert-base-uncased is used for MOSI and MOSEI. In the visual and audio modalities, this embodiment directly uses the features provided by the original datasets. In addition, this embodiment uses a single NVIDIA RTX 3090 GPU to develop KuDA for all datasets. The detailed settings of the optimal hyperparameters can be found in Table 2.

[0125] KuDA adopts a two-stage training method, see Algorithm 1 for details. In the first stage, this embodiment uses external data to pre-train the knowledge injection encoding module. For CH-SIMS and CH-SIMSv2, this embodiment uses the unimodal labels of the dataset itself for pre-training. For MOSI and MOSEI, considering the data scale, this embodiment translates the text of CH-SIMS and CH-SIMSv2 into English, and injects the sentiment knowledge of CH-SIMS into MOSI, and injects the sentiment knowledge of CH-SIMSv2 into MOSEI. It is worth noting that in order to make a fair comparison with the baseline, this embodiment only pre-trains the unimodal sentiment prediction task on all datasets, and does not involve the MSA task. In the second stage, in order to prevent the pre-trained knowledge from being overwritten, this embodiment freezes the adapter and decoder of each modality, and performs the MSA task based on the pre-trained knowledge.

[0126] Algorithm 1

[0127]

[0128] Tables 3 and 4 show the comparison results of the baseline and the proposed method on CH-SIMS, CH-SIMSv2, MOSI, and MOSEI. Since the distribution of modal importance in CH-SIMS and CH-SIMSv2 is more uniform, they are more complex than MOSI and MOSEI. As shown in Table 3, the proposed method outperforms all baselines in all indicators. It is worth noting that the method of this embodiment achieves excellent performance on the CH-SIMSv2 dataset. For example, compared with Self-MM and ALMT, the method of this embodiment achieves 8.45\% and 8.32\% improvements on Acc-5, and also achieves significant improvements on Acc-3. Therefore, achieving excellent performance in more challenging scenarios shows that KuDA can adjust the distribution of modal weights to complete dynamic fusion. This also shows that adjusting the dominant modality is crucial for MSA. In addition, although the distribution of modal importance in MOSI and MOSEI is not uniform, and the text modality plays an important role, it can be seen from Table 4 that KuDA still achieves SOTA performance in almost all indicators. Meanwhile, KuDA surpasses some text-centric training methods such as BBFN and ALMT.

[0129] Table 3

[0130]

[0131] Table 4

[0132]

[0133] This embodiment performs an ablation study to verify the effectiveness of each component, as shown in Table 5. By comparing "w / oDAF" and KuDA, this embodiment observes that removing DAF severely degrades the performance. This means that KuDA dynamically adjusts the attention weights between modalities for different scenarios to select the dominant modality. In addition, the performance under "w / o EKI" decreases, which indicates that emotional knowledge can further guide dynamic fusion. The performance decreases after removing other modules, indicating their effectiveness. Since increasing the utilization of vision and audio on MOSI introduces noise, Corr is improved.

[0134] Table 5

[0135]

[0136] To verify the impact of each modality, this example performs an ablation study by removing each modality, as shown in Table 6. When all modalities are available, KuDA achieves SOTA performance, which demonstrates the adaptive selection of the dominant modality to capture sentiment information. When any modality is removed, the performance degrades, but acceptable results can still be obtained, which proves that KuDA can still focus on the suboptimal modality. It is worth noting that by observing the performance degradation trend after removing a certain modality, it can be observed that the importance of each modality is evenly distributed in CH-SIMSv2, while the importance of the text modality is higher in MOSI.

[0137] Table 6

[0138]

[0139] In order to analyze the effects of different fusion techniques, this embodiment conducts some experiments, as shown in Table 7. Obviously, when facing complex scenes, the use of fusion methods based on ternary symmetry (TFN, LMF) or text center (BBFN, ALMT) will lead to performance degradation. This shows that not paying attention to the dominant mode or statically setting the dominant mode will limit the performance of MSA. However, it is most effective to use the dynamic attention fusion of this embodiment to dynamically fuse single modal features.

[0140] Table 7

[0141]

[0142] like Figure 7As shown, this example discusses the impact of CE loss on CH-SIMSv2 and MOSI by modifying α. This example compares MAE, Acc-5, and Acc-7 because these indicators show the ability of the method to predict fine-grained emotions. Compared with removing CE loss (α = 0), when α is 0.01, the model performance reaches STOA. This shows that CE loss can further highlight the contribution of the dominant modality. However, when α increases, the performance shows a downward trend, indicating that when CE loss increases, KuDA will enhance the retention of non-dominant modality features in multimodal representation, thereby limiting its performance.

[0143] To verify that KuDA can dynamically select the dominant modality, this example uses t-SNE to visualize the features of text, vision, audio, and multimodality on CH-SIMSv2, as shown in Figure 8 Then, this example selects two typical methods, CubeMLP (ternary symmetry) and ALMT (text center), to compare with KuDA. Figure 8 In the left figure of , it can be seen that in this embodiment, since CubeMLP treats the contribution of each modality equally, all unimodal features are averaged around the multimodal features. In addition, as Figure 8 As shown in the middle figure of , since ALMT is a text-centric method, in this embodiment, it can be observed that the text feature is located in the middle of the audio and visual, while all the unimodal features are distributed on the other side of the multimodal features. Figure 8 As can be seen from the figure on the right, the difference is that KuDA's multimodal features are divided into three clusters, close to text, audio, and visual features. This shows that KuDA dynamically selects the dominant modality to make the multimodal features closer to it.

[0144] like Fig. 9 Shown are three cases in CH-SIMS. The ground truth consists of unimodal labels (V, T, and A) and multimodal labels (M). The output shows the multimodal predictions from KuDA. The attention weights show the attention weight distribution of multimodal features with visual, textual, and audio modality features. In the attention weights, the left, middle, and right images of each case represent the multimodal and visual, textual, and audio modality weights, respectively. The partial attention of the dominant modality is marked with a dashed box.

[0145] In order to better demonstrate that the method of this embodiment can dynamically adjust the contribution of different modes, this embodiment selects three challenging cases for further analysis, such as Fig. 9As shown. In this embodiment, it can be observed that in case (a), although the text and audio express strong negative emotions, KuDA can still output correct predictions. This case shows that by adjusting vision as the dominant modality, KuDA effectively captures the speaker's expression and action information, thereby guiding the fusion of text and audio modalities. Similar label distributions also appear in cases (b) and (c). KuDA still makes correct predictions, which shows that it captures the text semantic information in case (b) and the audio intonation information in case (c) by adjusting the attention weights. At the same time, from Fig. 9 It can be seen from the attention weights that the dominant modality (with denser dark blocks) has a higher attention weight than other modalities. This once again proves the importance of dynamic attention fusion for the MSA task.

[0146] The embodiments described above are only descriptions of the preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary technicians in this field should all fall within the protection scope determined by the claims of the present invention.

Claims

1. A multimodal sentiment analysis method based on knowledge guidance and modal dynamic attention fusion, characterized in that: include: Extracting unimodal features from input multimodal data; Based on the unimodal features, obtaining a knowledge-enhanced representation of each modality; Predicting a unimodal sentiment value based on the knowledge enhanced representation, and converting the unimodal sentiment value into a sentiment ratio; Performing a preset dynamic attention process on the knowledge enhancement representation and the emotion ratio to obtain a final multimodal representation; Based on the final multimodal representation, a sentiment score is predicted.

2. The multimodal sentiment analysis method based on knowledge guidance and modal dynamic attention fusion according to claim 1 is characterized in that: The unimodal features include: semantic features of text modality, semantic features of visual modality and semantic features of audio modality.

3. The multimodal sentiment analysis method based on knowledge guidance and modal dynamic attention fusion according to claim 1 is characterized in that: Based on the single modal features, obtaining the knowledge enhanced representation of each modality includes: Encoding the unimodal features into global semantic representation and knowledge sentiment representation; The global semantic representation and the knowledge sentiment representation are combined to obtain a knowledge enhanced representation of each modality.

4. The multimodal sentiment analysis method based on knowledge guidance and modal dynamic attention fusion according to claim 3 is characterized in that: The global semantic representation includes: a global semantic representation of a textual modality and a global semantic representation of a visual and audio modality; The global semantics of the text modality is expressed as: H t , O t =BERT(I t θ BERT ) (1) Among them, H t is the global semantic representation of the text modality, I t is the original text sentence input, θ BERT are the parameters of the BERT model, BERT(.) is the initial text features extracted using the BERT model, t represents the hidden states except the last few layers; The global semantics of the visual and audio modalities are expressed as: Among them, H m Represents the global semantic representation of visual and audio modalities, O m Represents the intermediate hidden states of the visual and audio modalities, Encoder m () indicates the use of TransformerEncoder to extract the initial features of visual and audio modalities, I m represents the raw visual and audio features of the input, Parameters of TransformerEncoder representing visual and audio modalities; The knowledge sentiment is expressed as: in, represents the pre-trained parameters of the adapter for m-mode, K m Represents knowledge emotion expression, Indicates that the obtained data dimension belongs to T m ×d m , T m represents the length of knowledge sentiment representation, d m Represents the dimension of knowledge sentiment representation.

5. The multimodal sentiment analysis method based on knowledge guidance and modal dynamic attention fusion according to claim 1 is characterized in that: The knowledge enhancement representation and unimodal sentiment value of each modality are respectively: Among them, U m Represents knowledge enhancement representation, Represents a unimodal sentiment value, Decoder m () indicates using the Decoder module to predict the unimodal sentiment value. Represents the parameters of each single-mode decoder, and [·;·] indicates a connection.

6. The multimodal sentiment analysis method based on knowledge guidance and modal dynamic attention fusion according to claim 1 is characterized in that: The method of converting the unimodal sentiment value into a sentiment ratio is as follows: Among them, R m is the emotional ratio, D m It represents the intermediate variable of the unimodal sentiment value conversion, exp() represents the nth power of e, n is the calculated value in brackets, k represents the slope, represents the unimodal sentiment value, y represents the label value, and D t The intermediate variable representing the converted text modality sentiment value, D v Represents the intermediate variable for converting the visual modality sentiment value, D a Represents the intermediate variable for converting the audio modality emotion value.

7. The multimodal sentiment analysis method based on knowledge guidance and modal dynamic attention fusion according to claim 1 is characterized in that: The knowledge enhancement representation and the emotion ratio are subjected to a preset dynamic attention process to obtain a final multimodal representation, including: Mapping the knowledge enhancement representation to unify the length and dimension axes of the knowledge enhancement; Preset several stacked dynamic attention modules; The mapped knowledge-enhanced representation is used as the input of the first dynamic attention module; The output of the current dynamic attention module, the mapped knowledge enhancement representation and the sentiment ratio are used as the input of the next dynamic attention module until the output of the last dynamic attention module is obtained as the final multimodal representation.

8. The multimodal sentiment analysis method based on knowledge guidance and modal dynamic attention fusion according to claim 7 is characterized in that: The final multimodal representation is: Among them, F n is the final multimodal representation, F n-1 is the output of the current dynamic attention module, DAB is the dynamic attention block, is the knowledge enhancement representation after mapping, R m is the emotional ratio, is the dimension of multimodal fusion features, T f is the length of the fusion feature, d f is the dimension of fusion features.

9. The multimodal sentiment analysis method based on knowledge guidance and modal dynamic attention fusion according to claim 1 is characterized in that: Before predicting the sentiment score, the method further includes: using a preset joint loss to train the final multimodal representation; The preset joint loss is: in, is the multimodal sentiment prediction value, MLP() is a multi-layer perceptron, Mean() is the average value of the data in the length dimension, F L For the final multimodal, is the regression loss function, N is the total number of samples, y i is the label of sample i, is the multimodal sentiment prediction value of sample i, is the regression loss function, is the loss function for correlation evaluation, α is the hyperparameter for balancing loss contribution, is the preset joint loss.

Images