A multi-modal sentiment analysis method and system thereof

Abstract

The application belongs to the field of multi-modal sentiment analysis, and discloses a multi-modal sentiment analysis method and system, which comprises the following steps: S1, encoding voice, text and image modal sequences, then obtaining feature representations of all modes through 1D convolution, and taking the feature representations as input features of two-stage feature fusion; S2, designing a stacked Transformer; S3, the first stage utilizes the stacked Transformer to interact and complement the text mode with the visual and acoustic two non-text modes, and outputs the adapted text and non-text modes; S4, the second stage jointly represents the enhanced text mode information output by the first stage and the original input, and then extracts the potential adaptability between the fused representations; finally, the corresponding information obtained in the two stages is fused to predict the sentiment state. The application significantly improves the accuracy of multi-modal sentiment analysis classification.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of multi-modal sentiment analysis, and particularly relates to a multi-modal sentiment analysis method and system. BACKGROUND

[0002] In recent years, with the continuous development of social networks, we have witnessed an explosive growth of multi-modal data. More and more users tend to express their attitudes and emotions using various media forms (such as text + image, text + song, text + video, etc.). Mining and understanding these emotional elements from multi-modal data, i.e. multi-modal sentiment analysis (MSA), has become a hot research topic. Traditional unimodal sentiment analysis, such as text sentiment analysis, relies on words, phrases and their semantic relationships, which is not enough to identify complex emotional information. After adding facial expressions and intonation to text, multi-modal information (image, speech and transcribed text) provides a more vivid description and conveys more accurate and rich emotional information. Multi-modal fusion is the core problem in this research field, and the purpose is to integrate the information extracted from different unimodal modalities into multi-modal features for sentiment prediction.

[0003] Most of the previous work in this field focuses on early, late or hybrid fusion. Early fusion integrates immediately after feature extraction (usually by simply concatenating their representations). On the other hand, late fusion integrates after each modality makes a decision (e.g. classification or regression). Finally, hybrid fusion combines the advantages of the two methods mentioned above.

[0004] A large number of works confirm that modeling the interaction between different modalities has a positive effect on the final fusion, regardless of the way of fusion. Some works on this topic focus on the interaction between unimodal modalities, but these works ignore the potential adaptability between the fused representations, which may lead to the risk of losing actual information. SUMMARY

[0005] The present application aims to provide a multi-modal sentiment analysis method and system to solve the above technical problems.

[0006] To solve the above technical problems, the specific technical solutions of the present application are as follows:

[0007] A multi-modal sentiment analysis method based on two-stage stacked Transformer, comprising the following steps:

[0008] S1: encode the speech, text and image modality sequences, then obtain the feature representations of all modalities through 1-dimensional convolution, and take them as the input features of two-stage feature fusion;

[0009] S2: design a stacked Transformer, which is composed of a bidirectional cross-modal Transformer and a Transformer encoder, which is the core component of the two-stage inter-modal interaction, which accepts two inputs;

[0010] S3: the first stage uses stacked Transformer to interact with text modal and two non-text modal of visual and acoustic, and outputs the adapted text and non-text modal;

[0011] S4: the second stage combines the enhanced text modal information output by the first stage with the original input, and then extracts the potential adaptability between the fusion representation; Finally, the corresponding information obtained in the two stages is fused to predict the emotional state.

[0012] 2. The multi-modal sentiment analysis method of claim 1, wherein the specific steps of step S1 comprise:

[0013] S101: use COVAREP, FACE and BERT to extract initial vector features from audio and visual original data respectively:

[0014]

[0015] wherein h t The initial vector features, BERT is a pre-trained language model, X t is the text original data;

[0016] S102: then pass all input sequences through 1D time convolution to ensure that each element of the input sequence has sufficient perception of its neighborhood elements:

[0017]

[0018] wherein k{t, a, v} represents the size of the convolution kernel;

[0019] 3. The multi-modal sentiment analysis method of claim 1, wherein the specific steps of step S2 comprise:

[0020] First, use the bidirectional cross-modal Transformer to interact with the two input features, define the query vector, key vector and value vector of multi-head cross-modal attention as and wherein m∈{t, a}, the text-to-speech information is represented as:

[0021]

[0022]

[0023] wherein S t→a denotes the score matrix, CA i represents the i-th head in cross-modal attention, and the calculation of n heads in cross-modal attention is as follows:

[0024] MH-CA(S t→a , V t )=W′[CA1,..., CA n ] T

[0025] wherein is the weight matrix in multi-head attention; the weight accumulation mechanism is used to improve the adaptability of the text modality to the non-text modality in cross-modal attention:

[0026]

[0027] Finally, after cross-modal Transformer interaction, the output features are optimized by Transformer.

[0028] 4. The multi-modal sentiment analysis method of claim 1, wherein the specific steps of step S3 comprise:

[0029] Taking stacked Transformer as the core and the text modality as the center, the text modality is respectively interacted with the speech and image modalities, and the result after mutual complementation between single-peak modalities is output, the three features related to the text modality are spliced and taken as the center of the second stage:

[0030]

[0031] 5. The multi-modal sentiment analysis method of claim 1, wherein the specific steps of step S4 comprise:

[0032] The input of the stacked Transformer comes from the and wherein After self-attention refinement, the text information fused speech and image information are respectively interacted again with the text information fused speech and image information as the center.

[0033] 6. A multi-modal sentiment analysis system based on a two-stage stacked Transformer, comprising a multi-modal feature extraction module, a two-stage feature fusion module, and a sentiment prediction module.

[0034] The multi-modal feature extraction module is configured to extract initial features of speech, text, and image, and encode the initial features as input features of the two stages.

[0035] The double-stage feature fusion module is used for multi-stage fusion of the input multi-modal features, and the double stages capture the complementarity between the interaction between unimodal modalities and the potential fusion representation.

[0036] The emotion prediction module performs final emotion prediction on the output features after double-stage fusion.

[0037] The beneficial effects of the present application are:

[0038] (1) The present application designs a flexible multi-stage structure for a multi-modal emotion analysis task, which not only captures the interaction between unimodal modalities, but also focuses on the complementarity between potential fusion representations.

[0039] (2) In addition, in order to further improve the ability of the text modality to extract information from the text modality, a weight accumulation mechanism is proposed, which simply and effectively improves the performance of the stacked Transformer.

[0040] (3) The present application is very helpful for the multi-modal emotion analysis task, and can significantly improve the emotion prediction accuracy of multi-modal emotion analysis. DETAILED DESCRIPTION

[0041] Figure 1 is a flowchart of the multi-modal emotion analysis method based on double-stage stacked Transformer of the present application;

[0042] Figure 2 is a model schematic diagram of the multi-modal emotion analysis method and system based on double-stage stacked Transformer;

[0043] Figure 3 is a schematic diagram of the overall structure of the designed stacked Transformer;

[0044] Figure 4 is a structure block diagram of the multi-modal emotion analysis system based on double-stage stacked Transformer. DETAILED DESCRIPTION

[0045] The preferred embodiments of the present application will be described in detail below with reference to the accompanying drawings, so that the advantages and features of the present application can be more easily understood by those skilled in the art, and the protection scope of the present application can be more clearly and definitely defined.

[0046] Please refer to Figure 1 , Figure 2 and Figure 3 , the embodiments of the present application include:

[0047] A multi-modal emotion analysis method based on double-stage stacked Transformer, referred to as TSST, includes the following steps:

[0048] S1: encode the sequence of speech, text and image modalities, and then get the feature representation of all modalities through 1D convolution, and take it as the input feature of the two-stage feature fusion;

[0049] The initial multi-modal feature extraction includes the following steps:

[0050] S101: use COVAREP, FACE and BERT to extract initial vector features from audio and visual raw data respectively.

[0051]

[0052] wherein h t The initial vector feature, BERT is a pre-trained language model, X t is the text raw data.

[0053] S102: then pass all input sequences through 1D time convolution to ensure that each element of the input sequence has enough perception of its neighborhood elements

[0054]

[0055] wherein k{t, a, v} represents the size of the convolution kernel;

[0056] S2: a stacked Transformer is designed, which consists of a bidirectional cross-modal Transformer and a Transformer encoder, which is the core component of inter-modal interaction in the two-stage, accepts two inputs, first cross-modal interaction, then refine the output.

[0057] S201: first, use the bidirectional cross-modal Transformer to interact with the two input features, we define the query vector, key vector and value vector of multi-head cross-modal attention as and wherein m∈{t, a}, taking the text-to-speech information transfer as an example:

[0058]

[0059]

[0060] wherein S t→a represents the score matrix, CA i represents the i-th head in the cross-modal attention. At the same time, the calculation method of the n heads in the cross-modal attention is:

[0061] MH-CA(S t→a , V t)=W′[CA1,...,CA n ] T

[0062] wherein is the weight matrix in multi-head attention. In addition, we also use the weight accumulation mechanism to further improve the adaptability of the text modality to the non-text modality in cross-modal attention:

[0063]

[0064] Finally, after the cross-modal Transformer interaction, the output features will be optimized by the Transformer again.

[0065] S3: First stage: taking stacked Transformer as the core and taking the text modality as the center, respectively interacting with the speech and image modalities, outputting the results after the complementarity between the single modalities, Then we splice the three features related to the text modality, and further take them as the center of the second stage.

[0066]

[0067] S4: The second stage combines the enhanced text modality information output by the first stage with the original input to represent the potential adaptability between the fusion representations, and finally fuses the corresponding information obtained in the two stages to predict the emotional state.

[0068] Most of the steps are similar to S3, except that this stage focuses on the mutual adaptation between the fusion representations. The input of the stacked Transformer comes from the and wherein will be refined by self-attention, and then will be taken as the center to respectively interact with the speech and image information fused with the text information again.

[0069] Finally, the output of the second stage will be subjected to the final emotional prediction after the full connection layer.

[0070] It should be noted that in step S2, the two parts of the stacked Transformer, the cross-modal Transformer and the refinement Transformer, are stackable multiple layers. In addition, the model uses AdamW as the optimizer, and the learning rate is set to 0.00005. Combined with Figure 2 , the entire model is divided into four blocks: a feature extraction module, a single modality interaction module in the first stage, a fusion representation complementary module in the second stage, and a final prediction module.

[0071] In the embodiment of the present application, refer to Figure 4Also provided is a multi-modal sentiment analysis detection system based on a two-stage stacked Transformer, comprising:

[0072] A multi-modal feature extraction module is configured to extract initial features of voice, text and image, and encode the initial features as input features of the two stages.

[0073] A two-stage feature fusion module is configured to perform multi-stage fusion on the input multi-modal features, and the two stages capture the complementarity between the interaction between unimodal modes and the potential fusion representation.

[0074] A sentiment prediction module is configured to perform final sentiment prediction on the output features after two-stage fusion.

[0075] In the present application, experiments are performed on two public datasets MOSI and MOSEI. In order to quantitatively evaluate the detection results of the two-stage stacked Transformer (TSST), the binary classification accuracy (Acc-2) and the mean absolute error (MAE) are used as evaluation indexes.

[0076] Table 1

[0077]

[0078] Table 2

[0079]

[0080] All models in Table 1 use BERT to embed the text, and from the results, it can be found that compared with previous works, our framework has made significant improvements in various indicators. For some early works such as "MulT", which only captures the mutual adaptation between unimodal modes, all indicators of our model have been greatly improved on two datasets. Most importantly, the binary classification accuracy of the MOSEI dataset has been improved by more than 4%. Compared with other models involving Transformers ("MISA"), all indicators have been updated. In addition, for the sake of fairness, we also make a detailed comparison with the state-of-the-art model. We reproduce the best baseline "MMIM" under the same conditions. Our framework achieves 86.65% in Acc-2, which is 1.45% higher than "MMIM". Similar to Acc-2, we obtain an improvement of 1.1% in MAE. On the smaller MOSI dataset, our framework has similar performance to SOTA methods.

[0081] To further explore the contribution of TSST, we conduct an ablation study on the CMU-MOSEI dataset. The results are shown in Table 2. On the one hand, compared with the single stage, the indicators of the two-stage are significantly better: the Acc-2 indicator of the single stage is about 1.4% lower than that of the two-stage, and the MAE also decreases slightly. This proves that the cross-modal interaction modeling across multiple stages is beneficial, and the mutual adaptation of the fusion representation can bring some useful information. We also explore the benefits of the weight accumulation mechanism. The results show that it has a positive impact on the performance of the framework. When we remove these two cases, the overall performance of the model decreases significantly, and the binary classification accuracy decreases by about 2.3%. This further illustrates the effectiveness of the components introduced in this work.

[0082] On the other hand, we also conduct more than two fusion stages. We observe that further increases lead to performance degradation, and we hypothesize that this is due to overfitting of the dataset. Existing multi-modal sentiment analysis datasets are very small due to the laborious labeling process. For example, the development of existing datasets involves data cleaning and annotation by multiple annotators. The limited dataset size increases the risk of overfitting.

[0083] These results demonstrate the effectiveness of the method proposed by the present invention.

[0084] The above only describes the embodiments of the present invention, and does not limit the patent scope of the present invention, any equivalent structure or equivalent process transformation using the content of the specification and drawings, or direct or indirect application in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A multi-modal sentiment analysis method based on a two-stage stacked Transformer, characterized in that, The method comprises the following steps: S1: encode the voice, text and image modal sequence, then obtain the feature representation of all modalities through 1D convolution, and take the feature as the input feature of the two-stage feature fusion; S2: design a stacked Transformer, which is composed of a bidirectional cross-modal Transformer and a Transformer encoder, which is the core component of the inter-modal interaction in the two stages, and accepts two inputs; S3: the first stage uses the stacked Transformer to interact and complement the text modal with the visual and acoustic non-text modal, and outputs the adapted text and non-text modal; With the core of stacked Transformer, it interacts with speech and image modalities respectively, and outputs the complementary results between single modalities, Three features related to text modalities are spliced together as the center of the second stage: S4: the second stage combines the enhanced text modal information output by the first stage with the original input joint representation, then extracts the potential adaptability between the fusion representations, and finally fuses the corresponding information obtained in the two stages to predict the emotional state; The input of the stacked Transformer comes from the first stage and wherein After refinement by self-attention, it is centered and respectively interacts with the information of the speech and image fused with the text information again.

2. The multi-modal sentiment analysis method of claim 1, wherein, The specific steps of step S1 include: S101: use COVAREP, FACE and BERT to extract initial vector features from audio and visual raw data respectively: wherein represents the text h t The initial vector feature, BERT is a pre-training language model, X t is the original data of the text; S102: then pass all input sequences through 1D time convolution to ensure that each element of the input sequence has sufficient perception of its neighborhood elements: where k {t，a，v} denotes the size of the convolution kernel.

3. The multi-modal sentiment analysis method of claim 1, wherein, The specific steps of step S2 include: Firstly, the two input features are interacted by the bidirectional cross-modal Transformer, and the query vector, key vector and value vector of multi-head cross-modal attention are defined as and where m∈{t,a}, and the text-to-speech information transfer is represented as: where S t→a denotes the score matrix, CA i denotes the i-th head in cross-modal attention, and the n heads in cross-modal attention are calculated as MH-CA(S t→a , V t ) = W'[CA1,..., CA n ] T wherein is a weight matrix in multi-head attention; the weight accumulation mechanism is used to improve the adaptation capability of the text modality to the non-text modality in the cross-modal attention mechanism; Finally, after the cross-modal Transformer interaction, the output feature is optimized through the Transformer.

4. A multi-modal sentiment analysis system for performing the steps of the multi-modal sentiment analysis method according to any one of claims 1-3, characterized in that, It comprises a multi-modal feature extraction module, a two-stage feature fusion module and an emotion prediction module; The multi-modal feature extraction module is used to extract the initial features of voice, text and image, and encode them as the input features of the two stages; The two-stage feature fusion module is used to fuse the input multi-modal features in multiple stages, and the two stages capture the interaction between the single peak modal and the potential fusion representation. The emotion prediction module performs the final emotion prediction on the output feature after the two-stage fusion.

Images