Multi-modal sentiment analysis method based on feature decoupling and graph attention network

Abstract

The invention provides a multi-modal sentiment analysis method based on feature decoupling and a graph attention network in order to solve the problem that common features and unique feature relations of different modals are neglected when existing multi-modal sentiment analysis processes multi-modal data. Information of three modes of text, vision and voice is input through a network model, and firstly, different models after pre-training are used as feature extractors; then, through comparative learning, the features of each mode are decomposed into unique features of the modes and common features between the modes, effective decoupling of feature vectors is achieved, and therefore effective learning of a fusion model on emotion information is promoted; thirdly, constructing a static graph by utilizing the decoupled features, and learning multi-modal emotion features by adopting a graph attention network; and finally, the fusion features are classified through a linear layer to predict emotion tags.

Description

Technical Field

[0001] The present invention belongs to the field of multimodal sentiment analysis, and in particular relates to a multimodal sentiment analysis method based on feature decoupling and graph attention network. Background Art

[0002] With the surge in the number of Internet videos, analyzing the emotional information in them has become a hot topic in current research and plays an important role in many real-world applications, such as recommendation systems, social media analysis, and financial analysis. As a typical form of multimodal information expression, video not only includes text information used in traditional sentiment analysis, but also contains visual and speech information. The main difficulties are: 1) Data fusion: Data of different modalities have different characteristics and expressions, and how to effectively fuse these data is a challenge. For example, text data can be represented by vocabulary and grammatical features, while image data is represented by pixels or feature points. 2) Context understanding: Emotions are often closely related to specific contexts. For example, the same words may express different emotions in different situations. The model needs to be able to understand this contextual information in order to accurately perform sentiment analysis. 3) Complementarity and redundancy: Data of different modalities may provide complementary information or may be redundant. Effectively identifying and utilizing this complementary information while removing redundancy is a key task in multimodal sentiment analysis. Therefore, sentiment analysis of multimodal data has always been a very challenging task.

[0003] Previous research on multimodal sentiment analysis mainly focused on two aspects: modal representation learning and multimodal feature fusion.

[0004] In traditional representation learning methods, different modal features are extracted in various ways. For example, convolutional networks, long short-term memory networks, and attention mechanisms are used to extract multimodal feature representations; common features and unique features are projected into different subspaces to learn modal common and modal unique feature representations. In recent years, pre-trained encoders have been used to extract features. For example, models such as AFR-BERT and AOBERT use the pre-trained language model BERT to extract text features. This type of method can enable the model to learn richer feature representations and has been recognized by researchers.

[0005] Modal feature fusion usually follows three fusion paradigms: feature-level fusion, decision-level fusion, and hybrid fusion. Feature-level fusion methods usually use simple methods such as CONCAT splicing and addition to perform cross-modal fusion. For example, fusion features are generated by splicing features, but this method fails to capture the hierarchical correlation information between modalities. Decision-level fusion methods fuse the results based on independent sentiment analysis of multiple models. For example, independent deep neural network models are trained for each modality, and a voting mechanism is used to fuse the output results of each model, but this method is time-consuming and fails to fully utilize the interactive information between different modalities. Hybrid fusion methods first learn intra-modal representations and then perform inter-modal fusion. For example, tensor representations are constructed through the outer product operation of a single modal representation for fusion, and Cartesian products are used to explicitly simulate unimodal, bimodal, and trimodal interactions; multi-head attention is used for fusion, while considering the interactions within and between modalities; language-guided multi-level correlation fusion utilizes modality-related information from low-level to high-level. Although these methods have made significant progress in the field of multimodal feature fusion, they still have shortcomings in effectively utilizing the common and unique features between different modalities, which limits the performance of the fusion model. Summary of the invention

[0006] In order to address the problem that existing methods do not consider the unique features contained in different modalities and the relationship between common features, the present invention provides a multimodal sentiment analysis method based on feature decoupling and graph attention network.

[0007] The present invention adopts the following technical solution: a multimodal sentiment analysis method based on feature decoupling and graph attention network, the steps of which are as follows:

[0008] Step 1: Input a sequence of three modalities including text, image, and sound.

[0009] Step 2: Use the pre-trained multi-layer Transformer model as a feature extractor to extract modal features from the image modality and sound modality data. The image modality features and sound modality features are denoted as V and A, respectively.

[0010] Step 3: Use the pre-trained language model Bert to extract text modal features and obtain text modal features T.

[0011] Step 4: Use different MLP modules for decoupling, and decouple the extracted modal features into modality-related private features and modality-independent common features.

[0012] Step 5: Calculate the mean of the decoupled features as the sequence features of the mode.

[0013] Step 6: Use contrastive learning loss function to constrain the modal sequence features so that the MLP module can reasonably decouple the text modal features T, image modal features V, and sound modal features A into unique features and private features.

[0014] Step 7: Construct a static graph of the decoupled features to obtain graph data G that can be input into the fusion module.

[0015] Step 8: Use the graph attention neural network to fuse the graph data G and obtain the fusion feature f m .

[0016] Step 9: Fusion feature f m Multimodal tasks are classified through the MLP module, and the decoupled features are classified through the MLP module for single-modal tasks.

[0017] This method requires the construction of a feature extraction module, a feature decoupling module and a fusion module to form a multimodal sentiment analysis network. The feature extraction module is used to extract multimodal raw data into three specific low-dimensional feature representations. Among them, the pre-trained Bert language model is used to encode the text modality, and two independent Transformer models are used to encode the visual modality and speech modality respectively. The feature decoupling module uses contrastive learning to decouple the low-dimensional features of each modality into unique features related to the modality and common features unrelated to the modality. The fusion module is responsible for fusing the decoupled features. First, the decoupled features are converted into a static graph G. Then, the graph attention neural network is used to fuse G, and then a layer of average pooling layer is passed to obtain the fused features.

[0018] In the feature extraction module of the multimodal sentiment analysis method based on feature decoupling and graph attention network, different methods are used to extract features for data of different modalities. Specifically, a pre-trained Bert language model is used to extract features of the text modality, and the pre-training weights are base-uncase; for the visual modality and speech modality, two-layer Transformer models are constructed respectively for pre-training on the data set, and the training weights after training stabilization are used as the initialization parameters for feature extraction.

[0019] The above-mentioned multimodal sentiment analysis method based on feature decoupling and graph attention network uses intra-sample and inter-sample sample pairs in the feature decoupling module, and selects sample pairs according to the similarity of samples in the same batch, so that the model can learn the common features between different modalities and the unique features related to the modality, so as to achieve the purpose of decoupling features. Specifically, the MLP module is used to perform decoupling respectively, so that the features obtained by the feature extraction module are decomposed into common features and unique features, and the contrast learning loss function is used to constrain so that the features after projection can be correctly separated.

[0020] The above-mentioned multimodal sentiment analysis method based on feature decoupling and graph attention network adopts the graph attention method to fuse the decoupled features in the fusion module. Specifically, the present invention considers the relationship between the decoupled features and constructs them into a static graph, and then fuses them through the graph attention neural network. The construction of the static graph considers the following connections: (1) common features of different modalities; (2) decoupled features of the same modality; (3) adjacent common features.

[0021] In order to solve the problem of ignoring the connection between common features and unique features of different modalities when processing multimodal data, this paper proposes a novel multimodal sentiment analysis method based on feature decoupling and graph attention network. Through contrastive learning, the model can decompose the features of each modality into modality-unique features and common features between modalities, realize the effective decoupling of feature vectors, and thus promote the effective learning of sentiment information by the fusion model. Then, the decoupled features are used to construct a static graph, and the graph attention network is used to learn sentiment features to predict sentiment labels. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 This is the overall network model structure diagram of the present invention.

[0023] Figure 2 This is a structural diagram of the fusion graph data construction method 1 of the present invention.

[0024] Figure 3 This is a structural diagram of the fusion graph data construction method 2 of the present invention. DETAILED DESCRIPTION

[0025] In order to make the technical solution and purpose of the present invention easier to understand, the present invention will be described in detail below with reference to the accompanying drawings and embodiments.

[0026] The embodiment of the present invention provides a multimodal sentiment analysis method based on feature decoupling and graph attention network, referring to Figure 1 , the method is specifically implemented by the following steps:

[0027] Step 1: Input the text in a video, multiple frames of pictures, and the features of a speech after preprocessing. The openface model is used to extract the facial features of the picture data, and the librosa library is used to extract the audio features of the speech data.

[0028] Step 2: Use different feature extraction modules to extract modal features from data of different modalities.

[0029] Step 2.1: For text data, use the pre-trained text feature extractor Bert to extract text modal features.

[0030] Step 2.2: For facial features, a two-layer Transformer model is used for pre-training until it is stable. The stabilized Transformer model is used as a feature extractor to extract image modality features.

[0031] Step 2.3: For audio features, similar to facial features, a two-layer Transformer model is used for pre-training until it is stable. The stabilized Transformer model is used as a feature extractor to extract sound modality features.

[0032] Step 3: Decompose modal features into unique features and common features. The MLP module is used to decouple the features of different modalities into two independent subspaces: one subspace focuses on extracting modality-independent features to obtain cross-modal shared sentiment features; the other subspace focuses on extracting modality-related features to obtain unique sentiment features in each modality.

[0033] Step 4: Construct a contrastive learning loss function. The key to the contrastive learning loss function is to construct positive and negative sample pairs, and use two loss functions, inter-sample contrast loss and intra-sample contrast loss, to constrain features. For each sample i∈B, where B represents a batch of samples, the contrastive learning loss of sample i is Among them, z + represents a positive sample pair, z - Represents a negative sample pair. According to the source of the positive and negative sample pairs, they can be divided into in-sample sample pairs and sample pairs between samples Therefore, the positive sample pair And negative sample pairs

[0034] Step 4.1: Sample pair construction within a sample. The present invention uses the six features obtained by decoupling to form sample pairs, which are respectively recorded as: and in, and Represent the unique features and common features of the text modality of sample i respectively. Similarly, and Represent the unique features and common features of the visual and speech modalities of sample i respectively. For sample i, its common features are positive sample pairs within the sample. Common features and unique features constitute the negative sample pairs in the sample

[0035] Step 4.2: Sample pair construction. This process is based on the true labels and the pairwise similarities between samples. First, calculate the similarity s between any two samples. i,j, where i,j∈B. The calculation process is as follows: i,j =sim(u i ,u j )in, sim(a,b)=a T b / ||a||·||b|| represents the cosine similarity between vectors a and b, where ||·|| represents the modulus length and T represents the transposition operation. Next, select the k samples with the highest similarity to the true label and record them as the positive sample pair set S. For sample i∈B, its common features and the common features of the corresponding modality in S constitute the positive sample pair between samples. At the same time, select the k samples with the highest similarity to the true label and record them as the negative sample pair set For sample i∈B, its common features are The unique features of the corresponding modality in constitute the negative sample pairs between samples

[0036] Step 5: Graph data construction. The decoupled multimodal features are constructed into graph data, and then fused using a graph attention neural network. This paper designs two different graph data construction methods, refer to Figure 2 , Figure 3 , which are denoted as six-relation graph G6 and three-relation graph G3. Given a sample with m features of different modalities As the nodes of the graph, t∈[1,m]. The edges of the graph data can be composed of the following categories: (1) Common features of different modalities Same mode decoupling feature Adjacent shared features The six-relation graph G6 contains the three types of edges mentioned above, while the three-relation graph G3 only contains the parts (1) and (3) composed of common features.

[0037] Step 6: Graph data fusion. Using graph attention neural network to aggregate neighborhood information in the graph can achieve efficient communication and higher quality information aggregation. Based on this, graph attention neural network is used to aggregate neighborhood information in the graph. Where σ(·) represents the LeakyReLU activation function, represents the adjacent nodes of node v, h u and h v Represent the features of node u and node v respectively. The attention scores α of these two nodes u and v uv =Softmax(σ(a T [Wh v ,Wh u ])), where a represents the weighted attention vector and W represents the shared weight matrix.

[0038] Step 7: Multimodal fusion. The feature mean of all graph nodes is used as the feature f of multimodal fusion. m =Mean(h).

[0039] Step 8: Calculate the loss. The loss consists of three parts: contrastive learning loss l c , unimodal prediction loss l u , multimodal prediction loss l m .

[0040] Step 8.1: Calculate the contrastive learning loss. Take the mean of the sample contrastive loss Where n represents the size of the batch sample B.

[0041] Step 8.2: Calculate the multimodal prediction loss l m A multi-layer perceptron (MLP) with ReLU activation function is used as the classifier to obtain the final prediction results. The mean square error (MSE) is used to calculate the multimodal prediction loss. in, represents the multimodal prediction result of sample i, Represents the multimodal true result of sample i.

[0042] Step 8.3: Calculate the unimodal prediction loss l u . Take the mode x∈{T,A,V} as an example. The unique features of sample i and common features Should jointly participate in predicting the true value of x mode Therefore, the prediction loss for modality x is in, [,] indicates a merge operation. represents the prediction result of the x-modality of sample i. Therefore, the unimodal prediction loss in, represents the text modality prediction loss, represents the speech modality prediction loss, represents the image modality prediction loss.

[0043] Step 8.4: Calculate the total loss. Total loss loss = β1l c +β2l u +l m , where l c represents the contrastive learning loss, l u represents the unimodal prediction loss, l m represents the multimodal prediction loss, β1 and β2 represent the contribution of contrastive learning loss and unimodal prediction loss to the total loss, respectively.

[0044] Step 9: Training parameter setting. The experiments of the present invention are all performed in a hardware environment equipped with an NVIDIA RTX 3090 GPU. The hyperparameters used in the training model are as follows: the optimization algorithm is AdamW; k is 4; the learning rate is 0.0005; the feature vector dimension of the input in the graph neural network, the number of attention mechanism heads, and the number of network layers are 128, 4, and 3 respectively; β1 and β2 are 0.1 and 0.01 respectively.

Claims

1. A multimodal sentiment analysis method based on feature decoupling and graph attention network, characterized by: The following steps are involved: Step 1: Input a sequence of three modalities: text, image, and sound; Step 2: Use the pre-trained multi-layer Transformer model as a feature extractor to extract features from the image modality and sound modality data. The image modality features and sound modality features are denoted as V and A respectively. Step 3: Use the pre-trained language model Bert to extract text features and obtain text modality features T; Step 4: Use different MLP modules to decouple the extracted modal features into modality-related private features and modality-independent common features; Step 5: Calculate the mean of the decoupled features as the sequence features of the mode; Step 6: Use contrastive learning loss function to constrain modal sequence features so that the MLP module can reasonably decompose text modal features T, image modal features V, and sound modal features A into unique features and private features; Step 7: Construct a static graph of the decoupled features to obtain graph data G that can be input into the fusion module; Step 8: Use the graph attention neural network to fuse the graph data G to obtain the fusion feature f m ; Step 9: Fusion of features f m Multimodal tasks are classified through the MLP module, and the decoupled features are classified through the MLP module for single-modal tasks.

2. According to claim 1, a multimodal sentiment analysis method based on feature decoupling and graph attention network is characterized in that: The method includes three modules: feature extraction module, feature decoupling module and fusion module; the feature extraction module is used to extract multimodal raw data into three specific low-dimensional feature representations, among which the pre-trained Bert language model is used to extract features of the text modality, and two independent Transformer encoders are used to extract features of the visual modality and speech modality respectively; the feature decoupling module uses contrastive learning to decouple the low-dimensional features of each modality into modality-related private features and modality-independent common features; the fusion module is responsible for fusing the decoupled features. First, the decoupled features are converted into a static graph G, and then the static graph G is fused using a graph attention neural network, and then a layer of average pooling layer is passed to obtain the fused features.

3. The multimodal sentiment analysis method based on feature decoupling and graph attention network according to claim 2, characterized in that: In the feature extraction module, different methods are used to extract features for data of different modalities. Specifically, the pre-trained Bert language model is used to extract features of the text modality, and the pre-trained weights use base-uncase. For the image modality and sound modality, two-layer Transformer models are constructed and pre-trained on the dataset respectively, and the training weights after training stabilization are used as the initialization parameters for feature extraction.

4. A multimodal sentiment analysis method based on feature decoupling and graph attention network according to claim 2 or 3, characterized in that: In the feature decoupling module, intra-sample sample pairs and inter-sample sample pairs are used, and sample pairs are selected according to the similarity of samples in the same batch, so that the MLP module can learn the common features between different modalities and the unique features related to the modality, so as to achieve the purpose of decoupling features; specifically, the MLP module is used to perform decoupling respectively, so that the modal features obtained by the feature extraction module are decomposed into common features and unique features, and the contrast learning loss function is used to constrain so that the decoupled features can be correctly separated.

5. A multimodal sentiment analysis method based on feature decoupling and graph attention network according to claim 2 or 3, characterized in that: The fusion module uses a graph attention approach to fuse decoupled features. Specifically, a static graph is constructed and then fused through a graph attention neural network. The construction of the static graph takes into account the following connections: (1) common features of different modalities; (2) decoupled features of the same modality; and (3) adjacent common features.

Images