A multi-modal sentiment analysis method based on double-flow attention and gate fusion

Abstract

The application discloses a kind of multi-modal sentiment analysis methods based on double-flow attention and gate fusion, belong to natural language processing technical field, high-order semantic features of audio, image and text are extracted using pre-trained encoder in the method;The consistency and fine-grained difference of different modal sentiment contribution are captured by calculating the similar features and difference features between different modalities, realizing the deep interaction between different dimensional features, so as to mine the complementary information between modalities;Then use adaptive gate network to fuse the similar features and difference features of modal pair, while filtering out redundant sentiment and sentiment noise.The method can accurately mine sentiment features, effectively fuse the information of different modalities, solve the problem of insufficient multi-modal information fusion, and to some extent, alleviate the difficulty of multi-modal data independent and difficult to fuse;Help to improve the accuracy of multi-modal sentiment analysis prediction.

Description

Technical Field

[0001] This invention relates to the field of natural language processing technology, and more specifically to a multimodal sentiment analysis method based on the fusion of dual-stream attention and gating. Background Technology

[0002] Analyzing sentiment in multimodal data presents both an opportunity and a challenge for the field of sentiment analysis. Multimodal sentiment analysis involves the analysis and processing of information from multiple modalities. Compared to unimodal data, multimodal data not only provides more sentiment features, but the information from different modalities can also complement each other, helping machines better understand the emotions in text. Currently, the data from each modality are independent and highly diverse, so much work focuses on how to better integrate information from different modalities.

[0003] Multimodal fusion methods mainly fall into two categories: pre-fusion and post-fusion. Most pre-fusion methods simply piece together information from different modalities, ignoring the differences in the importance of each modality and making it difficult to accurately extract sentiment features. In post-fusion, most methods assume that the sentiment polarity of each modality is the same, ignoring the inconsistency in sentiment across modalities. Because the information from each modality exists in different semantic spaces, and these semantic spaces are both similar and different, fusing information between different modalities presents certain challenges, and the accuracy of multimodal sentiment analysis prediction needs improvement. Summary of the Invention

[0004] In view of this, the present invention provides a multimodal sentiment analysis method based on dual-stream attention and gating fusion that at least solves some of the above-mentioned technical problems. It can effectively fuse information from different modalities, solve the problems of difficulty in fusing multimodal information and insufficient fusion, and improve the accuracy of multimodal sentiment analysis prediction.

[0005] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0006] This invention provides a multimodal sentiment analysis method based on the fusion of two-stream attention and gating, comprising the following steps:

[0007] S1. Utilize deep learning models to vectorize multimodal information and extract high-order features from audio, images, and text;

[0008] S2. Calculate similarity attention and difference attention for modality pairs through dual-stream attention, perform interactive calculations in different semantic spaces, and obtain similarity vectors and difference vectors.

[0009] S3. An adaptive gating fusion network is used to fuse the similarity vectors and difference vectors of the modal pairs to obtain a multimodal vector representation;

[0010] S4. Based on a neural network classifier, perform sentiment polarity analysis and prediction on multimodal vector representations.

[0011] Preferably, in step S1, the COVAREP algorithm is used to extract acoustic features from the audio.

[0012] Preferably, in step S1, the FaceNet model is used to extract visual features from the image.

[0013] Preferably, in step S1, a pre-trained BERT model is used to extract high-order features of text semantics.

[0014] Preferably, in step S1, the latent vectors of audio and images are further mapped to query vectors through different linear transformations, and the latent vectors of text are mapped to key vectors and value vectors; the mapping method is as follows:

[0015]

[0016]

[0017] K t =H t ·W k +b k

[0018] V t =H t ·W v +b v

[0019] Among them, H a H v H t These represent the latent vectors for audio, images, and text, respectively. W k W v These are the learnable weight parameters in the fully connected layer. b k Q is the bias parameter in the fully connected layer. a Q v These are the query vectors mapped from the audio vector and the image vector, respectively, and K is the query vector. t V is the key vector mapped from the text vector. t It is the value vector mapped from the text vector.

[0020] Preferably, in step S2,

[0021] The formula for calculating similar vectors is:

[0022] α=Q·K T

[0023]

[0024] The formula for calculating the difference vector is:

[0025]

[0026] β=||QK||

[0027]

[0028] Where α represents the similarity representation matrix, β represents the difference representation matrix, Q represents the query vector, K represents the key vector, V represents the value vector, Affi represents the similarity vector, Diff represents the difference vector, softmax is the activation function, i represents the number of rows in the ||QK|| matrix, j represents the number of columns in the ||QK|| matrix, and d k This represents the dimension of the key vector.

[0029] Preferably, in step S3, a gated filtering matrix G∈R is trained using similarity vectors and difference vectors. L×L L represents the length of the output sequence; the gated filtering matrix learns its aggregation features by weighting similarity vectors and difference vectors, generates a fusion vector of modal pairs, selectively extracts interactive semantic information, and filters out useless emotional noise to obtain a multimodal vector representation.

[0030] Preferably, in step S4, the sentiment polarity analysis and prediction based on the multimodal vector representation using a neural network classifier specifically includes:

[0031] The two modalities are spliced ​​together to extract the sentiment semantic information of each modality. Then, the ReLU activation function and dropout method are used to alleviate the overfitting problem during training. Finally, a binary classifier is used to divide the sentiment into positive and negative categories.

[0032] Compared with the prior art, the present invention has at least the following beneficial effects:

[0033] This invention provides a multimodal sentiment analysis method based on dual-stream attention and gating fusion. It uses a deep learning model to extract high-order semantic features from audio, images, and text. By calculating similarity and difference features between different modalities, it captures the consistency and fine-grained differences in sentiment contributions across modalities, achieving deep interaction between features of different dimensions and thus mining complementary information between modalities. Then, an adaptive gating fusion network is used to fuse the similarity and difference features of modal pairs, while filtering out redundant sentiment and sentiment noise. This method not only solves the problem of insufficient multimodal information fusion but also alleviates, to some extent, the difficulty of fusing independent multimodal data, thereby improving the accuracy of multimodal sentiment analysis prediction.

[0034] Other features and advantages of the invention will be set forth in the following description, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures particularly pointed out in the written description and the accompanying drawings.

[0035] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0036] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0037] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used together with the embodiments of the invention to explain the invention and do not constitute a limitation thereof.

[0038] Figure 1 This is a flowchart illustrating a multimodal sentiment analysis method based on the fusion of dual-stream attention and gating, provided as an embodiment of the present invention.

[0039] Figure 2 This is a schematic diagram illustrating the principle framework of a multimodal sentiment analysis method based on the fusion of dual-stream attention and gating, as provided in an embodiment of the present invention.

[0040] Figure 3 This is a schematic diagram illustrating the principle of two-stream attention calculation provided in an embodiment of the present invention.

[0041] Figure 4 This is a schematic diagram of an adaptive gating fusion network provided in an embodiment of the present invention. Detailed Implementation

[0042] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are some embodiments of the present invention, but not all embodiments.

[0043] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0044] Reference Figure 1 and Figure 2 As shown, this invention provides a multimodal sentiment analysis method based on the fusion of dual-stream attention and gating, which includes the following steps:

[0045] S1. Utilize deep learning models to vectorize multimodal information and extract high-order features from audio, images, and text;

[0046] S2. Calculate similarity attention and difference attention for modality pairs through dual-stream attention, perform interactive calculations in different semantic spaces, and obtain similarity vectors and difference vectors.

[0047] S3. An adaptive gating fusion network is used to fuse the similarity vectors and difference vectors of the modal pairs to obtain a multimodal vector representation;

[0048] S4. Based on a neural network classifier, perform sentiment polarity analysis and prediction on multimodal vector representations.

[0049] The specific implementation methods of the present invention will be described in detail below:

[0050] ① Feature extraction of multimodal information:

[0051] Vivid multimodal data is converted into high-dimensional vector representations. Based on different information formats, appropriate encoders are selected to extract high-order semantic features from audio, images, and text. Preferably, in this embodiment, the COVAREP algorithm is used to extract acoustic features from audio, and the FaceNet model is used to extract visual features from images. For the text modality, a pre-trained BERT model is used to extract high-level semantic features. The extraction method is as follows:

[0052] X a =COVAREP(Acoustic)

[0053] X v =FaceNet (Visual)

[0054] X t =BERT(text)

[0055] Where Acoustic represents a piece of audio, Visual represents a picture, Text represents a piece of text, and X represents a piece of audio. a X v X t These represent the audio latent vector, image latent vector, and text latent vector obtained from audio, image, and text conversions, respectively.

[0056] It should be noted that: COVAREP is an open-source software package specifically designed for extracting features from speech signals, especially for emotional speech analysis; FaceNet is a deep learning-based face recognition system that can learn the feature representation of face images and map them to a high-dimensional vector space; BERT is a pre-trained language model based on Transformer that can learn itself from large-scale text data, thereby effectively acquiring contextual information of the text and obtaining better text vector representations through fine-tuning.

[0057] Furthermore, the mapping between the query vector, key vector, and value vector is performed:

[0058] The latent vectors of images and audio are mapped to query vectors Q through different linear transformations, and the latent vectors of text are mapped to key vectors K and value vectors V; the specific mapping methods are as follows:

[0059]

[0060]

[0061] K t =H t ·W k +b k

[0062] V t =H t ·W v +b v

[0063] Among them, H a H v H t These represent the latent vectors for audio, images, and text, respectively. W k W v These are all learnable weight parameters in fully connected layers. b k These are all bias parameters in the fully connected layer, Q a Q v These are the query vectors mapped from the audio vector and the image vector, respectively, and K is the query vector. t V is the key vector mapped from the text vector. t It is the value vector mapped from the text vector.

[0064] ② Extraction of similar and dissimilar features:

[0065] By calculating similarity and difference attention scores for modality pairs using two-stream attention, similarity and difference features are modeled, enabling deep interaction computation based on different semantic spaces. This allows for the mining of complementary information between different modalities, ultimately yielding the similarity vector Affi and the difference vector Diff for the two modality pairs. The principle is detailed in the attached figure. Figure 3 As shown, first, the dot product of vector Q and vector K is calculated to obtain the similarity representation matrix α∈R. L×L L represents the sequence length of the input information; a large variance in the characterization matrix may cause overfitting, and to reduce the variance of the matrix, it needs to be divided by . d k Let V represent the dimension of the latent vector, then use the softmax activation function to obtain the normalized matrix, and finally multiply it with the V vector to obtain the similarity vector Aiff∈R. L×D D represents the dimension of the output vector, and the calculation formula is as follows:

[0066] α=Q·K T

[0067]

[0068] The elements of vectors Q and K are subtracted positionally, and then summed to obtain the distance vector β∈R. L×L Finally, multiplying it with the vector V yields the difference vector Diff∈R. L×D The calculation formula is as follows:

[0069]

[0070] β=||QK||

[0071]

[0072] ③ Fusion of similar and dissimilar features

[0073] In this embodiment, an adaptive gated fusion network is used to fuse the similarity and difference features of modality pairs. The input to the gated network is the similarity feature matrix and the difference feature matrix of the modality pair, and the output is a dynamic weight matrix. This weight matrix learns the aggregation of similarity and difference features and is used to weight the similarity feature vector and the difference feature vector. Specifically, a gated filtering matrix G∈R is trained using Affi and Diff vectors. L×L L represents the length of the input sequence. This gating matrix learns its aggregation features through weighted similarity and difference vectors, selectively extracting interactive semantic information and filtering out useless emotional noise. An adaptive gating fusion network module is used to fuse the similarity and difference vectors to obtain the fusion vector F. The principle is as follows: Figure 4 As shown:

[0074] G=δ(Wg ·[Affi,Diff]+b g )

[0075] F = G⊙Affi + (1-G)⊙Diff

[0076] Where ⊙ represents the dot product operation, δ represents the activation function, and W g b represents the weight parameters in the fully connected layer. g This represents the bias parameters in the fully connected layer.

[0077] ④ Emotional polarity prediction classification:

[0078] In this embodiment, after obtaining the fused representation vectors of image-text and audio-text, the vectors of the two modality pairs are concatenated using a concat operation to obtain the fused representation vector of the three modalities. First, a max-pooling operation is used to reduce the dimensionality of the fused vector to simplify network complexity. To better represent the fused vector of the three modalities, a linear transformation network (DNN) is used to extract the sentiment semantic information of each modality. Finally, a softmax classifier is used to analyze and predict its sentiment polarity, predicting the multimodal vector representation as positive and negative categories. In this embodiment, the loss function for this task uses squared error, and L2 regularization and dropout techniques are selected to mitigate overfitting.

[0079] F a,t,v =Concat(F a,t F v,t )

[0080] M a,t,v =MaxPool(F a,t,v )

[0081] D a,t,v =DNN(M a,t,v )

[0082] label = softmax(D a,t,v )

[0083] Among them, F v,t F a,t F represents the fused representation vectors of image-text and audio-text, respectively. a,t,v M represents the fusion vector of audio, image, and text modalities. a,t,v D represents the dimensionality-reduced three-modal fusion vector. a,t,v This represents the multimodal representation vector after linear transformation, and label represents the final predicted sentiment category.

[0084] As described in the above embodiments, those skilled in the art will understand that the present invention provides a multimodal sentiment analysis method based on dual-stream attention and gating fusion. It mainly uses a deep learning model to extract high-order features of multimodal information. In the first stage, dual-stream attention is used to realize the interactive calculation of high-order features. In the second stage, a gating network is used to fuse modal pairs. Finally, a binary classifier is used to analyze and predict sentiment.

[0085] This invention first uses text as the primary modality and images and audio as secondary modalities. Then, semantic features from images and audio are extracted to supplement the undiscovered sentiment in the text, resulting in better classification performance. Since multimodal data reside in different semantic spaces, which are both similar and different, this method models similarity vectors and difference vectors to better describe the correlation and differences in sentiment between modalities. Similarity vectors can uncover the consistency of sentiment across different semantic spaces, while difference vectors can capture fine-grained sentiment differences. Specifically, this method uses a BERT pre-trained model and a BiGRU model to extract multimodal semantic features. These two encoders not only extract deep semantic features but also effectively encode the contextual information and relative position of the data. Modeling similarity and difference features enables deep semantic interaction, achieving information complementarity between modalities. An adaptive gated fusion network effectively fuses similarity and difference features, thereby capturing the consistency and fine-grained differences in sentiment, thus improving the performance of the sentiment analysis task.

[0086] The method of this invention not only solves the problem of insufficient fusion of multimodal information, but also alleviates the difficulty of fusion of independent multimodal data to a certain extent. Numerous experimental results show that this method is superior to most classic deep learning models, proving its effectiveness in feature extraction and modality fusion, thereby improving the accuracy and efficiency of multimodal sentiment analysis and prediction.

[0087] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0088] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A multimodal sentiment analysis method based on the fusion of dual-stream attention and gating, characterized in that, Includes the following steps: S1. Utilize deep learning models to vectorize multimodal information and extract acoustic features from audio, visual features from images, and high-order semantic features from text. S2. Calculate similarity attention and difference attention for modality pairs through dual-stream attention, perform interactive calculations in different semantic spaces, and obtain similarity vectors and difference vectors. S3. An adaptive gating fusion network is used to fuse the similarity vectors and difference vectors of the modal pairs to obtain a multimodal vector representation; S4. Sentiment polarity analysis and prediction based on neural network classifiers for multimodal vector representations; In step S1, the latent vectors of audio and images are mapped to query vectors through different linear transformations, and the latent vectors of text are mapped to key vectors and value vectors; the mapping method is as follows: in, , , These represent the latent vectors for audio, images, and text, respectively. , , , These are the learnable weight parameters in the fully connected layer. , , These are the bias parameters in the fully connected layer. , These are the query vectors mapped from the audio vector and the image vector, respectively. The key vector is the result of mapping the text vector. It is a value vector mapped from a text vector; In step S2 The formula for calculating similar vectors is: The formula for calculating the difference vector is: in, Represents the similarity representation matrix. Represents the difference in characterization matrix. Q Represents the query vector. K Represents the key vector. V Represents a value vector. Represents similar vectors, Represents the difference vector. For activation function, i express The number of rows in a matrix j express The number of columns in a matrix Indicates the dimension of the key vector; In step S3, a gated filtering matrix is ​​trained using similarity vectors and difference vectors; The gated filtering matrix learns its aggregation features by weighted similarity vectors and difference vectors, generates a fusion vector of modality pairs, selectively extracts interactive semantic information, and filters out useless emotional noise to obtain a multimodal vector representation. In step S4, after obtaining the fused representation vectors of image-text and audio-text, the vectors of the two modal pairs are concatenated to obtain the fused representation vectors of the three modalities. First, the dimensionality of the fused vectors is reduced by the max pooling operation. Next, a linear transformation network (DNN) is used to extract the sentiment semantic information of various modalities. Finally, the sentiment polarity is analyzed and predicted by a softmax classifier, and the multimodal vector representation is predicted as positive and negative categories.

2. The multimodal sentiment analysis method based on dual-stream attention and gating fusion according to claim 1, characterized in that, In step S1, the COVAREP algorithm is used to extract acoustic features from the audio.

3. The multimodal sentiment analysis method based on dual-stream attention and gating fusion according to claim 1, characterized in that, In step S1, the FaceNet model is used to extract visual features from the image.

4. The multimodal sentiment analysis method based on dual-stream attention and gating fusion according to claim 1, characterized in that, In step S1, a pre-trained BERT model is used to extract high-order features of text semantics.

Images