A Multimodal Sentiment Analysis Method Based on Tertiary Image Features and Graph Data Texts
By combining BiGCN and CNN models, image and text features are extracted, and the commonality and ironic recognition problems of sentiment analysis are solved, achieving more efficient multimodal sentiment analysis.
Patent Information
- Application Number
- CN202211144564.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-09-20
- Publication Date
- 2025-08-01
- Estimated Expiration
- 2042-09-20
AI Technical Summary
In the prior art, in sentiment analysis, there are problems such as poor generality of image sentiment analysis, errors caused by inter-user relationship information, and the inability to accurately identify satirical expressions.
Text features are extracted by BiGCN model, and low-level, intermediate and advanced features of the image are extracted in combination with CNN model, feature fusion is performed through the fusion layer, and multimodal sentiment analysis methods of BiGCN model and CNN model, combining relationships between users and ironic recognition, multimodal sentiment analysis is performed.
It improves the accuracy and versatility of emotion recognition, can better identify satirical expressions, reduces errors, and achieves the accuracy of multimodal emotion analysis.
Smart Images
Figure CN115512377B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of computer vision and natural language processing, and particularly relates to a multi-modal sentiment analysis method based on three-level image features and graph data text. Background Art
[0002] Image features related to sentiment can generally be divided into low-level visual features (such as color, lines, and texture), intermediate image aesthetics (such as composition, emphasis, and visual balance), and high-level image semantics. The sentiment of network images is mainly related to image semantics, the sentiment of art photos is mainly related to image aesthetics, and the sentiment of abstract paintings is mainly related to visual features. Different levels of deep representations such as image semantics, image aesthetics, and visual features have different degrees of influence on sentiment analysis of different types of images. Research on sentiment analysis for a single type of image has poor generality. For example, AlexNet is suitable for sentiment analysis of images with high-level semantic features; A-CNN is targeted at intermediate aesthetic features; T-CNN focuses on image sentiment analysis based on low-level visual features.
[0003] There are connections between users in social media, and their sentiments will also be affected to varying degrees. In the prior art, only text is considered when performing sentiment analysis, without considering the relationship between users, which will cause certain errors.
[0004] In addition, irony, as a language expression, widely exists in users' social interactions. Its literal meaning is opposite to the actual expressed meaning, but it is difficult for sentiment analysis in the prior art to recognize this point. Summary of the Invention
[0005] In order to overcome the above-mentioned disadvantages of the prior art, the purpose of the present invention is to provide a multi-modal sentiment analysis method based on three-level image features (low-level, intermediate, and high-level features) and graph data text. The problems to be solved include: (1) The generality problem of image sentiment analysis. (2) The error problem caused by insufficient consideration of the relationship information between users. (3) The problem of being unable to accurately identify ironic expressions.
[0006] In order to achieve the above purpose, the technical solution adopted by the present invention is:
[0007] A multi-modal sentiment analysis method based on three-level image features and graph data text, comprising the following steps:
[0008] Step 1, obtaining a comment data set composed of images and text;
[0009] Step 2, using the BiGCN model to extract text features and performing sentiment analysis to obtain a sentiment value M1;
[0010] Step 3: Use a CNN model to extract image features at different levels from the image;
[0011] The CNN model includes a main network, four branches, a fusion layer, and a classification layer; the main network includes four convolutional layers in sequence, and a branch is set after each convolutional layer; among them, the first convolutional layer and the second convolutional layer extract low-level features of the image, that is, visual features; the third convolutional layer extracts intermediate-level features of the image, that is, image aesthetic features; the fourth convolutional layer extracts high-level features of the image, that is, image semantic features; each branch includes a normalization layer and a fully connected layer, the output of each convolutional layer is normalized and then passed through the fully connected layer, and then feature fusion is performed in the fusion layer, and sentiment analysis is performed to obtain a sentiment value M2;
[0012] Step 4: Fuse the sentiment value M1 and the sentiment value M2 at the result level to obtain a fused sentiment value M;
[0013] Step 5: Fuse the text features and the image features for irony recognition, and determine the final sentiment value according to the irony recognition result The formula is as follows:
[0014]
[0015] Among them, S represents the irony recognition result, S = 1 means recognized as irony, and S = 0 means recognized as non-irony;
[0016] When represents positive; when represents neutral; when represents negative.
[0017] In one embodiment, in step 2, a dual-graph mode of the original graph and the feature graph is used to extract text features. The original graph represents the text and the relationship between texts, and is constructed based on the relationship between users, denoted as G(V, E), where V represents nodes, and each text in the comment dataset is a node, and E represents edges, representing the relationship between texts; V = {1,..., n}, and n is the number of texts in the comment dataset, that is, the number of nodes; the feature graph is constructed based on the potential relationship between comments, denoted as Gf.
[0018] In one embodiment, the original graph is filtered column by column, the feature graph is filtered row by row, and the mean value of the features obtained from both is taken to obtain text features through one layer of the BiGCN model.
[0019] In one embodiment, in the BiGCN model, Laplacian regularization on the original graph and the feature graph is added to obtain:
[0020]
[0021] Among them, Y is the feature matrix of the original graph, L1 is the normalization matrix of the original graph, and Y T is the feature matrix of the feature graph, L2 is the normalization matrix of the feature graph, and F = H (l) is the output of the l-th layer of the BiGCN model, and λ′1, λ′2 are Laplacian regularization hyperparameters. The optimal solution of this equation is equal to the solution of the differential equation:
[0022]
[0023] λ′1L1Y + λ′2YL2 = F - Y
[0024] Transform this differential equation into a bi-criterion optimization problem with equality constraints.
[0025]
[0026]
[0027]
[0028] Then, use the ADMM method for update iteration, approximate it with the first-order Taylor expansion to reduce the complexity, and take the mean of Y1 and Y2 as the approximate solution of the feature output Y after two iterations.
[0029] In one embodiment, update Y1 through a column low-pass filter and update Y2 through a row low-pass filter where p is a hyperparameter and I is the identity matrix;
[0030] Set the initial iteration value Y 0 = 0,
[0031] Iteratively update to obtain the feature output Y according to the following update formula:
[0032]
[0033]
[0034]
[0035] Y = ADMM(F, L1, L2)
[0036] In the formula, k is the iteration round;
[0037] Input Y into the second layer of the BiGCN model to obtain the text features.
[0038] In one embodiment, in step 3, in the CNN model, the probability formula and the loss formula are as follows:
[0039]
[0040]
[0041] Among them, h i is the output of the fully connected layer, i = 1, …, m, where m is the type of the final sentiment classification, and y i is the label, and it is the true label of the image.
[0042] In one embodiment, for step 4, the fusion calculation formula is as follows:
[0043]
[0044] δ is the proportion, which is determined by parameter tuning during the training process.
[0045] In one embodiment, for step 5, the image features and text features are fully connected to construct a two-layer BP neural network as the classification layer. The Sigmoid function is used as the activation function, the loss calculation uses cross entropy, and Adam is used as the optimizer to perform sarcasm recognition.
[0046] Compared with the prior art, the present invention combines the low, medium, and high-level features of the image and the text features of the graph data, greatly reducing the error of sentiment recognition and having good versatility at the same time; by introducing sarcasm recognition, it can more accurately identify the sentiment polarity. BRIEF DESCRIPTION OF THE DRAWINGS
[0047] Figure 1 is a schematic diagram of the text sentiment analysis framework of the present invention
[0048] Figure 2 is a schematic diagram of the image sentiment analysis framework of the present invention
[0049] Figure 3 is a schematic diagram of the sarcasm recognition network structure of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS
[0050] The following will describe the embodiments of the present invention in detail with reference to the drawings and embodiments.
[0051] Facing the problems of the generality of image sentiment analysis, the error problem caused by the insufficient consideration of the relationship information between users, and the problem of unable to accurately identify sarcastic expressions in multi-modal sentiment analysis, the present invention provides a multi-modal sentiment analysis method based on three-level image features and graph data text, and corresponding solutions are given from the following aspects.
[0052] First, in order to fully consider the relationship information among users when performing text sentiment analysis, the present invention uses a bidirectional low-pass filter neural network for sentiment analysis, extracts features from the original graph using a row filter; and extracts features from the feature graph constructed based on feature correlation using a column filter.
[0053] Second, to solve the generality of image sentiment analysis, the present invention effectively unifies different levels of deep representations in a CNN, adopts a structure of one main network and four branches, extracts different levels of deep representations from global and local aspects from different convolutional layers, and uses a multi-head attention mechanism for fusion of different levels of deep representations in the fusion layer.
[0054] Finally, to solve the problem that the intuitive meaning of the comments published by users is contrary to the actual expressed meaning, which affects the accuracy of sentiment analysis, the present invention applies irony recognition to sentiment analysis. Since images and texts, as independent wholes, may use irony, irony recognition can be performed on images and texts respectively.
[0055] Specifically, the present invention includes the following steps
[0056] Step 1, obtain a comment dataset composed of images and texts.
[0057] In the present invention, the comment dataset can be crawled from social websites. One piece of data often includes images and texts. Among them, the images and texts generally have obvious correlations.
[0058] Step 2, use the BiGCN model to extract text features and perform sentiment analysis to obtain a sentiment value M1.
[0059] For the sentiment analysis of texts, the present invention uses the BiGCN model for feature extraction.
[0060] Because there are certain connections among users in social media, the features of the original graph and the feature graph can be extracted using a dual-graph mode. Specifically, the original graph represents texts and the relationships among texts, is constructed based on the relationships among users, and is represented as G(V,E). B represents nodes, and each text in the comment dataset is a node. E represents edges, representing the relationships among texts. The relationship is defined as having the same topic words, and the topic words are obtained from a neural topic model; V = {1,…,n}, where n is the number of texts in the comment dataset, that is, the number of nodes. The feature graph is constructed based on the potential relationships among comments and is represented as Gf. The potential relationships among comments can be captured by BiGCN.
[0061] In this step, a general spectral GCN with a bidirectional low-pass filter is adopted to filter the original graph column by column and the feature map row by row. Then, the means of the features obtained from both are taken and passed through one layer of the BiGCN model to obtain the text features.
[0062] In a specific algorithm, by converting the text into a vector, the column vector x of the feature matrix of the text can be obtained. In the original graph, the i-th node is associated with the i-th element in the column vector x of the feature matrix, and x ∈ R n , the Laplacian matrix L of the original graph is L = D - A, where A is the adjacency matrix of the original graph and D is the degree matrix of the i-th node. Then, the smoothness Δx of x is measured by the Laplacian quadratic form, that is, Δx = x T Lx,
[0063] Meanwhile, the Laplacian matrix L = UΛU T , where U is the orthogonal eigenvector, serving as the basis of the Fourier transform, and Λ is the eigenvalue of L, serving as the graph frequency.
[0064] Fourier transform of the graph:
[0065] Inverse Fourier transform of the graph:
[0066] The graph filter is used to filter the input X:
[0067] g(L)X = Ug(Λ)U T X = Ug(Λ)F(X)
[0068] where g(L)X represents filtering X, g(Λ) = diag(g(λ1,…,λ N )), λ1,…,λ N represent the eigenvalues, N represents the number of eigenvalues, and g(λ1,…,λ N ) represents the filtering.
[0069] In BiGCN, in order to obtain better feature outputs, Laplacian regularization on the original graph and the feature map is added to get:
[0070]
[0071] where Y is the feature matrix of the original graph, L1 is the normalized matrix of the original graph, Y T is the feature matrix of the feature map, L2 is the normalized matrix of the feature map, F = H (k) is the output of the l-th layer of the BiGCN model. For example, l = 1, 2. The BiGCN framework of the l-th layer is as follows Figure 2 shown. λ′1, λ′2 are Laplacian regularization hyperparameters, and the optimal solution of this equation is equal to the solution of the differential equation:
[0072]
[0073] λ′1L1Y + λ′2YL2 = F - Y
[0074] Transform this differential equation into a bi - criterion optimization problem with equality constraints.
[0075]
[0076]
[0077]
[0078] Then, use the ADMM method for update iteration, approximate it with the first - order Taylor expansion to reduce the complexity, and take the mean of Y1 and Y2 as the approximate solution of Y after two iterations. Pass through a column low - pass filter to update Y1, and pass through a row low - pass filter to update Y2. The filtering is to obtain a better feature Y from the input original graph features. p is a hyperparameter, and I is the identity matrix.
[0079] Set the initial iteration value of Y 0 = 0,
[0080] Update to obtain the feature output Y according to the following update formula:
[0081]
[0082]
[0083]
[0084] Y = ADMM(F, L1, L2)
[0085] In the formula, k is the iteration round; input Y to the second layer to obtain the text feature.
[0086] Since it is difficult to quantitatively describe the feature correlation, D2 is the degree matrix of A2, L2 is a learnable symmetric matrix determined by H (l) , W is an upper - triangular learnable symmetric matrix, so the update calculation formula of L2 is as follows:
[0087] H k+1 = σ(ADMM(H (l) , L1, L2)W l )
[0088] W2 = sigmoid(W)
[0089] A2 = W2 + W2 T
[0090] L2 = I - D2 -1 / 2 A2D2 -1 / 2
[0091] W l A2 is the learnable symmetric matrix obtained by the l-th layer of the BiGCN model, and W2 is the normalized result of W.
[0092] Step 3: Use the CNN model to extract image features at different levels.
[0093] For the sentiment analysis of images, in this step, from the global and local perspectives, sentiment recognition is performed through image semantics, image aesthetics, and low-level visual features. The CNN model of the present invention extracts different levels of deep representations and then integrates the deep representations at different levels through a fusion layer for classification. The CNN uses convolutional layers and fully connected layers to extract representations layer by layer from the input image, followed by a Softmax layer, and the last layer converts into the probability output of the image emotion type. In the CNN model of the present invention, the probability formula and the loss formula are as follows:
[0094]
[0095]
[0096] where h i is the output of the fully connected layer, i = 1,..., h, n is the type of the final sentiment classification, y i is the label, is the true label of the image.
[0097] Different CNN models are similar in structure, and the different numbers of convolutional layers determine the different levels of feature extraction. In order to effectively unify the deep expressions at different levels, for example, in the present invention, referring to Figure 2 , the CNN model of the present invention includes a main network, four branches, a fusion layer, and a classification layer. The main network includes four convolutional layers in sequence, and a branch is set after each convolutional layer. The first convolutional layer and the second convolutional layer extract the low-level features of the image, that is, visual features; the third convolutional layer extracts the intermediate-level features of the image, that is, image aesthetic features; the fourth convolutional layer extracts the high-level features of the image, that is, image semantic features. Each branch includes a normalization layer and a fully connected layer. The output of each convolutional layer is normalized and then passed through the fully connected layer, and then feature fusion is performed in the fusion layer, and sentiment analysis is performed to obtain the sentiment value M2.
[0098] Exemplarily, the filter sizes of the four convolutional layers are 11×11, 5×5, 5×5, and 5×5 respectively. For each convolutional layer, the next are two fully connected layers, and a 1×1 convolutional layer is inserted between the pooling layer and the fully connected layer to unify the output dimensions. The fusion layer uses an attention mechanism to allocate weights for depth expressions at different levels to make the fusion result more reasonable.
[0099] Step 4, fuse the sentiment values M1 and M2 at the result level to obtain the fused sentiment value M. The fusion calculation formula is as follows:
[0100]
[0101] δ is the proportion, which is determined by parameter tuning during the training process.
[0102] Step 5, in order to reduce the impact of sarcasm on the final M value of sentiment analysis, fuse text features and image features for sarcasm recognition.
[0103] In this step, the image features and text features can be fully connected to construct a two-layer BP neural network as the classification layer. The Sigmoid function is used as the activation function, and the cross-entropy is used for loss calculation. Adam is used as the optimizer for sarcasm recognition. If the sarcasm recognition result S is 1, the sentiment value is reversed; if the sarcasm recognition S is 0, the sentiment value remains unchanged. The framework structure is as Figure 3 shown. According to the sarcasm recognition result, determine the final sentiment value The formula is as follows:
[0104]
[0105] Among them, S represents the sarcasm recognition result. S = 1 indicates that it is recognized as sarcasm, and S = 0 indicates that it is recognized as non-sarcasm;
[0106] When represents positive; when represents neutral; when represents negative.
[0107] In summary, the present invention conducts multi-modal sentiment analysis based on text images, greatly improving the utilization rate of data; extracts three-level features for images, fully considering the influence of low-level, intermediate-level, and high-level features on sentiment analysis; at the same time, due to the connection between text data, the BiGCN model is used to extract features of the original graph and the feature graph, fully considering the connection between text data, making sentiment analysis no longer independent and one-sided; and because the artificial addition of sarcasm techniques to data will affect the accuracy of sentiment analysis, finally, sarcasm recognition is performed on the full connection of image text features, ultimately increasing the accuracy of sentiment analysis.
Claims
1. A multi-modal sentiment analysis method based on three-level image features and graph data text, characterized in that, It includes the following steps: Step 1, obtain a comment dataset consisting of images and texts; Step 2, use the BiGCN model to extract text features and perform sentiment analysis to obtain a sentiment value M1; Step 3, use the CNN model to extract image features at different levels in the image; The CNN model includes a main network, four branches, as well as a fusion layer and a classification layer; the main network includes four convolutional layers in sequence, and a branch is set after each convolutional layer; among them, the first convolutional layer and the second convolutional layer extract low-level features of the image, that is, visual features; The third convolutional layer extracts intermediate features of the image, that is, image aesthetic features; the fourth convolutional layer extracts high-level features of the image, that is, image semantic features; each branch includes a normalization layer and a fully connected layer, the output of each convolutional layer is normalized and then passed through the fully connected layer, and then feature fusion is performed in the fusion layer and sentiment analysis is performed to obtain a sentiment value M2; Step 4, fuse the sentiment value M1 and the sentiment value M2 at the result level to obtain a fused sentiment value M; Step 5: Integrate the text features and the image features for sarcasm recognition, and determine the final sentiment value based on the sarcasm recognition result The formula is as follows: Among them, S represents the sarcasm recognition result, S = 1 indicates recognition as sarcasm, and S = 0 indicates recognition as non-sarcasm; When indicates positive; When indicates neutral; When indicates negative; In step 2, the text features are extracted using the dual-graph mode of the original graph and the feature graph. The original graph represents the text and the relationships between texts, which is constructed based on the relationships between users and is expressed as G(V, E). V represents nodes, and each text in the comment dataset is a node. E represents edges, representing the relationships between texts; V = {1,..., n}, where n is the number of texts in the comment dataset, that is, the number of nodes; the feature graph is constructed based on the potential relationships between comments and is expressed as Gf; the original graph is filtered column by column, the feature graph is filtered row by row, and the mean value of the features obtained from both is taken and passed through one layer of the BiGCN model to obtain the text features.
2. The multimodal sentiment analysis method based on three-level image features and graph data text according to claim 1, wherein In the BiGCN model, Laplacian regularization on the original graph and the feature graph is added to obtain: Among them, Y is the feature matrix of the original graph, L1 is the normalization matrix of the original graph, and Y T is the feature matrix of the feature graph, L2 is the normalization matrix of the feature graph, and F = H (l) is the output of the l-th layer of the BiGCN model, and λ′1, λ′2 are Laplacian regularization hyperparameters. The optimal solution of this equation is equal to the solution of the differential equation: λ′1L1Y + λ′2YL2 = F - Y Convert this differential equation into a bi-criterion optimization problem with equality constraints. Then use the ADMM method for update iteration, approximate with the first-order Taylor expansion to reduce the complexity, and take the mean value of Y1 and Y2 as the approximate solution of the feature output Y after two iterations.
3. The multimodal sentiment analysis method based on three-level image features and graph data text according to claim 2, wherein Pass through a column low-pass filter Update Y1 by passing through a row low-pass filter Update Y2, where p is a hyperparameter and I is the identity matrix; Set the initial iteration value Y 0 = 0, Iteratively update according to the following update formula to obtain the feature output Y: Y = ADMM(F, L1, L2) In the formula, k is the number of iteration rounds; Input Y into the second layer of the BiGCN model to obtain the text features.
4. The multimodal sentiment analysis method based on three-level image features and graph data text according to claim 1, wherein In step 3, in the CNN model, the probability formula and the loss formula are as follows: Among them, h i is the output of the fully connected layer, i = 1, …, m, where m is the number of categories for the final sentiment classification, y i is the label, and is the true label of the image.
5. The multimodal sentiment analysis method based on three-level image features and graph data text according to claim 1, characterized in that, In step 4, the fusion calculation formula is as follows: δ is the proportion, which is determined by parameter tuning during the training process.
6. The multimodal sentiment analysis method based on three-level image features and graph data text according to claim 1, characterized in that In step 5, fully connect the image features and the text features, construct a two-layer BP neural network as the classification layer, use the Sigmoid function as the activation function, calculate the loss using cross-entropy, and use Adam as the optimizer to perform sarcasm recognition.