A multi-modal sentiment classification method based on progressive neural network

Abstract

The application discloses a kind of multi-modal sentiment classification methods based on progressive neural network, which includes three modules of perception scanning, fine reading, heterogeneous graph reasoning;Perception scanning module is used to the feature representation of visual picture and corresponding text description, and the content information of picture and text is roughly perceived using VL-BERT;Fine reading module uses memory attention network to focus on important features in picture and text, models fine-grained complementary information between them;Heterogeneous graph reasoning module constructs multi-modal heterogeneous graph using the social relationship network between pictures, fuses the scan embedding and fine reading embedding generated in the previous two stages into graph node embedding, and uses graph convolutional neural network to perform multi-modal sentiment polarity reasoning.The application comprehensively utilizes feature representation, cross-modal attention mechanism, graph neural network and other technologies to infer the feature correlation and complementary information between multi-modal, which has a significant effect on improving the accuracy of multi-modal sentiment classification.

Description

Technical Field

[0001] This invention relates to a multimodal sentiment classification method based on a progressive neural network, belonging to the technical fields of multimodal data analysis and social network analysis. Background Technology

[0002] With the rapid development of information technology and the widespread adoption of mobile smart terminals, a large number of social networking platforms have emerged explosively. Every day, a vast number of users share images, text, and other information on different social platforms to showcase their personal experiences or attitudes towards life. Against this backdrop, how to fully mine and utilize social multimodal data to depict people's visions of a better life has aroused increasing research interest among scholars. In social multimodal analysis, the multimodal sentiment classification task requires models to accurately predict sentiment polarity based on information such as images, text, and network data provided by social users, attracting considerable attention. Because the data belongs to different modalities, there are complex feature relationships and insurmountable semantic gaps between cross-modal data, posing significant challenges to multimodal sentiment analysis. However, breakthroughs in multimodal sentiment analysis technology in social networks can not only promote the development of cross-modal data analysis but also have wide applications in scenarios such as business recommendations and political elections, which is of great significance to the development of related fields.

[0003] Despite the abundance of existing multimodal sentiment classification methods, they still suffer from several shortcomings: First, multimodal sentiment polarity is predicted through a single round of reasoning, lacking rereading strategies to update and refine multimodal sentiment information. This is inconsistent with the scanning and rereading methods humans use to understand multimodal content. Second, integrating social relationships into multimodal sentiment analysis is not easy; various types of relationships exist between social images, potentially influencing multimodal sentiment tendencies differently. Furthermore, the complex topological structure of image relationship networks makes it difficult for models to accurately model the associations between multimodal content and network structure, hindering effective cross-modal fusion. Therefore, an effective solution is urgently needed to deeply explore cross-modal feature associations and utilize rich heterogeneous relationship information for multimodal sentiment classification. Summary of the Invention

[0004] The technical problem to be solved by this invention is to overcome the shortcomings of existing multimodal sentiment classification technologies and provide a multimodal sentiment classification method based on a progressive neural network. This method first uses a perceptual scanning module to roughly perceive the content of images and text; then, it uses a fine reading module to explore the deep cross-modal complementary information between images and text; finally, it uses a graph neural network to perform cross-modal sentiment inference on a heterogeneous graph constructed based on image link relationships, thereby improving the accuracy of multimodal sentiment classification.

[0005] The technical solution adopted in this invention is as follows: a multimodal sentiment classification method based on a progressive neural network, which mainly includes three modules: perceptual scanning, fine reading, and heterogeneous graph inference. The perceptual scanning module models the correlation between images and corresponding text descriptions to roughly perceive the content they express. The fine reading module updates and refines cross-modal information through a memory network, focusing on important modal features and cross-modal associations to deeply understand the cross-modal complementary features between images and text. The design of the perceptual scanning and fine reading models is mainly to simulate the excellent skills of humans in progressively understanding cross-modal content. The heterogeneous graph inference module constructs a relationship network using the co-occurrence relationship between social images and integrates the multimodal features from the scanning and fine reading modules as an embedded representation of multimodal nodes in the social network. Cross-modal inference is achieved on the heterogeneous graph through a graph neural network, enabling the model to further capture the influence of links in the social network on multimodal sentiment and predict sentiment polarity.

[0006] The present invention provides a multimodal sentiment classification method based on a progressive neural network, the specific steps of which are as follows:

[0007] Step 1: Use the Faster R-CNN network to extract the target regions in the image and generate the image feature matrix. At the same time, use GloVe word vectors and LSTM to encode the text words corresponding to the image to generate the text feature matrix. Concatenate the two types of feature matrices and input them into VLBERT. VLBERT models the internal features between the image and the text, roughly perceives the content information of the two modalities, and generates scan embedding, image embedding and text embedding.

[0008] Step 2: A dual-branch memory network is used to perform mutual attention calculations on the image embeddings and text embeddings obtained from the perceptual scanning in Step 1, and the attention is used to update the memory units. The attention calculation and memory unit update are repeated multiple times to deeply learn the feature associations between images and text, explore fine-grained complementary features between different modalities, and finally generate detailed reading embeddings.

[0009] Step 3: Construct a heterogeneous graph by utilizing the links between social images, and fuse the scanning embeddings and detailed reading embeddings generated in Step 1 and Step 2 into a multimodal graph node embedding. Use a multi-layer graph convolutional neural network to perform multimodal sentiment polarity inference on the heterogeneous graph.

[0010] Step 4: Embed the graph nodes obtained from the graph convolutional neural network inference in Step 3 into the multimodal sentiment classifier to achieve multimodal sentiment polarity prediction.

[0011] In step one, the scanning embedding is the result generated by perceptual scanning of image and text features using VLBERT. It is the first vector output by VLBERT. Perceptual scanning has a rough understanding of the information contained in the image and text by modeling the feature interactions within and between modalities, and reflects this understanding in the scanning embedding.

[0012] In step two, the detailed embedding is the result of splicing and mapping two memory units in a dual-branch memory network. The memory network repeatedly interacts with the image embedding and text embedding generated by the perceptual scan to mine cross-modal complementary information between the two modalities. The detailed embedding can be seen as the result of deep interaction and complementary fusion between the image and the text.

[0013] In step three, the heterogeneous graph is a weighted undirected graph constructed using co-occurrence relationships between social images. If w co-occurrence relationships exist between images (e.g., taken at the same location, describing the same thing), the weight of the corresponding edge is w. Each graph node corresponds to a joint embedding of an image-text pair. A graph convolutional neural network is used to infer the influence of image link relationships on multimodal sentiment polarity, generating multimodal graph node embeddings that integrate image, text, and network information.

[0014] In step four, the multimodal sentiment classifier is constructed by combining a multilayer perceptron with a softmax activation function. The classifier uses negative log-likelihood loss as the loss function and embeds the multimodal graph nodes after heterogeneous inference as the input to the classifier to predict sentiment polarity.

[0015] The beneficial effects of this invention: Current multimodal sentiment classification methods mainly utilize single-round attention mechanisms to model the relationships between images and text, making it difficult to effectively extract complex sentiment associations between different modalities. Furthermore, existing methods neglect the influence of image link relationships on multimodal sentiment polarity, failing to effectively infer multimodal sentiment tendencies on heterogeneous relational networks. In contrast, the multimodal sentiment classification method based on progressive neural networks proposed in this invention has the following two advantages: First, this invention integrates the excellent human skills of progressively understanding multimodal content into the neural network structure design. Through perceptual scanning, detailed reading, and heterogeneous reasoning of multimodalities, the model can progressively and meticulously capture complementary features between different modalities. Second, this invention utilizes co-occurrence information between images to construct a relational graph network, fusing features obtained from scanning and detailed reading on the heterogeneous graph for cross-modal heterogeneous graph reasoning. By capturing heterogeneous relationships that influence sentiment classification, the accuracy of multimodal sentiment classification is improved. Attached Figure Description

[0016] Figure 1This is a flowchart illustrating a multimodal sentiment classification method based on a progressive neural network. Detailed Implementation

[0017] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0018] Example 1: As Figure 1 As shown, a multimodal sentiment classification method based on a progressive neural network aims to predict the sentiment polarity of each image-text pair, given an image, its corresponding descriptive text set, and the social attributes of the image. The specific implementation steps are as follows:

[0019] Step 1, Sensory Scan:

[0020] For a given image, this invention uses a Faster R-CNN network to extract salient detection targets in the image and represents the target features as E. O ={e o1 , ..., e oi , ..., e on}, where e oi Let E represent the feature vector corresponding to the i-th detected target; for the descriptive text corresponding to the image, each word in the text is encoded using GloVe word vectors, and the word sequence is modeled using LSTM, with the output feature sequence E. T ={e t1 , ..., e ti , ..., e tk} is the text feature matrix, where e ti This represents the vector corresponding to the i-th text word.

[0021] To facilitate calculation, this invention first maps the image target features and corresponding text features to a target space of the same dimension. This process can be described as: E′ O =σ(W1E O +b1), E′ T =σ(W2E T +b2), where W1 and W2 are learnable weight matrices, b1 and b2 are learnable bias terms, and σ is the non-linear activation function ReLU; then, this invention uses VLBERT to perform perceptual scanning of images and text; VLBERT can not only encode content information of different modalities, but also model cross-modal feature interactions, and E′ O and E′ T After concatenation and input into VLBERT, three types of features are output: scan embedding h. CLS Text embedding T and image embedding O; this process can be represented as: ([h CLS |T|O])=VLBERT([E′O ;E′ T ]), where, [E′ O ;E′ T ] represents E′ O and E′ T The feature concatenation, [·|·] is for concatenating elements line by line. During the input process, E′ O and E′ T Separated by the classifier [CLS], the delimiter [SEP], and the terminator ; scan embedding h CLS This is the first output vector of the last layer of VLBERT, representing the cross-modal joint embedding of the image and text after perceptual scanning; the text embedding T and the image embedding O are E′ O and E′ T The output matrices corresponding to the input positions represent the initial understanding of the content represented by each modality after joint scanning of text and images.

[0022] Step Two: Detailed Reading

[0023] Building upon perceptual scanning, this invention constructs a refined reading model between images and text. It employs a rereading strategy to update and refine cross-modal information, capturing fine-grained correlation features between different modalities to fully learn cross-modal complementary information. The refined reading model is implemented using a memory-based network on two non-independent branches. Specifically, this invention first uses a fully connected neural network to convert the image target embedding O and the text embedding T into memory-based M, respectively. O and M T Then, in one branch, compute the image target embedding O to memory M. T Attention vector α T In another branch, the text embedding T is computed against memory M. O Attention vector α O Next, in both branches, the attention vector is simultaneously multiplied by the corresponding memory unit, and the result is linearly mapped to update the value of the memory unit, with α... T The calculation and M T Taking the update as an example, this calculation process can be expressed as: α T =softmax(W αT (OM T )+b αT M T ←(α T ⊙M T W T +b T Among them, W αT and W T It is a learnable weight matrix, b αT and b T, where is the learnable bias term, ← represents update, and ⊙ is the element-wise multiplication operation after data broadcasting on the appropriate axis. While single-step attention can capture important feature relationships between different modalities to some extent, it may contain irrelevant information and lacks the ability to capture complex cross-modal complementary information. Therefore, this invention naturally extends attention from single-step to multi-hop to achieve fine-grained cross-modal relation extraction. In the extended attention network, the computation of the attention vector and the update of memory are repeated R times. Again, α... T and M T Taking the r-th calculation and update operation as an example, the process can be represented as follows:

[0024]

[0025] Where 1≤r≤R, the superscript r denotes the value calculated in the r-th round; through R rounds of attention calculation, the memory on both branches... and This can be seen as the result of intensive interaction between images and text, containing feature associations and complementary information between the two; finally, this invention further integrates... and To obtain detailed reading embedding h ATT The calculation process can be expressed as follows: Among them, W h and b h These are the learnable weight matrix and bias term, respectively. Through detailed reading of images and text, the embedding h is precisely analyzed. ATT This can be seen as the result of the model's deep understanding of the image and text content.

[0026] Step 3: Heterogeneous Graph Reasoning:

[0027] To leverage the social attributes of images, this invention transforms different co-occurrence relationships between images (taken at the same location, describing the same thing, etc.) into different types of image links, assigning a weight of 1 to each link. This creates a weighted undirected graph G = (H, E), where H represents the nodes of the graph and E represents the weighted edges. To capture the impact of image link relationships on multimodal sentiment polarity, this invention performs cross-modal sentiment inference on graph G. First, a multimodal bilinear pooling model is used to embed the scan embeddings h learned in steps one and two. CLS and intensive reading embedding h ATT Perform the following feature fusion: Where U and V are learnable weight matrices, and g is the preset number of factors. The fused joint feature h is the result of perceptual scanning and fine reading, and it is embedded as a node in the undirected graph G, i.e., h∈H. Next, this invention uses an F-layer graph convolutional neural network to perform multimodal sentiment reasoning on the heterogeneous graph G. Specifically, the graph convolution operation of the f-th layer (1≤f≤F) can be represented as: Among them, H (f) H represents the graph node embedding at level f, and the graph node embedding at level 0. (0) That is the original node code H. This represents the sum of the contiguous matrix and the identity matrix in graph G. That is the corresponding degree matrix, W (f) It is a learnable weight matrix, and σ is the non-linear activation function ReLU; this invention embeds the graph nodes of the F-th layer into H. (F) As a result of multimodal sentiment inference on heterogeneous graphs, H (F) It includes both the correlation and complementary features between images and text, as well as the heterogeneous information brought about by image link relationships. It is the result of deep inference of features from three modalities: images, text, and networks.

[0028] Step 4: Multimodal sentiment classification:

[0029] Based on the graph node embedding H obtained in step three (F) This invention employs a multilayer perceptron to construct a classifier and uses a negative log-likelihood function as the loss function for multimodal sentiment polarity classification (positive and negative). This process can be expressed as: p(h (F) = softmax(W) p h (F) ), L=-log(p(h (F) ), y), where h (F) ∈H (F) It is a graph node embedding on the F-th layer of the graph convolutional neural network, p(h (F) ) indicates that for h (F) The probability distribution for predicting sentiment polarity, W p It is a learnable weight matrix, L represents the loss function, and y is the true sentiment polarity label of the image-text pair on the graph node, with a value of 1 (positive) or 0 (negative).

[0030] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of protection of the claims.

Claims

1.A social multi-modal sentiment classification method based on progressive neural network, characterized in that: The method includes the following steps: A. Perceptual Scanning: For visual images and corresponding text descriptions, different neural networks are used to encode them into image features and text features respectively. These two types of features are mapped to the same feature space through a fully connected neural network, and VL-BERT is used to roughly perceive the content information of the images and text to generate image embeddings, text embeddings and scan embeddings. Image features are generated by using the Faster R-CNN network to extract salient detected objects in the image, resulting in an image object feature matrix. The image object feature matrix is ​​represented as E. O ={e o1 , ..., e oi ,...,e on }, where e oi This represents the feature vector corresponding to the i-th detected target; the text features are word feature matrices generated using GloVe word vectors and LSTM encoding of text words, and the word feature matrix E T ={e t1 , ..., e ti ,...,e tk }, where e ti This represents the vector corresponding to the i-th word in the text. Mapping image features and corresponding text features to a target space of the same dimension is described as: E′ O =σ(W1E O +b1), E′ T =σ(W2E T +b2), where W1 and W2 are learnable weight matrices, b1 and b2 are learnable bias terms, and σ is the non-linear activation function ReLU; E′ O and E′ T After concatenation and input into VL-BERT, three types of features are output: scan embedding h. CLS Text embedding (T) and image embedding (O) are represented as follows: ([h CLS |T|O])=VLBERT([E′ O ;E′ T ]), where, [E′ O ;E′ T ] represents E′ O and E′ T The feature concatenation, [·|·] is for concatenating elements line by line. During the input process, E′ O and E′ T Separated by the classifier [CLS], the delimiter [SEP], and the terminator ; scan embedding h CLS This is the first output vector of the last layer of VL-BERT, representing the cross-modal joint embedding of the image and text after perceptual scanning; the text embedding T and the image embedding O are E′ O and E′ T The output matrices corresponding to the input positions represent the initial understanding of the content represented by each modality after joint scanning of text and images; In step A, VL-BERT is a multimodal pre-trained model that concatenates text features and image features as input. Through feature self-attention learning within a modality and feature association learning across modalities, it roughly perceives the information of image and text content. In step A, the output of VL-BERT contains a vector and two matrices. The vector is the first output of VL-BERT, which represents the scan embedding. The outputs corresponding to the image and text input positions constitute the image embedding matrix and text embedding matrix obtained after perceptual scanning, respectively. B. Fine Reading: Based on the image embedding and text embedding generated by perceptual scanning, a two-branch memory attention network is constructed to calculate the attention values ​​of image to text and text to image respectively, and update the memory memory units. After feature concatenation, the memory memory units of the two branches are mapped to fine reading embeddings using a neural network. The detailed reading model is implemented using a memory-based network on two non-independent branches. Specifically, a fully connected neural network is used to convert the image target embedding O and the text embedding T into memory-based M, respectively. O and M T Then, in one branch, compute the image target embedding O to memory M. T Attention vector α T In another branch, the text embedding T is computed against memory M. O Attention vector α O Next, in both branches, the attention vector is simultaneously multiplied by the corresponding memory unit, and the result is linearly mapped to update the value of the memory unit, with α... T The calculation and M T The update process is represented as: α T =softmax(W αT (OM T )+b αT M T ←(α T ⊙M T W T +b T Among them, W αT and W T It is a learnable weight matrix, b αT and b T , which is a learnable bias term; ← indicates update; ⊙ is the element-wise multiplication operation after data broadcasting on the appropriate axis. Extending attention from a single step to multiple hops, in the extended attention network, the computation of the attention vector and the update of memory are repeated R times, and α is also... T and M T The process of the r-th calculation and update operation is represented as follows: Where 1≤r≤R, the superscript r denotes the value calculated in the r-th round; through R rounds of attention calculation, the memory on both branches... and Viewed as the result of dense interaction between images and text, it contains feature correlations and complementary information between the two; finally, it is fused. and To obtain detailed reading embedding h ATT The calculation process is expressed as follows: Among them, W h and b h These are the learnable weight matrix and the bias term, respectively. In step B, the memory attention network performs attention calculations on the image embedding matrix and text embedding matrix obtained from the perceptual scan, and uses the attention to update the memory memory units. The calculation and update process is repeated multiple times to achieve in-depth feature association learning between images and text. In step B, the precise embedding is obtained by concatenating the features of the memory units calculated by the dual-branch memory attention network and then performing feature mapping using a fully connected neural network. Through repeated calculations by the memory network, the precise embedding captures the complementary information between the image and the text. C. Heterogeneous Graph Reasoning: A weighted undirected graph is constructed using the relational network between social images. Scan embeddings and close reading embeddings are fused using a bilinear pooling model. The fused image-text embeddings are used as node embeddings in the heterogeneous graph. A graph convolutional neural network is used to perform cross-modal sentiment reasoning on the graph. In step C, the weighted undirected graph is a heterogeneous graph constructed using the co-occurrence relationships between social images. The model creates different image links for different co-occurrence relationships between images and assigns a weight of 1 to each link relationship. If there are w link relationships between two image nodes, the corresponding edge weight is w. Create a weighted undirected graph G = (H, E), where H represents the nodes of the graph and E represents the weighted edges. Perform cross-modal sentiment inference on graph G. First, use a multimodal bilinear pooling model to embed the scans learned in steps A and B into h. CLS and intensive reading embedding h ATT Perform the following feature fusion: Where U and V are learnable weight matrices, g is the preset number of factors, and the fused joint feature h is the result of perceptual scanning and fine reading, which is embedded as a node in the undirected graph G, i.e., h∈H; then, an F-layer graph convolutional neural network is used to perform multimodal sentiment inference on the heterogeneous graph G, i.e., the graph convolution operation of the f-th layer (1≤f≤F) is represented as: Among them, H (f) H represents the graph node embedding at level f, and the graph node embedding at level 0. (0) That is the original node code H. This represents the sum of the contiguous matrix and the identity matrix in graph G. That is the corresponding degree matrix, W (f) It is a learnable weight matrix, and σ is the non-linear activation function ReLU; In step C, the embedding of graph nodes in heterogeneous graph reasoning is the corresponding image-text joint embedding. This embedding encoding is derived from the feature fusion of scanning embedding and intensive reading embedding obtained from perceptual scanning and intensive reading. Multimodal sentiment reasoning on heterogeneous graphs is implemented using a multilayer graph convolutional neural network. D. Multimodal sentiment classification: The graph nodes obtained by heterogeneous graph reasoning are embedded into the sentiment classifier constructed by a multilayer perceptron to predict the multimodal sentiment polarity; In step D, the multimodal sentiment classification adopts a softmax classifier, which takes the negative log-likelihood function as the loss function and uses the graph node embedding after heterogeneous graph reasoning as the input of the classifier to predict the sentiment polarity of the multimodal data, and the specific process is as follows: Based on the graph node embedding H obtained in step C (F) A multilayer perceptron is used to construct a classifier, and a negative log-likelihood function is used as the loss function for multimodal sentiment polarity classification. This process is expressed as: p(h (F) = softmax(W) p h (F) ), L=-log(p(h (F) ), y), where h (F) ∈H (F) It is a graph node embedding on the F-th layer of the graph convolutional neural network, p(h (F) ) indicates that for h (F) The probability distribution for predicting sentiment polarity, W p It is a learnable weight matrix, L represents the loss function, and y is the true sentiment polarity label of the image-text pair on the graph node, with a value of 1 indicating positive or 0 indicating negative.

Images