Image-text sentiment analysis method based on reconstructed double-attention network

Abstract

The invention provides an image-text sentiment analysis method based on a reconstructed double-attention network, and the method comprises the steps: obtaining visual features through employing ResNet50 in visual feature extraction, introducing a space and channel reconstruction convolution module, reconstructing the visual feature space and channel position information, carrying out the fusion of key information of different positions, and carrying out the recognition of the image-text sentiment. And enhancing visual feature extraction. In text feature extraction, a BERT model is used to obtain text feature representation, and a bidirectional gating loop unit is used to pay attention to context relationships among low-level words, so that text semantic features are enhanced. Finally, an interactive attention mechanism is used for paying attention to feature interaction between modes, visual features and text features are fused, and the problem that key information cannot be well represented in the image-text double-model feature fusion stage is effectively solved.

Description

Technical Field

[0001] The present invention belongs to the fields of artificial intelligence, natural language processing and computer vision technology, and specifically relates to a method for analyzing image-text sentiment based on reconstructing a dual attention network. Background Art

[0002] With the rapid development of Internet technology, the information posted by users on social media has become more and more complex. Compared with the previous single text comment information, the comments on things are becoming more and more diverse. In order to describe their attitudes towards things in a more detailed way, people use images and text pairs to express their degree of liking for things. Therefore, the research on multimodal sentiment analysis has pushed the research on natural language processing sentiment analysis to a new height. For example, the sentence "He writes well" is easy to see from the literal meaning that it is a positive emotion, but people often use body expressions or facial expressions to express their attitude towards things in a deeper way.

[0003] Therefore, under different conditions, the emotions expressed in a sentence are completely different. When the speaker expresses other emotional expressions such as happy or negative expressions, they correspond to the speaker's positive and negative emotional attitudes, respectively. Therefore, it is particularly important to pay attention to the information of visual modality, making the results of sentiment analysis more reliable. It can provide enterprises, organizations and governments with the ability to explore the needs and attitudes of the general public towards the matter, understand the views of the masses in a timely manner, and help better understand the emotions and feedback of the public, which will play a positive role in the development of things.

[0004] For example, when buying goods online, people will check the reviews of the goods on the shopping platform. When traveling, they will check the reviews of the tourist destinations on Dianping.com. Therefore, the review information on social media is the most authentic way to understand consumers' views on things. Summary of the invention

[0005] The purpose of the present invention is to provide a method for sentiment analysis of images and texts based on a reconstructed dual attention network. By introducing a reconstructed unit convolution module, the connection density between channels is reduced and the redundancy of the spatial dimension is suppressed, and the key information representation of the base class model is enhanced, so as to effectively identify the key information in the vision. At the same time, the interactive attention mechanism is used to interact information between different modal features, which effectively enhances the attention to the visual feature space and channels, and effectively improves the problem of not being able to well represent key information in the feature fusion stage.

[0006] In order to achieve the above purpose, the present application provides a method for sentiment analysis of images and texts based on reconstructing a dual attention network, comprising the following steps:

[0007] S1. In text feature extraction, the input is original text data;

[0008] S2. Use the BERT network to capture text semantic features. When using the BERT network model to extract word features from text, extract key information about the word's position in the text, and obtain the importance of the word to the entire text, and efficiently represent the importance of the word to the text;

[0009] S3. Use a bidirectional recurrent gating network to enhance the text features extracted by the BERT network model. Since the BERT network model focuses too much on the feature values ​​between words, the bidirectional recurrent gating network has a good effect on learning low-level semantics. The combination of BERT and the bidirectional recurrent gating network can effectively realize multi-level and multi-depth text semantic information learning;

[0010] S4. Finally, the multi-level and multi-depth text semantic features are captured;

[0011] S5. In image feature extraction, the input is image data;

[0012] S6. Use ResNet50 to extract features from the original image data. The ResNet50 network can process images of different sizes, illuminations, and angles. By introducing residual blocks and deepening the network depth, the problem of vanishing gradients of visual features can be effectively reduced, making ResNet50 important in feature extraction;

[0013] S7. Enhance the visual features extracted by the base model by introducing a spatial reconstruction module. The spatial reconstruction module effectively solves the problem that CNN fails to fully pay attention to the supplementary role of key information in identifying image features, and effectively improves the representation efficiency of feature information;

[0014] S8. Enhance the visual features of the spatial reconstruction module by introducing a channel reconstruction module. Due to the channel redundancy of network features when CNN processes image features, it is difficult to effectively obtain the key information of the channel. The channel attention reconstruction module realizes comprehensive processing of key information of each channel, ignores unnecessary information, and improves network performance. The channel reconstruction module effectively improves the representation ability of image features;

[0015] S9. Perform feature fusion on base class features and reconstructed unit network feature enhancement;

[0016] S10. Use the interactive attention mechanism to achieve interactive feature fusion of text features and image features, and enhance the bimodal feature fusion representation capability;

[0017] S11. Finally, the sentiment polarity of the feature is obtained through Softmax;

[0018] Beneficial effects: Compared with the traditional image-text multimodal sentiment analysis algorithm model, the algorithm model of the present invention effectively and fully focuses on the visual feature space and channel feature space, and uses an interactive attention mechanism network to fully fuse the features of different modal information. Therefore, the model of the present invention is superior to the current mainstream model. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 is a flow chart of the method steps of the present invention;

[0020] Figure 2 It is a structural diagram of the channel reconstruction module of the present invention;

[0021] Figure 3 This is a structural diagram of the space reconstruction module of the present invention;

[0022] Figure 4 Interactive attention feature fusion structure diagram DETAILED DESCRIPTION

[0023] In order to more clearly describe the technical solution in the example of the present invention, the present invention will be described in detail below in conjunction with the drawings in the embodiment. The specific embodiment described in the present invention, as well as other embodiments based on the present invention, all belong to the protection scope of the present invention.

[0024] The purpose of this invention is to provide a method for sentiment analysis of images and texts based on reconstructed dual attention networks. This method mainly includes 7 modules, namely, text input layer, BERT feature extraction layer, bidirectional recurrent gating feature extraction layer, image input layer, channel reconstruction module layer, space reconstruction module layer, and interactive attention feature fusion module.

[0025] like Figure 1 Shown is a step flow chart of a method for image-text sentiment analysis based on reconstructing a dual attention network according to an example of the present invention.

[0026] An embodiment of the present invention provides a method for analyzing image-text sentiment based on reconstructing a dual attention network, comprising the following steps:

[0027] Step S1. In text feature extraction, the input is the original text data h;

[0028] Step S2. Use the BERT network to capture text semantic features. When using the BERT network model to extract text semantic features, since BERT uses a bidirectional encoder structure, it can well learn text semantic context information and pay close attention to the specific semantic characteristics of words in the text.

[0029] S21. Input the text h into the BERT pre-trained model network to obtain the text semantic feature h i ={h1,h2,…,hn}, where i∈[1,n].

[0030] S22. Use BERT network to analyze text semantic features h i Train to get b i =BERT(h i ).

[0031] Step S3. Use a bidirectional recurrent gating network to enhance the text features extracted by the BERT network model. Bidirectional GRU has a good effect on learning between low-level words. It can effectively solve the problem that the BERT network does not pay enough attention to words.

[0032] S31. b i Input into the forward recurrent gating network to obtain the forward GRU

[0033] S32. b i Input into the backward recurrent gating network to obtain the backward GRU

[0034] S33. Concatenate the forward and backward networks extracted by the bidirectional gating network to obtain the final feature vector This feature vector can well reflect the text context information and the feature information between text words.

[0035] Step S4. Finally, for each text feature, a multi-level and multi-depth text semantic feature R = {r1, r2, ..., r n}; R is the final state of text feature extraction.

[0036] Step S5. In image feature extraction, the input is image data; the image data feature representation is enhanced through 1×1 Conv.

[0037] Step S6. Use ResNet50 to extract features from the original image data. The ResNet50 network can process images of different sizes, illuminations, and angles through its deep structure and the use of residual learning, but the rich feature representation learned by the network enables ResNet50 to be robust to these changes to a certain extent.

[0038] S61. In ResNet50, the enhancement of input features is mainly achieved through residual blocks. This process can be expressed by a mathematical formula: Assuming the input feature is x, x is enhanced through a 1×1 Conv operation, and then the output after a residual block F(x) can be expressed as: X=F(x)+x, where F(x) is the convolution operation and nonlinear transformation part of the residual block, which usually includes several layers of convolution, batch normalization and ReLU activation function. The core idea of ​​ResNet50 is to add the input features directly to the feature map after convolution processing through jump connections, so that the gradient flow can still be maintained when the network is deeper, solving the gradient disappearance problem in deep networks. X=ReLU(BatchNorm(Conv(x)+X))

[0039] Step S7: Enhance the visual features extracted by the base class model by introducing a spatial reconstruction module.

[0040] S71. First, perform GN group normalization on the input feature X and perform channel grouping on the input feature X out , calculate the generated feature weight W in the feature map after Sigmoid and GN γ ={W1,W2,…,W n}, the calculation process is as follows:

[0041] X out =GN(X)

[0042]

[0043] W = Sigmoid(W γ )

[0044] S72. Use feature weights to perform feature separation operations on input features and generate feature subsets and Finally, the feature subsets are reorganized and transformed to generate the spatial refinement feature map X ω , the calculation process is as follows:

[0045]

[0046] F c =X ω1 ⊕X ω2

[0047] Step S8. Enhance the visual features of the space reconstruction module by introducing a channel reconstruction module.

[0048] S81. Reconstruct the space unit feature X ωThe features are divided into αC channel features and (1-α)C channel features, and the features are compressed by 1×1 convolution kernel to obtain X up and X low The features Y1 and Y2 are further transformed by using the global convolutional network (GWC) and the point convolutional network (PWC). The transformed features are pooled and softmaxed to obtain the feature weight vectors β1 and β2, and finally the channel refined feature F is obtained. s .

[0049] S8. Enhance the visual features of the spatial reconstruction module by introducing a channel reconstruction module.

[0050] Step S9: Perform feature fusion on the base class features and the reconstructed unit network feature enhancement. f =F c +F s ;

[0051] Step S10. Use the interactive attention mechanism to realize interactive feature fusion of text features and image features.

[0052] S101. In modal fusion, when text features are interactively fused to image features, image features are used as Queries and text features are used as Keys. Therefore, the interactive attention feature is expressed as

[0053] S102. In the interactive fusion of image features to text features, text features are used as Queries and image features are used as Keys. Therefore, the interactive attention feature is expressed as

[0054] Step S11. Finally, the sentiment polarity of the feature is obtained through Softmax;

[0055] Tables 1 and 2 list the comparison results of the present invention (IRDA) and other methods on the MVSA-Single and MVSA-Multi datasets. It can be seen that the present invention has higher scores than other methods in the text summarization model evaluation index.

[0056] Table 1 Comparative experimental results

[0057]

[0058] Table 2 Comparative experimental results

[0059]

Claims

1. A method for image-text sentiment analysis based on reconstructed dual attention network, comprising the following steps: (1) In text feature extraction, the input is raw text data; (2) Use the BERT network to capture text semantic features. When using the BERT network model to extract word features from text, it extracts key information about the word's position in the text and obtains the importance of the word to the entire text, efficiently expressing the importance of the word to the text; (3) Use a bidirectional recurrent gating network to enhance the text features extracted by the BERT network model. Since the BERT network model focuses too much on the feature values ​​between words, the bidirectional recurrent gating network has a good effect on learning low-level semantics. The combination of BERT and the bidirectional recurrent gating network can effectively realize multi-level and multi-depth text semantic information learning; (4) Ultimately, it captures multi-level and multi-depth text semantic features; (5) In image feature extraction, the input is image data; (6) Use ResNet50 to extract features from the original image data. The ResNet50 network can process images of different sizes, illuminations, and angles. By introducing residual blocks and deepening the network depth, the problem of vanishing gradients of visual features can be effectively reduced, making ResNet50 important in feature extraction; (7) The spatial reconstruction module is introduced to enhance the visual features extracted by the base class model. The spatial reconstruction module effectively solves the problem that CNN fails to fully pay attention to the supplementary role of key information in identifying image features, and effectively improves the representation efficiency of feature information; (8) Enhance the visual features of the spatial reconstruction module by introducing a channel reconstruction module. Due to the channel redundancy of network features that occurs when CNN processes image features, it is difficult to effectively obtain the key information of the channel. The channel attention reconstruction module realizes comprehensive processing of key information of each channel, ignores unnecessary information, and improves network performance. The channel reconstruction module effectively improves the representation ability of image features; (9) Feature fusion of base class features and reconstructed unit network feature enhancement; (10) Use the interactive attention mechanism to achieve interactive feature fusion of text features and image features, and enhance the bimodal feature fusion representation capability; (11) The sentiment polarity of the feature is finally obtained through Softmax.

Images