Multimodal sentiment analysis method and system based on image-text fusion

Abstract

The application discloses a kind of multi-modal sentiment analysis method and system based on image-text fusion, comprising: introducing STN and carrying out image feature extraction, obtains image feature vector;Using Bi-LSTM model to sequence modeling to text vector, capture the semantic information and context information of word in sentence, and the word information of different positions is combined, and the text features with more expression ability and distinguish degree are extracted;Determine the image key feature corresponding to the text keyword, and the text keyword corresponding to the image key feature, modal fusion is carried out based on image feature representation vector and text feature representation vector.The application adds spatial transformation network model STN and Bi-LSTM model in image and text feature extraction respectively, accurately extracts the features of multi-modal data.Meanwhile, the application combines text representation vector to position the key feature mapping in image, utilizes the interaction information between multi-modal data, provides better robustness and generalization ability.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of artificial intelligence, and relates to a multi-modal sentiment analysis method and system based on image-text fusion. BACKGROUND

[0002] Multi-modal image-text sentiment analysis technology is a sentiment analysis technology that combines multiple information such as text and image, which can more accurately analyze and understand people's emotions and attitudes. With the diversified development of the network, people no longer satisfy with only using text to publish their status on social media, but prefer to express their emotions in the way of combining text and image. Single modal sentiment analysis has many deficiencies, and its information is easily disturbed by other factors, thereby causing unsatisfactory sentiment analysis effect. Multi-modal sentiment analysis models based on neural networks are proposed and rapidly developed. Some previous researches mainly focus on extracting features from images and text, and then directly combining them to train a multi-modal sentiment classifier. This method ignores the characteristics of mutual strengthening and complementation between visual information and text information, and lacks a fine-grained framework to process the interaction of multi-modal content. In order to solve this problem, researchers propose a co-memory network for multi-modal sentiment analysis to iteratively model visual content and text bag-of-words model. However, in practical application, the size and content of the image are diverse, and the use of fixed-size image input will cause information loss and decrease in model robustness. In addition, the use of simple bag-of-words model cannot capture the context relationship of words in the text, which limits the performance of the model in processing long text. SUMMARY

[0003] The purpose of the application is to solve the problem that the use of fixed-size image input in the prior art will cause information loss and decrease in model robustness. At the same time, the use of simple bag-of-words model cannot capture the context relationship of words in the text, which limits the performance of the model in processing long text. The application provides a multi-modal sentiment analysis method and system based on image-text fusion.

[0004] To achieve the above purpose, the application adopts the following technical solutions:

[0005] The multi-modal sentiment analysis method based on image-text fusion comprises:

[0006] Based on the training sample of multi-modal sentiment analysis, the text data and image data of the training sample are obtained;

[0007] The image data is input into the spatial transformation network STN for spatial transformation network processing and feature extraction to obtain the image feature vector f image ;

[0008] Each word in the text data is represented as a text vector, and a bidirectional long short-term memory network (Bi-LSTM) model is used to sequence model the text vector to obtain the context information of the word, and the hidden state vector of the word is obtained and

[0009] The obtained word hidden state vector is spliced and weighted to obtain the final text feature vector h.

[0010] The text feature vector h is connected with the image feature vector f image to obtain the text representation vector and the image representation vector.

[0011] In the kth iteration, the previous text representation vector is connected with the image feature mapping vector f image , and the previous image representation vector is connected with the word embedding vector x t as the input of the attention layer, respectively, to obtain the image feature representation vector and the text feature representation vector.

[0012] Based on the image feature representation vector and the text feature representation vector, modal fusion is performed, and sentiment classification and correlation analysis of image-text sentiment tendency are performed.

[0013] Further improvements of the application are as follows:

[0014] Further, the image data is input into the spatial transformation network STN for spatial transformation network processing, specifically: the STN network includes a positioning network, a grid generator and a sampler; the positioning network is a convolutional neural network, and the input of the positioning network is an original image, and the output of the positioning network is a spatial transformation parameter θ; the grid generator receives the parameter θ from the positioning network, and converts the parameter θ into a corresponding affine transformation matrix, and then applies the matrix to the grid of the input data to generate a new grid, and each point on the new grid corresponds to a position on the input data; the sampler samples the input image according to the generated sampling grid, and generates new pixel values on the new coordinate positions.

[0015] Further, the image feature vector f image is obtained, specifically:

[0016] Based on ResNet50, the sampled image y i is subjected to feature extraction to obtain the feature vector of the image; the feature vector f image is extracted from the last convolutional layer of the convolutional neural network, which represents the features of the image, as shown in formula (1):

[0017] f image =CNN(yi (1)

[0018] Wherein, CNN stands for Convolutional Neural Network, f image This represents the feature vector of the image.

[0019] Furthermore, in obtaining the image feature vector f image Following this, it also includes: transferring the image feature vector f image The input is fed into a fully connected layer, where each feature map is flattened to obtain a feature vector corresponding to the image portion. Then, a CBAM attention model is used to weight each feature vector to obtain the final image feature vector. As shown in formula (2):

[0020]

[0021] Where, m i It is the feature vector corresponding to the i-th visual feature map, α i l represents the corresponding attention weight.

[0022] Furthermore, each word in the text data is represented as a text vector, and the text vectors are sequence-modeled using a Bi-LSTM network to obtain the contextual information of each word, thus obtaining the hidden state vector of that word. and Specifically:

[0023] The text sequence of text data is represented as a sequence of word vectors (x1, x2, ..., x...). T ), where x i It is the vector representation of the i-th word; for each input word x i The Bi-LSTM model traverses the path in both the forward and backward directions, using one LSTM model in each direction to obtain the hidden state vector of the word. and As shown in formulas (3) and (4):

[0024]

[0025]

[0026] in, This represents the feedforward LSTM model. This represents the backward LSTM model.

[0027] Furthermore, the obtained word hidden state vectors are concatenated and weighted to obtain the final text feature vector h. Specifically, the two obtained hidden state vectors are... and Concatenate along time steps to get a hidden state vector h i As shown in equation (5):

[0028]

[0029] Map the hidden state vector h i of each word to an attention score vector e i using a fully connected layer, as shown in equation (6):

[0030] e i = tanh(w h h i +b h ) (6)

[0031] where w h and b h are the weights and biases of the fully connected layer;

[0032] Normalize the attention score vector e i using a softmax function to get the attention weight vector a = [a1; a2;...; a T ] for each word, where a i represents the attention weight of the i-th word, as shown in equation (7):

[0033]

[0034] where w is the parameter vector in the attention mechanism;

[0035] Weight each word vector h i with its corresponding attention weight a i to get the final text feature vector h, as shown in equation (8):

[0036]

[0037] Further, concatenate the text feature vector h with the image feature vector f image to obtain the text representation vector and the image representation vector, specifically:

[0038] Concatenate the text feature vector h with the image feature vector f image to obtain the visual memory [f image , h], and use the visual memory [f image , h] as the input of a single-layer perceptron to calculate the text-guided visual hidden representation h 2i , as shown in equation (9):

[0039] h 2i= tanh(w 2im [f image ,h]+b 2im ) (9)

[0040] where w 2im and b 2im represent the weights and bias of the single-layer perceptron;

[0041] The new normalized visual attention weights a 2i are output based on the softmax function, as shown in equation (10):

[0042]

[0043] Based on the new attention weights, the subset of all image feature vectors is weighted, thereby updating the image representation vector v 2 img , as shown in equation (11):

[0044]

[0045] where m i is the feature vector corresponding to the i-th visual feature map;

[0046] The acquisition of the text representation vector is: connecting the image representation vector v with each word embedding vector x t , and then inputting each block of new text memory through a single-layer perceptron to obtain the image-guided text hidden representation h 2t , as shown in equation (12):

[0047] h 2t = tanh(w 2t [x t ,v l img ]+b 2t ) (12)

[0048] where w 2t and b 2t represent the weight matrix and bias vector of the second layer, respectively;

[0049] The new text attention weights a are normalized using the softmax function, as shown in equation (13):

[0050]

[0051] The weighted average of the word embedding vectors is calculated using the new attention weights, thereby updating the text representation vector v t 2 ext, as shown in equation (14):

[0052]

[0053] where x t is the word vector representation of the tth word.

[0054] Further, in the kth iteration, the previous text representation vector and the image feature mapping vector f image , and the previous image representation vector and the word embedding vector x t are respectively input into the attention layer to obtain the image feature representation vector and the text feature representation vector, as shown in equations (15) and (16):

[0055]

[0056]

[0057] Further, modal fusion is performed based on the image feature representation vector and the text feature representation vector, and sentiment classification and correlation analysis of image-text sentiment orientation are performed, specifically:

[0058] The image feature representation vector and the text feature representation vector are input into a softmax layer to obtain a final result y, which is used for sentiment analysis, as shown in equation (17):

[0059]

[0060] where w s and b s represent the weight matrix and the bias vector, respectively.

[0061] The multi-modal sentiment analysis system based on image-text fusion includes:

[0062] An acquisition module obtains text data and image data of a training sample based on a training sample of multi-modal sentiment analysis.

[0063] A spatial transformation network processing module inputs image data into a spatial transformation network STN for spatial transformation network processing and feature extraction to obtain an image feature vector f image .

[0064] A hidden state vector acquisition module represents each word in the text data as a text vector, and performs sequence modeling on the text vector through a bidirectional long short-term memory network Bi-LSTM model to obtain context information of the word, and obtains a hidden state vector and

[0065] a weighting module, which splices and weights the obtained word hidden state vector, to obtain a final text feature vector h;

[0066] a connecting module, which connects the text feature vector h and the image feature vector f image to obtain a text representation vector and an image representation vector;

[0067] a feature representation vector acquisition module, which, in the kth iteration, takes the previous text representation vector and the image feature mapping vector f image , and the previous image representation vector and the word embedding vector x t as inputs of an attention layer respectively, to obtain an image feature representation vector and a text feature representation vector;

[0068] a modal fusion module, which performs modal fusion based on the image feature representation vector and the text feature representation vector, and performs sentiment classification and correlation analysis of image-text sentiment orientation.

[0069] Compared with the prior art, the present application has the following beneficial effects:

[0070] The present application respectively adds a spatial transformation network model STN and a Bi-LSTM model in image and text feature extraction, which can more accurately extract features of multi-modal data. Meanwhile, the present application adopts an image representation vector to query keywords in the text in the aspect of image-text fusion, and combines a text representation vector to position key feature mapping in the image, which utilizes interactive information between multi-modal data, improves the sentiment feature expression ability of multi-modal data, and provides better robustness and generalization ability. BRIEF DESCRIPTION OF DRAWINGS

[0071] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments. It should be understood that the following drawings only show some embodiments of the present application, and therefore should not be regarded as a limitation on the scope, and for those skilled in the art, other related drawings can also be obtained without creative labor on the basis of these drawings.

[0072] Figure 1 It is a flowchart of the multi-modal sentiment analysis method based on image-text fusion of the present application;

[0073] Figure 2 It is a structural diagram of the multi-modal sentiment analysis system based on image-text fusion of the present application;

[0074] Figure 3 Another flowchart of the multi-modal sentiment analysis method based on the fusion of text and images of the present application;

[0075] Figure 4 A process diagram of the image feature extraction of the present application. DETAILED DESCRIPTION

[0076] In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. The components of the embodiments of the present application described and shown in the drawings can be arranged and designed in various different configurations.

[0077] Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present application without creative labor are within the scope of protection of the present application.

[0078] It should be noted that: similar reference numerals and letters represent similar items in the following drawings, and therefore, once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings.

[0079] In the description of the embodiments of the present application, it should be noted that if the terms "upper", "lower", "horizontal", "inner" and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, or the orientation or positional relationship in which the product of the present application is usually placed, which is only for the convenience of describing the present application and simplifying the description, and does not indicate or imply that the indicated device or element must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation on the present application. In addition, the terms "first", "second" and the like are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0080] In addition, if the term "horizontal" appears, it does not mean that the component must be absolutely horizontal, but can be slightly inclined. For example, "horizontal" only means that its direction is relatively more horizontal than "vertical", and does not mean that the structure must be completely horizontal, but can be slightly inclined.

[0081] In the description of the embodiments of the present application, it also needs to be explained that, unless otherwise explicitly specified and limited, if the terms "arrange", "install", "connect", "connect" appear, they should be understood in a broad sense, for example, it can be fixedly connected, or it can be detachably connected, or integrally connected, it can be mechanically connected, or it can be electrically connected, it can be directly connected, or indirectly connected through an intermediate medium, it can be the communication inside two elements. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0082] The present application will be further described in detail below in combination with the drawings:

[0083] Referring to Figure 1 , the present application discloses a kind of based on image-text fusion multi-modal sentiment analysis method, comprising:

[0084] S101, based on multi-modal sentiment analysis training sample, obtains the text data and image data of training sample.

[0085] S102, image data is input into space transformation network STN and is processed in space transformation network, and feature extraction is carried out, and image feature vector f image .

[0086] STN network includes positioning network, grid generator and sampler;The positioning network is convolutional neural network, and the input is original image, and the output is the parameter θ of space transformation;The grid generator receives the parameter θ from the positioning network, and converts the parameter θ into corresponding affine transformation matrix, then the matrix is applied to the grid of input data, generates a new grid, every point on the new grid corresponds to a position on the input data;The sampler samples input image according to the generated sampling grid, and generates new pixel values on new coordinate positions.

[0087] image feature vector f image is obtained, specifically:

[0088] ResNet50 is based on image y i After sampling, feature extraction is carried out, and the feature vector of image is obtained;Feature vector f image is extracted from the last convolutional layer of convolutional neural network, which represents the features of image, as shown in formula (1):

[0089] f image =CNN(y i ) (1)

[0090] Wherein, CNN is convolutional neural network, and f image It is the feature vector of image.

[0091] After obtaining the image feature vector f image Then, the image feature vector f image is input into a fully connected layer, each feature map is flattened to obtain a feature vector corresponding to the image part, and a CBAM attention model is used to weight each feature vector to obtain a final image feature vector As shown in formula (2):

[0092]

[0093] Wherein, m i is the feature vector corresponding to the i th visual feature map, is the corresponding attention weight.

[0094] S103, each word in the text data is represented as a text vector, and the text vector is sequentially modeled by a bidirectional long short-term memory network Bi-LSTM model to obtain the context information of the word, and the hidden state vectors of the word are obtained respectively and

[0095] The text sequence of the text data is represented as a sequence composed of word vectors (x1, x2,..., x T ), wherein x i is the vector representation of the i th word; for each input word x i , the Bi-LSTM model traverses from the front and back directions respectively, and uses an LSTM model in each direction to obtain the hidden state vectors of the word respectively and As shown in formula (3) and formula (4):

[0096]

[0097]

[0098] Wherein, represents the forward LSTM model, represents the backward LSTM model.

[0099] S104, the obtained word hidden state vectors are spliced and weighted to obtain a final text feature vector h.

[0100] The obtained two hidden state vectors and are spliced according to the time step to obtain a hidden state vector h i , as shown in formula (5):

[0101]

[0102] The hidden state vector h of each word is generated using a fully connected layer. i Mapped to attention score vector e i As shown in formula (6):

[0103] e i =tanh(w h h i +b h (6)

[0104] Among them, w h and b h These are the weights and biases of the fully connected layer;

[0105] The attention score vector e is evaluated using the softmax function. i Normalization is performed to obtain the attention weight vector α = [α1; α2; ...; α] for each word. T ], where α i The attention weight of the i-th word is represented by formula (7):

[0106]

[0107] Where w is the parameter vector in the attention mechanism;

[0108] Each word vector h i Its corresponding attention weight α i After weighting, the final text feature vector h is obtained, as shown in formula (8):

[0109]

[0110] S105, combine the text feature vector h with the image feature vector f image Perform a concatenation operation to obtain the text representation vector and the image representation vector.

[0111] Compare the text feature vector h with the image feature vector f image Make connections to obtain visual memory [f image [,h], and visual memory [f image The text-guided visual hidden representation h is computed using [h] as input to a single-layer perceptron. 2i As shown in formula (9):

[0112] h 2i =tanh(w 2im [f image ,h]+b 2im (9)

[0113] Among them, w2im and b 2im represent weights and bias of the single-layer perceptron;

[0114] output new normalized visual attention weights a based on the softmax function 2i as shown in equation (10):

[0115]

[0116] update the image representation vector v by weighting all subsets of image feature vectors based on the new attention weights 2 img as shown in equation (11):

[0117]

[0118] where m i is the feature vector corresponding to the i-th visual feature map;

[0119] The acquisition of the text representation vector is: concatenate the image representation vector with each word embedding vector x t , and then input each block of new text memory through a single-layer perceptron to obtain the image-guided text hidden representation h 2t as shown in equation (12):

[0120]

[0121] where w 2t and b 2t represent the weight matrix and bias vector of the second layer, respectively;

[0122] normalize the new text attention weights using the softmax function, as shown in equation (13):

[0123]

[0124] update the text representation vector by calculating the weighted average of the word embedding vectors using the new attention weights as shown in equation (14):

[0125]

[0126] where x t is the word vector representation of the t-th word.

[0127] S106, in the k-th iteration, concatenate the previous text representation vector with the image feature map vector f imageand the previous image representation vector with the word embedding vector x t respectively as the input of the attention layer, the image feature representation vector and the text feature representation vector are obtained; as shown in formula (15) and formula (16):

[0128]

[0129]

[0130] S107, modal fusion is carried out based on the image feature representation vector and the text feature representation vector, and sentiment classification and correlation analysis of image-text sentiment tendency are carried out.

[0131] The image feature representation vector and the text feature representation vector are input into a softmax layer to obtain a final result y, which is used for sentiment analysis, as shown in formula (17):

[0132]

[0133] wherein, w s and b s respectively represent a weight matrix and a bias vector.

[0134] Referring to Figure 2 , the application discloses a multi-modal sentiment analysis system based on image-text fusion, comprising:

[0135] An acquisition module obtains text data and image data of a training sample based on a training sample of multi-modal sentiment analysis;

[0136] A spatial transformation network processing module inputs the image data into a spatial transformation network STN for spatial transformation network processing and feature extraction to obtain an image feature vector f image ;

[0137] A hidden state vector acquisition module represents each word in the text data as a text vector, and acquires context information of the word by modeling the text vector through a bidirectional long short-term memory network Bi-LSTM model to obtain a hidden state vector h and

[0138] A weighting module splices and weights the acquired hidden state vectors of the words to obtain a final text feature vector h;

[0139] A connection module connects the text feature vector h with the image feature vector f imageThe connection is performed to obtain a text representation vector and an image representation vector;

[0140] The feature representation vector acquisition module, in the kth iteration, takes the previous text representation vector and the image feature mapping vector f image , and the previous image representation vector and the word embedding vector x t as inputs of the attention layer respectively to obtain an image feature representation vector and a text feature representation vector;

[0141] The modal fusion module performs modal fusion based on the image feature representation vector and the text feature representation vector, and performs sentiment classification and correlation analysis of the image-text sentiment tendency.

[0142] Embodiment:

[0143] Referring to Figure 3 , the implementation steps of the present application are as follows:

[0144] Step 1: Use the spatial transformation network model STN, the spatial attention mechanism and the channel attention mechanism to extract more discriminative image features, enhance effective features and suppress ineffective features or noise.

[0145] Step 2: Use the Bi-LSTM model and the attention mechanism to capture the semantic information and the context information of the words in the sentence, and combine the word information at different positions to extract more expressive and discriminative text features.

[0146] Step 3: Use two memory networks to process text and image data respectively to determine the image key features corresponding to the text key words and the text key words corresponding to the image key features, better utilize the interaction information between multi-modal data, and improve the sentiment feature expression ability of multi-modal data.

[0147] Referring to Figure 4 , the implementation content of step 1 is as follows:

[0148] 1.1) Input the original image x into the positioning network, and output the parameters θ of spatial transformation, as shown in formula (18):

[0149] θ=f loc (x) (18)

[0150] 1.2) The grid generator receives the vector θ from the positioning network and converts it into a corresponding affine transformation matrix, and then applies the matrix to the grid of the input data to generate a new grid. Each point on the new grid corresponds to a position on the input data, and the input data can be subjected to translation, rotation, scaling and other transformation operations, as shown in formula (19):

[0151] p' i = grid(p i ; θ) (19)

[0152] where p i represents the coordinates of the sampling points generated by the grid generator, p' i represents the coordinates of the sampling points mapped on the input image;

[0153] 1.3) The sampler samples the input image according to the generated sampling grid and generates new pixel values at the new coordinate positions. Usually, methods such as bilinear interpolation or nearest neighbor interpolation are used to calculate the new pixel values, as shown in equation (20):

[0154] y i = sampler(x, p' i ) (20)

[0155] where the input image x and the coordinates of the sampling grid p' i are input, and the sampled image y i is output;

[0156] 1.4) For the image processed by the spatial transformation network, the ResNet50 is used to extract the features of the sampled image y i , and the feature vector of the image is obtained. The feature vector f image is extracted from the last convolutional layer of the convolutional neural network to represent the features of the image, as shown in equation (21):

[0157] f image = CNN(y i ) (21)

[0158] where CNN is the convolutional neural network, and f image is the feature vector of the image;

[0159] 1.5) The extracted image feature vector is input into the fully connected layer, and each feature map is flattened to obtain a feature vector corresponding to the image part. The CBAM (Convolutional Block Attention Module) attention model is used to weight each feature vector to obtain the final image feature vector , as shown in equation (22):

[0160]

[0161] where m i is the feature vector corresponding to the i-th visual feature map, and w is the corresponding attention weight.

[0162] Step 2 involves the following steps:

[0163] 2.1) Represent the text sequence as a sequence of word vectors (x1, x2, ..., x...). T ), where x i It is the vector representation of the i-th word. For each input word x... i The Bi-LSTM model traverses the path in both the forward and backward directions, using one LSTM model in each direction to obtain the hidden state vector of the word. and As shown in formulas (23) and (24):

[0164]

[0165]

[0166] in, This represents the feedforward LSTM model. This represents a backward LSTM model;

[0167] 2.2) Obtain the two hidden state vectors and By concatenating the data according to time steps, a hidden state vector h is obtained. i As shown in formula (25):

[0168]

[0169] 2.3) Use a fully connected layer to store the hidden state vector h of each word. i Mapped to attention score vector e i As shown in formula (26):

[0170] e i =tanh(w h h i +b h (26)

[0171] Among them, w h and b h These are the weights and biases of the fully connected layer;

[0172] 2.4) Use the softmax function to apply the attention score vector e i Normalization is performed to obtain the attention weight vector α = [α1; α2; ...; α] for each word. T ], where α i The attention weight for the i-th word is represented by formula (27):

[0173]

[0174] where w is the parameter vector in the attention mechanism;

[0175] 2.5) Each word vector h i is weighted by its corresponding attention weight a i to obtain the final text feature vector h, as shown in equation (28):

[0176]

[0177] The implementation content of step 3 is:

[0178] 3.1) Connect the text feature vector h with the image feature vector f image , and then use the new visual memory [f image , h] as the input of the single-layer perceptron to calculate the text-guided visual hidden representation h 2i , as shown in equation (29):

[0179] h 2i = tanh(w 2im [f image , h] + b 2im ) (29)

[0180] where w 2im and b 2im represent the weights and bias of the single-layer perceptron;

[0181] 3.2) Use the softmax function to output a new normalized visual attention weight a 2i , as shown in equation (30):

[0182]

[0183] 3.3) According to the new attention weight, weight all subsets of image feature vectors, so as to update the image representation vector v 2 img , as shown in equation (31):

[0184]

[0185] where m i is the feature vector corresponding to the i-th visual feature mapping;

[0186] 3.4) Similarly, the text representation vector h

[0187] 3.4a) Connect the image representation vector v with each word embedding vector xt Connect, then input each new piece of text memory through a single-layer perceptron Get image-guided text hidden representation h 2t As shown in equation (32):

[0188] h 2t = tanh(w 2t [x t ,v l img ]+b 2t ) (32)

[0189] Where w 2t and b 2t represent the weight matrix and bias vector of the second layer, respectively;

[0190] 3.4b) Normalize the new text attention weights β t 2 using the softmax function, as shown in equation (33):

[0191]

[0192] 3.4c) Calculate the weighted average of word embedding vectors using the new attention weights to update the text representation vector As shown in equation (34):

[0193]

[0194] Where x t is the word vector representation of the t-th word;

[0195] 3.5) Use the stacked memory network to explore the subtle relationship between images and texts by iteratively querying the original memory matrix of images and texts, and finally complete sentiment classification. The specific implementation steps are as follows:

[0196] 3.5a) In the k-th iteration, input the previous text representation vector and the image feature mapping vector f image , and the previous image representation vector and the word embedding vector x t into the attention layer respectively, to get the image features and text features, as shown in equations (35) and (36):

[0197]

[0198]

[0199] 3.5b) Combine the final feature representation vectors of images and texts into the softmax layer to get the final result y for sentiment analysis, as shown in equation (37):

[0200]

[0201] where w s and b s represent the weight matrix and bias vector, respectively.

[0202] I. Experimental conditions

[0203] 1) Experimental environment

[0204] The environment of this experiment is Ubuntu 16.04 system, GPU is NVIDIA TITAN XP GPUs, CPU is Intel(R) Xeon(R) CPU E5-2620v4@2.10GHz, CUDA version is 10.0, and the experiment is carried out on PyCharm2019.2, among which Pytorch version is 1.0.0-GPU version and Python version is 3.6.

[0205] 2) Experimental data

[0206] This experiment uses two public multi-modal sentiment data sets: MVSA-Single and MVSA-Multi. MVSA-Single data: This data set includes 13,000 samples, each of which is a movie review. Each review is labeled as one of the positive, negative or neutral sentiment tendencies. Each sample in the data set contains features of two modalities: text and image. Among them, the text feature is the bag-of-words representation of the review text, and the image feature is the visual feature extracted from the movie poster. The data set provides vector representations of these features and corresponding sentiment labels.

[0207] MVSA-Multi data set: This data set includes 13,000 samples, but is more complex than the MVSA-Single data set. Each sample is still a movie review, but with three additional sentiment tendencies: anger, disgust and happiness, for a total of six sentiment labels. In addition to the text and image features contained in the MVSA-Single data set, the MVSA-Multi data set also includes action features, which are action trajectory features extracted from movie clips. Each sample in the data set contains features of these five modalities and corresponding sentiment labels.

[0208] 3) Evaluation index

[0209] (1) Accuracy: represents the proportion of all correctly predicted samples to the total samples, which is defined as:

[0210]

[0211] (2) F1 value: considering both the recall rate and the precision rate, and making both reach the highest, taking a balance, which is defined as:

[0212]

[0213] Wherein, the formula symbol explanation is shown in Table 1:

[0214] Table 1 Formula symbol explanation

[0215]

[0216] 4) Experimental setup

[0217] Each word is mapped to a 100-dimensional vector space, and this vector is called the embedding vector of the word. Each sentence is filled to 50 dimensions, that is, for the sentence less than 50 in length, 0 is used for padding, and for the sentence more than 50 in length, it is truncated. For the image part, InceptionV3 network is used to extract 2048 feature maps with a size of 55. During model training, Adam optimizer is used, and the learning rate is set to 0.0005. At the same time, the size of each batch is set to 16, and dropout technique and early-stopping technique are used to avoid overfitting.

[0218] II. Experimental content

[0219] The method of the present application and the existing method are compared with the above experimental data, and the performance parameter comparison results are shown in Table 2.

[0220] Table 2 Performance parameter comparison results of the method of the present application and the existing method

[0221]

[0222]

[0223] From Table 2, compared with the existing method, the present application achieves better results, which is due to the addition of the spatial transformation STN network and the Bi-LSTM model in the image and text feature extraction, which can more accurately extract the features of multi-modal data. At the same time, in the fusion aspect, the image representation vector is used to query the key words in the text, and the key feature mapping in the image is positioned by combining the text representation vector, which better utilizes the interaction information between multi-modal data.

[0224] The above merely describes the preferred embodiments of the present application, and is not used to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A multi-modal sentiment analysis method based on image-text fusion, characterized in that, The method comprises the following steps: Based on the training sample of multi-modal sentiment analysis, the text data and image data of the training sample are obtained; The image data is input into a spatial transformation network (STN) for spatial transformation network processing and feature extraction to obtain an image feature vector f image ; Each word in the text data is represented as a text vector, and the text vector is sequentially modeled by a bidirectional long short-term memory network Bi-LSTM model to obtain the context information of the word, and hidden state vectors of the word are obtained and The obtained word hidden state vector is spliced and weighted to obtain the final text feature vector h; concatenate the text feature vector h and the image feature vector f image concatenation, to obtain a text representation vector and an image representation vector; In the kth iteration, the previous text representation vector is concatenated with the image feature mapping vector f image and the previous image representation vector is concatenated with the word embedding vector x t to obtain the image feature representation vector and the text feature representation vector, respectively. Based on the image feature representation vector and the text feature representation vector, modal fusion is performed, and sentiment classification and correlation analysis of image-text sentiment tendency are performed.

2. The method of claim 1, wherein the method is based on a fusion of text and image. The image data is input into the spatial transformation network STN for spatial transformation network processing, specifically: the STN network includes a positioning network, a grid generator and a sampler; the positioning network is a convolutional neural network, the input of which is an original image, and the output is a spatial transformation parameter θ; the grid generator receives the parameter θ from the positioning network and converts the parameter θ into a corresponding affine transformation matrix, then applies the matrix to the grid of the input data to generate a new grid, and each point on the new grid corresponds to a position on the input data; the sampler samples the input image according to the generated sampling grid and generates new pixel values at the new coordinate positions. 3.The method of claim 2, wherein, The image feature vector f is obtained image Specifically, Based on ResNet50, the sampled image y is processed i Feature extraction is performed to obtain a feature vector of the image; a feature vector f is extracted from a last convolutional layer of the convolutional neural network image , representing the features of the image, as shown in equation (1): f image = CNN(y i ) (1) wherein CNN is a convolutional neural network, f image is a feature vector of the image.

4. The method of claim 3, wherein the method is based on a fusion of text and image. In the acquisition image feature vector f image After that, also include: the image feature vector f image Input into the full connection layer, flatten each feature map, get a feature vector corresponding to the image part, and use the CBAM attention model to weight each feature vector to get the final image feature vector As shown in formula (2): wherein m i is the feature vector corresponding to the i-th visual feature mapping, is the corresponding attention weight. 5.The method of claim 4, wherein, The text data is represented as a text vector for each word, and a bidirectional long short-term memory network Bi-LSTM model is used for sequence modeling of the text vector to obtain the context information of the word, and hidden state vectors of the word are obtained and Specifically: The text sequence of text data is represented as a sequence of word vectors (x1, x2, ..., x...). T ), where x i It is the vector representation of the i-th word; for each input word x i The Bi-LSTM model traverses the path in both the forward and backward directions, using one LSTM model in each direction to obtain the hidden state vector of the word. and As shown in formulas (3) and (4): wherein, denotes a forward LSTM model, denotes a backward LSTM model. 6.The method of claim 5, wherein, The obtained word hidden state vector is spliced and weighted to obtain a final text feature vector h, specifically: the two obtained hidden state vectors And are spliced according to time steps to obtain a hidden state vector h i As shown in formula (5): Based on a fully connected layer, the hidden state vector h i is mapped to an attention score vector e i as shown in equation (6): e i = tanh(w h h i + b h ) (6) where w h and b h are the weights and biases of the fully connected layer; The attention score vector e is normalized using a softmax function i to obtain an attention weight vector a = [a1; a2;...; ai;...; an] for each word, where ai represents the attention weight of the ith word, as shown in equation (7): T a = softmax(e) = exp(e1) / Σ exp(ei) i where exp represents the exponential function. Wherein, w is a parameter vector in the attention mechanism; Each word vector h i The attention weight α corresponding thereto i The final text feature vector h is obtained by weighting, as shown in equation (8):

7. The method of claim 6, wherein the method is based on a fusion of text and image. The text feature vector h and the image feature vector f are combined to obtain a combined feature vector h f. image The connection is made to obtain the text representation vector and the image representation vector, specifically: The text feature vector h is concatenated with the image feature vector f image The connection is made to obtain the visual memory [f image ,h], and the visual memory [f image ,h] is calculated as the input of a single-layer perceptron to calculate the text-guided visual hidden representation h 2i As shown in equation (9): h 2i = tanh(w 2im [f image ,h] + b 2im ) (9) where w 2im and b 2im represent the weights and biases of the single-layer perceptron; output a new normalized visual attention weight a based on a softmax function 2i As shown in equation (10): weighting all subsets of image feature vectors based on the new attention weights, thereby updating the image representation vector v 2 img As shown in equation (11): wherein m i is the feature vector corresponding to the i-th visual feature mapping; The acquisition text representation vector is: connecting image representation vector with each word embedding vector x t Connect and input each block of new text memory through a single-layer perceptron Get image-guided text hidden representation h 2t As shown in equation (12): where w 2t and b 2t denote the weight matrix and bias vector of the second layer, respectively; using a softmax function normalization, as shown in equation (13): updating the text representation vector by computing a weighted average of the word embedding vectors with new attention weights As shown in equation (14): where x t is the word vector representation of the tth word. 8.The method of claim 7, wherein, The previous text representation vector and the previous image representation vector image and the previous image representation vector and the previous image representation vector t respectively as inputs of the attention layers, to obtain the image feature representation vector and the text feature representation vector, as shown in equation (15) and equation (16): 9.The method of claim 8, wherein, Based on the image feature representation vector and the text feature representation vector, modal fusion is performed, and sentiment classification and correlation analysis of image-text sentiment tendency are performed, specifically: The image feature representation vector and the text feature representation vector are input into the softmax layer to obtain the final result y, which is used for sentiment analysis, as shown in formula (17): where w s and b s denote the weight matrix and bias vector, respectively.

10. A multi-modal sentiment analysis system based on image-text fusion, characterized in that, The method comprises the following steps: An acquisition module obtains text data and image data of a training sample based on a training sample of multi-modal sentiment analysis; a spatial transformation network processing module, which inputs image data into a spatial transformation network (STN) for spatial transformation network processing and feature extraction to obtain an image feature vector f image ; The hidden state vector acquisition module represents each word in the text data as a text vector, and acquires context information of the word by performing sequence modeling on the text vector through a bidirectional long short-term memory network (Bi-LSTM) model to obtain a hidden state vector of the word and A weighting module splices and weights the obtained word hidden state vector to obtain a final text feature vector h; a connection module, which connects the text feature vector h with the image feature vector f image connection is made to obtain a text representation vector and an image representation vector; a feature representation vector obtaining module, which obtains, in the kth iteration, a previous text representation vector and an image feature mapping vector f image and a previous image representation vector and a word embedding vector x t as inputs of attention layers respectively, to obtain an image feature representation vector and a text feature representation vector A modal fusion module performs modal fusion based on the image feature representation vector and the text feature representation vector, and performs sentiment classification and correlation analysis of image-text sentiment tendency.

Images