Multi-modal sentiment analysis method for fusing text and image

Abstract

The invention relates to the technical field of sentiment analysis, and discloses a multi-modal sentiment analysis method for fusing a text and an image, which effectively avoids one-sidedness and limitation possibly brought by single-modal analysis by integrating information of two different modalities of the text and the image, and fuses the information of the two modalities to obtain a multi-modal sentiment analysis result. The complete face of the emotion of the user can be captured more accurately; according to the method, the positive and negative polarities of the emotion are judged, namely whether the emotion of the user is positive or negative is distinguished, and the emotion intensity is further quantified; by means of the quantitative processing, the emotion analysis result is more specific and measurable, and the emotional state of the user can be deeply understood; meanwhile, according to the method, the change trend of the emotion is disclosed based on a time sequence, and visual and powerful data support is provided for dynamic change of the emotion of the user.

Description

Technical Field

[0001] The present invention relates to the technical field of sentiment analysis, and in particular to a multimodal sentiment analysis method integrating text and image. Background Art

[0002] With the rapid development of the Internet and multimedia technology, social media platforms, online comment systems, news reports and other media have generated massive amounts of text and image data. These data are not only carriers of information, but also intuitive reflections of users' emotional tendencies. Traditional sentiment analysis methods are mostly limited to a single text or image modality, but in real scenes, text and images often appear together, complementing each other and jointly constructing a more complete and delicate emotional expression.

[0003] For example, in a social media post, a user may post a photo of a smiling face and write "I feel particularly good today!" Here, the smiling face in the image and the positive expression in the text complement each other and convey strong positive emotions. If only the text or image is analyzed in a single mode, some emotional information may be missed, resulting in one-sidedness of sentiment analysis. Therefore, how to efficiently integrate the information of the two modes of text and image has become the key to improving the accuracy and comprehensiveness of sentiment analysis.

[0004] In addition, sentiment analysis should not only stop at judging whether sentiment is positive or negative, that is, simply classifying sentiment as positive or negative, but should further explore the intensity and changing trend of sentiment. The analysis of sentiment trends is of immeasurable value in understanding the dynamic changes of user emotions, predicting user behavior, and optimizing user experience. For example, in corporate brand management, by monitoring the sentiment trends in user comments, companies can promptly discover and respond to potential public relations crises, or seize opportunities to enhance their brand image.

[0005] In summary, effectively integrating the information of the two modalities of text and image can not only help to achieve more accurate and comprehensive sentiment tendency analysis, but also reveal the changes and trends of sentiment intensity, providing strong data support for decision-making. Therefore, the development of multimodal sentiment analysis technology that can accurately capture and integrate text and image sentiment information has far-reaching significance for improving the depth and breadth of sentiment analysis and promoting its application in various fields.

[0006] To this end, a multimodal sentiment analysis method that integrates text and images is proposed. Summary of the invention

[0007] In view of the deficiencies in the prior art, the present invention provides a multimodal sentiment analysis method that integrates text and image to solve the background technology problems.

[0008] To achieve the above object, the present invention provides the following technical solution: a multimodal sentiment analysis method integrating text and image, comprising the following steps:

[0009] Obtain text content, image content and their time sequence, use pre-trained word embedding models to extract text features, and use pre-trained neural networks to extract image features;

[0010] The cross-head attention mechanism (CMA) is used to fuse the text and image features with sentiment labels and classify them through the fully connected layer to form image-text feature pairs;

[0011] Use deep learning models to perform sentiment analysis on image-text feature pairs to determine the positive and negative polarity of sentiment;

[0012] The obtained image-text feature pairs are screened, and the image-text feature pairs with different positive and negative polarities of the image and text annotation emotions are deleted. Among the remaining image-text pairs, if one of the emotions of the image or text is neutral, the other positive or negative label is selected as the emotional label of the image-text feature pair;

[0013] Construct a sentiment intensity and polarity model. In a two-dimensional Cartesian coordinate system, the X-axis represents polarity and the Y-axis represents intensity. The filtered image and text feature pairs are brought into the model in chronological order, the sentiment is quantified, and the sentiment trend of text and image is obtained based on the chronological order.

[0014] Preferably, the X-axis indicates the degree of emotional pleasure (negative→positive), the Y-axis indicates the intensity of emotion (calm→excited), and the polarity score ranges from -5 to 5.

[0015] Preferably, when using the deep learning model to perform sentiment analysis on image-text feature pairs, a regression-based sentiment scoring method is adopted to convert sentiment judgment into a regression problem, thereby achieving quantitative scoring of the positive and negative polarity and intensity of the sentiment to evaluate its positive and negative polarity and intensity.

[0016] Preferably, before extracting text content features, the text is also preprocessed, including removing stop words, performing stem extraction or word form restoration to improve the accuracy of text feature extraction; before extracting image content features, the image is also preprocessed using data enhancement technology, including rotating, scaling, cropping or flipping the image, to improve the generalization ability of subsequent models when processing image features.

[0017] Preferably, when performing sentiment analysis on the image-text feature pairs, a sentiment dictionary is introduced to assist in judging the positive and negative polarity and intensity of the sentiment, so as to improve the accuracy and robustness of the sentiment analysis.

[0018] Preferably, when extracting features from text, named entity recognition (NER) technology is also used to identify key entities in the text, including but not limited to name pronouns such as names of people, places, and institutions, so as to further enrich text features and improve the accuracy of sentiment analysis.

[0019] Preferably, when adopting the cross multi-head attention mechanism (CMA) to fuse text and image features, the semantic association between text and image is considered, and by calculating the similarity or correlation between text and image features, the image-text feature pairs formed by text and image features with strong semantic association are given higher weights, so as to more accurately reflect their emotional intensity and polarity during the emotional quantification processing.

[0020] Preferably, when adopting the cross multi-head attention mechanism to fuse text and image features, a temporal attention mechanism is introduced to capture the temporal dependency between text and image, and the text and image feature pairs are grouped and sorted through this mechanism.

[0021] Preferably, in the text preprocessing stage, a sentiment dictionary is used to filter out words in the text that are not related to sentiment analysis, reduce the interference of noise information on the sentiment analysis results, and improve the accuracy and efficiency of sentiment analysis.

[0022] Compared with the prior art, the present invention has the following beneficial effects:

[0023] The present invention effectively avoids the one-sidedness and limitations that may be brought about by single-modal analysis by integrating information from two different modalities, text and image, and fuses the information from these two modalities to more accurately capture the complete picture of user emotions. The method not only determines the positive and negative polarity of emotions, that is, distinguishes whether the user's emotions are positive or negative, but also further quantifies the intensity of emotions. This quantification makes the results of sentiment analysis more specific and measurable, which helps to gain a deeper understanding of the user's emotional state. At the same time, the method also reveals the changing trend of emotions based on time series, providing intuitive and powerful data support for the dynamic changes of user emotions.

[0024] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 This is a flow chart of the multimodal sentiment analysis method for integrating text and image in the present invention;

[0026] Figure 2 It is a schematic diagram of the emotion trend of the present invention on a two-dimensional Cartesian coordinate system. Detailed implementation manners

[0027] Next, in combination with the accompanying drawings in the embodiments of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0028] Please refer to Figure 1 and Figure 2 , the present invention provides a multi-modal sentiment analysis method that fuses text and images, including the following steps:

[0029] I. Data preprocessing

[0030] 1. Text preprocessing:

[0031] Remove stop words: Use a stop word list to remove meaningless words in the text, such as "de", "le", etc.

[0032] Stemming / Lemmatization: For English text, use the Porter Stemmer algorithm for stemming or the Lemmatization technique for lemmatization to unify the vocabulary form.

[0033] Sentiment dictionary filtering: Use a sentiment dictionary to identify and remove words irrelevant to sentiment analysis, such as technical terms, jargon, etc., to reduce noise.

[0034] Specifically, each word in the preprocessed text is matched with the words in the sentiment dictionary. During the matching process, if a word is not in the sentiment dictionary, or its sentiment polarity is neutral and it contributes little to the overall sentiment value, then it can be considered that the word is irrelevant to sentiment analysis, and the identified irrelevant words are removed from the text to reduce the interference of noise on the sentiment analysis result.

[0035] 2. Image preprocessing:

[0036] Use data augmentation techniques to preprocess the images, and reasonably control the degree of data augmentation;

[0037] For example: (1) Simulate the situation of the image under different perspectives to help the model learn the features of the image under different angles:

[0038] Select a random rotation angle, which can be randomly selected within a preset range (such as -30° to 30°);

[0039] Apply a rotation transformation to the image to obtain the rotated image;

[0040] The label of the image is kept unchanged, as rotation does not change the content or category of the image.

[0041] (2) Simulate images at different sizes to help the model learn the features of images at different scales:

[0042] Select a random scaling factor, which can be randomly chosen within a preset range (such as 0.5 to 1.5).

[0043] Applies a scale transformation to an image, resulting in a scaled image. Note that the scaled image may need to be cropped or padded to fit the dimensions of the original image (if dimensional consistency is required).

[0044] Also, keep the labels of the images unchanged.

[0045] (3) Simulate the local conditions of the image at different perspectives to help the model learn the local features of the image:

[0046] Randomly select a cropping area, which can be anywhere in the image, but its size is usually fixed or randomly selected within a preset range;

[0047] Crop the image to obtain a cropped image;

[0048] If the size of the cropped image is smaller than the original image, you can choose to fill it (such as using zero padding, mean padding, etc.) or keep the original size (this depends on the specific application scenario and model requirements);

[0049] Keep the image labels unchanged.

[0050] (4) Simulate the image mirroring in the horizontal or vertical direction to increase data diversity:

[0051] Choose a random flip direction, either horizontal or vertical;

[0052] Apply a flip transformation to the image to obtain a flipped image;

[0053] Keep the image labels unchanged.

[0054] Improve the generalization ability of subsequent models when processing image features.

[0055] 2. Feature Extraction

[0056] 1. Text feature extraction:

[0057] Use pre-trained language models such as BERT or GPT to convert pre-processed text into high-dimensional vector representations;

[0058] For example, the BERT model is used to transform text. The BERT model transforms text into a high-dimensional vector representation through a multi-layer Transformer structure.

[0059] The original text is segmented into words to obtain a series of words or subword units, which are then mapped to the BERT word embedding matrix to obtain the initial word embedding vector. Position encoding and paragraph encoding are then added to indicate the position of the word in the sentence and paragraph.

[0060] Input vector E i It can be expressed as:

[0061] E i =W token [i]+W position [pos i ]+W segment [seg i ]

[0062] Among them, W token is the word embedding matrix, W position is the position encoding matrix, W segment is the paragraph encoding matrix, pos i and seg i are the position and paragraph index of word i respectively.

[0063] Transformer Layer

[0064] BERT uses a multi-layer Transformer structure to process input vectors. Each layer contains a self-attention mechanism and a position feed-forward network.

[0065] Self-Attention Mechanism:

[0066] The self-attention mechanism calculates the attention weights of each word to other words and performs weighted summation of the input vector according to these weights to obtain a new representation.

[0067] Attention weight A ij It can be expressed as:

[0068]

[0069] Among them, Q i and K j are the query vector and key vector respectively, which are obtained through linear transformation.

[0070] New representation O j It can be expressed as:

[0071]

[0072] Among them, V j is a vector of values.

[0073] Position feed-forward network:

[0074] The position feedforward network performs further nonlinear transformation on the representation of the output of the self-attention mechanism and further processes the output of the self-attention mechanism.

[0075] The mathematical expression of FFN is:

[0076] FFN(x)=max(0,xW 1 +b 1 )W 2 +b 2

[0077] Among them, x is the output of the self-attention mechanism, W 1 , W 2 is the weight matrix, b 1 , b 2 is the bias term, and max(0, ) represents the ReLU activation function.

[0078] Output vector representation

[0079] In the last layer of BERT, a vector representation (also called hidden state or context embedding) of each word or subword unit is obtained. For classification tasks, the vector representation of the special token [CLS] is used as the vector representation of the entire text.

[0080] Identifying Entities

[0081] Use named entity recognition (NER) technology to identify key entities in the text, such as names of people, places, and institutions, to enhance text features;

[0082] Taking the NER model based on LSTM+CRF as an example, the following are the key steps:

[0083] LSTM Layer

[0084] LSTM is used to capture long-term dependencies in text;

[0085] The output of LSTM h i It can be expressed as:

[0086] h i =LSTM(x i ,h i-1 )

[0087] Among them, x i is the vector representation of the i-th word (which may be the output of BERT or other word embedding representation), h i-1 is the LSTM output of the previous word.

[0088] Emission fraction

[0089] The emission score represents the probability that each word is labeled as a specific type of entity. It is calculated through a linear layer and a softmax function.

[0090] Emission fraction e i,j (i.e., the probability that the i-th word is labeled as the j-th entity type) can be expressed as:

[0091]

[0092] Among them, W e is the weight matrix of the linear layer, is the bias term for the j-th entity type, and K is the number of entity types.

[0093] Transfer score

[0094] The transition score represents the probability of transitioning from one entity type to another. This is usually calculated using a transition matrix.

[0095] Transfer score t j,k (i.e., the probability of transferring from the jth entity type to the kth entity type) can be expressed as:

[0096] t j,k =TransitionMatrix[j,k]

[0097] Among them, TransitionMatrix is ​​a pre-trained transition matrix.

[0098] CRF Layer

[0099] The CRF (Conditional Random Field) layer is used to consider the dependencies between tags and calculate the probability of the entire tag sequence;

[0100] The Conditional Random Field (CRF) layer is used to consider the dependencies between labels and calculate the probability of the entire label sequence.

[0101] Given a label sequence y = {y 1 ,y 2 ,…,y n}, its probability P(y|x) can be expressed as:

[0102]

[0103] Among them, Z(x) is a normalization factor, which represents the sum of the probabilities of all possible label sequences.

[0104] 2. Image feature extraction

[0105] Use pre-trained convolutional neural networks such as ResNet and VGG to extract image features;

[0106] Convolutional Layer

[0107] The convolutional layer extracts local features from the image, specifically represented as;

[0108]

[0109] Assume the input image is X, the convolution kernel is W, and the bias is b, then the output Y of the convolution layer can be expressed as:

[0110] Among them, (i, j) is the position on the output feature map, (m, n) is the size of the convolution kernel, X(i+m, j+n) is the pixel value of the input image at position (i+m, j+n), W(m, n) is the weight of the convolution kernel at position (m, n), and b is the bias term.

[0111] Activate Layer

[0112] The activation layer introduces nonlinear factors, allowing the network to learn complex features;

[0113] Using ReLU activation function:

[0114] f(x)=max(0,x)

[0115] Here, x is the output of the convolutional layer.

[0116] Pooling Layer

[0117] The pooling layer reduces the dimensionality of the feature map while retaining important features;

[0118] The maximum pooling is expressed as:

[0119]

[0120] Where k is the size of the pooling window, (i, j) is the position on the output feature map, and X(m, n) is the value of the input feature map at position (m, n).

[0121] Average Pooling:

[0122]

[0123] Among them, k, (i, j) and X(m, n) have the same meanings as in maximum pooling.

[0124] Fully connected layer

[0125] After feature extraction, one or more fully connected layers are used to classify or regress the extracted features. Each neuron in the fully connected layer is connected to all neurons in the previous layer.

[0126] Assume that the output of the previous layer is a, the weight of the fully connected layer is W, and the bias is b, then the output z of the fully connected layer can be expressed as:

[0127] z=W·a+b

[0128] where · represents matrix multiplication.

[0129] 3. Feature fusion and weighting

[0130] (1) The cross-headed multi-head attention mechanism (CMA) is used to fuse the text and image features with sentiment labels and classify them through a fully connected layer to form image-text feature pairs.

[0131] Input: text feature vector T and image feature vector I

[0132] Calculation: For each head h, calculate the attention weight:

[0133]

[0134] Where Q h , K h are query and key matrices respectively, d k The dimension of the key.

[0135] Output: Concatenate or average the outputs of each head to obtain the fused feature vector F.

[0136] (2) Semantic association weighting:

[0137] Calculate similarity or correlation between text and image features, such as cosine similarity

[0138] The image-text feature pairs are weighted according to their similarity, and feature pairs with high similarity are given higher weights.

[0139] (3) Temporal Attention Mechanism:

[0140] Capture the temporal dependencies between text and images and sort feature pairs according to timestamps.

[0141] Calculate temporal attention weights, such as LSTM or Transformer-based temporal attention mechanisms.

[0142] 4. Sentiment Analysis and Quantification

[0143] Use a deep neural network (DNN) or a convolutional neural network (CNN) to perform sentiment analysis on the fused feature vector F;

[0144] For example, using DNN implementation, the feature vector F is used as input, and nonlinear transformation is performed through multiple dense layers to finally output the result of sentiment classification; specifically, it includes:

[0145] Input layer: feature vector F;

[0146] Hidden layer: multiple layers of fully connected layers, each layer output is nonlinearly transformed through an activation function (such as ReLU);

[0147] Output layer: Use the softmax function to output the probability distribution of sentiment classification.

[0148] Specifically expressed as:

[0149] For the lth hidden layer, the output is a l It can be expressed as:

[0150] a l =σ(W l a l-1 +b l )

[0151] Where W l and b l are the weights and biases of the lth layer, and σ is the activation function.

[0152] For the output layer, the probability distribution p of sentiment classification can be expressed as:

[0153] p=softmax(W L a L-1 +b L )

[0154] Where L is the total number of layers, W L and B L are the weights and biases of the output layer.

[0155] After obtaining the DNN input, the sentiment dictionary is introduced to further assist in judging the positive and negative polarity and intensity of the sentiment;

[0156] Sentiment score calculation: For each word in the image-text feature pair, if it appears in the sentiment lexicon, a sentiment score is calculated based on its sentiment intensity and polarity (positive or negative).

[0157] Comprehensive judgment: Combining the output of DNN with the sentiment score calculated by the sentiment dictionary can further improve the accuracy of sentiment analysis; for example, a threshold is set. If the sentiment classification probability output by DNN is consistent with the sentiment polarity calculated by the sentiment dictionary and the score exceeds the threshold, the final judgment is that sentiment polarity.

[0158] Emotional quantitative scoring:

[0159] The regression model is used to transform sentiment judgment into a regression problem and output the sentiment intensity value s and polarity value p.

[0160] The polarity value p ranges from [-5,5], indicating the degree of emotional pleasure (negative → positive);

[0161] The intensity value s ranges from [0,10] and indicates the intensity of the emotion (calm → excited).

[0162] 5. Filtering and label assignment

[0163] Delete the image and text feature pairs with different positive and negative polarities of the image and text annotation emotions;

[0164] For the remaining image-text pairs, if one of the emotions of the image or text is neutral, then another positive or negative label is selected as the emotional label of the image-text feature pair.

[0165] 6. Build a model of sentiment intensity and polarity

[0166] Emotional quantification processing:

[0167] In a two-dimensional Cartesian coordinate system, the X-axis represents the emotional polarity (such as positive, negative, neutral), and the Y-axis represents the emotional intensity (such as strong, medium, weak)

[0168] Design a sentiment quantification function that accepts image-text feature pairs as input and outputs values ​​in two dimensions: sentiment polarity and sentiment intensity. This function can be a neural network model that is learned through training.

[0169] Substitute the filtered image-text feature pairs into the sentiment quantification function in time series to obtain the sentiment polarity and sentiment intensity values ​​at each time point.

[0170] Sentiment Trend Analysis:

[0171] Based on the sentiment polarity and sentiment intensity values ​​of the time series, plot the curves of sentiment intensity and polarity changing over time.

[0172] Analyze the changing trend of the curve and identify key nodes such as emotional highs, lows and turning points.

[0173] Explain the causes and effects of sentiment trends using contextual and background information.

[0174] Specifically, it can be expressed as:

[0175] Assume that the image-text feature pair is F t =(T t ,I t ), where T t is the text feature vector, I t is the image feature vector, and t is the time point. The emotion quantification function is Q(F t )=(P t ,S t ), where P t is the sentiment polarity value, S t is the emotion intensity value.

[0176] Feature Fusion: F t =Fuse(T t ,I t ), where Fuse is the fusion function.

[0177] Emotion Quantification: (P t ,S t )=Q(F t ), where Q is the sentiment quantification function.

[0178] Sentiment trend: can be plotted by (t,P t ) and (t,S t ) curve to show the changing trend of sentiment polarity and intensity over time.

[0179] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the embodiments here. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the claims of the present invention.

Claims

1. A multimodal sentiment analysis method integrating text and image, characterized in that: The following steps are involved: Obtain text content, image content and their time sequence, use pre-trained word embedding models to extract text features, and use pre-trained neural networks to extract image features; The cross-head attention mechanism (CMA) is used to fuse the text and image features with sentiment labels and classify them through the fully connected layer to form image-text feature pairs; Use deep learning models to perform sentiment analysis on image-text feature pairs to determine the positive and negative polarity of sentiment; The obtained image-text feature pairs are screened, and the image-text feature pairs with different positive and negative polarities of the image and text annotation emotions are deleted. Among the remaining image-text pairs, if one of the emotions of the image or text is neutral, the other positive or negative label is selected as the emotional label of the image-text feature pair; Construct a sentiment intensity and polarity model. In a two-dimensional Cartesian coordinate system, the X-axis represents polarity and the Y-axis represents intensity. The filtered image and text feature pairs are brought into the model in chronological order, the sentiment is quantified, and the sentiment trend of text and image is obtained based on the chronological order.

2. The multimodal sentiment analysis method integrating text and image according to claim 1, characterized in that: The X-axis indicates the degree of emotional pleasure (negative→positive), the Y-axis indicates the intensity of emotion (calm→excited), and the polarity score ranges from -5 to 5.

3. The multimodal sentiment analysis method integrating text and image according to claim 1, characterized in that: When using the deep learning model to perform sentiment analysis on image-text feature pairs, a regression-based sentiment scoring method is adopted to transform sentiment judgment into a regression problem, thereby achieving quantitative scoring of the positive and negative polarity and intensity of the sentiment to evaluate its positive and negative polarity and intensity.

4. The multimodal sentiment analysis method integrating text and image according to claim 1, characterized in that: Before extracting text content features, the text is also preprocessed, including removing stop words, stemming or word form restoration, to improve the accuracy of text feature extraction; before extracting image content features, the image is also preprocessed using data augmentation technology, including rotating, scaling, cropping or flipping the image, to improve the generalization ability of subsequent models when processing image features.

5. The multimodal sentiment analysis method integrating text and image according to claim 1, characterized in that: When performing sentiment analysis on the image-text feature pairs, a sentiment dictionary is introduced to assist in judging the positive and negative polarity and intensity of the sentiment, so as to improve the accuracy and robustness of the sentiment analysis.

6. The multimodal sentiment analysis method integrating text and image according to claim 1, characterized in that: When extracting text features, named entity recognition (NER) technology is also used to identify key entities in the text, including but not limited to name pronouns such as personal names, place names, and institution names, so as to further enrich text features and improve the accuracy of sentiment analysis.

7. The multimodal sentiment analysis method integrating text and image according to claim 1, characterized in that: When adopting the cross multi-head attention mechanism (CMA) to fuse text and image features, the semantic association between text and image is considered. By calculating the similarity or correlation between text and image features, the image-text feature pairs formed by text and image features with strong semantic association are given higher weights, so as to more accurately reflect their emotional intensity and polarity during sentiment quantification processing.

8. The multimodal sentiment analysis method integrating text and image according to claim 7, characterized in that: When using the cross-multi-head attention mechanism to fuse text and image features, the temporal attention mechanism is introduced to capture the temporal dependency between text and image, and the text and image feature pairs are grouped and sorted through this mechanism.

9. The multimodal sentiment analysis method integrating text and image according to claim 4, characterized in that: In the text preprocessing stage, the sentiment dictionary is used to filter out words in the text that are not related to sentiment analysis, reduce the interference of noise information on the sentiment analysis results, and improve the accuracy and efficiency of sentiment analysis.

Images