Multi-modal implicit sentiment analysis method based on multi-granularity attention mechanism

Abstract

The invention relates to a multi-modal implicit sentiment analysis method based on a multi-granularity attention mechanism, which belongs to the field of artificial intelligence, and comprises the following steps: S1, extracting visual features in image information by using a visual feature extraction strategy of multi-scale attention fusion; s2, generating a text description of the image information by adopting an image translation engine, splicing the text description with an original text, and extracting text features through a hierarchical attention mechanism; s3, carrying out normalization processing; s4, performing multi-modal fine-grained correlation fusion on the visual features and the text features by adopting a bidirectional cross attention mechanism, and extracting fine-grained features in the visual features and the text features through a multi-head attention mechanism and residual connection; and S5, fusing the fine-grained features by using an adaptive multi-modal classification module to complete sentiment classification. The multi-modal sentiment analysis method solves the problems that multi-modal sentiment analysis is insufficient in inter-modal correlation mining, insufficient in fine-grained information capture and large in noise interference.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence, and specifically relates to a multimodal implicit sentiment analysis method based on a multi-granularity attention mechanism. Background Art

[0002] With the popularity of the Internet and mobile devices, user-generated content on social media platforms has surged, containing a large amount of multimodal information, such as text, images, etc. These multimodal data provide unprecedented richness for people to express their personal opinions, attitudes and emotions. However, traditional sentiment analysis methods are mainly based on text data and cannot fully explore the sentiment features in multimodal information, which limits the accuracy and application scope of sentiment analysis.

[0003] Existing deep learning methods have made significant progress in multimodal sentiment analysis. Through deep neural networks, complementary sentiment features are extracted from multiple modalities such as vision, text, and audio, which significantly improves the accuracy of sentiment recognition. However, due to the significant differences in the distribution of data in different modalities, the effective fusion of multimodal sentiment features remains a difficult problem to be solved. Existing research has introduced domain separation methods to divide multimodal sentiment features into a common subspace shared by the modalities and a modality-specific private subspace to alleviate the problem of inconsistent distribution. However, sentiment information is often buried in a large amount of irrelevant noise. The representation of each modality contains not only shared sentiment features, but also its own unique sentiment details. How to extract effective sentiment signals and remove irrelevant noise has become a key challenge to improve the performance of sentiment analysis.

[0004] In multimodal sentiment analysis, the fusion of image and text features is crucial for generating accurate sentiment representations. However, a large amount of detail information is often lost during the fusion process. The reason is that images and texts differ significantly in terms of expression and information density. Images contain rich visual details and implicit emotions, while texts convey complex semantics through language structures. Simply fusing the features of the two may not be able to fully capture these subtleties, resulting in the loss of sentiment information. In addition, existing fusion methods may focus more on overall features and ignore key detail elements in images and texts. This loss of detail will affect the model's accurate understanding of emotions, especially when dealing with complex contexts such as sarcasm and metaphors. Therefore, how to retain and effectively utilize the detail information of images and texts in feature fusion remains a major challenge in the field of multimodal sentiment analysis.

[0005] At present, multimodal sentiment analysis models can already perform explicit sentiment analysis based on given sentiment feature words with considerable accuracy, but implicit sentiment analysis is still very difficult due to the hidden opinion features. Unlike explicit sentiment analysis, the research on implicit sentiment analysis is more challenging because in this task, the input text only contains "neutral" factual descriptions without explicit expression of opinions and attitudes. For such implicit sentiment scenarios, existing models cannot achieve satisfactory performance because they do not contain any explicit sentiment feature words. Summary of the invention

[0006] The purpose of the present invention is to provide a multimodal implicit sentiment analysis method based on a multi-granularity attention mechanism to solve the problems of insufficient mining of inter-modal correlation, insufficient capture of fine-grained information and large noise interference in existing multimodal sentiment analysis methods.

[0007] In order to achieve the above object, the technical solution of the present invention is as follows: The present invention relates to a multimodal implicit sentiment analysis method based on a multi-granularity attention mechanism, which comprises the following steps: S1. Extract visual features from image information using a multi-scale attention fusion visual feature extraction strategy; S2. Use the image translation engine to generate a text description of the image information, and concatenate the text description of the image information with the original text to obtain a combined text, and extract text features from the combined text through a hierarchical attention mechanism; S3. Map the visual features and text features into a unified embedding space and normalize them; S4. A bidirectional cross-attention mechanism is used to perform multimodal fine-grained correlation fusion on the normalized visual features and text features. The fine-grained correlation between visual and text features is captured through a multi-head attention mechanism and residual connection, and fine-grained features in visual and text features are extracted. S5. Use the adaptive multimodal classification module and the adaptive gating mechanism to fuse the fine-grained features in visual features and text features, and use the improved focal loss function for training to complete sentiment classification.

[0008] Preferably, the S1 uses a ResNet101 model to extract visual features from image information, and the specific steps include: S1.1. Using a multi-scale feature extraction strategy to extract the input image Adjust to different scales, and extract features of images of each scale through ResNet101 with shared parameters. The expression is: , in, Indicated in scale The scale feature map extracted below, For different scales The image below, are the trainable parameters of the ResNet101 model; S1.2. Introduce the attention mechanism to calculate the attention weight of each scale feature map , the expression is: , in, is the convolutional layer used to generate the attention map, represents the normalization function, which is used to transform the input values ​​into a probability distribution where each value is positive and the sum is 1. Representative scale feature map The importance weight of each position in Based on the attention weight of each scale feature map, a weighted feature map of each size is obtained , whose expression is: , in, Indicates bit-by-bit multiplication of elements; S1.3. Fuse weighted feature maps of all scales to form visual features , the fusion method is as follows: , in, represents the fusion function; S1.4. Perform dimensionality reduction on the fused visual features to obtain the final visual features , whose expression is: , in, is the activation function, and are the trainable weights and bias of the fully connected layers, respectively.

[0009] Preferably, in S2, the ViT-GPT-2 model is used to generate a text description of the image information, and the specific steps include: S2.1. Unify the resolution of image information, use ViT encoder to divide the image into fixed-size image blocks, and flatten the image blocks into embedded feature vector sequences, which are expressed as: , in, is the number of image patches, Indicates The feature vector of the image block, Indicates that the ViT model is used to encode the image block features. are the trainable parameters of the ViT encoder; S2.2. Generate a description of the image by receiving the feature vector sequence output by the ViT encoder through the GPT-2 decoder , its conditional probability is: , The generation probability of the entire description is: , in, represents the word generated at time step t, The text description representing the image information is composed of multiple The sequence composed of Represents a probability distribution function, which is used to calculate the probability of generating a certain word or the entire text description. Represents the GPT-2 decoder module, which is used to predict the word generated at the current time step t The probability of It represents the total length of the generated text description sequence, that is, the total number of words contained in the generated text. ∏ represents the probability multiplication symbol, which is used to calculate the joint probability of the entire sequence generation.

[0010] Preferably, the text description of the image information in S2 and the concatenation expression of the original text are: , in, Combined text, For the original text, is a text description of the image information, Represents a concatenation operation.

[0011] Preferably, the specific steps of extracting text features from the combined text by the hierarchical attention mechanism in S2 are: S2.3. Use the pre-trained language model to process the combined text and obtain the contextual embedding of each word , whose expression is: , in, , Word Contextual embeddings, represents the operation of the pre-trained language model, is the trainable parameter of the language model; S2.4. Introduce hierarchical attention mechanism to obtain text representation; S2.5. Based on the text representation, the final text feature is obtained through the fully connected layer and nonlinear activation function, and its expression is: , in, is the text feature, and are all trainable parameters of the fully connected layer, is a nonlinear activation function, Representation of text.

[0012] Preferably, the representation of the text in S2.4 is a word-level representation or a sentence-level representation; The method for obtaining the word-level representation includes: Calculate the importance weight of each word , the expression is as follows: , in, and are all trainable parameters, Represents the total length of the generated text description sequence, that is, the total number of words contained in the generated text. Indicates The feature vector of a word, is a trainable bias vector, represents the hyperbolic tangent activation function, which is used to introduce nonlinear characteristics so that the model can learn more complex representations. is the number of tokens in the text, represents the index of another token in the text, represents the exponential function; Based on the importance weight of each word , and get word-level representation , whose expression is: ; The method for obtaining the sentence-level representation includes: Divide the combined text into sentences, and the word-level representation of each sentence is ,in , calculate the importance weight of the sentence , the expression is as follows: , in, represents a trainable weight vector used to calculate the weight of sentence-level attention, Represents the index used to traverse all sentences, indicating the position of the sentence considered in the current calculation in the sentence set; Sentence-level representation based on sentence importance weights : ; Representation of text in S2.5 Word-level representation or sentence-level representation .

[0013] Preferably, the expression of the S3 mapping and normalization operation is: , , in, and Respectively represent the visual features and text features after normalization operation, is the visual feature obtained after fusion and dimensionality reduction. is the text feature, and Represent the linear mapping matrix of visual features and the linear mapping matrix of text features respectively, and Represent the bias items of visual features and text features respectively, Represents the layer normalization operation.

[0014] Preferably, the specific steps of S4 include: S4.1. Visual features as queries , text features as keys Sum , Computing Vision-to-Text Attention , the calculation formula is: , in, ; S4.2. Using text features as queries , visual features as keys Sum , calculate text-to-visual attention , the calculation formula is: , in, ; In the formula, and Respectively represent the visual features and text features after normalization operation, The linear mapping matrix of the query of the visual feature, the linear mapping matrix of the key of the text feature, the linear mapping matrix of the value of the text feature, the linear mapping matrix of the query of the text feature, the linear mapping matrix of the key of the visual feature, the linear mapping matrix of the value of the visual feature, is the normalized exponential function, Represents feature dimension; S4.3. Use multi-head attention mechanism to expand visual-to-text attention and text-to-visual attention to The process of multi-head cross attention is as follows: , , in, and They represent multi-head cross attention from vision to text and multi-head cross attention from text to vision, respectively. represents a multi-head cross-attention operation, and Respectively represent the output linear transformation matrix; S4.4. Based on multi-head cross attention, a residual connection network is introduced to perform layer normalization operations on visual features and text features, which can be expressed as: , , in, and Respectively represent the visual features and text features after layer normalization operation, Representation layer normalization operation; S4.5. The high-level feature information in the visual features and text features after the layer normalization operation is extracted through the feedforward network and the nonlinear activation function. The expression is: , , in, Represents high-level features in visual features, Represents high-level features in text features, and represents the weight matrix of the feedforward network, and represents the bias term of the feedforward network, represents a nonlinear activation function; S4.6. Gradually capture deeper fine-grained features by stacking multiple layers of cross-attention modules, and then gradually update the high-level feature information in visual features and text features and ; The expression for capturing deeper fine-grained features is: , , in, Represents deep and fine-grained features in visual features, Represents the deep and fine-grained features in text features, Indicates layer, Represents a cross-attention layer for mutual attention and information exchange between different modalities.

[0015] Preferably, the specific steps of the S5 sentiment classification include: S5.1. Align the dimensions of high-level features in visual features and text features. The expression is: , , in, represents the aligned visual features, represents the aligned text features, Represents high-level features in visual features, Represents high-level features in text features, They all represent linear transformation matrices, and Represent the bias terms of visual features and text features respectively; S5.2. Based on the subsequent visual features and text features, an adaptive gating mechanism is introduced to calculate the weight coefficients of visual features and text features through a multi-layer perceptron. The calculation formula is: , , in, represents the weight of the visual feature, represents the weight of the text feature, Represent the weight matrix of the multi-layer perceptron, denote the first and second bias terms of visual features and text features respectively, represents the Sigmoid activation function, represents a nonlinear activation function; S5.3. Apply the weights to the corresponding features by element-by-element multiplication and fuse them to obtain the fused features. The fusion formula is: , in, Indicates fusion features; S5.4. Input the fused features into the fully connected layer and map them to the sentiment classification space to obtain the classification score. The calculation formula is: , in, represents the weight matrix of the fully connected layer, represents the bias term of the fully connected layer, represents the unnormalized classification score; S5.5. An improved focal loss function is used to balance the categories, which is expressed as: , in, represents the value of the loss function, represents the class balance weight, represents the true label, represents the adaptive focusing parameter, represents the number of sentiment categories, Indicates The predicted probability of the class, The said The predicted probability of a class is calculated as: , in, Indicates the index of the category currently being calculated in the entire category collection. represents the unnormalized prediction score of the i-th class, represents the unnormalized prediction score of the jth class; The class balance weights The calculation formula is: , in, For the The number of samples of the class, is a hyperparameter, and ; The expression of the adaptive focusing parameter is: , in, is the basic focusing parameter, For the The sample frequency of the class, , is the total number of samples, is a hyperparameter; S5.6. Input the fused features into the adaptive multimodal classification module and select the category with the highest probability as the final result. The expression is: , in, Represents the output classification result, Indicates the category with the highest probability .

[0016] Compared with the prior art, the technical solution provided by the present invention has the following beneficial effects: 1. The multimodal implicit sentiment analysis method involved in the present invention effectively extracts rich and discriminative visual and text fine-grained features by designing a multi-scale attention fusion visual feature extractor and a text feature extractor based on a hierarchical attention mechanism. The method adaptively handles noise and nonlinear relationships in multimodal data, solves the problem of difficulty in capturing fine-grained associations between modalities in the prior art, and improves the accuracy and effectiveness of feature extraction.

[0017] 2. The present invention adopts a bidirectional cross-attention mechanism to achieve deep interaction and fusion between visual and textual features. By calculating attention in both the visual-to-text and text-to-visual directions, it captures the fine-grained correlation between multimodal data, narrows the modal differences, and enhances the richness and discrimination ability of multimodal representations, overcoming the deficiency of simple feature concatenation in the prior art that cannot deeply capture the complex relationships between the features of each modality.

[0018] 3. The present invention designs an adaptive multimodal classification module, dynamically adjusts the contribution of visual and textual modalities through an adaptive gating mechanism, and introduces an improved focal loss function, which enhances the model's attention to difficult-to-classify samples and minority categories, effectively solves the problems of category imbalance and difficulty in identifying implicit sentiment cues, improves the accuracy and robustness of implicit sentiment analysis, and makes up for the shortcomings of the prior art in processing difficult-to-classify samples and dynamic weight allocation. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 Architecture diagram of the multi-granularity attention fusion network model for multimodal implicit sentiment analysis.

[0020] Figure 2 Dataset test result diagram. DETAILED DESCRIPTION

[0021] In order to further understand the content of the present invention, the present invention is described in detail in conjunction with examples. The following examples are used to illustrate the present invention but are not used to limit the scope of the present invention.

[0022] Example: See attached Figure 1 The present invention relates to a multimodal implicit sentiment analysis method based on a multi-granularity attention mechanism, which comprises the following steps: S1. Use the ResNet101 model and the multi-scale attention fusion visual feature extraction strategy to extract visual features from image information. The specific steps include: S1.1. In order to capture image features of different scales, a multi-scale feature extraction strategy is used to extract the input image Adjust to different scales , and extract features from images of each scale through ResNet101 with shared parameters, and the expression is: (1), in, Indicated in scale The scale feature map extracted below, For different scales The image below, are the trainable parameters of the ResNet101 model; S1.2. In the feature extraction process, in order to allow the model to automatically focus on more important areas in the image, an attention mechanism is introduced to calculate the attention weight of each scale feature map , the expression is: (2), in, is the convolutional layer used to generate the attention map, represents the normalization function, which is used to transform the input values ​​into a probability distribution where each value is positive and the sum is 1. Representative scale feature map The importance weight of each position in Based on the attention weight of each scale feature map, a weighted feature map of each size is obtained , whose expression is: (3), in, Indicates bit-by-bit multiplication of elements; S1.3. Weighted features of all scales The images are fused to form visual features , the fusion method is as follows: (4), in, represents the fusion function; S1.4. In order to reduce the computational complexity and promote the interaction between features, a fully connected layer is introduced to reduce the dimensionality of the fused features to obtain the final visual features. , whose expression is: (5), in, is the activation function, and are the trainable weights and bias of the fully connected layers, respectively.

[0023] S2. In order to narrow the semantic gap between images and texts, the image translation engine (ViT-GPT-2) model is used to generate a text description of the image information, and the text description of the image information and the original text are concatenated to obtain a combined text. The text features are extracted from the combined text through a hierarchical attention mechanism. The specific steps include: S2.1. The resolution of the unified image information is 224×224, and the ViT encoder is used to divide the image into image blocks of fixed size, and the image blocks are flattened into an embedded feature vector sequence, which is expressed as: (6), in, is the number of image patches, Indicates The feature vector of the image block, Indicates that the ViT model is used to encode the image block features. are the trainable parameters of the ViT encoder; S2.2. Generate a description of the image by receiving the feature vector sequence output by the ViT encoder through the GPT-2 decoder , its conditional probability is: (7), The generation probability of the entire description is: (8), in, represents the token generated at time step t, The text description representing the image information, that is, the entire generated text description sequence (complete image description), is composed of multiple The sequence composed of Represents a probability distribution function, which is used to calculate the probability of generating a word or the entire text description. Represents the GPT-2 decoder module, which predicts the generated at the current time step t by inputting the previously generated context information and visual feature vector The probability of Represents the total length of the generated text description sequence, that is, the total number of words contained in the generated text, and ∏ represents the probability multiplication symbol, which is used to calculate the joint probability of the entire sequence generation; Then, the image description With the original text Splice to form a combined text , the expression is: (9), in, Combined text, For the original text, is a text description of the image information, Represents a splicing operation; S2.3. Combining text using the pre-trained language model BERT Processing to obtain the context embedding of each word , whose expression is: (10), in, , Word Contextual embeddings, represents the operation of the pre-trained language model, is the trainable parameter of the language model; S2.4. In order to capture fine-grained text features at different levels, a hierarchical attention mechanism is introduced to obtain text representation, which includes word-level representation or sentence-level representation; The method for obtaining the word-level representation includes: Calculate the importance weight of each word , the expression is as follows: (11), in, and are all trainable parameters, Represents the total length of the generated text description sequence, that is, the total number of words contained in the generated text. Indicates The feature vector of a word, is a trainable bias vector, represents the hyperbolic tangent activation function, which is used to introduce nonlinear characteristics so that the model can learn more complex representations. is the number of tokens in the text, represents the index of another token in the text, represents the exponential function; Based on the importance weight of each word , and get word-level representation , whose expression is: (12); The method for obtaining the sentence-level representation includes: Divide the combined text into sentences, and the word-level representation of each sentence is ,in , calculate the importance weight of the sentence , the expression is as follows: (13), in, represents a trainable weight vector used to calculate the weight of sentence-level attention, Represents the index used to traverse all sentences, indicating the position of the sentence considered in the current calculation in the sentence set; Sentence-level representation based on sentence importance weights : (14); S2.5. Based on the text representation, the final text feature is obtained through the fully connected layer and nonlinear activation function, and its expression is: (15), in, is the text feature, and are all trainable parameters of the fully connected layer, is a nonlinear activation function, For the representation of text, the representation of text Can be word-level representation or sentence-level representation , depending on whether sentence-level attention is performed.

[0024] S3. To eliminate visual features and text features The modal difference between them is used to map the visual features and text features into a unified embedding space and normalize them. The expression is: (16), (17), in, and Respectively represent the visual features and text features after normalization operation, is the visual feature obtained after fusion and dimensionality reduction. is the text feature, and Represent the linear mapping matrix of visual features and the linear mapping matrix of text features respectively, and Represent the bias items of visual features and text features respectively, The normalization operation at the representation layer is used to make the data mean 0 and the variance 1 to ensure the standardization of each modal feature.

[0025] S4. A bidirectional cross-attention mechanism (including visual-to-text attention and text-to-visual attention) is used to perform multimodal fine-grained correlation fusion on the normalized visual features and text features. The fine-grained correlation between visual and text features is captured through a multi-head attention mechanism and residual connection, and fine-grained features in visual features and text features are extracted. The specific steps include: S4.1. Visual features as queries , text features as keys Sum , Computing Vision-to-Text Attention , the calculation formula is: (18), in, ; S4.2. Using text features as queries , visual features as keys Sum , calculate text-to-visual attention , the calculation formula is: (19), in, ; In the formula, and Respectively represent the visual features and text features after normalization operation, A linear mapping matrix representing the query of visual features, a linear mapping matrix of the keys of text features, a linear mapping matrix of the values ​​of text features, a linear mapping matrix of the query of text features, a linear mapping matrix of the keys of visual features, and a linear mapping matrix of the values ​​of visual features, is the normalized exponential function, Represents the feature dimension, which is used to scale the inner product result to prevent the value from being too large and affecting the calculation stability of Softmax; S4.3. In order to enhance the expressiveness of the model, a multi-head attention mechanism is used to expand the visual-to-text attention and the text-to-visual attention into There are multiple parallel attention heads, each of which can independently focus on different information in the input features. The process of multi-head cross attention is as follows: (20), (twenty one), in, and They represent multi-head cross attention from vision to text and multi-head cross attention from text to vision, respectively. represents a multi-head cross-attention operation, and Respectively represent the output linear transformation matrix; S4.4. Based on multi-head cross attention, in order to enhance the representation ability of features and prevent gradient disappearance, a residual connection network is introduced to perform layer normalization operations on visual features and text features, which can be expressed as: (twenty two), (twenty three), in, and Respectively represent the visual features and text features after layer normalization operation, Representation layer normalization operation; S4.5. The high-level feature information in the visual features and text features after the layer normalization operation is extracted through the feedforward network and the nonlinear activation function. The expression is: (twenty four), (25), in, Represents high-level features in visual features, Represents high-level features in text features, and represents the weight matrix of the feedforward network, and represents the bias term of the feedforward network, Represents a nonlinear activation function, which is used to introduce nonlinearity and improve the expressiveness of the model; S4.6. Gradually capture deeper fine-grained features by stacking multiple layers of cross-attention modules, and then gradually update the high-level feature information in visual features and text features and ; The expression for capturing deeper fine-grained features is: (26), (27), in, Represents deep and fine-grained features in visual features, Represents the deep and fine-grained features in text features, Indicates layer, Represents a cross-attention layer for mutual attention and information exchange between different modalities.

[0026] S5. Use the adaptive multimodal classification module and the adaptive gating mechanism to fuse the fine-grained features in the visual features and text features, and use the improved focal loss function for training to complete the sentiment classification. The specific steps include: S5.1. Align the dimensions of high-level features in visual features and text features to ensure that each modality has the same feature dimension when fused. The expression of this operation is: (28), (29), in, represents the aligned visual features, represents the aligned text features, Represents high-level features in visual features, Represents high-level features in text features, They all represent linear transformation matrices, and Represent the bias terms of visual features and text features respectively; S5.2. Based on the subsequent visual features and text features, an adaptive gating mechanism is introduced to calculate the weight coefficients of visual features and text features through a multi-layer perceptron. The calculation formula is: (30), (31), in, represents the weight of the visual feature, represents the weight of the text feature, Represent the weight matrix of the multi-layer perceptron, denote the first and second bias terms of visual features and text features respectively, Represents the Sigmoid activation function, which is used to limit the weight coefficient to between [0, 1], Represents a nonlinear activation function, which is used to introduce nonlinearity; S5.3. Apply the weights to the corresponding features by element-by-element multiplication and fuse them to obtain the fused features. The fusion formula is: (32), in, represents the fusion feature, represents bit-wise multiplication; S5.4. Input the fused features into the fully connected layer and map them to the sentiment classification space to obtain the classification score. The calculation formula is: (33), in, represents the weight matrix of the fully connected layer, represents the bias term of the fully connected layer, represents the unnormalized classification score; S5.5. Use the improved focal loss function to balance the categories to deal with the problem of category imbalance and improve the model's attention to minority categories. Its expression is: (34), in, represents the value of the loss function, represents the class balance weight, represents the true label, represents the adaptive focusing parameter, represents the number of sentiment categories, Indicates The predicted probability of the class, The said The predicted probability of a class is calculated as: (35), in, Indicates the index of the category currently being calculated in the entire category collection. represents the unnormalized prediction score of the i-th class, represents the unnormalized prediction score of the jth class; The class balance weights The calculation formula is: (36), in, For the The number of samples of the class, is a hyperparameter, and ; The expression of the adaptive focusing parameter is: (37), in, is the basic focusing parameter, For the The sample frequency of the class, , is the total number of samples, is a hyperparameter; S5.6. Input the fused features into the adaptive multimodal classification module and select the category with the highest probability as the final result. The expression is: (38), in, Represents the output classification result, Indicates the category with the highest probability .

[0027] Experimental example To verify the effect of the method of this embodiment, the following experiment was conducted.

[0028] Benchmark datasets. Two well-known multimodal sentiment analysis benchmark datasets were used in the experiment: Twitter-2015 and Twitter-2017. These two datasets contain tweets posted on the Twitter platform from 2014 to 2015 and from 2016 to 2017, respectively. In these two datasets, each piece of data consists of a comment and a related picture, and the sentiment annotation of each piece of data is divided into three categories (positive, negative, and neutral).

[0029] Table 1. Statistics of Twitter-2015 and Twitter-2017 datasets.

[0030]

[0031] Experimental parameter settings. The experiment was completed on a computer equipped with the following hardware: the operating system is Ubuntu22.04, equipped with Intel(R) Xeon(R) Platinum 8488C * 2 and NVIDIA RTX A6000 * 4 graphics processors, and the memory capacity is 256 GB. Throughout the experiment, the PyTorch framework was used for model training, the AdamW optimizer was used, the learning rate was set to 0.0001, and the batch size was 32. The number of multi-scales was set to 3, the number of heads of the multi-head attention mechanism was 8, the number of layers of the cross attention module was 4, the number of hidden units in the fully connected layer was 512, and the GELU function was used as the activation function. To prevent overfitting, the Dropout ratio was set to 0.1, and the Early Stopping strategy was adopted, and the number of patience rounds was set to 5.

[0032] Evaluation indicators. In the experiment, accuracy (Accuracy, Acc) and macro F1 score (Macro F1, M-F1) are used as evaluation indicators. Among them, , is the number of samples predicted correctly, is the total number of samples; , is the positive prediction value, is the recall value.

[0033] Existing methods for experimental comparison. In order to prove the effectiveness of the method of this embodiment, several cutting-edge methods were selected in the experiment for effect comparison with the method of this embodiment. The existing methods include: IAN: Introducing the interactive attention network (IAN), generating representations for the target and context respectively through interactive learning, emphasizing the independent modeling of the two, and improving the performance of aspect-level sentiment classification. BERT-Based Model: BERT based on the bidirectional Transformer architecture learns deep bidirectional representations from unlabeled text, significantly improving the performance of natural language processing tasks. BERTABSA-ATT: The problem of the attention mechanism ignoring low-frequency sentiment words in aspect-level sentiment analysis is solved through a progressive self-supervised attention learning method. The model iteratively identifies active and misleading context words during training, and adds regularization terms to enhance model performance. CCR: Proposing a cross-modal consistent regression (CCR) method, combining CNN for image sentiment analysis and paragraph vector model for text sentiment analysis, and improving the sentiment analysis performance of large-scale social multimedia content through a multimodal regression model. DMAF: Introduces the Deep Multimodal Attention Fusion (DMAF) model, uses a hybrid fusion framework to extract the distinguishing features of visual and textual content and their internal correlations, and uses unimodal and multimodal attention models for fusion. AMGN: Proposes an attention-based modality-gated network (AMGN), learns the correlation between image and text modalities through a visual-semantic attention model, uses modality-gated LSTM to adaptively select modalities with stronger emotional signals, and applies semantic self-attention to focus on distinguishing features. TEMMA: Uses LSTM and Transformer-based multimodal multi-head attention to extract temporal dependencies, and combines early, late and model fusion techniques to improve the prediction of emotional states under stress. CLMLF: Proposes a contrastive learning and multi-layer fusion (CLMLF) method, aligns and fuses the word-level features of text and images through a multi-layer fusion module, and introduces two contrastive learning tasks based on labels and data to improve the performance of emotion detection in multimodal data. GEAR: A general debiasing framework is proposed based on inverse probability weighting (IPW), which reduces the spurious correlations in multimodal sentiment analysis by reducing the weights of high-bias samples, thereby enhancing the out-of-distribution generalization ability of the model.

[0034] Experimental results. Table 2 shows the experimental results of this model and the baseline models on the Twitter-2015 and Twitter-2017 datasets. Overall, this model achieves the best performance on both datasets. On the Twitter-2017 dataset, MGAFN-ISA achieves an accuracy of 90.91% and an F1 score of 90.52%, which is a significant improvement over the second-best GEAR (accuracy of 89.51% and F1 score of 89.26%). On the Twitter-2015 dataset, MGAFN-ISA achieves an accuracy of 93.10% and an F1 score of 93.50%, respectively, which also surpasses all baseline models.

[0035] From the perspective of model type, the performance of text-based unimodal models (IAN, BERT-Based Model, BERTABSA-ATT) is generally lower than that of multimodal models. This shows that it is difficult to fully capture implicit emotions by relying solely on text information, and the introduction of visual modality helps to improve the model's emotion recognition ability. In multimodal models, early methods such as CCR and DMAF have relatively low performance, probably because they are deficient in modal fusion and correlation mining. Models that use attention mechanisms and more advanced fusion strategies (such as TEMMA, DTRN, CLMLF) show better performance, indicating the importance of fine-grained modal interaction for sentiment analysis.

[0036] Table 2 Test results on the dataset

[0037] Test result analysis chart Figure 2 As shown in the figure, the accuracy and F1 score of the proposed method on the Twitter-2015 and Twitter-2017 datasets are significantly higher than those of other models, reflecting its excellent performance in emotion recognition tasks. In addition, Figure 2 It also demonstrates the stability and consistency of the present invention in different evaluation indicators, further proving that the present invention can effectively integrate text and visual information when processing multimodal data, and improve the accuracy and robustness of sentiment analysis. These data fully reflect the innovative advantages of the present invention in modal fusion strategy and sentiment recognition capabilities, and verify its potential value in practical applications.

[0038] The present invention is described in detail above in conjunction with the embodiments, but the contents described are only preferred embodiments of the present invention and cannot be considered to limit the scope of implementation of the present invention. All equivalent changes and improvements made according to the scope of application of the present invention should still fall within the scope of the patent coverage of the present invention.

Claims

1. A multimodal implicit sentiment analysis method based on a multi-granularity attention mechanism, characterized by: It includes the following steps: S1. Extract visual features from image information using a multi-scale attention fusion visual feature extraction strategy; S2. Use the image translation engine to generate a text description of the image information, and concatenate the text description of the image information with the original text to obtain a combined text, and extract text features from the combined text through a hierarchical attention mechanism; S3. Map the visual features and text features into a unified embedding space and normalize them; S4. A bidirectional cross-attention mechanism is used to perform multimodal fine-grained correlation fusion on the normalized visual features and text features. The fine-grained correlation between visual and text features is captured through a multi-head attention mechanism and residual connection, and fine-grained features in visual and text features are extracted. S5. Use the adaptive multimodal classification module and the adaptive gating mechanism to fuse the fine-grained features in visual features and text features, and use the improved focal loss function for training to complete sentiment classification.

2. The multimodal implicit sentiment analysis method based on multi-granularity attention mechanism according to claim 1 is characterized in that: S1 uses the ResNet101 model to extract visual features from image information, and the specific steps include: S1.

1. Using a multi-scale feature extraction strategy to extract the input image Adjust to different scales, and extract features of images of each scale through ResNet101 with shared parameters. The expression is: ， in, Indicated in scale The scale feature map extracted below, For different scales The image below, are the trainable parameters of the ResNet101 model; S1.

2. Introduce the attention mechanism to calculate the attention weight of each scale feature map , the expression is: ， in, is the convolutional layer used to generate the attention map, represents the normalization function, which is used to transform the input values ​​into a probability distribution where each value is positive and the sum is 1. Representative scale feature map The importance weight of each position in Based on the attention weight of each scale feature map, a weighted feature map of each size is obtained , whose expression is: , in, Indicates bit-by-bit multiplication of elements; S1.

3. Fuse weighted feature maps of all scales to form visual features , the fusion method is as follows: , in, represents the fusion function; S1.

4. Perform dimensionality reduction on the fused visual features to obtain the final visual features , whose expression is: , in, is the activation function, and are the trainable weights and bias of the fully connected layers, respectively.

3. The multimodal implicit sentiment analysis method based on multi-granularity attention mechanism according to claim 1 is characterized in that: In S2, the ViT-GPT-2 model is used to generate a text description of the image information, and the specific steps include: S2.

1. Unify the resolution of image information, use ViT encoder to divide the image into fixed-size image blocks, and flatten the image blocks into embedded feature vector sequences, which are expressed as: ， in, is the number of image patches, Indicates The feature vector of the image block, Indicates that the ViT model is used to encode the image block features. are the trainable parameters of the ViT encoder; S2.

2. Generate a description of the image by receiving the feature vector sequence output by the ViT encoder through the GPT-2 decoder , its conditional probability is: ， The generation probability of the entire description is: ， in, represents the word generated at time step t, The text description representing the image information is composed of multiple The sequence composed of Represents a probability distribution function, which is used to calculate the probability of generating a certain word or the entire text description. Represents the GPT-2 decoder module, which is used to predict the word generated at the current time step t The probability of It represents the total length of the generated text description sequence, that is, the total number of words contained in the generated text. ∏ represents the probability multiplication symbol, which is used to calculate the joint probability of the entire sequence generation.

4. The multimodal implicit sentiment analysis method based on multi-granularity attention mechanism according to claim 3 is characterized in that: The concatenation expression of the text description of the image information in S2 and the original text is: ， in, Combined text, For the original text, is a text description of the image information, Represents a concatenation operation.

5. The multimodal implicit sentiment analysis method based on multi-granularity attention mechanism according to claim 4 is characterized in that: The specific steps of S2 extracting text features from the combined text through the hierarchical attention mechanism are: S2.

3. Use the pre-trained language model to process the combined text and obtain the contextual embedding of each word , whose expression is: ， in, , Word Contextual embeddings, represents the operation of the pre-trained language model, is the trainable parameter of the language model; S2.

4. Introduce hierarchical attention mechanism to obtain text representation; S2.

5. Based on the text representation, the final text feature is obtained through the fully connected layer and nonlinear activation function, and its expression is: ， in, is the text feature, and are all trainable parameters of the fully connected layer, is a nonlinear activation function, Represents the text.

6. The multimodal implicit sentiment analysis method based on multi-granularity attention mechanism according to claim 5 is characterized in that: The text in S2.4 is represented as a word-level representation or a sentence-level representation; The method for obtaining the word-level representation includes: Calculate the importance weight of each word , the expression is as follows: ， in, and are all trainable parameters, Represents the total length of the generated text description sequence, that is, the total number of words contained in the generated text. Indicates The feature vector of a word, is a trainable bias vector, represents the hyperbolic tangent activation function, which is used to introduce nonlinear characteristics so that the model can learn more complex representations. is the number of tokens in the text, represents the index of another token in the text, represents the exponential function; Based on the importance weight of each word , and get word-level representation , whose expression is: ； The method for obtaining the sentence-level representation includes: Divide the combined text into sentences, and the word-level representation of each sentence is ,in , calculate the importance weight of the sentence , the expression is as follows: ， in, represents a trainable weight vector used to calculate the weight of sentence-level attention, Represents the index used to traverse all sentences, indicating the position of the sentence considered in the current calculation in the sentence set; Sentence-level representation based on sentence importance weights : ； Representation of text in S2.5 Word-level representation or sentence-level representation .

7. The multimodal implicit sentiment analysis method based on multi-granularity attention mechanism according to claim 1, characterized in that: The expressions for the S3 mapping and normalization operations are: ， ， in, and Respectively represent the visual features and text features after normalization operation, is the visual feature obtained after fusion and dimensionality reduction. is the text feature, and Represent the linear mapping matrix of visual features and the linear mapping matrix of text features respectively, and Represent the bias items of visual features and text features respectively, Represents the layer normalization operation.

8. The multimodal implicit sentiment analysis method based on multi-granularity attention mechanism according to claim 1, characterized in that: The specific steps of S4 include: S4.

1. Visual features as queries , text features as keys Sum , Computing Vision-to-Text Attention , the calculation formula is: ， in, ; S4.

2. Using text features as queries , visual features as keys Sum , calculate text-to-visual attention , the calculation formula is: ， in, ; In the formula, and Respectively represent the visual features and text features after normalization operation, A linear mapping matrix representing the query of visual features, a linear mapping matrix of the keys of text features, a linear mapping matrix of the values ​​of text features, a linear mapping matrix of the query of text features, a linear mapping matrix of the keys of visual features, and a linear mapping matrix of the values ​​of visual features, is the normalized exponential function, Represents feature dimension; S4.

3. Use multi-head attention mechanism to expand visual-to-text attention and text-to-visual attention to The process of multi-head cross attention is as follows: ， ， in, and They represent multi-head cross attention from vision to text and multi-head cross attention from text to vision, respectively. represents a multi-head cross-attention operation, and Respectively represent the output linear transformation matrix; S4.

4. Based on multi-head cross attention, a residual connection network is introduced to perform layer normalization operations on visual features and text features, which can be expressed as: ， ， in, and Respectively represent the visual features and text features after layer normalization operation, Representation layer normalization operation; S4.

5. The high-level feature information in the visual features and text features after the layer normalization operation is extracted through the feedforward network and the nonlinear activation function. The expression is: ， ， in, Represents high-level features in visual features, Represents high-level features in text features, and represents the weight matrix of the feedforward network, and represents the bias term of the feedforward network, represents a nonlinear activation function; S4.

6. Gradually capture deeper fine-grained features by stacking multiple layers of cross-attention modules, and then gradually update the high-level feature information in visual features and text features and ; The expression for capturing deeper fine-grained features is: ， ， in, Represents deep and fine-grained features in visual features, Represents the deep and fine-grained features in text features, Indicates layer, Represents a cross-attention layer for mutual attention and information exchange between different modalities.

9. The multimodal implicit sentiment analysis method based on multi-granularity attention mechanism according to claim 1, characterized in that: The specific steps of the S5 sentiment classification include: S5.

1. Align the dimensions of high-level features in visual features and text features. The expression is: ， ， in, represents the aligned visual features, represents the aligned text features, Represents high-level features in visual features, Represents high-level features in text features, Both represent linear transformation matrices, and Represent the bias terms of visual features and text features respectively; S5.

2. Based on the subsequent visual features and text features, an adaptive gating mechanism is introduced to calculate the weight coefficients of visual features and text features through a multi-layer perceptron. The calculation formula is: ， ， in, represents the weight of the visual feature, represents the weight of the text feature, Represent the weight matrix of the multi-layer perceptron, denote the first and second bias terms of visual features and text features respectively, represents the Sigmoid activation function, represents a nonlinear activation function; S5.

3. Apply the weights to the corresponding features by element-by-element multiplication and fuse them to obtain the fused features. The fusion formula is: ， in, Indicates fusion features; S5.

4. Input the fused features into the fully connected layer and map them to the sentiment classification space to obtain the classification score. The calculation formula is: ， in, represents the weight matrix of the fully connected layer, represents the bias term of the fully connected layer, represents the unnormalized classification score; S5.

5. An improved focal loss function is used to balance the categories, which is expressed as: ， in, represents the value of the loss function, represents the class balance weight, represents the true label, represents the adaptive focusing parameter, represents the number of sentiment categories, Indicates The predicted probability of the class, The said The predicted probability of a class is calculated as: ， in, Indicates the index of the category currently being calculated in the entire category collection. represents the unnormalized prediction score of the i-th class, represents the unnormalized prediction score of the jth class; The class balance weight The calculation formula is: ， in, For the The number of samples of the class, is a hyperparameter, and ; The expression of the adaptive focusing parameter is: ， in, is the basic focusing parameter, For the The sample frequency of the class, , is the total number of samples, is a hyperparameter; S5.

6. Input the fused features into the adaptive multimodal classification module and select the category with the highest probability as the final result. The expression is: ， in, Represents the output classification result, Indicates the category with the highest probability .

Images