Image-text sentiment analysis method and system based on multi-modal matching and interaction

Abstract

The invention provides an image-text sentiment analysis method and system based on multi-modal matching and interaction, and belongs to the field of natural language processing, and the method comprises the steps: extracting sentiment auxiliary information from an original text, splicing the sentiment auxiliary information with the original text to obtain an input text, extracting text sentiment features from the input text, extracting image emotion features from the original image; performing multi-modal matching on the text emotion features and the image emotion features to obtain enhanced text features and enhanced emotion features, and processing the enhanced text features and the enhanced emotion features by using an interactive attention mechanism to obtain text-guided visual sparse features and visual-guided text sparse features; fusing the text-guided visual sparse features and the visual-guided text sparse features through an attention layer and a full-connection layer to obtain image-text fusion features; and performing sentiment classification and sentiment consistency calibration based on the image-text fusion features to obtain an image-text sentiment analysis result.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and in particular to a method and system for analyzing image-text sentiment based on multimodal matching and interaction. Background Art

[0002] With the development of the Internet and communication technology, social media has become the main platform for people to share their lives and express their emotions. Users usually use a combination of images and text to express their opinions and emotions. This digital communication method generates a large amount of multimodal emotional data. Compared with single-modal data, multimodal data can provide more comprehensive emotional expression because there is inherent consistency and complementarity between different modalities, which helps to reveal the deep connection between data, thereby more accurately identifying and interpreting emotions.

[0003] In the field of multimodal sentiment analysis, feature learning, modal alignment, and multimodal fusion are three key technical links. Feature learning usually uses pre-trained models or deep neural networks to extract features from each modality, such as using BERT to process text and using CNN networks to process images. Modal alignment solves the differences in expression and feature space of different modal data to ensure that they can be effectively combined. The multimodal fusion stage uses a variety of strategies to combine the features of different modalities, including feature-level fusion, decision-level fusion, and consistency regression fusion.

[0004] However, when dealing with multimodal sentiment analysis, existing technologies are unable to effectively identify and extract key information that truly represents emotions. For example, texts are often mixed with a large number of words that are irrelevant to emotions. These redundant information will interfere with the model's accurate understanding of emotions. Negative texts may be paired with positive pictures. Such information will interfere with the model's accurate understanding of emotions and affect the accuracy of sentiment analysis.

[0005] Therefore, finding a sentiment analysis method that can filter out redundant information in the text and fully consider the sentiment consistency between different modalities is a technical problem that needs to be solved urgently by technical personnel in this field. Summary of the invention

[0006] The present invention provides a method and system for image-text sentiment analysis based on multimodal matching and interaction, which are used to solve the defects in the prior art that text is mixed with words irrelevant to sentiment and the sentiment expression of images and texts is inconsistent, realize the complementarity between image and text modalities, and improve the accuracy and robustness of image-text sentiment analysis.

[0007] The present invention provides a method for analyzing image-text sentiment based on multimodal matching and interaction, comprising the following steps: S1, extracting emotional auxiliary information from the original text and concatenating it with the original text to obtain input text, extracting text emotional features from the input text, and extracting image emotional features from the original image; S2, performing multimodal matching on the text emotion feature and the image emotion feature to obtain enhanced text feature and enhanced emotion feature, and processing the enhanced text feature and the enhanced emotion feature respectively using an interactive attention mechanism to obtain text-guided visual sparse feature and visually guided text sparse feature; S3, fusing the text-guided visual sparse features and the visual-guided text sparse features through an attention layer and a fully connected layer to obtain a picture-text fusion feature; S4. Perform sentiment classification and sentiment consistency calibration based on the image-text fusion features to obtain image-text sentiment analysis results.

[0008] According to a method for analyzing image-text sentiment based on multimodal matching and interaction provided by the present invention, the multimodal matching in step S2 specifically includes: By mapping and converting the image emotion feature and the text emotion feature in feature space, a mapping result is obtained, and the mapping result is input into a hyperbolic tangent class activation function for nonlinear transformation to obtain a cross matrix representing the interactive relationship between the image emotion feature and the text emotion feature; The image emotion feature is adjusted using the trainable weight matrix of the image emotion feature to obtain an image emotion adjustment feature; The text sentiment features are adjusted using the trainable weight matrix of the text sentiment features and the feature fusion is performed with the transpose of the cross matrix to obtain the feature fusion result; After adding the image emotion adjustment feature to the feature fusion result, a nonlinear transformation is performed through a hyperbolic tangent type activation function to obtain an image-text similarity matrix reflecting the similarity between the image emotion feature and the text emotion feature; The cross matrix is ​​multiplied by the image emotion feature and the text emotion feature respectively to obtain an enhanced image emotion feature and an enhanced text emotion feature.

[0009] According to a method for image-text sentiment analysis based on multimodal matching and interaction provided by the present invention, the interactive attention mechanism includes multi-head attention and sparse attention. The enhanced image sentiment features and the enhanced text sentiment features are processed based on the multi-head attention mechanism and the sparse attention mechanism to obtain text perception features, and text-guided visual sparse features and visually guided text sparse features are generated based on the text perception features.

[0010] According to a method for image-text sentiment analysis based on multimodal matching and interaction provided by the present invention, the enhanced text features and the enhanced sentiment features are processed using an interactive attention mechanism respectively, specifically comprising: Inputting the enhanced text sentiment features into a multi-head self-attention mechanism as a query vector, a key vector, and a value vector, and obtaining text perception features by scaling dot product attention calculation; The text-aware features are used as query vectors, and the enhanced image sentiment features are used as key vectors and value vectors, and text-guided visual sparse features are obtained through a sparse attention mechanism based on a sparsemax activation function, a feedforward neural network, and layer normalization processing; The enhanced image features are used as query vectors, and the text-aware features are used as key vectors and value vectors, and visually guided text sparse features are obtained through a sparse attention mechanism based on a sparsemax activation function, a feedforward neural network, and layer normalization processing.

[0011] According to a method for analyzing image-text sentiment based on multimodal matching and interaction provided by the present invention, step S4 specifically includes: Input the image-text fusion features into the fully connected layer and the softmax function for sentiment prediction to obtain intermediate feature representation and sentiment category probability distribution; The image-text fusion feature is input into a trained emotion consistency calibration model to obtain a calibrated fusion feature representation, wherein the trained emotion consistency calibration model includes a feature mapping module and a centroid-based feature calibration module, the feature mapping module is used to map the fusion feature to a task-related feature representation space, and the centroid-based feature calibration module is used to calibrate the feature representation based on the positive emotion centroid, the neutral emotion centroid and the negative emotion centroid.

[0012] According to a method for image-text sentiment analysis based on multimodal matching and interaction provided by the present invention, the method also includes: using a multi-task loss optimization training strategy to build a model and iteratively train sentiment classification and sentiment consistency calibration to obtain a trained sentiment classifier and a trained sentiment consistency calibration model, wherein the trained sentiment classifier is used to perform sentiment classification on the input image-text fusion features and output the sentiment category, and the trained sentiment consistency calibration model is used to calibrate the input image-text fusion features.

[0013] According to a method for image-text sentiment analysis based on multimodal matching and interaction provided by the present invention, the multi-task loss optimization training strategy includes the following steps: Step 1: construct a sentiment classifier and a sentiment consistency calibration model, and construct a multi-task comprehensive loss function based on the sentiment classifier and the sentiment consistency calibration model, wherein the comprehensive loss function includes a loss function for sentiment classification and a loss function for sentiment consistency calibration, and assign an initial weight to each loss function, and independently calculate the gradient of each loss function; Step 2: Calculate the adjusted gradient according to the task weight, integrate the adjusted gradients of each task, and update the parameters of the sentiment classifier and sentiment consistency calibration model based on the integrated total gradient. The calculation formula is: ; ; ; in, represents the adjusted gradient of the mth task, represents the weight of the nth task, n represents the task index, represents the original gradient of the mth task, represents the total gradient after integration of all tasks, m represents the task index except n, represents the updated parameters, Indicates the current parameters. represents the learning rate; Step 3: Use the performance of the sentiment classifier and sentiment consistency calibration model on the validation set or other indicators to dynamically adjust the task weights. The calculation formula is: ; in, represents the weight of the mth task, represents the learning rate of weight adjustment, Represents the performance evaluation indicator on the validation set; Step 4: Repeat steps 2 to 3 for iterative training until the sentiment classifier and the sentiment consistency calibration model reach the expected performance, thus completing the training of the sentiment classifier and the sentiment consistency calibration model.

[0014] The present invention also provides a picture-text sentiment analysis system based on multimodal matching and interaction, which implements the picture-text sentiment analysis method based on multimodal matching and interaction as described above, including: A feature extraction module, used to extract emotion auxiliary information from the original text, and extract multimodal emotion features from the emotion auxiliary information; wherein the multimodal emotion features include text emotion features and image emotion features; A multimodal matching module, used for performing multimodal matching on the text emotion feature and the image emotion feature to obtain enhanced text features and enhanced emotion features, and processing the enhanced text features and the enhanced emotion features respectively using an interactive attention mechanism to obtain text-guided visual sparse features and visually guided text sparse features; A feature fusion module, used for fusing the text-guided visual sparse features and the visual-guided text sparse features through an attention layer and a fully connected layer to obtain a picture-text fusion feature; The analysis module is used to perform sentiment classification and sentiment consistency calibration based on the image-text fusion features to obtain image-text sentiment analysis results.

[0015] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method for analyzing text and image sentiment based on multimodal matching and interaction as described above is implemented.

[0016] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-described methods for analyzing image-text sentiment based on multimodal matching and interaction.

[0017] The present invention provides a method and system for image-text sentiment analysis based on multimodal matching and interaction, which realizes two-way interactive fusion through an interactive attention mechanism. That is, by taking text-aware features as query vectors, a sparse attention mechanism, a feedforward neural network and layer normalization processing to obtain text-guided visual sparse features, and at the same time taking image features as query vectors to obtain visually guided text sparse features, thereby realizing the complementarity between image and text modalities and improving the accuracy and robustness of image-text sentiment analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1 It is a flow chart of the image-text sentiment analysis method provided by the present invention; Figure 2 This is the flow chart of the image and text sentiment analysis method provided by the present invention. Figure 1 ; Figure 3 This is the flow chart of the image and text sentiment analysis method provided by the present invention. Figure 2 ; Figure 4 It is a schematic diagram of the interactive attention mechanism of the image-text sentiment analysis method provided by the present invention; Figure 5 It is a schematic diagram of the sentiment consistency calibration of the image-text sentiment analysis method provided by the present invention; Figure 6 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0020] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0021] like Figure 1 and Figure 2 As shown, the present invention provides a method for image-text sentiment analysis based on multimodal matching and interaction, comprising the following steps: S1, extracting emotional auxiliary information from the original text and concatenating it with the original text to obtain input text, extracting text emotional features from the input text, and extracting image emotional features from the original image; S2, performing multimodal matching on the text emotion feature and the image emotion feature to obtain enhanced text feature and enhanced emotion feature, and processing the enhanced text feature and enhanced emotion feature respectively using an interactive attention mechanism to obtain text-guided visual sparse feature and visual-guided text sparse feature; S3, fusing the text-guided visual sparse features and the visual-guided text sparse features through an attention layer and a fully connected layer to obtain a picture-text fusion feature; S4. Perform sentiment classification and sentiment consistency calibration based on the image-text fusion features to obtain image-text sentiment analysis results.

[0022] The present invention realizes two-way interactive fusion through the interactive attention mechanism, realizes the complementarity between image and text modalities, and improves the accuracy and robustness of image-text sentiment analysis.

[0023] In one embodiment of the present invention, the specific implementation method of extracting emotional auxiliary information from the original text is: A text sentiment auxiliary information extraction method based on a large-scale language model is adopted. This method uses the fine-tuned and optimized ChatGLM3 model to process the input original text and extract the sentiment auxiliary information contained therein.

[0024] Specifically, based on the social media sentiment analysis dataset IsTS-CN, conversation samples are extracted, including original text, manually annotated sentiment keywords and auxiliary information, and the dataset is adapted based on LoRA fine-tuning to fine-tune the ChatGLM3 model. The adjusted dataset includes: "conversations" field: represents the conversation content, which is all the conversation texts in social media; "user" field: represents the user role, including the content posted by the user; "assistant" field: represents the assistant role, simulating the feedback content of sentiment auxiliary information; "text" field: represents the original text to be extracted; "keywords" field: represents the important sentiment keywords annotated manually, where LoRA fine-tuning achieves parameter optimization by injecting a trainable low-rank decomposition matrix into each layer of the model; Use the optimized model to process the input text and extract the sentiment auxiliary information.

[0025] As you can understand, LoRA fine-tuning achieves parameter optimization by injecting a trainable low-rank decomposition matrix into each layer of the model, including: Freeze the pre-trained weights of ChatGLM3 and only optimize the injected low-rank matrices A and B. The weights of the optimized ChatGLM3 are:

[0026] in, Represents the optimized ChatGLM3 model weight matrix, represents the original pre-trained weight matrix of the ChatGLM3 model, express The difference between and W is the weight adjustment amount that needs to be learned. represents the first low-rank matrix, represents the second low-rank matrix.

[0027] The LoRA method only requires a small amount of video memory to complete the fine-tuning task. For example, it can be run on a consumer-grade graphics card (NVIDIA 3090), occupying about 15GB of video memory, while full fine-tuning requires about 190GB.

[0028] By using the social media dataset IsTS-CN and the LoRA fine-tuning technology to optimize the ChatGLM3 model, the present invention can accurately extract the keywords that best represent the sentence sentiment from the original long text as auxiliary information, reduce the interference of irrelevant words in the text on sentiment analysis, and make the extraction of text sentiment auxiliary information more accurate and effective, thereby improving the accuracy of sentiment analysis.

[0029] A specific embodiment is used to illustrate: Taking the social media post "11.21 Forever love, forever tears in the eyes #× University ## Media Integration Center ## University Life" as an example, through the optimized ChatGLM3 model processing, the extracted emotional auxiliary information is "Forever love, forever tears in the eyes" and written into the "keywords" field of the dataset.

[0030] In one embodiment of the present invention, step S1 specifically includes: Extract emotional auxiliary information from the original text and concatenate the emotional auxiliary information with the original text as the input text , enter the text After being converted into a text word sequence, it is input into BERT (Bidirectional Encoder Representations from Transformers) for feature extraction to obtain text sentiment features. The calculation formula is:

[0031] Among them, T represents the sentiment feature of the text, Represents the operation function for feature extraction using the BERT model, Represents input text, represents the global feature vector for the entire input text, The feature vector representing the first word in a text word sequence, The feature vector representing the second word in the text word sequence, Represents the feature vector of the Nth word in the text word sequence, represents a real number matrix space with H rows and (L+1) columns, where H represents the dimension of the text sentiment feature and L represents the maximum length of the text word sequence; The original image I is cropped to a size of 256×256 and divided into L image blocks of 16×16; The separated images are processed by ViT (Vision Transformer) to extract the image emotion feature vector. The calculation formula is:

[0032] in, Represents the emotional characteristics of the image, represents the operation function for feature extraction using the visual transformer model, I represents the original image, Represents the image sentiment feature representation of the entire image, Represents the image sentiment feature representation of the first image block, Represents the image sentiment feature representation of the second image block, represents the image emotion feature representation of the Lth image block, represents a real matrix space with d rows and (L+1) columns, d represents the dimension of the feature representation vector, and L represents the number of blocks after the image is divided; like Figure 3 and Figure 4 As shown in the figure, Vc represents the image sentiment feature, Tc represents the text sentiment feature, Fc represents the image-text fusion feature of Vc and Tc, F1 represents the first image-text fusion feature, F2 represents the second image-text fusion feature, and F L represents the Lth image-text fusion feature. In one embodiment of the present invention, the multimodal matching in step S2 specifically includes: By mapping and transforming the image emotion feature and the text emotion feature in feature space, a mapping result is obtained, and the mapping result is input into a hyperbolic tangent class activation function for nonlinear transformation to obtain a cross matrix representing the interactive relationship between the image emotion feature and the text emotion feature; wherein the calculation formula of the cross matrix is:

[0033] in, represents the cross matrix, represents the transpose of the image sentiment feature V, A trainable weight matrix representing feature interactions, Represents the sentiment feature of the text, represents the hyperbolic tangent class activation function; The image emotion feature is adjusted using the trainable weight matrix of the image emotion feature to obtain an image emotion adjustment feature; The text sentiment features are adjusted using the trainable weight matrix of the text sentiment features and the feature fusion is performed with the transpose of the cross matrix to obtain the feature fusion result; After adding the image emotion adjustment feature to the feature fusion result, a nonlinear transformation is performed through a hyperbolic tangent activation function to obtain a picture-text similarity matrix reflecting the similarity between the image emotion feature and the text emotion feature, wherein the calculation formula of the picture-text similarity matrix is:

[0034] in, represents the image-text similarity matrix, A trainable weight matrix representing the emotional features of an image, Represents the emotional characteristics of the image, A trainable weight matrix representing the characteristics of text sentiment features, represents the transpose of the cross matrix; The cross matrix is ​​multiplied by the image sentiment feature and the text sentiment feature respectively to obtain the enhanced image sentiment feature and the enhanced text sentiment feature. The calculation formula is:

[0035]

[0036] in, represents the enhanced image emotional features, Represents the enhanced text sentiment features.

[0037] In one embodiment of the present invention, the interactive attention mechanism includes multi-head attention and sparse attention. The enhanced image sentiment features and the enhanced text sentiment features are processed based on the multi-head attention mechanism and the sparse attention mechanism to obtain text perception features, and text-guided visual sparse features and visually guided text sparse features are generated based on the text perception features.

[0038] Furthermore, the enhanced text features and the enhanced sentiment features are processed respectively using an interactive attention mechanism, specifically including: The enhanced text sentiment features are input into the multi-head self-attention mechanism as query vector, key vector and value vector, and the text perception features are obtained by scaling dot product attention calculation; the calculation formula is:

[0039] in, represents text-aware features, represents the multi-head attention mechanism, represents the softmax activation function, represents the query vector, represents the transpose of the key vector, d represents the dimension of the attention head, represents a value vector; The text perception feature is used as the query vector, and the enhanced image sentiment feature is used as the key vector and value vector. The text-guided visual sparse feature is obtained through the sparse attention mechanism based on the sparsemax activation function, the feedforward neural network and the layer normalization processing. The calculation formula is:

[0040] in, represents text-guided visual sparse features, Representation layer normalization operation, represents the feedforward neural network operation, Represents the sparsemax activation function; The enhanced image features are used as query vectors, the text-aware features are used as key vectors and value vectors, and the visually guided text sparse features are obtained through a sparse attention mechanism based on a sparsemax activation function, a feedforward neural network, and layer normalization processing. The calculation formula is:

[0041] in, Representing visually guided textual sparse features.

[0042] The present invention adopts a bidirectional fusion method, that is, by taking text-perceived features as query vectors, and obtaining text-guided visual sparse features through a sparse attention mechanism, a feedforward neural network and layer normalization processing, and at the same time taking image features as query vectors to obtain visually guided text sparse features, it improves the previous unidirectional modality-dominated model architecture, effectively enhances the complementarity between images and text, and improves the fusion effect of cross-modal features.

[0043] In one embodiment of the present invention, the calculation formula of step S3 is:

[0044] Among them, F represents the image and text fusion feature, Represents a concatenation operation.

[0045] like Figure 5 As shown, M0 represents the positive emotion centroid, M1 represents the neutral emotion centroid, and M2 represents the negative emotion centroid. In one embodiment of the present invention, step S4 specifically includes: The image-text fusion feature is input into the fully connected layer and the softmax function for sentiment prediction to obtain the intermediate feature representation and sentiment category probability distribution. The calculation formula is:

[0046]

[0047] in, represents the intermediate feature representation, represents the linear transformation operation of the fully connected layer, represents the trainable weight matrix of the fully connected layer, p represents the probability distribution of emotion categories, Represents the trainable weight matrix of the softmax layer; The image-text fusion feature is input into a trained emotion consistency calibration model to obtain a calibrated fusion feature representation, wherein the trained emotion consistency calibration model includes a feature mapping module and a centroid-based feature calibration module, the feature mapping module is used to map the fusion feature to a task-related feature representation space, and the centroid-based feature calibration module is used to calibrate the feature representation based on the positive emotion centroid, the neutral emotion centroid and the negative emotion centroid.

[0048] It can be understood that the training steps of the sentiment consistency calibration model are: Obtain image-text pair samples and their corresponding sentiment labels from the training dataset, fuse these samples through multimodal matching, attention layer and fully connected layer to obtain image-text fusion features, and map the image-text fusion features to the task-related fusion feature representation space through feature mapping; Based on the true emotional labels of the samples, the centroids of the three categories of samples, positive emotion, neutral emotion and negative emotion, are calculated respectively. The centroid of each category of emotion is obtained by the average value of the fusion feature representation of all samples in this category. The calculation formula of the emotional centroid is:

[0049] in, Represents the centroid vector of the sentiment label, corresponding to positive, neutral, and negative sentiment labels, respectively. 0 represents a positive label, 1 represents a neutral label, 2 represents a negative label, i represents the sample index, and B represents the number of samples. represents the indicator function, represents the true sentiment label of the i-th sample, represents the fusion feature representation of the i-th sample; The distance between the fused feature representation of each sample and the centroid of its corresponding emotion category is calculated, and the distance between the sample and its emotion centroid is minimized by optimizing the loss function of emotion consistency calibration, so that samples of the same emotion category are more clustered in the feature space; the loss function is:

[0050] in, represents the loss function of sentiment consistency calibration, j represents the sample index other than i, Represents the L2 distance between the jth sample and its corresponding sentiment centroid.

[0051] During the training process of the emotion consistency calibration model, by continuously optimizing the loss function, we learn the feature representation that can effectively distinguish different emotion categories, and finally obtain a trained emotion consistency calibration model. This model can effectively calibrate the input image and text fusion features for emotion consistency, thereby improving the accuracy of subsequent emotion classification.

[0052] The present invention calculates the positive centroid, neutral centroid and negative centroid of the fused feature representation and uses the L2 loss function to minimize the distance between the sample and its emotional centroid, effectively solving the feature offset problem in the cross-modal feature fusion process, ensuring that the emotional features of different modalities remain consistent in the feature space and improving the emotional expressiveness.

[0053] In one embodiment of the present invention, the image-text sentiment analysis method also includes: using a multi-task loss optimization training strategy to construct and iteratively train models for sentiment classification and sentiment consistency calibration to obtain a trained sentiment classifier and a trained sentiment consistency calibration model, wherein the trained sentiment classifier is used to perform sentiment classification on the input image-text fusion features and output the sentiment category, and the trained sentiment consistency calibration model is used to calibrate the input image-text fusion features.

[0054] In one embodiment of the present invention, the multi-task loss optimization training strategy includes the following steps: Step 1: construct a sentiment classifier and a sentiment consistency calibration model, and construct a multi-task comprehensive loss function based on the sentiment classifier and the sentiment consistency calibration model, wherein the comprehensive loss function includes a loss function for sentiment classification and a loss function for sentiment consistency calibration, and assign an initial weight to each loss function, and independently calculate the gradient of each loss function; Step 2: Calculate the adjusted gradient according to the task weight, integrate the adjusted gradients of each task, and update the parameters of the sentiment classifier and sentiment consistency calibration model based on the integrated total gradient. The calculation formula is: ; ; ; in, represents the adjusted gradient of the mth task, represents the weight of the nth task, n represents the task index, represents the original gradient of the mth task, represents the total gradient after integration of all tasks, m represents the task index except n, represents the updated parameters, Indicates the current parameters. represents the learning rate; Step 3: Use the performance of the sentiment classifier and sentiment consistency calibration model on the validation set or other indicators to dynamically adjust the task weights. The calculation formula is: ; in, represents the weight of the mth task, represents the learning rate of weight adjustment, Represents the performance evaluation indicator on the validation set; Step 4: Repeat steps 2 to 3 for iterative training until the sentiment classifier and the sentiment consistency calibration model reach the expected performance, thus completing the training of the sentiment classifier and the sentiment consistency calibration model.

[0055] It can be understood that the task includes a sentiment classification task and a sentiment consistency calibration task. When m=1, the task is a sentiment classification task, and when m=2, the task is a sentiment consistency calibration task. The performance evaluation index on the validation set can be accuracy, F1 score, etc., and the present invention does not make specific restrictions on this.

[0056] The present invention calculates the adjusted gradient through task weights and dynamically adjusts the task weights according to the performance on the validation set, thereby optimizing the sentiment classification task, improving the overall calibration capability of sentiment features, and effectively solving the loss imbalance problem in multi-task training, making samples of the same sentiment category closer in the feature space, ensuring that each task can be optimized collaboratively, thereby improving the overall performance.

[0057] In one embodiment of the present invention, the following experimental process is given to evaluate the performance of the TEMLO (ext-guided Enhanced Multi-task Loss Optimization) model by designing comparative experiments and ablation experiments. The model is experimentally verified on the IsTS-CN dataset and the CSMSA dataset, and a qualitative analysis is performed.

[0058] (1) Dataset introduction: IsTS-CN dataset: It is derived from the Weibo platform and uses keywords related to college students such as "graduate entrance examination" and "graduation season" as search objects. The dataset contains 10,347 picture-text pairs, covering 34,807 pictures, with an average of about 3 pictures per piece of data. In terms of sentiment distribution, there are 5,449 positive sentiment data, 2,191 neutral sentiment data, and 2,707 negative sentiment data.

[0059] CSMSA dataset: It comes from the most popular social software. The original data contains 28,000 emoji text pairs and 16,000 different emojis. After cleaning, the final valid data is 1,570. This dataset is the first annotated Chinese emoji-based dataset for multimodal sentiment analysis. The original dataset has 4 division methods, and this paper adopts the more difficult Hard Task-2 division method.

[0060] Table 1 Dataset statistics

[0061] (2) Evaluation indicators: The evaluation indicators use accuracy and macro-average F1 value to evaluate the performance of the sentiment analysis model. The specific calculation formula is:

[0062]

[0063]

[0064] in, Indicates the accuracy, Indicates the number of samples predicted as positive in the positive class, TN indicates the number of samples predicted as negative in the negative class, FP indicates the number of samples predicted as positive in the negative class, FN indicates the number of samples predicted as negative in the positive class, F1 indicates the harmonic mean of precision and recall, Precision indicates the precision rate, Recall indicates the recall rate, represents the macro average, It represents the evaluation index calculated on the h-th category, h represents the category index, h∈{1,2,3}, corresponding to positive, neutral, and negative emotions respectively.

[0065] (3) Baseline method: A text pre-training language model (BERT, Bidirectional Encoder Representations from Transformers, bidirectional transformer model) is proposed to effectively extract the semantic features of text for Weibo text sentiment analysis. A text pre-training language model based on Chinese improvement (RoBERTa, Robustly Optimized BERT Pretraining Approach) is proposed. It is an improvement and optimization of BERT, contains more layers and parameters, and can more effectively extract text semantic features. A picture pre-training model (ResNet, Residual Neural Network, residual neural network) is proposed, which uses residual structure to alleviate the degradation problem in deep networks. It is the model for extracting the initial visual features of images in this paper. A model based on multimodal encoding and fusion (MBERT, Multimodal BERT, multimodal BERT model) is proposed. The embedded vectors of text and image are obtained by pre-training models BERT and ResNet respectively, and the image and text feature vectors are spliced ​​and input into the multimodal encoding layer (BERT) for fusion calculation. A model based on fine-grained text fusion (VistaNet, Visual Sentiment Attention Network) is proposed. It uses the visual attention mechanism of image-guided text to determine the importance of different sentences in the document for document sentiment classification. A feature layer interactive fusion (SAMSAM, Semantic Aspect Multimodal Sentiment Analysis Model) model is proposed. The feature input is divided into text, image, OCR text and image style, and a feature interaction layer is designed to fuse different types of feature information. A fusion interactive attention (SA-JIA, Sentiment-Aligned Joint Interactive Attention) model is proposed. Under the guidance of both text and image, new image-text features are obtained, reducing noise while enhancing the emotional consistency expression of image-text modalities. Then, after interactive attention processing, the text and image are interactively fused. A multi-layer fusion (CLMLF, Contrastive Learning Multi-Layer Fusion) model based on two contrastive learning tasks is proposed to learn common emotion-related features in multimodal data based on label and data-based contrastive learning.A Multi-View Calibration Network (MVCN) model is proposed, and a feature-constrained calibration subtask is designed to solve the feature offset in the representation space. A Semantic-Guided Attention Multi-Task Learning (SGAMTL) model is proposed, which uses the fusion of multiple layers of image features to improve the emotional relevance between images and text. A TEMLO method is proposed, which uses text information as a guide, fuses image features through an interactive attention mechanism, and combines a multi-task loss optimization strategy to improve model performance.

[0066] (4) Experimental results: Table 2 Experimental results of sentiment classification of image-text comparison model

[0067] It can be seen from Table 2 that the blank space indicates no experimental data.

[0068] Table 3 Ablation experiment results

[0069] Note: w / o IA means removing interactive attention, w / o AIE means removing auxiliary information extraction, and w / o MLO means removing multi-task loss optimization.

[0070] The results of comparative experiments show that the accuracy of unimodal methods is generally lower than that of multimodal methods, which indicates that in social media datasets, both text and image modalities contain rich emotional information. Therefore, in sentiment analysis tasks, it is necessary to integrate the information of these two modalities. In terms of text and image sentiment analysis, the TEMLO method proposed in this paper achieved the best results on both datasets, and its performance was better than that of existing research methods, which fully verified the effectiveness of the method proposed in this paper.

[0071] The ablation experiment results further show that removing any module designed in this paper will lead to a decrease in model performance. Among them, the performance decrease is most significant when the sentiment auxiliary information module is removed, indicating the key role of sentiment auxiliary information in sentiment analysis. This also shows that the interference information in the text will significantly affect the accuracy of sentiment analysis, thus verifying the effectiveness of this method in eliminating interference and improving analysis accuracy.

[0072] The present invention provides a picture-text sentiment analysis system based on multimodal matching and interaction, which implements the picture-text sentiment analysis method based on multimodal matching and interaction as described above, including: A feature extraction module, used to extract emotion auxiliary information from the original text, and extract multimodal emotion features from the emotion auxiliary information; wherein the multimodal emotion features include text emotion features and image emotion features; A multimodal matching module, used for performing multimodal matching on the text emotion feature and the image emotion feature to obtain enhanced text features and enhanced emotion features, and processing the enhanced text features and the enhanced emotion features respectively using an interactive attention mechanism to obtain text-guided visual sparse features and visually guided text sparse features; A feature fusion module, used for fusing the text-guided visual sparse features and the visual-guided text sparse features through an attention layer and a fully connected layer to obtain a picture-text fusion feature; The analysis module is used to perform sentiment classification and sentiment consistency calibration based on the image-text fusion features to obtain image-text sentiment analysis results.

[0073] The present invention provides a description of a graphic and text sentiment analysis device based on multimodal matching and interaction below. The graphic and text sentiment analysis device based on multimodal matching and interaction described below and the graphic and text sentiment analysis method based on multimodal matching and interaction described above can refer to each other.

[0074] Figure 6 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 6 As shown, the electronic device may include: a processor 610, a communication interface 620, a memory 630 and a communication bus 640, wherein the processor 610, the communication interface 620 and the memory 630 communicate with each other through the communication bus 640. The processor 610 may call the logic instructions in the memory 630 to execute the image-text sentiment analysis method, which includes: extracting sentiment auxiliary information from the original text, extracting text sentiment features from the sentiment auxiliary information, and extracting image sentiment features from the original image; performing multimodal matching on the text sentiment features and the image sentiment features to obtain enhanced text features and enhanced sentiment features, and processing the enhanced text features and the enhanced sentiment features respectively using the interactive attention mechanism to obtain text-guided visual sparse features and visual-guided text sparse features; fusing the text-guided visual sparse features and the visual-guided text sparse features through the attention layer and the fully connected layer to obtain image-text fusion features; performing sentiment classification and sentiment consistency calibration based on the image-text fusion features to obtain image-text sentiment analysis results.

[0075] In addition, the logic instructions in the above-mentioned memory 630 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0076] On the other hand, the present invention also provides a computer program product, which includes a computer program, and the computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the image-text sentiment analysis method provided by the above-mentioned methods, and the method includes: extracting sentiment auxiliary information from the original text, extracting text sentiment features from the sentiment auxiliary information, and extracting image sentiment features from the original image; performing multimodal matching on the text sentiment features and the image sentiment features to obtain enhanced text features and enhanced sentiment features, and processing the enhanced text features and the enhanced sentiment features using an interactive attention mechanism respectively to obtain text-guided visual sparse features and visually guided text sparse features; fusing the text-guided visual sparse features and the visually guided text sparse features through an attention layer and a fully connected layer to obtain image-text fusion features; performing sentiment classification and sentiment consistency calibration based on the image-text fusion features to obtain image-text sentiment analysis results.

[0077] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the image-text sentiment analysis method provided by the above-mentioned methods, the method comprising: extracting sentiment auxiliary information from the original text, extracting text sentiment features from the sentiment auxiliary information, and extracting image sentiment features from the original image; performing multimodal matching on the text sentiment features and the image sentiment features to obtain enhanced text features and enhanced sentiment features, and processing the enhanced text features and the enhanced sentiment features respectively using an interactive attention mechanism to obtain text-guided visual sparse features and visually guided text sparse features; fusing the text-guided visual sparse features and the visually guided text sparse features through an attention layer and a fully connected layer to obtain image-text fusion features; performing sentiment classification and sentiment consistency calibration based on the image-text fusion features to obtain image-text sentiment analysis results.

[0078] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0079] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0080] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for sentiment analysis of images and texts based on multimodal matching and interaction, characterized in that: The following steps are involved: S1, extracting emotional auxiliary information from the original text and concatenating it with the original text to obtain input text, extracting text emotional features from the input text, and extracting image emotional features from the original image; S2, performing multimodal matching on the text emotion feature and the image emotion feature to obtain enhanced text feature and enhanced emotion feature, and processing the enhanced text feature and the enhanced emotion feature respectively using an interactive attention mechanism to obtain text-guided visual sparse feature and visually guided text sparse feature; S3, fusing the text-guided visual sparse features and the visual-guided text sparse features through an attention layer and a fully connected layer to obtain a picture-text fusion feature; S4. Perform sentiment classification and sentiment consistency calibration based on the image-text fusion features to obtain image-text sentiment analysis results.

2. According to claim 1, a method for image-text sentiment analysis based on multimodal matching and interaction is characterized in that: The multimodal matching in step S2 specifically includes: By mapping and converting the image emotion feature and the text emotion feature in feature space, a mapping result is obtained, and the mapping result is input into a hyperbolic tangent class activation function for nonlinear transformation to obtain a cross matrix representing the interactive relationship between the image emotion feature and the text emotion feature; The image emotion feature is adjusted using the trainable weight matrix of the image emotion feature to obtain an image emotion adjustment feature; The text sentiment features are adjusted using the trainable weight matrix of the text sentiment features and the feature fusion is performed with the transpose of the cross matrix to obtain the feature fusion result; After adding the image emotion adjustment feature to the feature fusion result, a nonlinear transformation is performed through a hyperbolic tangent type activation function to obtain an image-text similarity matrix reflecting the similarity between the image emotion feature and the text emotion feature; The cross matrix is ​​multiplied by the image emotion feature and the text emotion feature respectively to obtain an enhanced image emotion feature and an enhanced text emotion feature.

3. According to claim 1, a method for image-text sentiment analysis based on multimodal matching and interaction is characterized in that: The interactive attention mechanism includes multi-head attention and sparse attention. Based on the multi-head attention mechanism and the sparse attention mechanism, the enhanced image sentiment features and the enhanced text sentiment features are processed to obtain text perception features, and text-guided visual sparse features and visually guided text sparse features are generated based on the text perception features.

4. The method for image-text sentiment analysis based on multimodal matching and interaction according to claim 3 is characterized in that: The enhanced text features and the enhanced sentiment features are processed respectively using an interactive attention mechanism, specifically including: Inputting the enhanced text sentiment features into a multi-head self-attention mechanism as a query vector, a key vector, and a value vector, and obtaining text perception features by scaling dot product attention calculation; The text-aware features are used as query vectors, and the enhanced image sentiment features are used as key vectors and value vectors, and text-guided visual sparse features are obtained through a sparse attention mechanism based on a sparsemax activation function, a feedforward neural network, and layer normalization processing; The enhanced image features are used as query vectors, and the text-aware features are used as key vectors and value vectors, and visually guided text sparse features are obtained through a sparse attention mechanism based on a sparsemax activation function, a feedforward neural network, and layer normalization processing.

5. The method for image-text sentiment analysis based on multimodal matching and interaction according to claim 1, characterized in that: Step S4 specifically includes: Input the image-text fusion features into the fully connected layer and the softmax function for sentiment prediction to obtain intermediate feature representation and sentiment category probability distribution; The image-text fusion feature is input into a trained emotion consistency calibration model to obtain a calibrated fusion feature representation, wherein the trained emotion consistency calibration model includes a feature mapping module and a centroid-based feature calibration module, the feature mapping module is used to map the fusion feature to a task-related feature representation space, and the centroid-based feature calibration module is used to calibrate the feature representation based on the positive emotion centroid, the neutral emotion centroid and the negative emotion centroid.

6. The method for image-text sentiment analysis based on multimodal matching and interaction according to claim 1, characterized in that: The image-text sentiment analysis method also includes: using a multi-task loss optimization training strategy to construct a model and iteratively train sentiment classification and sentiment consistency calibration to obtain a trained sentiment classifier and a trained sentiment consistency calibration model, wherein the trained sentiment classifier is used to perform sentiment classification on the input image-text fusion features and output the sentiment category, and the trained sentiment consistency calibration model is used to calibrate the input image-text fusion features.

7. The method for image-text sentiment analysis based on multimodal matching and interaction according to claim 6, characterized in that: The multi-task loss optimization training strategy includes the following steps: Step 1: construct a sentiment classifier and a sentiment consistency calibration model, and construct a multi-task comprehensive loss function based on the sentiment classifier and the sentiment consistency calibration model, wherein the comprehensive loss function includes a loss function for sentiment classification and a loss function for sentiment consistency calibration, and assign an initial weight to each loss function, and independently calculate the gradient of each loss function; Step 2: Calculate the adjusted gradient according to the task weight, integrate the adjusted gradients of each task, and update the parameters of the sentiment classifier and sentiment consistency calibration model based on the integrated total gradient. The calculation formula is: ； ； ； in, represents the adjusted gradient of the mth task, represents the weight of the nth task, n represents the task index, represents the original gradient of the mth task, represents the total gradient after integration of all tasks, m represents the task index except n, represents the updated parameters, Indicates the current parameters. represents the learning rate; Step 3: Use the performance of the sentiment classifier and sentiment consistency calibration model on the validation set or other indicators to dynamically adjust the task weights. The calculation formula is: ； in, represents the weight of the mth task, represents the learning rate of weight adjustment, Represents the performance evaluation indicator on the validation set; Step 4: Repeat steps 2 to 3 for iterative training until the sentiment classifier and the sentiment consistency calibration model reach the expected performance, thus completing the training of the sentiment classifier and the sentiment consistency calibration model.

8. A graphic and text sentiment analysis system based on multimodal matching and interaction, characterized in that: Implementing the method for analyzing image-text sentiment based on multimodal matching and interaction as described in any one of claims 1 to 7, comprising: A feature extraction module, used to extract emotion auxiliary information from the original text, and extract multimodal emotion features from the emotion auxiliary information; wherein the multimodal emotion features include text emotion features and image emotion features; A multimodal matching module, used for performing multimodal matching on the text emotion feature and the image emotion feature to obtain enhanced text features and enhanced emotion features, and processing the enhanced text features and the enhanced emotion features respectively using an interactive attention mechanism to obtain text-guided visual sparse features and visually guided text sparse features; A feature fusion module, used for fusing the text-guided visual sparse features and the visual-guided text sparse features through an attention layer and a fully connected layer to obtain a picture-text fusion feature; The analysis module is used to perform sentiment classification and sentiment consistency calibration based on the image-text fusion features to obtain image-text sentiment analysis results.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the image-text sentiment analysis method based on multimodal matching and interaction as described in any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the image-text sentiment analysis method based on multimodal matching and interaction as described in any one of claims 1 to 7 is implemented.

Images