A multimodal aspect-level sentiment analysis method that fuses images and text

Abstract

This invention relates to the field of image and text fusion technology, specifically to a multimodal aspect-level sentiment analysis method that integrates images and text at multiple levels. The method includes the following steps: Step 1, multi-granularity visual alignment, including coarse-grained alignment, fine-grained alignment, and character-level alignment; Step 2, multi-scale visual fusion, first capturing multi-scale visual features and obtaining corresponding hierarchical visual representations using visual data from multi-granularity visual alignment, then supervising visual aspects and opinions, and finally performing dynamic visual fusion based on prompts; Step 3, text-guided multimodal training, obtaining multimodal aspect-level sentiment analysis results by minimizing the KL loss between the text output space and the text + image context output space. This invention can perform multimodal aspect-level sentiment analysis more effectively.

Description

Technical Field

[0001] This invention relates to the field of image and text fusion technology, and more specifically, to a multimodal aspect-level sentiment analysis method that integrates images and text at multiple levels. Background Technology

[0002] The rapid development of social networks has brought not only convenience and speed to interpersonal communication, allowing people to share their lives with those thousands of miles away without leaving home, but also a more realistic expression of emotions. Unlike when social networks were first established and content was purely text-based, today, to facilitate users' more authentic expression of their thoughts, the modes of expression have evolved from being limited to single textual descriptions to a combination of multimodal content such as emoticons, emojis, images, and videos. Just as the transmission of emotions in face-to-face communication relies more on factors such as body language and tone of voice, the diverse content on numerous social platforms, led by WeChat Moments, Weibo, and Twitter, makes it difficult to efficiently and accurately classify user sentiment at this stage. Traditional single-modal text sentiment analysis can no longer effectively utilize the diverse content of the big data era.

[0003] Multimodal sentiment analysis is a crucial step towards fully utilizing multimodal information for sentiment-based representation learning. It allows businesses to understand user concerns and social trends, enabling them to better adjust company policies and improve user satisfaction. However, most current multimodal sentiment analysis methods treat multimodal data holistically to determine user sentiment polarity, failing to reveal user sentiment judgments regarding specific aspects. Therefore, focusing on aspect-level terms allows for a holistic understanding while also considering aspect-level sentiment information. However, extracting the corresponding sentiment information from images based on aspect-level terms is extremely challenging.

[0004] Therefore, some researchers have considered jointly extracting aspect terms and classifying their corresponding sentiments based on text-image relationship detection on annotated datasets. However, this approach mainly focuses on global cross-modal interactions and cannot effectively address the fine-grained correspondence between images and text. While existing techniques can fuse text and visual information to some extent by extracting feature information from text and images through text and visual encoders and mapping them to the same-dimensional mapping space before interaction, images and text represent two modalities with a significant semantic gap. Irrelevant visual objects during interaction often negatively impact the fusion of text and visual modalities. Therefore, these fusion methods are limited by the lack of semantic alignment between visual objects and text content before fusion; due to their intermodal misalignment, the interaction problem is not fully resolved. Thus, while pre-training on a large amount of labeled data can be used for intermodal alignment, it consumes significant human and computational resources, and few studies explore bridging the semantic gap between modalities and effectively utilizing visual information for coarse-grained to fine-grained cross-modal fusion. Summary of the Invention

[0005] The present invention provides a multimodal aspect-level sentiment analysis method that integrates images and text at multiple levels. This method can address the discrepancy between the performance and cost represented by the human and computational resources required for intermodal alignment and the need to label datasets.

[0006] A multimodal aspect-level sentiment analysis method for multi-level fusion of images and text according to the present invention is characterized by comprising the following steps:

[0007] Step 1: Multi-granularity visual alignment

[0008] This includes coarse-grained alignment, fine-grained alignment, and character-grained alignment.

[0009] Step 2: Multi-scale visual fusion

[0010] First, multi-scale visual features are captured and corresponding hierarchical visual representations are obtained by utilizing multi-granularity visual alignment visual data. Then, visual aspects and opinions are supervised, and finally, dynamic visual fusion based on prompt is performed.

[0011] Step 3: Text-guided multimodal training

[0012] By minimizing the KL loss between the text output space and the text + image context output space, we obtain the results of multimodal aspect-level sentiment analysis.

[0013] Preferably, in coarse-grained alignment, images are processed through an image description model to establish a holistic relationship between vision and language, aiming to generate meaningful and effective image descriptions. This represents the semantic information of visual content at a coarse-grained level and connects the feature spaces between modalities. Specifically, the image captioning tool ClipCap is applied to generate high-quality captions for the scene, denoted as C:

[0014] C = Caption(G)

[0015] Where C represents the overall description of the image generated from the Caption, the Caption is used as a coarse-grained text alignment map for the entire image, and G represents the input image.

[0016] Preferably, in fine-grained alignment, the LightFace face detector is first used to identify all faces and convert them into textual face attributes; then, facial expression description templates are used to generate facial descriptions.

[0017] D = Face D description(G)

[0018] After obtaining facial attributes in text form, the prediction confidence based on the face detector is sorted in descending order, and attributes with low prediction confidence are filtered out.

[0019] As a preferred approach, for character granular alignment, Google's Tesseract OCR engine is used to achieve character granular alignment by accurately recognizing and extracting text from images.

[0020] Oc = OCR(G)

[0021] Where Oc represents the concatenated sequence of English words extracted by the OCR model; concatenating Oc with C and D forms the text-visual alignment of the visual context, denoted as Vc = (C, [SEP], D, [SEP], Oc, [SEP]); at this stage, visual information is mapped to the text space, and after concatenating it with the text input T, a T+V input is formed; a [SEP] marker is inserted between the T input and the visual context Vc, and T+V and T are processed by a Transformer-based model to obtain the final hidden representation H. L T+V With H L T It is fed into the CRF layer; for a label sequence y = (y1, y2, ..., yn), given the hidden representation H L The probability of the label sequence y is defined as follows:

[0022]

[0023] p(y|H L ) = Softmax(s(H L ,y))

[0024] in, From label y j to y j+1 The randomly initialized transformation matrix, Indicates from H L Label y of linear transformation j The emission matrix, H L It is the hidden representation of the Lth layer, p(y|H L ) is the conditional probability.

[0025] Preferably, in step 2, specifically: a four-block ResNet is used as the visual encoder, and YOLOv5x6 is used as the object detector; at most, the three regions with the highest confidence scores are retained. b =(O1,O2,…,O z );

[0026] The multi-scale image input is fed into the visual encoder, where deep information is upsampled and shallow information is added element by element; this process extracts the multi-scale feature map F = (F1, F2, ..., F...). r Subsequently, average pooling is performed to enhance visual recognition capabilities within the image.

[0027]

[0028]

[0029] Where [F1,F2,…,F r ] G and This represents visual features obtained by fusing multi-scale feature maps, including global image features and object features; Ave represents the fusion of F... i Projected onto an average pooling layer of the same size; This represents the l-th multi-scale visual fusion feature.

[0030] Preferably, in step 2, an ANP detector is used as supervision for visual aspects and opinions. The distribution P of the first N predicted adjective-noun pairs is calculated using the first N predicted probabilities:

[0031]

[0032] Where r = 4, and W ∈ R d×N and b∈R N This represents the trainable parameters, and d represents the dimension of the text representation in BERT.

[0033] To make the predicted distribution P closer to the true top-N adjective-noun pair distribution A, the standard cross-entropy loss L is used. V To extract fine-grained information from image input:

[0034] L V =-Alog(P)

[0035] This loss can reduce the difference between the predicted distribution and the basic fact distribution.

[0036] Preferably, in step 2, a dynamic attention mechanism is used to project multi-level visual information as a prompt onto the l-th layer of BERT in the text modality; multiple normalized vectors are calculated using a dynamic projector, and these vectors determine the degree of visual feature transformation for each block within BERT; first, logits are calculated. As a mapped semaphore:

[0037]

[0038] Wherein, MLP represents the feature dimensionality reduction layer; multi-head self-attention is used to combine the transformed multi-scale visual features with the key / value vectors of the context representation of each layer of BERT; Let k be the k-th multi-scale visual fusion feature, where This indicates that the transformed multi-scale visual features are input into the l-th layer of BERT;

[0039]

[0040] Represents the weight matrix. This indicates that the transformed features obtained by multi-scale visual fusion features through multi-head self-attention and dimensionality reduction are split into two parts;

[0041] These transformed visual features are then concatenated with the original keyword and value vectors from the previous layer in BERT, which act as new keys and values ​​during the attention process; the computation of visual cues and text-based fusion attention is as follows:

[0042]

[0043] in, as well as These represent the query, key, and value in the new attention matrix, respectively.

[0044] Preferably, in step 3, minimizing the KL-divergence of the probability distribution obtained by feeding the output into the BERT model is equivalent to calculating the cross-entropy loss between the two distributions:

[0045]

[0046] in and It is a probability distribution;

[0047] The loss function is the negative log-likelihood function of the true label sequence, as shown below:

[0048]

[0049] The final objective function is defined as follows:

[0050] L MTVAF =λ·L T +μ·L V +γ·L T+V

[0051] Where λ, μ and γ∈[0,1] are hyperparameters used to control the contribution of each module.

[0052] This invention proposes a pre-trained model that uses text as the core and images as auxiliary information, injecting multi-granular information from images into the text through hierarchical feature extraction. Since the size variance of object information in images often varies significantly, the model utilizes receptive fields of different sizes found in feature pyramids and employs a dynamic attention mechanism to fuse hierarchical visual features into each layer of BERT.

[0053] The multi-level alignment module designed in this invention can align the semantic space of the visual modality with the text space before fusion. Due to the significant semantic discrepancy between the two modalities, this invention proposes translating images into text at the global, local, and character granular levels. Utilizing image description models, face attribute templates, and an OCR engine to obtain visual contexts at three granularities can bridge the semantic gap between modalities. To reduce the misleading effect of irrelevant visual context, the impact of visual noise on text representation prediction results is reduced by minimizing the KL loss between the text output space and the text + image context output space.

[0054] The universal multimodal alignment and fusion method designed in this invention not only effectively utilizes the content of image modalities, but also aligns and fuses important information of each modality during the training process of the pre-trained model, further enhancing the ability of attribute-level sentiment analysis. Attached Figure Description

[0055] Figure 1 This is a flowchart of a multimodal aspect-level sentiment analysis method that integrates images and text in an embodiment. Detailed Implementation

[0056] To further understand the content of this invention, a detailed description of the invention will be provided in conjunction with the accompanying drawings and embodiments. It should be understood that the embodiments are merely illustrative and not limiting of the invention.

[0057] Example

[0058] like Figure 1 As shown, this embodiment provides a multimodal aspect-level sentiment analysis method that fuses images and text at multiple levels, characterized by the following steps:

[0059] Step 1: Multi-granularity visual alignment (module)

[0060] This includes coarse-grained alignment, fine-grained alignment, and character-grained alignment.

[0061] Step 2: Multi-scale visual fusion (module)

[0062] First, multi-scale visual features are captured and corresponding hierarchical visual representations are obtained by utilizing multi-granularity visual alignment visual data. Then, visual aspects and opinions are supervised, and finally, dynamic visual fusion based on prompt is performed.

[0063] Step 3: Text-guided multimodal training (module)

[0064] By minimizing the KL loss between the text output space and the text + image context output space, we obtain the results of multimodal aspect-level sentiment analysis.

[0065] Multi-granularity visual alignment module

[0066] In coarse-grained alignment, images are processed through an image description model to establish a holistic relationship between vision and language, aiming to generate meaningful and effective image descriptions. This represents the semantic information of visual content at a coarse-grained level, connecting the feature spaces between modalities. Specifically, the image captioning tool ClipCap is applied to generate high-quality captions for the scene, represented as C:

[0067] C = Vaption(G)

[0068] Where C represents the overall description of the image generated from the Caption, the Caption is used as a coarse-grained text alignment map for the entire image, and G represents the input image.

[0069] In fine-grained alignment, the LightFace face detector is first used to identify all faces and convert them into textual face attributes. Then, facial expression description templates [referencing the paper FITE (Face-Sensitive Image-to-Emotional-Text Cross-modal Translation for Multimodal Aspect-based Sentiment Analysis)] are used to generate facial descriptions.

[0070] D = FaceDescription(G)

[0071] After obtaining facial attributes in text form, the prediction confidence based on the face detector is sorted in descending order, and attributes with low prediction confidence are filtered out.

[0072] In character granular alignment, Google's Tesseract OCR engine is applied to achieve character granular alignment by accurately recognizing and extracting text from images:

[0073] Oc = OCR(G)

[0074] Where Oc represents the concatenated sequence of English words extracted by the OCR model; concatenating Oc with C and D forms the text-visual alignment of the visual context, denoted as Vc = (C, [SEP], D, [SEP], Oc, [SEP]); at this stage, visual information is mapped to the text space, and after concatenating it with the text input T, a T+V input is formed; a [SEP] marker is inserted between the T input and the visual context Vc, and T+V and T are processed by a Transformer-based model to obtain the final hidden representation H. L T+V With H L T It is fed into the CRF layer; for a label sequence y = (y1, y2, ..., yn), given the hidden representation H L The probability of the label sequence y is defined as follows:

[0075]

[0076] p(y|H L ) = Softmax(s(H L ,y))

[0077] in, From label y j to y j+1 The randomly initialized transformation matrix, Indicates from H L Label y of linear transformationj The emission matrix, H L It is the hidden representation of the Lth layer, p(y|H L ) is the conditional probability.

[0078] Multi-scale visual fusion module

[0079] 1) Multi-scale visual features

[0080] Convolutional Neural Networks (CNNs) have shallow and deep layers with different receptive fields, making them suitable for processing objects of varying sizes, which is especially important when dealing with fine-grained images. With this in mind, this module aims to capture multi-scale visual features and obtain corresponding hierarchical visual representations by leveraging the multi-granularity transalignment of visual data implemented in the previous module.

[0081] To achieve this, global and region images are used as supplementary visual information. Global images help capture large-scale abstract concepts, such as entity context and overall emotional cues. On the other hand, region images are key visual cues for small-scale details and guide visual feature learning. By integrating semantic and spatial information from deep and shallow features, we obtain multi-scale visual features. Specifically, a four-block ResNet structure is used as the visual encoder, and YOLOv5x6 is used as the object detector; at most, the three regions with the highest confidence scores are retained. b =(O1,O2,…,O z );

[0082] The multi-scale image input is fed into the visual encoder, where deep information is upsampled and shallow information is added element by element; this process extracts the multi-scale feature map F = (F1, F2, ..., F...). r Subsequently, average pooling is performed to enhance visual recognition capabilities within the image.

[0083]

[0084]

[0085] Where [F1,F2,…,F r ] G and This represents visual features obtained by fusing multi-scale feature maps, including global image features and object features; Ave represents the fusion of F... i Projected onto an average pooling layer of the same size; This represents the l-th multi-scale visual fusion feature.

[0086] 2) Top-N visual opinions

[0087] When merging a large amount of fine-grained visual information into two inputs, it becomes apparent that the visual features obtained from multi-scale networks also require a comprehensive exploration of their fine-grained relationships. Inspired by VLP-MABSA, we employ ANP as supervision for visual aspects and opinions. These aspects and opinions are derived from the pre-trained ANP detector DeepSentiBank, which predicts the category distribution of 2089 pairs of adjective-noun pairs across the entire image, reflecting visual opinion information.

[0088] Because relying solely on predicted ANPs leads to error propagation problems, and using the entire distribution for supervision introduces redundant noise, it is recommended to use adjective-noun pairs with the top N predicted probabilities for the bootstrapping model. The top N adjective-noun pairs in the input image constitute an ordered list of visual opinions, which may cause the model to overemphasize fine-grained visual information. The distribution P of the top N predictions is calculated as follows:

[0089]

[0090] Where r = 4, and W ∈ R d×N and b∈R N This represents the trainable parameters, and d represents the dimension of the text representation in BERT.

[0091] To make the predicted distribution P closer to the true top-N adjective-noun pair distribution A, the standard cross-entropy loss L is used. V To extract fine-grained information from image input:

[0092] L V =-Alog(P)

[0093] This loss reduces the difference between the predicted distribution and the basic fact distribution, thereby enhancing the model's ability to capture the desired visual aspects of opinion relationships.

[0094] 3) Dynamic visual fusion based on prompt

[0095] In multimodal sentiment analysis, while the image modality plays a crucial role, the text modality remains dominant in recognizing entities and their emotions. Therefore, a dynamic attention mechanism is used to project multi-level visual information as a prompt onto the l-th layer of BERT within the text modality; multiple normalized vectors are computed using a dynamic projector, which determine the degree of visual feature transformation within each block of BERT; first, logits are calculated. As a mapped semaphore:

[0096]

[0097] Wherein, MLP represents the feature dimensionality reduction layer; multi-head self-attention is used to combine the transformed multi-scale visual features with the key / value vectors of the context representation of each layer of BERT; Let k be the k-th multi-scale visual fusion feature, where This indicates that the transformed multi-scale visual features are input into the l-th layer of BERT;

[0098]

[0099] Represents the weight matrix. This indicates that the transformed features obtained by multi-scale visual fusion features through multi-head self-attention and dimensionality reduction are split into two parts;

[0100] These transformed visual features are then concatenated with the original keyword and value vectors from the previous layer in BERT, which act as new keys and values ​​during the attention process; the computation of visual cues and text-based fusion attention is as follows:

[0101]

[0102] in, as well as These represent the query, key, and value in the new attention matrix, respectively.

[0103] Text-guided multimodal training module

[0104] Given the diversity and multi-layered nature of visual information, while it provides visual cues from different angles when inputting into BERT, it can easily draw attention to excessively long visual text, thus obscuring the dominant role of textual information in the backpropagation of gradient loss. Furthermore, the lack of annotation labels to supervise the alignment between textual and visual information also presents a significant challenge. Therefore, minimizing the KL-divergence of the probability distribution obtained by feeding the output into the BERT model is equivalent to calculating the cross-entropy loss between these two distributions:

[0105]

[0106] in and It is a probability distribution;

[0107] Because textual visual information introduces noise into the aspect-sentiment pairing process of multimodal aspect-level sentiment analysis models, this embodiment employs a text-centric approach to convey important information from the multimodal context. Therefore, only It is backpropagated. The loss function is the negative log-likelihood function of the true label sequence, as shown below:

[0108]

[0109] The final objective function is defined as follows:

[0110] L MTVAF =γ·L T +γ·L V +γ·L T+V

[0111] Where λ, μ, and γ ∈ [0, 1] are hyperparameters used to control the contribution of each module. The multi-level alignment module designed in this embodiment can align the semantic space of the visual modality with the text space before fusion. Due to the large semantic deviation between the two modalities, this embodiment proposes to translate images into text at the global, local, and character granularities. By using image description models, face attribute templates, and OCR engines to obtain visual contexts at three granularities, the semantic gap between modalities can be bridged. In order to reduce the misleading effect of irrelevant visual context, the influence of visual noise on the text representation prediction results is reduced by minimizing the KL loss between the text output space and the text + image context output space.

[0112] The general multimodal alignment and fusion design in this embodiment not only effectively utilizes the content of image modalities, but also aligns and fuses important information of each modality during the training process of the pre-trained model, further enhancing the ability of attribute-level sentiment analysis.

[0113] The present invention and its embodiments have been described above illustratively. This description is not restrictive, and the figures shown are only one embodiment of the present invention; the actual structure is not limited thereto. Therefore, if those skilled in the art are inspired by this description and design similar structures and embodiments without departing from the spirit of the present invention, such designs should fall within the protection scope of the present invention.

Claims

1. A multimodal aspect-level sentiment analysis method that fuses images and text at multiple levels, characterized in that: Includes the following steps: Step 1: Multi-granularity visual alignment This includes coarse-grained alignment, fine-grained alignment, and character-grained alignment. In coarse-grained alignment, images are processed through an image description model to establish a holistic relationship between vision and language, aiming to generate meaningful and effective image descriptions. This represents the semantic information of visual content at a coarse-grained level, connecting the feature spaces between modalities. Specifically, the image captioning tool ClipCap is applied to generate high-quality captions for the scene, represented as C: ； Where C represents the overall description of the image generated from the Caption, the Caption is used as a coarse-grained text alignment map for the entire image, and G represents the input image; In fine-grained alignment, the LightFace face detector is first used to identify all faces and convert them into textual face attributes; then, facial expression description templates are used to generate facial descriptions. : ； After obtaining facial attributes in text form, the prediction confidence based on the face detector is sorted in descending order, and attributes with low prediction confidence are filtered out. In character granular alignment, Google's Tesseract OCR engine is applied to achieve character granular alignment by accurately recognizing and extracting text from images: ； Where Oc represents the concatenated sequence of English words extracted by the OCR model; concatenating Oc with C and D forms the text-visual alignment method of the visual context, represented as... At this stage, visual information is mapped to the text space and, after concatenating it with the text input T, forms the T+V input. A [SEP] marker is inserted between the T input and the visual context Vc. T+V and T are then processed by a Transformer-based model to obtain the final hidden representation H. L T+V With H L T It is fed into the CRF layer; for a label sequence y = (y1, y2, ..., yn), given the hidden representation H L The probability of the label sequence y is defined as follows: ； ； in, From label y j to y j+1 The randomly initialized transformation matrix, Indicates from H L Label y of linear transformation j The emission matrix, It is the hidden representation of the Lth layer. It is conditional probability; Step 2: Multi-scale visual fusion First, multi-scale visual features are captured and corresponding hierarchical visual representations are obtained by utilizing multi-granularity visual alignment visual data. Then, visual aspect and opinion supervision are performed. Finally, dynamic visual fusion based on prompt is achieved. Step 2 specifically involves using a four-block ResNet as the visual encoder and YOLOv5x6 as the object detector; retaining at most the three regions with the highest confidence scores. ; The multi-scale image input is fed into a visual encoder, where deep information is upsampled and shallow information is added element-wise; this process extracts multi-scale feature maps. Subsequently, average pooling is performed to enhance visual recognition capabilities within the image. ； ； in and This represents visual features obtained by fusing multi-scale feature maps, including global image features and object features; Ave represents the fusion of F... i Projected onto an average pooling layer of the same size; This represents the l-th multi-scale visual fusion feature; In step 2, an ANP detector is used as supervision for visual aspects and opinions. The distribution P of the first N predicted adjective-noun pairs is calculated using the first N predicted probabilities: ； Where r=4, and This represents the trainable parameters, and d represents the dimension of the text representation in BERT. To make the predicted distribution P closer to the true top-N adjective-noun pair distribution A, standard cross-entropy loss is used. To extract fine-grained information from image input: ； This loss can reduce the difference between the predicted distribution and the basic fact distribution; In step 2, a dynamic attention mechanism is used to project multi-layered visual information as a prompt onto the l-th layer of BERT in the text modality; multiple normalized vectors are calculated using a dynamic projector, and these vectors determine the degree of visual feature transformation for each block within BERT; first, the calculation... As a mapped semaphore: ； Wherein, MLP represents the feature dimensionality reduction layer; multi-head self-attention is used to combine the transformed multi-scale visual features with the key / value vectors of the context representation of each layer of BERT; Let k be the k-th multi-scale visual fusion feature, where This indicates that the transformed multi-scale visual features are input into the l-th layer of BERT; ； Represents the weight matrix. This indicates that the transformed features obtained by multi-scale visual fusion features through multi-head self-attention and dimensionality reduction are split into two parts; These transformed visual features are then concatenated with the original keyword and value vectors from the previous layer in BERT, which act as new keys and values ​​during the attention process; the computation of visual cues and text-based fusion attention is as follows: ； in, , as well as These represent the query, key, and value in the new attention matrix, respectively. Step 3: Text-guided multimodal training By minimizing the KL loss between the text output space and the text and image context output spaces, multimodal aspect-level sentiment analysis results are obtained.

2. The multi-modal aspect-level sentiment analysis method for multi-level fusion of images and text according to claim 1, characterized in that: In step 3, minimizing the KL-divergence of the probability distribution obtained by feeding the output into the BERT model is equivalent to calculating the cross-entropy loss between the two distributions: ； in and It is a probability distribution; loss function It is the negative log-likelihood function of the true label sequence, as shown below: ； final objective function The definition is as follows: ； in and These are hyperparameters used to control the contribution of each module.

Images