A multi-modal sentiment classification method based on contrastive learning and aspect enhancement
By employing a multimodal sentiment classification method based on contrastive learning and aspect enhancement, the challenge of fusing text and image information is addressed, achieving modality alignment and feature fusion, thereby improving the accuracy of multimodal sentiment classification.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- KUNMING UNIV OF SCI & TECH
- Filing Date
- 2023-10-17
- Publication Date
- 2026-07-14
AI Technical Summary
Existing technologies struggle to effectively integrate text and image information in multimodal sentiment analysis, leading to modal representation bias and semantic gaps that affect the accuracy of multimodal sentiment classification.
We employ a multimodal sentiment classification method based on contrastive learning and aspect enhancement. By constructing an aspect-guided multimodal contrastive learning module and a cross-modal feature fusion layer, we achieve modal alignment and feature fusion of text and images. We also optimize multimodal representation using triple contrastive learning and multimodal contrastive loss.
It significantly improves the accuracy of multimodal sentiment classification, bridges the semantic gap between text and images, and enhances the performance of multimodal sentiment classification.
Smart Images
Figure CN117407525B_ABST
Abstract
Description
Technical Field
[0001] This invention provides a multimodal sentiment classification method based on contrastive learning and aspect enhancement. By capturing aspect-enhanced multimodal information, it improves the performance of aspect-oriented multimodal sentiment classification and belongs to the field of natural language processing technology. Background Technology
[0002] Multimodal Aspect-Based Sentiment Analysis (MABSA) aims to integrate text and image information to accurately identify the sentiment polarity of specific aspects within a sentence. This technology has enormous application potential across various fields, including social media, healthcare, and education. One of the key challenges of this task is effectively fusing two distinct modalities—text and images—to analyze sentiment across aspects. In many downstream applications of MABSA, precisely matched image-text data pairs are often rare. This is due to the complementarity of image and text data, which creates a significant semantic gap between the two modalities. Therefore, accurately aligning text and images can be challenging. Aspect-based multimodal fusion aims to facilitate overall semantic fusion between text and images by using aspect representations as a cross-modal semantic fulcrum. How to enable networks to learn better aspect-oriented multimodal representations is one of the most critical issues in MABSA.
[0003] Many recent studies have attempted to address the challenge of visual-text alignment in MABSA by employing several aspect-oriented multimodal fusion strategies. However, most of these works tend to focus on aspect-oriented feature-level fusion by directly using visual, textual, and aspect representations as input, often ignoring the representational differences between different modalities. Significant representational differences exist between two modalities in different multimodal feature spaces. Therefore, directly fusing two different types of non-aligned modal representations inevitably leads to modal representation bias.
[0004] Modal alignment and feature-level fusion are two coupled problems that must be considered in the process of multimodal fusion. By properly aligning the representations of images and text in the modal space, it is easier to fuse their representations in the multimodal feature space. Alignment and fusion are two interdependent subtasks, both crucial for successful multimodal fusion. Furthermore, aspect representations, which often contain overlapping semantic information of images and text, are a suitable fulcrum for improving cross-modal alignment and feature fusion. To address the aspect-oriented cross-modal fusion problem that arises in MABSA, this invention proposes a multimodal sentiment classification method based on contrastive learning and aspect enhancement to facilitate multimodal fusion, which can significantly improve the accuracy of aspect-oriented sentiment polarity recognition. Summary of the Invention
[0005] This invention proposes a multimodal sentiment classification method based on contrastive learning and aspect enhancement to address the technical problems mentioned above. By bridging the modality gap, improving cross-modal alignment and feature fusion, and obtaining aspect-enhanced multimodal representations, the performance of multimodal sentiment classification is improved. The proposed method provides strong support for applications such as multimodal aspect-level sentiment analysis, modality alignment, and feature fusion.
[0006] The technical solution of this invention is: a multimodal sentiment classification method based on contrastive learning and aspect enhancement, wherein the specific steps of the multimodal sentiment classification method based on contrastive learning and aspect enhancement are as follows:
[0007] Step 1: Enter text, which includes specific aspects and related images;
[0008] Step 2: Obtain the features of text, specific aspects, and images;
[0009] Step 3: Construct an aspect-guided multimodal contrastive learning module to obtain aspect representations for text enhancement and image enhancement, and to achieve modal alignment;
[0010] Step 4: Establish an aspect-oriented cross-modal feature fusion layer with symmetrical cross-modal interaction to effectively fuse the aspect representations of text enhancement and image enhancement.
[0011] Step 5: Build a multimodal sentiment classifier for sentiment classification.
[0012] Furthermore, the specific steps of Step 2 are as follows:
[0013] Step 2.1: Use the Vit-Base-Patch16-2 pre-trained model to encode the image to obtain the corresponding image features;
[0014] Step 2.2: Use the Bert-base-uncased pre-trained model to encode the text and specific aspects to obtain text representations, which include text features and aspect features.
[0015] Furthermore, the specific steps of Step 3 are as follows:
[0016] Step 3.1: Input text features, aspect features and image features into the multimodal contrast learning module, and obtain text-enhanced aspect representation and image-enhanced aspect representation through aspect-text interaction and aspect-image interaction with aspect as the pivot.
[0017] Step 3.2: Employ aspect-guided triple contrastive learning to achieve modal alignment.
[0018] Furthermore, the specific steps of Step 4 are as follows:
[0019] Step 4.1: Perform cross-modal feature fusion on the text enhancement aspect representation and the image enhancement aspect representation obtained in Step 3.1;
[0020] Step 4.2: Perform multimodal comparative learning on the fused image and text multimodal features to promote feature fusion.
[0021] Furthermore, the specific steps of Step 5 are as follows:
[0022] Step 5.1: Input the multimodal representation obtained after multimodal contrastive learning in Step 4.2 into a traditional Transformer Encoder module;
[0023] Step 5.2: Use the softmax layer for final sentiment classification.
[0024] Furthermore, the specific steps of Step 3.1 are as follows:
[0025] Step 3.1.1: Using two stacked traditional cross-attention modules, the aspect-text interaction is captured by using the aspect and text representations Et and Es as the query and key / valuation, respectively.
[0026] H ts =LN(E t +CrossAtt(E t E s E s ))
[0027] in, It is the aspect representation of text enhancement, CrossAtt and LN represent the normalization operations of the cross attention module and layer; n and d represent the dimensions of the aspect representation matrix of text enhancement;
[0028] Step 3.1.2: Using two stacked traditional cross-attention modules, the aspect-image interaction is captured by treating the aspect and image representations as the query and key / valua, respectively.
[0029] H tv =LN(E t +CrossAtt(E t E v E v ))
[0030] in, It is an aspect representation of image enhancement, and CrossAtt and LN represent the normalization operations of the cross attention module and the layer.
[0031] Furthermore, the specific steps of Step 3.2 are as follows:
[0032] Step 3.2.1, Representing aspects of text enhancement H ts and aspects of image enhancement represented by H tv We will use an average to obtain an intermediate representation:
[0033]
[0034] in The aspect representing the average;
[0035] Step 3.2.2: Perform a triple contrastive learning operation of modality alignment on the text-enhanced aspect representation, the image-enhanced aspect representation, and the averaged aspect representation:
[0036]
[0037] in This indicates a triple comparison loss. Let represent the contrast loss between α and β, where α,β∈{H} ta H tv H ts}, This represents the following operation:
[0038]
[0039] in and This represents the following operation:
[0040]
[0041]
[0042] Where, α i and β j are the normalized representations of the i-th feature in α and the j-th feature in β, respectively, where N is the batch size and τ is the temperature parameter.
[0043] Furthermore, the specific steps of Step 4.1 are as follows:
[0044] Step 4.1.1: Obtain the text-enhanced aspect representation H ts and aspects of image enhancement represented by H tv This indicates that a mixup operation is being performed.
[0045] H s2v =w1H ts +(1-w1)H tv
[0046] H v2s =w2H tv +(1-w2)H ts
[0047] in, The shallow aspect representation shows that w1 and w2 are hybrid hyperparameters;
[0048] Step 4.1.2 Next, represent the shallow aspect H s2v H v2s The following operations are performed within the two-layer attention module that has a multi-head self-attention mechanism and a cross-attention layer:
[0049]
[0050]
[0051] SelfATT(·) and CrossATT(·) represent multi-head self-attention mechanism and cross-attention mechanism, respectively; The LN representation layer normalization operation represents visual-text augmentation aspect representation and text-visual augmentation aspect representation.
[0052] Furthermore, the specific steps of Step 4.2 are as follows:
[0053] Step 4.2.1: Represent the visual-text enhancement aspects and text-to-visual enhancement aspects Perform multimodal contrastive learning operations:
[0054]
[0055] in This represents the multimodal contrast loss. express and The contrast loss between them and This represents the following operation:
[0056]
[0057]
[0058] Where α represents β represents α i and β j are the normalized representations of the i-th feature in α and the j-th feature in β, respectively, where N is the batch size and τ is the temperature parameter.
[0059] Furthermore, the specific steps of Step 5.1 are as follows:
[0060] Step 5.1.1: Represent the visual-text enhancement aspects and text-to-visual enhancement aspects The last dimension is concatenated and fed into a single-layer Transformer encoder, as shown in the following operation:
[0061]
[0062] TransEnc c This represents a Transformer encoder layer, and concat represents the concatenation operation in the last dimension. Aspects representing visual and textual enhancements;
[0063] The specific steps of Step 5.2 are as follows:
[0064] Step 5.2.1 Then, the pooled aspect representation is fed into the softmax layer for sentiment classification:
[0065] p(y|H)=Softmax(Pooling(H a )W′)
[0066] Where Softmax represents the Softmax operation in the last dimension; pooling represents the average pooling operation in the length dimension. These are trainable parameters; the classification loss function is defined as:
[0067]
[0068] Where N represents the batch size;
[0069] Finally, by integrating classification losses Triple contrast loss Contrast loss with multimodal Three types of losses are used to obtain the total loss function, which serves as a measure of difference in classification. The total loss is expressed as:
[0070]
[0071] The hyperparameters λ1 and λ2 are used to balance different training losses.
[0072] The beneficial effects and advantages of this invention are:
[0073] 1. This invention proposes a multimodal sentiment classification method based on contrastive learning and aspect enhancement to solve the problems of modality alignment and feature fusion in multimodal aspect-level sentiment analysis. This invention can obtain aspect-enhanced multimodal representations to improve the performance of multimodal sentiment classification.
[0074] 2. This invention constructs an aspect-oriented modality alignment strategy with triple contrastive learning, which effectively bridges the semantic gap between images and text;
[0075] 3. This invention proposes an aspect-oriented cross-modal feature fusion layer with symmetric cross-modal interaction. By utilizing symmetric two-layer cross-modal interaction and aspect-enhanced multimodal contrastive learning, it obtains multimodal aspect representations, thereby enhancing the multimodal fusion capability.
[0076] 4. The method proposed in this invention achieves state-of-the-art (SOTA) results on both benchmark datasets of the MABSA task. Visualization experiments and case studies demonstrate the effectiveness and superiority of the method proposed in this invention. Attached Figure Description
[0077] Figure 1 This is a flowchart from the present invention;
[0078] Figure 2 This is a detailed flowchart of the multimodal sentiment classification method based on contrastive learning and aspect enhancement proposed in this invention. Detailed Implementation
[0079] The present invention will be further described below with reference to the accompanying drawings.
[0080] This invention addresses the challenges of cross-modal alignment and feature fusion in multimodal aspect-level sentiment analysis. It proposes a multimodal sentiment classification method based on contrastive learning and aspect enhancement. By utilizing an aspect-oriented modal alignment strategy with triple contrastive learning, it effectively bridges the semantic gap between images and text. A symmetrical cross-modal feature fusion layer with cross-modal interaction is constructed. Multimodal aspect representations are obtained through symmetrical two-layer cross-modal interaction and aspect enhancement multimodal contrastive learning, thereby enhancing the multimodal fusion capability.
[0081] like Figures 1-2 As shown, the specific steps of the multimodal sentiment classification method based on contrastive learning and aspect enhancement proposed in this invention are as follows:
[0082] Step 1: Enter text, which includes specific aspects and related images;
[0083] Step 2: Obtain the features of text, specific aspects, and images;
[0084] The specific steps of Step 2 are as follows:
[0085] Step 2.1: Use the Vit-Base-Patch16-2 pre-trained model to encode the image to obtain the corresponding image features;
[0086] Step 2.2: Use the Bert-base-uncased pre-trained model to encode the text and specific aspects to obtain text representations, which include text features and aspect features.
[0087] Step 3: Construct an aspect-guided multimodal contrastive learning module to obtain aspect representations for text enhancement and image enhancement, and to achieve modal alignment;
[0088] The specific steps of Step 3 are as follows:
[0089] Step 3.1: Input text features, aspect features and image features into the multimodal contrast learning module, and obtain text-enhanced aspect representation and image-enhanced aspect representation through aspect-text interaction and aspect-image interaction with aspect as the pivot.
[0090] The specific steps of Step 3.1 are as follows:
[0091] Step 3.1.1: Using two stacked traditional cross-attention modules, the aspect-text interaction is captured by using the aspect and text representations Et and Es as the query and key / valuation, respectively.
[0092] H ts =LN(E t +CrossAtt(E t E s E s ))
[0093] in, It is the aspect representation of text enhancement, CrossAtt and LN represent the normalization operations of the cross attention module and layer; n and d represent the dimensions of the aspect representation matrix of text enhancement;
[0094] Step 3.1.2: Using two stacked traditional cross-attention modules, the aspect-image interaction is captured by treating the aspect and image representations as the query and key / valua, respectively.
[0095] H tv =LN(E t +CrossAtt(E t E v E v ))
[0096] in, It is an aspect representation of image enhancement, and CrossAtt and LN represent the normalization operations of the cross attention module and the layer.
[0097] Step 3.2: Employ aspect-guided triple contrastive learning to achieve modal alignment.
[0098] The specific steps of Step 3.2 are as follows:
[0099] Step 3.2.1, Representing aspects of text enhancement H ts and aspects of image enhancement represented by H tv We will use an average to obtain an intermediate representation:
[0100]
[0101] in The aspect representing the average;
[0102] Step 3.2.2: Perform a triple contrastive learning operation of modality alignment on the text-enhanced aspect representation, the image-enhanced aspect representation, and the averaged aspect representation:
[0103]
[0104] in This indicates a triple comparison loss. Let represent the contrast loss between α and β, where α,β∈{H} ta H tv H ts}, This represents the following operation:
[0105]
[0106] in and This represents the following operation:
[0107]
[0108]
[0109] Where, α i and β j are the normalized representations of the i-th feature in α and the j-th feature in β, respectively, where N is the batch size and τ is the temperature parameter.
[0110] Step 4: Establish an aspect-oriented cross-modal feature fusion layer with symmetrical cross-modal interaction to effectively fuse the aspect representations of text enhancement and image enhancement.
[0111] The specific steps of Step 4 are as follows:
[0112] Step 4.1: Perform cross-modal feature fusion on the text enhancement aspect representation and the image enhancement aspect representation obtained in Step 3.1;
[0113] The specific steps of Step 4.1 are as follows:
[0114] Step 4.1.1: Obtain the text-enhanced aspect representation H ts and aspects of image enhancement represented by H tv This indicates that a mixup operation is being performed.
[0115] H s2v =w1H ts +(1-w1)H tv
[0116] H v2s =w2H tv +(1-w2)H ts
[0117] in, The shallow aspect representation shows that w1 and w2 are hybrid hyperparameters;
[0118] Step 4.1.2 Next, represent the shallow aspect H s2v H v2s The following operations are performed within the two-layer attention module that has a multi-head self-attention mechanism and a cross-attention layer:
[0119]
[0120]
[0121] SelfATT(·) and CrossATT(·) represent multi-head self-attention mechanism and cross-attention mechanism, respectively; The LN representation layer normalization operation represents visual-text augmentation aspect representation and text-visual augmentation aspect representation.
[0122] Step 4.2: Perform multimodal comparative learning on the fused image and text multimodal features to promote feature fusion.
[0123] The specific steps of Step 4.2 are as follows:
[0124] Step 4.2.1: Represent the visual-text enhancement aspects and text-to-visual enhancement aspects Perform multimodal contrastive learning operations:
[0125]
[0126] in This represents the multimodal contrast loss. express and The contrast loss between them and This represents the following operation:
[0127]
[0128]
[0129] Where α represents β represents α i and β j are the normalized representations of the i-th feature in α and the j-th feature in β, respectively, where N is the batch size and τ is the temperature parameter.
[0130] Step 5: Build a multimodal sentiment classifier for sentiment classification.
[0131] The specific steps of Step 5 are as follows:
[0132] Step 5.1: Input the multimodal representation obtained after multimodal contrastive learning in Step 4.2 into a traditional Transformer Encoder module;
[0133] The specific steps of Step 5.1 are as follows:
[0134] Step 5.1.1: Represent the visual-text enhancement aspects and text-to-visual enhancement aspects The last dimension is concatenated and fed into a single-layer Transformer encoder, as shown in the following operation:
[0135]
[0136] TransEnc c This represents a Transformer encoder layer, and concat represents the concatenation operation in the last dimension. Aspects representing visual and textual enhancements;
[0137] Step 5.2: Use the softmax layer for final sentiment classification.
[0138] The specific steps of Step 5.2 are as follows:
[0139] Step 5.2.1 Then, the pooled aspect representation is fed into the softmax layer for sentiment classification:
[0140] P(y|H)=Softmax(Pooling(H a )W′)
[0141] Where Softmax represents the Softmax operation in the last dimension; pooling represents the average pooling operation in the length dimension. These are trainable parameters; the classification loss function is defined as:
[0142]
[0143] Where N represents the batch size;
[0144] Finally, by integrating classification losses Triple contrast loss Contrast loss with multimodal Three types of losses are used to obtain the total loss function, which serves as a measure of difference in classification. The total loss is expressed as:
[0145]
[0146] The hyperparameters λ1 and λ2 are used to balance different training losses.
[0147] To demonstrate the effectiveness of the proposed method, extensive experiments were conducted on two benchmark datasets for the MABSA task: Twitter2015 and Twitter2017. Details of the Twitter2015 and Twitter2017 datasets are shown in Table 1. To accurately evaluate the method's performance, accuracy (Acc) and Macro-F1 score (F1) were used as evaluation metrics. These two metrics can objectively and accurately reflect the effectiveness of sentiment analysis. The experimental results are shown in Table 2.
[0148] Table 1 shows the details of the two MABSA benchmark datasets.
[0149]
[0150] Comparison results of the two benchmark datasets:
[0151] Table 2 shows the comparison results of the two MABSA benchmark datasets.
[0152]
[0153]
[0154] The results were compared on two MABSA benchmark datasets to demonstrate the effectiveness of the proposed method, as shown in Table 2. It can be seen that (1) the proposed method outperforms other state-of-the-art (SOTA) MABSA methods on both datasets, with F1 scores improved by 1.60% and 1.05%, respectively. (2) Compared to unimodal methods, this method significantly improves upon all unimodal methods (such as pure text or pure image ABSA methods). Experimental results demonstrate the effectiveness of text and visual modalities in emotion classification. (3) Compared to other MABSA methods, the proposed method significantly outperforms other SOTA MABSA algorithms, confirming the advantages of the proposed model algorithm.
[0155] The specific embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the above embodiments. Within the scope of knowledge possessed by those skilled in the art, various changes can be made without departing from the spirit of the present invention.
Claims
1. A multimodal sentiment classification method based on contrastive learning and aspect enhancement, characterized in that, Includes the following steps: Step 1: Enter text, which includes specific aspects and related images; Step 2: Obtain the features of text, specific aspects, and images; Step 3: Construct an aspect-guided multimodal contrastive learning module to obtain aspect representations for text enhancement and image enhancement, and to achieve modal alignment; Step 4: Establish an aspect-oriented cross-modal feature fusion layer with symmetrical cross-modal interaction to effectively fuse the aspect representations of text enhancement and image enhancement. Step 5: Construct a multimodal sentiment classifier for sentiment classification; The specific steps of Step 3 are as follows: Step 3.1: Input text features, aspect features and image features into the multimodal contrast learning module, and obtain text-enhanced aspect representation and image-enhanced aspect representation through aspect-text interaction and aspect-image interaction with aspect as the pivot. Step 3.2: Employ aspect-guided triple contrastive learning to achieve modal alignment; The specific steps of Step 4 are as follows: Step 4.1: Perform cross-modal feature fusion on the text enhancement aspect representation and the image enhancement aspect representation obtained in Step 3.1; Step 4.2: Perform multimodal contrastive learning on the fused image and text multimodal features to promote feature fusion; The specific steps of Step 5 are as follows: Step 5.1: Input the multimodal representation obtained after multimodal contrastive learning in Step 4.2 into a traditional Transformer Encoder module; Step 5.2: Perform final sentiment classification using a softmax layer; The specific steps of Step 3.1 are as follows: Step 3.1.1: Using two stacked traditional cross-attention modules, the aspect-text interaction is captured by using the aspect and text representations Et and Es as the query and key / valuation, respectively. ； in, It is an aspect of text enhancement. LN represents the normalization operation of the cross-attention module and layer; n and d represent the aspects of text enhancement and the dimensions of the matrix. Step 3.1.2: Using two stacked traditional cross-attention modules, the aspect-image interaction is captured by treating the aspect and image representations as the query and key / valua, respectively. ； in, It is an aspect of image enhancement. LN represents the normalization operation for cross-attention modules and layers; The specific steps of Step 3.2 are as follows: Step 3.2.1, Aspects of text enhancement and aspects of image enhancement We will use an average to obtain an intermediate representation: ； in The aspect representing the average; Step 3.2.2: Perform a triple contrastive learning operation of modality alignment on the text-enhanced aspect representation, the image-enhanced aspect representation, and the averaged aspect representation: ； in This indicates a triple comparison loss. express and The difference in loss between them, and This represents the following operation: ； in and This represents the following operation: ； ； in, and They are The i-th feature and The normalized representation of the j-th feature, where N is the batch size. It is a temperature parameter; The specific steps of Step 4.1 are as follows: Step 4.1.1: Obtain the aspect representation of text enhancement. and aspects of image enhancement This indicates that a mixup operation is being performed. ； ； in, This indicates a shallow aspect. and It is a mixture of hyperparameters; Step 4.1.2 Next, represent the shallow aspects. The following operations are performed within the two-layer attention module that has a multi-head self-attention mechanism and a cross-attention layer: ； ； in and This refers to the multi-head self-attention mechanism and the cross-attention mechanism; Normalization operation of LN representation layer for visual-text augmentation aspect representation and text-visual augmentation aspect representation; The specific steps of Step 4.2 are as follows: Step 4.2.1: Represent the visual-text enhancement aspects and text-to-visual enhancement aspects Perform multimodal contrastive learning operations: ； in This represents the multimodal contrast loss. express and The contrast loss between them and This represents the following operation: ； ； in express , express , and They are The i-th feature and The normalized representation of the j-th feature, where N is the batch size. It is a temperature parameter.
2. The multimodal sentiment classification method based on contrastive learning and aspect enhancement as described in claim 1, characterized in that: The specific steps of Step 2 are as follows: Step 2.1: Use the Vit-Base-Patch16-2 pre-trained model to encode the image to obtain the corresponding image features; Step 2.2: Use the Bert-base-uncased pre-trained model to encode the text and specific aspects to obtain text representations, which include text features and aspect features.
3. The multimodal sentiment classification method based on contrastive learning and aspect enhancement as described in claim 1, characterized in that: The specific steps of Step 5.1 are as follows: Step 5.1.1: Represent the visual-text enhancement aspects and text-to-visual enhancement aspects The last dimension is concatenated and fed into a single-layer Transformer encoder, as shown in the following operation: ； in This represents a Transformer encoder layer; concat represents the concatenation operation in the last dimension. Aspects representing visual and textual enhancements; The specific steps of Step 5.2 are as follows: Step 5.2.1 Then, the pooled aspect representation is fed into the softmax layer for sentiment classification: ； Where Softmax represents the Softmax operation in the last dimension; pooling represents the average pooling operation in the length dimension. These are trainable parameters; the classification loss function is defined as: ； Where N represents the batch size; Finally, by integrating classification losses Triple contrast loss Contrast loss with multimodal Three types of losses are used to obtain the total loss function, which serves as a measure of difference in classification. The total loss is expressed as: ； hyperparameters and Used to balance different training losses.