A multi-modal sarcasm detection method based on fact and sentiment opposition

Abstract

The application discloses a kind of based on fact and emotional antagonism multi-modal irony detection method, step 1, obtain the data tuple including text and image to be detected, obtain image feature data and text feature data by two feature extraction branches;Step 2, realize feature alignment processing using implicit cross-modal attention mechanism;Step 3, realize the semantic and emotional inconsistency modeling of multi-modal data;Step 4, calculate semantic inconsistency;Step 5, obtain the difference of image feature and text feature emotional polarity;Step 6, based on the difference of image feature and text feature emotional polarity and inconsistency value carries out irony detection, obtains detection result.Compared with prior art, the application shows excellent prediction performance on multi-modal irony detection dataset, exceeds all existing methods.The application can accurately detect whether given image belongs to satirical emotional category.

Description

Technical Field

[0001] This invention belongs to the fields of natural language processing and computer vision processing technology, and in particular relates to a method for multimodal ironic sentiment detection based on the opposition between facts and emotions. Background Technology

[0002] Sarcasm is a humorous and popular way to express user opinions, often meaning that the expressed attitude is the opposite of the literal meaning. Due to the development of social media, multimodal data combining text and images has become the mainstream method for users to express sarcasm, and sarcasm detection (SD) plays an important role in applications such as product evaluation analysis and opinion mining.

[0003] In the early stages, researchers primarily studied satire evoked by textual data. Since online data such as Twitter often contains category labels, these labels were used to build a natural language satire corpus. Based on the information from this constructed satire corpus, researchers explored tasks such as detecting satire in textual data and mining opinions and sentiments.

[0004] Due to the widespread availability of image-text multimodal data, multimodal sarcasm detection (MSD) has received increasing attention in recent years. Unlike unimodal sarcasm detection, mining the relationships between modalities is a key issue in MSD. Some early researchers analyzed handcrafted features and deep representations of images and text, employing feature concatenation methods for multimodal prediction. Later, attention-based modal fusion methods became the main research direction in MSD, such as using multi-level fusion strategies to deeply fuse image and text representations. Influenced by the significant success of the Transformer self-attention model, cross-modal self-attention modules were used to discover the correlation between image and text modalities. To better utilize the mapping relationship between images and text for each instance, many researchers have focused on graph network-based modeling methods. Further, by utilizing visual question answering (VQA) algorithms to output bounding boxes of objects in the image for fine-grained cross-modal matching, and constructing cross-modal graph networks, the sarcasm-related interaction information between multimodalities can be grasped.

[0005] Existing techniques implicitly model intermodal inconsistencies through intermodal interactions, but the exact knowledge learned about these inconsistencies remains unknown. Summary of the Invention

[0006] Based on the aforementioned existing technologies and inspired by human perception processes, this invention proposes a multimodal irony detection method based on the opposition between facts and emotions. Based on a dual network perception structure, it achieves explicit mining of semantic inconsistencies in ironic data from both factual and emotional perspectives, thereby realizing a multimodal irony sentiment detection method.

[0007] To achieve the above-mentioned objectives, the present invention adopts the following technical solution:

[0008] A multimodal irony detection method based on factual and emotional opposition, the system comprising the following steps

[0009] Step 1: Obtain the data tuple to be detected, including text and images, and obtain image feature data and text feature data through two feature extraction branches;

[0010] Step 2: The image feature data and text feature data obtained in Step 1 are taken as two modalities respectively. The implicit cross-modal attention mechanism is used to realize the feature alignment between the two modalities, so as to obtain aligned image features and aligned text features, which are then merged into the same feature space.

[0011] Step 3: Based on the aligned image features and aligned text feature distributions from Step 2, model the semantic and sentiment inconsistencies of multimodal data;

[0012] Step 4: Input the aligned image features and aligned text features into the semantic inconsistency model of Step 3. Process the model to obtain enhanced satirical image features and enhanced satirical text features. Calculate the probability that the similarity of the obtained enhanced image features and enhanced text features belongs to the Gaussian distribution of satirical data and the Gaussian distribution of non-satirical data, respectively, and calculate the semantic inconsistency.

[0013] Step 5: Introduce emotional information from the emotional dictionary, further enhance the emotional features of the image through continuous comparative learning, and obtain the difference between the emotional polarity of the image features and the text features as the emotional inconsistency.

[0014] Step 6: Based on the differences and inconsistencies in the semantic and emotional polarities of the image features and text features from Steps 4 and 5, perform ironic sentiment detection to obtain the detection results.

[0015] Compared with the prior art, the present invention can achieve the following beneficial technical effects:

[0016] 1) By leveraging the opposition between fact and sentiment in multimodal data, a dual-perception network was constructed to detect multimodal irony data through modeling the display of inconsistencies;

[0017] 2) A channel-weighted correction method for factual semantic inconsistency is proposed, enhancing the image-text features related to satire. By performing Gaussian modeling on satirical and non-satirical data and calculating the distance between the sample and the two distributions, factual inconsistency is adaptively calculated;

[0018] 3) By introducing unlabeled sentiment information through a sentiment dictionary, a method of continuous contrastive learning of sentiment is proposed to further enhance the sentiment features of images. Sentiment opposition is modeled using the polarity difference between text and image sentiment.

[0019] 4) In the prediction stage, multimodal image and text data are fed into two modules respectively, and the output features are used to fuse them to determine whether it is satirical;

[0020] 5) It demonstrates excellent prediction performance on the multimodal irony detection dataset, surpassing all existing methods. For a given image, this invention can accurately detect whether it belongs to the irony sentiment category. Attached Figure Description

[0021] Figure 1 This is an overall flowchart of a multimodal irony detection method based on factual and emotional opposition according to the present invention;

[0022] Figure 2 This is a schematic diagram of a multimodal irony detection method based on factual and emotional opposition according to the present invention;

[0023] Figure 3 This is a visual representation of the prediction results of the present invention. Detailed Implementation

[0024] The present invention will be further described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0025] like Figure 1 The diagram shows the overall flowchart of a multimodal irony detection method based on factual and emotional opposition according to the present invention. The specific implementation steps are as follows:

[0026] Step 1: Obtain the data tuple to be detected, which includes text and images. Image feature data is obtained through image feature extraction, and text feature data is obtained through text feature extraction. The specific explanations of the two feature extraction branches are as follows:

[0027] Step 1-1: Use the ViT network as the backbone network for image feature extraction. The ViT network is based on the Transformer structure and mainly consists of a self-attention module, a multi-head attention module, and a multilayer perceptron. The specific image feature extraction process is as follows: Given a batch of RGB images, use bilinear interpolation to scale them to 256×256 pixels, crop the images into 196 image blocks, add positional encoding, and then feed them into the ViT network to obtain the encoded image features;

[0028] Steps 1-2: Use the BERT model as the backbone network for text feature extraction. The BERT model is also mainly based on the Transformer structure, including a self-attention module, a multi-head attention module, and a multilayer perceptron. The specific text feature extraction process is as follows: Given a batch of text data, first, the text length is padded using a padding strategy, and then word segmentation is performed. Each word segment is converted into a corresponding code and then fed into the BERT model to obtain the text feature code.

[0029] Step 2: The image and text feature data obtained from the two feature extraction branches are treated as two modalities. An implicit cross-modal attention module (CAM) is used to align the features between the two modalities, allowing them to acquire interactive information and merge into the same feature space. Common explicit alignment processes require that the representations of image and text features remain consistent through loss; however, irony detection relies on contradictions between modalities, and such strong constraints compromise the potential inconsistencies within the representations. For example... Figure 2 The diagram shown illustrates the cross-modal attention mechanism model of this invention. Both the input and output are image or text feature maps. The specific operation is described below:

[0030] Step 2-1: For a given image feature e p ∈R m×C and a text feature e w ∈R k×C First, the features are multiplied by the spatial dimension to obtain a two-dimensional relation matrix RM∈R. m×k The expression is as follows:

[0031]

[0032] Here, T represents transpose, and Conv indicates that the convolution operation is implemented by two convolutional layers. The larger the value of RM, the stronger the correlation of the features.

[0033] Step 2-2: For the visual modality, calculate the inter-channel variance of image features in each batch of data, and generate the image activation vector v. p , activate the image vector vp Perform a dot product with the original image features to obtain the aligned image features e. ap e ap =e p ·v p Similarly, the text activation vector v w Performing a dot product with the original text features generates aligned text features e. aw e aw =e wp ·v w ;

[0034] Step 3: Modeling semantic and sentiment inconsistencies in multimodal data based on aligned image features and aligned text feature distributions; counterfactual reasoning is crucial for understanding irony. Facts describe the existence of objects or events, which can often be perceived through semantic information. The semantic inconsistency modeling module aims to strengthen invariant representations that lead to irony and utilize distributions to model inconsistencies in multimodal data;

[0035] Step 4: Align the image features e ap and aligned text features e aw The input semantic and sentiment inconsistency model is then processed as follows:

[0036] By using weight adjustment, the image and text features most relevant to satire are found. After obtaining discriminative semantic image and text features of satire, the similarity distribution between satirical and non-satirical samples is maintained. Gaussian modeling is applied to the data to determine the probability of semantic inconsistency in the current sample. The specific processing is as follows:

[0037] Step 4-1: Introduce a channel-level weighted strategy to learn satirical representations. The motivation for this strategy is the observation that some image-text regions are related but unrelated to the satirical object. Specifically, as the semantic and sentiment inconsistency model is trained, satirical-related representations are gradually activated by the loss function, and these activated features vary significantly across different instances. This invention leverages this characteristic by enhancing image features with larger channel-dimensional variance to strengthen satirical-related content, as shown in the following expression:

[0038] r p =e ap ·σ(ReLu(FC(e ap (2)

[0039] Where, r p This represents the irony-related feature representation of the enhanced image, where σ represents the variance of the maintained Gaussian distribution, and e ap For implicitly aligned image features, ReLu() represents the activation function, and FC() represents the fully connected layer.

[0040] Similarly, the enhanced text feature invariant representation r is obtained using a channel-level weighting strategy. w The expression is as follows:

[0041] r w =e aw ·σ(ReLU(FC(e aw ))) (3)

[0042] Where, r w This represents the enhanced text features related to irony, where σ represents the variance of the maintained Gaussian distribution, and e aw For the aligned features of this paper, ReLu() represents the activation function and FC() represents the fully connected layer.

[0043] Step 4-2: During the training process, set up two memory queues. The enhanced image and text features belonging to satirical data and the enhanced image and text features belonging to non-satirical data from past data are stored separately. Gaussian distributions are used for modeling, and the two Gaussian distributions are denoted as the satirical data Gaussian distribution D. s Gaussian distribution of non-ironic data D ns .

[0044] The mean and variance values ​​μ and σ of the Gaussian distribution maintained by the satirical data are obtained as follows:

[0045]

[0046]

[0047] Where Sim represents the cosine similarity function, and m represents the number of data points in this batch. This represents the enhanced image features and enhanced text features that belong to the category of irony.

[0048] In equations (4) and (5) It can also be replaced with The enhanced image features and enhanced text features represent non-ironic data; specifically as follows:

[0049] Furthermore, the mean and variance μ of the Gaussian distribution maintained by the non-ironic data are obtained. n and σ n The expression is as follows:

[0050]

[0051]

[0052] Where Sim represents the cosine similarity function, and m represents the number of data points in a batch. This represents the enhanced image features and enhanced text features belonging to non-ironic data;

[0053] The similarity between the obtained enhanced image features and the enhanced text features is calculated to determine whether they belong to the Gaussian distribution D of the satirical data. s Gaussian distribution of non-ironic data D ns The probability P s P ns Define semantic inconsistency λ SID The expression is as follows:

[0054] λ SID =|P ns -P s | (6)

[0055] Step 5: After adding sentiment information based on text features using the Sentiment Dictionary SenticNet, the difference between the sentiment polarity of image features and text features is obtained. This step designs a continuous contrastive learning method to enhance the learned sentiment representation; specifically, it includes the following processing:

[0056] Step 5-1: SenticNet is a widely used sentiment dictionary that can output word-level continuous sentiment polarity values. Introducing unlabeled sentiment information, a sentiment continuity contrastive learning method is proposed to further enhance image sentiment features. We explicitly model sentiment opposition using the polarity difference between image and text sentiment. This is achieved through implicitly aligned text features e... aw The input is fed into a Siamese network layer to predict the sentiment label for each word; specifically, this Siamese network layer consists of a projection network layer for extracting sentiment embeddings and a classifier for obtaining sentiment polarity values, and then the text sentiment loss function L is calculated using MSE loss. ts The expression is as follows:

[0057]

[0058] Where, p i q represents the predicted value. i This represents the sentiment polarity value output in SenticNet;

[0059] Emotional information plays an important role in the detection of multimodal sarcasm;

[0060] Step 5-2: Since the features are already implicitly aligned in the cross-modal attention module, this invention also utilizes a projection layer and classifier with shared parameters to process the image. To further enhance the image sentiment representation, this invention proposes a continuous contrastive learning strategy, constructing continuous supervised labels to clearly describe the intensity of sentiment polarity. For image-text pairs, a large polarity difference means that features should be pushed apart accordingly; otherwise, they should be brought closer. Therefore, a sentiment association graph is constructed between the image and text predictions. The expression is as follows:

[0061]

[0062] Where i and j are both indices of the feature vector. Let represent the predicted value of the i-th image feature. Let represent the predicted value of the j-th text feature, and the similarity matrix of the features. It can be calculated by the following formula:

[0063]

[0064] in, and Let i and j represent the image and text features output by the projection layer, respectively, where i and j are indices of the feature vectors. The loss function for continuous graph contrastive learning is calculated using Kullback-Leible (KL) divergence.

[0065]

[0066] The difference λ between the sentiment polarity of image features and text features is obtained. SSC The expression is as follows:

[0067] λ SSC =|p v -p t | (11)

[0068] Where, p v p is the predicted value of the image features. t The predicted value for text features;

[0069] Step 6: Calculate the difference λ between image features and text features in terms of sentiment polarity. SSC and inconsistency value λ SID To predict the ironic sentiment, the detection result y is obtained, expressed as follows:

[0070] y = sigmoid(y f +λ SID +λ SSC (12)

[0071] Where sigmoid() represents the activation function, y f The value is the predicted value obtained by fusing image and text features and passing it through a classification layer.

[0072] The multimodal model based on factual and emotional inconsistency proposed in this invention is mainly implemented based on image and text encoders, semantically enhanced distributed modeling module, and twin emotion contrast learning module.

[0073] Table 1 shows the evaluation effect of the present invention on actual image and text data. It demonstrates the performance of the present invention on a multimodal satirical dataset.

[0074] Table 1

[0075]

[0076]

[0077] The method of this invention significantly outperforms unimodal irony detection methods. Utilizing complementary information from multimodal data, this invention improves accuracy by 21.76% and 5.74% compared to visually optimal and text-optimal methods, respectively. On one hand, detecting irony expressed in text with richer semantics is relatively more effective than using only image sentiment data. On the other hand, images, as important content for expressing irony, can significantly improve the performance of multimodal irony detection.

[0078] Compared to the optimal multimodal method, DIP achieves improved accuracy of 2.28%, 1.18%, and 2.04% across the three backbone network implementations, respectively. In terms of mean accuracy, the DIP method outperforms the optimal CMGCN method by 4.13%. This result demonstrates that DIP is particularly adept at identifying data categorized as satirical. Furthermore, our Macro mean metric is also improved by at least 1.44%, proving the effectiveness of this invention in distinguishing between satirical and non-satirical data. Moreover, compared to previous methods that implicitly model inconsistencies, the accuracy of this invention is improved by more than 2%. Therefore, the explicit modeling of factual and sentiment inconsistencies proposed in this invention is more effective for multimodal satire detection.

[0079] Finally, it should be noted that the above description is merely a preferred embodiment of the present invention. Any modifications made by those skilled in the art under the guidance of the present invention, without departing from the spirit and claims of the invention, are considered to be understandable to those skilled in the art. Therefore, the present invention is not limited to the above embodiments, and all variations and modifications to the technical solutions of the present invention fall within the scope of protection of this application.

Claims

1. A multimodal irony detection method based on factual and emotional opposition, characterized in that, The method specifically includes the following steps: Step 1: Obtain the data tuple to be detected, which includes text and images. Obtain image feature data and text feature data through two feature extraction branches. Step 2: The image feature data and text feature data obtained in Step 1 are taken as two modalities respectively. The implicit cross-modal attention mechanism is used to realize the feature alignment between the two modalities, so as to obtain aligned image features and aligned text features, which are then merged into the same feature space. Step 3: Based on the aligned image features and aligned text features in Step 2, use distribution to model the semantic and sentiment inconsistencies in multimodal data to strengthen the invariant representations that lead to irony. Step 4: Input the aligned image features and aligned text features into the channel-level weighting module for processing to obtain the enhanced image features and enhanced text features. Calculate the similarity between the enhanced image features and the enhanced text features. Based on the Gaussian distribution of satirical data and the Gaussian distribution of non-satirical data constructed during training, calculate the probability that the similarity belongs to the two distributions respectively, and use the absolute difference between the two probabilities as the semantic inconsistency value. Step 5: Introduce emotional information from the emotional dictionary, further enhance the emotional features of the image through continuous comparative learning, and obtain the difference between the emotional polarity of the image features and the text features as the emotional inconsistency. Step 6: Based on the difference and inconsistency values ​​of the emotional polarity between the image features and text features from Step 5, perform irony detection to obtain the detection results.

2. The multimodal irony detection method based on factual and emotional opposition as described in claim 1, characterized in that, Step 2 of the cross-modal attention mechanism further includes the following processing: Step 2-1: For a given image feature and a text feature By performing dot products on the features according to the spatial dimension, a two-dimensional relation matrix is ​​obtained. The expression is as follows: （1）； in, Indicates transpose. Indicates the convolution operation; Step 2-2: For the visual modality, sum the values ​​of each image feature relative to the text feature to generate an image activation vector. Image activation vector Perform a dot product with the original image features to obtain aligned image features. , Similarly, the image activation vector Perform a dot product with the original text features to generate aligned text features. , .

3. The multimodal irony detection method based on factual and emotional opposition as described in claim 1, characterized in that, Step 4 further includes the following processing: Step 4-1: Introduce a channel-level weighting strategy to learn ironic representations and obtain enhanced image features, expressed as follows: （2）； in, This represents the irony-related features of the enhanced image. This represents the variance of the maintained Gaussian distribution. Image features that are implicitly aligned. This represents the activation function. This represents a fully connected layer; Similarly, channel-level weighting strategies are used to obtain enhanced text feature invariant representations. The expression is as follows: （3）； in, This indicates enhanced textual features related to irony. This represents the variance of the maintained Gaussian distribution. For the alignment of text features, This represents the activation function. Indicates a fully connected layer; Step 4-2: Set up two memory queues during training. , The system stores the enhanced image and text features of satirical data and the enhanced image and text features of non-satirical data from past data, respectively. Gaussian distributions are used for modeling, and the two Gaussian distributions are denoted as the satirical data Gaussian distributions. Gaussian distribution of non-ironic data ; Obtain the mean and variance of the Gaussian distribution maintained by the satirical data. and The expression is as follows: （4-1）； （5-1）； Among them, S Let m represent the cosine similarity function, and m represent the number of data points in a batch. This represents the enhanced image features and enhanced text features belonging to satirical data; And, obtain the mean and variance values ​​of the Gaussian distribution maintained by the non-ironic data. and The expression is as follows: （4-2）； （5-2）； Among them, S Let m represent the cosine similarity function, and m represent the number of data points in a batch. This represents the enhanced image features and enhanced text features belonging to non-ironic data; The similarity between the obtained enhanced image features and the enhanced text features is calculated to determine whether they belong to a Gaussian distribution of satirical data. Gaussian distribution of non-ironic data probability Semantic inconsistency in definition The expression is as follows: （6）。 4. The multimodal irony detection method based on factual and emotional opposition as described in claim 3, characterized in that, Step 5 further includes the following processing: Step 5-1: Using implicitly aligned text features The input is fed into a Siamese network layer to predict the sentiment label for each word; specifically, this Siamese network layer consists of a projection network layer for extracting sentiment embeddings and a classifier for obtaining sentiment polarity values, and then the text sentiment loss function is calculated using MSE loss. The expression is as follows: （7）； in, Indicates the predicted value. This represents the sentiment polarity value output in SenticNet; Step 5-2: Based on the strategy of continuous contrastive learning, construct continuous supervised labels to clearly describe the intensity of sentiment polarity, and construct a sentiment correlation graph between image and text prediction values. The expression is as follows: （8）； in, and These are all indices of the feature vector. Indicates the first Predicted values ​​of image features, Indicates the first Predicted values ​​of text features, and the similarity matrix of the features. It can be calculated by the following formula: ； in, and These represent the image features and text features output by the projection layer, respectively. and These are all indices of the feature vectors; The loss function for continuous graph contrast learning is calculated using Kullback-Leible (KL) divergence, serving as supervision for image feature learning. （9）； The difference between the sentiment polarity of image features and text features is obtained. The expression is as follows: （10）； in, These are the predicted values ​​for image features. These are the predicted values ​​for text features.

5. The multimodal irony detection method based on factual and emotional opposition as described in claim 4, characterized in that, In step 6, the detection result The expression is as follows: ； in, This represents the activation function. The value is the predicted value obtained by fusing image and text features and passing it through a classification layer.

Images