A multi-modal sarcasm recognition system fusing relational discrepancy information

Abstract

The application relates to a multi-modal sarcasm recognition system fusing relationship difference information. The system comprises a graphic-text feature extraction module, a graphic-text relationship fusion module, a graphic-text difference fusion module and a sarcasm recognition module. The graphic-text feature extraction module is used for extracting feature representation of graphic-text data; the graphic-text relationship fusion module is used for matching and combining text image consistent information for prediction; the graphic-text difference fusion module is used for extracting inconsistency information in the text in the mode, extracting inconsistency information between the graphic and the text in the mode, and combining the inconsistency information for prediction; and the sarcasm recognition module is used for judging whether sarcasm exists in the multi-modal data.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of multi-modal sarcasm recognition, and more particularly, to a multi-modal sarcasm recognition system fusing relationship difference information. BACKGROUND

[0002] Sarcasm is a complex linguistic behavior and also a special way of expressing emotions. People often use exaggeration, metaphor and other methods to express the content opposite to the real meaning. With the continuous enrichment of the cultural attributes of social platforms, the meaning of tweets published by users is usually not limited to the surface, and sarcasm is one of the expression methods frequently used by users. In the current user viewpoint mining of social platforms, there are great limitations in analyzing user viewpoints and attitudes by using only traditional sentiment analysis and viewpoint mining methods, and it is difficult to effectively identify the inconsistent information implied in the viewpoints. Therefore, the detection method for identifying sarcasm is beneficial to analyzing the real viewpoints and attitudes of users and improving the accuracy of sentiment analysis and viewpoint mining tasks.

[0003] At the same time, with the continuous development of social media and network culture, people are not limited to using text as the only way to express their own viewpoints, and multi-modal tweets such as text, pictures and videos are widely published on various social platforms. Among them, there are not a few tweets that achieve the effect of sarcasm through the combination of text and images. In the multi-modal context, sarcasm is no longer a purely linguistic phenomenon, and due to the nature of social media short text, the information of the text is not always sufficient, and the opposite relationship is more expressed through cross-modal means. Therefore, it is not enough to judge sarcasm in multi-modal information from the perspective of text analysis alone.

[0004] In recent years, the research on multi-modal fusion technology has developed rapidly. The multi-modal fusion task is to design a modal fusion strategy to make the data of different modalities complement each other and provide more rich information for downstream tasks. At present, sentiment analysis, image-text retrieval and other tasks based on multi-modal fusion technology are becoming mature and widely used.

[0005] In the field of sarcasm research, compared with traditional pure text sarcasm detection, multi-modal sarcasm detection can better adapt to social media platforms with rich data types. Sarcasm is a method of implicit emotional expression, and identifying sarcasm requires in-depth understanding of the semantic information of the data. Pure text sarcasm detection generally lacks context information, resulting in poor recognition effect. Multi-modal sarcasm detection can supplement the interaction clues of multi-modal information, enrich the context information, and detect the multi-level relationship contrast of sarcasm tasks to achieve a relatively ideal effect. As a downstream task of multi-modal fusion, the research focus of multi-modal sarcasm recognition is around the semantic information of each modality, and the potential relationship between the modal data is analyzed to provide more rich context information for the sarcasm recognition model. The current multi-modal sarcasm recognition method applied to social media first extracts the feature representation of image and text according to the characteristics of each modal data, and then uses attention mechanism, graph neural network and other deep learning algorithms to fuse the data of different modalities, and better utilizes the data correlation and difference information of each modality. The existing mainstream multi-modal sarcasm model has carried out a lot of research on multi-modal fusion strategy, which can be divided into front-end fusion, middle-end fusion and back-end fusion. These methods can fully utilize the information between modalities for sarcasm detection. However, the semantic information of sarcasm not only comes from the incongruity generated by analyzing the information between modalities after fusing the image and text data, but also may exist in the text itself or the picture itself, that is, the sarcasm information within the modalities. Therefore, while fusing the data between image and text modalities, how to explore the inconsistent information within the image and text modalities to improve the sarcasm recognition accuracy is the main challenge of multi-modal sarcasm recognition task. SUMMARY

[0006] The purpose of the present application is to provide a multi-modal sarcasm recognition system that fuses relationship difference information, based on image and text dual modal data in the field of social media, and performs hierarchical feature extraction analysis on the content of each modality, including topic data extracted from the text modality, text data and object information recognized from the image modality. At the same time, for different modal information, in the aspect of modal relationship fusion, a graph-text relationship fusion module based on self-attention mechanism is designed; in the aspect of modal difference fusion, a graph-text difference fusion module based on the combination of gating mechanism and alienation matrix is designed; finally, the relationship and difference information of each modal data is fused by using splicing mechanism, and the fused result is applied to sarcasm recognition.

[0007] To achieve the above purpose, the technical scheme of the present application is as follows: a multi-modal sarcasm recognition system that fuses relationship difference information, comprising:

[0008] A graph-text feature extraction module for extracting feature representation of graph-text data;

[0009] a text-image relationship fusion module for matching and combining consistent information of text and image for prediction;

[0010] a text-image difference fusion module for extracting inconsistent information in text of the same modality, extracting inconsistent information between text and image of different modalities, and combining inconsistent information for prediction;

[0011] a sarcasm identification module for determining whether sarcasm exists in the multi-modal data.

[0012] In an embodiment of the present application, the text-image feature extraction module extracts features of primary information of the image modality, the text modality, and the topic modality.

[0013] In an embodiment of the present application, the text-image relationship fusion module, by referring to the idea of multi-head attention mechanism, designs an image-text matching layer to capture and fuse consistent information between text and image.

[0014] In an embodiment of the present application, the text-image difference fusion module extracts fusion vectors of image entities and image text, and then captures inconsistent information between the fusion vectors and the text information modality; the same module is used to capture inconsistent information between the text and the topic modality.

[0015] In an embodiment of the present application, the sarcasm identification module determines whether sarcasm exists in the data according to the text-image features extracted by the text-image relationship and difference modules.

[0016] Compared with the prior art, the present application has the following beneficial effects: the present application can fully exploit the hierarchical information possessed by the modal content itself, and can further extract and fuse the correlation and incoordination within and between modalities, thereby effectively improving the accuracy of sarcasm identification. BRIEF DESCRIPTION OF DRAWINGS

[0017] Figure 1 The present application is a multi-modal sarcasm identification system that fuses relationship and difference information.

[0018] Figure 2 The present application is a multi-modal sarcasm identification system that fuses relationship and difference information. DETAILED DESCRIPTION

[0019] The technical solutions of the present application will be specifically described below with reference to the accompanying drawings.

[0020] As shown in Figure 1 , 2 , the present application is a multi-modal sarcasm identification system that fuses relationship and difference information, comprising:

[0021] 1) Image-text feature extraction module: used for pre-processing image and text data, extracting corresponding feature representation for different modal content and its attribute characteristics; 2) Image-text relationship fusion module: a self-attention relationship fusion mechanism is designed to fuse image-text feature representation and extract the correlation degree between image-text data; 3) Image-text difference fusion module: a mechanism combining a gating mechanism and a distance matrix is designed to extract the difference degree within and between modalities; 4) Satire recognition module: input the feature representation of the correlation and difference of multi-modal data into the classifier to determine whether the multi-modal data contains satire.

[0022] 1. Image-text feature extraction module

[0023] The following describes how the feature extraction module extracts the feature representation of different modal data.

[0024] ① Text data feature extraction

[0025] For text data, the present application uses a pre-trained BERT model to obtain text representation. The text encoding is the output of the last layer of the BERT encoder, which can be described as T = {t1, t2, …, t n}, where, is the word embedding representation of word i.

[0026] ② Topic text data feature extraction

[0027] Similarly, for topic text data, the present application also uses a pre-trained BERT model to obtain text representation, and the topic text encoding can be described as H = {h1, h2, …, h n}, where, is the word embedding representation of word i.

[0028] ③ Image data feature extraction

[0029] For image data, the present application uses a ResNet-152 network model to extract image features. In order to obtain more rich image feature information, the present application truncates the last fully connected layer and obtains the output of the last convolutional layer:

[0030]

[0031] where each r i represents a 2048-dimensional vector of each region on the image. Therefore, the image I can be represented as:

[0032]

[0033] Finally, in order to project the visual features into the same dimension as the text features, the present application linearly converts the encoded image representation to:

[0034] G = W v ResNet(I)

[0035] ④Image Text Feature Extraction

[0036] Further, the present application extracts the text in the image through Paddle OCR. Then, the image text is input into the pre-trained BERT, and the special character null is filled for the image without image text. The image text encoding can be described as O = {o1, o2, …, o n}, wherein, is the word embedding representation of word i.

[0037] ⑤Image Object Feature Extraction

[0038] The present application extracts the local features of the image object through Faster-RCNN. First, based on the feature map of the image, 16 target object candidate boxes are generated using the RPN module, and then ResNet-152 is used to extract the features of the candidate boxes.

[0039]

[0040] wherein, p i represents a 2048-dimensional vector of each target object candidate box.

[0041] 2、Image-Text Relationship Fusion Module

[0042] The following describes how the image-text relationship fusion module represents the relationship between image-text data and extracts the association representation between the sarcasm modalities.

[0043] To extract the data combination association representation of the multi-modal data text and image, this paper designs an image-text relationship matching layer based on the self-attention mechanism, taking the text features extracted in the foregoing as the query, and the image features as the key and value, which can be used to capture the internal representation of the image-text interaction sequence.

[0044]

[0045] MATT(T,G) = [ATT1(T,G), …, ATT h (T,G)]W o

[0046] Then, the residual connection of the text feature T and the output of the self-attention layer MATT(T,G) is:

[0047] Z = LN(T + MATT(T,G))

[0048] Using an MLP feedforward network and another residual connection on Z to obtain the first text-image matching layer encoding output TIM lm (T,G) :

[0049] TIM(T,G) = LN(Z + MLP(Z))

[0050] The final representation of the text-image matching layer can describe TIM lm The encoding of the [CLS] token of (T,G) is

[0051] 3、Text-image difference fusion module

[0052] The text-image difference fusion module is described below. How does it represent the difference between the text-image data modalities and the internal difference of the text modality.

[0053] ① Intra-modal information difference extraction

[0054] The invention regards the contradiction between the original text and the topic in it as intra-modal inconsistency. For those data without topic labels, a special mark is used instead. A disassociation matrix C is introduced to simulate the interaction between text and topic labels. The calculation formula of C is:

[0055] C = tanh(T T W b H)

[0056] Where, and represent the text features and topic features respectively. N and M are predefined hyperparameters, representing the maximum length of the text and topic input sequences respectively. is a learnable parameter containing weights.

[0057] After calculating the disassociation matrix , the disassociation matrix on the text feature position is maximized to obtain the attention of the topic label. Specifically, the weight vector is calculated by applying the column maximum pooling operation on the matrix C.

[0058] T H = aH T

[0059] Where represents the intra-modal inconsistency information of text and topic labels.

[0060] ② Inter-modal information difference extraction

[0061] The image text and image object extracted from the image are mapped to the same dimensional space through a linear layer. Among them, due to the existence of noise in the image itself, the text extracted from the image is not always helpful for better prediction of the text. The application designs a G filtering gate mechanism based on a tanh activation function to remove redundant noise and enhance the features of key parts, ensuring that valuable information is passed to subsequent layers, while useless information is prevented. The formula of G is:

[0062]

[0063] Get filtered image object features And image text Input into the GA fusion module to ensure that one modality can obtain its relevant complementary information OP from other modalities.

[0064]

[0065] The formula of the GA fusion module is as follows:

[0066]

[0067]

[0068]

[0069] g k Aim to establish the correlation between the elements of the image object modal representation And the image text modal feature According to the importance of the correlation in the picture, design the weight coefficient a j , Finally get the correlation summary of image object and image text.

[0070] Finally, input the text information T and image object text information OP into the alienation matrix C to extract the inconsistent information between modalities:

[0071] C=tanh(T T W b OP)

[0072] T OP =aH T

[0073] 4, Satire recognition module

[0074] The following describes how the satire recognition module performs satire recognition on multi-modal data.

[0075] After obtaining the associated representation T G of the text image, the intra-modal inconsistency representation T Hand inter-modal inconsistency representation T OP After that, they are connected to make prediction. The prediction part includes a linear dimension reduction layer and a Softmax function to distribute probability to each class. Among them, the given text is classified into the class with the highest probability. This process can be described as:

[0076]

[0077] where, is the training of learnable parameters along with the model, is the final model classification result. The present application adopts cross-entropy as the loss function of the model:

[0078]

[0079] where, y i is the label of the i-th example in the data (1 for sarcasm, 0 for non-sarcasm), is the model prediction result, L2 is the standard L2 regular norm, and λ is the weight that measures the norm.

[0080] The above is the preferred embodiment of the present application, any changes made according to the technical solutions of the present application, as long as the generated function does not exceed the scope of the technical solutions of the present application, belongs to the protection scope of the present application.

Claims

1. A multi-modal sarcasm recognition system fusing relational discrepancy information, characterized in that, Comprise: A text-image feature extraction module for extracting feature representation of text-image data; A text-image relationship fusion module for matching and combining consistent information of text and image for prediction, which is implemented as follows: To extract the multimodal data text, image data combination association representation, design a graph-text relationship matching layer based on self-attention mechanism, and the text features As a query, image features As a key and value, to capture the internal representation of image and text interaction sequence; MATT(T,G) = [ATT1(T,G),...,ATT h (T,G)]W o Then the residual connection is performed between the text feature T and the output of the self-attention layer MATT(T, G) as follows: Z = LN(T + MATT(T, G)) Using an MLP feedforward network on Z and another residual connection to obtain the first text image matching layer encoding output, TIM lm (T,G): TIM(T, G) = LN(Z + MLP(Z)) The final representation of the text-image matching layer describes the TIM lm the encoding of the [CLS] token of (T, G) is A text-image difference fusion module for extracting inconsistent information in the text within the modal and the inconsistent information between the text and image in the modal, and combining the inconsistent information for prediction; A sarcasm identification module for determining whether there is sarcasm in the multi-modal data. After obtaining the associated representation T G , intra-modal inconsistency representation T H and inter-modal inconsistency representation T OP , they are concatenated for prediction, which includes a linear dimension reduction layer and a Softmax function to distribute probabilities to each class; where the given text is classified into the class with the highest probability, which is described as follows: wherein, is the learnable parameter training with the model, is the result of the final model classification; cross-entropy is used as the loss function of the model: where y i is the label of the i-th example in the data, 1 for sarcasm and 0 for non-sarcasm; is the prediction result of the model; L2 is the standard L2 regular norm; and λ is the weight measuring the norm.

2. The multi-modal sarcasm recognition system that fuses relational discrepancy information according to claim 1, wherein, The text-image feature extraction module extracts the features of the primary information of the image modal, the text modal and the topic modal.

3. The multi-modal sarcasm recognition system that fuses relational discrepancy information according to claim 1, wherein, The text-image relationship fusion module designs an image-text matching layer to capture and fuse consistent information between text and image by referring to the idea of multi-head attention mechanism.

4. The multi-modal sarcasm recognition system fusing relational discrepancy information according to claim 1, wherein, The text-image difference fusion module extracts the fusion vector of image entity and image text, and then captures the inconsistent information between it and the text information modal; the same module is used to capture the inconsistent information between the text and the topic modal.

5. A multimodal irony recognition system that integrates relational difference information according to claim 1, characterized in that, The sarcasm identification module determines whether there is sarcasm in the data according to the text-image features extracted by the text-image relationship fusion module and the text-image difference fusion module.

Images