A multimodal sarcasm detection method based on joint attention and graph learning enhancement

Abstract

The present invention belongs to the technical field of multimodal sarcasm detection of artificial intelligence, and specifically relates to a multimodal sarcasm detection method based on joint attention and graph learning enhancement, including: extracting initial image features and initial text features of an image-text pair; performing joint cross-attention learning processing on the initial image features and initial text features to obtain joint attention image features and joint attention text features; enhancing the joint attention text features by using graph learning; combining the enhanced text features with the joint attention image features, and using the combined features to predict the sarcasm tendency. The present invention uses a joint attention mechanism to simultaneously capture semantic inconsistencies within and between modalities, and uses graph learning to better integrate the intrinsic connections captured in a single modality into multimodal information, so as to solve the problem of inconsistent sarcasm information between modalities in multimodal sarcasm detection tasks.

Description

Technical Field

[0001] The present invention relates to the technical field of multimodal sarcasm detection of artificial intelligence, and in particular to a multimodal sarcasm detection method based on joint attention and graph learning enhancement. Background Art

[0002] Sarcasm is a way of expressing emotions, which conveys information by criticizing or mocking someone, something, or a certain point of view. It can prompt people to rethink certain views or behaviors, thereby achieving the purpose of criticism, reflection, or entertainment. Therefore, sarcasm detection is particularly important in tasks such as analyzing implicit emotions and understanding literary works. Early sarcasm detection was usually based only on text analysis, but in the real world, sarcasm often involves multiple forms of expression, including language, images, etc. For example, on online social platforms, people prefer to use emoticons combined with text to express their opinions. This generates a large amount of multimodal data related to sarcasm. In this case, visual information becomes impossible to ignore. Multimodal sarcasm detection came into being.

[0003] However, in the multimodal sarcasm detection task, the amount of sarcasm information contained in the text modality and the image modality is not equal. The sarcasm information in the text modality is richer and more expressive. In addition, the attention mechanism widely used in previous studies has limitations in the multimodal sarcasm detection task, and may highlight irrelevant words due to the input of text data. Summary of the invention

[0004] The purpose of the present invention is to propose a multimodal sarcasm detection method based on joint attention and graph learning enhancement to solve the problems existing in the above-mentioned prior art. The joint attention mechanism can be introduced to simultaneously capture the inconsistency problems within and between modalities in the multimodal sarcasm detection task; the high-order relations within the text modality can also be used to enhance the learning of the planar attention mechanism by using a double affine transformation method to solve the problem of inconsistent sarcasm information between modalities in the multimodal sarcasm detection task and the limitations of the attention mechanism itself.

[0005] To achieve the above object, the present invention provides the following solutions:

[0006] A multimodal sarcasm detection method based on joint attention and graph learning enhancement, including:

[0007] Extracting initial image features and initial text features of the image-text pair;

[0008] Performing joint cross attention learning processing on the initial image features and the initial text features to obtain joint attention image features and joint attention text features;

[0009] Enhance the text features of the joint attention using graph learning;

[0010] The enhanced text features are combined with the image features with joint attention, and the combined features are used to predict the sarcasm tendency.

[0011] Optionally, extracting initial image features and initial text features of the image-text pair includes:

[0012] Extracting the initial image features of the image-text pair through a pre-trained image encoder;

[0013] The initial text features of the image-text pair are extracted through a pre-trained text encoder.

[0014] Optionally, performing joint cross-attention learning processing on the initial image features and the initial text features includes:

[0015] Connecting the initial image features and the initial text features to obtain joint features;

[0016] Perform cross-attention learning on the joint features, the initial image features and the initial text features, respectively, to obtain an image correlation matrix and a text correlation matrix;

[0017] Combining the image correlation matrix and the text correlation matrix with the initial image features and the initial text features, respectively, and then using the learnable weight matrix and the RELU function, respectively, to calculate the attention map of the image features and the attention map of the text features;

[0018] The image attention map is used to calculate new image features, and the text attention map is used to obtain new text features. The two new features are connected to obtain new joint features. The above process is repeated to obtain the final joint attention image features and joint attention text features. The present invention finds that the iterative fusion form is conducive to the fusion of information between different modalities, so repeated operations are required.

[0019] Optionally, the image correlation matrix is:

[0020]

[0021] in, is the image correlation matrix, is the learnable weight matrix between I and J, j and i are the subscripts representing the joint vector and image vector respectively, I is the image feature, ( ) is the activation function, J is the joint feature, is the transpose of the image features, are the feature dimensions of image and text features;

[0022] The text correlation matrix is:

[0023]

[0024] in, is the text correlation matrix, is the learnable weight matrix between T and J, t is the subscript representing the image, is the transpose of the text features.

[0025] Optionally, the attention map of the image feature is:

[0026]

[0027] in, is the attention map of image features, for The learnable weight matrix between and T, is the text correlation matrix, ( ) is the activation function;

[0028] The attention map of the text feature is:

[0029]

[0030] in, is the attention map of text features, for A learnable weight matrix between and I.

[0031] Using the map of text and image, calculate the text features that combine the joint features and image features .

[0032]

[0033]

[0034] in, represents the first joint attention operation. represents the second joint attention operation, i.e. the iterative part, is the final text output of the joint attention module, is the final image output of the joint attention mechanism, Used as the next step of graph dependency enhancement operation, Used as a modal fusion operation before classification. is the text feature output of the first joint attention module, is the image output of the first joint attention module, is the text feature attention map for the second joint attention operation, is the image feature attention map for the second joint attention operation, is the learnable weight of the text modality, is the learnable weight of the image modality.

[0035] Optionally, enhancing the text features of the joint attention using graph learning includes:

[0036] Constructing a text graph for the jointly-attended text features;

[0037] Use the graph attention mechanism to process the text graph and obtain the text features of the preset advanced features ;

[0038] The text features Perform joint attention processing to obtain text features ;

[0039] The text features and text features Perform feature exchange and iterate on the exchange results.

[0040] Optionally, constructing the text graph includes:

[0041] The tokens in the joint attention text features are regarded as nodes of a graph, and the dependencies between words in the joint attention text features are extracted as edges of the graph; and the constructed text graph is undirected and includes self-loops.

[0042] Optionally, processing the text graph using the graph attention mechanism includes:

[0043]

[0044]

[0045]

[0046] Among them, W and are all learnable parameters of the first layer of GAT, is a scalar indicating the correlation score between node i and its neighboring node j. represents the feature representation of node j at layer (l), is the unnormalized score of the attention coefficient of node i to node j in the (l)th layer, is a learnable weight vector The transpose of is the feature of node i, is the node adjacent to the i node, is a collection of nodes, is the unnormalized score of the attention coefficient of node i to node k in the (l)th layer, is the attention weight of node i itself, is the feature of node i in the (l+1)th layer, ( ) is the activation function, ( ) is the activation function.

[0047] Optionally, iterating the exchange result includes:

[0048]

[0049]

[0050]

[0051]

[0052]

[0053]

[0054] in, and are the learnable weight matrices of the graph and the attention mechanism, respectively. for arrive The projection of for arrive The projection of , K is the number of interactive iterative learning, Based on and The attention weight calculated by the relationship between Based on and The attention weight calculated by the relationship between for The updated representation in layer K+1 is To represent the graph structure features of the Kth layer, for The updated representation at layer K+1 is The joint attention feature of the text of the Kth layer is Name it , used for the next fusion part, For the (K)th layer arrive The projection of For the (K)th layer arrive The projection of ( ) is the normalization operation.

[0055] Optionally, using the combined features to predict sarcasm tendency includes:

[0056]

[0057]

[0058]

[0059]

[0060] Among them, Mean( ) means applying the average function to the feature, LN( ) means using normalization, and is a trainable learning parameter, is the probability of predicting the label, For The result obtained after average pooling is: is the text feature after the final image enhancement. For The result after average pooling is To fuse the image features of the joint features, For the general and The result obtained after fusion and linear transformation is and is the corresponding bias vector.

[0061] The beneficial effects of the present invention are:

[0062] This paper introduces joint feature representation into the cross-modal cross-attention mechanism for the first time to simultaneously capture the inconsistencies between and within modalities.

[0063] The present invention considers both the graph-based representation learned from the text-corresponding dependency graph and the cross-modal representation learned from the joint attention mechanism. By performing BiAffine transformations between the two representations, the dependency graph guides the representation learning of the joint attention mechanism. This solves the problem of unequal irony information and the limitations of the attention mechanism on the MSD task.

[0064] The present invention feeds the two representations again to the enhancement module in a progressive mechanism to obtain a more detailed feature representation. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0066] Figure 1 A schematic flow chart of a multimodal sarcasm detection method based on joint attention and graph learning enhancement according to an embodiment of the present invention;

[0067] Figure 2 A schematic diagram of a network architecture according to an embodiment of the present invention;

[0068] Figure 3 Schematic diagram of comparison results between an embodiment of the present invention and several most advanced methods. DETAILED DESCRIPTION

[0069] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0070] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0071] like Figure 1-Figure 2 As shown, this embodiment proposes a multimodal sarcasm detection method based on joint attention and graph learning enhancement, including:

[0072] Extracting initial image features and initial text features of the image-text pair;

[0073] Performing joint cross-attention learning processing on the initial image features and the initial text features to obtain joint attention image features and joint attention text features;

[0074] Enhance text features with joint attention using graph learning;

[0075] The enhanced text features are combined with the image features with joint attention, and the combined features are used to predict the sarcasm tendency.

[0076] Furthermore, extracting initial image features and initial text features of the image-text pair includes:

[0077] Extract the initial image features of the image-text pair through the pre-trained image encoder;

[0078] The initial text features of image-text pairs are extracted through a pre-trained text encoder.

[0079] Specifically, in this embodiment, the specific steps of extracting image and text features are as follows:

[0080] First, given an image-text pair Following the previous method, this embodiment first resizes the given image I to a size of 224x224, divides each image into r patches, and then reshapes these patches into a sequence, represented as . ViT, which is trained on ImageNet for classification, is selected as the pre-trained model to extract visual features. Regarding the text T consisting of n tokens, this embodiment uses the pre-trained BERT to generate a feature representation for each token as follows:

[0081] .

[0082] Furthermore, the joint cross-attention learning process of the initial image features and the initial text features includes:

[0083] Connect the initial image features and the initial text features to obtain joint features;

[0084] Perform cross-attention learning on the joint features, initial image features and initial text features respectively to obtain image correlation matrix and text correlation matrix;

[0085] The image correlation matrix and the text correlation matrix are combined with the initial image features and the initial text features respectively, and then the attention map of the image features and the attention map of the text features are calculated using the learnable weight matrix and the RELU function respectively;

[0086] The image attention map is used to calculate new image features, and the text attention map is used to obtain new text features. The two new features are then connected to obtain new joint features. The above process is repeated to obtain the final joint attention image features and joint attention text features. This embodiment finds that the iterative fusion form is conducive to the fusion of information between different modalities, so repeated operations are required.

[0087] Specifically, in this embodiment, the specific steps of constructing the joint attention module are as follows:

[0088] The visual-text representation I, T obtained in the first step is connected to the two modalities to obtain the joint representation J of the visual and textual modalities.

[0089]

[0090] in, Indicates the connection between the front and back. Similarly, The joint feature representation of image and text is ,

[0091]

[0092] J is fed into the text joint cross attention module and the image joint cross attention module, and fused with the original text features and the original image features. In the text joint cross attention module, J and T generate the correlation matrix :

[0093]

[0094] in, is the learnable weight matrix between T and J.

[0095] Similarly, J and I are fed into the image joint cross attention module and fused with the original text features and the original image features. In the text joint cross attention module, J and I generate the correlation matrix :

[0096]

[0097] in, is the learnable weight matrix between I and J.

[0098] The correlation matrix of each modality can be used to calculate the semantic correlation within and between modalities. Combine the joint matrix X with the text feature T, and use the learnable weight matrix to calculate the attention map of each modality. Use RELU to activate it.

[0099]

[0100]

[0101] in, yes and T, yes and the learnable weight matrix between I, These are the attention maps generated by the text modality and image modality respectively.

[0102] Finally, in order to make the obtained features more refined, this module feeds the features calculated by each modality using the attention map into the joint attention module again. The text and image features after two layers of joint attention modules are obtained, namely the final joint attention image features and joint attention text features.

[0103]

[0104] .

[0105] This embodiment introduces the joint feature representation into the cross-modal cross-attention mechanism for the first time to simultaneously capture the inconsistencies between and within modalities.

[0106] Furthermore, the use of graph learning to enhance the text features of joint attention includes:

[0107] Construct a text graph for the text features of the joint attention;

[0108] Use the graph attention mechanism to process the text graph and obtain the text features of the preset advanced features ,

[0109] Text features Perform joint attention processing to obtain text features ;

[0110] Text features and text features Perform feature exchange and iterate on the exchange results.

[0111] Specifically, in this embodiment, the specific steps of using graph learning to enhance the text modality of joint attention are as follows:

[0112] Since the feature representations obtained by the general attention mechanism-based methods are often planar and limited. Therefore, this embodiment chooses to use the graph neural network method to use text features with high-level structures to strengthen the joint attention module. First, this embodiment constructs a text graph for the input text. This embodiment regards the tokens in the text as nodes of the graph, and uses spacy to extract the dependencies between words as the edges of the graph. In addition, the constructed text graph is undirected and includes self-loops.

[0113] Next, this embodiment uses the graph attention mechanism (GAT) to process the text graph. GAT can calculate the weight of each node when aggregating the features of neighboring nodes and capture the complex relationship between nodes and edges in the graph.

[0114]

[0115]

[0116]

[0117] Where W and a are the learnable parameters of the (l)th layer GAT. is a scalar indicating the correlation score between node i and its neighbor node j. Represents the feature representation of node j at the (l)th layer.

[0118] This embodiment obtains text features with high-level structures through GAT , the text features are obtained through the joint attention module Then, and They are input into biaffine together to exchange their features, and finally the results are reused in biaffine for iteration.

[0119]

[0120]

[0121]

[0122]

[0123]

[0124]

[0125] in, and are the learnable weight matrices of the graph and the attention mechanism, respectively. yes arrive The projection of yes arrive K is the number of interactive iterative learning.

[0126] This embodiment considers both the graph-based representation learned from the text-corresponding dependency graph and the cross-modal representation learned from the joint attention mechanism. By performing BiAffine conversion between the two representations, the dependency graph guides the representation learning of the joint attention mechanism. This solves the problem of unequal irony information and the limitations of the attention mechanism on the MSD task.

[0127] In this embodiment, in a progressive mechanism, the two representations are fed again to the enhancement module to obtain a more detailed feature representation.

[0128] Specifically, this embodiment combines text features with image features, defines model loss and optimizes the model. The output T of the dependency reinforcement module and the output I of the joint attention module are projected into the global embedding, and the ironic tendency is predicted.

[0129]

[0130]

[0131]

[0132]

[0133] Among them, Mean( ) means applying the average function to the feature, and LN( ) means using normalization. and is a trainable learning parameter. is the possibility of predicting the label, there are two cases.

[0134] This embodiment uses the most commonly used cross entropy loss to optimize the model of this embodiment. Learning rate , the model is optimized by the Adam algorithm.

[0135]

[0136] This embodiment proposes a novel multimodal sarcasm detection method, which can introduce a joint attention mechanism to simultaneously capture the inconsistency problem between modalities and modalities in the multimodal sarcasm detection task. It can also use a double affine transformation method to use the high-order relationship within the text modality to strengthen the learning of the plane attention mechanism to solve the problem of inconsistent sarcasm information between modalities in the multimodal sarcasm detection task and the limitations of the attention mechanism itself.

[0137] The implementation process of this embodiment mainly includes:

[0138] Step 1: Extract image and text features. Given an image-text pair, the feature extraction module generates image features and text features through pre-trained image encoders and text encoders respectively, and these features are passed to the corresponding modules for processing.

[0139] Step 2: Build a joint attention module to capture both intra-modal and inter-modal inconsistencies. Connect the text features generated in the first step with the image features to obtain joint features. Then perform cross-attention learning on the joint features with the corresponding unimodal features. This enables the learned features to include both image and text feature weights.

[0140] Step 3: Use graph learning to enhance the text modality of joint attention. Use dependencies to build the graph structure of the text modality. Use the graph attention mechanism to learn the dependencies between words in the text as a supplementary high-level structure. The text features of the learned joint attention obtained in the second step interact with the high-level structure of the graph through a double affine transformation to obtain mutually enhanced text features. In a progressive mechanism, the two representations are fed back to the reinforcement module to obtain a more detailed feature representation.

[0141] Step 4: Combine text features with image features, define model loss and optimize the model. The model is optimized based on the loss function, learning rate and optimizer.

[0142] The comparison results with several state-of-the-art comparison methods show the performance of the present invention; the comparison results of DGJA with several state-of-the-art methods (accuracy\%, Value\%) Figure 3 shown.

[0143] The above embodiments are only descriptions of the preferred embodiments of the present invention, and are not intended to limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary technicians in this field should fall within the protection scope determined by the claims of the present invention.

Claims

1. A multimodal sarcasm detection method based on joint attention and graph learning enhancement, characterized in that: include: Extracting initial image features and initial text features of the image-text pair; Performing joint cross attention learning processing on the initial image features and the initial text features to obtain joint attention image features and joint attention text features; The text features of the joint attention are enhanced by using graph learning, that is, the updated text features and the original text features after graph learning are subjected to a double affine transformation to enhance the text features, so that the processed text contains both the planar structure of the attention mechanism and the high-level structure of the dependency graph; The enhanced text features are combined with the image features with joint attention, and the combined features are used to predict the sarcasm tendency.

2. The multimodal sarcasm detection method based on joint attention and graph learning enhancement according to claim 1, characterized in that: Extracting initial image features and initial text features of image-text pairs includes: Extracting the initial image features of the image-text pair through a pre-trained image encoder; The initial text features of the image-text pair are extracted through a pre-trained text encoder.

3. The multimodal sarcasm detection method based on joint attention and graph learning enhancement according to claim 1, characterized in that: Performing joint cross-attention learning processing on the initial image features and the initial text features includes: Connecting the initial image features and the initial text features to obtain joint features; Perform cross-attention learning on the joint features, the initial image features and the initial text features, respectively, to obtain an image correlation matrix and a text correlation matrix; Combining the image correlation matrix and the text correlation matrix with the initial image features and the initial text features, respectively, and then using the learnable weight matrix and the RELU function, respectively, to calculate the attention map of the image features and the attention map of the text features; The image attention map is used to calculate the new image features, and the text attention map is used to obtain the new text features. The two new features are then connected to obtain the new joint features. The above process is repeated to obtain the final joint attention image features and joint attention text features.

4. The multimodal sarcasm detection method based on joint attention and graph learning enhancement according to claim 3, characterized in that: The image correlation matrix is: in, is the image correlation matrix, is the learnable weight matrix between I and J, j and i are the subscripts representing the joint vector and image vector respectively, I is the image feature, ( ) is the activation function, J is the joint feature, is the transpose of the image features, are the feature dimensions of image and text features; The text correlation matrix is: in, is the text correlation matrix, is the learnable weight matrix between T and J, t is the subscript representing the image, is the transpose of the text features.

5. The multimodal sarcasm detection method based on joint attention and graph learning enhancement according to claim 3, characterized in that: The attention map of the text feature is: in, is the attention map of text features, for The learnable weight matrix between and T, is the text correlation matrix, ( ) is the activation function; The attention map of the image feature is: in, is the attention map of image features, for A learnable weight matrix between and I; Using the map of text and image, calculate the text features that combine the joint features and image features ; in, represents the first joint attention operation. represents the second joint attention operation, i.e. the iteration part, is the final text output of the joint attention module, is the final image output of the joint attention mechanism, Used as the next step of graph dependency enhancement operation, Used as a modal fusion operation before classification. is the text feature output of the first joint attention module, is the image output of the first joint attention module, is the text feature attention map for the second joint attention operation, is the image feature attention map for the second joint attention operation, is the learnable weight of the text modality, is the learnable weight of the image modality.

6. The multimodal sarcasm detection method based on joint attention and graph learning enhancement according to claim 1, characterized in that: Using graph learning to enhance the text features of the joint attention includes: Constructing a text graph for the jointly-attended text features; Use the graph attention mechanism to process the text graph and obtain the text features of the preset advanced features , The text features Perform joint attention processing to obtain text features ; The text features and text features Perform feature exchange and iterate on the exchange results.

7. The multimodal sarcasm detection method based on joint attention and graph learning enhancement according to claim 6, characterized in that: Constructing the text graph includes: The tokens in the text features of the joint attention are regarded as nodes of a graph, and the dependencies between words in the text features of the joint attention are extracted as edges of the graph; and the constructed text graph is undirected and includes self-loops.

8. The multimodal sarcasm detection method based on joint attention and graph learning enhancement according to claim 6, characterized in that: Using the graph attention mechanism to process text graphs includes: Among them, W and are all learnable parameters of the first layer of GAT, is a scalar indicating the correlation score between node i and its neighboring node j. represents the feature representation of node j at layer (l), is the unnormalized score of the attention coefficient of node i to node j in the (l)th layer, is a learnable weight vector The transpose of is the feature of node i, is the node adjacent to the i node, is a collection of nodes, is the unnormalized score of the attention coefficient of node i to node k in the (l)th layer, is the attention weight of node i itself, is the feature of node i in the (l+1)th layer, ( ) is the activation function, ( ) is the activation function.

9. The multimodal sarcasm detection method based on joint attention and graph learning enhancement according to claim 6, characterized in that: Iterating the exchange results includes: in, and are the learnable weight matrices of the graph and the attention mechanism, respectively. for arrive The projection of for arrive The projection of K is the number of interactive iterative learning, K = 3, Based on and The attention weight calculated by the relationship between Based on and The attention weight calculated by the relationship between for The updated representation in layer K+1 is, To represent the graph structure features of the Kth layer, for The updated representation at layer K+1 is The joint attention feature of the text of the Kth layer is Name it , used for the next fusion part, For the (K)th layer arrive The projection of For the (K)th layer arrive The projection of ( ) is the normalization operation.

10. The multimodal sarcasm detection method based on joint attention and graph learning enhancement according to claim 9, characterized in that: The combined features used to predict sarcasm include: Among them, Mean( ) means applying the average function to the feature, LN( ) means using normalization, and is a trainable learning parameter, is the probability of predicting the label, For The result obtained after average pooling is: is the text feature after the final image enhancement. For The result after average pooling is To fuse the image features of the joint features, For the general and The result obtained after fusion and linear transformation is and are the corresponding bias vectors respectively.

Images