Multimodal sarcasm detection method, device, computer equipment and storage medium

Abstract

The present invention relates to the technical field of natural language processing, and in particular to a multimodal sarcasm detection method, which comprises the steps of obtaining document data to be tested and a preset multimodal sarcasm detection model, inputting the document data to be tested into an encoding module, performing encoding processing on a text to be tested and an image to be tested respectively, and obtaining a text encoding representation and an image encoding representation; inputting the text encoding representation and the image encoding representation into a semantic extraction module for semantic feature extraction, and obtaining a text semantic feature representation and an image semantic feature representation; inputting the text semantic feature representation and the image semantic feature representation into an emotion tendency detection module for emotion tendency detection, and obtaining a text emotion tendency feature representation and an image emotion tendency feature representation; inputting the text emotion tendency feature representation and the image emotion tendency feature representation into a sarcasm detection module for sarcasm detection, and obtaining a sarcasm detection result of the document data to be tested.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and in particular to a multimodal sarcasm detection method, device, computer equipment and storage medium. Background Art

[0002] Sarcasm is a ubiquitous linguistic phenomenon that represents the difference between literal meaning and implicit intent. Sarcasm detection can help better understand social media comments and infer people's true emotions and opinions. Sarcasm detection can help better understand the comments made by users and infer people's true emotions and opinions on social media platforms.

[0003] Current multimodal sarcasm detection methods capture the commonalities and differences between text and images by constructing a decomposed relational network, or use attention mechanisms and graph neural networks to model the inconsistencies within and between modalities to build a strategy for image-text fusion features to improve the performance of multimodal sarcasm detection. However, since sarcastic image-text pairs often contain opposite emotional information, while non-sarcastic image-text emotions tend to be consistent, the above methods often only start from the image-text fusion features of a single piece of data, and do not consider the relevance of the emotional information of the image and text to sarcasm detection, resulting in a decrease in the accuracy of multimodal sarcasm detection. Summary of the invention

[0004] Based on this, the present invention provides a multimodal sarcasm detection method, device, computer equipment and storage medium, which extracts semantic information from the document data to be tested, and further extracts text-level and image-level emotional tendency feature representations based on the semantic information for sarcasm recognition, fully considering the relevance of emotional information in text and image modalities for sarcasm detection, and improving the accuracy and efficiency of multimodal sarcasm detection. The technical method is as follows:

[0005] In a first aspect, an embodiment of the present application provides a multimodal taunt detection method, comprising the following steps:

[0006] Obtaining document data to be tested and a preset multimodal sarcasm detection model, wherein the document data to be tested includes text to be tested and an image to be tested; the multimodal sarcasm detection model includes an encoding module, a semantic extraction module, a sentiment tendency detection module, and a sarcasm detection module;

[0007] Inputting the document data to be tested into the encoding module, encoding the text to be tested and the image to be tested respectively to obtain a text encoding representation and an image encoding representation;

[0008] Inputting the text encoding representation and the image encoding representation into the semantic extraction module to extract semantic features, thereby obtaining text semantic feature representation and image semantic feature representation;

[0009] Inputting the text semantic feature representation and the image semantic feature representation into the sentiment tendency detection module to perform sentiment tendency detection, thereby obtaining the text sentiment tendency feature representation and the image sentiment tendency feature representation;

[0010] The text sentiment tendency feature representation and the image sentiment tendency feature representation are input into the sarcasm detection module to perform sarcasm detection, and the sarcasm detection result of the document data to be tested is obtained.

[0011] In a second aspect, an embodiment of the present application provides a multimodal taunting detection device, comprising:

[0012] A data acquisition module, used to obtain document data to be tested and a preset multimodal sarcasm detection model, wherein the document data to be tested includes text to be tested and an image to be tested; the multimodal sarcasm detection model includes an encoding module, a semantic extraction module, a sentiment tendency detection module and a sarcasm detection module;

[0013] A document encoding module, used for inputting the document data to be tested into the encoding module, encoding the text to be tested and the image to be tested respectively, and obtaining a text encoding representation and an image encoding representation;

[0014] A semantic information extraction module, used for inputting the text encoding representation and the image encoding representation into the semantic extraction module to extract semantic features, so as to obtain text semantic feature representation and image semantic feature representation;

[0015] The emotional information extraction module is used to input the text semantic feature representation and the image semantic feature representation into the emotional tendency detection module to perform emotional tendency detection, and obtain the text emotional tendency feature representation and the image emotional tendency feature representation;

[0016] The sarcasm detection result acquisition module is used to input the text sentiment tendency feature representation and the image sentiment tendency feature representation into the sarcasm detection module to perform sarcasm detection and obtain the sarcasm detection result of the document data to be tested.

[0017] In a third aspect, an embodiment of the present application provides a computer device, comprising: a processor, a memory, and a computer program stored in the memory and executable on the processor; when the computer program is executed by the processor, the steps of the multimodal taunt detection method described in the first aspect are implemented.

[0018] In a fourth aspect, an embodiment of the present application provides a storage medium, wherein the storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the multimodal taunt detection method as described in the first aspect are implemented.

[0019] In this embodiment, a multimodal sarcasm detection method, apparatus, computer device and storage medium are provided. By extracting semantic information from the document data to be tested, text-level and image-level sentiment tendency feature representations are further extracted based on the semantic information for sarcasm recognition. The relevance of sentiment information in text and image modalities to sarcasm detection is fully considered, thereby improving the accuracy and efficiency of multimodal sarcasm detection.

[0020] For better understanding and implementation, the present invention is described in detail below with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 A schematic diagram of a flow chart of a multimodal taunting detection method provided in the first embodiment of the present application;

[0022] Figure 2 A schematic diagram of the process of S2 in the multimodal taunting detection method provided in the first embodiment of the present application;

[0023] Figure 3 A schematic diagram of the process of S3 in the multimodal taunting detection method provided in the first embodiment of the present application;

[0024] Figure 4 A schematic diagram of the process of S4 in the multimodal taunting detection method provided in the first embodiment of the present application;

[0025] Figure 5 A schematic diagram of the process of S5 in the multimodal taunting detection method provided in the first embodiment of the present application;

[0026] Figure 6 A schematic diagram of a flow chart of a multimodal taunting detection method provided in the second embodiment of the present application;

[0027] Figure 7 A schematic diagram of the process of S64 in the multimodal taunting detection method provided in the third embodiment of the present application;

[0028] Figure 8 A schematic diagram of the structure of a multimodal taunting detection device provided in a fourth embodiment of the present application;

[0029] Fig. 9 A schematic diagram of the structure of a computer device provided in the fifth embodiment of the present application. DETAILED DESCRIPTION

[0030] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0031] The terms used in this application are for the purpose of describing the target embodiments only and are not intended to limit the application. The singular forms of "a", "said" and "the" used in this application and the appended claims are also intended to include plural forms unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0032] It should be understood that, although the terms first, second, third, etc. may be used in the present application to describe various information, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" / "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determination".

[0033] See also Figure 1 , Figure 1 The flowchart of the multimodal taunting detection method provided in the first embodiment of the present application includes the following steps:

[0034] S1: Obtain the document data to be tested and the preset multimodal sarcasm detection model.

[0035] The execution subject of the multimodal taunting detection method of the present application is a detection device of the multimodal taunting detection method (hereinafter referred to as the detection device).

[0036] In an optional embodiment, the detection device may be a computer device, a server, or a server cluster formed by a combination of multiple computer devices.

[0037] The detection device obtains the document data to be tested. In an optional embodiment, the document data to be tested originates from social media, wherein social media is a content production and exchange platform based on user relationships on the Internet, mainly including social networking sites, Weibo, WeChat, Facebook, Twitter, blogs, forums, podcasts, etc. The training document data includes the opinions, insights, experiences, and views shared by users on social media.

[0038] Specifically, the detection device can be connected to the social media to obtain the document data to be tested published on the social media, wherein the training document data includes the text to be tested and the image to be tested.

[0039] The detection device obtains a preset multimodal sarcasm detection model, wherein the multimodal sarcasm detection model comprises an encoding module, a semantic extraction module, a sentiment tendency detection module and a sarcasm detection module.

[0040] S2: Input the document data to be tested into the encoding module, perform encoding processing on the text to be tested and the image to be tested respectively, and obtain a text encoding representation and an image encoding representation.

[0041] In this embodiment, the detection device inputs the document data to be tested into the encoding module, and performs encoding processing on the text to be tested and the image to be tested respectively to obtain a text encoding representation and an image encoding representation.

[0042] The encoding module includes a word embedding module and an object detection module. Figure 2 , Figure 2 The flowchart of S2 in the multimodal taunting detection method provided in the first embodiment of the present application includes steps S21 to S22, which are specifically as follows:

[0043] S21: Input the text to be tested into the word embedding module for encoding processing to obtain the text encoding representation.

[0044] The word embedding module adopts the BERT (Bidirectional Encoder Representation from Transformers) word embedding model, which is used to convert the vectors of several words in the sentence to be tested into corresponding state vectors.

[0045] In this embodiment, the detection device inputs the text to be tested into the word embedding module for encoding processing to obtain the text encoding representation, wherein the text encoding representation includes word embedding vectors of several words.

[0046] Specifically, the detection device inputs the sentence to be tested into the BERT word embedding model, maps each word in the text to be tested into a low-dimensional vector space, and obtains the hidden layer vectors of several words of the text to be tested output by the BERT word embedding model after querying the pre-trained BERT matrix as the word embedding vector, and performs encoding processing to obtain the text encoding representation.

[0047] S22: Divide the image to be tested into a plurality of image sub-regions, input the plurality of image sub-regions into the target detection module for target detection, and obtain the image coding representation.

[0048] The target detection module adopts the ViT model for multi-target detection and can determine predefined objects related to entities in the image.

[0049] In this embodiment, the detection device divides the image to be tested into several image sub-regions, inputs the several image sub-regions into the target detection module for target detection, and obtains the image coding representation, wherein the image coding representation includes detection vectors of several image sub-regions.

[0050] S3: Input the text coding representation and the image coding representation into the semantic extraction module to extract semantic features, so as to obtain text semantic feature representation and image semantic feature representation.

[0051] In this embodiment, the detection device inputs the text encoding representation and the image encoding representation into the semantic extraction module to extract semantic features, thereby obtaining text semantic feature representation and image semantic feature representation, and utilizing the semantic information of text and image to mine more effective text-level and image-level sarcasm features.

[0052] The semantic extraction module includes a text semantic extraction module and an image semantic extraction module; see Figure 3 , Figure 3 The flowchart of S3 in the multimodal taunting detection method provided in the first embodiment of the present application includes steps S31 to S32, which are specifically as follows:

[0053] S31: Use the text encoding representation as the input representation of the first-layer text semantic feature extraction layer of the text semantic extraction module, perform multi-head self-attention processing and forward propagation processing in sequence according to the text encoding representation, obtain the output representation of the first-layer text semantic feature extraction layer as the input representation of the next-layer text semantic feature extraction layer, until the output representation of the last-layer text semantic feature extraction layer is obtained as the text semantic feature representation.

[0054] The text semantic extraction module adopts RoBERTa (Robustly Optimized BERT Pretraining Approach). The text semantic extraction module includes a plurality of stacked text semantic feature extraction layers.

[0055] In this embodiment, the detection device uses the text encoding representation as the input representation of the first-layer text semantic feature extraction layer of the text semantic extraction module, and performs multi-head self-attention processing and forward propagation processing in sequence according to the text encoding representation and the preset text semantic feature extraction algorithm to obtain the output representation of the first-layer text semantic feature extraction layer as the input representation of the next-layer text semantic feature extraction layer, until the output representation of the last-layer text semantic feature extraction layer is obtained as the text semantic feature representation, wherein the text semantic feature extraction algorithm is:

[0056]

[0057] In the formula, is the multi-head self-attention feature representation of the l-th text semantic feature extraction layer after normalization, Norm(·) is the normalization function, MHSA(·) is the multi-head self-attention function, is the output representation of the l-1th text semantic feature extraction layer, FFN(·) is the feedforward neural network function, It is the output representation of the lth text semantic feature extraction layer.

[0058] S32: Use the image coding representation as the input representation of the first-layer image semantic feature extraction layer of the image semantic extraction module, perform forward propagation processing and multi-head self-attention processing in sequence according to the image coding representation, obtain the output representation of the first-layer image semantic feature extraction layer as the input representation of the next-layer image semantic feature extraction layer, until the output representation of the last-layer image semantic feature extraction layer is obtained as the image semantic feature representation.

[0059] The image semantic extraction module adopts CLIP (Contrastive Language-Image Pre-Training) contrastive language-image pre-training model, and the image semantic extraction module has several stacked image semantic feature extraction layers.

[0060] In this embodiment, the detection device uses the image coding representation as the input representation of the first-layer image semantic feature extraction layer of the image semantic extraction module, and performs forward propagation processing and multi-head self-attention processing in sequence according to the image coding representation and the preset image semantic feature extraction algorithm to obtain the output representation of the first-layer image semantic feature extraction layer as the input representation of the next-layer image semantic feature extraction layer, until the output representation of the last-layer image semantic feature extraction layer is obtained as the image semantic feature representation, wherein the image semantic feature extraction algorithm is:

[0061]

[0062] In the formula, is the multi-head self-attention feature representation of the l-th image semantic feature extraction layer, is the output representation of the l-1th image semantic feature extraction layer, It is the output representation of the lth image semantic feature extraction layer.

[0063] By utilizing the multi-head self-attention mechanism and feedforward neural network, we attempt to make the model focus on the correlation between different parts of the overall unimodal input to filter out contextual information irrelevant to sarcasm, better capture the intra-modal contextual sarcasm clues from different subspaces, and improve the accuracy of sarcasm detection.

[0064] S4: Input the text semantic feature representation and the image semantic feature representation into the sentiment tendency detection module to perform sentiment tendency detection, and obtain the text sentiment tendency feature representation and the image sentiment tendency feature representation.

[0065] In this embodiment, the detection device inputs the text semantic feature representation and the image semantic feature representation into the sentiment tendency detection module to perform sentiment tendency detection, and obtains the text sentiment tendency feature representation and the image sentiment tendency feature representation.

[0066] See also Figure 4 , Figure 4 The flowchart of S4 in the multimodal taunting detection method provided in the first embodiment of the present application includes steps S41 to S42, which are specifically as follows:

[0067] S41: Calculate sentiment scores according to word embedding vectors of several words in the text encoding representation to obtain sentiment score vectors of several words, add the sentiment score vectors of several words in pairs to obtain sentiment association vectors between several words, and construct a text sentiment adjacency matrix.

[0068] In this embodiment, the detection device calculates the sentiment scores according to the word embedding vectors of the plurality of words in the text encoding representation to obtain the sentiment score vectors of the plurality of words.

[0069] Specifically, the detection device uses an external sentiment knowledge tool (SenticNet) to confirm whether there is a corresponding word embedding vector in the external sentiment knowledge tool according to the word embedding vectors of several words in the text encoding representation. If so, the sentiment score is calculated according to the word embedding vector and a preset sentiment score function to obtain the sentiment score vector of the word, otherwise the sentiment score vector of the word is set to 0, wherein the sentiment score vector is:

[0070]

[0071] In the formula, s(ti ) is the sentiment score vector of the i-th word, t i is the word embedding vector of the ith word, and SenticNet(·) is the sentiment score function.

[0072] The detection device adds the sentiment score vectors of several words in pairs, thereby focusing on the overall sentiment consistency of the text to be tested and paying more attention to words with strong emotions, obtaining the sentiment association vectors between several words, and constructing a text sentiment adjacency matrix, wherein the sentiment association vector is:

[0073]

[0074] In the formula, A i，j is the sentiment association vector between the i-th word and the j-th word, is the sentiment score vector of the jth word, t j is the word embedding vector of the jth word.

[0075] S42: Normalizing the text sentiment adjacency matrix to obtain the normalized text sentiment adjacency matrix; extracting sentiment tendency features based on the text semantic feature representation and the normalized text sentiment adjacency matrix to obtain the text sentiment tendency feature representation; performing graph convolution processing based on the image semantic feature representation to obtain the image sentiment tendency feature representation.

[0076] In this embodiment, the detection device normalizes the text emotion adjacency matrix to obtain a normalized text emotion adjacency matrix, wherein the normalized text emotion adjacency matrix is:

[0077]

[0078] In the formula, is the text sentiment adjacency matrix, A i is the sentiment association vector of the word in the i-th row in the text sentiment adjacency matrix.

[0079] The detection device performs graph convolution processing according to the text semantic feature representation and the normalized text sentiment adjacency matrix to obtain the text graph convolution feature representation, and performs graph convolution processing according to the image semantic feature representation to obtain the image sentiment tendency feature representation.

[0080] Specifically, the sentiment tendency detection module includes a residual graph convolutional neural network. The detection device performs graph convolution processing according to the text semantic feature representation, the normalized text sentiment adjacency matrix and the residual graph convolutional neural network to obtain the text sentiment tendency feature representation, wherein the text sentiment tendency feature representation is:

[0081]

[0082] In the formula, S T is the feature representation of the text sentiment tendency, ReLU(·) is the linear activation function, is the normalized text sentiment adjacency matrix, H T is the text semantic feature representation, W t is the first weight parameter of the residual graph convolutional neural network, b t is the first bias parameter of the residual graph convolutional neural network.

[0083] The detection device performs graph convolution processing according to the image semantic feature representation and the residual graph convolutional neural network to obtain the image emotional tendency feature representation, wherein the image emotional tendency feature representation is:

[0084] S V =W v H V +b v

[0085] In the formula, S V is the feature representation of the image sentiment tendency, W v is the second weight parameter of the residual graph convolutional neural network, H V is the image semantic feature representation, b v is the second bias parameter of the residual graph convolutional neural network.

[0086] S5: Inputting the text sentiment tendency feature representation and the image sentiment tendency feature representation into the sarcasm detection module to perform sarcasm detection, and obtaining the sarcasm detection result of the document data to be tested.

[0087] In this embodiment, the detection device inputs the text sentiment tendency feature representation and the image sentiment tendency feature representation into the sarcasm detection module to perform sarcasm detection, and obtains the sarcasm detection result of the document data to be tested.

[0088] By extracting semantic information from the document data to be tested, and further extracting text-level and image-level sentiment feature representations based on the semantic information, sarcasm recognition is performed. The relevance of sentiment information in text and image modalities to sarcasm detection is fully considered, thereby improving the accuracy and efficiency of multimodal sarcasm detection.

[0089] See also Figure 5 , Figure 5 The flowchart of S5 in the multimodal taunting detection method provided in the first embodiment of the present application includes steps S51 to S52, which are specifically as follows:

[0090] S51: Adopting the interactive attention method, performing feature interaction according to the text sentiment tendency feature representation and the image sentiment tendency feature representation, and obtaining a text-to-image interactive attention feature representation.

[0091] In this embodiment, the detection device adopts an interactive attention method, and according to the text emotion tendency feature representation, the image emotion tendency feature representation and the preset first interactive attention calculation algorithm, the text emotion tendency feature representation and the image emotion tendency feature representation are interactively fused using the interactive attention method, and the image and text are compressed and mapped to the same embedding space to align the modal features of the image and text, and obtain the text-to-image interactive attention feature representation, wherein the first interactive attention calculation algorithm is:

[0092]

[0093] In the formula, H t2v is the text-to-image interactive attention feature representation, Softmax() is the normalized exponential function, W q is the first weight parameter of the taunt detection module, W k is the second weight parameter of the taunt detection module, W e is the third weight parameter of the taunt detection module, b q is the first bias parameter of the taunt detection module, b k is the second bias parameter of the taunting detection module, b e is the third bias parameter of the taunt detection module, d is the dimension parameter, and T is the transposition symbol.

[0094] S52: Perform global feature extraction on the text-to-image interactive attention feature representation to obtain a text-to-image global feature representation, perform a sarcasm probability vector calculation based on the text-to-image global feature representation to obtain a predicted sarcasm probability vector, and obtain a sarcasm detection result for the document data to be tested based on the predicted sarcasm probability vector.

[0095] In this embodiment, the detection device performs global feature extraction on the text-to-image interactive attention feature representation to obtain a text-to-image global feature representation to achieve the perception of rich unimodal contextual information and suppress some information irrelevant to sarcasm, thereby mining subtle sarcasm clues at the text and image levels.

[0096] Specifically, the text-to-image interactive attention feature representation includes several elements, which are used to indicate tensors of corresponding dimensions. The detection device takes the first element in the text-to-image interactive attention feature representation as the text-to-image global feature representation.

[0097] The detection device calculates a sarcasm probability vector according to the text-to-image global feature representation to obtain a predicted sarcasm probability vector, wherein the sarcasm probability vector algorithm is:

[0098]

[0099] Where P s To predict the mockery probability vector, W n is the fourth weight parameter of the taunt detection module, is the global feature representation of text to image, b n is the fourth bias parameter of the taunt detection module.

[0100] The detection device obtains a ridicule detection result of the document data to be tested according to the predicted ridicule probability vector. Specifically, the detection device obtains a ridicule detection result corresponding to a probability vector with the largest dimension according to the predicted ridicule probability vector as the ridicule detection result of the document data to be tested.

[0101] In an optional embodiment, the step S6 is also included: training the multimodal sarcasm detection model, see Figure 6 , Figure 6 The flowchart of S6 in the multimodal taunting detection method provided in the second embodiment of the present application includes steps S61 to S64, which are specifically as follows:

[0102] S61: Obtain a training document dataset, input the training document dataset into the multimodal sarcasm detection model, and obtain text sentiment tendency feature representations, image sentiment tendency feature representations, and predicted sarcasm probability vectors of a plurality of training document data.

[0103] In this embodiment, the detection device obtains a training document data set, inputs the training document data set into the multimodal sarcasm detection model, obtains text sentiment tendency feature representation, image sentiment tendency feature representation and predicted sarcasm probability vector of a plurality of training document data

[0104] S62: Calculate the sentiment tendency probability vectors respectively according to the text sentiment tendency feature representations and the image sentiment tendency feature representations of the training document data, and obtain the text sentiment tendency probability vectors and the image sentiment tendency probability vectors of the training document data.

[0105] In this embodiment, the detection device calculates the sentiment tendency probability vectors according to the text sentiment tendency feature representations and the image sentiment tendency feature representations of the training document data, and obtains the text sentiment tendency probability vectors and the image sentiment tendency probability vectors of the training document data.

[0106] Specifically, the detection device adopts a binary linear layer to obtain the text sentiment tendency probability vector according to the text sentiment tendency feature representation and a preset text sentiment tendency probability algorithm, wherein the text sentiment tendency probability algorithm is:

[0107]

[0108] In the formula, is the text sentiment probability vector, is the first weight parameter of the linear layer, is the first bias parameter of the linear layer.

[0109] The detection device obtains the image emotion tendency probability vector according to the image emotion tendency feature representation and a preset image emotion tendency algorithm, wherein the image emotion tendency probability algorithm is:

[0110]

[0111] In the formula, is the image sentiment tendency probability vector, is the second weight parameter of the linear layer, is the second bias parameter of the linear layer.

[0112] S63: Performing divergence loss calculation based on the text sentiment tendency probability vectors and the image sentiment tendency probability vectors of the training document data to obtain a divergence loss value.

[0113] In this embodiment, the detection device performs divergence loss calculation based on the text sentiment tendency probability vectors, image sentiment tendency probability vectors and a preset first loss algorithm of the training document data to obtain a divergence loss value for model training, wherein the first loss algorithm is:

[0114]

[0115] In the formula, is the divergence loss value, and KL[] is the divergence function.

[0116] Based on the KL divergence method, when the probability predictions of positive and negative sentiment tendencies are consistent, the model is encouraged to choose one from positive and negative instead of neutral, which ensures the accuracy of model training while improving the efficiency of training, generates the corresponding sentiment tendency probability distribution for the image, and encourages the network to achieve better sentiment feature extraction.

[0117] S64: obtaining a plurality of true ridicule probability vectors of the training document data, performing classification loss calculation according to the predicted ridicule probability vectors of the plurality of training document data and the true ridicule probability vector, and obtaining a classification loss value.

[0118] In this embodiment, the detection device obtains a real ridicule probability vector of a plurality of training document data, performs classification loss calculation according to the predicted ridicule probability vector of the plurality of training document data, the real ridicule probability vector and a preset second loss algorithm, and obtains a classification loss value, wherein the second loss algorithm is:

[0119]

[0120] In the formula, is the classification loss value, N is the total number of training document data, Y i is the true mockery probability vector of the i-th training document data, is the predicted sarcasm probability vector for the i-th training document data.

[0121] S65: Training the multimodal ridicule detection model according to the divergence loss value and the classification loss value.

[0122] In this embodiment, the detection device trains the multimodal sarcasm detection model according to the divergence loss value and the classification loss value.

[0123] See also Figure 7 , Figure 7 The flowchart of S65 in the multimodal taunting detection method provided in the third embodiment of the present application includes steps S651 to S653, which are specifically as follows:

[0124] S651: Obtain text-to-image global feature representation and image-to-text global feature representation of a number of training document data.

[0125] In this embodiment, the detection device obtains text-to-image global feature representations and image-to-text global feature representations of several training document data. Specifically, for obtaining the text-to-image global feature representations, reference may be made to steps S61 to S62, which will not be repeated here.

[0126] For obtaining the image-to-text global feature representation, the detection device adopts an interactive attention method to obtain the image-to-text interactive attention feature representation according to the text sentiment tendency feature representation, the image sentiment tendency feature representation and a preset second interactive attention calculation algorithm, wherein the second interactive attention calculation algorithm is:

[0127]

[0128] In the formula, Hv2t is the image-to-text interactive attention feature representation.

[0129] The detection device performs global feature extraction on the image-to-text interactive attention feature representation to obtain the image-to-text global feature representation.

[0130] S652: Calculate similarity loss based on the text-to-image global feature representation and the image-to-text global feature representation of the plurality of training document data to obtain a similarity loss value.

[0131] In this embodiment, the detection device performs similarity loss calculation based on the text-to-image global feature representation and the image-to-text global feature representation of the plurality of training document data to obtain a similarity loss value.

[0132] Specifically, the detection device obtains the mocking labels of the plurality of training document data, and according to the mocking labels, regards other training document data that are consistent with the mocking labels of the current training document data as positive examples, and constructs a plurality of positive example sets of the training document data.

[0133] In order to obtain better feature representation and thus facilitate downstream tasks, the detection device adopts a dual contrastive learning method, aligns different feature spaces according to the text-to-image global feature representation, image-to-text global feature representation of several training document data in the positive example set and a preset third loss algorithm, performs similarity loss calculation, and obtains a similarity loss value, wherein the third loss algorithm is:

[0134]

[0135] In the formula, is the similarity loss value, B is the preset batch, is the positive example set, is the global feature representation from image to text, is the text-to-image global feature representation of the i-th training document data in the current batch, is the text-to-image global feature representation of the j-th training document data in the positive example set, is the text-to-image global feature representation of the k-th training document data in the current batch, τ is the temperature coefficient, and sim() is the similarity function.

[0136] S653: Training the multimodal sarcasm detection model according to the divergence loss value, the classification loss value, and the similarity loss value.

[0137] In this embodiment, the detection device obtains a total loss value according to the divergence loss value, the classification loss value, the similarity loss value and a preset total loss algorithm, and trains the multimodal sarcasm detection model according to the total loss value, effectively combining the emotional information and the sarcasm label, and improving the recognition accuracy of the sarcasm in the social media, wherein the total loss algorithm is:

[0138]

[0139] In the formula, is the total loss value, α is the first hyperparameter, β is the second hyperparameter, and γ is the third hyperparameter.

[0140] See also Figure 8 , Figure 8 This is a schematic diagram of the structure of a multimodal taunting detection device provided in the fourth embodiment of the present application. The device can implement all or part of the multimodal taunting detection method through software, hardware, or a combination of both. The device 8 includes:

[0141] The data acquisition module 81 is used to obtain the document data to be tested and the preset multimodal sarcasm detection model, wherein the document data to be tested includes the text to be tested and the image to be tested; the multimodal sarcasm detection model includes an encoding module, a semantic extraction module, a sentiment tendency detection module and a sarcasm detection module;

[0142] The document encoding module 82 is used to input the document data to be tested into the encoding module, and respectively encode the text to be tested and the image to be tested to obtain a text encoding representation and an image encoding representation;

[0143] A semantic information extraction module 83, used for inputting the text coding representation and the image coding representation into the semantic extraction module to extract semantic features, so as to obtain text semantic feature representation and image semantic feature representation;

[0144] The emotional information extraction module 84 is used to input the text semantic feature representation and the image semantic feature representation into the emotional tendency detection module to perform emotional tendency detection, and obtain the text emotional tendency feature representation and the image emotional tendency feature representation;

[0145] The sarcasm detection result acquisition module 85 is used to input the text emotion tendency feature representation and the image emotion tendency feature representation into the sarcasm detection module to perform sarcasm detection and obtain the sarcasm detection result of the document data to be tested.

[0146] In an embodiment of the present application, a data acquisition module is used to obtain document data to be tested and a preset multimodal sarcasm detection model, wherein the document data to be tested includes a text to be tested and an image to be tested; the multimodal sarcasm detection model includes an encoding module, a semantic extraction module, an emotional tendency detection module, and a sarcasm detection module; the document data to be tested is input into the encoding module through the document encoding module, and the text to be tested and the image to be tested are encoded and processed respectively to obtain a text encoding representation and an image encoding representation; the text encoding representation and the image encoding representation are input into the semantic extraction module through the semantic information extraction module to extract semantic features, and obtain a text semantic feature representation and an image semantic feature representation; the text semantic feature representation and the image semantic feature representation are input into the emotional tendency detection module through the emotional information extraction module to perform emotional tendency detection, and obtain a text emotional tendency feature representation and an image emotional tendency feature representation; the text emotional tendency feature representation and the image emotional tendency feature representation are input into the sarcasm detection module through the sarcasm detection result acquisition module to perform sarcasm detection, and obtain a sarcasm detection result of the document data to be tested. By extracting semantic information from the document data to be tested, and further extracting text-level and image-level sentiment feature representations based on the semantic information, sarcasm recognition is performed. The relevance of sentiment information in text and image modalities to sarcasm detection is fully considered, thereby improving the accuracy and efficiency of multimodal sarcasm detection.

[0147] Please refer to Fig. 9 , Fig. 9 The structural diagram of the computer device provided for the fifth embodiment of the present application, the computer device 9 includes: a processor 91, a memory 92, and a computer program 93 stored in the memory 92 and executable on the processor 91; the computer device can store a plurality of instructions, and the instructions are suitable for the processor 91 to load and execute the method steps of the first to third embodiments above. The specific execution process can refer to the specific description of the first to third embodiments, which will not be repeated here.

[0148] Among them, the processor 91 may include one or more processing cores. The processor 91 uses various interfaces and lines to connect various parts in the server, and executes various functions and processes data of the multimodal ridicule detection device 8 by running or executing instructions, programs, code sets or instruction sets stored in the memory 92, and calling the data in the memory 92. Optionally, the processor 91 can be implemented in at least one hardware form of digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field-Programmable Gate Array, FPGA), and programmable logic array (Programble Logic Array, PLA). The processor 91 can integrate one or more combinations of a central processing unit 91 (Central Processing Unit, CPU), a graphics processor 91 (Graphics Processing Unit, GPU) and a modem. Among them, the CPU mainly processes the operating system, user interface and application programs; the GPU is responsible for rendering and drawing the content to be displayed on the touch display screen; the modem is used to process wireless communication. It can be understood that the above-mentioned modem may not be integrated into the processor 91, and it can be implemented by a single chip.

[0149] Among them, the memory 92 may include a random access memory 92 (Random Access Memory, RAM), and may also include a read-only memory 92 (Read-Only Memory). Optionally, the memory 92 includes a non-transitory computer-readable storage medium. The memory 92 can be used to store instructions, programs, codes, code sets or instruction sets. The memory 92 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as touch instructions, etc.), instructions for implementing the above-mentioned various method embodiments, etc.; the data storage area may store data involved in the above-mentioned various method embodiments, etc. The memory 92 may also be optionally at least one storage device located away from the aforementioned processor 91.

[0150] An embodiment of the present application also provides a storage medium, which can store multiple instructions. The instructions are suitable for being loaded by a processor and executing the method steps of the first to third embodiments shown. The specific execution process can refer to the specific description of the first to third embodiments shown, which will not be repeated here.

[0151] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0152] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0153] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the target application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each target application, but such implementation should not be considered to be beyond the scope of the present invention.

[0154] In the embodiments provided by the present invention, it should be understood that the disclosed devices / terminal equipment and methods can be implemented in other ways. For example, the device / terminal equipment embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0155] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0156] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0157] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and when the computer program is executed by a processor, the steps of the above-mentioned method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form, etc.

[0158] The present invention is not limited to the above-mentioned embodiments. If various changes or modifications to the present invention do not depart from the spirit and scope of the present invention, and if these changes and modifications fall within the scope of the claims and equivalent technologies of the present invention, the present invention is also intended to include these changes and modifications.

Claims

1. A multimodal taunting detection method, characterized in that: The following steps are involved: Obtaining document data to be tested and a preset multimodal sarcasm detection model, wherein the document data to be tested includes text to be tested and an image to be tested; the multimodal sarcasm detection model includes an encoding module, a semantic extraction module, a sentiment tendency detection module, and a sarcasm detection module; Inputting the document data to be tested into the encoding module, encoding the text to be tested and the image to be tested respectively to obtain a text encoding representation and an image encoding representation; Inputting the text encoding representation and the image encoding representation into the semantic extraction module to extract semantic features, thereby obtaining text semantic feature representation and image semantic feature representation; Inputting the text semantic feature representation and the image semantic feature representation into the sentiment tendency detection module, calculating sentiment scores according to the word embedding vectors of several words in the text encoding representation, obtaining sentiment score vectors of several words, adding the sentiment score vectors of several words in pairs, obtaining sentiment association vectors between several words, and constructing a text sentiment adjacency matrix; Normalizing the text sentiment adjacency matrix to obtain a normalized text sentiment adjacency matrix; extracting sentiment tendency features based on the text semantic feature representation and the normalized text sentiment adjacency matrix to obtain a text sentiment tendency feature representation; performing graph convolution processing based on the image semantic feature representation to obtain an image sentiment tendency feature representation; The text emotion tendency feature representation and the image emotion tendency feature representation are input into the sarcasm detection module, and an interactive attention method is adopted to perform feature interaction according to the text emotion tendency feature representation, the image emotion tendency feature representation and a preset first interactive attention calculation algorithm to obtain a text-to-image interactive attention feature representation, wherein the first interactive attention calculation algorithm is: In the formula, H t2v is the text-to-image interactive attention feature representation, Softmax() is the normalized exponential function, W q is the first weight parameter of the taunt detection module, W k is the second weight parameter of the taunt detection module, W e is the third weight parameter of the taunt detection module, b q is the first bias parameter of the taunt detection module, b k is the second bias parameter of the taunt detection module, b e is the third bias parameter of the taunting detection module, d is the dimension parameter, and T is the transposition symbol; Perform global feature extraction on the text-to-image interactive attention feature representation to obtain a text-to-image global feature representation, perform sarcasm probability vector calculation based on the text-to-image global feature representation to obtain a predicted sarcasm probability vector, and obtain a sarcasm detection result of the document data to be tested based on the predicted sarcasm probability vector.

2. The multimodal taunting detection method according to claim 1, characterized in that: The encoding module includes a word embedding module and a target detection module; The step of inputting the document data to be tested into the encoding module, encoding the text to be tested and the image to be tested respectively, and obtaining a text encoding representation and an image encoding representation comprises the following steps: Inputting the test text into the word embedding module for encoding processing to obtain the text encoding representation, wherein the text encoding representation includes word embedding vectors of several words; The image to be tested is divided into a plurality of image sub-regions, and the plurality of image sub-regions are input into the target detection module for target detection to obtain the image coding representation, wherein the image coding representation includes detection vectors of the plurality of image sub-regions.

3. The multimodal taunting detection method according to claim 2, characterized in that: The semantic extraction module includes a text semantic extraction module and an image semantic extraction module; the text semantic extraction module includes a plurality of stacked text semantic feature extraction layers, and the image semantic extraction module includes a plurality of stacked image semantic feature extraction layers; The step of inputting the text encoding representation and the image encoding representation into the semantic extraction module to extract semantic features and obtain text semantic feature representation and image semantic feature representation comprises the following steps: The text encoding representation is used as the input representation of the first-layer text semantic feature extraction layer of the text semantic extraction module, and multi-head self-attention processing and forward propagation processing are sequentially performed according to the text encoding representation to obtain the output representation of the first-layer text semantic feature extraction layer as the input representation of the next-layer text semantic feature extraction layer, until the output representation of the last-layer text semantic feature extraction layer is obtained as the text semantic feature representation; The image coding representation is used as the input representation of the first-layer image semantic feature extraction layer of the image semantic extraction module, and forward propagation processing and multi-head self-attention processing are performed in sequence according to the image coding representation to obtain the output representation of the first-layer image semantic feature extraction layer as the input representation of the next-layer image semantic feature extraction layer, until the output representation of the last-layer image semantic feature extraction layer is obtained as the image semantic feature representation.

4. The multimodal taunting detection method according to claim 3, characterized in that: The method further comprises the steps of: training the multimodal sarcasm detection model, wherein the training of the multimodal sarcasm detection model comprises the steps of: Obtain a training document data set, input the training document data set into the multimodal sarcasm detection model, and obtain text sentiment tendency feature representations, image sentiment tendency feature representations, and predicted sarcasm probability vectors of a plurality of training document data; Calculate sentiment tendency probability vectors according to the text sentiment tendency feature representations and image sentiment tendency feature representations of the plurality of training document data, and obtain the text sentiment tendency probability vectors and image sentiment tendency probability vectors of the plurality of training document data; Performing a divergence loss calculation based on the text sentiment tendency probability vectors and the image sentiment tendency probability vectors of the training document data to obtain a divergence loss value; Obtaining a number of true ridicule probability vectors of training document data, performing classification loss calculation based on the predicted ridicule probability vectors of the training document data and the true ridicule probability vectors, and obtaining a classification loss value; The multimodal mockery detection model is trained according to the divergence loss value and the classification loss value.

5. The multimodal taunting detection method according to claim 4, characterized in that: The training of the multimodal sarcasm detection model further includes the steps of: Obtaining a text-to-image global feature representation and an image-to-text global feature representation of a plurality of training document data; Performing similarity loss calculation based on the text-to-image global feature representation and the image-to-text global feature representation of the plurality of training document data to obtain a similarity loss value; The multimodal mockery detection model is trained according to the divergence loss value, the classification loss value, and the similarity loss value.

6. A device applied to the multimodal taunting detection method according to any one of claims 1 to 5, characterized in that: include: A data acquisition module, used to obtain document data to be tested and a preset multimodal sarcasm detection model, wherein the document data to be tested includes text to be tested and an image to be tested; the multimodal sarcasm detection model includes an encoding module, a semantic extraction module, a sentiment tendency detection module and a sarcasm detection module; A document encoding module, used for inputting the document data to be tested into the encoding module, encoding the text to be tested and the image to be tested respectively, and obtaining a text encoding representation and an image encoding representation; A semantic information extraction module, used for inputting the text encoding representation and the image encoding representation into the semantic extraction module to extract semantic features, so as to obtain text semantic feature representation and image semantic feature representation; The emotional information extraction module is used to input the text semantic feature representation and the image semantic feature representation into the emotional tendency detection module to perform emotional tendency detection, and obtain the text emotional tendency feature representation and the image emotional tendency feature representation; The sarcasm detection result acquisition module is used to input the text sentiment tendency feature representation and the image sentiment tendency feature representation into the sarcasm detection module to perform sarcasm detection and obtain the sarcasm detection result of the document data to be tested.

7. A computer device, characterized in that: The invention comprises a processor, a memory and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps of the multimodal ridicule detection method according to any one of claims 1 to 5 are implemented.

8. A storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by the processor, the steps of the multimodal ridicule detection method according to any one of claims 1 to 5 are implemented.

Images