Multi-modal sarcasm detection method and system based on dual uncertainty guided network

Abstract

The application provides a kind of multi-modal sarcasm detection method and system based on double uncertainty guide network, it is related to artificial intelligence and emotional processing technical field, comprising: obtaining text data and image data;Text data and image data are respectively encoded, and text feature sequence and image feature sequence are obtained;Text feature sequence and image feature sequence are processed by cross-modal graph semantic interaction, text relationship graph and image relationship graph are constructed, and semantic interaction is carried out by graph attention network and cross-modal attention mechanism, to obtain enhanced text features and image features;The decision uncertainty quantization processing is carried out to the enhanced text features and image features;Fusion latent variable and enhanced features, and carry out multi-modal feature fusion by channel attention mechanism, to obtain fusion features;Sarcasm classification prediction is carried out based on fusion features, and the classification result is output.The application solves the problem of multi-modal data and model decision uncertainty in multi-modal sarcasm detection.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and emotion processing technology, and in particular to a multimodal irony detection method and system based on a dual uncertainty guided network. Background Technology

[0002] The statements in this section are merely background information relating to this disclosure and do not necessarily constitute prior art.

[0003] Irony is a common and subtle way of expressing emotions in human communication, often conveying a true emotional attitude contrary to the literal meaning through irony, exaggeration, or contextual contrast. Early research on irony detection mainly focused on unimodal methods, either modeling clues such as semantic inconsistencies at the text level or mining visual cues from object or region-level features at the image level.

[0004] With the booming development of social media, multimodal content combining text and images has become the mainstream form of expressing opinions. Images may imply emotions contrary to text, and text may also use images to create contrast, making satire more subtle and introducing significant data uncertainty. To address this challenge, multimodal satire detection (MSD) has become a research focus, with its core being the identification of inconsistencies between cross-modal data.

[0005] Existing technologies have proposed various multimodal fusion strategies to improve detection performance. For example, Schifanella et al. used feature concatenation; Cai, Pan, Xu et al. introduced various attention mechanisms to enhance intermodal interactions; and Kang, Li et al. used memory networks or modal-level attention for fusion. To further mine deep features, graph neural networks (GNNs) have been widely used in the MSD field: Liang et al. constructed intramodal and cross-modal graphs and used interactive GCNs to learn inconsistency features; Lu et al. combined graph attention and dynamic routing on the "fact / sentiment" dual-branch; and Wei et al. aggregated global information through graph structures.

[0006] However, these existing methods generally overlook a key issue: uncertainty. This issue manifests itself in two main ways: First, data uncertainty, where the inherent noise, metaphors, and ellipsis in social media content lead to semantic ambiguity within and between modalities; second, decision uncertainty, where existing methods largely follow a deterministic paradigm, over-relying on point estimations (such as similarity or probability outputs) without quantifying the reliability of the model's decisions themselves. This makes the models prone to overconfident and erroneous judgments when faced with difficult samples (such as noisy or modally conflicting samples), resulting in insufficient reliability. Summary of the Invention

[0007] To address at least one of the technical problems mentioned above, this invention proposes a dual-uncertainty guided network, which effectively solves the problem of uncertainty in multimodal data and model decision-making in multimodal irony detection.

[0008] The first aspect of this invention provides a multimodal irony detection method based on a dual-uncertainty guided network, comprising:

[0009] Acquire multimodal input data, which includes text data and image data;

[0010] The text data and image data are encoded separately to obtain text feature sequences and image feature sequences;

[0011] Cross-modal graph semantic interaction processing is performed on the text feature sequence and image feature sequence to construct text relationship graph and image relationship graph. Then, semantic interaction is performed through graph attention network and cross-modal attention mechanism to obtain enhanced text features and image features.

[0012] The enhanced text features and image features are subjected to decision uncertainty quantification processing, including generating latent variables through a gated weighted variational information bottleneck strategy and combining a labeled Gaussian prior strategy to quantify decision uncertainty;

[0013] The latent variables and enhanced features are fused together, and multimodal feature fusion is performed through a channel attention mechanism to obtain fused features;

[0014] Based on the fusion features, perform irony classification prediction and output the classification result.

[0015] Furthermore, the encoding process for the text data and image data includes:

[0016] The BERT model is used to encode text data to obtain text feature sequences. , where n is the text length and d is the embedding dimension;

[0017] The ViT model is used to encode image data in blocks to obtain image feature sequences. , where m is the number of image blocks.

[0018] Furthermore, constructing the text relationship graph includes: treating each word in the text as a node and using the spaCy tool to calculate the similarity between words. ,like Greater than the threshold Then, an edge is created between word i and word j to form a text relationship graph. ;

[0019] The construction of the image relationship graph includes: treating each image patch as a node and calculating the cosine similarity between image patches. ,like Greater than the threshold Then, an edge is established between image blocks i and j to form an image relationship graph. .

[0020] Furthermore, the cross-modal graph semantic interaction processing includes: using a graph attention network to aggregate node information in the text relationship graph and the image relationship graph, with the specific node update formula being:

[0021] ;

[0022] in, Representing nodes respectively and The representation at layer l, For attention weights, This is the weight matrix. For the set of neighboring nodes, This is the activation function.

[0023] Furthermore, the gated weighted variational information bottleneck strategy in the decision uncertainty quantification process includes:

[0024] Text features and image features Calculate the weights using a gating network respectively and :

[0025] ;

[0026] ;

[0027] in, It is the sigmoid activation function. , This is the weight matrix. , For bias terms;

[0028] Variational information bottleneck processing is applied to the weighted features to generate latent variables. and :

[0029]

[0030] ;

[0031] in, and These are the mean values ​​of the text and image modal features, respectively. , For standard normal distribution noise, ⊙ represents element-wise multiplication.

[0032] Furthermore, the labeled Gaussian prior strategy in the decision uncertainty quantification process includes the following steps:

[0033] Enhanced text features Text CLS representation Image CLS representation

[0034] Calculate the cosine similarity between the text CLS representation and the image CLS representation;

[0035] Based on the label information of the training samples, Gaussian probability distribution models corresponding to positive and negative samples are dynamically constructed respectively.

[0036] Based on the Gaussian probability distribution model, calculate the probability that the cosine similarity belongs to the positive or negative class;

[0037] The semantic margin is defined based on the probabilities of belonging to the positive and negative classes, and a marginal loss function is constructed based on the semantic margin to optimize the confidence of the model decision.

[0038] A second aspect of the present invention provides a multimodal irony detection system based on a dual-uncertainty guided network, comprising:

[0039] A multimodal data acquisition module is used to acquire multimodal input data, which includes text data and image data;

[0040] A multimodal feature encoding module is used to encode the text data and image data respectively to obtain text feature sequences and image feature sequences;

[0041] The cross-modal graph semantic interaction module is used to perform cross-modal graph semantic interaction processing on the text feature sequence and image feature sequence, construct text relationship graph and image relationship graph, and perform semantic interaction with cross-modal attention mechanism through graph attention network to obtain enhanced text features and image features;

[0042] The decision uncertainty quantification module is used to perform decision uncertainty quantification processing on the enhanced text features and image features, including generating latent variables through a gated weighted variational information bottleneck strategy and combining a label Gaussian prior strategy to quantify decision uncertainty.

[0043] The multimodal fusion module is used to fuse the latent variables and enhanced features, and to perform multimodal feature fusion through a channel attention mechanism to obtain fused features;

[0044] The classification prediction module is used to perform irony classification prediction based on the fused features and output the classification result.

[0045] A third aspect of the present invention provides an electronic device including a memory, a processor, and a program stored in the memory and running on the processor, wherein the processor executes the program to implement the steps of the multimodal irony detection method based on a dual uncertainty guided network as described in the first aspect of the present invention.

[0046] A fourth aspect of the present invention provides a computer-readable storage medium having a program stored thereon that, when executed by a processor, implements the steps of the multimodal irony detection method based on a dual uncertainty guided network as described in the first aspect of the present invention.

[0047] A fifth aspect of the present invention provides a computer program product comprising software code, wherein the program in the software code performs the steps of the multimodal irony detection method based on a dual uncertainty guided network as described in the first aspect of the present invention.

[0048] Compared with existing technologies, the multimodal irony detection method and system based on a dual-uncertainty guided network provided by this invention has the following beneficial effects:

[0049] (1) To address the data uncertainty and semantic ambiguity caused by multimodal data noise, metaphors, etc., this invention constructs an intramodal relation graph to aggregate semantically significant nodes and suppress noise, and explicitly models the alignment relationship between modalities to amplify inconsistency signals, thereby improving the discriminativeness of single-modal representation, alleviating cross-modal ambiguity caused by text-image semantic mismatch, and significantly reducing the semantic ambiguity caused by data uncertainty.

[0050] (2) In view of the technical problems of decision uncertainty in existing models and the resulting overconfidence and unreliable judgment on difficult samples, this invention dynamically adjusts modal weights through gating networks and uses variational information bottlenecks to compress redundant features and retain key clues, thereby achieving the technical effect of suppressing noise interference between modalities, enhancing the compactness and robustness of feature representation, and thus reducing decision uncertainty.

[0051] (3) In order to further solve the above-mentioned decision uncertainty problem and achieve self-calibrated robust discrimination, this invention constructs a label-guided Gaussian distribution space, calculates semantic margins and constructs a marginal loss function, which forces the feature space to be compact within classes and separated between classes. When the sample is at the decision boundary or modal conflict, the output confidence is dynamically reduced instead of being forcibly misclassified, thereby significantly improving the technical effect of the model's discrimination reliability and decision credibility on noisy and contradictory samples.

[0052] (4) Through the collaborative work of the above modules, the method provided by the present invention achieves robust semantic representation and controllable decision confidence in complex multimodal irony detection scenarios, which significantly alleviates the dual uncertainty of data and decision caused by noise, modality mismatch and expression subjectivity. Attached Figure Description

[0053] The accompanying drawings, which form part of this disclosure, are used to provide a further understanding of this disclosure. The illustrative embodiments of this disclosure and their descriptions are used to explain this disclosure and do not constitute an undue limitation of this disclosure.

[0054] Figure 1 This is a flowchart of the multimodal irony detection method based on a dual uncertainty guiding network provided in Embodiment 1 of the present invention;

[0055] Figure 2 This is a schematic diagram of the MAG-TE model framework provided in Embodiment 1 of the present invention;

[0056] Figure 3 This is the first thermogram under different parameter combinations provided in Embodiment 1 of the present invention;

[0057] Figure 4 This is the second thermogram under different parameter combinations provided in Embodiment 1 of the present invention;

[0058] Figure 5 This is a scatter plot of two types of samples in the embedding space without the introduction of LGPS, as provided in Embodiment 1 of the present invention.

[0059] Figure 6 This is a scatter plot of two types of samples in the embedding space when LGPS is introduced, as provided in Embodiment 1 of the present invention;

[0060] Figure 7 This is a schematic diagram of a multimodal irony detection system based on a dual uncertainty guidance network provided in Embodiment 2 of the present invention. Detailed Implementation

[0061] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used in this invention have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0062] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, unless the context clearly indicates otherwise, the singular form is intended to include the plural form as well. Furthermore, it should be understood that the terms “comprising” and “having”, and any variations thereof, are intended to cover non-exclusive inclusion, for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0063] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0064] All data acquisition in this embodiment is carried out in accordance with laws and regulations and with user consent, and the data is used legally.

[0065] Terminology Explanation:

[0066] Multi-modal Sarcasm Detection (MSD): This method identifies and understands the semantics of sarcasm or irony by combining different modalities (such as text, images, videos, etc.).

[0067] The Cross-modal Graph Semantic Interaction Module (CGSIM) effectively interacts the semantic information of text and images by constructing a cross-modal graph structure and combining graph attention networks with cross-modal attention mechanisms, thus alleviating the problem of data uncertainty caused by multimodal semantic ambiguity.

[0068] The Decision Uncertainty Quantification Module (DUQM) quantifies and enhances the model's ability to handle uncertainty during the decision-making process by using the gated weighted variational information bottleneck (GVIB) and the labeled Gaussian prior strategy (LGPS).

[0069] Cross-modal Graph Attention Strategy (CGAS): A strategy designed based on graph attention and cross-modal attention mechanisms that can deeply mine graph structure information and enable cross-modal information interaction.

[0070] Gated Weighted Variational Information Bottleneck (GVIB): Inspired by the variational information bottleneck, and with the addition of a gating structure that assigns different weights to different modal information, GVIB can further enhance feature representation in multimodal irony detection tasks and claims that the loss function constrains the model.

[0071] The Decision Uncertainty Quantification Module (Label Gaussian Prior Strategy, LGPS) constructs a label-guided Gaussian distribution space, calculates semantic margins, and builds a marginal loss function. This forces the feature space to be compact within classes and separated between classes. Furthermore, it dynamically reduces the output confidence when samples are at the decision boundary or in modal conflict, rather than forcibly misclassifying them. This significantly improves the model's discriminative reliability and decision credibility on noisy and contradictory samples.

[0072] Dual Uncertainty Guided Network (DUGN): This invention is a model constructed to effectively address the uncertainty in data and model decision-making in the field of multimodal irony detection.

[0073] Semantic margin: A metric used to quantify the difference in a model’s confidence in judging a sample as either satirical or non-satirical.

[0074] Example 1

[0075] like Figure 1 This embodiment provides a multimodal irony detection method based on a dual-uncertainty guiding network, including:

[0076] Acquire multimodal input data, which includes text data and image data;

[0077] The text data and image data are encoded separately to obtain text feature sequences and image feature sequences;

[0078] Cross-modal graph semantic interaction processing is performed on the text feature sequence and image feature sequence to construct text relationship graph and image relationship graph. Then, semantic interaction is performed through graph attention network and cross-modal attention mechanism to obtain enhanced text features and image features.

[0079] The enhanced text features and image features are subjected to decision uncertainty quantification processing, including generating latent variables through a gated weighted variational information bottleneck strategy and combining a labeled Gaussian prior strategy to quantify decision uncertainty;

[0080] The latent variables and enhanced features are fused together, and multimodal feature fusion is performed through a channel attention mechanism to obtain fused features;

[0081] Based on the fusion features, perform irony classification prediction and output the classification result.

[0082] This embodiment provides a MAG-TE model, the framework of which is as follows: Figure 2 As shown, it mainly consists of three main components: 1) a multimodal feature encoding module, which uses BERT and ViT to encode multimodal data for text and images respectively; 2) a cross-modal graph semantic interaction module (CGSIM), which effectively interacts the semantic information of text and images by constructing a cross-modal graph structure and combining graph attention network with cross-modal attention mechanism, thereby alleviating the semantic ambiguity problem in multimodal data and improving the accuracy of irony detection; 3) a decision uncertainty quantification module (LGPS), which quantifies and enhances the model's ability to handle uncertainty in the decision-making process through gated weighted variational information bottleneck (GVIB) and label Gaussian prior strategy (LGPS).

[0083] Specifically, the encoding process for the text data and image data includes:

[0084] The BERT model is used to encode text data to obtain text feature sequences. , where n is the text length and d is the embedding dimension;

[0085] The ViT model is used to encode image data in blocks to obtain image feature sequences. , where m is the number of image blocks.

[0086] Specifically, constructing the text relationship graph includes: treating each word in the text as a node and using the spaCy tool to calculate the similarity between words. ,like Greater than the threshold Then, an edge is created between word i and word j to form a text relationship graph. ;

[0087] The construction of the image relationship graph includes: treating each image patch as a node and calculating the cosine similarity between image patches. ,like Greater than the threshold Then, an edge is established between image blocks i and j to form an image relationship graph. .

[0088] Given a ,in It is text. Let D be an image and D be a dataset. In this section, , where n represents the length of the text; Where m represents the It was divided into m small blocks. The text was encoded using BERT:

[0089] ;

[0090] Where i is the i-th word, d represents the embedding dimension. For an image, ViT is used to embed each small block of the image:

[0091] ;

[0092] Where j is the j-th small block d represents the embedding dimension.

[0093] Specifically, the cross-modal graph semantic interaction processing includes: using a graph attention network to aggregate node information in the text relationship graph and the image relationship graph, with the specific node update formula being:

[0094] ;

[0095] in, Representing nodes respectively and The representation at layer l, For attention weights, This is the weight matrix. For the set of neighboring nodes, This is the activation function.

[0096] Obtained from MFEM and As input to CGSIM, the first step is graph construction. For a text-based relationship graph, each... As nodes in the graph, spaCy2 is used to extract the relationship strength between each word in the text. To construct the edges, where , This represents the similarity between the i-th and j-th words in the text. For an image relationship graph, each... As nodes in the graph, the cosine similarity of each patch is calculated to extract the strength of the relationship between each patch in the image. To construct the edges , This is the cosine similarity between the i-th and j-th patches in the image. This yields the text relationship graph. Relationship between images .

[0097] Then and To input our cross-modal graph attention strategy (CGAS), we first introduce a graph attention network (GAT) to aggregate information from the image and text relationship graphs. and Each node is set as Its neighboring nodes are The GAT update node is:

[0098] ;

[0099] in, Representing nodes respectively and The representation at layer l, It is attention weight. It is the weight matrix of the l-th layer. It is a node The set of neighboring nodes, It is the activation function. Finally, we get... and .

[0100] Through GAT, nodes can dynamically adjust the weight of information propagation based on their relationship with neighboring nodes, thereby strengthening the information transmission of key nodes and suppressing redundant or irrelevant semantic information. Each node adaptively weights and aggregates features based on its semantic relevance to neighboring nodes.

[0101] After GAT processing, a cross-modal attention mechanism (CAM) is used to provide bidirectional interactive guidance for the text and image relationship graph, as follows:

[0102] ;

[0103] ;

[0104] in, These are text and image query vectors, respectively. These are the key vectors for text and images, respectively, with the superscript T indicating transpose. These are the value vectors of the text and the image, respectively. It is the attention weight matrix. It represents the dimension of the vector. After CGSIM, the semantic alignment of text and image features is further optimized, alleviating the semantic ambiguity between images and text and reducing erroneous judgments caused by semantic ambiguity between modalities.

[0105] Specifically, the gated weighted variational information bottleneck strategy in the quantification of decision uncertainty includes:

[0106] Text features and image features Calculate the weights using a gating network respectively and :

[0107] ;

[0108] ;

[0109] in, It is the sigmoid activation function. , This is the weight matrix. , For bias terms;

[0110] Variational information bottleneck processing is applied to the weighted features to generate latent variables. and :

[0111]

[0112] ;

[0113] in, and These are the mean values ​​of the text and image modal features, respectively. , For standard normal distribution noise, ⊙ represents element-wise multiplication.

[0114] Received and This alleviates the semantic ambiguity problem in multimodal data, but existing methods still cannot solve the uncertainty in decision-making. Therefore, the inventors designed a decision uncertainty quantification module. First, based on the variational information bottleneck, they designed a gated weighted variational information bottleneck (GVIB). Each modal feature first enters a gated network for dynamic weight calculation. Then, weighting operations are performed on text and image features, dynamically adjusting the contribution of each modality to obtain the weighted features.

[0115] ;

[0116] ;

[0117] in and These are the calculated weighting coefficients. It is an activation function that ensures the weight values ​​are between 0 and 1. and It is the weight matrix of the gated network. and This is the bias term. The features weighted by the gated network are then input into the VIB to retain relevant information for the irony detection task while suppressing irrelevant or redundant content. In the VIB, the representations of text and image modalities are modeled using a normal distribution to obtain the mean and standard deviation. Then, latent variables are generated through reparameterization techniques. Details are as follows:

[0118] ;

[0119] ;

[0120] in and These are the mean values ​​of the text and image modal features, respectively. and It is the standard deviation. and It is noise from a standard normal distribution. and These are potential variables in the generated text and images. This represents element-wise multiplication. VIB also generates a loss function:

[0121]

[0122] ;

[0123] in Represents the input Seeking expectations, and The posterior distributions representing text and image features. and It is the prior distribution of text and image features. and These are hyperparameters that control the KL divergence of text and image modalities respectively, helping to balance information bottlenecks and task fitting. It is the KL divergence, which measures the difference between the posterior and prior distributions, controls the size of the information bottleneck, and promotes more compact feature representations.

[0124] Specifically, the labeled Gaussian prior strategy in the decision uncertainty quantification process includes the following steps:

[0125] Enhanced text features Text CLS representation Image CLS representation

[0126] Calculate the cosine similarity between the text CLS representation and the image CLS representation;

[0127] Based on the label information of the training samples, Gaussian probability distribution models corresponding to positive and negative samples are dynamically constructed respectively.

[0128] Based on the Gaussian probability distribution model, calculate the probability that the cosine similarity belongs to the positive or negative class;

[0129] The semantic margin is defined based on the probabilities of belonging to the positive and negative classes, and a marginal loss function is constructed based on the semantic margin to optimize the confidence of the model decision.

[0130] Furthermore, Gaussian probability distribution models corresponding to positive and negative samples are dynamically constructed, including:

[0131] Calculate the mean cosine similarity of positive samples based on the labels of the positive samples in the current batch of training data. and standard deviation ;

[0132] Calculate the mean cosine similarity of negative samples based on the labels of the negative samples in the current batch of training data. and standard deviation ;

[0133] The Gaussian probability distribution model is defined by the mean μ and standard deviation σ.

[0134] Furthermore, the formula for calculating the probability of cosine similarity belonging to the positive and negative classes is as follows:

[0135] .

[0136] In one specific embodiment, GVIB can effectively enhance the feature representation in a multimodal irony detection task, reducing uncertainty in the model's decision-making process. The latent variables generated by GVIB and the feature representation enhanced by CGSIM are fused using self-attention:

[0137]

[0138]

[0139] To further enhance the model's ability to handle decision uncertainty, a labeled Gaussian prior strategy is introduced. First, the... and Perform pooling operations to obtain its CLS representation as follows: and Used to calculate cosine similarity It is used to represent the semantic consistency between text and visual features, serving as the basis for subsequent Gaussian distribution calculations. CLS representation is a processed global vector that represents a summary or extracted information from the input data. Secondly, it utilizes label information... We construct a Gaussian distribution to represent positive and negative samples respectively, and calculate the mean and standard deviation of the cosine similarity between the positive and negative samples:

[0140] ;

[0141] ;

[0142] in and These represent the number of positive and negative samples, respectively. , and , These represent the mean and standard deviation of the cosine similarity between positive and negative samples, respectively. Then, based on the probability density function of the Gaussian distribution, the probability of each sample belonging to the positive and negative classes is calculated:

[0143] ;

[0144] Next, we define semantic margins. This is used to measure the difference in credibility between positive and negative classes. A larger difference... This indicates that the model has high confidence in its ironic decision regarding this sample, and the final objective loss function is:

[0145] .

[0146] Semantic margin is the difference between the model's predicted probabilities for the satirical and non-satirical categories, aiming to measure the model's clarity regarding the decision boundary. Specifically:

[0147] 1) If the semantic margin is large (close to 1), it means that the model is very certain in its judgment of the sample, and the boundaries between categories are clear, and the model's output has high confidence.

[0148] 2) If the semantic margin is small (close to 0), it means that the model is uncertain about the discrimination of the sample, the boundary between categories is blurred, and the model has low confidence in making decisions.

[0149] The role of semantic margins in the model:

[0150] 1) High semantic margin: When the semantic margin is large, the model has high confidence in its classification decision for a particular sample. The model can clearly distinguish between ironic and non-ironic samples, especially when semantic alignment is good and signals do not conflict. In this case, the model's decision is reliable and robust.

[0151] 2) Low semantic margin: When the "semantic margin" is small, the model's classification decision for the sample is uncertain, indicating that the sample may be in the boundary region of the class or there is significant conflict between modalities. For example, when the information from text and image modalities is inconsistent, the model may generate lower decision confidence and tend to output lower probability differences. In this case, the model's output will be more cautious to avoid making overconfident and erroneous judgments.

[0152] Dynamic adjustment and application:

[0153] During the decision-making process, the dynamic changes in semantic margins affect the model's output confidence. Specifically, if the model determines that a sample is located in a "hard-to-determine" region, and there is significant conflict between cross-modal information (such as text and image information), the semantic margin will decrease accordingly, and the model will lower its decision confidence for that sample. In this case, the model's decision confidence will be significantly reduced to avoid making incorrect judgments due to overconfidence.

[0154] Experimental setup

[0155] 1) Dataset: The model was evaluated on the public benchmark dataset MMSD, which is a dataset constructed based on tweets from Twitter as samples. MMSD contains 24,635 samples, of which the training set, validation set, and test set contain 19,815, 2,410, and 2,409 samples, respectively. Each sample consists of text and images, and some texts also contain tagged words and emoji words.

[0156] 2) Implementation details: Pre-trained language models BERT and ViT are used to embed text and image modalities respectively. The learning rate is 1e-5, the number of training epochs is 25, the batch size is 32, and the loss function coefficients are constrained. and They are 0.05 and 1.0 respectively.

[0157] 3) Baseline: The method provided by this invention is compared with several recent advanced methods: single-modal methods include ResNet, Vit, TextCNN, BERT; multimodal methods include HFM, D&R, Att-BERT, HKEModel, DMSD-CL, DIP, G2SAM, AFIM, SAPL.

[0158] Results and Analysis

[0159] The superiority of DUGN is demonstrated through a detailed comparison of performance metrics with other baseline models (see Table 1). It can be observed that: 1) Neither single-text nor single-image modal methods outperform multimodal models, indicating that multimodal methods provide more comprehensive satirical information, and utilizing both text and image information improves the model's satirical detection performance. 2) DUGN outperforms all models across all metrics. Specifically, compared to methods that have not addressed data and decision uncertainty in recent years, the method provided in this invention achieves F1 scores 2.47%, 1.64%, 2.29%, 4.23%, and 1.47% higher than DIP, G2SAM, AFIM, SAPL, and KFGC-Net, respectively; compared to the LLM-based model DMSD-CL, DUGN also shows significant improvements across all evaluation metrics. This demonstrates that DUGN effectively addresses the uncertainty of multimodal data and model decision-making through CGSIM and DUQM.

[0160] Table 1. Performance comparison and evaluation metrics of DUGN on the MMSD dataset with all baselines.

[0161]

[0162] In addition, the inventors conducted ablation studies to evaluate the impact of each component on the model (as shown in Table 2). First, the lack of graph construction led to a decrease in model performance, indicating that graph structures are more conducive to the model's mining of text and image features. Second, the lack of CGAS showed a significant decrease in model performance, indicating that cross-modal interaction further optimized semantic alignment and alleviated the semantic ambiguity between images and text. Third, the lack of GVIB also resulted in a decrease in model performance, indicating that without GVIB, the model lacks the ability to dynamically remove redundancy and suppress uncertainty in cross-modal features, leading to increased noise interference between modalities and insufficient extraction of key information. Finally, the lack of LGPS showed a significant decrease in model performance, indicating that the model lost the ability to explicitly model and distinguish the semantic boundaries of positive and negative class samples, leading to increased confusion of cross-modal features in the decision space and decreased inter-class separability. This also shows that the uncertainty of model decisions is very important in irony detection.

[0163] Table 2. Ablation studies of different components using the MAG-TE model

[0164]

[0165] To evaluate the impact of each loss term on the model, we studied two hyperparameters of the total loss, keeping all other settings unchanged except for the hyperparameter under investigation. Figure 3 and Figure 4 The two heatmaps provided show the different ( , Performance changes under different combinations. It is quite obvious that: 1) When When the weights are small (close to 0), the redundancy removal / uncertainty suppression effect of the information bottleneck is insufficient, cross-modal noise is difficult to suppress, and performance is low; as the weights increase, performance gradually improves; when the weights are too large, excessive compression leads to information loss and underfitting, resulting in reduced performance. 2) Too small a weight will result in insufficient prototype separation / semantic boundary modeling, and positive and negative classes will easily overlap in the feature space; too large a weight will excessively widen the inter-class margin, amplify the penalty for hard examples / boundary samples, and lead to misclassification and decreased stability. Based on the above research, the models provided in this invention all adopt: =0.05, =1.0.

[0166] To further explore the impact of the proposed LGPS on model decision uncertainty, t-SNE was used to visualize the features in two dimensions. Specifically, under two configurations—"without LGPS" and "with LGPS"—the fused multimodal feature representation was extracted from the penultimate layer of the classifier. t-SNE dimensionality reduction was then performed on the same batch of samples with the same random seed and hyperparameter settings to ensure comparability of the results. The visualization results are shown below. Figure 5 and Figure 6 As shown, from Figure 5 It can be seen that without LGPS, the two classes of samples have a large overlap in the embedding space, with loose cluster distribution and blurred boundaries. This indicates that the model is deficient in both intra-class compactness and inter-class separability, and is prone to confusion between ironic and non-ironic semantics during model decision-making. In contrast, Figure 6 The distribution after introducing LGPS is shown: intra-class samples are more tightly clustered, inter-class decision boundaries are significantly clear, ironic and non-ironic samples form a high-density clustered distribution, and the overlapping area is significantly reduced.

[0167] This experimental result is highly consistent with the design goals of LGPS. LGPS enhances the discriminative power of model decisions by explicitly labeling and guiding prototype separation and semantic boundary constraints, thereby compressing intra-class variance and expanding inter-class margins. The visualization results intuitively verify that LGPS effectively addresses the uncertainty problem in decision-making during multimodal irony detection.

[0168] Example 2

[0169] like Figure 7 As shown, this embodiment provides a multimodal irony detection system based on a dual-uncertainty guided network, including:

[0170] A multimodal data acquisition module is used to acquire multimodal input data, which includes text data and image data;

[0171] A multimodal feature encoding module is used to encode the text data and image data respectively to obtain text feature sequences and image feature sequences;

[0172] The cross-modal graph semantic interaction module is used to perform cross-modal graph semantic interaction processing on the text feature sequence and image feature sequence, construct text relationship graph and image relationship graph, and perform semantic interaction with cross-modal attention mechanism through graph attention network to obtain enhanced text features and image features;

[0173] The decision uncertainty quantification module is used to perform decision uncertainty quantification processing on the enhanced text features and image features, including generating latent variables through a gated weighted variational information bottleneck strategy and combining a label Gaussian prior strategy to quantify decision uncertainty.

[0174] The multimodal fusion module is used to fuse the latent variables and enhanced features, and to perform multimodal feature fusion through a channel attention mechanism to obtain fused features;

[0175] The classification prediction module is used to perform irony classification prediction based on the fused features and output the classification result.

[0176] Example 3

[0177] Embodiment 3 of the present invention provides an electronic device.

[0178] An electronic device includes a memory, a processor, and a program stored in the memory and running on the processor. When the processor executes the program, it implements the steps in the multimodal irony detection method based on a dual uncertainty guided network as described in Embodiment 1 of the present invention.

[0179] The detailed steps are the same as those of the multimodal irony detection method based on a dual uncertainty guided network provided in Example 1, and will not be repeated here.

[0180] Example 4

[0181] Embodiment 4 of the present invention provides a computer-readable storage medium.

[0182] A computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the steps of the multimodal irony detection method based on a dual uncertainty guided network as described in Embodiment 1 of the present invention.

[0183] The detailed steps are the same as those of the multimodal irony detection method based on a dual uncertainty guided network provided in Example 1, and will not be repeated here.

[0184] Example 5

[0185] Embodiment 5 of the present invention provides a computer program product.

[0186] A computer program product includes software code, wherein the program in the software code performs the steps of the multimodal irony detection method based on a dual uncertainty guided network as described in Embodiment 1 of the present invention.

[0187] The detailed steps are the same as those of the multimodal irony detection method based on a dual uncertainty guided network provided in Example 1, and will not be repeated here.

[0188] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The solutions in the embodiments of the present invention can be implemented using various computer languages, such as the object-oriented programming language Java and the interpreted scripting language JavaScript.

[0189] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, as well as combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0190] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0191] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0192] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0193] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

[0194] The above description is merely a preferred embodiment of this practice and is not intended to limit the scope of this practice. Various modifications and variations can be made to this practice by those skilled in the art. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of this practice should be included within the protection scope of this practice.

Claims

1. A multi-modal sarcasm detection method based on a dual uncertainty guided network, characterized in that, The method comprises the following steps: acquiring multi-modal input data, wherein the multi-modal input data comprises text data and image data; encoding the text data and the image data respectively to obtain a text feature sequence and an image feature sequence; performing cross-modal graph semantic interaction processing on the text feature sequence and the image feature sequence to construct a text relation graph and an image relation graph, and performing semantic interaction through a graph attention network and a cross-modal attention mechanism to obtain enhanced text features and image features; quantifying decision uncertainty of the enhanced text features and image features, including generating latent variables through a gated weighted variational information bottleneck strategy, and quantifying decision uncertainty in combination with a label Gaussian prior strategy; fusing the latent variables and the enhanced features, and performing multi-modal feature fusion through a channel attention mechanism to obtain fused features; performing satire classification prediction based on the fused features, and outputting a classification result.

2. The method of claim 1, wherein, The encoding of the text data and the image data comprises the following steps: The text data is encoded using a BERT model to obtain a text feature sequence where n is the length of the text and d is the embedding dimension; The image data is encoded in blocks using a ViT model to obtain an image feature sequence where m is the number of image blocks.

3. The method of claim 1, wherein, The constructing text relation graph includes: taking each word in the text as a node, calculating the similarity between the words using the spaCy tool , if is greater than a threshold , an edge is established between the word i and the word j, forming a text relation graph ; The constructing the image relation graph comprises: taking each image block as a node, and calculating cosine similarity between the image blocks , if is greater than a threshold value , an edge is established between the image blocks i and j to form the image relation graph .

4. The method of claim 3, wherein, The cross-modal graph semantic interaction processing comprises aggregating node information of the text relation graph and the image relation graph using the graph attention network, and a specific node update formula is as follows: ； wherein, respectively represent nodes and representations of nodes at the l-th layer, is an attention weight, is a weight matrix, is a set of neighbor nodes, is an activation function.

5. The method of claim 1, wherein, The gated weighted variational information bottleneck strategy in the decision uncertainty quantification processing comprises the following steps: on text features and image features respectively by a gating network and : ； ； wherein, sigmoid is a sigmoid activation function, , W is a weight matrix, , bias is a bias term; performing variational information bottleneck processing on the weighted features to generate latent variables and : ； where, and are the mean of text and image modality features, respectively, , is a standard normal distributed noise, and ⊙ denotes element-wise multiplication.

6. The method of claim 1, wherein, The label Gaussian prior strategy in the decision uncertainty quantification processing comprises the following steps: performing pooling operations on the enhanced text features and image features to obtain text CLS representations and image CLS representations; calculating a cosine similarity between the text CLS representations and the image CLS representations; dynamically constructing Gaussian probability distribution models corresponding to positive samples and negative samples based on label information of training samples; calculating probabilities that the cosine similarity belongs to the positive class and the negative class according to the Gaussian probability distribution models; defining semantic margins according to the probabilities that belong to the positive class and the negative class, and constructing a marginal loss function based on the semantic margins to optimize the confidence of model decision.

7. A multi-modal sarcasm detection system based on dual uncertainty guided network, characterized in that, The method comprises the following steps: a multi-modal data acquisition module is configured to acquire multi-modal input data, wherein the multi-modal input data comprises text data and image data; a multi-modal feature encoding module is configured to encode the text data and the image data respectively to obtain a text feature sequence and an image feature sequence; a cross-modal graph semantic interaction module is configured to perform cross-modal graph semantic interaction processing on the text feature sequence and the image feature sequence to construct a text relation graph and an image relation graph, and perform semantic interaction through a graph attention network and a cross-modal attention mechanism to obtain enhanced text features and image features; a decision uncertainty quantification module is configured to quantify decision uncertainty of the enhanced text features and image features, including generating latent variables through a gated weighted variational information bottleneck strategy, and quantifying decision uncertainty in combination with a label Gaussian prior strategy; a multi-modal fusion module is configured to fuse the latent variables and the enhanced features, and perform multi-modal feature fusion through a channel attention mechanism to obtain fused features; a classification prediction module is configured to perform satire classification prediction based on the fused features, and output a classification result.

8. An electronic device comprising a memory, a processor, and a computer program stored on the memory and running on the processor, characterized in that, The processor implements the steps of the method for multi-modal sarcasm detection based on a dual uncertainty guided network according to any one of claims 1-6 when executing the program.

9. A computer readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the steps of the method for multi-modal sarcasm detection based on a dual uncertainty guided network according to any one of claims 1-6.

10. A computer program product comprising software code, characterized in that, The program in the software code implements the steps of the method for multi-modal sarcasm detection based on a dual uncertainty guided network according to any one of claims 1-6.

Images