Multimodal sarcasm detection method based on generative counterfactual reasoning and multi-granularity conflict quantification

Abstract

The application discloses a kind of based on generative counterfactual reasoning and multi-granularity conflict quantification multimodal sarcasm detection method, the method includes the following steps: step 1: multi-modal semantic anchor point analysis and hierarchical visual feature extraction;Step 2: double-path counterfactual semantic deduction and contrast baseline construction;Step 3: macro fact alignment and sentence-level semantic conflict measurement;Step 4: micro local mapping and word-level rhetoric contradiction quantification;Step 5: multi-granularity conflict feature dynamic fusion and adaptive determination.The application can accurately understand the deep context of picture-text interaction and accurately identify the existence of implicit sarcastic semantics in multi-modal data, improve the limitations of traditional discriminative sarcasm detection methods in handling complex picture-text contradictions and context-dependent sarcasm, and significantly improve the detection accuracy and reliability of multi-modal sarcastic expression through counterfactual comparison space construction and multi-granularity conflict quantification.

Description

Technical Field

[0001] This invention relates to a multimodal irony detection method based on generative counterfactual reasoning and multigranular conflict quantification, belonging to the fields of Internet and artificial intelligence technology. Background Technology

[0002] Multimodal irony detection, a crucial frontier research area at the intersection of natural language processing and affective computing, aims to identify implicit cross-modal semantic contrasts and intent reversals by aligning heterogeneous information flows through algorithmic models. The core challenge of multimodal irony detection lies in capturing the logical contrast between literal expressions and the objectively realistic visual environment. For example, a picture of an empty parking lot paired with the text "The welcome center is bustling today," or a picture of dangerously exposed electrical wires paired with the text "That was a beautiful job." Recognizing such semantic reversals relies not only on single textual features but also requires the model to possess multimodal contextual cognitive reasoning capabilities to capture these deep pragmatic contradictions. Therefore, this is not only a cross-modal feature fusion task but also a comprehensive test of the machine's complex cognitive reasoning capabilities, possessing significant academic theoretical value and broad application prospects.

[0003] Despite significant technological advancements, accurate detection of multimodal satire remains a major challenge. Early and traditional discriminative satire detection models (such as hierarchical fusion models or graph neural networks (GCNs) primarily rely on shallow feature fusion or complex cross-modal interaction networks, attempting to forcibly align static visual regions with textual markers to find conflict cues. However, these methods are fundamentally limited to mining surface-level statistical correlations and severely lack deep cognitive reasoning capabilities. They operate within closed semantic spaces and often exhibit extremely poor robustness and generalization ability when faced with complex satirical expressions in open-world scenarios that heavily rely on implicit common sense rather than explicit visual features.

[0004] With the development of large-scale multimodal models, the research frontier is shifting from discriminative classification to generative understanding. Existing methods attempt to leverage the powerful external commonsense knowledge base of pre-trained large models for zero-shot or few-shot sarcasm detection, such as by enhancing the decision context of text through thought chains or image-dense descriptions. However, directly applying general-purpose large models faces a significant alignment gap: large models are primarily pre-trained to predict the next label, rather than specifically optimized for pragmatic conflict (inconsistency) detection. More critically, existing large-model-based methods typically treat the model merely as a direct classifier or simple interpreter, analyzing given input samples in isolation. This completely ignores the crucial cognitive mechanism of counterfactual reasoning in human sarcasm recognition. The lack of a clear, non-sarcastic semantic comparison baseline makes it extremely difficult for existing large models to distinguish genuine implicit sarcasm from ordinary background noise or loose semantic associations.

[0005] Addressing the cognitive limitations of existing research in multimodal satire detection, particularly the limitations in counterfactual baseline construction, this invention introduces a multimodal satire detection method based on generative counterfactual reasoning and multi-granular conflict quantification. Unlike traditional methods relying on static feature matching or large-scale isolated model prediction, this invention utilizes a large-scale multimodal model fine-tuned by instructions as a constructor for comparative semantic anchors, actively synthesizing both objective factual contexts describing the visual scene and satirical contexts explicitly expressing satirical intent. Based on these two generated prototypes, this invention constructs a ternary contrast space. Combined with grammatically guided hierarchical fine-grained visual feature extraction, the system can accurately quantify the semantic tension of the original input deviating from visual facts and tending towards subjective satire at both the macro-sentence level and the micro-word level. This method explicitly simulates the human process of comparative cognition and counterfactual reasoning, significantly improving the ability to resolve implicit conflicts in complex multimodal contexts.

[0006] More importantly, this method demonstrates extremely broad practical value in real-world applications. Multimodal satire is prevalent in various rich media interactive scenarios, including social media posts, short video comments, online forums, and news images. Accurately identifying satirical images not only helps improve the understanding boundaries of AI systems regarding complex human intentions but also plays an irreplaceable role in areas such as online public opinion analysis, brand reputation management, automatic multimedia content review, and intelligent human-computer interaction. For example, on social media platforms, this invention can be used to screen for sarcastic or insinuating comments that do not match the text in real time, assisting platforms in conducting more accurate public opinion monitoring and content governance; in corporate public opinion monitoring, this method can help brands accurately identify potential negative jokes under image comments, avoiding misjudgments due to incorrect extraction of literal positive emotions; and in the secure alignment of multimodal large models, this technology can also serve as an important advanced semantic understanding filter.

[0007] Ultimately, this invention constructs a new paradigm for image-text satire detection with deep cognitive reasoning capabilities by integrating large-scale dual-path counterfactual generation with multi-granularity geometric space conflict measurement. This method effectively overcomes the dual technical bottlenecks of existing discriminative models lacking reasoning ability and generative models lacking comparative baselines, achieving a leap from simple feature fitting to deep cognitive reasoning. It not only theoretically advances the development of affective computing and cross-modal alignment technologies but also demonstrates significant practical value and scalability in industrial-grade multimedia semantic understanding applications. Summary of the Invention

[0008] To address the problems and shortcomings of existing technologies, this invention proposes a multimodal satire detection method based on generative counterfactual reasoning and multi-granularity conflict quantification. Addressing the issues of existing discriminative methods' over-reliance on surface statistical features and lack of cognitive reasoning ability, and existing generative methods' lack of comparative baselines and difficulty in capturing implicit text-image contradictions, this invention constructs a detection framework from two dimensions: counterfactual deduction and multi-granularity conflict modeling. This invention introduces a large-scale multimodal model to construct a dual-path counterfactual reasoning mechanism, dynamically synthesizing objective factual prototypes describing visual truth and satirical semantic prototypes reflecting subjective biases, thereby constructing an explicit semantic comparison space for the original input. Based on this, this invention constructs a hybrid measurement system composed of macro-level sentence-level semantic conflict and micro-level word-level semantic conflict through grammatically guided hierarchical feature extraction. This method can not only capture global semantic drift at the sentence level through a ternary comparison space, but also accurately pinpoint micro-level rhetorical contradictions caused by specific keywords through dual factual baselines, thereby comprehensively improving the model's modeling ability and decision robustness for complex, implicit multimodal satirical expressions.

[0009] To achieve the above objectives, the technical solution of the present invention is as follows: a multimodal irony detection method based on generative counterfactual reasoning and multi-granularity conflict quantification, the method comprising the following steps:

[0010] Step 1: Multimodal semantic anchor point parsing and hierarchical visual feature extraction;

[0011] Step 2: Dual-path counterfactual semantic inference and baseline construction;

[0012] Step 3: Macro-level fact alignment and sentence-level semantic conflict measurement;

[0013] Step 4: Quantifying the contradiction between microscopic local mapping and word-level rhetoric;

[0014] Step 5: Dynamic fusion and adaptive determination of multi-granularity conflict features.

[0015] As an improvement of this invention, step 1: Multimodal semantic anchor parsing and hierarchical visual feature extraction. First, deep grammar parsing technology is used to perform syntactic dependency analysis on the input text, extracting core structures such as adjective modification, negation modification, and adverbial modification chains as potential ironic rhetorical units. Polarity determination and sentiment weight calculation are performed using a sentiment dictionary, and a set of high-weight sentiment keywords is selected as semantic anchors. Subsequently, a pre-trained visual encoder is used to extract the global visual feature sequence of the input image to preserve the complete spatial and texture details of the image. Simultaneously, driven by the high-weight sentiment keywords, a grammar-guided fine-grained visual attention mechanism is constructed. By calculating the semantic relevance between text keywords and image block sequences, local visual features strictly aligned with key entities are dynamically extracted. The core objective of this step is to break the rigid limitations of traditional fixed bounding boxes, actively guide the precise and multi-level selection of visual information through the core semantics of the text, change the limitations of passive splicing of traditional modal features, and provide high-quality, strongly aligned cross-modal underlying data support for the accurate quantification of subsequent multi-level text-image counterfactual conflicts. The implementation of this step can be divided into the following sub-steps:

[0016] Sub-step 1-1: Syntactic parsing and sentiment weighting of core semantic anchors in the text. In this step, deep syntax parsing technology is used to analyze the original input text. Dependency parsing was performed to extract core structures containing adjective modification, negation modification, and adverbial modification chains as potential ironic rhetorical units; positive sentiment scores for each rhetorical unit were obtained by combining a sentiment lexicon. With negative sentiment score And based on whether there is a negation marker in its dependency path, the sentiment weight of a single word is calculated. The calculation formula is as follows:

[0017]

[0018] Furthermore, to effectively filter out redundant words with low information content, a preset threshold for sentiment saliency assessment is set. (In a preferred embodiment of the present invention, the preset threshold) (Set to 0.2). The sentiment weights of each word will be... With preset threshold By comparing and filtering words with sentiment weights below a certain threshold, a set of high-weight sentiment keywords is selected and generated. The set The set and its corresponding emotional weights will play a dual role: on the one hand, the set As semantic anchors for cross-modal feature alignment, they are used to actively guide the dynamic extraction of local visual features in subsequent steps; on the other hand, they retain the sentiment weight scalars of each high-weight keyword. It is used as a calculation factor to measure the rhetorical contribution of the corresponding keywords when quantifying word-level semantic conflicts.

[0019] Sub-steps 1-2 involve global visual feature extraction based on a pre-trained visual encoder. In this step, the input image is adjusted to a standard resolution and then input into a pre-trained visual encoder (such as CLIP-ViT). The visual encoder divides the image into... A number of non-overlapping image patches, among which The image side length, Let be the side length of the image patch; perform linear projection on each image patch to obtain the corresponding embedding vector. The embedding vector is then coupled with a learnable classification label. and position encoding matrix Concatenate to form the input sequence The input sequence The calculation formula is as follows:

[0020]

[0021] The input sequence The input is processed in the Transformer module of the pre-trained visual encoder, and the output is a raw visual feature sequence containing complete spatial and texture details. :

[0022]

[0023] To ensure that subsequent large-scale multimodal models can fully perceive the global semantics and local details of the image, the original visual feature sequence is preserved. As a global visual feature, it is denoted as Wherein, the image patch feature sequence corresponding to the original visual feature sequence after removing the classification label is denoted as... ,in Indicates the first Feature vectors of image patches.

[0024] Sub-steps 1-3 involve grammar-driven dynamic fine-grained local visual feature alignment. In this step, the high-weight sentiment keyword set generated in sub-step 1-1 is used as the basis for this alignment. To drive this, a grammar-guided fine-grained visual attention mechanism is constructed; for sets... Any keyword in Map it to text semantic vector The text semantic vector is calculated using scaled dot product attention. With the image patch feature sequence Each image patch The strength of the association between them, i.e., attention weights :

[0025]

[0026] in, The dimension of the feature vector; based on the attention weight distribution, the features of each image patch are dynamically weighted and aggregated to extract local visual features that are strictly aligned with specific rhetorical entities. :

[0027]

[0028] Traverse the set All high-weight sentiment keywords were used to construct a set of local visual features. This step actively guides the multi-level filtering of visual information through the core semantics of the text, breaking the rigid limitation of discrete bounding boxes in traditional object detection methods. It achieves fine-grained image-text soft alignment at the image patch level, providing high-quality local feature representations for the accurate quantification of subsequent multi-level counterfactual conflicts.

[0029] As an improvement of this invention, step 2: Dual-path counterfactual semantic inference and contrast baseline construction. By actively constructing a counterfactual semantic space, multimodal irony detection is elevated from traditional passive feature fusion to an active generative cognitive reasoning level. The core of this step lies in using a domain-adapted large-scale multimodal model to simulate the human contrastive cognitive process. By generating semantic anchors that are logically consistent with the original input but have different emotional polarities, the implicit pragmatic conflict between text and images is made explicit. Compared to direct classification prediction, this invention constructs a ternary contrast space containing factual prototypes and ironic prototypes, ensuring that the model can examine the original input from both objective physical facts and subjective rhetorical intentions. This effectively solves the problem of insufficient robustness of traditional discriminative methods in dealing with implicit irony that is highly dependent on context due to a lack of logical reasoning ability. By executing lightweight adaptation, dynamic construction of generated signals, and bidirectional semantic inference in stages, this invention establishes a robust decision reference system, significantly enhancing the system's analytical accuracy and interpretability for complex modal conflicts. The implementation of this step can be divided into the following sub-steps.

[0030] Sub-step 2-1: Domain Adaptation of the Multimodal Large Model Based on a Lightweight Adapter. In this step, a low-rank adapter (LoRA) is used to fine-tune the Transformer decoder of the pre-trained multimodal large model (such as LLaVA-1.5-13B) to enhance its instruction-following ability in satirical contexts; during the fine-tuning process, the pre-trained weight matrix is ​​maintained. Freezing is achieved by introducing two trainable low-rank factorization matrices. Implement weight updates, resulting in the updated weight matrix. Represented as:

[0031]

[0032] Among them, rank satisfy (In a preferred embodiment of the present invention, (Set to 8), the model can be efficiently fine-tuned through the parameters to transform it into a domain expert model with the ability to understand irony and pragmatics.

[0033] Sub-step 2-2: Construction of the adaptive grammar-guided signal generation. In this step, the model constructs the signal based on the high-weight sentiment keyword set extracted in step 1-1. And its syntactic dependency relations, constructing a set of grammatical modification pairs composed of rhetorical words and core words. ,in Indicates a modifier, The headword it modifies (e.g., "delicious", "toast") is then defined; subsequently, an adaptive input signal is defined. In standard scenarios, let To provide refined generation-driven processing; in sparse scenarios where extreme brevity leads to syntactic analysis failure, the adaptive input signal... Automatically backtrack to the original input text ,Right now This is to ensure the continuity and integrity of semantic input.

[0034] Sub-steps 2-3 involve generating the semantic prototype of the subjective satirical path. In this step, the model constructs the subjective satirical path, setting the model as the "satirer" role based on the adaptive input signal. Imposing constraints of forced rationalization and stylistic exaggeration: requiring the model to ignore the actual contradictions between the image and the text, and to treat the signal... The semantics in the image (such as "successful toast") are treated as absolute facts about the image content, and a virtual narrative scene is constructed using ironic, exaggerated, or satirical tones, outputting a prototype of ironic semantic description with strong rhetorical color. .

[0035] Sub-steps 2-4 involve generating the semantic prototype of the objective fact path. In this step, the model constructs the objective fact path, setting the model as an "objective verifier" and utilizing the global visual features extracted in steps 1-2. For signals The system performs physical-level verification of entities and attributes, including existence checks and de-embellishment processes. If an object in the signal is actually missing from the image or its visual attributes (such as "charred") conflict with textual embellishments (such as "delicious"), the model is forced to correct the description or remove subjective adjectives, outputting an objective factual semantic description prototype that restores the physical visual truth. The above dual-path generation process can be formally represented as follows:

[0036]

[0037] in, Used to determine the polarity of instructions; through the dual-path deduction, this invention establishes a subjective semantic anchor point within the semantic space. With objective semantic anchors This provides a benchmark for the accurate quantification of subsequent multi-granularity conflicts.

[0038] As an improvement of this invention, step 3: Macro-fact alignment and sentence-level semantic conflict measurement. This step aims to quantify the relative degree of deviation of the original input text from visual objective facts and towards subjective satirical intent from the perspective of macro-context. Traditional models usually rely only on the surface similarity of the original text and images for matching, which is easily masked by rhetoric and leads to misjudgment. This invention innovatively uses the subjective satirical description and objective factual description generated in step 2 as absolute subjective anchors and objective anchors in the semantic space. Based on the spatial geometric projection assumption, if the original text has satirical intent, its global semantic representation should show high affinity with the subjective anchor and significantly exclude the objective anchor; conversely, if it is declarative factual text, the three tend to converge highly. By constructing this measurement index in the ternary contrastive semantic space, this system effectively filters redundant background information noise in long texts and accurately captures the macro-semantic drift features of inconsistency. The implementation of this step can be divided into the following sub-steps:

[0039] Sub-step 3-1, Global Semantic Space Vector Mapping. In this step, the model uses a pre-trained text encoder (such as Sentence-BERT) to map the original input text. Step 2 generates a prototype of subjective ironic semantic description. and the prototype of semantic description of objective facts Perform encoding feature extraction to obtain the corresponding global semantic vector, denoted as follows: , as well as .

[0040] Sub-step 3-2 involves calculating sentence-level semantic conflict (SSC) based on the ternary contrast space. In this step, the model computes the original text vector within the global semantic space. Connecting objective anchor points With subjective anchor point The projection deviation along the semantic axis is used to quantify the degree of macro-level conflict. Specifically, the cosine similarity between the original text and the subjective anchor point is calculated, and the cosine similarity between the original text and the objective anchor point is subtracted to obtain the basic conflict score. Introduce a bias term as a slack variable. (In a preferred embodiment of the present invention, The system then applies a nonlinear activation function to filter out noise and enhance the robustness of the metric, ultimately yielding a sentence-level semantic conflict score. The calculation formula is as follows:

[0041]

[0042]

[0043] Among them, when When the value is positive, it indicates that although the original text was triggered by the visual scene, its macro-semantic core has deviated from visual reality and is moving towards a forced-generated ironic prototype, thus accurately capturing the macro-level contradiction between text and image at the sentence level.

[0044] As an improvement to this invention, step 4: Micro-level local mapping and word-level rhetorical contradiction quantification. Although the sentence-level semantic conflict module can effectively capture macro-contextual misalignment, its sensitivity may be diluted by a large number of irrelevant neutral words in long sentences, while the true satirical intent is often anchored on specific keywords that carry strong emotional weight. To overcome the limitations of macro-representation, this step introduces a fine-grained dual verification mechanism. For the extracted core rhetorical units, the micro-semantic distance between them and the objective textual fact baseline and the local visual fact baseline is calculated respectively. This not only allows for the perception of global semantic deviations but also accurately locates and quantifies the micro-semantic reversal caused by the misuse of specific adjectives or modifiers. The implementation of this step can be divided into the following sub-steps:

[0045] Sub-step 4-1: Extracting contextual features of rhetorical anchors. In this step, the model uses a pre-trained context-aware text encoder to extract contextual features from the original input text. and the generated objective fact semantic description prototype Encode the set of high-weight sentiment keywords extracted in step 1-1. any of the core words Extract its contextual embedding representation in the original text. And the corresponding contextual embedding representation in the description of objective facts. .

[0046] Sub-step 4-2 involves calculating micro-distances based on dual fact baselines. In this step, the model calculates the micro-distance for each core vocabulary. The micro-semantic conflict is calculated in both textual and visual modalities. First, textual semantic conflict is calculated to measure the deviation between the original rhetorical vocabulary and the generated objective text description:

[0047]

[0048] in, Represents the cosine similarity function; Indicates core vocabulary Contextual embedding representation in the original input text; Indicates core vocabulary The context embedding representation corresponding to the generated objective fact description.

[0049] Secondly, computational visual semantic conflict A projection matrix is ​​introduced to map local visual features to the text semantic space, and the degree of adversarial relationship between the projection matrix and the original word embedding is calculated to capture conflicts that violate the original visual truth. The calculation formula is as follows:

[0050]

[0051] in, This represents a learnable projection matrix; This indicates the keywords extracted in steps 1-3 that are related to the core vocabulary. Strictly aligned local visual features.

[0052] Sub-step 4-3: Dynamic aggregation of Keyword-level Semantic Conflict (KSC). In this step, the model combines the core words retained in step 1-1. Corresponding sentiment weight scalar The text and visual conflict scores of all high-weight keywords are weighted and aggregated to quantify the global word-level semantic conflict (KSC) score. The calculation formula is as follows:

[0053]

[0054] in, The higher the score, the more intense the contrast between the local rhetoric in the original text and the objective visual and factual descriptions, thus accurately revealing the micro-level irony triggering mechanism.

[0055] As an improvement of this invention, step 5: Dynamic fusion and adaptive judgment of multi-granularity conflict features. This step aims to establish a robust comprehensive judgment criterion, organically coordinating macro-level sentence-level contextual misalignment with micro-level word-level rhetorical contradictions. Because ironic expressions vary greatly across different scenarios, a single-dimensional indicator cannot encompass their diversity: some samples appear harmonious in the macro-context, yet subtly contain irony in individual keywords; while other samples demonstrate irony through the contrast of the overall context, without prominent local lexical conflicts. Therefore, this invention proposes to achieve complementary advantages of the two granularity conflict information through a dynamic balancing coefficient: macro-conflict provides a global contextual framework to correct local ambiguities, while micro-conflict provides precise local anchor points to prevent semantic drift. Finally, through adaptive nonlinear mapping, the continuous mixed conflict scalar is transformed into an ironic probability distribution that conforms to human cognition, thereby establishing the final high-precision decision boundary. The implementation of this step can be divided into the following sub-steps:

[0056] Sub-step 5-1 calculates the Hybrid Conflict Score (HCS). In this step, the model introduces a balancing coefficient to adjust the weights of macro- and micro-level conflict contributions. The sentence-level semantic conflict score obtained in step 3 and the word-level semantic conflict score obtained in step 4 are linearly weighted and combined to calculate the Hybrid Conflict Score (HCS). The calculation formula is as follows:

[0057]

[0058] in, The balance coefficient is represented (in a preferred embodiment of the invention, it is set) To achieve optimal synergy between the two. This represents the sentence-level semantic conflict score calculated in step 3; This represents the word-level semantic conflict score calculated in step 4.

[0059] Sub-step 5-2, Adaptive Satirical Judgment Based on Probability Mapping. In this step, to map continuous deviation scalars to conditional probabilities that a sample belongs to the satirical category, a prediction layer based on the Sigmoid activation function is constructed. The model is trained to automatically learn the non-linear mapping relationship between conflict features and the classification target, outputting the predicted probability. The calculation formula is as follows:

[0060]

[0061] in, This represents the Sigmoid activation function; This represents learnable scaling weights used to adaptively adjust the model's sensitivity to conflict scores; This represents a learnable bias term used to implicitly encode the classification decision boundary. Based on predicted probabilities... The model outputs the final multimodal irony classification result: when If the input sample is not satirical, it is determined to be a satirical sample; otherwise, it is determined to be a non-satirical sample.

[0062] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the multimodal irony detection method based on generative counterfactual reasoning and multigranular conflict quantification.

[0063] A storage medium storing a computer program that, when executed by a processor, implements the multimodal irony detection method based on generative counterfactual reasoning and multigranular conflict quantification.

[0064] Compared with the prior art, the advantages of the present invention are as follows:

[0065] (1) This invention proposes a multimodal satire detection method based on generative counterfactual reasoning and multigranular conflict quantification. By constructing subjective satire paths and objective fact paths, a large-scale multimodal model is used to actively infer and generate counterfactual semantic prototypes, effectively compensating for the lack of cognitive reasoning ability in traditional methods when dealing with complex cross-modal data. This method provides key semantic anchors for identifying implicit text-image contradictions and significantly enhances the model's ability to understand and capture multimodal satirical expressions.

[0066] (2) This invention introduces a multi-granularity conflict measurement system at both the macro-sentence level and the micro-word level, significantly improving the accuracy and robustness of detection. Compared to traditional discriminative models that rely solely on surface statistical correlations, this invention quantifies macro-contextual deviations through sentence-level semantic conflict and accurately locates micro-contradictions driven by specific rhetoric by combining word-level semantic conflict. This multi-dimensional quantification mechanism can analyze explicit and implicit emotional signals more meticulously, effectively filtering noise interference, and has achieved internationally leading detection levels on multiple public benchmark datasets.

[0067] (3) This invention possesses high interpretability and broad applicability. The reasoning process based on the ternary contrast space achieves decision transparency through specific conflict scores, and can be widely applied in fields such as social media sentiment monitoring, brand reputation management, intelligent content review, and intelligent human-computer interaction. When processing massive heterogeneous data from social networks, this method can identify satirical intentions in user content in real time, assisting enterprises or institutions in promptly grasping the true trend of public sentiment; at the same time, its lightweight adaptation mechanism ensures efficient operation under different computing power resource environments, providing intelligent and humanized semantic understanding solutions for various intelligent systems.

[0068] (4) This invention possesses stronger implicit irony detection capabilities and adaptability to open scenarios. Compared to existing detection methods that rely on single-modal features, shallow image-text alignment relationships, or surface sentiment cues, this invention, by introducing a generative counterfactual reasoning mechanism and combining sentence-level and word-level conflict information analysis, can more effectively identify complex samples that are semantically consistent on the surface but contain deep-seated irony. Furthermore, this method maintains good detection stability and adaptability even when facing multimodal data with strong noise interference, implicit semantic expression, and loose image-text relationships, thereby further enhancing the practical value of the method. Attached Figure Description

[0069] Figure 1 This is an overall model diagram of an embodiment of the present invention;

[0070] Figure 2 This is a flowchart illustrating the method processing of an embodiment of the present invention. Detailed Implementation

[0071] To enhance understanding of the present invention, the invention will be further explained below with reference to specific embodiments.

[0072] Example 1: A Multimodal Irony Detection Method Based on Generative Counterfactual Reasoning and Multigranular Conflict Quantification. Addressing the shortcomings of traditional discriminative models lacking reasoning ability and generative models lacking comparative baselines in multimodal irony detection, this invention constructs a detection framework from two dimensions: counterfactual semantic deduction and multigranular conflict measurement. This invention introduces a large-scale multimodal model to construct a dual-path counterfactual reasoning mechanism, dynamically synthesizing objective factual prototypes describing visual truth and ironic semantic prototypes reflecting subjective ironic intent. Combined with grammatically guided hierarchical visual feature extraction, a hybrid conflict quantification system consisting of macro-level sentence-level semantic conflict and micro-level word-level semantic conflict is constructed. This invention not only focuses on semantic drift at the global level but also locks in core rhetorical contradictions through dual factual baseline verification at the micro-level, thereby comprehensively improving the modeling ability and classification accuracy for complex and implicit multimodal ironic expressions.

[0073] For the specific model, please refer to [link / details]. Figure 1 The detailed implementation steps are as follows:

[0074] Step 1: Multimodal Semantic Anchor Point Parsing and Hierarchical Visual Feature Extraction. First, deep grammar parsing technology is used to perform syntactic dependency analysis on the input text, extracting core structures such as adjective modification, negation modification, and adverbial modification chains as potential ironic rhetorical units. Polarity determination and sentiment weight calculation are performed using a sentiment lexicon, and a set of high-weight sentiment keywords is selected as semantic anchor points. Subsequently, a pre-trained visual encoder is used to extract the global visual feature sequence of the input image to preserve the complete spatial and texture details of the image. Simultaneously, driven by the high-weight sentiment keywords, a grammar-guided fine-grained visual attention mechanism is constructed. By calculating the semantic relevance between text keywords and image block sequences, local visual features strictly aligned with key entities are dynamically extracted. The core objective of this step is to break the rigid limitations of traditional fixed bounding boxes, actively guiding the precise and multi-level selection of visual information through the core semantics of the text, changing the limitations of passive splicing of traditional modal features, and providing high-quality, strongly aligned cross-modal underlying data support for the accurate quantification of multi-level text-image counterfactual conflicts. The implementation of this step can be divided into the following sub-steps:

[0075] Sub-step 1-1: Syntactic parsing and sentiment weighting of core semantic anchors in the text. In this step, deep syntax parsing technology is used to analyze the original input text. Dependency parsing was performed to extract core structures containing adjective modification, negation modification, and adverbial modification chains as potential ironic rhetorical units; positive sentiment scores for each rhetorical unit were obtained by combining a sentiment lexicon. With negative sentiment score And based on whether there is a negation marker in its dependency path, the sentiment weight of a single word is calculated. The calculation formula is as follows:

[0076]

[0077] Furthermore, to effectively filter out redundant words with low information content, a preset threshold for sentiment saliency assessment is set. (In a preferred embodiment of the present invention, the preset threshold) (Set to 0.2). The sentiment weights of each word will be... With preset threshold By comparing and filtering words with sentiment weights below a certain threshold, a set of high-weight sentiment keywords is selected and generated. The set The set and its corresponding emotional weights will play a dual role: on the one hand, the set As semantic anchors for cross-modal feature alignment, they are used to actively guide the dynamic extraction of local visual features in subsequent steps; on the other hand, they retain the sentiment weight scalars of each high-weight keyword. It is used as a calculation factor to measure the rhetorical contribution of the corresponding keywords when quantifying word-level semantic conflicts.

[0078] Sub-steps 1-2 involve global visual feature extraction based on a pre-trained visual encoder. In this step, the input image is adjusted to a standard resolution and then input into a pre-trained visual encoder (such as CLIP-ViT). The visual encoder divides the image into... A number of non-overlapping image patches, among which The image side length, Let be the side length of the image patch; perform linear projection on each image patch to obtain the corresponding embedding vector. The embedding vector is then coupled with a learnable classification label. and position encoding matrix Concatenate to form the input sequence The input sequence The calculation formula is as follows:

[0079]

[0080] The input sequence The input is processed in the Transformer module of the pre-trained visual encoder, and the output is a raw visual feature sequence containing complete spatial and texture details. :

[0081]

[0082] To ensure that subsequent large-scale multimodal models can fully perceive the global semantics and local details of the image, the original visual feature sequence is preserved. As a global visual feature, it is denoted as Wherein, the image patch feature sequence corresponding to the original visual feature sequence after removing the classification label is denoted as... ,in Indicates the first Feature vectors of image patches.

[0083] Sub-steps 1-3 involve grammar-driven dynamic fine-grained local visual feature alignment. In this step, the high-weight sentiment keyword set generated in sub-step 1-1 is used as the basis for this alignment. To drive this, a grammar-guided fine-grained visual attention mechanism is constructed; for sets... Any keyword in Map it to text semantic vector The text semantic vector is calculated using scaled dot product attention. With the image patch feature sequence Each image patch The strength of the association between them, i.e., attention weights :

[0084]

[0085] in, The dimension of the feature vector; based on the attention weight distribution, the features of each image patch are dynamically weighted and aggregated to extract local visual features that are strictly aligned with specific rhetorical entities. :

[0086]

[0087] Traverse the set All high-weight sentiment keywords were used to construct a set of local visual features. This step actively guides the multi-level filtering of visual information through the core semantics of the text, breaking the rigid limitation of discrete bounding boxes in traditional object detection methods. It achieves fine-grained image-text soft alignment at the image patch level, providing high-quality local feature representations for the accurate quantification of subsequent multi-level counterfactual conflicts.

[0088] Step 2: Dual-Path Counterfactual Semantic Inference and Contrast Baseline Construction. By actively constructing a counterfactual semantic space, multimodal irony detection is elevated from traditional passive feature fusion to an active generative cognitive reasoning level. The core of this step lies in using a domain-adapted large-scale multimodal model to simulate the human comparative cognitive process. By generating semantic anchors that are logically consistent with the original input but have vastly different emotional polarities, the implicit pragmatic conflict between text and images is explicitly revealed. Compared to direct classification prediction, this invention constructs a ternary contrast space containing factual prototypes and ironic prototypes, ensuring that the model can examine the original input from both objective physical facts and subjective rhetorical intentions. This effectively solves the problem of insufficient robustness of traditional discriminative methods in handling implicit irony that is highly dependent on context due to a lack of logical reasoning ability. By executing lightweight adaptation, dynamic construction of generated signals, and bidirectional semantic inference in stages, this invention establishes a robust decision reference system, significantly enhancing the system's analytical accuracy and interpretability for complex modal conflicts. The implementation of this step can be divided into the following sub-steps:

[0089] Sub-step 2-1: Domain Adaptation of the Multimodal Large Model Based on a Lightweight Adapter. In this step, a low-rank adapter (LoRA) is used to fine-tune the Transformer decoder of the pre-trained multimodal large model (such as LLaVA-1.5-13B) to enhance its instruction-following ability in satirical contexts; during the fine-tuning process, the pre-trained weight matrix is ​​maintained. Freezing is achieved by introducing two trainable low-rank factorization matrices. Implement weight updates, resulting in the updated weight matrix. Represented as:

[0090]

[0091] Among them, rank satisfy (In a preferred embodiment of the present invention, (Set to 8), the model can be efficiently fine-tuned through the parameters to transform it into a domain expert model with the ability to understand irony and pragmatics.

[0092] Sub-step 2-2: Construction of the adaptive grammar-guided signal generation. In this step, the model constructs the signal based on the high-weight sentiment keyword set extracted in step 1-1. And its syntactic dependency relations, constructing a set of grammatical modification pairs composed of rhetorical words and core words. ,in Indicates a modifier, The headword it modifies (e.g., "delicious", "toast") is then defined; subsequently, an adaptive input signal is defined. In standard scenarios, let To provide refined generation-driven processing; in sparse scenarios where extreme brevity leads to syntactic analysis failure, the adaptive input signal... Automatically backtrack to the original input text ,Right now This is to ensure the continuity and integrity of semantic input.

[0093] Sub-steps 2-3 involve generating the semantic prototype of the subjective satirical path. In this step, the model constructs the subjective satirical path, setting the model as the "satirer" role based on the adaptive input signal. Imposing constraints of forced rationalization and stylistic exaggeration: requiring the model to ignore the actual contradictions between the image and the text, and to treat the signal... The semantics in the image (such as "successful toast") are treated as absolute facts about the image content, and a virtual narrative scene is constructed using ironic, exaggerated, or satirical tones, outputting a prototype of ironic semantic description with strong rhetorical color. .

[0094] Sub-steps 2-4 involve generating the semantic prototype of the objective fact path. In this step, the model constructs the objective fact path, setting the model as an "objective verifier" and utilizing the global visual features extracted in steps 1-2. For signals The system performs physical-level verification of entities and attributes, including existence checks and de-embellishment processes. If an object in the signal is actually missing from the image or its visual attributes (such as "charred") conflict with textual embellishments (such as "delicious"), the model is forced to correct the description or remove subjective adjectives, outputting an objective factual semantic description prototype that restores the physical visual truth. The above dual-path generation process can be formally represented as follows:

[0095]

[0096] in, Used to determine the polarity of instructions; through the dual-path deduction, this invention establishes a subjective semantic anchor point within the semantic space. With objective semantic anchors This provides a benchmark for the accurate quantification of multi-granularity conflicts in the future.

[0097] Step 3: Macro-level Fact Alignment and Sentence-level Semantic Conflict Measurement. This step aims to quantify the relative degree of deviation of the original input text from visual objective facts and towards subjective satirical intent from the perspective of macro-context. Traditional models usually rely solely on the surface similarity of the original text and images for matching, which is easily masked by rhetoric and leads to misjudgments. This invention innovatively uses the subjective satirical description and objective factual description generated in Step 2 as absolute subjective and objective anchors in the semantic space. Based on the spatial geometric projection assumption, if the original text has satirical intent, its global semantic representation should show high affinity with the subjective anchor while significantly rejecting the objective anchor; conversely, if it is declarative factual text, the three tend to converge highly. By constructing this measurement index in the ternary contrastive semantic space, this system effectively filters redundant background information noise in long texts and accurately captures the macro-level semantic drift features of inconsistency. The implementation of this step can be divided into the following sub-steps:

[0098] Sub-step 3-1, Global Semantic Space Vector Mapping. In this step, the model uses a pre-trained text encoder (such as Sentence-BERT) to map the original input text. Step 2 generates a prototype of subjective ironic semantic description. and the prototype of semantic description of objective facts Perform encoding feature extraction to obtain the corresponding global semantic vector, denoted as follows: , as well as .

[0099] Sub-step 3-2 involves calculating sentence-level semantic conflict (SSC) based on the ternary contrast space. In this step, the model computes the original text vector within the global semantic space. Connecting objective anchor points With subjective anchor point The projection deviation along the semantic axis is used to quantify the degree of macro-level conflict. Specifically, the cosine similarity between the original text and the subjective anchor point is calculated, and the cosine similarity between the original text and the objective anchor point is subtracted to obtain the basic conflict score. Introduce a bias term as a slack variable. (In a preferred embodiment of the present invention, The system then applies a nonlinear activation function to filter out noise and enhance the robustness of the metric, ultimately yielding a sentence-level semantic conflict score. The calculation formula is as follows:

[0100]

[0101]

[0102] Among them, when When the value is positive, it indicates that although the original text was triggered by the visual scene, its macro-semantic core has deviated from visual reality and is moving towards the ironic prototype of forced generation, thus accurately capturing the macro-text-image contradiction at the sentence level.

[0103] Step 4: Micro-level Local Mapping and Word-level Rhetorical Contradictions Quantification. Although the sentence-level semantic conflict module can effectively capture macro-contextual misalignment, its sensitivity may be diluted by a large number of irrelevant neutral words in long sentences, while the true satirical intent is often anchored on specific keywords carrying strong emotional weight. To overcome the limitations of macro-representation, this step introduces a fine-grained dual verification mechanism. For the extracted core rhetorical units, the micro-semantic distance between them and the objective textual fact baseline and the local visual fact baseline is calculated respectively. This not only allows for the perception of global semantic deviations but also accurately locates and quantifies the micro-semantic reversals caused by the misuse of specific adjectives or modifiers. The implementation of this step can be divided into the following sub-steps:

[0104] Sub-step 4-1: Extraction of contextual features from rhetorical anchors. In this step, the model uses a pre-trained context-aware text encoder to extract contextual features from the original input text. and the generated objective fact semantic description prototype Encode the set of high-weight sentiment keywords extracted in step 1-1. any of the core words Extract its contextual embedding representation in the original text. And the corresponding contextual embedding representation in the description of objective facts. .

[0105] Sub-step 4-2 involves calculating micro-distances based on dual fact baselines. In this step, the model calculates the micro-distance for each core vocabulary. The micro-semantic conflict is calculated in both textual and visual modalities. First, textual semantic conflict is calculated to measure the deviation between the original rhetorical vocabulary and the generated objective text description:

[0106]

[0107] in, Represents the cosine similarity function; Indicates core vocabulary Contextual embedding representation in the original input text; Indicates core vocabulary The context embedding representation corresponding to the generated objective fact description.

[0108] Secondly, computational visual semantic conflict A projection matrix is ​​introduced to map local visual features to the text semantic space, and the degree of adversarial relationship between the projection matrix and the original word embedding is calculated to capture conflicts that violate the original visual truth. The calculation formula is as follows:

[0109]

[0110] in, This represents a learnable projection matrix; This indicates the keywords extracted in steps 1-3 that are related to the core vocabulary. Strictly aligned local visual features.

[0111] Sub-step 4-3: Dynamic aggregation of Keyword-level Semantic Conflict (KSC). In this step, the model combines the core words retained in step 1-1. Corresponding sentiment weight scalar The textual and visual conflict scores of all high-weight keywords are weighted and aggregated to quantify the global word-level semantic conflict (KSC) score. The calculation formula is as follows:

[0112]

[0113] in, The higher the score, the more intense the contrast between the local rhetoric and the objective visual and factual descriptions in the original text, thus accurately revealing the micro-level irony triggering mechanism.

[0114] Step 5: Dynamic Fusion and Adaptive Judgment of Multi-Granularity Conflict Features. This step aims to establish a robust comprehensive judgment criterion that organically coordinates macro-level sentence-level contextual misalignment with micro-level word-level rhetorical contradictions. Because ironic expressions vary greatly across different scenarios, a single-dimensional indicator cannot encompass their diversity: some samples appear harmonious in the macro-context but subtly contain irony in individual keywords; while other samples demonstrate irony through the contrast of the overall context, without prominent local lexical conflicts. Therefore, this invention proposes to achieve complementary advantages of the two granularity conflict information through a dynamic balancing coefficient: macro-conflict provides a global contextual framework to correct local ambiguities, while micro-conflict provides precise local anchor points to prevent semantic drift. Finally, an adaptive nonlinear mapping transforms the continuous mixed conflict scalar into an irony probability distribution that conforms to human cognition, thereby establishing the final high-precision decision boundary. The implementation of this step can be divided into the following sub-steps:

[0115] Sub-step 5-1 calculates the Hybrid Conflict Score (HCS). In this step, the model introduces a balancing coefficient to adjust the weights of macro- and micro-level conflict contributions. The sentence-level semantic conflict score obtained in step 3 and the word-level semantic conflict score obtained in step 4 are linearly weighted and combined to calculate the Hybrid Conflict Score (HCS). The calculation formula is as follows:

[0116]

[0117] in, The balance coefficient (in a preferred embodiment of the invention, is set) represents the balance coefficient. To achieve optimal synergy between the two. This represents the sentence-level semantic conflict score calculated in step 3; This represents the word-level semantic conflict score calculated in step 4.

[0118] Sub-step 5-2, Adaptive Satirical Judgment Based on Probability Mapping. In this step, to map continuous deviation scalars to conditional probabilities that a sample belongs to the satirical category, a prediction layer based on the Sigmoid activation function is constructed. The model is trained to automatically learn the non-linear mapping relationship between conflict features and the classification target, outputting the predicted probability. The calculation formula is as follows:

[0119]

[0120] in, This represents the Sigmoid activation function; This represents learnable scaling weights used to adaptively adjust the model's sensitivity to conflict scores; This represents a learnable bias term used to implicitly encode the classification decision boundary. Based on predicted probabilities... The model outputs the final multimodal irony classification result: when If the input sample is not satirical, it is determined to be a satirical sample; otherwise, it is determined to be a non-satirical sample.

[0121] Based on the same inventive concept, the present invention also provides a multimodal irony detection method and apparatus based on generative counterfactual reasoning and multigranular conflict quantification, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is loaded into the processor, it implements the multimodal irony detection method based on generative counterfactual reasoning and multigranular conflict quantification as described above.

[0122] Those skilled in the art will recognize that the embodiments described herein are intended to help readers understand the principles of the invention, and should be understood as being for illustrative purposes only and not for limiting the scope of the invention; after reading this invention, any modifications, equivalent substitutions or improvements to the invention by those skilled in the art will fall within the protection scope defined by the claims of this application.

Claims

1. A multimodal irony detection method based on generative counterfactual reasoning and multi-granularity conflict quantification, characterized in that, The method includes the following steps: Step 1: Multimodal semantic anchor point parsing and hierarchical visual feature extraction; Step 2: Dual-path counterfactual semantic inference and baseline construction; Step 3: Macro-level fact alignment and sentence-level semantic conflict measurement; Step 4: Quantifying the contradiction between microscopic local mapping and word-level rhetoric; Step 5: Dynamic fusion and adaptive determination of multi-granularity conflict features.

2. The multimodal irony detection method based on generative counterfactual reasoning and multi-granularity conflict quantification according to claim 1, characterized in that, Step 1: Multimodal Semantic Anchor Point Parsing and Hierarchical Visual Feature Extraction. First, deep grammar parsing technology is used to perform syntactic dependency analysis on the input text, extracting core structures such as adjective modification, negation modification, and adverbial modification chains as potential ironic rhetorical units. A sentiment lexicon is then used to determine polarity and calculate sentiment weights, generating a set of high-weight sentiment keywords as semantic anchors. Subsequently, a pre-trained visual encoder is used to extract the global visual feature sequence of the input image to preserve the complete spatial and textural details of the image. Simultaneously, driven by high-weight sentiment keywords, a grammar-guided fine-grained visual attention mechanism is constructed. By calculating the semantic relevance between text keywords and image block sequences, local visual features strictly aligned with key entities are dynamically extracted. The core objective of this step is to break the rigid limitations of traditional fixed bounding boxes, actively guiding the precise and multi-level selection of visual information through the core semantics of the text, changing the limitations of passively splicing traditional modal features, and providing high-quality, strongly aligned cross-modal underlying data support for the accurate quantification of multi-layered text-image counterfactual conflicts. This step includes the following sub-steps: Sub-step 1-1: Syntactic parsing and sentiment weighting of core semantic anchors in the text. In this step, deep syntax parsing technology is used to analyze the original input text. Dependency parsing was performed to extract core structures containing adjective modification, negation modification, and adverbial modification chains as potential ironic rhetorical units; positive sentiment scores for each rhetorical unit were obtained by combining a sentiment lexicon. With negative sentiment score And based on whether there is a negation marker in its dependency path, the sentiment weight of a single word is calculated. The calculation formula is as follows: To effectively filter out redundant words with low information content, a preset threshold is set for sentiment saliency assessment. The emotional weight of each word With preset threshold By comparing and filtering words with sentiment weights below a certain threshold, a set of high-weight sentiment keywords is selected and generated. ,gather The corresponding emotional weights will play a dual role: on the one hand, the set As semantic anchors for cross-modal feature alignment, they are used to actively guide the dynamic extraction of local visual features in subsequent steps; on the other hand, they retain the sentiment weight scalars of each high-weight keyword. This is used as a calculation factor to measure the rhetorical contribution of the corresponding keywords when quantifying word-level semantic conflicts. Sub-steps 1-2 involve global visual feature extraction based on a pre-trained visual encoder. In this step, the input image is adjusted to a standard resolution and then input into the pre-trained visual encoder. The visual encoder then divides the image into... A number of non-overlapping image patches, among which The image side length, Let be the side length of the image patch; perform linear projection on each image patch to obtain the corresponding embedding vector. And the embedding vector is paired with a learnable classification label. and position encoding matrix Concatenate to form the input sequence Input sequence The calculation formula is as follows: Input sequence The input is processed in the Transformer module of the pre-trained visual encoder, and the output is a raw visual feature sequence containing complete spatial and texture details. : To ensure that subsequent large-scale multimodal models can fully perceive the global semantics and local details of images, the original visual feature sequences are preserved. As a global visual feature, it is denoted as The image patch feature sequence corresponding to the original visual feature sequence after removing the classification label is denoted as . ,in Indicates the first Feature vectors of image patches Sub-steps 1-3 involve grammar-driven dynamic fine-grained local visual feature alignment, where the high-weight sentiment keyword set generated in sub-step 1-1 is used as the basis for this alignment. To drive this, a grammar-guided fine-grained visual attention mechanism is constructed; for sets... Any keyword in Map it to text semantic vector And use scaled dot product attention to compute text semantic vectors. With image patch feature sequence Each image patch The strength of the association between them, i.e., attention weights : in, The dimension of the feature vector is given; based on the attention weight distribution, the features of each image patch are dynamically weighted and aggregated to extract local visual features that are strictly aligned with specific rhetorical entities. : Traversing a collection All high-weight sentiment keywords are used to construct a set of local visual features. This step proactively guides the multi-level filtering of visual information through the core semantics of the text.

3. The multimodal irony detection method based on generative counterfactual reasoning and multi-granularity conflict quantification according to claim 1, characterized in that, Step 2: Dual-path counterfactual semantic inference and baseline construction. By actively constructing a counterfactual semantic space, multimodal irony detection is elevated from traditional passive feature fusion to an active generative cognitive reasoning level. By constructing a ternary comparison space containing factual prototypes and ironic prototypes, the model ensures that it can examine the original input from both objective physical facts and subjective rhetorical intentions. Through phased execution of lightweight adaptation, dynamic construction of generated signals, and bidirectional semantic inference processes, a robust decision reference system is established. This step is implemented in the following sub-steps: Sub-step 2-1, Domain Adaptation of Multimodal Large Model Based on Lightweight Adapter: In this step, a low-rank adapter (LoRA) is used to fine-tune the Transformer decoder of the pre-trained multimodal large model to enhance its instruction compliance ability in ironic contexts; during the fine-tuning process, the pre-trained weight matrix is ​​maintained. Freezing is achieved by introducing two trainable low-rank factorization matrices. Implement weight updates, resulting in the updated weight matrix. Represented as: Among them, rank satisfy By efficiently fine-tuning the parameters, the model can be transformed into a domain expert model with the ability to understand irony and pragmatics. Sub-step 2-2, Construction of Adaptive Syntax-Guided Generative Signal: In this step, the model constructs the signal based on the set of high-weight sentiment keywords extracted in step 1-1. And its syntactic dependency relations, constructing a set of grammatical modification pairs composed of rhetorical words and core words. ,in Indicates a modifier, The headword it modifies is indicated; subsequently, an adaptive input signal is defined. In standard scenarios, let To provide refined generation-driven processing; and to adapt input signals in sparse scenarios where extreme brevity leads to syntactic analysis failure. Automatically backtrack to the original input text ,Right now To ensure the continuity and integrity of semantic input, Sub-steps 2-3 involve generating the semantic prototype of the subjective satirical path. In this step, the model constructs the subjective satirical path, setting the model as the "satirer" role based on adaptive input signals. Imposing constraints of forced rationalization and stylistic exaggeration: requiring the model to ignore the actual contradictions between the image and the text, and to treat the signal... The semantics in the image are treated as absolute facts about the image content, and a virtual narrative scene is constructed using ironic, exaggerated, or satirical tones, outputting a prototype of ironic semantic description with strong rhetorical color. , Sub-steps 2-4 involve generating the semantic prototype of the objective fact path. In this step, the model constructs the objective fact path, setting the model as an "objective verifier" and utilizing the global visual features extracted in steps 1-2. For signals The system performs physical-level verification of entities and attributes, including existence checks and de-embellishment processes: if an object in the signal is actually missing in the image or its visual attributes conflict with textual modifiers, the model is forced to correct the description or remove subjective adjectives, outputting an objective factual semantic description prototype that restores the physical visual truth. The above dual-path generation process can be formally represented as follows: in, Used to determine the polarity of instructions; through dual-path deduction, subjective semantic anchors are established within the semantic space. With objective semantic anchors This provides a benchmark for the accurate quantification of subsequent multi-granularity conflicts.

4. The multimodal irony detection method based on generative counterfactual reasoning and multi-granularity conflict quantification according to claim 1, characterized in that, Step 3: Macro-fact alignment and sentence-level semantic conflict measurement. This step aims to quantify the relative degree of deviation of the original input text from visual objective facts and towards subjective satirical intent from the macro-context level. The subjective satirical description and objective factual description generated in Step 2 are used as absolute subjective anchors and objective anchors in the semantic space. Based on the spatial geometric projection assumption, if the original text has satirical intent, its global semantic representation should show high affinity with the subjective anchors while significantly rejecting the objective anchors; conversely, if it is declarative factual text, the three tend to converge. The implementation of this step is divided into the following sub-steps: Sub-step 3-1, Global Semantic Space Vector Mapping: In this step, the model uses a pre-trained text encoder (such as Sentence-BERT) to map the original input text. Step 2 generates a prototype of subjective ironic semantic description. and the prototype of semantic description of objective facts Perform encoding feature extraction to obtain the corresponding global semantic vector, denoted as follows: , as well as , Sub-step 3-2 involves calculating sentence-level semantic conflict (SSC) based on the ternary contrast space. In this step, the model computes the original text vector within the global semantic space. Connecting objective anchor points With subjective anchor point The projection deviation along the semantic axis is used to quantify the degree of macro-conflict. Specifically, the cosine similarity between the original text and the subjective anchor point is calculated, and the cosine similarity between the original text and the objective anchor point is subtracted to obtain the basic conflict score. Introduce a bias term as a slack variable. Furthermore, a nonlinear activation function is applied to filter out noise and enhance the robustness of the metric, ultimately yielding a sentence-level semantic conflict score. The calculation formula is as follows: Among them, when When the value is positive, it indicates that although the original text was triggered by the visual scene, its macro-semantic core has deviated from visual reality and is moving towards a forced-generated ironic prototype, thus accurately capturing the macro-level contradiction between text and image at the sentence level.

5. The multimodal irony detection method based on generative counterfactual reasoning and multi-granularity conflict quantification according to claim 1, characterized in that, Step 4: Micro-level local mapping and word-level rhetorical contradiction quantification. A fine-grained dual verification mechanism is introduced. For the extracted core rhetorical units, the micro-semantic distance between them and the objective textual fact baseline and the local visual fact baseline is calculated respectively. The implementation of this step is divided into the following sub-steps: Sub-step 4-1, extracting contextual features of rhetorical anchors: In this step, the model uses a pre-trained context-aware text encoder to extract contextual features from the original input text. and the generated objective fact semantic description prototype Encode the set of high-weight sentiment keywords extracted in step 1-1. any of the core words Extract its contextual embedding representation in the original text. And the corresponding contextual embedding representation in the description of objective facts. , Sub-step 4-2 involves calculating the micro-distance based on dual fact baselines. In this step, the model performs micro-distance calculations for each core vocabulary. The micro-semantic conflict is calculated in both textual and visual modalities. First, textual semantic conflict is calculated to measure the deviation between the original rhetorical vocabulary and the generated objective text description: in, Represents the cosine similarity function; Indicates core vocabulary Contextual embedding representation in the original input text; Indicates core vocabulary The context embedding representation corresponding to the generated objective fact description. Secondly, computational visual semantic conflict A projection matrix is ​​introduced to map local visual features to the text semantic space, and the degree of adversarial relationship between the projection matrix and the original word embedding is calculated to capture conflicts that violate the original visual truth. The calculation formula is as follows: in, This represents a learnable projection matrix; This indicates the keywords extracted in steps 1-3 that are related to the core vocabulary. Strictly aligned local visual features Sub-step 4-3, Keyword-level Semantic Conflict (KSC) dynamic aggregation: In this step, the model combines the core words retained in step 1-1. Corresponding sentiment weight scalar The textual and visual conflict scores of all high-weight keywords are weighted and aggregated to quantify the global word-level semantic conflict (KSC) score. The calculation formula is as follows: in, The higher the score, the more intense the contrast between the local rhetoric and the objective visual and factual descriptions in the original text, thus accurately revealing the micro-level irony triggering mechanism.

6. The multimodal irony detection method based on generative counterfactual reasoning and multi-granularity conflict quantification according to claim 1, characterized in that, Step 5: Dynamic Fusion and Adaptive Judgment of Multi-Granularity Conflict Features. This step aims to establish a robust comprehensive judgment criterion that organically coordinates macro-level sentence-level contextual misalignment with micro-level word-level rhetorical contradictions. The implementation of this step is divided into the following sub-steps: Sub-step 5-1 calculates the Hybrid Conflict Score (HCS). In this step, the model introduces a balancing coefficient to adjust the weights of macro- and micro-conflict contributions. The sentence-level semantic conflict score obtained in step 3 and the word-level semantic conflict score obtained in step 4 are linearly weighted and combined to calculate the Hybrid Conflict Score (HCS). The calculation formula is as follows: in, Indicates the balance coefficient; This represents the sentence-level semantic conflict score calculated in step 3; This represents the word-level semantic conflict score calculated in step 4. Sub-step 5-2, Adaptive Satirical Judgment Based on Probability Mapping: In this step, to map continuous deviation scalars to conditional probabilities that samples belong to the satirical category, a prediction layer based on the Sigmoid activation function is constructed. The model is trained to automatically learn the non-linear mapping relationship between conflict features and the classification target, outputting the predicted probability. The calculation formula is as follows: in, This represents the Sigmoid activation function; This represents learnable scaling weights used to adaptively adjust the model's sensitivity to conflict scores; This represents a learnable bias term used to implicitly encode the classification decision boundary, based on predicted probabilities. The model outputs the final multimodal irony classification result: when If the input sample is not satirical, it is determined to be a satirical sample; otherwise, it is determined to be a non-satirical sample.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, it implements the multimodal irony detection method based on generative counterfactual reasoning and multigranular conflict quantification as described in any one of claims 1 to 6.

8. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it implements the multimodal irony detection method based on generative counterfactual reasoning and multigranular conflict quantification as described in any one of claims 1 to 6.

Images