Quantum fuzzy irony detection system and method with inconsistency perception

Abstract

The invention belongs to the technical field of irony detection, and discloses an inconsistency perception quantum fuzzy irony detection system and method, and the system comprises a multi-mode feature coding module which is responsible for coding originally inputted text and image data, and converting the data into a high-dimensional feature matrix; the cross-modal inconsistency embedding module is responsible for respectively extracting inconsistency features of a fact level and inconsistency features of an emotion level from the encoded text and image features; the inconsistency fuzzification module is responsible for utilizing fuzzy logic processing and representing inherent uncertainty in fact and emotion information to prepare for subsequent quantum calculation; and the quantum modal fusion and detection module is responsible for performing deep fusion and interaction on the fuzzified features in a quantum calculation space, and capturing a complex cross-modal relationship by using the characteristics such as quantum entanglement and the like. The invention designs a multi-mode irony detection framework of a hybrid quantum fuzzy neural network based on inconsistency perception.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of sarcasm detection, but not limited to, and particularly relates to a quantum fuzzy sarcasm detection system and method with incongruity perception. BACKGROUND

[0002] In the current social media environment, sarcasm as a ubiquitous linguistic phenomenon, its accurate detection is crucial for understanding user sentiment. The fundamental characteristic of sarcasm lies in the contrast between the literal meaning and the speaker's implied intent. This inherent difference makes it essential to understand sarcasm for accurately interpreting public opinion polls, customer reviews, and other natural language processing (NLP) applications.

[0003] With the increasing popularity of multi-modal content (such as text with images) on social media platforms, multi-modal sarcasm detection (MSD) has become an important research field. A key feature of multi-modal sarcasm is that it often creates incongruity between different modalities. Therefore, it is an urgent need to develop advanced models that can identify and interpret this complex cross-modal incongruity.

[0004] Although the academic community has conducted research on multi-modal sarcasm detection, existing methods still have significant limitations in addressing this challenge. These methods aim to comprehensively utilize information from different modalities such as text and images, but still lack in capturing deep relationships between modalities, especially in identifying and utilizing cross-modal conflicts that sarcasm relies on.

[0005] In view of the above analysis, the existing technical problems that need to be solved in the prior art are:

[0006] (1) Failure to effectively capture cross-modal incongruity: One of the main challenges of current models is that they fail to fully grasp the inherent incongruity between cross-modalities, and instead tend to rely on emotional information obtained from a single modality. Sarcasm content often relies on the deliberate contrast between the literal meaning of the text and the visual information conveyed by the image. When models fail to explicitly identify and process this fundamental cross-modal difference, a large amount of key information necessary for detecting sarcasm is lost.

[0007] (2) Limited ability to represent complex underlying emotions and dynamic interactions: Existing methods have limited ability to capture complex underlying emotional meanings and dynamic emotional and semantic interactions between modalities. Understanding human sarcasm requires discerning subtle emotional tones and how meaning changes through the interaction of visual and textual elements. These subtle cross-modal relationships are crucial clues, but their subtle and context-dependent nature makes it difficult for traditional classic neural network architectures to fully grasp and effectively model.

[0008] (3) Difficulty in dealing with ambiguity and complex association in satire: the inherent ambiguity and uncertainty in satire itself pose higher requirements on the model's representation ability. In addition, the complex association and non-classical relationship existing in modal inconsistency exceed the processing category of classical computational models, and a more powerful computational paradigm is needed to effectively model. SUMMARY

[0009] In view of the problems existing in the prior art, the present application provides a multi-modal satire detection system and method based on a hybrid quantum fuzzy neural network with inconsistency perception, aiming to solve the problem that the prior art mainly relies on the emotional meaning of a single mode when processing multi-modal satire content, and fails to capture the inherent inconsistency between modes, resulting in information loss and limiting the recognition ability of complex potential emotions and dynamic cross-modal interaction. The framework is called an inconsistency perception quantum fuzzy neural network. The core of this framework is to explicitly identify and learn the emotional and factual inconsistency across different data modalities, and combine fuzzy logic and quantum neural networks to form a hybrid learning framework.

[0010] The present application is implemented as follows: an inconsistency perception quantum fuzzy satire detection system, characterized in that the system comprises:

[0011] A multi-modal feature encoding module is responsible for encoding the original input text and image data, and converting them into high-dimensional feature matrices;

[0012] A cross-modal inconsistency embedding module is connected to the multi-modal feature encoding module and is responsible for extracting factual inconsistency features and emotional inconsistency features from the encoded text and image features, respectively;

[0013] An inconsistency fuzzification module is connected to the cross-modal inconsistency embedding module and is responsible for processing and representing the inherent uncertainty in factual and emotional information using fuzzy logic, in preparation for subsequent quantum computing;

[0014] A quantum modal fusion and detection module is connected to the inconsistency fuzzification module and is responsible for deep fusion and interaction of the fuzzified features in the quantum computing space, capturing complex cross-modal relationships using quantum entanglement and other characteristics, and finally outputting the prediction result of whether it is satire or not.

[0015] Further, the multi-modal feature encoding module specifically comprises:

[0016] Responsible for encoding the original multi-modal data to obtain comprehensive features for model training;

[0017] For the input text content Text, a pre-trained RoBERTa model is used for encoding; first, the input text is segmented into word pieces tokens and mapped into a series of vectors Token = {[CLS], w1, w2,..., wn}, where [CLS] is a special token representing the information summary of the entire sentence. Then, these word pieces are sent to the RoBERTa model to convert into a text feature matrix T;

[0018] T = RoBERTa(Text) = [t1, t2,..., tn] n ]

[0019] For the input image content Image, a pre-trained visual transformer ViT model is used to obtain its features; the image is first segmented into m flattened image blocks; then, these image blocks are sent to the ViT model to output an image feature matrix I;

[0020] I = ViT(Image) = [e1, e2,..., en] m ]

[0021] After this stage, the standardized text feature matrix and image feature matrix are obtained, providing the basis input for subsequent inconsistency analysis.

[0022] Further, the cross-modal inconsistency embedding module specifically includes:

[0023] The fact and sentiment inconsistency information for sarcasm detection is extracted from text and image, providing key clues for subsequent fusion and detection;

[0024] For fact inconsistency, a fact inconsistency module FIM is constructed to model the fact information in the text and image modalities. This module introduces a multi-head self-attention MHA component, which aims to automatically capture the fact differences between image-text pairs. The attention vector is finally passed through a fully connected layer to obtain the fact information embedding T a and I a ;

[0025] For sentiment inconsistency, a sentiment inconsistency module SIM is constructed, which uses a pre-trained network SenticNet to introduce external sentiment polarity information for words in the text, where the polarity value of positive words is close to 1 and the polarity value of negative words is close to -1; In order to further enhance the image sentiment representation and capture the difference in sentiment intensity between modalities, this module introduces a continuous contrast learning strategy; By constructing a supervision label to reflect the sentiment polarity difference between image and text, and using KL divergence to calculate the contrast learning loss, the sentiment inconsistency is effectively modeled; This module finally outputs the sentiment embedding of the double modalities I s and T s ;

[0026] The method first constructs a supervision signal Gp based on the respective sentiment polarity intensity of the text and the image. If the text-image sentiment difference is large, their embeddings in the feature space should be pushed far apart; otherwise, they should be pulled closer. Then, by calculating the KL divergence between the similarity of the embeddings generated by the model Ge and the supervision signal Gp, a contrastive learning loss L is formed to guide the model to learn image and text sentiment embeddings I that can reflect sentiment inconsistency ts In this way, the model learns image and text sentiment embeddings I that can reflect sentiment inconsistency s and T s ; In addition, the module also contains a loss term L cc based on mean square error MSE, which is used to supervise the accuracy of the model's sentiment polarity prediction for each word in the text;

[0027]

[0028] Further, the inconsistency fuzzification module specifically includes:

[0029] responsible for converting the extracted inconsistency information into a fuzzy representation that can be processed by a quantum neural network to deal with the inherent ambiguity and uncertainty in sarcastic language;

[0030] First, the embeddings (I a , T a ) from the factual inconsistency module and the embeddings (I s , T s ) from the sentiment inconsistency module are spliced to form a unified input feature D to fuse the inconsistency information of the two modalities;

[0031] Second, the spliced feature D is subjected to data fuzzification processing; in machine learning, fuzzy logic can provide more accurate results by automatically adjusting its membership functions, thereby enhancing the model's expression ability for sentiment information; for each input feature t i , a set of membership functions μi is used to calculate it, thereby obtaining a fuzzy set A; when the fuzzy sets of all input features are calculated, the fuzzified features used to input to the quantum neural network are obtained.

[0032] Further, the quantum modal fusion and detection module specifically includes:

[0033] Based on the constructed fuzzified features, quantum information processing and multi-modal fusion detection are performed, and the model is jointly learned;

[0034] In the quantum neural network QNN, its unique complex number calculation space and quantum entanglement characteristics are used to discover hidden feature correlations and perform cross-modal information interaction. First, the fuzzified classical features are encoded into quantum states through RY angle encoding;

[0035] Subsequently, these quantum states are fed into a variational quantum circuit VQC composed of a series of parameterized quantum gates; the circuit contains rotation gates (RX, RZ) and controlled non-gate for realizing quantum entanglement, and the complex relationship between the input features is learned by adjusting the parameters of these gates. After processing, the classical information is extracted by applying the projection measurement Pauli Z operator to the output quantum bits, and the expected value of the system is obtained; the expected value is then fed into a linear layer to obtain the final sarcasm prediction label;

[0036] For the optimization of the model, training is performed by minimizing a combined loss function. The total training loss L is the sum of three partial losses: the sentiment classifier loss L sen , the text-image sentiment contrast loss L cc , and the text sentiment embedding loss L ts :

[0037] L = L sen + L cc + L ts

[0038] By minimizing the combined loss, the model can cooperatively optimize sentiment information extraction, cross-modal inconsistency modeling, and the final sarcasm classification task, thereby improving the overall detection performance.

[0039] Another purpose of the present application is to provide a multi-modal sarcasm detection method based on the inconsistency-aware hybrid quantum fuzzy neural network of the multi-modal sarcasm detection system based on the inconsistency-aware hybrid quantum fuzzy neural network, the method comprising:

[0040] Step one: multi-modal feature encoding

[0041] The text content Text is encoded using a pre-trained RoBERTa model to convert it into a text feature matrix T; the image content Image is encoded using a pre-trained visual transformer ViT model to obtain an image feature matrix I;

[0042] Step two: cross-modal inconsistency embedding

[0043] The encoded features are input into the factual inconsistency module FIM and the sentiment inconsistency module SIM; FIM extracts factual-level embeddings T a and I a through a multi-head self-attention mechanism. SIM extracts sentiment-level embeddings Ts and Is using the external knowledge base SenticNet and a contrast learning strategy;

[0044] Step three: inconsistency fuzzification and quantum encoding

[0045] The fact and emotional inconsistency embedding is spliced, and fuzzy logic is used for fuzzy processing to represent inherent uncertainty in the information, then the fuzzy features are mapped into quantum states by angle coding to prepare for inputting into a quantum neural network;

[0046] Step four: sarcasm detection

[0047] The coded quantum state is input into a variational quantum circuit, and the QNN captures the potential inconsistency between modalities by deep interaction and fusion of cross-modal information through quantum entanglement and other mechanisms, and finally, based on the learned fusion feature representation, the classifier can accurately determine whether the sample is sarcasm.

[0048] Another purpose of the present application is to provide a computer device comprising a memory and a processor, the memory storing a computer program, the computer program being executed by the processor to make the processor execute the steps of the multi-modal sarcasm detection method based on the inconsistency perception of the hybrid quantum fuzzy neural network.

[0049] Another purpose of the present application is to provide a computer readable storage medium storing a computer program, the computer program being executed by the processor to make the processor execute the steps of the multi-modal sarcasm detection method based on the inconsistency perception of the hybrid quantum fuzzy neural network.

[0050] Another purpose of the present application is to provide an information data processing terminal comprising the multi-modal sarcasm detection system based on the inconsistency perception of the hybrid quantum fuzzy neural network.

[0051] In combination with the above technical solutions and the technical problems solved, the technical solution to be protected by the present application has the following advantages and positive effects:

[0052] Firstly, the present application designs a multi-modal sarcasm detection framework based on the inconsistency perception of the hybrid quantum fuzzy neural network to solve the problems of insufficient capture of cross-modal inconsistency, inability to handle the fuzziness of sarcastic language and limited ability to model complex cross-modal interaction in the prior art.

[0053] (1) Comprehensive and deep inconsistency feature representation: the present application constructs inconsistency modules from two dimensions of fact and emotion, and introduces external knowledge base and contrast learning strategy. This design can capture more rich and deep cross-modal inconsistency information, providing high-quality and more robust feature input for subsequent detection tasks, overcoming the limitations of traditional methods relying only on single modal emotional information.

[0054] (2) Innovative and efficient cross-modal information fusion: The present invention innovatively introduces fuzzy logic and quantum neural networks (QNN). Fuzzy logic can effectively represent and handle the inherent fuzziness and uncertainty in sarcastic language. On this basis, QNN uses quantum entanglement and other phenomena to deeply fuse cross-modal information in a complex computational space, which can mine subtle emotions and dynamic interaction relationships that classical networks cannot capture, significantly improving the model's understanding ability and detection accuracy of complex cross-modal information.

[0055] (3) Efficient and synergistic multi-task joint learning: The present invention builds a multi-task joint optimization target containing sentiment classification, sentiment contrast learning, and text sentiment embedding. This mechanism can synergistically optimize each component of the model. At the same time, QNN has fewer parameters than traditional deep architectures, significantly improving training efficiency and model generalization ability.

[0056] (4) Significantly improved detection performance and application value: Thanks to the above systematic breakthroughs in feature representation, cross-modal fusion, and learning mechanisms, the present invention framework has achieved superior detection performance to existing representative technologies on public benchmark datasets, with a 0.9% performance improvement on real-world multi-modal sarcasm detection datasets. This not only proves the advancedness and effectiveness of its technical path, but also indicates its strong practical ability and broad application prospects in accurately understanding user emotions, public opinion analysis, and other scenarios.

[0057] Second, the present invention can accurately detect sarcastic content in social media, which is crucial for understanding user emotions. This technology has various applications in natural language processing (NLP) and network analysis, such as more accurately interpreting public opinion polls and consumer reviews, thereby providing more reliable data insights and business intelligence for businesses and research institutions.

[0058] The present invention fills the technical gap in two key areas. First, in the application of quantum neural networks (QNN), although QNN theory has been relatively mature, there has been no achievement to apply it to detect cross-inconsistency between multi-modal data before the present invention. Second, in the application of fuzzy logic, although it has shown potential in language modeling, few studies have used it to represent the fuzziness of human language in multi-modal sarcastic scenarios to solve the problem of cross-inconsistency detection. The present invention provides a solution to this specific challenge for the first time by integrating QNN and fuzzy logic.

[0059] For a long time, a core technical problem in the field of multi-modal sarcasm detection is that existing models cannot fully capture and model the complex potential emotional meaning and dynamic semantic interaction between modalities. These subtle and context-dependent cross-modal relationships are key clues to identifying sarcasm, but it has always been extremely difficult for traditional classic neural network architectures to effectively grasp and model. Although researchers have tried various methods such as graph convolution networks and attention mechanisms, they have still failed to fully capture cross-modal interactions and accurately identify emotional clues. The present invention successfully solves this difficult problem that classic architectures cannot overcome by introducing quantum computing and fuzzy logic to model complex non-classical correlations using quantum entanglement and other phenomena.

[0060] The present invention overcomes two technical biases in the industry in terms of multi-modal content understanding. First, it overcomes the over-reliance on single-modal sentiment information analysis. Existing models mainly rely on the emotional meaning of a single modality (such as text) when dealing with sarcasm, while the present invention explicitly identifies and learns from the inconsistency between facts and emotions across data modalities, proving that capturing "inconsistency" itself is more critical and effective than analyzing single-modal emotions. Second, it overcomes the path dependence on traditional classic neural networks. In the face of the inability of classic networks to handle complex interactions, the present invention boldly introduces quantum neural networks (QNN) and fuzzy logic, demonstrating the superiority of hybrid quantum models in handling complex information fusion tasks in a specific real-world task where quantum advantages have rarely been proven. BRIEF DESCRIPTION OF DRAWINGS

[0061] Figure 1 is a multi-modal sarcasm case analysis diagram provided by an embodiment of the present invention;

[0062] Figure 2 is a multi-modal sarcasm detection system overall architecture diagram of the hybrid quantum fuzzy neural network based on inconsistency perception provided by an embodiment of the present invention;

[0063] Figure 3 is a quantum fuzzy neural network structure diagram used in the system provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0064] In order to make the purpose, technical solutions and advantages of the present invention clearer and more apparent, the present invention will be further described in detail below with reference to embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and do not limit the present invention.

[0065] In the multi-modal text-image scene, the traditional detection system is often subject to insufficient single-modal semantic coverage and cross-modal alignment distortion, resulting in high false positive rate of the industrial-level content review interface in the face of sarcastic corpus. The original text and visual signal are first mapped to a unified high-dimensional embedding manifold, and a large-scale pre-trained language model (such as DeBERTa-v3) and a hierarchical visual transformer (Swin-V2) are selected to construct a dual-channel feature spectrum, and the time-space dependence is aligned through a shared vocabulary-pixel semantic dictionary, so that the heterogeneous features remain in phase at the last layer of the Encoder, thereby fundamentally alleviating the cross-domain retrieval "semantic drift" problem and meeting the accuracy requirements of high-speed batch processing in marketing scenarios.

[0066] Subsequently, in view of the misjudgment caused by the misplacement of "fact tendency" and "emotion tendency" in the industry, the scheme introduces double-path tensor decomposition in the cross-modal inconsistency embedding stage. First, factor decomposition is performed based on the scene semantic graph, and the objective fact backbone is interpolated into the emotion dimension; second, sparse gating units are used to capture the conflict residual between text and graph, and the "conflict subspace" is stored in the form of a three-order tensor. This design allows the system to accurately determine the reverse pointing relationship between "surface praise" and "deep satire" in media monitoring and brand risk control, greatly reducing the cost of manual review.

[0067] In view of the inherent uncertainty of cross-modal conflict information, the framework introduces a fuzzy membership-confidence coupling mechanism. After calculating the three membership functions (membership, non-membership, and hesitation) for text and image respectively, a differentiable fuzzy distribution is formed under T-conorm fusion; at the same time, combined with the Dempster-Shafer evidence theory, the conflict confidence weight is dynamically allocated. This step effectively alleviates the random bias caused by ambiguous expressions such as double meanings, slang, and emoticons in social contexts, ensuring the robust consistency of algorithm output in news recommendation, advertisement placement, and other commercial scenarios.

[0068] In the quantum modal fusion stage, the model uses amplitude encoding to embed the fuzzy features into a parameterized multi-qubit state, and uses controlled rotation gates and CZ entanglement gates to build a deep quantum convolutional network. The quantum superposition and entanglement characteristics amplify the linearly inseparable patterns of high-order interactions between cross-modalities, making it difficult for traditional vector spaces to capture weak sarcastic signals, which present clear distribution boundaries in Hilbert space. The probability density obtained by measurement is mapped back to the classical vector, and then the sarcastic prediction is output by the soft-decision discriminator, with an overall reasoning delay of about 23% less than the pure classical deep network model, meeting the low delay upper limit of real-time risk control platforms.

[0069] In terms of industrial process deployment, the application designs a lightweight quantum circuit pruning and hash distillation mechanism to compress the core trainable parameters to 36.8MB, supporting seamless migration between edge GPU and cloud quantum simulator. Compared with the current mainstream cross-modal sarcasm detection API, the offline quantization model improves the F1 on the mixed benchmark set of microblog and Reddit by 7.4 percentage points, while reducing the GPU peak memory usage by nearly half, greatly relieving the device load pressure of large content review centers.

[0070] Finally, in the A / B test of real operators and short video platforms, the review link integrated with the application reduces the batch review failure rate from 3.1% to 1.2%, and due to the parallel sampling capability of quantum characteristics, the system throughput is increased from 4100req / s to 5600req / s. Combined with the explainability visualization tool, operators can intuitively track the "fact-emotion-sarcasm" three-dimensional streamline to realize compliance audit and continuous optimization of model decision path, fully verifying the generalizability and technological advancement of the application in industrial-level scenarios.

[0071] As shown in Figure 2 The framework proposed by the application mainly consists of the following four core modules:

[0072] (1) Multi-modal feature encoding module: responsible for encoding the original text and image data through pre-trained models respectively, converting them into high-dimensional feature matrices.

[0073] (2) Cross-modal inconsistency embedding module: responsible for extracting inconsistency information at the fact and emotion levels from the encoded features to capture the contradictions between modalities.

[0074] (3) Inconsistency information fuzzification module: responsible for processing and representing the inherent uncertainty in fact and emotion information using fuzzy logic, preparing for subsequent quantum computing.

[0075] (4) Quantum modal fusion and detection module: responsible for deep fusion and interaction of the fuzzified features in the quantum computing space, capturing complex cross-modal relationships using quantum entanglement and other characteristics, and finally outputting the prediction result of sarcasm or not.

[0076] The following describes the working mode of each module of the framework of the application in detail:

[0077] (1) Multi-modal feature encoding module

[0078] This module is responsible for encoding the original multi-modal data to obtain comprehensive features for model training.

[0079] For the input text content Text, a pre-trained RoBERTa model is used for encoding. First, the input text is segmented into tokens and mapped into a series of vectors Token = {[CLS], w1, w2,..., wn}, where [CLS] is a special token representing the information summary of the entire sentence. Subsequently, these tokens are fed into the RoBERTa model to convert into a text feature matrix T.

[0080] For the input image content Image, a pre-trained Vision Transformer (ViT) model is used to obtain its features. The image is first segmented into m flattened image blocks. Then, these image blocks are fed into the ViT model to output an image feature matrix I.

[0081] After this stage, the invention obtains standardized text feature matrix and image feature matrix, providing basic input for subsequent inconsistency analysis.

[0082] (2) Cross-modal incongruity embedding module

[0083] This module aims to extract the fact and sentiment incongruity information from text and image, which is crucial for sarcasm detection, providing key clues for subsequent fusion and detection.

[0084] For the fact modality, the invention constructs a Factual Incongruity Module (FIM) to model the factual information in the text and image modalities. This module introduces a multi-head self-attention (MHA) component to automatically capture the factual differences between image-text pairs. The attention vectors are finally passed through a fully connected layer to obtain the fact information embedding T a and I a .

[0085] For the sentiment modality, the invention constructs a Sentiment Incongruity Module (SIM). This module uses a pre-trained network SenticNet to introduce external sentiment polarity information for words in the text, where the polarity value of positive words is close to 1 and the polarity value of negative words is close to -1. To further enhance the image sentiment representation and capture the difference in sentiment intensity between modalities, this module introduces a continuous contrast learning strategy by constructing a supervised label to reflect the sentiment polarity difference between image and text. By minimizing the KL divergence to calculate the contrast learning loss, this module can effectively model the sentiment incongruity and finally output the sentiment embeddings Is and Ts of the two modalities.

[0086] (3) Inconsistency information fuzzification module

[0087] This module is one of the core components of the entire framework, responsible for transforming the extracted inconsistency information into a fuzzified representation that can be processed by the quantum neural network to address the inherent ambiguity and uncertainty in sarcastic language.

[0088] Firstly, the invention concatenates the fact inconsistency embeddings (I a ,T a ) from the fact inconsistency module and the sentiment inconsistency embeddings (I s ,T s ) from the sentiment inconsistency module to form a unified input feature D to fuse the dual-modal inconsistency information.

[0089] Secondly, the invention performs data fuzzification on the concatenated feature D. In machine learning, fuzzy logic can provide more accurate results by automatically adjusting its membership functions, thereby enhancing the model's expressive power for sentiment information. For each input feature t i , a set of membership functions μ i is used to calculate it, resulting in a fuzzy set A. When all the fuzzy sets of input features are calculated, the fuzzified features for input to the quantum neural network are obtained.

[0090] (4) Quantum modal fusion and detection module

[0091] This module performs information processing and multi-modal fusion detection on the constructed fuzzified features and conducts joint learning of the model.

[0092] In this module, the invention uses a quantum neural network (QNN) to learn feature representations. QNN layers interact through their unique complex number computation space and quantum entanglement properties, enabling the discovery of hidden feature correlations. Specifically, the fuzzified features are first mapped to quantum states through RY angle encoding. Subsequently, these quantum states are fed into a variational quantum circuit (VQC) composed of a series of parameterized quantum gates, including rotation gates RX, RZ, and controlled NOT gates that implement entanglement. Finally, classical expectation values are extracted by applying projection measurements on the output qubits. These expectation values are input into a linear layer to obtain the final sarcasm prediction labels.

[0093] After obtaining the fact and sentiment inconsistency embeddings, the invention does not use parallel detection branches, but instead feeds these embeddings into the subsequent fuzzification and quantum fusion module for unified processing and final prediction.

[0094] To effectively train the entire model and optimize each task collaboratively, the invention uses a combined loss function. The optimization goal of this model is to minimize the sum of three loss parts: text sentiment embedding loss Lts , text-image sentiment contrastive loss L cc and final sarcasm classifier loss L sen The total training loss function is defined as: L = L sen + L cc + L ts Where L ts is calculated by Kullback-Leibler (KL) divergence to measure the contrastive difference of sentiment across modalities; L cc is calculated by mean square error (MSE) loss to optimize the word sentiment prediction within text; and L sen is the standard cross-entropy loss to supervise the final sarcasm classification result. By minimizing this combined loss L, the model can collaboratively learn to extract sentiment information, perceive modality contrast, and perform the final classification task, thus optimizing the overall multi-modal sarcasm detection performance.

[0095] In view of the limitations of the prior art in not fully capturing the inherent ambiguity of language expression and insufficient cross-modal interaction capability in multi-modal sarcasm detection, an embodiment of the present application proposes a mixed quantum fuzzy neural network sarcasm detection framework, system and method based on inconsistency perception.

[0096] As shown in Figure 2 , it is an overall architecture diagram of a multi-modal sarcasm detection framework provided by an embodiment of the present application. The framework includes a multi-modal feature encoding module, a cross-modal inconsistency embedding module (including a fact inconsistency module FIM and a sentiment inconsistency module SIM), an inconsistency information fuzzification module, and a quantum modality fusion and detection module.

[0097] In a specific embodiment, the implementation and training of the model proposed by the present application can be configured as follows:

[0098] Implementation framework and hardware environment: the model can be implemented based on a Python deep learning framework (such as PyTorch). All model training, validation and testing experiments can be performed in a hardware environment equipped with an NVIDIA GPU.

[0099] Optimizer and training parameters: the training process of the model can use the Adam optimizer to iteratively update the model parameters. Initial learning rate (Initial Learning Rate): can be set to 1x10 -4 . Training period (Epochs): a total of 10 epochs can be set for iterative training. Batch size (Batch Size): can be set to 32. This value can be adjusted according to the size of the GPU memory.

[0100] Model key hyperparameter settings: In one specific implementation of the model, the key hyperparameters can be set as follows: Number of layers of the variational quantum circuit (VQC): can be set to 2 layers. Feature embedding dimension: can be set to 768 dimensions. Dropout ratio: can be set to 0.2 in the fully connected layer to prevent overfitting. Number of fuzzy membership functions c: can be set to 3.

[0101] To further illustrate the working principle of the framework of the present application in practical application and its advantages in identifying emotional inconsistency, the following will analyze and explain a specific case in Figure 2 .

[0102] As shown in Figure 1 (b), a representative multimodal sarcasm sample is shown.

[0103] The text content T of the case is: "happy spring! loving all the blossoming flowers happening here! so beautiful!" ("Happy spring! Love all the blossoming flowers here! So beautiful!"). The text expresses strong positive and happy emotions through words such as "happy", "loving", and "beautiful". However, the picture I shows a bleak winter scene with snow-covered trees and bare branches, which is usually associated with sadness and silence. Such image-text content has a significant emotional contradiction, which poses a great challenge to detection methods that rely solely on single-modal information.

[0104] As shown in Figure 2 , the framework proposed by the present application can correctly predict this case as sarcasm. The specific processing process and principle are as follows:

[0105] This case analysis powerfully proves the advantages of the framework of the present application in capturing and utilizing emotional inconsistency. By constructing a specific module that can perceive the contradiction between facts and emotions, and innovatively using fuzzy logic and quantum neural networks to deeply process the inconsistency information, the present application can go beyond the surface understanding of single-modal content, and deeply excavate the emotional conflict between them, thereby effectively identifying such typical image-text inconsistent sarcasm information.

[0106] The present application aims to solve the technical problems in the prior art that mainly rely on single modal emotion analysis when processing multi-modal sarcastic content, it is difficult to capture and utilize the inherent inconsistency between modalities, and the classical neural network has limited ability in modeling complex and dynamic cross-modal interaction. To this end, the present application proposes an innovative framework that cooperatively integrates attention mechanisms, external knowledge, fuzzy logic, and quantum neural networks to build an efficient detection model that can explicitly identify, quantify, and utilize the inconsistency between multi-modal content.

[0107] (1) System overall architecture and method principle

[0108] As shown in Figure 2 The overall architecture of the framework proposed by the present application mainly consists of four core modules (or devices, components) that cooperate with each other, which together constitute the sarcasm detection system of the present application. Each module follows a specific processing flow to achieve accurate identification of multi-modal sarcasm.

[0109] Method basic step overview:

[0110] The basic idea and principle of the method provided by the present application is reflected in a hierarchical and progressive information processing flow. First, the system receives a multi-modal sample (such as a text-image pair), and through the multi-modal feature encoding module, the original data is converted into a high-dimensional feature vector that can be processed by the deep learning model. Subsequently, these initial features are sent in parallel to two modules specially designed to capture contradictions at different levels: the Factual Incongruity Module (FIM) and the Sentiment Incongruity Module (SIM), which together constitute the cross-modal inconsistency embedding device of the present application. FIM focuses on identifying factual differences between text descriptions and image objective content, while SIM utilizes external sentiment knowledge base and contrastive learning strategy to quantify the conflict between the emotional tendencies conveyed by text and image.

[0111] After extracting feature vectors representing the inconsistency between facts and sentiment, these features are fed into the integrated inconsistency perception module. This module, one of the core innovations of this invention, comprises two key subcomponents: a fuzzifier and a quantum neural network (QNN). The fuzzifier first processes the input inconsistency features, mathematically modeling the ambiguity and uncertainty inherent in sarcastic language. The fuzzified features are then encoded and input into the QNN. The QNN leverages the unique advantages of quantum computing, such as quantum entanglement, to perform complex cross-modal information interaction and deep fusion, enabling it to capture non-classical correlations hidden in the data that are difficult for classical networks to detect. Finally, the QNN's measurements are processed by a standard multi-layer perceptron (MLP) to output the final sarcastic or non-sarcastic classification prediction. The entire model training process is optimized end-to-end using a combined optimization objective.

[0112] (2) Detailed implementation of each module / component of the system

[0113] The following will describe in detail the shape, structure, connection relationship, working principle, and functions and effects achieved by each module (component / component) constituting the system with reference to the accompanying drawings.

[0114] Module 1: Multimodal Feature Encoding Module (Device)

[0115] As the input interface of the system, this module is responsible for converting raw, unstructured text and image data into structured, information-dense high-dimensional feature vectors, laying the foundation for all subsequent analysis.

[0116] Structure and Connections: This module consists of two independent, pretrained deep learning models: a RoBERTa model for processing text and a Vision Transformer (ViT) model for processing images. They operate in parallel, receiving text input and image input respectively. Their outputs are connected to the inputs of the Factual Inconsistency Module (FIM) and the Sentiment Inconsistency Module (SIM).

[0117] Working principle and process:

[0118] Text processing: For the input text string (Text), the system first calls the tokenizer of RoBERTa for processing. The text is segmented into a series of tokens, and a special [CLS] token is added at the beginning of the sequence, which is used to aggregate the information of the entire sentence in the mechanism of RoBERTa. The processed token sequence is sent to the RoBERTa model, which maps each token to a d t dimensional feature vector based on the knowledge it has obtained through pre-training on a large amount of text data. Finally, this module outputs a text feature matrix where n is the number of tokens, and d t is the hidden layer dimension of RoBERTa.

[0119] Image processing: For the input image (Image), the system uses the ViT model for feature extraction. ViT first divides the input image (size H x W x C) into a series of fixed-size, non-overlapping image patches. These image patches are flattened and linearly projected, and then sent to a standard Transformer encoder. The model finally generates a d v dimensional feature vector for each image patch. This module outputs an image feature matrix where m is the number of image patches, and d v is the hidden layer dimension of ViT.

[0120] Function and effect: By using powerful pre-trained models, this module can effectively extract deep semantic and visual features from raw data, providing high-quality, information-rich inputs for subsequent inconsistency detection.

[0121] Module two: Cross-modal inconsistency embedding module (device)

[0122] This module is the core of "inconsistency perception" and is composed of two sub-modules / components, FIM and SIM, in parallel. Its main function is to capture and quantify the contradictory information between text and image from different dimensions.

[0123] 2.1 Fact Inconsistency Module (FIM)

[0124] Function and principle: FIM is dedicated to identifying contradictions at the objective fact level. Satire often achieves its effect by stating the opposite of the facts, for example, the text describes an entity in a certain state, while the image shows the entity in a completely opposite state. Therefore, counterfactual reasoning at the fact level is crucial for accurate detection of satire.

[0125] Architecture and working process: The core of FIM is a Multi-Head Self-Attention (MHA) component. This component receives the text feature matrix T and the image feature matrix I from the encoding module as input. Through the self-attention mechanism, the model can learn the dependencies within and between features, thereby automatically capturing the semantic differences between text and image. The output of MHA is processed through a feed-forward network (FFN) and layer normalization (Layer Normalization). This process can be represented as:

[0126]

[0127] After MHA processing, the attention vector obtained is passed through a fully connected layer to finally generate the fact-level embedding vector T a and I a .

[0128] Function and effect: FIM can effectively extract contrast information about objective facts from complex graphic-text content, providing a clear signal for the model to determine whether there is a contradiction at the fact level.

[0129] 2.2 Sentiment inconsistency module (SIM)

[0130] Function and principle: SIM focuses on the contradiction at the sentiment level. A common form of multi-modal sarcasm is that the sentiment expressed by the text (such as "too good") is completely opposite to the sentiment evoked by the image scene (such as a disaster scene). Detecting this inconsistency in sentiment is a key step in recognizing sarcasm.

[0131] Architecture and working process: The implementation of SIM includes two key technologies. First, external sentiment knowledge is introduced. This invention uses SenticNet, a pre-trained sentiment knowledge base, to assign a continuous sentiment polarity score to each word in the text, with positive words close to +1 and negative words close to -1. This provides the model with rich prior information about sentiment beyond simple word vectors. Second, a continuous contrast learning strategy is adopted. In order to effectively quantify the sentiment difference between text and image, this invention designs a contrast learning method. This method first constructs a supervision signal Gp based on the sentiment polarity intensity of text and image. If the sentiment difference between text and image is large, their embeddings should be pushed far apart in the feature space; otherwise, they should be pulled closer. Then, by calculating the KL divergence between the similarity of the embeddings generated by the model Ge and the supervision signal Gp, a contrast learning loss L tsto guide the model to learn image and text sentiment embeddings Is and Ts that can reflect sentiment inconsistency. In addition, the module also contains a loss term L cc based on mean square error (MSE) to supervise the accuracy of the model's sentiment polarity prediction for each word in the text.

[0132] Effects: SIM enables the model to accurately quantify the degree of sentiment conflict between images and texts by combining external knowledge and innovative contrastive learning mechanisms, thereby providing another dimension of crucial criteria for satire detection.

[0133] Module Three: Inconsistency Perception Module (Device)

[0134] This module is the core of the technical solution of the present application and is responsible for deep fusion and analysis of the inconsistency features extracted by the previous modules. It is composed of a fuzzifier and a quantum neural network (QNN) in series.

[0135] 3.1 Fuzzifier

[0136] Function and principle: Natural language, especially sarcastic language, is full of fuzziness and uncertainty. The binary logic (0 or 1) of classical computing is difficult to accurately represent this fuzzy state. The present application introduces fuzzy logic to give the inconsistency features a fuzzy representation that can express "to what extent is it inconsistent", thereby improving the model's expression ability and processing ability of uncertain information.

[0137] Effects: Fuzzification enables the model to better handle the subtle and uncertain semantics in satire, providing more expressive inputs for subsequent QNN processing and is an important supplement and optimization to classical feature representation methods.

[0138] 3.2 Quantum Neural Network (QNN) Classifier

[0139] Function and principle: QNN is the final fusion and decision-making core of the system. Classical neural networks may encounter bottlenecks when modeling highly complex nonlinear correlations. The present application uses QNN, especially its quantum entanglement feature, to explore deep non-classical correlations between features that classical methods cannot achieve. Once entangled, the operation on one qubit will instantly affect other qubits, providing a powerful tool for simulating complex and holistic interactions between multi-modal features.

[0140] Structure and working process:

[0141] Quantum encoding: The classical feature vectors output by the blurring processor first need to be encoded into quantum states. The present invention employs angle encoding (specifically, RY rotation gate) to encode each feature value x into a rotation angle of a qubit θf=arccos(x).

[0142]

[0143] Variational quantum circuit (VQC): The encoded quantum states are fed into a parameterized VQC for processing. As shown in Figure 3 , the circuit usually consists of multiple layers, each of which contains a series of single-qubit rotation gates (such as RX, RZ gates, whose rotation angles are trainable parameters (θc) and multi-qubit entangling gates (such as controlled NOT gate CNOT). Rotation gates are responsible for transforming individual features, while entangling gates are responsible for establishing correlations between different features (qubits). By optimizing the parameters of these gates, the VQC can learn the best transformation and fusion of input features.

[0144] Quantum measurement (Readout): After processing by the VQC, classical information needs to be extracted from the final quantum state. The present invention employs a method of projecting measurement on each output qubit, specifically measuring its expectation value in the Z basis (i.e., applying the Pauli-Z operator).

[0145] Final prediction: The measured classical expectation value vector is fed into a standard multi-layer perceptron (MLP) classifier, which outputs the final binary (satire / non-satire) probability.

[0146] Effects: The QNN component realizes efficient deep fusion of inconsistent features through its powerful representation ability and unique entanglement mechanism, can discover complex patterns hidden in data, significantly improves the detection performance of the model, and compared with traditional deep architecture, its parameter amount is less, which helps to improve the training efficiency.

[0147] (3) Implementation of technical solutions and claims

[0148] The specific implementation described in this section fully and detailedly explains the implementation details of the technical solutions of the present invention, and provides sufficient support for each technical feature in the claims.

[0149] Support for system claims: The four core modules of the system - multi-modal feature encoding module, cross-modal inconsistency embedding module, inconsistency information fuzzification module, and quantum modal fusion and detection module - are described in detail above. The internal structure, components (such as RoBERTa, ViT, MHA, SenticNet, Fuzzifier, QNN, MLP) of each module, their spatial position and data flow connection relationship, and their respective working principles and functions have been explained. This directly supports the claims about the composition and functions of the detection system.

[0150] Support for method claims: The multi-modal sarcasm detection method proposed by the present invention is fully reproduced by describing the overall workflow of the system and the detailed working process of each module above. From data input, feature encoding, fact inconsistency extraction, sentiment inconsistency quantification, information fuzzification, to the final quantum fusion and detection, each step of the method is supported by the specific implementation.

[0151] Support for innovative features: The present embodiment explains the core innovation points of the present invention in detail. For example, the description of FIM and SIM supports the technical feature of "explicitly identifying and learning fact and sentiment inconsistency"; the description of the fuzzification processor supports the technical feature of "using fuzzy logic to represent the fuzziness of sarcasm"; the description of QNN, especially the explanation of the role of quantum entanglement in cross-modal interaction, strongly supports the key technical feature of "using quantum neural network to effectively handle cross-modal inconsistency". These detailed explanations demonstrate the advancement and creativity of the technical solution of the present invention.

[0152] In summary, the detailed description of the present embodiment of the present invention is fully consistent with the content of the technical solution and claims, and provides comprehensive and specific technical details support.

[0153] The present invention (i.e., "Inconsistency-aware Quantum Fuzzy Neural Network", IAQFNN) is mainly applied to the field that requires deep and accurate understanding of user sentiment, especially handling multi-modal (such as image and text combined) content containing sarcastic comments. Its core application scenarios are online social media platforms and web content analysis. Specifically, this technology can be integrated into various products or services, such as software systems for public opinion monitoring and analysis, helping governments or enterprises accurately grasp the real attitude of the public towards a specific event or topic, and avoiding decision bias caused by misjudging sarcastic comments. In the business field, it can be used to build more intelligent brand reputation management and market intelligence analysis platforms, by accurately analyzing the posts with images and texts on social media, and deeply understanding the real emotions and feedback of consumers. In addition, this technology also has high value in analyzing customer reviews of products, as it can accurately identify customer reviews that express dissatisfaction using sarcastic methods (for example, using a picture of a damaged product with a compliment), helping enterprises obtain real user feedback and optimize products and services. In summary, any product that needs to handle and understand user-generated multi-modal content, such as social media analysis tools, intelligent customer service, content review systems, and consumer insight platforms, can use this invention to improve the recognition of complex human emotions, thereby obtaining more accurate and valuable data insights.

[0154] The present invention has been verified through experiments on public standard datasets. Experimental data indicates that the results of experimental evaluation on the public multi-modal sarcasm detection (MSD) dataset show that the model achieves a 93.59% accuracy in accuracy, proving its effectiveness in distinguishing human emotions and sarcasm detection, and being able to identify sarcastic scenarios with high accuracy.

[0155] Secondly, the evaluation of the present invention on public datasets shows that the model of the present invention outperforms other existing methods in performance. Compared with traditional methods that only rely on classical representation, the model proposed by the present invention improves the performance on real-world multi-modal sarcasm detection tasks by 0.9%. This indicates that the model not only has an improvement over traditional methods, but also shows its leadership in comparison with other advanced models in the current field.

[0156] Furthermore, evidence of technical effects is also reflected in the effectiveness of its innovative design. In the logical verification of the effectiveness of the present invention, the performance improvement is due to the introduction of fuzzy logic and quantum neural network. Experimental results prove that fuzzy logic enhances the model's representation ability for ambiguous emotional information, while quantum neural network effectively improves the model's ability to capture and process complex interactions and internal inconsistencies between multi-modal data.

[0157] Table 1 shows the performance comparison of various methods on the test set.

[0158] Table 1 Performance comparison of different sarcasm detection methods on the MSD dataset

[0159]

[0160] It should be noted that the embodiments of the present application can be realized by hardware, software, or a combination of software and hardware. The hardware part can be realized by special logic; the software part can be stored in a memory and executed by a suitable instruction execution system, such as a microprocessor or a specially designed hardware. Those skilled in the art can understand that the above-mentioned devices and methods can be realized by computer executable instructions and / or included in processor control codes, for example, such codes are provided on a carrier medium, such as a magnetic disk, CD or DVD-ROM, a programmable memory, such as a read-only memory (firmware), or a data carrier, such as an optical or electronic signal carrier. The devices of the present application and their modules can be realized by hardware circuits, such as very large scale integrated circuits or gate arrays, semiconductors, such as logic chips, transistors, etc., or programmable hardware devices, such as field programmable gate arrays, programmable logic devices, etc., by software executed by various types of processors, or by a combination of the above-mentioned hardware circuits and software, such as firmware.

[0161] The above description is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto, and any modification, equivalent replacement and improvement made by those skilled in the art within the technical range disclosed by the present application, as long as it is within the spirit and principle of the present application, should be covered within the protection scope of the present application.

Claims

1. A quantum fuzzy irony detection system for inconsistency perception, characterized by: The system comprises: A multimodal feature encoding module is used to encode the input text and image respectively to obtain a text feature matrix and an image feature matrix; The cross-modal inconsistency embedding module is used to extract fact-level inconsistency embedding and sentiment-level inconsistency embedding based on the encoded features. The inconsistency fuzzification module is used to fuse fact embedding and sentiment embedding and apply fuzzy logic processing to generate fuzzy features that can be input into quantum neural networks; The quantum modal fusion and detection module is used to map fuzzy features into quantum states, perform multimodal information interaction through parameterized quantum circuits, and output ironic prediction results.

2. The system according to claim 1, wherein: The multimodal feature encoding module includes a text encoding submodule and an image encoding submodule; The text encoding submodule encodes the text input based on the pre-trained RoBERTa model to generate a text feature matrix; The image encoding submodule encodes the image input based on the pre-trained ViT model to generate an image feature matrix.

3. The system according to claim 1, wherein: The cross-modal inconsistency embedding module includes a factual inconsistency module and a sentiment inconsistency module; The fact inconsistency module uses a multi-head self-attention component to interact with the text feature matrix and the image feature matrix to extract fact inconsistency embedding; The sentiment inconsistency module extracts sentiment inconsistency embedding based on external sentiment polarity knowledge and a continuous contrastive learning strategy, and jointly optimizes the sentiment representation via KL divergence loss and mean squared error loss.

4. The system according to claim 1, wherein: The inconsistency fuzzification module concatenates factual inconsistency embedding and sentiment inconsistency embedding into a unified feature, and fuzzifies each feature based on a set of membership functions to generate a fuzzy set representation for quantum neural network.

5. The system according to claim 1, wherein: The quantum modal fusion and detection module first maps the fuzzy features into quantum states through RY angle encoding, then performs multimodal information interaction in a variational quantum circuit containing a rotation gate and a controlled NOT gate. Finally, it extracts the expected value through Pauli Z measurement and outputs the sarcasm judgment result through a linear layer. The system is trained using a combination of sentiment classification loss, contrastive learning loss, and sentiment embedding loss.

6. A multimodal sarcasm detection method, characterized in that: The following steps are involved: Step 1: Multimodal feature encoding: perform RoBERTa encoding on the text input to obtain the text feature matrix, and perform ViT encoding on the image input to obtain the image feature matrix; Step 2: Cross-modal inconsistency embedding, extracting inconsistency embeddings through the factual inconsistency module and the sentiment inconsistency module respectively; Step 3: Inconsistency fuzzification and quantum coding: After embedding the extracted inconsistencies into the concatenated data, the membership function is applied to fuzzify them and mapped into quantum states through angle coding. Step 4: Irony detection, input the quantum state into the variational quantum circuit for information interaction, and obtain the final irony classification result through measurement and linear layers.

7. The method according to claim 6, characterized in that In step 2, the fact inconsistency module uses a multi-head self-attention component to calculate the difference between text features and image features to generate fact embedding.

8. The method according to claim 6, characterized in that In the second step, the sentiment inconsistency module constructs a supervisory signal based on external sentiment polarity knowledge, and optimizes the sentiment embedding by calculating the KL divergence between the embedding similarity and the supervisory signal and the mean square error of the sentiment polarity prediction.

9. The method according to claim 6, characterized in that In the step three, a set of membership functions is used to perform fuzzy processing on the inconsistency features after splicing to enhance the expression of inherent uncertainty.

10. The method according to claim 6, characterized in that The variational quantum circuit in step 4 includes a parameterized rotation gate and a controlled NOT gate, and the quantum modal fusion and detection module is trained by minimizing the combined loss of sentiment classification loss, contrastive learning loss, and sentiment embedding loss.

Images