A common sense reasoning-based multi-modal question-text inconsistency detection method and system

Abstract

The application provides a multi-modal question-text inconsistency detection method and system based on common sense reasoning, and relates to the technical field of artificial intelligence. The method comprises the following steps: acquiring knowledge clues related to the multi-modal content of a post; constructing a deep common sense reasoning question for verifying the authenticity of the knowledge clues; obtaining a first answer by answering the common sense reasoning question based on the multi-modal content of the post; obtaining a second answer by performing open domain question answering on the common sense reasoning question; comparing whether the first answer and the second answer are consistent; if yes, the post does not have contradictions and false parts; if no, the post has contradictions and false parts; obtaining all potential inconsistency features in the post by comparing the answers; extracting the multi-modal features of the post with contradictions and false parts, fusing the multi-modal features and the inconsistency features to obtain fused features, inputting the fused features into a preset question-text inconsistency label prediction model, and outputting the question-text inconsistency label of the post.

Description

Technical Field

[0001] The present invention relates to the technical field of artificial intelligence, and in particular to a multimodal question-text discrepancy detection method and system based on common sense reasoning. Background Art

[0002] With the rapid development of social media, people often share their daily activities or express their opinions through a large number of posts. However, in order to gain a competitive advantage, malicious creators may deliberately create tempting or deceptive content in titles, posts, and tags to attract readers to click on more related links. Driven by curiosity, readers are lured to click, only to find that the content contains no valid information and is irrelevant to the title, which makes them disappointed and offended. This phenomenon is called "title-text discrepancy". These low-quality posts are very harmful. Therefore, it is of great significance to study multimodal title-text discrepancy detection.

[0003] Currently, to address discrepancies between the title and the text, existing technologies use rule-based methods to identify posts with discrepancies. However, this approach relies heavily on manually constructed rules, often based on experts' understanding of the post's structure, language features, and content logic. These rules aim to capture various inconsistencies that may exist within the post, but require significant human resources and are costly. The rules also struggle to adapt to innovations in post content and lack scalability. Existing technologies also disclose the use of neural models to identify title-text discrepancies. These neural models make predictions based on the association between post content and tags. While this method is highly scalable and adaptable to various post content types, it is a black box model, lacking transparency and interpretability for identifying title-text discrepancies. This makes it difficult to understand the fine-grained reasoning process, impacting the accuracy of title-text discrepancy detection. Summary of the Invention

[0004] In order to solve the problem that the above-mentioned existing technologies lack explainability and transparency in question-text discrepancy detection, which affects the detection accuracy, the present invention proposes a multimodal question-text discrepancy detection method and system based on common sense reasoning, which ensures that the question-text discrepancy detection is explainable and transparent, and improves the detection accuracy.

[0005] In order to achieve the above technical effects, the technical solutions of the present invention are as follows:

[0006] A multimodal question-text discrepancy detection method based on common sense reasoning includes the following steps:

[0007] S1. Obtain knowledge clues related to the multimodal content of the post;

[0008] S2. Construct a deep commonsense reasoning question to verify the authenticity of the knowledge clue, answer the commonsense reasoning question based on the multimodal content of the post to obtain a first answer, and conduct open-domain question answering on the commonsense reasoning question to obtain a second answer;

[0009] S3. Compare the first and second answers to see if they are consistent. If so, the post contains no contradictions or falsehoods. If not, the post contains contradictions or falsehoods. By comparing the answers, all potential inconsistencies in the post are identified, and then proceed to S4.

[0010] S4. Extract the multimodal features of the posts containing contradictory and false parts, fuse the multimodal features with the inconsistency features to obtain fused features, input the fused features into a preset title-text discrepancy label prediction model, and output the title-text discrepancy label of the post.

[0011] Preferably, a multimodal retrieval enhancement method is used to obtain knowledge clues related to the multimodal content of the post, wherein the knowledge clues include context clues of the post content and common sense clues that help perceive deceptive content.

[0012] Preferably, the common sense reasoning problem includes basic sub-problems and complex problems, and the construction of a deep common sense reasoning problem for verifying the authenticity of the knowledge clue includes: constructing basic sub-problems and complex problems.

[0013] Preferably, constructing the basic sub-problems comprises the following steps:

[0014] S201. Retrieve the question template of the knowledge clue by using relationship type matching;

[0015] S202. Instruct the knowledge clue to generate the basic sub-questions based on the question template.

[0016] Preferably, constructing the complex question comprises the following steps:

[0017] S211. Select an appropriate sub-problem from the basic sub-problem;

[0018] S212. Combine appropriate sub-problems into higher-order complex problems.

[0019] Preferably, the constructing of the complex problem further comprises the following steps:

[0020] S213. Verify the high-order complex problem and obtain the problem quality validity test result.

[0021] Preferably, a preset open-domain question-answering model is used to perform open-domain question-answering on the common sense reasoning question to obtain the second answer.

[0022] Preferably, the multimodal features include visual features, text features, cross-modal features, language features and author portrait features.

[0023] Preferably, the calculation expression of the title-text mismatch label prediction model is as follows:

[0024]

[0025] in, is the prediction result of the post title not matching the label, Softmax(.) is the activation function, W is the learned parameter matrix, o is the input vector of the fusion feature, and b is the bias vector.

[0026] A multimodal question-text discrepancy detection system based on common sense reasoning, including:

[0027] The acquisition module is used to obtain knowledge clues related to the multimodal content of the post;

[0028] A question construction and processing module is used to construct a deep common sense reasoning question for verifying the authenticity of the knowledge clue, answer the common sense reasoning question based on the multimodal content of the post to obtain a first answer, and conduct open domain question answering on the common sense reasoning question to obtain a second answer;

[0029] a comparison processing module, configured to compare whether the first answer and the second answer are consistent; if so, the post contains no contradictions or falsehoods; if not, the post contains contradictions or falsehoods, and to obtain all potential inconsistency features in the post by comparing the answers;

[0030] The feature fusion prediction module is used to extract the multimodal features of posts with contradictory and false parts, fuse the multimodal features and the inconsistency features into a preset title-text discrepancy label prediction model, and output the title-text discrepancy label of the post.

[0031] Compared with the prior art, the beneficial effects of the technical solution of the present invention are:

[0032] The present invention proposes a multimodal question-text discrepancy detection method based on common sense reasoning. First, knowledge clues related to the multimodal content of the post are obtained, so that the collection of evidence clues is more comprehensive and accurate, and can fully cover all potentially deceptive content in the post; secondly, deep common sense reasoning questions are constructed to question each potentially deceptive knowledge clue. The deep common sense reasoning questions have controllable difficulty and can flexibly inquire about complex and hidden semantic relationships, thereby gradually verifying the authenticity of the knowledge clues. This method is more in-depth and effective than traditional simple matching questions; then, the first answer to the multimodal content answer of the source post and the second answer of the open domain question and answer are compared to determine whether they are consistent, so as to infer all potential inconsistency features in the post. By comparing the answers from the two sources, the complex question-text discrepancy detection task is subdivided, and an explainable analysis process is provided to facilitate the location of the specific induction point of the question-text discrepancy; finally, a question-text discrepancy label prediction model is used to fuse all potential inconsistency features and multimodal features in the post, and output the post's question-text discrepancy label, thereby ensuring the accuracy and reliability of the question-text discrepancy label detection result. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 A flowchart of a multimodal question-text discrepancy detection method based on common sense reasoning proposed in an embodiment of the present invention is shown;

[0034] Figure 2 A comparison chart showing the performance of multimodal question-text discrepancy detection proposed in an embodiment of the present invention;

[0035] Figure 3 A structural diagram of a multimodal question-text discrepancy detection system based on common sense reasoning proposed in an embodiment of the present invention;

[0036] Figure 4 A diagram showing the working principle of a multimodal question-text discrepancy detection system based on common sense reasoning proposed in an embodiment of the present invention;

[0037] Figure 5 A structural diagram of a search module proposed in an embodiment of the present invention is shown;

[0038] Figure 6 A diagram showing the structure of a questioning module proposed in an embodiment of the present invention;

[0039] Figure 7 A diagram showing examples of questions used in the chain thinking prompts proposed in an embodiment of the present invention.

[0040] 1. Retrieval module; 11. Context clue retrieval submodule; 12. External common sense clue retrieval submodule; 2. Questioning module; 21. Basic sub-question generation submodule; 22. Complex question generation submodule; 23. Question quality and validity verification submodule; 3. Consistency verification and prediction module. DETAILED DESCRIPTION

[0041] The accompanying drawings are included to provide a further understanding of the present application and are incorporated in and constitute a part of this specification, illustrate embodiments contemplated for carrying out the present application, and, together with the description, serve to explain principles of the present application.

[0042] It is to be understood that some of the details of the apparatus described in the ensuing discussion can be implcatedly disclosed and are therefore not to be necessarily repeated herein for the sake of brevity and conciseness.

[0043] In order to facilitate the understanding of the present embodiment, first, the prior art information of the present embodiment is introduced as follows:

[0044] With the development of social networks, malicious creators often post misinformative posts to gain profits. These online posts are usually filled with exaggerated, terrifying, offensive, and deceptive content and quickly spread on the network, thereby damaging user experience. In the face of such a large number of posts, manual review is not feasible. Therefore, machine detection has become a hot research topic.

[0045] In the academic field, one research direction is to analyze social behaviors, papers “The role of user profiles for fake news detection” and “Detect rumors in microblog posts using propagation structure via kernel learning” point out that these social behaviors include the posting behavior of the creator and user status data such as the sharing, commenting, clicking and reading time of the post, however, these research works have the cold start problem, i.e. cannot detect new published posts without historical behavior data; paper “Detecting clickbait in chinese social media by prompt learning” proposes another method mainly based on the content of the post. It uses language features such as appropriate words, stance, punctuation to infer the label of the title-text inconsistency, but these features are mostly handcrafted, time-consuming, costly and difficult to extend. Researchers turn to data-driven neural network methods, which can fully encode the contextual semantics of the post, however, posts often use irrelevant but attractive titles to induce users to click on annoying or deceptive content articles; in order to discover this inconsistency, paper “Attention-fused deep relevancy matching network for clickbait detection” proposes to measure the matching degree of the title and the text by n-gram overlap or twin networks, however, the title is short text, while the text is long text, the attributes of these two kinds of text are different, the amount of information they contain is different, and the language and statistical features are also different, the method based on simple word matching to calculate the relevance will introduce a lot of irrelevant noise, thus affecting the accuracy; paper “Clickbait detection via contrastive variational modelling of text and label” points out that this problem can be solved by reducing the text to short text and then matching, but this method has a ceiling effect, i.e. the performance of the model is limited by the reduction model; in addition, paper “Identifying clickbait: A multi-strategy approach using neural networks” points out that inconsistency not only exists within the text modality, but also may exist between various modalities, for example, a rumor text will be accompanied by an attractive but irrelevant actress cover photo, many fusion-based methods have been proposed to combine multi-modal features for prediction, however, they are mostly black box models, which cannot analyze the reasons for the title-text inconsistency;

[0046] In the patent field, Yang Zhenyu et al. proposed a personalized news recommendation method based on fake news identification. By extracting features and detecting fake news from users' historical clicked news and candidate news, the recommendation probability value was calculated to provide personalized news recommendations; Chen Xiaoshuai et al. disclosed an embodiment of a method for more accurately identifying clickbait titles by generating multimodal features through the integration of media features and text features; Chen Yiming et al. provided a method that combines content classification and audit sub-models to detect whether the content title meets the set conditions. The method selects the appropriate audit sub-model through classification labels, and ultimately improves the accuracy and reliability of the title audit; Different from the above studies, the present invention proposes to solve this task from the perspective of asking questions. The present invention questions every suspicious point in the post, and finds the answer to each question in the context of the post. It can also use open domain question answering to retrieve the answer from the Internet with a wide range of common sense facts, and infer the result by comparing the two types of answers.

[0047] The paper "Pathqg: Neural question generation from facts" points out that inconsistencies can be found through answer comparison and explains the reasons for result prediction. This question-and-answer-based verification technique has good interpretability and wide application range, such as fact-checking in the paper "QACheck: A demonstration system for question-guided multi-hop fact-checking", fake news identification in the paper "Muser: A multi-step evidence retrieval enhancement framework for fake news detection", and error correction in the paper "Evidence-based factual error correction". However, the paper "Socratic question generation: A novel dataset, models, and evaluation" points out that it is challenging to ask a question that hits the nail on the head. A good question can speed up the discovery of evidence, while a bad question wastes time on ineffective tests. Previous research has mainly focused on shallow questions, and the paper "Answer-driven deep question generation based on reinforcement learning" points out that the answer can be found through word matching. It is based on a variant of the encoder-decoder framework, including Seq2Seq, pre-trained models, dual learning models, graph-based models, and adversarial network models, to convert input text into questions. These trivial questions are difficult to detect complex and disguised inconsistencies. The paper "Reasoning circuits: Few-shot multi-hop question generation with structured rationales" studies multi-hop questions by using reasoning chains, but its answer process only covers limited context of the post, and it is difficult to detect common sense inconsistencies and other difficult problems that do not match the post. Therefore, the paper "Generating deep questions with commonsense reasoning ability from the text by disentangled adversarial inference" proposes common sense reasoning questions, but the difficulty is uncontrollable.The paper "Difficulty controllable generation of reading comprehension questions" controls difficulty by using latent variables, which are obscure and lack interpretability. In contrast, this paper proposes generating deep commonsense reasoning questions in a complex manner similar to stacking blocks, which allows for flexible control of difficulty.

[0048] The technical solution of the present invention is further described below with reference to the accompanying drawings and embodiments.

[0049] Example 1

[0050] like Figure 1 As shown, this embodiment proposes a multimodal question-text mismatch detection method based on common sense reasoning, including the following steps:

[0051] S1. Obtain knowledge clues related to the multimodal content of the post;

[0052] In S1, a multimodal retrieval enhancement method is used to obtain knowledge clues related to the multimodal content of the post, where the knowledge clues include context clues of the post content and common sense clues that help perceive deceptive content.

[0053] In posts where the title and text do not match, the inconsistencies usually exist in fine-grained entities and relationships in the context, some of which even involve external common sense, which is shared by most people and includes facts about the physical world, history, celebrities, events, etc. Although not explicitly mentioned in the post, they are indispensable nodes in the reasoning process, and only with these nodes can a reasoning chain be formed. In order to facilitate the raising of reasonable questions to discover inconsistencies in posts, the present invention retrieves all relevant knowledge clues from the context.

[0054] These knowledge clues usually come from two sources, namely S=S P ∪S C , S represents knowledge clues, S P represents contextual clues from the post content, while S C represents a set of common sense clues that help perceive deceptive content. The knowledge clue S is expressed as S∈(o,r,s), where r represents a relationship, o and s represent the head entity and tail entity, respectively.

[0055] To facilitate reasoning, for each picture v∈V in the post, V represents the picture set and v represents the picture in the picture set, its text description is obtained through the existing preset language-image pre-training model ALIP; considering that some clues of mismatch between the title and the text are the text in the picture, the existing preset OCR model is used to extract the picture text in the picture; then the text description and the picture text t∈T are input into the dependency parser to obtain theP Fine-grained entities and relationships.

[0056] In order to obtain S C The traditional, simple approach is to introduce an external knowledge graph (KG), but its coverage is limited. Furthermore, the outdated knowledge contained in the KG cannot match the new events discussed in the post. Therefore, we propose to derive clues by performing hint learning on a pre-set large language model (LLM). LLMs cover a wider range of common sense facts than KGs. A three-step retrieval enhancement technique is used to mitigate the hallucination problem of large language models (LLMs). The three steps are as follows:

[0057] First, content related to the post is retrieved from the web and used to guide large language models (LLMs) to generate higher quality knowledge clues. The image w∈V of each post is used as a query, and a search engine is used to collect the top w′ images and articles similar to it. These articles describe facts, events, participants, etc. related to the image. Similarly, based on entity queries, the top v′ text records are searched for each text description and image text t∈T.

[0058] Then, in order to reduce noise, an existing preset retrieval-based language model is used, specifically the RECOMP model. Under the guidance of the query words, the RECOMP model generates a short summary based on the retrieval results.

[0059] Finally, useful information is extracted from these retrieved results and condensed into retrieval enhancement prompts; the external common sense clue retrieval submodule 12 uses the existing excellent multimodal encoder MUSTIE model to represent each retrieved image or text as e g ; Similarly, the text and image of the post are encoded as and

[0060] This embodiment selects the top N candidates that are most relevant to the content of post x, and their relevance is expressed by Calculate, where λ represents the weight, represents the text encoding representation of the post, Represents the image encoding representation. This embodiment creates a prompt customized for the task of this embodiment. The prompt is as follows: "Based on the provided context: [context], considering the above information in [context] and your knowledge, please tell me the relationship between [entity-1] and [entity-2]." The prompt contains two inputs: the retrieved content as the context, the entity o mentioned in the post, and the classic common sense relationship type r. The type is predefined and collected from the classic common sense relationship in the external knowledge graph KG. The prompt can ask the large language model LLMs to output a common sense entity s. This entity can be used to prompt higher-order common sense entities. Through k rounds of repetition, k-order triples {(o, r, s)}∈S can be collected. C .

[0061] S2. Construct a deep commonsense reasoning question to verify the authenticity of the knowledge clue, answer the commonsense reasoning question based on the multimodal content of the post to obtain a first answer, and conduct open-domain question-answering on the commonsense reasoning question to obtain a second answer; wherein the second answer is obtained by conducting open-domain question-answering on the commonsense reasoning question using a preset open-domain question-answering model;

[0062] In S2, the common sense reasoning problem includes basic sub-problems and complex problems. The construction of a deep common sense reasoning problem for verifying the authenticity of the knowledge clue includes: constructing basic sub-problems and complex problems; constructing the basic sub-problems includes the following steps:

[0063] S201. Retrieve the question template of the knowledge clue by using relationship type matching;

[0064] S202. Instruct the knowledge clue to generate the basic sub-questions based on the question template.

[0065] Complex problems can usually be decomposed into a series of simpler basic sub-problems, each of which corresponds to a reasoning step, thus forming a reasoning chain; in turn, these basic sub-problems should be able to be recombined into a complex problem. Inspired by this, this embodiment creates a series of basic sub-problems as the basis for combination. Specifically,

[0066] Generate a single-hop basic sub-question for each retrieved triple, using one entity as the answer and the rest as the question object. Since this is a simple factual question generation task (QG), a preset BART model was selected to complete this task. The preset BART model uses the BART model architecture. The BART model performs well in this simple task. To enhance fluency, a question template is introduced to guide generation. Specifically,

[0067] Using one entity as the answer and other entities as question objects, a first-order sub-question is generated for each retrieved triple; in order to improve the fluency of the results, a question template is introduced to guide the generation. Based on previous work, patterns are extracted from a large amount of unlabeled question data as templates.

[0068] Then, for each triple, relation type matching is used to retrieve a question template for the knowledge clue. Based on the question template, a powerful end-to-end BART model is used to generate a sub-question for each triple. To reduce computational costs, prefix tuning techniques are used. Specifically, prefix information, similar to the "conversion of triples to questions based on templates," is developed to guide BART model generation.

[0069] First, the original text of prefix, common sense knowledge and template are embedded as pr, t and u respectively. t ; Then, concatenate them together and record them as z = [pr; t; u t ] is provided to the encoder, and then transmitted to the decoder after being encoded by the encoder. The decoding process of the decoder is as follows:

[0070]

[0071] Among them, |P idx | represents the matrix P θ The length of the given prefix, z i is the embedding representation of the i token in the input z, and h i It is z i The hidden state of is the encoder in the BART model, is the decoder in the BART model; h ＜i Refers to all hidden states before the i-th one. During the training process, the BART model parameter φ is fixed, and the parameter θ involved in the prefix template is trainable.

[0072] Finally, a pool of sub-questions is generated for all triples S is the total number of generated subproblems.

[0073] Constructing the complex problem involves the following steps:

[0074] S211. Select an appropriate sub-problem from the basic sub-problem;

[0075] S212. Combine appropriate sub-problems into higher-order complex problems.

[0076] In the complex problem generation submodule 22, these simpler basic subproblems are gradually assembled into a higher-order complex problem in a building block manner. The assembly process has two stages: one is to select appropriate subproblems, and the other is to merge them into a higher-order complex problem. Such results can be used to iteratively combine higher-order complex problems, which can improve interpretability and effectively control the intermediate reasoning process, thereby obtaining better results. Specifically:

[0077] First, we need to find pairs of combinable basic sub-questions. Not every clause can be assembled into a reasonable question. Blind assembly may lead to unreasonable and unanswerable results. Clauses suitable for assembly should have some logical connection to make the results reasonable and answerable. For ease of operation, we adopt the strategy of enumeration + verification. Basic sub-questions form a sub-question pool, and potential pairs of basic sub-questions are extracted from the sub-question pool. If a i It's q j The entity mentioned in only accepts this pair (q i ,a i ) and (q j ,a j ) and consider them as related. Considering that some entities with the same semantics may have different expressions, the Spacy toolkit is used to align them. In addition, it is stipulated that a j Should not appear in q i to avoid reasoning loops. Spacy is an industry-wide toolkit for natural language tasks such as word segmentation, extraction, and alignment.

[0078] Then, we formulate high-level complex questions. Common sense questions usually contain complex reasoning structures, which are crucial for the direction of the questions. Therefore, we design a variety of typical structures as prior knowledge to guide the generation, covering almost all types of 2 to 4 hop reasoning in mainstream applications and datasets. Each structure is a directed acyclic graph, where the nodes represent a basic sub-problem and the edges represent the reasoning relationship. To combine a set of combinable basic sub-problems The simplest approach to solving complex problems is to train a classic Seq2Seq model. However, Seq2Seq models rely heavily on large amounts of training data, which requires annotations for each reasoning step. Current datasets lack such annotations, and manual annotation is expensive. To address this issue, we employ Large Language Models (LLMs), a powerful few-shot learner. We design chained thought prompts to guide LLMs, leveraging their superior contextual learning capabilities to help generate results that integrate multiple subproblems.

[0079] The prompts for the k-th reasoning structure include instructions, some examples, input and output placeholders. Considering that a question can be expressed in many ways, a variety of prompts are developed to capture this capability. A set of examples E are collected for the k-th reasoning type. k ,like Figure 3 As shown, the underlined text represents the answer to each basic sub-question. A basic sub-question q i The answer to another basic sub-problem q j This suggests that these elementary sub-questions are chained together through their answers, thus forming an effective reasoning process;

[0080] By E k Extract a subset E from i As input, this embodiment can form a variety of prompts instead of a single fixed prompt. This helps to calculate the following formula (2) for each group of basic sub-questions Q k Several deep-seated problems are decoded and expressed as follows:

[0081]

[0082] E i ={e i :e i ~E k};

[0083] Where m is the sample size, p G (·|·) represents the GPT-3 model using kernel sampling, where p=0.5 and E is the set of examples. Y k Refers to the complex problem y generated i The problem set is composed of Q, which is a set of sub-problems, e i That is, from the subset E i The selected inference type.

[0084] The constructing of the complex problem further comprises the following steps:

[0085] S213. Verify the high-order complex problem and obtain the problem quality validity test result.

[0086] The combination result may not necessarily be a common sense reasoning problem, but a shallow problem. For example, for a combination question containing multiple clauses, the answer can be found in the text through simple matching, without the need to use hidden common sense knowledge for complex reasoning. In addition, the required difficulty may not be achieved, and the answer can be obtained through shortcuts such as direct matching without the need for reasoning. To solve this problem, a validator is developed in the question quality validity verification submodule 23 to comprehensively measure the quality of the results. It is a weighted classifier;

[0087] The score calculation formula for each high-level complex question Q is

[0088] where β i is a learnable weight, x is the clue text fragment referenced when generating Q, A is the expected answer, and v i (·) is the judgment function. Q with a score exceeding the threshold λ2 is considered qualified.

[0089] In order to check whether the high-order complex questions Q generated by the qualified method are solvable, the closed-domain question answering model SG-Net is used to predict its answer. The accuracy of the SG-Net model is very high. On the popular question answering dataset SQuAD 2.0, its F1 score is only 1.79% lower than that of humans. Reward the case of consistent answers, where a1=SG_Net(x,Q), g(·) is a cosine similarity function, e a and It is the embedding representation of answers a and a1 based on the DeBERTa-v3-large model, where the DeBERTa-v3-Large model is a natural language processing model based on pre-training technology and self-attention mechanism.

[0090] In addition, we further check the satisfaction of Q in terms of complexity to avoid shortcut reasoning and reach the expected difficulty level. In order to analyze the intermediate reasoning details, we first parse Q into an AMR tree. Each subtree should correspond to a basic subproblem Q. i , whose height represents the reasoning step.

[0091] Infer the total height d ′ And compare it with the given difficulty level d, the formula is:

[0092]

[0093] v2 represents the difference score; smaller differences indicate higher differences. Detecting shortcuts typically requires a chain of reasoning as a gold standard, but this chain is not provided. Given a standard answer, this embodiment uses a simple matching question-answering GA to perform indirect detection. If this complex question can be solved by the GA, then a shortcut is likely present.

[0094] Based on this finding, the answer a ″ =GA(x,Q) and the true answer are compared for similarity. The formula is v3=1-g(e a ,e a″). Similarly, check the shortcuts of each sub-problem in the intermediate reasoning steps. In order to facilitate separation analysis, replace the answer of the previous step with the corresponding sub-tree to form an independent sub-problem Q i .

[0095] Then, the result of the inference model is used as its reference answer a (i) =SG_Net(x,Q i ), and compared with the results of the question-answering model GA for step perception, the formula is A good question should focus on the context of the input rather than going beyond the scope, so a classifier is used to capture this dependency, which is formulated as v5 = σ(W[e x ;e Q ]), where W is the weight, [;] is the concatenation operator, σ(·) is the logistic function used to predict context relevance, and e x and e Q They are respectively based on the BiDAF model, question-aware context, and context-aware question encoding. Among them, the BiDAF model is good at capturing correlations through cross-attention.

[0096] To learn the verifier, there are two stages; in the first stage, it is preliminarily trained using some labeled data. This requires constructing positive and negative training samples, where each sample (x, Q, A) can be considered a positive example. It is observed that the entity words in the question play an important role in the reasoning direction. When these entities change, the corresponding answers are usually different. Therefore, a batch of negative examples are created by replacing the entity words. Considering that the entities in different questions are usually different, the other samples Q are used. j Entity replacement in Q i , thereby generating multiple training samples; Q with true answers is regarded as positive examples, and the others are negative examples; based on this data, the cross entropy loss is used to train the verifier.

[0097] Given that data is often limited, in the second phase, the verifier is gradually updated using extended training examples generated by the generator. This involves chaining thought prompts with diverse examples to generate multiple candidate samples, with the highest-scoring examples considered positive. Negative examples are obtained through entity replacement as described previously. This extended data helps better train the verifier. Simultaneously, this validated feedback helps the generator filter out low-quality noise and output better extended data. After multiple iterations, the optimal verifier and generator are achieved.

[0098] S3. Compare the first and second answers to see if they are consistent. If so, the post contains no contradictions or falsehoods. If not, the post contains contradictions or falsehoods. By comparing the answers, all potential inconsistencies in the post are identified, and then proceed to S4.

[0099] S4. Extract the multimodal features of the posts containing contradictory and false parts, fuse the multimodal features with the inconsistency features to obtain fused features, input the fused features into a preset title-text discrepancy label prediction model, and output the title-text discrepancy label of the post.

[0100] In this embodiment, multimodal retrieval enhancement technology is first used to collect all evidence clues related to the post content. This method is not limited to text content, but also includes multiple forms such as images and videos, making the collection of evidence clues more comprehensive and accurate, and able to fully cover all potentially deceptive content in the post. Secondly, deep common sense reasoning questions are constructed to question each potentially deceptive knowledge clue. Deep common sense reasoning questions have controllable difficulty and can flexibly inquire about complex and hidden semantic relationships, thereby gradually verifying the authenticity of the knowledge clues. This approach is more in-depth and effective than traditional simple matching questions. Then, the first answer to the multimodal content answer of the source post and the second answer of the open domain question and answer are compared to determine whether they are consistent, to infer all potential inconsistency features in the post. By comparing the answers from the two sources, the complex question-text discrepancy detection task is subdivided, and an explainable analysis process is provided to facilitate the location of the specific triggering points of the question-text discrepancy. Finally, a question-text discrepancy label prediction model is used to integrate all potential inconsistency features and multimodal features in the post to output the post's question-text discrepancy label, thereby ensuring the accuracy and reliability of the question-text discrepancy label detection results.

[0101] Example 2

[0102] This embodiment further explains S4; extracting an answer from the closed context of post x To guide the direction of questioning. However, posts with inconsistent question-text labels may contain deceptive content, making the answer wrong. Therefore, the authenticity of the answer can be verified to infer that the question-text is inconsistent. If an incorrect or contradictory answer is found, this indicates that the post may be inconsistent with the question-text. Here, a well-performing open domain question answering model is used. The open domain question answering model is specifically a VE model. The VE model retrieves widely known facts on the Internet and performs reasoning to obtain another answer. The answer is supported by many well-known facts. It can be considered relatively credible. If the two answers do not match, it is very likely that the post contains contradictions and false parts, and the question and text do not match.

[0103] The existing F1 score compares similarity based on simple word matching, however, it does not work when two answers have only a few or no overlapping words but are semantically similar, in order to solve this problem, the embodiment proposes to fine-tune a transformer for comparison; considering that the questions have different difficulties and purposes, their importance is also different. In order to represent this, the embodiment adds an attention mechanism to each question to learn the importance weight, specifically, first encode the post x, question set {Q1, …, Qn} and answer pair {A1, …, An} based on the BERT model, respectively represented as n}

[0104] In order to embed separation, a special <cls>Mark and use between two answers <sep>, which learns the importance of each question using additive attention.

[0105] The post is represented as a query, the question as a key, and the answer as a value, as shown in the following calculation formula:

[0106]

[0107] where σ is a trade-off factor and W is a set of learnable weights.

[0108] In addition to answer consistency, the embodiment additionally employs multi-modal features to better predict the final label, including visual features, text features, cross-modal features, language features, and author portrait features. These features are good at capturing language features in post content that do not match the question.

[0109] By connecting all these features, the input vector o of the fusion features can be obtained to represent the post; then the input vector o is passed through the question-post mismatch label prediction model to calculate the question-post mismatch label, where the question-post mismatch label prediction model is a multilayer perceptron classifier, and the calculation expression of the question-post mismatch label prediction model is as follows:

[0110]

[0111] where, is the predicted result of the question-post mismatch label of the post, Softmax(.) is an activation function, W is a learned parameter matrix, o is the input vector of the fusion features, and b is a bias vector; the multilayer perceptron classifier is optimized by binary cross-entropy loss, i.e. If the question-post mismatch or the question has reached the maximum difficulty level d, terminate the question generation process and output the label. Otherwise, continue to verify the post by generating high-order complex questions.

[0112] The following describes five types of modal features and extraction methods:

[0113] For the extraction method of visual features, the post mainly contains two types of visual information: cover pictures and text illustrations. The features of the images in these posts are extracted through Swin-T technology. Swin-T technology uses a Transformer pre-training model as a basic framework and extracts hierarchical information of images through a sliding window-based self-attention mechanism. In practical applications, a considerable number of posts involve public figures, and most of the pictures contain portraits. Therefore, face recognition and object detection are performed through DNN and RetinaNet networks, respectively.

[0114] As for the method of extracting text features, for the title and text in the post, they are first segmented and encoded to obtain the embedded code of each word, which is input into the BERT pre-trained model to obtain a word feature set with a dimension of 768; in addition, considering that there may be deceptive text in the cover image, OCR technology is used to extract such text and encode it.

[0115] Regarding the extraction method of cross-modal matching features, posts with inconsistent titles and texts often have inconsistent content. For example, the cover image contains a person, but the main text does not mention her. Therefore, whether the content of each part of the post is consistent is one of the important bases for judging the inconsistency between the title and text. The modalities of each part of the information are different, and there is a heterogeneous gap between the corresponding features, which cannot be directly unified for modeling. To solve this problem, CT Transformer technology is used to achieve cross-modal matching between the visual features V and text features T extracted above. CT Transformer technology first adds modality type encoding to the original features, and then uses a multi-head attention mechanism to output text-aware visual features F vt =CT((TW t ),(VW v ), and visually perceived text features F tv =CT((VW v ),(TW t )), where W t 、W v is the weight matrix. Relying on the multi-head attention mechanism, the model can characterize the deceptive behavior of mismatching the title and the text from multiple perspectives.

[0116] Regarding the extraction of cross-modal matching features, posts with mismatched titles and content have distinct linguistic characteristics in terms of semantics and deceptive tactics. Language features are modeled from the following six perspectives:

[0117] Text and cover Figure 1 Consistency, using cover images that cause strong visual impact is one of the common methods used by malicious authors when writing posts that do not match the title. However, these cover images often have no connection with the actual content of the article. With the help of the pre-trained model CLIP, the fusion features of the main text b and the cover image t are extracted [b c ,t c ]=CLIP(b,t) to capture this cross-modal inconsistency, where d c Dimension the text features and cover image features.

[0118] Consistency between the text and the title: Malicious authors often create information gaps between the title and the reader's cognition, tricking them into clicking on the post link to fill their curiosity. However, the text of these posts is often of poor quality and cannot be aligned with the title content. To measure whether there is a difference between the content of the text and the title, we first input them into the pre-trained BERT model to obtain the corresponding features b and h, and then use the Siamese network to evaluate the similarity between the text and the title. bh =Siamese(b,h).

[0119] The consistency between the cover image and the title. The inconsistency between the title and the cover image will arouse the reader's urge to click on the post link. To determine whether there is a content conflict between the cover image and the title, a text generator pre-trained on the MS COCO dataset is used to generate a new title with the cover image as input, and then the cosine similarity of the generated title and the original title's BERT feature is calculated.

[0120] Title sentiment polarity: Deceptive titles usually rely on extreme emotional tones to resonate with readers and thus arouse their curiosity. The sentiment classifier outputs two-dimensional features to represent the sentiment polarity (positive / neutral / negative) and intensity value respectively. The value of each dimension is normalized to the range [0,1].

[0121] Title lexical analysis: Deceptive titles often include elements that attract readers' attention, such as numbers, exclamation marks, and emoticons. Therefore, a lexical statistical feature was constructed to record the number of punctuation marks (such as "!", "?", "~"), emoticons, pronouns, affirmative words, vague words, and other elements that appear in post titles.

[0122] Common word analysis in titles. Malicious authors often include trendy internet terms, sensitive words, names of public figures, and place names in titles. We construct a common word statistical feature to record the number of occurrences of these common words.

[0123] Regarding the extraction method of author portrait features, the author portrait can reflect the quality of the post to a certain extent. If an author frequently posts posts with inconsistent titles and texts in the past, then he or she can be identified as a malicious author, and the posts he or she posts in the future are also likely to be deceptive. In order to characterize each author u j Whether the author is malicious or not is determined by extracting portrait features based on his / her personal profile, including the author's account age, nickname, number of followers, number of followers, number of posts published, number of low-quality posts published in the past, time since the first post was published, and time since the last post was published.

[0124] Example 3

[0125] To further evaluate the multimodal question-text discrepancy detection method based on commonsense reasoning proposed in Example 1, this example conducts qualitative and quantitative analysis experiments to comprehensively evaluate the method. This example is evaluated on three classic datasets: CLDInst, Clickbait17, and FakeNewsNet. These datasets were collected through crowdsourcing and contain 4k / 3k, 9k / 29k, and 5k / 17k samples of posts with question-text discrepancies and non-question-text discrepancies, respectively. The samples come from popular social media platforms such as Twitter and Instagram. The CLDInst, Clickbait17, and FakeNewsNet datasets are all existing datasets. A significant portion of these samples involve complex multi-hop reasoning and hidden commonsense knowledge, which is suitable for the multimodal question-text discrepancy detection method based on commonsense reasoning proposed in Example 1. To evaluate detection performance, four typical metrics from the classification field were used: accuracy (ACC), precision (PRE), recall (REC), and F1 score (F1).

[0126] These metrics are high when the model is able to distinguish between discrepancies and non-discrepancies. To account for class imbalance, all evaluation models were trained using oversampling. To ensure statistical significance, the model was run five times and the average performance is reported. Figure 2 A performance comparison chart between a multimodal question-text discrepancy detection method based on common sense reasoning proposed in Example 1 and other models is shown. Extensive experimental results on three data sets show that the method proposed in Example 1 far outperforms other baseline methods. Therefore, a multimodal question-text discrepancy detection method based on common sense reasoning proposed in Example 1 can effectively detect multimodal question-text discrepancies and provide an explainable reasoning process.

[0127] Example 4

[0128] This embodiment aims to learn a detector F(y|x) to predict whether a given post x is inconsistent with the title and text. y=1 indicates that the title and text do not match. On the contrary, y=0 indicates a normal post. Post x includes multimodal content such as a text title, a cover image, and its links. These contents may contain deceptive content, contradictions, and various inconsistencies, such as a cover image that is attractive but irrelevant to the article. In addition, creators use various techniques to generate new variants to evade detection. It is very challenging to detect these disguised posts, which requires complex multi-step reasoning and common sense reasoning. Therefore, this embodiment proposes to introduce question-asking and question-answering technology into the detection process, through a set of deep common sense reasoning questions {Q1,…,Q n to challenge every content detail C in post x, the answer can be found in the limited multi-modal content body of post x Next, an open-domain question answering model is used to retrieve relevant evidence in external knowledge sources and derive another answer By comparing Inconsistent features that can help prediction can be found in this way. In this way, the present application decomposes the complex detection task into multiple small questioning steps, allowing for interpretable analysis of unknown reasoning processes.

[0129] As shown in Figure 3 and Figure 4 , the embodiment proposes a multi-modal question-text inconsistency detection system based on common sense reasoning. Specifically, first, knowledge clues related to the multi-modal content of the post are collected; then, a question generator generates questions from easy to difficult, challenging each potential deceptive clue; finally, the answer is derived by reasoning on the clue and external common sense knowledge, and the inconsistency features are found by comparing the answers, and the inconsistency features are fused with the classic language features to predict the result of the question-text inconsistency label, wherein the multi-modal question-text inconsistency detection system comprises a relevant evidence clue retrieval module 1, a clue-related questioning module 2, and a consistency verification and prediction module 3.

[0130] The retrieval module 1 is used to obtain knowledge clues related to the multi-modal content of the post.

[0131] In the retrieval module 1, inconsistencies usually exist in fine-grained entities and relationships in the context, and some even involve external common sense, which is shared by most people, including facts about the physical world, history, celebrities, events, etc.; Although not explicitly mentioned in the post, they are indispensable nodes in the reasoning process, and with these nodes, a reasoning chain can be formed. In order to facilitate the discovery of inconsistencies in the post by asking reasonable questions, the present application retrieves all relevant knowledge clues from the context.

[0132] These knowledge clues usually come from two sources, i.e. P ∪S C , S represents knowledge clues, S P represents context clues from post content, and S C represents a set of common sense clues that help to perceive deceptive content. The expression form of knowledge clues S is S∈(o,r,s), wherein r represents a relationship, and o and s represent head entity and tail entity, respectively.

[0133] Referring to Figure 5 , the retrieval module 1 comprises a context clue retrieval submodule 11 related to question-text inconsistency and an external common sense clue retrieval submodule 12;

[0134] In the context clue retrieval submodule 11, for the convenience of reasoning, for each picture v∈V in the post, V represents the picture set and v represents the picture in the picture set, its text description is obtained through the existing preset language-image pre-training model ALIP; considering that some clues of mismatch between title and text are text in the picture, the existing preset OCR model is used to extract the picture text in the picture; then the text description and the picture text t∈T are input into the dependency parser to obtain the information used to construct S P Fine-grained entities and relationships;

[0135] In the external common sense clue retrieval submodule 12, in order to obtain S C The traditional, simple approach is to introduce an external knowledge graph (KG), but its coverage is limited. Furthermore, the outdated knowledge contained in the KG cannot match the new events discussed in the post. Therefore, in the external common sense clue retrieval submodule 12, it is proposed to derive clues by performing prompt learning on a preset large language model (LLM). LLMs cover a wider range of common sense facts than KGs. The external common sense clue retrieval submodule 12 uses a three-step retrieval enhancement technique to alleviate the hallucination problem of large language models (LLMs). The three steps are as follows:

[0136] First, content related to the post is retrieved from the web and used to guide large language models (LLMs) to generate higher quality knowledge clues. The image w∈V of each post is used as a query, and a search engine is used to collect the top w′ images and articles similar to it. These articles describe facts, events, participants, etc. related to the image. Similarly, based on entity queries, the top v′ text records are searched for each text description and image text t∈T.

[0137] Then, in order to reduce noise, an existing preset retrieval-based language model is used, specifically the RECOMP model. Under the guidance of the query words, the RECOMP model generates a short summary based on the retrieval results.

[0138] Finally, useful information is extracted from these retrieved results and condensed into retrieval enhancement prompts; the external common sense clue retrieval submodule 12 uses the existing excellent multimodal encoder MUSTIE model to represent each retrieved image or text as e g ; Similarly, the text and pictures of the post are coded as e t and e v .

[0139] This embodiment selects the top N candidates that are most relevant to the content of post x, and their relevance is expressed by Calculate, where λ represents the weight, represents the text encoding representation of the post, Represents the image encoding representation. This embodiment creates a prompt customized for the task of this embodiment. The prompt is as follows: "Based on the provided context: [context], considering the above information in [context] and your knowledge, please tell me the relationship between [entity-1] and [entity-2]." The prompt contains two inputs: the retrieved content as the context, the entity o mentioned in the post, and the classic common sense relationship type r. The type is predefined and collected from the classic common sense relationship in the external knowledge graph KG. The prompt can ask the large language model LLMs to output a common sense entity s. This entity can be used to prompt higher-order common sense entities. Through k rounds of repetition, k-order triples {(o, r, s)}∈S can be collected. C .

[0140] Questioning module 2 is used to construct in-depth common sense reasoning questions to verify the authenticity of the knowledge clues, answer the common sense reasoning questions based on the multimodal content of the post to obtain a first answer, and conduct open-domain question answering on the common sense reasoning questions to obtain a second answer;

[0141] In questioning module 2, in order to discover discrepancies between the title and the text and various inconsistency features in the post, questioning module 2 generates a set of reasonable and to-the-point common sense reasoning questions to question the clue S by constructing deep common sense reasoning questions to verify the authenticity of the knowledge clue; the difficulty of common sense reasoning questions will increase to verify S through indirect and high-order relations; here, "difficulty" refers to the number of reasoning steps of common sense reasoning questions, and the common sense reasoning problems include basic sub-problems and complex problems; specifically, single-hop basic sub-problems are generated based on s∈S, and then they are combined to obtain multi-hop complex questions; in addition, a verifier is designed to ensure that the generated questions meet requirements such as language fluency, complexity matching, and grammatical validity.

[0142] See also Figure 6 The questioning module 2 consists of a basic sub-question generation sub-module 21, a complex question generation sub-module 22 and a question quality validity verification sub-module 23.

[0143] The basic sub-problem generation sub-module 21 is used to construct the basic sub-problem, including the following steps:

[0144] S201. Retrieve the question template of the knowledge clue by using relationship type matching;

[0145] S202. Instruct the knowledge clue to generate the basic sub-questions based on the question template.

[0146] Complex problems can usually be decomposed into a series of simpler basic sub-problems, each of which corresponds to a reasoning step, thus forming a reasoning chain; in turn, these basic sub-problems should be able to be recombined into a complex problem. Inspired by this, this embodiment creates a series of basic sub-problems as the basis for combination. Specifically,

[0147] Generate a single-hop basic sub-question for each retrieved triple, using one entity as the answer and the rest as the question object. Since this is a simple factual question generation task (QG), a preset BART model was selected to complete this task. The preset BART model uses the BART model architecture. The BART model performs well in this simple task. To enhance fluency, a question template is introduced to guide generation. Specifically,

[0148] Using one entity as the answer and other entities as question objects, a first-order sub-question is generated for each retrieved triple; in order to improve the fluency of the results, a question template is introduced to guide the generation. Based on previous work, patterns are extracted from a large amount of unlabeled question data as templates.

[0149] Then, for each triple, relation type matching is used to retrieve a question template for the knowledge clue. Based on the question template, a powerful end-to-end BART model is used to generate a sub-question for each triple. To reduce computational costs, prefix tuning techniques are used. Specifically, prefix information, similar to "converting triples to questions based on templates," is developed to guide BART model generation.

[0150] First, the original text of prefix, common sense knowledge and template are embedded as pr, t and u respectively. t ; Then, concatenate them together and record them as z = [pr; t; u t ] is provided to the encoder, and then transmitted to the decoder after being encoded by the encoder. The decoding process of the decoder is as follows:

[0151]

[0152] Among them, |P idx | represents the matrix P θ The length of the given prefix, z i is the embedding representation of the i token in the input z, and h i It is z i The hidden state of is the encoder in the BART model, is the decoder in the BART model; h ＜i refers to all hidden states before the i-th one. During the training process, the BART model parameter φ is fixed, and the parameters θ involved in the prefix template are trainable. Finally, a sub-problem pool is generated for all triples S is the total number of generated subproblems.

[0153] The complex question generation submodule 22 is used to construct the complex question and includes the following steps:

[0154] S211. Select an appropriate sub-problem from the basic sub-problem;

[0155] S212. Combine appropriate sub-problems into higher-order complex problems.

[0156] In the complex problem generation submodule 22, these simpler basic subproblems are gradually assembled into a higher-order complex problem in a building block manner. The assembly process has two stages: one is to select appropriate subproblems, and the other is to merge them into a higher-order complex problem. Such results can be used to iteratively combine higher-order complex problems, which can improve interpretability and effectively control the intermediate reasoning process, thereby obtaining better results. Specifically:

[0157] First, we need to find pairs of combinable basic sub-questions. Not every clause can be assembled into a reasonable question. Blind assembly may lead to unreasonable and unanswerable results. Clauses suitable for assembly should have some logical connection to make the results reasonable and answerable. For ease of operation, we adopt the strategy of enumeration + verification. Basic sub-questions form a sub-question pool, and potential pairs of basic sub-questions are extracted from the sub-question pool. If a i It is q j The entity mentioned in only accepts this pair (q i ,a i ) and (q j ,a j ) and consider them as related. Considering that some entities with the same semantics may have different expressions, the Spacy toolkit is used to align them. In addition, it is stipulated that a j Should not appear in q i To avoid reasoning loops. Spacy is an industry-wide toolkit for natural language tasks such as word segmentation, extraction, and alignment.

[0158] Then, we formulate high-level complex questions. Common sense questions usually contain complex reasoning structures, which are crucial for the direction of the questions. Therefore, we design a variety of typical structures as prior knowledge to guide the generation, covering almost all types of 2 to 4 hop reasoning in mainstream applications and datasets. Each structure is a directed acyclic graph, where the nodes represent a basic sub-problem and the edges represent the reasoning relationship. To combine a set of combinable basic sub-problems The simplest approach to solving complex problems is to train a classic Seq2Seq model. However, Seq2Seq models rely heavily on large amounts of training data, which requires annotations for each reasoning step. Current datasets lack such annotations, and manual annotation is expensive. To address this issue, we employ Large Language Models (LLMs), a powerful few-shot learner. We design chained thought prompts to guide LLMs, leveraging their superior contextual learning capabilities to help generate results that integrate multiple subproblems.

[0159] The prompts for the k-th reasoning structure include instructions, some examples, input and output placeholders. Considering that a question can be expressed in many ways, a variety of prompts are developed to capture this capability. A set of examples E are collected for the k-th reasoning type. k ,like Figure 7 As shown, the underlined text represents the answer to each basic sub-question. A basic sub-question q i The answer to another basic sub-problem q j This shows that these basic sub-questions are linked together through their answers, thus forming an effective reasoning process; k Extract a subset E from i As input, this embodiment can form a variety of prompts instead of a single fixed prompt. This helps to calculate the following formula for each set of basic sub-questions Q k Several deep-seated problems are decoded and expressed as follows:

[0160]

[0161] E i ={e i :e i ~E k };

[0162] Where m is the sample size, p G (·|·) represents the GPT-3 model using kernel sampling, where p=0.5 and E is the set of examples. Y k Refers to the complex problem y generated i The problem set is composed of Q, which is a set of sub-problems, e i That is, from the subset E i The selected inference type.

[0163] The question quality validity verification submodule 23 is used to verify the high-order complex question and obtain the question quality validity verification result;

[0164] The combination result may not necessarily be a common sense reasoning problem, but a shallow problem. For example, for a combination question containing multiple clauses, the answer can be found in the text through simple matching, without the need to use hidden common sense knowledge for complex reasoning. In addition, the required difficulty may not be achieved, and the answer can be obtained through shortcuts such as direct matching without the need for reasoning. To solve this problem, a validator is developed in the question quality validity verification submodule 23 to comprehensively measure the quality of the results. It is a weighted classifier;

[0165] The score calculation formula for each high-level complex question Q is

[0166] where β i is a learnable weight, x is the clue text fragment referenced when generating Q, A is the expected answer, and v i (·) is the judgment function. Q with a score exceeding the threshold λ2 is considered qualified.

[0167] In order to check whether the qualified high-order complex questions Q are solvable, the closed-domain question answering model SG-Net is used to predict its answer. The accuracy of the SG-Net model is very high. On the popular question answering dataset SQuAD 2.0, its F1 score is the harmonic mean of precision and recall, which is only 1.79% lower than that of humans. Reward consistent answers, where the answer predicted by the SG-Net model matches an expected answer or a standard answer, where a1 = SG_Net(x,Q), g(·) is a cosine similarity function, and e a and It is the embedding representation of answers a and a1 based on the DeBERTa-v3-large model, where the DeBERTa-v3-Large model is a natural language processing model based on pre-training technology and self-attention mechanism.

[0168] In addition, when the high-order complex problem Q is solvable, we further check the satisfaction of Q in terms of complexity to avoid shortcut reasoning and reach the expected difficulty level. In order to analyze the intermediate reasoning details, we first parse Q into an AMR tree. Each subtree should correspond to a basic subproblem Q. i , whose height represents the reasoning step.

[0169] Infer the total height d ′ And compare it with the given difficulty level d, the formula is:

[0170]

[0171] v2 represents the difference score; smaller differences indicate higher differences. Detecting shortcuts typically requires a chain of reasoning as a gold standard, but this chain is not provided. Given a standard answer, this embodiment uses a simple matching question-answering GA to perform indirect detection. If this complex question can be solved by the GA, then a shortcut is likely present.

[0172] Based on this finding, the answer a ″ =GA(x,Q) and the true answer are compared for similarity. The formula is v3=1-g(e a ,e a″ ). Similarly, check the shortcuts of each sub-problem in the intermediate reasoning steps. In order to facilitate separation analysis, replace the answer of the previous step with the corresponding sub-tree to form an independent sub-problem Q i .

[0173] Then, the result of the inference model is used as its reference answer a (i) =SG_Net(x,Q i ), and compared with the results of the question-answering model GA for step perception, the formula is A good question should focus on the context of the input rather than going beyond the scope, so a classifier is used to capture this dependency, which is formulated as v5 = σ(W[e x ;e Q ]), where W is the weight, [;] is the concatenation operator, σ(·) is the logistic function used to predict context relevance, and e x and e Q They are respectively based on the BiDAF model, question-aware context, and context-aware question encoding. Among them, the BiDAF model is good at capturing correlations through cross-attention.

[0174] To learn the verifier, there are two stages; in the first stage, it is preliminarily trained using some labeled data. This requires constructing positive and negative training samples, where each sample (x, Q, A) can be considered a positive example. It is observed that the entity words in the question play an important role in the reasoning direction. When these entities change, the corresponding answers are usually different. Therefore, a batch of negative examples are created by replacing the entity words. Considering that the entities in different questions are usually different, the other samples Q are used. j Entity replacement in Q i , thereby generating multiple training samples; Q with true answers is regarded as positive examples, and the others are negative examples; based on this data, the cross entropy loss is used to train the verifier.

[0175] Considering that data is usually limited, in the second stage, the extended training instances generated by the generator gradually update the verifier; that is, use the chain of thought prompts with various instances to generate multiple candidate samples, and consider the sample with the highest score as a positive example; the negative examples are obtained by entity replacement as described above, and these extended data can help better train the verifier. At the same time, these verified feedbacks can help the generator filter out low-quality noise and better output extended data. After several iterations, the optimal verifier and generator can be obtained.

[0176] The consistency verification and prediction module 3 is configured to compare whether the first answer and the second answer are consistent, if yes, the post does not have contradictory and false parts, and if no, the post has contradictory and false parts, obtain all potential inconsistency features in the post by comparing the answers, extract the multi-modal features of the post with contradictory and false parts, input the multi-modal features and the inconsistency features into a preset question-text inconsistency label prediction model, and output the question-text inconsistency label of the post.

[0177] In the consistency verification and prediction module 3, one answer is extracted from the closed context of the post x to guide the direction of asking questions. However, the post with the question-text inconsistency label may contain deceptive content, resulting in the answer being incorrect. Therefore, the authenticity of the answer can be verified to infer the question-text inconsistency. If an incorrect or contradictory answer is found, it indicates that the post may be question-text inconsistency. Here, a well-performing open-domain question answering model, specifically a VE model, is used to obtain another answer by retrieving widely known facts on the Internet and reasoning. Since there are many well-known facts to support the answer , it can be considered relatively credible. If the two answers do not match, it is likely that the post has contradictory and false parts and question-text inconsistency occurs.

[0178] The existing F1 score compares similarity based on simple word matching, however, it does not work when two answers have only a few or no overlapping words but are semantically similar. To solve this problem, the embodiment proposes to fine-tune a transformer for comparison; considering that questions have different difficulties and purposes, their importance also differs. To represent this, the embodiment adds an attention mechanism to each question to learn an importance weight, specifically, first encode the post x, the questions {Q1, …, Q n} and the answer pairs based on the BERT model, respectively represented as and

[0179] To embed separation, a special token is attached to each question and answer pair. <cls>Mark and use between two answers <sep>, using additive attention to learn the importance of each question.

[0180] Consider posts as queries, questions as keys, and answers as values, as shown in the following calculation formula:

[0181]

[0182] where σ is a trade-off factor and W is a set of learnable weights.

[0183] In addition to the consistency of answers,

[0184] This embodiment also uses multimodal features to better predict the final label. Multimodal features include visual features, text features, cross-modal features, language features, and author profile features. These features are good at capturing the language features in the content of posts with inconsistent titles.

[0185] By connecting all these features, the input vector o of the fused feature can be obtained to represent the post; the input vector o is then passed through the title-text mismatch label prediction model to calculate the title-text mismatch label, wherein the title-text mismatch label prediction model is a multi-layer perceptron classifier, and the calculation expression of the title-text mismatch label prediction model is as follows:

[0186]

[0187] in, is the prediction result of the post title not matching the label, Softmax(.) is the activation function, W is the learned parameter matrix, o is the input vector of the fusion feature, and b is the bias vector; the multilayer perceptron classifier is optimized by binary cross entropy loss, that is, If the question text is found to be inconsistent or the question has reached the maximum difficulty level d, the question generation process is terminated and the label is output. Otherwise, the post is verified by generating high-level complex questions.

[0188] The following describes five types of modal features and their extraction methods:

[0189] Posts primarily contain two types of visual information: cover images and body illustrations. We use Swin-T technology to extract features from these images. Swin-T leverages a pre-trained Transformer model as its foundational framework, extracting hierarchical image information through a sliding window-based self-attention mechanism. In practice, a significant number of posts involve public figures, and images often contain portraits. To this end, we employ DNN and RetinaNet networks for facial recognition and object detection, respectively.

[0190] As for the method of extracting text features, for the title and text in the post, they are first segmented and encoded to obtain the embedded code of each word, which is input into the BERT pre-trained model to obtain a word feature set with a dimension of 768; in addition, considering that there may be deceptive text in the cover image, OCR technology is used to extract such text and encode it.

[0191] Regarding the extraction method of cross-modal matching features, posts with inconsistent titles and texts often have inconsistent content. For example, the cover image contains a person, but the main text does not mention her. Therefore, whether the content of each part of the post is consistent is one of the important bases for judging the inconsistency between the title and text. The modalities of each part of the information are different, and there is a heterogeneous gap between the corresponding features, which cannot be directly unified for modeling. To solve this problem, CT Transformer technology is used to achieve cross-modal matching between the visual features V and text features T extracted above. CT Transformer technology first adds modality type encoding to the original features, and then uses a multi-head attention mechanism to output text-aware visual features F vt =CT((TW t ),(VW v ), and visually perceived text features F tv =CT((VW v ),(TW t )), where W t 、W v is the weight matrix. Relying on the multi-head attention mechanism, the model can characterize the deceptive behavior of mismatching the title and the text from multiple perspectives.

[0192] Regarding the extraction of cross-modal matching features, posts with mismatched titles and content have distinct linguistic characteristics in terms of semantics and deceptive tactics. Language features are modeled from the following six perspectives:

[0193] Text and cover Figure 1 Consistency, using cover images that cause strong visual impact is one of the common methods used by malicious authors when writing posts that do not match the title. However, these cover images often have no connection with the actual content of the article. With the help of the pre-trained model CLIP, the fusion features of the main text b and the cover image t are extracted [b c ,t c ]=CLIP(b,t) to capture this cross-modal inconsistency, where d c Dimension the text features and cover image features.

[0194] Consistency between the text and the title: Malicious authors often create information gaps between the title and the reader's cognition, tricking them into clicking on the post link to fill their curiosity. However, the text of these posts is often of poor quality and cannot be aligned with the title content. To measure whether there is a difference between the content of the text and the title, we first input them into the pre-trained BERT model to obtain the corresponding features b and h, and then use the Siamese network to evaluate the similarity between the text and the title. bh =Siamese(b,h).

[0195] The consistency between the cover image and the title. The inconsistency between the title and the cover image will arouse the reader's urge to click on the post link. To determine whether there is a content conflict between the cover image and the title, a text generator pre-trained on the MS COCO dataset is used to generate a new title with the cover image as input, and then the cosine similarity of the generated title and the original title's BERT feature is calculated.

[0196] Title sentiment polarity: Deceptive titles usually rely on extreme emotional tones to resonate with readers and thus arouse their curiosity. The sentiment classifier outputs two-dimensional features to represent the sentiment polarity (positive / neutral / negative) and intensity value respectively. The value of each dimension is normalized to the range [0,1].

[0197] Title lexical analysis: Deceptive titles often include elements that attract readers' attention, such as numbers, exclamation marks, and emoticons. Therefore, a lexical statistical feature was constructed to record the number of punctuation marks (such as "!", "?", "~"), emoticons, pronouns, affirmative words, vague words, and other elements that appear in post titles.

[0198] Common word analysis in titles. Malicious authors often include trendy internet terms, sensitive words, names of public figures, and place names in titles. We construct a common word statistical feature to record the number of occurrences of these common words.

[0199] Regarding the extraction method of author portrait features, the author portrait can reflect the quality of the post to a certain extent. If an author frequently posts posts with inconsistent titles and texts in the past, then he or she can be identified as a malicious author, and the posts he or she posts in the future are also likely to be deceptive. In order to characterize each author u j Whether the author is malicious or not is determined by extracting portrait features based on his / her personal profile, including the author's account age, nickname, number of followers, number of followers, number of posts published, number of low-quality posts published in the past, time since the first post was published, and time since the last post was published.

[0200] Obviously, the above embodiments of the present application are only examples for clearly explaining the present application, and are not intended to limit the embodiments of the present application. Based on the above description, other different forms of changes or variations can be made by those skilled in the art. Here, it is not necessary and impossible to exhaust all the embodiments. Any modification, equivalent replacement and improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the claims of the present application.< / sep> < / cls> < / sep> < / cls> ​​

Claims

1. A multimodal question-text discrepancy detection method based on common sense reasoning, characterized by: The following steps are involved: S1. Obtain knowledge clues related to the multimodal content of the post; S2. Construct a deep commonsense reasoning question to verify the authenticity of the knowledge clue, answer the commonsense reasoning question based on the multimodal content of the post to obtain a first answer, and conduct open-domain question answering on the commonsense reasoning question to obtain a second answer; S3. Compare the first and second answers to see if they are consistent. If so, the post contains no contradictions or falsehoods. If not, the post contains contradictions or falsehoods. By comparing the answers, all potential inconsistencies in the post are identified, and then proceed to S4. S4. Extracting multimodal features from posts containing contradictory and false parts, fusing the multimodal features with the inconsistency features to obtain fused features, inputting the fused features into a preset title-text discrepancy label prediction model, and outputting a title-text discrepancy label for the post; wherein, a multimodal retrieval enhancement method is used to obtain knowledge clues related to the multimodal content of the post, wherein the knowledge clues include context clues of the post content and common sense clues that help perceive deceptive content; The common sense reasoning problem includes basic sub-problems and complex problems, and the construction of a deep common sense reasoning problem for verifying the authenticity of the knowledge clue includes: constructing basic sub-problems and complex problems; Constructing the basic sub-problem includes the following steps: S201. Retrieve the question template of the knowledge clue by using relationship type matching; S202. Generating the basic sub-questions by guiding the knowledge clues based on the question template; Constructing the complex problem involves the following steps: S211. Select an appropriate sub-problem from the basic sub-problem; S212. Combine appropriate sub-problems into higher-order complex problems.

2. The multimodal question-text discrepancy detection method based on common sense reasoning according to claim 1 is characterized in that: The constructing of the complex problem further comprises the following steps: S213. Verify the high-order complex problem and obtain the problem quality validity test result.

3. The multimodal question-text discrepancy detection method based on common sense reasoning according to claim 1 is characterized in that: Perform open-domain question answering on the common sense reasoning question using a preset open-domain question answering model to obtain the second answer.

4. The multimodal question-text discrepancy detection method based on common sense reasoning according to claim 1 is characterized in that: The multimodal features include visual features, text features, cross-modal features, language features and author portrait features.

5. The multimodal question-text discrepancy detection method based on common sense reasoning according to claim 4 is characterized in that: The calculation expression of the title-text mismatch label prediction model is as follows: in, The predicted result is that the title of the post does not match the label. (.) is the activation function, is the learned parameter matrix, is the input vector of the fusion feature, and b is the bias vector.

6. A multimodal question-text discrepancy detection system based on common sense reasoning, the system being implemented based on the multimodal question-text discrepancy detection method based on common sense reasoning according to any one of claims 1 to 5, characterized in that: include: A retrieval module is used to obtain knowledge clues related to the multimodal content of the posts; A questioning module is used to construct deep common sense reasoning questions to verify the authenticity of the knowledge clues, answer the common sense reasoning questions based on the multimodal content of the post to obtain a first answer, and conduct open-domain question answering on the common sense reasoning questions to obtain a second answer; The consistency verification and prediction module is used to compare whether the first answer and the second answer are consistent. If so, the post does not contain contradictory and false parts; if not, the post contains contradictory and false parts, and all potential inconsistency features in the post are obtained by comparing the answers; the multimodal features of the posts with contradictory and false parts are extracted, and the multimodal features and the inconsistency features are fused and input into a preset title-text discrepancy label prediction model to output the title-text discrepancy label of the post.

Images