Multimodal question inconsistency detection method and system based on common sense reasoning

Abstract

The invention provides a multi-modal question inconsistency detection method and system based on common sense reasoning, and relates to the technical field of artificial intelligence, and the method comprises the steps: obtaining knowledge clues related to post multi-modal content; constructing a deep common sense reasoning question for verifying the authenticity of the knowledge clues, answering the common sense reasoning question based on the post multi-modal content to obtain a first answer, and performing open domain question answering on the common sense reasoning question to obtain a second answer; comparing whether the first answer and the second answer are consistent or not, and if yes, determining that the post does not have contradictions and false parts; if not, determining that the post has contradictions and false parts, and obtaining all potential inconsistent features in the post through answer comparison; extracting the multi-modal features of the posts with contradictions and false parts, fusing the multi-modal features and the inconsistency features to obtain fused features, inputting the fused features into a preset question inconformity label prediction model, and outputting question inconformity labels of the posts.

Description

Technical Field

[0001] The present invention relates to the technical field of artificial intelligence, and in particular to a multimodal question-text mismatch detection method and system based on common sense reasoning. Background Art

[0002] With the rapid development of social media, people often share their daily activities or express their opinions through a large number of posts. However, in order to gain a competitive advantage, malicious creators may deliberately create tempting or deceptive content in titles, posts, and tags to attract readers to click on more related links. Driven by curiosity, readers are lured to click, only to find that the content has no valid information and is irrelevant to the title, which makes them disappointed and offended. This phenomenon is called "title-text mismatch". These low-quality posts are very harmful. Therefore, it is of great significance to study multimodal title-text mismatch detection.

[0003] At present, in order to deal with the inconsistency between the title and the text, the existing technology adopts a rule-based method to identify posts with inconsistent title and text, but this method mainly relies on manually constructed rules. These rules are often based on experts' understanding of the post structure, language features and content logic, and are intended to capture various inconsistencies that may exist within the post. However, they require a lot of human resources and are costly. The rules are difficult to adapt to the innovation of post content and are not scalable. The existing technology also discloses the use of neural models to identify inconsistent title and text. The neural model predicts through the association between post content and tags. Although this method is highly scalable and adaptable to various forms of post content, it is a black box model, not transparent, and lacks explainability for identifying inconsistent title and text, making it difficult to understand the fine-grained reasoning process, affecting the accuracy of inconsistent title and text detection. Summary of the invention

[0004] In order to solve the problem that the above-mentioned prior art lacks explainability and transparency in question-text discrepancy detection, which affects the detection accuracy, the present invention proposes a multimodal question-text discrepancy detection method and system based on common sense reasoning, which ensures that the question-text discrepancy detection is explainable and transparent, and improves the detection accuracy.

[0005] In order to achieve the above technical effects, the technical solution of the present invention is as follows:

[0006] A multimodal question-text mismatch detection method based on common sense reasoning includes the following steps:

[0007] S1. Obtain knowledge clues related to the multimodal content of the post;

[0008] S2. Construct a deep common sense reasoning question for verifying the authenticity of the knowledge clue, answer the common sense reasoning question based on the multimodal content of the post to obtain a first answer, and conduct open domain question answering on the common sense reasoning question to obtain a second answer;

[0009] S3. Compare whether the first answer and the second answer are consistent. If yes, the post does not have any contradiction or false part; if no, the post does have any contradiction or false part. Obtain all potential inconsistency features in the post by comparing the answers, and execute S4.

[0010] S4. Extract the multimodal features of the posts with contradictory and false parts, fuse the multimodal features with the inconsistency features to obtain fused features, input the fused features into a preset title-text discrepancy label prediction model, and output the title-text discrepancy label of the post.

[0011] Preferably, a multimodal retrieval enhancement method is used to obtain knowledge clues related to the multimodal content of the post, and the knowledge clues include context clues of the post content and common sense clues that help perceive deceptive content.

[0012] Preferably, the common sense reasoning problem includes basic sub-problems and complex problems, and the construction of a deep common sense reasoning problem for verifying the authenticity of the knowledge clue includes: constructing basic sub-problems and complex problems.

[0013] Preferably, constructing the basic sub-problems comprises the following steps:

[0014] S201. Retrieving the question template of the knowledge clue by using relationship type matching;

[0015] S202. Instructing the knowledge clues to generate the basic sub-questions based on the question template.

[0016] Preferably, constructing the complex problem comprises the following steps:

[0017] S211. Selecting a suitable sub-problem from the basic sub-problem;

[0018] S212. Combine appropriate sub-problems into higher-order complex problems.

[0019] Preferably, the step of constructing the complex problem further comprises the following steps:

[0020] S213. Verify the high-order complex problem to obtain a problem quality validity test result.

[0021] Preferably, a preset open-domain question-answering model is used to perform open-domain question-answering on the common sense reasoning question to obtain the second answer.

[0022] Preferably, the multimodal features include visual features, text features, cross-modal features, language features and author portrait features.

[0023] Preferably, the calculation expression of the title-text mismatch label prediction model is as follows:

[0024]

[0025] in, is the prediction result of the post title not matching the label, Softmax(.) is the activation function, W is the learned parameter matrix, o is the input vector of the fusion feature, and b is the bias vector.

[0026] A multimodal question-text mismatch detection system based on common sense reasoning, comprising:

[0027] An acquisition module, used to obtain knowledge clues related to the multimodal content of the post;

[0028] A question construction processing module, used to construct a deep common sense reasoning question for verifying the authenticity of the knowledge clue, answer the common sense reasoning question based on the multimodal content of the post to obtain a first answer, and conduct open domain question answering on the common sense reasoning question to obtain a second answer;

[0029] A comparison processing module, used to compare whether the first answer and the second answer are consistent. If yes, the post does not have any contradictions or false parts; if no, the post does have any contradictions or false parts, and all potential inconsistency features in the post are obtained by comparing the answers;

[0030] The feature fusion prediction module is used to extract the multimodal features of posts with contradictory and false parts, fuse the multimodal features and the inconsistency features into a preset title-text mismatch label prediction model, and output the title-text mismatch label of the post.

[0031] Compared with the prior art, the technical solution of the present invention has the following beneficial effects:

[0032] The present invention proposes a multimodal title-text discrepancy detection method based on common sense reasoning. First, knowledge clues related to the multimodal content of a post are obtained, so that the collection of evidence clues is more comprehensive and accurate, and all potential deceptive content in the post can be fully covered; secondly, a deep common sense reasoning question is constructed to question each potential deceptive knowledge clue. The deep common sense reasoning question has controllable difficulty and can flexibly inquire about complex and hidden semantic relationships, thereby gradually verifying the authenticity of the knowledge clues. This method is more in-depth and effective than traditional simple matching questions; then, the first answer to the multimodal content answer of the source post and the second answer of the open domain question and answer are compared to see whether they are consistent, so as to infer all potential inconsistency features in the post. By comparing the answers from the two sources, the complex title-text discrepancy detection task is subdivided, and an explainable analysis process is provided to facilitate the location of the specific induction point of the title-text discrepancy; finally, a title-text discrepancy label prediction model is used to fuse all potential inconsistency features and multimodal features in the post, and the title-text discrepancy label of the post is output, thereby ensuring the accuracy and reliability of the detection result of the title-text discrepancy label. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 A flowchart of a multimodal question-text mismatch detection method based on common sense reasoning proposed in an embodiment of the present invention is shown;

[0034] Figure 2 A comparison diagram of the multi-modal title-text discrepancy detection performance proposed in an embodiment of the present invention is shown;

[0035] Figure 3 A structural diagram of a multimodal question-text discrepancy detection system based on common sense reasoning proposed in an embodiment of the present invention;

[0036] Figure 4 A working principle diagram of a multimodal question-text discrepancy detection system based on common sense reasoning proposed in an embodiment of the present invention;

[0037] Figure 5 A structural diagram of a search module proposed in an embodiment of the present invention is shown;

[0038] Figure 6 A structural diagram of a questioning module proposed in an embodiment of the present invention is shown;

[0039] Figure 7 A diagram showing examples of questions used in the chain thinking prompts proposed in an embodiment of the present invention.

[0040] 1. Retrieval module; 11. Context clue retrieval submodule; 12. External common sense clue retrieval submodule; 2 Questioning module; 21. Basic sub-question generation submodule; 22. Complex question generation submodule; 23. Question quality validity verification submodule; 3 Consistency verification and prediction module. DETAILED DESCRIPTION

[0041] The accompanying drawings are only used for illustrative purposes and are not to be construed as limiting the present invention;

[0042] It is understandable to those skilled in the art that some well-known contents may be omitted in the drawings;

[0043] In order to facilitate understanding of this embodiment, first, the prior art information of this embodiment is introduced as follows:

[0044] With the development of social networks, malicious creators often publish posts that do not match the content in order to make profits. These online posts are usually full of exaggerated, horrific, offensive and deceptive content, and spread quickly on the Internet, thus damaging the user experience. Faced with such a large number of posts, manual review is not feasible. Therefore, machine detection has become a hot research topic.

[0045] In the academic field, one research direction is to analyze social behaviors. The papers "The role of user profiles for fake news detection" and "Detect rumors in microblog posts using propagation structure via kernel learning" point out that these social behaviors include the creator's posting behavior and user status data, such as the sharing, comments, clicks and reading time of the post. However, these research works have the problem of cold start, that is, they cannot detect newly published posts without historical behavior data; the paper "Detecting clickbait in Chinese social media by prompt learning" proposes another method that is mainly based on the content of the post. It uses language features, such as appropriate words, positions, and punctuation marks to infer labels that do not match the title, but most of these features are manually formulated, which is time-consuming, labor-intensive, costly, and difficult to expand. Researchers turned to data-driven neural network methods, which can fully encode the contextual semantics of posts. However, posts often use irrelevant but attractive titles to induce users to click on annoying or deceptive articles. In order to detect this inconsistency, the paper "Attention-fused deep relevancy matching network for clickbait detection" proposed to measure the matching degree of title and text through n-gram overlap or twin networks. However, the title is a short text, while the text is a long text. The two types of text have different properties, different amounts of information, different language and statistical characteristics. Calculating relevance based on simple word matching methods will introduce a lot of irrelevant noise, thus affecting accuracy. The paper "Clickbait detection via contrastive variational modelling of text and label" pointed out that this problem can be solved by summarizing the text into short texts and then matching them, but this method has a ceiling effect, that is, the model performance is limited by the inductive model. In addition, the paper "Identifying clickbait: A multi-strategy approach using neural networks” pointed out that inconsistency exists not only within the text modality, but also between various modalities. For example, a rumor text may be accompanied by an attractive but irrelevant cover photo of an actress. Many fusion-based methods have been proposed to combine multimodal features for prediction. However, most of them are black-box models and cannot analyze the reasons for the inconsistency between the title and the text.

[0046] In the patent field, Yang Zhenyu et al. proposed a personalized news recommendation method based on fake news identification. By extracting features and detecting fake news from users' historical clicked news and candidate news, the recommendation probability value was calculated to provide personalized news recommendations; Chen Xiaoshuai et al. disclosed an embodiment that generated multimodal features by integrating media features and text features to more accurately identify clickbait titles; Chen Yiming et al. provided a method that combines content classification and audit sub-models to detect whether the content title meets the set conditions. The method selects the appropriate audit sub-model through classification labels, and ultimately improves the accuracy and reliability of the title audit; Different from the above studies, the present invention proposes to solve this task from the perspective of asking questions. The present invention questions every suspicious point in the post, and finds the answer to each question in the context of the post. It can also use open domain question answering to retrieve the answer from the Internet with extensive common sense facts, and infer the result by comparing the two types of answers.

[0047] The paper "Pathqg: Neural question generation from facts" points out that by comparing answers, inconsistencies can be found and the reasons for the prediction of the results can be explained. This question-answering-based verification technology has good interpretability and a wide range of applications, such as fact checking in the paper "QACheck: A demonstration system for question-guidedmulti-hop fact-checking", fake news identification in the paper "Muser: A multi-step evidenceretrieval enhancement framework for fake news detection", and error correction in the paper "Evidence-based factual error correction". However, the paper "Socraticquestion generation: A novel dataset, models, and evaluation" points out that it is challenging to ask a question that hits the point. A good question can accelerate the discovery of evidence, while a bad question wastes time on invalid tests. Previous studies have mainly focused on shallow questions. The paper "Answer-driven deepquestion generation based on reinforcement learning" points out that its answers can be found by word matching. It is based on variants of the encoder-decoder framework, including Seq2Seq, pre-trained models, dual learning models, graph-based models, and adversarial network models, etc., to convert input text into questions. These trivial questions have difficulty detecting complex and disguised inconsistencies. The paper "Reasoning circuits: Few-shot multi-hop question generation with structured rationales" studies multi-hop questions by using reasoning chains, but its answer process only covers the limited context of the post, and it is difficult to detect common sense inconsistencies such as the question text not matching the post. Therefore, the paper "Generating deep questions with common sense reasoning ability from the text by disentangled adversarial inference" raises the problem of common sense reasoning, but its difficulty is uncontrollable.The paper "Difficulty controllable generation of reading comprehension questions" controls the difficulty by using latent variables, which are obscure and lack interpretability. In contrast, the present invention proposes to generate deep common sense reasoning questions in a composite way similar to stacking blocks, which can flexibly control the difficulty.

[0048] The technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0049] Example 1

[0050] like Figure 1 As shown, this embodiment proposes a multimodal question-text mismatch detection method based on common sense reasoning, including the following steps:

[0051] S1. Obtain knowledge clues related to the multimodal content of the post;

[0052] In S1, a multimodal retrieval enhancement method is used to obtain knowledge clues related to the multimodal content of the post, where the knowledge clues include context clues of the post content and common sense clues that help perceive deceptive content.

[0053] In posts where the title and text do not match, the inconsistencies usually exist in fine-grained entities and relationships in the context, and some even involve external common sense, which is shared by most people and includes facts about the physical world, history, celebrities, events, etc. Although they are not explicitly mentioned in the posts, they are indispensable nodes in the reasoning process, and only with these nodes can a reasoning chain be formed. In order to facilitate the raising of reasonable questions to discover inconsistencies in posts, the present invention retrieves all relevant knowledge clues from the context.

[0054] These knowledge clues usually come from two sources, namely S = S P ∪S C , S represents knowledge clues, S P represents contextual clues from the post content, while S C represents a set of common sense clues that help perceive deceptive content. The knowledge clue S is expressed as S∈(o,r,s), where r represents a relationship, o and s represent the head entity and tail entity, respectively.

[0055] To facilitate reasoning, for each picture v∈V in the post, V represents the picture set and v represents the picture in the picture set, its text description is obtained through the existing preset language-image pre-training model ALIP; considering that some clues of mismatch between the title and the text are the text in the picture, the existing preset OCR model is used to extract the picture text in the picture; then the text description and the picture text t∈T are input into the dependency parser to obtain theP Fine-grained entities and relationships.

[0056] In order to obtain S C The traditional and simple method is to introduce an external knowledge graph KG, but its coverage is limited. In addition, the outdated knowledge contained in the external knowledge graph KG cannot match the new events discussed in the post; therefore, it is proposed to derive clues by learning the hints of the preset large language model LLMs. LLMs cover a wider range of common sense facts than KG. A three-step retrieval enhancement technique is used to alleviate the hallucination problem of large language models LLMs. The three steps are as follows:

[0057] First, content related to the post is retrieved from the web and used to guide large language models (LLMs) to generate higher quality knowledge clues. The image w∈V of each post is used as a query, and the search engine is used to collect the top w′ images and articles similar to it, which describe facts, events, participants, etc. related to the image; similarly, based on entity queries, the top v′ text records are searched for each text description and image text t∈T.

[0058] Then, in order to reduce noise, an existing preset retrieval-based language model is used, and the language model is specifically the RECOMP model. Under the guidance of the query words, the RECOMP model generates a short summary based on the retrieval results;

[0059] Finally, useful information is extracted from these retrieved results and condensed into retrieval enhancement prompts; the external common sense clue retrieval submodule 12 uses the existing excellent multimodal encoder MUSTIE model to represent each retrieved image or text as e g ; Similarly, the text and image of the post are encoded as and

[0060] This embodiment selects the top N candidates that are most relevant to the content of post x, and their relevance is expressed by Calculate, where λ represents the weight, represents the text encoding of the post, Represents the image encoding representation. This embodiment creates a prompt customized for the task of this embodiment. The prompt is as follows: "Based on the provided context: [context], considering the above information in [context] and your knowledge, please tell me the relationship between [entity-1] and [entity-2]." The prompt contains two inputs: the retrieved content as the context, the entity o mentioned in the post, and the classic common sense relationship type r. The type is predefined and collected from the classic common sense relations in the external knowledge graph KG. The prompt can ask the large language model LLMs to output a common sense entity s. This entity can be used to prompt higher-order common sense entities. Through k rounds of repetition, k-order triples {(o, r, s)}∈S can be collected. C .

[0061] S2. Construct a deep common sense reasoning question for verifying the authenticity of the knowledge clue, answer the common sense reasoning question based on the multimodal content of the post to obtain a first answer, and conduct an open domain question and answer on the common sense reasoning question to obtain a second answer; wherein the second answer is obtained by conducting an open domain question and answer on the common sense reasoning question using a preset open domain question and answer model;

[0062] In S2, the common sense reasoning problem includes basic sub-problems and complex problems, and the construction of a deep common sense reasoning problem for verifying the authenticity of the knowledge clue includes: constructing basic sub-problems and complex problems; constructing the basic sub-problems includes the following steps:

[0063] S201. Retrieving the question template of the knowledge clue by using relationship type matching;

[0064] S202. Instructing the knowledge clues to generate the basic sub-questions based on the question template.

[0065] Complex problems can usually be decomposed into a series of simpler basic sub-problems, each of which corresponds to a reasoning step, thus forming a reasoning chain; in turn, these basic sub-problems should be able to be recombined into a complex problem. Inspired by this, this embodiment creates a series of basic sub-problems as the basis for combination. Specifically,

[0066] Generate a single-hop basic sub-question for each retrieved triple, using one entity as the answer and the rest as the question object. Since this is a simple factual question generation task (QG), a preset BART model is selected to complete this task. The preset BART model adopts the BART model architecture. The BART model performs well in this simple task. In order to enhance fluency, a question template is introduced to guide generation. Specifically,

[0067] Using one entity as the answer and other entities as question objects, a first-order sub-question is generated for each retrieved triple. In order to improve the fluency of the results, a question template is introduced to guide the generation. Based on the previous work, patterns are extracted from a large amount of unlabeled question data as templates.

[0068] Then, for each triple, the question template of the knowledge clue is retrieved by using relation type matching. Based on the question template, a powerful end-to-end BART model is used to generate a sub-question for each triple. In order to reduce the computational cost, prefix tuning technology is used. Specifically, prefix information similar to "converting triples to questions according to templates" is developed to guide BART model generation.

[0069] First, the original text and template of prefix, common sense knowledge are embedded as pr, t and u respectively. t ; Then, concatenate them together and record them as z = [pr; t; u t ] is provided to the encoder, and then transmitted to the decoder after being encoded by the encoder. The decoding process of the decoder is specifically as follows:

[0070]

[0071] Among them, |P idx |Represented by the matrix P θ The length of the given prefix, z i is the embedding representation of the i token in the input z, and h i Yes i The hidden state of is the encoder in the BART model, is the decoder in the BART model; h ＜i refers to all hidden states before the i-th one. During the training process, the BART model parameter φ is fixed, and the parameter θ involved in the prefix template is trainable.

[0072] Finally, a pool of sub-questions is generated for all triples S is the total number of generated subproblems.

[0073] Constructing the complex problem includes the following steps:

[0074] S211. Selecting a suitable sub-problem from the basic sub-problem;

[0075] S212. Combine appropriate sub-problems into higher-order complex problems.

[0076] In the complex problem generation submodule 22, based on these simpler basic subproblems, a higher-order complex problem is gradually assembled in a building block manner. The assembly process has two stages, one is to select appropriate subproblems, and the other is to merge them into a higher-order complex problem. Such results can be used to iteratively combine higher-order complex problems, which can improve interpretability and effectively control the intermediate reasoning process, thereby obtaining better results, specifically:

[0077] First, we need to find pairs of combinable basic sub-questions. Not every clause can be assembled into a reasonable question. Blind assembly may lead to unreasonable and unanswerable results. Clauses suitable for assembly should have some logical connection to make the results reasonable and answerable. For ease of operation, we adopt the strategy of enumeration + verification. Basic sub-questions form a sub-question pool, and potential pairs of basic sub-questions are extracted from the sub-question pool. If a i Yes j The entity mentioned in only accepts this pair (q i ,a i ) and (q j ,a j ) and consider them as related. Considering that some entities with the same semantics may have different expressions, the Spacy toolkit is used to align them. In addition, it is stipulated that a j Should not appear in q i to avoid reasoning loops. Spacy is an industry-wide toolkit for natural language tasks such as word segmentation, extraction, and alignment;

[0078] Then, we formulate high-level complex questions. Common sense questions usually contain complex reasoning structures, which are crucial to the direction of the questions. Therefore, we designed a variety of typical structures as prior knowledge to guide the generation, covering almost all types of 2 to 4 hop reasoning in mainstream applications and datasets. Each structure is a directed acyclic graph, where nodes represent a basic sub-question and edges represent reasoning relationships. To combine a set of combinable basic sub-questions The simplest way to convert to a complex problem is to train a classic Seq2Seq model. However, the Seq2Seq model relies heavily on a large amount of training data, which needs to be annotated for each reasoning step. The current dataset does not have such annotations. In addition, manual annotation is expensive. To solve this problem, with the help of large language models (LLMs), which are powerful few-shot learners, chained thinking prompts are designed to guide large language models (LLMs), using their excellent contextual learning capabilities to help generate results that integrate multiple sub-problems.

[0079] The prompts for the k-th reasoning structure include instructions, some examples, and input and output placeholders. Considering that a question can be asked in a variety of ways, a variety of prompts are developed to capture this capability. A set of examples E are collected for the k-th reasoning type. k ,like Figure 3 As shown in the figure, the underlined text indicates the answer to each basic sub-question. i The answer to another basic sub-problem q j In , this shows that these basic sub-questions are chained together through their answers, thus forming an effective reasoning process;

[0080] By E k Extract a subset E from i As input, this embodiment can form a variety of prompts instead of a single fixed prompt. This helps to calculate each set of basic sub-questions Q by the following calculation formula (2): k Several deep-level problems are decoded, and the expressions are as follows:

[0081]

[0082] E i ={e i :e i ~E k};

[0083] Where m is the sample size, p G (·|·) represents the GPT-3 model using kernel sampling, where p=0.5 and E is the example set. Y k refers to the complex problem y generated i The problem set is composed of. Q is the sub-problem set, e i That is, from the subset E i The selected inference type.

[0084] The constructing of the complex problem further comprises the following steps:

[0085] S213. Verify the high-order complex problem to obtain a problem quality validity test result.

[0086] The combination result may not necessarily be a common sense reasoning problem, but a shallow problem. For example, for a combination problem containing multiple clauses, the answer can be found in the text through simple matching, without the need to use hidden common sense knowledge for complex reasoning. In addition, the required difficulty may not be achieved, and the answer can be obtained through shortcuts such as direct matching without the need for reasoning. To solve this problem, a validator is developed in the question quality validity verification submodule 23 to comprehensively measure the quality of the results. It is a weighted classifier;

[0087] The score calculation formula for each high-level complex question Q is

[0088] where β i is a learnable weight, x is the clue text fragment referenced when generating Q, A is the expected answer, and v i (·) is the judgment function. The score exceeds the threshold λ 2 The Q is considered qualified.

[0089] In order to check whether the high-order complex questions Q generated by the qualified system are solvable, the closed-domain question answering model SG-Net is used to predict its answer. The accuracy of the SG-Net model is very high. On the popular question answering dataset SQuAD 2.0, its F1 score is only 1.79% lower than that of humans. Rewards are given for consistent answers, where a 1 =SG_Net(x,Q), g(·) is a cosine similarity function, e a and is the answer a and a based on the DeBERTa-v3-large model 1 The DeBERTa-v3-Large model is a natural language processing model based on pre-training technology and self-attention mechanism.

[0090] In addition, we further check whether Q satisfies the complexity in order to avoid shortcut reasoning and reach the expected difficulty level. In order to analyze the intermediate reasoning details, we first parse Q into an AMR tree. Each subtree should correspond to a basic subproblem Q. i , whose height indicates the reasoning step.

[0091] Inference calculation of total height d ′ And compare it with the given difficulty level d, the formula is:

[0092]

[0093] v 2 Represents the difference score, the smaller the difference, the higher the difference score. In order to detect shortcuts, a reasoning chain is usually required as a gold standard, but this chain is not provided. Since the standard answer is given, this embodiment uses a simple matching question-answering model GA, and the question-answering model GA performs detection in an indirect way. If this complex problem can be solved by the question-answering model GA, then there is a high probability that a shortcut exists.

[0094] Based on this finding, answer a ″ =GA(x,Q) compares the similarity with the true answer, the formula is v 3 =1-g(e a ,ea″ ). Similarly, check the shortcuts of each subproblem in the intermediate reasoning steps. In order to facilitate separation analysis, replace the answer of the previous step with the corresponding subtree to form an independent subproblem Q i .

[0095] Then, the result of the inference model is used as its reference answer a (i) =SG_Net(x,Q i ), and compared with the result of question-answering model GA step-aware, the formula is A good question should focus on the context of the input, rather than going beyond the scope, so a classifier is used to capture this dependency, and its formula is v 5 =σ(W[e x ;e Q ]), where W is the weight, [;] is the concatenation operator, σ(·) is the logistic function used to predict contextual relevance, and e x and e Q They are respectively based on the BiDAF model, question-aware context, and context-aware question encoding. Among them, the BiDAF model is good at capturing correlations through cross-attention.

[0096] To learn the verifier, there are two stages; in the first stage, it is preliminarily trained using some labeled data. This requires constructing positive and negative training samples, where each sample (x, Q, A) can be considered a positive example. It is observed that the entity words in the question play an important role in the reasoning direction, and when these entities change, the corresponding answers are usually different; therefore, a batch of negative examples are created by replacing the entity words. Considering that the entities in different questions are usually different, using other samples Q j Entity replacement in Q i , thereby generating multiple training samples; Q with true answers is considered as positive examples, and the others are negative examples; based on these data, the cross entropy loss is used to train the verifier.

[0097] Considering that data is usually limited, in the second stage, the verifier is gradually updated through the extended training examples generated by the generator; that is, chained thinking prompts with various examples are used to generate multiple candidate samples, and the samples with the highest scores are regarded as positive examples; negative examples are obtained through entity replacement as described above. These extended data can help better train the verifier. At the same time, these verified feedback can help the generator filter out low-quality noise and better output extended data. After multiple iterations, the optimal verifier and generator can be obtained.

[0098] S3. Compare whether the first answer and the second answer are consistent. If yes, the post does not have any contradiction or false part; if no, the post does have any contradiction or false part. Obtain all potential inconsistency features in the post by comparing the answers, and execute S4.

[0099] S4. Extract the multimodal features of the posts with contradictory and false parts, fuse the multimodal features with the inconsistency features to obtain fused features, input the fused features into a preset title-text discrepancy label prediction model, and output the title-text discrepancy label of the post.

[0100] In this embodiment, first, multimodal retrieval enhancement technology is used to collect all evidence clues related to the content of the post. This method is not limited to text content, but also includes multiple forms such as images and videos, so that the collection of evidence clues is more comprehensive and accurate, and can fully cover all potential deceptive content in the post. Secondly, a deep common sense reasoning problem is constructed to question each potential deceptive knowledge clue. The deep common sense reasoning problem has controllable difficulty and can flexibly inquire about complex and hidden semantic relationships, thereby gradually verifying the authenticity of the knowledge clues. This method is more in-depth and effective than traditional simple matching problems. Then, the first answer to the multimodal content answer of the source post and the second answer to the open domain question and answer are compared to see if they are consistent, so as to infer all potential inconsistent features in the post. By comparing the answers from the two sources, the complex question-text discrepancy detection task is subdivided, and an explainable analysis process is provided to facilitate the location of the specific induction point where the question-text discrepancy occurs. Finally, the question-text discrepancy label prediction model is used to fuse all potential inconsistent features and multimodal features in the post, and the question-text discrepancy label of the post is output, thereby ensuring the accuracy and reliability of the detection result of the question-text discrepancy label.

[0101] Example 2

[0102] This embodiment further explains S4; extracting an answer from the closed context of post x To guide the direction of the question. However, posts with the question-text mismatch label may contain deceptive content, resulting in the answer being wrong. Therefore, the authenticity of its answer can be verified to infer the question-text mismatch. If a wrong or contradictory answer is found, this indicates that the post may be a question-text mismatch. Here, a well-performing open domain question answering model is used. The open domain question answering model is specifically a VE model. The VE model retrieves well-known facts on the Internet for reasoning to obtain another answer. The answer is supported by many well-known facts. It can be considered relatively credible. If the two answers do not match, it is very likely that the post contains contradictions and false parts, and the question and text do not match.

[0103] The existing F1 score compares similarities based on simple word matching. However, it does not work when two answers have only a few or no overlapping words but are semantically similar. To solve this problem, this embodiment proposes to fine-tune a transformer for comparison; considering that questions have different difficulties and purposes, their importance also varies. To characterize this, this embodiment adds an attention mechanism to each question to learn the importance weight. Specifically, firstly, based on the BERT model, post x, question {Q 1 ,…,Q n} and the answer is Encode and represent them as and

[0104] To embed separation, a special <cls>Mark and use between two answers <sep>, using additive attention to learn the importance of each question.

[0105] The post is considered as a query, the question is considered as a key, and the answer is considered as a value, as shown in the following calculation formula:

[0106]

[0107] where σ is a trade-off factor and W is a set of learnable weights.

[0108] In addition to answer consistency, this embodiment also uses multimodal features to better predict the final label. Multimodal features include visual features, text features, cross-modal features, language features, and author portrait features. These features are good at capturing the language features in the content of the post that does not match the question.

[0109] By connecting all these features, the input vector o of the fused feature can be obtained to represent the post; then the input vector o is passed through the title-text mismatch label prediction model to calculate the title-text mismatch label, wherein the title-text mismatch label prediction model is a multi-layer perceptron classifier, and the calculation expression of the title-text mismatch label prediction model is as follows:

[0110]

[0111] in, is the prediction result of the post title not matching the label, Softmax(.) is the activation function, W is the learned parameter matrix, o is the input vector of the fusion feature, and b is the bias vector; the multilayer perceptron classifier is optimized by binary cross entropy loss, that is, If the question text is found to be inconsistent or the question has reached the maximum difficulty level d, the question generation process is terminated and the label is output. Otherwise, continue to verify the post by generating high-level complex questions.

[0112] The following is an explanation of the five types of modal features and their extraction methods:

[0113] As for the means of extracting visual features, posts mainly contain two kinds of visual information: cover images and text illustrations. The features of the images in these posts are extracted through Swin-T technology. Swin-T technology uses the Transformer pre-trained model as the basic framework and extracts the hierarchical information of the image through the self-attention mechanism based on the sliding window. In actual applications, a considerable number of posts involve public figures, and most of the pictures contain portraits. For this purpose, facial recognition and target detection are performed through DNN and RetinaNet networks respectively.

[0114] As for the means of extracting text features, for the title and text in the post, they are first segmented and encoded to obtain the embedded code of each word, which is input into the BERT pre-trained model to obtain a word feature set with a dimension of 768; in addition, considering that there may be deceptive text in the cover image, OCR technology is used to extract such text and encode it.

[0115] Regarding the extraction method of cross-modal matching features, posts with inconsistent titles and texts often have inconsistencies in content. For example, the cover image contains a person, but the main text does not mention her. Therefore, whether the content of each part of the post is consistent is one of the important bases for judging the inconsistency between the title and the text. The different modalities of each part of the information are different, and there is a heterogeneous gap between the corresponding features, which cannot be directly unified for modeling. To solve this problem, CT Transformer technology is used to realize cross-modal matching between the visual features V and text features T extracted above. CT Transformer technology first adds modality type encoding to the original features, and then uses a multi-head attention mechanism to output text-aware visual features F vt =CT((TW t ),(VW v ), and visually perceived text features F tv =CT((VW v ),(TW t )), where W t , W v is the weight matrix. Relying on the multi-head attention mechanism, the model can characterize the deceptive behavior of the title-text discrepancy from multiple angles.

[0116] Regarding the extraction method of cross-modal matching features, posts with inconsistent titles and texts have distinct language characteristics in terms of semantics and deceptive routines. Language features are modeled from the following six perspectives:

[0117] Text and cover Figure 1 Consistency, using cover images that cause strong visual impact is one of the common methods used by malicious authors when writing posts that do not match the title. However, these cover images often have no connection with the actual content of the article. Using the pre-trained model CLIP to extract the fusion features of the main text b and the cover image t [b c ,t c ]=CLIP(b,t) to capture this cross-modal inconsistency, where d c Dimension the text features and cover image features.

[0118] Consistency between the text and the title. Malicious authors often create information gaps between the title and the reader's cognition to trick readers into clicking on the post link to fill the gap in curiosity. However, the text of these posts is usually of poor quality and cannot be aligned with the title content. To measure whether there is a difference between the content of the text and the title, first input them into the pre-trained model BERT to obtain the corresponding features b and h, and then use the Siamese network to evaluate the similarity between the text and the title. bh =Siamese(b,h).

[0119] The consistency between the cover image and the title. The inconsistency between the title and the cover image will arouse the reader's urge to click on the post link. In order to determine whether there is a content conflict between the cover image and the title, a text generator pre-trained on the MS COCO dataset is used to generate a new title with the cover image as input, and then the cosine similarity of the generated title and the BERT feature of the original title is calculated.

[0120] Title sentiment polarity. Deceptive titles usually rely on extreme emotional colors to arouse readers' resonance and thus arouse their curiosity. With the help of the sentiment classifier, the two-dimensional features are output to represent the sentiment polarity (positive / neutral / negative) and intensity value respectively. The value of each dimension is standardized to the range [0,1].

[0121] Title lexical analysis: Deceptive titles usually introduce elements that are easy to attract readers' attention, such as numbers, exclamation marks, and emoticons. Therefore, a lexical statistical feature is constructed to record the number of times punctuation marks (such as "!", "?", "~"), emoticons, pronouns, affirmative words, vague words, and other elements appear in the post title.

[0122] Analysis of common words in titles. Malicious authors often add Internet terms, sensitive words, names of public figures, place names, etc. that cater to current trends in titles. A common word statistical feature is constructed to record the number of occurrences of these common words.

[0123] As for the extraction method of author portrait features, the author portrait can reflect the quality of the post to a certain extent. If an author frequently posts posts with inconsistent titles and texts in the past, he or she can be identified as a malicious author, and the posts he or she posts later are also likely to be deceptive. In order to characterize each author u j Whether it is malicious or not, the portrait features are extracted based on the author's personal information, including the author's account age, nickname, number of followers, number of followers, number of posts published, number of low-quality posts published in the past, the time since the first post was published, and the time since the last post was published.

[0124] Example 3

[0125] In order to further evaluate the multimodal title-text discrepancy detection method based on common sense reasoning proposed in Example 1, this embodiment conducts qualitative and quantitative analysis experiments to comprehensively evaluate the multimodal title-text discrepancy detection method based on common sense reasoning proposed in Example 1. This embodiment is evaluated on three classic datasets, including CLDInst, Clickbait17, and FakeNewsNet. These datasets are collected by crowdsourcing and contain 4k / 3k, 9k / 29k, and 5k / 17k samples of title-text discrepancy / non-title-text discrepancy posts, respectively. The samples come from popular social media platforms such as Twitter and Instagram. Among them, the datasets CLDInst, Clickbait17, and FakeNewsNet are all existing datasets. A considerable number of samples involve complex multi-hop reasoning and hidden common sense knowledge, which is suitable for the task of a multimodal title-text discrepancy detection method based on common sense reasoning proposed in Example 1. In order to evaluate the detection performance, four typical indicators in the classification field are used, including accuracy (ACC), precision (PRE), recall (REC), and F1-score (F1).

[0126] These metrics are high when the model is able to distinguish between mismatches / non-mismatches well. All evaluation models were trained using oversampling techniques to account for class imbalance. Five replicates were run to ensure statistical significance and the average performance is reported. Figure 2 The performance comparison chart between the multimodal question-text discrepancy detection method based on common sense reasoning proposed in Example 1 and other models is shown. Extensive experimental results on three data sets show that the method proposed in Example 1 far exceeds other baseline methods. Therefore, the multimodal question-text discrepancy detection method based on common sense reasoning proposed in Example 1 can effectively detect multimodal question-text discrepancies and provide an explainable reasoning process.

[0127] Example 4

[0128] This embodiment aims to learn a detector F(y|x) to predict whether a given post x is inconsistent with the title and text. y=1 indicates that the title and text do not match. On the contrary, y=0 indicates a normal post. Post x includes multimodal content such as text title, cover image and its link. These contents may contain deceptive content, contradictions and various inconsistencies, such as a cover image that is attractive but irrelevant to the article. In addition, creators will use various techniques to generate new variants to evade detection. It is very challenging to detect these disguised posts, which requires complex multi-step reasoning and common sense reasoning. Therefore, this embodiment proposes to introduce question-asking and question-answering technology into the detection process, through a set of deep common sense reasoning questions from simple to complex {Q 1 ,…,Q n} to question each content detail C in post x, and the answer can be found in the finite multimodal content body of post x Next, an open-domain question-answering model is used to retrieve relevant evidence from external knowledge sources and derive another answer. By comparison Inconsistent features that are helpful for prediction can be discovered. In this way, the present invention breaks down complex detection tasks into multiple small questioning steps, allowing for explainable analysis of unknown reasoning processes.

[0129] like Figure 3 and Figure 4 As shown, this embodiment proposes a multimodal title-text discrepancy detection system based on common sense reasoning. Specifically, first, knowledge clues related to the multimodal content of the post are collected; then, questions are generated from easy to difficult through a question generator to question each potential deceptive clue, and finally, the clues and external common sense knowledge are reasoned and answered, and the inconsistency features are discovered by comparing the answers. The result of the title-text discrepancy label is predicted by fusing the shown inconsistency features with the classic language features, wherein the multimodal title-text discrepancy detection system includes a retrieval module 1 for relevant evidence clues, a clue-related questioning module 2, and a consistency verification and prediction module 3;

[0130] Retrieval module 1, used to obtain knowledge clues related to the multimodal content of the post;

[0131] In retrieval module 1, inconsistencies usually exist in fine-grained entities and relations in the context, and some even involve external common sense, which is shared by most people and includes facts about the physical world, history, celebrities, events, etc.; although they are not explicitly mentioned in the post, they are indispensable nodes in the reasoning process, and only with these nodes can a reasoning chain be formed. In order to facilitate the raising of reasonable questions to discover inconsistencies in posts, the present invention retrieves all relevant knowledge clues from the context.

[0132] These knowledge clues usually come from two sources, namely S = S P ∪S C , S represents knowledge clues, S P represents contextual clues from the post content, while S C represents a set of common sense clues that help perceive deceptive content. The knowledge clue S is expressed as S∈(o,r,s), where r represents a relationship, o and s represent the head entity and tail entity, respectively.

[0133] See also Figure 5 , the retrieval module 1 includes a context clue retrieval submodule 11 related to the inconsistency between the title and the text and an external common sense clue retrieval submodule 12;

[0134] In the context clue retrieval submodule 11, for the convenience of reasoning, for each picture v∈V in the post, V represents the picture set, and v represents the picture in the picture set, its text description is obtained through the existing preset language-image pre-training model ALIP; considering that some clues of mismatch between title and text are text in the picture, the existing preset OCR model is used to extract the picture text in the picture; then the text description and the picture text t∈T are input into the dependency parser, and the image text used to construct S is obtained. P Fine-grained entities and relationships;

[0135] In the external common sense clue retrieval submodule 12, in order to obtain S C The traditional simple method is to introduce an external knowledge graph KG, but its coverage is limited. In addition, the outdated knowledge contained in the external knowledge graph KG cannot match the new events discussed in the post; therefore, in the external common sense clue retrieval submodule 12, it is proposed to derive clues by prompting learning of the preset large language model LLMs. LLMs cover a wider range of common sense facts than KG. The external common sense clue retrieval submodule 12 adopts a three-step retrieval enhancement technology to alleviate the hallucination problem of large language models LLMs. The three steps are as follows:

[0136] First, content related to the post is retrieved from the web and used to guide large language models (LLMs) to generate higher quality knowledge clues. The image w∈V of each post is used as a query, and the search engine is used to collect the top w′ images and articles similar to it, which describe facts, events, participants, etc. related to the image; similarly, based on entity queries, the top v′ text records are searched for each text description and image text t∈T.

[0137] Then, in order to reduce noise, an existing preset retrieval-based language model is used, and the language model is specifically the RECOMP model. Under the guidance of the query words, the RECOMP model generates a short summary based on the retrieval results;

[0138] Finally, useful information is extracted from these retrieved results and condensed into retrieval enhancement prompts; the external common sense clue retrieval submodule 12 uses the existing excellent multimodal encoder MUSTIE model to represent each retrieved image or text as e g ; Similarly, the text and image of the post are encoded as e t and e v .

[0139] This embodiment selects the top N candidates that are most relevant to the content of post x, and their relevance is expressed by Calculate, where λ represents the weight, represents the text encoding of the post, Represents the image encoding representation. This embodiment creates a prompt customized for the task of this embodiment. The prompt is as follows: "Based on the provided context: [context], considering the above information in [context] and your knowledge, please tell me the relationship between [entity-1] and [entity-2]." The prompt contains two inputs: the retrieved content as the context, the entity o mentioned in the post, and the classic common sense relationship type r. The type is predefined and collected from the classic common sense relations in the external knowledge graph KG. The prompt can ask the large language model LLMs to output a common sense entity s. This entity can be used to prompt higher-order common sense entities. Through k rounds of repetition, k-order triples {(o, r, s)}∈S can be collected. C .

[0140] Questioning module 2 is used to construct a deep common sense reasoning question for verifying the authenticity of the knowledge clue, answer the common sense reasoning question based on the multimodal content of the post to obtain a first answer, and conduct open domain question answering on the common sense reasoning question to obtain a second answer;

[0141] In questioning module 2, in order to discover the inconsistent content and various inconsistent features in the post, questioning module 2 generates a set of reasonable and to-the-point common sense reasoning questions to question clue S by constructing deep common sense reasoning questions to verify the authenticity of the knowledge clues; the difficulty of common sense reasoning questions will increase to verify S through indirect and high-order relationships; here "difficulty" refers to the number of reasoning steps of common sense reasoning questions, and the common sense reasoning problems include basic sub-problems and complex problems; specifically, single-hop basic sub-problems are generated based on s∈S, and then they are combined to obtain multi-hop complex problems; in addition, a verifier is designed to ensure that the generated questions meet the requirements of language fluency, complexity matching and grammatical validity.

[0142] See also Figure 6 The questioning module 2 is composed of a basic sub-question generation sub-module 21, a complex question generation sub-module 22 and a question quality validity verification sub-module 23.

[0143] The basic sub-problem generation sub-module 21 is used to construct the basic sub-problem, including the following steps:

[0144] S201. Retrieving the question template of the knowledge clue by using relationship type matching;

[0145] S202. Instructing the knowledge clues to generate the basic sub-questions based on the question template.

[0146] Complex problems can usually be decomposed into a series of simpler basic sub-problems, each of which corresponds to a reasoning step, thus forming a reasoning chain; in turn, these basic sub-problems should be able to be recombined into a complex problem. Inspired by this, this embodiment creates a series of basic sub-problems as the basis for combination. Specifically,

[0147] Generate a single-hop basic sub-question for each retrieved triple, using one entity as the answer and the rest as the question object. Since this is a simple factual question generation task (QG), a preset BART model is selected to complete this task. The preset BART model adopts the BART model architecture. The BART model performs well in this simple task. In order to enhance fluency, a question template is introduced to guide generation. Specifically,

[0148] Using one entity as the answer and other entities as question objects, a first-order sub-question is generated for each retrieved triple. In order to improve the fluency of the results, a question template is introduced to guide the generation. Based on the previous work, patterns are extracted from a large amount of unlabeled question data as templates.

[0149] Then, for each triple, the question template of the knowledge clue is retrieved by using relation type matching. Based on the question template, a powerful end-to-end BART model is used to generate a sub-question for each triple. In order to reduce the computational cost, prefix tuning technology is used. Specifically, prefix information similar to "converting triples to questions according to templates" is developed to guide BART model generation.

[0150] First, the original text and template of prefix, common sense knowledge are embedded as pr, t and u respectively. t ; Then, concatenate them together and record them as z = [pr; t; u t ] is provided to the encoder, and then transmitted to the decoder after being encoded by the encoder. The decoding process of the decoder is specifically as follows:

[0151]

[0152] Among them, |P idx |Represented by the matrix P θ The length of the given prefix, z i is the embedding representation of the i token in the input z, and h i Yes i The hidden state of is the encoder in the BART model, is the decoder in the BART model; h ＜i refers to all hidden states before the i-th one. During the training process, the BART model parameter φ is fixed, and the parameter θ involved in the prefix template is trainable. Finally, a sub-question pool is generated for all triples S is the total number of generated subproblems.

[0153] The complex question generation submodule 22 is used to construct the complex question and includes the following steps:

[0154] S211. Selecting a suitable sub-problem from the basic sub-problem;

[0155] S212. Combine appropriate sub-problems into higher-order complex problems.

[0156] In the complex problem generation submodule 22, based on these simpler basic subproblems, a higher-order complex problem is gradually assembled in a building block manner. The assembly process has two stages, one is to select appropriate subproblems, and the other is to merge them into a higher-order complex problem. Such results can be used to iteratively combine higher-order complex problems, which can improve interpretability and effectively control the intermediate reasoning process, thereby obtaining better results, specifically:

[0157] First, we need to find pairs of combinable basic sub-questions. Not every clause can be assembled into a reasonable question. Blind assembly may lead to unreasonable and unanswerable results. Clauses suitable for assembly should have some logical connection to make the results reasonable and answerable. For ease of operation, we adopt the strategy of enumeration + verification. Basic sub-questions form a sub-question pool, and potential pairs of basic sub-questions are extracted from the sub-question pool. If a i Yes j The entity mentioned in only accepts this pair (q i ,a i ) and (q j ,a j ) and consider them as related. Considering that some entities with the same semantics may have different expressions, the Spacy toolkit is used to align them. In addition, it is stipulated that a j Should not appear in q i to avoid reasoning loops. Spacy is an industry-wide toolkit for natural language tasks such as word segmentation, extraction, and alignment;

[0158] Then, we formulate high-level complex questions. Common sense questions usually contain complex reasoning structures, which are crucial to the direction of the questions. Therefore, we designed a variety of typical structures as prior knowledge to guide the generation, covering almost all types of 2 to 4 hop reasoning in mainstream applications and datasets. Each structure is a directed acyclic graph, where nodes represent a basic sub-question and edges represent reasoning relationships. To combine a set of combinable basic sub-questions The simplest way to convert to a complex problem is to train a classic Seq2Seq model. However, the Seq2Seq model relies heavily on a large amount of training data, which needs to be annotated for each reasoning step. The current dataset does not have such annotations. In addition, manual annotation is expensive. To solve this problem, with the help of large language models (LLMs), which are powerful few-shot learners, chained thinking prompts are designed to guide large language models (LLMs), using their excellent contextual learning capabilities to help generate results that integrate multiple sub-problems.

[0159] The prompts for the k-th reasoning structure include instructions, some examples, and input and output placeholders. Considering that a question can be asked in a variety of ways, a variety of prompts are developed to capture this capability. A set of examples E are collected for the k-th reasoning type. k ,like Figure 7 As shown in the figure, the underlined text indicates the answer to each basic sub-question. i The answer to another basic sub-problem q j This shows that these basic sub-questions are linked together through their answers, thus forming an effective reasoning process; by k Extract a subset E from i As input, the present embodiment can form a variety of prompts, rather than a single fixed prompt. This helps to calculate each set of basic sub-questions Q by the following formula k Several deep-level problems are decoded, and the expressions are as follows:

[0160]

[0161] E i ={e i :e i ~E k };

[0162] Where m is the sample size, p G (·|·) represents the GPT-3 model using kernel sampling, where p=0.5 and E is the example set. Y k refers to the complex problem y generated i The problem set is composed of. Q is the sub-problem set, e i That is, from the subset E i The selected inference type.

[0163] The question quality validity verification submodule 23 is used to verify the high-order complex question and obtain the question quality validity verification result;

[0164] The combination result may not necessarily be a common sense reasoning problem, but a shallow problem. For example, for a combination problem containing multiple clauses, the answer can be found in the text through simple matching, without the need to use hidden common sense knowledge for complex reasoning. In addition, the required difficulty may not be achieved, and the answer can be obtained through shortcuts such as direct matching without the need for reasoning. To solve this problem, a validator is developed in the question quality validity verification submodule 23 to comprehensively measure the quality of the results. It is a weighted classifier;

[0165] The score calculation formula for each high-level complex question Q is

[0166] where β i is a learnable weight, x is the clue text fragment referenced when generating Q, A is the expected answer, and v i (·) is the judgment function. The score exceeds the threshold λ 2 The Q is considered qualified.

[0167] In order to check whether the qualified generated high-order complex questions Q are solvable, the closed domain question answering model SG-Net is used to predict its answer. The accuracy of the SG-Net model is very high. On the popular question answering dataset SQuAD 2.0, its F1 score is the harmonic mean of precision and recall, which is only 1.79% lower than that of humans; Rewards are given for consistent answers, where the answer predicted by the SG-Net model matches an expected answer or a standard answer, where a 1 =SG_Net(x,Q), g(·) is a cosine similarity function, e a and is the answer a and a based on the DeBERTa-v3-large model 1 The DeBERTa-v3-Large model is a natural language processing model based on pre-training technology and self-attention mechanism.

[0168] In addition, when the high-order complex problem Q is solvable, we further check whether Q satisfies the complexity in order to avoid shortcut reasoning and reach the expected difficulty level. In order to analyze the intermediate reasoning details, we first parse Q into an AMR tree. Each subtree should correspond to a basic subproblem Q. i , whose height indicates the reasoning step.

[0169] Inference calculation of total height d ′ And compare it with the given difficulty level d, the formula is:

[0170]

[0171] v 2 Represents the difference score, the smaller the difference, the higher the difference score. In order to detect shortcuts, a reasoning chain is usually required as a gold standard, but this chain is not provided. Since the standard answer is given, this embodiment uses a simple matching question-answering model GA, and the question-answering model GA performs detection in an indirect way. If this complex problem can be solved by the question-answering model GA, then there is a high probability that a shortcut exists.

[0172] Based on this finding, answer a ″ =GA(x,Q) compares the similarity with the true answer, the formula is v 3 =1-g(e a ,e a″ ). Similarly, check the shortcuts of each subproblem in the intermediate reasoning steps. In order to facilitate separation analysis, replace the answer of the previous step with the corresponding subtree to form an independent subproblem Q i .

[0173] Then, the result of the inference model is used as its reference answer a (i) =SG_Net(x,Q i ), and compared with the result of question-answering model GA step-aware, the formula is A good question should focus on the context of the input, rather than going beyond the scope, so a classifier is used to capture this dependency, and its formula is v 5 =σ(W[e x ;e Q ]), where W is the weight, [;] is the concatenation operator, σ(·) is the logistic function used to predict contextual relevance, and e x and e Q They are respectively based on the BiDAF model, question-aware context, and context-aware question encoding. Among them, the BiDAF model is good at capturing correlations through cross-attention.

[0174] To learn the verifier, there are two stages; in the first stage, it is preliminarily trained using some labeled data. This requires constructing positive and negative training samples, where each sample (x, Q, A) can be considered a positive example. It is observed that the entity words in the question play an important role in the reasoning direction, and when these entities change, the corresponding answers are usually different; therefore, a batch of negative examples are created by replacing the entity words. Considering that the entities in different questions are usually different, using other samples Q j Entity replacement in Q i , thereby generating multiple training samples; Q with true answers is considered as positive examples, and the others are negative examples; based on these data, the cross entropy loss is used to train the verifier.

[0175] Considering that data is usually limited, in the second stage, the verifier is gradually updated through the extended training examples generated by the generator; that is, chained thinking prompts with various examples are used to generate multiple candidate samples, and the samples with the highest scores are regarded as positive examples; negative examples are obtained through entity replacement as described above. These extended data can help better train the verifier. At the same time, these verified feedback can help the generator filter out low-quality noise and better output extended data. After multiple iterations, the optimal verifier and generator can be obtained.

[0176] The consistency verification and prediction module 3 is used to compare whether the first answer and the second answer are consistent. If so, the post does not have contradictory and false parts; if not, the post has contradictory and false parts, and all potential inconsistent features in the post are obtained by comparing the answers; the multimodal features of the posts with contradictory and false parts are extracted, and the multimodal features and the inconsistent features are fused and input into a preset title-text inconsistency label prediction model to output the title-text inconsistency label of the post.

[0177] In consistency verification and prediction module 3, an answer is extracted from the closed context of post x To guide the direction of the question. However, posts with the question-text mismatch label may contain deceptive content, resulting in the answer being wrong. Therefore, the authenticity of its answer can be verified to infer the question-text mismatch. If a wrong or contradictory answer is found, this indicates that the post may be a question-text mismatch. Here, a well-performing open domain question answering model is used. The open domain question answering model is specifically a VE model. The VE model retrieves well-known facts on the Internet for reasoning to obtain another answer. The answer is supported by many well-known facts. It can be considered relatively credible. If the two answers do not match, it is very likely that the post contains contradictions and false parts, and the question and text do not match.

[0178] The existing F1 score compares similarities based on simple word matching. However, it does not work when two answers have only a few or no overlapping words but are semantically similar. To solve this problem, this embodiment proposes to fine-tune a transformer for comparison; considering that questions have different difficulties and purposes, their importance also varies. To characterize this, this embodiment adds an attention mechanism to each question to learn the importance weight. Specifically, firstly, based on the BERT model, post x, question {Q 1 ,…,Q n } and the answer is Encode and represent them as and

[0179] To embed separation, a special <cls>Mark and use between two answers <sep>, using additive attention to learn the importance of each question.

[0180] The post is considered as a query, the question is considered as a key, and the answer is considered as a value, as shown in the following calculation formula:

[0181]

[0182] where σ is a trade-off factor and W is a set of learnable weights.

[0183] In addition to answer consistency,

[0184] This embodiment also uses multimodal features to better predict the final label, and the multimodal features include visual features, text features, cross-modal features, language features and author portrait features. These features are good at capturing the language features in the content of the post that does not match the title.

[0185] By connecting all these features, the input vector o of the fused feature can be obtained to represent the post; then the input vector o is passed through the title-text mismatch label prediction model to calculate the title-text mismatch label, wherein the title-text mismatch label prediction model is a multi-layer perceptron classifier, and the calculation expression of the title-text mismatch label prediction model is as follows:

[0186]

[0187] in, is the prediction result of the post title not matching the label, Softmax(.) is the activation function, W is the learned parameter matrix, o is the input vector of the fusion feature, and b is the bias vector; the multilayer perceptron classifier is optimized by binary cross entropy loss, that is, If the question text is found to be inconsistent or the question has reached the maximum difficulty level d, the question generation process is terminated and the label is output. Otherwise, continue to verify the post by generating high-level complex questions.

[0188] The following is an explanation of the five types of modal features and their extraction methods:

[0189] As for the means of extracting visual features, posts mainly contain two kinds of visual information: cover images and text illustrations. The features of the images in these posts are extracted through Swin-T technology. Swin-T technology uses the Transformer pre-trained model as the basic framework and extracts the hierarchical information of the image through the self-attention mechanism based on the sliding window. In actual applications, a considerable number of posts involve public figures, and most of the pictures contain portraits. For this purpose, facial recognition and target detection are performed through DNN and RetinaNet networks respectively.

[0190] As for the means of extracting text features, for the title and text in the post, they are first segmented and encoded to obtain the embedded code of each word, which is input into the BERT pre-trained model to obtain a word feature set with a dimension of 768; in addition, considering that there may be deceptive text in the cover image, OCR technology is used to extract such text and encode it.

[0191] Regarding the extraction method of cross-modal matching features, posts with inconsistent titles and texts often have inconsistencies in content. For example, the cover image contains a person, but the main text does not mention her. Therefore, whether the content of each part of the post is consistent is one of the important bases for judging the inconsistency between the title and the text. The different modalities of each part of the information are different, and there is a heterogeneous gap between the corresponding features, which cannot be directly unified for modeling. To solve this problem, CT Transformer technology is used to realize cross-modal matching between the visual features V and text features T extracted above. CT Transformer technology first adds modality type encoding to the original features, and then uses a multi-head attention mechanism to output text-aware visual features F vt =CT((TW t ),(VW v ), and visually perceived text features F tv =CT((VW v ),(TW t )), where W t , W v is the weight matrix. Relying on the multi-head attention mechanism, the model can characterize the deceptive behavior of the title-text discrepancy from multiple angles.

[0192] Regarding the extraction method of cross-modal matching features, posts with inconsistent titles and texts have distinct language characteristics in terms of semantics and deceptive routines. Language features are modeled from the following six perspectives:

[0193] Text and cover Figure 1 Consistency, using cover images that cause strong visual impact is one of the common methods used by malicious authors when writing posts that do not match the title. However, these cover images often have no connection with the actual content of the article. Using the pre-trained model CLIP to extract the fusion features of the main text b and the cover image t [b c ,t c ]=CLIP(b,t) to capture this cross-modal inconsistency, where d c Dimension the text features and cover image features.

[0194] Consistency between the text and the title. Malicious authors often create information gaps between the title and the reader's cognition to trick readers into clicking on the post link to fill the gap in curiosity. However, the text of these posts is usually of poor quality and cannot be aligned with the title content. To measure whether there is a difference between the content of the text and the title, first input them into the pre-trained model BERT to obtain the corresponding features b and h, and then use the Siamese network to evaluate the similarity between the text and the title. bh =Siamese(b,h).

[0195] The consistency between the cover image and the title. The inconsistency between the title and the cover image will arouse the reader's urge to click on the post link. In order to determine whether there is a content conflict between the cover image and the title, a text generator pre-trained on the MS COCO dataset is used to generate a new title with the cover image as input, and then the cosine similarity of the generated title and the BERT feature of the original title is calculated.

[0196] Title sentiment polarity. Deceptive titles usually rely on extreme emotional colors to arouse readers' resonance and thus arouse their curiosity. With the help of the sentiment classifier, the two-dimensional features are output to represent the sentiment polarity (positive / neutral / negative) and intensity value respectively. The value of each dimension is standardized to the range [0,1].

[0197] Title lexical analysis: Deceptive titles usually introduce elements that are easy to attract readers' attention, such as numbers, exclamation marks, and emoticons. Therefore, a lexical statistical feature is constructed to record the number of times punctuation marks (such as "!", "?", "~"), emoticons, pronouns, affirmative words, vague words, and other elements appear in the post title.

[0198] Analysis of common words in titles. Malicious authors often add Internet terms, sensitive words, names of public figures, place names, etc. that cater to current trends in titles. A common word statistical feature is constructed to record the number of occurrences of these common words.

[0199] As for the extraction method of author portrait features, the author portrait can reflect the quality of the post to a certain extent. If an author frequently posts posts with inconsistent titles and texts in the past, he or she can be identified as a malicious author, and the posts he or she posts later are also likely to be deceptive. In order to characterize each author u j Whether it is malicious or not, the portrait features are extracted based on the author's personal information, including the author's account age, nickname, number of followers, number of followers, number of posts published, number of low-quality posts published in the past, the time since the first post was published, and the time since the last post was published.

[0200] Obviously, the above embodiments of the present invention are only examples for clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the embodiments here. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the claims of the present invention.< / sep> < / cls> < / sep> < / cls>

Claims

1. A multimodal question-text mismatch detection method based on common sense reasoning, characterized in that: The following steps are involved: S1. Obtain knowledge clues related to the multimodal content of the post; S2. Construct a deep common sense reasoning question for verifying the authenticity of the knowledge clue, answer the common sense reasoning question based on the multimodal content of the post to obtain a first answer, and conduct open domain question answering on the common sense reasoning question to obtain a second answer; S3. Compare whether the first answer and the second answer are consistent. If yes, the post does not have any contradiction or false part; if no, the post does have any contradiction or false part. Obtain all potential inconsistency features in the post by comparing the answers, and execute S4. S4. Extract the multimodal features of the posts with contradictory and false parts, fuse the multimodal features with the inconsistency features to obtain fused features, input the fused features into a preset title-text discrepancy label prediction model, and output the title-text discrepancy label of the post.

2. The multimodal question-text mismatch detection method based on common sense reasoning according to claim 1 is characterized in that: A multimodal retrieval enhancement method is used to obtain knowledge clues related to the multimodal content of the post, wherein the knowledge clues include context clues of the post content and common sense clues that help perceive deceptive content.

3. The multimodal question-text mismatch detection method based on common sense reasoning according to claim 1 is characterized in that: The common sense reasoning problem includes basic sub-problems and complex problems, and the construction of a deep common sense reasoning problem for verifying the authenticity of the knowledge clues includes: constructing basic sub-problems and complex problems.

4. The multimodal question-text mismatch detection method based on common sense reasoning according to claim 2 is characterized in that: Constructing the basic sub-problem includes the following steps: S201. Retrieving the question template of the knowledge clue by using relationship type matching; S202. Instructing the knowledge clues to generate the basic sub-questions based on the question template.

5. The multimodal question-text mismatch detection method based on common sense reasoning according to claim 4 is characterized in that: Constructing the complex problem includes the following steps: S211. Selecting a suitable sub-problem from the basic sub-problem; S212. Combine appropriate sub-problems into higher-order complex problems.

6. The multimodal question-text mismatch detection method based on common sense reasoning according to claim 5 is characterized in that: The constructing of the complex problem further comprises the following steps: S213. Verify the high-order complex problem to obtain a problem quality validity test result.

7. The multimodal question-text mismatch detection method based on common sense reasoning according to claim 1 is characterized in that: The common sense reasoning question is answered in an open domain using a preset open domain question answering model to obtain the second answer.

8. The multimodal question-text mismatch detection method based on common sense reasoning according to claim 1 is characterized in that: The multimodal features include visual features, text features, cross-modal features, language features and author portrait features.

9. The multimodal question-text mismatch detection method based on common sense reasoning according to claim 8 is characterized in that: The calculation expression of the title-text mismatch label prediction model is as follows: in, is the prediction result of the post title not matching the label, Softmax(.) is the activation function, W is the learned parameter matrix, o is the input vector of the fusion feature, and b is the bias vector.

10. A multimodal question-text mismatch detection system based on common sense reasoning, characterized in that: include: A retrieval module is used to obtain knowledge clues related to the multimodal content of the posts; A questioning module, for constructing deep common sense reasoning questions for verifying the authenticity of the knowledge clues, answering the common sense reasoning questions based on the multimodal content of the post to obtain a first answer, and performing open domain question answering on the common sense reasoning questions to obtain a second answer; The consistency verification and prediction module is used to compare whether the first answer and the second answer are consistent. If so, the post does not contain contradictory and false parts; if not, the post contains contradictory and false parts, and all potential inconsistent features in the post are obtained by comparing the answers; the multimodal features of the posts with contradictory and false parts are extracted, and the multimodal features and the inconsistent features are fused and input into a preset title-text inconsistency label prediction model to output the title-text inconsistency label of the post.

Images