A fact checking method based on multi-modal evidence fusion and discourse relation enhancement
By constructing a heterogeneous graph of discourse relations and fusing multimodal evidence, and utilizing graph neural networks and cross-attention mechanisms, the problem of the spread of misinformation in social media was solved, improving the accuracy and reliability of the judgment of the authenticity of statements.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- BEIHANG UNIV
- Filing Date
- 2024-08-23
- Publication Date
- 2026-07-24
AI Technical Summary
Existing technologies enable the rapid spread of misinformation on social media and make it difficult to effectively determine the authenticity of claims. The multimodal evidence interaction relationships have not been fully explored, resulting in poor accuracy and reliability in judgments.
By employing multimodal evidence fusion and discourse relation enhancement methods, a heterogeneous discourse relation graph is constructed. Then, using graph neural networks and cross-attention mechanisms, statement and evidence features are fused to determine authenticity.
This improves the accuracy and reliability of multimodal fact-checking models in determining the authenticity of claims, effectively addressing the spread of misinformation on social media.
Smart Images
Figure CN119089381B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of computer science and technology, specifically relating to a fact-checking method based on multimodal evidence fusion and discourse relation enhancement. Background Technology
[0002] With the rapid development of the internet and the rise of social media, the speed of information dissemination has continuously increased. Due to the universal nature of social media, information spreads virally at extremely fast speeds. However, due to imperfect or absent review mechanisms, social media has become a breeding ground for various types of misinformation. This widespread dissemination of false information makes it difficult for people to discover the truth behind events and may even seriously affect social order.
[0003] To address the growing prevalence of misinformation on social media platforms, two tasks have been proposed: fake news detection and fact verification. The former typically involves a binary classification task to determine the veracity of the news item to be detected, while the latter requires the system to retrieve evidence related to the claim from a reliable knowledge base and determine whether the claim is true, false, or lacks sufficient information (NEI) based on this evidence.
[0004] For fact-verification tasks, some researchers focus on improving the evidence retrieval process to enhance the relevance between retrieved evidence and claims. Others focus on improving the fact-verification process itself, enabling models to better learn the representations of claims and evidence, thereby determining the veracity of claims.
[0005] Nevertheless, most past research has been limited to the text modality and has simply spliced together the features of the statement and the evidence. While a few studies have explored the interaction between statements and evidence through attention mechanisms, they have neglected the interaction between the evidence itself. Other studies have used graph neural networks for fact-checking tasks and used the information transmission between graph nodes to study the interaction between the evidence, but they have not fully explored the semantic structure information contained within the evidence, resulting in poor accuracy and reliability in judging the authenticity of statements and failing to effectively judge the statements to be detected. Summary of the Invention
[0006] The purpose of this invention is to provide a fact-checking method based on multimodal evidence fusion and discourse relation enhancement, which realizes automatic fact-checking through multimodal evidence fusion and discourse relation enhancement, improves the accuracy and reliability of multimodal fact-checking models in judging the authenticity of claims, and can effectively judge the claims to be detected.
[0007] The specific technical solution adopted by this invention is as follows:
[0008] A fact-checking method based on multimodal evidence fusion and discourse relation enhancement includes the following steps:
[0009] Step 1, Evidence Retrieval: For the statement to be detected, retrieve relevant evidence from the evidence database;
[0010] Step 2: Constructing an evidence relationship graph: Using natural language processing tools, extract discourse relationships from relevant evidence. Based on the extracted discourse relationships, construct a discourse relationship heterogeneity graph. Then, use a graph neural network to fuse and extract the features of the discourse relationship heterogeneity graph to form the discourse relationship heterogeneity graph evidence features.
[0011] Step 3, Authenticity Assessment: Using complete relevant evidence as the unit, the characteristics of the statement and the characteristics of the evidence are integrated to analyze whether the relevant evidence supports the statement to be assessed.
[0012] Furthermore, the relevant evidence in step 1 includes textual evidence and image evidence.
[0013] Furthermore, step 2 includes the following steps:
[0014] Step 101: Construct relevant indexes for the textual evidence to form a new index library;
[0015] Step 102: Train a semantic similarity model and a multimodal fact-verification model for retrieving relevant textual evidence;
[0016] Step 103: Train a multimodal fact-verification model for retrieving claim-related image evidence;
[0017] Step 104: Use the trained semantic similarity model to extract features of the statement and the evidence index, calculate cosine similarity, sort according to similarity, and select the top 5 most relevant pieces of evidence as the retrieved text evidence.
[0018] Step 105: Use the trained multimodal fact-checking model to extract features of the claims and features of the image evidence, calculate cosine similarity, sort according to similarity, and select the top 5 most relevant images as retrieved image evidence.
[0019] Furthermore, the semantic similarity model and the multimodal fact-verification model in step 1 are trained using a contrastive learning method. During the training process, evidence related to the statement is selected as positive samples, and relevant evidence unrelated to the statement is randomly selected as negative samples for contrastive learning to participate in the training.
[0020] Furthermore, step 2 includes the following steps:
[0021] Step 201: Analyze the discourse relations of textual evidence using text relation extraction tools;
[0022] Step 202: Based on the extracted discourse relations, construct a directed heterogeneous graph as an evidence relation graph to form a discourse relation heterogeneous graph;
[0023] Step 203: Connect the statement and related image evidence to nodes of the heterogeneous discourse relation graph as node features. Use the attention mechanism of the graph neural network to control message transmission. Use the attention mechanism to perform weighted summation of node features to complete the fusion and extraction of node features of the heterogeneous discourse relation graph, and obtain the final heterogeneous discourse relation graph evidence features.
[0024] Furthermore, the discourse relation heterogeneous graph includes declaration nodes, text evidence nodes, and image evidence nodes, as well as five types of edges: selection relation, parallel relation, causal relation, inverse causal relation, and ordinary connection relation.
[0025] Furthermore, step 3 includes the following steps:
[0026] Step 301: Segment the text evidence, input the segmented text evidence and image evidence together into the multimodal fact verification model, and use the multimodal fact verification model to extract text evidence features, image evidence features and the features of the statement to be detected;
[0027] Step 302: Calculate the attention distribution of statements and evidence through a cross-attention mechanism, where statement features are used as the query and evidence features are used as the key and value for feature fusion;
[0028] Step 303: Input the fused features into the pooling layer and the linear layer to determine the authenticity of the claims, which are categorized as true, false, and insufficient information.
[0029] The technical effects achieved by this invention are as follows:
[0030] This invention provides a fact-checking method based on multimodal evidence fusion and discourse relation enhancement, which realizes automatic fact-checking through multimodal evidence fusion and discourse relation enhancement. It solves the lack of automatic multimodal fact-checking, improves the accuracy and reliability of multimodal fact-checking models in judging the authenticity of claims, and can effectively judge the claims to be detected. It effectively addresses the problem of the spread of false information on social media and is of great significance in reducing the spread of false information. Attached Figure Description
[0031] Figure 1 This is a flowchart of the present invention. Detailed Implementation
[0032] To make the objectives and advantages of this invention clearer, the invention will be specifically described below with reference to embodiments. It should be understood that the following text is merely used to describe one or more specific embodiments of the invention and does not strictly limit the scope of protection specifically claimed by the invention.
[0033] Example 1:
[0034] like Figure 1 As shown, a fact-checking method based on multimodal evidence fusion and discourse relation enhancement includes the following steps:
[0035] Step 1, Evidence Retrieval: For the statement to be detected, retrieve relevant evidence from the evidence database; use a large semantic similarity model to extract important information from the evidence text and eliminate redundant content from interfering with the retrieval.
[0036] Among them, effectively retrieving relevant evidence from the evidence base involves calculating the semantic similarity between the statement to be detected and the evidence in the evidence base, and retrieving evidence related to the statement. That is, for the statement to be retrieved, relevant evidence that may be helpful in judging its authenticity is retrieved from the corresponding evidence knowledge base.
[0037] The statement to be tested refers to sentences or paragraphs published on social media, news articles or other information sources that need to be verified for their authenticity; the statement usually contains a specific statement, claim or opinion, and it is necessary to search and analyze relevant evidence to determine whether it is true, false or cannot be determined by existing information;
[0038] The relevant evidence in step 1 includes textual evidence and image evidence;
[0039] Textual evidence refers to written materials related to a claim to be verified. These materials can include news articles, research reports, official statements, blog posts, social media posts, etc. Textual evidence provides detailed information and background information to support or refute a claim, and is usually analyzed and relevant features are extracted through natural language processing techniques.
[0040] Visual evidence refers to pictures or images related to a claim to be verified. These images can include photographs, screenshots, charts, illustrations, etc., providing visual support or rebuttal information. Visual evidence is analyzed and its features extracted using computer vision technology to help verify the authenticity of the claim.
[0041] In step 1, the use of a large semantic similarity model to extract important information from the evidence text and eliminate redundant content from interfering with the retrieval involves using a large semantic similarity model to extract important information from the text evidence, making a summary summary, and using the summary as an index of relevant evidence. When performing similarity calculations, the summary content generated by the large semantic similarity model is used to eliminate redundant content, thereby reducing interference with the retrieval results.
[0042] Specifically, step 1 includes the following steps:
[0043] Step 101: Construct relevant indexes for the textual evidence to form a new index library;
[0044] During the index construction process, a large semantic similarity model is used to summarize the textual evidence and predict the evidence statements that may be related to the evidence text. The prompt prefix used is "which statements may the following evidence text be related to", and the final extracted index is built into an index library.
[0045] Step 102: Train a semantic similarity model and a multimodal fact-checking model for retrieving relevant textual evidence. The training is based on the pre-trained model SBERT used to calculate semantic similarity. The training adopts a contrastive learning method. For each statement, relevant evidence is selected as positive examples, and some evidence from the remaining corpus is selected as negative examples for contrastive learning. During the training process, the cosine similarity between statement features and evidence features is calculated to determine the degree of association between evidence and statement. InfoNceLoss is used as the loss function used in retrieval training.
[0046] Step 103: Train the multimodal fact-checking model CLIP for retrieving claim-related image evidence. The training method is similar to that of the semantic similarity model for retrieving claim-related text evidence.
[0047] Step 104: Use the trained semantic similarity model to extract features of the statement and the evidence index, calculate cosine similarity, sort according to similarity, and select the top 5 most relevant pieces of evidence as the retrieved text evidence.
[0048] Step 105: Use the trained multimodal fact-checking model to extract features of the claims and features of the image evidence, calculate cosine similarity, sort according to similarity, and select the top 5 most relevant images as retrieved image evidence.
[0049] Here, in step 1, when training using a semantic similarity model and the CLIP multimodal fact-verification model combined with a contrastive learning method, evidence relevant to the claim is selected as positive samples during training, while relevant evidence unrelated to the claim is randomly selected as negative samples for contrastive learning to participate in training. This improves the accuracy of retrieving relevant textual and image evidence from the evidence base.
[0050] Step 2: Constructing an evidence relationship graph: Introducing a discourse relationship enhancement module, using existing natural language processing tools (Graphene) as the discourse relationship enhancement module to extract corresponding discourse relationships from relevant evidence, constructing a discourse relationship heterogeneity graph based on the extracted discourse relationships, and fusing and extracting the features of the discourse relationship heterogeneity graph through a graph neural network to form discourse relationship heterogeneity graph evidence features;
[0051] Step 2 includes the following steps:
[0052] Step 201: Use the text relation extraction tool Graphene to analyze the discourse relations of text evidence. Graphene can convert complex sentences into simple independent sentences and identify the relations between sentences, thus completing text cleaning and discourse analysis.
[0053] The relationships extracted by Graphene include BACKGROUND, CAUSE, CONTRAST, etc. To simplify the representation of discourse relationship graphs, they are simplified into selection relationships, parallel relationships, and causal relationships, and their corresponding relationships are as follows: Selection relationship: DISJUNCTION; Parallel relationship: LIST, CONTRAST; Causal relationship: BACKGROUND, CAUSE, CONDITION, PURPOSE, CAUSE C, RESULT, RESULT C.
[0054] Step 202: Based on the extracted discourse relations, construct a directed heterogeneous graph as an evidence relation graph to form a discourse relation heterogeneous graph. The discourse relation heterogeneous graph includes declaration nodes, text evidence nodes, and image evidence nodes, as well as five types of edges: selection relation, parallel relation, causal relation, inverse causal relation, and ordinary connection relation. The specific construction of the discourse relation heterogeneous graph is completed using DGL. DGL (Deep Graph Library) is an open source library for graph neural networks (GNNs) designed to simplify the construction, training, and application of graph neural networks.
[0055] Step 203: Connect the statement and related image evidence to nodes of the heterogeneous discourse relation graph as node features. Use the attention mechanism of the graph neural network to control message transmission, alleviate the long-distance dependency problem, and enhance the ability of the multimodal fact-verification model to understand evidence and judge the authenticity of statements. Use the attention mechanism to perform weighted summation on the node features to complete the fusion and extraction of node features of the heterogeneous discourse relation graph, and obtain the final heterogeneous discourse relation graph evidence features.
[0056] In step 2, CLIP is used to encode the text evidence nodes and image evidence nodes in the constructed heterogeneous discourse relation graph to obtain the initial feature representations of the nodes. Unlike conventional graph neural networks, heterogeneous graph networks have multiple types of nodes and edges, so the impact of different types of nodes and edges needs to be considered during message passing; the nodes (i, j) associated through the meta-path φ are represented by α. i，j The attention weight represents the importance of node i to node j; the attention weight is calculated using the following formula.
[0057]
[0058] Among them, h i and h j Let α represent the feature representations of nodes i and j respectively. Φ || represents the attention vector, and || represents concatenation. Let α represent the set of neighboring nodes of node i under the metapath φ. ij h represents the attention weights. k The characteristics of the neighboring nodes of node i are represented.
[0059] Then, this study superimposes two layers of the discourse relation heterogeneous graph attention network and inputs its output node features into a bidirectional GRU module. Through its gating mechanism, it can effectively handle dependencies in long sequences, making up for the shortcomings of graph neural networks in capturing long-range dependencies, and obtaining the final node representation.
[0060] Finally, the node features are weighted and summed using an attention mechanism to obtain the final discourse relation heterogeneous graph features, as shown in the formula below:
[0061] α i =Softmax(LeakyReLU(H i ||C))
[0062] G=∑ i∈N α i H i
[0063] Where H i Let α represent the feature of the i-th node in the graph, C represent the declared feature, and α represent the feature of the i-th node. i is the weighting coefficient of the i-th node, N represents all nodes in the graph, and G represents the final discourse relation heterogeneous graph features.
[0064] The module for judging the authenticity of statements to be detected classifies whether a statement is true, false, or cannot be judged based on the extracted features of the statement and evidence. By integrating multimodal evidence and analyzing the discourse relation structure, the module ultimately improves the model's performance in judging the authenticity of statements.
[0065] Step 2 uses the natural language processing tool Graphene as an evidence discourse relation enhancement module to analyze relevant textual evidence and extract discourse relations. Based on the extracted discourse relations, a multimodal heterogeneous graph is constructed, and the extracted discourse relations are classified and summarized, ultimately simplified into four types of relations: selection, coordination, causation, and inverse causation. The resulting multimodal heterogeneous graph contains three types of nodes: declaration nodes, text evidence nodes, and image evidence nodes, as well as five types of edges: selection relations, coordination relations, causal relations, inverse causal relations, and ordinary connection relations. CLIP is used to extract features from the text nodes and image nodes in the constructed discourse relation heterogeneous graph to obtain the initial feature representations of the nodes.
[0066] In the training process of the multimodal fact-verification model, a heterogeneous graph attention network in graph neural networks is used to fuse and extract features of multimodal heterogeneous graphs. The heterogeneous graph attention network is stacked with two layers, and its output node features are input into a bidirectional GRU module. The gating mechanism is used to process the dependencies in long sequences, and the attention mechanism is used to perform weighted summation of node features to obtain the final discourse heterogeneous graph features, so as to enhance the multimodal fact-verification model's ability to understand evidence and judge the authenticity of claims.
[0067] Step 3, Authenticity Assessment: Using complete relevant evidence as the unit, the characteristics of the statement and the characteristics of the evidence are integrated to analyze whether the relevant evidence supports the statement to be assessed;
[0068] Here, in step 3, feature extraction and fusion methods are used to comprehensively utilize evidence from different modalities to improve the overall judgment ability of the model; the heterogeneous graph features are combined with the features of claims and evidence to enhance the model's understanding of evidence and its ability to judge the authenticity of claims.
[0069] In step 3, during the evidence fusion and analysis process, the statement features and evidence features are fused through an attention mechanism to improve the overall judgment ability of the model. The attention distribution of the statement and evidence is calculated through a cross-attention mechanism, with the statement features as the query and the evidence features as the key and value. Finally, the semantic understanding features are obtained by calculating the average features. The heterogeneous graph features are combined with the features of the statement and evidence to enhance the model's understanding of the evidence and its ability to judge the authenticity of the statement. The final classification results include three categories: true, false, and uncertain.
[0070] By analyzing the relevant evidence of the statement to be detected, the relevant evidence can be visualized and added as new features to the training of the multimodal fact-verification model. This completes the construction of the evidence discourse relationship graph and uses it to train the neural network, thereby improving the multimodal fact-verification model's ability to judge the authenticity of statements.
[0071] Specifically, step 3 includes the following steps:
[0072] Step 301: The retrieved text evidence is processed by NLTK into sentences. The processed text evidence and image evidence are then input into the multimodal fact verification model CLIP. The multimodal fact verification model CLIP is used to extract text evidence features, image evidence features, and the statement features to be detected. The features used here are the average features of the output of the last hidden layer of the multimodal fact verification model CLIP after removing the first [CLS].
[0073] Step 302: Calculate the attention distribution of statements and evidence through a cross-attention mechanism, where statement features are used as queries and evidence features are used as keys and values for feature fusion, and capture the interaction information and gaps by calculating the difference and product between the evidence and statement features.
[0074] Step 303: Input the fused features into the pooling layer and the linear layer to determine the authenticity of the claims. The categories include supported, refuted, and non-information-indeterminate (NEI), where non-information-indeterminate means that it cannot be determined. The loss function used is cross-entropy loss.
[0075] Through the above technical solutions, this invention realizes automatic fact-checking through multimodal evidence fusion and discourse relation enhancement, solves the lack of multimodal automatic fact-checking, improves the accuracy and reliability of multimodal fact-checking models in judging the authenticity of claims, can effectively judge the claims to be detected, effectively addresses the problem of the spread of false information on social media, and is of great significance in reducing the spread of false information.
[0076] The above description is merely a preferred embodiment of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention. Structures, devices, and operating methods not specifically described or explained in this invention are implemented according to conventional methods in the art unless otherwise specified or limited.
Claims
1. A fact-checking method based on multimodal evidence fusion and discourse relation enhancement, characterized in that: Includes the following steps: Step 1, Evidence Retrieval: For the statement to be detected, retrieve relevant evidence from the evidence database; Step 2: Constructing an evidence relationship graph: Using natural language processing tools, extract discourse relationships from relevant evidence. Based on the extracted discourse relationships, construct a discourse relationship heterogeneity graph. Then, use a graph neural network to fuse and extract the features of the discourse relationship heterogeneity graph to form the discourse relationship heterogeneity graph evidence features. Step 3, Authenticity Assessment: Using complete relevant evidence as the unit, the characteristics of the statement and the characteristics of the evidence are integrated to analyze whether the relevant evidence supports the statement to be assessed; Step 2 includes the following steps: Step 201: Analyze the discourse relations of textual evidence using text relation extraction tools; Step 202: Based on the extracted discourse relations, construct a directed heterogeneous graph as an evidence relation graph to form a discourse relation heterogeneous graph; Step 203: Connect the statement and related image evidence to the nodes of the heterogeneous discourse relation graph as node features, use the attention mechanism of the graph neural network to control message transmission, and use the attention mechanism to perform weighted summation of the node features to complete the fusion and extraction of the node features of the heterogeneous discourse relation graph, and obtain the final heterogeneous discourse relation graph evidence features. The discourse relation heterogeneous graph includes declaration nodes, text evidence nodes, and image evidence nodes, as well as five types of edges: selection relation, parallel relation, causal relation, inverse causal relation, and ordinary connection relation.
2. The fact-checking method based on multimodal evidence fusion and discourse relation enhancement according to claim 1, characterized in that: The relevant evidence in step 1 includes textual evidence and image evidence.
3. The fact-checking method based on multimodal evidence fusion and discourse relation enhancement according to claim 1, characterized in that: Step 2 includes the following steps: Step 101: Construct relevant indexes for the textual evidence to form a new index library; Step 102: Train a semantic similarity model and a multimodal fact-verification model for retrieving relevant textual evidence; Step 103: Train a multimodal fact-verification model for retrieving claim-related image evidence; Step 104: Use the trained semantic similarity model to extract features of the statement and the evidence index, calculate cosine similarity, sort according to similarity, and select the top 5 most relevant pieces of evidence as the retrieved text evidence. Step 105: Use the trained multimodal fact-checking model to extract features of the claims and features of the image evidence, calculate cosine similarity, sort according to similarity, and select the top 5 most relevant images as retrieved image evidence.
4. The fact-checking method based on multimodal evidence fusion and discourse relation enhancement according to claim 3, characterized in that: The semantic similarity model and multimodal fact-verification model in step 1 are trained using a contrastive learning method. During the training process, evidence related to the statement is selected as positive samples, and relevant evidence unrelated to the statement is randomly selected as negative samples for contrastive learning to participate in the training.
5. The fact-checking method based on multimodal evidence fusion and discourse relation enhancement according to claim 3, characterized in that: The text relationship extraction tool is Graphene.
6. The fact-checking method based on multimodal evidence fusion and discourse relation enhancement according to claim 3, characterized in that: Step 3 includes the following steps: Step 301: Segment the text evidence, input the segmented text evidence and image evidence together into the multimodal fact verification model, and use the multimodal fact verification model to extract text evidence features, image evidence features and the features of the statement to be detected; Step 302: Calculate the attention distribution of statements and evidence through a cross-attention mechanism, where statement features are used as the query and evidence features are used as the key and value for feature fusion; Step 303: Input the fused features into the pooling layer and the linear layer to determine the authenticity of the claims, which are categorized as true, false, and insufficient information.