Multi-modal fact-checking method and system based on fine-grained multi-source knowledge enhancement

Abstract

The application discloses a multi-modal fact verification method and system based on fine-grained multi-source knowledge enhancement, and belongs to the technical field of information detection. The method comprises the following steps: acquiring tweet information and evidence information; extracting fine-grained knowledge information from the tweet information and the evidence information; extracting text information semantic hidden representations and picture information semantic hidden representations of the tweet information, the evidence information and the knowledge information; constructing a heterogeneous graph containing the tweet information, the evidence information and the knowledge information based on the text information semantic hidden representations and the picture information semantic hidden representations; calculating new node representations of the heterogeneous graph based on a graph attention network; and classifying the new node representations of the heterogeneous graph to obtain a multi-modal fact verification result of the tweet information. The application can more accurately predict the authenticity of a tweet.

Description

Technical Field

[0001] This invention relates to the field of information detection technology, specifically to a multimodal fact-checking method and system based on fine-grained multi-source knowledge enhancement. Background Technology

[0002] With the widespread dissemination of misinformation and fake news, fact-checking has become a major research focus. It aims to determine the veracity of news, tweets, and other content through relevant evidence. In recent years, the rapid development of multimedia information and platforms has made the spread of multimodal misinformation more convenient and efficient, making multimodal fact-checking a new research hotspot in the field.

[0003] Recent multimodal fact-checking methods utilize text and image information to extract modality-specific feature representations, and then use different fusion methods, such as concatenation and attention networks, to fuse these modality-specific feature representations to obtain multimodal information feature representations. These representations are then used by a classifier to predict factual labels for tweets or news content.

[0004] However, these methods only consider coarse-grained global information, thus ignoring fine-grained knowledge information that contains more complex and important semantic information, which to some extent affects the performance of multimodal fact-checking methods. Furthermore, these methods only consider evidence from a single source as the prediction standard, without comprehensively considering multi-source knowledge information, which can easily lead to bias or the use of false information as evidence to predict tweets or news. Summary of the Invention

[0005] To address the aforementioned issues, this invention discloses a multimodal fact-checking method and system based on fine-grained multi-source knowledge enhancement. This method utilizes fine-grained, multi-source knowledge information to assist fact-checking tasks involving multimodal information. Specifically, during the data processing stage, relevant fine-grained knowledge information is acquired from multiple knowledge sources, and this information is then integrated to more accurately predict the authenticity of tweets. This fine-grained, multi-source knowledge information increases the reliability of evidence to a certain extent, while also supplementing missing semantic information in coarse-grained information, thus improving model performance.

[0006] To achieve the above objectives, the technical solution provided by the present invention is as follows:

[0007] A multimodal fact-checking method based on fine-grained multi-source knowledge enhancement, the method comprising:

[0008] Obtain tweet information and evidence information; wherein, the tweet information includes: tweet text information and tweet image information, and the evidence information includes: evidence text information and evidence image information;

[0009] Extracting fine-grained knowledge information from tweets and evidence;

[0010] Extract semantic latent representations of textual and image information from tweets, evidence, and knowledge.

[0011] Based on semantic implicit representation of text information and semantic implicit representation of image information, a heterogeneous graph containing the tweet information, the evidence information, and the knowledge information is constructed.

[0012] Calculate a new node representation of the heterogeneous graph based on a graph attention network;

[0013] The new node representations of the heterogeneous graph are classified to obtain the multimodal fact-checking results of the tweet information.

[0014] Furthermore, the knowledge information includes: text entity knowledge, image entity knowledge, and key text content.

[0015] Furthermore, the semantic latent representation of textual information and semantic latent representation of image information for extracting tweet information, evidence information, and knowledge information includes:

[0016] Based on a pre-trained language model, feature encoding is performed on text data to obtain the semantic implicit representation of text information; wherein, the text data includes: tweet text information, evidence text information, text entity knowledge, and key text content;

[0017] Feature encoding is performed on image data based on a pre-trained visual model to obtain the semantic latent representation of image information; wherein, the image data includes: tweet image information, evidence image information, and image entity knowledge.

[0018] Furthermore, the heterogeneous graph G = (V, E) includes the node set V, which comprises nodes corresponding to the semantic latent representation of text information and nodes corresponding to the semantic latent representation of image information, and the edge set E comprises an adjacency matrix and an edge feature matrix.

[0019] Furthermore, computing a new node representation of the heterogeneous graph based on a graph attention network includes:

[0020] The nodes in the heterogeneous graph are initialized based on semantic latent representations of textual and image information to obtain node representations. Where i∈(1,||V||);

[0021] Node representation based on the output of layer l-1 of the graph attention network Obtain the weights of the l-th layer Where, j∈(1,||V||);

[0022] For node i, combined with the weights Node representation of neighbor node j Perform aggregation and base the results on the node representation of node i. Obtain the updated node representation of node i

[0023] Node representation based on the output of the last layer of a graph attention network Obtain the new node representation of the heterogeneous graph.

[0024] Furthermore, the classification of the new node representations of the heterogeneous graph to obtain the multimodal fact-checking results of the tweet information includes:

[0025] The new node representation of the heterogeneous graph is subjected to average pooling.

[0026] The average pooling results are categorized to obtain the multimodal fact-checking results of the tweet information.

[0027] A multimodal fact-checking system based on fine-grained multi-source knowledge enhancement, the system comprising:

[0028] The data acquisition and processing module is used to acquire tweet information and evidence information; wherein, the tweet information includes: tweet text information and tweet image information, and the evidence information includes: evidence text information and evidence image information;

[0029] The knowledge extraction module is used to extract fine-grained knowledge information from tweet information and evidence information;

[0030] The feature encoding module is used to extract semantic latent representations of textual and image information from tweets, evidence, and knowledge.

[0031] The heterogeneous graph construction module is used to construct a heterogeneous graph containing the tweet information, the evidence information, and the knowledge information based on the semantic implicit representation of text information and the semantic implicit representation of image information.

[0032] The knowledge-guided graph fusion module is used to compute new node representations of the heterogeneous graph based on a graph attention network.

[0033] The multimodal fact-checking and classification module is used to classify the new node representations of the heterogeneous graph to obtain the multimodal fact-checking results of the tweet information.

[0034] Furthermore, the feature encoding module, heterogeneous graph construction module, knowledge-guided graph fusion module, and multimodal fact-checking and classification module are jointly trained based on cross-entropy loss.

[0035] An electronic device, characterized in that the electronic device comprises: a processor and a memory storing computer program instructions; the processor, when executing the computer program instructions, implements the multimodal fact-checking method based on fine-grained multi-source knowledge enhancement as described above.

[0036] A computer-readable storage medium, characterized in that the computer-readable storage medium stores computer program instructions, which, when executed by a processor, implement the multimodal fact-checking method based on fine-grained multi-source knowledge enhancement as described above.

[0037] Compared with the prior art, the technical solution of the present invention has the following advantages:

[0038] In terms of practical fact-checking systems, this invention, compared to general multimodal fact-checking systems, places greater emphasis on the integration and interaction of fine-grained, multi-source knowledge information with the original tweet and supporting evidence. This avoids neglecting fine-grained semantic information and thus impacting system performance by focusing only on coarse-grained information. Compared to traditional multimodal fact-checking systems, this invention focuses on comprehensively predicting and judging the authenticity of tweets by combining multi-source knowledge and supporting evidence, which is more in line with practical multimodal fact-checking scenarios. It also effectively avoids the negative impact of inaccurate evidence from a single source on the system.

[0039] At the technical level, the multimodal fact-checking method based on fine-grained multi-source knowledge enhancement designed in this invention introduces image entity and text entity information. Simultaneously, it utilizes a large language model with strong generative capabilities to extract key text information, further acquiring multi-source, fine-grained knowledge. Furthermore, it employs a knowledge-oriented graph fusion method to fully integrate tweets, evidence information, and knowledge information, resulting in a latent representation containing richer semantics. Experiments on public datasets show that, compared to other existing multimodal fact-checking methods, this invention achieves an absolute performance improvement of 2.64% and 1.69% on the FACTIFY and MOCHEG datasets, respectively. Attached Figure Description

[0040] Figure 1 This is a flowchart of a multimodal fact-checking method based on fine-grained multi-source knowledge enhancement, according to an embodiment of the present invention.

[0041] Figure 2 This is a block diagram of a multimodal fact-checking system based on fine-grained multi-source knowledge enhancement, according to an embodiment of the present invention. Detailed Implementation

[0042] The present invention will be described in detail below with reference to the accompanying drawings and embodiments. It should be noted that the described embodiments are only intended to facilitate the understanding of the present invention and do not constitute any limitation thereof.

[0043] Figure 1 This is a flowchart of a multimodal fact-checking method based on fine-grained multi-source knowledge enhancement provided by an embodiment of the present invention, which specifically includes the following steps:

[0044] Step 1: Obtain tweets and evidence containing multimodal information from social media platforms and Wikipedia.

[0045] This invention obtains tweets and evidence containing multimodal information from social media platforms and Wikipedia. The data is divided into training data and test data. For each sample C in the training and test data, it includes the tweet text information C. T Tweet image information C I Evidence text information E T Evidence image information E I .

[0046] Step 2: Based on the obtained tweets and evidence information, extract fine-grained knowledge information from multiple knowledge acquisition sources (such as text entities, image entities, and large models).

[0047] Based on the tweet information and evidence information, a knowledge extraction module is constructed. It uses text entity extraction methods (such as TAGME), image entity extraction methods (such as Faster R-CNN), and general large language models (such as ChatGPT) to extract fine-grained knowledge information from the tweets and evidence information, specifically including text entity knowledge, image entity knowledge, and key text content.

[0048] Step 3: Extract the semantic latent representation of the text information and the semantic latent representation of the image information for the tweet, evidence information and knowledge information respectively.

[0049] Based on tweet information, evidence information, and knowledge information, a feature encoding module is constructed. This encoding module is then used to process the content and dissemination data to obtain a latent representation.

[0050] The specific processing steps include:

[0051] Step 3.1: If the data is text (tweet text information, evidence text information, text entity knowledge, text key content), input sample text x, and extract the semantic latent representation h of the text information through a pre-trained language model. Define Θ1 as the trainable network parameters. Specifically:

[0052] h = f text ({[CLS],w1,…,w n};Θ1)

[0053] Where [CLS] represents a special character inserted at the beginning of a sentence, w iThis represents the i-th word of the input text sequence, where n is the text length and f is the input text length. text This represents any pre-trained language model, such as the DeBERTa model.

[0054] Step 3.2: If the information is image-based (e.g., tweet image, evidence image, image entity knowledge), input a sample image v. Using a pre-trained visual model (e.g., ViT, Swin Transformer), extract the semantic latent representation o of the image information. Define Θ2 as a trainable network parameter. Details are as follows:

[0055] o = f image (v;Θ2)

[0056] Step 4: Based on the obtained semantic latent representations of text information and image information, use the heterogeneous graph construction method to construct the tweets, evidence information and knowledge information into graph structure information.

[0057] Based on the semantic latent representation h of the text information and the semantic latent representation o of the image information obtained in step 3, a heterogeneous graph construction method is introduced to reconstruct the data into graph structure data G = (V, E). The details are as follows:

[0058]

[0059] in, It is an adjacency matrix. Let be the edge feature matrix, and ||V|| represents the total number of nodes, and cos(*) represents the cosine similarity.

[0060] Step 5: Based on the obtained graph structure information, design a knowledge-oriented graph fusion method to fuse the multimodal information contained in tweets, evidence information and knowledge information to obtain the multimodal information latent representation.

[0061] Based on the graph structure data G and node representation V obtained in step 5, a knowledge-oriented graph fusion method is designed to update the representation of each node in the graph structure data and finally obtain a new node representation. Specifically as follows:

[0062]

[0063] in, The feature representation of node i in layer l. The feature representation of node i at level l-1, Let Λ be the set of neighboring nodes of node i. The parameters are learnable, and LeakyReLU(*) represents the activation function.

[0064] Step 6: Based on the obtained multimodal information latent representation, use a multimodal fact-checking classifier to predict the authenticity label of the tweet.

[0065] Based on the new node representation obtained in step 5 Predict factual tags for tweets using a multimodal fact-checking classifier. Right now:

[0066]

[0067] Where f(*) represents the classifier, meanpool(*) represents the average pooling operation, and θ cls This represents the classifier parameters.

[0068] Step 7: Model training.

[0069] During the training phase, this invention calculates the classification loss using cross-entropy loss, as follows:

[0070]

[0071] Where y represents the distribution of real fake news tags.

[0072] like Figure 2 As shown, embodiments of the present invention also provide a multimodal fact-checking system, specifically including:

[0073] The data acquisition and processing module is used to acquire tweet information and evidence information; wherein, the tweet information includes: tweet text information and tweet image information, and the evidence information includes: evidence text information and evidence image information;

[0074] The knowledge extraction module is used to extract fine-grained knowledge information from tweet information and evidence information;

[0075] The feature encoding module is used to extract semantic latent representations of textual and image information from tweets, evidence, and knowledge.

[0076] The heterogeneous graph construction module is used to construct a heterogeneous graph containing the tweet information, the evidence information, and the knowledge information based on the semantic implicit representation of text information and the semantic implicit representation of image information.

[0077] The knowledge-guided graph fusion module is used to compute new node representations of the heterogeneous graph based on a graph attention network.

[0078] The multimodal fact-checking and classification module is used to classify the new node representations of the heterogeneous graph to obtain the multimodal fact-checking results of the tweet information.

[0079] Experimental test:

[0080] Experiments were conducted using two publicly available datasets, FACTIFY and MOCHEG. The results show that traditional multimodal fact-checking methods perform relatively poorly on both datasets. In contrast, the method proposed in this invention integrates fine-grained, multi-source knowledge information, utilizes graph-structured data to enhance the interaction and fusion between different modalities, granularities, and knowledge types, and pays attention to both coarse-grained and fine-grained knowledge information. This results in better performance on the datasets; for example, the proposed method achieves an absolute improvement of 2.64% and 1.69% in detection performance on the FACTIFY and MOCHEG datasets, respectively.

[0081] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A multimodal fact-checking method based on fine-grained multi-source knowledge enhancement, characterized in that, The method includes: Obtain tweet information and evidence information; wherein, the tweet information includes: tweet text information and tweet image information, and the evidence information includes: evidence text information and evidence image information; Fine-grained knowledge information is extracted from tweet information and evidence information; wherein, the knowledge information includes: text entity knowledge, image entity knowledge and key text content; the key text content is obtained by extracting key information from tweet information and evidence information through a large language model; Extract semantic latent representations of textual and image information from tweets, evidence, and knowledge. Based on semantic latent representations of textual and image information, a heterogeneous graph is constructed that includes the tweet information, the evidence information, and the knowledge information; wherein, the heterogeneous graph Node set This includes nodes corresponding to the semantic implicit representation of text information and nodes corresponding to the semantic implicit representation of image information, as well as the edge set. Including the adjacency matrix and the edge feature matrix; Calculate a new node representation of the heterogeneous graph based on a graph attention network; The new node representations of the heterogeneous graph are classified to obtain the multimodal fact-checking results of the tweet information.

2. The method according to claim 1, characterized in that, The semantic latent representations of textual and image information extracted from tweet information, evidence information, and knowledge information include: Based on a pre-trained language model, feature encoding is performed on text data to obtain the semantic implicit representation of text information; wherein, the text data includes: tweet text information, evidence text information, text entity knowledge, and key text content; Feature encoding is performed on image data based on a pre-trained visual model to obtain the semantic latent representation of image information; wherein, the image data includes: tweet image information, evidence image information, and image entity knowledge.

3. The method according to claim 1, characterized in that, The computation of new node representations of the heterogeneous graph based on a graph attention network includes: The nodes in the heterogeneous graph are initialized based on semantic latent representations of textual and image information to obtain node representations. ;in, ; Based on graph attention network Layer output node representation , obtained the Layer weights ;in, ; For nodes Combined with the weights for neighbor nodes Node representation Perform aggregation, and based on the aggregation result and nodes Node representation , obtain node Updated node representation ; Node representation based on the output of the last layer of a graph attention network This yields a new node representation of the heterogeneous graph. .

4. The method according to claim 1, characterized in that, The classification of the new node representations of the heterogeneous graph to obtain the multimodal fact-checking results of the tweet information includes: The new node representation of the heterogeneous graph is subjected to average pooling. The average pooling results are categorized to obtain the multimodal fact-checking results of the tweet information.

5. A multimodal fact-checking system based on fine-grained multi-source knowledge enhancement, characterized in that, The system includes: The data acquisition and processing module is used to acquire tweet information and evidence information; wherein, the tweet information includes: tweet text information and tweet image information, and the evidence information includes: evidence text information and evidence image information; The knowledge extraction module is used to extract fine-grained knowledge information from tweet information and evidence information; wherein, the knowledge information includes: text entity knowledge, image entity knowledge and key text content; the key text content is obtained by extracting key information from tweet information and evidence information through a large language model; The feature encoding module is used to extract semantic latent representations of textual and image information from tweets, evidence, and knowledge. A heterogeneous graph construction module is used to construct a heterogeneous graph containing the tweet information, the evidence information, and the knowledge information based on the semantic latent representation of text information and the semantic latent representation of image information; wherein, the heterogeneous graph Node set This includes nodes corresponding to the semantic implicit representation of text information and nodes corresponding to the semantic implicit representation of image information, as well as the edge set. Including the adjacency matrix and the edge feature matrix; The knowledge-guided graph fusion module is used to compute new node representations of the heterogeneous graph based on a graph attention network. The multimodal fact-checking and classification module is used to classify the new node representations of the heterogeneous graph to obtain the multimodal fact-checking results of the tweet information.

6. The system according to claim 5, characterized in that, The feature encoding module, heterogeneous graph construction module, knowledge-guided graph fusion module, and multimodal fact-checking and classification module are jointly trained based on cross-entropy loss.

7. An electronic device, characterized in that, The electronic device includes: a processor and a memory storing computer program instructions; when the processor executes the computer program instructions, it implements the multimodal fact-checking method based on fine-grained multi-source knowledge enhancement as described in any one of claims 1-4.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program instructions, which, when executed by a processor, implement the multimodal fact-checking method based on fine-grained multi-source knowledge enhancement as described in any one of claims 1-4.

Images