Multimodal fact-checking method for enhancing large language model based on retrieval and reasoning

Abstract

The application discloses a kind of multi-modal fact checking methods of large language model based on retrieval and reasoning enhancement, it is to the sample to be checked containing declaration image and declaration text, first utilize semantic enhancement retrieval module in the uniformity of news domain alignment graph-text feature space, retrieve the evidence image consistent with declaration semantics from candidate evidence;Again, utilize knowledge enhancement retrieval module to carry out entity extraction, knowledge graph / retrieval and refine internet evidence text, obtain background knowledge and high-quality text evidence;Finally, declaration and evidence are jointly input into multi-modal large language model, through the two-stage self-refining reinforcement fine-tuning of cold start, group relative strategy optimization, so that model output contains the authenticity determination result of reasoning process.The application can improve the accuracy and consistency of fact checking based on the joint optimization of multi-modal evidence retrieval and reasoning, so as to provide more reliable automated multi-modal authenticity determination result for news, social media and other scenes.

Description

Technical Field

[0001] This invention belongs to the field of multimodal information processing and fact-checking technology, and particularly relates to a multimodal fact-checking method that combines retrieval enhancement and reasoning enhancement. Background Technology

[0002] With the development of social media and mobile internet, multimodal content containing images and text has become an important vehicle for the spread of rumors. Single-modal verification methods that rely solely on text or images cannot fully utilize complementary multimodal clues, resulting in insufficient robustness in cross-domain, cross-language, or "inconsistent image and text" scenarios.

[0003] Existing multimodal fact-checking methods typically face the following problems: First, evidence retrieval often relies on general image-text similarity or rule templates, making it difficult to perform semantic alignment for the news domain; second, evidence texts often contain a large amount of redundancy and noise, affecting downstream reasoning; third, although large language models have strong reasoning capabilities, they are prone to illusions or unstable reasoning formats if they lack high-quality evidence and training constraints. Summary of the Invention

[0004] This invention aims to solve the above problems by proposing a multimodal fact-checking method based on a large language model enhanced by retrieval and reasoning. It seeks to improve the accuracy of evidence matching, the stability of reasoning, and the interpretability of conclusions through the collaborative design of semantically enhanced retrieval, knowledge-enhanced retrieval, and two-stage self-refinement reinforcement training.

[0005] To achieve the above objectives, the present invention adopts the following technical solution: The present invention provides a multimodal fact-checking method based on a retrieval and reasoning-enhanced large language model, characterized by the following steps: Step 1: Obtain the first The pre-processed statements pending verification for each news event include: Image of a statement about a news event With the Statement text for a news event and the corresponding verification conclusions ;when When, it indicates that the veracity of the statement to be verified is true. When this occurs, it indicates that the veracity of the statement to be verified is determined to be false; Get the The preprocessed candidate evidence set corresponding to each news event includes: A collection of candidate evidence images for a news event and its corresponding candidate evidence text set ;in, Indicates the first The first news event One candidate evidence image, express The corresponding candidate evidence text, , The number of candidate pieces of evidence; Will The Middle The best-matching evidence image for a news event is denoted as ,Will middle The corresponding best-matching evidence text is denoted as ; Step 2: Construct a semantically enhanced retrieval module, including: a pre-trained image encoder, a pre-trained text encoder, an image domain adapter, and a text domain adapter, and then apply them to... , as well as , Encoding and domain adaptation are performed to obtain the corresponding first... The image representation vector of a news event's statement. , Declare text representation vector and the The image representation vector of each candidate evidence , No. Candidate evidence text representation vectors Thus constructing the first A positive sample set of news events and the A positive sample set of news events ; Step 3, based on and The loss for constructing the semantic enhancement retrieval module includes: the fact NCE loss in the image-to-text direction. And the fact of NCE loss in the text-to-image direction This is used to train the semantic enhancement retrieval module, obtain the trained semantic enhancement retrieval module, and output the optimal image to the text set. and the optimal text-to-image set ; Step 4: Use the trained semantic enhancement retrieval module to perform retrieval on the first... Image of the statement corresponding to each news event With the declaration text and images of various candidate pieces of evidence related to this news event. and its corresponding candidate evidence text Encoding and domain adaptation are performed to obtain the first... The optimal representation vector of a news event's statement image. , Declare the optimal representation vector of the text , No. Optimal representation vector of candidate evidence images and the The optimal representation vector of candidate evidence texts And use equation (3) to calculate the first... The pending verification statement for the first news event and its first Semantic consistency score among candidate evidence Thus, a semantic consistency score set is obtained. ; and on After sorting in descending order, select the first few... The candidate evidence image is used as the first A collection of semantically enhanced evidence images for a news event; This indicates the preset number of evidence images. Indicates the number of candidate pieces of evidence; (3) In equation (3), Indicates the first The first news event The optimal representation vector for each candidate evidence image; express The corresponding optimal representation vector of the candidate evidence text; Step 5: Construct a knowledge-enhanced retrieval module, including a search agent and a refinement agent, which are used to obtain the first... Knowledge text of a news event and the A collection of evidence texts after preprocessing of a news event ; Step Six: , , , and Construct the first according to the preset template Tips for a news event The data is then input into a multimodal large language model for processing, thereby outputting the first... Structured reasoning results and predictive verification conclusions for individual news events ,when When, it indicates that the prediction of the first The veracity of a news event is determined to be true, when... When, it indicates that the prediction of the first The authenticity of the news event was determined to be false; Step 7: Construct cross-entropy loss using equation (5) It is used to perform cold start training on a multimodal large language model to obtain a multimodal large language model after cold start training; (5) In equation (5), Represents the probability value; Step 8: Use the group relative strategy optimization method to perform reinforcement learning on the multimodal large language model after cold start training to obtain the multimodal large language model after reinforcement learning; Step 9: The final multimodal fact-checking model is composed of the trained semantic enhancement retrieval module, the knowledge enhancement retrieval module, and the multimodal large language model after reinforcement learning. It is used to perform evidence retrieval and fact reasoning on the input multimodal claims and output the optimal structured reasoning result and its optimal predicted verification conclusion.

[0006] The multimodal fact-checking method based on retrieval and reasoning-enhanced large language models described in this invention is characterized in that step two includes the following steps: Step 2.1: The pre-trained image encoder respectively... and The initial declared image features are obtained through processing. With the Initial candidate evidence image features ; Step 2.2: The pre-trained text encoder is respectively... and The initial declaration text features are obtained through processing. With the Initial candidate text features ; Step 2.3, the image domain adapter... and Perform domain adaptation mapping to obtain the corresponding domain... The image representation vector of a news event's statement. and the The image representation vector of each candidate evidence ,make The corresponding best-matching image representation vector is denoted as ; Step 2.4, the text field adapter... and Perform domain adaptation mapping to obtain the corresponding declaration text representation vector. and the Candidate evidence text representation vectors ,make The corresponding best-matching evidence text representation vector is denoted as ; Step 2.5, from and Composition of declaration image—declaration text pair ; Depend on and Composition of declaration text — declaration image pair ; Depend on and Composition of Evidence Images—Evidence Text Pairs ; Depend on and Composition of Evidence Text-Evidence Image Pair ; Depend on and Composition of candidate image-evidence text pairs ; Depend on and Composition of Evidence Text-Candidate Image Pairs ; Depend on and Composition of Evidence Images—Candidate Text Pairs ; Depend on and Composition of candidate text-evidence image pairs ; Declaring image—declaring text pair Evidence Image - Evidence Text Pair Candidate image-evidence text pair Evidence Image-Candidate Text Pair Constituting the first A positive sample set of images to text from a news event. ; From declaration text—declaration image pair Evidence text-evidence image pair Evidence text—candidate image pair Candidate text-evidence image pair Constituting the first A collection of positive text-to-image samples for a news event. .

[0007] Furthermore, step three includes the following steps: Step 3.1: Construct the factual NCE loss from image to text using equation (1). : (1) In equation (1), The total number of news events; For the first The image representation vector of each news event, and ∈ ; For the first The text representation vector of a news event, and ∈ ; Indicates the first The first news event One text candidate representation vector, Represents the dot product of vectors; Temperature coefficient; Step 3.2: Construct the factual NCE loss for the text-to-image direction using equation (2). : (2) In equation (2), Indicates the first The first news event Image candidate representation vectors.

[0008] Furthermore, step five includes the following steps: Step 5.1, Search for proxy pairs and Entity extraction is performed to obtain the first... A collection of entities for a news event and to After entity linking, in the knowledge graph With knowledge base Search The relationships, attributes, and descriptions of each entity in the code are used to generate the first... Knowledge text of a news event ; Step 5.2, Refine the agent to obtain the first A collection of evidence texts related to a news event And perform segmentation to obtain the first segment. A collection of text fragments from a news event Therefore, based on equation (4), Redundancy is filtered out to obtain the first... A collection of evidence texts after preprocessing of a news event ,in, For the first The first news event A text fragment, Number of text fragments: (4) In equation (4), Indicates the first The first news event A text fragment; Indicates to Encoded semantic vector; Indicates to Encoded semantic vector; Represents the cosine similarity function; This is the redundancy filtering threshold.

[0009] Furthermore, step eight includes the following steps: Step 8.1: Construct an accuracy reward system ,when When, it indicates that the authenticity judgment result output by the multimodal large language model is consistent with the authenticity verification conclusion. This indicates that the authenticity judgment result output by the multimodal large language model differs from the actual verification conclusion; Step 8.2, Build Format Rewards ,when When, it indicates that the structured reasoning results output by the multimodal large language model after cold start training include preset reasoning fields and conclusion fields. This indicates that the structured reasoning results output by the multimodal large language model after cold start training do not include the preset reasoning fields and conclusion fields; Step 8.3: Construct the total reward ; Step 8.4: Set the prompt words The input is repeatedly processed in the multimodal large language model after cold start training to obtain... Candidate structured reasoning results ,in, Indicates the first The candidate structured reasoning results; Step 8.5, calculate the first... Candidate structured reasoning results Total reward and its advantage value ; Step 8.6: Using the multimodal large language model trained after cold start as the policy model, construct the group relative policy optimization objective using equation (6). This is used to train the policy model, resulting in a multimodal large language model after reinforcement learning. (6) In equation (6), This represents the policy model during the current training round. This represents the policy model from the previous training round. Representational strategy model, For the clipping function, This is the cutting factor; for divergence regularization term, for The weighting coefficients of the divergence regularization term.

[0010] The present invention provides an electronic device, including a memory and a processor, characterized in that the memory is used to store a program supporting the processor in performing the method described therein, and the processor is configured to execute the program stored in the memory.

[0011] The present invention discloses a computer-readable storage medium storing a computer program, characterized in that the computer program is executed by a processor to perform the steps of the method described thereon.

[0012] Compared with the prior art, the beneficial effects of the present invention include: 1. To address the problem that existing multimodal fact-checking methods often directly use general image-text similarity or rely solely on text evidence, making it difficult to achieve fine-grained image semantic alignment in the news domain, thus resulting in low recall rate and high false matching rate of evidence images, this invention introduces a pre-trained image encoder, a pre-trained text encoder, and image domain adapters and text domain adapters in steps two and three. Based on bidirectional Fact NCE contrastive learning, it simultaneously constrains the consistency between image-text and text-image, enabling the statement and candidate evidence to be aligned and measured in a unified feature space. This significantly improves the retrieval accuracy and stability of evidence images and reduces erroneous verification conclusions caused by image semantic drift.

[0013] 2. To address the problem in existing technologies where "declaration images - evidence images" often lack explicit training constraints and the model tends to learn only coarse-grained thematic similarities while ignoring detailed differences in the same event (such as location, people, time, landmarks, etc.), this invention further constructs cross-image-text positive sample pairs (including declaration images and evidence text, and evidence images and declaration text) in the semantic enhancement retrieval module. By using a loss function to map multi-source descriptions of the same event to neighboring regions, the model's ability to consistently model event-level semantics is enhanced. This ensures that the retrieved evidence images are not only visually similar but also falsifiable / corroborative in terms of news semantics, thereby increasing the proportion of usable evidence in subsequent reasoning stages.

[0014] 3. To address the problem that existing multimodal fact-checking methods often lack background knowledge when relying solely on webpage fragments or title text, leading to incomplete reasoning chains or even unfounded conjectures, this invention, in step five, maps key entities in the statement image to a knowledge graph and structured knowledge base through entity extraction and entity linking. A search agent then obtains high-order semantic relationships related to the entities and refined background sentences. These background knowledge texts and evidence texts are then input into a multimodal large language model for reasoning. This technique can complete the context at the entity level, improving the ability to determine implicit facts, ambiguous place / organization names, and cross-language aliases, thereby fundamentally reducing the probability of "illusory reasoning due to missing evidence."

[0015] 4. To address the problem that existing retrieval enhancement methods often contain a large number of repetitive, noisy, or contradictory fragments in multi-source webpage evidence, and that direct splicing can cause prompt word inflation, interfere with model attention, and increase inference uncertainty, this invention sets up a refinement agent in step five to segment and vectorize the retrieved evidence text, remove redundancy based on similarity, and reorganize it to obtain a final evidence text with higher information density and lower redundancy. This step ensures the preservation of key evidence while reducing interference from irrelevant fragments, reducing the length of inference input and computational overhead, and improving the consistency and interpretability of the conclusions.

[0016] 5. To address the issues that existing multimodal large language models often rely on template-based supervised fine-tuning for fact-checking tasks, making it difficult to form stable deep reasoning strategies and constrain output formats, this invention constructs a three-stage self-optimizing training process in steps seven to nine: first, cold-start supervised training establishes the basic capability of "template-based output + evidence-based reasoning"; then, rejection sampling reinforces the correct reasoning trajectory; finally, group-based relative strategy optimization, driven by rule-based reward signals (accuracy reward and format reward), continuously explores and converges to a better reasoning strategy. Through the above training mechanism, this invention can improve reasoning depth and robustness while ensuring the parsability of structured output, thereby improving the overall accuracy and practicality of multimodal fact-checking. Attached Figure Description

[0017] Figure 1 A comparison of FACTCOMPASS multimodal fact-checking paradigms and a schematic diagram of the framework of this invention; Figure 2 This is a schematic diagram of the overall architecture of the FACTCOMPASS invention. Detailed Implementation

[0018] In this embodiment, taking a news scenario as an example, the overall process of a multimodal fact-checking method based on a retrieval and reasoning-enhanced large language model is as follows: Figure 1 As shown, it includes the following steps: Step 1: Obtain the first The verification statements corresponding to each news event are preprocessed. The preprocessing may include: uniformly scaling and normalizing the pixel size of the statement image; denoising and cleaning the statement text, segmenting it into words, and truncating it to a preset maximum length to ensure consistent input encoding, thus obtaining the preprocessed verification statements, including: the first... Image of a statement about a news event With the Statement text for a news event and the corresponding verification conclusions The verifiable results can be obtained from manual verification or authoritative fact-checking platforms, and serve as a monitoring signal during the training phase.

[0019] when When, it indicates that the veracity of the statement to be verified is true. When the result is true, it indicates that the veracity of the statement to be verified is false. This embodiment uses a binary classification of true and false as an example, but it can also be extended to multi-level conclusions or output confidence scores.

[0020] The first [item] was obtained by searching the internet for the declaration text or image. The preprocessed candidate evidence set corresponding to each news event is obtained, and the evidence image and its corresponding text are saved for each candidate evidence, thus obtaining the first... A collection of candidate evidence images for a news event and its corresponding candidate evidence text set ;in, Indicates the first The first news event One candidate evidence image, express Corresponding candidate evidence text, such as news headlines, summaries, webpage text snippets, or image captions, are used to provide cross-modal evidence together with the evidence images. , The number of candidate evidences can be set to an upper limit based on computing power, and can be deduplicated and filtered in advance to reduce noise and duplication.

[0021] Will The Middle The best-matching evidence image for a news event is denoted as ,Will middle The corresponding best-matching evidence text is denoted as Preferably, candidate evidence can be coarsely ranked using general image-text similarity to obtain the best matching evidence as a text anchor point, which facilitates subsequent knowledge-enhanced retrieval to complete the background around the same event and reduce cross-event noise.

[0022] Step Two, as follows Figure 2 The diagram illustrates the construction of a semantically enhanced retrieval module, used to learn the alignment relationship between claims and evidence in a unified feature space of the news domain, thereby achieving fine-grained cross-modal retrieval. This module specifically includes: a pre-trained image encoder, a pre-trained text encoder, an image domain adapter, and a text domain adapter, which are extracted by the pre-trained encoder. , as well as , The general representation of the image and text is then encoded and adapted by the domain adapter to complete the domain distribution correction and output the aligned representation, thus obtaining the corresponding first... The image representation vector of a news event's statement. , Declare text representation vector and the The image representation vector of each candidate evidence , No. Candidate evidence text representation vectors Thus constructing the first A positive sample set of news events and the A positive sample set of news events The positive sample set is used for subsequent fact NCE contrastive learning, including statement image-text pairs, evidence image-text pairs, and cross-modal cross-pairing, to enhance the ability to model event-level semantic consistency.

[0023] Step 2.1: The pre-trained image encoder respectively... and The initial declared image features are obtained through processing. With the Initial candidate evidence image features In practice, the pre-trained image encoder preferably uses a visual encoder such as CLIP; the input image is encoded and pooled to obtain a feature vector, which can be regarded as the initial visual semantic representation.

[0024] Step 2.2: The pre-trained text encoder is respectively... and The initial declaration text features are obtained through processing. With the Initial candidate text features In practice, the pre-trained text encoder preferably uses the CLIP text encoder or a similar Transformer; after word segmentation and truncation, the text outputs text features for alignment with visual features.

[0025] Step 2.3: The image domain adapter adopts a two-layer MLP and introduces a residual connection structure. and Perform domain adaptation mapping to project the initial visual features onto the news domain alignment space, thereby obtaining the corresponding first... The image representation vector of a news event's statement. and the The image representation vector of each candidate evidence ,make The corresponding best-matching image representation vector is denoted as The image representation vector corresponding to the best matching evidence can serve as an important anchor point for subsequent retrieval and reasoning stages.

[0026] Step 2.4: The text domain adapter adopts a structure symmetrical to that of the image domain adapter. and Domain adaptation mapping is performed to map the initial text features to a news domain space consistent with visual representation, thereby obtaining the declaration text representation vector. and the Candidate evidence text representation vectors ,make The corresponding best-matching evidence text representation vector is denoted as The text representation vector corresponding to the best matching evidence can be used for query construction and evidence refinement in subsequent knowledge-enhanced retrieval.

[0027] In practice, the domain adapter can adopt a lightweight Adapter / residual feedforward network to map general pre-trained features to the news domain alignment space, thereby reducing training costs while maintaining the general pre-trained representation.

[0028] Step 2.5, from and Composition of declaration image—declaration text pair It corresponds to the image and text pairing of the same declaration, used to ensure that the image and text representation of the declaration are consistent.

[0029] Depend on and Composition of declaration text — declaration image pair ; is with Symmetrical reverse pairings are used to construct positive samples in the text-to-image direction.

[0030] Depend on and Composition of Evidence Images—Evidence Text Pairs It corresponds to the image and text pairing within the same candidate evidence, used to maintain the consistency of the evidence's images and text.

[0031] Depend on and Composition of Evidence Text-Evidence Image Pair ; is with Symmetrical reverse pairing for positive sample augmentation in the text-to-image direction.

[0032] Depend on and Composition of candidate image-evidence text pairs By introducing cross-pairing of declarative images and evidentiary text, robustness to "title rewriting / description difference" scenarios can be improved.

[0033] Depend on and Composition of Evidence Text-Candidate Image Pairs ; is with Symmetrical cross-pairing is used to supplement positive sample coverage in the text-to-image direction.

[0034] Depend on and Composition of Evidence Images—Candidate Text Pairs Further, by introducing cross-pairing of evidence images and candidate texts, the ability to align multi-source descriptions is enhanced.

[0035] Depend on and Composition of candidate text-evidence image pairs ; is with Symmetrical cross pairings are used to refine event-level consistency constraints in the text-to-image direction.

[0036] Declaring image—declaring text pair Evidence Image - Evidence Text Pair Candidate image-evidence text pair Evidence Image-Candidate Text Pair Constituting the first A positive sample set of images to text from a news event. Treating multiple image-text pairings within the same news event as a unified positive sample set can simultaneously constrain the event-level semantic consistency of "statement-evidence" and "evidence-evidence".

[0037] From declaration text—declaration image pair Evidence text-evidence image pair Evidence text—candidate image pair Candidate text-evidence image pair Constituting the first A collection of positive text-to-image samples for a news event. The positive sample set in the text-to-image direction is constructed symmetrically with the image-to-text direction, which can avoid the model from overfitting only on unidirectional similarity.

[0038] Step 3, based on and The fact NCE loss of the semantic enhancement retrieval module is constructed using a contrastive learning approach. It uses image-text pairings within the same news event as positive samples and other event samples within the same batch as negative samples. A temperature coefficient is used to adjust the difficulty of differentiation, thereby learning a more discriminative news domain alignment space. Specifically, this loss includes: the fact NCE loss in the image-to-text direction. And the fact of NCE loss in the text-to-image direction This is used to train the semantic enhancement retrieval module, obtain the trained semantic enhancement retrieval module, and output the optimal image to the text set. and the optimal text-to-image set During this training phase, only a few parameters such as the domain adapter may be updated. After training is completed, the parameters of the semantic enhancement retrieval module are fixed for evidence ranking during the inference phase.

[0039] Step 3.1: Construct the factual NCE loss from image to text direction using equation (1). The image-to-text approach, using the statement / evidence image as the query and the text as the candidate, helps to strengthen the consistency modeling between the evidence text and the evidence image caption.

[0040] (1) In equation (1), The total number of news events; For the first The image representation vector of each news event, and ∈ ; For the first The text representation vector of a news event, and ∈ ; Indicates the first The first news event One text candidate representation vector, Represents the dot product of vectors; The temperature coefficient is used; the feature dimension can be set according to the pre-trained encoder, for example, 512; the temperature coefficient can be tuned on the training set to balance the convergence speed and the ability to distinguish difficult negative samples.

[0041] Step 3.2: Construct the factual NCE loss for the text-to-image direction using equation (2). The text-to-image direction is symmetrical to step 3.1, which avoids the model relying solely on unidirectional similarity and improves the stability of cross-modal alignment.

[0042] (2) In equation (2), Indicates the first The first news event A number of candidate image representation vectors; by summing over all news events, sufficient negative samples can be constructed within a batch and the discriminative power of contrastive learning can be improved; in practice, vectorized computation can be used to improve training efficiency.

[0043] Step 4: During the inference phase, use the trained semantic enhancement retrieval module to keep its parameters fixed, and perform forward computation on the... Image of the statement corresponding to each news event With the declaration text and images of various candidate pieces of evidence related to this news event. and its corresponding candidate evidence text Perform encoding and domain adaptation, and output the optimal representation vector for each claim / evidence, including: The optimal representation vector of a news event's statement image. , Declaring the optimal representation vector of the text , No. Optimal representation vector of candidate evidence images and the The optimal representation vector of candidate evidence texts And use equation (3) to calculate the first... The pending verification statement for the first news event and its first Semantic consistency score among candidate evidence Thus, a semantic consistency score set is obtained. In this embodiment, the semantic consistency score can integrate bidirectional similarity such as "declaration image-evidence text" and "declaration text-evidence image", thereby improving robustness against image-text mismatch and cross-event interference.

[0044] right After sorting in descending order, select the first few... The candidate evidence image is used as the first A collection of semantically enhanced evidence images for a news event; This indicates the preset number of evidence images. This indicates the number of candidate evidences. In practice, the number of candidates can be preset to a small value to balance the sufficiency of evidence with the length of the prompt words. For example, the first 3 to 5 evidence images can be retained for subsequent reasoning.

[0045] (3) In equation (3), Indicates the first The first news event The optimal representation vector for each candidate evidence image; express The corresponding candidate evidence text optimal representation vector; the score given by equation (3) is used to measure the semantic alignment between the statement and the candidate evidence in the unified space. The larger the score, the more likely they are to come from the same event context, and thus can be used for evidence ranking and screening.

[0046] Step 5: Construct a knowledge-enhanced retrieval module to supplement the entity background knowledge related to the declaration and to denoise and enhance the information density of the webpage evidence text. This module, together with the semantic enhancement retrieval module, constitutes a multi-level retrieval module. Specifically, this module includes: a search agent focusing on external knowledge retrieval and a refinement agent focusing on text evidence refinement. The search agent is used, for example, to obtain entity relationships and descriptions from knowledge graphs / knowledge bases; the refinement agent is used, for example, for sentence segmentation, vectorized filtering, and redundancy removal and rearrangement, thereby obtaining the relevant information. Knowledge text of a news event and the A collection of evidence texts after preprocessing of a news event ; Step 5.1: The search agent can use object detection models, entity recognition models, or third-party entity detection services to detect visual entities such as people, places, organizations, and landmarks from the declaration image, and extract named entities by combining the declaration text, thereby enabling the search to... and Entity extraction is performed to obtain the first... A collection of entities from a news event and to Entity links are established, with a preference for links to knowledge graphs such as DBpedia or self-built graphs. Furthermore, entity descriptions, attributes, and relationships can be obtained from structured knowledge bases such as Wikidata or encyclopedic knowledge sources before being linked to the knowledge graph. With knowledge base Search The relationships, attributes, and descriptions of each entity in the code are used to generate the first... Knowledge text of a news event This knowledge text can be organized into complete background sentences or a list of key points, which can be used as external knowledge supplements when multimodal large language models are reasoning.

[0047] Step 5.2, Refine the agent to obtain the first A collection of evidence texts related to a news event Furthermore, the evidence is segmented into segments, such as by periods, semicolons, and line breaks, to reduce noise in long texts and facilitate subsequent filtering, thereby obtaining the first... A collection of text fragments from a news event Therefore, based on equation (4), Redundancy is filtered out to obtain the first... A collection of evidence texts after preprocessing of a news event ,in, For the first The first news event A text fragment, The number of text fragments is used to filter redundancy. In practice, the method of filtering redundancy is to calculate the cosine similarity of the fragment vectors. Fragments that exceed the threshold are considered redundant and deleted to avoid the inflation of prompt words and highlight key information. Subsequently, the retained fragments can be rearranged and reorganized to form the final evidence text with higher information density and more coherence.

[0048] (4) In equation (4), Indicates the first The first news event A text fragment; Indicates to Encoded semantic vector; Indicates to Encoded semantic vector; Represents the cosine similarity function; This is the redundancy filtering threshold; this redundancy filtering threshold can be set on the development set, for example, from 0.7 to 0.9; the smaller the threshold, the stricter the deduplication, but it may mistakenly delete key evidence with similar semantics, and the parameter can be adjusted according to the actual effect.

[0049] Step Six: Design a preset template that includes a declaration image, declaration text, a set of semantically enhanced evidence images, refined evidence text, and knowledge text. Explicitly require the output of structured results for "inference fields" and "conclusion fields," such as JSON key-value pairs or fixed-field text, for automatic parsing and auditing. , , , and Construct the first according to the preset template Tips for a news event The data is then input into a multimodal large language model for processing, thereby outputting the first... Structured reasoning results and predictive verification conclusions for individual news events ,when When, it indicates the prediction of the first The veracity of a news event is determined to be true, when... When, it indicates the prediction of the first The authenticity of a news event is determined to be false; through structured prompts and structured outputs, the illusion caused by the free generation of large language models can be reduced and the interpretability of the conclusions can be improved.

[0050] Step 7: Construct cross-entropy loss using equation (5) It is used to perform cold start training on a multimodal large language model to obtain a multimodal large language model after cold start training. Cold start training can be supervised and fine-tuned using a small amount of high-quality chained inference data, such as distilling annotations with inference processes from a high-performance multimodal large language model. The model is first taught the basic ability of "inference based on evidence and template" through cross-entropy loss.

[0051] (5) In equation (5), The cross-entropy loss represents the probability value; it measures the difference between the model's output distribution and the true label distribution. In implementation, mini-batch training combined with a learning rate warm-up can be used to stabilize convergence. Specifically, data can be augmented by rejecting samples to strengthen correct inference trajectories.

[0052] Step 8, Group Relative Policy Optimization, involves sampling multiple candidate inference results for each prompt word and performing intra-group relative comparisons, achieving stable optimization without a separate value network. Simultaneously, a divergence regularization term can be added to limit policy deviation and improve training stability. Therefore, the group relative policy optimization method is used to perform reinforcement learning on the cold-start trained multimodal large language model, resulting in a fine-tuned multimodal large language model used to output structured reasoning processes and verification conclusions.

[0053] Step 8.1: Construct an accuracy reward system ,when When, it indicates that the authenticity judgment result output by the multimodal large language model is consistent with the authenticity verification conclusion, when When the accuracy of the multimodal large language model outputs a result that differs from the actual verification conclusion, it indicates that the accuracy reward is used to directly constrain the correctness of the conclusion. It can also be extended to a continuous reward based on confidence level or partial matching.

[0054] Step 8.2, Build Format Rewards ,when When, it indicates that the structured reasoning results output by the multimodal large language model after cold start training include preset reasoning fields and conclusion fields. When the format reward is set to 0, it indicates that the structured reasoning results output by the multimodal large language model after cold start training do not include the preset reasoning fields and conclusion fields. This format reward is used to constrain the output to be parsable, ensuring that the reasoning fields and conclusion fields are complete, thereby facilitating automatic evaluation and traceability.

[0055] Step 8.3: Construct the total reward The total reward can also be set as a weighted sum, with the weights adjusted according to the importance of accuracy and interpretability.

[0056] Step 8.4: Set the prompt words The input is repeatedly processed in the multimodal large language model after cold start training to obtain... Candidate structured reasoning results ,in, Indicates the first The number of candidate structured inference results can be preset, for example, 8 inference results are sampled each time to form relative comparisons within the group.

[0057] Step 8.5, calculate the first... Candidate structured reasoning results Total reward and its advantage value This advantage value can normalize the rewards of candidate reasoning results in the same group, so as to compare relative superiority and inferiority and serve as a training signal for policy updates.

[0058] Step 8.6: Using the multimodal large language model trained after cold start as the policy model, construct the group relative policy optimization objective using equation (6). This is used to train the policy model to obtain a multimodal large language model after reinforcement learning. By limiting the policy update magnitude through the pruning function and combining it with the divergence regularization term to suppress excessive deviation from the reference model, the stability and generalization ability of the reinforcement learning stage can be improved.

[0059] (6) In equation (6), This represents the policy model during the current training round. This represents the policy model from the previous training round. Representational strategy model, For the clipping function, This is the cutting factor; for divergence regularization term, for The weight coefficients of the divergence regularization term; the pruning coefficients and divergence regularization weights can be tuned on the development set; the reference model can be a frozen copy of the cold-start model, used to limit policy drift and avoid reward hacking.

[0060] Step Nine: The final multimodal fact-checking model, composed of the trained semantic-enhanced retrieval module, knowledge-enhanced retrieval module, and the reinforcement-learned multimodal large language model, is used to perform evidence retrieval and fact reasoning on the input multimodal claims, and output the optimal structured reasoning result and its optimal predicted verification conclusion. The final model operates in a pipeline manner of "semantic-enhanced retrieval - knowledge-enhanced retrieval - structured reasoning discrimination". The modular design facilitates independent replacement of the encoder, retrieval source, or inference model during engineering deployment, and can be tailored and accelerated according to computational constraints.

[0061] In this embodiment, an electronic device includes a memory and a processor. The memory stores a program that supports the processor in executing the above-described method, and the processor is configured to execute the program stored in the memory.

[0062] In this embodiment, a computer-readable storage medium stores a computer program, which is executed by a processor to perform the steps of the above method.

[0063] Experimental verification: To quantitatively evaluate the effectiveness and robustness of the method of the present invention in multimodal fact-checking tasks, this embodiment selects a publicly available benchmark dataset for comparative experiments. English data sets include NewsCLIPpings (Grace Luo, TrevorDarrell, and Anna Rohrbach. Newsclippings: Automatic generation of out-of-context multimodal media. arXiv:2104.05893, 2021), OOC-MFC (Shengkang Wang, Hongzhan Lin, Ziyang Luo, Zhen Ye, Guang Chen, and Jing Ma. MFC-Bench: Benchmarking Multimodal Fact-Checking with Large Vision-Language Models.arXiv:2406.11288, 2024) and VERITE (Stefanos-Iordanis Papadopoulos, ChristosKoutlis, Symeon Papadopoulos, and Panagiotis C. Petrantonakis. Verite: arobust benchmark for multimodal misinformation detection for unimodal bias. International Journal of Multimedia Information Retrieval, 13(1):4, The Chinese dataset used was a subset of CCMF and SCMF from the CMFC benchmark (Fanrui Zhang, Jiawei Liu, Jingyi Xie, Qiang Zhang, Yongchao Xu, and Zheng-Jun Zha. ESCNet: Entity-enhanced and Stance Checking Network for Multi-Modal Fact-Checking. In Proceedings of the ACM Web Conference 2024, 2429–2440). Evaluation metrics were presented in Tables 1 and 2 as binary classification accuracy (Acc, %); the ablation experiments (Table 3) further reported precision (Prec), recall (Rec), and F1 score (%).The comparative methods cover traditional multimodal verification models and multimodal large language model solutions, including BERT (Jacob Devlin et al., NAACL 2019), CLIP (Alec Radford et al., ICML 2021), CCN (Sahar Abdelnabi, Rakibul Hasan, and Mario Fritz, CVPR 2022), SNIFFER (Peng Qi, Zehong Yan, Wynne Hsu, and Mong Li Lee, CVPR 2024), RED-DOT / AITR (Stefanos-Iordanis Papadopoulos et al., 2025), and DEFAME (Tobias Braun et al., ICML 2025).

[0064] Table 1. Comparison of Accuracy (Acc, %) on the English Multimodal Fact Check Dataset

[0065] As shown in Table 1, the method of this invention achieves the best accuracy on the OOC-MFC and VERITE English benchmarks (90.2% and 84.1%, respectively), which is significantly better than earlier methods based solely on general representations (such as BERT and CLIP). This indicates that introducing semantically enhanced retrieval can effectively alleviate the performance bottleneck caused by the semantic mismatch between claim images and evidence images. Compared with direct end-to-end inference of multimodal large language models (e.g., GPT-4o achieves 84.4% on OOC-MFC and 73.6% on VERITE), this invention significantly improves the factual consistency of inference through "evidence image retrieval + evidence text refinement + rule reward-driven inference optimization," demonstrating that the synergistic mechanism of retrieval and inference enhancement can provide the model with more reliable external evidence and reduce unfounded generation. Furthermore, this invention achieves an accuracy of 92.2% on the NewsCLIPpings dataset. Although it lags behind some specialized methods (e.g., AITR's 93.3%), it is still significantly better than most baselines and maintains good cross-dataset generalization performance.

[0066] Table 2 Comparison of Accuracy (Acc, %) on the Chinese Multimodal Fact Check Dataset

[0067] As shown in Table 2, the method of this invention achieves 88.4% and 87.7% accuracy on the Chinese CCMF and SCMF datasets, respectively, both the highest in the table. Compared to ESCNet, a representative method designed for Chinese multimodal fact-checking (86.2% and 84.9% on CCMF and SCMF, respectively), this invention improves accuracy by 2.2 and 2.8 percentage points, respectively. Compared to zero-shot inference schemes that rely solely on the model's own knowledge (such as some multimodal large language models), this invention, by introducing knowledge-enhanced retrieval, can complete entity background and relational information, and suppresses the interference of noisy evidence on the inference chain through refined proxies. Therefore, it can also maintain a stable improvement in Chinese scenarios.

[0068] Table 3 Ablation experiment results on the OOC-MFC dataset (Acc / Prec / Rec / F1, %)

[0069] Table 3 presents the component ablation results on the OOC-MFC dataset. It can be seen that: (1) after removing cold start training, the accuracy dropped from 90.2% to 79.9%, and Prec / Rec / F1 also dropped significantly, indicating that the cold start stage provides a controllable inference starting point for the model, which can significantly reduce the instability and mode collapse in the early stage of training; (2) removing semantic enhancement retrieval or evidence image input resulted in a decrease in accuracy of 3.7 percentage points and 4.4 percentage points, respectively, verifying that "evidence image semantic alignment + multimodal evidence collaboration" is crucial for identifying mismatched false information; (3) removing refined proxy will reduce the accuracy to 84.5%, indicating that redundancy removal and reorganization of evidence text can improve evidence density and reduce noise misleading; (4) removing knowledge graph and knowledge base will result in a decrease of 2.8 percentage points and 1.8 percentage points, respectively, indicating that structured background knowledge helps to fill the context gap of short texts; (5) in terms of inference optimization, removing rejection sampling or removing group relative strategy optimization will reduce performance (a decrease of 1.5 percentage points and 3.0 percentage points, respectively), indicating that the three-stage training based on rule rewards can gradually enhance the inference depth and output consistency.

[0070] In summary, this invention improves the relevance between evidence images and text through a semantic enhancement retrieval module, completes entity background and refines evidence text through a knowledge enhancement retrieval module, and enhances the reasoning ability and format controllability of a multimodal large language model by combining a three-stage self-optimization training mechanism. Therefore, it can achieve stable and interpretable performance improvements on various public benchmarks.

Claims

1. A multimodal fact-checking method based on a retrieval and reasoning-enhanced large language model, characterized in that, Includes the following steps: Step 1: Obtain the first The pre-processed statements pending verification for each news event include: Image of a statement about a news event With the Statement text for a news event and the corresponding verification conclusions ;when When, it indicates that the veracity of the statement to be verified is true. When this occurs, it indicates that the veracity of the statement to be verified is determined to be false; Get the The preprocessed candidate evidence set corresponding to each news event includes: A collection of candidate evidence images for a news event and its corresponding candidate evidence text set ;in, Indicates the first The first news event One candidate evidence image, express The corresponding candidate evidence text, , The number of candidate pieces of evidence; Will The Middle The best-matching evidence image for a news event is denoted as ,Will middle The corresponding best-matching evidence text is denoted as ; Step 2: Construct a semantically enhanced retrieval module, including: a pre-trained image encoder, a pre-trained text encoder, an image domain adapter, and a text domain adapter, and then apply them to... , as well as , Encoding and domain adaptation are performed to obtain the corresponding first... The image representation vector of a news event's statement. , Declare text representation vector and the The image representation vector of each candidate evidence , No. Candidate evidence text representation vectors Thus constructing the first A positive sample set of news events and the A positive sample set of news events ; Step 3, based on and The loss for constructing the semantic enhancement retrieval module includes: the fact NCE loss in the image-to-text direction. And the fact of NCE loss in the text-to-image direction This is used to train the semantic enhancement retrieval module, obtain the trained semantic enhancement retrieval module, and output the optimal image to the text set. and the optimal text-to-image set ; Step 4: Use the trained semantic enhancement retrieval module to perform retrieval on the first... Image of the statement corresponding to each news event With the declaration text and images of various candidate pieces of evidence related to this news event. and its corresponding candidate evidence text Encoding and domain adaptation are performed to obtain the first... The optimal representation vector of a news event's statement image. , Declare the optimal representation vector of the text , No. Optimal representation vector of candidate evidence images and the The optimal representation vector of candidate evidence texts And use equation (3) to calculate the first... The pending verification statement for the first news event and its first Semantic consistency score among candidate evidence Thus, a semantic consistency score set is obtained. ; and on After sorting in descending order, select the first few... The candidate evidence image is used as the first A collection of semantically enhanced evidence images for a news event; This indicates the preset number of evidence images. Indicates the number of candidate pieces of evidence; (3) In equation (3), Indicates the first The first news event The optimal representation vector for each candidate evidence image; express The corresponding optimal representation vector of the candidate evidence text; Step 5: Construct a knowledge-enhanced retrieval module, including a search agent and a refinement agent, which are used to obtain the first... Knowledge text of a news event and the A collection of evidence texts after preprocessing of a news event ; Step Six: , , , and Construct the first according to the preset template Tips for a news event The data is then input into a multimodal large language model for processing, thereby outputting the first... Structured reasoning results and predictive verification conclusions for individual news events ,when When, it indicates that the prediction of the first The veracity of a news event is determined to be true, when... When, it indicates that the prediction of the first The authenticity of the news event was determined to be false; Step 7: Construct cross-entropy loss using equation (5) It is used to perform cold start training on a multimodal large language model to obtain a multimodal large language model after cold start training; (5) In equation (5), Represents the probability value; Step 8: Use the group relative strategy optimization method to perform reinforcement learning on the multimodal large language model after cold start training to obtain the multimodal large language model after reinforcement learning; Step 9: The final multimodal fact-checking model is composed of the trained semantic enhancement retrieval module, the knowledge enhancement retrieval module, and the multimodal large language model after reinforcement learning. It is used to perform evidence retrieval and fact reasoning on the input multimodal claims and output the optimal structured reasoning result and its optimal predicted verification conclusion.

2. The multimodal fact-checking method based on a retrieval and reasoning-enhanced large language model according to claim 1, characterized in that, Step two includes the following steps: Step 2.1: The pre-trained image encoder respectively... and The initial declared image features are obtained through processing. With the Initial candidate evidence image features ; Step 2.2: The pre-trained text encoder is respectively... and The initial declaration text features are obtained through processing. With the Initial candidate text features ; Step 2.3, the image domain adapter... and Perform domain adaptation mapping to obtain the corresponding domain... The image representation vector of a news event's statement. and the The image representation vector of each candidate evidence ,make The corresponding best-matching image representation vector is denoted as ; Step 2.4, the text field adapter... and Perform domain adaptation mapping to obtain the corresponding declaration text representation vector. and the Candidate evidence text representation vectors ,make The corresponding best-matching evidence text representation vector is denoted as ; Step 2.5, from and Composition of declaration image—declaration text pair ; Depend on and Composition of declaration text — declaration image pair ; Depend on and Composition of Evidence Images—Evidence Text Pairs ; Depend on and Composition of Evidence Text-Evidence Image Pair ; Depend on and Composition of candidate image-evidence text pairs ; Depend on and Composition of Evidence Text-Candidate Image Pairs ; Depend on and Composition of Evidence Images—Candidate Text Pairs ; Depend on and Composition of candidate text-evidence image pairs ; Declaring image—declaring text pair Evidence Image - Evidence Text Pair Candidate image-evidence text pair Evidence Image-Candidate Text Pair Constituting the first A positive sample set of images to text from a news event. ; From declaration text—declaration image pair Evidence text-evidence image pair Evidence text—candidate image pair Candidate text-evidence image pair Constituting the first A collection of positive text-to-image samples for a news event. .

3. The multimodal fact-checking method based on a retrieval and reasoning-enhanced large language model according to claim 2, characterized in that, Step three includes the following steps: Step 3.1: Construct the factual NCE loss from image to text using equation (1). : (1) In equation (1), The total number of news events; For the first The image representation vector of each news event, and ∈ ; For the first The text representation vector of a news event, and ∈ ; Indicates the first The first news event One text candidate representation vector, Represents the dot product of vectors; Temperature coefficient; Step 3.2: Construct the factual NCE loss for the text-to-image direction using equation (2). : (2) In equation (2), Indicates the first The first news event Image candidate representation vectors.

4. The multimodal fact-checking method based on a retrieval and reasoning-enhanced large language model according to claim 3, characterized in that, Step five includes the following steps: Step 5.1, Search for proxy pairs and Entity extraction is performed to obtain the first... A collection of entities for a news event and to After entity linking, in the knowledge graph With knowledge base Search The relationships, attributes, and descriptions of each entity in the code are used to generate the first... Knowledge text of a news event ; Step 5.2, Refine the agent to obtain the first A collection of evidence texts related to a news event And perform segmentation to obtain the first segment. A collection of text fragments from a news event Therefore, based on equation (4), Redundancy is filtered out to obtain the first... A collection of evidence texts after preprocessing of a news event ,in, For the first The first news event A text fragment, Number of text fragments: (4) In equation (4), Indicates the first The first news event A text fragment; Indicates to Encoded semantic vector; Indicates to Encoded semantic vector; Represents the cosine similarity function; This is the redundancy filtering threshold.

5. The multimodal fact-checking method based on a retrieval and reasoning-enhanced large language model according to claim 4, characterized in that, Step eight includes the following steps: Step 8.1: Construct an accuracy reward system ,when When, it indicates that the authenticity judgment result output by the multimodal large language model is consistent with the authenticity verification conclusion. This indicates that the authenticity judgment result output by the multimodal large language model differs from the actual verification conclusion; Step 8.2, Build Format Rewards ,when When, it indicates that the structured reasoning results output by the multimodal large language model after cold start training include preset reasoning fields and conclusion fields. This indicates that the structured reasoning results output by the multimodal large language model after cold start training do not include the preset reasoning fields and conclusion fields; Step 8.3: Construct the total reward ; Step 8.4: Set the prompt words The input is repeatedly processed in the multimodal large language model after cold start training to obtain... Candidate structured reasoning results ,in, Indicates the first The candidate structured reasoning results; Step 8.5, calculate the first... Candidate structured reasoning results Total reward and its advantage value ; Step 8.6: Using the multimodal large language model trained after cold start as the policy model, construct the group relative policy optimization objective using equation (6). This is used to train the policy model, resulting in a multimodal large language model after reinforcement learning. (6) In equation (6), This represents the policy model during the current training round. This represents the policy model from the previous training round. Representational strategy model, For the clipping function, This is the cutting factor; for divergence regularization term, for The weighting coefficients of the divergence regularization term.

6. An electronic device, comprising a memory and a processor, characterized in that, The memory is used to store a program that supports a processor in executing the method of any one of claims 1-5, the processor being configured to execute the program stored in the memory.

7. A computer-readable storage medium storing a computer program thereon, characterized in that, The computer program is executed by the processor to perform the steps of the method according to any one of claims 1-5.

Images