Multi-modal fact checking method based on retrieval enhancement generation

Abstract

The invention discloses a multi-modal fact checking method based on retrieval enhancement generation. The method comprises the steps that a multi-modal fact to be checked is acquired, the multi-modal fact to be checked comprises information of multimedia and / or information of a fact text, and the multimedia comprises a fact image and a fact video; under the condition that the to-be-checked multi-modal facts meet the retrieval enhancement generation conditions, the input to-be-checked multi-modal facts are retrieved through the retrieval enhancement generation model, and background knowledge texts corresponding to the to-be-checked multi-modal facts are determined; the input to-be-checked multi-modal fact and the background knowledge text are processed through the trained target checking model, a fact checking result corresponding to the to-be-checked multi-modal fact is determined, and the fact checking result comprises fact truth and / or fact abnormity. According to the technical scheme of the invention, the accuracy and efficiency of multi-modal fact verification can be improved.

Description

Technical Field

[0001] The present invention relates to the field of computer application technology, and in particular to a multimodal fact-checking method based on retrieval-enhanced generation. Background Art

[0003] In related technologies, methods such as keyword matching or text similarity calculation are usually used to determine the authenticity of information. However, the above methods are more suitable for the verification of single-modal facts, that is, text-based facts. For multi-modal facts, it is difficult to ensure the accuracy and efficiency of fact verification. Summary of the invention

[0004] The present invention provides a multimodal fact-checking method based on retrieval enhanced generation to solve the technical problem that the prior art is difficult to ensure the accuracy and efficiency of multimodal fact-checking.

[0005] According to one aspect of the present invention, a multimodal fact-checking method based on retrieval-enhanced generation is provided, wherein the method comprises:

[0006] Acquire a multimodal fact to be verified, wherein the multimodal fact to be verified includes multimedia information and / or fact text information, and the multimedia includes a fact image and a fact video;

[0007] In the case where the multimodal fact to be verified meets the retrieval enhancement generation condition, the input multimodal fact to be verified is retrieved through the retrieval enhancement generation model to determine the background knowledge text corresponding to the multimodal fact to be verified;

[0008] The trained target verification model is used to process the input multimodal facts to be verified and the background knowledge text to determine the fact verification result corresponding to the multimodal facts to be verified.

[0009] The fact-checking result includes factual truth and / or factual anomaly, and the target verification model is obtained by training a preliminary verification model based on multimodal fact samples.

[0010] The technical solution of the embodiment of the present invention is to obtain a multimodal fact to be verified, wherein the multimodal fact to be verified includes multimedia information and / or fact text information, and the multimedia includes fact images and fact videos; when the multimodal fact to be verified meets the retrieval enhancement generation condition, the input multimodal fact to be verified is retrieved through the retrieval enhancement generation model to determine the background knowledge text corresponding to the multimodal fact to be verified; the input multimodal fact to be verified and the background knowledge text are processed by the trained target verification model to determine the fact verification result corresponding to the multimodal fact to be verified, wherein the fact verification result includes fact truth and / or fact abnormality, and the target verification model is obtained by training the preliminary verification model based on the multimodal fact sample. The technical solution of the present invention realizes the authenticity verification of multimodal facts, and adopts the method of retrieving background knowledge through retrieval enhancement generation technology, combining background knowledge and deep learning model to verify facts, thereby improving the accuracy and efficiency of multimodal fact verification.

[0011] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0013] Figure 1 is a flow chart of a multimodal fact checking method based on retrieval enhanced generation provided according to the first embodiment of the present invention;

[0014] Figure 2 is a flow chart of a multimodal fact checking method based on retrieval enhanced generation provided according to the second embodiment of the present invention;

[0015] Figure 3 is a flow chart of performing fact checking through a target checking model provided according to an embodiment of the present invention;

[0016] Figure 4 It is a flow chart of a target verification model obtained by training according to an embodiment of the present invention. DETAILED DESCRIPTION

[0017] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0018] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0019] Embodiment 1

[0020] Figure 1 The first embodiment of the present invention provides a flowchart of a multimodal fact checking method based on retrieval enhancement generation. This embodiment is applicable to the case of authenticity verification of multimodal facts. Figure 1 As shown, the method includes:

[0021] S110: Acquire multimodal facts to be verified, wherein the multimodal facts to be verified include multimedia information and / or fact text information, and the multimedia includes fact images and fact videos.

[0022] The multimodal facts to be verified can be understood as multimodal facts to be verified. Optionally, the facts can be news or posts spread through the network in the name of facts. The multimodal facts can be understood as facts containing at least two forms such as text, image and video.

[0023] The fact text may be understood as the text contained in the multimodal fact to be verified.

[0024] The fact image may be understood as an image included in the multimodal fact to be verified.

[0025] The fact video can be understood as the video included in the multimodal fact to be verified.

[0026] Optionally, after obtaining the multimodal fact to be verified, the method further includes:

[0027] Determining stored knowledge text in a target database that is associated with the multimodal fact to be verified;

[0028] When the number of the stored knowledge texts does not exceed a preset threshold, the to-be-verified multimodal fact is regarded as a multimodal fact that satisfies the retrieval enhancement generation condition;

[0029] When the number of texts in the stored knowledge text exceeds a preset threshold, the multimodal fact to be verified is regarded as a multimodal fact that does not meet the retrieval enhancement generation condition.

[0030] The target database may be understood as a database storing background knowledge. The background knowledge may be understood as text that further explains facts. Exemplarily, the background knowledge may include concept explanations and / or associated facts.

[0031] The stored knowledge text may be understood as background knowledge stored in the target database and associated with the multimodal fact to be verified.

[0032] The preset threshold value can be understood as a preset threshold value. The preset threshold value is associated with the search enhancement generation condition. In the implementation of the present invention, the preset threshold value can be preset according to the scene requirements, and is not specifically limited here.

[0033] The retrieval enhancement generation condition can be used to determine whether the multimodal fact to be verified requires retrieval enhancement generation.

[0034] S120: When the multimodal fact to be verified satisfies the retrieval enhancement generation condition, the input multimodal fact to be verified is retrieved through the retrieval enhancement generation model to determine the background knowledge text corresponding to the multimodal fact to be verified.

[0035] The retrieval-augmented generation model can be understood as a model with retrieval-augmented generation function. The retrieval-augmented generation (RAG) refers to a technology that uses information from private or proprietary data sources to assist in text generation.

[0036] The background knowledge text may be understood as the text of the background knowledge associated with the multimodal fact to be verified. The background knowledge may be understood as the text further explaining the fact. Exemplarily, the background knowledge may include concept explanations and / or associated facts.

[0037] Optionally, the step of retrieving the input multimodal fact to be verified by using the retrieval enhancement generation model to determine the background knowledge text corresponding to the multimodal fact to be verified includes:

[0038] The input multimodal facts to be verified are converted by using a retrieval enhancement generation model and preset prompt words to determine a search statement;

[0039] Searching the search statement through a target search engine to determine a plurality of search web pages and web page text corresponding to each of the search web pages;

[0040] The background knowledge text corresponding to the multimodal fact to be verified is determined according to the webpage text corresponding to each of the search webpages.

[0041] The preset prompt words can be used to guide the search enhancement generation model to convert the multimodal facts to be verified into effective search statements. In the embodiment of the present invention, the preset prompt words can be set in advance according to the application scenario and user needs, and are not specifically limited here.

[0042] The search statement can be understood as a statement that can be used for engine search. Obviously, the search statement is associated with the multimodal fact to be verified.

[0043] The target search engine can be understood as an engine with a search function. In the embodiment of the present invention, the target search engine can be preset according to the scene requirements, and is not specifically limited here. Optionally, the target search engine can include Google, Yahoo or Baidu, etc.

[0044] The search web page may be understood as a web page determined by searching and related to the search statement.

[0045] The webpage text may be understood as webpage content in text form in the search webpage.

[0046] Optionally, determining the background knowledge text corresponding to the multimodal fact to be verified according to the webpage text corresponding to each search webpage includes:

[0047] For each webpage text corresponding to the search webpage, determining the degree of relevance between the webpage text and the multimodal fact to be verified;

[0048] Determining at least one target webpage associated with the multimodal fact to be verified among the plurality of search webpages according to the relevance;

[0049] The background knowledge text corresponding to the multimodal fact to be verified is determined according to the webpage text corresponding to the target webpage and the preset prompt word.

[0050] In an embodiment of the present invention, the method for determining the degree of relevance between the webpage text and the multimodal fact to be verified can be preset according to the scenario requirements and is not specifically limited here. Optionally, the degree of relevance may include a relevance ratio. The relevance ratio may be the percentage of the relevant text in the search webpage to the webpage text.

[0051] Optionally, determining at least one target webpage associated with the to-be-verified multimodal fact from among the plurality of search webpages according to the relevance degree includes:

[0052] A correlation threshold is determined, and the search webpage whose correlation ratio exceeds the correlation threshold is used as the target webpage. The correlation threshold can be preset according to scene requirements and is not specifically limited here.

[0053] The target web page can be understood as a web page in the search web page that is highly relevant to the multimodal fact to be verified.

[0054] S130, processing the input multimodal facts to be verified and the background knowledge text through the trained target verification model to determine the fact verification result corresponding to the multimodal facts to be verified.

[0055] The target verification model may be understood as a model having a fact-checking function.

[0056] The fact-checking result may be understood as a verification result that characterizes whether the multimodal fact to be verified is a true fact. Optionally, the fact-checking result may include a true fact and / or an abnormal fact.

[0057] The target verification model is obtained by training a preliminary verification model based on multimodal fact samples.

[0058] Optionally, the target verification model may include a multi-task learning model and a joint attention mechanism, and the multi-task learning model includes multiple sub-network models.

[0059] Optionally, the multimodal fact checking based on retrieval-enhanced generation further includes:

[0060] When the multimodal fact to be verified does not satisfy the retrieval enhancement generation condition, the input multimodal fact to be verified and the stored knowledge text are processed by the trained target verification model to determine the fact verification result corresponding to the multimodal fact to be verified.

[0061] In an embodiment of the present invention, when the multimodal fact to be verified does not satisfy the retrieval enhancement generation condition, that is, when the target database stores knowledge text sufficient for fact-verifying the input multimodal fact to be verified, fact-verification can be performed directly without the need for retrieval enhancement generation, which can effectively improve the efficiency of fact-verification while ensuring the accuracy of fact-verification.

[0062] The technical solution of the embodiment of the present invention is to obtain a multimodal fact to be verified, wherein the multimodal fact to be verified includes multimedia information and / or fact text information, and the multimedia includes fact images and fact videos; when the multimodal fact to be verified meets the retrieval enhancement generation condition, the input multimodal fact to be verified is retrieved through the retrieval enhancement generation model to determine the background knowledge text corresponding to the multimodal fact to be verified; the input multimodal fact to be verified and the background knowledge text are processed by the trained target verification model to determine the fact verification result corresponding to the multimodal fact to be verified, wherein the fact verification result includes fact truth and / or fact abnormality, and the target verification model is obtained by training the preliminary verification model based on the multimodal fact sample. The technical solution of the present invention realizes the authenticity verification of multimodal facts, and adopts the method of retrieving background knowledge through retrieval enhancement generation technology, combining background knowledge and deep learning model to verify facts, thereby improving the accuracy and efficiency of multimodal fact verification.

[0063] Embodiment 2

[0064] Figure 2 The flowchart of a multimodal fact-checking method based on retrieval enhancement generation provided in the second embodiment of the present invention is that the target verification model completed through training in the above embodiment processes the input multimodal facts to be verified and the background knowledge text, and determines the fact-checking results corresponding to the multimodal facts to be verified and refines them. Figure 2 As shown, the method includes:

[0065] S210: Obtain multimodal facts to be verified.

[0066] S220: When the multimodal fact to be verified satisfies the retrieval enhancement generation condition, the input multimodal fact to be verified is retrieved through the retrieval enhancement generation model to determine the background knowledge text corresponding to the multimodal fact to be verified.

[0067] Figure 3 FIG. 1 is a flow chart of performing fact checking through a target checking model according to an embodiment of the present invention. Figure 3As shown, optionally, the target verification model includes a multi-task learning model, and the multi-task learning model includes multiple sub-network models.

[0068] Among them, MMOE (Multi-gate Mixture-of-Experts) represents the sub-network model in the multi-task learning model; DM-MMOE (Duo Men-Multi-gate Mixture-of-Experts) represents the sub-network model including the gating mechanism; CO-Attention (Coordinate Attention) represents the common attention mechanism.

[0069] S230: Determine pixel change characteristics of the multimedia.

[0070] The pixel change feature may indicate whether the pixels of the multimedia have been changed. In the embodiment of the present invention, if the pixel change feature indicates that the pixels of the multimedia have been changed, it is considered that the multimedia may have been maliciously tampered with.

[0071] Specifically, the input multimedia may be analyzed by a visual geometry group (VGG) model to determine the pixel change feature corresponding to the multimedia.

[0072] S240, determining the image semantic features corresponding to the multimedia, the text semantic features corresponding to the factual text, and the background semantic features corresponding to the background knowledge text through the multiple sub-network models.

[0073] The image semantic feature can be understood as a feature that characterizes the semantics of the multimedia. Optionally, the image semantic feature can be a text describing the information of the multimedia. Specifically, the semantic feature of the multimedia is recognized by a masked autoencoder (MAE) to determine the image semantic feature.

[0074] The text semantic feature can be understood as a feature that characterizes the semantics of the factual text. Optionally, the text semantic feature can be a text that describes the information of the factual text. In an embodiment of the present invention, the text semantic feature is more concise and clear than the factual text.

[0075] The background semantic features can be understood as features that characterize the semantics of the background knowledge text. In an embodiment of the present invention, the background semantic features are more concise and clear than the background knowledge text. Specifically, the factual text and the background knowledge text are respectively identified by a deep bidirectional language representation model (Bidirectional Encoder Representations from Transformers, BERT), and the text semantic features and the background semantic features are determined in turn.

[0076] S250, determining the network weight of each of the sub-network models, and determining the fact verification result corresponding to the multimodal fact to be verified according to the network weight, the pixel change feature, the image semantic feature, the text semantic feature and the background semantic feature.

[0077] The network weight can be understood as the weight of each sub-network model. Specifically, the network weight can be calculated through the gating mechanism contained in DM-MMOE (Duo Men-Multi-gate Mixture-of-Experts). In an embodiment of the present invention, the actual situation of the task processed by the sub-network model is identified through the gating mechanism to perform real-time allocation of the network weight and ensure that the sum of the network weights of the allocated multiple sub-network models is 1.

[0078] Optionally, the target verification model further includes a common attention mechanism; the determining of the fact verification result corresponding to the multimodal fact to be verified according to the network weight, the pixel change feature, the image semantic feature, the text semantic feature and the background semantic feature includes:

[0079] Determining respectively, by the common attention mechanism, a first fusion feature between the image semantic feature and the text semantic feature, and a second fusion feature between the text semantic feature and the background semantic feature;

[0080] Determine a fact verification result corresponding to the multimodal fact to be verified based on the multiple network weights, the pixel change features, the image semantic features, the text semantic features, the background semantic features, the first fusion features, and the second fusion features.

[0081] The first fusion feature can be understood as a fusion feature between the image semantic feature and the text semantic feature, and the second fusion feature can be understood as a fusion feature between the text semantic feature and the background semantic feature.

[0082] Optionally, determining the first fusion feature between the image semantic feature and the text semantic feature through the common attention mechanism includes:

[0083] Determine respectively, by a multi-head attention mechanism, a first multi-head attention feature corresponding to the image semantic feature and a second multi-head attention feature corresponding to the text semantic feature;

[0084] The first multi-head attention feature and the second multi-head attention feature are fused through a common attention mechanism to obtain a first fused feature.

[0085] Optionally, determining a plurality of first multi-head attention features corresponding to the image semantic features through a multi-head attention mechanism includes:

[0086] Performing a linear transformation on the image semantic features through a multi-head self-attention mechanism to obtain a query matrix, a key matrix, and a value matrix corresponding to the image semantic features;

[0087] Determine a first multi-head attention feature corresponding to the image semantic feature based on the query matrix, the key matrix and the value matrix.

[0088] The technical solution of the embodiment of the present invention determines the pixel change characteristics of the multimedia; determines the image semantic characteristics corresponding to the multimedia, the text semantic characteristics corresponding to the factual text, and the background semantic characteristics corresponding to the background knowledge text through multiple sub-network models; determines the network weight of each sub-network model, and determines the fact verification result corresponding to the multimodal fact to be verified based on the network weight, the pixel change characteristics, the image semantic characteristics, the text semantic characteristics, and the background semantic characteristics. The technical solution of the present invention can ensure the accuracy of fact verification through the target verification model.

[0089] As an optional embodiment of the above embodiment, it is described in the form of steps. Specifically, the specific steps of the multimodal fact checking based on retrieval enhancement generation can be as follows:

[0090] 1. Generate background knowledge text of multimodal posts to be verified based on retrieval enhancement technology.

[0091] First, the content T of the multimodal post to be verified is converted into a search statement Q that can be recognized by the target search engine. Specifically, the conversion can be performed by constructing a large language model (LLM) containing the prompt word. The prompt can be used to guide the LLM to convert T into a valid Q.

[0092] Further, by calling the target search engine, for example, Google Application Programming Interface (Google API), to retrieve, the search results returned are multiple search web pages; extracting the web page text [C1, C2, ..., Cn] in each of the search web pages;

[0093] Furthermore, the correlation between the webpage text and the content T is calculated, the webpage text is sorted according to the correlation, and the webpage text C corresponding to the final reference target webpage is screened;

[0094] Finally, the background knowledge text K is generated by combining the web page text C and the preset prompt.

[0095] In an embodiment of the present invention, before performing retrieval enhancement generation, it is determined whether the knowledge text stored in the target data is sufficient for fact-checking the multimodal post. If the stored knowledge text is sufficient, there is no need to perform retrieval enhancement to generate the background knowledge text K. This can further improve the efficiency of fact-checking while ensuring the accuracy of fact-checking.

[0096] 2. Pixel feature extraction is performed through VGG and sub-network models respectively to determine the pixel features of images in the multimodal posts to be verified. Semantic feature extraction is performed through MAE to determine the semantic features of images in the multimodal posts to be verified. Semantic feature extraction is performed through BERT to determine the semantic features of texts in the multimodal posts to be verified and the semantic features of background knowledge texts.

[0097] First, extract the pixel features of the image. Load the pre-trained VGG network and network weights, where VGG can be VGG19; resize the image to 224×224 pixels, and normalize the 224×224 pixel image to make it meet the input and format requirements of the VGG network; input the pre-processed image into the VGG network, and extract the pixel features of the image through the network's convolutional layer, activation layer, and pooling layer;

[0098] Furthermore, the semantic features of the image are extracted through the MAE model. The pre-trained MAE model is loaded, the image is encoded through the encoder-decoder structure of the MAE model, and then a part of the image is reconstructed through the mask mechanism to obtain the intrinsic semantic features of the image;

[0099] Furthermore, the BERT model parameter volume is set to 110M. Its structure can include 12 Transformer layers, each layer is equipped with 12 self-attention mechanism heads, and the hidden layer dimension can be 768. Install the Transformers dependency library, and on the basis of the Transformers dependency library, load the word segmenter of the BERT model, preprocess the text through the word segmenter, and perform semantic recognition on the preprocessed text through the BERT model to obtain semantic features.

[0100] 3. Construct a multimodal MMoE to refine text features.

[0101] First, the MMoE model can contain 3 groups of expert networks, each of which can be a sub-network model responsible for learning different representations of the input data. The expert networks work in parallel, and each expert network can focus on capturing certain specific features or patterns in the data. The calculation formula for feature refinement through the sub-network model is as follows:

[0102]

[0103] Among them, E i represents the sub-network model, x represents the input, represents the network weight of the ith sub-network model, represents the network bias of the i-th sub-network model, E i (x) represents the output of the i-th sub-network model.

[0104] Furthermore, for each sub-network model, there is a corresponding gating network, and the function of the gating network is to dynamically assign network weights to each sub-network model. The assigned network weights determine the contribution of each sub-network model to the output of the fact-checking result. In an embodiment of the present invention, the gating network softmax function ensures that the sum of all expert weights is 1. The calculation formula corresponding to the gating network can be as follows:

[0105]

[0106] Among them, g k (x) represents the output of the gating function of the kth task, represents the network weight of the gating network for the kth task, represents the network bias of the gating network for the kth task.

[0107]

[0108] Among them, y k (x) represents the final output of the kth task, and n represents the number of sub-network models.

[0109] Furthermore, MMoE is used to analyze the semantic features of images in posts. is , semantic features of the text in the post t and semantic features of background knowledge text k Perform feature refinement to obtain The first three features are used for verification prediction of a single dimension, and the last three features are used for cross-modal consistency learning;

[0110] Furthermore, multi-head self-attention is used to enhance feature representation. For example, text features are calculated separately as well as The multi-head attention feature of For example, the reference formula is as follows:

[0111]

[0112] Among them, Q t represents the query matrix, K t represents the bond matrix, V t represents the value matrix, as well as Represent three different linear transformations respectively.

[0113]

[0114] Among them, Z t express The multi-head attention feature of , H represents the number of heads in the multi-head attention, and h represents the h-th head.

[0115] Furthermore, for and Perform the same operation as above and get The corresponding multi-head attention feature Z k , The corresponding multi-head attention feature Z i .

[0116] Furthermore, a first fusion feature between the image semantic feature and the text semantic feature, and a second fusion feature between the text semantic feature and the background semantic feature are determined through a common attention mechanism. The reference is as follows:

[0117]

[0118] h T-I =GELU(Linear(MLP(Z it ,Z ti )));

[0119] Among them, h T-IIn this embodiment of the present invention, the same operation as above is performed to obtain the second fusion feature h T-K .

[0120] 4. Through MLP and Sigmoid function h T-I and h T-K The feature is mapped to a score, and the fact-checking result of the multimodal post to be checked is determined based on the score.

[0121] Figure 4 FIG. 1 is a flow chart of a target verification model obtained through training according to an embodiment of the present invention. Figure 4 As shown, for the training of the target verification model of the model, the loss function can be calculated based on the binary cross entropy loss. The difference between the model prediction value and the actual label is measured by this function, and the parameters are updated using the Adam optimizer. This optimizer combines the advantages of momentum and adaptive learning rate to improve the stability and convergence speed of the training process. The learning rate can be set according to needs. Repeat the above steps for 50 rounds to obtain the model parameters with the best verification set results, and determine the target verification model based on the parameters.

[0122] The technical solution of the present invention realizes the authenticity verification of multimodal facts; performs fact verification through a target verification model that includes both DM-MMoE and a joint attention mechanism, effectively improving the accuracy of fact verification; introduces retrieval enhancement generation technology, combines multimodal facts and retrieved background knowledge to perform fact verification, effectively improving the target verification model's ability to understand and process multimodal facts to be verified in dynamic and real-time application scenarios, and further ensuring the accuracy of authenticity verification of multimodal facts.

Claims

1. A multimodal fact-checking method based on retrieval-enhanced generation, It is characterized in that include: Acquire a multimodal fact to be verified, wherein the multimodal fact to be verified includes multimedia information and / or fact text information, and the multimedia includes a fact image and a fact video; In the case where the multimodal fact to be verified meets the retrieval enhancement generation condition, the input multimodal fact to be verified is retrieved through the retrieval enhancement generation model to determine the background knowledge text corresponding to the multimodal fact to be verified; The trained target verification model is used to process the input multimodal facts to be verified and the background knowledge text to determine the fact verification result corresponding to the multimodal facts to be verified. The fact-checking result includes factual truth and / or factual anomaly, and the target verification model is obtained by training a preliminary verification model based on multimodal fact samples.

2. The method according to claim 1, It is characterized in that The step of searching the input multimodal fact to be verified by using the retrieval enhancement generation model to determine the background knowledge text corresponding to the multimodal fact to be verified includes: The input multimodal facts to be verified are converted by using a retrieval enhancement generation model and preset prompt words to determine a search statement; Searching the search statement through a target search engine to determine a plurality of search web pages and web page text corresponding to each of the search web pages; The background knowledge text corresponding to the multimodal fact to be verified is determined according to the webpage text corresponding to each of the search webpages.

3. The method according to claim 2, It is characterized in that The determining, based on the webpage text corresponding to each of the search webpages, the background knowledge text corresponding to the multimodal fact to be verified includes: For each webpage text corresponding to the search webpage, determining the degree of relevance between the webpage text and the multimodal fact to be verified; Determining at least one target webpage associated with the multimodal fact to be verified among the plurality of search webpages according to the relevance; The background knowledge text corresponding to the multimodal fact to be verified is determined according to the webpage text corresponding to the target webpage and the preset prompt word.

4. The method according to claim 1, It is characterized in that After obtaining the multimodal facts to be verified, the method further includes: Determining stored knowledge text in a target database that is associated with the multimodal fact to be verified; When the number of the stored knowledge texts does not exceed a preset threshold, the to-be-verified multimodal fact is regarded as a multimodal fact that satisfies the retrieval enhancement generation condition; When the number of texts in the stored knowledge text exceeds a preset threshold, the multimodal fact to be verified is regarded as a multimodal fact that does not meet the retrieval enhancement generation condition.

5. The method according to claim 4, It is characterized in that Also includes: When the multimodal fact to be verified does not satisfy the retrieval enhancement generation condition, the input multimodal fact to be verified and the stored knowledge text are processed by the trained target verification model to determine the fact verification result corresponding to the multimodal fact to be verified.

6. The method according to claim 1, It is characterized in that The target verification model includes a multi-task learning model, and the multi-task learning model includes multiple sub-network models; the target verification model completed by training processes the input multimodal facts to be verified and the background knowledge text to determine the fact verification results corresponding to the multimodal facts to be verified, including: determining pixel change characteristics of the multimedia; Determine the image semantic features corresponding to the multimedia, the text semantic features corresponding to the factual text, and the background semantic features corresponding to the background knowledge text through the plurality of sub-network models; Determine the network weight of each of the sub-network models, and determine the fact verification result corresponding to the multimodal fact to be verified according to the network weight, the pixel change feature, the image semantic feature, the text semantic feature and the background semantic feature.

7. The method according to claim 6, It is characterized in that The target verification model also includes a common attention mechanism; the fact verification result corresponding to the multimodal fact to be verified is determined according to the network weight, the pixel change feature, the image semantic feature, the text semantic feature and the background semantic feature, including: Determining respectively, by the common attention mechanism, a first fusion feature between the image semantic feature and the text semantic feature, and a second fusion feature between the text semantic feature and the background semantic feature; Determine a fact verification result corresponding to the multimodal fact to be verified based on the multiple network weights, the pixel change features, the image semantic features, the text semantic features, the background semantic features, the first fusion features, and the second fusion features.

8. The method according to claim 7, It is characterized in that The determining of a first fusion feature between the image semantic feature and the text semantic feature through the common attention mechanism includes: Determine respectively, by a multi-head attention mechanism, a first multi-head attention feature corresponding to the image semantic feature and a second multi-head attention feature corresponding to the text semantic feature; The first multi-head attention feature and the second multi-head attention feature are fused through a common attention mechanism to obtain a first fused feature.

Images