A multimodal fact-checking method based on causal inference

Abstract

The present invention belongs to the field of natural language processing and multimodal technology, and discloses a multimodal fact-checking method based on causal inference, including: performing cross-modal cross-evidence retrieval on the information to be checked to obtain candidate cross-evidence; sorting the candidate cross-evidence to obtain cross-modal cross-evidence information, including image evidence and text evidence; establishing a causal graph, and building a visual causal model based on the causal graph, inputting the image evidence into the visual causal model, identifying the object-level causal features and non-causal features in the image evidence, and removing the non-causal features to obtain the image causal object features; under the constraints of the causal graph, inputting the image causal object features, the information to be checked and the text evidence into the counterfactual reasoning model to obtain the model prediction results. The present invention can solve complex multimodal information verification problems, provide users with high-quality and accurate answers, and meet the existing and potential needs of various industries.

Description

Technical Field

[0001] The present invention belongs to the field of natural language processing and multimodal technology, and in particular relates to a multimodal fact-checking method based on causal inference. Background Art

[0002] In the digital age, the rapid spread of information has made the generation and spread of fake news a global challenge. Therefore, the development of effective fact-checking technologies is crucial to maintaining a healthy information ecosystem. In recent years, people have increasingly realized that relying solely on the analysis of image-text pairs is not enough to detect false information, and it is usually necessary to integrate and cross-check external information. For example, a popular method is cross-modal cross-retrieval, which combines multimodal information and retrieved evidence for verification. However, these methods often ignore the possible false correlations in the evidence, leading to wrong judgments.

[0003] In recent years, automated false information detection has received significant research attention, exploring methods to identify false information in textual, visual, and multimodal formats. Although multiple datasets have been developed for text-based fact-checking, Glockner et al. pointed out that most datasets use "leaked evidence" from verified articles, which leads to an unrealistic setting that is not suitable for early detection and detection of emerging false information. In the context of multimodal false information, most studies only consider image-text pairs without integrating external information or evidence, with only a few exceptions.

[0004] Abdelnabi et al. used the NewsCLIPings dataset and extended it to include external information, which is simply called NewsCLIPings+. NewsCLIPings is an algorithmically generated dataset that uses a multimodal encoder CLIP and scene and character matching computer vision models to decontextualize image and text pairs. Since the "fake" pairs in NewsCLIPings are algorithmically generated, it ensures that there is no "leaked" evidence. Recent studies have used NewsCLIPings+ for evidence-based multimodal fact-checking, but this algorithmic generation method introduces uncertainty about the model's ability to effectively generalize to real-world multimodal false information.

[0005] Recently, researchers have attempted to train multimodal false information detection models on algorithmically generated datasets such as NewsCLIPings and MEIR and evaluate their generalization capabilities on real-world false information. Due to the presence of unimodal bias in widely adopted benchmarks such as VMU Twitter and COSMOS, Papadopoulos et al. developed VERITE, an evaluation benchmark for real-world multimodal false information detection that takes unimodal bias into account. Therefore, a multimodal fact-checking method based on causal inference is proposed to address the shortcomings of the existing technology. Summary of the invention

[0006] The purpose of this invention is to propose a multimodal fact-checking method based on causal inference, which uses causal graphs and integrates counterfactual reasoning to enable the model to learn true multimodal fact-checking technology, thereby reducing reliance on shortcut reasoning and improving overall accuracy.

[0007] To achieve the above object, the present invention provides a multimodal fact-checking method based on causal inference, comprising:

[0008] Performing cross-modal cross-evidence retrieval on the information to be verified to obtain candidate cross-evidence, wherein the information to be verified includes text modal information and image modal information of the content to be verified;

[0009] Sorting the candidate cross-evidence to obtain cross-modal cross-evidence information, wherein the cross-modal cross-evidence information includes image evidence and text evidence;

[0010] Establishing a causal graph, and building a visual causal model based on the causal graph, inputting the image evidence into the visual causal model, identifying object-level causal features and non-causal features in the image evidence, and removing the non-causal features to obtain image causal object features; wherein the visual causal model is obtained by training with a training set, and the training set includes text information, image information, and answers to fact-checking tasks;

[0011] Under the constraints of the causal graph, the image causal object features, the information to be verified and the text evidence are input into the counterfactual reasoning model to obtain the model prediction result.

[0012] Optionally, cross-modal cross-evidence retrieval is performed on the information to be verified, and candidate cross-evidence is obtained including:

[0013] Based on the image evidence library, the text modality information of the content to be verified is retrieved to obtain candidate image evidence;

[0014] Based on the text evidence library, the image modality information of the content to be verified is retrieved to obtain text candidate evidence;

[0015] The candidate cross evidence is constructed based on the image candidate evidence and the text candidate evidence.

[0016] Optionally, establishing a causal graph, and constructing a visual causal model based on the causal graph includes:

[0017] Establishing the cause-effect graph, wherein the cause-effect graph includes image information, text information, and fused multimodal information, the image information includes image information to be verified and text evidence, and the text information includes text information to be verified and image evidence;

[0018] The mixed factors are backdoor adjusted based on the causal graph to construct the visual causal model.

[0019] Optionally, the method for adjusting the backdoor is:

[0020]

[0021] Among them, P(Y|do(T)) represents the true causal relationship from the text information T to the final prediction result Y, P(Y|T,τ) represents the prediction in each new data segmentation τ, and P(τ) represents the mixed selection probability of specific videos and sentences.

[0022] Optionally, inputting the image evidence into the visual causal model, and identifying object-level causal features and non-causal features in the image evidence includes:

[0023] Using a pre-trained detection convolutional network to detect the image evidence, and cropping the detected bounding box to obtain target visual features;

[0024] A multi-layer perceptron and a long short-term memory network are used to perform temporal embedding encoding on the target visual features to obtain object features;

[0025] Interacting the bounding box and the object category through a non-local network to obtain detected interaction features;

[0026] The object feature and the detected interaction feature are fused through channel connection, and given a question sentence, a word parser and time embedding are used to obtain question embedding;

[0027] The problem embedding and the object features are mapped through the multi-layer perceptron, the score of each object feature is calculated, and the causal attribute labels of the object are classified, thereby obtaining the object-level causal features and non-causal features.

[0028] Optionally, a method of performing spatiotemporal interaction on the bounding box and the object category through a non-local network to obtain a detected interaction feature is:

[0029]

[0030] Among them, L g represents the detected interaction features, NonL(·) represents channel connection, MLP(·) is a multi-layer perceptron, is the bounding box and C is the object category.

[0031] Optionally, the method for classifying the causal attribute labels of the objects is:

[0032]

[0033] in, is a causally related object feature, GS represents Gumbel-Softmax, q g is the problem embedding, represents matrix multiplication, is the object feature, Indicates multiplication by index, is the target visual feature.

[0034] Optionally, the visual causal model is trained by a training set to satisfy a predefined causal formula, and the causal formula is:

[0035]

[0036] Among them, L c is the cross entropy loss, XE is the cross entropy loss, The calculation model is input (T c ,I) after the predicted label, T c Refers to text information, including the text information to be verified T v Object features that are causally relevant I refers to the image information, including the image information to be verified i v and textual evidence T e , a is the label label.

[0037] Optionally, the model prediction result is:

[0038]

[0039] in,

[0040]

[0041] In the formula, Y represents the news label, To predict the label based on the text modality only, To predict the label based only on the image modality, To predict labels based on multimodal information at the same time, C is the fused multimodal feature. In the reasoning path T→Y, T is the text information, I is the image information, t, c * 、i * They represent the case where t is set to the initial value and processed in the factual scenario, and c, i are blocked and c in the counterfactual scenario. * 、i * For situations that are not handled, the same applies to other reasoning paths.

[0042] Optionally, after obtaining the model prediction results, the following is also included:

[0043] The model prediction result is judged. If the model prediction result is true, the fact is correct, otherwise, the fact is wrong.

[0044] The present invention has the following beneficial effects:

[0045] By utilizing causal graphs and integrating counterfactual reasoning, the proposed method enables the model to learn true multimodal fact-checking technology, thereby reducing reliance on shortcut reasoning and improving overall accuracy; a large number of experiments conducted on benchmark datasets show that the method of the present invention is effective in improving fact-checking performance and mitigating the impact of false correlations.

[0046] From a causal perspective, the present invention uses causal tools to remove non-causally related parts from image evidence, reducing the model's reliance on false correlations, thereby making more accurate decisions; the counterfactual reasoning model uses shortcut reasoning as the manifestation of two direct causal effects in the results, providing a new perspective and technology to learn true multimodal reasoning. Compared with various benchmark methods, the present invention has demonstrated significant progress in fact-checking tasks. At the same time, the counterfactual reasoning model does not rely on a specific model and can be applied to many fact-checking architectures to reduce shortcut reasoning; by combining causal reasoning and counterfactual reasoning techniques, the present invention can solve complex multimodal information verification problems, provide users with high-quality and accurate answers, and meet the existing and potential needs of various industries. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0048] Figure 1 An example diagram of the NewsCLIPings+ dataset proposed in an embodiment of the present invention;

[0049] Figure 2 This is a flow chart of a multimodal fact-checking method based on causal inference according to an embodiment of the present invention;

[0050] Figure 3 A causal graph for fact checking proposed in an embodiment of the present invention, wherein (a) is the proposed causal graph, and (b) is a causal graph with interference factors;

[0051] Figure 4 A schematic diagram of a visual causal model proposed in an embodiment of the present invention;

[0052] Figure 5 The predicted labels in different scenarios proposed by the embodiments of the present invention, where (a) is a possible scenario of disconnected reasoning, where labels are predicted based only on the text modality, (b) is another possible disconnected reasoning scenario, where labels are predicted based only on the image modality, and (c) is true multimodal reasoning, where labels are predicted based on information obtained from multimodal fusion;

[0053] Figure 6 This is an example diagram of the data set proposed in the embodiment of the present invention. DETAILED DESCRIPTION

[0054] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0055] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0056] A common problem in previous fact-checking methods is that they assume that all retrieved evidence is valid, without considering potential false associations. Figure 1 As shown, the image and the image evidence are unrelated, yet the model of previous methods learned the wrong pattern, that is, both the image and the image evidence show the ocean, which leads to the wrong decision result. In addition, although both the text and the image are correct, they come from different news and become fake news when combined together. Therefore, the model needs to detect both the image and the text to determine whether the news is fake news. Therefore, the visual evidence may contain object features that are unrelated to the text information, and a simple matching algorithm may lead the model to draw incorrect conclusions. This limitation highlights the need for more sophisticated techniques to distinguish between causally related parts of the evidence and spurious related parts.

[0057] To address these challenges, this paper proposes a multimodal fact-checking method based on causal inference, which uses causal tools to identify and eliminate potentially misleading parts of evidence. By explicitly capturing visual semantic features that are causally related to the text, the impact of non-causal features on the model is reduced, thereby revealing the true association. Figure 1 As shown, the part (boxed area) that is causally related to the text information is marked, and other non-causal features are removed. The method of the present invention aims to improve the generalization ability of the model by eliminating misleading correlations that may affect decision-making, so that the model can learn a true multimodal fact-checking method.

[0058] In addition, the present invention also highlights another common problem in current fact-checking models, namely the reliance on shortcut reasoning. This reasoning makes judgments based on information from only a single modality, ignoring the complexity of multimodal evidence. Figure 1 As shown in the figure, the text information "Tiffany is located northeast of Aberdeen" is true, and the image is also true. However, there is a cruise ship in the picture. When only single-modal information is considered, everything seems to be accurate. However, multimodal information is inconsistent. If the model relies on shortcut reasoning, the judgment of such information is easily wrong. To address this limitation, a counterfactual reasoning model is introduced, which distinguishes the true causal effects in shortcut reasoning and multimodal reasoning by explicitly modeling causal relationships.

[0059] like Figure 2 As shown, this embodiment provides a multimodal fact-checking method based on causal inference, including the following steps:

[0060] Perform cross-modal cross-evidence retrieval on the information to be verified to obtain candidate cross-evidence, wherein the information to be verified includes text modal information and image modal information of the content to be verified;

[0061] Furthermore, cross-modal cross-evidence retrieval is performed on the information to be verified, and candidate cross-evidence is obtained including:

[0062] Based on the image evidence database, the text modal information of the content to be verified is retrieved to obtain candidate image evidence;

[0063] Based on the text evidence database, image modality information of the content to be verified is retrieved to obtain text candidate evidence;

[0064] The image candidate evidence and the text candidate positive evidence constitute the candidate cross evidence.

[0065] Sorting candidate cross-evidence to obtain cross-modal cross-evidence information, wherein the cross-modal cross-evidence information includes image evidence and text evidence;

[0066] Specifically, the retrieved evidence, i.e., candidate cross-evidence, is sorted to obtain TOP-K cross-modal cross-evidence information.

[0067] Establish a causal graph, and build a visual causal model based on the causal graph, input image evidence into the visual causal model, identify object-level causal features and non-causal features in the image evidence, remove non-causal features, and obtain image causal object features; the visual causal model is obtained by training a training set, which includes text information, image information, and answers to fact-checking tasks;

[0068] Under the constraints of the causal graph, the image causal object features, the information to be verified and the text evidence are input into the counterfactual reasoning model to obtain the model prediction results.

[0069] Specifically, expert knowledge is used to construct a causal graph, and backdoor adjustments are made to mixed factors based on the causal graph to build a visual causal model. The visual causal model targets the image evidence part of the cross-modal cross-evidence information of TOP-K, identifies object-level causal features in the image, removes redundant non-causal object features in the image, and obtains image causal object features. Under the constraints of the causal graph, the part containing only the image causal object features, as well as the text evidence and the original information to be verified, are sent to the counterfactual reasoning module; through the counterfactual reasoning model, the reasoning shortcuts are forced to be eliminated, that is, the use of single modality information for prediction is avoided to obtain the model prediction results.

[0070] Furthermore, establishing a causal graph and constructing a visual causal model based on the causal graph includes:

[0071] Establishing a causal graph, wherein the causal graph includes image information, text information, and fused multimodal information, the image information includes image information to be verified and text evidence, and the text information includes text information to be verified and image evidence;

[0072] Backdooring the confounding factors based on the causal graph and building a visual causal model.

[0073] Furthermore, the image evidence is input into the visual causal model to identify object-level causal features and non-causal features in the image evidence, including:

[0074] Use a pre-trained detection convolutional network to detect image evidence and crop the detected bounding box to obtain the target visual features;

[0075] Multi-layer perceptron and long short-term memory network are used to perform temporal embedding encoding on the target visual features to obtain the object features;

[0076] The bounding boxes and object categories are interacted through a non-local network to obtain the detected interaction features;

[0077] The object features and the detected interaction features are fused through channel connection, and given a question sentence, the question embedding is obtained using the word parser and time embedding;

[0078] The problem embedding and object features are mapped through a multi-layer perceptron, the score of each object feature is calculated, and the causal attribute labels of the object are classified to obtain object-level causal features and non-causal features.

[0079] Specifically, the task of multimodal fact-checking (MFC) is defined as follows: Given a dataset in, Represents the picture and text pair to be verified. and Represents an image evidence array containing M elements and a text evidence array containing M elements, respectively. represents the overall decision label, i.e. true information (0) or false information (1). Its main goal is to train a classifier f:

[0080] The multimodal causal relationships in fact-checking tasks are formalized through structured causal models. Figure 3 As shown in (a), nodes T, I, and C represent text information, image information, and fused multimodal information, respectively. According to the proposed causal graph, the final prediction result Y receives input from three branches: the direct influence of text T and image I input on Y, T→Y and I→Y, and the indirect influence of text T and image I input on Y through fused feature C, that is, T(I)→C→Y.

[0081] We formalize the cross-modal causal relationships of the visual causal module through structured causal modeling. Figure 3 In (b), the key factors in the multimodal fact-checking task are represented as nodes in the causal graph, and the relationships between the factors are represented as edges constructed by the current model by default, including text information T, image information I, fused multimodal information C, label Y and confounding factors U, where the confounding factor is the image evidence I e The non-causally related part.

[0082] T←U→Y indicates that the spurious correlation between the confounding factor U of the retrieved image evidence and the predicted label Y is based on the textual information T. Specifically, objects in the visual evidence that are unrelated to the textual form may be wrongly associated with the answer through simple image information matching.

[0083] As is known to all, backdoor adjustments help to remove spurious correlations, thus improving the generalization ability of the model. The true causal relationship from T to Y is expressed as P(Y|do(T)), where do(·) represents the intervention operation. P(Y|do(T)) can be obtained by transforming the original training data into new data T = {τ1,...,τ h} to cut off the link from U→T to prevent this backdoor path. Each τ represents a confounding layer (a total of h), guiding the model to find invariance from confounding factors. The backdoor adjustment can be expressed as:

[0084] P(Y|do(T))=∑ τ∈T P(Y|T,τ)P(τ) (1)

[0085] Among them, P(Y|T,τ) represents the prediction in each new data segmentation τ, and P(τ) represents the probability of mixed selection of a specific video and sentence, which is calculated as h is the number of hybrid layers. The probability estimates of the image are calculated through two independent architectures. In this way, the interaction is cut off. Figure 3 The confounding effect of T←U→Y in (b) is shown in Figure 2. Taking image data as an example, it is expensive to traverse the confounding factors of all images. When the number of images in the dataset is M, formula (1) expands the training data from M to M in one training cycle. 2 , increasing memory consumption. Therefore, in order to balance the size of the confounding set T and the training speed, the confounding factor is found in each cycle and combined with the original data T. After K training cycles, the visual part of the equation can be approximately adjusted, that is:

[0086] In order to help the model accurately identify causal-related features in fact-checking tasks, a visual causal model is used to disentangle confounding factors and causal factors at the object level. Figure 4 As shown in the figure, before the image evidence information is fed into the backbone model for multimodal fact-checking training, a visual causal model is proposed to identify object-level causal features and non-causal features in image evidence. First, the image-level visual features are divided into object-level visual features by the detected bounding boxes. The category information, location information and interaction relationship of each object are modeled by an interaction detector. The indicator determines whether the features of different objects are causally related or irrelevant based on the text information prompts; then the causal features are fed into the backbone model for further prediction.

[0087] For image evidence I e , a pre-trained detection convolutional network (Detection ConvNet) is used to detect object-level locations and treat them as bounding boxes Where N is the number of detected objects. Then, the corresponding regions of interest (ROIs) are cropped from the detected bounding boxes to obtain the target visual features. The expression is

[0088] For each object feature, a multi-layer perceptron (MLP) and a long short-term memory network (LSTM) are used to temporally embed the visual features:

[0089]

[0090] in, is the global N object representation, and d is the hidden dimension of the LSTM. In addition to the visual encoding, bounding box position correlation is also modeled in an interaction detection flow Object categories C, and the spatiotemporal interactions between and within objects. Implemented by a nonlocal network with MLP:

[0091]

[0092] Among them, NonL(·) represents channel connection, Represents the detected interaction features. The outputs of the two feature extraction streams are then fused through channel connections: Among them, all objects are candidates for causal or irrelevant objects, and their identities are determined by the semantics of the question. Therefore, given a question sentence, word parser and event embedding are applied to obtain the global question representation q g , that is, question embedding, which is implemented through LSTM:

[0093] q g =LSTM(Parser(q)) (4)

[0094] The parser uses the pre-trained GloVe model, and the LSTM is implemented by the question embedding model of the MFC backbone. For a specific problem, the causally related objects are usually a collection of objects O g , and are internally interconnected. Reasoning about its relationship with irrelevant objects is redundant and prone to spurious associations. Therefore, hard segmentation is used in the indicator to explicitly divide object features into causally related features and irrelevant features, rather than soft GCN reasoning or attention-based models. Specifically, the question is embedded in q via MLP g and object features O g The indicator computes the score of each object feature and uses Gumbel-Softmax to classify the causal attribute label of the object:

[0095]

[0096] Among them, GS represents Gumbel-Softmax, · represents matrix multiplication, Indicates multiplication by index.

[0097] In order to ensure the domain consistency of the training set and the test set, the backbone model is trained using the original data (T, I, A), where A is the answer to the fact-checking task, T refers to the text information, and I refers to the image information. The backbone model loss L o as follows:

[0098]

[0099] Where XE is the cross entropy loss, Calculate the predicted label of the model after input (T, I), where T refers to the text information, including the text information to be verified T v and image evidence I e , and I refers to the image information, including the image information to be verified i v and textual evidence T e .

[0100] The visual causal module aims to capture both causal object features and non-causal object features for further robust prediction. During the training phase, the causal visual features have the same ability as question answering, which is subject to the cross entropy loss L c The formula of causal visual feature loss is as follows:

[0101]

[0102] L=L o +λL c (8)

[0103] Among them, T c Represents the text information feature T related to the text information v and object evidence characteristics

[0104] In order to separate the impact of the shortcut path from the total causal effect, counterfactual causal inference is used to block out other effects. Figure 5 As shown in (a) in , to model T→Y, the counterfactual formula is used:

[0105]

[0106] Formula (9) describes the situation when T is set to the original value t and C and I are blocked. Here c * and i * is the counterfactual marker. t and c * ,i * represent the treatment of t in the factual scenario and c in the counterfactual scenario. * ,i *If the situation is not handled. Figure 5 (b) and Figure 5 As shown in (c) in , the definitions of the other two effects are the same:

[0107]

[0108] and

[0109]

[0110] Causal effect: According to the counterfactual definition, the total effect of t, i, c on Y can be decomposed into the natural direct effect of T on Y and the natural direct effect of I on Y discussed earlier and the effect of C on Y. These two natural direct effects lead to the disconnected reasoning problem. The effect of C→Y is the expected multi-hop reasoning.

[0111] like Figure 5 (a) and Figure 5 As shown in (b) of the above, the effect of T on Y when I and C are blocked and the effect of I on Y when T and C are blocked can be easily obtained by setting T or I to counterfactual values. However, the effect of C on Y cannot be obtained by changing T / I to T * ,I * To obtain. Use the total indirect effect (TIE) to express the effect of C on Y, the formula is:

[0112]

[0113] The main problem is is unknown. It is also difficult to use counterfactual samples of T, I, C to obtain its value, because T, I should be the factual value of t, i. In the embodiment, it is assumed that the model will * The probability of guessing the output under no processing conditions is expressed as:

[0114]

[0115] Where C is the output and a learnable parameter.

[0116] Training Objective: From the above discussion, the total causal effect can be estimated:

[0117]

[0118] It should be noted that Y can be the true label of any answer span prediction, supporting fact identification, or answer type prediction. In the fact-checking task, Y represents the news label (true or false).

[0119] Reasoning: As mentioned above, the goal of this method is to exclude natural direct effects (T→Y, I→Y) and use the real multi-hop effect C→Y to reduce the shortcut reasoning of the fact-checking model. Therefore, the following formula is used for reasoning:

[0120] Y t,i,c -C (15)

[0121] Parameterization of the neural network fact-checking model F:

[0122] F(t,i)-C (16)

[0123] The time consumption of this method is equivalent to that of the existing method.

[0124] Specifically, after obtaining the model prediction result, the following step is further included: judging the model prediction result. If the model prediction result is true, the fact is correct; otherwise, the fact is wrong.

[0125] verify:

[0126] To evaluate the generalization ability of our method in real-world multimodal false information, we train the model on the NewsCLIPings+ dataset and evaluate its performance on the Out-Of-Context (OOC) pairs of the VERITE benchmark. The VERITE dataset is divided into three categories: True, OOC, and MisCaptioned (MC), which represent true information, true text-image pairs from different news (false information caused by inconsistency between text and image), and correct image information but fake text information. "External evidence information" is collected through Google API.

[0127] Assuming that NewsCLIPings+ and VERITE follow different distributions, the former contains algorithmically generated samples and the latter contains real-world false information, causal tools can capture causal features, enabling the model to learn a true multimodal fact-checking method and improve the generalization ability of the model. “True vs. OOC Accuracy” is reported on the VERITE dataset, that is, the accuracy in the out-of-context (OOC) category.

[0128] A unified feature extraction method was applied to both datasets. The evidence was ranked and then the top two evidence images were selected for each instance. Each evidence image was processed to detect up to six objects, and only objects with a detection score higher than 0.9 were retained. In the object detection module, the labeled representation of text information was provided by GloVe, and the multi-layer perceptron (MLP) was implemented through fully connected layers. This method was trained for a maximum of 100 epochs, with an early stopping strategy at 10 epochs, and the Adam optimizer was used with a learning rate set to lr=1e-4. When tuning the hyperparameters, the following configuration was considered: t=6 Transformer layers, z=2048 as the dimension of the feedforward network, and h=8 as the number of attention heads. The dropout rate was set to 0.1 and the batch size was 256.

[0129] In the Counterfactual Reasoning model, the reference state * is defined as the state where the signal of the text modality or image modality is masked, that is, t and i are not given (null values). To achieve this, a tensor filled with scalar values ​​0 is used to represent the reference value t * and i * . In this way, the input does not contain any semantic information. For example, t * Refers to the masked text information and image evidence retrieved based on text information, filled with scalar value 0. Similarly, i * Refers to masking image information and text evidence based on image information retrieval.

[0130] As shown in Table 1, the accuracy of Truevs.OOC is reported on VERITE with M texts and K images as evidence for training, and the proposed method significantly improves the performance compared with other methods. These results show that the proposed method can more effectively capture the model with better generalization ability to real-world out-of-context false information (VERITE) from algorithmically generated samples (NewsCLIPings+).

[0131] Table 1

[0132]

[0133] In previous methods, the performance is best when M,K=1, while the performance decreases when M,K>1 introduces additional but less relevant evidence. Table 1 shows that our method shows comparable results to M,K=1 when M,K=2 by removing non-causal features from the evidence, which proves the effectiveness of our method.

[0134] In order to eliminate the false correlation that may lead to model judgment bias and cut off the influence of shortcut reasoning T→Y and I→Y, the effectiveness of the method is verified by removing the visual causal model and the counterfactual reasoning model respectively. As shown in Table 2, the ablation experiment results of different models are shown, where -vcm means removing the visual causal model (Visual CausalModule) from the method. and Represents the exclusion of single shortcut reasoning paths I→Y and T→Y, respectively, and -cfI represents the removal of the counterfactual reasoning model. After removing the visual causal model, in addition to the overall performance degradation, the effect of using M,K=2 is worse than that of using M,K=1 due to the introduction of more confounding factors.

[0135] Table 2

[0136]

[0137] For the counterfactual reasoning model, an ablation experiment is performed by excluding the influence of one shortcut reasoning to verify the effectiveness of the model proposed in this invention. The effect of reducing shortcut reasoning by excluding one of T→Y or I→Y is explored. As shown in Table 2, it can be seen that excluding one shortcut reasoning path does not effectively improve our experimental results. Removing the dual influence of T→Y and I→Y performs better in reducing shortcut reasoning, because if one shortcut reasoning is blocked, the model can always use another shortcut reasoning.

[0138] like Figure 6 Figure 2 shows two examples from the dataset. In the left half of the image, the visual causal model detects the causally relevant parts of the image evidence. The model uses causal reasoning to exclude possible spurious related features in the image, which, if not excluded, may mislead the model to make incorrect judgments. For example, in a picture containing multiple objects, the model will identify those objects that are truly related to the text information, while excluding irrelevant background or minor objects. In the right half of the image, the counterfactual reasoning model combines information from text and images for multimodal reasoning. By explicitly modeling causal relationships, the model is able to distinguish between the direct causal effects of shortcut reasoning and the true causal effects of multimodal reasoning, thereby avoiding the model relying solely on a single modal information to make judgments. For example, when verifying the authenticity of a piece of news, the module not only considers the objects in the picture, but also combines the text information to ensure the accuracy and reliability of the judgment.

[0139] For example, when a person is playing table tennis next to a TV in a room, the model can easily identify the TV and then understand that a person is watching TV. The model of this method identifies people, table tennis and other causally related objects, and excludes interference factors such as the TV. The same person was playing games at first, and then went to watch TV. It is wrong to say that he was playing games in the scene of watching TV. Playing games is real, and the scene of watching TV is also real, but they are not the same thing happening at the same time. The model of the present invention can learn real knowledge based on scenes and events at the same time, so as to verify the authenticity of events.

[0140] Some data samples are randomly selected for experiments, wherein in 90% of the samples, the assumption is reasonable, and 10% of the data samples encounter a situation where the object cannot be detected, so the present invention uses the entire image features of the evidence image itself.

[0141] Application scenario (multimodal data to be verified is fact-checked in an existing credible evidence database, which includes existing multimodal factual data, such as real multimodal news, archives, etc.)

[0142] (1) News and media industry;

[0143] Application scenarios: Journalists, media organizations, and fact-checkers need to verify multimodal information (text, images) in news reports to verify their authenticity.

[0144] Input: User question (e.g., “Is this photo relevant to the event depicted and does it support the authenticity of the event?”).

[0145] Output: Verification results, including causal correlation analysis of image and text information, and detailed description of excluding false correlation features.

[0146] Effect: Quickly and accurately verify multimodal information in news reports, and improve the credibility and accuracy of news reports.

[0147] (2) Social media and public relations industry;

[0148] Application scenario: Social media platforms and public relations experts need to verify the multimodal content posted by users to prevent the spread of false information.

[0149] Input: User questions (e.g., “Do the pictures and descriptions in this post match, or are they misleading?”).

[0150] Output: Verification results, including causal analysis of images and descriptions, and identification of spurious correlation features.

[0151] Effect: Effectively prevent the spread of false information, improve the credibility of platform content, and maintain the public image.

[0152] (3) Government and legal professions;

[0153] Application scenarios: Government agencies and legal experts need to verify the authenticity of multimodal evidence in cases to support decision-making and legal judgments.

[0154] Input: User questions (such as: "Are the images and text descriptions in this evidence consistent? Can they be used as valid evidence?").

[0155] Output: Verification results, including detailed analysis of causal reasoning to determine the consistency and relevance of images and text descriptions.

[0156] Effect: Provide reliable evidence verification, support legal judgments and policy making, and ensure the fairness and accuracy of decision-making.

[0157] (4) Scientific research and education industry;

[0158] Application scenarios: Researchers and educators need to verify the multimodal information in research data to ensure the reliability of research results.

[0159] Input: User questions (e.g., “Do the images and text in this research paper match? Is there any data bias?”).

[0160] Output: Verification results, including causal analysis and counterfactual reasoning results, identifying potential data biases and spurious correlation features.

[0161] Effect: Ensure the accuracy of research data and improve the credibility and quality of academic research.

[0162] Input summary:

[0163] Users input queries into the system by asking natural language questions. These questions can involve news reports, social media content, legal evidence, or the authenticity verification of scientific research data. For example:

[0164] “Does the photo relate to the events depicted and support the reality of the events?”

[0165] “Do the pictures and descriptions in this post match, or are they misleading?”

[0166] “Are the pictures and text descriptions in this evidence consistent? Can they be used as valid evidence?”

[0167] “Do the images and text in this research paper match? Is there any data bias?”

[0168] Output summary:

[0169] The system provides accurate answers and relevant background information through causal reasoning and counterfactual reasoning modules. The output includes:

[0170] Causal correlation analysis between image and text information.

[0171] Detailed description of the features to exclude spurious correlations.

[0172] Causal analysis and counterfactual reasoning results.

[0173] By using this method, users can quickly and accurately verify the authenticity of multimodal information and meet various query needs. The specific effects include: improving user experience. The present invention can quickly process and answer user questions and improve user satisfaction; saving time and energy. Users do not need to search through a large amount of information and literature. The present invention automatically completes information screening and reasoning; supporting decision-making and research, providing reliable data support for journalists, legal experts, scientific researchers, etc., to help them conduct more in-depth research and reporting; wide application potential. In addition to news and social media, this method can also be extended to government, law, scientific research and other fields to meet the needs of more industries and users.

[0174] Existing or potential demand for services:

[0175] 1) News and media industry needs;

[0176] Target users: Journalists, media organizations, fact-checkers.

[0177] Requirement: Quickly and accurately verify multimodal information in news reports to improve the credibility and accuracy of news reports.

[0178] 2) Social media and public relations industry needs;

[0179] Target users: Social media platforms, public relations experts.

[0180] Requirements: Effectively prevent the spread of false information, improve the credibility of platform content, and maintain public image.

[0181] 3) Government and legal industry needs;

[0182] Target users: Government agencies, legal experts.

[0183] Requirements: Provide reliable evidence verification, support legal judgments and policy making, and ensure the fairness and accuracy of decision-making.

[0184] 4) Scientific research and education industry needs;

[0185] Target users: researchers and educators.

[0186] Need: Ensure the accuracy of research data and improve the credibility and quality of academic research.

[0187] By combining causal reasoning and counterfactual reasoning techniques, the present invention can solve complex multimodal information verification problems, provide users with high-quality and accurate answers, and meet the existing and potential needs of various industries.

[0188] The above is only a preferred specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A multimodal fact-checking method based on causal inference, characterized in that: include: Performing cross-modal cross-evidence retrieval on the information to be verified to obtain candidate cross-evidence, wherein the information to be verified includes text modal information and image modal information of the content to be verified; Sorting the candidate cross-evidence to obtain cross-modal cross-evidence information, wherein the cross-modal cross-evidence information includes image evidence and text evidence; Establishing a causal graph, and building a visual causal model based on the causal graph, inputting the image evidence into the visual causal model, identifying object-level causal features and non-causal features in the image evidence, and removing the non-causal features to obtain image causal object features; wherein the visual causal model is obtained by training with a training set, and the training set includes text information, image information, and answers to fact-checking tasks; Under the constraints of the causal graph, the image causal object features, the information to be verified and the text evidence are input into a counterfactual reasoning model to obtain a model prediction result; The step of establishing a causal graph and constructing a visual causal model based on the causal graph includes: Establishing the cause-effect graph, wherein the cause-effect graph includes image information, text information, and fused multimodal information, the image information includes image information to be verified and text evidence, and the text information includes text information to be verified and image evidence; Performing backdoor adjustments on mixed factors based on the causal graph to construct the visual causal model; Inputting the image evidence into the visual causal model, identifying object-level causal features and non-causal features in the image evidence includes: Using a pre-trained detection convolutional network to detect the image evidence, and cropping the detected bounding box to obtain target visual features; A multi-layer perceptron and a long short-term memory network are used to perform temporal embedding encoding on the target visual features to obtain object features; Interacting the bounding box and the object category through a non-local network to obtain detected interaction features; The object feature and the detected interaction feature are fused through channel connection, and given a question sentence, a word parser and time embedding are used to obtain question embedding; Mapping the problem embedding and the object features through the multi-layer perceptron, calculating the score of each object feature, and classifying the causal attribute labels of the object, thereby obtaining the object-level causal features and non-causal features; The model prediction results are: in, In the formula, Y represents the news label, To predict the label based on the text modality only, To predict the label based only on the image modality, To predict labels based on multimodal information at the same time, C is the fused multimodal feature, T is the text information, and I is the image information. In the reasoning path T→Y, t, c * 、i * They represent the case where t is set to the initial value and processed in the factual scenario, and c, i are blocked and c in the counterfactual scenario. * 、i * Situation that was not addressed.

2. The multimodal fact-checking method based on causal inference as claimed in claim 1, characterized in that: Cross-modal cross-evidence retrieval is performed on the information to be verified, and candidate cross-evidence is obtained including: Based on the image evidence library, the text modality information of the content to be verified is retrieved to obtain candidate image evidence; Based on the text evidence library, the image modality information of the content to be verified is retrieved to obtain text candidate evidence; The candidate cross evidence is constructed based on the image candidate evidence and the text candidate evidence.

3. The multimodal fact-checking method based on causal inference as claimed in claim 1, characterized in that: The method for performing the backdoor adjustment is: Among them, P(Y|do(T)) represents the true causal relationship from the text information T to the final prediction result Y, P(Y|T, τ) represents the prediction in each new data segmentation τ, and P(τ) represents the mixed selection probability of specific videos and sentences.

4. The multimodal fact-checking method based on causal inference according to claim 1, characterized in that: The method of performing spatiotemporal interaction on the bounding box and the object category through a non-local network to obtain the detected interaction features is: Among them, L g represents the detected interaction features, NonL(·) represents channel connection, MLP(·) is a multi-layer perceptron, is the bounding box and C is the object category.

5. The multimodal fact-checking method based on causal inference as claimed in claim 1, characterized in that: The method to classify the causal attribute labels of an object is: in, is a causally related object feature, GS represents Gumbel-Softmax, q g is the problem embedding, represents matrix multiplication, is the object feature, ο means multiplication by index, is the target visual feature.

6. The multimodal fact-checking method based on causal inference according to claim 1, characterized in that: The visual causal model is trained by the training set to satisfy a predefined causal formula, which is: Among them, L c is the result of cross entropy loss calculation, XE is the result of cross entropy loss calculation, The calculation model is input (T c , I) after the predicted label, T c Refers to text information, including the text information to be verified T v Object features that are causally relevant I refers to the image information, including the image information to be verified i v and textual evidence T e , a is the label label.

7. The multimodal fact-checking method based on causal inference as claimed in claim 1, characterized in that: After obtaining the model prediction results, it also includes: The model prediction result is judged. If the model prediction result is true, the fact is correct, otherwise, the fact is wrong.

Images