Multi-modal rumor detection method based on anti-factual reasoning and causal intervention

Abstract

The invention discloses a multi-modal rumor detection method fusing texts, images and social propagation structures, and belongs to the technical field of natural language processing, computer vision and causal reasoning. Specifically, the invention provides a unified causal inference framework, and hybrid deviation in multi-modal data is effectively stripped by integrating text anti-fact causal inference and an image dot product causal intervention mechanism. Under the framework, social propagation structure features are further fused, and a multi-head collaborative attention mechanism is adopted, so that deep alignment and semantic enhancement in cross-modal features are realized. Adversarial samples are generated through projection gradient descent for adversarial training, and model parameters are optimized in combination with anti-fact loss, so that the classification accuracy and generalization ability of the model are improved. According to the rumor detection method, a causal reasoning normal form is introduced into a rumor detection task, the effectiveness of an anti-fact and intervention mechanism in a complex information scene is verified, and a new theoretical support and method path are provided for constructing a credible multi-modal information system.

Description

Technical Field

[0001] This invention relates to the fields of natural language processing, computer vision, and causal reasoning, and specifically to a multimodal rumor detection method that integrates text, images, and social communication structures, belonging to the category of artificial intelligence information credibility recognition technology. Background Technology

[0002] With the rapid development of social networks, the way information is disseminated has undergone profound changes. However, at the same time, the generation and spread of malicious information such as rumors have become increasingly convenient. In public emergencies, vividly illustrated false information can quickly trigger social panic, causing serious public safety and health risks. To address this problem, researchers have proposed multimodal rumor detection technology, which has evolved from early rule-based and traditional classification methods to deep learning systems that integrate text, images, and social behavior features. In recent years, the introduction of large language models has improved semantic understanding capabilities, and cross-modal attention mechanisms, contrastive learning, and optimal transport methods have significantly enhanced intermodal collaboration and representation alignment. However, most existing methods are based on statistical correlation learning and have failed to effectively characterize the causal mechanisms of rumor generation and spread, resulting in insufficient robustness and interpretability in real-world complex scenarios.

[0003] The introduction of causal inference provides a new theoretical perspective for rumor detection research, helping to overcome the reliance of traditional methods on surface correlation and improve the generalization and interpretability of models. Early causal methods mainly focused on a single modality, such as using causal graphs to model language style biases in text or estimating the causal effect of user behavior on social features through instrumental variables. However, multimodal information is widely intertwined in the social media environment, making single-modal causal modeling difficult to handle. To address this, researchers have proposed multimodal causal inference methods, such as counterfactual generation, multi-perspective causal learning, and mediator variable modeling, exploring preliminary approaches to modeling cross-modal causal effects. However, most existing methods rely on traditional causal adjustment strategies, which are disconnected from deep learning representation learning, thus remaining insufficient in end-to-end modeling and modal co-optimization.

[0004] Based on existing research, multimodal rumor detection faces two main challenges: First, methods generally rely on statistical correlations between modalities rather than true causality, making them susceptible to training data bias, data selection bias, and confirmation bias, resulting in limited generalization ability when dealing with out-of-distribution data. Second, existing models often independently model text, image, and social media propagation features, neglecting potential confounding paths and interaction biases among them, thus limiting the model's robustness and interpretability. Particularly in the image and text modalities, there are pseudo-correlation problems such as language-induced bias and contextual confounding bias; if not addressed, these can easily lead to misjudgments of information authenticity.

[0005] To overcome the aforementioned problems, achieve end-to-end multimodal causal representation learning, and improve the robustness and interpretability of rumor detection, this invention proposes a multimodal rumor detection method that integrates counterfactual generation and causal intervention. Summary of the Invention

[0006] The purpose of this invention is to address the shortcomings of existing multimodal rumor detection methods, which generally rely on statistical correlation and neglect causal mechanism modeling. This invention proposes a multimodal rumor detection method (CCMRD) based on counterfactual reasoning and causal intervention. This method can systematically model the causal paths between text, image, and social media communication modalities, effectively mitigating spurious correlations and confounding biases, thereby improving the robustness, generalization ability, and interpretability of the detection model.

[0007] To achieve the above objectives, the technical problems to be solved by this invention include: first, how to jointly model multimodal inputs from a causal perspective to eliminate language-induced bias and contextual confounding bias in text and image modalities; second, how to extract the true causal effects between modalities through counterfactual reasoning and causal intervention mechanisms to avoid model dependence on surface correlations; and third, how to achieve end-to-end causal representation learning within a deep learning framework, thereby achieving the unification of multimodal collaborative optimization and causal reasoning.

[0008] To address the aforementioned technical problems, this invention proposes a multimodal rumor detection method (CCMRD) based on counterfactual reasoning and causal intervention. The technical solution includes the following steps: Step 1: Multimodal feature extraction;

[0009] Step 1.1 Text modality feature extraction: A convolutional neural network with pooling operation is used to extract semantic features of text sentences. ,in The text information is used for rumor detection, where B is the batch size, L is the sequence length, and D is the embedding dimension.

[0010] Step 1.2 Image modality feature extraction: Image features are extracted using a pre-trained ResNet-50. ,in Image information for rumor detection.

[0011] Step 1.3 Propagation Structure Graph Feature Extraction: Based on the graph structure (adjacency matrix) A and node embeddings V, a graph attention network is used to generate graph features. ,in For a set of nodes, These represent post nodes, user nodes, and comment nodes, respectively. This is the initial edge matrix.

[0012] Step 2: Counterfactual causal inference of text. By assessing the importance of text features, a mask is generated to cover some features in order to construct a counterfactual scenario and then analyze the causal effect of the model.

[0013] Step 2.1 Evaluate the feature importance of the text; Calculate the importance score S, which reflects the contribution of features to the prediction, using a trainable neural network. The importance score S is calculated, where For text feature vectors, , , , For trainable parameters, This refers to the Sigmoid function.

[0014] Step 2.2 Apply masks to generate counterfactual features. Based on the importance score S, select the Top-K method to generate masks. The mask ratio is controlled by mask_ratio. It refers to the first The first sample Whether a word is masked is determined by... The calculation yielded, where Next application The mask generates counterfactual features by setting important features to zero, simulating a counterfactual scenario where key features are removed.

[0015] Step 2.3 Construct the counterfactual loss function. By calculating the loss between the counterfactual prediction and the original prediction, the causal reasoning ability of the model is optimized. The counterfactual loss function considers three aspects of loss: causal effect, consistency, and importance regularization.

[0016] Step 2.3.1 Loss of Causal Effect. Loss of Causal Effect Furthermore, the differences in predicted distributions (KL) and predicted categories are also considered. and confidence level difference Composition. The difference in the predicted distribution, KL, is obtained using the Kullback-Leibler divergence. This represents the proportion of samples in a batch whose predictions have changed, reflecting the difference between the original prediction category and the counterfactual prediction category. is the average difference in the model's prediction confidence levels on the original and counterfactual inputs, and is the difference between the maximum probabilities of the original and counterfactual predictions. The specific formula for calculating the loss function is as follows: , , , ,in: , These are the predicted probability distributions under the original and counterfactual scenarios, respectively, where i represents the i-th sample; The indicator function is a binary function. , For the original predicted category and For counterfactual prediction category, This refers to expectations. And... , which are preset hyperparameters.

[0017] Step 2.3.2 Consistency Loss. Consistency loss controls the difference between the counterfactual prediction and the original prediction, ensuring consistency between the mask ratio and the degree of prediction change. ,in It is the model's classification score for the original input text, representing the model's "confidence" in believing that the text belongs to each category; It is the model's classification score for counterfactual text input; To prevent division by 0.

[0018] Step 2.3.3 Importance Regularization Loss. To promote the sparsity and diversity of importance scores and control the distribution of importance scores, this loss function is designed: ,in, It is a sparsity loss, obtained from the expectation of the importance score S. It's variance loss, which encourages mask diversity. Mask consistency loss, where M refers to the mask. For hyperparameters;

[0019] Step 2.3.4 The total loss function of the counterfactual reasoning module is:

[0020] ;

[0021] Step 2.4 Causal Effect Assessment. Counterfactual inference is performed by comparing the original predictions. Counterfactual prediction To evaluate the causal effect of features, based on the do-logic approach to causal inference, the causal effect obtained is as follows: .

[0022] Step 3: Image causal intervention mechanism;

[0023] Step 3.1 Construct a confusion dictionary; it is necessary to construct a dictionary that approximates a hierarchical confusion dictionary. Where N represents the size of the hyperparameters, each This represents a context prototype. Based on the image set for rumor detection, the target object in each training image is masked according to its bounding box, generating a context image set I for rumor detection. This context image set I is then input into a pre-trained backbone network φ(·) to obtain the context feature set. ,in This is the number of training samples. To compute the contextual prototype, K-Means++ and principal component analysis are used to learn U, so that each... A form representing a contextual cluster. Each cluster It is set as the average feature of each cluster in K-Means++, i.e. ,in It is the first The number of contextual features in each cluster.

[0024] Step 3.2 Approximates the intervention distribution using the NWGM weighted average method; this invention provides two strategies to represent the influence of confounding factors: dot product attention and added attention. Dot product attention: Additive attention: ,in , , It is a mapping matrix. Different strategies are used to... It is approximately a weighted integral of all context prototypes. ,in These are weighting coefficients, used to measure each... In relation to the original features The importance of post-interaction, and .

[0025] Step 3.3 Constructing causal intervention image features; Learning a model through image features. Image features obtained after causal intervention .

[0026] Step 4: Multimodal Feature Fusion. The multimodal fusion module uses a multi-layer collaborative attention mechanism to capture interaction information between different modalities. It enhances cross-modal features by learning attention weights between features of different modalities, and uses the Sigmoid() and LeakyReLu() functions to obtain the final multimodal fused features. Where: the text feature of the i-th sample obtained by counterfactual reasoning is... The image features after causal intervention are For modal pairs Image features are computed using an iterative collaborative attention mechanism. Enhanced text features The process iterates a total of l times. In the initial rounds, the Q, K, and V matrices are respectively... , , ,in, It is a linear transformation matrix. The output of the j-th layer attention mechanism can be expressed as: Where H is the number of heads, It is the matrix of the linear transformation of the output layer. The j-th (j>1) layer... Use the output of the previous layer Iterative updates. The output of the last collaborative attention mechanism is used as the enhanced text feature. For modal pairs Similarly, it can also be obtained Enhanced image features Similarly, for other modal pairs, the results can be obtained through the iterative collaborative attention mechanism described above. .Will Input Sigmoid() and get the output. ;Will Input them into LeakReLU() respectively to get the output. The outputs of identical combined mode pairs are multiplied together and then concatenated to form the final fused feature. This process can be formally represented as .

[0027] Step 5: Data Classification and Prediction. This involves combining the final features of the merged posts. By passing the data to a fully connected layer, we can obtain the posts. Let Pi be the probability of a rumor, which can be expressed as: ,if Then the post is considered to be... Rumors, i.e., prediction tags .

[0028] Thus, from step 1 to step 5, a multimodal rumor detection method based on counterfactual reasoning and causal intervention has been completed.

[0029] Compared with the prior art, the present invention has the following beneficial effects: (1) The present invention constructs a unified multimodal causal modeling framework from the perspective of causal reasoning, which can systematically alleviate the problems of pseudo-correlation and confounding bias; (2) Through text counterfactual reasoning and image context intervention, key causal features are effectively identified and non-causal factors are eliminated, thereby improving the interpretability of the model; (3) Social propagation causal modeling is introduced to enhance the robustness and cross-domain generalization ability of the model in real complex scenarios. Attached Figure Description

[0030] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0031] Figure 1This is a schematic diagram of a multimodal rumor detection method based on counterfactual reasoning and causal inference, as described in an embodiment of the present invention.

[0032] Figure 2 The process of constructing a confusion dictionary.

[0033] Figure 3 This is a framework diagram of the feature fusion module of the present invention. Detailed Implementation

[0034] It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.

[0035] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0036] In this embodiment, as Figure 1 As shown, a multimodal rumor detection method based on counterfactual reasoning and causal inference includes the following steps:

[0037] Step 1: Multimodal feature extraction. CNN, pre-trained ResNet-50, and graph attention network are used to extract text features, image features, and social graph structure features, respectively.

[0038] Step 2: Counterfactual Reasoning Stage: Perform counterfactual causal inference on text features. The specific operation is as follows, using a trainable neural network. Obtain the word-level importance score S; select the TopK method to generate the mask. ; through formula Generate counterfactual samples Calculate the difference between the original and counterfactual predictions; construct the counterfactual loss function by considering the losses from causal effects, consistency, and importance regularization. ;in For the loss of causal effect, Consistency loss, importance regularization loss .

[0039] Step 3: Image Dot Product Causal Intervention Stage: Perform dot product causal intervention on image features. The specific operations are as follows:

[0040] (1) such as Figure 2 As shown, a contextual prototype heterogeneous dictionary U is constructed using K-Means++;

[0041] (2) The intervention distribution is approximated using the NWGM weighted average method;

[0042] (3) Constructing post-intervention image representations through dot product or additive attention mechanisms ;

[0043] Step 4: Multimodal fusion stage: such as Figure 3 As shown, , The social graph structural features G are fused into a multimodal fusion module, and cross-modal interaction is achieved using a multi-layer multi-head collaborative attention mechanism; a self-supervised approach is used to force propagation of structural alignment.

[0044] Step 5: Data Classification and Prediction. The total loss is obtained by calculating the classification loss, cross-modal feature alignment loss, and counterfactual loss using the cross-entropy loss function. The training and optimization are completed; finally, the rumor prediction result Y_pred is output through the classifier.

[0045] To better understand this example, this embodiment uses two real-world social media rumor datasets for testing: WEIBO and PHEME. The PHEME dataset is constructed from five tweets related to breaking news on the Twitter platform, each containing tagged text, a corresponding image, and a comment. The WEIBO dataset comes from Xinhua News Agency and Weibo platforms, containing a large number of tagged texts, images, and comments. Three modalities were used to detect rumors in the experiment; therefore, missing values ​​were handled, and samples that failed to simultaneously contain text, images, and propagation structure information were removed. For the PHEME and WEIBO datasets, they were divided into training, validation, and test sets in a 7:2:1 ratio. Statistical information for the preprocessed datasets is shown in Table 1.

[0046] Table 1. Dataset Statistics

[0047]

[0048] Initial settings for the experiment: For the unified modal context encoder, the input lengths of both text input T and image input I were set to a maximum of 300. The batch size was set to 64, the number of epochs to 20, and the learning rate to 5e-4. The Adam optimizer was used, with a learning rate of 0.002 during training. In the counterfactual causal inference part, a mask ratio of 0.15 was used. Hyperparameters , , , , The values ​​were set to 0.4, 0.3, 0.3, 0.4, 0.3, 0.3, 0.08, 0.03, and 0.01, respectively. The cluster size N of the confounding factor U was set to 128, and a dot product causal intervention strategy was used.

[0049] To verify the effectiveness of the method in this embodiment, the following baseline model was selected as a comparison model. In the comparison model,

[0050] EANN is a GAN-based model that uses text and image data to extract invariant features of events and applies them to emerging events.

[0051] MVAE uses a bimodal variable autoencoder combined with a binary classifier for multimodal rumor detection.

[0052] GLAN co-encodes local semantic and global structural information and applies a global-local attention network for rumor detection.

[0053] MFAN uses a multimodal feature-enhanced attention network for rumor detection, leveraging textual, visual, and social graph features, and improving social graph feature learning by inferring hidden links.

[0054] MRDMPL utilizes only text and image features and achieves better alignment between the two modalities by simultaneously applying cue learning to both features.

[0055] REPORT combines user information and propagation structure features, learns the relationships between users through graph neural networks, and models the information propagation process using a tree structure, thereby improving the accuracy and robustness of the model.

[0056] Accuracy was used as the evaluation metric for the experiment. In addition, considering class imbalance, precision, recall, and F1 score were used as supplementary evaluation metrics. The comparison results are shown in Table 2.

[0057] Table 2 Performance comparison with state-of-the-art methods on the PHEME and WEIBO datasets (%)

[0058]

[0059] The experimental results of this method and the comparison method are shown in Table 2. The performance comparison between the CCMRD model and its comparative models is presented, with the best and second-best results highlighted in bold and underlined, respectively. Since the performance metrics of the REPORT model on the Weibo dataset are not publicly available, and some performance metrics of the comparative models (precision (Pre), recall (Re), and F1 score) are not publicly available, missing values ​​in the corresponding cells of Table 2 are indicated by "—" to ensure the scientific rigor of the data comparison analysis.

[0060] The accuracy on the PHEME dataset is 91.42%, and the F1 score is 91.37%. On the WEIBO dataset, the accuracy is 94.17%, and the F1 score is 93.39%. Compared to other multimodal rumor detection methods, this method improves accuracy by approximately 3 percentage points. The table shows that MRDMPL, REPORT, and MFAN perform well in rumor detection, indicating that multimodal structures consider not only text and image features but also propagation graphs for identification.

[0061] To better validate the effectiveness of the proposed causal inference framework, its performance was compared with existing causal inference methods such as SpotFake+CCD, MCAN+CCD, and HMCAN+CCD on the PHEME dataset. These three methods combine different models with the causal inference framework to perform causal intervention on text and counterfactual inference on images, thus achieving multimodal rumor detection. The experimental results are shown in Table 3.

[0062] Table 3. Performance comparison with other multimodal causal inference models on the PHEME dataset (%)

[0063]

[0064] Table 3 shows that our method outperforms the other three methods. This indicates that the causal inference method of the CCMRD model is more effective for multimodal rumor detection.

[0065] To evaluate the effectiveness of different modules in the proposed method, this study conducts an ablation study on the impact of counterfactual inference and causal intervention modules on rumor detection performance. Ablation experiments were performed on the PHEME and WEIBO datasets, ensuring all training parameters were identical. Accuracy (Acc) and F1 score were selected as evaluation metrics. The experimental results are shown in Table 4.

[0066] The comparison method is as follows:

[0067] w / o TCF(Add) indicates that CCMRD has removed the text counterfactual inference module and the image uses additive causal intervention;

[0068] w / o TCF(Dot) indicates that CCMRD has removed the text counterfactual inference module and uses dot product causal intervention for the image. The text features obtained from CNN, the image features after dot product causal intervention, and the social graph features are fused together.

[0069] w / o ICI indicates that CCMRD has removed the image causal intervention module and only uses text features, image features, and social graphs obtained through counterfactual reasoning for feature fusion and inference.

[0070] TCF --> TCI (Add) indicates that the text portion did not use counterfactual reasoning but instead employed additive causal intervention, while the image portion did not undergo causal intervention;

[0071] TCF --> TCI (Dot) indicates that the text portion did not use counterfactual reasoning but instead employed dot product causal intervention, while the image portion did not undergo causal intervention;

[0072] TCI(Add)+ICI(Dot) indicates that the text part uses an additive causal intervention method, while the image part uses a dot product causal intervention method.

[0073] Table 4 Ablation experiments of the CCMRD model on the PHEME and WEIBO datasets (%)

[0074]

[0075] The ablation experiments of these variants showed that:

[0076] (1) The results of CCMRD, w / o TCF(Add), w / o TCF(Dot) and w / o ICI on the PHEME and WEIBO datasets show that in multimodal rumor detection, the performance of the model decreases when different causal inference methods are removed. Among them, the text counterfactual inference method contributes more to the model performance. Therefore, it is necessary to use causal inference to remove the rumor detection bias.

[0077] (2) Different causal intervention strategies yielded different results for the images. In the PHEME dataset, the accuracy was 88.31% (a decrease of 3.11%) and the F1 score was 85.80% (a decrease of 5.57%) when only additive causal intervention was applied to the images, while the accuracy was 88.83% (a decrease of 2.59%) and the F1 score was 86.77% (a decrease of 1.13%) when using the dot product causal intervention strategy. In the WEIBO dataset, the accuracy was 90.81% (a decrease of 2.97%) and the F1 score was 90.15% (a decrease of 3.2%) when only additive causal intervention was applied to the images, while the accuracy was 91.15% (a decrease of 2.63%) and the F1 score was 90.52% (a decrease of 2.83%) when using the dot product causal intervention strategy. Therefore, the dot product causal intervention strategy is superior to the additive causal intervention strategy.

[0078] (3) As can be seen from the last three rows of Table 4, when the counterfactual reasoning of the text is replaced with a causal intervention strategy similar to that of the image, the model performance drops rapidly. It can be considered that counterfactual reasoning is better than causal intervention methods in removing text bias.

Claims

1. A multimodal rumor detection method based on counterfactual reasoning and causal intervention, characterized in that, Includes the following steps: Step 1: Multimodal feature extraction; We used CNN, pre-trained ResNet-50, and graph attention network to extract text features, image features, and social graph structure features, respectively. Step 2: Counterfactual Causal Inference of Text; Perform counterfactual causal inference on text features. The specific operation is as follows, using a trainable neural network. Obtain the word-level importance score S; select the TopK method to generate the mask. ; through formula Generate counterfactual samples Calculate the difference between the original and counterfactual predictions; construct the counterfactual loss function by considering the losses from causal effects, consistency, and importance regularization. ;in For the loss of causal effect, Consistency loss, importance regularization loss ; Step 3: Image Causal Intervention Mechanism; Causal intervention is applied to image features, and the specific operations are as follows: (1) Construct a contextual prototype heterogeneous dictionary U using K-Means++; (2) The intervention distribution is approximated using the NWGM weighted average method; (3) Constructing post-intervention image representations through dot product or additive attention mechanisms ; Step 4: Multimodal fusion; , The social graph structural features G are fused into a multimodal fusion module, and cross-modal interaction is achieved using a multi-layer multi-head collaborative attention mechanism; a self-supervised approach is used to force propagation of structural alignment. Step 5: Data classification and prediction; The total loss is obtained by calculating the three types of losses—classification loss, cross-modal feature alignment loss, and counterfactual loss—using the cross-entropy loss function. The training and optimization are completed; finally, the rumor prediction result Y_pred is output through the classifier.

2. The multimodal rumor detection method based on counterfactual reasoning and causal intervention as described in claim 1, characterized in that, Step 1 includes the following sub-steps: Step 1.1, Text Modal Feature Extraction: A convolutional neural network with pooling operations is used to extract semantic features of the text sentences. ,in For the text information used in rumor detection, B is the batch size, L is the sequence length, and D is the embedding dimension. Step 1.2, Image modality feature extraction: Image features are extracted using a pre-trained ResNet-50: ,in Image information for rumor detection; Step 1.3, Propagation Structure Graph Feature Extraction: Based on the graph structure (adjacency matrix) A and node embeddings V, a graph attention network is used to generate graph features. ,in For a set of nodes, These represent post nodes, user nodes, and comment nodes, respectively. This is the initial edge matrix.

3. The multimodal rumor detection method based on counterfactual reasoning and causal intervention as described in claim 1, characterized in that, Step 2 includes the following sub-steps: Step 2.1: Evaluate the feature importance of the text. This is achieved by using a trainable neural network to calculate an importance score S that reflects the contribution of each feature to the prediction. The importance score S is calculated, where For text feature vectors, , , , For trainable parameters, For the Sigmoid function; Step 2.2 Apply masks to generate counterfactual features. Based on the importance score S, select the Top-K method to generate the mask. The mask ratio is controlled by mask_ratio. It refers to the first The first sample Whether a word is masked is determined by... The calculation yielded, where Next application Masking generates counterfactual features by setting important features to zero, simulating a counterfactual scenario where key features are removed; Step 2.3: Construct the counterfactual loss function. By calculating the loss between the counterfactual prediction and the original prediction, the causal reasoning ability of the model is optimized. The counterfactual loss function considers the loss from three aspects: causal effect, consistency, and importance regularization. Step 2.4, Causal Effect Assessment, Counterfactual Inference by comparing the original predictions. Counterfactual prediction To evaluate the causal effect of features, based on the do-logic approach to causal inference, the causal effect obtained is as follows: .

4. The multimodal rumor detection method based on counterfactual reasoning and causal intervention as described in claim 3, characterized in that, Step 2.3 includes: Step 2.3.1, Loss of Causal Effect: Loss of Causal Effect Furthermore, the differences in predicted distributions (KL) and predicted categories are also considered. and confidence level difference The difference between the composition and the predicted distribution KL is obtained using the Kullback-Leibler divergence. The proportion of the difference between the original prediction category and the counterfactual prediction category measures the proportion of samples in a batch whose predictions have changed. The loss function is calculated as follows: ... , , , ,in: , These are the predicted probability distributions under the original and counterfactual scenarios, respectively, where i represents the i-th sample; The indicator function is a binary function. , For the original predicted category and For counterfactual prediction category, This refers to expectations, and , which are preset hyperparameters; Step 2.3.2, Consistency Loss: Consistency loss controls the difference between the counterfactual prediction and the original prediction, ensuring consistency between the mask ratio and the degree of prediction change. ,in It is the model's classification score for the original input text, representing the model's "confidence" in believing that the text belongs to each category; It is the model's classification score for counterfactual text input; Prevent division by zero; Step 2.3.3, Importance Regularization Loss: To promote the sparsity and diversity of importance scores and control the distribution of importance scores, this loss function is designed: ,in, It is a sparsity loss, obtained from the expectation of the importance score S. It's variance loss, which encourages mask diversity. Mask consistency loss, where M refers to the mask. For hyperparameters; Step 2.3.4, the total loss function of the counterfactual reasoning module is: 。 5. The multimodal rumor detection method based on counterfactual reasoning and causal intervention as described in claim 1, characterized in that, Step 3 includes the following sub-steps: Step 3.1, construct a confusion dictionary; this requires constructing a dictionary that approximates a hierarchical confusion dictionary. Where N represents the size of the hyperparameters, each To represent a context prototype, a context image set I is generated by masking the target object in each training image based on the bounding box of the target object in the image set for rumor detection. The context image set I is then input into a pre-trained backbone network φ(·) to obtain the context feature set. ,in This refers to the number of training samples. To compute the contextual prototype, K-Means++ and principal component analysis are used to learn U, so that each... A form representing a context cluster, each cluster It is set as the average feature of each cluster in K-Means++, i.e. ,in It is the first The number of context features in each cluster; Step 3.2: Approximate the intervention distribution using the NWGM weighted average method; this invention provides two strategies to represent the influence of confounding factors: dot product attention and added attention. Dot product attention: Additive attention: ,in , , It is a mapping matrix, which uses different strategies to... It is approximately a weighted integral of all context prototypes. ,in These are weighting coefficients, used to measure each... In relation to the original features The importance of post-interaction, and ; Step 3.3: Construct causal intervention image features; learn a model using image features. Image features obtained after causal intervention .

6. The multimodal rumor detection method based on counterfactual reasoning and causal intervention as described in claim 1, characterized in that, Step 4 uses a multi-layer collaborative attention mechanism to capture interaction information between different modalities. It learns the attention weights between features of different modalities to enhance cross-modal features and uses the Sigmoid() and LeakyReLu() functions to obtain the final multimodal fusion features.

7. The multimodal rumor detection method based on counterfactual reasoning and causal intervention as described in claim 1, characterized in that, Step 5, data classification prediction, includes: combining the final merged features of the posts. By passing the data to a fully connected layer, we can obtain the posts. Let Pi be the probability of a rumor, which can be expressed as: ,if Then the post is considered to be... Rumors, i.e., prediction tags .

Images