A multi-modal rumor detection method based on information bottleneck and counterfactual intervention

Abstract

This invention discloses a multimodal rumor detection method based on information bottlenecks and counterfactual intervention, belonging to the fields of natural language processing and causal reasoning. Specifically, the invention includes: inputting text, images, and social propagation graphs; extracting modal features; performing variational information bottleneck encoding on each modal feature to compress label irrelevance and confounding redundancy; constructing context-aware cross-modal interactions to obtain enhanced representations; further generating fine-grained masks through a learnable mask generator and using differentiable discrete sampling to achieve counterfactual causal intervention on the representations, generating multiple counterfactual samples to estimate causal effects; and weighting and enhancing the multimodal features based on causal effects to complete the classification output. This method can effectively suppress spurious correlations induced by potential confounding factors, improve robustness across events and out-of-distribution conditions, and provide interpretable evidence. On the PHEME and WEIBO public datasets, this method significantly improves performance compared to other baseline models.

Description

Technical Field

[0001] This invention relates to the fields of natural language processing, artificial intelligence and causal reasoning, and specifically to a multimodal rumor detection method based on information bottlenecks and counterfactual intervention, which falls under the category of artificial intelligence information credibility identification technology. Background Technology

[0002] The spread of misinformation and false content on social media has been proven to significantly interfere with public perception and social decision-making, and to continuously erode public trust through large-scale dissemination. As platform content formats evolve, false content is no longer limited to textual narratives but increasingly employs multimodal combinations of text-image-dissemination structures to enhance persuasiveness and dissemination efficiency. This makes multimodal rumor / fake news detection a crucial research direction in social media mining and a core issue in social media mining and trustworthy AI. However, existing methods are mostly correlation-driven, using complex fusion alignment modules to capture cross-modal statistical associations to improve performance. But statistical correlation is not equivalent to causation. In real-world scenarios, unobserved confounding factors (such as event context, collective sentiment, and differences between platform and user groups) may simultaneously influence textual wording, visual choices, and dissemination patterns, inducing models to learn "pseudo-correlated but predictable" shortcut cues. This leads to a significant performance degradation under cross-event migration and distribution shifts, and the decision-making basis becomes difficult to explain. Causal inference methods can provide strong explanations, but existing methods often remain at the coarse-grained or post-hoc analysis level, lacking end-to-end differentiable intervention mechanisms. Information compression methods can suppress redundant noise, but they are difficult to explicitly distinguish between causal and non-causal information and may still retain spurious patterns induced by hybridity.

[0003] To address the spurious correlations caused by modal confounding factors and redundant noise, we propose a unified framework, CCI-IB, which integrates the Variational Information Bottleneck (VIB) and Counterfactual Causal Inference (CCI), achieving simultaneous optimization of redundancy compression and causal identification. By introducing the VIB structure into the model, we minimize the mutual information between the latent representation and the input while maintaining sufficient label information, thereby learning compact and more robust multimodal representations, effectively...

[0004] This approach mitigates the interference of high-dimensional redundancy and noise on intervention estimation. Building upon this, a fine-grained, learnable feature-level counterfactual intervention mechanism is designed to explicitly characterize causal and confounding paths between modalities using a structural causal model. By intervening in different modal cues and estimating their causal effects, non-causal interference introduced by shortcut factors is suppressed. Through the synergistic modeling of VIB redundancy compression and counterfactual intervention, CCI-IB demonstrates stronger generalization stability in distribution shift assessment and provides more interpretable causal evidence for model decisions. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of existing technologies. Addressing two key issues—identifying causal relationships and spurious statistical shortcuts in biased datasets, preventing model interference from false correlations, and accurately extracting causal substructures from entangled multimodal inputs to understand the complex interactions and dependencies between different modalities—this invention proposes a multimodal rumor framework that combines counterfactual causal intervention with variational information bottlenecks. Specifically, it utilizes variational information bottleneck techniques to compress and separate stable causal representations driven by potential causal factors from statistical shortcut features E caused by bias, thereby effectively eliminating the confusing paths between statistical shortcut features E, causal features U, and the predicted result Y. Simultaneously, an intelligent counterfactual causal inference module performs differentiable intervention on the causal components at the feature level, directly blocking the prediction path from the original modal features X to the predicted Y, allowing the model to infer solely based on the estimated causal representations.

[0006] The technical solution of the present invention includes the following steps:

[0007] Step 1: Analyze text, images, and social communication structures. Figure 3 Single-modal feature extraction is performed on multimodal rumor data of various modalities.

[0008] Step 1.1, Text Modality Feature Extraction: BERT is used to extract semantic features from text sentences. These features help us better understand the contribution of text content to rumor detection. These features can be formally described as follows: Describe, in which For the text information used in rumor detection, B is the batch size, L is the sequence length, and D is the embedding dimension.

[0009] Step 1.2, Image Modal Feature Extraction: Image features are extracted using a pre-trained ResNet-50 to ensure the effectiveness and accuracy of image information. This can be described as follows: ,in Image information for rumor detection;

[0010] Step 1.3: Propagation structure graph feature extraction. Based on the adjacency matrix A and node embedding V, a graph convolutional network (GCN) is used to generate graph features. ,in For a set of nodes, These represent post nodes, user nodes, and comment nodes, respectively. This is the initial edge matrix.

[0011] Step 2: Variational information bottleneck coding;

[0012] Step 2.1: Mapping Input Features with a Variational Encoder. First, the m-th modal feature of the input is processed using a variational encoder (VAE). , VAE maps input features to the mean of a Gaussian distribution. and variance ,Right now ,in express The parameters.

[0013] Step 2.2: The reparameterization technique is used to obtain latent variables. The encoding representation is then transferred using the reparameterization technique to obtain the bottleneck latent variables for the three modalities. The formula for this process is: ,in It is noise in a standard normal distribution;

[0014] Step 2.3: Calculate the VIB training loss and obtain the latent variables in the VAE. Then, based on the VIB objective, the VIB loss for each mode is calculated. The formula for VIB loss is: ,in Represents the approximate posterior distribution With prior distribution Kullback-Leibler divergence between them Given a standard normal prior distribution, the variational approximation distribution... , To balance compression and prediction sufficiency, the VIB module optimizes the aforementioned loss function, compressing the input data for each modality and retaining only the information relevant to the target label Y while removing redundant or irrelevant information. Specifically, the VIB module maximizes the mutual information between the input features and the target label, ensuring that the representation of each modality includes information relevant to the target label while removing irrelevant noise or redundant information.

[0015] Step 3: Context-aware cross-modal interaction;

[0016] Step 3.1: Obtain the three modal bottleneck representations output from Step 2. Their equivalent features are represented as T, I, and G, where T represents the text modal feature representation, I represents the image modal feature representation, and G represents the propagation graph structure modal feature representation;

[0017] Step 3.2: Construct a multi-head attention-based bidirectional text-image interaction module to characterize the contextual dependencies between text and images. This module uses text as the query and images as keys and values ​​to compute an enhanced text representation. Image-enhanced representations are calculated using images as queries and text as keys and values. ,satisfy , ,in Each by The linear mapping yields MultiHead(.), which represents the multi-head attention operator.

[0018] Step 3.3: Fuse the features obtained after the interaction between the text and image to obtain a joint text-image representation. Specifically, it is to and Perform splicing and nonlinear transformation function Mapping, satisfying , among which, among which Indicates text features and image features splicing This represents a nonlinear transformation function. These are learnable parameters. For activation functions;

[0019] Step 3.4: Construct a cross-modal interaction module for propagation graph and graph-text joint representation, and use multi-head attention to characterize the interaction relationship between the propagation graph modality and the graph-text modality; using the propagation graph representation as the query and the graph-text joint representation as the key and value, obtain the propagation graph enhanced representation. ,satisfy: ;

[0020] Step 3.5: To suppress overpropagation and feature drift caused by cross-modal interaction, a residual interpolation mechanism is introduced to fuse the interaction enhancement representation and the original representation according to weights, obtaining the final cross-modal interaction output. ,satisfy: ,in These are hyperparameters used to balance the transition characteristics of the m-mode. With original features Finally, the above This serves as input to subsequent cross-modal fusion discrimination and / or counterfactual causal intervention reasoning modules, thereby improving the contextual consistency and robustness of multimodal representations.

[0021] Step 4: Intelligent Counterfactual Causal Learning Module;

[0022] Step 4.1, first, define the fact representation. The system learns sample-specific intervention strategies through a lightweight mask generator and utilizes modality-level mask strength generated by a global controller and an importance predictor. Importance of feature dimensions ,in Indicates the modal level mask strength. This indicates the importance of each feature dimension, specifically, These are learnable parameters that are progressively optimized through a training process to generate the optimal mask; combined with the original mask for a specific modality. , Finally, the unbindified mask is obtained. ;

[0023] Step 4.2: Use the mask matrix generated by the mask generator to perform element-wise multiplication on the fact representation to generate N counterfactual feature samples. ,in, This represents element-wise multiplication. These are the first corresponding modes. A counterfactual mask; to enable discrete interventions to support end-to-end backpropagation optimization, this invention introduces a soft mask generation mechanism based on Gumbel-Softmax during the training phase, applying a mask with temperature parameters. The Sigmoid function generates a soft mask. During the inference phase, a hard mask is used, i.e. ;

[0024] Step 4.3, causal effect estimation, firstly, present the factual representation. Counterfactual representation Then, by comparing factual and counterfactual representations, we... The causal effect was estimated and the average of multiple counterfactual samples was calculated. To obtain the final causal effect;

[0025] Step 4.4, counterfactual feature enhancement, based on the counterfactual causal effect obtained in Step 4.3, through... , , Features obtained after counterfactual causality effect is enhanced ,in The feedforward neural network MLP is used. Finally, the counterfactual causality regularization loss is composed of five losses: contrastive loss, sparsity loss, balance loss, diversity loss, and independence loss. The specific formula is as follows: ,in, It is a comparison of losses. It is sparsity loss. It is a balancing loss. It is a loss of diversity. It is the independence constraint loss, and the specific calculation formula is as follows: , , , , ,in For target similarity, For the purpose of sparsity, It is the variance function. These are the corresponding weight hyperparameters. Combining the causal loss, VIB loss, and the overall classification loss obtained through cross-entropy loss, the final total loss function is: ,in, and It is a hyperparameter that modulates the effects of causal loss and VIB loss.

[0026] Step 5: Fusion and Prediction; Concatenate the enhanced features into a single data structure. By inputting into a fully connected layer, you can obtain the post. The probability of it being a rumor , can be represented as ,if Then the post is considered to be... Rumors, i.e., prediction tags .

[0027] Thus, from step 1 to step 5, a multimodal rumor detection method based on information bottlenecks and counterfactual intervention has been completed.

[0028] Compared with the prior art, the present invention has the following beneficial effects:

[0029] (1) This invention systematically characterizes the hybrid problem in multimodal rumor detection from the perspective of structural causal model and formalizes it into a counterfactual causal inference task.

[0030] (2) A multimodal learning framework that integrates variational information bottleneck and counterfactual causal intervention is proposed, which realizes the synergistic optimization of causal feature selection and information compression, overcomes the limitations of existing methods in multimodal learning, and improves the robustness and accuracy of the model through causal reasoning.

[0031] (3) By introducing the simulated annealing algorithm, the process of generating counterfactual masks can be effectively controlled, ensuring that the model has sufficient flexibility in the exploration process, and improving the accuracy of decision-making by hardening the mask in the later stage of training.

[0032] (4) The effectiveness and superiority of our method were demonstrated on the public datasets PHEME and WEIBO. Attached Figure Description

[0033] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0034] Figure 1 This is the causal graph for rumor detection described in an embodiment of the present invention.

[0035] Figure 2 This is the CCI-IB framework for a multimodal rumor detection method based on information bottlenecks and counterfactual intervention, as described in this embodiment of the invention.

[0036] Figure 3 This is a diagram of the intelligent counterfactual causal learning module in a robust multimodal rumor detection method based on information bottlenecks and counterfactual causal intervention, as described in an embodiment of the present invention.

[0037] Figure 4 This is the global mask strength map on PHEME in the robust multimodal rumor detection method based on information bottleneck and counterfactual causal intervention described in this embodiment of the invention.

[0038] Figure 5 This is the feature importance distribution on PHEME in a robust multimodal rumor detection method based on information bottleneck and counterfactual causal intervention as described in an embodiment of the present invention.

[0039] Figure 6 This is a bar chart comparing the sparsity of a robust multimodal rumor detection method based on information bottlenecks and counterfactual causal intervention as described in an embodiment of the present invention on PHEME.

[0040] Figure 7 This is a mask heatmap on PHEME for a robust multimodal rumor detection method based on information bottlenecks and counterfactual causal intervention, as described in an embodiment of the present invention.

[0041] Figure 8 This is a graph showing the performance metrics and total training time of a robust multimodal rumor detection method based on information bottlenecks and counterfactual causal intervention, as described in an embodiment of the present invention, on the PHEME dataset, as a function of the value of N.

[0042] Figure 9This is a graph showing the performance metrics and total training time of a robust multimodal rumor detection method based on information bottlenecks and counterfactual causal intervention, as described in an embodiment of the present invention, as a function of the value N on the WEIBO dataset.

[0043] Figure 10 In the robust multimodal rumor detection method based on information bottleneck and counterfactual causal intervention described in this embodiment of the invention, different datasets of the PHEME dataset are used. and A comparison chart of F1 scores under different values.

[0044] Figure 11 In the robust multimodal rumor detection method based on information bottleneck and counterfactual causal intervention described in this embodiment of the invention, different datasets are used in the WEIBO dataset. and A comparison chart of F1 scores under different values. Detailed Implementation

[0045] It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.

[0046] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0047] In this embodiment, as Figure 1 As shown, the causal graph for multimodal rumor detection considered in this invention is presented. Figure 1 (b) Causal graph of existing multimodal rumor detection Figure 1 (a) Make a comparison.

[0048] Figure 1 (a) refers to the existing method of multimodal rumor detection, which uses a structural causal model (SCM) to describe the causal dependencies between variables, as follows:

[0049] (1) This indicates that the observed feature X is composed of both potential causal factors U and statistical shortcut factors EE. For example, U represents the true facts of the event, while E captures misleading features (such as platform or topic context).

[0050] (2) This indicates that the two are intertwined due to the presence of unknown confounding factors.

[0051] (3) U→Y: This means that U directly affects label Y and is the key factor that truly determines whether it is a rumor;

[0052] (4) X→Y: This represents the path from the observed feature X to the label Y for prediction.

[0053] The causal inference structure in existing methods has the following problems (such as...) Figure 1 (a) shows that the observed feature X is simultaneously influenced by both a statistical shortcut factor E and a potential causal factor U, and there is a correlation between E and U (E→U). This structure makes the model prone to learning the "induced statistical shortcut feature E," that is, it tends to establish a seemingly effective but actually unstable X→y prediction path. According to d-separation theory, if there is an unblocked path between two variables, they are correlated. Figure 1 In (a), there are two unblocked paths between X and Y: (1) (2) .

[0054] Both of these paths introduce spurious correlations. Therefore, to eliminate bias, both paths must be blocked simultaneously. Based on this, we designed... Figure 1 (b) shows the causal inference structure, where the red crosses in the figure indicate blocked paths. Figure 1 (b) expresses a goal-oriented reasoning structure, which, while still starting from observed feature X in the reasoning stage, decomposes it into two types of information: one corresponding to U, i.e., stable causal evidence; and the other corresponding to E, i.e., induced statistical shortcut features. This structure must satisfy two key constraints:

[0055] (1) Cut off shortcuts (for path 1): The predicted path should only go through U→Y, rather than directly from X to Y. That is, in the prediction process, the shortcut information in the original features is not used, but reasoning is performed through the estimated causal part U;

[0056] (2) Decoupling and independence (for path 2): The causal part U should carry as little information as possible about the shortcut feature E, that is, minimize the statistical dependency between U and E. By forcing them to be independent, the connection between E and U is blocked.

[0057] In this embodiment, as Figure 2 As shown, a multimodal rumor detection method based on information bottlenecks and counterfactual intervention includes the following steps:

[0058] Step 1: Single-modal feature extraction. Features are extracted from each modality using the following methods: For the text modality, semantic features are extracted from the text using the BERT model; for the image modality, image features are extracted using the pre-trained ResNet-50 model; for the propagation network structure modality, features are extracted from the adjacency matrix of the propagation network based on the graph convolutional network (GCN) to obtain graph structure features.

[0059] Step 2: Variational Information Bottleneck Encoding. Variational information bottleneck encoding is performed on the features of each modality to obtain a compressed bottleneck latent variable representation. By constructing a VIB loss function, redundant information is compressed while maintaining the relevance of each modality to the rumor detection task; the VIB loss function formula is as follows: ,in, Represents the VIB loss function. It is the posterior distribution of the latent variables. It is the probability of generating the model. It is an information bottleneck term, representing a constraint on the amount of information.

[0060] Step 3: Context-aware cross-modal interaction. Based on the context-aware cross-modal interaction mechanism, the bottleneck representations of each modality are interactively modeled to obtain enhanced cross-modal representations, thereby improving the multimodal robustness of the model.

[0061] Step 4: Intelligent counterfactual causal learning module, such as Figure 3 As shown, a fine-grained mask is generated through a learnable mask generator, and counterfactual intervention is applied to the enhanced representation based on differentiable discrete sampling to generate at least one counterfactual representation. The causal effect is estimated by calculating the prediction difference between the factual representation and the counterfactual representation, and the features are weighted and enhanced based on the causal effect. A counterfactual causal regularization loss is constructed, consisting of contrast loss, sparsity loss, balance loss, and diversity loss. The formula for the counterfactual causal regularization loss is: ,in, It is a comparison of losses. It is sparsity loss. It is a balancing loss. It is a loss of diversity. These are the corresponding weight hyperparameters; combining the causal loss, VIB loss, and the overall classification loss obtained through cross-entropy loss, the final total loss function is: ,in, and It is a hyperparameter that modulates the effects of causal loss and VIB loss;

[0062] Step 5: Fusion and prediction. The enhanced multimodal features are fused, and the fused features are input into the classifier for rumor detection and prediction, and the rumor detection results are output.

[0063] To better understand this example, this embodiment uses two real-world social media rumor datasets for testing: WEIBO and PHEME. The PHEME dataset is constructed from five tweets related to breaking news on the Twitter platform, each containing tagged text, a corresponding image, and a comment. The WEIBO dataset comes from Xinhua News Agency and Weibo platforms, containing a large number of tagged texts, images, and comments. Three modalities were used to detect rumors in the experiment; therefore, missing values ​​were handled, and samples that failed to simultaneously contain text, images, and propagation structure information were removed. For the PHEME and WEIBO datasets, they were divided into training, validation, and test sets in a 7:2:1 ratio. The statistics of the preprocessed datasets are shown in Table 1.

[0064] Table 1. Dataset Statistics

[0065]

[0066] Initial settings for the experiment: For the unified modal context encoder, the input lengths of both text input T and image input I were set to a maximum of D=768. The batch size was set to 256, and the number of epochs was set to 50. The model was trained using the Adam optimizer with an initial learning rate of 0.001. The weight decay of the main classification model was set to 1e−5, while the decay factor of the auxiliary representation learning module was 1e−4. Gradient clipping with a maximum norm of 1.0 was used to ensure training stability. For counterfactual inference, N=5 counterfactual samples were generated for each instance. The mask generator output a continuous feature-level mask [0,1], enabling distinguishable interventions. During training, a temperature annealing strategy was used to progressively sharpen the mask distribution, with the temperature decreasing linearly. Reducing the hyperparameter from 5.0 to 0.5 allows the model to transition from soft exploratory masking to near-discrete causal decision-making. Take values ​​of 1 and 0.3 respectively; ; ; ; We set the values ​​to 1, 0.5, 0.3, and 0.2 respectively. This was applied to the PHEME dataset. Use values ​​of 0.9 and 0.8 on the WEIBO dataset. We used values ​​of 0.9, 0.2, etc. In terms of hardware, all experiments were trained on an NVIDIA GeForce GTX 1080 Ti GPU. For software, we used Python 3.9 and the PyTorch deep learning framework.

[0067] To verify the effectiveness of the method in this embodiment, the following baseline model was selected as a comparison model. In the comparison model,

[0068] EANN (Wang et al., 2018) is a GAN-based model. It utilizes text and image data to extract invariant features of events and applies them to newly emerging events.

[0069] MVAE (Khattar et al., 2019) uses a bimodal variable autoencoder combined with a binary classifier for multimodal rumor detection.

[0070] GLAN (Yuan et al., 2019) co-encodes local semantic and global structural information and applies a global-local attention network for rumor detection.

[0071] MFAN (Zheng et al., 2022) uses a multimodal feature-enhanced attention network for rumor detection, leveraging textual, visual, and social graph features, and improving social graph feature learning by inferring hidden links.

[0072] MRDMPL (Chen et al., 2024) utilizes only text and image features and achieves better alignment between the two modalities by simultaneously applying cue learning to both features.

[0073] REPORT (Liu et al., 2024) combines user information and propagation structure features, learns the relationships between users through graph neural networks, and uses a tree structure to model the information propagation process, thereby improving the accuracy and robustness of the model.

[0074] MFCL (Han Chen et al., 2025) integrates the intrinsic features of text, images, and propagation networks to capture key cross-modal associations for accurate detection. Through contrastive learning techniques, it effectively extracts features while alleviating the challenge of insufficient labeled data.

[0075] DCE-RD (Kaiwei Zhang et al., 2023) is a rumor detection method based on dynamic causal effects. It combines user information and propagation structure, and improves detection performance through graph neural networks (GNN) and causal inference techniques.

[0076] Accuracy was used as the evaluation metric for the experiment. In addition, considering class imbalance, precision, recall, and F1 score were used as supplementary evaluation metrics. The comparison results are shown in Table 2.

[0077] Table 2 Performance comparison with state-of-the-art methods on the PHEME and WEIBO datasets (%)

[0078]

[0079] The experimental results of this method and the comparison methods are shown in Table 2. Table 2 compares the performance of CCI-IB with several representative rumor detection methods on the PHEME and WEIBO benchmark datasets. It can be observed that CCI-IB achieves superior results in all four metrics (Acc / Pre / Re / F1) on both datasets, with significant advantages across multiple dimensions, fully validating the effectiveness and advancement of the method. On the PHEME dataset, CCI-IB achieves an F1 score of 90.20%; on the WEIBO dataset, CCI-IB achieves an F1 score as high as 98.71%, significantly outperforming all existing comparison methods.

[0080] EANN and MVAE, as earlier multimodal rumor detection models, both achieved F1 scores below 80% on both datasets; particularly on WEIBO, MVAE's F1 score was only 70.34%, indicating that simple multimodal alignment or generative modeling is ill-suited for complex social media scenarios. In contrast, CCI-IB achieved an F1 score improvement of over 28% on WEIBO, demonstrating a significant advantage.

[0081] GLAN utilizes graph structures to model propagation relationships, achieving an F1 score of 78.51% on PHEME. MFAN / MFCL further introduces a multimodal attention mechanism, improving the F1 score to 86–87% on PHEME and approximately 91–92% on WEIBO. However, CCI-IB still achieves significant improvements over MFCL, increasing the F1 score by 2.61% on PHEME and 6.76% on WEIBO. This indicates that CCI-IB does not rely on single-modal enhancement but achieves substantial improvements at the multimodal fusion paradigm level.

[0082] MRDMPL and DCE-RD have demonstrated strong performance on WEIBO (F1 ≈ 92–93%), representing a strong baseline for current Chinese rumor detection. REPORT shows extremely high precision (93.8%) on PHEME, but significantly low recall (82.8%), indicating a prediction bias issue. In contrast, CCI-IB achieves a better balance between precision and recall, with Pre / Re ratios of 90.27% / 90.15% on PHEME and close to 98.7% on WEIBO.

[0083] In summary, Table 2 presents a performance comparison on the PHEME and WEIBO datasets. Overall, CCI-IB achieves superior results across all metrics, demonstrating strong effectiveness. In PHEME, CCI-IB achieves an F1 score of 90.2%, outperforming baseline models such as MFCL (87.59% F1), MFAN (86.05% F1), and MRDMPL (83.6% F1), indicating a significant improvement beyond attention fusion or graph augmentation modeling alone. On Weibo, CCI-IB's F1 score further improves to 98.71%, far exceeding competing methods including MFCL (91.95% F1) and DCE-RD (93.21% F1), demonstrating superior promotion capabilities on large-scale real-world Chinese social media. Notably, CCI-IB maintains a good balance between precision and recall, while some earlier methods exhibited biased behavior. These results collectively validate that CCI-IB provides a strong and continuous improvement over previous advanced models in English and Chinese rumor detection benchmarks.

[0084] Table 3. Results of cross-event analysis in the PHEME dataset (%)

[0085]

[0086] Table 3 shows the experimental results of the method in this example and the generalization performance of CCI-IB on different events. Experiments were conducted on the PHEME dataset using a cross-event setting, where one event was used as the test set and the remaining events as the training set. The results are shown in Table 3. As can be seen from the table, the model achieved average accuracy, precision, recall, and F1-score of 89.15%, 89.07%, 89.15%, and 89.07% respectively on the five unseen events, with a standard deviation of only 1.81% for F1, indicating that the method exhibits strong and relatively stable generalization ability in cross-event scenarios. Specifically, the model performed best on Sydney Siege (Acc=92.27%, F1=92.35%), while its performance was relatively lower on Ferguson and Charlie Hebdo (F1 scores of 87.00% and 87.97%, respectively). Among them, socially controversial events (Ferguson) and internationally complex events (Charlie Hebdo) are more difficult to detect, and their F1 scores are significantly lower than those of structurally clear emergencies (Sydney Siege), with overall fluctuations controlled within approximately 5.35 percentage points.

[0087] These results demonstrate that the model can capture rumor-discriminating features shared across events, rather than relying excessively on surface patterns from single events. However, significant distributional biases still exist between different events, and model performance degrades when test events differ significantly from training events in semantic expression, narrative style, or contextual structure. Notably, the Precision and Recall values ​​for each event are highly similar (difference ≤ 0.005), indicating that the model does not exhibit significant bias between rumor and non-rumor categories, thus validating its robustness under class imbalance conditions. Overall, these experimental results fully demonstrate the effectiveness and robustness of CCI-IB on unknown events.

[0088] In this embodiment, the global mask strength represents the overall degree to which the model "preserves key information" in that mode, such as... Figure 4 As shown, the distributions of Real and Fake news in the text modality are generally high (mean approximately 0.65–0.75) and relatively concentrated; the distribution of image modality is in the middle (approximately 0.45–0.6), with significantly greater fluctuations than text; the graph structure modality has the lowest overall strength (approximately 0.3–0.4), but there is still a discernible distribution shift between real and fake news. The distribution of Fake News in Text / Image is slightly biased towards lower values. Global mask strength analysis shows that the model exhibits the strongest causal dependence in the text modality, while it tends to intervene more significantly in the graph structure modality. Compared to real news, fake news has a generally lower global strength across multiple modalities, indicating that the model needs stronger counterfactual intervention to weaken the influence of non-causal features when identifying fake news.

[0089] In this embodiment, as Figure 5 As shown in the feature importance distribution plot on WEIBO, the Real / Fake curves highly overlap across the three modalities, with the overall importance values ​​concentrated in the 0.45–0.55 range, without any "extreme peaks in a few dimensions." The feature-level importance distribution exhibits high consistency between real and fake news, and shows a smooth distribution across all modalities. This indicates that the model does not rely on a few anomalous features for decision-making, but rather selectively intervenes in different modal features through a masking mechanism while maintaining the stability of the global importance structure, thereby achieving causal discrimination.

[0090] In this embodiment, as Figure 6As shown in the histogram comparing sparsity on WEIBO, the graph structure modality is the sparsest (≈98% occluded), indicating that most structural features are not causally necessary for the final judgment. The text modality has the lowest sparsity, further confirming that text is the primary causal evidence. Fake News is slightly sparser on the Text modality, indicating that the model is less "trustworthy" of fake text and requires stronger intervention. Mask sparsity analysis further shows that different modalities contribute significantly to causal decision-making. The graph structure modality exhibits the highest sparsity, indicating that the model relies on only a very small number of structural features for judgment; while the text modality has the lowest sparsity, verifying its dominant position as core causal evidence. Meanwhile, Fake News has slightly higher sparsity on the text modality, reflecting that the model exerts stronger counterfactual suppression on text features when processing false information.

[0091] In this embodiment, as Figure 7 As shown, the mask heatmap on WEIBO has an overall yellowish tint (≈0.4–0.6) and exhibits a distinct horizontal stripe structure. Horizontal stripes represent stable causal dimensions, indicating that certain feature dimensions are consistently preserved or suppressed across a large number of samples. This means the model learns consistent feature suppression and preservation patterns across different samples. The "low-value regions" are denser in the image and graph structure modalities of FakeNews, suggesting that compared to real news, fake news exhibits greater feature suppression in the image and graph structure modalities. This indicates that the model does not "mask randomly according to samples," but rather learns a counterfactual intervention pattern shared across samples, effectively weakening the influence of non-causal or noisy features through the counterfactual masking mechanism.

[0092] In this embodiment, to determine the optimal number of counterfactual samples, comprehensive experimental analysis was conducted on the PHEME and WEIBO datasets. As shown in Figures 8 and 9, the accuracy and F1 score on both datasets peaked at N=5. When N<5, the model performance was slightly insufficient; while when N>5, the performance not only failed to improve but also slightly decreased, while the training time increased significantly. Considering both prediction performance and computational efficiency, N=5 provides sufficient counterfactual diversity to support robust model learning while avoiding the introduction of excessive redundant information or noise, achieving an optimal balance between accuracy and training cost. Therefore, in all experiments in this paper, we uniformly set the number of counterfactual samples to 5 to ensure the efficiency and stability of model learning.

[0093] In this embodiment, as Figure 10 and Figure 11 As shown, the VIB loss weights are different on the PHEME and WEIBO datasets, respectively. Below, model performance varies with counterfactual loss weights. The trend of change. Overall, the experimental results consistently show that: and There are significant synergistic effects between them, and their proper configuration is a key factor determining model performance and stability. On the PHEME dataset, When the value is 0.9, the performance exhibits a typical unimodal distribution, and at that time... The optimal value (F1 = 0.9020) is reached at 0.8, indicating that there can be an effective synergy between strong information bottleneck constraints and moderate counterfactual regularization. While slightly inferior (F1 = 0.8995) to 0.6, it demonstrates stability over a wider parameter range, exhibiting strong robustness. And when... When = 0.1, performance increases with... Increased persistent degradation indicates that weak constraints are insufficient to support effective counterfactual learning. On the WEIBO dataset, larger... This also significantly improves performance and stability. = 0.9 performs best across the entire interval, and in The global best F1 score of 0.9871 is achieved at 0.2, with minimal overall fluctuation (F1 > 0.985). = 0.7 and 1.0 also show a similar trend, further validating the crucial role of strong KL constraints in learning discriminative representations. In contrast, = 0.1 indicates a significant performance drop, while a moderate value ( = 0.3 / 0.5) for More sensitive, but relatively less stable. From a mechanistic perspective, larger... This helps compress redundant information and learn compact and discriminative latent representations, thus providing a stable foundation for counterfactual learning; while moderate... This further strengthens the model's reliance on causal correlation features, improving its robustness to perturbations and distribution shifts. Conversely, too small... This can lead to insufficient representation constraints, causing counterfactual interventions to amplify noise; while excessive constraints can... This could introduce excessive intervention, disrupting the original discrimination structure. Furthermore, optimality varies across different datasets. The differences (PHEME was larger, WEIBO was smaller) indicate that the intensity of counterfactual intervention needs to be adjusted based on data characteristics. Nevertheless, both sets of experiments consistently show that when... When the value is ≥ 0.6 / 0.7, the model can achieve a wider range of performance. It maintains stable and superior performance within the specified range. In summary, this reveals the information bottleneck constraint ( ) provides a stable representation foundation, while counterfactual regularization ( Building upon this foundation, enhancing causal discrimination capabilities is crucial, and the synergistic optimization of both is key to achieving high performance and robustness. This conclusion provides important guidance for model parameter selection and the integration of information bottlenecks with causal learning.

[0094] To evaluate the effectiveness of different modules in the proposed method, this study conducts an ablation study on the impact of different modalities, the VIB module, and the counterfactual causal inference module on rumor detection performance. Ablation experiments were performed on the PHEME and WEIBO datasets, ensuring all training parameters were identical. Accuracy (Acc) and F1 score were selected as evaluation metrics. The experimental results are shown in Table 4.

[0095] The comparison method is as follows:

[0096] Text_Only indicates that CCI-IB uses only text information for modeling;

[0097] Image_Only indicates that CCI-IB uses only image information for modeling;

[0098] Graph_Only indicates that CCI-IB only uses propagation structure graph information for modeling;

[0099] Text_Image indicates that CCI-IB uses text and image information for modeling;

[0100] Text_Graph indicates that CCI-IB uses text and propagation structure graph information for modeling;

[0101] Image_Graph indicates that CCI-IB uses image and propagation structure graph information for modeling;

[0102] wo_Counterfactual indicates that the CCI-IB has removed the intelligent counterfactual inference module;

[0103] wo_vib indicates that the VIB module has been removed from CCI-IB;

[0104] wo_TempAnnealing indicates that CCI-IB uses the simulated annealing algorithm.

[0105] Table 4 Ablation experiments of the CCI-IB model on the PHEME and WEIBO datasets (%)

[0106]

[0107] The ablation experimental results of these variants indicate that:

[0108] Overall, CCI-IB achieved state-of-the-art performance on both the PHEME and WEIBO datasets, outperforming all comparable methods across all four metrics: Accuracy, Precision, Recall, and F1 score. This demonstrates the model's stability and generalization ability across different language environments (English PHEME and Chinese WEIBO). On the PHEME dataset, CCI-IB achieved an F1 score of 90.20%, an improvement of approximately +5.54% compared to the best single-modal model (Graph_Only, 84.66%). On the WEIBO dataset, CCI-IB achieved an F1 score of 98.71%, a significant improvement of +2.53% compared to Graph_Only (96.18%). This indicates that the proposed multimodal fusion and causal enhancement mechanisms are not simply additive but can further extract discriminative information on top of a strong baseline.

[0109] In terms of unimodal performance, Text_Only and Image_Only significantly outperformed Graph_Only on both datasets, especially on WEIBO. Graph_Only demonstrated the strongest unimodal capability in both PHEME (F1=84.66%) and WEIBO (F1=96.18%), indicating that propagation structure and user interaction information are crucial for rumor detection. This phenomenon is consistent with existing research, namely that the propagation patterns of rumors are highly distinguishable.

[0110] In the bimodal combination, significant complementary differences emerge between the different modalities. Text_Image performs well on PHEME (F1=88.50%), but its performance drops significantly on WEIBO (F1=80.77%), indicating that visual information is subject to significant noise in Chinese social media. Text_Graph achieves near-optimal performance on WEIBO (F1=98.02%), approaching the complete model, demonstrating a strong complementary relationship between text semantics and propagation structure. Image_Graph's overall performance is inferior to Text_Graph, further validating the core role of the text modality in multimodal rumor detection. Although Text_Graph has achieved high performance on WEIBO, CCI-IB further improves its performance on both datasets, indicating that trimodal joint modeling can effectively alleviate biases in single or bimodal models. The model does not simply concatenate multimodal features but achieves deep interaction through a structured mechanism.

[0111] As seen in the last three rows of Table 4, in the ablation analysis of key modules, after removing counterfactual learning, the F1 score for PHEME decreased from 90.20% to 88.09% (a decrease of 2.11%), and for WEIBO, it decreased from 98.71% to 94.53% (a decrease of 4.18%). This significant decrease indicates that counterfactual modeling plays a crucial role in removing spurious correlations and suppressing modal biases, especially in high-structure signal datasets like WEIBO. Regarding the temperature annealing mechanism, removing it resulted in a significant performance decrease on PHEME (F1 score decreased by 6.01%), while the decrease was smaller on WEIBO but still existed (F1 score decreased by 0.14%). This demonstrates that temperature annealing plays an important role in model training stability and causal alignment. Furthermore, after removing VIB, the F1 score decreased by 1.09% in the PHEME dataset and by 2.12% in the WEIBO dataset, indicating that VIB helps to compress redundant features, improve the quality of discriminative representations, and has a more significant effect in multimodal high-redundancy scenarios (WEIBO).

[0112] Based on the above analysis, the following conclusions can be drawn: (1) The propagation graph structure is the most discriminative single information source in rumor detection; (2) The text modality is the core hub of multimodal fusion; (3) The proposed Counterfactual + VIB + TempAnnealing mechanism has a clear, quantifiable and complementary contribution to improving performance; (4) The consistent advantages of CCI-IB on different datasets verify its robustness and transferability.

Claims

1. A method for multi-modal rumor detection based on information bottleneck and counterfactual intervention, characterized in that, Includes the following steps: Step 1: Single-modal feature extraction. For multimodal rumor detection samples containing text, images, and propagation network structures, the following methods are used to extract features from each modality: For the text modality, the BERT model is used to extract semantic features from the text; for the image modality, the pre-trained ResNet-50 model is used to extract image features; for the propagation network structure modality, feature extraction is performed on the adjacency matrix of the propagation network based on graph convolutional networks to obtain graph structure features. Step 2: Variational Information Bottleneck Encoding. First, a variational encoder is used to map the input features of each modality onto the mean and variance of a Gaussian distribution. Then, latent variables are obtained through reparameterization techniques. Next, the training loss function of VIB is calculated, and the input data is compressed by optimizing this loss, retaining information relevant to the target label and removing redundant or irrelevant parts. By maximizing the mutual information between the input features and the target label, VIB optimizes the representation of each modality, enabling it to effectively extract causal information related to the label while reducing noise and redundancy. Step 3: Context-aware cross-modal interaction. Based on the context-aware cross-modal interaction mechanism, the bottleneck representations of each modality are interactively modeled to obtain enhanced cross-modal representations, thereby improving the multimodal robustness of the model. Step 4: The intelligent counterfactual causal learning module generates fine-grained masks through a learnable mask generator and performs counterfactual intervention on the enhanced representations based on differentiable discrete sampling, generating 5 counterfactual representations. It estimates the causal effect by calculating the prediction difference between the factual and counterfactual representations, and weights the features based on the causal effect. It also constructs a counterfactual causal regularization loss composed of contrast loss, sparsity loss, balance loss, diversity loss, and independence constraints. The formula for the counterfactual causal regularization loss is: ,in, It is a comparison of losses. It is sparsity loss. It is a balancing loss. It is a loss of diversity. It is the independence constraint loss. These are the corresponding weight hyperparameters; combining the causal loss, VIB loss, and the overall classification loss obtained through cross-entropy loss, the final total loss function is: ,in, and It is a hyperparameter that modulates the effects of causal loss and VIB loss; Step 5: Fusion and prediction. The enhanced multimodal features are fused, and the fused features are input into the classifier for rumor detection and prediction, and the rumor detection results are output.

2. The multimodal rumor detection method based on information bottlenecks and counterfactual intervention as described in claim 1, characterized in that, Step 1 includes the following sub-steps: Step 1.1, Text Modality Feature Extraction: BERT is used to extract semantic features from text sentences. These features help us better understand the contribution of text content to rumor detection. These features can be formally described as follows: Describe, in which For the text information used in rumor detection, B is the batch size, L is the sequence length, and D is the embedding dimension. Step 1.2, Image Modal Feature Extraction: Image features are extracted using a pre-trained ResNet-50 to ensure the effectiveness and accuracy of image information. This can be described as follows: ,in Image information for rumor detection; Step 1.3: Propagation structure graph feature extraction. Based on the adjacency matrix A and node embedding V, a graph convolutional network (GCN) is used to generate graph features. ,in For a set of nodes, These represent post nodes, user nodes, and comment nodes, respectively. Let be the initial edge matrix.

3. The multimodal rumor detection method based on information bottlenecks and counterfactual intervention as described in claim 1, characterized in that, Step 2 includes the following sub-steps: Step 2.1: The variational encoder (VAE) maps the input features. First, the m-th modal feature of the input is processed using the variational encoder. , VAE maps input features to the mean of a Gaussian distribution. and variance ,Right now ,in express Parameters; Step 2.2: The reparameterization technique is used to obtain latent variables. The encoding representation is then transferred using the reparameterization technique to obtain the bottleneck latent variables for the three modalities. The formula for this process is: ,in It is noise in a standard normal distribution; Step 2.3: Calculate the VIB training loss and obtain the latent variables in the VAE. Then, based on the VIB objective, the VIB loss for each mode is calculated. The formula for the VIB loss is as follows: ,in Represents the approximate posterior distribution With prior distribution Kullback-Leibler divergence between them Given a standard normal prior distribution, the variational approximation distribution... , To balance the hyperparameters of compression and prediction sufficiency, the VIB module compresses the input data of each modality by optimizing the loss function mentioned above, retaining only the information related to the target label Y, while removing redundant or irrelevant information. Specifically, the VIB module maximizes the mutual information between the input features and the target label, so that the representation of each modality can contain information related to the target label while removing irrelevant noise or redundant information.

4. The multimodal rumor detection method based on information bottlenecks and counterfactual intervention as described in claim 1, characterized in that, Step 3 includes the following sub-steps: Step 3.1: Obtain the three modal bottleneck representations output from Step 2. Their equivalent features are represented as T, I, and G, where T represents the text modal feature representation, I represents the image modal feature representation, and G represents the propagation graph structure modal feature representation; Step 3.2: Construct a multi-head attention-based bidirectional text-image interaction module to characterize the contextual dependencies between text and images. This module uses text as the query and images as keys and values ​​to compute an enhanced text representation. Image-enhanced representations are calculated using images as queries and text as keys and values. ,satisfy , ,in Each by The linear mapping yields MultiHead(.), which represents the multi-head attention operator. Step 3.3: Fuse the features obtained after the interaction between the text and image to obtain a joint text-image representation. Specifically, it is to and Perform splicing and nonlinear transformation function Mapping, satisfying , among which, among which Indicates text features and image features splicing This represents a nonlinear transformation function. These are learnable parameters. For activation functions; Step 3.4: Construct a cross-modal interaction module for propagation graph and graph-text joint representation, and use multi-head attention to characterize the interaction relationship between the propagation graph modality and the graph-text modality; using the propagation graph representation as the query and the graph-text joint representation as the key and value, obtain the propagation graph enhanced representation. ,satisfy: ; Step 3.5: To suppress overpropagation and feature drift caused by cross-modal interaction, a residual interpolation mechanism is introduced to fuse the interaction enhancement representation and the original representation according to weights, obtaining the final cross-modal interaction output. ,satisfy: ,in These are hyperparameters used to balance the transition characteristics of the m-mode. With original features Finally, the above This serves as input to subsequent cross-modal fusion discrimination and / or counterfactual causal intervention reasoning modules, thereby improving the contextual consistency and robustness of multimodal representations.

5. The multimodal rumor detection method based on information bottlenecks and counterfactual intervention as described in claim 1, characterized in that, Step 4 includes the following sub-steps: Step 4.1, first, define the fact representation. The system learns sample-specific intervention strategies through a lightweight mask generator and utilizes modality-level mask strength generated by a global controller and an importance predictor. Importance of feature dimensions ,in Indicates the modal level mask strength. This indicates the importance of each feature dimension, specifically, These are learnable parameters that are progressively optimized through a training process to generate the optimal mask; combined with the original mask for a specific modality. , Finally, the unbindified mask is obtained. ; Step 4.2: Use the mask matrix generated by the mask generator to perform element-wise multiplication on the fact representation to generate N counterfactual feature samples. ,in, This represents element-wise multiplication. These are the first corresponding modes. A counterfactual mask; to enable discrete interventions to support end-to-end backpropagation optimization, this invention introduces a soft mask generation mechanism based on Gumbel-Softmax during the training phase, applying a mask with temperature parameters. The Sigmoid function generates a soft mask. During the inference phase, a hard mask is used, i.e. ; Step 4.3, causal effect estimation, firstly, present the factual representation. Counterfactual representation Then, by comparing factual and counterfactual representations, we... The causal effect was estimated and the average of multiple counterfactual samples was calculated. To obtain the final causal effect; Step 4.4, counterfactual feature enhancement, based on the counterfactual causal effect obtained in Step 4.3, through... , , Features obtained after counterfactual causality effect is enhanced ,in The feedforward neural network MLP is used, and the counterfactual causality regularization loss is composed of five losses: contrastive loss, sparsity loss, balance loss, diversity loss, and independence loss. The specific formula is as follows: ,in, It is a comparison of losses. It is sparsity loss. It is a balancing loss. It is a loss of diversity. It is the independence constraint loss, and the specific calculation formula is as follows: , , , , ,in For target similarity, For the purpose of sparsity, It is the variance function. These are the corresponding weight hyperparameters. Combining the causal loss, VIB loss, and the overall classification loss obtained through cross-entropy loss, the final total loss function is: ,in, and It is a hyperparameter that modulates the effects of causal loss and VIB loss.

6. The multimodal rumor detection method based on information bottlenecks and counterfactual intervention as described in claim 1, characterized in that, Step 5, data classification and prediction, involves concatenating the enhanced features into a single data structure. By inputting into a fully connected layer, you can obtain the post. The probability of it being a rumor , can be represented as ,if Then the post is considered to be... Rumors, i.e., prediction tags .

Images