Multi-stage nested hierarchical fusion rumor detection method
By employing a multi-stage nested hierarchical fusion method, this approach utilizes BERT and VGG-19 to extract features and combines them with a multi-head attention mechanism and loss function. This addresses the issue of insufficient intermodal interaction relationships in multimodal fake information detection, thereby improving the accuracy and robustness of fake information detection.
Patent Information
- Application Number
- CN202511704428.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-19
- Publication Date
- 2026-02-27
AI Technical Summary
Existing multimodal misinformation detection methods have shortcomings in terms of intermodal alignment, robustness to noisy data, and model interpretability. They are unable to capture the synergistic and conflicting relationships between local features and global semantics across modalities, leading to a decline in misinformation detection performance.
We employ a multi-stage nested hierarchical fusion approach, extracting text and image features using BERT and VGG-19, enhancing the interaction between local and global features through multi-head attention and multi-level learning, and combining InfoNCE loss and cross-entropy loss functions to achieve comprehensive processing of multimodal information.
It significantly improves the accuracy and robustness of multimodal misinformation detection, and enhances the model's detection capabilities in complex fake news scenarios.
Smart Images
Figure CN121580288A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to a multi-stage, nested, layered, and fusion-based multimodal method for detecting misinformation, belonging to the field of misinformation detection technology. Background Technology
[0002] In recent years, the rapid popularization of social networking platforms has greatly expanded the ways in which users generate and disseminate information. Content formats have evolved from simple text to multimodal expressions integrating text, images, audio, and video. While this richness of content formats enhances the persuasiveness of information, it also significantly increases the concealment and difficulty in identifying misinformation. Multimodal misinformation, by combining misleading images, videos, and inflammatory text, confuses the public in a more deceptive way. This not only easily leads to widespread dissemination but also amplifies the negative social impact of misinformation, erodes the credibility of the online information environment, and even poses a threat to public safety and social stability. Against this backdrop, developing technical methods to effectively identify multimodal misinformation has become a key challenge in information trust computing and content security governance, possessing significant academic research value and urgent practical application needs.
[0003] Traditional methods for detecting multimodal fake content often rely on single-modal feature analysis, making it difficult to capture semantic inconsistencies and collaborative deception patterns across modalities. In recent years, deep learning-based multimodal fusion models have gradually become mainstream research, enhancing the ability to identify logical contradictions and contextual conflicts by jointly modeling heterogeneous information such as text, images, and audio. However, existing methods still have shortcomings in terms of intermodal alignment, robustness to noisy data, and model interpretability.
[0004] The relationships between multimodal information are complex, both within a single modality and between different modalities. Relying on traditional intermodal interaction relationships can easily overlook the impact of local features on the global picture. For example, in a news story about a fire, from a global perspective, the image does show a fire being put out, and the text describes the time of the fire. However, from a local perspective, if the fire scene is in the ocean, then it is likely fake news. Therefore, local text features play an irreplaceable role in the overall image, and local image features play an irreplaceable role in the overall text. Thus, the role of local features should be emphasized.
[0005] However, current mainstream methods in multimodal fake news detection research largely rely on the interaction of overall modal representations or the alignment of local units, failing to fully explore the synergistic and conflicting relationships between local cues and global semantics. While these methods have achieved considerable results on most benchmark datasets, their architectures often lack sensitivity to fine-grained inconsistencies within and between modalities. In real-world dissemination scenarios, fake news often implants misleading information through subtle manipulations such as altering local image regions and replacing key text entities, without affecting overall semantic coherence, creating subtle semantic gaps between local content and the global context. Traditional models, failing to construct a multi-level reasoning mechanism from local regions to overall semantics, are limited in recognizing such implicit contradictions, especially when fake news has high visual-text surface alignment, where detection performance tends to decline significantly. Therefore, achieving synergistic perception and contradiction capture of cross-modal local-global semantics is a key challenge in improving the robustness and interpretability of fake news identification.
[0006] To effectively utilize multimodal data and deeply explore the feature interactions between different modalities, researchers need to explore more optimized models that can simultaneously consider the interaction of local and global information, and perform multi-scale decomposition and multi-level parsing learning to enhance the accuracy and robustness of fake information detection. Through deep learning, attention mechanisms, and residual neural networks, models that capture local and global features can be constructed to serve the information needs of multimodal fake information detection. Summary of the Invention
[0007] To effectively overcome the limitation of local features in feature interaction during multimodal misinformation detection, this invention provides a multi-stage, nested, hierarchical fusion method for multimodal misinformation detection. Specifically, it includes the following steps: First, multimodal data from social media platforms such as Weibo and Twitter are acquired and the text and image data are processed. Then, a text feature extractor and an image feature extractor are constructed. The extracted text and image features are then projected into the same space for local semantic consistency detection. Global semantic consistency detection is then performed on the acquired global features, generating a joint semantic consistency result. Next, multi-stage learning is performed. In each stage, features are decomposed into multiple scales, and a multi-head attention mechanism is used to enhance the interaction between local features and another modality. The enhanced modal features are then concatenated with the original features, and inter-modal interaction is performed again. Finally, the multi-level results and the results of multi-stage learning are fused. Finally, a loss function is established by combining the fusion result and the semantic consistency result, thereby achieving misinformation detection. This invention can effectively improve the accuracy and efficiency of multimodal misinformation detection.
[0008] 1. A multi-stage nested hierarchical fusion rumor detection method, characterized by comprising the following steps:
[0009] Step 1: Dataset Acquisition. Acquire multimodal news data from social networks (such as Weibo and Twitter) to build the dataset needed for training the model. Each data point in the training dataset includes multimodal news data and a label indicating whether it is false information. The multimodal news data includes text data and image data.
[0010] Step 2: Extracting basic image and text features. The preprocessed text data from Step 1 is input into a pre-trained BERT model, whose parameters are frozen to obtain sentence-level feature vectors. The preprocessed image data from Step 1 is input into a pre-trained VGG-19 model to obtain image-level feature vectors. The obtained word vectors and primitive vectors are mapped to the same latent space, then normalized using the Normalize method and activated using ReLU.
[0011] Step 3 involves local matching of image and text features and enhanced feature extraction. The word vectors and primitive vectors are decomposed and enhanced at multiple scales, and then enhanced through multi-level interaction and multi-stage learning to form a feature matrix. The pair of features of the most closely aligned text and image modalities are extracted and subjected to multiple batch normalization and linear transformation of the weight matrix. Finally, the softmax activation function is used to generate the predicted probability distribution of the two classes.
[0012] Step 4, social media information detection: The local fragment-level matching results obtained in Step 3 and the relevance matching results obtained through similarity learning are input into a binary classifier. The sum of the result vectors at the two levels is projected into the target space of real information and fake information by the fully connected layer with the softmax function in the binary classifier, and the detection result of social media information is obtained.
[0013] In step 2:
[0014] The extraction of local text features specifically includes: encoding text information word by word using the Local Feature Extractor BERT, extracting the representation of each word in the context, and capturing the local information of each word in the context through BERT's own attention mechanism. Let a text segment be represented as B = {b1, b2, ..., b...} n}, where n is the number of tokens. This represents obtaining phrase-level information for each word using convolutional kernels of different sizes, b represents the word obtained from word segmentation in step 2, and T represents the local features of the segmented word extracted after BERT processing.
[0015]
[0016] Then, average pooling is performed on T to obtain x.t :
[0017]
[0018] The extraction of local image features specifically includes: dividing the input image into fixed-size local image blocks using the VGG-19 local feature extractor, and performing average pooling to obtain x. v :
[0019]
[0020] In step 3:
[0021] For each local image patch x extracted in step 2 v and text features x t All are mapped to the same space, and after normalization using the Normalize method and activation using ReLU, h is obtained. v and h t :
[0022]
[0023] For each x t Each x is treated as Q. v This operation utilizes an improved multi-head attention mechanism to enhance information interaction between different modalities, while x v With x t The multiscale nature of x can be adjusted according to specific experimental conditions, but x v With x t The initial number of scales should be the same.
[0024]
[0025] The enhanced feature r is concatenated with the original feature to obtain the original feature of the second level.
[0026]
[0027] The final enhanced modal features are obtained by performing an outer product on the feature vectors obtained through multiple levels (assuming there are m levels).
[0028]
[0029] Considering the randomness of the interaction, we adopt a multi-stage learning approach, performing multiple multi-scale hierarchical interactions on the original features, and finally forming a feature matrix from the feature vectors of the multiple stages.
[0030]
[0031] The diagonal elements are then used as matching multimodal pairs, where each element corresponds to a pair of features from the most closely aligned text and image modalities in the feature space.
[0032] z = dim(x mm )
[0033] After batch normalization and a linear transformation using the weight matrix, the hidden layer output is obtained. Then, another linear transformation is performed using another weight matrix, and finally, the softmax activation function is used to generate the predicted probability distributions for the two classes.
[0034] In step 4:
[0035] Classification loss is defined as Cross-entropy with the true label y:
[0036]
[0037] To encourage consistency between patterns and perform similarity learning, the projected features are first normalized: cosine similarity is calculated.
[0038] To facilitate alignment between multimodal features, InfoNCEloss is introduced to maximize the similarity of matched sample pairs in the feature space while minimizing the similarity between mismatched sample pairs, thereby enhancing the model's understanding of cross-modal data. It is deeply integrated with the model's multi-stage interaction mechanism to ensure optimized alignment of text and image features at different stages.
[0039]
[0040] This invention addresses the limitations of existing technologies in handling the interaction between local and global features. It proposes a method to enhance this interaction, enabling features extracted from different modalities to fully interact with another modality, achieving comprehensive processing of multimodal information. To address the issue of insufficient feature interaction, this invention provides a multi-stage nested hierarchical fusion rumor detection method that effectively captures the complex semantic relationships between image and text modalities, thereby improving the semantic understanding of cross-modal data. By deeply fusing multimodal local features learned through multi-stage learning, this invention significantly improves the accuracy of multimodal misinformation detection, enhances the model's robustness and generalization ability in complex fake news scenarios, and is applicable to the fields of multimodal data processing and misinformation detection. Attached Figure Description
[0041] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0042] Figure 1 This is a flowchart of a multi-stage nested layered fusion rumor detection method according to the present invention;
[0043] Figure 2 This is a system architecture diagram of a multi-stage nested layered fusion rumor detection method according to the present invention; Detailed Implementation
[0044] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0045] Before introducing the technical solution of this invention, the technical terms involved will first be explained or described:
[0046] BERT (Bidirectional Encoder Representations from Transformers) is a pre-trained language model that uses a bidirectional Transformer architecture to consider contextual information simultaneously, enabling it to effectively understand the meaning of words and sentences in text.
[0047] VGG-19 is a deep convolutional neural network consisting of 16 stacked convolutional layers and 3 fully connected layers. Its main function is to achieve efficient image feature extraction through a deep architecture.
[0048] Self-attention is a mechanism for calculating the similarity between each element in a sequence and other elements. It re-represents each element by weighted summation, enabling the model to focus on important information related to the current element in the sequence and capture the dependencies between elements.
[0049] Reference Figure 1 and Figure 2 A multi-stage nested hierarchical fusion rumor detection method comprises three modules: a feature extraction module, a multi-stage nested hierarchical interaction module, and a contrastive learning module. The functions of each module are as follows:
[0050] (1) Feature Extraction Module: The feature extraction process of this method includes two stages: data preprocessing and feature extraction. First, labeled multimodal news data is acquired to construct a training set. Then, data preprocessing is performed: for text data, word segmentation is performed and the data is converted into word sequences, while all sequences are standardized to a fixed length through truncation or zero-padding; for image data, it is uniformly scaled to a fixed size required by the model. The preprocessed data will be input into the model for subsequent global and local feature extraction. For text feature extraction, a BERT pre-trained model is used to extract features x. t The image was extracted using a VGG-19 pre-trained model. v .
[0051] 1. Text Feature Extraction: The text information is encoded word by word using the Local Feature Extractor (BERT), extracting the representation of each word in the context. BERT's attention mechanism captures the local information of each word in the context. Let a text be represented as B = {b1, b2, ..., b...} n}, where n is the number of tokens. represents the phrase-level information of each word obtained through convolutional kernels of different sizes, b represents the word obtained from the word segmentation process in step two, and T represents the local features of the segmented word extracted after BERT processing. The specific formula is as follows:
[0052]
[0053] Then, average pooling is performed on T to obtain x. t The specific formula is as follows:
[0054]
[0055] 2. Image Feature Extraction: The input image is divided into fixed-size local image patches using the VGG-19 local feature extractor, and average pooling is performed to obtain x. v The specific formula is as follows:
[0056]
[0057] (2) Multi-stage nested hierarchical interaction module: used to enhance feature interaction between different modalities, such as Figure 2 As shown. Each local image patch x extracted from (l) v and text features x t All features are mapped to the same space, normalized using the Normalize method, activated using ReLU, and then enhanced through a multi-scale, multi-level multi-head attention mechanism. After multiple stages of enhancement, the multi-stage hierarchical results are finally deeply fused.
[0058] 1. Consistency processing: Each extracted local image patch x v and text features x t All are mapped to the same space, and after normalization using the Normalize method and activation using ReLU, h is obtained. v and h t The specific formula is as follows:
[0059] The above method further unifies the dimensions of text and images, making it easier to maintain consistent dimensions in the next interactive task and preventing different shapes from causing the model to malfunction.
[0060] 2. Feature enhancement: For each x... t Each x is treated as Q. v This operation utilizes an improved multi-head attention mechanism to enhance information interaction between different modalities, while x v With x t The multiscale nature of x can be adjusted according to specific experimental conditions, but x v With x t The initial number of scales should be the same, and the specific formula is as follows:
[0061]
[0062] The enhanced feature r is concatenated with the original feature to obtain the original feature of the second level. The specific formula is as follows:
[0063] The final enhanced modal features are obtained by performing an outer product on the feature vectors obtained through multiple levels (assuming there are m levels). The specific formula is as follows:
[0064]
[0065] The beneficial effect of the above method is that it decomposes the original text and image features into more fine-grained multi-scale features, and then interacts with the corresponding modalities respectively. This allows for a more comprehensive acquisition of the interaction between each local feature and the corresponding global feature. The enhanced features are then concatenated with the original features containing the original information, and different modal interactions are performed. Finally, the results of each interaction are combined by outer product to obtain the fully interactive local features.
[0066] 3. Stage Learning: Considering the randomness of the interaction, we adopt a multi-stage learning method, performing multiple multi-scale hierarchical interactions on the original features, and finally forming a feature matrix from the feature vectors of the multiple stages. The specific formula is as follows:
[0067]
[0068] Extract its diagonal elements as matching multimodal pairs, where each element corresponds to a pair of features of the most closely aligned text and image modalities in the feature space.
[0069] z = dim(x mm )
[0070] After batch normalization and a linear transformation using the weight matrix, the hidden layer output is obtained. Then, another linear transformation is performed using another weight matrix, and finally, the softmax activation function is used to generate the predicted probability distributions for the two classes.
[0071]
[0072] The above method performs R rounds of reinforcement learning in step 2, thereby further enhancing the interaction between different modalities. It also selects diagonal elements as the best-matching modal pairs for detection. By measuring the consistency between different modalities, it helps the model correct potential contradictions or inconsistencies between modalities, thereby enhancing the global grasp of the authenticity of information.
[0073] (3) Comparative learning module:
[0074] Classification loss is defined as Cross-entropy with the true label y:
[0075]
[0076] To encourage consistency between patterns, we perform similarity learning. We first normalize the projected features: we calculate the cosine similarity.
[0077]
[0078] To facilitate alignment between multimodal features, InfoNCE loss is introduced to maximize the similarity of matching sample pairs in the feature space while minimizing the similarity between mismatched sample pairs, thereby enhancing the model's understanding of cross-modal data. It is deeply integrated with the model's multi-stage interaction mechanism to ensure optimized alignment of text and image features at different stages.
[0079]
[0080] The beneficial effects of the above scheme are that it enhances the consistency between modalities and improves the feature alignment between modalities through similarity learning, thereby making it easier to select the best-matching samples and distinguish the least-matching samples. This represents the final loss function, which the model will learn to make better predictions given the input.
[0081] This invention addresses the limitations of existing technologies in handling the interaction between local and global features. It proposes a method to enhance this interaction, enabling features extracted from different modalities to fully interact with another modality, achieving comprehensive processing of multimodal information. To address the issue of insufficient feature interaction, this invention provides a multi-stage nested hierarchical fusion rumor detection method that effectively captures the complex semantic relationships between image and text modalities, thereby improving the semantic understanding of cross-modal data. By deeply fusing multimodal local features learned through multi-stage learning, this invention significantly improves the accuracy of multimodal misinformation detection, enhances the model's robustness and generalization ability in complex fake news scenarios, and is applicable to the fields of multimodal data processing and misinformation detection.
[0082] The embodiments of the present invention have been described in detail above with reference to the accompanying drawings, but the present invention is not limited to the described embodiments. For those skilled in the art, various changes, modifications, substitutions, and variations can be made to these embodiments without departing from the principles and spirit of the present invention, and these variations still fall within the protection scope of the present invention.
Claims
1. A multi-stage nested hierarchical fusion rumor detection method, characterized in that, Includes the following steps: Step 1: Dataset Acquisition. Acquire multimodal news data from social networks (such as Weibo and Twitter) to build the dataset needed for training the model. Each data point in the training dataset includes multimodal news data and a label indicating whether it is false information. The multimodal news data includes text data and image data. Step 2: Extracting basic image and text features. The preprocessed text data from Step 1 is input into a pre-trained BERT model, whose parameters are frozen to obtain sentence-level feature vectors. The preprocessed image data from Step 1 is input into a pre-trained VGG-19 model to obtain image-level feature vectors. The obtained word vectors and primitive vectors are mapped to the same latent space, then normalized using the Normalize method and activated using ReLU. Step 3 involves local matching of image and text features and enhanced feature extraction. The word vectors and primitive vectors are decomposed and enhanced at multiple scales, and then enhanced through multi-level interaction and multi-stage learning to form a feature matrix. The pair of features of the most closely aligned text and image modalities are extracted and subjected to multiple batch normalization and linear transformation of the weight matrix. Finally, the softmax activation function is used to generate the predicted probability distribution of the two classes. Step 4, social media information detection: The local fragment-level matching results obtained in Step 3 and the relevance matching results obtained through similarity learning are input into a binary classifier. The sum of the result vectors at the two levels is projected into the target spaces of real information and fake information using a fully connected layer with a softmax function in the binary classifier, and the detection results of social media information are obtained.
2. The multi-stage nested hierarchical fusion rumor detection method according to claim 1, characterized in that: In step 2: The extraction of local text features specifically includes: encoding text information word by word using the Local Feature Extractor BERT, extracting the representation of each word in the context, and capturing the local information of each word in the context through BERT's own attention mechanism. Let a text segment be represented as B = {b1, b2, ..., b...} n }, Where n is the number of tokens. This represents the phrase-level information of each word obtained through convolutional kernels of different sizes, b represents the word obtained from the word segmentation process in step two, and T represents the local features of the segmented word extracted after BERT processing. Then, average pooling is performed on T to obtain x. t : The extraction of local image features specifically includes: dividing the input image into fixed-size local image blocks using the VGG-19 local feature extractor, and performing average pooling to obtain x. v :
3. The multi-stage nested hierarchical fusion rumor detection method according to claim 1, characterized in that, In step 3: For each local image patch x extracted in step 2 v and text features x t All are mapped to the same space, and after normalization using the Normalize method and activation using ReLU, h is obtained. v and h t : For each x t Each x is treated as Q. v This operation utilizes an improved multi-head attention mechanism to enhance information interaction between different modalities, while x v With x t The multiscale nature of x can be adjusted according to specific experimental conditions, but x v With x t The initial number of scales should be the same. The enhanced feature r is concatenated with the original feature to obtain the original feature of the second level. The final enhanced modal features are obtained by performing an outer product on the feature vectors obtained through multiple levels (assuming there are m levels). Considering the randomness of the interaction, we adopt a multi-stage learning approach, performing multiple multi-scale hierarchical interactions on the original features, and finally forming a feature matrix from the feature vectors of the multiple stages. The diagonal elements are then used as matching multimodal pairs, where each element corresponds to a pair of features from the most closely aligned text and image modalities in the feature space. z=dim(x mm ) After batch normalization and a linear transformation using the weight matrix, the hidden layer output is obtained. Then, another linear transformation is performed using another weight matrix, and finally, the softmax activation function is used to generate the predicted probability distributions for the two classes.
4. The multi-stage nested hierarchical fusion rumor detection method according to claim 1, characterized in that, In step 4: Classification loss is defined as Cross-entropy with the true label y: To encourage consistency between patterns, we perform similarity learning. We first normalize the projected features: we calculate the cosine similarity. To facilitate alignment between multimodal features, InfoNCE loss is introduced to maximize the similarity of matching sample pairs in the feature space while minimizing the similarity between mismatched sample pairs, thereby enhancing the model's understanding of cross-modal data. It is deeply integrated with the model's multi-stage interaction mechanism to ensure optimized alignment of text and image features at different stages.
5. A computer device comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the adaptive enhancement method for non-uniformly illuminated images as described in any one of claims 1 to 4.
6. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the adaptive enhancement method for non-uniformly illuminated images as described in any one of claims 1 to 4.