Multimodal false news detection method and system based on queue contrast learning and gating fusion
By employing a queue contrastive learning and gating fusion method, the problems of feature discrimination and fusion strategy in multimodal fake news detection are solved, achieving high accuracy and robustness in fake news detection, applicable to social media platforms.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-03
- Publication Date
- 2026-03-13
AI Technical Summary
Existing technologies are unable to effectively address the problems of weak feature discrimination, coarse fusion strategies, and ambiguous decision boundaries in multimodal fake news, resulting in low detection efficiency and insufficient accuracy.
We employ a queue-based contrastive learning and gating fusion approach. By optimizing the cross-modal contrastive loss and supervised contrastive loss through a dynamic negative sample queue, and combining a multi-layer gating fusion module and an ArcFace classification head, we achieve cross-modal feature representation and decision boundary sharpening.
It significantly improves the accuracy and robustness of fake news detection and is suitable for the automatic identification of multimodal information on social media platforms.
Smart Images
Figure CN121658726A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of multimodal learning and fake news detection technology, and in particular to a multimodal fake news detection method and system based on queue contrastive learning and gating fusion. Background Technology
[0002] With the widespread use of social media, multimodal fake news (false information that combines text with manipulated images and misleading visual elements) spreads rapidly, seriously distorting public perception and threatening social stability. Traditional single-text fake news detection methods struggle to address the challenges posed by modal inconsistencies, while existing multimodal detection schemes have numerous limitations.
[0003] In existing technologies, some methods rely on pre-trained models such as CLIP and BLIP combined with attention fusion, but lack task adaptation for the features of fake news modality mismatch; contrastive learning-related schemes are limited by the small number of negative samples in batches and weak supervision signals, resulting in insufficient feature discrimination; multimodal fusion often adopts simple splicing or static weight allocation, which cannot dynamically capture the complementary relationship and redundant information between modalities; at the same time, some models have problems such as complex training, strong data dependence, and poor robustness in low-resource scenarios.
[0004] These methods fail to address the three core pain points of semantic inconsistency, weak feature discrimination, and coarse fusion methods simultaneously. They are unable to achieve high accuracy and strong generalization ability in fake news identification while ensuring detection efficiency. Therefore, there is an urgent need for a multimodal fake news detection scheme that takes into account feature quality, fusion flexibility, and training stability. Summary of the Invention
[0005] The purpose of this invention is to provide a multimodal fake news detection method and system based on queue contrastive learning and gating fusion with high detection accuracy and robustness, so as to solve the problems of weak feature discrimination, coarse multimodal fusion strategy and vague decision boundary in the prior art.
[0006] The technical solution to achieve the purpose of this invention is: a multimodal fake news detection method based on queue contrastive learning and gating fusion, comprising the following steps:
[0007] Step 1: Extract the original features of the news content using a text encoder and a visual encoder, and map them to a unified space using a projection network;
[0008] Step 2: Introduce a contrastive learning module containing a dynamic negative sample queue to collaboratively optimize the cross-modal contrastive loss, supervised contrastive loss, and anti-collapse regularization term in order to learn highly discriminative cross-modal feature representations;
[0009] Step 3: Design a multi-layer gated fusion module to capture fine-grained cross-modal associations through multi-head cross-attention, and use a dynamic gated network to adaptively generate fusion weights for text and images for each sample to obtain deeply fused multimodal features;
[0010] Step 4: The news classification model uses a classification head based on the ArcFace mechanism. By introducing angular intervals, the decision boundary is sharpened to complete the detection of the authenticity of news.
[0011] A multimodal fake news detection system based on queue contrastive learning and gating fusion is proposed. This system implements the aforementioned multimodal fake news detection method based on queue contrastive learning and gating fusion. The system includes an original feature extraction module and:
[0012] The original feature extraction module uses a text encoder and a visual encoder to extract the original features of the news content and maps them to a unified space through a projection network.
[0013] The contrastive learning module includes a dynamic negative sample queue and collaboratively optimizes the cross-modal contrastive loss, supervised contrastive loss, and anti-collapse regularization term to learn highly discriminative cross-modal feature representations.
[0014] The fusion module employs multi-layer gated fusion, which captures fine-grained cross-modal associations through multi-head cross-attention and uses a dynamic gated network to adaptively generate fusion weights for text and images for each sample, thereby obtaining deeply fused multimodal features.
[0015] The classification module uses a classification head based on the ArcFace mechanism. By introducing angular intervals, the decision boundary is sharpened to complete the detection of the authenticity of news.
[0016] A mobile terminal includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the multimodal fake news detection method based on queue contrastive learning and gating fusion.
[0017] Compared with the prior art, the present invention has the following significant advantages: (1) By introducing a contrastive learning framework with a dynamic negative sample queue, the scale of negative samples is greatly expanded, and the discriminativeness and robustness of features are effectively improved by combining supervision signals and anti-collapse regularization; (2) A multi-layer gating fusion module is designed, which can dynamically and adaptively fuse text and image information according to the specific content of each news article, and accurately capture key cross-modal inconsistencies; (3) The ArcFace classification head is used to sharpen the decision boundary, which significantly improves the classification accuracy of the model in complex scenarios and is suitable for the automatic identification of fake news containing multimodal information such as text and images on social media platforms. Attached Figure Description
[0018] Figure 1 This is a flowchart of the multimodal fake news detection method based on queue contrastive learning and gating fusion of the present invention. Detailed Implementation
[0019] This invention provides a multimodal fake news detection method based on queue contrastive learning and gating fusion, comprising the following steps:
[0020] Step 1: Extract the original features of the news content using a text encoder and a visual encoder, and map them to a unified space using a projection network;
[0021] Step 2: Introduce a contrastive learning module containing a dynamic negative sample queue to collaboratively optimize the cross-modal contrastive loss, supervised contrastive loss, and anti-collapse regularization term in order to learn highly discriminative cross-modal feature representations;
[0022] Step 3: Design a multi-layer gated fusion module to capture fine-grained cross-modal associations through multi-head cross-attention, and use a dynamic gated network to adaptively generate fusion weights for text and images for each sample to obtain deeply fused multimodal features;
[0023] Step 4: The news classification model uses a classification head based on the ArcFace mechanism. By introducing angular intervals, the decision boundary is sharpened to complete the detection of the authenticity of news.
[0024] As a specific example, step 1 is as follows:
[0025] Step 1.1: Extract the raw features of the news text and images using pre-trained text encoders and visual encoders, respectively. The raw features include the global features of the news text. Global features of news photos The pre-trained text encoder is a RoBERTa model, and the visual encoder is a ViT-B / 16 model.
[0026] Step 1.2: Map the original features to a unified contrastive learning space using a projection network to obtain text projection features. and image projection features The projection network is a multilayer perceptron containing two fully connected layers and a GELU activation function.
[0027] As a specific example, step 2 is as follows:
[0028] Step 2.1: Introduce a contrastive learning module containing a dynamic negative sample queue. The dynamic negative sample queue is used to store features of historical batches as additional negative samples. The queue capacity is fixed and is dynamically maintained during training through cyclic enqueue and dequeue operations.
[0029] Step 2.2: Input the text projection features and image projection features into the contrastive learning module containing a dynamic negative sample queue. The total loss of the contrastive learning module includes cross-modal contrastive loss, supervised contrastive loss, and anti-collapse regularization term. The total loss weight of the contrastive learning module increases linearly in the first few rounds of training.
[0030] Step 2.3: Collaborately optimize the cross-modal contrastive loss, supervised contrastive loss, and anti-collapse regularization term to learn highly discriminative cross-modal feature representations.
[0031] As a specific example, step 2.1 involves dynamic maintenance during training through cyclic enqueue and dequeue operations, specifically as follows:
[0032] In the first round of training, a warm start is performed, where features are simply added to the queue without being removed.
[0033] Starting from the second round, after each training batch is completed, the features of the current batch are added to the queue, and the oldest features are removed.
[0034] As a specific example, the total contrastive learning loss mentioned in step 2.2 includes cross-modal contrastive loss, supervised contrastive loss, and anti-collapse regularization term, as follows:
[0035] (1) Cross-modal contrast loss, which uses historical features in the queue as a large-scale negative sample;
[0036] (2) Monitored contrast loss, activated after two rounds of warm start, using label information to bring similar samples closer and push away dissimilar samples;
[0037] (3) Anti-collapse regularization term, which is composed of VICReg loss and Barlow Twins loss weighted sum, is used to maintain feature diversity, constrain the variance, invariance and covariance of features, and prevent feature dimension collapse.
[0038] As a specific example, the multi-layer gating fusion module described in step 3 stacks multiple basic gating fusion units; each basic gating fusion unit specifically performs the following operations:
[0039] First, using text features as queries and image features as keys and values, and vice versa, two multi-head cross-attention calculations are performed to obtain enhanced text features and image features.
[0040] Then, the enhanced text features, image features and their interaction information are input into a gating network, and the text fusion weights and image fusion weights are output through the Softmax function;
[0041] Finally, the enhanced features are weighted using fusion weights and then subjected to nonlinear transformation and residual connection through a feedforward network to obtain the output features of the basic gated fusion unit.
[0042] As a specific example, step 3 is as follows:
[0043] The features optimized through contrastive learning are input into a multi-layer gated fusion module, which stacks two basic gated fusion units. For the l-th unit (l=1,2), the input is the text features from the previous layer. and image features Initially ;
[0044] First, cross-attention enhancement is performed to obtain enhanced features of the news text. Enhanced features of news photos :
[0045]
[0046] CrossAttn is an 8-head cross-attention mechanism;
[0047] Then, the enhanced features and their interaction information will be... Input a two-layer fully connected layer with GELU activation, and output the text weights through the Softmax function. Image weights ;
[0048] Finally, the enhanced features are weighted and fused, and nonlinear transformation and residual connection are performed through a feedforward network to obtain the output of this layer. The feedforward network includes layer normalization, linear transformation, GELU and Dropout.
[0049] After two stacked layers, the final text and image features are averaged and then passed through a projection layer to obtain a deep-fused multimodal feature vector. .
[0050] As a specific example, the news classification model described in step 4 uses a classification head based on the ArcFace mechanism. By introducing angular intervals to sharpen the decision boundary, it completes the detection of the authenticity of news, as detailed below:
[0051] First, L2 normalization is performed on the deep fusion multimodal feature vectors and classification weight matrix;
[0052] Secondly, a dynamic margin is added from the perspective of the target category, and after multiplying by a scaling factor, a learnable class bias is added to obtain the final classification logits; the dynamic margin starts from an initial value in the early stage of training and increases linearly to the target value with each training round.
[0053] Finally, the probability of the news being true or false is output through the Softmax function; the loss function of the news classification model adopts label-smoothed cross-entropy loss and is optimized by combining a class-balanced weighting strategy.
[0054] This invention also provides a multimodal fake news detection system based on queue contrastive learning and gating fusion. This system is used to implement the aforementioned multimodal fake news detection method based on queue contrastive learning and gating fusion. The system includes an original feature extraction module and:
[0055] The original feature extraction module uses a text encoder and a visual encoder to extract the original features of the news content and maps them to a unified space through a projection network.
[0056] The contrastive learning module includes a dynamic negative sample queue and collaboratively optimizes the cross-modal contrastive loss, supervised contrastive loss, and anti-collapse regularization term to learn highly discriminative cross-modal feature representations.
[0057] The fusion module employs multi-layer gated fusion, which captures fine-grained cross-modal associations through multi-head cross-attention and uses a dynamic gated network to adaptively generate fusion weights for text and images for each sample, thereby obtaining deeply fused multimodal features.
[0058] The classification module uses a classification head based on the ArcFace mechanism. By introducing angular intervals, the decision boundary is sharpened to complete the detection of the authenticity of news.
[0059] The present invention also provides a mobile terminal, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the multimodal fake news detection method based on queue contrastive learning and gating fusion.
[0060] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.
[0061] Example
[0062] like Figure 1 As shown, the present invention provides a multimodal fake news detection method based on queue contrastive learning and gating fusion, comprising the following steps:
[0063] Step 1: Extract the original features of the news content using a text encoder and a visual encoder, and map them to a unified space using a projection network;
[0064] Step 2: Introduce a contrastive learning module containing a dynamic negative sample queue to collaboratively optimize the cross-modal contrastive loss, supervised contrastive loss, and anti-collapse regularization term in order to learn highly discriminative cross-modal feature representations;
[0065] Step 3: Design a multi-layer gated fusion module to capture fine-grained cross-modal associations through multi-head cross-attention, and use a dynamic gated network to adaptively generate fusion weights for text and images for each sample to obtain deeply fused multimodal features;
[0066] Step 4: The news classification model uses a classification head based on the ArcFace mechanism. By introducing angular intervals, the decision boundary is sharpened to complete the detection of the authenticity of news.
[0067] Further, in step 1, pre-trained text encoders and visual encoders are used to extract the original features of news text and images, respectively, and the original features are mapped to a unified contrastive learning space through a projection network to obtain text projection features and image projection features.
[0068] The RoBERTa model was used as a text encoder to extract global features from the news text. The ViT-B / 16 model was used as the visual encoder to extract global features from news images. .Will and By inputting each of the two fully connected projection networks (with the middle layer using the GELU activation function), we obtain text projection features of dimension 512. and image projection features .
[0069] Further, in step 2, the text projection features and image projection features are input into a contrastive learning module containing a dynamic negative sample queue. This module collaboratively optimizes the cross-modal contrastive loss, supervised contrastive loss, and anti-collapse regularization term to learn a highly discriminative cross-modal feature representation. Specifically:
[0070] Construct a dynamic negative sample queue with a capacity of 4096. In the first round of training, perform a warm start by simply adding features to the queue without removing any. Starting from the second round, after each training batch, add the features of the current batch to the queue and remove the oldest features.
[0071] The total loss of the contrastive learning module consists of three parts: (1) cross-modal contrastive loss, which uses historical features in the queue as large-scale negative samples; (2) supervised contrastive loss, which is activated after two rounds of warm start, using label information to bring similar samples closer and push away dissimilar samples; and (3) anti-collapse regularization term, which is composed of a weighted sum of VICReg loss and Barlow Twins loss, used to maintain feature diversity. The total loss weight of this contrastive learning module increases linearly from 0 to 1 in the first 5 rounds of training.
[0072] Further, in step 3, the features optimized through contrastive learning are input into a multi-layer gated fusion module. A multi-head cross-attention mechanism captures fine-grained cross-modal associations, and a dynamic gated network adaptively generates fusion weights for text and images for each sample, resulting in a deeply fused multimodal feature vector. Specifically:
[0073] The multi-layer gated fusion module stacks two basic gated fusion units. For the l-th unit, l=1,2, its input is the text features from the previous layer. and image features (Initial) ).
[0074] First, perform cross-attention enhancement:
[0075]
[0076] CrossAttn is an 8-head cross-attention mechanism.
[0077] Then, the enhanced features and their interaction information will be... Input a two-layer fully connected layer (GELU activation), and output the text weights through the Softmax function. Image weights .
[0078] Finally, the enhanced features are weighted and fused, and then nonlinearly transformed and residual connected through a feedforward network (including layer normalization, linear transformation, GELU and Dropout) to obtain the output of this layer.
[0079] After two stacked layers, the final text and image features are averaged and then passed through a projection layer to obtain deep fused features. .
[0080] Furthermore, in step 4, the news is classified as true or false based on its feature vector and a news classification model. Specifically:
[0081] The news classification model described is a classification head based on the ArcFace mechanism. Firstly, it involves deep fusion features. and classification weight matrix Perform L2 normalization. Then, from the perspective corresponding to the target category... Add a dynamic interval In addition, a learnable class bias is introduced. To alleviate the data imbalance problem, the final classification logits are calculated using the following formula: , where the scale factor Dynamic interval The error rate linearly increased from 0.15 to 0.30 during the first 10 training rounds. The model employed a cross-entropy loss with a label smoothing coefficient of 0.1 and was optimized using a class-balanced weighting strategy.
[0082] In summary, this invention provides a multimodal fake news detection method based on queue contrastive learning and gating fusion. There are many methods and approaches to implement this technical solution. The above description is only a preferred embodiment of this invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this invention, and these improvements and modifications should also be considered within the scope of protection of this invention.
Claims
1. A multimodal fake news detection method based on queue contrastive learning and gating fusion, characterized in that, Includes the following steps: Step 1: Extract the original features of the news content using a text encoder and a visual encoder, and map them to a unified space using a projection network; Step 2: Introduce a contrastive learning module containing a dynamic negative sample queue to collaboratively optimize the cross-modal contrastive loss, supervised contrastive loss, and anti-collapse regularization term in order to learn highly discriminative cross-modal feature representations; Step 3: Design a multi-layer gated fusion module to capture fine-grained cross-modal associations through multi-head cross-attention, and use a dynamic gated network to adaptively generate fusion weights for text and images for each sample to obtain deeply fused multimodal features; Step 4: The news classification model uses a classification head based on the ArcFace mechanism. By introducing angular intervals, the decision boundary is sharpened to complete the detection of the authenticity of news.
2. The multimodal fake news detection method based on queue contrastive learning and gating fusion as described in claim 1, characterized in that, Step 1 is described in detail as follows: Step 1.1: Extract the raw features of the news text and images using pre-trained text encoders and visual encoders, respectively. The raw features include the global features of the news text. Global features of news photos The pre-trained text encoder is a RoBERTa model, and the visual encoder is a ViT-B / 16 model. Step 1.2: Map the original features to a unified contrastive learning space using a projection network to obtain text projection features. and image projection features The projection network is a multilayer perceptron containing two fully connected layers and a GELU activation function.
3. The multimodal fake news detection method based on queue contrastive learning and gating fusion as described in claim 2, characterized in that, Step 2 is described in detail below: Step 2.1: Introduce a contrastive learning module containing a dynamic negative sample queue. The dynamic negative sample queue is used to store features of historical batches as additional negative samples. The queue capacity is fixed and is dynamically maintained during training through cyclic enqueue and dequeue operations. Step 2.2: Input the text projection features and image projection features into the contrastive learning module containing a dynamic negative sample queue. The total loss of the contrastive learning module includes cross-modal contrastive loss, supervised contrastive loss, and anti-collapse regularization term. The total loss weight of the contrastive learning module increases linearly in the first few rounds of training. Step 2.3: Collaborately optimize the cross-modal contrastive loss, supervised contrastive loss, and anti-collapse regularization term to learn highly discriminative cross-modal feature representations.
4. The multimodal fake news detection method based on queue contrastive learning and gating fusion as described in claim 3, characterized in that, In step 2.1, dynamic maintenance is performed during the training process through cyclic enqueue and dequeue operations, specifically as follows: In the first round of training, a warm start is performed, where features are simply added to the queue without being removed. Starting from the second round, after each training batch is completed, the features of the current batch are added to the queue, and the oldest features are removed.
5. The multimodal fake news detection method based on queue contrastive learning and gating fusion according to claim 3, characterized in that, The total contrastive learning loss mentioned in step 2.2 includes cross-modal contrastive loss, supervised contrastive loss, and anti-collapse regularization term, as detailed below: (1) Cross-modal contrast loss, which uses historical features in the queue as a large-scale negative sample; (2) Monitored contrast loss, activated after two rounds of warm start, using label information to bring similar samples closer and push away dissimilar samples; (3) Anti-collapse regularization term, which is composed of VICReg loss and Barlow Twins loss weighted sum, is used to maintain feature diversity, constrain the variance, invariance and covariance of features, and prevent feature dimension collapse.
6. The multimodal fake news detection method based on queue contrastive learning and gating fusion as described in claim 3, characterized in that, The multi-layer gating fusion module described in step 3 stacks multiple basic gating fusion units; each basic gating fusion unit performs the following operations: First, using text features as the query and image features as the key and value, and vice versa, two multi-head cross-attention calculations are performed to obtain the enhanced text features and image features. Then, the enhanced text features, image features and their interaction information are input into a gating network, and the text fusion weights and image fusion weights are output through the Softmax function; Finally, the enhanced features are weighted using fusion weights and then subjected to nonlinear transformation and residual connection through a feedforward network to obtain the output features of the basic gated fusion unit.
7. The multimodal fake news detection method based on queue contrastive learning and gating fusion as described in claim 6, characterized in that, Step 3 is as follows: The features optimized through contrastive learning are input into a multi-layer gated fusion module, which stacks two basic gated fusion units. For the l-th unit (l=1,2), the input is the text features from the previous layer. and image features Initially ; First, cross-attention enhancement is performed to obtain enhanced features of the news text. Enhanced features of news photos : ； CrossAttn is an 8-head cross-attention mechanism; Then, the enhanced features and their interaction information will be... Input a two-layer fully connected layer with GELU activation, and output the text weights through the Softmax function. Image weights ; Finally, the enhanced features are weighted and fused, and nonlinear transformation and residual connection are performed through a feedforward network to obtain the output of this layer. The feedforward network includes layer normalization, linear transformation, GELU and Dropout. After two stacked layers, the final text and image features are averaged and then passed through a projection layer to obtain a deep-fused multimodal feature vector. .
8. The multimodal fake news detection method based on queue contrastive learning and gating fusion according to claim 1, characterized in that, Step 4 describes a news classification model that uses a classification head based on the ArcFace mechanism. It sharpens the decision boundary by introducing angular intervals to detect the authenticity of news, as detailed below: First, L2 normalization is performed on the deep fusion multimodal feature vectors and classification weight matrix; Secondly, a dynamic margin is added from the perspective of the target category, and after multiplying by a scaling factor, a learnable class bias is added to obtain the final classification logits; the dynamic margin starts from an initial value in the early stage of training and increases linearly to the target value with each training round. Finally, the probability of the news being true or false is output through the Softmax function; the loss function of the news classification model adopts label-smoothed cross-entropy loss and is optimized by combining a class-balanced weighting strategy.
9. A multimodal fake news detection system based on queue contrastive learning and gating fusion, characterized in that, This system is used to implement the multimodal fake news detection method based on queue contrastive learning and gating fusion as described in any one of claims 1 to 8, the system comprising an original feature extraction module, and: The original feature extraction module uses a text encoder and a visual encoder to extract the original features of the news content and maps them to a unified space through a projection network. The contrastive learning module includes a dynamic negative sample queue and collaboratively optimizes the cross-modal contrastive loss, supervised contrastive loss, and anti-collapse regularization term to learn highly discriminative cross-modal feature representations. The fusion module employs multi-layer gated fusion, which captures fine-grained cross-modal associations through multi-head cross-attention and uses a dynamic gated network to adaptively generate fusion weights for text and images for each sample, thereby obtaining deeply fused multimodal features. The classification module uses a classification head based on the ArcFace mechanism. By introducing angular intervals, the decision boundary is sharpened to complete the detection of the authenticity of news.
10. A mobile terminal, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the multimodal fake news detection method based on queue contrastive learning and gating fusion as described in any one of claims 1 to 8.