Multi-modal false news detection method and system based on test stage adaptation strategy
Through the multimodal fake news detection framework that adapts to the strategy during the testing stage, and using the self-supervised masked autoencoder to optimize the model, the problem of insufficient generalization ability in multimodal fake news detection is solved, and higher detection accuracy and flexibility are achieved, and training costs are reduced.
Patent Information
- Application Number
- CN202510454232.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-04-11
- Publication Date
- 2025-08-29
AI Technical Summary
The existing multimodal fake news detection methods lack generalization capabilities when processing novel data, and relying on a large amount of labeled data leads to high costs and insufficient data set size and diversity. They ignore the unique characteristics of multimodal data and fail to effectively deal with the distribution differences between training data and test data.
Using a multimodal fake news detection framework based on the test stage adaptation strategy, the masked autoencoder task optimization model is used to dynamically adapt to new data features in the test stage, combined with advanced feature extraction and multimodal fusion technology, a self-supervised masked autoencoder is used to capture subtle features, and the model's robustness is improved through multimodal masking filtering and integrating information.
Improve the accuracy and robustness of fake news detection, reduce dependence on labeled data, and improve the model's adaptability in dynamic social media environments.
Smart Images
Figure CN120561785A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of fake news detection, and in particular relates to a multimodal fake news detection method and system based on a test phase adaptation strategy. Background Art
[0002] The rapid development of online social media platforms has made access to news more convenient, but at the same time, the risk of the rapid spread of fake news has also increased. This risk primarily stems from the lack of effective content verification mechanisms. On social media, users share information at an extremely high rate, and fake news often spreads before real news, causing public misunderstanding and social unrest. Because news is typically presented in multiple formats such as text, images, and video, single-mode analysis is no longer sufficient. Multimodal fake news detection, as an emerging method that can comprehensively utilize multiple information sources to improve the accuracy of fake news identification, has attracted increasing research attention in recent years. Existing research primarily focuses on the analysis and fusion of multimodal content, emphasizing the complementary advantages of different modalities to facilitate fake news identification.
[0003] However, traditional fake news detection methods often rely on the use of labeled datasets for training, which can lead to insufficient generalization capabilities when dealing with novel, unseen data. This is especially true in the dynamic social media environment, where the form and content of fake news are constantly changing, making the models face severe challenges in practical application. Furthermore, the process of labeling these datasets is not only time-consuming and labor-intensive, but often results in insufficient dataset size and diversity, which in turn impacts model performance. While recent research has focused on improving the performance of fake news detection by integrating multimodal information, it often overlooks the unique characteristics of multimodal fake news data itself, focusing too much on classification loss and failing to effectively address the distribution differences between training and test datasets.
[0004] To address these challenges, the present invention proposes a multimodal fake news detection framework based on a test-phase adaptation strategy, applying a test-time training strategy to multimodal fake news detection for the first time. Unlike traditional methods that rely solely on training data and classification loss, the present invention optimizes the model through masked autoencoder tasks, allowing the model to continuously adapt to new data features during the test phase. The core of the framework lies in integrating advanced feature extraction and multimodal fusion technologies through a test-phase adaptation strategy algorithm. Through the self-supervised masked autoencoder, the model of the present invention is able to capture subtle features in multimodal fake news, going beyond simple binary classification and focusing on deeper subtle features. At the same time, multimodal masking further improves the robustness of the model by filtering and integrating relevant information from different modalities, can effectively cope with changes in data distribution, and enhance its adaptability to new models. This innovative approach not only improves the accuracy of fake news detection, but also provides new ideas and methods for future related research. Summary of the Invention
[0005] The present invention provides a multimodal fake news detection method and system based on a test-phase adaptation strategy, which is committed to addressing the challenges currently faced in the field of fake news detection. In particular, when processing multimodal data (such as text and images), by introducing advanced graph neural networks and generative adversarial networks, it aims to improve the accuracy and robustness of fake news detection.
[0006] To achieve the above objectives, the technical solution of the present invention includes the following contents.
[0007] A multimodal fake news detection method based on a test phase adaptation strategy includes a training phase and a test-training phase. The training phase performs training based on the cross-entropy loss of news samples and the reconstruction loss of text and images in the news samples to obtain a pre-trained fake news detection model. The test-training phase fine-tunes the pre-trained fake news detection model based on the reconstruction loss of text and images in the news to be detected, and uses the fine-tuned fake news detection model to detect the news to be detected. The fake news detection model includes: a text feature encoding network, an image feature encoding network, a mask autoencoder, and a classifier.
[0008] Furthermore, training is performed based on the cross entropy loss of the news samples and the reconstruction loss of the text and image in the news samples to obtain a pre-trained fake news detection model, including:
[0009] Obtain text samples and image samples from news samples;
[0010] Based on a text feature encoding network, obtaining text feature encoding of the text sample; wherein the text feature encoding network includes: a BERT model;
[0011] Based on an image feature coding network, obtaining image feature coding of the image sample; wherein the image feature coding network includes: a ViT model;
[0012] Performing mask processing on the text sample and the image sample respectively;
[0013] The text and image in the news sample, the masked text sample, and the masked image sample are fed into the masked autoencoder to calculate the reconstruction loss of the text and image in the news sample;
[0014] The text feature encoding of the text sample and the image feature encoding of the image sample are fed into a masked autoencoder to obtain a fusion feature of the news sample;
[0015] Using the classifier to classify the fusion features of the news sample, and obtaining the cross entropy loss of the news sample according to the classification result and the label of the news sample;
[0016] Parameters of a fake news detection model are adjusted based on the cross entropy loss of the news sample and the reconstruction loss of the text and image.
[0017] Furthermore, obtaining the fusion feature of the news sample according to the text feature coding of the text sample and the image feature coding of the image sample includes:
[0018] Based on the text feature coding of the text sample and the image feature coding of the image sample, a similarity matrix between the text word vector and the local visual feature is calculated to obtain a locally aligned image-text feature;
[0019] Based on the locally aligned image-text features, an attention network is used to capture the semantic contradiction between the text feature encoding of the text sample and the global visual features to obtain a global reasoning result;
[0020] The local aligned image-text features and global reasoning results are input into several groups of Transformer modules respectively to obtain the hierarchical features C i ;
[0021] Splicing the hierarchical features C i , and obtain the fusion features of the news sample.
[0022] Furthermore, the text and image in the news sample, the masked text sample, and the masked image sample are fed into a masked autoencoder to calculate the reconstruction loss of the text and image in the news sample, including:
[0023] The text decoder is used to recover the masked tokens in the masked text sample, and combined with the images in the news sample, the text mask language cross entropy loss is calculated;
[0024] The masked pixel blocks in the masked image sample are reconstructed through the image decoder and combined with the text in the news sample to calculate the image mask modeling loss;
[0025] By combining the text mask language cross entropy loss and the image mask modeling loss, we can obtain the reconstruction loss of the text and image in the news sample.
[0026] Furthermore, the pre-trained fake news detection model is fine-tuned based on the reconstruction loss of the text and image in the news to be detected, and the news to be detected is detected using the fine-tuned fake news detection model, including:
[0027] Obtain the text and image of the news to be detected;
[0028] Based on a text feature encoding network, obtaining text feature encoding of the news to be detected; wherein the text feature encoding network includes: a BERT model;
[0029] Based on the image feature coding network, the image feature coding of the news to be detected is obtained; wherein the image feature coding network includes: a ViT model;
[0030] Mask the text and image of the news to be detected respectively;
[0031] The text and image of the news to be detected, as well as the masked text and masked image, are fed into the masked autoencoder, the reconstruction loss of the text and image is calculated, and the parameters of the masked autoencoder are fine-tuned based on the reconstruction loss of the text and image.
[0032] The text feature encoding and image feature encoding of the news to be detected are fed into the fine-tuned mask autoencoder to obtain the fused features of the news to be detected;
[0033] The classifier is used to classify the fusion features of the news sample to obtain a false news detection result of the news to be detected.
[0034] A multimodal fake news detection system based on a test phase adaptation strategy, including a training module and a test-training module;
[0035] The training module is used to perform training based on the cross entropy loss of the news sample and the reconstruction loss of the text and image in the news sample to obtain a pre-trained fake news detection model;
[0036] The training and testing module is used to fine-tune the pre-trained fake news detection model based on the reconstruction loss of text and image in the news to be detected, and use the fine-tuned fake news detection model to detect the news to be detected;
[0037] The fake news detection model includes: a text feature encoding network, an image feature encoding network, a mask autoencoder and a classifier.
[0038] An electronic device, comprising: a processor and a memory storing computer program instructions; when the processor executes the computer program instructions, it implements any of the above-mentioned multimodal fake news detection methods based on the test phase adaptation strategy.
[0039] A computer-readable storage medium having computer program instructions stored thereon, wherein the computer program instructions, when executed by a processor, implement any of the above-mentioned multimodal fake news detection methods based on a test phase adaptation strategy.
[0040] A computer program product, characterized in that when the computer program product is run on a computer device, the computer device is caused to execute any of the above-mentioned multimodal fake news detection methods based on the test phase adaptation strategy.
[0041] Compared with the prior art, the present invention has at least the following beneficial effects.
[0042] 1) By combining multimodal features, the present invention can fully utilize text and visual information to improve the accuracy of false news detection.
[0043] 2) The self-supervised learning mechanism is adopted to reduce the dependence on large amounts of labeled data and reduce the cost of model training.
[0044] 3) The introduction of training strategies during testing enables the model to better adapt to new samples in practical applications, thereby improving the flexibility and robustness of the system. BRIEF DESCRIPTION OF THE DRAWINGS
[0045] Figure 1 Flowchart of the multimodal fake news detection method based on the adaptation strategy in the testing phase. DETAILED DESCRIPTION
[0046] The present invention is further described in detail below through specific embodiments and drawings.
[0047] The multimodal fake news detection method based on a test-phase adaptation strategy proposed in this paper introduces a feature alignment module—the Masked Autoencoder Learning Module—to improve the model's adaptability to test samples. The core function of this module is to effectively align features extracted during training with those encountered during the test phase, thereby enhancing the model's robustness and accuracy, enabling it to better cope with various uncertainties and changing environments.
[0048] Specifically, the present invention is divided into a training phase and a test-time training phase. First, during the training phase, the model learns features using a large amount of labeled data to establish a basic feature representation. Next, during the test-time training phase, the model dynamically adjusts based on real-time data to optimize feature adaptability, ensuring efficient processing of new data. The adjusted model is then able to accurately classify and judge unseen samples.
[0049] This structured approach ensures a comprehensive understanding of the framework and its operational mechanisms, highlighting how each component works together to improve the overall effectiveness of fake news detection. Particularly in the rapidly changing social media landscape, this approach effectively addresses the diversity and complexity of fake news through precise feature alignment and self-supervised learning mechanisms.
[0050] By implementing these strategies, the present invention aims to comprehensively improve the accuracy and reliability of fake news detection, and by leveraging advanced self-supervised learning and feature alignment techniques, establish a more robust and flexible detection system, thereby providing users with more reliable information screening services.
[0051] Figure 1 4 is a flow chart of the multimodal fake news detection method based on the test phase adaptation strategy of the present invention.
[0052] 1. Training phase.
[0053] Step 1: Preprocess the multimodal news data.
[0054] After the present invention obtains the multimodal post P={W,R}, W is text data, R is image data, the text data is unified to 256 length, and the image data is unified to 224*224 size.
[0055] Step 2: Use BERT to extract news text feature encoding, and use ViT to extract image feature encoding in the news dataset.
[0056] ●Text feature extraction: Use the BERT model to encode the text W to obtain word vector features, extract dynamic contextual semantic features, and aggregate global features through the [CLS] tag.
[0057] Visual feature extraction: Use the ViT model to segment the image R into N image blocks, and extract local and global visual features through linear projection and Transformer encoder.
[0058] Step 3: Fuse the news text feature coding and image feature coding to generate fused features.
[0059] Local alignment unit: This unit identifies consistency between images and text (for example, matching an object in an image with its text description) by calculating a similarity matrix between text word vectors and local visual features. This process ensures effective alignment of local features, laying the foundation for subsequent global reasoning.
[0060] Global Reasoning Unit: Based on image-text alignment, this unit models global dependencies across modalities and utilizes an attention network to capture potential semantic contradictions. The results of global reasoning are used to determine the authenticity of the news being tested, ensuring a holistic understanding of the combined image and text information, and serve as one of the inputs to the hierarchical encoding unit.
[0061] Attention calculation in attention mechanism network:
[0062]
[0063] Among them, Q, K and V are generated by linear transformations of image and text features respectively. Specifically, the query Q matrix is obtained by linearly transforming the text feature T, in the form of Q=Wq×T, where Wq is a learnable weight matrix. The matrix K is obtained by linearly transforming the visual feature I, in the form of K=Wk×IK, where Wk is also a learnable weight matrix; the value V is also obtained by another linear transformation of the visual feature I.
[0064] The form is V = Wv × I, where Wv is another set of learnable weight matrices.
[0065] Hierarchical Encoding Unit: This unit uses three sets of Transformer modules to obtain hierarchical semantic features C0, C1, and C2 of text and visual information. Each Transformer module receives input from the image and text features processed by the local alignment unit and the results of the global inference unit. These hierarchical features are ultimately concatenated into a fused feature C.
[0066] Step 4: Classification and loss function
[0067] The fusion feature C is output through the fully connected layer and the Softmax classifier to obtain the probability distribution The cross entropy loss is defined as:
[0068]
[0069] Where Y is the true label and N is the number of samples.
[0070] Step 5: Generate masked text and masked image data.
[0071] The present invention uses a masked self-encoding algorithm to perform masking processing on text and image, and obtains masked text and masked image data respectively.
[0072] Step 6: Calculate the reconstruction loss of text and image.
[0073] Masked Autoencoder (MAE) Design:
[0074] Masking strategy: Text masking: randomly masking 15% of tokens while retaining key context information;
[0075] For image masking, 75% of image patches are randomly blocked, retaining only edge regions requiring the model to learn the overall content.
[0076] Cross-modal reconstruction mechanism:
[0077] Text reconstruction: Based on the complete image and masked text, the masked token is restored through the text decoding restorer.
[0078] Image reconstruction: Based on the complete text and mask image, the masked pixel blocks are reconstructed through the image decoder.
[0079] Reconstruction loss function of text and image:
[0080]
[0081] in:
[0082] represents the text mask language cross entropy loss;
[0083] Image mask modeling loss.
[0084] Step 7: Update the model parameters based on the cross entropy loss and the reconstruction loss of text and image.
[0085] 2. Test and training phase.
[0086] The test-training phase is similar to the training phase, except that, before fusing the text and image feature encodings of the news item to be tested, the present invention first masks the text and image of the news item to be tested, and fine-tunes the parameters of the masked autoencoder based on the masked image and text. The present invention then uses the fine-tuned masked autoencoder to fuse the text and image feature encodings, thereby obtaining fake news detection results for the news item to be tested.
[0087] The following uses the Weibo dataset as an example to illustrate the technical effects of the present invention.
[0088] Comparing the results of the method of the present invention with those of other methods, the false news recognition accuracy is shown in Table 1, and the F1 value is shown in Table 2:
[0089] Table 1. Accuracy of false news identification on the Weibo dataset
[0090] Dataset Method of the present invention FND-CLIP LIIMR Weibo 0.902 0.841 0.767
[0091] In Table 1, FND-CLIP and LIIMR are comparative methods used for comparison with the method of the present invention.
[0092] Table 2. F1 value of fake news identification on the Weibo dataset
[0093] Dataset Method of the present invention FND-CLIP LIIMR Weibo 0.901 0.831 0.773
[0094] The results in Tables 1 and 2 respectively demonstrate that the method of the present invention can significantly improve the effect of multimodal fake news detection.
[0095] The above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit the same. Those skilled in the art may modify or make equivalent substitutions for the technical solutions of the present invention without departing from the spirit and scope of the present invention. The scope of protection of the present invention shall be based on the claims.
Claims
1. A multimodal fake news detection method based on a test phase adaptation strategy, characterized by: The method includes a training phase and a test-training phase. The training phase is based on the cross-entropy loss of news samples and the reconstruction loss of text and images in the news samples to obtain a pre-trained fake news detection model. The test-training phase is based on the reconstruction loss of text and images in the news to be detected to fine-tune the pre-trained fake news detection model, and the news to be detected is detected using the fine-tuned fake news detection model. The fake news detection model includes: a text feature encoding network, an image feature encoding network, a mask autoencoder and a classifier.
2. The method according to claim 1, characterized in that The pre-trained fake news detection model is trained based on the cross entropy loss of the news sample and the reconstruction loss of the text and image in the news sample, including: Obtain text samples and image samples from news samples; Based on a text feature encoding network, obtaining text feature encoding of the text sample; wherein the text feature encoding network includes: a BERT model; Based on an image feature coding network, obtaining image feature coding of the image sample; wherein the image feature coding network includes: a ViT model; Performing mask processing on the text sample and the image sample; The text and image in the news sample, the masked text sample, and the masked image sample are fed into the masked autoencoder to calculate the reconstruction loss of the text and image in the news sample; The text feature encoding of the text sample and the image feature encoding of the image sample are fed into a masked autoencoder to obtain a fusion feature of the news sample; Using the classifier to classify the fusion features of the news sample, and obtaining the cross entropy loss of the news sample according to the classification result and the label of the news sample; Parameters of a fake news detection model are adjusted based on the cross entropy loss of the news sample and the reconstruction loss of the text and image.
3. The method according to claim 2, characterized in that Obtaining the fusion feature of the news sample according to the text feature coding of the text sample and the image feature coding of the image sample includes: Based on the text feature coding of the text sample and the image feature coding of the image sample, a similarity matrix between the text word vector and the local visual feature is calculated to obtain a locally aligned image-text feature; Based on the locally aligned image-text features, an attention network is used to capture the semantic contradiction between the text feature encoding of the text sample and the global visual features to obtain a global reasoning result; The local aligned image-text features and global reasoning results are input into several groups of Transformer modules respectively to obtain the hierarchical features C i ; Splicing the hierarchical features C i , and obtain the fusion features of the news sample.
4. The method according to claim 2, characterized in that The text and image in the news sample, the masked text sample, and the masked image sample are fed into a masked autoencoder to calculate the reconstruction loss of the text and image in the news sample, including: The masked tokens in the masked text sample are recovered through the text decoding restorer, and combined with the images in the news sample, the text mask language cross entropy loss is calculated; The image decoder is used to reconstruct the masked pixel blocks in the masked image sample and combine them with the text in the news sample to calculate the image mask modeling loss. By combining the text mask language cross entropy loss and the image mask modeling loss, we can obtain the reconstruction loss of the text and image in the news sample.
5. The method according to claim 1, wherein Fine-tuning the pre-trained fake news detection model based on the reconstruction loss of text and image in the news to be detected, and detecting the news to be detected using the fine-tuned fake news detection model, including: Obtain the text and image of the news to be detected; Based on a text feature encoding network, obtaining text feature encoding of the news to be detected; wherein the text feature encoding network includes: a BERT model; Based on the image feature coding network, the image feature coding of the news to be detected is obtained; wherein the image feature coding network includes: a ViT model; Mask the text and images of the news to be detected; The text and image of the news to be detected, as well as the masked text and masked image, are fed into the masked autoencoder, the reconstruction loss of the text and image is calculated, and the parameters of the masked autoencoder are fine-tuned based on the reconstruction loss of the text and image. The text feature encoding and image feature encoding of the news to be detected are fed into the fine-tuned mask autoencoder to obtain the fused features of the news to be detected; The classifier is used to classify the fusion features of the news sample to obtain a false news detection result of the news to be detected.
6. A multimodal fake news detection system based on a test phase adaptation strategy, characterized by: Includes training module and test training module; The training module is used to perform training based on the cross entropy loss of the news sample and the reconstruction loss of the text and image in the news sample to obtain a pre-trained fake news detection model; The training and testing module is used to fine-tune the pre-trained fake news detection model based on the reconstruction loss of text and image in the news to be detected, and use the fine-tuned fake news detection model to detect the news to be detected; The fake news detection model includes: a text feature encoding network, an image feature encoding network, a mask autoencoder and a classifier.
7. An electronic device, characterized in that: The electronic device includes: a processor and a memory storing computer program instructions; when the processor executes the computer program instructions, it implements the multimodal fake news detection method based on the test phase adaptation strategy as described in any one of claims 1 to 5.
8. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer program instructions, which, when executed by a processor, implement the multimodal fake news detection method based on a test phase adaptation strategy as described in any one of claims 1 to 5.
9. A computer program product, characterized in that When the computer program product runs on a computer device, it causes the computer device to execute the multimodal fake news detection method based on the testing phase adaptation strategy according to any one of claims 1 to 5.