A Multimodal Fake Information Detection Method

By encoding text and images using ResNet50 and BERT, and training with Transformer layers and various loss functions, a multimodal fake information detection model is constructed. This solves the problem of low detection accuracy in existing technologies and achieves higher accuracy and robustness in fake information identification.

CN117932391BActive Publication Date: 2026-11-17XINJIANG UNIVERSITY
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202311511630.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-11-10
Publication Date
2026-11-17
Estimated Expiration
2043-11-10

AI Technical Summary

Technical Problem

Existing methods for detecting misinformation have shortcomings in target training and misclassification of difficult samples, resulting in low accuracy of detection results.

Method used

We use ResNet50 and BERT to encode text and images respectively, and fuse them through a Transformer layer. We then train the model by combining data augmentation, contrastive learning loss, category-aware attention module, and cross-entropy loss to build a multimodal fake information detection model.

Benefits of technology

It improves the accuracy and robustness of the false information detection model, solves the problem of misclassification of difficult samples, and enhances the model's generalization ability and detection effect.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117932391B_ABST
    Figure CN117932391B_ABST
Patent Text Reader

Abstract

This invention discloses a multimodal misinformation detection method, belonging to the field of multimodal misinformation detection technology, including the following steps: performing data augmentation processing on the images and text contained in the tweet; inputting the original image and the processed image into an image encoder ResNet50 to obtain image hidden vectors, and inputting the original text and the processed text into a text encoder BERT to obtain text hidden vectors; inputting the image hidden vectors and text hidden vectors into an image-text feature fusion encoder Transformer to fuse the image and text encodings to obtain multimodal fusion features; inputting the multimodal fusion features into an attention layer and a fully connected layer to obtain multimodal features for learning; and inputting the multimodal features into a training target to complete the training of the multimodal misinformation detection model. This invention solves the problem of low detection accuracy caused by defects in target training and sample selection in existing technologies.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of multimodal misinformation detection technology, and in particular to a method for multimodal misinformation detection. Background Technology

[0002] With the development of technology and the improvement of people's living standards, various infrastructures have been gradually improved. The popularization of communication devices has promoted the development of various multimedia platforms. People no longer rely on traditional newspapers, magazines, and television for information. Using the internet, people can speak freely and express their opinions anytime, anywhere. Features such as comments and reposts promote interaction among netizens. Multimedia platforms, with their text, images, and videos, are widely popular, and the number of Chinese netizens has experienced explosive growth. Among these netizens are numerous marketing accounts that spread false information to gain attention. Driven by these marketing accounts and the reposting and commenting of uninformed "enthusiastic" netizens, other users easily believe and actively forward false information. Compared to traditional text information, users are more easily attracted to information with images or videos. Adhering to the principle that "a picture is worth a thousand words," images and videos often increase the credibility of posts, making people more willing to believe they are true. This false information can influence people's decisions and disrupt political, economic, and social stability. With the widespread use of multimodal data (including images, text, and videos), the spread of false information is wider and its harm is stronger. Therefore, the importance of multimodal fusion in the detection of misinformation is becoming increasingly prominent. Only by comprehensively utilizing various types of data can we more accurately identify and combat misinformation, and protect the public's right to know and interests.

[0003] However, existing methods for detecting misinformation have certain shortcomings in target training and misclassification of difficult samples, resulting in low accuracy of detection results. Summary of the Invention

[0004] To address the aforementioned shortcomings in existing technologies, this invention provides a multimodal false information detection method that solves the problem of low detection accuracy caused by defects in target training and sample selection in existing technologies.

[0005] To achieve the above-mentioned objectives, the technical solution adopted by this invention is: a multimodal fake information detection method, comprising the following steps:

[0006] S1: Perform data augmentation on the images and text contained in the tweet;

[0007] S2: Input the original image and the processed image into the image encoder ResNet50 to obtain the image hidden vector, and input the original text and the processed text into the text encoder BERT to obtain the text hidden vector;

[0008] S3: Input the image hidden vector and the text hidden vector into the image-text feature fusion encoder Transformer to fuse the image and text encodings and obtain multimodal fusion features;

[0009] S4: Input the multimodal fusion features into the attention layer and the fully connected layer to obtain multimodal features for learning;

[0010] S5: Input the multimodal features used for learning into the training target to complete the training of the multimodal fake information detection model;

[0011] S6: Test the trained multimodal misinformation detection model, and obtain the final multimodal misinformation detection model based on the model's classification results to complete the detection of multimodal misinformation.

[0012] The beneficial effects of the above scheme are as follows: This invention uses ResNet50 and BERT to extract and encode text and images respectively, and then inputs them into the Transformer layer to fuse the two modalities, learn the deep semantic features contained therein, and input the fused features into the learning target proposed in this scheme for better learning, thereby realizing false information detection. This invention solves the problem that existing technologies have defects in target training and misclassification of difficult samples, resulting in low detection accuracy.

[0013] Furthermore, data augmentation processing in S1 includes: back-translating text, and rotating, cropping, and flipping images.

[0014] The beneficial effects of the above-mentioned further solutions are: by preprocessing text and images, the data can be expanded and the model's generalization ability can be improved, while the robustness of the model is enhanced.

[0015] Furthermore, the image-text feature fusion encoder Transformer in S3 employs a multi-head self-attention mechanism.

[0016] The beneficial effect of the above-mentioned further scheme is that it adopts a multi-head self-attention mechanism to capture the intrinsic relationship between text and image features. This mechanism assigns higher attention weights to text tags and image fragments with explicit or implicit relationships, thereby promoting the alignment and fusion between modalities.

[0017] Furthermore, the training objectives in S5 include contrastive learning loss, category-aware attention module, and cross-entropy loss.

[0018] The beneficial effect of the above-mentioned further scheme is that by utilizing contrastive learning loss, category-aware attention module and cross-entropy loss, features can be learned better, thereby improving detection accuracy.

[0019] Furthermore, comparing the learning loss L con for:

[0020]

[0021] Where N is the total number of samples in the batch, i is an index from 1 to N, P(i) is the set of positive samples, p is a single positive sample, exp is an exponential function with the natural constant e as the base, sim(·) is the cosine similarity, and z i Let z be the hidden vector of the original sample. p For z i The hidden vector of the positive samples, τ is the scalar temperature parameter, z a Let z be the hidden vector of the negative sample, a be a single negative sample, and A(i) be the hidden vector of the negative sample. i The index set of negative samples.

[0022] The beneficial effect of the above-mentioned further scheme is that, through comparative learning, similar species are brought closer together and dissimilar species are kept away, and the need for false information detection can be met by comparative learning.

[0023] Furthermore, the attention score α between the embedded vector and the corresponding class vector in the category-aware attention module... i,j for:

[0024]

[0025] Among them, c yj Here are the class labels for the tweets, the superscript * represents the adjoint matrix, C is the number of tweets in the mini-batch, k is the number of tweets selected in the mini-batch, and c... k Regarding tweets and z in mini-batch i Class tags;

[0026] α i,j By incorporating it into the contrastive learning loss, a new contrastive learning loss L is obtained. CA for:

[0027]

[0028] Where, α i,p α represents the attention score between the training samples and the positive samples. i,a The attention scores are for the training samples and the negative samples.

[0029] The beneficial effects of the above-mentioned further solution are: by using a category-aware attention module, the problem of difficult negative samples in the model can be solved. Difficult samples can significantly affect sample mining and may cause the trained model to fall into suboptimal local minima. In order to evaluate the relationship between tweet embeddings and their class labels, this invention calculates the compatibility between an embedding vector and its corresponding class vector. Through gradient descent, difficult negative samples will be able to be correctly represented as their class.

[0030] Furthermore, the cross-entropy loss L RD for:

[0031]

[0032] Where R represents the features of the sample, w RD b represents the weights obtained by training the model. RD This represents the bias in the model.

[0033] The beneficial effect of the above further scheme is that the cross-entropy loss function can measure the distance between two distributions, and minimizing this function can achieve good learning results.

[0034] Furthermore, the total loss L of the multimodal fake information detection model is:

[0035] L=λ1·L RD +λ2L CA

[0036] Where λ1 is the first weight and λ2 is the second weight.

[0037] The beneficial effects of the above-mentioned further scheme are: the model combines contrastive learning loss and cross-entropy learning loss for the overall learning objective and assigns them different weights, which makes the model learn better and more accurate in identifying false information. Attached Figure Description

[0038] Figure 1 This is a flowchart of a multimodal misinformation detection method.

[0039] Figure 2 This is a diagram for comparative learning.

[0040] Figure 3 This is a schematic diagram of the category-aware attention module. Detailed Implementation

[0041] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0042] like Figure 1 As shown, a multimodal misinformation detection method includes the following steps:

[0043] S1: Perform data augmentation on the images and text contained in the tweet;

[0044] S2: Input the original image and the processed image into the image encoder ResNet50 to obtain the image hidden vector, and input the original text and the processed text into the text encoder BERT to obtain the text hidden vector;

[0045] S3: Input the image hidden vector and the text hidden vector into the image-text feature fusion encoder Transformer to fuse the image and text encodings and obtain multimodal fusion features;

[0046] S4: Input the multimodal fusion features into the attention layer and the fully connected layer to obtain multimodal features for learning;

[0047] S5: Input the multimodal features used for learning into the training target to complete the training of the multimodal fake information detection model;

[0048] S6: Test the trained multimodal misinformation detection model, and obtain the final multimodal misinformation detection model based on the model's classification results to complete the detection of multimodal misinformation.

[0049] Data augmentation in S1 includes back-translation of text and operations such as rotation, cropping, and flipping of images.

[0050] The S3 image-text feature fusion encoder Transformer employs a multi-head self-attention mechanism.

[0051] The training objectives in S5 include contrastive learning loss, category-aware attention module, and cross-entropy loss.

[0052] like Figure 2 As shown, the contrast learning loss L con for:

[0053]

[0054] Where N is the total number of samples in the batch, i is an index from 1 to N, P(i) is the set of positive samples, p is a single positive sample, exp is an exponential function with the natural constant e as the base, sim(·) is the cosine similarity, and z i Let z be the hidden vector of the original sample. p For z i The hidden vector of the positive samples, τ is the scalar temperature parameter, z a Let z be the hidden vector of the negative sample, a be a single negative sample, and A(i) be the hidden vector of the negative sample. i The index set of negative samples.

[0055] like Figure 3 As shown, the attention score α between the embedded vector and the corresponding class vector in the category-aware attention module. i,j for:

[0056]

[0057] Among them, c yj Here are the class labels for the tweets, the superscript * represents the adjoint matrix, C is the number of tweets in the mini-batch, k is the number of tweets selected in the mini-batch, and c... k Regarding tweets and z in mini-batch i Class tags;

[0058] α i,j By incorporating it into the contrastive learning loss, a new contrastive learning loss L is obtained. CA for:

[0059]

[0060] Where, α i,p α represents the attention score between the training samples and the positive samples. i,a The attention scores are for the training samples and the negative samples.

[0061] Cross-entropy loss L RD for:

[0062]

[0063] Where R represents the features of the sample, w RD b represents the weights obtained by training the model. RD This represents the bias of the model. The total loss L of the multimodal fake information detection model is:

[0064] L=λ1·L RD +λ2L CA

[0065] Where λ1 is the first weight and λ2 is the second weight.

[0066] In one embodiment of the present invention, the encoder includes a text encoder (BERT), an image encoder (ResNet50), and an image-text feature fusion encoder (Transformer), which together constitute a multi-layer fusion encoder. Furthermore, for the final training objective, instead of using a simple cross-entropy loss function, the contrastive learning loss and the cross-entropy loss are summed together, and a category-aware attention module addresses the problem of difficult negative samples during training.

[0067] The multimodal misinformation detection method proposed in this invention can accurately identify misinformation categories, suppress the spread of misinformation, and reduce negative impacts.

[0068] Those skilled in the art will recognize that the embodiments described herein are intended to help the reader understand the principles of the invention, and should be understood that the scope of protection of the invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific modifications and combinations based on the technical teachings disclosed in this invention without departing from the spirit of the invention, and these modifications and combinations are still within the scope of protection of the invention.

Claims

1. A multimodal method for detecting misinformation, characterized in that, Includes the following steps: S1: Perform data augmentation on the images and text contained in the tweet; S2: Input the original image and the processed image into the image encoder ResNet50 to obtain the image hidden vector, and input the original text and the processed text into the text encoder BERT to obtain the text hidden vector; S3: Input the image hidden vector and the text hidden vector into the image-text feature fusion encoder Transformer to fuse the image-text hidden vectors and obtain multimodal fusion features; S4: Input the multimodal fusion features into the attention layer and the fully connected layer to obtain multimodal features for learning; S5: Input the multimodal features used for learning into the training target to complete the training of the multimodal fake information detection model; The training objectives in S5 include contrastive learning loss, category-aware attention module, and cross-entropy loss. The contrastive learning loss for: in, This represents the total number of samples in the batch. For a number from 1 to index, For the positive sample set, For a single positive sample, For the natural constant An exponential function with base 0. For cosine similarity, The hidden vector of the original sample. For about The hidden vector of the positive samples, For scalar temperature parameters, The hidden vector of the negative sample. For a single negative sample, For about The index set of negative samples; The attention score between the embedded vector and the corresponding class vector in the category-aware attention module for: in, Superscript is a class tag for tweets. For the adjoint matrix, The number of tweets in the mini-batch. These are the tweets selected from the mini-batch. For tweets and in mini-batch Class tags; Will By incorporating it into the contrastive learning loss, a new contrastive learning loss is obtained. for: in, The attention scores for training samples and positive samples. Attention scores for training samples and negative samples; The cross-entropy loss for: in, For the characteristics of the sample, The weights are obtained by training the model. This represents the bias of the model; S6: Test the trained multimodal misinformation detection model, and obtain the final multimodal misinformation detection model based on the model's classification results to complete the detection of multimodal misinformation.

2. The multimodal fake information detection method according to claim 1, characterized in that, The data augmentation process in S1 includes: back-translating the text, and rotating, cutting, and flipping the image.

3. The multimodal fake information detection method according to claim 1, characterized in that, The image-text feature fusion encoder Transformer in S3 employs a multi-head self-attention mechanism.

4. The multimodal fake information detection method according to claim 1, characterized in that, The total loss of the multimodal fake information detection model for: in, As the first weight, It is the second weight.

Citation Information

Patent Citations

  • Multimodal false information detection model based on confrontation

    CN113239926A

  • False information detection method and device based on attention mechanism multi-feature fusion, electronic equipment and storage medium

    CN113761377A