Multi-modal false news detection method based on multi-source evidence fusion

Abstract

The invention relates to the technical field of deep learning and big data, in particular to a multi-modal false news detection method based on multi-source evidence fusion, which comprises the following steps: acquiring multi-modal news data, and inputting the multi-modal news data into a trained false news detection model to obtain a false news detection result; the false news detection model comprises a text and image information extraction module, a cross-modal fusion module, a double-flow cross-encoder and a classifier; according to the method, the external data corresponding to the news are automatically retrieved to form multiple groups of reference evidences, and the data from different sources can mutually verify, so that the information deviation caused by single content is reduced, and the judgment of false news has higher objectivity and stability; according to the double-flow cross-encoder, deep association can be established between texts and images of different sources, and semantic supplement relations or contradictions between the sources are automatically captured, so that final judgment is established on the basis of more comprehensive evidence fusion, and the overall judgment precision is improved.

Description

Technical Field

[0001] This invention relates to the fields of deep learning and big data technology, and in particular to a multimodal fake news detection method based on multi-source evidence fusion. Background Technology

[0002] In recent years, with the rapid development of mobile internet and social media, the speed of information dissemination has significantly increased, and the threshold for producing news content has been continuously lowered. Users can quickly publish and forward information through various channels such as social media platforms, video platforms, and forums, making the internet one of the main sources of information for the public. However, while information dissemination is efficient and convenient, a large amount of fake news, misleading content, and false information also spreads rapidly online, seriously impacting the social environment, public safety, and user perception.

[0003] Traditional methods of identifying fake news primarily rely on manual review, with professionals verifying each news item to confirm its authenticity. However, with the exponential growth of internet information, manual review alone is not only inefficient and costly, but also fails to meet real-time requirements. Once fake news spreads rapidly within a short period, it often causes irreversible social impact. Furthermore, due to the subjectivity and fatigue inherent in manual review, its consistency and stability cannot be guaranteed.

[0004] In recent years, the rapid development of technologies such as deep learning, natural language processing, and computer vision has provided new research directions for fake news detection. Related systems can assist in identifying suspicious content through text analysis, image recognition, and content consistency assessment. However, existing fake news detection technologies still have significant shortcomings: First, many methods rely solely on a single modality (such as text or image) for judgment, failing to comprehensively understand the news content and resulting in limited accuracy. Second, they lack the ability to jointly analyze multi-source evidence, failing to effectively integrate structured and unstructured data such as news text, accompanying images, publisher characteristics, and social media dissemination behavior. Third, existing models lack the ability to judge semantic consistency across modalities, making it difficult to handle complex forms of fake news such as discrepancies between text and images, forged images, and manipulated data.

[0005] Therefore, how to build a fake news detection method based on deep learning and big data technologies that can integrate multi-source evidence, understand the internal relationships of multimodal information, and has high robustness and real-time performance has become an important problem that urgently needs to be solved in the current technology field. Summary of the Invention

[0006] To address the aforementioned problems in the prior art, this invention employs a multimodal fake news detection method based on multi-source evidence fusion, comprising: acquiring multimodal news data; inputting the multimodal news data into a trained fake news detection model to obtain fake news detection results; the fake news detection model includes: a text and image information extraction module, a cross-modal fusion module, a two-stream cross-encoder, and a classifier; the training process of the fake news detection model includes:

[0007] S1. Obtain a multimodal news dataset, which includes multiple multimodal news items. Data for each multimodal news item Includes data samples from multiple different sources j ;in, For multimodal news, For multimodal news Multi-source retrieval evidence, where n is multimodal news data. The index, J, is the number of sources;

[0008] S2, Multimodal news data Data samples for each source j By inputting the text and image information extraction modules respectively, multimodal news data is obtained. Text-side information for each source j and image side information ;

[0009] S3, Multimodal news data Text-side information for each source and image side information Input the data into the cross-modal fusion module to obtain multimodal news data. Text-side features and image-side features ;

[0010] S4, Multimodal news data Text-side features and image-side features Input dual-stream cross-encoder to obtain multimodal news data Integrated text-side features and image-side features ;

[0011] S5, Multimodal News Data Integrated text-side features and image-side features Perform input classifier to obtain multimodal news data The classification results and their probabilities;

[0012] S6. Based on multimodal news data The loss function value is calculated based on the probability of the classification results. The parameters of the fake news detection model are updated based on the loss function value. When the loss function value is minimized, the trained fake news detection model is obtained.

[0013] The beneficial effects of this invention are:

[0014] 1. This invention not only processes original news text and images but also automatically retrieves corresponding external materials to form multiple sets of reference evidence. Data from different sources can corroborate each other, thereby reducing information bias caused by single content and making the judgment of fake news more objective and stable. 2. This invention uses optical character recognition technology to extract implicit text from images and models it in a unified manner with the text. This allows the model to simultaneously capture implicit information at the image level and explicit semantics at the text level, effectively improving the ability to identify complex forms of fake news such as forged images and embedded text tampering. 3. This invention extracts cross-modal representations through the CLIP model and performs source-intra-alignment in the InfoFusion module, enabling text features and image features to be projected into a consistent semantic space. This helps improve the consistency and comparability of descriptions between different modalities, thereby improving the ability to judge fake news. 4. The dual-stream cross-encoder of this invention can establish deep associations between text and images from different sources, automatically capturing semantic complementarity or contradictions between sources. This allows the final judgment to be based on a more comprehensive evidence fusion, thereby improving the overall judgment accuracy. Attached Figure Description

[0015] Figure 1 This is a flowchart illustrating a multimodal fake news detection method based on multi-source evidence fusion, according to an embodiment of the present invention.

[0016] Figure 2 This is a structural diagram of the InfoFusion module of a multimodal fake news detection method based on multi-source evidence fusion according to an embodiment of the present invention;

[0017] Figure 3 This is a data preprocessing flowchart of a multimodal fake news detection method based on multi-source evidence fusion according to an embodiment of the present invention;

[0018] Figure 4 This is an overall model structure diagram of a multimodal fake news detection method based on multi-source evidence fusion according to an embodiment of the present invention. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] like Figure 1 , Figure 4 As shown, this embodiment of the invention employs a multimodal fake news detection method based on multi-source evidence fusion, comprising: acquiring multimodal news data, inputting the multimodal news data into a trained fake news detection model, and obtaining fake news detection results; the fake news detection model includes:; the training process of the fake news detection model includes:

[0021] S1. Obtain a multimodal news dataset, which includes multiple multimodal news items. Multimodal news data Includes J data samples from different sources j. ;in, For multimodal news, For multimodal news Multi-source retrieval evidence, where n is multimodal news data. The index, J is the number of sources; multimodal news Includes: titles, text, images, and video frame extracts;

[0022] like Figure 2 As shown, this demonstrates how to obtain multimodal news data. include:

[0023] S11. Obtain the HTML page of the target news webpage through web crawlers or public data interfaces, standardize the HTML page of the target news webpage, and obtain multimodal news. ;

[0024] Standardizing the HTML pages of the target news webpage includes:

[0025] Structured information, including titles, text, and media content, is parsed from HTML pages. Within the media content, not only are existing images extracted from the HTML page, but also frames are extracted from accompanying video content, selecting keyframes as supplementary images. Multimodal news is then constructed in the format of "title + text + image + extracted video frame images." This provides foundational data for subsequent multimodal analysis.

[0026] S12. Use search engine APIs for multimodal news. A joint search was performed using the titles and images (excluding video frame extracts) to obtain the top J-1 related web pages; the HTML pages of the top J-1 related web pages were obtained, and the text and images of the HTML pages of the top J-1 related web pages were extracted to obtain multi-source search evidence. .

[0027] Multimodal news The combined retrieval of titles and images includes: using the title as the text query term and the features generated by the image as auxiliary search conditions to obtain the most relevant content pages for the news through a search engine.

[0028] To ensure the diversity and independence of evidence, this invention selects the top J-1 relevant web pages in the search results as external reference evidence for the original news, providing a sufficient data foundation for subsequent multi-source information comparison and cross-source fusion. Preferably, in this embodiment of the invention, the top 3 relevant web pages are selected.

[0029] S2, Multimodal news data Data samples from each source By inputting the text and image information extraction modules respectively, multimodal news data is obtained. Text-side information for each source j and image side information ;

[0030] The text and image information extraction module includes an optical character discrimination model, which does not perform parameter updates. The optical character discrimination model includes a text detection model and a text recognition model. The text and image information extraction module processes multimodal news data. Data samples from source j The processing includes:

[0031] S21, Data Sample Each picture Input the text detection model to perform text region detection and obtain data samples. Each picture Detection box set Where p is the index of the image for each source data sample;

[0032] This step is used to automatically locate regions in image I that may contain text, for subsequent character recognition processing. Specifically, the text detection model maps image I to a set of detection boxes, including:

[0033]

[0034] in, For the input image, This is a text detection model based on deep learning. This is the parameter set for the text detection model. For the first image of image I Each detection box, i.e., the bounding box of the detected text region, is defined by the coordinates of its top-left corner. ,width and height definition; This is the set of all bounding boxes for image I.

[0035] S22. Based on the detection box set In the corresponding image Image patches are cropped from the text and then input into a text recognition model to obtain data samples. Each picture Text information ;

[0036] For each detected text region in the original image I, the detection box This step converts the image pixel sequence into the corresponding character sequence, as expressed by the following formula:

[0037]

[0038]

[0039] in, It is an operation function, based on the bounding box. From the original image The corresponding image blocks are cropped from the image. It is a text recognition model based on deep learning. It is the parameter set of the text recognition model. It is the first one identified The text information of each text region, i.e., a string sequence. , It is a predefined character vocabulary. For image The collection of text information from all text regions.

[0040] S23, Data Sample All images Text information With the corresponding text By concatenating the data, we can obtain the text-side information of source j. ;

[0041] S24. Sample the data from source j. Each picture Adjust the size to a fixed value to obtain the image side information of source j. .

[0042] S3, Multimodal news data Text-side information for each source and image side information Input the data into the cross-modal fusion module to obtain multimodal news data. Text-side features and image-side features ;

[0043] This step enables deep interaction between the text and image modalities within each evidence source, resulting in more consistent and semantically coupled multimodal feature representations. This step comprises two main components: the calculation of the cross-modal relevance weight λ and a cross-attention-based bidirectional feature fusion mechanism. Its core idea is to dynamically control the fusion ratio based on the consistency between the text and image, making the information interaction process more robust and interpretable.

[0044] Specifically, the cross-modal fusion module includes: a cross-modal representation module and an information fusion module (InfoFusion module); the cross-modal fusion module processes multimodal news data. Text-side information for each source and image side information The processing includes:

[0045] S31, Multimodal news data Text-side information for each source and image side information Input the data into the cross-modal representation module to obtain multimodal news data. Text lateral vector features of each source Image lateral vector features and cross-modal consistency weights ;

[0046] The cross-modal representation module includes: a pre-trained CLIP model and a weight calculation module; the pre-trained CLIP model includes: a text representation model and an image representation model; the cross-modal representation module provides text-side information from source j. and image side information The processing includes: extracting textual information from source j. and image side information Input the corresponding representation model to obtain the text side vector features. and image lateral vector features ;Text side vector features and image lateral vector features Input the weight calculation module to obtain multimodal news data. Cross-modal consistency weights of source j .

[0047] The corresponding formula is shown below:

[0048]

[0049]

[0050] in, For text representation models, This is an image representation model.

[0051] The weight calculation module calculates text side vector features. and image lateral vector features The processing includes: processing the text side vector features separately. and image lateral vector features Global average pooling is performed to obtain text and image vector representations at a uniform scale. The cosine similarity between these uniformly scaled text and image vector representations is then calculated, and the cosine similarity is mapped to the 0-1 interval using the sigmoid function to obtain multimodal news data. Cross-modal consistency weights of source j The corresponding formula is shown below:

[0052]

[0053] in, For global average pooling, The cosine similarity function is used. The λ is the Sigmoid function; the closer λ is to 1, the higher the consistency between the image and text information, and the better the cross-modal fusion should be; the closer λ is to 0, the weaker the correlation between the image and text, and more of the original features should be preserved to avoid the spread of misinformation.

[0054] S32, Multimodal news data Text lateral vector features of each source Image lateral vector features and cross-modal consistency weights Input the InfoFusion module to perform modal alignment within the source to obtain multimodal news data. Text lateral vector features after source alignment and image lateral vector features ;

[0055] like Figure 3 As shown, the InfoFusion module introduces a cross-attention mechanism. The process of cross-modal fusion can be represented as follows:

[0056]

[0057]

[0058] in, , , Let Q, K, and V be the learnable mapping matrices for the query, key, and value vectors, respectively. Let h represent the dimension of the hidden layer vector representation of the pre-trained model, and NormProj(·) represent the layer normalization and linear mapping function.

[0059] S33, Text lateral vector features after aligning all sources and image lateral vector features By splicing the data, multimodal news data can be obtained. Text-side features and image-side features .

[0060] Text-side features and image-side features Represented as:

[0061]

[0062]

[0063] S4, Multimodal news data Text-side features and image-side features The input dual-stream cross-encoder integrates information from different source evidence to obtain multimodal news data. Integrated text-side features and image-side features ;

[0064] The two-stream cross-encoder consists of multiple layers of sequentially cascaded two-stream cross-encoder layers, including: intra-modal self-attention modules, inter-modal cross-attention modules, and residual blocks. The two-stream cross-encoder processes multimodal news data. Text-side features and image-side features The processing includes: processing multimodal news data Text-side features and image-side features The input consists of multiple sequentially concatenated two-stream cross-coding layers. The output of the last two-stream cross-coding layer is used as the integrated text-side feature. and image-side features .

[0065] Specifically, multimodal news data Text-side features and image-side features Input the first dual-stream cross-coding layer, input the output of the first dual-stream cross-coding layer into the second dual-stream cross-coding layer, and so on, input the output of the previous dual-stream cross-coding layer into the current dual-stream cross-coding layer, until the output of the last dual-stream cross-coding layer is obtained.

[0066] In this embodiment of the invention, L is 3.

[0067] The dual-stream cross-coding layer of layer l processes the output of the previous layer l-1, including:

[0068] S41. The text-side features output from the two-stream cross-coding layer of the previous layer l-1. and image-side features The text-side self-attention features of the l-th layer are obtained by performing self-attention calculations within the modal self-attention module and inputting the self-attention modules into the modal self-attention module. Image-side self-attention features ;

[0069] The specific formula for self-attention is:

[0070]

[0071]

[0072] in, This represents the layer index of the Two-Stream Cross-Encoder, starting with 1, and LN indicates layer normalization.

[0073] S42. The text-side self-attention features of layer l. Image-side self-attention features The intermodal cross-attention module is input separately to calculate the intermodal cross-attention, thus obtaining the text-side cross-attention features of the l-th layer. Image-side cross-attention features ;

[0074] The specific formula for cross-attention is:

[0075]

[0076]

[0077] S43, Incorporating text-side cross-attention features Image-side cross-attention features By inputting the residual blocks separately, we obtain the text-side features of the l-th layer dual-stream cross-coding layer output. and image-side features .

[0078] Residual block text-side cross-attention features of layer l Image-side cross-attention features The processing includes: incorporating the text-side cross-attention features of layer l. Image-side cross-attention features The inputs are fed into a feedforward neural network (FFN); the outputs of the FFN are then connected to the text-side cross-attention features of layer l using residual connections. Image-side cross-attention features The features are summed and then normalized using LN layer normalization to obtain the text-side features output by the l-th layer. and image-side features The specific formula is as follows:

[0079]

[0080]

[0081] S5, Multimodal News Data Integrated text-side features and image-side features Perform input classifier to obtain multimodal news data The classification results and their probabilities;

[0082] The classifier consists of two linear layers, integrating text-side features. and image-side features The probability of being identified as fake news is obtained through each of the linear layers.

[0083]

[0084] in, For linear layers, The activation function is O, which represents the final output probability. If the probability is greater than 0.5, it is fake news; otherwise, it is real news.

[0085] S6. Based on multimodal news data The loss function value is calculated based on the probability of the classification results. The parameters of the fake news detection model are updated based on the loss function value. When the loss function value is minimized, the trained fake news detection model is obtained.

[0086] The loss function is the cross-entropy loss function between the probability of the classification result and the true class.

[0087] The above-described embodiments further illustrate the purpose, technical solution, and advantages of the present invention. It should be understood that the above-described embodiments are merely preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made to the present invention within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A multi-modal fake news detection method based on multi-source evidence fusion, characterized in that, The method comprises the following steps: The multi-modal news data is acquired, and the multi-modal news data is input into a trained fake news detection model to obtain a fake news detection result. The fake news detection model comprises a text and image information extraction module, a cross-modal fusion module, a dual-flow cross-encoder, and a classifier. S1, acquire a multi-modal news dataset, the multi-modal news dataset comprising a plurality of multi-modal news data ; each multi-modal news data comprises a plurality of data samples of different sources j ; wherein is multi-modal news, is multi-modal news of multi-source retrieval evidence, n is the index of multi-modal news data , and j is the number of sources; S2, the multi-modal news data data samples in each source j input into the text and image information extraction module respectively, to obtain multi-modal news data text side information in each source j and picture side information ; S3, inputting the text side information and the picture side information of each source into a cross-modal fusion module respectively to obtain text side features and image side features of the multi-modal news data ;​​​​​ S4, inputting the text side feature and the image side feature of the multimodal news data into a dual-stream cross-encoder to obtain integrated text side feature and image side feature of the multimodal news data ;​​​​​ S5、inputting the integrated text-side features and image-side features into a classifier to obtain a classification result and a probability of the multi-modal news data integrated text-side features and image-side features S6、outputting the classification result and the probability of the multi-modal news data ​ S6、According to the classification result of the multi-modal news data a probability calculation loss function value, updating the parameters of the fake news detection model according to the loss function value, and obtaining the trained fake news detection model when the loss function value is minimum.

2. The multi-modal fake news detection method based on multi-source evidence fusion according to claim 1, characterized in that, Acquiring multi-modal news data comprising: An HTML page of a target news webpage is acquired, the HTML page of the target news webpage is standardized to obtain a multi-modal news ; the multi-modal news includes a title, a text, a picture and a video frame picture Adopt search engine to the multimodal news of title and picture carry out joint search, get the relevant webpage of the top J-1; get the HTML page of the relevant webpage of the top J-1, respectively extract the text and picture of the HTML page of the relevant webpage of the top J-1, get the search evidence .

3. The multi-modal fake news detection method based on multi-source evidence fusion according to claim 1, characterized in that, The text and image information extraction module comprises an optical character discrimination model, and the optical character discrimination model comprises a text detection model and a text recognition model. Data samples of the source j The processing comprises: S21, Data Sample Each picture Input the text detection model to obtain data samples Each picture Detection box set Where p is the index of the image; S22、according to the detection frame set In the corresponding picture The cropped image block is input into the text recognition model to obtain the text information of each picture of the data sample ;​ S23, splice the text information of all pictures of the data sample ;​​​​ S24, resizing each picture of the data sample to a fixed size, obtaining picture side information of the picture of source j .​ 4. The multi-modal fake news detection method based on multi-source evidence fusion according to claim 1, characterized in that, The cross-modal fusion module comprises a cross-modal representation module and an information fusion module; the cross-modal fusion module processes the multi-modal news data text side information of each source and picture side information processing includes: S31、the multi-modal news data text side information of each source and picture side information are respectively input into a cross-modal representation module to obtain text side vector features , image side vector features and cross-modal consistency weights of each source; S32, text-side vector features of each source , image-side vector features , and cross-modal consistency weights The input information fusion module performs modal alignment within the source to obtain aligned text-side vector features and image-side vector features of each source. S33, Text lateral vector features after aligning all sources and image lateral vector features By splicing the data, multimodal news data can be obtained. Text-side features and image-side features .

5. The multi-modal fake news detection method based on multi-source evidence fusion according to claim 4, characterized in that, The cross-modal representation module comprises a pre-trained CLIP model and a weight calculation module; the pre-trained CLIP model comprises a text representation model and an image representation model; the cross-modal representation module processes the multi-modal news data Text side information of source j And picture side information The processing comprises: inputting the text side information And picture side information Into the corresponding representation model respectively to obtain text side vector features And image side vector features ; inputting the text side vector features And image side vector features Into the weight calculation module to obtain the cross-modal consistency weight of source j of the multi-modal news data .​ 6. The multi-modal fake news detection method based on multi-source evidence fusion according to claim 5, characterized in that, The weight calculation module processes the text-side vector features and the image-side vector features , including: performing global average pooling on the text-side vector features and the image-side vector features respectively to obtain vector representations of the text and the image in a unified scale; calculating a cosine similarity between the vector representations of the text and the image in the unified scale, and mapping the cosine similarity to a 0-1 interval by using a Sigmoid function to obtain a cross-modal consistency weight of the source j of the multi-modal news data .

7. The multi-modal fake news detection method based on multi-source evidence fusion according to claim 4, characterized in that, The information fusion module processes the text side vector features of source j , the image side vector features , and the cross-modal consistency weight include: wherein, wherein, , , are the query vector Q, the key vector K and the value vector V learnable mapping matrices, respectively, and NormProj(·) denotes a layer normalization and linear mapping function, is a cross-attention mechanism.

8. The multi-modal fake news detection method based on multi-source evidence fusion according to claim 1, characterized in that, The dual-stream cross-encoder comprises multiple layers of dual-stream cross-coding layers connected in series. Each dual-stream cross-coding layer includes: an intra-modal self-attention module, an inter-modal cross-attention module, and a residual block. The dual-stream cross-encoder processes multimodal news data. Text-side features and image-side features The processing includes: processing multimodal news data Text-side features and image-side features The input consists of multiple sequentially concatenated two-stream cross-coding layers. The output of the last two-stream cross-coding layer is used as the integrated text-side feature. and image-side features ; The training process of the fake news detection model comprises the following steps: S41, input the text side feature output by the dual-stream cross-encoding layer of the last layer l-1 and the image side feature respectively into the intra-modal self-attention module to obtain the text side self-attention feature and the image side self-attention feature ; S42, input the text side self-attention feature and the image side self-attention feature into the inter-modal cross-attention module respectively to obtain a text side cross-attention feature and an image side cross-attention feature and the image side cross-attention feature respectively and the image side cross-attention feature ; S43, text-side cross-attention features and image-side cross-attention features are respectively input into the residual block to obtain text-side features of the l-th layer double-flow cross-encoding layer output and image-side features .

9. The multi-modal fake news detection method based on multi-source evidence fusion according to claim 1, characterized in that, The classifier includes two linear layers; the classifier processes the integrated text-side features and image-side features The processing includes: wherein, is a linear layer, is an activation function, O represents an output probability, and O is greater than 0.5 is fake news, otherwise is true news.

10. The multi-modal fake news detection method based on multi-source evidence fusion according to claim 1, characterized in that, The processing process of the dual-flow cross-encoder of the lth layer comprises the following steps: The loss function is a cross-entropy loss function of the probability of the classification result and the real category.

Images