Multi-source false news detection method and system based on multi-modal fusion

Abstract

The invention relates to the technical field of artificial intelligence and false information detection, in particular to a multi-source false news detection method and system based on multi-modal fusion, and the method comprises the steps: carrying out the feature extraction of a text, including sentence-level embedding and word-level embedding, and screening features through a text expert system in combination with a weighting coefficient output by a text gating module; carrying out feature extraction on the image by using a masking automatic encoder to extract deep features, and screening the features through an image expert system in combination with a weighting coefficient output by an image gating module; using a target detection algorithm to extract a subgraph of an important region of the image, inputting the text, the complete image and the subgraph into CLIP to obtain features, calculating a similarity matrix of the text and the image region through a cross-modal alignment module, and splicing the alignment features with the output features of the text expert system and the output features of the image expert system to obtain fusion features; and carrying out modal re-correction on the fusion features by using a dual Transform structure. According to the invention, the accuracy and robustness of multi-modal false news detection are improved.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and fake news detection technology, and in particular to a multi-source fake news detection method and system based on multimodal fusion. Background Technology

[0002] With the rapid development of social media and online news platforms, the speed and scale of information dissemination have reached unprecedented levels. However, the generation and spread of fake news has also become an increasingly serious information security problem globally. The harm of fake news is multi-layered and extremely serious; it not only affects individuals but also impacts the entire social order. Its most direct harm is the transmission of misinformation to individuals. Based on this information, people may make decisions harmful to themselves, such as incurring health risks, financial losses, emotional distress, and creating prejudice and antagonism. It also harms society, including but not limited to eroding public trust and hindering the effective operation of public affairs (public health, democratic processes, public safety, etc.). Furthermore, the rapid advancements in recent years, such as the GPT series and LLaMA representing large language models, and image generation models like DALL·E and StableDiffusion, have enabled artificial intelligence to generate high-quality text and images on a large scale. While this technological progress brings convenience to content creation, it also significantly increases the concealment and complexity of fake news generation, posing new challenges to traditional fake news detection methods. Especially in today's information ecosystem, news content is often presented in a combination of text and images, making it difficult for single-modal detection methods to fully capture the potential multi-dimensional falsehoods in news. Therefore, there is an urgent need to develop a fake news detection method for multimodal data, capable of simultaneously identifying real and fake content generated by both humans and generative models, in order to address the detection needs of multi-source and diverse fake information and improve the intelligence level and technological breadth of fake information governance.

[0003] To address the aforementioned problems and challenges, researchers have explored and studied the issue from different directions. Early pioneers defined misinformation and set the tone for data selection, analysis, and detection methods throughout the task. In another direction, misinformation detection has gradually expanded from a single-modal approach focusing on text to multimodal analysis incorporating images, exploring the issue of image-text modal inconsistency. Recently, some researchers have also focused on the source of text, such as determining whether text was generated by large models like LLM or ChatGPT, and have proposed comprehensive judgment methods for further research and evaluation.

[0004] Existing research generally focuses on detecting text, images, and videos generated by large models, or solely on detecting fake information. These are all binary classification tasks, and methods for identifying information generated by large models only detect a single modality, lacking fusion approaches. There is currently no research specifically addressing multi-source fake information detection. Existing studies either determine the authenticity of information or determine the source of the information individually—whether it was generated by a large model or manually. These tasks are independent and struggle to handle complex tasks. Using multiple independent tasks with sequential outputs leads to accumulated errors due to fixed inputs, outputs, and feature extraction, resulting in more severe errors and unsatisfactory performance. These challenges highlight the necessity of developing more robust and adaptive multi-source fake news detection models. Summary of the Invention

[0005] To overcome the shortcomings of existing technologies, such as the inability to simultaneously determine the source of news (human / large model) and its authenticity, the lack of effective association of text and image features, and the failure to correct multi-stage modal bias, this invention proposes a multi-source fake news detection method and system based on multimodal fusion. This method achieves joint and accurate detection of news sources (human / large model) and authenticity, thereby improving the accuracy and robustness of multimodal fake news detection.

[0006] To achieve the above objectives, the technical solution adopted is:

[0007] This invention provides a multi-source fake news detection method based on multimodal fusion, comprising the following steps:

[0008] Step 1, Task Definition: Set up news on social media to include text and images. The task objective is to determine the source and authenticity of the news, which may be generated manually or by a large model.

[0009] Step 2, Feature Extraction: Feature extraction is performed on both text and image. Text feature extraction includes sentence-level embedding and word-level embedding, and features are selected by a text expert system using the weighted coefficients output by the text gating module. Image feature extraction is performed using a masked autoencoder (MAE) to extract deep features, and features are selected by an image expert system using the weighted coefficients output by the image gating module.

[0010] Step 3, Modal Alignment and Fusion: Use the object detection algorithm to extract important sub-images of the image, input the text, the complete image and the sub-images into CLIP to obtain features, calculate the similarity matrix between the text and image regions through the cross-modal alignment module, and obtain the alignment features through normalization, pooling and dimension mapping. Concatenate the alignment features with the output features of the text expert system and the output features of the image expert system to obtain the fused features.

[0011] Step 4, Modal Recorrection: Modal recorrection is performed on the fused features using a dual Transformer structure;

[0012] Step 5: Output detection results: Input the corrected fusion features into the classifier and output four classification results, including: real news by human, fake news by human, real news by large model, and fake news by large model.

[0013] According to the multi-modal fusion-based multi-source fake news detection method of the present invention, further, in step 1, tags are defined during task definition. Where 0 and 1 represent real news and fake news written by humans, respectively, and 2 and 3 represent real news and fake news written by large models, respectively.

[0014] According to the multi-modal fusion-based multi-source fake news detection method of the present invention, the feature extraction process of the text in step 2 is further as follows:

[0015] Sentence-level embeddings are obtained using SimCSE, and word-level embeddings are obtained using RoBERTa.

[0016] Sentence-level embeddings are input into the text gating module to calculate the weighting coefficients used to control the text expert system;

[0017] The word-level embeddings and the weighting coefficients output by the text gating module are input into the text expert system, and the text features are output after weighted fusion.

[0018] According to the multi-modal fusion-based multi-source fake news detection method of the present invention, the feature extraction process of the image in step 2 is further as follows:

[0019] Use MAE to extract deep features from images;

[0020] The features extracted by MAE are passed through a linear attention module to output an optimized image embedding.

[0021] The image output by the linear attention module is embedded into the input image gating module to generate N gating values, i.e., weighting coefficients. The number of gating values ​​is the same as the number of experts in the image expert system.

[0022] The deep features output by MAE and the weighting coefficients output by the image gating module are input into the image expert system, and the image features are output after weighted fusion.

[0023] According to the multi-modal fusion-based multi-source fake news detection method of the present invention, both the text gating module and the image gating module are implemented based on MLP, and the gating function is defined as:

[0024] G(x)=Softmax(W2·SiLU(W1·x+b1)+b2)

[0025] Where x is the sentence-level text embedding or the image embedding output by the linear attention module, W1 and W2 are weight matrices, b1 and b2 are bias terms, SiLU is the activation function, and Softmax is the normalization function.

[0026] According to the multi-modal fusion-based multi-source fake news detection method of the present invention, the text expert system and the image expert system are both composed of five convolutional kernels of different sizes. The feature dimensions are consistent through differential stride and padding. After activation function processing, salient features are retained by max pooling.

[0027] According to the multi-modal fusion-based multi-source fake news detection method of the present invention, the process of obtaining fusion features in step 3 further includes:

[0028] Input the news text into the CLIP text encoder, and output the text features T. c ;

[0029] The news image I is input into the YOLOv8 object detection algorithm to extract multiple important region sub-images. Each important region sub-image and the original complete image are then used as inputs to the CLIP image encoder to extract visual features, which are then concatenated and mapped to the text features T via a fully connected layer. c Same-dimensional embedding space, output image features I c ;

[0030] Through formula T′ c =Unsequeeze(T) c ,2) and I′ c =Unsequeeze(I c 1) Extended text features T c and image features I c The dimension is defined by Unsequeeze(·,2), where Unsequeeze(·,2) represents the dimension expansion operation function, with parameter 2 specifying the position of the expanded dimension; Unsequeeze(·,1) represents the dimension expansion operation function, with parameter 1 specifying the position of the expanded dimension.

[0031] Through formula Calculate the similarity matrix, where D is the feature dimension and softmax is the normalization function;

[0032] After normalizing the similarity matrix A according to the image dimensions, the spatial dimensions are compressed by global average pooling. Then, it is mapped to the target dimension through a small MLP to obtain the alignment feature F. a ;

[0033] The splicing alignment features, text expert system output features, and image expert system output features are combined and processed by MLP to obtain the fused feature F.f Then, the feature F′ is obtained by weighting through a modal expert system and a modal gating module. f The processing formula for the modal expert system is: F′ f =Linear(SiLU(linear(F) f In this context, linear represents the inner linear transformation, SiLU represents the activation function, and Linear represents the outer linear transformation; the modal gating module is implemented based on MLP.

[0034] According to the multi-modal fusion-based multi-source fake news detection method of the present invention, the calculation formula for modal recorrection in step 4 is further as follows:

[0035] α = Transformer α (F′ f )

[0036] β = Transformer β (F′ f )

[0037]

[0038] Among them, Transformer α and Transformer β These are two independent Transformer networks that do not share weights. α and β are the correction parameters of the Transformer output, and μ and δ are the features F′. f The mean and standard deviation;

[0039] After recalibration, the final features are obtained through linear transformation by the modal expert system and the final gating module. The final gating module is implemented based on MLP.

[0040] According to the multi-modal fusion-based multi-source fake news detection method of the present invention, during the training phase, the cross-entropy loss of text experts, image experts, modal experts and the final classifier is weighted and summed as the total loss.

[0041] Furthermore, the present invention also provides a multi-source fake news detection system based on multimodal fusion, the system comprising:

[0042] The feature extraction module is used to extract features from text and images respectively. Text feature extraction includes sentence-level embedding and word-level embedding, and features are selected by a text expert system in combination with the weighted coefficients output by the text gating module. Image feature extraction uses a masked autoencoder (MAE) to extract deep features, and features are selected by an image expert system in combination with the weighted coefficients output by the image gating module.

[0043] The modal alignment and fusion module is used to extract important sub-images of an image using an object detection algorithm. It inputs text, the complete image, and the sub-image into CLIP to obtain features. The similarity matrix between the text and image regions is calculated through the cross-modal alignment module. After normalization, pooling, and dimensionality mapping, the alignment features are obtained. The alignment features are then concatenated with the output features of the text expert system and the image expert system to obtain the fused features.

[0044] The modality recalibration module is used to perform modality recalibration on fused features using a dual Transformer structure;

[0045] The detection result output module is used to input the corrected fusion features into the classifier and output four classification results, including: human-made real news, human-made fake news, big model real news, and big model fake news.

[0046] The beneficial effects achieved by adopting the above technical solution are:

[0047] 1. This invention innovatively achieves joint four-class classification detection of multi-source (human / large model) and authenticity, breaking through the limitations of existing technologies that can only determine the authenticity of news from a single dimension or a single source. By constructing a training system that includes four categories of labels—human-generated true news, human-generated fake news, large model-generated true news, and large model-generated fake news—the previously independent tasks of "authenticity determination" and "source identification" are integrated into a unified detection process, avoiding the error accumulation problem caused by multiple tasks being executed sequentially. This innovation broadens the scope of fake news detection, accurately addresses the challenge of the concealment of fake content generated by large models, and meets the core needs of multi-source fake information governance in the current complex information ecosystem.

[0048] 2. This invention employs the YOLOv8 object detection algorithm to extract important image regions as sub-images, achieving precise alignment with text entities. It also strengthens image-text association verification by combining cross-modal similarity matrix calculation. This design specifically addresses the deficiency of existing technologies in adequately associating image and text features, effectively capturing the typical fake news characteristic of "text not matching image." For example, when text describes a specific event but key image regions lack corresponding entities, the model can quickly identify anomalies by comparing the association strength between sub-images and text words.

[0049] 3. This invention employs a multi-stage gating module and an expert system for hierarchical feature optimization, combined with a dual-Transformer modality recalibration strategy. The gating module dynamically adjusts the importance of different features through adaptive weighting coefficients, while the expert system accurately extracts key information at multiple scales. Together, they achieve layer-by-layer feature optimization. The modality recalibration stage specifically corrects modal biases generated during multi-stage processing by unifying feature numerical scales and strengthening semantic relationships within modalities. This comprehensive strategy effectively addresses the problem of existing technologies failing to consider multi-stage modal relationship correction, ensuring efficient collaboration between text and image features throughout the entire process and significantly reducing the impact of feature biases on detection results.

[0050] 4. This invention significantly improves the ability to identify fake news generated by large models by employing a multi-module collaborative architecture of "feature extraction, modality alignment, bias correction, and hierarchical optimization." The model can accurately capture the unique features of content generated by large models, and the distinction between source attributes and authenticity is strengthened through training on four types of data. Furthermore, the collaborative design of multiple modules enhances the model's robustness under different data scenarios, enabling efficient detection of both complex, manually generated fake news and covertly false content generated by large models. This makes the fake news detection results more reliable and effectively improves the practicality of the technology in real-world scenarios such as social media and online news platforms. Attached Figure Description

[0051] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings of the embodiments of the present invention will be briefly described below. The drawings are merely illustrative of some embodiments of the present invention and are not intended to limit the scope of the present invention to all embodiments.

[0052] Figure 1 This is a flowchart illustrating the multi-source fake news detection method based on multimodal fusion according to an embodiment of the present invention.

[0053] Figure 2 This is a schematic diagram of the structure of a text expert system according to an embodiment of the present invention. Detailed Implementation

[0054] The exemplary solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Unless otherwise defined, the technical or scientific terms used in this invention should have the ordinary meaning understood by one of ordinary skill in the art.

[0055] This invention discloses a multi-source fake news detection method based on multimodal fusion, such as... Figure 1 As shown, it includes the following steps:

[0056] Step S101, Task Definition: Set that news on social media includes text and images. The task objective is to determine the source and authenticity of the news. The source can be human or a large model.

[0057] Suppose S is a news article on social media, consisting of varying numbers of characters. After word embedding, the text T is represented as T = [t1, t2, t3, ..., tt2]. l The total number of words is L; meanwhile, the image I for each news item is represented as I = [i1, i2, i3, ..., i...]. n ], n is at least 1. Each news item has a real tag. Where 0 and 1 represent real and fake news written by humans, respectively, and 2 and 3 represent real and fake news written by a large model, respectively. Given text T and its accompanying image I, the final determination is whether the news was generated manually or by a large model, and also whether the news is true or false.

[0058] Step S102, Feature Extraction: Perform feature extraction on both text and image. For text, feature extraction includes sentence-level embedding and word-level embedding. The weighted coefficients output by the text gating module are combined with the text expert system to select features. For image, feature extraction is performed using a masked autoencoder (MAE) to extract deep features. The weighted coefficients output by the image gating module are combined with the image expert system to select features.

[0059] For text feature extraction, the methods used are as follows: Figure 1The upper part of the feature extraction section is shown. ① First, unsupervised SimCSE is used, which employs contrastive learning and attention masks, achieving excellent sentence-level text embeddings with an extremely simple and fast structure. ② Then, the sentence-level embeddings obtained from SimCSE are input into a text gate module (Text Gate) built from a three-layer MLP, outputting N gate values ​​(weighting coefficients). These values ​​provide weighted calculations for subsequent text expert systems to extract information from different aspects. It is worth noting that this scheme uses multiple sets of gate modules for progressive weighting. The Text Gate and subsequent Img Gate, Fusion Gate, and Final Gate are all based on MLP implementations, differing only in their dimensionality choices. The gate function is defined as: G(x) = Softmax(W2·SiLU(W1·x+b1)+b2), where x is the input feature vector of the gate function, W1 and W2 are weight matrices, b1 and b2 are bias terms, SiLU is the activation function, and Softmax is the normalization function. ③ Based on the original text, the RoBERTa model, pre-trained for Chinese, is used to obtain more accurate word-level embeddings; ④ The word-level embeddings and the weighted coefficients output by the text gating module are input into the text expert system. Each expert focuses on providing features at different scales, and the features output by each expert are multiplied by their corresponding weighted coefficients; finally, all weighted features are merged to form the filtered and enhanced text feature T. e .

[0060] like Figure 2 As shown, the text expert system consists of five convolutional kernels of different sizes. Differential strides and padding ensure consistent feature dimensions. The design aims to: "By configuring convolutional kernels with different receptive fields (e.g., sizes of 1, 3, 5, 7, etc.), each expert branch focuses on extracting local dependencies and patterns at different ranges within the input sequence. Differential strides further enhance this scale diversity. Although the initial parameters of each branch differ, by precisely calculating and applying different padding amounts to each branch, it is ensured that all five branches output feature maps of the same size after processing input sequences of arbitrary lengths. This allows for weighted fusion of features at different scales in subsequent steps." Subsequently, the outputs of each branch are processed by activation functions and then subjected to max-pooling layers to retain the most significant activation values ​​within each feature channel, while simultaneously converting sequences of different lengths into fixed-length feature vectors.

[0061] For image feature extraction, the methods used are as follows: Figure 1The lower half of the feature extraction section shows: ① Deep features are extracted from the image using MAE; ② The features extracted by MAE are passed through a linear attention module, which transforms and filters high-dimensional features to reduce dimensionality and focus on more critical information, outputting an optimized image embedding; ③ The image output from the linear attention module is embedded into an input image gating module, generating N gating values, i.e., weighting coefficients, the same number as the number of experts in the image expert system; ④ The deep features output by MAE and the weighting coefficients output by the image gating module are input into the image expert system for weighted fusion, outputting image feature I. e The structure of the image expert system is the same as that of the text expert system (five parallel convolutional branches).

[0062] Step S103, Modal Alignment and Fusion: Use the object detection algorithm to extract important sub-images of the image, input the text, the complete image and the sub-images into CLIP to obtain features, calculate the similarity matrix between the text and image regions through the cross-modal alignment module, and obtain the alignment features through normalization, pooling and dimension mapping, and concatenate the alignment features with the output features of the text expert system and the image expert system to obtain the fused features.

[0063] After obtaining the initial features of the text and image respectively, the alignment problem between image and text features still needs to be addressed. If features across multiple modalities cannot be effectively correlated and aligned, subsequent performance will be severely affected. The specific implementation process of this step is as follows:

[0064] ① Input the news text into the CLIP text encoder, and the output dimension is... Text features T c Where B is the batch size, N is the length of the text sequence, and D is the dimension of the uniform modality feature.

[0065] ② For news images, we attempt to associate a certain area in the image with these embedded texts. Such an area is called an important area. The advantage of doing this is that if a large number of key areas cannot be associated with important text information, it is more likely that the news is fake news. To align important regions with embedded text features, the news image I is first input into the YOLOv8 object detection algorithm for region detection and extraction. Multiple sub-images are obtained from one or more accompanying images. These sub-images are then filtered according to the following rules: First, sub-images that are too small are discarded, and only those occupying a certain proportion of the original image are selected. Second, sub-images with an intersection-union ratio exceeding a certain threshold are also discarded, retaining larger sub-images. The core idea is not concerned with what is actually in the sub-image, but rather with the region being easily observable in the image and therefore deserving and necessary description in the text. This is considered from the reader's perspective; if the accompanying image in a news article is unclear and not mentioned in the news text, it is highly likely to be fake news. If there are multiple accompanying images, according to writing conventions, the order of text description should correspond to the order in which the images appear. Sub-images are extracted according to the order in which the images appear, so the order in which the sub-images appear should also correspond to the order in which the text is associated. Subsequently, the visual features of each important region sub-image and the original complete image are extracted and concatenated as input to the CLIP image encoder, and then mapped to the text feature T through a fully connected layer. c The same D-dimensional embedding space, with output dimension of Image features I c Where M is the sum of the number of valid subimages and the number of original complete images, i.e., the number of image regions, and T c with I c The dimension D is kept consistent to meet the requirements of cross-modal computing.

[0066] ③Text features T c With image features I c The input is fed into the cross-modal alignment module. This module first expands the dimensions of the text and image features, then calculates the association strength between each text embedding and each image region to obtain a similarity matrix. This matrix is ​​used to determine whether the sub-image is associated with the text, as described above. The calculation formula is:

[0067] T′ c =Unsequeeze(T) c ,2)

[0068] I' c =Unsequeeze(I c ,1)

[0069]

[0070] Here, Unsequeeze(·,2) represents the dimension expansion operation function, and parameter 2 specifies the position of the expanded dimension. Through this operation, the original T is expanded. c The dimension from Expand to The newly added dimension is used for dimensional matching with the image feature matrix, providing dimensional compatibility support for subsequent calculations of the similarity matrix between text and image regions, and avoiding matrix multiplication errors caused by inconsistencies in the dimensions of text and image features; Unsequeeze(·,1) represents the dimension expansion operation function, with parameter 1 specifying the position of the expanded dimension, transforming the original I... c The dimension from Expand to softmax is a normalization function.

[0071] ④ The similarity matrix A is normalized according to the image dimension. The reason for operating on the image dimension instead of the text dimension is that in fake news detection, text is the primary modality, while images are a secondary modality. After normalization, global average pooling is used to pool the attention weights, compressing the spatial dimension. Finally, a small MLP is used to map the pooled feature dimensions to the target dimension, obtaining the aligned features F. a :

[0072] F a =W2·Relu(MLP(W1·MaxPool(A)))

[0073] Where W1 and W2 represent weight matrices, ReLU is the activation function, MLP is a multilayer perceptron, and MaxPool is the max pooling operation.

[0074] ⑤ Segmentation Alignment Feature F a Text expert system output features T e Image expert system output features I e Then, MLP is used for spatial transformation and high-dimensional feature extraction to obtain the fused feature F. f Then, the feature F′ is obtained by weighting using a modal expert system and a modal gating module. f Modal expert systems differ from text and image expert systems in that their processing formula is:

[0075] F′ f =Linear(SiLU(linear(F) f )))

[0076] Here, linear is the inner linear transformation, SiLU is the activation function, and linear is the outer linear transformation; its main purpose is to focus on different fusion strategies or semantic dimensions, and then optimize the selection by combining gating.

[0077] Step S104, Modal recalibration: Use a dual Transformer structure to perform modal recalibration on the fused features.

[0078] CLIP works by jointly training images and text through contrastive learning, aiming to make semantically corresponding images and text as close as possible in the same embedding space. This step achieves excellent cross-modal alignment and is widely used in tasks such as zero-shot classification, retrieval, and generation. However, CLIP is essentially just a basic feature extractor. Subsequent network structures or task requirements generally involve further processing and refining the features from CLIP, which may introduce new modal biases.

[0079] To address potential modal bias issues after fusion, the features are recalibrated before classification. This module receives the fused features F′ from step S103. f And perform the following adaptive distribution normalization operation:

[0080] The normalization parameters required for normalization are dynamically learned from the fused features using two independent, lightweight Transformer networks that do not share weights.

[0081] α = Transformer α (F′ f )

[0082] β = Transformer β (F′ f )

[0083] Among them, Transformer α and Transformer β These are two independent Transformer networks that do not share weights, where α is the scaling parameter and β is the translation parameter.

[0084] Using the generated α and β, the fused feature F′ f Perform adaptive distribution standardization:

[0085]

[0086] Where μ and δ are characteristic F′ f The mean and standard deviation, this operation first through Forced standardization is performed to eliminate bias, followed by adaptive scaling and translation using α and β, enabling the network to recover the most effective feature distribution according to task requirements.

[0087] After the above operations, modal recalibration can be achieved. Especially for image-text matching tasks, mean + variance matching is a low-cost yet highly effective alignment method. After recalibration, the same operations are performed using the modal expert system and the final gating module to conduct a linear transformation and obtain the final features. The data is then fed into a fully connected network to perform a classification task.

[0088] Step S105: Output detection results: Input the corrected fusion features into the classifier and output four classification results including: real news by human, fake news by human, real news by large model, and fake news by large model.

[0089] This scheme uses multiple gating modules and expert systems. Therefore, when defining the loss function, in order to force the gating modules and expert systems to perform effective information extraction and alignment, the final classification task is added after each expert system, and the loss is calculated separately and added to the final loss function.

[0090] L loss =λ1L t_e +λ2L i_e +λ3L fu_e +λ4L c

[0091] Among them, L t_e L i_e L fu_e L c λ1, λ2, λ3, and λ4 represent the cross-entropy losses of the text expert, image expert, modality expert, and final classifier, respectively, and the weighting coefficients are represented by λ1, λ2, λ3, and λ4. The sum of all weighting coefficients is 1.

[0092] Corresponding to the above method, embodiments of the present invention also disclose a multi-source fake news detection system based on multimodal fusion, the system comprising:

[0093] The feature extraction module is used to extract features from text and images respectively. Text feature extraction includes sentence-level embedding and word-level embedding, and features are selected by a text expert system in combination with the weighted coefficients output by the text gating module. Image feature extraction uses a masked autoencoder (MAE) to extract deep features, and features are selected by an image expert system in combination with the weighted coefficients output by the image gating module.

[0094] The modal alignment and fusion module is used to extract important sub-images of an image using an object detection algorithm. It inputs text, the complete image, and the sub-image into CLIP to obtain features. The similarity matrix between the text and image regions is calculated through the cross-modal alignment module. After normalization, pooling, and dimensionality mapping, the alignment features are obtained. The alignment features are then concatenated with the output features of the text expert system and the image expert system to obtain the fused features.

[0095] The modality recalibration module is used to perform modality recalibration on fused features using a dual Transformer structure;

[0096] The detection result output module is used to input the corrected fusion features into the classifier and output four classification results, including: human-made real news, human-made fake news, big model real news, and big model fake news.

[0097] To verify the effectiveness of this scheme, further explanations and illustrations are provided below based on experimental data.

[0098] Table 1 compares the method of this invention (abbreviated as MSFEND) with other existing methods. Existing methods do not consider multi-source data, so their performance is relatively poor. Existing models are designed only for the truth or falsehood of news, which is a binary classification. This invention, however, is a four-class classification: real news written by humans, fake news written by humans, real news generated by a large model, and fake news generated by a large model. Directly training existing models on four types of data and obtaining results results in poor performance.

[0099] Table 1 Comparison of Experimental Results

[0100]

[0101] Table 2 shows the experimental results of using different weights for different loss functions, which is essentially a hyperparameter experiment. The loss is easy to understand; it involves adding a classifier to the features obtained from text experts, image experts, and modality experts respectively, and calculating the loss based on the final classification result. The advantage of this approach is that it ensures the loss does not vanish during model propagation, acting as a residual and mitigating the gradient vanishing problem.

[0102] Table 2 Experimental results on loss hyperparameters

[0103]

[0104] Table 3 is the ablation experiment, which involves disabling certain modules to render them ineffective, and then examining which module has the greatest impact on the final result to determine the extent of the innovation.

[0105] Table 3 Ablation experiments with different modules removed

[0106]

[0107] Finally, it should be noted that the above-described embodiments are merely specific implementations of the present invention, used to illustrate the technical solutions of the present invention, and not to limit it. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed in the present invention, or make equivalent substitutions for some of the technical features; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A multi-source fake news detection method based on multimodal fusion, characterized in that, Includes the following steps: Step 1, Task Definition: Set up news on social media to include text and images. The task objective is to determine the source and authenticity of the news, which may be generated manually or by a large model. Step 2, Feature Extraction: Feature extraction is performed on both text and image. Text feature extraction includes sentence-level embedding and word-level embedding, and features are selected by a text expert system using the weighted coefficients output by the text gating module. Image feature extraction is performed using a masked autoencoder (MAE) to extract deep features, and features are selected by an image expert system using the weighted coefficients output by the image gating module. Step 3, Modal Alignment and Fusion: Use the object detection algorithm to extract important sub-images of the image, input the text, the complete image and the sub-images into CLIP to obtain features, calculate the similarity matrix between the text and image regions through the cross-modal alignment module, and obtain the alignment features through normalization, pooling and dimension mapping. Concatenate the alignment features with the output features of the text expert system and the output features of the image expert system to obtain the fused features. Step 4, Modal recalibration: Modal recalibration is performed on the fused features using a dual Transformer structure; Step 5: Output detection results: Input the corrected fusion features into the classifier and output four classification results, including: real news by human, fake news by human, real news by large model, and fake news by large model.

2. The multi-source fake news detection method based on multimodal fusion according to claim 1, characterized in that, In step 1, tags were also defined during task definition. Where 0 and 1 represent real news and fake news written by humans, respectively, and 2 and 3 represent real news and fake news written by large models, respectively.

3. The multi-source fake news detection method based on multimodal fusion according to claim 1, characterized in that, The feature extraction process for the text in step 2 is as follows: Sentence-level embeddings are obtained using SimCS, and word-level embeddings are obtained using RoBERTa. Sentence-level embeddings are input into the text gating module to calculate the weighting coefficients used to control the text expert system; The word-level embeddings and the weighting coefficients output by the text gating module are input into the text expert system, and the text features are output after weighted fusion.

4. The multi-source fake news detection method based on multimodal fusion according to claim 3, characterized in that, The process of feature extraction from the image in step 2 is as follows: Use MAE to extract deep features from images; The features extracted by MAE are passed through a linear attention module to output an optimized image embedding. The image output by the linear attention module is embedded into the input image gating module to generate N gating values, i.e., weighting coefficients. The number of gating values ​​is the same as the number of experts in the image expert system. The deep features output by MAE and the weighting coefficients output by the image gating module are input into the image expert system, and the image features are output after weighted fusion.

5. The multi-source fake news detection method based on multimodal fusion according to claim 4, characterized in that, Both the text gating module and the image gating module are implemented based on MLP, and the gating function is defined as follows: G(x)=Softmax(W2·SiLU(W1·x+b1)+b2) Where x is the sentence-level text embedding or the image embedding output by the linear attention module, W1 and W2 are weight matrices, b1 and b2 are bias terms, SiLU is the activation function, and Softmax is the normalization function.

6. The multi-source fake news detection method based on multimodal fusion according to claim 4, characterized in that, Both the text expert system and the image expert system consist of five convolutional kernels of different sizes. The feature dimensions are consistent through differential stride and padding. After activation function processing, salient features are preserved through max pooling.

7. The multi-source fake news detection method based on multimodal fusion according to claim 1, characterized in that, Step 3, the process of obtaining fusion features, specifically includes: Input the news text into the CLIP text encoder, and output the text features T. c ; The news image I is input into the YOLOv8 object detection algorithm to extract multiple important region sub-images. Each important region sub-image and the original complete image are then used as inputs to the CLIP image encoder to extract visual features, which are then concatenated and mapped through a fully connected layer to the text features T. c Same-dimensional embedding space, output image features I c ; Through formula T' c =Unsequeeze(T) c ,2) and I' c =Unsequeeze(I c 1) Extended text features T c and image features I c The dimension is defined by Unsequeeze(·,2), where Unsequeeze(·,2) represents the dimension expansion operation function, with parameter 2 specifying the position of the expanded dimension; Unsequeeze(·,1) represents the dimension expansion operation function, with parameter 1 specifying the position of the expanded dimension. Through formula Calculate the similarity matrix, where D is the feature dimension and softmax is the normalization function; After normalizing the similarity matrix A according to the image dimensions, the spatial dimensions are compressed by global average pooling. Then, it is mapped to the target dimension through a small MLP to obtain the alignment feature F. a ; The splicing alignment features, text expert system output features, and image expert system output features are combined and processed by MLP to obtain the fused feature F. f The feature F' is then obtained by weighting the modal expert system and modal gating module. f The processing formula for the modal expert system is: F' f =Linear(SiLU(linear(F) f In this context, linear represents the inner linear transformation, SiLU represents the activation function, and Linear represents the outer linear transformation; the modal gating module is implemented based on MLP.

8. The multi-source fake news detection method based on multimodal fusion according to claim 7, characterized in that, The formula for calculating modal recalibration in step 4 is: a=Transformer α (F' f ) β=Transformer β (F' f ) Among them, Transformer α and Transformer β These are two independent Transformer networks that do not share weights. α and β are the correction parameters of the Transformer output, and μ and δ are the features F'. f The mean and standard deviation; After recalibration, the final features are obtained through linear transformation by the modal expert system and the final gating module. The final gating module is implemented based on MLP.

9. The multi-source fake news detection method based on multimodal fusion according to claim 1, characterized in that, During the training phase, the weighted sum of the cross-entropy losses of text experts, image experts, modality experts, and the final classifier is used as the total loss.

10. A multi-source fake news detection system based on multimodal fusion, characterized in that, The system includes: The feature extraction module is used to extract features from text and images respectively. Text feature extraction includes sentence-level embedding and word-level embedding, and features are selected by a text expert system in combination with the weighted coefficients output by the text gating module. Image feature extraction uses a masked autoencoder (MAE) to extract deep features, and features are selected by an image expert system in combination with the weighted coefficients output by the image gating module. The modal alignment and fusion module is used to extract important sub-images of an image using an object detection algorithm. It inputs text, the complete image, and the sub-image into CLIP to obtain features. The similarity matrix between the text and image regions is calculated through the cross-modal alignment module. After normalization, pooling, and dimensionality mapping, the alignment features are obtained. The alignment features are then concatenated with the output features of the text expert system and the image expert system to obtain the fused features. The modality recalibration module is used to perform modality recalibration on fused features using a dual Transformer structure; The detection result output module is used to input the corrected fusion features into the classifier and output four classification results, including: human-made real news, human-made fake news, big model real news, and big model fake news.

Images