Human reading heuristic multi-modal false news detection method and system

Abstract

The invention belongs to the technical field of false news detection, and provides a human reading heuristic multi-modal false news detection method and system. According to the technical scheme, a prompt text related to an image is generated based on a predefined template; encoding the original news text data and the prompt text to obtain original text features and prompt text features; extracting spatial domain features and frequency domain features of the image; enhancing original text features, prompt text features and spatial domain features and frequency domain features of the image; fusing the enhanced original text feature and the prompt text feature to obtain an enhanced text feature, and fusing the spatial domain feature of the enhanced prompt text feature image to obtain an enhanced visual feature; and feature interaction is carried out based on the enhanced image frequency domain features, the enhanced text features and the enhanced visual features to capture mutual information between the same mode and different modes, final multi-mode features are obtained through splicing, and false news classification is carried out based on the multi-mode features to obtain a classification result. And the detection accuracy is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of fake news detection, and in particular relates to a human reading heuristic multimodal fake news detection method and system. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] Online social media has become a crucial channel for sharing everyday information. However, it has also facilitated the spread of false information, including crisis rumors, hoaxes, and pseudoscience, often in the form of deliberately fabricated fake news. Fake news refers to information that is intentionally fabricated, distorted, or exaggerated to mislead the public or serve a specific purpose, and therefore must be deliberate and verifiable. It is particularly prevalent in the digital age, spreading rapidly through social media and the internet. Fake news erodes the credibility of information and disrupts communication, making it crucial to identify and prevent its spread at an early stage.

[0004] Fake news is increasingly being published on social media platforms in the form of images and text. The combination of text and images conveys more information, and their vibrant colors are more likely to attract readers' attention. Posts containing both text and images are shared more frequently than text-only posts, making the combination of text and images more effective in detecting fake news. There is a need to design effective methods to detect the authenticity of multimodal news composed of text and images to mitigate the significant negative impact of fake news.

[0005] Fake News Detection (FND) refers to the identification and verification of the authenticity of news articles. This is a multimodal task that requires comprehensive consideration of various factors to determine whether they contain false or misleading information. Current FND methods face several challenges: many of them focus on specific modalities such as text or images, while neglecting to integrate modalities into a more comprehensive feature set. Although some methods attempt to extract frequency-domain features from images to improve feature quality, the techniques used are often outdated and inefficient. Furthermore, most FND methods fail to incorporate external knowledge, which could deepen the semantic understanding of fake news. This leads to a significant semantic gap between latent visual features and their high-level semantic representations. Furthermore, FND methods often encounter problems such as data imbalance, which undermines their effectiveness. In summary, most existing feature fusion techniques do not fully consider the relevance and importance of features across different modalities. Summary of the Invention

[0006] In order to solve at least one technical problem existing in the above-mentioned background technology, the present invention provides a human reading-inspired multimodal fake news detection method and system, which simulates human reading patterns, enhances multimodal features and captures cross-modal interactions, fully utilizes the interdependence between multimodal features, and improves the accuracy of fake news detection.

[0007] In order to achieve the above object, the present invention adopts the following technical solutions:

[0008] A first aspect of the present invention provides a human reading heuristic multimodal fake news detection method, comprising the following steps:

[0009] Generate image-related prompt text based on acquired news image data and predefined templates;

[0010] Encode the acquired original news text data and prompt text respectively to obtain original text features and prompt text features;

[0011] Extract the spatial domain features and frequency domain features of the image respectively;

[0012] Enhance original text features, hint text features, image spatial domain features and frequency domain features;

[0013] The enhanced original text features and the prompt text features are fused to obtain enhanced text features, and the spatial domain features of the enhanced prompt text feature image are fused to obtain enhanced visual features;

[0014] Based on the enhanced image frequency domain features, enhanced text features and enhanced visual features, feature interaction is performed to capture the mutual information between the same and different modalities, and the final multimodal features are spliced together. The fake news classification is performed based on the multimodal features to obtain the classification results.

[0015] Furthermore, when encoding the obtained original news text data and prompt text, the sentences are decomposed into continuous word sequences, and the word sequences and the pre-trained BERT model are combined to extract text context information. The text context information is applied with an average pooling operation to obtain the features of the entire text from all words.

[0016] Furthermore, the VGG19 network is used for image spatial domain feature extraction.

[0017] Furthermore, when extracting image frequency domain features, it includes:

[0018] The image is preprocessed and wavelet transformed to obtain a tensor with multiple channels;

[0019] A tensor with multiple channels is subjected to depthwise separable convolution to obtain a frequency domain feature vector; the depthwise separable convolution includes depthwise convolution and pointwise convolution, the depthwise convolution convolves the tensor of each channel, and the pointwise convolution combines the output of each channel to obtain a frequency domain feature vector.

[0020] Furthermore, when enhancing the original text features, prompt text features, spatial domain features and frequency domain features of the image, it includes calculating the query matrix, keyword matrix and value matrix of each modality respectively, and based on the query matrix, keyword matrix and value matrix of each modality, combining multi-head self-attention calculation to obtain the enhanced features of the corresponding modality.

[0021] Furthermore, the calculation formula for the enhanced text feature is:

[0022]

[0023] The calculation formula for enhanced visual features is:

[0024]

[0025] in, is the output linear transformation, Q p is the query matrix corresponding to the original text features, K t and V t is the key matrix and value matrix corresponding to the original text features, h represents the hth attention head, H represents the number of attention heads, K v and V v are the key matrix and value matrix corresponding to the spatial domain features, and d is the feature dimension of the query vector and keyword vector.

[0026] Furthermore, when false news is classified based on multimodal features to obtain the classification results, the multimodal feature representation is input into the FC layer, and the output of the FC layer generates the distribution of classification labels through the softmax function.

[0027] A second aspect of the present invention provides a human reading heuristic multimodal fake news detection system, comprising:

[0028] A prompt generation module, which is used to generate prompt text related to the image based on the acquired news image data and a predefined template;

[0029] A text feature extraction module is used to encode the acquired original news text data and prompt text to obtain original text features and prompt text features respectively;

[0030] An image feature extraction module is used to extract the spatial domain features and frequency domain features of the image respectively;

[0031] A feature enhancement module is used to enhance the original text features, the hint text features, the spatial domain features and the frequency domain features of the image;

[0032] The multimodal fusion module is used to fuse the enhanced original text features and the prompt text features to obtain enhanced text features, and to fuse the spatial domain features of the enhanced prompt text feature image to obtain enhanced visual features. Based on the enhanced image frequency domain features, enhanced text features, and enhanced visual features, feature interaction is performed to capture the mutual information between the same and different modalities, and the final multimodal features are obtained by splicing.

[0033] The classification module is used to classify fake news based on multimodal features to obtain classification results.

[0034] A third aspect of the present invention provides a computer-readable storage medium.

[0035] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the human reading heuristic multimodal false news detection method as described above.

[0036] A fourth aspect of the present invention provides a computer device.

[0037] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the steps of the human reading heuristic multimodal fake news detection method described above are implemented.

[0038] Compared with the prior art, the present invention has the following beneficial effects:

[0039] The present invention integrates external knowledge and performs feature interaction based on enhanced image frequency domain features, enhanced text features and enhanced visual features to capture the mutual information between the same and different modalities, making full use of the interdependence between multimodal features and improving the accuracy of fake news detection.

[0040] Advantages of additional aspects of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The accompanying drawings, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0042] Figure 1 This is a flow chart of a human reading heuristic multimodal fake news detection method provided by an embodiment of the present invention;

[0043] Figure 2This is a visualization result of the classification features of the existing model provided by the embodiment of the present invention;

[0044] Figure 3 This is a visualization result of the model classification features of the present invention provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0045] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0046] It should be noted that the following detailed descriptions are illustrative and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present invention belongs.

[0047] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or combinations thereof.

[0048] To address the challenges faced by current FND methods, this paper proposes HRMulFND, which stands for human reading-inspired multimodal FND using DeepSeek cues and spatial-frequency information alignment. Unlike traditional FND methods, HRMulFND offers four distinct advantages:

[0049] (1) It utilizes text-image information and external cues in the feature extraction process. A large-scale language model (LLM) is used for the original news text to generate relevant external cues based on the accompanying news images. (2) It extracts text features from the original news text and external cues to obtain high-level semantic representations. (3) For news image information, a combination of wavelet transform and depthwise separable convolution is innovatively applied to extract the frequency domain features of the image, which are then aligned with the spatial features. (4) In the feature fusion stage, a multimodal feature alignment strategy is designed to simulate the human reading scene. A multi-head self-attention (MSA) mechanism is used to enhance the feature representation in each modality. Then, the external cue features are integrated through a joint attention mechanism to enhance the visual features of the original text and image and narrow the semantic gap between them. Finally, a joint attention mechanism is applied in pairs between the image frequency domain features, the enhanced original text features, and the enhanced image visual features to achieve efficient cross-modal feature interaction.

[0050] Example 1

[0051] This embodiment provides a human reading-inspired multimodal fake news detection method, comprising the following steps:

[0052] Step 1: Generate image-related prompts based on the acquired news image data and predefined templates;

[0053] In this embodiment, the defined template can adopt chain thinking (ToC), graph thinking (ToG), or tree thinking (ToT). For example, a simple template such as "Please describe the content of the image" is used for the image of the post. The image and the corresponding model are input into a large language model, such as DeepSeek-V3, to obtain prompts related to the image.

[0054] Step 2: Encode the acquired original news text data and the generated prompts to obtain prompt text features and original text features respectively;

[0055] The specific steps include:

[0056] Step 201: Input the generated prompt into the pre-trained BERT to extract the prompt text features, specifically including:

[0057] Step 2021: take the prompt text as input and mark it, breaking the sentence into a continuous sequence of words;

[0058] Step 2022: Combine the word sequence and the pre-trained BERT model to extract the prompt text context information V j1 , expressed as:

[0059]

[0060] Among them, V j1 represents the feature vector of the j-th sentence, represents the i-th word embedded in the j-th sentence;

[0061] Step 2023: extract the context information V j1 Apply the average pooling operation to obtain the feature F of the entire prompt text from all words p ;

[0062] Step 2024, use the FC layer with size 768×2742 and 2742×128 to transform the feature F p Reshaped to 128 dimensions to match the input size of the subsequent co-attention module.

[0063] Step 202: Pass the first encoded feature through a fully connected (FC) layer to reshape the feature into 128 dimensions to match the input size of the subsequent attention module.

[0064] Step 203: Input the news text data into the pre-trained BERT to extract text features, specifically including:

[0065] Step 2031: Take the original text in the news as input, mark it, and decompose the sentence into a continuous sequence of words according to the vocabulary provided by Wolf et al.;

[0066] Step 2032: Combine the word sequence and the pre-trained BERT model to extract context information V j2 , expressed as:

[0067]

[0068] in, represents the k-dimensional vector of the i-th word in the j-th sentence, V j2 Represents the feature vector of the jth sentence;

[0069] Step 2033: extract the context information V j2 Apply the average pooling operation to obtain the features F of the entire original text from all words t ;

[0070] Step 2034, use the FC layer with size 768×2742 and 2742×128 to transform the feature F t Reshaped to 128 dimensions to match the input size of the subsequent co-attention module.

[0071] Step 3: Extract the spatial domain features and frequency domain features of the image respectively;

[0072] The specific steps include:

[0073] Step 301: Combine news image data and a spatial domain feature extraction network to extract spatial domain features;

[0074] Specifically, the image size is normalized to 224×224 pixels, which is consistent with the ImageNet pre-training dimension, and then it is input into the 19-layer VGGNet. During the training process, the pre-training parameters of VGG19 are frozen to reduce overfitting; the output feature V of the penultimate layer of VGG19 is selected g , the measurement value is 4096×1, and then FC layers with sizes of 768×2742 and 2742×128 are applied to resize the features to 128 dimensions to meet the input requirements of the subsequent attention module.

[0075] The resulting visual representation is the spatial domain feature, denoted as F v , the derivation is as follows:

[0076] F v =σ(WV g ) (3),

[0077] Among them, V g is the visual feature representation obtained from the pre-trained VGG19, and W is the weight matrix of the FC layer in the visual feature extractor.

[0078] Step 302: extracting frequency domain features by combining news image data and frequency feature extraction network;

[0079] Specifically, fake news images often exhibit periodic recompression or tampering features in the frequency domain, highlighting the importance of frequency domain features in multimodal FND tasks. Traditional FND methods usually obtain frequency representations of social images through discrete cosine transform (DCT). However, these methods often face problems such as insensitivity to local details, loss of spatial information in the frequency domain, and poor localization of forged areas. Wavelet transform has recently been widely used in various visual tasks due to its excellent performance. Depthwise separable convolution is another widely adopted optimization strategy that decomposes the convolution operation into depth and point-by-point stages, significantly reducing model parameters and computational requirements. Drawing inspiration from these advances, this embodiment introduces a new combination of wavelet transform and depthwise separable convolution, which enables the model of the present invention to effectively extract frequency domain image features. This approach solves the complexity of complex forgery scenes and enhances the robustness of the FND method. Specifically comprising the following steps:

[0080] Step 3021: image preprocessing and wavelet transform;

[0081] Given an input image First resize the image to 128×128 in height and width and convert it to grayscale:

[0082]

[0083] Among them, I (i) Represents the i-th channel of the input image.

[0084] Apply 2D discrete wavelet transform (Haar wavelet) to grayscale image I gray , to obtain four subband coefficients: low-frequency subband LL and three high-frequency subbands LH, HL and HH:

[0085] DWT2(I gray )=(LL,(LH,HL,HH)) (5),

[0086] The size of each subband is half of the input, that is, These subbands are then stacked into a tensor with four channels,

[0087] Step 3022: Process the output of the wavelet transform through depthwise separable convolution to obtain a frequency domain feature vector;

[0088] Specifically, depthwise separable convolution consists of two parts: depthwise convolution and pointwise convolution.

[0089] The depthwise convolution performs a 3×3 convolution kernel on each input channel independently. For the input tensor X∈ The depthwise convolution is calculated as follows:

[0090] Y (i) =X (i) *K (i) ,i∈{1,2,…,C in} (6),

[0091] Among them, * represents convolution operation, K (i) is the convolution kernel of the i-th channel, C in Indicates the number of input channels, output Keep the same number of input channels.

[0092] Point-wise convolution uses a 1×1 convolution kernel to combine the outputs of each input channel and is calculated as follows:

[0093]

[0094] Among them, P (i,j) is the point-by-point convolution kernel connecting the i-th input channel and the j-th output channel. The final output is

[0095] Step 3023: For two depth-separable convolution blocks, the output is recursively updated as follows:

[0096] X1=ReLU(Pointwise(ReLU(Depthwise(W)))) (8),

[0097] X2=ReLU(Pointwise(ReLU(Depthwise(X1)))) (9),

[0098] Finally, the feature size is fixed to 128×1×1 through the adaptive pooling layer to ensure that the spatial resolution of the feature map is independent of the input image size. After flattening, a 128-dimensional feature vector F is obtained. s .

[0099] Step 4: Fuse the prompt text features with the original text features to obtain enhanced text features, and fuse the prompt text features with the image's spatial domain features to obtain enhanced visual features. Perform feature interaction based on the enhanced image frequency domain features, enhanced text features, and enhanced visual features to obtain the final multimodal features. Fake news is classified based on the multimodal features to obtain the classification results.

[0100] In the real world, users typically follow a pattern when reading articles containing both images and text. They first quickly glance at the image to get a general idea of the context, then read the accompanying text, continuously integrating information from both sources as they read. This interaction relies on prior knowledge and how the visual system analyzes the content, enhancing the user's overall understanding of the news.

[0101] Inspired by this pattern, this paper proposes a new multimodal feature alignment strategy that better mimics human reading habits and effectively captures the synergy between news text and images. Many current multimodal FND methods ignore the inherent reading patterns and the beneficial interactions between different features, resulting in suboptimal deep feature integration.

[0102] Enhanced Co-Attention Block: This embodiment uses a co-attention mechanism to learn attention weights between different input features, enhance the features of different inputs, and capture mutual information between the same and different modalities.

[0103] The steps include:

[0104] Step 401: Enhance the feature representation of each modality in text and image data;

[0105] Specifically, for text and image modalities, multi-head self-attention (MSA) is first employed to enhance the feature representation in each modality. For example, for the original text feature F t , the query matrix, keyword matrix and value matrix are calculated as follows:

[0106]

[0107] in, is a linear transformation, where H is the number of attention heads.

[0108] Then, the MSA features of the text modality are obtained as follows:

[0109]

[0110] Among them, h represents the h-th attention head, is the output linear transformation, d is Q t and K t The dimensions of the vector.

[0111] F p 、F v and F s Perform the same operation to obtain the corresponding feature R p 、R v and R s .

[0112] Step 402: Fusing the enhanced original text features and the prompt text features to obtain enhanced text modality features, and fusing the enhanced prompt text features and the spatial domain features to obtain enhanced visual features.

[0113] In this embodiment, a collaborative attention mechanism is used to generate enhanced multimodal features. Specifically, in order to utilize the prompt text features to perform collaborative attention with the original text and visual features, an operation similar to the above self-attention process is first performed, but F t Replace with R p To obtain the query matrix Q p , and F t Convert to R t To obtain the key matrix K t Sum matrix V t .

[0114] Then, the enhanced text modality feature R is obtained as follows pt :

[0115]

[0116] in, is the output linear transformation, R pt represents the enhanced original text features, and the hint text features are merged according to their relevance. H represents the number of attention heads.

[0117] Following the same process as above, the query matrix remains unchanged, and F t is converted to R v To obtain the key matrix K v Sum matrix V v .

[0118] The final cross-modal enhanced visual feature R pv Obtained as follows:

[0119]

[0120] Among them, R pv Represents enhanced visual features that incorporate hint text features based on their relevance.

[0121] Step 403: Combine the enhanced image frequency domain features, enhanced text features, and enhanced visual features using a collaborative attention mechanism and then splice them to obtain the final multimodal features;

[0122] Specifically, after obtaining two enhanced features R pt and R pv After that, each pair of the three features (i.e., “R s ”, “R pt ” and “R pv ”) Apply the joint attention mechanism in step 402 to obtain two visual enhancement features and four cross-modal enhancement features: R′ sv , R′ vs , R′ st , R′ ts , R′ vt and R′ tv .

[0123] Finally, these features are concatenated to obtain the final multimodal features:

[0124] F m =concat(R′ sv ,R′ vs ,R′ st ,R′ ts ,R′ vt ,R′ tv ) (14),

[0125] Among them, R′ sv is the frequency domain visual feature, R′ vs is the visual frequency domain feature, R′ st is the frequency domain text feature, R′ ts is the frequency domain feature of the text, R′ vt is the visual text feature, R′ tv is the visual feature of the text;

[0126] Step 404: Combine the multimodal features and input them into the FC layer to obtain the final fake news classification result;

[0127] In order to efficiently fuse multimodal features to obtain the final feature representation F m After that, it is input to the FC layer. Then, the output of the FC layer is passed through the softmax function to generate the distribution of the classification label, which is calculated as follows:

[0128] y p =softmax(W c F m +b c ) (15), where W c and b cRepresents the parameters of the FC layer used in the cross entropy loss function:

[0129]

[0130] Where y is the true label of the sample, Represents the predicted probability of the sample. Finally, 0 represents the prediction of fake news and 1 represents the prediction of real news.

[0131] Example

[0132] Two well-known datasets are used to evaluate the performance of the model, Weibo and Twitter.

[0133] The Weibo dataset includes real news from Xinhua News Agency and other authoritative Chinese news sources, as well as fake news from the official Weibo platform from 2012 to 2016. The texts are mainly in Chinese.

[0134] The Twitter dataset from the MediaEval benchmark program is a key resource for evaluating multimodal performance. It is divided into two distinct parts: a training set and a test set. The training set includes approximately 8,000 fake news stories and 6,000 real news stories, while the test set consists of approximately 2,000 news stories. The focus is on detecting fake news by extracting features from both text and images, and news stories containing only images or text are excluded during the training process.

[0135] In the experiments, the Weibo and Twitter datasets were split into training and test sets with a ratio of 7:3. To avoid overfitting, the parameters of the pre-trained BERT and VGG19 models were frozen during the training phase. To extract features from raw text and external cues, the BERT-base-uncased model was used for the Twitter dataset, and the BERT-base-chinese model was used for the Weibo dataset. The proposed model was trained on an NVIDIA RTX 2060 GPU using the AdamW optimizer. The batch size was set to 128, the training epochs to 50, and the learning rate to 3e-5. Accuracy, precision, recall, and F1 score were used as metrics to evaluate the performance of the model.

[0136] To demonstrate the effectiveness of the proposed model, its performance is tested using several state-of-the-art methods.

[0137] 1) Single-modal approach

[0138] BERT: This is a large-scale pre-trained model that is commonly used to extract unimodal text features in FND. This paper uses it as the backbone for capturing text features.

[0139] VGG19: It is a large-scale pre-trained model that is commonly used to extract unimodal image features in FND. This paper uses it as the backbone to capture image features.

[0140] 2) Multimodal methods

[0141] EANN: It uses a text neural network and pre-trained VGG19 to extract text and visual modal features respectively. Then, it concatenates them to form a multimodal feature of fake news.

[0142] MVAE: It uses bidirectional LSTM and pre-trained VGG19 for text and image extraction respectively, and then encodes the combined features.

[0143] SpotFake: It uses BERT for text extraction and VGG-19 for image feature extraction, concatenates these features, and then classifies fake news.

[0144] SpotFake+: It uses the pre-trained XLNet to extract text features and the VGG19 model to extract image features, which are then fed into the FC network to determine the classification results.

[0145] MCAN: It extracts spatial-frequency features from image and text features, and then adopts multiple co-attention modules to integrate these multimodal features for rumor detection.

[0146] MPFN: It is a multi-level fusion network that captures the features of each modality at different levels and achieves intra-layer and inter-layer modality fusion.

[0147] MRAN: It uses a multi-layer encoding network to extract text features and uses VGG19 to learn image features. These representations are input into FND's relational attention network.

[0148] CSFND: It captures local contextual features and combines them with global semantic features to form contextual semantic representation. In addition, a contextual testing strategy is designed for FND.

[0149] Performance evaluation and error analysis

[0150] Tables 1 and 2 provide detailed comparisons of the proposed HRMulFND model with the baselines for the two datasets.

[0151] Table 1 Performance comparison of the proposed method and 10 SOTA methods on the Weibo dataset

[0152]

[0153] Table 2 Performance comparison of the proposed method and 10 SOTA methods on the Twitter dataset

[0154]

[0155]

[0156] As shown in Tables 1 and 2, the model achieves a maximum accuracy of 90.8% and 89.2% on the Weibo and Twitter datasets, respectively. Based on these results, we draw the following conclusions:

[0157] (1) The proposed HRMulFND model shows excellent and consistent performance on both datasets, surpassing all baselines. This improvement is due to several key factors: First, the LLM hint bridges the semantic knowledge gap of the visual features of fake news text and improves the model's understanding ability. Second, the combination of wavelet transform and depthwise separable convolution effectively extracts frequency domain features from images and captures key image details. Third, the proposed multimodal feature alignment strategy mimics human reading patterns and leverages semantic relevance and cross-modal complementary features to improve classification results.

[0158] (2) For unimodal models, BERT performs worst on the Twitter dataset due to sample imbalance, with over 70% of posts focused on a single event. The more balanced event distribution of the Weibo dataset enables BERT to learn distinguishable features, ensuring data diversity. In addition, the text-based BERT outperforms the image-based VGG19 model because the Weibo dataset contains images of higher quality and quantity than Twitter. However, VGG19 has difficulty extracting strong features from the images in the Weibo dataset.

[0159] (3) Multimodal methods outperform unimodal methods on both datasets, highlighting the value of multimodal information in FND. Among multimodal techniques, EANN and MVAE perform poorly because they learn features from different modalities independently. These methods ignore the semantic differences between modalities, resulting in different embedding spaces and suboptimal fusion results.

[0160] (4) The performance of SpotFake and SpotFake+ varies significantly across datasets, as the Twitter dataset is unbalanced, resulting in poor classification performance. For the balanced Weibo dataset, pre-trained models such as BERT and XLNet can significantly improve model performance. Our model outperforms SpotFake+, achieving a better balance in precision, recall, and F1 score, while also being less complex in design.

[0161] (5) MRAN and CSFND achieve stable and decent performance on both datasets because they use attention-based feature fusion instead of simpler concatenation methods, which better capture cross-modal relationships. However, these models are limited by the scope and scale of extracted features, which hinders their ability to fully exploit the rich information in news images, thus limiting further performance improvements.

[0162] (6) MCAN and MPFN enhance image features through the DCT+CNN method for frequency domain extraction and use a multi-level fusion strategy for interactive cross-modal mixing. However, this technique for extracting frequency domain features is outdated and has difficulty processing images with complex textures and edges. In addition, the fusion mechanism does not fully consider multimodal interactions, resulting in the loss of valuable information during the fusion process.

[0163] In order to intuitively demonstrate the classification advantage of the proposed model rather than just browsing the monotonous numbers, the present invention plots a 3D UMAP representation of the model classification performance. Figure 2 In , the original features make classification difficult due to the overlap of samples from different classes. However, Figure 3 The results show that the learned features of our model clearly separate the samples into two distinct clusters: fake news and real news, with a clear gap between the two. These visualizations demonstrate the effectiveness of our HRMulFND model in enhancing the separability of different classes in the latent feature space.

[0164] Example 2

[0165] This embodiment provides a human reading-inspired multimodal fake news detection system, including:

[0166] A prompt generation module, which is used to generate prompt text related to the image based on the acquired news image data and a predefined template;

[0167] A text feature extraction module is used to encode the acquired original news text data and prompt text to obtain original text features and prompt text features respectively;

[0168] An image feature extraction module is used to extract the spatial domain features and frequency domain features of the image respectively;

[0169] A feature enhancement module is used to enhance the original text features, the hint text features, the spatial domain features and the frequency domain features of the image;

[0170] The multimodal fusion module is used to fuse the enhanced original text features and the prompt text features to obtain enhanced text features, and to fuse the spatial domain features of the enhanced prompt text feature image to obtain enhanced visual features. Based on the enhanced image frequency domain features, enhanced text features, and enhanced visual features, feature interaction is performed to capture the mutual information between the same and different modalities, and the final multimodal features are obtained by splicing.

[0171] The classification module is used to classify fake news based on multimodal features to obtain classification results.

[0172] It should be noted that the specific implementation of the human reading-inspired multimodal false news detection system in the embodiment of the present invention is similar to the specific implementation of the human reading-inspired multimodal false news detection method in the embodiment of the present invention. Please refer to the description of the method section for details. To reduce redundancy, it is not repeated here.

[0173] Example 3

[0174] This embodiment provides a computer-readable storage medium having a computer program stored thereon. When the program is executed by a processor, the steps in the human reading heuristic multimodal fake news detection method described above are implemented.

[0175] Example 4

[0176] This embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the steps in the human reading heuristic multimodal fake news detection method described above are implemented.

[0177] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage and optical storage, etc.) containing computer-usable program code.

[0178] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0179] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0180] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0181] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing related hardware through a computer program. The program can be stored in a computer-readable storage medium, and when executed, the program can include the processes in the above-described method embodiments. The storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0182] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the present invention.

Claims

1. Human reading heuristic multimodal fake news detection method, characterized by: The steps include: Generate image-related prompt text based on acquired news image data and predefined templates; Encode the acquired original news text data and prompt text respectively to obtain original text features and prompt text features; Extract the spatial domain features and frequency domain features of the image respectively; Enhance original text features, hint text features, image spatial domain features and frequency domain features; The enhanced original text features and the prompt text features are fused to obtain enhanced text features, and the spatial domain features of the enhanced prompt text feature image are fused to obtain enhanced visual features; Based on the enhanced image frequency domain features, enhanced text features and enhanced visual features, feature interaction is performed to capture the mutual information between the same and different modalities, and the final multimodal features are spliced together. The fake news classification is performed based on the multimodal features to obtain the classification results.

2. The human reading heuristic multimodal fake news detection method according to claim 1, wherein: When encoding the obtained original news text data and prompt text, the sentences are decomposed into continuous word sequences. The word sequences and the pre-trained BERT model are combined to extract text context information. The text context information is applied with an average pooling operation to obtain the features of the entire text from all words.

3. The human reading heuristic multimodal fake news detection method according to claim 1, wherein: The VGG19 network is used to extract image spatial domain features.

4. The human reading heuristic multimodal fake news detection method according to claim 1, wherein: Image frequency domain feature extraction includes: The image is preprocessed and wavelet transformed to obtain a tensor with multiple channels; A tensor with multiple channels is subjected to depthwise separable convolution to obtain a frequency domain feature vector; the depthwise separable convolution includes depthwise convolution and pointwise convolution, the depthwise convolution convolves the tensor of each channel, and the pointwise convolution combines the output of each channel to obtain a frequency domain feature vector.

5. The human reading heuristic multimodal fake news detection method according to claim 1, wherein: When enhancing the original text features, prompt text features, spatial domain features and frequency domain features of the image, it includes calculating the query matrix, keyword matrix and value matrix of each modality respectively. Based on the query matrix, keyword matrix and value matrix of each modality, the enhanced features of the corresponding modality are obtained in combination with multi-head self-attention calculation.

6. The human reading heuristic multimodal fake news detection method according to claim 1, wherein: The calculation formula for enhanced text features is: The calculation formula for enhanced visual features is: in, is the output linear transformation, Q p is the query matrix corresponding to the original text features, K t and V t is the key matrix and value matrix corresponding to the original text features, h represents the hth attention head, H represents the number of attention heads, K v and V v are the key matrix and value matrix corresponding to the spatial domain features, and d is the feature dimension of the query vector and keyword vector.

7. The human reading heuristic multimodal fake news detection method according to claim 1, wherein: When classifying fake news based on multimodal features and obtaining the classification results, the multimodal feature representation is input into the FC layer, and the output of the FC layer generates the distribution of classification labels through the softmax function.

8. A human reading-inspired multimodal fake news detection system, characterized by: include: A prompt generation module, which is used to generate prompt text related to the image based on the acquired news image data and a predefined template; A text feature extraction module is used to encode the acquired original news text data and prompt text to obtain original text features and prompt text features respectively; An image feature extraction module is used to extract the spatial domain features and frequency domain features of the image respectively; A feature enhancement module is used to enhance the original text features, the hint text features, the spatial domain features and the frequency domain features of the image; A multimodal fusion module is used to fuse the enhanced original text features and the prompt text features to obtain enhanced text features, and to fuse the spatial domain features of the enhanced prompt text feature image to obtain enhanced visual features; Based on the enhanced image frequency domain features, enhanced text features and enhanced visual features, feature interaction is performed to capture the mutual information between the same and different modalities, and the final multimodal features are obtained by splicing. The classification module is used to classify fake news based on multimodal features to obtain classification results.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the human reading heuristic multimodal fake news detection method according to any one of claims 1 to 7 are implemented.

10. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps in the human reading heuristic multimodal fake news detection method according to any one of claims 1 to 7 are implemented.

Images