Multi-modal false news detection method and system based on coarse and fine granularity feature fusion

Abstract

The invention relates to the technical field of false news detection, and provides a multi-modal false news detection method and system based on coarse and fine granularity feature fusion, and the method comprises the following steps: carrying out the coarse granularity feature extraction and the fine granularity feature extraction through a plurality of feature encoders based on contrast training for each modal of obtained target news to be detected, multi-granularity features are obtained; for the obtained multi-granularity features, carrying out average pooling on the multi-modal fine-granularity features, and splicing the pooled fine-granularity features with the coarse-granularity features of the corresponding modals to obtain coarse-granularity and fine-granularity fusion features of each modal; and converting the obtained coarse-grained features and the coarse-grained and fine-grained fusion features of each mode into features of the same dimension, splicing the features, and inputting the spliced features into a false news detection classifier to obtain a detection result that whether the target news is false news. According to the method, accurate and efficient false news detection is realized through comparative learning, coarse and fine granularity fusion and the like.

Description

Technical Field

[0001] The present disclosure relates to the technical field related to fake news detection, and more specifically, to a multimodal fake news detection method and system based on coarse and fine granularity feature fusion. Background Art

[0002] The statements in this section merely provide background information related to the present disclosure and do not necessarily constitute prior art.

[0003] With the rapid development of internet technology, online social networks (OSNs) have become widespread and indispensable platforms for people to receive, share, and exchange information. However, the quality of widely disseminated news varies greatly, posing a threat to the credibility of news media and increasing the difficulty for the public to judge the trustworthiness of information. The spread and proliferation of false news has had significant negative impacts across various sectors. In this context, ensuring the accuracy and credibility of information has become increasingly urgent. Therefore, research on false news detection technology has a positive and far-reaching impact.

[0004] In recent years, deep learning has demonstrated significant advantages in fake news detection, particularly in multimodal detection. Convolutional neural networks (CNNs), recurrent neural networks (RNNs), and pre-trained models such as BERT and GPT have been widely used in feature extraction and semantic understanding. To improve detection accuracy, researchers have also proposed various feature extractors and data augmentation techniques to enhance data quality and multimodal fusion capabilities. However, existing methods still have the following shortcomings:

[0005] 1) Single feature extraction dimension: Fake news texts have complex structures and diverse forms of expression. Single feature extraction models often fail to capture their multidimensional semantic features, resulting in limited ability to identify fake information.

[0006] 2) Current multimodal detection methods mostly focus on the consistency between text tags and image patches, ignoring the importance of object-level granularity and relationship modeling. They lack a hierarchical understanding of information and find it difficult to achieve fusion modeling of global and local information.

[0007] Most existing multimodal technologies only consider the level of consistency between each tag and the image patch, making the current model too limited to the matching of "local elements (such as words and local parts of the image)" and ignoring higher-level "object-level", "scene-level" and "semantic relationship" alignment modeling, resulting in limited multimodal understanding capabilities in fake news detection and inaccurate detection results.

[0008] 3) Semantic shift problem in multimodal representation transfer: There is often semantic inconsistency or conflict between images and text in fake news. Existing multimodal models (such as BERT and VAE) have difficulty identifying such semantic differences during feature transfer, resulting in a decrease in model generalization ability and prone to misjudgment in practical applications. Summary of the Invention

[0009] To address the above-mentioned issues, this paper proposes a multimodal fake news detection method and system based on coarse- and fine-grained feature fusion. This method extracts coarse- and fine-grained features of text and images through a feature encoder, and uses contrastive learning to enhance the ability to recognize multimodal semantic consistency and differences, effectively alleviating the problem of domain shift and improving the accuracy and generalization performance of fake news detection.

[0010] In order to achieve the above objectives, the present disclosure adopts the following technical solutions:

[0011] One or more embodiments provide a multimodal fake news detection method based on coarse-grained and fine-grained feature fusion, comprising the following steps:

[0012] For each modality of the target news to be detected, multiple feature encoders are used to perform coarse-grained and fine-grained feature extraction to obtain multi-granularity features.

[0013] During feature encoder training, the loss is calculated by comparing coarse-grained and fine-grained features. The goal is to construct a loss function for training by bringing the image and text embedding vectors of each real news item closer together in the multimodal vector space and pushing the image and text embedding vectors of different news items further apart.

[0014] For the obtained multi-granularity features, average pooling is performed on the fine-grained features of the multiple modalities respectively, and the pooled fine-grained features are spliced with the coarse-grained features of the corresponding modality to obtain the coarse-grained and fine-grained fusion features of each modality;

[0015] The obtained coarse-grained features and the coarse-grained and fine-grained fusion features of each modality are transformed into features of the same dimension and then spliced. The features are then input into the fake news detection classifier to obtain the detection result of whether the target news is fake news.

[0016] One or more embodiments provide a multimodal fake news detection system based on coarse-grained and fine-grained feature fusion, including:

[0017] The feature extraction module is configured to perform coarse-grained and fine-grained feature extraction on each modality of the target news to be detected through multiple feature encoders to obtain multi-granularity features;

[0018] During feature encoder training, the loss is calculated by comparing coarse-grained and fine-grained features. The goal is to construct a loss function for training by bringing the image and text embedding vectors of each real news item closer together in the multimodal vector space and pushing the image and text embedding vectors of different news items further apart.

[0019] The coarse-grained and fine-grained feature concatenation module is configured to perform average pooling on the obtained multi-granularity features, respectively, and concatenate the pooled fine-grained features with the coarse-grained features of the corresponding modality to obtain the coarse-grained and fine-grained fusion features of each modality;

[0020] The classification module is configured to transform the obtained coarse-grained features and the coarse-grained and fine-grained fusion features of each modality into features of the same dimension, then splice them together and input them into the fake news detection classifier to obtain the detection result of whether the target news is fake news.

[0021] An electronic device includes a memory and a processor, and computer instructions stored in the memory and executed on the processor. When the computer instructions are executed by the processor, the steps of the above-mentioned multimodal fake news detection method based on coarse and fine granularity feature fusion are completed.

[0022] A computer-readable storage medium is used to store computer instructions. When the computer instructions are executed by a processor, the steps in the above-mentioned multimodal fake news detection method based on coarse and fine granularity feature fusion are completed.

[0023] Compared with the prior art, the present invention has the following beneficial effects:

[0024] This paper significantly enhances the model's ability to identify multimodal information inconsistencies at both the global and local levels through the collaborative modeling of coarse and fine-grained features. The introduced contrastive learning loss function effectively alleviates the domain shift problem caused by multi-source data and improves semantic consistency between different modalities, thereby enhancing the generalization performance of the fake news detection model in complex environments with multiple scenarios and multiple semantics. Furthermore, this method can more fully exploit the potential correlation features between images and text, improving the accuracy and stability of fake news detection.

[0025] The advantages of the present disclosure and additional advantages will be described in detail in the following specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The accompanying drawings, which constitute a part of the present disclosure, are used to provide a further understanding of the present disclosure. The exemplary embodiments of the present disclosure and their descriptions are used to explain the present disclosure but do not constitute a limitation of the present disclosure.

[0027] Figure 1 is a flow chart of the detection method of Example 1 of the present disclosure;

[0028] Figure 2 Schematic diagram of the overall framework adopted by the multimodal fake news detection method based on coarse and fine granularity feature fusion in Example 1 of the present disclosure; DETAILED DESCRIPTION

[0029] The present disclosure will be further described below with reference to the accompanying drawings and embodiments.

[0030] It should be noted that the following detailed descriptions are exemplary and intended to provide further explanation of the present disclosure. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present disclosure belongs.

[0031] It should be noted that the terms used herein are only for the purpose of describing specific embodiments and are not intended to limit the exemplary embodiments according to the present disclosure. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or combinations thereof. It should be noted that, in the absence of conflict, the various embodiments in the present disclosure and the features in the embodiments can be combined with each other. The embodiments will be described in detail below with reference to the accompanying drawings.

[0032] Explanation of terms:

[0033] Multimodal fake news: news content that uses multiple different types of media elements, typically text and images, to spread false or inaccurate information;

[0034] Multimodal fake news identification: Identify the truth or falsity of news based on multimodal news data;

[0035] BERT (Bidirectional Encoder Representations from Transformers): A pre-trained natural language processing model based on the Transformer architecture that can effectively generate context-dependent vector representations of words or sentences for various NLP tasks.

[0036] Variational Autoencoder (VAE): A generative model that combines the concepts of variational inference and autoencoders. Its core goal is to learn the underlying probability distribution of input data and to generate new data samples from this distribution. Essentially, given the data, it seeks a probabilistic model that can generate similar data.

[0037] CLIP (Contrastive Language-Image Pretraining): A contrastive language-image pretraining model. Using contrastive learning, the image encoder and text encoder map images and text into the same feature space, learning associations by maximizing the similarity of matching image-text pairs and minimizing the similarity of non-matching pairs.

[0038] Fully Connected Layer: A basic layer type in a neural network, where each neuron is connected to every neuron in the previous layer. Fully connected layers map input features to the output space and are a key component of the model's final classification or regression task.

[0039] Example 1

[0040] In the technical solutions disclosed in one or more embodiments, Figures 1 to 2 As shown in FIG, a multimodal fake news detection method based on coarse-grained and fine-grained feature fusion includes the following steps:

[0041] Step 1: For each modality of the target news to be detected, multiple feature encoders are used to perform coarse-grained and fine-grained feature extraction to obtain multi-granularity features;

[0042] During feature encoder training, the loss is calculated by comparing coarse-grained and fine-grained features. The goal is to construct a loss function for training by bringing the image and text embedding vectors of each real news item closer together in the multimodal vector space and pushing the image and text embedding vectors of different news items further apart.

[0043] Step 2: For the obtained multi-granularity features, average pool the fine-grained features of the multi-modal features respectively, and concatenate the pooled fine-grained features with the coarse-grained features of the corresponding modality to obtain the coarse-grained and fine-grained fusion features of each modality;

[0044] Step 3: The obtained coarse-grained features and the coarse-grained and fine-grained fusion features of each modality are transformed into features of the same dimension and then spliced. The features are then input into the fake news detection classifier to obtain the detection result of whether the target news is fake news.

[0045] In this embodiment, the method first uses feature encoders of different structures to extract coarse-grained features (such as overall semantics, global context) and fine-grained features (such as word-level or local image area features) for each modality (such as text and image) to obtain multi-level feature expressions. During the training phase, in order to improve the model's ability to distinguish the consistency and difference between multimodal information, a contrastive learning mechanism is introduced to embed the images and texts in real news samples close to each other in the multimodal feature space, while the images and texts between false and real news samples are embedded farther apart, so that the model learns more robust semantic alignment capabilities. Subsequently, by applying an average pooling operation to the fine-grained features, its overall semantic representation is extracted and spliced with the coarse-grained features of the corresponding modality to further enhance the expression ability of the internal information of the modality. Finally, the coarse-grained features and the fused multimodal features are uniformly mapped to the same dimension and then spliced, and input into the subsequently constructed classifier to complete the discrimination of true and false news.

[0046] This embodiment significantly enhances the model's ability to identify multimodal information inconsistencies at both the global and local levels through the collaborative modeling of coarse and fine-grained features. The introduced contrastive learning loss function effectively mitigates the domain shift problem caused by multi-source data and improves semantic consistency between different modalities, thereby enhancing the generalization performance of the fake news detection model in complex environments with multiple scenarios and multiple semantics. Furthermore, this method can more fully exploit the potential correlation features between images and text, improving the accuracy and stability of fake news detection.

[0047] In multimodal learning, a "token" usually refers to a word or a subword unit in the text, and an "image patch" is to divide the image into multiple small areas (patches) for processing. The current mainstream models often only focus on whether a word (or subword) in the text is "matched" or "related" to a local area in the image. For example, whether the word "dog" and the dog's face in the image are semantically related. In this embodiment, the coarse-grained features and fine-grained features of each modality are fused to capture the inconsistency of multimodal information at the global and local levels. That is, it is not just about comparing the correspondence between words and local images, but about understanding the correspondence between a group of words (phrases, sentences) and an image object or the entire scene. In other words, modeling should be performed at the "combination" level (rather than a single element) to detect fake news, thereby improving the accuracy of fake news identification.

[0048] In the field of artificial intelligence and information processing, modality refers to the form of information presentation or perception. In other words, different modalities refer to the different channels through which information is presented, such as text, image, voice, video, etc. In an online news article, common modal information includes text modality, image modality, video modality, and audio modality. In this embodiment, text modality and image modality are mainly used for explanation.

[0049] In step 1, coarse-grained and fine-grained feature extraction is performed on the text modal features, including the following:

[0050] Step 11: Extract fine-grained features (Token-level) of text: Use the pre-trained Bert model to extract the contextual semantics of the text and generate an embedding vector (fine-text Feature) for each token, which is the fine-grained text feature embedding.

[0051] Specifically, the BERT model acts as a text encoder to segment text data into words, where each word or subword is a token. It then generates a vector representation for each token (i.e., each word or subword), which contains the semantic features of the token in context.

[0052] Use Bert to extract the contextual semantics of the text and generate the embedding of each token:

[0053]

[0054] in, n is the text length, d is the embedding dimension, T is the text data, and R represents the real number field.

[0055] After training on a large corpus, the BERT model in this embodiment can learn contextual information, have better generalization capabilities for other tasks, and can express the specific meaning of words in different contexts, capturing text details, situational differences, semantic dependencies, etc. For example, the meaning of "virus" in medical news and political commentary is different, and BERT can identify this contextual difference.

[0056] Step 12: Extract coarse-grained text features: Use CLIP text encoder to extract sentence-level global semantic embedding (coarse-text feature) to obtain coarse-grained text feature embedding The formula is:

[0057]

[0058] Coarse-grained text feature embedding, that is, the entire sentence corresponds to a vector (sentence vector), which expresses the overall semantics of the sentence, such as "is it positive, sarcastic, or factual"; it is used for overall matching when aligned with image modality.

[0059] In this embodiment, a text vectorization method is used to convert text data into a numerical form that can be understood and processed by computers, and feature extraction focuses on accurately extracting key information from the text. These two important processes have laid a solid foundation for the application of text data in various deep learning tasks. The BERT model is used for vectorization processing: it can generate a text representation with context-awareness, so that the generated text representation not only contains the semantic information of a single word, but also fully considers the meaning of the word in a specific context, thereby greatly improving the quality and accuracy of the text semantic representation. Using CLIP as a text encoder can extract overall semantic features, thereby better capturing the multimodal information inconsistency in fake news at both the global and local levels.

[0060] In step 1, coarse-grained and fine-grained feature extraction is performed for the image modality, including the following:

[0061] Step 1-1: Extract fine-grained image features: Use the pre-trained VAE model (variational autoencoder) to extract the local potential features of the image (fine-image feature), which is the fine-grained image feature embedding

[0062] Specifically, the image is first input into the image encoder of the pre-trained VAE model. The encoder extracts the parameters (such as mean and variance) of each dimension distribution in the image latent space, and generates a potential feature representation of the image through sampling based on these parameters.

[0063] In this embodiment, a 4096-dimensional image embedding vector is finally obtained as a fine-grained feature representation of the image.

[0064] The VAE model doesn't directly output a fixed vector representation of the image. Instead, the features it extracts focus on deep representations such as local image details and texture structure, capturing patterns of change in microscopic regions within the image. Compared to directly output encoding results, the latent space features generated by the VAE model have greater expressiveness and generalization capabilities. They can effectively address common issues in fake news images, such as local image forgery and detail manipulation, and help improve the model's ability to perceive unusual details.

[0065] Step 1-2, image coarse-grained feature extraction: Use the pre-trained model CLIP image encoder to extract the overall semantic embedding features of the image (coarse-image Feature) to obtain the coarse-grained image feature embedding

[0066] Specifically, the image encoder in the CLIP framework (which can be based on the ResNet or Vision Transformer architecture) is used to process the image as a whole to generate a 512-dimensional global semantic vector for the image. This vector serves as a sentence-level embedding of the image to express the overall semantic information of the image.

[0067] Coarse-grained image feature embedding can summarize the overall visual theme of an image, for example, identifying whether an image depicts a disaster scene, a medical setting, a close-up of a person, or other event-related content. This feature helps determine the consistency of image and text in multimodal tasks, enabling the identification of out-of-context images in fake news and improving the robustness and accuracy of the system.

[0068] In step 2, the obtained multi-granularity features are average-pooled on the fine-grained features of the multi-modal features, and the pooled fine-grained features are concatenated with the coarse-grained features of the corresponding modality to obtain the coarse-grained and fine-grained fused features (fused-image Feature) of each modality;

[0069] In order to preserve the fine-grained feature information of the BERT model and combine it with the global semantics provided by CLIP, this embodiment extracts fine-grained text features from the BERT model. Perform average pooling on it and Converted into global semantic representation, from the fine-grained image features of the VAE model Perform average pooling on it and Transformed into a global semantic representation, as follows:

[0070] Step 21: Perform average pooling on the fine-grained text features. The formula is as follows:

[0071]

[0072] Step 22: Average pooling of fine-grained image features. The formula is as follows:

[0073]

[0074] Step 23: Concatenate the fine-grained text features and coarse-grained text features after average pooling:

[0075]

[0076] Step 24: Concatenate the fine-grained image features and the coarse-grained image features after average pooling:

[0077]

[0078] Where m represents the number of regions where the image is segmented, and Pool(·) represents the pooling function.

[0079] Specifically, step 23 and step 24 can be spliced using a fully connected neural network (fc);

[0080] Multimodal features are a supplement to unimodal features and are used to enhance the semantic representation of unimodal features. Previous works usually use a single network to mine coarse and fine features from one modality.

[0081] In step 3, for news containing both text and image modalities, the coarse-grained image features, coarse-grained text features, and coarse-grained and fine-grained fusion features of each modality are converted into features of the same dimension. These features are then concatenated through a fusion module (CONCATENATE) and input into a fake news detection classifier to determine whether the target news is fake news, either real or fake.

[0082] Furthermore, the obtained coarse-grained features and the coarse-grained and fine-grained fusion features of each modality are transformed into features of the same dimension and then spliced together. This can be achieved in the fusion module. The fusion module (CONCATENATE) is configured to perform an attention operation on each input feature and perform fusion and splicing based on the attention mechanism.

[0083] Specifically, the attention mechanism includes a text attention mechanism and an image attention mechanism. The text attention mechanism and the image attention mechanism each consist of two collaborative attention layers. By connecting these two collaborative attention layers, the multimodal correlation features M output by the fusion module (CONCATENATE) are obtained. t and M v :

[0084]

[0085] Among them, CA means that the calculation is performed using the Co-Attention layer, and T, TS, V, and VS are calculated as Q, K, and V in the collaborative attention layer respectively. Represents a join operation.

[0086] The obtained coarse-grained image features, coarse-grained text features and two splicing features are transformed into features of the same dimension through the fully connected layer (fc) and then concatenated.

[0087] The spliced multimodal correlation features are input into the binary classification model fake news detector. The binary classification model fake news detector can use a support vector machine to obtain the detection result of whether the target news is fake news. Specifically:

[0088]

[0089] Where F(·) represents the concatenation operation, Y represents the predicted label, cos represents the calculated cosine similarity, and L represents the loss function.

[0090] The concatenated multimodal correlation features are then fed into a constructed binary classification model, which acts as a fake news detector to determine whether the input news is fake. This binary classification model can be built based on a variety of machine learning or deep learning architectures, including fully connected neural networks (FCNNs), support vector machines (SVMs), and deep classification networks that combine an attention mechanism and a Transformer architecture.

[0091] The input layer of the binary classification model receives the fused feature vector, which is then processed through several hidden layers for nonlinear mapping and feature interaction. The final output layer uses a Sigmoid or Softmax function to generate a classification result of 0 or 1, representing "real news" or "fake news" respectively.

[0092] To improve classification accuracy, this binary classification model uses labeled datasets of real and fake news during training and performs supervised learning optimization using a cross-entropy loss function. To prevent the model from overfitting to domain-specific data, dropout mechanisms, L2 regularization, or contrastive learning can be incorporated into the training phase, making the model more generalizable and robust to domain shifts.

[0093] A further technical solution includes a method for training a feature encoder for coarse-grained and fine-grained feature extraction using contrastive learning, comprising the following steps:

[0094] Step S1: Obtain multimodal news data and pre-process it as a training set;

[0095] Specifically, we use web crawler technology and preprocessing operations to obtain a multimodal news dataset as a training set;

[0096] Step S2: Based on the constructed multiple feature encoders, multi-granularity features of the images and texts of the target news in the training set are extracted to obtain fine-grained text feature embedding, coarse-grained text feature embedding, fine-grained image feature embedding, and coarse-grained image feature embedding;

[0097] In this embodiment, for text modality data, the feature encoder includes a BERT model for extracting fine-grained text features and a CLIP text encoder for extracting coarse-grained text features; for image modality data, the feature encoder includes a VAE model for extracting fine-grained image features and a CLIP image encoder for extracting coarse-grained image features;

[0098] Step S3: Embedding coarse-grained image features and coarse-grained text feature embedding Perform L2 norm normalization and fine-grained image feature embedding and fine-grained text feature embedding Perform L2 norm normalization;

[0099] Step S4: Perform matrix dot product operation on the normalized feature embedding to obtain the similarity matrix Sim. Specifically, the cosine similarity is used for calculation. The formula is as follows:

[0100]

[0101] Where * represents the bitwise multiplication of matrix elements, and e is the reciprocal of the temperature coefficient in contrastive learning.

[0102] The temperature coefficient in the contrastive learning loss function of the traditional CLIP model is optimized directly from training, rather than being a hyperparameter. This example uses the temperature coefficient obtained from Taiyi-CLIP training as a hyperparameter. The temperature coefficient controls the model's ability to discriminate against negative samples. A larger value for t increases the penalty for difficult negative examples. A larger value in the i-th row and j-th column of the similarity matrix indicates a greater similarity between the image feature vector of the i-th news item and the text feature vector of the j-th news item.

[0103] Step S4: Construct a loss function with the goal of bringing the image and text embedding vectors of each real news item closer in the multimodal vector space and pushing the image and text vectors of different news items further apart. Based on the obtained similarity matrix Sim, calculate the loss function value and adjust the parameters of each feature encoder. Iterate the training until the feature extraction accuracy is met, and obtain the trained feature encoder.

[0104] In this embodiment, Infinity loss is used for training. The N-classification loss Li for images and the N-classification loss Lt for text are defined as follows. The total contrast loss L is the mean of the two. The training goal is to bring the image and text embedding vectors of each real news item closer together in the multimodal vector space, and push the image and text vectors of different news items further apart. The specific loss function calculation formula is as follows:

[0105]

[0106] in, Represents the coarse-grained classification loss of the image, represents the coarse-grained classification loss of the text, Sim represents the obtained similarity matrix, subscripts i and j represent rows and columns respectively; N represents the number of samples.

[0107] This embodiment introduces an auxiliary loss function for contrastive learning. Through this loss function, the model can effectively reduce the impact of domain shift, allowing the model to maintain good performance across different data distributions and scenarios. By introducing an auxiliary loss function for contrastive learning to reduce domain shift, and through feature alignment, domain adversarial training allows the encoder to generate features that are difficult to distinguish between domains. Using contrastive loss constraints to bring positive sample features from different domains closer together, thereby minimizing the distance between features in different domains, unifying feature distributions and optimizing global semantic consistency between text and images, it can significantly improve the effectiveness of fake news detection.

[0108] In the above scheme, a multimodal dataset is obtained through web crawler technology and preprocessing operations, and multi-granular features of the image and text of the target news are extracted. The fine-grained features are average-pooled to obtain a unified vector dimension, which is then spliced with the coarse-grained features. The coarse-grained and fine-grained features of the image and text features are contrastively learned respectively, and the loss is fed back into the model. The coarse-grained features of the text and image and the two fused features are spliced and put into the classifier to classify whether the target news is fake news. Accurate and efficient fake news detection is achieved through contrastive learning and coarse- and fine-grained fusion.

[0109] Example 2

[0110] Based on Example 1, this embodiment provides a multimodal fake news detection system based on coarse-grained and fine-grained feature fusion, including:

[0111] The feature extraction module is configured to perform coarse-grained and fine-grained feature extraction on each modality of the target news to be detected through multiple feature encoders to obtain multi-granularity features;

[0112] During feature encoder training, the loss is calculated by comparing coarse-grained and fine-grained features. The goal is to construct a loss function for training by bringing the image and text embedding vectors of each real news item closer together in the multimodal vector space and pushing the image and text embedding vectors of different news items further apart.

[0113] The coarse-grained and fine-grained feature concatenation module is configured to perform average pooling on the obtained multi-granularity features, respectively, and concatenate the pooled fine-grained features with the coarse-grained features of the corresponding modality to obtain the coarse-grained and fine-grained fusion features of each modality;

[0114] The classification module is configured to transform the obtained coarse-grained features and the coarse-grained and fine-grained fusion features of each modality into features of the same dimension, then splice them together and input them into the fake news detection classifier to obtain the detection result of whether the target news is fake news.

[0115] Example 3

[0116] Based on Example 1, this embodiment provides an electronic device, including a memory and a processor, and computer instructions stored in the memory and running on the processor. When the computer instructions are executed by the processor, the steps of the multimodal fake news detection method based on coarse and fine granularity feature fusion described in the above Example 1 are completed.

[0117] Example 4

[0118] Based on Example 1, this embodiment provides a computer-readable storage medium for storing computer instructions. When the computer instructions are executed by a processor, the steps of the multimodal fake news detection method based on coarse and fine granularity feature fusion described in Example 1 are completed.

[0119] The foregoing description is merely a preferred embodiment of the present disclosure and is not intended to limit the present disclosure. Those skilled in the art will readily appreciate that various modifications and variations are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present disclosure shall be included within the scope of protection of the present disclosure.

[0120] Although the above describes the specific implementation methods of the present disclosure in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present disclosure. Those skilled in the art should understand that on the basis of the technical solution of the present disclosure, various modifications or variations that can be made by those skilled in the art without creative work are still within the scope of protection of the present disclosure.

Claims

1. A multimodal fake news detection method based on coarse and fine granularity feature fusion, characterized by: The steps include: For each modality of the target news to be detected, multiple feature encoders are used to perform coarse-grained and fine-grained feature extraction to obtain multi-granularity features. During feature encoder training, the loss is calculated by comparing coarse-grained and fine-grained features. The goal is to construct a loss function for training by bringing the image and text embedding vectors of each real news item closer together in the multimodal vector space and pushing the image and text embedding vectors of different news items further apart. For the obtained multi-granularity features, average pooling is performed on the fine-grained features of the multiple modalities respectively, and the pooled fine-grained features are spliced with the coarse-grained features of the corresponding modality to obtain the coarse-grained and fine-grained fusion features of each modality; The obtained coarse-grained features and the coarse-grained and fine-grained fusion features of each modality are transformed into features of the same dimension and then spliced. The features are then input into the fake news detection classifier to obtain the detection result of whether the target news is fake news.

2. The multimodal fake news detection method based on coarse and fine granularity feature fusion according to claim 1 is characterized in that: For text modal features, coarse-grained and fine-grained feature extraction is performed, including the following: Use the pre-trained BERT model to extract the contextual semantics of the text and generate an embedding vector for each token as fine-grained text feature embedding; The CLIP text encoder is used to extract global semantic embedding at the sentence level to obtain coarse-grained text feature embedding.

3. The multimodal fake news detection method based on coarse and fine granularity feature fusion according to claim 1 is characterized in that: For image modalities, coarse-grained and fine-grained feature extraction is performed, including the following: Use the pre-trained VAE model to extract local latent features of the image as fine-grained image feature embedding; The pre-trained CLIP image encoder is used to extract the overall semantic embedding of the image and obtain the coarse-grained image feature embedding.

4. The multimodal fake news detection method based on coarse and fine granularity feature fusion according to claim 1 is characterized in that: For the obtained multi-granularity features, average pooling is performed on the fine-grained features of the multi-modal features respectively, and the pooled fine-grained features are spliced with the coarse-grained features of the corresponding modality to obtain the coarse-grained and fine-grained fusion features of each modality, including the following steps: Perform average pooling on fine-grained text features; Average pooling of fine-grained image features; Concatenate the fine-grained text features after average pooling with the coarse-grained text features; The fine-grained image features after average pooling are concatenated with the coarse-grained image features.

5. The multimodal fake news detection method based on coarse- and fine-grained feature fusion according to claim 4 is characterized in that: The fine-grained image features after average pooling and the coarse-grained image features are concatenated using a fully connected neural network.

6. The multimodal fake news detection method based on coarse-grained and fine-grained feature fusion according to claim 1, characterized in that: The obtained coarse-grained features and the coarse-grained and fine-grained fusion features of each modality are transformed into features of the same dimension and then spliced together. Attention operation is performed on each input feature, and the fusion and splicing are performed after processing based on the attention mechanism.

7. The multimodal fake news detection method based on coarse and fine granularity feature fusion according to claim 1 is characterized in that: Also included is a method for training a feature encoder for coarse-grained and fine-grained feature extraction using contrastive learning, comprising the following steps: Obtain multimodal news data and preprocess it as a training set; Based on the constructed multiple feature encoders, multi-granular features of the target news images and texts in the training set are extracted to obtain fine-grained text feature embedding, coarse-grained text feature embedding, fine-grained image feature embedding, and coarse-grained image feature embedding; Perform L2 norm normalization on coarse-grained image feature embedding and coarse-grained text feature embedding, and perform L2 norm normalization on fine-grained image feature embedding and fine-grained text feature embedding; Perform matrix dot product operation on the normalized feature embedding to obtain the similarity matrix; A loss function is constructed with the goal of bringing the image and text embedding vectors of each real news item closer in the multimodal vector space and pushing the image and text vectors of different news items further apart. Based on the obtained similarity matrix Sim, the loss function value is calculated and the parameters of each feature encoder are adjusted. The training is iterated until the feature extraction accuracy is met, and the trained feature encoder is obtained.

8. A multimodal fake news detection system based on coarse-grained and fine-grained feature fusion, characterized by: include: The feature extraction module is configured to perform coarse-grained and fine-grained feature extraction on each modality of the target news to be detected through multiple feature encoders to obtain multi-granularity features; During feature encoder training, the loss is calculated by comparing coarse-grained and fine-grained features. The goal is to construct a loss function for training by bringing the image and text embedding vectors of each real news item closer together in the multimodal vector space and pushing the image and text embedding vectors of different news items further apart. The coarse-grained and fine-grained feature concatenation module is configured to perform average pooling on the obtained multi-granularity features, respectively, and concatenate the pooled fine-grained features with the coarse-grained features of the corresponding modality to obtain the coarse-grained and fine-grained fusion features of each modality; The classification module is configured to transform the obtained coarse-grained features and the coarse-grained and fine-grained fusion features of each modality into features of the same dimension, then splice them together and input them into the fake news detection classifier to obtain the detection result of whether the target news is fake news.

9. An electronic device, characterized in that: The system comprises a memory and a processor, and computer instructions stored in the memory and executed on the processor, wherein when the computer instructions are executed by the processor, the steps of the multimodal fake news detection method based on coarse and fine granularity feature fusion according to any one of claims 1 to 7 are completed.

10. A computer-readable storage medium, characterized in that Used to store computer instructions, which, when executed by a processor, complete the steps of the multimodal fake news detection method based on coarse and fine granularity feature fusion as described in any one of claims 1 to 7.

Images