False news detection method based on multi-modal fusion and comparative learning

Abstract

The invention relates to a false news detection method based on multi-modal fusion and comparative learning, and the method comprises the steps: obtaining news, coding the news into coarse-grained text features and coarse-grained visual features, and extracting fine-grained text features of the news; orderly splicing the coarse-grained text features and the fine-grained text features to obtain multi-grained text features, mapping the multi-grained text features and the coarse-grained visual features to the same feature space, and performing cross-modal alignment to obtain multi-grained multi-modal features; inputting the multi-granularity multi-modal features into a multi-head self-attention mechanism model to obtain final features, and inputting the final features into a classifier to obtain a prediction label; the self-attention mechanism model and the classifier are obtained through training of a training set. According to the method, false news can be effectively detected through multi-modal fusion and comparative learning (FND-MC), and the method is obviously superior to a current most advanced method.

Description

Technical Field

[0001] The present invention relates to the technical field of news detection, and in particular to a fake news detection method based on multimodal fusion and contrastive learning. Background Art

[0002] With the rapid development of Internet technology and social media, it has become more convenient and faster for the public to obtain information. However, the rapid and widespread dissemination of information has also created fertile soil for the spread of fake news. The continuous spread of fake news and misinformation on social media platforms poses a major challenge to maintaining network security. Given the large amount of fake news that continues to pour in every day, it is not enough to rely solely on manual review and expert fact-checking. Therefore, more and more researchers have turned their attention to the automatic detection of fake news. Although various methods have been proposed in recent years to verify and authenticate information on social media, the application of emerging multimodal learning models and contrastive learning in fake news detection has received limited attention:

[0003] Existing fake news detection methods mainly focus on analyzing the text content of news articles. Initially, traditional learning methods, such as decision trees, were used for fake news detection. With the rapid development of deep learning, models such as convolutional neural networks, recurrent neural networks, and graph neural networks were subsequently applied to the field of fake news detection and showed good performance. In addition, some detection methods significantly improved the detection efficiency by deeply analyzing the content of news articles while considering metadata (such as publishing time, user profile, and comments). Although these purely text-based detection methods can effectively identify fake news, it should be noted that in the current Internet, fake news often combines text and images to enhance the credibility and appeal of false information. In order to improve the detection of multimodal fake news, some methods extract text and image features separately and then obtain a multimodal representation by simple splicing, which greatly improves the detection performance of fake news. However, relying solely on concatenation for cross-modal fusion brings two major disadvantages. First, in fake news detection, the number and importance of text and image features are unbalanced, and splicing may not be able to handle them effectively. Second, splicing cannot capture the interactive information between text and image, which limits the model's understanding of cross-modal data. To address these issues, some methods have designed more advanced multimodal representation learning paradigms to extract complex cross-modal features. However, these methods often ignore the alignment relationship between modes and the impact of inter-news relations on fake news detection performance. In addition, emerging pre-trained multimodal learning models, such as CLIP, and contrastive learning, have shown excellent performance in various downstream tasks. Although these advanced techniques help to learn multimodal information more effectively, further research and exploration are needed to systematically incorporate them into the field of fake news detection. Summary of the invention

[0004] In order to solve the problems existing in the above-mentioned prior art, the purpose of the present invention is to propose a fake news detection method based on multimodal fusion and contrastive learning. First, an advanced multimodal learning model is used to extract text and visual features from news articles. Secondly, these features are integrated using text-visual feature fusion and multi-head self-attention mechanism to integrate the global information of the news. Finally, the effectiveness of the present invention is verified by conducting a large number of experiments on two real-world public datasets. The experimental results show that multimodal fusion and contrastive learning (FND-MC) can effectively detect fake news, significantly outperforming the current state-of-the-art methods.

[0005] To achieve the above object, the present invention provides the following solutions:

[0006] A fake news detection method based on multimodal fusion and contrastive learning, comprising:

[0007] Acquire news, encode the news into coarse-grained text features and coarse-grained visual features, and extract fine-grained text features of the news;

[0008] The coarse-grained text features and the fine-grained text features are sequentially spliced ​​to obtain multi-grained text features, the multi-grained text features and the coarse-grained visual features are mapped to the same feature space, and cross-modal alignment is performed to obtain multi-granular multimodal features;

[0009] The multi-granularity multimodal features are input into a multi-head self-attention mechanism model to obtain final features, and the final features are input into a classifier to obtain predicted labels; the self-attention mechanism model and the classifier are both trained using a training set; the training set includes a ReCOvery data set and a gossip cop data set.

[0010] Optionally, encoding the news into coarse-grained text features and coarse-grained visual features respectively, and extracting fine-grained text features of the news includes:

[0011] The CLIP model is used to encode the text information and visual information of the news respectively to obtain the coarse-grained text features and the coarse-grained visual features;

[0012] The BERT model is used to extract fine-grained text features of the text information.

[0013] Optionally, mapping the multi-granularity text feature and the coarse-grained visual feature to the same feature space comprises:

[0014] e t′ =e t W t ;

[0015] ev′ =v c W v ;

[0016] Among them, e t′ represents multi-granularity text features mapped to the same feature space, e v′ represents the coarse-grained visual features mapped to the same feature space, e t represents multi-granularity text features, v c represents the coarse-grained visual features, W t , W v represents learnable parameters.

[0017] Optionally, using the MSE loss function for cross-modal alignment includes:

[0018]

[0019] in, represents the MSE loss function, e t′ represents multi-granularity text features mapped to the same feature space, e v′ represents the coarse-grained visual features mapped to the same feature space, and n represents the number of feature vectors.

[0020] Optionally, obtaining multi-granularity multimodal features includes:

[0021] The multi-granularity text features and the coarse-grained visual features after cross-modal alignment are connected to obtain the multi-granularity multimodal features.

[0022] Optionally, obtaining the final feature includes:

[0023] The multi-granular multimodal features are input into the multi-head self-attention mechanism model, and the query matrix of each attention head is calculated respectively. Key Matrix Sum Matrix obtaining the final feature;

[0024] in, and represents the trainable parameters, e x It is a multi-granularity multi-modal feature.

[0025] Optionally, obtaining the predicted label includes:

[0026] Input the final feature into the pooling layer of the classifier to downsample the final feature, and set the target probability of the dropout layer of the classifier to obtain the preprocessed feature;

[0027] The preprocessed features are input into the sigmoid layer of the classifier to generate the predicted labels.

[0028] Optionally, generating the predicted label includes:

[0029]

[0030] Among them, W C and b c is a learnable parameter, is the predicted label.

[0031] Optionally, the method further comprises:

[0032] The overall goals of setting up the self-attention mechanism model and the classifier include:

[0033]

[0034] Among them, α is the weight coefficient, is the key loss function, is the MSE loss function.

[0035] Optionally, obtaining the key loss function includes:

[0036]

[0037] Among them, y represents the true label of the news, is the predicted label.

[0038] The beneficial effects of the present invention are:

[0039] The present invention firstly uses an advanced multimodal learning model to extract text and visual features from news articles. Secondly, these features are integrated using text-visual feature fusion and multi-head self-attention mechanism to integrate the global information of the news. Finally, the present invention verifies the effectiveness of the present invention by conducting a large number of experiments on two real-world public datasets. The experimental results show that multimodal fusion and contrastive learning (FND-MC) can effectively detect fake news, significantly outperforming the current state-of-the-art methods.

[0040] The present invention proposes a fake news detection method based on multimodal fusion and contrastive learning. The method considers both the relationship between different modalities and the relationship between different news articles, and effectively detects fake news. The present invention also adopts a multi-head self-attention mechanism to obtain more expressive multimodal features of news articles, and conducts extensive experiments on two real-world datasets. The results show that the framework of the present invention is significantly better than the current most advanced system. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0042] Figure 1 This is a flow chart of a fake news detection method based on multimodal fusion and contrastive learning according to an embodiment of the present invention;

[0043] Figure 2 A schematic diagram of true and false news in a data set according to an embodiment of the present invention;

[0044] Figure 3 The training process of the model of the embodiment of the present invention on the ReCOvery dataset;

[0045] Figure 4 The figure shows the training process of the model of the embodiment of the present invention on the gossipcop dataset. DETAILED DESCRIPTION

[0046] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0047] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0048] like Figure 1 As shown in the figure, the task of fake news detection aims to determine whether a given news story is fake or not, and is usually defined as a binary classification problem. Specifically, given a news story x = (t, v), a true label y, where y∈{1,0}, y = 1 indicates fake news, y = 0 indicates true news, t and v represent the text content and image content of the news, respectively. The goal of this invention is to predict the truth or falsity of news by integrating the multimodal features of news. In other words, in To predict the label, F represents the FND-MC model.

[0049] The present embodiment discloses a fake news detection method based on multimodal fusion and contrastive learning, including: obtaining news, encoding the news into coarse-grained text features and coarse-grained visual features, and extracting fine-grained text features of the news; sequentially splicing the coarse-grained text features and the fine-grained text features to obtain multi-granularity text features, mapping the multi-granularity text features and the coarse-grained visual features to the same feature space, and performing cross-modal alignment to obtain multi-granularity multimodal features; inputting the multi-granularity multimodal features into a multi-head self-attention mechanism model to obtain final features, inputting the final features into a classifier to obtain predicted labels; the self-attention mechanism model and the classifier are both trained using a training set; the training set includes a ReCOvery data set and a gossip cop data set.

[0050] Furthermore, the news is encoded into coarse-grained text features and coarse-grained visual features respectively, and the fine-grained text features of the news are extracted, including: using the CLIP model to encode the text information and visual information of the news respectively to obtain coarse-grained text features and coarse-grained visual features; and using the BERT model to extract the fine-grained text features of the text information of the news.

[0051] Furthermore, mapping multi-granularity text features and coarse-grained visual features into the same feature space includes:

[0052] e t′ =e t W t ;

[0053] e v′ =v c W v ;

[0054] Among them, e t′ represents multi-granularity text features mapped to the same feature space, e v′ represents the coarse-grained visual features mapped to the same feature space, e t represents multi-granularity text features, v c represents the coarse-grained visual features, W t , W v represents learnable parameters.

[0055] Furthermore, using the MSE loss function for cross-modal alignment includes:

[0056]

[0057] in, represents the MSE loss function, e t′ represents multi-granularity text features mapped to the same feature space, e v′ represents the coarse-grained visual features mapped to the same feature space, n represents.

[0058] Specifically:

[0059] Multimodal feature extraction:

[0060] The pre-trained multimodal learning model CLIP can effectively understand the semantic relationship between text and image and encode them into the same feature space. Therefore, the present invention adopts CLIP as a coarse-grained multimodal feature encoder. Given a news x = (t, v), CLIP encodes the text information t and the visual information v into coarse-grained text features t and v respectively. c and coarse-grained visual features v c Among them, t c , Among them, d c is the dimension of the coarse-grained multimodal feature space. For the text information t, the present invention further uses BERT to extract fine-grained text features Among them, d b Represents the length of fine-grained text features.

[0061] Multimodal feature fusion:

[0062] Cross-modal alignment. In order to improve the representation learned in each modality, this paper introduces cross-modal alignment to establish meaningful connections between different modalities. First, the paper sequentially concatenates text features of different granularities to generate multi-granularity text features. t , as shown in Formula 1.

[0063]

[0064] in, Indicates a connection operation. Represents the feature space dimension.

[0065] Then, the multi-granularity text features e t and visual features v c Mapped to the same feature space:

[0066]

[0067] Among them, W t and W v is a learnable parameter. Then, the present invention uses the MSE loss function for cross-modal alignment:

[0068]

[0069] Obtaining aligned fine-grained visual features and multi-granular text features Then, by connecting and To generate multi-granular multimodal features of the input news x

[0070] Furthermore, obtaining multi-granular multimodal features includes:

[0071] The multi-granularity text features and coarse-grained visual features after cross-modal alignment are connected to obtain multi-granularity multimodal features.

[0072] Furthermore, obtaining the final features includes:

[0073] Input multi-granular multimodal features into the multi-head self-attention mechanism model and calculate the query matrix of each attention head separately Key Matrix Sum Matrix Get the final features;

[0074] in, and represents the trainable parameters, e x It is a multi-granularity multi-modal feature.

[0075] Furthermore, the formula for obtaining the final features is:

[0076]

[0077] e=Concat(head1,...,head n );

[0078] Among them, n represents the number of attention heads, Concat represents the concatenation operation, e represents the final of the news, and T represents.

[0079] Specifically:

[0080] Multi-head self-attention mechanism:

[0081] In this process, for multi-granular multimodal features e x , calculate each attention head separately i The query matrix Key Matrix Sum Matrix in, and is a trainable parameter. The final features of the news can be expressed as:

[0082]

[0083] Where n is the number of attention heads, Concat is the concatenation operation, and e is the final representation of the news.

[0084] Furthermore, obtaining the predicted label includes:

[0085] Input the final features into the pooling layer of the classifier to downsample the final features, and set the target probability of the dropout layer of the classifier to obtain the preprocessed features;

[0086] The preprocessed features are input into the sigmoid layer of the classifier to generate the predicted labels.

[0087] Furthermore, generating a predicted label includes:

[0088]

[0089] Among them, W C and b c is a learnable parameter, is the predicted label.

[0090] Furthermore, the method further comprises:

[0091] The overall goals of setting up the self-attention model and classifier include:

[0092]

[0093] Among them, α is the weight coefficient, is the key loss function, is the MSE loss function.

[0094] Furthermore, obtaining the key loss functions includes:

[0095]

[0096] Among them, y represents the true label of the news, is the predicted label.

[0097] Specifically:

[0098] Classifier:

[0099] In order to improve the performance and efficiency of the fake news detection framework proposed in this paper, two pooling layers are used to downsample the multimodal representation e of the news. Subsequently, a dropout layer is set with a probability of 0.5 to reduce the complex co-adaptation between neurons and prevent overfitting. After the preprocessing layer, the obtained news representation e is d Feed it into a sigmoid layer to generate the predicted label:

[0100]

[0101] Among them, W C and b c is a learnable parameter, is the predicted label. The cross entropy loss can be expressed as:

[0102]

[0103] Among them, y represents the true label of the news, c represents the category of the label, including 0 or 1, As a prediction label, it is a value between [0-1], which means that the news vector e d The probability of predicting it as fake news. Then, The model is fed into the cross entropy loss function for training, and the real label y of the news is used as a reference. During the training process, the model will make the cross entropy loss smaller and smaller, that is, the model's predicted label for the news vector is closer and closer to the real label of the news. When the training is completed, the model has the ability to distinguish true from false news.

[0104] The overall goal of the model framework of the present invention is defined as follows:

[0105]

[0106] Where α is a weight coefficient, and its value range is [0,1]. The present invention proposes a strategy to simplify the model training process. Since the goal of the framework is to accurately classify true and false news, the present invention uses the key loss function The weight of is fixed to 1, and the weight α is optimized. This strategy effectively improves the efficiency of hyperparameter search and reduces the computing resources required for model optimization.

[0107] experiment:

[0108] Datasets: We conducted extensive experiments on two publicly available datasets to evaluate the performance of the proposed framework, namely ReCOvery and gossipcop. The ReCOvery dataset collects news articles related to COVID-19 published on 22 reliable and 38 unreliable websites from January to May 2020. The dataset includes text content, images, titles, publication dates, author information, etc. The gossipcop dataset comes from the FakeNewsNet repository, which collects a large number of articles about celebrity gossip and entertainment news on Twitter. This dataset is widely used in the field of fake news detection, and it includes features such as text, images, user profiles, comments, and forwarding relationships. Detailed statistical information of the datasets used in the experiment is shown in Figure 2 shown.

[0109] The dataset is divided into training set, validation set and test set in a ratio of 8:1:1. First, the news samples in the training set are input into the model for training. During this process, the model will evaluate its error on the training set by calculating the training loss, and minimize its classification error as much as possible through the loss function. At the same time, the model will also calculate the validation loss on the validation set to verify the performance of the model. The validation loss is intended to tell researchers whether the model parameters need to be adjusted. The training loss is measured after each batch, and each batch is 32 samples. The validation loss is measured after each epoch, for a total of 100 epochs. As the training progresses, the training loss and validation loss steadily decrease and stabilize at a specific point, that is, the training loss curve and the validation loss curve achieve a good fit. Figure 3 Represents the training process of the model on the ReCOvery dataset. Figure 4 Represents the training process of the model on the gossipcop dataset.

[0110] Experimental details: In the experiment, for data input, the length of the input text sequence is set to 256, and the size of the input image is set to 224×224. For news items with multiple images, the present invention randomly selects one image. In the experiment of pre-training BERT, the BERT-base-case model is used to encode English data, and the BERT-base-Chinese model is used to encode Chinese data. The output dimension of BERT is set to 768. The pre-trained CLIP model used is "ViT-B / 32", d c =500. Due to the lack of pre-trained Chinese text models for CLIP, the present invention uses public API1 to translate Chinese text into English. Since the maximum token limit of CLIP text input is set to 77 tokens, the experiment uses a summary generation model to generate summary sentences for texts with more than 50 tokens, which are then used as input for CLIP. The number of positive faces of the self-attention mechanism is set to 8. To ensure a fair comparison, the model parameters of Bert and CLIP are frozen throughout the experiment. The batch size is set to 32, the learning rate is 5×10-6, and the model is trained for 100 epochs using the Adam optimizer. An early stopping strategy is used during training to avoid overfitting.

[0111] Baseline Models: The proposed method is benchmarked against the following state-of-the-art multimodal fake news detection methods to verify its superiority.

[0112] (1) SpotFake: The text and visual features extracted by BERT and VGG-19 are concatenated to generate a multimodal representation, which is then classified using a fully connected layer.

[0113] (2) SAFE: It uses convolutional neural networks (CNNs) to extract text and image features and exploits the similarity relationship between the two patterns for fake news detection.

[0114] (3) BTIC: We use BERT to concatenate features extracted from text with image patches using ResNet to generate multimodal news representations, and introduce contrastive learning to enhance model performance.

[0115] (4) : We use distribution differences to quantify the ambiguity between different unimodal features and enhance fake news detection performance through adaptive aggregation and cross-modal correlation of unimodal features.

[0116] (5) COOLANT: Introduces an attention-guided module to adaptively weight unimodal and cross-modal features to generate more coherent multimodal news representations.

[0117] (6) TTEC: Improving fake news detection performance through data augmentation using a back-translation strategy.

[0118] (7)MMFN: It uses the features encoded from text and images in the CLIP pre-trained model to evaluate cross-modal relevance, effectively solving the challenge of fake news detection in cross-modal ambiguous scenarios.

[0119] Results and Analysis: Table 1 gives a detailed overview of the performance of FND-MC and all baseline methods on the ReCOvery and gossip datasets. The experimental results show that FMD-MC outperforms all baselines in terms of Mac.F1, reflecting the overall performance of the model, thus verifying the effectiveness of the proposed framework. It is worth noting that TTEC is the best performing model on the ReCOvery dataset, and it shows more significant performance improvement than BTIC by extracting the complete structure of the image as visual features. The proposed method FND-MC shows an increase of 1.69% and 1.8% in Mac.F1 on the ReCOvery and gossip datasets, respectively, compared with TTEC. The proposed method not only captures richer multimodal information, but also better understands the cross-modal relationship between text and visual modalities, which helps to detect fake news more effectively. CAFE effectively resolves the ambiguity between text and visual information by calculating the KL divergence between text and visual vectors, thereby achieving excellent fake news detection performance. Compared with CAFE, FND-MC enhances the final news representation by integrating text vectors and visual vectors, thereby achieving a more semantically meaningful final feature representation of news. MMFN also uses an emerging multimodal learning model for fake news detection, achieving good results on two datasets. Compared with MMFN, FND-MC fully considers the cross-modal alignment between news text and visual vectors, thereby improving the fake news detection performance. The multi-head self-attention mechanism allows the method of the present invention to pay more attention to the key information in the final news representation, enabling it to learn a more generalized feature representation. This enhances the adaptability and discriminability of the method when dealing with implicit or new types of fake news.

[0120] Table 1

[0121]

[0122]

[0123] Ablation experiment: This paper verifies the structural rationality of the proposed method at the module level. Ablation experiments usually focus on the overall performance of each component within the method, rather than the changes in a single category. To this end, this paper uses Mac.F1-score, Mac.Precision, and Mac.Recall as evaluation indicators to mitigate any evaluation bias that may arise from the imbalanced distribution of true and false news in the dataset.

[0124] At the module level, the present invention designs two variants:

[0125] FC: This variant removes the cross-modal alignment module and directly concatenates textual and visual features of different granularities, which are then fed into the multi-head self-attention mechanism.

[0126] FM: This variant removes the joint learning of the multi-head self-attention mechanism and directly concatenates the cross-modal aligned text and visual features to form the final news representation for classification.

[0127] Table 2 gives the ablation study results of FND-MC at the module level. We observe that the overall performance of both FC and FM variants has declined compared to the original FMC framework, indicating that the cross-modal alignment module and the multi-head self-attention mechanism play a key role in improving the performance of multimodal fake news detection. In addition, FC performs better than FM, which shows that the multi-head self-attention mechanism is crucial for capturing key information in news features and enhancing the differentiation of the final news representation. Based on the results of the module-level ablation study, we conclude that the combination of the multi-granularity feature fusion module and the joint learning module effectively enhances the performance of multimodal fake news detection.

[0128] Table 2

[0129]

[0130]

[0131] This paper proposes a fake news detection method based on multimodal fusion and contrastive learning (FND-MC). This method emphasizes the potential of leveraging multimodal information of different granularities in fake news detection, an aspect often overlooked by most existing methods. In FND-MC, the application of multi-head attention mechanism plays a crucial role in capturing the complex dependencies in news content, thereby enhancing the representation ability of news features.

[0132] Extensive experimental results on publicly available real-world datasets not only demonstrate the superiority of FMC, but also show the effectiveness of multi-granularity feature fusion in the task of fake news detection. Fake news detection models may raise ethical issues and social impacts related to privacy protection, freedom of speech, and bias. The present invention conducts fairness tests in different datasets and environments to ensure consistent performance for all groups. However, there are several other issues that need to be addressed. First, if the training data contains biases, the model may reflect these biases, leading to discrimination against certain groups. Therefore, it is crucial to identify and eliminate biases in the data. Second, care must be taken to ensure that the implementation of fake news detection does not suppress legitimate freedom of speech and that the model does not inadvertently block or restrict legitimate news content.

[0133] Finally, fake news detection models may produce false positives, which may unfairly affect the reputation of individuals or organizations, highlighting the need to evaluate the accuracy and risk of the results. In this study, many aspects of the model deserve further investigation. First, the framework architecture and learning strategy can be further optimized to improve the adaptability to different news types and formats. Second, exploring a wider variety of pre-trained models can enhance the framework's ability to handle multimodal data. Finally, achieving a balance between performance and computational efficiency is critical to enabling the present invention to effectively monitor and identify fake news in real time. In future work, the present invention will continue to explore and address these issues.

[0134] The embodiments described above are only descriptions of the preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary technicians in this field should all fall within the protection scope determined by the claims of the present invention.

Claims

1. A fake news detection method based on multimodal fusion and contrastive learning, characterized in that: include: Acquire news, encode the news into coarse-grained text features and coarse-grained visual features, and extract fine-grained text features of the news; The coarse-grained text features and the fine-grained text features are sequentially spliced ​​to obtain multi-grained text features, the multi-grained text features and the coarse-grained visual features are mapped to the same feature space, and cross-modal alignment is performed to obtain multi-granular multimodal features; The multi-granularity multimodal features are input into a multi-head self-attention mechanism model to obtain final features, and the final features are input into a classifier to obtain predicted labels; the self-attention mechanism model and the classifier are both trained using a training set; the training set includes a ReCOvery data set and a gossip cop data set.

2. The fake news detection method based on multimodal fusion and contrastive learning according to claim 1, characterized in that: Encoding the news into coarse-grained text features and coarse-grained visual features, and extracting fine-grained text features of the news includes: The CLIP model is used to encode the text information and visual information of the news respectively to obtain the coarse-grained text features and the coarse-grained visual features; The BERT model is used to extract fine-grained text features of the text information.

3. The fake news detection method based on multimodal fusion and contrastive learning according to claim 1, characterized in that: Mapping the multi-granularity text features and the coarse-grained visual features to the same feature space includes: have been t′ =e t W t ; e v′ 9. v c W v 100. Among them, e t′ represents multi-granularity text features mapped to the same feature space, e v′ represents the coarse-grained visual features mapped to the same feature space, e t represents multi-granularity text features, v c represents the coarse-grained visual features, W t , W v represents learnable parameters.

4. The fake news detection method based on multimodal fusion and contrastive learning according to claim 1, characterized in that: Using the MSE loss function for cross-modal alignment includes: in, represents the MSE loss function, e t′ represents multi-granularity text features mapped to the same feature space, e v′ represents the coarse-grained visual features mapped to the same feature space, and n represents the number of feature vectors.

5. The fake news detection method based on multimodal fusion and contrastive learning according to claim 1, characterized in that: Acquiring multi-granular multimodal features includes: The multi-granularity text features and the coarse-grained visual features after cross-modal alignment are connected to obtain the multi-granularity multimodal features.

6. The fake news detection method based on multimodal fusion and contrastive learning according to claim 1, characterized in that: Obtaining the final features includes: The multi-granular multimodal features are input into the multi-head self-attention mechanism model, and the query matrix of each attention head is calculated respectively. Key Matrix Sum value matrix V i e =e x W i V , obtain the final feature; Among them, W i Q , W i K and W i V represents the trainable parameters, e x It is a multi-granularity multi-modal feature.

7. The fake news detection method based on multimodal fusion and contrastive learning according to claim 1, characterized in that: Obtaining the predicted label includes: Input the final feature into the pooling layer of the classifier to downsample the final feature, and set the target probability of the dropout layer of the classifier to obtain the preprocessed feature; The preprocessed features are input into the sigmoid layer of the classifier to generate the predicted labels.

8. The fake news detection method based on multimodal fusion and contrastive learning according to claim 7, characterized in that: Generating the predicted label includes: Among them, W C and b c is a learnable parameter, is the predicted label.

9. The fake news detection method based on multimodal fusion and contrastive learning according to claim 1, characterized in that: The method further includes: The overall goals of setting up the self-attention mechanism model and the classifier include: Among them, α is the weight coefficient, is the key loss function, is the MSE loss function.

10. The fake news detection method based on multimodal fusion and contrastive learning according to claim 9, characterized in that: Obtaining the key loss function includes: Among them, y represents the true label of the news, is the predicted label.

Images