Multi-modal false news detection method based on multi-level feature fusion and comparative learning

Abstract

The invention discloses a multi-modal false news detection method based on multi-level feature fusion and comparative learning. The method comprises the following steps: acquiring text information and image information; respectively extracting multi-modal features of the text information and the image information, and obtaining text multi-modal features and visual multi-modal features; performing comparative learning on the text multi-modal features and the visual multi-modal features to obtain text embedding and visual embedding; fusing the text embedding and the visual embedding to obtain a multi-modal fusion feature; and inputting the multi-modal features into a classifier to obtain a classification result. According to the method, the problem of modal information splitting in a traditional method is relieved.

Description

Technical Field

[0001] The present invention belongs to the technical field of fake news detection, and in particular relates to a multimodal fake news detection method based on multi-level feature fusion and contrastive learning. Background Art

[0002] Social media and online platforms in the digital age have revolutionized the ubiquitous dissemination of information, with its speed and reach increasing exponentially. However, the open nature of this information ecosystem has also provided a technological breeding ground for the growth and spread of fake news. As a new cybersecurity threat, fake news is defined as misleading information that distorts objective facts through systematically fabricated multimodal content such as text and images. This has prompted academics to prioritize automated fake news detection technology, with research continuing to deepen through the intersection of computer vision, natural language processing, and multimodal learning.

[0003] Early research on fake news detection mainly adopted a unimodal analysis paradigm, that is, verifying the authenticity of news by analyzing text or image content separately. In the text dimension, research methods are generally divided into two categories: one focuses on mining potential clues at the language level such as vocabulary selection, syntactic features, and sentiment polarity; the other combines metadata such as publisher background, user comment features, and dissemination time series to assist in judgment. Detection in the visual dimension focuses on the extraction of underlying image attributes and high-level semantic features. Although such methods have certain effectiveness in unimodal counterfeiting scenarios, they expose obvious flaws when faced with new types of fake news that are collaboratively forged by multiple modalities. Because counterfeiters often use cross-modal consistency to enhance deception, unimodal analysis methods have difficulty identifying semantic conflicts between text and visual content. Although subsequent research has shifted toward multimodal fusion and achieved significant results, existing frameworks still have significant shortcomings. First, most of these methods only extract single textual and visual features for fusion, neglecting to fully utilize the multimodal information of news from the perspective of features at different levels. Second, the transfer learning potential of pre-trained multimodal large models and the cross-modal alignment advantages of contrastive learning are underexploited, resulting in semantic bias and information redundancy in multimodal representations. In fake news detection, extracting textual and visual features with rich representational meaning is crucial to improving fake news detection performance. Summary of the Invention

[0004] To solve the above technical problems, the present invention proposes a multimodal fake news detection method based on multi-level feature fusion and contrastive learning, which can effectively improve the fake news detection performance.

[0005] To achieve the above objectives, the present invention provides a multimodal fake news detection method based on multi-level feature fusion and contrastive learning, comprising:

[0006] Get text information and image information;

[0007] Extracting multimodal features of the text information and image information respectively to obtain text multimodal features and visual multimodal features;

[0008] Performing comparative learning on the text multimodal features and the visual multimodal features to obtain text embedding and visual embedding;

[0009] Fusing the text embedding and the visual embedding to obtain multimodal fusion features;

[0010] The multimodal features are input into a classifier to obtain a classification result.

[0011] Optionally, extracting multimodal features of the text information and the image information respectively, and obtaining text multimodal features and visual multimodal features includes:

[0012] Extracting first-level multimodal features of the text information and image information respectively to obtain first-level text features and first-level visual features;

[0013] Extracting second-level multimodal features of the text information and the image information respectively to obtain second-level text features and second-level visual features;

[0014] Performing a cascade operation on the first-level text features and the second-level text features to obtain text multimodal features;

[0015] A cascade operation is performed on the first-level visual features and the second-level visual features to obtain visual multimodal features.

[0016] Optionally, obtaining the first-level text features and the first-level visual features includes:

[0017] An ALIGN encoder is used to extract first-level multimodal features of the text information and image information to obtain the first-level text features and the first-level visual features.

[0018] Optionally, obtaining the second-level text features and the second-level visual features includes:

[0019] Using the pre-trained model XLnet to perform second-level encoding on the text information to obtain second-level text features;

[0020] The pre-trained model SWIN-T is used to perform second-level encoding on the image information to obtain second-level visual features.

[0021] Optionally, performing comparative learning on the text multimodal features and the visual multimodal features to obtain text embedding and visual embedding includes:

[0022] Get posts containing text-image pairs and tag information;

[0023] Calculating text-to-visual similarity scores and visual-to-text similarity scores in the post;

[0024] Calculating the cross entropy loss of the textual multimodal features and the visual multimodal features respectively based on the similarity scores;

[0025] Obtaining contrastive learning loss based on the cross entropy loss;

[0026] By using the contrastive learning loss, similar text embeddings and visual embeddings are brought closer together in the feature space, while dissimilar text embeddings and visual embeddings are further apart in the feature space.

[0027] By continuously optimizing the contrastive learning loss, we obtain deeply aligned text embedding and visual embedding.

[0028] Optionally, calculate the text-to-visual similarity score by:

[0029]

[0030] in, is the text-to-visual similarity score, τ is a learnable temperature parameter, is a pair of text embedding and visual embedding, j is the counting symbol of the number of pictures, which can be 1, 2, 3...N, where N means there are N pictures;

[0031] Calculating the visual-to-text similarity score involves:

[0032]

[0033] in, is the visual-to-text similarity score, is a pair of visual embedding and text embedding.

[0034] Optionally, fusing the text embedding and the visual embedding to obtain a multimodal fusion feature includes:

[0035] Calculating query matrices, key matrices, and value matrices for the text embedding and visual embedding, respectively;

[0036] Based on the query matrix, the key matrix and the value matrix, obtaining multi-level text features and multi-level visual features;

[0037] The multi-level text features and the multi-level visual features are fused to obtain the multimodal fusion features.

[0038] Optionally, obtaining multi-level text features based on the query matrix, key matrix, and value matrix includes:

[0039]

[0040] Among them, e tv' is a multi-level text feature, Q v is the text embedding query matrix, is the transpose of the text embedding key matrix, V t is the text embedding value matrix, d and H represent the dimension size and the number of attention heads respectively, is the output linear transformation matrix;

[0041] Based on the query matrix, the key matrix, and the value matrix, obtaining multi-level visual features includes:

[0042]

[0043] Among them, e vt′ is a multi-level visual feature, Q t is the visual embedding query matrix, is the transpose of the visual embedding key matrix, V v is the visual embedding value matrix, is the output linear transformation matrix.

[0044] Compared with the prior art, the present invention has the following advantages and technical effects:

[0045] This paper employs a multi-level feature extraction strategy combining ALIGN, XLNET, and SWIN-T. This strategy preserves the cross-modal associations of low-level semantic alignment while capturing contextual sensitivity and visual hierarchical features, enhancing the completeness of feature representation. By integrating cross-modal alignment constraints with a text-visual co-attention fusion mechanism, it enables dynamic interaction and complementary information mining among multi-level features, alleviating the problem of modal information fragmentation in traditional methods. This paper provides a new technical approach for multimodal fake news detection. Future work could further explore strategies for the fusion of temporal dynamic features and consider the embedding of external information, such as social media dissemination paths, to address the challenges of more complex real-world scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The accompanying drawings, which constitute part of this application, are intended to provide a further understanding of this application. The exemplary embodiments and descriptions of this application are intended to explain this application and do not constitute an improper limitation on this application. In the accompanying drawings:

[0047] Figure 1 This is a flow chart of a multimodal fake news detection method based on multi-level feature fusion and contrastive learning in an embodiment of the present invention. DETAILED DESCRIPTION

[0048] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in this application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0049] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0050] This embodiment proposes a multimodal fake news detection method based on multi-level feature fusion and contrastive learning. Figure 1 As shown, the specific steps include:

[0051] Get text information and image information;

[0052] Extracting multimodal features of text information and image information respectively to obtain text multimodal features and visual multimodal features;

[0053] Perform comparative learning on textual multimodal features and visual multimodal features to obtain text embedding and visual embedding;

[0054] Fuse text embedding and visual embedding to obtain multimodal fusion features;

[0055] Input the multimodal features into the classifier to obtain the classification results.

[0056] Specifically, this embodiment innovatively constructs a multi-level feature extraction module, which captures high-level (second-level) and low-level (first-level) features of text and images by integrating multiple pre-trained models; the core goal of single-modal models such as XLNet (text) and SWIN-T (image) is to deeply explore the details and semantics within the modality. For example, XLNet captures long-distance dependencies and complex semantic relationships through permutation language modeling, and its high-level features include abstract information such as grammatical structure and logical association. The hierarchical window attention mechanism of SWIN-T gradually aggregates local to global visual patterns, and high-level features may include high-level visual concepts such as object component combinations and scene topology. Therefore, the text and visual embeddings encoded by XLNet and SWIN-T are called high-level features. The goal of ALIGN is to map graphic features to a unified low-dimensional space through contrastive learning, so it pays more attention to basic cross-modal correlations (such as color, shape, simple action descriptions) rather than deep abstraction of a single modality. Therefore, the text and visual embeddings encoded by ALIGN are called low-level features. A cross-modal comparative learning mechanism is designed to strengthen the deep alignment of the semantic space of images and texts; and a common attention network with dynamic weight allocation is introduced to realize the hierarchical fusion of multi-level features, and finally complete the authenticity judgment of news through joint representation.

[0057] More specifically, Figure 1 As shown, the proposed framework consists of three major components: multimodal feature extraction, multi-level feature fusion, and joint learning. In the multimodal feature extraction module, this embodiment uses the pre-trained ALIGN model to extract semantically aligned low-level text and visual features, while simultaneously using the pre-trained XLNET and SWIN-T models to extract context-aware text features and hierarchical visual features. By employing this feature extraction approach that combines multiple pre-trained models, FAD not only captures more comprehensive news features but also enhances its generalization capabilities across various news types.

[0058] The multi-level feature fusion module first generates multi-level text features and multi-level visual features by sequentially concatenating text and visual features from different levels. Cross-modal alignment is then used to encourage the framework to learn the consistency of multi-level text and visual features in a common feature space, thereby establishing meaningful connections between them. Finally, the aligned multi-level text and visual features are fed into the text-visual co-attention fusion to generate a multi-level multimodal news representation. Text-visual co-attention focuses on the complementary and important aspects of the two modal features, achieving a balance between delicate processing and global understanding of news text and visual information, thereby improving classification performance.

[0059] Furthermore, the multimodal features of text information and image information are extracted respectively, and obtaining text multimodal features and visual multimodal features includes:

[0060] Extracting first-level multimodal features of text information and image information respectively to obtain first-level text features and first-level visual features;

[0061] Extracting second-level multimodal features of text information and image information respectively to obtain second-level text features and second-level visual features;

[0062] Perform a cascade operation on the first-level text features and the second-level text features to obtain text multimodal features;

[0063] The first-level visual features and the second-level visual features are cascaded to obtain visual multimodal features.

[0064] Furthermore, obtaining the first-level text features and the first-level visual features includes:

[0065] The ALIGN encoder is used to extract the first-level multimodal features of text information and image information to obtain the first-level text features and the first-level visual features.

[0066] Furthermore, obtaining the second-level text features and the second-level visual features includes:

[0067] Use the pre-trained model XLnet to perform second-level encoding on text information and obtain second-level text features;

[0068] The pre-trained model SWIN-T is used to perform second-level encoding on the image information to obtain the second-level visual features.

[0069] Specifically, low-level multimodal feature extraction. The pre-trained multimodal learning model ALIGN can effectively understand the semantic relationship between text and image, and can encode text and image into the same feature space. Therefore, this embodiment uses ALIGN as a low-level multimodal feature encoder. Given news x = (t, v), ALIGN encodes text information t and visual information v into low-level text features t c and low-level visual features υ c . Where t c , d c is the dimension of the low-level multimodal feature space.

[0070] High-level multimodal feature extraction. This embodiment uses the pre-trained models XLNET and SWIN-T, which are widely used in multimodal fake news detection, as high-level multimodal feature encoders. For example, for a given news item x = (t, v), XLNET encodes the text information t into high-level text features where d b Represents the length of high-level text features. Accordingly, SWIN-T encodes the visual information v into high-level image features where d r Dimensions representing high-level visual features.

[0071] Furthermore, comparative learning of textual multimodal features and visual multimodal features is performed to obtain text embedding and visual embedding, including:

[0072] Get posts containing text-image pairs and tag information;

[0073] Compute text-to-visual and visual-to-text similarity scores for posts;

[0074] Based on the similarity scores, the cross entropy loss of text multimodal features and visual multimodal features is calculated separately;

[0075] According to the cross entropy loss, obtain the contrastive learning loss;

[0076] Through contrastive learning loss, similar text embeddings and visual embeddings are brought closer in the feature space, while dissimilar text embeddings and visual embeddings are further apart in the feature space.

[0077] By continuously optimizing the contrastive learning loss, we obtain deeply aligned text embedding and visual embedding.

[0078] Specifically, multimodal feature processing:

[0079] Feature cascade: This work aims to fuse unimodal features at different levels to generate multi-level multimodal news representation. For example, for the input news x, in order to enhance the information expression capability of unimodal features, this embodiment sequentially cascades text and visual features at different levels to generate multi-level text features e t and multi-level visual features e v :

[0080]

[0081] in, Indicates cascade operation, and

[0082] Cross-modal alignment:

[0083] For a batch of N posts P for training N , where each post P i ∈P N Contains text-image pairs and label information, represented as (i represents the i-th post, y i is the label of the post), the multi-level multimodal news feature input by the cross-modal alignment module is represented as The cross-modal alignment module predicts posts P separately through contrastive learning. i Text-to-visual similarity in and visual-to-text similarity In-foNCE is used as the contrast loss. The similarity score is measured by calculating the dot product using formula (2) and

[0084]

[0085] In formula (2), τ is a learnable temperature parameter. Next, use formula (3) to calculate the cross entropy loss L t→v and L v→t :

[0086]

[0087] In formula (3) and Represents the vector corresponding to the true label using one-hot encoding. Specifically, and Represents positive sample pairs, and the number of positive sample pairs is N. and Represents negative sample pairs, the number of negative sample pairs is N 2 -N. The contrastive learning loss is calculated as follows:

[0088]

[0089] So get the aligned text embedding e t' and deep visual embeddings v' .

[0090] Furthermore, text embedding and visual embedding are fused to obtain multimodal fusion features including:

[0091] Compute the query matrix, key matrix, and value matrix of text embedding and visual embedding respectively;

[0092] Based on the query matrix, key matrix and value matrix, multi-level text features and multi-level visual features are obtained;

[0093] Multi-level text features and multi-level visual features are fused to obtain multimodal fusion features.

[0094] Specifically, the joint attention mechanism:

[0095] For aligned multi-level text features e t′ , using a method similar to the multi-head self-attention mechanism to calculate its query matrix, key matrix and value matrix respectively: and in is the linear transformation matrix, d and H represent the dimension size and the number of attention heads respectively. This embodiment uses the same method to obtain the aligned multi-level visual features e v′ The query matrix, key matrix and value matrix of Q v ,K v and V v Formula (4) is used to calculate the multi-level text features e that enhance visual information tv′ and multi-level visual features that enhance text information vt′ :

[0096]

[0097] in is the output linear transformation matrix. Then e tv′ and e vt′ Perform cascade operations to generate multi-level multimodal features e of the input news x x :

[0098]

[0099] Here Represents a cascade operation. The multi-level multimodal features generated by the multi-level feature fusion module will serve as the input of the joint learning module.

[0100] The present embodiment is described in detail below:

[0101] 1. Dataset:

[0102] Extensive experiments are conducted on two real-world public datasets to evaluate the performance of the proposed framework in this embodiment, namely ReCOvery and MR 2 .

[0103] ReCOvery collected news articles about COVID-19 published between January and May 2020 on 22 reliable and 38 unreliable websites. The dataset contains information such as the news content, images, titles, publication dates, and authors. Similar to the baseline method, this example removed duplicate news and plain text news during the data cleaning process. 2 Including MR 2 -E English dataset and MR 2 -C Chinese dataset. MR 2 -E English dataset and MR 2 The -C Chinese dataset is constructed by collecting news posts from two of the most popular social platforms, Twitter and Weibo. 2 In addition to text, images, and release dates, it also includes comments, forwarding relationships, likes, user names, geographic locations, personal tags, and other social metadata information. 2 During the construction phase, posts with similar content were removed, and duplicate and low-resolution images were removed. This dataset is a high-quality multimodal dataset for training and evaluating fake news detection models. Detailed statistics of the dataset used in the experiments are shown in Table 1.

[0104] Table 1

[0105] Dataset Real News Fake News ReCOvery 1297 562 <![CDATA[MR 2 -E]]> 2318 1418 <![CDATA[MR 2 -C]]> 2609 1754

[0106] 2 Experimental details:

[0107] Evaluation indicators:

[0108] This example uses the four most commonly used evaluation indicators in the field of fake news detection, namely accuracy, precision, recall, and F1-score.

[0109]

[0110] TP (True Positive) indicates that a positive example is actually predicted. TN (True Negative) indicates that a negative example is actually predicted. FP (False Positive) indicates that a negative example is actually predicted as a positive. FN (False Negative) indicates that a positive example is actually predicted as a negative.

[0111] Accuracy (Acc) indicates the proportion of correctly predicted samples to the total number of samples, thus providing an intuitive overview of performance. Precision (Pre) indicates the proportion of samples predicted to be positive that are actually positive. In fake news detection, the higher the precision, the smaller the FP, which means the smaller the probability that true news (actually negative) is misclassified as fake news (predicted to be positive). Recall (Rec) indicates the proportion of the actual number of positive samples in the predicted positive samples to the positive samples in the total number of samples. The larger the recall, the smaller the FN, which means the smaller the probability that fake news is predicted to be true news. F1-score, as a comprehensive indicator, reflects the comprehensive performance and robustness of the model. In the experiment, this embodiment takes the average value of five independent repeated experiments as the result of each indicator.

[0112] Experimental Environment,The fake news detection framework proposed in this embodiment is implemented in the,experimental environment shown in Table 2.

[0113] Table 2

[0114]

[0115]

[0116] Parameter setting,In the experiment, for data input, the length of the input text sequence is set to 256, and the input image size is set to 289×289. For news posts containing multiple images at the same time, this embodiment will randomly select an image. 2 -C, the experiment uses Chinese-XLnet to encode Chinese data, and the XLNET output dimension is 768. The output dimension of the pre-trained SWIN-T is set to 2048. In the pre-trained ALIGN model, the ALIGN visual encoder is EfficientNet-L2 and the text encoder is BERT-Large. Since the ALIGN model currently has no pre-trained Chinese text model, this example uses the public API 1 Translate Chinese text into English. Since the upper limit of ALIGN text input is set to 64 tokens, the experiment uses a summary generation model to generate summary sentences as ALIGN input for texts longer than 20. The number of heads of the self-attention mechanism is set to 8. To ensure a fair comparison, the model parameters of XLnet, SWIN-T, and ALIGN are frozen throughout the experiment. Throughout the experiment, the batch size is 32, the learning rate is 4×10-6, and the model is trained using the Adam optimizer for 100 epochs. The early stopping strategy is used during training to avoid overfitting. The dataset is split into training, validation, and test sets in a ratio of 8:1:1. The two key loss weights are set to α=0.8 and β=0.2.

[0117] 3. Baseline Model:

[0118] This example uses the following state-of-the-art multimodal fake news detection methods as comparison methods to verify the superiority of the proposed framework:

[0119] (1) SpotFake: The text and visual features extracted by BERT and VGG-19 are concatenated to generate a multimodal representation, and then classified using a fully connected layer.

[0120] (2) SAFE: Use convolutional neural networks (CNN) to extract text and image features and exploit the similarity relationship between the two types of modalities for fake news detection.

[0121] (3) BTIC: BERT and ResNet are used to extract features of text and image blocks respectively and concatenate them to generate multimodal representation of news, and contrastive learning is introduced to improve model performance.

[0122] (4) : We use the distribution divergence between different unimodal features to quantify inter-modal ambiguity and improve fake news detection performance by adaptively aggregating unimodal features and cross-modal associations.

[0123] (5) COOLANT: An attention guidance module is designed to guide the adaptive weighting of unimodal features and cross-modal features to generate more reasonable multimodal news representations.

[0124] (6) TTEC: Using back-translation strategy for data augmentation in fake news detection to improve detection performance.

[0125] (7)MMFN: Using text and image features encoded by the CLIP pre-trained model to evaluate cross-modal relevance can effectively solve the problem of fake news detection in cross-modal ambiguous scenarios.

[0126] 4. Performance comparison:

[0127] Table 3-4 details the performance of FAD compared with all baseline methods in ReCOvery, GossipCop, MR 2 -E and MR 2 -Fake news performance on the C dataset. Experimental results show that the comprehensive performance of FAD is significantly better than the current state-of-the-art fake news detection methods, which proves the superiority of the framework proposed in this example.

[0128] It can be observed that TTEC is the best model on the ReCOvery dataset. It extracts the complete structure of the image as a visual feature, which makes its performance more obvious than BTIC. The framework FAD proposed in this embodiment has a better performance on ReCOvery, GossipCop, MR2 -E and MR 2 -C significantly outperforms TTEC and BTIC. Specifically, FAD's accuracy on the four datasets outperforms TTEC by 0.61%, 2.12%, 1.6%, and 0.84%, respectively, and outperforms BTIC by 2.22%, 3.94%, 4.01%, and 1.06%, respectively. Both TTEC and BTIC employ a simple sequential concatenation of features to generate multimodal news representations during the feature fusion phase. While this is a common and intuitive approach, simple feature concatenation ignores the interactions between modalities, preventing the model from effectively capturing the interdependencies between image content and text descriptions. In contrast, in the FAD framework proposed in this embodiment, low-level features capture the overall semantics and content of text and images, while high-level features provide a more nuanced representation of the specific details of the text and images. The multi-level feature fusion module integrates textual and visual features from different levels to generate multi-level, multimodal features of the news. Therefore, the framework of this embodiment not only captures richer multimodal information but also better understands the cross-modal connections between text and visual modalities, which helps to better identify fake news. MMFN leverages the emerging multimodal pre-training model CLIP to improve fake news detection. However, it still focuses on resolving ambiguity by weighing the importance of unimodal and multimodal features in the fake news detection decision process; it also ignores the important role of contrastive learning in fake news detection. As a result, FAD's accuracy improves over MMFN by 1.68%, 2.74%, 3.74%, and 2.21% on different datasets, respectively. Furthermore, COOLANT also achieves good fake news detection performance, but FAD's accuracy outperforms COOLANT by 2.85%, 2.3%, 2.4%, and 1.52% on different datasets, respectively.

[0129] Table 3

[0130]

[0131]

[0132] Table 4

[0133]

[0134] The above are merely preferred embodiments of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A multimodal fake news detection method based on multi-level feature fusion and contrastive learning, characterized by: include: Get text information and image information; Extracting multimodal features of the text information and image information respectively to obtain text multimodal features and visual multimodal features; Performing comparative learning on the text multimodal features and the visual multimodal features to obtain text embedding and visual embedding; Fusing the text embedding and the visual embedding to obtain multimodal fusion features; The multimodal features are input into a classifier to obtain a classification result.

2. The multimodal fake news detection method based on multi-level feature fusion and contrastive learning according to claim 1 is characterized in that: Extracting the multimodal features of the text information and the image information respectively, and obtaining the text multimodal features and the visual multimodal features includes: Extracting first-level multimodal features of the text information and image information respectively to obtain first-level text features and first-level visual features; Extracting second-level multimodal features of the text information and the image information respectively to obtain second-level text features and second-level visual features; Performing a cascade operation on the first-level text features and the second-level text features to obtain text multimodal features; A cascade operation is performed on the first-level visual features and the second-level visual features to obtain visual multimodal features.

3. The multimodal fake news detection method based on multi-level feature fusion and contrastive learning according to claim 2 is characterized in that: Obtaining the first-level text features and the first-level visual features includes: An ALIGN encoder is used to extract first-level multimodal features of the text information and image information to obtain the first-level text features and the first-level visual features.

4. The multimodal fake news detection method based on multi-level feature fusion and contrastive learning according to claim 2 is characterized in that: Obtaining the second-level text features and the second-level visual features includes: Using the pre-trained model XLnet to perform second-level encoding on the text information to obtain second-level text features; The pre-trained model SWIN-T is used to perform second-level encoding on the image information to obtain second-level visual features.

5. The multimodal fake news detection method based on multi-level feature fusion and contrastive learning according to claim 1 is characterized in that: Comparative learning of the text multimodal features and the visual multimodal features to obtain text embedding and visual embedding includes: Get posts containing text-image pairs and tag information; Calculating text-to-visual similarity scores and visual-to-text similarity scores in the post; Calculating the cross entropy loss of the textual multimodal features and the visual multimodal features respectively based on the similarity scores; Obtaining contrastive learning loss based on the cross entropy loss; By using the contrastive learning loss, similar text embeddings and visual embeddings are brought closer together in the feature space, while dissimilar text embeddings and visual embeddings are further apart in the feature space. By continuously optimizing the contrastive learning loss, we obtain deeply aligned text embedding and visual embedding.

6. The multimodal fake news detection method based on multi-level feature fusion and contrastive learning according to claim 5 is characterized in that: Calculating the text-to-visual similarity score involves: in, is the text-to-visual similarity score, τ is a learnable temperature parameter, is a pair of text embedding and visual embedding, j is the counting symbol of the number of pictures, which can be 1, 2, 3...N, where N means there are N pictures; Calculating the visual-to-text similarity score involves: in, is the visual-to-text similarity score, is a pair of visual embedding and text embedding.

7. The multimodal fake news detection method based on multi-level feature fusion and contrastive learning according to claim 1 is characterized in that: Fusing the text embedding and visual embedding to obtain multimodal fusion features includes: Calculating query matrices, key matrices, and value matrices for the text embedding and visual embedding, respectively; Based on the query matrix, the key matrix and the value matrix, obtaining multi-level text features and multi-level visual features; The multi-level text features and the multi-level visual features are fused to obtain the multimodal fusion features.

8. The multimodal fake news detection method based on multi-level feature fusion and contrastive learning according to claim 7 is characterized in that: Based on the query matrix, key matrix, and value matrix, obtaining multi-level text features includes: Among them, e tv' is a multi-level text feature, Q v is the text embedding query matrix, is the transpose of the text embedding key matrix, V t is the text embedding value matrix, d and H represent the dimension size and the number of attention heads respectively, is the output linear transformation matrix; Based on the query matrix, the key matrix, and the value matrix, obtaining multi-level visual features includes: Among them, e vt' is a multi-level visual feature, Q t is the visual embedding query matrix, is the transpose of the visual embedding key matrix, V v is the visual embedding value matrix, is the output linear transformation matrix.

Images