False news detection method based on CLIP contrast learning and semantic alignment

Abstract

The application discloses a fake news detection method based on CLIP contrast learning and semantic alignment, and comprises the following steps: extracting N-HIN features and text features and visual features of different levels of news with true and false labels; constructing a fake news detection model; inputting the N-HIN features and the text features and the visual features of different levels of news with true and false labels into the fake news detection model to obtain an optimized fake news detection model; inputting news to be detected into the optimized fake news detection model, classifying the news to be detected, and obtaining classified true news and fake news. The application effectively integrates different modal features to improve the performance of fake news detection.

Description

Technical Field

[0001] This invention belongs to the fields of social media data mining and internet content security technology, and particularly relates to a fake news detection method based on CLIP contrastive learning and semantic alignment. Background Technology

[0002] More and more people are choosing to share their opinions and obtain information on convenient social media platforms; however, this has also facilitated the rapid and widespread spread of fake news. "Fake news" can be understood as reports fabricated or written with the aim of misleading the public, and news posts on social media are easier to manipulate and forge compared to traditional printed materials such as newspapers and magazines. With hundreds of millions of social media users constantly generating unverified news posts, relying solely on expert debunking and manual verification to combat the spread of fake news is far from sufficient. Therefore, there is an urgent need for efficient and automated fake news detection methods.

[0003] Early fake news detection efforts primarily relied on textual data. Whether employing machine learning methods such as Naive Bayes classifiers, decision trees, and support vector machines, or deep learning methods such as Long Short-Term Memory and Convolutional Neural Networks, classification typically involved extracting features from the news text, such as keywords, word frequencies, and syntactic structures, as representations of the source posts. Additionally, some text-based fake news detection efforts considered other cues, such as sentiment and grammatical errors. However, with technological advancements and the proliferation of social media, news posts often include multiple formats, including text and images, making text-based fake news detection insufficiently comprehensive and accurate. Therefore, researchers have focused on mining textual and visual features within news content to improve fake news detection. Recent work has applied the emerging multimodal learning technique CLIP to fake news detection with promising results. However, a common limitation of these methods is that they all ignore the social context of the news, which can provide more clues for identifying carefully crafted fake news. While existing techniques consider social context features, they neglect multimodal semantic alignment.

[0004] Simply using textual and visual features from posts may be insufficient to identify carefully crafted fake news. However, user participation and comments can provide additional assistance in effectively identifying such fake news. Research has revealed that a small minority of users disseminate the vast majority of news, and older, more politically conservative users are more likely to view and share misinformation. Therefore, analyzing user characteristics can provide more evidence for fake news detection. Furthermore, since post content and user comments are inherently interconnected, user comments can provide important clues to confirm potential misinformation within posts and offer interpretability for fake news detection. Heterogeneous Information Networks (HINs) can represent rich social contextual information. Inspired by this, a news-oriented heterogeneous information network (N-HIN) is constructed using news posts, users, and comments, and its features are extracted as supplementary information to improve fake news detection. Currently, fake news detection faces two major challenges: First, textual features, visual features, and N-HIN features are heterogeneous, making it difficult to integrate features from multiple modalities into a single framework for effective aggregation to generate the final news representation and improve decision-making performance. Secondly, emerging pre-trained multimodal learning models (such as CLIP) and powerful cross-modal contrastive learning methods are not entirely applicable to fake news detection. How to effectively utilize these advanced technologies to improve fake news detection remains an unresolved issue. Summary of the Invention

[0005] To address the aforementioned technical issues, this invention proposes a fake news detection method based on CLIP contrastive learning and semantic alignment, which effectively integrates features from different modalities to improve the performance of fake news detection.

[0006] To achieve the above objectives, this invention provides a fake news detection method based on CLIP contrastive learning and semantic alignment, comprising the following steps:

[0007] Extract N-HIN features, as well as textual and visual features at different levels, from news articles with true / false labels;

[0008] To construct a fake news detection model, the N-HIN features of the news with true and false labels, as well as text and visual features at different levels, are input into the fake news detection model to obtain an optimized fake news detection model.

[0009] The news to be detected is input into the optimized fake news detection model, and the news to be detected is classified to obtain the classified real news and fake news.

[0010] According to the fake news detection method based on CLIP contrastive learning and semantic alignment provided by the present invention, the fake news detection model includes a cross-modal association generation module, a multimodal semantic alignment module, and a multimodal feature aggregation module;

[0011] The cross-modal association generation module is used to learn the correlation between deep text features and deep visual features in different levels of text features to obtain cross-modal association features;

[0012] The multimodal semantic alignment module is used to semantically align the news content features with the N-HIN features to obtain the semantically aligned news content features and N-HIN features.

[0013] The multimodal feature aggregation module is used to adaptively weight and aggregate the cross-modal association features, semantically aligned news content features, and N-HIN features to generate the final news representation and classify it to obtain the classification result.

[0014] According to the fake news detection method based on CLIP contrastive learning and semantic alignment provided by the present invention, the cross-modal association generation module includes a contrastive learning submodule and an association generation submodule;

[0015] The contrastive learning submodule is used for alignment between the deep text features and the deep visual features;

[0016] The association generation submodule is used to obtain cross-modal association features between the deep text features and the deep visual features through a text-visual modality fusion network.

[0017] According to the fake news detection method based on CLIP contrastive learning and semantic alignment provided by the present invention, the method for aligning the deep text features and the deep visual features includes:

[0018] Input text-image pairs and label information, normalize the text-image pairs and label information, calculate similarity and cross-entropy loss for the normalized images, and obtain aligned deep text features and deep visual features.

[0019] The fake news detection method based on CLIP contrastive learning and semantic alignment provided by the present invention includes a method for obtaining cross-modal correlation features between deep text features and deep visual features through a text-visual modal fusion network, comprising:

[0020] Based on the aligned deep text features and deep visual features, calculate the intermodal weight graph between text embedding and visual embedding and between visual embedding and text embedding;

[0021] The aligned deep text features and deep visual features are updated by updating the intermodal weight graph to obtain the updated deep text features and deep visual features.

[0022] Based on the updated deep text features and deep visual features, an interaction matrix is ​​generated by outer product. The interaction matrix is ​​then flattened and input into a fully connected layer for transformation to obtain the cross-modal association features.

[0023] According to the fake news detection method based on CLIP contrastive learning and semantic alignment provided by the present invention, the method for obtaining the news content features includes:

[0024] Based on the text features and visual features of the news with true and false labels, text features that enhance visual information and visual features that enhance text information are obtained by calculation.

[0025] The text features of the enhanced visual information and the visual features of the enhanced text information are concatenated to obtain the news content features.

[0026] The fake news detection method based on CLIP contrastive learning and semantic alignment provided by the present invention includes a method for semantically aligning news content features with the N-HIN features to obtain semantically aligned news content features and N-HIN features, comprising:

[0027] The news content features and the N-HIN features are transformed into a shared semantic space. The news content features and the N-HIN features are semantically aligned through joint learning of a specific multilayer perceptron and a semantic sharing layer. The alignment process is optimized using a cosine similarity loss function to obtain the semantically aligned news content features and N-HIN features.

[0028] According to the fake news detection method based on CLIP contrastive learning and semantic alignment provided by the present invention, the method for adaptively weighting and aggregating cross-modal association features, semantically aligned news content features, and N-HIN features to generate the final news representation and perform classification includes:

[0029] The cross-modal association features, semantically aligned news content features, and N-HIN features are concatenated into a specific feature matrix, and the feature matrix is ​​compressed into a specific feature vector through adaptive pooling.

[0030] The specific feature vectors are weighted and aggregated to obtain the final representation of the news.

[0031] The final representation of the news is input into a fully connected network for classification, resulting in classified real and fake news.

[0032] Technical effects of this invention: This invention discloses a fake news detection method based on CLIP contrastive learning and semantic alignment. It jointly models textual, visual, and N-HIN features into a unified framework, extracts textual and visual features at different levels, and uses contrastive learning to achieve more complex alignment between deep textual and deep visual features. It uses an association generation task and text-visual COattention to fuse textual and visual features at different levels, and utilizes social context features to set a semantic alignment task to achieve semantic alignment between news content features and N-HIN features to improve fake news detection. It effectively integrates features from different modalities to improve the performance of fake news detection. Attached Figure Description

[0033] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0034] Figure 1 This is a flowchart illustrating the fake news detection method based on CLIP contrastive learning and semantic alignment according to an embodiment of the present invention.

[0035] Figure 2 This is a schematic diagram illustrating the accuracy results analysis of this invention on PHEME and Gossipcop.

[0036] Figure 3 For embodiments of the present invention in MR 2 -E and MR 2 A schematic diagram of the Accuracy results analysis on -C;

[0037] Figure 4 This is a schematic diagram of the ROC-AUC curves on PHEME and Gossipcop in an embodiment of the present invention;

[0038] Figure 5 MR is an embodiment of the present invention. 2 -E and MR 2 A schematic diagram of the ROC-AUC curve on -C;

[0039] Figure 6 This is an example of the ablation study of SARD on PHEME and Gossipcop according to embodiments of the present invention;

[0040] Figure 7 As an embodiment of the present invention, SARD in MR 2 -E and MR 2 Ablation studies on -C indicate;

[0041] Figure 8 SARD and its variants are embodiments of the present invention in MR2 T-SNE visualization of the final news representation learned on the -E test set, where (a) is w / o CT, (b) is w / o AG, (c) is w / o CA, (d) is w / o SA, (e) is w / o WG, and (f) is SARD;

[0042] Figure 9 The key hyperparameters α and β in the embodiments of the present invention are in MR 2 The impact of the -E dataset on the model performance metric Accuracy. Detailed Implementation

[0043] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0044] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0045] like Figure 1 As shown, this embodiment provides a fake news detection method based on CLIP contrastive learning and semantic alignment, including the following steps:

[0046] Extract N-HIN features, as well as textual and visual features at different levels, from news articles with true / false labels;

[0047] To construct a fake news detection model, the N-HIN features of news with true and false labels, as well as text and visual features at different levels, are input into the fake news detection model to obtain an optimized fake news detection model.

[0048] The news to be detected is input into the optimized fake news detection model, which classifies the news to be detected and obtains the classified real news and fake news.

[0049] Furthermore, N-HIN features of news articles tagged with true or false information, as well as textual and visual features at different levels, are extracted, specifically including:

[0050] Multimodal feature generation using CLIP. Features of text and images are extracted using the CLIP encoder. Specifically, for each multimodal post p... i The text information t in i and visual information v i Semantically aligned text embeddings were obtained using CLIP text encoder and CLIP image encoder, respectively. and visual embedding

[0051] The CLIP model aims to understand the semantic relationship between image content and related text descriptions through pre-training on large-scale image-text pairs. Therefore, it may overlook features that are irrelevant to direct semantic matching but crucial for news detection tasks, such as text style features, syntactic features, and image noise features. In contrast, pre-trained models like BERT and ResNet can capture more comprehensive deep unimodal features. For text information t i Alternatively, BERT is used to generate deeper text features containing more trivial clues. In order to capture more comprehensive image information v i The features are generated using a pre-trained ResNet50 model pre-trained on ImageNet. Specifically It uses a fully connected layer to transform the output of the penultimate layer of ResNet50 into deep text features. Obtained with the same dimensions.

[0052] A heterogeneous information network (N-HIN) feature generation approach for news applications uses the textual features of posts and comments as the initial embeddings for post and comment nodes in the N-HIN. For user nodes, BERT encoding of user profiles is used to represent user features and serves as the initial embedding for user nodes. The post P is obtained by extracting features from different heterogeneous information sources using a Hierarchical Graph Attention Network (HGAT). i Social context information. For N-HIN Each post node in Its neighboring nodes also belong to Neighboring nodes include nodes P i Connected user nodes, comment nodes, and node P i It itself will be node P i As a neighboring node, it can leverage a collaborative self-attention mechanism to reinforce the target node P. i HGAT includes node-level attention and pattern-level attention. Node-level attention aims to learn the feature representation of node P. i A set of weights for neighbors of the same type; this set of weights reflects the relationship between each neighbor node of the same type and the target node P. i The degree of importance. In other words, node-level attention learns which entities within a set of entities of the same type are important to the target node. For example, with node P... i A group of associated user nodes is referred to as node P. i For neighboring nodes of the same type, this set of weights can reflect which user nodes are relevant to node P.i This is important. Then, based on weights, neighboring nodes of the same type are aggregated into pattern nodes. Pattern-level attention aims to learn the attention weights between different pattern nodes to reflect the importance of different types of entities to the target node. Finally, the target node P is obtained by fusing the pattern nodes. i The final expression

[0053] Furthermore, the fake news detection model includes a cross-modal association generation module, a multimodal semantic alignment module, and a multimodal feature aggregation module;

[0054] The cross-modal association generation module is used to learn the correlation between deep text features and deep visual features in different levels of text features to obtain cross-modal association features;

[0055] The multimodal semantic alignment module is used to semantically align news content features with N-HIN features to obtain semantically aligned news content features and N-HIN features.

[0056] The multimodal feature aggregation module is used to adaptively weight and aggregate cross-modal association features, semantically aligned news content features, and N-HIN features to generate the final news representation and perform classification to obtain the classification results.

[0057] Furthermore, the cross-modal association generation module includes a contrastive learning submodule and an association generation submodule;

[0058] The contrastive learning submodule, used for alignment between deep textual features and deep visual features, includes:

[0059] Input text-image pairs and label information, normalize the text-image pairs and label information, calculate similarity and cross-entropy loss on the normalized images, and obtain aligned deep text features and deep visual features.

[0060] Specifically, inspired by the CLIP contrastive learning approach, a contrastive learning module was designed to achieve alignment between deep text and visual embeddings. This is implemented for a batch of N posts P used for training. N Each post P i ∈P N Includes text-image pairs and label information, represented as i represents the i-th post, y i For the tags of the posts, the normalized feature representation of the input to the contrast learning module is as follows: Cross-modal alignment in the contrastive learning module is achieved by predicting post P separately. i text-to-vision similarity Similarity with vision-to-text This is achieved by using In-foNCE as the contrast loss in this process. The similarity score is measured by calculating the dot product using formula (1). and

[0061]

[0062] In formula (1), τ is a learnable temperature parameter. For the i-th deep text feature, Let be the j-th deep visual feature, N be the number of posts used for training, i be the count number of the deep text feature, j be the count code of the deep visual feature, t be the text information, and v be the image information. Next, we use formula (2) to calculate the cross-entropy loss. and

[0063]

[0064] In formula (2) and This represents the vector corresponding to the actual label, encoded using one-hot encoding. Specifically, and This represents a positive sample pair, and the number of positive sample pairs is N. and This represents a negative sample pair, and the number of negative sample pairs is N. 2 -N. The loss function of the contrastive learning module can be calculated using formula (3):

[0065]

[0066] The aligned deep text embeddings are obtained by training a contrastive learning module. t' and deep visual embedding e v' As input to the association generation module.

[0067] The association generation submodule is used to obtain cross-modal association features between deep text features and deep visual features through a text-visual modality fusion network, including:

[0068] Based on the aligned deep text features and deep visual features, calculate the intermodal weight graph between text embedding and visual embedding and between visual embedding and text embedding;

[0069] The aligned deep text features and deep visual features are updated by updating the intermodal weight graph, thus obtaining the updated deep text features and deep visual features.

[0070] An interaction matrix is ​​generated by outer product of the updated deep text features and deep visual features. The interaction matrix is ​​then flattened and input into a fully connected layer for transformation to obtain cross-modal association features.

[0071] Specifically, this includes: effectively improving news detection by capturing the semantic interactions and relevance between two modalities. Therefore, an association generation module is used to fuse aligned deep textual and deep visual features to obtain more comprehensive and accurate cross-modal association features. For the aligned deep unimodal embedding e... t' and e v' First, the intermodal weight graphs f between text embedding and visual embedding and between visual embedding and text embedding are calculated using formula (4). t→v and f v→t :

[0072]

[0073] The square root of the dimension of the deep unimodal embedding is used to normalize the original feature map. The original deep text and visual embeddings are updated through the intermodal weight map, as shown in Equation (5):

[0074]

[0075] To capture the complex modal interactions between deep text and visual embeddings, an interaction matrix e is used. f' This represents the cross-modal association between deep text embeddings and deep visual embeddings. In formula (6), by adjusting e... tv′ and e vt′ The interaction matrix e is generated by performing an outer product between them. f′ :

[0076] In formula (6) e represents the outer product. tv′ To integrate deep text features that incorporate deep image information, e vt′ This is a deep visual feature that incorporates deep textual information. The final generated cross-modal association feature is e. f From the interaction matrix e f′ After being flattened into a vector, it is input into a fully connected layer for transformation and serves as the input to the multimodal feature aggregation module.

[0077] Furthermore, methods for obtaining news content characteristics include:

[0078] Based on the textual and visual features of news with true and false labels, textual features that enhance visual information and visual features that enhance textual information are obtained through calculation.

[0079] By combining the text features of enhanced visual information with the visual features of enhanced text information, the features of news content can be obtained.

[0080] Specifically, CLIP encodes textual and visual features into the same mathematical space, making it highly suitable for Textual-Visual Co-Attention mechanisms to embed semantically rich news content. For example, for post P... i Text features in It uses a method similar to multi-head self-attention to calculate its query matrix, key matrix, and value matrix respectively: and in It is a linear transformation matrix, where d and H represent the dimension and the number of heads, respectively. Visual deep features can be obtained using the same method. The query matrix, key matrix, and value matrix: and Next, we use formula (7) to calculate the text features that enhance the visual information. And enhanced visual features of text information.

[0081]

[0082]

[0083] In formula (7) This is the output linear transformation matrix. Next, we will... and To obtain post P by splicing i News content characteristics in As shown in formula (8):

[0084]

[0085] Here This represents the concatenation operation. News content features m are obtained through a Textual-Visual Co-Attention mechanism. c As input to the semantic alignment module.

[0086] Furthermore, methods for semantically aligning news content features with N-HIN features to obtain semantically aligned news content features and N-HIN features include:

[0087] The news content features and N-HIN features are transformed into a shared semantic space. The news content features and N-HIN features are semantically aligned through joint learning of a specific multilayer perceptron and a semantic sharing layer. The alignment process is optimized by using a cosine similarity loss function to obtain the semantically aligned news content features and N-HIN features.

[0088] Specifically, it includes:

[0089] In the field of fake news detection, semantic differences between features from different sources often pose a challenge to detection accuracy. The semantic differences between news content features and N-HIN features blur the line between real and fake information. This not only increases the difficulty of identifying fake news but also introduces additional complexity to automated fake news detection, a point that existing work has failed to address. To fill this gap, a semantic alignment task is proposed, aiming to achieve semantic alignment between news content features and N-HIN features through joint learning of a specific multilayer perceptron (MLP) and a semantic sharing layer. Specifically, the semantic relevance between news content embeddings and N-HIN embeddings is first defined. If the news content embedding and the N-HIN embedding come from the same real news article, these two embeddings are defined as positively correlated samples, denoted by the label y′=1. Otherwise, they are negatively correlated samples, denoted by the label y′=0. To better assist the relevance learning task, a new auxiliary dataset D′ is generated by randomly sampling positively and negatively correlated samples.

[0090] In the semantic alignment task, for the input news content embedding m c and N-HIN embedding m g First, the two embeddings are transformed into a shared semantic space using a specific multilayer perceptron (MLP). The subsequent semantic sharing layer is responsible for learning and refining the shared semantic representation. The shared semantic representation is then fed into an average pooling layer to integrate information, and finally, a fully connected layer performs binary classification. The semantic alignment module is trained using positive and negative samples, and the cosine similarity loss function in formula (9) is used as the optimization objective.

[0091]

[0092] cos(·) represents the normalized cosine similarity. The cosine similarity loss function aims to maximize the cosine similarity of the embedding vectors between positive samples and minimize the cosine similarity between negative samples until a preset boundary d is reached (set to 0.2 based on empirical research). Backpropagation is used to optimize parameters during the joint training of the semantic alignment module, generating semantically aligned news content features. and N-HIN characteristics

[0093] Furthermore, methods for adaptively weighting and aggregating cross-modal association features, semantically aligned news content features, and N-HIN features to generate the final news representation and perform classification include:

[0094] The cross-modal association features, semantically aligned news content features, and N-HIN features are concatenated into a specific feature matrix, and the feature matrix is ​​compressed into a specific feature vector through adaptive pooling.

[0095] The news is finally represented by a weighted aggregation of specific feature vectors.

[0096] The final news representation is input into a fully connected network for classification, resulting in real and fake news.

[0097] Specifically, it includes:

[0098] Considering the different roles of various modal features in the decision-making process, it is necessary to adjust the importance of each feature in the multimodal feature aggregation process based on its contribution. Inspired by the success of the Squeezed and Incentivized Network (SE-Net), a weight generation mechanism is proposed to adaptively weight each feature. Specifically, firstly, given three L×1 features... and e f The features are concatenated into an L×3 feature set, where L represents the feature length. This L×3 feature set is then compressed into a 1×3 vector s using adaptive pooling. Each feature type corresponds to one channel, and vector s contains the initial weights for each channel. Equation (10) describes a simple gating mechanism to capture the dependencies between different channels:

[0099] a = f S (W2f LR (W1s)) (10)

[0100] Where f LR (·) represents the LeakyReLU function. and This represents the weight matrix of two fully connected layers, where C represents the dimension (set to C=3) and r is the scaling parameter (set to r=1). S (·) represents the sigmoid function, which normalizes the output range to [0-1]. a={a c a g a f} represents the three given features and e f The corresponding attention weights determine the final representation of the post. Calculated using formula (11):

[0101]

[0102] This represents a splicing operation.

[0103] The final representation of a given post Input a fully connected network to predict the tags representing whether a post is real or fake news. As shown in formula (12):

[0104]

[0105] Then, the function formula (13) uses cross-entropy to calculate the prediction loss:

[0106]

[0107] In summary, SARD's overall goal can be defined as:

[0108] A strategy to simplify the model training process was adopted by fixing a key loss function. Weighting and optimizing other losses and We use weights to reduce the number of hyperparameters and maintain the balance of the overall loss function. Specifically, we use the classifier loss that is directly related to this task. The weights are set to 1 to ensure that model training focuses on the most important task objective. Other losses are optimized. and The weights can directly control the relative proportions of the impact of different loss terms on model performance. This strategy effectively simplifies the SARD parameter tuning process while reducing the computational resources and time required for model optimization and improving hyperparameter search efficiency.

[0109] Specific experimental examples:

[0110] This invention uses three real-world public datasets to evaluate the proposed fake news detection model SARD, namely PHEME, Gossipcop, and MR. 2 PHEME consists of tweets collected from Twitter about five different events, including text, images, and comments. The GossipCop dataset, from the FakeNewsNet repository, collects a large number of articles from Twitter about celebrity gossip and entertainment news. This dataset is widely used in the field of fake news detection, and it includes features such as text, images, user profiles, comments, and retweet relationships. MR 2 It can be divided into MR 2 -E English Dataset and MR 2 -C Chinese dataset, MR 2 -E English dataset and MR2 The -C Chinese dataset was constructed by collecting posts from the social media platforms Twitter and Weibo. 2 This includes text, images, comments, reposting relationships, likes, posting dates, and user information such as name, location, and personal tags. In addition, MR... 2 During the construction phase, posts with similar content, duplicate images, and low-resolution images were removed, resulting in a high-quality multimodal dataset for training and evaluating fake news detection models. For the PHEME dataset, user profiles were scraped using the official Twitter API based on user ID. This invention employs the same method as the baseline model to remove posts containing only text or only images from the dataset. Detailed statistical information of the datasets used in the experiments is shown in Table 1.

[0111] Table 1

[0112]

[0113] In this invention, the batch size was 64, the learning rate was 0.001, the Adam optimizer was used to train the model for 50 epochs, and an early stopping strategy was employed to avoid overfitting. For data input, the text length was set to 200 words, and the image size was set to 224×224. For news posts containing multiple images, one image was randomly selected. Regarding the choice of pre-trained BERT models, the experiment used the BERT-base-cased model to encode English data collected from Twitter, and the BERT-base-chinese model to encode Chinese data collected from Weibo. The output dimension of the penultimate layer of the pre-trained ResNet50 feature set was 2048. The pre-trained CLIP model in SARD was "ViT-B / 32". Since there is currently no pre-trained Chinese text model for CLIP, the Google Translation API was used to translate Chinese text into English. Because the upper limit for CLIP text input is set to 77 tokens, a summary generation model was used to generate summary sentences for texts longer than 50 words as CLIP input. In HGAT

[46] , the number of attention heads is set to 8, and the dimensions of the node-level representation and the pattern-level representation are set to 12 and (8*K), respectively. To ensure fair comparison, the model parameters of BERT, ResNet, and CLIP were frozen throughout the experiment. In Textual-Visual Co-Attention, the number of attention heads is set to 8. In the semantic alignment module, the specific multilayer perceptron (MLP) consists of three fully connected layers, each containing 64 hidden units. In special notes, the learnable temperature parameter τ in formula (1) is initialized to 0.07.

[0114] To facilitate comparison with other studies in this field, accuracy, precision, recall, and F1-score were selected as model evaluation metrics in the experiment. The calculation formula is shown in (15).

[0115]

[0116] TP (True Positive) represents the number of fake news items correctly identified by the model, TN (True Negative) represents the number of real news items correctly identified by the model, FN (False Negative) represents the number of fake news items misidentified as real news by the model, and FP (False Positive) represents the number of real news items misidentified as fake news by the model.

[0117] In the experiment, the dataset was divided into training, validation, and test sets in an 8:1:1 ratio. On the validation set, the optimal hyperparameters α = 0.3 and β = 0.5 were determined based on accuracy. The proposed fake news detection framework SARD was implemented in the experimental environment shown in Table 2.

[0118] Table 2

[0119]

[0120]

[0121] Considering the different data types of source data used in existing methods, the baselines are divided into the following four types. In addition to methods based on plain text or plain images, the proposed SRAD model is compared with twelve existing robust baselines in the experiments.

[0122] Based on plain text or plain images: This invention uses pre-trained models BERT and ResNet to verify the effectiveness of textual and visual features in news content for fake news detection. This is because these two pre-trained models can extract comprehensive unimodal features and are widely used in current fake news detection methods. In the experiments, BERT-encoded text features and ResNet-encoded visual features are directly fed into the classifier to determine whether the news is true or false.

[0123] Based on text and social context: dEFEND develops a sentence-comment co-attention sub-network to jointly model text content and comments for fake news detection. FinerFact proposes a fine-grained reasoning framework and designs a priori-aware dual-channel kernel graph network to capture subtle differences between evidence. TR-HGAN utilizes a topological imbalance smoothing strategy to measure the topological weights of nodes and leverages an attention mechanism for graph convolutional learning to detect fake news. DANES is a deep neural network ensemble architecture that combines social and textual contextual information for fake news detection.

[0124] Based on text and images: CAFE models the ambiguity between different modalities by calculating KL divergence, and uses this to measure the importance of cross-modal features in fake news detection. HMCAN performs multimodal fusion through a multimodal contextual attention network and designs a hierarchical encoding network to capture rich semantic information to assist in fake news detection. FND-CLIP introduces an attention layer that adaptively outputs channel scores to measure the importance of each modality, used to dynamically adjust the use of single-modal features and fused features. MMFN achieves more accurate fake news detection by integrating single-modal features and multi-granularity multimodal fusion features.

[0125] Based on text, image, and social context: att-RNN is the first RNN using an attention mechanism to fuse image features, text features, and social context features for news detection. DGExplain detects fake news by leveraging cross-modal association features between text and visual information, as well as comment information. DCNN uses DeepFM, a combination of deep learning and the FM algorithm, to learn social features related to news, while textCNN and VGG19 learn text and image features of news, and concatenate these three features to obtain multimodal features. MFAN is the first to attempt to simultaneously consider textual, visual, and social graph features within a single framework, fusing features from different sources through a cross-modal attention network for fake news detection.

[0126] The data types and main models used in the baseline method are shown in Table 3.

[0127] Table 3

[0128] BERT Text BERT Resnet Image Resnet dEFEND Txet+social BiGRU, Co-Attention FinerFact Txet+social BERT, MLP, APPNP, Bi-ChannelKGAT TR-HGAN Txet+social CNN, self-attention, HGAN DANES Txet+social RNN, CNN, BiGRU, LSTM CAFE Text + image BERT, ResNet34, VAE HMCAN Text + image BERT,Resnet50,contextualtransformer FND-CLIP Text + image CLIP,BERT,ResNet,modality-wiseattention MMFN Text + image CLIP,BERT,Swin-T,Co-attentionTransfomer att-RNN2017 Text + image + social RNN, LSTM, VGG-19, Attentionmechanism DGExplain Text + image + social LSTM, BiGRU DCNN Text + image + social DeepFM, VGG19, textCNN MFAN Text + image + social BERT,Resnet50,SignedGAT,Co-Attention

[0129] Table 4 details the SARD and all baseline methods in PHEME, Gossipcop, and MR. 2Performance on the datasets. Experiments on three datasets show that SARD outperforms all state-of-the-art methods in both accuracy (ACC) and F1 score. First, for unimodal methods, the experimental results show that the text-based method (BERT) outperforms the image-based method (ResNet) on all three datasets, indicating that the textual information of news is the core data for fake news detection tasks. Figure 2 As shown, both text-based and social context-based methods and text-based and image-based methods outperform the unimodal method, demonstrating that the social context and visual information of news can provide important supplements to the fake news detection task.

[0130] Table 4

[0131]

[0132]

[0133] Among text- and social context-based methods, TR-HGAN performs better, demonstrating that modeling the social context of news as a graph structure can significantly improve the performance of fake news detectors. The superior performance of MFAN compared to DCNN and att-RNN further supports this. FND-CLIP and MMFN achieved good results on all three datasets, indicating the crucial role of novel multimodal learning techniques (such as CLIP) in improving fake news detector performance. Among text, image, and social context-based methods, DGExplain achieved excellent results, demonstrating that incorporating social context information in addition to considering news text and images can effectively improve fake news detection. MFAN comprehensively utilizes text, image, and social graph features, but it focuses on fusing features from different modalities using a cross-modal attention mechanism, thus ignoring the impact of different modalities on the final decision of the fake news detection model. In contrast, SARD outperforms all state-of-the-art methods because it uses advanced multimodal learning techniques to extract text and image features at different levels and achieves more complex alignment between text and image features. Furthermore, the semantic alignment task in SARD can learn the shared semantics of news content features and N-HIN features, thereby deriving invariant features of fake news. This is beneficial for the model to perform better when faced with fake news of newly arriving events.

[0134] like Figure 3 As shown, SARD in PHEME, Gossipcop, MR 2 -E and MR 2The accuracy on -C is improved by 1.55%, 3.21%, 1.68%, and 1.59% compared to state-of-the-art methods, respectively. Experimental results show that SARD can effectively integrate textual, visual, and N-HIN features through the joint operation of its three modules, thereby learning a better final representation of news. In this process, SARD utilizes advanced multimodal learning techniques and contrastive learning ideas to achieve alignment and fusion between features from different sources, capturing more comprehensive information and thus more effectively detecting fake news. Table 5 shows the performance of SARD and other state-of-the-art methods in MR. 2 -E and MR 2 Performance comparison on -C. Figure 4-5 The ROC-AUC curves of SARD are shown on four datasets.

[0135] Table 5

[0136]

[0137] The ablation study aims to conduct a series of ablation experiments on the SARID model of this invention to explore the contribution of each component to the model's performance. For the proposed SARID model, this invention designs five different ablation versions:

[0138] 1) w / o CT: The contrastive learning submodule has been removed, and e v and e t Directly input the associated submodule.

[0139] 2) w / o AG: This indicates that the associated generation submodule has been deleted and replaced with a simple connection.

[0140] 3) w / o CA: This indicates that the Textual-Visual Co-Attention mechanism has been removed, and m is used instead. v and m t A simple connection method.

[0141] 4) w / o SA: This indicates that the semantic alignment task has been removed.

[0142] 5) w / o WG: This indicates that the weight generation mechanism has been removed, and e is directly used. f , and The link serves as the final representation of the news.

[0143] Quantitative analysis: Table 6 shows the ablation studies in PHEME, Gossipcop, and MR. 2 -E and MR 2 The results on -C show that removing any component leads to a decrease in the overall performance of the SARD method proposed in this invention. This invention uses... Figure 6-7This further demonstrates intuitively the effectiveness of each component in improving the performance of the model of this invention.

[0144] Table 6

[0145]

[0146] Furthermore, the results show that w / o CA performs poorly, indicating that fully utilizing the complementary relationship between news text and images to fuse and generate news content features is necessary for improving fake news detection. The performance degradation of w / o SA proves the effectiveness of the semantic alignment between the news content features and N-HIN features explored in this invention for fake news detection. Looking at the performance of w / o SA on the entire dataset, compared to MR... 2 -E and MR 2 The performance on -C is significantly lower with no SA on PHEME and Gossipcop, which confirms that the semantic alignment task can derive event-invariant features, thereby enhancing the model's ability to identify the same type of fake news.

[0147] Typesetting analysis: Ablation studies, while validating the effectiveness of model components, can also reveal the internal mechanisms and working principles of the model to verify the rationality of the model design. Therefore, in MR... 2 On the -E test set, the final news representations learned by SARD and its ablation version are visualized using T-SNE to further analyze the impact of each component on the model's learning of news features.

[0148] exist Figure 8 In the analysis, it can be observed that compared to the other five variants, SARD's classification boundaries are the clearest, indicating that the news representation learned by SARD has better representational power and is therefore easier to distinguish. For example... Figure 8 As shown in (a)-(e), the news representations learned by the five variants, besides having unclear classification boundaries, also exhibit numerous instances of misclassification. Therefore, each variant plays a crucial role in improving the fake news detector, and SARD, through the combined effect of its five components, can effectively enhance fake news detection.

[0149] The impact of two important hyperparameters, α and β, in the SARD overall loss function on fake news detection performance was explored. The overall loss function (Eq14) consists of contrastive learning loss. Semantic alignment loss and classifier loss These three parts constitute the invention. Since the ultimate goal of this invention in the fake news detection task is to achieve better classification of true and false news, the classifier loss weight is set to 1. The invention was experimentally tested on MR... 2On the -E dataset, a three-dimensional model is constructed to represent the model performance metric Accuracy for different values ​​of hyperparameters α and β in order to find the optimal hyperparameters. For example... Figure 9 As shown, α and β are increased from 0.0 to 1.0 in increments of 0.1. The model accuracy for each pair of α and β is represented by the peak value on the graph. It can be seen that when α = 0.3 and β = 0.5, the corresponding... Figure 9 The highest peak value, i.e., the highest accuracy, is reached on the dataset. This setting also shows better performance on other datasets. Therefore, in the experiments, the two important hyperparameters α and β of the proposed fake news detection model SARD were set to 0.3 and 0.5, respectively.

[0150] This invention discloses a fake news detection method based on CLIP contrastive learning and semantic alignment. It jointly models textual, visual, and N-HIN features into a unified framework, extracts textual and visual features at different levels, and uses contrastive learning to achieve more complex alignment between deep textual and deep visual features. It uses an association generation task and text-visual COattention to fuse textual and visual features at different levels, and utilizes social context features to set a semantic alignment task to achieve semantic alignment between news content features and N-HIN features to improve fake news detection. It effectively integrates features from different modalities to improve the performance of fake news detection.

[0151] The above are merely preferred embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A fake news detection method based on CLIP contrastive learning and semantic alignment, characterized in that, include: Extract N-HIN features, as well as textual and visual features at different levels, from news articles with true / false labels; To construct a fake news detection model, the N-HIN features of the news with true and false labels, as well as text and visual features at different levels, are input into the fake news detection model to obtain an optimized fake news detection model. The news to be detected is input into the optimized fake news detection model, and the news to be detected is classified to obtain the classified real news and fake news; The fake news detection model includes a cross-modal association generation module, a multimodal semantic alignment module, and a multimodal feature aggregation module; The cross-modal association generation module is used to learn the correlation between deep text features and deep visual features in different levels of text features to obtain cross-modal association features; The multimodal semantic alignment module is used to semantically align the news content features with the N-HIN features to obtain the semantically aligned news content features and N-HIN features. The multimodal feature aggregation module is used to adaptively weight and aggregate the cross-modal association features, semantically aligned news content features, and N-HIN features to generate the final news representation and classify it to obtain the classification result. The cross-modal association generation module includes a contrastive learning submodule and an association generation submodule; The contrastive learning submodule is used for alignment between the deep text features and the deep visual features; The association generation submodule is used to obtain cross-modal association features between the deep text features and the deep visual features through a text-visual modality fusion network; The method for aligning the deep text features and the deep visual features includes: Input text-image pairs and label information, normalize the text-image pairs and label information, calculate similarity and cross-entropy loss for the normalized images, and obtain aligned deep text features and deep visual features. Methods for obtaining cross-modal correlation features between deep text features and deep visual features through text-visual modality fusion networks include: Based on the aligned deep text features and deep visual features, calculate the intermodal weight graph between text embedding and visual embedding and between visual embedding and text embedding; The aligned deep text features and deep visual features are updated by updating the intermodal weight graph to obtain the updated deep text features and deep visual features. Based on the updated deep text features and deep visual features, an interaction matrix is ​​generated by outer product. The interaction matrix is ​​then flattened and input into a fully connected layer for transformation to obtain the cross-modal association features. The methods for obtaining the features of the news content include: Based on the text features and visual features of the news with true and false labels, text features that enhance visual information and visual features that enhance text information are obtained by calculation. The text features of the enhanced visual information and the visual features of the enhanced text information are concatenated to obtain the news content features.

2. The fake news detection method based on CLIP contrastive learning and semantic alignment as described in claim 1, characterized in that, The method for semantically aligning news content features with the N-HIN features to obtain semantically aligned news content features and N-HIN features includes: The news content features and the N-HIN features are transformed into a shared semantic space. The news content features and the N-HIN features are semantically aligned through joint learning of a specific multilayer perceptron and a semantic sharing layer. The alignment process is optimized using a cosine similarity loss function to obtain the semantically aligned news content features and N-HIN features.

3. The fake news detection method based on CLIP contrastive learning and semantic alignment as described in claim 1, characterized in that, The method for adaptively weighting and aggregating cross-modal association features, semantically aligned news content features, and N-HIN features to generate the final news representation and perform classification includes: The cross-modal association features, semantically aligned news content features, and N-HIN features are concatenated into a specific feature matrix, and the feature matrix is ​​compressed into a specific feature vector through adaptive pooling. The specific feature vectors are weighted and aggregated to obtain the final representation of the news. The final representation of the news is input into a fully connected network for classification, resulting in classified real and fake news.

Images