False news detection method and device based on contrast learning and multi-modal fusion

Abstract

The application discloses a false news detection method and device based on contrast learning and multi-modal fusion, which comprises the following steps: through a co-attention network, each feature in initial text features, initial image features and initial graph structure features is respectively subjected to modal enhancement with the remaining two modal features; then, the co-attention network serially connects the enhanced text features, the enhanced image features and the enhanced graph structure features; finally, target multi-modal output features are subjected to contrast learning and input into a full connection layer to obtain a prediction probability of the to-be-detected news being false news. The co-attention network is used for fusing the three kinds of modal features to obtain enhanced features with the three kinds of modal information, so that the semantic consistency between different modalities is ensured, and the accuracy of false news detection is improved. Through contrast learning, the probability representation between samples with the same label is narrowed, so that the true news and the false news can be better distinguished, and the accuracy of false news detection is improved.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence, and more specifically, to a method and apparatus for detecting fake news based on contrastive learning and multimodal fusion. Background Technology

[0002] In today's information age, the internet has achieved leapfrog development. Social media, with its rich information, convenient information sharing, and rapid and widespread dissemination, has become the platform for most people to obtain and exchange information.

[0003] The rapid development of social networks has made them an ideal venue for news dissemination, which has also led to the rapid spread of fake news and distorted viewpoints. The development of multimedia technology has provided opportunities for self-media news, transforming it from simple text posts into multimedia posts with images or videos. Compared to text-only news, these posts contain richer information and attract a wider audience. Similarly, fake news takes full advantage of this to attract and mislead readers.

[0004] Fake news detection methods mainly utilize the co-attention mechanism based on text, images, and social media. Figure 3 This method involves pairwise fusion of features across different modalities, followed by concatenation of the resulting enhanced features into a final multimodal feature set. However, existing fake news detection methods neglect the interactions between the three modalities. Furthermore, the significant differences in the number and quality of modalities contained in different news samples lead to modal imbalances. While attention fusion mechanisms can adaptively allocate weights to different modalities, they cannot directly constrain and enhance semantic consistency between them, thus failing to fully guarantee high semantic coordination among different modalities. Consequently, existing fake news detection methods are relatively weak in fusing multimodal features, making it difficult to guarantee the accuracy of fake news detection. Summary of the Invention

[0005] In view of this, the present invention provides a method and apparatus for detecting fake news based on contrastive learning and multimodal fusion, which solves the problem that existing multimodal fusion methods for detecting fake news are relatively weak in terms of integrating multimodal features, and the accuracy of fake news detection is difficult to guarantee.

[0006] To achieve the above objectives, the following solution is proposed:

[0007] A method for detecting fake news based on contrastive learning and multimodal fusion, the method comprising:

[0008] The pre-trained feature extraction model extracts the initial text features, initial image features, and initial graph structure features of the news to be detected. The initial graph structure features are the structural features of the social graph.

[0009] By using a common attention network, each feature in the initial text features, initial image features, and initial graph structure features is modally enhanced with the other two modal features to obtain enhanced text features, enhanced image features, and enhanced graph structure features;

[0010] By concatenating enhanced text features, enhanced image features, and enhanced graph structure features through a common attention network, the target multimodal output features are obtained.

[0011] The target multimodal output features are compared and learned, and then input into a fully connected layer to obtain the predicted probability that the news to be detected is fake news.

[0012] Preferably, the process of extracting initial text features, initial image features, and initial graph structure features of the news to be detected by the pre-trained feature extraction model includes:

[0013] Initial text features of the news to be detected are extracted using a pre-trained convolutional neural network.

[0014] Initial image features of the news to be detected are extracted using a pre-trained ResNet50;

[0015] The initial graph structure features of the news to be detected are extracted using a pre-trained graph attention network.

[0016] Preferably, the process of modality enhancement of the initial text features and the other two modal features includes:

[0017] The initial text features are enhanced based on the initial image features to obtain cross-modal text features;

[0018] Enhanced text features are obtained by enhancing cross-modal text features based on the initial graph structure features.

[0019] Preferably, the enhancement of the initial text features based on the initial image features includes:

[0020] The initial text features are enhanced using the following formula:

[0021]

[0022] in, For cross-modal text features, This is the initial image feature query vector. As initial image features, This is the transpose of the initial text feature chain vector. This is the initial text feature key vector. This is the initial vector of text feature values. R is the number of heads. For linear transformation, h represents the head index in the multi-head attention mechanism, i is the index variable, d represents the feature dimension, H is the number of heads in the multi-head attention mechanism, and Q represents the relevance to the query;

[0023] The following formula is used to enhance cross-modal text features:

[0024]

[0025] in, To enhance text features, This is the initial graph structure feature query vector. These are the initial graph structural features. It is a linear transformation.

[0026] A fake news detection device based on contrastive learning and multimodal fusion, the device comprising:

[0027] The feature extraction unit extracts the initial text features, initial image features, and initial graph structure features of the news to be detected through a pre-trained feature extraction model. The initial graph structure features are the structural features of the social graph.

[0028] The feature enhancement unit uses a common attention network to enhance each feature in the initial text features, initial image features, and initial graph structure features with the other two modal features, resulting in enhanced text features, enhanced image features, and enhanced graph structure features.

[0029] The feature fusion unit concatenates enhanced text features, enhanced image features, and enhanced graph structure features through a common attention network to obtain the target multimodal output features;

[0030] The news detection unit compares and learns the multimodal output features of the target and inputs them into a fully connected layer to obtain the predicted probability that the news to be detected is fake news.

[0031] According to specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0032] This invention provides a method and apparatus for detecting fake news based on contrastive learning and multimodal fusion. The method uses a co-attention network to enhance each feature from the initial text features, initial image features, and initial graph structure features with the remaining two modal features. Then, the co-attention network concatenates the enhanced text features, enhanced image features, and enhanced graph structure features. Finally, the target multimodal output features are subjected to contrastive learning and input into a fully connected layer to obtain the predicted probability that the news to be detected is fake news. This invention fuses three modal features through a co-attention network to obtain enhanced features with information from all three modalities, ensuring semantic consistency between different modalities and improving the accuracy of fake news detection.

[0033] This invention maps the differences between real and fake news to a feature space through contrastive learning, making the representations of positive samples as close as possible. By narrowing the probability representations between samples with the same label through contrastive learning, this invention can better distinguish between real and fake news, thus improving the accuracy of fake news detection. Attached Figure Description

[0034] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0035] Figure 1 A flowchart of a fake news detection method based on contrastive learning and multimodal fusion provided in this embodiment of the invention;

[0036] Figure 2 This invention provides an architecture diagram of a fake news detection network model based on contrastive learning and multimodal fusion.

[0037] Figure 3 A multimodal feature fusion network architecture diagram provided in an embodiment of the present invention;

[0038] Figure 4 This is a schematic diagram of model performance test results provided in an embodiment of the present invention;

[0039] Figure 5 This is a schematic diagram of a fake news detection device based on contrastive learning and multimodal fusion, provided as an embodiment of the present invention. Detailed Implementation

[0040] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0041] First, combined Figure 1 This invention introduces a method for detecting fake news based on contrastive learning and multimodal fusion, as provided in an embodiment of the present invention. Figure 1 As shown, the detection methods include:

[0042] Step S01: Extract the initial text features, initial image features, and initial graph structure features of the news to be detected using a pre-trained feature extraction model.

[0043] Specifically, embodiments of the present invention utilize text, visuals, and social media. Figure 3 Multimodal feature fusion is performed using various features. Embodiments of this invention achieve this through, as follows: Figure 2 The network model architecture implementation shown is as follows: Figure 2 As shown, the network model is divided into three parts. The first part is feature extraction, which uses three different sub-models to extract features from text, images, and social graphs respectively. In the social graph, nodes represent posts, comments, or users, and edges represent follow relationships between users, post / comment posting relationships between users, and membership relationships between posts and comments.

[0044] In terms of text feature extraction, this invention employs a CNN fusion strategy to extract the semantic features of sentences. An initial text feature of the news to be detected is extracted using a pre-trained convolutional neural network (CNN). By using convolutional kernels of different sizes, the CNN can obtain feature representations of different granularities. Smaller kernels can capture word-level features, while larger kernels can capture a wider range of sentence-level features. This multi-scale feature representation contributes to a more comprehensive understanding of the text. This invention obtains feature maps using convolutional layers with different receptive field sizes. Then, max pooling layers are used to extract the maximum value of each feature map, and finally, the maximum values ​​of all feature maps are concatenated to form the overall initial text features.

[0045] In terms of image feature extraction, this embodiment of the invention extracts initial image features of the news to be detected using a pre-trained ResNet50. The second-layer output of ResNet50 is extracted and transformed through a fully connected layer to obtain initial image features of the same dimension as the initial text features. The ResNet50 can be pre-trained based on the ImageNet database. This fully utilizes the high-level features learned by ResNet50 on ImageNet, further improving the performance of fake news detection. The pre-trained ResNet50 ensures the quality and reliability of the image features.

[0046] In terms of graph structure feature extraction from social graphs, this embodiment of the invention extracts initial graph structure features of the news to be detected through a pre-trained Graph Attention Network (GAT). A social graph is constructed based on the social network, where nodes represent users, comments, and posts; edges represent relationships such as interactions and following between them. For user nodes, the average of the embedding vectors of their posts and comments is used as the initial embedding to comprehensively reflect the user's characteristics. For post and comment nodes, the text features of the posts and comments are used as the initial embedding. The Graph Attention Network can extract rich features from the social graph, providing strong support for subsequent social network analysis.

[0047] Step S02: Modal enhancement is performed by combining each feature in the initial text features, initial image features, and initial graph structure features with the other two modal features through a common attention network.

[0048] Specifically, such as Figure 2 The second part of the network model architecture shown is feature fusion, which effectively integrates the text features, image features, and graph structure features of a post. This embodiment of the invention provides a... Figure 3 The multimodal feature fusion network shown is used to construct effective contextual information across different modules and extract higher-order complementary information from it. For example... Figure 2 As shown, the multimodal feature fusion network consists of multiple co-attention modules.

[0049] The extracted features from the three modalities are fused using a co-attention network to obtain enhanced features with information from all three modalities. This invention utilizes text, vision, and social media... Figure 3 These features are fused using a multimodal co-attention network to obtain enhanced text features, enhanced image features, and enhanced graph structure features.

[0050] The process of initial text feature fusion enhancement is as follows:

[0051] Cross-modal text features are obtained by enhancing the initial text features based on the initial image features. For the initial text features... use and The query vector, chain vector, and value vector are used as text features. For linear transformation, R is the number of heads. When using visual features to enhance text features, Replace with Then we obtained the cross-modal features. for:

[0052]

[0053] in, For cross-modal text features, This is the initial image feature query vector. As initial image features, This is the transpose of the initial text feature chain vector. This is the initial text feature chain vector. This is the initial vector of text feature values. R is the number of heads. For linear transformation, h represents the head index in the multi-head attention mechanism, i is the index variable, d represents the feature dimension, H is the number of heads in the multi-head attention mechanism, and Q represents the relevance to the query.

[0054] To further enhance the text features using the initial graph structure features, the cross-modal text features obtained above are combined with the initial graph structure features through a co-attention module. The result is an enhanced text feature derived from the initial image features and the initial graph structure features.

[0055]

[0056] in, To enhance text features, This is the initial graph structure feature query vector. These are the initial graph structural features. It is a linear transformation.

[0057] Step S03: The enhanced text features, enhanced image features, and enhanced graph structure features are concatenated through a common attention network.

[0058] Specifically, after the co-attention network enhances each feature, the enhanced features are fused. The initial text features, initial image features, and initial graph structure are enhanced based on the multimodal co-attention mechanism described in step S02, resulting in three enhanced features for each modality. Finally, these features are concatenated to form the final target multimodal output feature.i :

[0059]

[0060] in, To enhance text features, To enhance image features, To enhance graph structure features.

[0061] Step S04: Compare and learn the target multimodal output features and input them into the fully connected layer.

[0062] Specifically, fake news detection requires models to capture subtle features and differences in textual data, features that are often difficult to learn through traditional supervised learning methods. Contrastive learning can help models learn more discriminative feature representations because it encourages models to map the differences between real and fake news into a feature space. In this way, the model can better distinguish between real and fake news. This invention adds contrastive learning at the end of the model, such as... Figure 2 As shown, the third part of the network model architecture is contrastive learning, which is used to bring the probability representations of samples with the same label closer together. This makes the representations of positive samples as close as possible while keeping them away from negative samples. Contrastive learning is introduced through feature learning across different modalities to improve the model's cross-modal performance.

[0063] The target multimodal features o i The input is a fully connected layer, which has a two-layer structure: the first layer has a dimension of 512 and uses the ReLU activation function; the second layer has a dimension of 2 and outputs the predicted probability that the news to be detected is fake news.

[0064]

[0065] Among them, W c Let be the weight matrix of the fully connected layer, and b be the bias term of the fully connected layer.

[0066] This invention provides a method and apparatus for detecting fake news based on contrastive learning and multimodal fusion. The method uses a co-attention network to enhance each feature from initial text features, initial image features, and initial graph structure features with the remaining two modal features. Then, the co-attention network concatenates the enhanced text features, enhanced image features, and enhanced graph structure features. Finally, the target multimodal output features are subjected to contrastive learning and input into a fully connected layer to obtain the predicted probability that the news to be detected is fake news. This invention fuses three modal features through a co-attention network to obtain enhanced features with information from all three modalities, ensuring semantic consistency between different modalities and improving the accuracy of fake news detection.

[0067] This invention utilizes contrastive learning to map the differences between real and fake news into a feature space, making the representations of positive samples as close as possible. By using contrastive learning to narrow down the probability representations of samples with the same labels, this invention can better distinguish between real and fake news, thereby improving the accuracy of fake news detection.

[0068] The present invention describes the model training process as follows:

[0069] 1. Obtain the dataset and select the training and testing datasets.

[0070] Two public datasets are collected, and each dataset is divided into a training set and a test set. For example, the two datasets could be WEIBO and PHEME. The WEIBO dataset is based on the Weibo platform and covers microblog information from various fields. Each post contains three elements (Twitter ID, text, and image). The PHEME dataset is based on five breaking news stories, each containing a set of posts. Each dataset contains a large amount of tagged text and images.

[0071] 2. Model Training and Optimization

[0072] During the model training phase, this embodiment of the invention first requires model initialization, including the initialization of weights and biases. To accelerate the training process and avoid overfitting, pre-trained model weights are used, particularly the weights of ResNet50 and the Graph Attention Network (GAT). These pre-trained models perform well on their respective tasks and can provide rich feature representations, thereby helping to improve the performance of fake news detection.

[0073] Next, the model is trained using the stochastic gradient descent optimization algorithm. During training, the model's weights and biases are updated based on the gradient of the loss function. The loss function includes classification loss and contrastive learning loss, and the importance of both can be balanced by adjusting the hyperparameter β.

[0074] This invention uses the cross-entropy function as the loss function and introduces mode alignment:

[0075]

[0076] in λ is the probability predicted by the model, y is the true label, and λ is the true label. c and λ a These are weighting coefficients. The alignment loss can be derived using cosine similarity.

[0077] Because cross-entropy loss learns information between classes through inter-class competition, it only focuses on the accuracy of predicting the probability of the correct label, ignoring the differences of other incorrect labels. This leads to suboptimal generalization and instability. Good generalization requires capturing the similarity between samples within a class and comparing samples within the same class with samples from other classes.

[0078] Given the output features of news in a batch: Where N is the number of inputs in the batch. Define the output features of the news. i When used as an anchor point, news features O with the same label will be included. j O j As a positive sample pair, i.e. y i =y j Among them, y i and y j They are o i and O j If the labels are specified, then the contrastive loss function is:

[0079]

[0080] The final loss function is

[0081]

[0082] in It's a loss of framework, L sup It is the contrastive learning loss, where β is a hyperparameter.

[0083] 3. Model Evaluation and Result Analysis

[0084] After training, the model's performance is evaluated using metrics such as accuracy, recall, and F1 score. These metrics comprehensively reflect the model's ability to identify fake news. By comparing the model's performance metrics on the test set, its generalization ability and robustness can be assessed. Furthermore, the model of this invention is compared with existing fake news detection models to verify its advantages.

[0085] Experimental results are as follows Figure 4 As shown, the model of this invention outperforms existing multimodal fake news detection models in terms of accuracy, recall, and F1 score. This is mainly due to improvements in feature extraction, multimodal feature fusion, and contrastive learning. Specifically, this invention utilizes text, images, and social media... Figure 3 By incorporating the features of various modalities and combining a multimodal co-attention network and a contrastive learning mechanism, the model of this invention can more comprehensively understand news content and capture the correlation and complementarity between different modalities.

[0086] The following describes the fake news detection device based on contrastive learning and multimodal fusion provided by the embodiments of the present invention. The fake news detection device based on contrastive learning and multimodal fusion described below can be referred to in correspondence with the fake news detection method based on contrastive learning and multimodal fusion described above.

[0087] First, combine Figure 5 This paper introduces a fake news detection device based on contrastive learning and multimodal fusion, such as... Figure 5 As shown, the fake news detection device based on contrastive learning and multimodal fusion may include:

[0088] The feature extraction unit 100 extracts the initial text features, initial image features, and initial graph structure features of the news to be detected through a pre-trained feature extraction model. The initial graph structure features are the structural features of the social graph.

[0089] The feature enhancement unit 200 uses a common attention network to enhance each feature in the initial text features, initial image features, and initial graph structure features with the other two modal features, resulting in enhanced text features, enhanced image features, and enhanced graph structure features.

[0090] The feature fusion unit 300 concatenates enhanced text features, enhanced image features, and enhanced graph structure features through a common attention network to obtain target multimodal output features;

[0091] The news detection unit 400 compares and learns the multimodal output features of the target and inputs them into a fully connected layer to obtain the predicted probability that the news to be detected is fake news.

[0092] This invention also provides a storage medium that stores a program suitable for a processor to execute, the program being used to implement the various processing flows in the aforementioned fake news detection scheme based on contrastive learning and multimodal fusion.

[0093] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0094] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0095] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for detecting fake news based on contrastive learning and multimodal fusion, characterized in that the method... The method comprises the steps of: extracting initial text features, initial image features and initial graph structure features of the news to be detected by a pre-trained feature extraction model, wherein the initial graph structure features are structural features of a social graph; enhancing each feature in the initial text features, the initial image features and the initial graph structure features by a co-attention network, to obtain enhanced text features, enhanced image features and enhanced graph structure features; concatenating the enhanced text features, the enhanced image features and the enhanced graph structure features by the co-attention network to obtain target multi-modal output features; performing contrastive learning on the target multi-modal output features and inputting the target multi-modal output features into a fully connected layer to obtain a prediction probability that the news to be detected is fake news; the process of enhancing the initial text features and the other two modal features comprises: enhancing the initial text features based on the initial image features to obtain cross-modal text features; enhancing the cross-modal text features based on the initial graph structure features to obtain the enhanced text features; the process of enhancing the initial text features based on the initial image features comprises: enhancing the initial text features by the following formula: ， wherein, is a cross-modal text feature, is an initial image feature query vector, , is an initial image feature, is a transpose of an initial text feature key vector, is an initial text feature key vector, is a value vector of an initial text feature, , is a linear transformation, is a linear transformation, is an initial text feature, h represents a head index in a multi-head attention mechanism, i is an index variable, d represents a feature dimension, and H is a number of heads in the multi-head attention mechanism. enhancing the cross-modal text features by the following formula: ， wherein, to enhance the text features, to obtain the initial graph structure feature query vector, , to obtain the initial graph structure feature, to obtain the linear transformation, to obtain the linear transformation.

2. The false news detection method based on contrastive learning and multi-modal fusion according to claim 1, characterized in that, the process of extracting the initial text features, the initial image features and the initial graph structure features of the news to be detected by the pre-trained feature extraction model comprises: extracting the initial text features of the news to be detected by a pre-trained convolutional neural network; extracting the initial image features of the news to be detected by a pre-trained ResNet50; extracting the initial graph structure features of the news to be detected by a pre-trained graph attention network.

3. A false news detection device based on contrastive learning and multi-modal fusion, characterized in that, The device comprises: a feature extraction unit configured to extract initial text features, initial image features and initial graph structure features of the news to be detected by a pre-trained feature extraction model, wherein the initial graph structure features are structural features of a social graph; a feature enhancement unit configured to enhance each feature in the initial text features, the initial image features and the initial graph structure features by a co-attention network, to obtain enhanced text features, enhanced image features and enhanced graph structure features; a feature fusion unit configured to concatenate the enhanced text features, the enhanced image features and the enhanced graph structure features by the co-attention network to obtain target multi-modal output features; a news detection unit configured to perform contrastive learning on the target multi-modal output features and input the target multi-modal output features into a fully connected layer to obtain a prediction probability that the news to be detected is fake news; the process of enhancing the initial text features and the other two modal features comprises: enhancing the initial text features based on the initial image features to obtain cross-modal text features; enhancing the cross-modal text features based on the initial graph structure features to obtain the enhanced text features; the process of enhancing the initial text features based on the initial image features comprises: enhancing the initial text features by the following formula: ， wherein, is a cross-modal text feature, is an initial image feature query vector, , is an initial image feature, is a transpose of an initial text feature key vector, is an initial text feature key vector, is a value vector of an initial text feature, , is a linear transformation, is a linear transformation, is an initial text feature, h represents a head index in a multi-head attention mechanism, i is an index variable, d represents a feature dimension, and H is a number of heads in the multi-head attention mechanism. enhancing the cross-modal text features by the following formula: ， wherein, to enhance the text features, to obtain the initial graph structure feature query vector, , to obtain the initial graph structure features, to obtain the linear transformation, to obtain the linear transformation.

Images