False news detection method based on comment-context double-collaborative mask Transform model

Abstract

The invention discloses a false news detection method based on a comment-context double-collaborative mask Transform model, and the method comprises the following steps: S1, obtaining an image, a text and a user comment of a social media post, respectively inputting the images, the text and the user comment into a pre-trained deep learning network, and obtaining an image region feature representation, a text feature representation and a user comment feature representation; s2, inputting the image region feature representation, the text feature representation and the user comment feature representation obtained in the step S1 into a multi-mode dual-collaborative mask Transform network; s3, introducing a comment-context encoder, automatically identifying and screening out key comments, and filtering out irrelevant or unimportant comments; and S4, calculating a loss value according to a prediction result, training a false news detector and optimizing the model performance, and finally outputting the authenticity prediction probability of the article through a full connection layer in cooperation with an activation function. According to the method, multi-modal features are fused by utilizing a double-collaborative mask Transform network, and a comment-context encoder is introduced, so that efficient detection of false news is realized.

Description

Technical Field

[0001] The present invention relates to the field of multimedia technology, and in particular to a fake news detection method based on a comment-context dual collaborative mask Transformer model. Background Art

[0002] In recent decades, more and more people have participated in publishing and consuming online news, a trend that can be attributed to the growing popularity and convenience of social media platforms. With the growing user base, social media sites have become a hub for a wide variety of information. However, the authenticity of information is far from reliable, as users often neglect to carefully review the reliability of the content they share. This laxity in verification has paved the way for the rampant spread of a large amount of fake news. Moreover, this phenomenon of large-scale dissemination of fake news has a harmful impact on both individuals and society as a whole, as they maliciously distort and fabricate factual information. In order to avoid unnecessary losses and negative impacts, methods need to be developed to help automatically detect and identify fake news so that users can obtain authentic information.

[0003] Currently, efforts to detect fake news have long been a research focus. Early methods relied on user reports and expert confirmation, which are time-consuming and labor-intensive. Therefore, researchers have explored automated methods, which can be broadly divided into two types: (1) The first is based on handcrafted features, which design features from post content and user social context, and then use SVM or decision tree classifiers to identify fake news. However, handcrafted features may not capture the complex content of fake news. (2) The second method is based on deep learning, which uses neural networks to capture deep features. In recent years, the composition of news content has changed from pure text to multimedia formats such as pictures and videos, which provide complementary information for identifying fake news. User comments also contain key clues to determine true and false news. Although detection methods for multimodal information have some effectiveness, most of them still cannot properly detect the internal relationships between multimodal data. The challenge of multimodal feature fusion lies in dealing with the heterogeneity of various modalities and interpreting the potential correlations between them. In general, there is an unbalanced and complementary relationship between different modalities. However, in different modalities, there is some irrelevant contextual information, and these irrelevant elements may interfere with each other, thus affecting the effectiveness of fake news detection. In addition, user comments can provide valuable clues and supplementary information, which can assist the model to better identify and detect false information. However, current detection methods using user reviews often have difficulty processing user comments because comments usually contain serious noise. In addition, most of such methods do not make full use of all relevant comments, or only treat all relevant user comments equally, which will greatly reduce the role of key user comments.

[0004] Therefore, how to provide a fake news detection method based on the comment-context dual collaborative mask Transformer model is an urgent problem that technicians in this field need to solve. Summary of the invention

[0005] One purpose of the present invention is to propose a fake news detection method based on a comment-context dual collaborative masked Transformer model. The present invention adopts a method based on a comment-context dual collaborative masked Transformer network, fully integrates the multimodal features of images, texts and user comments in social media posts, constructs fine internal associations, and effectively filters noise through a cross-attention network and a comment-context encoder, accurately identifies key comments, and realizes efficient detection of fake news. This method not only improves the ability to capture fine-grained relationships between multimodal data, but also significantly improves the accuracy and robustness of false information detection, providing a more reliable tool for social media information identification.

[0006] According to an embodiment of the present invention, a fake news detection method based on a comment-context dual collaborative mask Transformer model includes the following steps:

[0007] S1. Obtain images, texts and user comments of social media posts, and input them into the pre-trained deep learning network to obtain image region feature representation, text feature representation and user comment feature representation;

[0008] S2, inputting the image region feature representation, text feature representation and user comment feature representation obtained in step S1 into the multimodal dual collaborative mask Transformer network;

[0009] S3, introduce comment-context encoder to automatically identify and filter out key comments and filter out irrelevant or unimportant comments;

[0010] S4. Calculate the loss value based on the prediction results, train the fake news detector and optimize the model performance, and finally output the authenticity prediction probability of the article through the fully connected layer and activation function.

[0011] As stated in the task description, our model input is multimodal news P = {T, V, C}, where T represents text content, V represents visual content, and C represents corresponding user comments. In order to better obtain fine-grained image and text feature representations of social media posts. For text information, we use the pre-trained model BERT to extract text features; for image information, we use ResNet50 to obtain image feature representations.

[0012] Optionally, the S1 specifically includes:

[0013] S11. For text information, use the pre-trained model BERT to extract text features and generate feature representations including word semantics and context information;

[0014] Given a text content T, model T as a word sequence T = {w1,w2,…,w m}, where m represents the number of words in the text, and the processed text features are represented as P T = {P t1 ,P t2 ,...P tm}, where P ti Represents the i-th word w i The characteristics of the word representation method P ti is calculated by the pre-trained BERT model:

[0015] P T = {P t1 ,P t2 ,...P tm}=BERT(W);

[0016] in, is the output layer hidden state of the corresponding tag in BERT, d t is the dimension of word embedding;

[0017] S12. For the visual content V, use the pre-trained ResNet50 model to extract regional features and obtain a set of P V = {P v1 ,P v2 ,...P vn}, where n represents the number of regions in the image, P vj represents the average pooled convolution feature of the jth region;

[0018] The pre-trained model remains unchanged during the training process. For the provided visual content V, the function of the penultimate pooling layer in the visual feature extractor can be described as:

[0019] P V = {P v1 ,P v2 ,...P vn} = ResNet50(V);

[0020] in, d v represents the dimension of image embedding;

[0021] S13. For the comment content C, use BERT to extract content features. Given a set of comment sequences C = {c1,...,c o}, where o represents the number of comments, and the transformed features are Indicates that Indicates the hth comment C h The feature transformation of the comment indicates Calculated using pre-trained BERT:

[0022]

[0023] in, represents the pooled output of the hidden state layer in BERT, d t is the dimension of word embedding.

[0024] Optionally, the dual collaborative mask Transformer network in step S2 is composed of a text-guided image mask Transformer network, a text-guided comment mask Transformer network and two cross-attention modules, such as Figure 2 shown.

[0025] In the Transformer model, the self-attention mechanism is indispensable and constitutes the core of the encoder-decoder architecture in sequence-to-sequence processing, which helps to establish long-range dependencies between input and output. L*D , the expression of the single-head self-attention mechanism is as follows:

[0026]

[0027] where the product of softmax(·) and V represents the input combination weighted by the attention score derived from the interaction between the query Q and the key value K, and the scaling factor in the denominator is It helps to suppress the growth of the magnitude of the dot product, thus stabilizing the metric.

[0028] Optionally, the text-guided image mask Transformer network converts the pre-trained text features P T As input 2, the image feature P V As input 1, calculate the similarity matrix and mask matrix M:

[0029]

[0030] l=Mean(FFN([P T ||P V ]));

[0031] in, Indicates P T The i-th word in P V The similarity score between the j-th regions in represents the scaling factor, Indicates P T The i-th word in P V The mask matrix between the jth regions in , l represents the context similarity threshold, FFN represents a two-layer fully connected network, and || represents a connection operation; it should be noted that we do not set the threshold l as a fixed constant, but instead use a two-layer fully connected network to map them into a one-dimensional space to obtain l based on the intrinsic features of text and image. This enables the network to adaptively assign different thresholds l to different samples, thereby improving the robustness of the model.

[0032] The modified single-head multimodal self-attention is defined as:

[0033]

[0034] Among them, MaskedAtt represents mask attention, Softmax represents the normalization function, represents different linear transformations that project the input into queries, keys, and values, represents an element-wise multiplication operation, Represents the scaling factor.

[0035] Optionally, the text-guided comment mask Transformer network transforms the text feature P T As input 2, the comment features As input 1, calculate the similarity score matrix and mask matrix M1:

[0036]

[0037] u=Mean(FFN([P T ||P C ]));

[0038] in, Indicates P T The i-th word in P C The similarity score between the h-th comments in Indicates P T The i-th word in P C The mask matrix between the h-th comments in , u represents the context similarity threshold, FFN represents a two-layer fully connected network, and || represents a connection operation;

[0039] The improved single-head multimodal self-attention is described as:

[0040]

[0041] Among them, MaskedAtt represents mask attention, Softmax represents the normalization function, denote different linear transformations that project the input into queries, keys, and values, respectively. represents an element-wise multiplication operation, Represents the scaling factor.

[0042] Optionally, the two cross-attention modules encode the output of the text-guided image mask Transformer network and the output of the text-guided comment mask Transformer network respectively. Figure 2 As shown, for the left cross attention, P TV As input 4 (i.e. Q), P TC As input 3 (i.e., K and V). Similarly, for the right side of the cross attention, P TC As input 4 (i.e. Q), P TV As input 3 (ie K and V).

[0043] Optionally, the comment-context encoder in step S3 is used to explore the internal relationship between key comments and articles, providing key auxiliary information for detecting fake news;

[0044] Extract the indices of the top k reviews from the masked attention matrix in the text-guided review mask transformer:

[0045]

[0046] U=TopK((M a ) T W a ,k);

[0047] Among them, U∈R k ,M a ∈R n×o and W a ∈R n×1 is a training parameter, TopK represents the top-k operation row by row, k represents the hyperparameter of the number of selected comments, Softmax represents the normalization function, denote different linear transformations that project the input into query, key, and value, respectively. T represents text feature representation, P V represents image feature representation, represents an element-wise multiplication operation, Represents the scaling factor.

[0048] Optionally, the fake news detector in step S4 predicts the probability distribution of fake news on the input through a dual collaborative mask Transformer network and a comment-context encoder network, including:

[0049] Use the prediction results of the dual collaborative mask Transformer network as the final prediction;

[0050] The difference between the predicted distribution and the true labels is minimized by the cross entropy loss function.

[0051] Specifically, both the dual collaborative masked Transformer network and the comment-context encoder network predict the probability distribution of fake news for the input. We use the prediction of the former model (denoted as ) as the final prediction of our model. The prediction of the latter model is It is not incorporated into the final prediction but only used by the loss function to better train the former model for review extraction.

[0052] The dual collaborative masked Transformer network is designed to integrate multimodal feature information, effectively capture fine-grained internal relationships between modalities, and filter out noise and irrelevant context. Therefore, in order to enhance the effectiveness in fake news detection, we adopt the cross entropy loss To train our model, this loss minimizes the true label y∈R 2 With the predicted distribution The difference between:

[0053]

[0054] The comment-context encoder network is used to determine meaningful user comments by mapping and sorting the masked attention weight matrix. In the comment-context encoder network, the model predicts fake news labels based on the selected comments, motivating us to design the selection loss To enhance the probability associated with the predicted y:

[0055]

[0056] in, is the predicted distribution of the comment context encoder network, is the true label.

[0057] The aggregate loss used to train the model is:

[0058]

[0059] in, is the cross entropy loss used to train the dual co-masked Transformer network to predict fake news labels. represents the selection loss, which is used to train the review-context encoder network to select key user reviews and enhance the probability associated with the prediction. Among them, τ is a hyperparameter as an element to balance the two components.

[0060] The beneficial effects of the present invention are:

[0061] First, the present invention realizes the effective fusion of multimodal data (including images, text and user comments), thereby significantly improving the accuracy of fake news detection. Pre-trained deep learning models (such as BERT and ResNet50) are used to extract text, image and comment features respectively, so that the model can better capture the fine-grained multimodal feature information in social media posts. Then, the dual collaborative masked Transformer network deeply fuses the multimodal features through the text-guided image mask and comment mask Transformer networks, filters out irrelevant contextual information between different modalities, and further enhances the understanding of the fine-grained internal relations between modalities.

[0062] By introducing two cross-attention networks, the present invention can further explore and coordinate the internal associations between text, images, and comments, ensuring that the model not only focuses on information within a single modality, but also fully captures the interactive relationship between multiple modalities. This feature effectively avoids the problem of decreased accuracy in existing detection methods due to insufficient fusion of modal information. At the same time, the introduction of the comment-context encoder network realizes the screening and optimization of user comments, automatically identifies key comments, filters out irrelevant or noisy comments, and provides key clues to reveal the truth behind unverified articles. This function enables the model to better utilize the potential important information in user comments when identifying false information, significantly improving the accuracy of detection.

[0063] Finally, by inputting the fused multimodal features into the fake news detector and training with the cross entropy loss function, the model can accurately classify the authenticity of social media posts. With this method, the model achieved higher accuracy, recall and F1 score in the COVID-19 fake news detection task, effectively alleviating the challenges in the multimodal information fusion and user comment screening process, improving the ability to detect false information in social media, and providing users with a more reliable information identification tool. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0065] Figure 1This is a flowchart of the fake news detection method based on the comment-context dual collaborative mask Transformer model proposed by the present invention;

[0066] Figure 2 A schematic diagram of a model of a fake news detection method based on a comment-context dual collaborative mask Transformer model proposed in the present invention;

[0067] Figure 3 Schematic diagram of the mask mechanism module of the fake news detection method based on the comment-context dual collaborative mask Transformer model proposed in the present invention. DETAILED DESCRIPTION

[0068] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, which only illustrate the basic structure of the present invention in a schematic manner, and therefore only show the components related to the present invention.

[0069] refer to Figure 1-3 , a fake news detection method based on comment-context dual collaborative mask Transformer model includes the following steps:

[0070] S1. Obtain images, texts and user comments of social media posts, and input them into the pre-trained deep learning network to obtain image region feature representation, text feature representation and user comment feature representation;

[0071] S2, inputting the image region feature representation, text feature representation and user comment feature representation obtained in step S1 into the multimodal dual collaborative mask Transformer network;

[0072] S3, introduce comment-context encoder to automatically identify and filter out key comments and filter out irrelevant or unimportant comments;

[0073] S4. Calculate the loss value based on the prediction results, train the fake news detector and optimize the model performance, and finally output the authenticity prediction probability of the article through the fully connected layer and activation function.

[0074] As stated in the task description, our model input is multimodal news P = {T, V, C}, where T represents text content, V represents visual content, and C represents corresponding user comments. In order to better obtain fine-grained image and text feature representations of social media posts. For text information, we use the pre-trained model BERT to extract text features; for image information, we use ResNet50 to obtain image feature representations.

[0075] In this implementation, S1 specifically includes:

[0076] S11. For text information, use the pre-trained model BERT to extract text features and generate feature representations including word semantics and context information;

[0077] Given a text content T, model T as a word sequence T = {w1,w2,…,w m}, where m represents the number of words in the text, and the processed text features are represented as P T = {P t1 ,P t2 ,...P tm}, where P ti Represents the i-th word w i The characteristics of the word representation method P ti is calculated by the pre-trained BERT model:

[0078] P T = {P t1 ,P t2 ,...P tm}=BERT(W);

[0079] in, is the output layer hidden state of the corresponding tag in BERT, d t is the dimension of word embedding;

[0080] S12. For the visual content V, use the pre-trained bottom-up attention model ResNet50 to extract regional features and obtain a set of P V = {P v1 ,P v2 ,...P vn}, where n represents the number of regions in the image, P vj represents the average pooled convolution feature of the jth region;

[0081] The pre-trained model remains unchanged during the training process. For the provided visual content V, the function of the penultimate pooling layer in the visual feature extractor can be described as:

[0082] P V = {P v1 ,P v2 ,...P vn} = ResNet50(V);

[0083] in, d v represents the dimension of image embedding;

[0084] S13. For the comment content C, use BERT to extract content features. Given a set of comment sequences C = {c1,...,c o}, where o represents the number of comments, and the transformed features are Indicates that Indicates the hth comment C h The feature transformation of the comment indicates Calculated using pre-trained BERT:

[0085]

[0086] in, represents the pooled output of the hidden state layer in BERT, d t is the dimension of word embedding.

[0087] In this embodiment, the dual collaborative mask Transformer network in step S2 is composed of a text-guided image mask Transformer network, a text-guided comment mask Transformer network and two cross-attention modules, such as Figure 2 shown.

[0088] In the Transformer model, the self-attention mechanism is indispensable and constitutes the core of the encoder-decoder architecture in sequence-to-sequence processing, which helps to establish long-range dependencies between input and output. L*D , the expression of the single-head self-attention mechanism is as follows:

[0089]

[0090] where the product of softmax(·) and V represents the input combination weighted by the attention score derived from the interaction between the query Q and the key value K, and the scaling factor in the denominator is It helps to suppress the growth of the magnitude of the dot product, thus stabilizing the metric.

[0091] In this embodiment, the text-guided image mask Transformer network converts the pre-trained text feature P T As input 2, the image feature P V As input 1, calculate the similarity matrix and mask matrix M:

[0092]

[0093] l=Mean(FFN([P T ||P V ]));

[0094] in, Indicates P T The i-th word in P V The similarity score between the j-th regions in represents the scaling factor, Indicates P T The i-th word in P V The mask matrix between the jth regions in , l represents the context similarity threshold, FFN represents a two-layer fully connected network, and || represents a connection operation; it should be noted that we do not set the threshold l as a fixed constant, but instead use a two-layer fully connected network to map them into a one-dimensional space to obtain l based on the intrinsic features of text and image. This enables the network to adaptively assign different thresholds l to different samples, thereby improving the robustness of the model.

[0095] The modified single-head multimodal self-attention is defined as:

[0096]

[0097] Among them, MaskedAtt represents mask attention, Softmax represents the normalization function, represents different linear transformations that project the input into queries, keys, and values, represents an element-wise multiplication operation, Represents the scaling factor. To help understand, we give a schematic diagram of the mask strategy, as shown in Figure 3 shown.

[0098] In this embodiment, the text-guided comment mask Transformer network transforms the text feature P T As input 2, the review feature P Ch As input 1, calculate the similarity score matrix and mask matrix M1:

[0099]

[0100] u=Mean(FFN([P T ||P C ]));

[0101] in, Indicates P T The i-th word in P C The similarity score between the h-th comments in Indicates P T The i-th word in P C The mask matrix between the h-th comments in , u represents the context similarity threshold, FFN represents a two-layer fully connected network, and || represents a connection operation;

[0102] The improved single-head multimodal self-attention is described as:

[0103]

[0104] Among them, MaskedAtt represents mask attention, Softmax represents the normalization function, denote different linear transformations that project the input into queries, keys, and values, respectively. represents an element-wise multiplication operation, Represents the scaling factor.

[0105] In this embodiment, the two cross-attention modules encode the output of the text-guided image mask Transformer network and the output of the text-guided comment mask Transformer network respectively. Figure 2 As shown, for the left cross attention, P TV As input 4 (i.e. Q), P TC As input 3 (i.e., K and V). Similarly, for the right side of the cross attention, P TC As input 4 (i.e. Q), P TV As input 3 (ie K and V).

[0106] In this embodiment, the comment-context encoder in step S3 is used to explore the internal relationship between key comments and articles, and provide key auxiliary information for detecting fake news;

[0107] Extract the indices of the top k reviews from the masked attention matrix in the text-guided review mask transformer:

[0108]

[0109] U=TopK((M a ) T W a ,k);

[0110] Among them, U∈R k ,M a ∈R n×o and W a ∈R n×1 is a training parameter, TopK represents the top-k operation row by row, k represents the hyperparameter of the number of selected comments, Softmax represents the normalization function, denote different linear transformations that project the input into query, key, and value, respectively. T represents text feature representation, P V represents image feature representation, represents an element-wise multiplication operation, Represents the scaling factor.

[0111] In this embodiment, the fake news detector in step S4 predicts the probability distribution of fake news on the input through a dual collaborative masked Transformer network and a comment-context encoder network, including: using the prediction result of the dual collaborative masked Transformer network as the final prediction; and minimizing the difference between the predicted distribution and the true label through a cross entropy loss function.

[0112] Specifically, both the dual collaborative masked Transformer network and the comment-context encoder network predict the probability distribution of fake news for the input. We use the prediction of the former model (denoted as ) as the final prediction of our model. The prediction of the latter model is It is not incorporated into the final prediction but only used by the loss function to better train the former model for review extraction.

[0113] The dual collaborative masked Transformer network is designed to integrate multimodal feature information, effectively capture fine-grained internal relationships between modalities, and filter out noise and irrelevant context. Therefore, in order to enhance the effectiveness in fake news detection, we adopt the cross entropy loss To train our model, this loss minimizes the true label y∈R 2 With the predicted distribution The difference between:

[0114]

[0115] The comment-context encoder network is used to determine meaningful user comments by mapping and sorting the masked attention weight matrix. In the comment-context encoder network, the model predicts fake news labels based on the selected comments, motivating us to design the selection loss To enhance the probability associated with the predicted y:

[0116]

[0117] in, is the predicted distribution of the comment context encoder network, is the true label.

[0118] The aggregate loss used to train the model is:

[0119]

[0120] in, is the cross entropy loss used to train the dual co-masked Transformer network to predict fake news labels. represents the selection loss, which is used to train the review-context encoder network to select key user reviews and enhance the probability associated with the prediction. Among them, τ is a hyperparameter as an element to balance the two components.

[0121] Embodiment 1:

[0122] In order to verify the feasibility of the present invention in implementation, we use two publicly available datasets, ReCOVery and MMCoVaR, for experiments. ReCOVery contains COVID-19 news posts covering multimodal information from January 2020 to May 2020. MMCoVaR contains multimodal social media posts related to specific COVID-19 vaccines, and the post editing time is from February 2020 to May 2021. We divide the two datasets into training and test sets in a ratio of 8:2.

[0123] For multimodal embedding of social media posts, we use the pre-trained ResNet to extract visual features and the pre-trained BERT model to extract text features in post text and comments. The image embedding dimension is 2048 and the text embedding dimension is 768. To adapt to this model, the visual region feature dimension is converted from 2048 to 768 by introducing a 2D convolutional layer. Our model is trained using the Adaptive Moment Estimation (Adam) optimizer for a total of 200 epochs, with a learning rate of 0.001 and a mini-batch size of 64. The false information detection task belongs to binary classification, and accuracy is generally used as the main evaluation indicator. However, its reliability may be reduced in the case of unbalanced sample data. Therefore, during the experiment, we added precision, recall, and weighted F1-score as supplementary evaluation indicators to address the problem caused by data imbalance. Table 1 shows the comparison of the proposed method with other methods on the ReCOVery dataset. Table 2 shows the comparison of the proposed method with other methods on the MMCoVaR dataset.

[0124] Table 1 Comparison of different methods in false information detection in the ReCOVery dataset

[0125] method Acc(%) P(％) R(％) F1(％) HSA 77.9 73.7 73.6 73.6 ExFaux 76.3 71.9 69.5 70.4 dEFEND 85.6 82.6 81.3 82.3 BTIC 76.3 71.9 69.5 70.4 SAFE 83.1 80.3 78.9 79.5 EANN 84.7 81.6 83.4 82.4 SpotFake 68.1 63.7 65.0 64.1 MVAE 82.5 81.3 75.5 77.4 DGExplain 89.7 89.0 86.1 87.3 C2DCMTN(Ours) 93.6 92.3 87.6 89.8

[0126] Table 2 Comparison of different methods in false information detection in the MMCoVaR dataset

[0127] method Acc(%) P(％) R(％) F1(％) HSA 80.3 78.2 78.5 78.4 ExFaux 76.9 78.4 69.4 70.7 dEFEND 85.6 84.7 83.1 83.8 BTIC 82.9 82.3 79.1 80.3 SAFE 78.8 77.3 74.9 75.7 EANN 83.3 81.9 81.0 81.4 SpotFake 69.6 67.0 62.0 62.3 MVAE 81.5 80.5 83.4 80.8 DGExplain 89.5 89.6 87.1 88.1 C2DCMTN(Ours) 93.8 94.8 88.4 91.5

[0128] Among them, HSA (Hierarchical Social Analysis) is a method for false information detection using user comments and hierarchical social network structures in the media. ExFaux is a graph-based false image interpretation framework that can provide content explanations for detection results. dEFEND is an explainable false information detection method. It uses the correlation between news text and user comments to both classify false news and determine how user comments provide explanations for classification results. BTIC uses a BERT-based multimodal framework for unreliable news detection, and uses contrastive learning strategies for text and visual information in suspicious articles to utilize this information. SAFE is a similarity-centered multimodal method for false information detection. It extracts text and visual elements from news materials and explores their mutual relationships to obtain the final representation. EANN includes a multimodal feature extractor, a false news detector, and a post-processing discriminator. SpotFake uses pre-trained BERT to extract text features, while using VGG-19 pre-trained on the ImageNet dataset to extract image features, thereby assisting in judging the authenticity of the post. MVAE obtains multimodal representation by introducing multimodal features into a bimodal variational autoencoder (VAE). The multimodal latent representation generated by the bimodal VAE is then classified by a binary classifier. DGExplain is a generative method that analyzes the inter-modal connections between visual and textual information to identify the authenticity of multimodal posts related to COVID-19.

[0129] The experimental results are shown in Table 1 and Table 2. By observing the data in the table, we can draw the following conclusions:

[0130] (1) Feature fusion-based methods, such as MVAE, SpotFake, EANN, and ExFaux, performed poorly in both datasets. The poor performance of these methods can be attributed to their neglect of the intricate relationship between visual and textual elements in social media posts. They infer the credibility of the content to be detected directly from the mixed multimodal content without considering this mutual correlation;

[0131] (2) DGExplain outperforms other baseline methods, which shows that it accurately generates cross-modal features and combines the consistency evaluation between generated and original multimodal features to assist in the detection of fake news. At the same time, the incorporation of DGExplain's content-comment graph greatly enhances the integration of generated, original multimodal features and user comments, which in turn helps in the detection of false information;

[0132] (3) The proposed C2DCMTN outperforms all baseline methods on both datasets. The results show that the proposed multimodal dual collaborative gather transformer network can fully capture the fine-grained feature relationships between and within modalities, retrieve and collect key parts of modal information, and ultimately achieve good detection results.

[0133] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A fake news detection method based on comment-context dual collaborative masked Transformer model, characterized in that: The steps include: S1. Obtain images, texts and user comments of social media posts, and input them into the pre-trained deep learning network to obtain image region feature representation, text feature representation and user comment feature representation; S2, inputting the image region feature representation, text feature representation and user comment feature representation obtained in step S1 into the multimodal dual collaborative mask Transformer network; S3, introduce comment-context encoder to automatically identify and filter out key comments and filter out irrelevant or unimportant comments; S4. Calculate the loss value based on the prediction results, train the fake news detector and optimize the model performance, and finally output the authenticity prediction probability of the article through the fully connected layer and activation function.

2. The fake news detection method based on comment-context dual collaborative mask Transformer model according to claim 1 is characterized in that: The S1 specifically includes: S11. For text information, use the pre-trained model BERT to extract text features and generate feature representations including word semantics and context information; Given a text content T, model T as a word sequence T = {w1,w2,…,w m }, where m represents the number of words in the text, and the processed text features are represented as P T = {P t1 ,P t2 ,...P tm }, where P ti Represents the i-th word w i The characteristics of the word representation method P ti is calculated by the pre-trained BERT model: P T ={P t1 ,P t2 ,...P tm }=BERT(W); in, is the output layer hidden state of the corresponding tag in BERT, d t is the dimension of word embedding; S12. For the visual content V, use the pre-trained ResNet50 model to extract regional features and obtain a set of P V = {P v1 ,P v2 ,...P vn }, where n represents the number of regions in the image, P vj represents the average pooled convolution feature of the jth region; The pre-trained model remains unchanged during the training process. For the provided visual content V, the function of the penultimate pooling layer in the visual feature extractor is described as: P V ={P v1 ,P v2 ,...P vn }=ResNet50(V); in, d v represents the dimension of image embedding; S13. For the comment content C, use BERT to extract content features. Given a set of comment sequences C = {c1,...,c o }, where o represents the number of comments, and the transformed features are Indicates that Indicates the hth comment C h The feature transformation of the comment indicates Calculated using pre-trained BERT: in, represents the pooled output of the hidden state layer in BERT, d t is the dimension of word embedding.

3. The fake news detection method based on comment-context dual collaborative mask Transformer model according to claim 1 is characterized in that: The dual collaborative mask Transformer network in step S2 consists of a text-guided image mask Transformer network, a text-guided comment mask Transformer network and two cross-attention modules.

4. The fake news detection method based on comment-context dual collaborative mask Transformer model according to claim 3 is characterized in that: The text-guided image mask Transformer network transforms the pre-trained text features P T As input 2, the image feature P V As input 1, calculate the similarity matrix and mask matrix M: l=Mean(FFN(P T ||P V ])); in, Indicates P T The i-th word in P V The similarity score between the j-th regions in represents the scaling factor, Indicates P T The i-th word in P V The mask matrix between the j-th regions in , l represents the context similarity threshold, FFN represents a two-layer fully connected network, and || represents a connection operation; The modified single-head multimodal self-attention is defined as: Among them, MaskedAtt represents mask attention, Softmax represents the normalization function, represents different linear transformations that project the input into queries, keys, and values, represents an element-wise multiplication operation, Represents the scaling factor.

5. The fake news detection method based on comment-context dual collaborative mask Transformer model according to claim 3 is characterized in that: The text-guided comment mask Transformer network transforms the text feature P T As input 2, the comment features As input 1, calculate the similarity score matrix and mask matrix M1: u=Mean(FFN([P T ||P C ])); in, Indicates P T The i-th word in P C The similarity score between the h-th comments in Indicates P T The i-th word in P C The mask matrix between the h-th comments in , u represents the context similarity threshold, FFN represents a two-layer fully connected network, and || represents a connection operation; The improved single-head multimodal self-attention is described as: Among them, MaskedAtt represents mask attention, Softmax represents the normalization function, denote different linear transformations that project the input into queries, keys, and values, respectively. represents an element-wise multiplication operation, Represents the scaling factor.

6. The fake news detection method based on comment-context dual collaborative mask Transformer model according to claim 3 is characterized in that: The two criss-cross attention modules encode the output of the text-guided image mask Transformer network and the output of the text-guided comment mask Transformer network, respectively.

7. The fake news detection method based on comment-context dual collaborative mask Transformer model according to claim 1 is characterized in that: The comment-context encoder in step S3 is used to explore the internal relationship between key comments and articles, providing key auxiliary information for detecting fake news; Extract the indices of the top k reviews from the masked attention matrix in the text-guided review mask transformer: U=TopK((M a ) T IN a ,k); Among them, U∈R k ,M a ∈R n×o and W a ∈R n×1 is a training parameter, TopK represents the top-k operation row by row, k represents the hyperparameter of the number of selected comments, Softmax represents the normalization function, denote different linear transformations that project the input into query, key, and value, respectively. T represents text feature representation, P V represents image feature representation, represents an element-wise multiplication operation, Represents the scaling factor.

8. The fake news detection method based on comment-context dual collaborative mask Transformer model according to claim 1 is characterized in that: The fake news detector in step S4 predicts the probability distribution of fake news on the input through a dual collaborative mask Transformer network and a comment-context encoder network, including: Use the prediction results of the dual collaborative mask Transformer network as the final prediction; The difference between the predicted distribution and the true labels is minimized by the cross entropy loss function.

Images