Multimodal fake news detection method based on dynamic propagation social graph

By introducing dynamic propagation social graphs and multimodal dynamic fusion layers in the fake news detection model, combining Transformer and dynamic symbol attention blocks, the shortcomings of the existing models in capturing dynamic fusion and timing similarity intensity are solved, and high accuracy is achieved and robust fake news detection effects.

CN118568261BActive Publication Date: 2024-12-06NANKAI UNIV

Patent Information

Application Number
CN202410713566.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-06-04
Publication Date
2024-12-06
Estimated Expiration
2044-06-04

AI Technical Summary

Technical Problem

Existing fake news detection models are difficult to capture dynamic fusion and timing similarity intensity, and cannot fully and accurately capture the complex communication patterns and graph structure information of news in social networks.

Method used

A multimodal fake news detection model based on dynamic propagation social graph is designed. Through timing similarity intensity weighting and multimodal dynamic fusion layer, local and global semantic information in the propagation sequence is captured, and Transformer perceives the propagation sequence. Semantics, combining dynamic symbol attention block and graph structure characteristics to perform false news detection.

Benefits of technology

Fake news detection with high accuracy and robustness is achieved, significantly improving detection performance, especially in handling heterogeneous patterns and timing information.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118568261B_ABST
    Figure CN118568261B_ABST
Patent Text Reader

Abstract

The present invention discloses a multimodal false news detection method based on a dynamic propagation social graph, which belongs to the field of information security technology and is implemented through a multimodal false news detection model. The multimodal false news detection model includes five parts: a propagation sequence capture layer, a feature representation layer, multimodal dynamic fusion, enhanced graph structure extraction, and neighbor semantic perception. The existing methods have the problem of difficulty in capturing temporal intensity and deep social graph representation. Considering the multimodality of news content and the relationship between entities, it first uses the temporal permeation fusion block to perform multimodal dynamic fusion on various types of nodes in the propagation sequence, and then uses dynamic symbol weighting to enhance the propagation graph social graph features. Finally, Transformer captures the global semantics of the propagation mode and fuses multi-domain information. Experimental results on three data sets demonstrate the effectiveness of the proposed model in the task of false news detection.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of information security technology and is a multimodal fake news detection model (DPSG) based on dynamic propagation social graphs involving multimodal fusion, social network heterogeneous graphs, attention mechanisms and Transformer. Experimental results on three data sets prove the effectiveness of the proposed model in the fake news detection task. Background Art

[0002] With the rise of social media, people are more inclined to get news from social media than from traditional ways of getting news. However, while social media brings convenience to people's lives, it also promotes the spread of false news. Multimodal false news has a stronger visual impact, spreads faster and more widely in social media, and causes more destructive harm to the public. Therefore, it is necessary to design a fast and accurate false news detection model.

[0003] Traditional fake news detection models are mainly text-based unimodal models, which use semantic features extracted from news texts, position-based features, sentiment features, etc. for detection. With the development of social media, news content has gradually become richer, including multimodal forms such as text and images. Therefore, some multimodal detection models have been proposed. However, news does not exist independently in social networks, and these models only focus on news content without considering the social context of news in social networks.

[0004] In social networks, news has various types of relationships with entities such as users and posts, which can be represented by heterogeneous graphs. Figure 2 The following is an example of a news heterogeneous graph. Figure 2 In the example, user 7 published news 3 and forwarded news 2, and user 3 forwarded news 1 and news 2. News 1 and news 3, as well as news 1 and news 2, are likely to have the same tags (or similar news content). If we only focus on the news content, we cannot capture the attached Figure 2 The inner connection between news 1 and news 3, and news 1 and news 2 is shown in FIG. The nodes that have such inner connection with the target news node are called news neighbor nodes, user neighbor nodes, and post neighbor nodes.

[0005] Graph Neural Networks (GNNs) such as Graph Attention Networks (GAT) and Graph Convolutional Networks (GCN) learn the feature representation of target nodes by aggregating neighbor nodes, and have achieved good results in processing graph-structured data. However, most existing graph models are mainly used to process isomorphic graphs rather than heterogeneous graphs. Although some models for heterogeneous graphs have been proposed, some of them still have problems with edge isomorphism or node isomorphism, and cannot fully and accurately capture the complexity of news heterogeneous graphs. Recently, researchers applied Transformer to the processing of heterogeneous graphs, capturing structural information through the encoder-decoder structure of Transformer.

[0006] Although the above methods have good performance in the fake news detection task, they have the following shortcomings: (1) It is difficult to capture dynamic fusion. Existing models give equal attention when performing multimodal fusion or aggregating neighborhood context information, without considering the strength of temporal similarity. Specifically, after a piece of news is published, the posts closer to the publishing time can better reflect the news content and the user's real-time reaction. Because as time goes by, users will obtain new information or have a deeper understanding of the news content. (2) Shallow graph structure feature representation. By attaching Figure 2 It can be seen that the importance of deeply capturing the graph structure information in the propagation pattern is important for improving the performance of fake news detection. Most existing models cannot simultaneously capture the complex relationship between nodes and edges in the graph structure and the dependency relationship between nodes in the propagation pattern. Summary of the invention

[0007] The purpose of this invention is to overcome the shortcomings of existing models and build a fake news detection model that considers the dynamic social graph of propagation mode, which is called a multimodal fake news detection model based on dynamic propagation social graph. The model has high prediction accuracy and strong robustness, and has better performance in fake news prediction than existing models.

[0008] The technical solution of the present invention is:

[0009] A multimodal fake news detection model based on dynamic propagation social graph (DPSG) is proposed. The temporal similarity strength is designed to weight the neighbors in the propagation sequence and perform multimodal dynamic fusion. Then, the propagation social graph of the target node is constructed, and the local context structure is captured by the dynamic symbol weighting method using the enhanced node representation and graph structure. Finally, Transformer is used to perceive the global semantics in the propagation sequence and perform fake news detection by fusing multi-domain information.

[0010] The multimodal false news detection method based on dynamic propagation social graph is implemented by a multimodal false news detection model. The architecture diagram of the multimodal false news detection model based on dynamic propagation social graph of the present invention includes five parts: propagation sequence capture layer, feature representation layer, multimodal dynamic fusion, enhanced graph structure extraction and neighbor semantic perception. The specific implementation includes the following steps:

[0011] Step 1: Propagation sequence capture layer: Construct a news heterogeneous graph G = (V, E) based on the three types of entities in the dataset: news, posts, and users, where V = {N, P, U} represents the node set of the three node types of news, posts, and users, and E represents the edge set between them. m n 、m p and m u Represents the number of news nodes, post nodes and user nodes respectively, traversing the news node n i ∈N, for each starting node n i Perform the following steps T times: return to the starting node with probability p, or randomly select n i Neighbor nodes join N(n i ) set until the specified number of neighbor nodes of the corresponding type is collected to ensure that each type of neighbor node is sampled. i ) News neighbor nodes N in the set n 、Post neighbor node N p and user neighbor node N u Sort by sampling frequency and keep the first k neighbor nodes with the highest frequency, k = k n +k p +k u ;

[0012] Step 2: Feature representation layer: First, the text is padded or truncated into a token sequence of length n using a tokenizer T = {[CLS], o1, o2, ..., o n ,[SEP]}, then input T into the fine-tuned RoBERTa model to obtain the embedding vector, and use the embedding vector of [CLS] to represent the embedding vector of the entire token sequence e, d t Represents the embedding dimension. Using the above method, the following standardized embedding vector representations are obtained in turn: News title e t 、News content c , post information p , user description u For visual feature representation, the pre-trained ResNet18 model is used to obtain fine-grained image feature vectors, and the image representation is obtained after normalization. where d vRepresents the dimension of the image feature vector. Some numerical attributes and category attributes are converted into sparse feature representation using the one-hot method. The numerical attributes are normalized to ensure the same numerical range, and other feature vectors of the post are obtained. Other feature vectors of users where d op , d ou Represent the dimensions of other feature vectors of posts and users respectively;

[0013] Step 3: Multimodal dynamic fusion. In order to accurately obtain the feature representation of each type of node in the heterogeneous graph, a multimodal dynamic fusion layer is designed, which consists of two parts: temporal similarity strength and temporal permeation fusion blocks.

[0014] Furthermore, in the temporal similarity strength described in step 3:

[0015] Among the news neighbor nodes and post neighbor nodes of the target news node, the closer the neighbor nodes are to the target news node, the more their content can reflect the social context of the target news node when it is released. Therefore, the temporal similarity strength is designed, and the news neighbor N n The oth neighbor in the target news n i Time interval The calculation is shown in formula (1)

[0016]

[0017] in Indicates target news n i The release time, t o Indicates news neighbor N n The publishing time of the oth neighbor in ;

[0018] News Neighbor N n The oth neighbor in the target news n i The temporal similarity strength As shown in formula (2)

[0019]

[0020] Using the above method, we can obtain the news neighbors and post neighbors for the target news n i The temporal similarity strength s n 、s p ,in

[0021] Furthermore, in the timing permeation fusion block described in step 3:

[0022] The temporal permeation fusion block is designed to capture the multimodal dynamic fusion features of three types of nodes: news, users, and posts. The self-attention mechanism is used to capture the internal dependencies of each attribute feature. i Taking the news neighbors of as an example, we first concatenate the attribute features of the news nodes, including title, content, and vision, and perform linear transformation into a unified dimension d. The title feature E of the news neighbors t The acquisition of is shown in formula (3)

[0023]

[0024] in, is a trainable parameter, concat(·) represents a vector concatenation operation;

[0025] Using multi-head self-attention mechanism to capture semantic dependency features of news headlines in parallel The calculation is shown in formula (4)

[0026]

[0027] in,

[0028] In the same way, the semantic dependency features of news content of news neighbor nodes can be obtained: Visual semantic dependency features and temporal semantic dependency features Then, time-series weighted permeation can permeate the time-series information of each neighbor node into the attribute features, capturing the dynamic relationship between neighbors. The calculation is shown in formula (5)

[0029]

[0030] in, and Represents the target news n i The temporal semantic dependency features of news type neighbors and the semantic dependency features of news titles,

[0031] Formula (5) can be used to obtain the news content temporal penetration characteristics of news node types: and visual temporal penetration features Build an average pooling layer to fuse the three attribute features of the news node type together to obtain the fused feature representation of the news node The calculation is shown in formula (6)

[0032]

[0033] in, MeanPool(·) represents the average pooling operation.

[0034] The same method as above can be used to obtain the target news n i Fusion features Fusion features of post neighbors It is obtained by fusing time series permeation information and other features, while the fusion features of user neighbors do not need to consider time series permeation, because there is no dynamic relationship between user neighbors, so the fusion features of user nodes It is obtained by fusing the descriptive features enhanced by internal dependencies and other features, where

[0035] Step 4: Enhanced graph structure extraction. It consists of three parts: graph structure enhancement, dynamic symbol attention block and multimodal alignment.

[0036] Furthermore, in the graph structure enhancement described in step 4:

[0037] The extraction of graph structure features mainly involves two parts of information: the information contained in the node itself and the relationship between nodes. For the information part of the node itself, the fusion features obtained by the multimodal dynamic fusion module are used to obtain the fusion features of the target news. Fusion features with its neighboring nodes According to the propagation trajectory, the sequence N(n i ) are spliced ​​together in order to form the target news n i The dynamic neighbor embedding sequence E fus ∈R l×d , the first line in the sequence is the target news n i Fusion features, where l = 1 + k n +k p +k u , the relationship between nodes is expressed by building an adjacency matrix. The construction of the adjacency matrix mainly obtains the complex connection relationship between nodes from two perspectives. One is to find the connection relationship between l nodes in the heterogeneous graph G, and the other is to supplement the edge connection with the cosine similarity. The latter is because in social networks, personal privacy or data crawling issues may inevitably lead to incompleteness of the acquired data. According to the homogeneity of the network, similar nodes are more likely to be connected. The cosine similarity between each node is calculated based on the dynamic neighbor embedding sequence, and the nodes with cosine similarity greater than 0.5 are also added with edge connections. After supplementation, an l×l adjacency matrix A is constructed. If there is an edge between two nodes, it is marked as 1, otherwise it is marked as 0. The supplemented adjacency matrix contains a more comprehensive connection relationship between nodes.

[0038] Furthermore, in the dynamic symbol attention block described in step 4:

[0039] Build a dynamic symbolic attention block to calculate the dynamic attention coefficients between nodes and weight neighbor nodes by symbol to fuse them into a deep and rich graph structure feature.

[0040] The dynamic attention coefficient is mainly calculated through the graph attention mechanism network. In this process, since the temporal semantics are permeated in the dynamic neighbor embedding sequence, the graph attention mechanism also integrates the dynamic relationship between nodes when calculating the attention coefficient. The dynamic attention coefficient E between nodes in the dynamic neighbor embedding sequence + The calculation is shown in formula (7)

[0041] E + =LeakyReLU((E fus Wa1)+(E fus Wa2) T ) (7)

[0042] Among them, E fus Indicates target news n i Dynamic neighbor embedding sequence, W, a1 and a2 are trainable parameters, a1 and a2 are the first d rows and the last d rows of weight a, respectively, a∈R 2d , E fus ∈R l×d , W∈R d×d , E + ∈R l×l ;

[0043] Dynamic attention coefficient E + Standardization, as shown in formula (8)

[0044] E + ′=softmax(E + ) (8)

[0045] In the dynamic attention coefficient E + There will be some negative values ​​in E, which reflect the negative correlation between nodes, such as posts opposing the target news. This relationship is also very important for the fake news detection task. However, after the softmax(·) function, the negative values ​​will become smaller and this negative correlation will be ignored. Therefore, let E - =-E + Then, it is standardized as shown in formula (9):

[0046] E - ′=softmax(E - ) (9)

[0047] Based on the standardized dynamic attention coefficient E + ′ and E -′, according to the connection between nodes in the neighbor matrix A, weight and aggregate the neighbor nodes by sign, concatenate the aggregate features that capture the positive and negative correlations, and obtain the dynamic sign attention block output feature f through the fully connected layer g , calculated as shown in formula (10)

[0048]

[0049] Among them, ReLU(·) is the activation function, || represents the concatenation operation, Wg is a trainable parameter, and W g ∈R 2d×d , f g ∈R l ×d ;

[0050] The dynamic neighbor embedding sequence is input into the dynamic symbol attention block h times, and the neighbor nodes are weighted and aggregated with different dynamic attention coefficients. After splicing the output, it is input into the dynamic symbol attention block again to obtain a more comprehensive and integrated graph structure feature F. g , the calculation process is shown in formula (11)

[0051]

[0052] in, represents the graph structure feature obtained for the jth time, j∈[1,h], DSA(·) represents the dynamic symbol attention block, σ(·) represents the sigmoid activation function, and F g ∈R l×d .

[0053] Furthermore, in the multimodal alignment described in step 4:

[0054] Multi-head attention is used to enhance the graph structure features and fusion features of the target news and news neighbor nodes, and the graph structure features and text-visual fusion features are transformed into the same feature space, as shown in formula (12):

[0055]

[0056] in, and Represents the target news n i and the graph structure characteristics of its news neighbor nodes, || represents the splicing operation,

[0057] Then, the MSE loss function is used to measure the graph modality alignment, and the calculation is shown in formula (13):

[0058]

[0059] Step 5: Neighbor semantic perception. After obtaining the fusion features of the target news and its neighbors of various types and the graph structure features, the encoder-decoder structure of Transformer is used to perceive the contextual semantics of the nodes in the news propagation sequence, and the perception features of the target news are obtained for training the classifier.

[0060] Furthermore, in the Transformer structure input described in step 5: the Encoder input includes the target news node n i Its N(n i The fusion feature representation of all neighbors in ) is E fus , corresponding position embedding E pos , graph structure feature F g and node type embedding E type , for E fus and F g Formula (12) is used to enhance and transform into the same feature space to obtain the enhanced feature representation E' fus and F' g ,Encoder input sequence H enc As shown in formula (14)

[0061] H enc =E' fus +E pos +F' g +E type (14)

[0062] Among them, H enc ∈R l×d ;

[0063] The input of the decoder includes the target news n i Its N(n i The enhanced fusion feature representation of news neighbors in Corresponding position embedding Enhanced graph feature representation and node type embedding Decoder input sequence H dec As shown in formula (15)

[0064]

[0065] in,

[0066] The obtained H enc and H dec Follow Target News iThe propagation sequence is fed into the encoder and decoder of the Transformer model respectively to capture the relationship between nodes, learn the information interaction and transmission between nodes, aggregate the contextual semantic information of neighboring nodes, and obtain the target news n i The perceptual feature representation H awa .

[0067] Furthermore, in the training classifier described in step 5: the target news n i The perceptual feature representation H awa The input is sent to the fully connected layer, and the sigmoid activation function is used to predict whether the target news is fake news. The calculation is shown in formula (16)

[0068]

[0069] Among them, W awa and b awa are trainable weights and biases, respectively, σ(·) represents the sigmoid activation function;

[0070] The cross entropy loss function is used for calculation, as shown in formula (17):

[0071]

[0072] The overall loss L is shown in formula (18):

[0073]

[0074] where λ a and λ c Used to balance L a and L c Two loss functions.

[0075] Furthermore, the method of the multi-head self-attention mechanism in step 3 of the present invention is: the multi-head self-attention mechanism inputs E t Mapped into three matrices: query, key, and value, using Q′=E t W t Q , K′=E t W t K and V′=E t W t V Way E t Linear transformation H times, parallel execution of attention mechanism, multi-head attention mechanism calculation is shown in formula (19) (20)

[0076]

[0077] Among them, d krepresents the dimension of K′, W M ∈R d×d , d represents the unified projection dimension, and || represents the concatenation operation.

[0078] Advantages and beneficial effects of the present invention:

[0079] The present invention provides a new idea and method for the field of fake news detection, which has the advantages of high accuracy and strong robustness, and proposes a multimodal fake news detection model (DPSG) based on a dynamic propagation social graph. The model uses a multimodal dynamic fusion module to fuse the multimodal features of each node in the propagation sequence in a time-order weighted manner. The graph structure features are obtained through the symbol attention block, and the global semantics are perceived by Transformer, so as to accurately capture the propagation mode. The experimental results on multiple data sets show that the model shows significant superiority in the fake news detection task. By analyzing the experimental results, in the GNN-based method, the model shows better performance than other methods, especially in constructing models of heterogeneous nodes and heterogeneous edges. In addition, the model also achieves good results in capturing time series information and global semantic information, and has achieved significant improvements in accuracy and F1 scores compared with other baseline models. In summary, the present invention provides an innovative solution for the fake news detection task, which has important application prospects and practical significance. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] Figure 1 This is a flow chart of the multimodal fake news detection model based on the dynamic propagation social graph of the present invention.

[0081] Figure 2 This is an example of a news heterogeneous graph.

[0082] Figure 3 This is an architecture diagram of the multimodal fake news detection model based on a dynamic propagation social graph of the present invention.

[0083] Figure 4 The accuracy changes with the learning rate on the three data sets.

[0084] Figure 5 The accuracy changes with batch size on the three datasets.

[0085] Figure 6 The accuracy varies with the unified projection dimension on the three datasets.

[0086] Figure 7 (a)-(c) are the changes in accuracy with loss weight on PHEME, PolitiFact and GossipCop, respectively.

[0087] Figure 8 (a)-(c) are the comparisons of experimental results between each variant and the DPSG model of the present invention on PHEME, PolitiFact and GossipCop, respectively. DETAILED DESCRIPTION

[0088] The specific implementation of the present invention is described in detail below in conjunction with the accompanying drawings and embodiments. This embodiment is implemented based on the technical solution of the present invention so that technicians in the relevant field can better understand the technical features and functional features of the present invention, but the protection scope of the present invention is not limited to the following embodiments.

[0089] A multimodal fake news detection model based on dynamic propagation social graph (DPSG), with Figure 1 The flowchart of the algorithm of the present invention is shown in FIG. Figure 3 The figure shows the architecture diagram of the multimodal fake news detection model based on dynamic propagation social graph of the present invention, which includes five parts: propagation sequence capture layer, feature representation layer, multimodal dynamic fusion, enhanced graph structure extraction and neighbor semantic perception. In this embodiment, the present invention uses the Pytorch platform to run all experiments on a server equipped with Intel Core i7-8700K CPU3.70GHz×12, GeForce RTX 2080Ti GPU and Ubuntu 18.04.3LTS.

[0090] Three datasets are used to evaluate the performance of the proposed model on the fake news detection task, namely PHEME, PolitiFact and GossipCop. The PHEME dataset consists of tweets of different breaking news on Twitter. The PolitiFact and GossipCop datasets are from FakeNewsNet. Both datasets include rich text, images and other information. The statistical information of the dataset after deleting blank news is shown in Table 1.

[0091] Table 1 Dataset statistics

[0092] Dataset PHEME PolitiFact GossipCop Fake News 2402 385 4913 Real News 4023 401 15446 Total news 6425 786 20359 Posts 98929 440467 1192766 user 51043 468210 429628 Total number of nodes 156397 909463 1642753

[0093] For the feature representation layer, the text of the dataset is processed by the RoBERTa model, and the news text is further processed by the T5 model to obtain the embedding vector, with an output dimension of 768. For the images in the dataset, the ResNet model is used to obtain the embedding vector, with an output dimension of 512. For the RWR method of the PSG layer, p=0.5, T==10000. The attention heads of the multi-head attention mechanism of the MDF and EGSE layers are h=8. For the encoder-decoder structure layer of the Transformer of the NSG layer, it is set to 1 on the PHEME and Gossipcop datasets, and to 6 on the PolitiFact dataset. During the experiment, 10% of the dataset is used as the test set of the experiment, and the remaining 90% is divided into a training set and a validation set in a ratio of 4:1. The experimental results select the experimental results with the best performance among the five running results. Each model is trained for a maximum of 40 times, and the patience value is set to 5.

[0094] The following models are used as baselines for comparative experiments with the DPSG model of the present invention to verify the superiority of the model in the fake news detection task:

[0095] 1) SVM-RBF: SVM classifier with RBF kernel function is used to classify manually extracted features for rumor detection.

[0096] 2) DTC: Use artificially extracted dataset features to train a decision tree classifier to evaluate the authenticity of the message.

[0097] 3) HetGNN: Build a BiLSTM-based graph convolutional network model to simultaneously learn heterogeneous structures and node information in heterogeneous graphs.

[0098] 4) IARNet: Constructs a two-level attention mechanism to learn heterogeneous graph information and uses graph convolution for fake news detection.

[0099] 5) HMGNN: This model constructs super nodes to connect global and local heterogeneous graphs, pays attention to different edge types in heterogeneous graphs, and uses graph convolutional networks for fraud detection.

[0100] 6) BERT: A semantic processing model based on Transformer bidirectional encoder.

[0101] 7) HGT: This model obtains graph structure features based on Transformer for specific types of nodes and edges in heterogeneous graphs.

[0102] 8) HetTransformer: Utilizes the Transformer structure to capture the global contextual semantics of news propagation patterns in heterogeneous graphs for fake news detection.

[0103] 9) EBGCN: To address the unreliable problem of propagation structure in social media, a Bayesian approach is used to build a robust graph structure in the rumor detection process.

[0104] 10) MFAN: Using GAT to learn the interactive fusion of text, visual and social graph multimodal features for rumor detection.

[0105] Accuracy is often used to evaluate the performance of fake news prediction models, and is defined as the ratio of the number of correct predictions to the total number. However, considering the imbalanced distribution of positive and negative examples in the dataset, the Precision, Recall, and F1 scores of fake news and real news are further used to evaluate the experimental results of the model. P and N are used to indicate whether a given news is real news or fake news, respectively, and T and F are used to indicate whether the model predicts that a given news is real news or fake news, respectively. The evaluation standard formulas are shown in (22)-(24).

[0106]

[0107] In order to optimize the performance and generalization ability of the DPSG model, the learning rate α, batch size, unified projection dimension d, L a and L c The loss weight λ a and λ c Five hyperparameters were experimented on three data sets. The specific process is as follows:

[0108] 1) The learning rate in the model determines the step size in each iteration. A too small learning rate will cause the gradient to disappear, while a too large learning rate will cause the gradient to be too large. A suitable learning rate can make the model converge quickly on the training set and achieve good performance on the test set. During the experiment, the learning rate α was set to 10 -i , i∈[1,5], and keep other hyperparameters unchanged. The experimental results are shown in the attached Figure 4 As shown in Figure 2, on the three datasets, the accuracy of the model increases first and then decreases with the increase of i value, which shows that too small and too large learning rates will affect the performance of the model. -2 When α=10, the accuracy reaches the highest value of 95.5% and 97.5% respectively. On the PolitiFact dataset, when α=10 -4 The model performance is optimal only when . This shows that for complex data sets (such as PHEME and GossipCop), the model may need a larger learning rate to better fit the data and accelerate model convergence. The PolitiFact data set has only 786 news nodes and is relatively simple. A smaller data set can better control the model training process and prevent the model from overfitting.

[0109] 2) Set the batch size value to 2 i ,i∈[2,6]. The experimental results are shown in the attached Figure 5 As shown in the figure, on the three datasets, the model performance first improves and then deteriorates with the increase of batch size. This may be because when the batch size is too small, the model parameter update direction will be inaccurate, affecting the convergence performance of the model. When the batch size is too large, the model is prone to fall into local minima and ignore the global minimum. On the PHEME, PolitiFact, and GossipCop datasets, the model performance is optimal when the batch size values ​​are set to 8, 32, and 16, respectively. Among the three datasets, the PHEME dataset has the least number of nodes, and a smaller batch size helps the model update parameters frequently, improving the sensitivity and generalization ability of the model. The PolitiFact dataset is medium, and a larger batch size may help the model make more full use of data for training. The GossipCop dataset has the largest number of nodes, and a medium batch size can effectively balance the performance and efficiency of the model.

[0110] 3) Fix the values ​​of other hyperparameters and uniformly project the relationship between dimension d and model performance on the three datasets as shown in the attached figure. Figure 6 As shown, d∈[64,256]. Figure 6 It can be seen that when d∈[64,128], the model accuracy increases with the increase of dimension, which indicates that when the dimension is too small, the model will be underfitted. When d∈[128,256], the model accuracy decreases with the increase of dimension, which may be due to the overfitting problem of the model caused by the large dimension. When d=128, the PHEME, PolitiFact and GossipCop datasets all achieved the highest accuracy, which are 95.5%, 98.5% and 99.1% respectively.

[0111] 4) The overall model loss is determined by the cross entropy loss function L c And the MSE loss function L a It consists of two parts, and the loss weights λ for the two parts a and λ c We conducted hyperparameter experiments and selected 11 combinations to explore the impact of different weights on model performance. The experimental results on three datasets are shown in the attached figure. Figure 7 As shown. With the loss weight λ a and λ cThe model accuracy varies between [0.895, 0.985] on the PolitiFact dataset, and is relatively stable on the PHEME and GossipCop datasets, ranging from [0.948, 0.955] to [0.985, 0.991]. The hyperparameter combinations (1.8, 1.2), (2.15, 1.55), and (2.5, 1.4) achieve the best results of 95.5%, 98.5%, and 99.1% on the PHEME, PolitiFact, and GossipCop datasets, respectively. Figure 7 It is found that on each data set, the accuracy of the model fluctuates in a relatively stable range, which indicates that the model is relatively robust and has a certain generalization ability.

[0112] The effect of the present invention can be verified by the following performance analysis:

[0113] 1. Comparative experiment

[0114] Table 2 shows the experimental results of the accuracy, precision, recall and F1 score of the DPSG of the present invention and the baseline model on three data sets. The best results are marked in bold. The following analysis results can be obtained from the data in Table 2:

[0115] 1) Methods using feature engineering (such as HetGNN, HetTransformer, and MFAN) are stronger in predicting fake news than methods using manually extracted dataset features (SVM-RBF and DTC). The method using manually extracted features has the highest accuracy of 71.3%. This may be because the manually extracted features focus more on capturing features from the news content and ignore the contextual semantic information of the news dissemination pattern.

[0116] 2) Among the GNN-based methods, the DPSG method achieved better experimental results in terms of accuracy, fake news F1 score, and real news F1 score on the three datasets than the HetGNN, IARNet, HMGNN, and MFAN methods. On the PolitiFact dataset, the accuracy, fake news F1 score, and real news F1 score were 6%, 6.6%, and 5.4% higher than the top-ranked HetGNN method, respectively. The reason why the experimental results of the DPSG and HetTransformer methods are better than those of the GNN-based methods may be that the encoder-decoder structure of the Transformer can more effectively integrate multi-domain information such as news titles, content, and images.

[0117] 3) When capturing news propagation patterns, models that construct heterogeneous nodes and edges (such as DPSG, HetGNN, HGT, and HetTransformer) have better performance in predicting fake news tasks than models that construct homogeneous nodes and heterogeneous edges (such as HMGNN).

[0118] 4) The reason why DPSG is better than the HetGNN method in capturing timing information may be that the HetGNN method uses BiLSTM to process sequence information. At this time, the time intervals between nodes in the sequence are the same, and the time differences between nodes are not captured.

[0119] 5) Among the Transformer-based methods, DPSG outperforms BERT, HGT, and HetTransformer on three datasets. On the PHEME dataset, DPSG is 13%, 16.5%, and 10.8% higher than the top-ranked HetTransformer in accuracy, F1 score of fake news, and F1 score of real news, respectively. The BERT method only uses the Transformer to capture news text features while ignoring visual information and propagation patterns, HGT only feeds text features instead of multimodal features into the encoder, and HetTransformer only captures global semantic information in propagation patterns while ignoring local semantic information and dynamic feature fusion.

[0120] 6) On the PHEME, PolitiFact, and GossipCop datasets, the DPSG method of the present invention achieved 95.5%, 98.5%, and 99.1% accuracy, respectively, which is a 6.2% improvement in accuracy compared to the optimal baseline. This shows that DPSG can capture more comprehensive features for fake news prediction tasks by considering the dynamic fusion between multiple modalities and the local and global contextual semantic information of the propagation mode in the news heterogeneous graph.

[0121] Table 2 Experimental results of different baselines and the proposed model on three datasets

[0122]

[0123]

[0124] 2. Ablation experiments of each module

[0125] In order to demonstrate the effectiveness of each part proposed in the DPSG model of the present invention, the model is decomposed into the following three variants: -Temporal similarity strength, -EGSE, -Transformer decoder. -Temporal similarity strength means removing the dynamic information in the temporal permeation fusion block, -EGSE means removing the graph structure features from the DPSG model, and -Transformer decoder means removing the decoder of the Transformer in the NSA module.

[0126] The experimental results are shown in Table 3, from which it can be observed that:

[0127] 1) In general, the performance of each variant is worse than that of the DPSG model, with the accuracy dropping from 0.7% to 7.7%. The experimental results of each variant and DPSG show the effectiveness of each module in improving the fake news prediction performance.

[0128] 2) Removing the temporal similarity intensity leads to a decrease in the performance of the model on the three datasets, with the accuracy dropping by 0.7%, 6.2%, and 1.2%, respectively. This shows that considering dynamic information when acquiring social context semantic information is in line with reality and plays a positive role in the fake news detection task.

[0129] 3)-EGSE variants perform worse than the overall model on the three datasets, decreasing by 2.3%, 7.7%, and 3.1% on fake news F1, and by 1.2%, 7.8%, and 1% on real news F1. This shows that the EGSE module can effectively learn local context semantics and capture the graph structure characteristics of the target news.

[0130] 4)-Transformer decoder experimental results show that the use of Transformer decoder structure can capture the global semantic information of the propagation sequence and promote the fusion of features between neighbor nodes and target nodes. Using only the output of the encoder for prediction leads to a decrease in model performance. Specifically, the accuracy on the three datasets decreased by 0.8%, 7.7% and 1.8% respectively.

[0131] Table 3 Ablation experimental results of modules in the proposed model

[0132]

[0133] 3. Ablation Experiment of Enhanced Graph Structure Extraction Module

[0134] In order to further evaluate the effectiveness of each part of the EGSE module, the following five variants are proposed: -Enhancement, -DSA, -DSA (positive), -DSA (negative), and -Alignment. -Enhancement means that the incompleteness of the data is not considered in the graph structure enhancement part. -DSA means that symbolic attention weighting is not used, and -DSA (positive) and -DSA (negative) respectively mean that the positive correlation and negative correlation between nodes are not considered when performing symbolic attention weighting. -Alignment means that the modal alignment between graph structure features and text-visual fusion features is not considered. The experimental results of each variant and the DPSG model of the present invention on three data sets are compared in the attached figure. Figure 8 As shown. Figure 8 The following observations can be obtained from the experimental results:

[0135] 1) In general, removing any part of the EGSE module will affect the performance of the model in fake news detection, and the accuracy rate drops from 0.5% to 12.3%. The necessity of considering graph structure features has been verified. The designed way to capture graph structure features has also been proven to be effective.

[0136] 2) When capturing graph structural features, not considering the problem of data incompleteness will cause the model performance to drop by 0.7%, 3.1%, and 3.6% on the three datasets, respectively.

[0137] 3) After removing the dynamic symbol attention block, the model accuracy dropped from 1.2% to 5%. Removing positive correlations reflects lower performance compared to removing negative correlations, with the accuracy dropping by at least 0.8% on the three datasets. This indicates that positive correlations are more beneficial to improving model performance.

[0138] 4) After removing the modal alignment between text-visual features and graph features, the -Alignment variant performs worse than the overall model, especially on the PolitiFact dataset, where the accuracy, fake news F1, and real news F1 indicators dropped by 12.3%, 12.4%, and 12.7%, respectively. This shows that the design of graph modal alignment is effective in promoting the fusion between graph structural features and text-visual features.

[0139] It should be further explained that the above implementation modes are only used to further illustrate the technical solution of the present invention, and are not used to limit the protection scope of the present invention. Any obvious adjustments and modifications made to the present invention within the technical concept of the present invention should fall within the protection scope of the present invention.

Claims

1. A multimodal fake news detection method based on dynamic propagation social graph is implemented through a multimodal fake news detection model. The multimodal fake news detection model includes five parts: propagation sequence capture layer, feature representation layer, multimodal dynamic fusion, enhanced graph structure extraction and neighbor semantic perception. It is characterized by The steps include: Step 1: Propagation sequence capture layer: Construct a news heterogeneous graph G = (V, E) based on the three types of entities in the dataset: news, posts, and users, where V = {N, P, U} represents the node set of the three node types of news, posts, and users, and E represents the edge set between them. m n 、m p and m u Represents the number of news nodes, post nodes and user nodes respectively, traversing the news node n i ∈N, for each starting node n i Perform the following steps T times: return to the starting node with probability p, or randomly select n i Neighbor nodes join N(n i ) set until the specified number of neighbor nodes of the corresponding type is collected to ensure that each type of neighbor node is sampled. i ) News neighbor nodes N in the set n 、Post neighbor node N p and user neighbor node N u Sort by sampling frequency and keep the first k neighbor nodes with the highest frequency, k = k n +k p +k u ; Step 2: Feature representation layer: First, the text is padded or truncated into a token sequence of length n using a tokenizer T = {[CLS], o1, o2, ..., o n ,[SEP]}, then input T into the fine-tuned RoBERTa model to obtain the embedding vector, and use the embedding vector of [CLS] to represent the embedding vector of the entire token sequence e, d t represents the embedding dimension, and the following standardized embedding vector representations are obtained in turn: News title e t 、News content c , post information p , user description u For visual feature representation, the pre-trained ResNet18 model is used to obtain fine-grained image feature vectors, and the image representation is obtained after normalization. where d v Represents the dimension of the image feature vector, some numerical attributes and category attributes, and uses the one-hot method to convert them into sparse feature representation. The numerical attributes are normalized to ensure the same numerical range, and other feature vectors of the post are obtained Other feature vectors of users where d op , d ou Represent the dimensions of other feature vectors of posts and users respectively; Step 3: Multimodal dynamic fusion: In order to accurately obtain the feature representation of each type of node in the heterogeneous graph, a multimodal dynamic fusion layer is designed, which consists of two parts: temporal similarity strength and temporal penetration fusion blocks; Step 4: Enhanced graph structure extraction: In the news heterogeneous graph, there are complex relationships between nodes and edges. In order to deeply extract graph structure features, the target news and its neighbor sequences captured in the propagation trajectory are enhanced in graph structure. Then, a dynamic symbol attention block is established to deeply extract graph structure features between neighbors and between neighbors and target nodes. Finally, the graph structure features and fusion features are multimodally aligned to achieve the fusion and interaction of multimodal information among text, vision and graph structure. Step 5: Neighborhood semantic perception: After obtaining the fusion features and graph structure features of the target news and its various types of neighbors, the encoder-decoder structure of Transformer is used to perceive the contextual semantics of the nodes in the news propagation sequence to obtain the perceptual features of the target news for training the classifier.

2. The method according to claim 1, characterized in that In the temporal similarity strength in step 3: among the news neighbor nodes and post neighbor nodes of the target news node, the closer the neighbor nodes are to the target news node, the more their content can reflect the social context of the target news node when it is released. Therefore, the temporal similarity strength is designed, and the news neighbor N n The oth neighbor in the target news n i Time interval The calculation is shown in formula (1) in Indicates target news n i The release time, t o Indicates news neighbor N n The publishing time of the oth neighbor in ; News Neighbor N n The oth neighbor in the target news n i The temporal similarity strength As shown in formula (2) Using the above method, we can obtain the news neighbors and post neighbors for the target news n i The temporal similarity strength s n 、s p ,in 3. The method according to claim 1, characterized in that In step 4, graph structure enhancement: The extraction of graph structure features mainly involves two parts of information: the information contained in the node itself and the relationship between nodes. For the information part of the node itself, the fusion features obtained by the multimodal dynamic fusion module are used to obtain the fusion features of the target news. Fusion features with its neighboring nodes According to the propagation trajectory, the sequence N(n i ) are spliced ​​together in order to form the target news n i The dynamic neighbor embedding sequence E fus ∈R l×d , the first line in the sequence is the target news n i Fusion features, where l = 1 + k n +k p +k u , the relationship between nodes is represented by building an adjacency matrix. The construction of the adjacency matrix mainly obtains the complex connection relationship between nodes from two perspectives. One is to find the connection relationship between l nodes in the heterogeneous graph G, and the other is to supplement the edge connection with the help of cosine similarity. The cosine similarity between each node is calculated according to the dynamic neighbor embedding sequence, and the nodes with cosine similarity greater than 0.5 are also added with edge connections. After supplementation, an l×l adjacency matrix A is constructed. If there is an edge between two nodes, it is marked as 1, otherwise it is marked as 0. The supplemented adjacency matrix contains a more comprehensive connection relationship between nodes.

Citation Information

Patent Citations

  • False news detection method and device, electronic equipment and storage medium

    CN113469214A

  • False news detection method and system based on prompt learning and similar perception fusion

    CN116226384A

Cited By

  • A method for reconstructing and tracing a cross-platform false information propagation tree based on a heterogeneous propagation graph

    CN122986133A