A fake news detection method based on graph convolutional neural network

Abstract

This invention discloses a fake news detection method based on graph convolutional neural networks (GCNNs), belonging to the field of GCNN machine learning technology. The method includes the following steps: S1, using a pre-trained model to extract image modality feature sets and text modality feature sets from the news set to be detected; S2, using the image modality features and text modality features as nodes in the initial undirected graph to construct initial undirected graphs for the image modality and text modality respectively, the initial undirected graphs being represented by an initial adjacency matrix and node features; S3, inputting the initial undirected graphs into a graph structure learning network for graph structure optimization to obtain an optimized graph structure; S4, inputting the optimized graph structure into a two-layer cross-modal shared weight graph convolutional neural network to finally obtain cross-modal features; S5, inputting the obtained cross-modal features into a modality discriminator to learn modality-invariant features; S6, inputting the modality-invariant features obtained in S5 into a classifier to obtain the probability that the news item is fake news.

Description

Technical Field

[0001] This invention belongs to the field of graph convolutional neural network machine learning technology, specifically relating to a fake news detection method based on graph convolutional neural networks. Background Technology

[0002] Fake news detection technology refers to a technical means that uses computer technology and data science methods to automatically analyze and judge news content in order to identify reports that fail to reflect the true nature of objective facts and contain false elements. Specifically, fake news refers to "news" reported based on false "news facts," lacking any objective factual basis. Specifically, fake news detection technology combines various algorithms and models such as natural language processing, machine learning, and deep learning. It analyzes and fuses multimodal information from news text and images to extract key features, and then uses a classifier to classify and evaluate whether the news is true or false based on these features. However, in practical applications, many fake news detection methods fail to fully extract and analyze the text and image information of the news, resulting in poor learned representations, which is detrimental to the fake news detection task. Furthermore, many fake news detection methods fail to recognize the significant modal differences between image and text information, leading to low accuracy in subsequent fake news detection tasks.

[0003] Graph Convolutional Neural Networks (GNNs) are a machine learning technique whose core idea is to extend convolution operations to graph data. They utilize the adjacency relationships between nodes for information transfer and feature extraction. Specifically, GNNs use the feature information of a node's neighboring nodes to update the representation of each node, and then gradually aggregate information from a wider range of neighboring nodes through multiple convolutional operations. However, in practical applications, GNNs are highly sensitive to graph structure. Excessive noise in the graph structure or incorrect connections in the original graph will result in poor representations learned by the GNN, hindering subsequent tasks such as fake news detection. Existing graph structure optimization methods use attention mechanisms or feature similarity to reweight the original graph's connections without altering the existing connections, thus failing to optimize the original graph to its optimal state.

[0004] Therefore, reducing the interference of noisy data in graph structures on subsequent fake news detection tasks, and minimizing the differences between modalities, remains a challenging problem. Summary of the Invention

[0005] Purpose of the invention: To address the problem that existing fake news detection methods fail to adequately extract and analyze the text and image information of news and fail to notice the significant modal differences between images and text information, this invention provides a fake news detection method based on graph convolutional neural networks.

[0006] This invention proposes a highly efficient graph structure optimization module based on a graph convolutional neural network. This module effectively supplements the latent connections of the original graph, removes unreasonable connections, and optimizes the weight relationships of existing connections, enabling the graph neural network to fully extract and analyze the textual and image information of news. Simultaneously, to mitigate intermodal differences and learn invariant representations between different modalities, this invention also designs a modality discriminator and enhances the discriminative ability of features through cross-modal supervised contrastive loss. Through the modality discriminator, this invention reduces modality differences and ultimately improves the accuracy of fake news detection.

[0007] Technical solution: A fake news detection method based on graph convolutional neural networks, comprising the following steps:

[0008] S1. Use a pre-trained model to extract the image modal feature set and text modal feature set of the news collection to be detected;

[0009] S2. Using the image modal features and text modal features obtained in S1 as nodes of the initial undirected graph, construct the initial undirected graphs for the image modality and the text modality respectively. The initial undirected graphs are represented by the initial adjacency matrix and node features.

[0010] S3. Input the initial undirected graph obtained in S2 into the graph structure learning network to optimize the graph structure and obtain the optimized graph structure.

[0011] S4. Input the optimized graph structure from S3 into a two-layer graph convolutional neural network with shared weights across modalities to finally obtain cross-modal features;

[0012] S5. Input the cross-modal features obtained in S4 into the modality discriminator to learn modality-invariant features, further mitigating the differences between modalities;

[0013] S6. Input the modality-invariant features obtained in S5 into the classifier to obtain the probability that the news is fake news, and use this to determine if it is fake news.

[0014] Furthermore, in S1:

[0015] 1) For text data in news collections, a bidirectional encoder is used to represent the pre-trained model to extract its text modal features;

[0016] 2) For image data in the news collection, a multimodal contrastive learning pre-trained model is used to extract its image modal features.

[0017] Furthermore, the specific operation method of S1 is as follows:

[0018] 1) Assume the news collection consists of n news items containing text and images, t iv represents the text in the i-th news item. i Let y represent the image in the i-th news item. i The set of text modalities representing the tags of the i-th news item is denoted as T = [t1, ..., t2]. i ,…,t n The set of image modalities is denoted as V = [v1, ..., v]. i ,…,v n ];

[0019] 2) For the i-th news item, where C indicates the type of news tag. Let Y represent a C-dimensional real vector space; the tag matrix Y corresponding to the news collection is represented as: Let represent a C×n dimensional real matrix space;

[0020] 3) For the i-th news item, t i The input is fed into a bidirectional encoder to represent a pre-trained model to extract its textual modal features. i ,in d u The dimension of the text modal features; the set U of news text modal features is represented as:

[0021]

[0022] 4) For the i-th news item, v i Input to a multimodal contrastive learning pre-trained model to extract its image modal features q i ,in d q The dimension of the image modal features; the set Q of news image modal features is represented as:

[0023]

[0024] Furthermore, in S2:

[0025] 1) Construction of the initial undirected graph of the image modality: For each sample feature in the set of image modality features, calculate its cosine similarity with any sample feature in the set, and connect the nodes with a cosine similarity greater than a preset threshold to construct the initial undirected graph of the image modality.

[0026] 2) Construction of the initial undirected graph of the text modality: For each sample feature in the set of text modality features, calculate its cosine similarity with any sample feature in the set, and connect the nodes with a cosine similarity greater than a pre-set threshold to construct the initial undirected graph of the text modality.

[0027] Furthermore, the specific operation method of S2 is as follows:

[0028] Step 1) Represent the feature extractor F as F(X,θ) f ), where X∈{U,Q}, is used to represent the set U of text modal features or the set Q of image modal features extracted from the news collection by the pre-trained model, θ f This represents the parameters to be learned, in order to further extract features;

[0029] Step 2) Calculate the pairwise sample features u in sets U and Q respectively. i and u j q i and q j The cosine similarity between nodes, where i, j∈[1,n]; nodes with a cosine similarity greater than a set threshold are considered. Connect to get an edge Thus, the initial undirected graph δ is constructed. X ; Initial undirected graph δ X Represented as δ X =(Λ) X ,Ψ X ), Λ X Represents nodes in an undirected graph The set of Ψ X Represents the edges in an undirected graph The set of; the initial undirected graph δ X The initial adjacency matrix is ​​represented as A X .

[0030] Furthermore, the specific steps of S3 are as follows:

[0031] 1) Input the initial adjacency matrix and node features obtained in S2 into a two-layer graph convolutional neural network of the graph structure learning network to obtain node embeddings;

[0032] In each iteration, A X The input X is fed into a two-layer graph convolutional neural network, and the obtained node features Z are processed. X Represented as Z X =GCN Z (A X ,X), Where d z This represents the dimension of the embedding, where n represents the number of node embeddings.

[0033] 2) Transpose the node embedding matrix, and then multiply the transpose matrix with the node embedding matrix to obtain the similarity matrix S. X ;

[0034] 3) For dense similarity matrices S XK-nearest neighbor sparsification is performed, meaning that for each node, only the edges connecting the K nodes with the highest feature similarity to that node are retained. The final similarity matrix is ​​as follows:

[0035] 4) The optimized adjacency matrix From the initial adjacency matrix A X and the calculated similarity matrix The summation of the elements forms the structure, i.e.

[0036] Furthermore, in S4:

[0037] The set B of latent feature representations X Represented as: Where X∈{U,Q}, and GCN B (·,·) represents a two-layer graph convolutional neural network used to learn feature representations. B X The i-th feature in B represents U B represents the cross-modal feature representation of the text modality. Q A cross-modal feature representation of an image modality.

[0038] Furthermore, in S5:

[0039] 1) Modal discriminator;

[0040] The modality discriminator D consists of a single-layer neural network and a normalized exponential function, denoted as D(M; θ) d ), where θ d The input parameters, the set of input text and image features M, are represented as follows:

[0041]

[0042] Where, d k The dimension of a mode, m i This represents the i-th feature in set M;

[0043] The output P of the modality discriminator θ (m i )=D(m i ;θ d ) is m i The probability of a text feature or an image feature; the modality label of the feature is used. It means that among them Represents set Y m The modality label of the i-th feature. m i Belongs to the text modality. m iBelongs to image modality; loss L of modality discriminator m It is expressed as follows:

[0044]

[0045] in, Describes the distribution from (M, Y) m Random variables obtained by sampling from ) To calculate the expected value of the mathematical expression;

[0046] 2) Introduce cross-modal supervised contrast loss into the modal discriminator;

[0047] For m i Let P(i) be a term in M ​​that is connected to m. i A(i) is the set of indexes of features of the same category in M, excluding P(i). The cross-modal supervised contrastive loss L... S It is expressed as follows:

[0048]

[0049] Where |P(i)| represents the number of features in the index set of P(i), and τ = 0.5.

[0050] Furthermore, in step S6:

[0051] 1) Classifier:

[0052] Classifier It consists of a single-layer neural network and a normalized exponential function, where θ c For learnable parameters, Represents the mode-invariant features obtained from S5; the output of the classifier. for The probability of a news item being fake or real is calculated. If the probability of it being fake is greater than the probability of it being real, then the news item is considered fake; otherwise, it is considered real.

[0053] Classifier loss L c Represented as:

[0054]

[0055] in, Indicates the distribution from Random variables obtained by sampling To calculate the expected value of the mathematical expression;

[0056] 2) Regarding the parameter θ of the feature extractor f The parameters θ of the modality discriminator d The parameters θ of the classifier cOptimizations will be made; details are as follows:

[0057] According to L m L s L c The total loss L can be obtained. total L total Represented as: L total =L c +βL s -ηL m , where β and η are equilibrium factors;

[0058] By maximizing the loss mode discriminator loss L m To update parameter θ d θ f θ c This mitigates the modal discrepancy between text modal features and image modal features. A gradient inversion layer is employed before the modality discriminator, and the parameters are optimized using stochastic gradient descent. The optimization process is as follows:

[0059]

[0060] in, Represents the optimized θ f θ c ;

[0061]

[0062] in, Represents the optimized θ d .

[0063] This invention also provides a multimodal fake news detection system based on a graph convolutional neural network for fake news detection, the multimodal fake news detection system comprising:

[0064] 1) Modal data preprocessing module, including a multimodal contrastive learning pre-trained model for extracting image modal features of the news to be detected and a bidirectional encoder representation pre-trained model for extracting text modal features of the news to be detected;

[0065] 2) Graph structure learning network, including an initial undirected graph construction module for constructing initial undirected graphs for image modalities and text modalities, and a graph structure optimization module for optimizing the obtained initial undirected graphs;

[0066] 3) The modality-invariant feature learning module consists of two layers of graph convolutional neural networks with shared weights across modalities and a modality discriminator. The optimized graph structure is first input into the two layers of graph convolutional neural networks with shared weights across modalities to obtain cross-modal features. The cross-modal features are then input into the modality discriminator to learn modality-invariant feature representations.

[0067] 4) Fake news detection module, including a classifier, inputs modality-invariant feature representations into the classifier to obtain the probability that the news item belongs to fake news or real news.

[0068] Beneficial effects:

[0069] 1) The optimized graph adjacency matrix of this invention supplements the potential connection relationships in the original graph, deletes unreasonable connections, and optimizes the weight relationship of the original connections, so that the graph neural network can learn features better, which is beneficial to improving the accuracy of fake news detection tasks.

[0070] 2) This invention learns modality-invariant feature representations by using a generative adversarial network composed of a modality discriminator and a graph structure learning network. This reduces the modality differences between text features and image features, resulting in better representation of the learned modality-invariant features, which is beneficial for the classifier to classify and evaluate fake news. Attached Figure Description

[0071] Figure 1 This is a flowchart illustrating the steps of the fake news detection method based on graph convolutional neural networks in this invention.

[0072] Figure 2 This is a schematic diagram of the graph structure optimization network for fake news detection based on graph convolutional neural networks in an embodiment of the present invention;

[0073] Figure 3 This is a schematic diagram of the modal data preprocessing module and the graph structure learning network in the embodiment of the present invention;

[0074] Figure 4 This is a structural diagram of the modality discriminator in an embodiment of the present invention;

[0075] Figure 5 This is a diagram of the classifier structure in this invention, which consists of a single-layer neural network and a normalized exponential function (Softmax activation function). Detailed Implementation

[0076] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings, but the scope of protection of the present invention is not limited to the embodiments described.

[0077] like Figure 1 As shown, this invention provides a fake news detection method based on graph convolutional neural networks, comprising the following steps:

[0078] S1. Obtain multimodal data of the news collection to be detected: Use a pre-trained model to extract the image modal feature set and text modal feature set of the news collection to be detected.

[0079] Specifically: 1) For text data in the news collection, a bidirectional encoder representation pre-trained model (BERT) is used to extract its text modal features; 2) For image data in the news collection, a multimodal contrastive learning pre-trained model (CLIP) is used to extract its image modal features.

[0080] The specific operation method of S1 is as follows:

[0081] 1) Assume the news collection consists of n news items containing text and images, t i v represents the text in the i-th news item. i Let y represent the image in the i-th news item. i The set of text modalities representing the tags of the i-th news item is denoted as T = [t1, ..., t2]. i ,…,t n The set of image modalities is denoted as V = [v1, ..., v]. i ,…,v n ].

[0082] 2) For the i-th news item, where C indicates the type of news tag. Let Y represent a C-dimensional real vector space. The tag matrix Y corresponding to the news collection is represented as: Let represent a C×n dimensional real matrix space.

[0083] 3) For the i-th news item, t i The input is fed into the Bidirectional Encoder Representation Pretrained Model (BERT) to extract its textual modal features. i ,in d u The dimension of the text modal features is represented by: The set U of news text modal features is represented as:

[0084] 4) For the i-th news item, v i Input to a multimodal contrastive learning pre-trained model (CLIP) to extract its image modal features q i ,in d q Let Q represent the dimension of the image modal features. The set of modal features for news images is represented as:

[0085] S2. Using the image modal features and text modal features obtained in S1 as nodes of the initial undirected graph, construct the initial undirected graphs of the image modality and the text modality respectively. The initial undirected graphs are represented by the initial adjacency matrix and node features.

[0086] 1) Construction of the initial undirected graph of the image modality: For each sample feature in the set of image modality features, calculate its cosine similarity with any sample feature in the set, and connect the nodes with cosine similarity greater than a pre-set threshold to construct the initial undirected graph of the image modality.

[0087] 2) Construction of the initial undirected graph of the text modality: For each sample feature in the set of text modality features, calculate its cosine similarity with any sample feature in the set, and connect the nodes with a cosine similarity greater than a pre-set threshold to construct the initial undirected graph of the text modality.

[0088] The specific operation method of S2 is as follows:

[0089] Step 1) In this invention, the feature extractor F is represented as F(X,θ) f ), where X∈{U,Q}, is used to represent the set U of text modal features or the set Q of image modal features extracted from the news collection by the pre-trained model, θ f This represents the parameters to be learned in order to further extract features.

[0090] Step 2) Calculate the pairwise sample features u in sets U and Q respectively. i and u j q i and q j The cosine similarity between nodes, where i, j∈[1,n]; nodes with a cosine similarity greater than a set threshold are considered. Connect to get an edge Thus, the initial undirected graph δ is constructed. X .

[0091] Each sample feature is represented as a node in the initial undirected graph. The cosine similarity between pairwise sample features is represented as nodes on the initial undirected graph. The connection relationships, i.e., the edges of the initial undirected graph. Where x∈{u,q}, is used to represent the preprocessed text modal feature u or image modal feature q.

[0092] Initial undirected graph δ X Represented as δ X =(Λ) X ,Ψ X ), Λ X Represents nodes in an undirected graph The set of Ψ X Represents the edges in an undirected graph A set of.

[0093] Initial undirected graph δ X The initial adjacency matrix is ​​represented as AX A X Given an n×n matrix, when the node... There are edges season Conversely in Denotes the initial adjacency matrix A X The element in the i-th row and j-th column.

[0094] S3. Optimize the graph structure: Input the initial undirected graph obtained in S2 into the graph structure learning network to optimize the graph structure and obtain the optimized graph structure.

[0095] Graph neural networks are highly sensitive to graph structures. However, the initial undirected graph obtained through S2 may contain inappropriate connections, making it difficult to achieve good representations in subsequent graph neural network learning. To ensure that the graph structure learning part obtains the optimal representation of the graph structure, thus better serving subsequent graph representation learning, S3 inputs the adjacency matrix and node features of the initial undirected graph into the graph structure learning network to obtain a similarity matrix. Finally, the initial adjacency matrix and the similarity matrix are added to obtain the optimized adjacency matrix. This operation allows the optimized graph structure to remove unreasonable connections and optimize the weight relationships of the original connections.

[0096] The specific steps for S3 are as follows:

[0097] 1) Input the initial adjacency matrix and node features obtained in S2 into a two-layer graph convolutional neural network of the graph structure learning network to obtain node embeddings.

[0098] To adjust the connection relationships and connection weights, in each iteration, A is... X The input X is fed into a two-layer graph convolutional neural network, and the obtained node features Z are processed. X Represented as Z X =GCN Z (A X ,X), (GCN Z (·,·) represents a two-layer graph convolutional neural network used to obtain node features. Where d z This represents the dimension of the embedding, and n represents the number of embedded nodes.

[0099] 2) To measure the similarity between features, the node embedding is first transposed, and then the transposed matrix is ​​multiplied by the node embedding matrix to obtain the similarity matrix.

[0100] This invention first focuses on the node feature Z X Transpose the matrix to obtain Again ZX Performing matrix multiplication yields an n×n similarity matrix S. X S X Each element Let represent the dot product between the embeddings of the i-th node and the j-th node. Each element in the matrix can be represented as the similarity between the i-th node and the j-th node.

[0101] 3) Perform K-Nearest Neighbor (KNN) sparsification on the dense similarity matrix. That is, for each node, this invention only retains the edges formed by connecting the K nodes with the highest feature similarity to that node.

[0102] To reduce computational costs and memory usage, this invention addresses the issue of dense similarity matrices S. X Sparsity reduction techniques were employed. Specifically, for each node, this invention retains only the K elements with the highest similarity to that node, resulting in a final similarity matrix.

[0103] 4) The optimized adjacency matrix is ​​formed by summing the elements of the initial adjacency matrix and the calculated similarity matrix. The optimized adjacency matrix, obtained by adding the original adjacency matrix and the similarity matrix, is expressed as:

[0104] S4. Obtain cross-modal features: Input the optimized graph structure from S3 into a two-layer graph convolutional neural network with shared weights across modalities to finally obtain cross-modal features.

[0105] A two-layer graph convolutional neural network is used to learn cross-modal feature representations for each modality. In order to learn cross-modal features, the weight parameters of the graph convolutional neural network are shared across modalities.

[0106] The set B of latent feature representations X Represented as: Where X∈{U,Q}, and GCN B (·,·) represents a two-layer graph convolutional neural network used to learn feature representations. B X The i-th feature in B represents U B represents the cross-modal feature representation of the text modality. Q A cross-modal feature representation of an image modality.

[0107] S5. Obtain modality-invariant features: Input the cross-modal features obtained in S4 into the modality discriminator to learn modality-invariant features, further mitigating the differences between modalities;

[0108] Specifically, a generative adversarial network is constructed by using the graph convolutional neural network in S4 as the generator and the modality discriminator as the discriminator. The modality discriminator is used to mitigate modality discriminants and is designed to identify whether the input comes from an image or text modality.

[0109] Significant modal differences exist between text features and image features, which can negatively impact fake news detection. Therefore, in this step, the cross-modal features obtained through S4 are input into the modality discriminator to reduce modality differences. U and B Q It is input into the modality discriminator, where B U B represents the cross-modal feature representation of the text modality. Q This discriminator, which represents cross-modal feature representations of image modalities, aims to identify whether the input comes from a text or image modality. It learns modality-invariant feature representations through a generative adversarial network consisting of two layers of graph convolutional neural networks sharing weights across the cross-modal domains.

[0110] 1) Modal discriminator:

[0111] The modality discriminator D consists of a single-layer neural network and a normalized exponential function (Softmax activation function), denoted as D(M; θ d ), where θ d The input parameters, the set of input text and image features M, are represented as follows:

[0112]

[0113] Where, d k The dimension of a mode, m i Let i represent the i-th feature in set M.

[0114] The output P of the modality discriminator θ (m i )=D(m i ;θ d ) is m i The probability of being a text feature or an image feature.

[0115] Modal labels of features It means that among them Represents set Y m The modality label of the i-th feature. m i Belongs to the text modality. m i It belongs to the image modality.

[0116] Based on cross-entropy, the loss L of the modality discriminator m It is expressed as follows:

[0117]

[0118] in, Describes the distribution from (M, Y) m Random variables obtained by sampling from ) The expected value is calculated.

[0119] 2) This invention introduces cross-modal supervised contrast loss into the modal discriminator;

[0120] To enhance the discriminative power of the learned features, this invention designs a cross-modal supervised contrastive loss. Under the constraint of this loss, the similarity of features from the same category will increase, while the similarity of features from different categories will decrease.

[0121] For m i Let P(i) be a term in M ​​that is connected to m. i A(i) is the set of indexes of features of the same category in M, excluding P(i). The cross-modal supervised contrastive loss L... S It is expressed as follows:

[0122]

[0123] Where |P(i)| represents the number of features in the index set of P(i), and τ is a parameter set to τ=0.5 based on experience.

[0124] S6. Fake News Judgment: Input the modality-invariant features obtained in S5 into a classifier to obtain the probability that the news is fake news, and use this to judge fake news.

[0125] After all data samples pass through the modality discriminator trained in S5, stable modality-invariant features are obtained and input into the classifier for classification.

[0126] 1) Classifier:

[0127] Classifier It consists of a single-layer neural network and a normalized exponential function (Softmax activation function), where θ c For learnable parameters, This represents the mode-invariant features obtained from S5. The classifier's output. for The probability of a news item being fake or real is calculated. If the probability of it being fake is greater than the probability of it being real, then the news item is considered fake; otherwise, it is considered real.

[0128] Classifier loss L c Represented as:

[0129]

[0130] in, Indicates the distribution from Random variables obtained by sampling To calculate the expected value of the mathematical expression;

[0131] 2) Parameter optimization: Optimize the parameters θ of the feature extractor. f The parameters θ of the modality discriminator d The parameters θ of the classifier c Optimize;

[0132] According to L m L s L c The total loss L can be obtained. total L total Represented as: L total =L c +βL s -ηL m , where β and η are balance factors.

[0133] By maximizing the loss mode discriminator loss L m To update parameter θ d θ f θ c This approach mitigates the modal discrepancies between text and image modal features. A gradient inversion layer is used before the modality discriminator, and stochastic gradient descent is employed to optimize the parameters. Specifically, when the cross-modal feature representation obtained from two layers of graph convolutional neural networks sharing cross-modal weights is passed to the gradient inversion layer, since the gradient inversion layer does not alter the data during forward propagation, the feature representation is then passed to the modality discriminator to obtain modality predictions. The cross-modal supervised contrastive loss and the modality discriminator loss are then calculated based on the modality labels. The signs of the cross-modal supervised contrastive loss and the modality discriminator loss are inverted as the gradients pass through the gradient inversion layer. This means that during backpropagation, the cross-modal supervised contrastive loss increases while the modality discriminator loss decreases, resulting in improved modality discriminator performance. This leads to an increase in the parameter θ of the feature extractor. f The update reduces the cross-modal supervised contrastive loss and increases the loss of the modality discriminator, making the acquired features harder for the modality discriminator to distinguish. This adversarial training method encourages the feature extractor to learn modality-invariant features. The optimization process is represented as follows:

[0134]

[0135] in, Represents the optimized θ f θ c .

[0136]

[0137] in, Represents the optimized θ d .

[0138] Working principle:

[0139] This invention provides a multimodal fake news detection method based on graph neural networks. The entire method can be implemented in six stages: the first stage is the preprocessing of news data; the second stage is the construction of initial undirected graphs for text and image modalities; the third stage is the optimization of the graph structure; the fourth stage is the acquisition of cross-modal features; the fifth stage is the acquisition of modality-invariant features; and the sixth stage is the fake news discrimination and parameter optimization. In the preprocessing of news data, a bidirectional encoder representation pre-trained model (BERT) and a multimodal contrastive learning pre-trained model (CLIP) are used to extract news data features. In the initial undirected graph construction part, the cosine similarity between pairwise sample features is calculated, and nodes with a cosine similarity greater than a pre-set threshold are connected to construct an undirected graph. In the graph structure optimization part, an optimized graph structure and adjacency matrix are obtained through a graph structure learning network. In the cross-modal feature acquisition part, cross-modal features are learned through a two-layer graph convolutional neural network with shared weight parameters. In the modality-invariant feature acquisition part, the similarity of different categories of features is adjusted through a modality discriminator and cross-modal supervised contrastive loss to obtain invariant features between text and image modalities. In the fake news discrimination and parameter optimization part, a normalized exponential function (Softmax activation function) is used as the structure of the classifier to optimize the parameters and improve the performance of fake news detection.

[0140] Based on the multimodal fake news detection method based on graph neural networks proposed in this invention, this invention also proposes a multimodal fake news detection system, such as... Figure 2 As shown, the multimodal fake news detection system includes:

[0141] 1) Modal data preprocessing module;

[0142] For text data, this invention uses a bidirectional encoder representation pre-trained model (BERT) to extract text modal features; for image data, this invention uses a multimodal contrastive learning pre-trained model (CLIP) to extract image modal features. Furthermore, features are further extracted and modality-invariant feature representations are learned through subsequent graph structure learning networks and modality-invariant feature learning modules.

[0143] 2) The graph structure learning network consists of an initial undirected graph construction module and a graph structure optimization module. The specific methods for the undirected graph construction module and the graph structure optimization module are as follows:

[0144] The initial undirected graph construction module first calculates the cosine similarity between the features of each sample in both modes for each mode, and connects the nodes whose cosine similarity is greater than a pre-set threshold, thereby constructing an undirected graph for each mode. The initial undirected graph can be represented by an adjacency matrix and node features.

[0145] In the graph structure optimization module, in each iteration, the present invention uses a two-layer graph neural network to extract node features and calculate the similarity matrix of node features. The similarity matrix is ​​added to the original graph adjacency matrix to obtain the optimized adjacency matrix. When calculating the similarity matrix, the dense similarity matrix is ​​sparsified by K-nearest neighbors (KNN).

[0146] The structural diagrams of the modal data preprocessing module and the graph structure learning network are shown in Figure 3.

[0147] 3) The modality-invariant feature learning module consists of a two-layer graph convolutional neural network with shared weights across modalities and a modality discriminator;

[0148] First, the optimized graph is fed into a two-layer graph convolutional neural network with shared weights across modalities, ultimately yielding cross-modal features. The shared weight feature representations of text and images are then fed into a modality discriminator, designed to identify whether the input originates from an image or text modality. Through a generative adversarial network (GAN) composed of a graph structure learning network, modality-invariant feature representations can be learned. The modality discriminator structure diagram is shown below. Figure 4 As shown.

[0149] 4) The fake news detection module inputs the modality-invariant feature representation into a classifier to obtain the probability that the news item is either fake or real. If the probability of being fake is greater than the probability of being real, the news item is judged as fake; otherwise, it is real. The classifier structure diagram is shown below. Figure 5 As shown.

[0150] As described above, although the invention has been shown and described with reference to specific preferred embodiments, it should not be construed as limiting the invention itself. Various changes in form and detail may be made without departing from the spirit and scope of the invention.

Claims

1. A method for fake news detection based on graph convolutional neural network, characterized in that, Includes the following steps: S1. Use a pre-trained model to extract the image modal feature set and text modal feature set of the news collection to be detected; S2. Using the image modal features and text modal features obtained in S1 as nodes of the initial undirected graph, construct the initial undirected graphs for the image modality and the text modality respectively. The initial undirected graphs are represented by the initial adjacency matrix and node features. S3. Input the initial undirected graph obtained in S2 into the graph structure learning network to optimize the graph structure and obtain the optimized graph structure. S4. Input the optimized graph structure from S3 into a two-layer graph convolutional neural network with shared weights across modalities to finally obtain cross-modal features; S5. Input the cross-modal features obtained in S4 into the modality discriminator to learn modality-invariant features, further mitigating the differences between modalities; S6. Input the modality-invariant features obtained in S5 into the classifier to obtain the probability that the input news is fake news, and use this to judge fake news; The specific steps for S3 are as follows: 1) Input the initial adjacency matrix and node features obtained in S2 into a two-layer graph convolutional neural network of the graph structure learning network to obtain node embeddings; In each iteration, and The input is fed into a two-layer graph convolutional neural network to obtain node features. Represented as , ,in Indicates the dimension of embedding. Indicates the number of embedded nodes; Represents the initial adjacency matrix. Represent node characteristics; 2) Transpose the node embedding matrix, and then multiply the transpose matrix with the node embedding matrix to obtain the similarity matrix. ; 3) For dense similarity matrices K-nearest neighbor sparsification is performed, meaning that for each node, only the edges connecting the K nodes with the highest feature similarity to that node are retained. The final similarity matrix is ​​as follows: ; 4) The optimized adjacency matrix From the initial adjacency matrix and the calculated similarity matrix The summation of the elements forms the structure, i.e. .

2. The method for detecting fake news based on graph convolutional neural networks according to claim 1, characterized in that, In S1: 1) For text data in news collections, a bidirectional encoder is used to represent the pre-trained model to extract its text modal features; 2) For image data in the news collection, a multimodal contrastive learning pre-trained model is used to extract its image modal features.

3. The fake news detection method based on graph convolutional neural networks according to claim 1, characterized in that, The specific operation method of S1 is as follows: 1) Assume the news collection is composed of A news item consists of text and images. Indicates the first The text in the news article, Indicates the first The image in the news article, Indicates the first The set of tags and text modalities for a news item is denoted as . The set of image modalities is denoted as ; 2) For the first News items, including , Indicates the type of news tag. Represent a A 3D real vector space; the tag matrix corresponding to the news collection. Represented as: , Represent a A dimensional space of real matrices; 3) For the first This news item will The input is fed into a bidirectional encoder to represent a pre-trained model to extract its textual modal features. ,in , The dimension representing the modal features of a text; the set of modal features of a news text. Represented as: ; 4) For the first This news item will Input to a multimodal contrastive learning pre-trained model to extract its image modal features ,in , The dimension representing the modal features of an image; the set of modal features of a news image. Represented as: .

4. The fake news detection method based on graph convolutional neural networks according to claim 1, characterized in that, In S2: 1) Construction of the initial undirected graph of the image modality: For each sample feature in the set of image modality features, calculate its cosine similarity with any sample feature in the set, and connect the nodes with a cosine similarity greater than a preset threshold to construct the initial undirected graph of the image modality. 2) Construction of the initial undirected graph for the text modality: For each sample feature in the set of text modality features, calculate its cosine similarity with any sample feature in the set, and connect the nodes with a cosine similarity greater than a pre-set threshold to construct the initial undirected graph of the text modality.

5. The fake news detection method based on graph convolutional neural networks according to claim 3, characterized in that, The specific operation method of S2 is as follows: Step 1) Feature extractor Represented as ,in , used to represent the set of text modal features extracted from a news collection by a pre-trained model. or a set of image modal features , This represents the parameters to be learned, in order to further extract features; Step 2) Calculate separately , Features of pairs of samples in the set and , and The cosine similarity between them, where Nodes with a cosine similarity greater than a set threshold will be selected. , Connect to get an edge Thus, an initial undirected graph is constructed. Initial undirected graph Represented as , Represents nodes in an undirected graph The set, Represents the edges in an undirected graph A set; an initial undirected graph The initial adjacency matrix is ​​represented as .

6. The fake news detection method based on graph convolutional neural networks according to claim 5, characterized in that, In S4: Set of latent feature representations Represented as: ;in ,in This represents a two-layer graph convolutional neural network used to learn feature representations. express The Middle Each feature represents, Cross-modal feature representations of text modalities A cross-modal feature representation of an image modality.

7. The fake news detection method based on graph convolutional neural networks according to claim 6, characterized in that, In S5: 1) Modal discriminator; Modal discriminator It consists of a single-layer neural network and a normalized exponential function, and is expressed as follows: ,in The input parameters are the input text and image feature sets. Represented as: ； in, The dimension representing the mode. Represents a set The Middle One feature; Output of modality discriminator yes The probability of a text feature or an image feature; the modality label of the feature is used. It means that among them Represents a set The Middle Modal labels for each feature express Belongs to the text modality. express Belongs to image modality; loss of modality discriminator It is expressed as follows: ； in, Indicates the distribution from Random variables obtained by sampling To calculate the expected value of the mathematical expression; 2) Introduce cross-modal supervised contrast loss into the modal discriminator; for ,set up for Zhongyu A set of indexes with features of the same category. for Except In addition to the index set of features, cross-modal supervised contrastive loss It is expressed as follows: ； in, express The number of features in the index set, .

8. The fake news detection method based on graph convolutional neural networks according to claim 7, characterized in that, In step S6: 1) Classifier: Classifier It consists of a single-layer neural network and a normalized exponential function, where For learnable parameters, Represents the mode-invariant features obtained from S5; the output of the classifier. for The probability of a news item being fake or real is calculated. If the probability of it being fake is greater than the probability of it being real, then the news item is considered fake; otherwise, it is considered real. Classifier loss Represented as: ； in Indicates the distribution from Random variables obtained by sampling To calculate the expected value of the mathematical expression; 2) Parameters of the feature extractor Parameters of the modality discriminator Classifier parameters Optimizations will be made; details are as follows: according to , , The total loss can be obtained. , Represented as: ,in , As a balance factor; By maximizing the loss mode discriminator loss To update parameters , , This mitigates the modal discrepancy between text modal features and image modal features. A gradient inversion layer is employed before the modality discriminator, and the parameters are optimized using stochastic gradient descent. The optimization process is as follows: ； in , Indicates the optimized , ; ； in, Indicates the optimized .

9. A multimodal fake news detection system based on the method of claim 1, characterized in that, The multimodal fake news detection system includes: 1) Modal data preprocessing module, including a multimodal contrastive learning pre-trained model for extracting image modal features of the news to be detected and a bidirectional encoder representation pre-trained model for extracting text modal features of the news to be detected; 2) Graph structure learning network, including an initial undirected graph construction module for constructing initial undirected graphs for image modalities and text modalities, and a graph structure optimization module for optimizing the obtained initial undirected graphs; 3) The modality-invariant feature learning module consists of two layers of graph convolutional neural networks with shared weights across modalities and a modality discriminator. The optimized graph structure is first input into the two layers of graph convolutional neural networks with shared weights across modalities to obtain cross-modal features. The cross-modal features are then input into the modality discriminator to learn modality-invariant feature representations. 4) Fake news detection module, including a classifier, inputs modality-invariant feature representations into the classifier to obtain the probability that the input news belongs to fake news or real news.

Images