A robust graph convolutional neural network method based on spatio-temporal sparse learning

Abstract

The application discloses a robust graph convolutional network method based on space-time sparse learning. The method realizes spatial sparsity on each node through a TopK function, and proposes an attention mechanism based on time sparsity, that is, different weights are assigned to each dimension of a feature space according to different activation frequencies. The application provides an improved graph convolutional neural network, which has high robustness while maintaining the original network accuracy, and improves the anti-interference ability of the model in the face of noise.

Description

Technical Field

[0001] This invention belongs to the field of graph adversarial attacks and defenses, and particularly relates to a method for improving the robustness of a model to perturbations using spatiotemporal sparse learning. Background Technology

[0002] In recent years, the successful application of graph neural networks (GNNs) on various graph-structured data, such as social networks, chemical composition structures, and biological genes and proteins, has attracted increasing attention. However, recent work has pointed out that GNNs are vulnerable to adversarial attacks, which could potentially cripple safety-critical GNN applications, such as autonomous driving and medical diagnostics.

[0003] The main idea of ​​GNN adversarial attacks is to change the topological information of the graph structure or the feature information of the nodes to intentionally interfere with the classifier. In terms of adversarial attacks on generated graphs, Dai et al. studied the non-target avoidance attack RL-S2V based on reinforcement learning[1]. Zugner proposed a poisoning attack method Nettack, which can change the training data to misclassify the target node[2]. In addition, Zugner et al. also proposed to use meta-gradients to solve the min-max problem in the attack during the training process and proposed an attack method that reduces the overall classification performance[3]. On the other hand, Xu et al. simplified the discrete problem of the graph through convex relaxation, and thus proposed a gradient-based topological attack[4].

[0004] Regarding methods to improve robustness on graphs, Wu et al. improved the robustness of the model by preserving neighboring nodes with high Jaccard similarity to their nodes [5]. The RGCN model uses a Gaussian distribution as the hidden representation of nodes in all convolutional layers of the graph and reduces the variance of the Gaussian distribution by absorbing the effects of adversarial attacks through an attention mechanism [6]. Zugner et al. proposed a robustness proof based on convex relaxation for perturbations only to node properties and used semi-supervised properties to improve the robustness of the model [7]. The PA-GNN model learns the ability to penalize perturbations through information from other clean graphs in similar domains and transfers it to the target poisoned graph [8].

[0005] Unlike existing methods for improving robustness on graphs, this patent proposes to construct a robust feature space on the graph from the perspective of spatiotemporal sparsity and propagate a robust feature representation of each node in the graph, without targeting specific perturbations. The motivation of this patent is to show that robust models are different from conventional models in that robust models tend to learn more meaningful and salient data features[9]. This phenomenon prompted this patent to use sparse representation to construct robust feature representations, retaining only salient features to reduce the influence of weakly correlated features, thereby improving the robustness of the model. At the same time, such sparse representation has been widely used in computational neuroscience

[10] .

[0006] Therefore, this patent proposes a sparse representation learning framework (ST-SparseGCN) to improve the robustness of GNN models. The proposed framework can not only learn to activate the most salient features through spatial sparsity, but also expand the latent active feature set through temporal sparsity learning, enabling dynamic selection of active features from a larger pool of salient features, thereby enhancing the model's generalization ability. Summary of the Invention

[0007] Purpose of the invention: This invention provides a robust graph convolutional neural network method based on spatiotemporal sparse learning (ST-SparseGCN), which effectively improves the model's ability to adapt to the effects of perturbations, increases the accuracy of node identification under perturbation conditions, and enhances the robustness and robustness of the model.

[0008] Technical solution: To achieve the above objectives, the technical solution adopted by this invention is as follows:

[0009] A robust graph convolutional neural network method based on spatiotemporal sparse learning (ST-SparseGCN) includes the following steps:

[0010] Step 1) Given graph structure data, including feature vector matrix X representing node features, adjacency matrix A representing the connection relationships in the graph, and corresponding node category labels, initialize the activation frequency matrix.

[0011] Step 2) Input the graph structure data into the graph convolutional neural network layer to obtain the hidden feature representation of each node.

[0012] Step 3) Input the hidden feature representation of each node into the spatial sparsification module to obtain the sparse hidden feature representation of each node.

[0013] Step 4) Update the activation frequency matrix based on the activation positions represented by the sparse hidden features.

[0014] Step 4-1) Count the dimensions of the feature values ​​that are not set to 0 in the sparse hidden feature representation.

[0015] Step 4-2) In the time weight matrix, add 1 to the position of the same dimension as the one being counted.

[0016] Step 5) Generate a time weight matrix by activating the frequency matrix.

[0017] Step 6) Assign the time weight matrix to the sparse hidden feature representation of each node to obtain the spatiotemporal sparse hidden feature representation of each node.

[0018] Step 7) Repeat steps 2) through 6) to construct a multi-layer graph convolutional neural network and train the entire model.

[0019] Step 8) Using the trained model, evaluate the categories of test nodes with and without perturbations.

[0020] Beneficial effects:

[0021] 1) This invention presents a robust graph convolutional neural network method based on spatiotemporal sparse learning. It explores how to construct a robust feature space to represent nodes to defend against adversarial attacks and tries to mine the robust hidden features of each node in graph structure data.

[0022] 2) To verify the effectiveness of the proposed ST-SparseGCN model, extensive experiments were conducted on multiple datasets. Experimental results show that ST-SparseGCN can significantly improve the robustness and stability of GCN models in terms of classification accuracy. Furthermore, the method also experimentally analyzes the parameter sensitivity of the model. Attached Figure Description

[0023] Figure 1 This is a flowchart illustrating the specific implementation of the method of the present invention;

[0024] Figure 2 A schematic diagram illustrating the concept of spatiotemporally sparse learning;

[0025] Figure 3 This is a schematic diagram of the overall structure of the method of the present invention;

[0026] Figure 4 This is a diagram comparing TopK and ReLU functions;

[0027] Figure 5 This is a schematic diagram of the parameter sensitivity experiment of the method of the present invention; Detailed Implementation

[0028] This invention provides an embodiment of a robust graph convolutional neural network method based on spatiotemporal sparse learning. To enable those skilled in the art to better understand the technical solutions in this embodiment and to make the above-mentioned objectives, features, and advantages of this invention more apparent, the technical solutions of this invention are further described in detail below with reference to the accompanying drawings:

[0029] like Figure 1 As shown, a robust graph convolutional neural network method based on spatiotemporal sparse learning is proposed. On the basis of graph convolutional neural networks, it constructs robust feature representations through spatiotemporal sparse learning. It can not only learn to activate the most salient features through spatial sparsity, but also expand the potential active feature set through temporal sparsity learning, so as to dynamically select active features from a larger pool of salient features, thereby enhancing the model's generalization ability.

[0030] The basic conceptual structure of spatiotemporal sparse learning is as follows: Figure 2As shown, spatial sparsity is primarily used to transform the dense node vector of the GNN output into a sparse high-dimensional vector, where only the top K salient features are activated. During GNN training, temporal sparsity further sparsifies the active features along the time dimension. More specifically, the duty cycle of each effective feature size is sparse, thus avoiding excessive use of each effective feature.

[0031] Complete network structure such as Figure 3 As shown, the network structure includes a graph convolutional network and a spatiotemporal sparse learning architecture. The output of the ST-Sparse layer serves as the input to the GCN layer, and the output of the GCN layer serves as the input to the next ST-Sparse layer. Finally, the class prediction value for each node is generated through the Softmax function. A flowchart illustrating the specific implementation of this invention is shown below. Figure 1 As shown, the process is as follows:

[0032] Step 1) Given graph structure data, including feature vectors representing node features, adjacency matrices representing connection relationships in the graph, and corresponding node category labels, initialize the activation frequency matrix.

[0033] Prepare the node feature information X and edge adjacency information A of the graph structure data, and randomly initialize the weight parameters of the entire model network with a mean of 0 and a variance of 0.01. Initialize the activation frequency matrix, where the size is the dimension of the previous layer GCN output, and all values ​​are 0.

[0034] Step 2) Input the graph structure data into the graph convolutional neural network layer to obtain the hidden feature representation of each node. Specifically, input the adjacency matrix A and the feature vector matrix X of the graph into the graph convolutional network layer. The graph convolutional network layer can be represented as:

[0035]

[0036] in, This is achieved by adding an identity matrix I to the adjacency matrix A. N The adjacency matrix W constructed later contains self-loops. (l) It is the weight matrix that needs to be trained. Here, is the degree matrix, and σ(·) is the activation function. In the first layer, the input is the feature matrix H of the node. (0) =X, and in other layers it is a matrix composed of spatiotemporally sparse hidden feature representations learned through spatiotemporal sparse learning.

[0037] Step 3) Input the hidden feature representation of each node into the spatial sparsification module to obtain the sparse hidden feature representation of each node. That is, the matrix composed of the hidden feature representations of all nodes output by the graph convolutional layer is sparsified using the TopK function, retaining only the top K features with the largest hidden feature representation values ​​for each node, and setting all other features to 0. This can be formally represented as follows:

[0038] s i =TopK(h i ,k α (2)

[0039] Among them, s i h represents the sparse hidden feature representation. i k represents the hidden feature representation. α k represents the number of activated features. α =αm, where m is h i The dimension size is α, where α is the proportion of the number of activated features to the total feature dimension.

[0040] Step 4) Update the activation frequency matrix based on the activation positions represented by the sparse hidden features. The specific steps are as follows:

[0041] Step 4-1) Count the dimensions of the feature values ​​that are not set to 0 in the sparse hidden feature representation.

[0042] Step 4-2) In the time weight matrix, add 1 to the position of the same dimension as the one being counted.

[0043] Step 5) Generate the time weight matrix by activating the frequency matrix. Specifically, the method of this invention generates the time weight matrix using an exponential smoothing function, the expression of which is:

[0044]

[0045] in, B is the time weight matrix. t Let γ be the activation frequency matrix, and γ be a hyperparameter.

[0046] Step 6) Assign the time weight matrix to the sparse hidden feature representation of each node to obtain the spatiotemporal sparse hidden feature representation of each node. This is achieved by multiplying the time weight matrix by the sparse hidden feature representation generated by the TopK function, and its expression is:

[0047]

[0048] in, This represents a spatiotemporally sparse hidden feature representation.

[0049] Step 7) Repeat steps 2) through 6) to construct a multi-layer graph convolutional neural network and train the entire model. After computation, the predicted node classification is obtained through the Softmax function and used as the model output. The loss function L during training is the sum of the output sample and the partially labeled sample V. L The cross-entropy of class labels between them is used to learn the optimal parameters θ in a semi-supervised manner.

[0050]

[0051] Where θ represents the set of all parameters, c v This represents the label of a given labeled sample in the training set, and z after training. v This represents the class probability of each node instance.

[0052] Step 8) Using the trained model, evaluate the categories of test nodes with and without perturbations. A test node with a perturbation is defined in two ways: firstly, the node has incorrect connections with other nodes, or has lost some connections with its original neighbors (i.e., it has abnormal edge connections with other nodes in the graph, or it has lost some edges with its neighbors); secondly, the node itself has abnormal features. The method of this invention will generate perturbation nodes using three methods: DICE, Nettack, and Meta-Self.

[0053] To evaluate the robustness and effectiveness of the ST-SparseGCN method, experiments were conducted on the deep learning framework PyTorch and the deep learning extension library PyG. Its performance was compared with Graph Convolutional Networks (GCNs) and Robust Graph Convolutional Networks (RGCNs) for node-level semi-supervised classification. For fair comparison, the method used the same general parameters for different models in the experiments. Specifically, the method proposes fixing the number of layers to 2 and the number of training iterations to 1000. For all comparison models, the method proposes fixing the learning rate of the Adam optimization algorithm to 0.01. Furthermore, a single hyperparameter in the ST-SparseGCN model was adjusted based on the validation set to achieve optimal robustness. The method also reports that the final results for all experiments were obtained through an average of 10 runs.

[0054] To conduct a comprehensive study of ST-SparseGCN, the proposed method was evaluated on three well-known graph datasets: Cora, Citeseer, and PubMed, where nodes represent documents and edges represent citations. The method then uses the sparse bag-of-words feature vector of each node as input to the model. Table 1 lists the basic information for each dataset. Furthermore, the method uses the same training, test, and validation sets on the same datasets in the experiments to fairly evaluate the performance of different models.

[0055] Table 1. Basic Information about the Dataset

[0056] Number of nodes Number of sides feature category Cora 2708 (1 graph) 5429 1433 7 Citeseer 3327 (1 graph) 4732 3703 6 Pubmed 19717 (1 graph) 44338 500 3

[0057] To accurately measure the impact of perturbations, the method of this invention first evaluates the performance of ST-SparseGCN and baseline models GCN and RGCN on different clean datasets. The mean accuracy with standard deviation is reported in Table 2. This result demonstrates that graph convolutional models based on spatiotemporal sparse learning rules still exhibit excellent performance on clean datasets. Although ST-SparseGCN does not show significant performance improvement on clean datasets compared to all baseline models, it lays the foundation for further discussion of the robustness of the model in the presence of perturbation attacks.

[0058] Table 2 shows the accuracy on the unperturbed dataset.

[0059] Cora Citeseer Pubmed GCN 81.5±0.6 71.1±0.7 79.0±0.6 RGCN 81.9±0.5 71.5±0.7 79.1±0.6 ST-SparseGCN 82.3±0.6 71.6±0.7 79.5±0.5

[0060] This invention uses three different attack methods—DICE, Nettack, and Meta-Self—to generate perturbed nodes and evaluates robust graph convolutional neural network methods based on spatiotemporal sparse learning. Table 3 presents the average accuracy of different models and attack methods. The following observations can be drawn from Table 3:

[0061] 1) The accuracy of all models decreases rapidly with increasing perturbation magnitude. This indicates that graph network models are susceptible to perturbations.

[0062] 2) ST-SparseGCN can achieve higher performance under different conditions because ST-SparseGCN builds a powerful function space in each layer of GCN;

[0063] 3) When the disturbance amplitude is large or small, the accuracy difference between the three models decreases.

[0064] 4) On the PubMed dataset, ST-SparseGCN does not show a significant performance improvement over RGCN, which this invention attributes to the dataset itself. Specifically, the PubMed dataset is a simple dataset containing a bag of word vectors of length 500 and word vectors from three categories. Therefore, the performance of the PubMed dataset degrades slowly under perturbation.

[0065] Table 3 shows the accuracy on the perturbed dataset.

[0066]

[0067] Experiments have verified that the robust graph convolutional neural network method based on spatiotemporal sparse learning can construct a robust feature space well and exhibit good defense performance against various adversarial attacks.

[0068] Figure 4 This demonstrates a comparison between the TopK function and the ReLU function. Figure 4 The figure shows the activation rates of the hidden feature representations of the output nodes using GCN with TopK and ReLU functions. The TopK function maintains a constant, low activation rate throughout the training cycle, while the ReLU function shows a high activation rate as the training cycle increases, which is detrimental to the model learning robust features. Figure 4 The image 'b' shows the function graphs of the TopK and ReLU functions. The activation objects of the TopK function are contained within the ReLU function, so using the TopK function alone will not additionally activate features that were not originally activated in the ReLU function.

[0069] at the same time, Figure 5 The parameter sensitivity experiments for this model are provided. Figure 5 :a and Figure 5 The results show that models with temporal sparsity can exhibit better performance on varying degrees of spatial sparsity. Furthermore, the accuracy of ST-SparseGCN remains largely unchanged when the hyperparameter α controlling spatial sparsity is within a certain range. However, setting α too small degrades performance, possibly because the number of activated features is insufficient for the model to distinguish between different nodes.

[0070] same, Figure 5 c and Figure 5 The :d indicates the impact of different output feature dimensions on the model. Whether using a clean or perturbed dataset, performance degrades significantly when the dimensionality is too low. Once the dimensionality reaches a certain size and continues to increase, performance remains almost constant.

[0071] The above description is merely a preferred embodiment of the present invention, and the scope of protection of the present invention is not limited to the above embodiments. For those skilled in the art, improvements and modifications obtained without departing from the inventive concept should also be considered within the scope of protection of the present invention.

[0072] The following are the relevant documents retrieved:

[0073] [1]H.Dai et al., "Adversarial attack on graph structured data," in 35thInternational Conference on Machine Learning,ICML 2018, 2018, vol.3, pp.1799–1808.

[0074] [2] D. Zügner, A. Akbarnejad, and S. Günnemann, “Adversarial attacks on neural networks for graph data,” in Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2018, pp.2847–2856.

[0075] [3] D.Zügner and S.Günnemann, “Adversarial attacks on graph neuralnetworks via meta learning,” in 7th International Conference on LearningRepresentations,ICLR 2019, 2019, pp.1–15.

[0076] [4]H.Xu et al., "Adversarial Attacks and Defenses in Images, Graphs andText: A Review," 2019.

[0077] [5]H.Wu,C.Wang,Y.Tyshetskiy,A.Docherty,K.Lu,and L.Zhu,“AdversarialExamples on Graph Data:Deep Insights into Attack and Defense,”2016.

[0078] [6]M.Jin,H.Chang,W.Zhu,and S.Sojoudi,“Power up!Robust GraphConvolutional Network against Evasion Attacks based on Graph Powering,”pp.1–14,2019.

[0079] [7]D.Zügner and S.Günnemann,“Certifiable robustness and robusttraining for graph convolutional networks,”in Proceedings of the ACM SIGKDDInternational Conference on Knowledge Discovery and Data Mining,2019,pp.246–256.

[0080] [8]X.Tang,Y.Li,Y.Sun,H.Yao,P.Mitra,and S.Wang,“Robust Graph NeuralNetwork Against Poisoning Attacks via Transfer Learning,”in WSDM 2020,2019.

[0081] [9]D.Tsipras,S.Santurkar,L.Engstrom,A.Turner,and A.Madry,“Robustnessmay be at odds with accuracy,”in 7th International Conference on LearningRepresentations,ICLR 2019,2019,pp.1–24.

[0082]

[10] S.Ahmad and J.Hawkins,“How do neurons operate on sparsedistributed representations?A mathematical theory of sparsity,neurons andactive dendrites,”pp.1–23,2016.

Claims

1. A robust graph convolutional neural network method based on spatiotemporal sparse learning, characterized in that, Includes the following steps: Step 1) Given graph structure data, including feature vectors to represent node features, adjacency matrices to represent the connection relationships in the graph, and corresponding node category labels, initialize the activation frequency matrix, where the graph structure data is the citation network of the paper, and the node features represent the sparse bag-of-words of the paper. Step 2) Input the graph structure data into the graph convolutional neural network layer to obtain the hidden feature representation of each node, where the node represents the paper; Step 3) Input the hidden feature representation of each node into the spatial sparsification module to obtain the sparse hidden feature representation of each node; Step 4) Update the activation frequency matrix based on the activation positions represented by the sparse hidden features; Step 5) Generate a time weight matrix by activating the frequency matrix; Step 6) Assign the time weight matrix to the sparse hidden feature representation of each node to obtain the spatiotemporal sparse hidden feature representation of each node. Step 7) Repeat steps 2) to 6) to construct a multi-layer graph convolutional neural network and train the entire model. Step 8) Using the trained model, evaluate the categories of test nodes with and without perturbations, where the categories are used to represent the research domain of the paper, and the trained model is used to predict the paper category in the citation network of the paper. The spatial sparsification module in step 3) is implemented by the TopK function. For the hidden feature representation of each node, the TopK function only retains the top K largest feature values ​​among all feature values ​​in the node's hidden feature representation and sets all other feature values ​​to 0. The sparse hidden feature representation in step 3) refers to a hidden feature representation that retains only a small number of feature values, which is implemented by the TopK function; Step 4) includes the following steps: Step 4 1. Dimension of the feature values ​​that are not set to 0 in the statistical sparse hidden feature representation; Step 4 2. In the time weight matrix, increment the corresponding position of the dimension being counted by 1; The specific steps of step 5) involve generating a time weight matrix using an exponential smoothing function, the expression of which is: in, This is the time weight matrix. To activate the frequency matrix, This is a hyperparameter.

2. The robust graph convolutional neural network method based on spatiotemporal sparse learning according to claim 1, characterized in that, Step 1) Initializing the time weight matrix means: initializing an all-zero matrix with the same dimension as the hidden feature representation.

3. The robust graph convolutional neural network method based on spatiotemporal sparse learning according to claim 1, characterized in that, Step 2) inputs the graph structure data into the graph convolutional neural network layer. The graph convolutional neural network layer refers to a semi-supervised node classification model that can learn the hidden representation of each node. The hidden vectors of all nodes in its (l+1)th layer can be recursively represented by the hidden vectors of the lth layer, as shown below: Among them, is Through the adjacency matrix Add an identity matrix The subsequently constructed adjacency matrix contains self-loops. It is the weight matrix that needs to be trained. It is a degree matrix. It's an activation function; in the first layer, our input is the feature matrix of the node. In other layers, the matrix consists of spatiotemporally sparse hidden feature representations learned through spatiotemporal sparse learning.

4. The robust graph convolutional neural network method based on spatiotemporal sparse learning according to claim 1, characterized in that, The generation steps of the spatiotemporal sparse hidden feature representation in step 6) are as follows: the sparse hidden feature representation is generated by the time weight matrix dot product through the graph convolutional network layer and the Top function.

5. The robust graph convolutional neural network method based on spatiotemporal sparse learning according to claim 1, characterized in that, In step 7), after the multi-layer model is calculated, the estimated node classification is obtained through the Softmax function and used as the model output. The loss function during training is also mentioned. Output samples and partially labeled samples The cross-entropy of class labels between them is used to learn the optimal parameters in a semi-supervised manner. : in, Represents the set of all parameters. This represents the label of a given labeled sample in the training set, after training. This represents the class probability of each node instance.

6. The robust graph convolutional neural network method based on spatiotemporal sparse learning according to claim 1, characterized in that, In step 8), a test node with disturbance refers to two things: firstly, the node has incorrect connections with other nodes, or has lost some connections with its original neighboring nodes, that is, it has abnormal edge connections with other nodes on the graph, or has lost some edges with neighboring nodes; secondly, the node itself has abnormal features.

Images