Citation network clustering method based on graph autoencoder

Abstract

The present invention provides a reference network clustering method based on graph autoencoder. First, the negative samples corresponding to the original data are obtained by noise processing; then, the encoder is used to obtain the node-level representation of the data, and the graph-level representation is obtained by the graph pooling function, and the decoder is used to reconstruct the adjacency matrix, and the overall loss function is optimized through the reverse gradient propagation of the neural network. The representation when the network reaches the optimal value is the final representation of the data; finally, the clustering result of the original data is obtained by spectral clustering of the data representation. The present invention makes full use of the mutual information between the original data and the learned representation, the mutual information between the learned representation and the graph-level representation, and the first-order and second-order spatial topological structure of the graph data, and can achieve better clustering effect.

Description

Technical Field

[0001] The present invention belongs to the technical field of graph representation learning, and specifically relates to a citation network clustering method based on graph autoencoder. Background Art

[0002] In recent years, the application of graph structured data has been growing, and graph analysis has become a hot research direction. Citation network is a kind of graph structured data, which takes academic documents as vertices in graph structured data and the mutual citation relationship between documents as edges in graph structured data. In the form of vertices and edges, the citation network shows the citation level of each document in the overall literature. Graph clustering is a basic method of graph analysis. Its main purpose is to divide all vertices in graph structured data into different groups or classes according to the similarity between vertices in graph structured data, so that the similarity between vertices in the same class is greater than the similarity between vertices in different classes. At present, the basic processing process of most graph clustering methods is: first, the graph representation learning technology is used to preprocess the graph structured data, and then clustering is performed. It can be mainly divided into the following three types of methods:

[0003] (1) Methods based on probabilistic models. This type of method usually regards nodes in graph data as words in a sentence, and obtains low-dimensional graph embedding vectors by randomly walking the nodes in the graph. The most representative one is the DeepWalk algorithm proposed by Perozzi et al. in the document "B.Perozzi, R.Al-Rfou, and S.Skiena.Deepwalk:Online Learning ofSocialRepresentations.in Proceedings of International Conference on KnowledgeDiscovery and Data Mining, 2014, pp.701-710", which uses truncated random walks to learn a network representation. However, this method does not consider the properties of the nodes themselves, but only the topological structure of the graph.

[0004] (2) Methods based on matrix factorization. This type of method factors the feature matrix of the original data to obtain a low-dimensional feature matrix. The most representative one is the M-NMF algorithm proposed by X. Wang et al. in the document "X. Wang, P. Cui, J. Wang, J. Pei, W. Zhu, and S. Yang. "Community Preserving Network Embedding. in Proceedings of AAAI Conference on Artificial Intelligence, vol. 33, no. 1, 2017". This method not only considers the microstructure of the graph when learning the graph representation, but also considers the mesoscopic community structure of the graph data. However, this method also does not consider the attributes of the nodes themselves.

[0005] (3) Methods based on deep learning. This type of method uses neural networks to perform feature selection on the original graph data. The most representative one is the DGI algorithm proposed by P. Velickovic et al. in the paper "P. Velickovic, W. Fedus, W. Hamilton, B. Nadler, P. Liò, Y. Bengio, and R. Hjelm. Deep Graph Infomax. in Proceedings of International Conference on Learning Representations, 2019". This method uses a graph convolutional network to learn the representation of graph nodes, while considering the mutual information between graph-level and node-level features. The problem with this method is that it does not consider the mutual information between the original features and the learned node-level representations, and uses a single-layer graph convolutional neural network, which does not consider the second-order similarity of nodes in the graph. Summary of the invention

[0006] In order to overcome the deficiency of the prior art that it is unable to simultaneously utilize the mutual information of graph data, the node attributes in the graph, and the topology of the graph data, the present invention provides a reference network clustering method based on a graph autoencoder. First, the negative samples corresponding to the original data are obtained by noise processing; then, the encoder is used to obtain the node-level representation of the data, and the graph-level representation is obtained by the graph pooling function, and the adjacency matrix is ​​reconstructed by the decoder. The overall loss function is optimized through the reverse gradient propagation of the neural network. The representation when the network reaches the optimal value is the final representation of the data; finally, the clustering result of the original data is obtained by spectral clustering of the data representation. The present invention makes full use of the mutual information between the original data and the learned representation, the mutual information between the learned representation and the graph-level representation, and the first-order and second-order spatial topological structure of the graph data, and can achieve better clustering effect.

[0007] A citation network clustering method based on graph autoencoder, characterized by the following steps:

[0008] Step 1: Input the reference network data and perform noise processing on it, taking the input original reference network data as the positive sample and the reference network data after noise processing as its corresponding negative sample;

[0009] Step 2: Input the positive sample data and negative sample data into the encoder respectively to obtain their corresponding node-level representations. The encoder is a two-layer graph convolutional neural network, which is calculated as follows:

[0010]

[0011] Among them, H represents the obtained node-level representation, Represents the adjacency matrix of the input data with self-connection, according to Calculated, A represents the adjacency matrix of the input reference network data, the element value of 0 in the matrix indicates that the two nodes in the reference network are not connected, the element value of 1 indicates that the two nodes in the reference network are connected, I represents the unit matrix of the same size as the matrix A, its main diagonal element is 1, and the rest of the elements are 0, Represents an adjacency matrix with self-connection The degree matrix of The sum of the elements of each row of i represents the matrix The row number of the matrix The column number, l represents the number of layers of the encoder, H (l) represents the input features of the encoder’s l-th layer convolutional neural network, W (l) represents the weight of the encoder's l-th layer convolutional neural network. When l = 0, H (l) =X, X is the feature matrix of the input reference network data, σ(·) is the activation function;

[0012] Step 3: Use average graph pooling function Process the node-level representation H of the positive sample data to obtain its graph-level representation Right now

[0013] Step 4: Input the node-level representation output by the encoder into the decoder to obtain the reconstructed adjacency matrix. The decoder uses the inner product form and is calculated as follows:

[0014] A′=σ(HH T ) (2)

[0015] Among them, A′ is the reconstructed adjacency matrix;

[0016] Step 5: Optimize the overall loss function through reverse gradient propagation of the neural network until the overall loss reaches the minimum or no longer changes, and complete the network parameter optimization. The node-level representation output by the encoder at this time is used as the final representation of the data;

[0017] The neural network is all the processing and networks from step 1 to step 4;

[0018] The overall loss function L total The calculation formula is as follows:

[0019] L total =L encoder +βL decoder (3)

[0020] Among them, L encoder is the loss function of the encoding stage, L decoder is the loss function of the decoding stage, β is the hyperparameter 1, and its value range is [0,2];

[0021] The encoding phase loss function L encoder The calculation formula is:

[0022] L encoder =L 1 +αL 2 (4)

[0023] Among them, L 1 represents the local mutual information loss, L 2 represents the global mutual information loss, α is the second hyperparameter, and its value range is [0,1]; L 1 and L 2 The calculation formulas are as follows:

[0024]

[0025]

[0026] Among them, N represents the number of nodes in the initial reference network data, M represents the number of nodes in the reference network data after noise addition, X represents the feature matrix of the positive sample data, and A represents the adjacency matrix of the positive sample data. Represents the feature matrix of negative sample data, Represents the adjacency matrix of negative sample data, E (X,A) represents the expectation of positive sample data, represents the expectation of negative sample data, represents the node-level representation of the i-th positive sample data, represents the graph-level representation, represents the node-level representation of the j-th negative sample data, represents the original features of the i-th positive sample data, represents the original features of the jth negative sample data, D L (·) represents the local mutual information discriminator, W score is the score matrix, σ(·) is the activation function, and D G (·) represents the global mutual information discriminator,

[0027] Decoding stage loss function L decoder The calculation formula is:

[0028]

[0029] Among them, y ij is the value of the node in the i-th row and j-th column of the adjacency matrix A of the positive sample data, is the value of the node in the i-th row and j-th column of the reconstructed adjacency matrix A′;

[0030] Step 6: Perform spectral clustering on the final representation of the data, and use the clustering result label of the data representation as the clustering label of its corresponding data to obtain the clustering result of the original data.

[0031] The beneficial effects of the present invention are as follows: due to the use of an encoder in the form of a graph convolutional neural network, the representation not only contains spatial topological information, but also takes into account the attribute information of the node itself; for the node-level representation of the middle layer, not only the mutual information between it and the graph-level representation is considered, but also the mutual information between it and the original data is considered, and the two-layer graph convolutional neural network used can consider the second-order similarity information between nodes, so as to better ensure that the learned representation can maximize the retention of the discriminant information of the original data, thereby ensuring better clustering results; due to the use of spectral clustering to cluster the learned representation, good clustering effects can also be achieved for data that does not obey a strict distribution. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 This is a flow chart of the citation network clustering method based on graph autoencoder of the present invention. DETAILED DESCRIPTION

[0033] The present invention is further described below in conjunction with the accompanying drawings and embodiments. The present invention includes but is not limited to the following embodiments.

[0034] like Figure 1 As shown, the present invention provides a citation network clustering method based on graph autoencoder, and its specific implementation process is as follows:

[0035] 1. Graph structure data preprocessing

[0036] In order to obtain negative samples of graph structure data so as to maximize the mutual information between representations later, the present invention first performs noise processing on the original graph data, such as keeping the adjacency matrix of the original data unchanged and shuffling the rows of the feature matrix. Through noise processing, negative graph structure data samples are obtained, that is, the original data is used as the positive sample and the noised data is used as its corresponding negative sample.

[0037] 2. Obtaining node-level representation of data

[0038] The positive sample data and negative sample data are input into the encoder respectively. The encoder is a two-layer graph convolutional neural network. The nodes in the graph can consider the first-order and second-order similarity information, thereby obtaining a node-level low-dimensional representation containing spatial topological information. The specific formula is as follows:

[0039]

[0040] Among them, H represents the obtained node-level representation, Represents the adjacency matrix of the input data with self-connection, according to Calculated, A represents the adjacency matrix of the input reference network data, the element value of 0 in the matrix indicates that the two nodes in the reference network are not connected, the element value of 1 indicates that the two nodes in the reference network are connected, I represents the unit matrix of the same size as the matrix A, its main diagonal element is 1, and the rest of the elements are 0, Represents an adjacency matrix with self-connection The degree matrix of The sum of the elements of each row of i represents the matrix The row number of the matrix The column number, l represents the number of layers of the encoder, H (l) represents the input features of the encoder’s l-th layer convolutional neural network, W (l) represents the weight of the encoder's l-th layer convolutional neural network. When l = 0, H (l) =X, X is the feature matrix of the input reference network data, and σ(·) is the activation function.

[0041] 3. Obtaining graph-level representation

[0042] Using the learned node-level representation of the positive sample, we can obtain the graph-level representation by using an average graph pooling function. To perform graph pooling on node-level representations:

[0043]

[0044] in, That is the graph-level representation.

[0045] 4. Obtain the reconstructed citation network adjacency matrix

[0046] The decoder is used to reconstruct the adjacency matrix, that is, the low-dimensional representation of the positive sample node level learned by the encoder is input into the decoder, and then the adjacency matrix is ​​reconstructed. The decoder of the present invention adopts the inner product form:

[0047] A′=σ(HH T ) (10)

[0048] Where A′ is the reconstructed adjacency matrix.

[0049] 5. Overall optimization parameters

[0050] By performing reverse gradient propagation on the loss functions of the encoding and decoding stages, the network parameters are gradually optimized as the overall loss gradually decreases, thereby learning the best representation. That is, the entire process of steps 1-4 is used as the entire neural network processing process, and the overall loss function is optimized until the overall loss reaches the minimum or no longer changes, and the network parameter optimization is completed. The node-level representation output by the encoder at this time is used as the final representation of the data.

[0051] The loss function L in the encoding stage encoder The calculation formula is:

[0052] L encoder =L 1 +αL 2 (11)

[0053] Among them, L 1 represents the local mutual information loss, L 2 represents the global mutual information loss, α is the second hyperparameter, and its value range is [0,1]. 1 and L 2 The calculation formulas are as follows:

[0054]

[0055]

[0056] Where N represents the number of nodes in the initial citation network data (positive sample), M represents the number of nodes in the citation network data after noise addition (negative sample), X represents the feature matrix of the initial citation network data (positive sample), and A represents the adjacency matrix of the initial citation network data (positive sample). Represents the feature matrix of the reference network data (negative samples) after adding noise, represents the adjacency matrix of the reference network data (negative samples) after adding noise, E (X,A) represents the expectation of positive sample data, represents the expectation of negative sample data, represents the node-level representation of the i-th positive sample data, represents the graph-level representation, represents the node-level representation of the j-th negative sample data, represents the original features of the i-th positive sample data, represents the original features of the jth negative sample data, D L (·) represents the local mutual information discriminator, which is a bilinear function To score the positive and negative sample pairs, where W score is a learnable score matrix, yes is the transpose of , σ(·) is the activation function, D G (·) represents the global mutual information discriminator, which is a bilinear function To score the positive and negative sample pairs, yes The transpose of .

[0057] Decoding stage loss function L encoder The calculation formula is:

[0058]

[0059] Among them, y ij is the value (0 or 1) of the node in the i-th row and j-th column of the adjacency matrix A of the initial reference network data (positive sample), is the value (between 0 and 1) of the corresponding node in the i-th row and j-th column of the reconstructed reference network data adjacency matrix A′.

[0060] The overall loss function L total The calculation formula is as follows:

[0061] L total =L encoder +βL decoder (15)

[0062] Among them, L encoder is the loss function of the encoding stage, L decoder is the loss function of the decoding stage, β is a hyperparameter with a value range of [0,2].

[0063] 6. Spectral clustering of learned representations

[0064] Spectral clustering does not have strict requirements on the distribution of data. It obtains the similarity matrix by measuring the distance between data. Therefore, spectral clustering is performed on the final representation of the data, and the clustering result label of the data representation is used as the clustering label of the corresponding data to obtain the clustering result of the original data.

[0065] To verify the effectiveness of the method of the present invention, the CPU isThe simulation experiment was carried out using PYCHARM software on an i7-10700F 2.90GHz CPU, 16G memory, WINDOWS 10 operating system and TiTan RTX. Three public data sets were used in the experiment, namely the citation network data of Cora, Citeseer, and Pubmed databases. The K-means method, DNGR method, GAE method, ARGA_AX method, and DBGAN method were selected as comparison methods.Among them, the K-means method is a method proposed in the literature "S.Lloyd.Least Squares Quantization in Pcm.IEEE Transactions on Information Theory,pp.129–137,1982."; the DNGR method is a method proposed in the literature "S.Cao,W.Lu,andQ.Xu.Deep Neural Networks for Learning Graph Representations.in Proceedingsof AAAI Conference on Artificial Intelligence,2016."; the GAE method is a method proposed in the literature "T.Kipf and M.Welling.Semi-supervised Classification with Graph ConvolutionalNetworks.arXiv preprint arXiv:1609.02907,2016."; the ARGA_AX method is a method proposed in the literature "S.Pan,R.Hu,S.Fung,G.Long,J.Jiang,and C.Zhang.Learning Graph Embedding wihAdversarial Training Methods.IEEE Transactions on Cybernetics, pp.2475-2487, 2020."; The DBGAN method is the method proposed in the literature "S.Zheng, Z.Zhu, X.Zhang, Z.Liu, J.Cheng, and Y.Zhao.Distribution-induced Bidirectional Generative Adversarial Network forGraph Representation Learning.In Proceedings ofIEEE Conference on Computer Vision and Pattern Recognition, 2020, pp.7222-7231."

[0066] The accuracy ACC, normalized mutual information NMI and adjusted Rand index ARI are calculated as evaluation indicators of clustering results of different methods, as shown in Table 1. It can be seen that on the Citeseer dataset, the method of the present invention is superior to other methods in ACC, NMI and ARI indicators; on the Cora dataset, the ARI of DBGAN is slightly higher, but the ACC and NMI are lower than those of the method of the present invention; on the Pubmed dataset, the NMI of the method of the present invention is superior to other methods, and the ACC and ARI of DBGAN are slightly higher. Overall, the method of the present invention has a good clustering effect.

[0067] Table 1

[0068]

Claims

1. A citation network clustering method based on graph autoencoder, characterized by Here are the steps: Step 1: Input the reference network data and perform noise processing on it, taking the input original reference network data as the positive sample and the reference network data after noise processing as its corresponding negative sample; Step 2: Input the positive sample data and negative sample data into the encoder respectively to obtain their corresponding node-level representations. The encoder is a two-layer graph convolutional neural network, which is calculated as follows: Among them, H represents the obtained node-level representation, Represents the adjacency matrix of the input data with self-connection, according to Calculated, A represents the adjacency matrix of the input reference network data, the element value of 0 in the matrix indicates that the two nodes in the reference network are not connected, the element value of 1 indicates that the two nodes in the reference network are connected, I represents the unit matrix of the same size as the matrix A, its main diagonal element is 1, and the rest of the elements are 0, Represents an adjacency matrix with self-connection The degree matrix of The sum of the elements of each row of i represents the matrix The row number of the matrix The column number, l represents the number of layers of the encoder, H (l) represents the input features of the encoder’s l-th layer convolutional neural network, W (l) represents the weight of the encoder's l-th layer convolutional neural network. When l = 0, H (l) =X, X is the feature matrix of the input reference network data, σ(·) is the activation function; Step 3: Use average graph pooling function Process the node-level representation H of the positive sample data to obtain its graph-level representation Right now Step 4: Input the node-level representation output by the encoder into the decoder to obtain the reconstructed adjacency matrix. The decoder uses the inner product form and is calculated as follows: A′=σ(HH T ) (2) Among them, A′ is the reconstructed adjacency matrix; Step 5: Optimize the overall loss function through reverse gradient propagation of the neural network until the overall loss reaches the minimum or no longer changes, and complete the network parameter optimization. The node-level representation output by the encoder at this time is used as the final representation of the data; The neural network is all the processing and networks from step 1 to step 4; The overall loss function L total The calculation formula is as follows: L total =L encoder +βL decoder (3) Among them, L encoder is the loss function of the encoding stage, L decoder is the loss function of the decoding stage, β is the hyperparameter 1, and its value range is [0,2]; The loss function L in the encoding stage encoder The calculation formula is: L encoder =L1+αL2 (4) Among them, L1 represents the local mutual information loss, L2 represents the global mutual information loss, α is the second hyperparameter, and its value range is [0,1]. The calculation formulas of L1 and L2 are as follows: Among them, N represents the number of nodes in the initial reference network data, M represents the number of nodes in the reference network data after noise addition, X represents the feature matrix of the positive sample data, and A represents the adjacency matrix of the positive sample data. Represents the feature matrix of negative sample data, Represents the adjacency matrix of negative sample data, E (X,A) represents the expectation of positive sample data, represents the expectation of negative sample data, represents the node-level representation of the i-th positive sample data, represents the graph-level representation, represents the node-level representation of the j-th negative sample data, represents the original features of the i-th positive sample data, represents the original features of the jth negative sample data, D L (·) represents the local mutual information discriminator, W score is the score matrix, σ(·) is the activation function, and D G (·) represents the global mutual information discriminator, Decoding stage loss function L decoder The calculation formula is: Among them, y ij is the value of the node in the i-th row and j-th column of the adjacency matrix A of the positive sample data, is the value of the node in the i-th row and j-th column of the reconstructed adjacency matrix A′; Step 6: Perform spectral clustering on the final representation of the data, and use the clustering result label of the data representation as the clustering label of its corresponding data to obtain the clustering result of the original data.

Images