A Variational Graph Autoencoder Method Based on Multi-Task Learning

Abstract

The present invention discloses a variational graph autoencoder method based on multi-task learning, comprising the following steps: S1: preprocessing source data; S2: dividing graph data sets; S3: inputting the training set obtained in S22 into a shallow graph convolution layer to obtain a shallow shared embedding representation H; S4: inputting the shallow shared embedding representation H obtained in S3 into two different downstream network frameworks respectively to obtain respective embedding representations; S5: performing link prediction tasks and semi-supervised node classification tasks on the two different embedding representations obtained in S4 respectively. This method can make the embedding representation closer to the real distribution of the sample space, and has strong competitiveness and robustness in link prediction tasks.

Description

Technical Field

[0001] The present invention relates to the field of computer data analysis, and in particular to a variational graph autoencoder method based on multi-task learning. Background Art

[0002] With the continuous development of deep learning technology, more and more complex application scenarios cannot be represented by simple Euclidean data, such as molecular structures, recommendation systems, citation networks, and social networks. These application data, that is, non-Euclidean data, can be represented by graphs. Graph data includes nodes and edges. Nodes have their own attribute characteristics, and different nodes have different numbers of neighbor nodes. Traditional convolutional neural networks or recurrent neural networks cannot be used to represent graph data. In recent years, graph neural networks have attracted great attention from researchers. Compared with convolutional neural networks and recurrent neural networks, graph neural networks can embed node features into low-dimensional space by retaining topological structure information and node feature information, and have strong performance. Among them, graph autoencoders and variational graph autoencoders are effective frameworks for graph unsupervised learning (link prediction, node clustering, graph generation).

[0003] However, multi-task learning of graph data has not attracted much attention from researchers. In fact, learning multiple related tasks together can improve the overall generalization ability of the tasks. Existing graph neural network frameworks based on multi-task learning directly use the learned shared representation as the input of downstream tasks, which means that different downstream tasks use a common embedding representation for learning, and do not focus on learning the specific embedding information of a single task. In fact, using a common embedding representation for different tasks may be detrimental to the learning of each task, because the learned shared embedding representation may also learn the noise of other tasks. Summary of the invention

[0004] The purpose of the present invention is to provide a variational graph autoencoder method based on multi-task learning to address the deficiencies of the prior art. This method can make the embedded representation closer to the actual distribution of the sample space, and has strong competitiveness and robustness in link prediction tasks.

[0005] The technical solution for achieving the purpose of the present invention is:

[0006] A variational graph autoencoder method based on multi-task learning comprises the following steps:

[0007] S1: Preprocess the source data. The specific process is as follows:

[0008] S11: Process the source data in the citation network into graph data G = (V, E), where V is a node set and E is an edge set. Assume that a paper in the citation network is regarded as a node in the graph, the author and research direction of the paper are regarded as the characteristics of the node, an undirected edge is established between the paper and the cited paper, and the category to which the paper belongs is regarded as a label. Thus, a citation network constitutes a graph data set;

[0009] S12: Using the graph data set obtained in S11, obtain the degree matrix, adjacency matrix, and feature matrix corresponding to the graph;

[0010] S2: Divide the graph data set. The specific process is as follows:

[0011] S21: Mask part of the data in the graph dataset for semi-supervised learning;

[0012] S22: Divide the data in the matrix obtained in S12 into a training set, a validation set and a test set;

[0013] S3: Input the training set obtained in S22 into the shallow graph convolution layer to obtain the shallow shared embedding representation H, that is, input the adjacency matrix and feature matrix into the shallow graph convolution layer, and use the message propagation mechanism H = σ(AXW), where A is the adjacency matrix, X is the node feature matrix, W is the learnable parameter matrix, and σ is the activation function to aggregate the feature information and topological structure information of the current neighbor nodes to update the feature information of the current node, thereby obtaining the shallow shared embedding representation H;

[0014] S4: Input the shallow shared embedding representation H obtained in S3 into two different downstream network frameworks to obtain their respective embedding representations. The specific process is as follows:

[0015] S41: Input the shallow shared embedding representation H obtained in S3 into the graph convolutional network for link prediction to obtain the embedding representations Z_mean and Z_log;

[0016] S42: Add Z_mean and Z_log using Gaussian distribution to obtain an embedding representation Z that conforms to the Gaussian distribution;

[0017] S43: Use Z as a fake sample input to the discriminator, and based on the generative adversarial mechanism, make the embedded representation Z closer to the original sample distribution;

[0018] S44: Input the shallow representation H obtained in S3 into the graph convolutional network for node classification to obtain the embedded representation Z_nc;

[0019] S5: The two different embedding representations obtained in S4 are used for link prediction and semi-supervised node classification respectively. The specific process is as follows:

[0020] S51: Input the embedded representation Z obtained in S4 into the inner product layer to reconstruct the adjacency matrix for the link prediction task;

[0021] S52: Input the embedding representation Z obtained in S4 into the graph convolution layer to reconstruct the feature matrix as an auxiliary task for link prediction;

[0022] S53: Input the embedding representation Z_nc obtained in S4 into the graph convolutional network for node classification;

[0023] S54: Calculate the loss function and use the gradient descent algorithm to update the iteration parameters. After multiple iterations, the loss function can converge. The final loss function formula is:

[0024]

[0025] Where C is a set of node labels. If node i belongs to class c, y is the class label to which the node belongs. is the softmax probability that node i belongs to class c. When node i has a label, MASK i =1, otherwise MASK i =0,E q(Z|X，A) [logp(A|Z)]-KL[q(Z|X,A)||p(Z)] is the reconstruction loss of the adjacency matrix, where KL[q(·)||p(·)] is the relative entropy between the generated sample and the original sample. is the cross entropy loss for semi-supervised node classification, is the reconstruction loss of the feature matrix.

[0026] The beneficial effects of this technical solution are:

[0027] This technical solution is based on multi-task joint learning of unsupervised link prediction tasks and semi-supervised node classification tasks. Unlike other multi-task-based graph neural network frameworks, which directly use shared representations as inputs for different prediction or classification tasks, this technical solution only obtains shared representations at a shallow level and inputs the shared representations into exclusive network frameworks designed for different downstream tasks. In addition, in order to make the embedded representation of the link prediction task more robust, this technical solution adds an adversarial generative network framework, which uses the generator-discriminator game mechanism to make the embedded representation closer to the actual distribution of the sample space. Experimental results on three real citation network datasets show that the framework proposed in this technical solution is very competitive in link prediction tasks, and even outperforms the most advanced methods on one of the datasets.

[0028] This method can make the embedded representation closer to the actual distribution of the sample space, and is highly competitive and robust in link prediction tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 Schematic diagram of the process of the embodiment. DETAILED DESCRIPTION

[0030] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments, but the present invention is not limited thereto.

[0031] Example:

[0032] This example is applicable to data in non-Euclidean spaces, such as social networks, citation networks, and molecular structures.

[0033] Reference Figure 1 , a variational graph autoencoder method based on multi-task learning, comprising the following steps:

[0034] S1: Preprocess the source data. The specific process is as follows:

[0035] S11: In this example, graph datasets are collected in the citation network. The number of categories in each graph dataset is different. Each paper in the dataset has its own label. The graph dataset in the citation network is processed into graph data G = (V, E), where V is a node set and E is an edge set. Assume that a paper is regarded as a node in the graph, the author and research direction of the paper are regarded as the characteristics of the node, and an undirected edge is established between the paper and the cited paper. The category to which the paper belongs is regarded as a label. A citation network constitutes a graph dataset. The details of the three citation network graph datasets are shown in Table 1:

[0036] Table 1 Dataset

[0037] Dataset Number of nodes Number of edges Feature Dimension Number of categories Cora 2708 5429 1433 7 Citeseer 3327 4732 3703 6 Pubmed 19717 44338 500 3 ;

[0038] S12: Using the graph data set obtained in S11, obtain the degree matrix, adjacency matrix, and feature matrix corresponding to the graph;

[0039] S2: Divide the graph data set. The specific process is as follows:

[0040] S21: Mask part of the data in the graph dataset for semi-supervised learning;

[0041] S22: Divide the data in the matrix obtained in S12 into a training set, a validation set and a test set;

[0042] S3: Input the training set obtained in S22 into the shallow graph convolution layer to obtain the shallow shared embedding representation H, that is, input Cora = (A, X) into the shallow graph convolution layer, and use the message propagation mechanism H = σ (AXW), where A is the adjacency matrix, X is the node feature matrix, W is the learnable parameter matrix, and σ is the activation function to aggregate the feature information and topological structure information of the current neighbor nodes to update the feature information of the current node to obtain the shallow shared embedding representation H. The formula of the graph convolution network is:

[0043]

[0044] Among them, the activation function σ(·) = ReLU(·), W is the weight matrix, and D is the degree matrix of the adjacency matrix;

[0045] S4: Input the shallow shared embedding representation H obtained in S3 into two different downstream network frameworks to obtain their respective embedding representations. The specific process is as follows:

[0046] S41: Input the shallow shared embedding representation H obtained in S3 into the graph convolutional network for link prediction to obtain the embedding representations Z_mean and Z_log;

[0047] S42: Add Z_mean and Z_log using Gaussian distribution to obtain an embedding representation Z that conforms to the Gaussian distribution;

[0048] S43: Use Z as a fake sample input to the discriminator, and based on the generative adversarial mechanism, make the embedded representation Z closer to the original sample distribution;

[0049] S44: Input the shallow representation H obtained in S3 into the graph convolutional network for node classification to obtain the embedded representation Z_nc;

[0050] S5: The two different embedding representations obtained in S4 are used for link prediction and semi-supervised node classification respectively. The specific process is as follows:

[0051] S51: Input the embedded representation Z obtained in S4 into the inner product layer to reconstruct the adjacency matrix for the link prediction task, and obtain the reconstructed adjacency matrix:

[0052]

[0053] The loss function for reconstructing the adjacency matrix is:

[0054] L re =E q(Z|X,A) [logp(A|Z)]-KL[q(Z|X,A)||p(Z)];

[0055] S52: Input the embedding representation Z obtained in S4 into the graph convolution layer to reconstruct the feature matrix as an auxiliary task for link prediction, and obtain the reconstructed feature matrix:

[0056]

[0057] The loss function for reconstructing the feature matrix is:

[0058]

[0059] S53: Input the embedding representation Z_nc obtained in S4 into the graph convolutional network for node classification. The loss function of the point classification task is:

[0060]

[0061] S54: Calculate the loss function and use the gradient descent algorithm to update the iteration parameters. After multiple iterations, the loss function can converge. The final loss function formula is:

[0062]

[0063] Where C is a set of node labels. If node i belongs to class c, y is the class label to which the node belongs. is the softmax probability that node i belongs to class c. When node i has a label, MASK i =1, otherwise MASK i =0,E q(Z|X，A) [logp(A|Z)]-KL[q(Z|X,A)||p(Z)] is the reconstruction loss of the adjacency matrix, where KL[q(·)||p(·)] is the relative entropy between the generated sample and the original sample. is the cross entropy loss for semi-supervised node classification, is the reconstruction loss of the feature matrix.

[0064] After 50 iterations, the loss function has converged and the training is stopped.

[0065] The experimental results of the three graph datasets are shown in Table 2 and Table 3:

[0066] Table 2 Link prediction: AUC and AP scores

[0067]

[0068] Table 3 Node classification: accuracy

[0069] Methods Cora Pubmed Citeseer GCN 0.815 0.790 0.703 Planetoid 0.757 0.772 0.947 DeepWalk 0.972 0.653 0.432 MTGAE 0.790 0.804 0.718 This example 0.809 0.861 0.666 .

Claims

1. A variational graph autoencoder method based on multi-task learning, characterized in that: The steps include: S1: Preprocess the source data. The specific process is as follows: S11: Process the source data in the citation network into graph data G = (V, E), where V is a node set and E is an edge set. Assume that a paper in the citation network is regarded as a node in the graph, the author and research direction of the paper are regarded as the characteristics of the node, an undirected edge is established between the paper and the cited paper, and the category to which the paper belongs is regarded as a label. Thus, a citation network constitutes a graph data set; S12: Using the graph data set obtained in S11, obtain the degree matrix, adjacency matrix, and feature matrix corresponding to the graph; S2: Divide the graph data set. The specific process is as follows: S21: Mask part of the data in the graph dataset for semi-supervised learning; S22: Divide the data in the matrix obtained in S12 into a training set, a validation set and a test set; S3: Input the training set obtained in S22 into the shallow graph convolution layer to obtain the shallow shared embedding representation H, that is, input the adjacency matrix and feature matrix into the shallow graph convolution layer, and use the message propagation mechanism H=σ(AXW), where A is the adjacency matrix, X is the node feature matrix, W is the learnable parameter matrix, and σ is the activation function to aggregate the feature information and topological structure information of the current neighbor nodes to update the feature information of the current node, thereby obtaining the shallow shared embedding representation H; S4: Input the shallow shared embedding representation H obtained in S3 into two different downstream network frameworks to obtain their respective embedding representations. The specific process is as follows: S41: Input the shallow shared embedding representation H obtained in S3 into the graph convolutional network for link prediction to obtain the embedding representations Z_mean and Z_log; S42: Add Z_mean and Z_log using Gaussian distribution to obtain an embedding representation Z that conforms to the Gaussian distribution; S43: Use Z as a fake sample input to the discriminator, and based on the generative adversarial mechanism, make the embedded representation Z closer to the original sample distribution; S44: Input the shallow representation H obtained in S3 into the graph convolutional network for node classification to obtain the embedded representation Z_nc; S5: The two different embedding representations obtained in S4 are used for link prediction and semi-supervised node classification respectively. The specific process is as follows: S51: Input the embedded representation Z obtained in S4 into the inner product layer to reconstruct the adjacency matrix for the link prediction task; S52: Input the embedding representation Z obtained in S4 into the graph convolution layer to reconstruct the feature matrix as an auxiliary task for link prediction; S53: Input the embedding representation Z_nc obtained in S4 into the graph convolutional network for node classification; S54: Calculate the loss function and use the gradient descent algorithm to update the iteration parameters. After multiple iterations, the loss function can converge. The final loss function formula is: Where C is a set of node labels. If node i belongs to class c, y is the class label to which the node belongs. is the softmax probability that node i belongs to class c. When node i has a label, MASK i =1, otherwise MASK i =0,E q(Z|X,A) [logp(A|Z)]-KL[q(Z|X,A)||p(Z)] is the reconstruction loss of the adjacency matrix, where KL[q(·)||p(·)] is the relative entropy between the generated sample and the original sample. is the cross entropy loss for semi-supervised node classification, is the reconstruction loss of the feature matrix.

Images