A graph representation learning method and device based on generative adversarial network

Abstract

The unsupervised graph representation learning method and device based on generative adversarial network include: pre-embedding generation stage, using LLE dimension reduction method to compress the features of original data, and recording the result after dimension reduction as pre-embedding; generative adversarial stage; learning the embedding representation of nodes specifically through a generative adversarial network model; the generative adversarial network is composed of two parts, a generator and a discriminator, each of which has a node embedding layer, and based on the idea of ​​adversarial learning, each promotes the other to optimize the embedding representation of nodes; the generative adversarial stage is executed multiple times until the generative adversarial model converges; at this time, the node embedding layers Z^G and Z^D in the model are the embedding representation matrices of the graph nodes finally learned. The present invention effectively removes redundant information and enables effective optimization of the GAN model.

Description

Technical Field

[0001] The present invention relates to a representation learning method and device for graph structure data. Background Art

[0002] There are all kinds of intricate relationships in the real world, which constitute many huge relationship graphs. In order to better discover valuable information from these graph-structured data, graph representation learning methods are often used in the field of machine learning. This type of method converts the data that constitutes the node information in the graph into a low-dimensional vector form, called an embedding representation, which is then applied to downstream tasks such as node classification, link prediction, and recommendation. This type of method is also called graph embedding, network representation learning, and network embedding method.

[0003] Graph representation learning methods can be roughly divided into supervised graph representation learning and unsupervised graph representation learning according to the different learning processes. The most advanced graph representation learning methods usually adopt supervised models based on graph neural networks, or unsupervised models based on random walks or autoencoders. Supervised models require a large amount of reliable label information in the data; unsupervised models do not have strict requirements on the data set, but the embedded representations learned by them have lower accuracy in expressing information than those of supervised models.

[0004] In the real world, it is extremely difficult to obtain enough reliable labeled data for large-scale datasets, so using unsupervised models to implement graph representation learning is a more ideal solution. Existing unsupervised methods have achieved some results, but they still lag behind supervised methods in terms of the accuracy of embedded representations, which limits their application in practical application scenarios that lack labels, such as life safety, risk prediction and other important fields. Therefore, it is of great practical significance to develop a new unsupervised graph representation learning method that can accurately express the embedded representation of information. Summary of the invention

[0005] In order to overcome the problem of low accuracy of expression information of embedded representation obtained by existing unsupervised graph representation learning methods, the present invention provides an unsupervised graph representation learning method and device based on a generative adversarial network.

[0006] The present invention utilizes the graph structure, converts the original data into low-dimensional pre-embedding through the LLE (local linear embedding) method, and uses a generative adversarial model to learn and optimize the embedded representation, thereby realizing graph representation learning.

[0007] The unsupervised graph representation learning method based on generative adversarial networks is divided into a pre-embedding generation phase and a generative adversarial phase. The specific steps are as follows:

[0008] Step 1: Pre-embedding generation stage: Use the LLE dimensionality reduction method to compress the features of the original data and record the result after dimensionality reduction as pre-embedding;

[0009] 1.1 Original feature matrix of initial graph nodes Use the KNN algorithm to find the K nearest neighbors of each node sample, reconstruct the node through the K neighbor nodes of each node, and calculate the weight A ij , the reconstruction error is shown in formula (1):

[0010]

[0011] Where N represents the number of nodes, d0 represents the dimension of the original feature vector, and the weight A ij represents the contribution of the jth data point to the i-th reconstruction. To calculate the weight, the cost function needs to be minimized under two constraints: First, each node vector x i It can only be rebuilt through neighbor nodes. If node j is not in the neighbor set, then A ij =0; secondly, the sum of the rows of the weight matrix is ​​one, that is, ∑ j A ij =1.

[0012] 1.2 The optimal reconstruction weight matrix A obtained by formula (1) is used to calculate the embedding matrix after dimensionality reduction And satisfying d<<d0, the embedding loss is shown in formula (2):

[0013]

[0014] During the training process, the weight matrix A remains unchanged, and the embedding matrix result is obtained through formula (2).

[0015] Step 2: Generative adversarial phase: Through a generative adversarial network model, the embedded representation of the node is specifically learned; the generative adversarial network consists of two parts: the generator and the discriminator. Each of them has a node embedding layer. Based on the idea of ​​adversarial learning, they push each other to optimize the embedded representation of the node;

[0016] 2.1 Before training begins, initialize the generator and discriminator, and use the pre-embedding matrix obtained in the previous step as the initial value of their respective node embedding layers;

[0017] 2.2 Node pair sampling: Starting from node i, based on the adjacency weight A random walk can get a path Path i ; Among them, the adjacency weight of node i is an N-dimensional vector, where N represents the number of nodes; The component in the jth dimension The calculation formula is shown in formula (3) (4):

[0018]

[0019] in, represents the set of neighbor nodes of node i, represents the embedding representation of node i in the generator G, Z G represents the node embedding layer in the generator G;

[0020] Path i is the set of all nodes that are passed when randomly walking with node i as the starting node; for small-scale data sets, when the next node of the walk already has Path i When the number of steps reaches a certain value, the walk stops. For large-scale data sets, the walk stops when the number of steps reaches a certain value.

[0021] is the set of node pairs about node i used for generator training, Path i Each pair of adjacent nodes in the form a node pair to join is the set of node pairs about node i used for discriminator training, Path i The head and tail nodes in the form a node pair and join By performing multiple random walks starting from node i, and Get a sufficient number of node pairs for subsequent training steps;

[0022] 2.3 Discriminator training; using the Adam algorithm to minimize the discriminator loss function L D , optimize the node embedding layer Z of the discriminator D D ; The calculation formula of the discriminator objective function is shown in formula (5):

[0023]

[0024] in, represents the embedding representation of node i in the discriminator D,<h,t> represents a node pair consisting of the head node h and the tail node t, norm is a normalization function;

[0025] 2.4 Generator training; using the Adam algorithm to minimize the generator loss function L G , optimize the node embedding layer Z of the generator G G ; The calculation formula of the generator objective function is shown in formula (6):

[0026]

[0027] Step 3: Execute the generative adversarial phase multiple times until the generative adversarial model converges; at this point, the node embedding layer Z in the model G and Z DThis is the embedding representation matrix of the graph nodes finally learned.

[0028] The second aspect of the present invention relates to an unsupervised graph representation learning device based on a generative adversarial network, comprising a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, they are used to implement the unsupervised graph representation learning method based on a generative adversarial network of the present invention.

[0029] The third aspect of the present invention relates to a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the unsupervised graph representation learning method based on a generative adversarial network of the present invention.

[0030] The present invention combines the above technologies to propose an unsupervised graph representation learning method based on a generative adversarial network; in order to solve the problem that the embedding representation obtained by the existing unsupervised graph representation learning method has low accuracy in expressing information, a generator for obtaining node positive and negative sample pairs based on neighbor weighted random walks and a discriminator structure responsible for distinguishing whether there is a real connection between node pairs are used to automatically learn and optimize node embedding in an adversarial manner; in addition, in order to solve the problem that pre-embedding has a large impact on the training of the generative adversarial model, the LLE dimensionality reduction technology is used to obtain high-quality pre-embedding.

[0031] The advantages of the present invention are: (1) LLE dimensionality reduction produces high-quality pre-embeddings, ensuring effective optimization of the GAN model. (2) The generative adversarial model selects the existence of a node pair connection as the adversarial point, and uses random walk sampling instead of the strategy of randomly generating node embeddings to effectively remove redundant information. (3) As an unsupervised graph representation learning method, there are no strict requirements for the data set, and the learned embedding representation is equal to or even exceeds the supervised learning method in terms of information expression accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 It is an overall flow chart of the method of the present invention.

[0033] Figure 2 It is a specific illustration of the method of the present invention. DETAILED DESCRIPTION

[0034] The specific implementation of the present invention is described below with reference to embodiments.

[0035] Example 1

[0036] This embodiment relates to a document classification and document recommendation method using the unsupervised graph representation learning method based on a generative adversarial network of the present invention.

[0037] Suppose we perform graph representation learning on the Citeseer citation dataset. This dataset has 3327 nodes and 4552 edges, and the feature dimension of each node is 3703 dimensions.

[0038] The graph representation learning method based on generative adversarial network is used to learn graph representation on Citeseer dataset. The steps are as follows: 1. Obtain pre-embedding;

[0039] 1.1 For the original feature matrix X with the initial graph node dimension of 3327×3703, the KNN algorithm is used to find the 12 nearest neighbors of each node sample, and the node is reconstructed through the 12 neighbor nodes of each node. The weight matrix A for the optimal reconstruction is calculated, and the dimension of the weight matrix A is 3327×3327.

[0040] 1.2 Fix the optimal reconstruction weight matrix A obtained in 1.1 and use formula (2) to calculate the embedding matrix Z with a dimension of 3327×64 after dimensionality reduction.

[0041] 2. Adversarial training;

[0042] 2.1 Initialize the Generative Adversarial Network.

[0043] (1) Initialize the generator embedding matrix Z G , the dimension is 3327×64, and the initial value is the value of the pre-embedded matrix in the previous step.

[0044] (2) Initialize the discriminator embedding matrix Z D , the dimension is 3327×64, and the initial value is the value of the pre-embedded matrix in the previous step.

[0045] 2.2 Obtain training samples.

[0046] 2.3 Node pair set Sample D and the embedding matrix Z D Calculate the discriminator loss function L D .

[0047] 2.4 Use the Adam algorithm to minimize the discriminator loss function L D , optimize the discriminator embedding matrix Z D .

[0048] 2.5 Node pair collection Sample G , embedding matrix Z D and the embedding matrix Z G Calculate the generator loss function L G .

[0049] 2.6 Use the Adam algorithm to minimize the generator loss function L G , optimize the generator embedding matrix ZG .

[0050] 3. Repeat the training to obtain the node embedding representation;

[0051] 3.1 Take steps 2.2 to 2.6 as a complete iteration process, and repeat the iteration until 200 iterations or the loss function L G , L D The value of is no longer significantly reduced. The changes in the model's internal embedding matrix during the entire iteration process are as follows: Figure 2 shown.

[0052] 3.2 Embedding Representation Matrix Z of Size 3327×64 D and Z G , which is the 64-dimensional embedding representation result of the Citeseer dataset node.

[0053] 4 Input the newly added document features into the model to achieve document subject classification and related document recommendation;

[0054] When a newly published document is added to the citation database, the newly constructed graph network is input into the trained model to obtain the embedding results of the document nodes. The subject classification of the documents is carried out using the node embedding similarity between the documents, and high-quality related document recommendations are provided to researchers to help them discover the latest research results in the target field.

[0055] This concludes the specific implementation steps of graph representation learning for the Citeseer dataset using the iterative aggregation graph representation learning method based on generative adversarial networks.

[0056] Example 2

[0057] This embodiment relates to an unsupervised graph representation learning device based on a generative adversarial network, including a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, they are used to implement the method of Example 1.

[0058] Example 3

[0059] This embodiment relates to a computer-readable storage medium on which a program is stored. When the program is executed by a processor, the method of Embodiment 1 is implemented.

[0060] The contents described in the embodiments of this specification are merely an enumeration of the implementation forms of the inventive concept. The protection scope of the present invention should not be regarded as limited to the specific forms described in the embodiments. The protection scope of the present invention also extends to equivalent technical means that can be conceived by those skilled in the art based on the inventive concept.

Claims

1. The literature classification and literature recommendation method using the unsupervised graph representation learning method based on generative adversarial networks includes the following steps: Step 1: Pre-embedding generation stage: Use the LLE dimensionality reduction method to compress the features of the original data and record the result after dimensionality reduction as pre-embedding; 1.1 Original feature matrix of initial graph nodes Use the KNN algorithm to find the K nearest neighbors of each node sample, reconstruct the node through the K neighbor nodes of each node, and calculate the weight A ij , the reconstruction error is shown in formula (1): Where N represents the number of nodes, d0 represents the dimension of the original feature vector, and the weight A ij represents the contribution of the jth data point to the i-th reconstruction; in order to calculate the weight, it is necessary to minimize the cost function under two constraints: first, each node vector x i It can only be rebuilt through neighbor nodes. If node j is not in the neighbor set, then A ij =0; secondly, the sum of the rows of the weight matrix is ​​one, that is, ∑ j A ij =1; 1.2 The optimal reconstruction weight matrix A obtained by formula (1) is used to calculate the embedding matrix after dimensionality reduction And satisfying d<<d0, the embedding loss is shown in formula (2): During the training process, the weight matrix A remains unchanged, and the embedding matrix result is obtained through formula (2); Step 2: Generative adversarial phase: Through a generative adversarial network model, the embedded representation of the node is specifically learned; the generative adversarial network consists of two parts: the generator and the discriminator. Each of them has a node embedding layer. Based on the idea of ​​adversarial learning, they push each other to optimize the embedded representation of the node; 2.1 Before training begins, initialize the generator and discriminator, and use the pre-embedding matrix obtained in the previous step as the initial value of their respective node embedding layers; 2.2 Node pair sampling; Starting from node i, based on the adjacency weight A random walk can get a path Path i ; Among them, the adjacency weight of node i is an N-dimensional vector, where N represents the number of nodes; The component in the jth dimension The calculation formula is shown in formula (3) (4): in, represents the set of neighbor nodes of node i, represents the embedding representation of node i in the generator G, Z G represents the node embedding layer in the generator G; Path i is the set of all nodes that are passed when randomly walking with node i as the starting node; for small-scale data sets, when the next node of the walk already has Path i When the number of steps reaches a certain value, the walk stops. For large-scale data sets, the walk stops when the number of steps reaches a certain value. is the set of node pairs about node i used for generator training, Path i Each pair of adjacent nodes in the form a node pair to join is the set of node pairs about node i used for discriminator training, Path i The head and tail nodes in the form a node pair and join By performing multiple random walks starting from node i, and Get a sufficient number of node pairs for subsequent training steps; 2.3 Discriminator training; using the Adam algorithm to minimize the discriminator loss function L D , optimize the node embedding layer Z of the discriminator D D ; The calculation formula of the discriminator objective function is shown in formula (5): in, represents the embedding representation of node i in the discriminator D,<h,t> represents a node pair consisting of the head node h and the tail node t, norm is a normalization function; 2.4 Generator training; using the Adam algorithm to minimize the generator loss function L G , optimize the node embedding layer Z of the generator G G ; The calculation formula of the generator objective function is shown in formula (6): Step 3: Execute the generative adversarial phase multiple times until the generative adversarial model converges; at this point, the node embedding layer Z in the model G and Z D That is the embedding representation matrix of the graph node finally learned; Step 4. Input the newly added document features into the model to achieve document subject classification and related document recommendation; When a newly published document is added to the citation database, the newly constructed graph network is input into the trained model to obtain the embedding results of the document nodes, and the node embedding similarities between documents are used to perform subject classification and recommendation of the documents.

2. A document classification and document recommendation device using an unsupervised graph representation learning method based on a generative adversarial network, characterized in that: It includes a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, it is used to implement the document classification and document recommendation method based on the unsupervised graph representation learning method based on the generative adversarial network as described in claim 1.

3. A computer-readable storage medium, characterized in that: A program is stored thereon, and when the program is executed by a processor, the document classification and document recommendation method based on the unsupervised graph representation learning method based on the generative adversarial network as described in claim 1 is implemented.

Images