Simple unsupervised graph representation learning method

Abstract

The invention discloses a simple unsupervised graph representation learning method for effective and efficient comparative learning. Specifically, a multi-loss function proposed in the method uses complementarity between structural information and neighborhood information to expand inter-class differences, and an upper bound loss is added to realize a finite distance between positive sample representation and an anchor point so as to reduce intra-class differences. Therefore, the combination of expanding the inter-class difference and reducing the intra-class difference realizes a smaller generalization error, so that an effective model is obtained. In addition, the method removes a data enhancement and discriminator which is widely used in a previous graph contrast learning method, can output low-dimensional embedding, and improves the efficiency of the model. Experimental results on various real-world data sets verify the effectiveness and the high efficiency of the method, and compared with the most advanced method, the method is better in performance.

Description

Technical Field

[0001] The present invention belongs to the field of graph representation learning, and more specifically, relates to a simple unsupervised graph representation learning method. Background Art

[0002] Due to the widespread application of graph neural networks (GNNs), unsupervised graph representation learning (UGRL) has also received great attention recently. UGRL does not require a large number of labeled nodes for training, and it can output discriminative representations by simultaneously learning representations and preserving the local structure of samples. The discriminative representation ensures that downstream tasks output effective models, making UGRL show remarkable performance in practical applications. As one of the representative methods of UGRL, contrastive learning is proposed to maximize the complementary information (MI) between input content and related content.

[0003] The key difference between graph contrastive learning methods lies in the definition of input content and its related content, for example, Deep Graph Infomax (DGI) maximizes the mutual information between node representation and graph summary. Graph Mutual Information (GMI) maximizes the mutual information between input graph and output graph. GRACE maximizes the mutual information between two views of each node through various data augmentations such as attribute masking or edge perturbations.

[0004] Although previous methods are effective in many tasks of representation learning, they usually rely on data augmentation to generate input content and its related content that maximizes mutual information, resulting in expensive computational costs for the training process. Therefore, these previous methods are usually inefficient, especially for large-scale datasets, where the computational cost increases dramatically with the increase in the number of samples or the embedding dimension. In fact, the reasons why previous contrastive learning methods have weak scalability include data augmentation, high-dimensional embedding, contrastive loss, etc. First, previous works usually generate multiple views as related content through data augmentation, for example, by removing edges and masking node features to generate multiple views. Therefore, the computational cost of data augmentation (including data generation and data encoding) accounts for about 20%-40% of the training time. Second, existing works improve the representation quality by increasing the dimension of embedding, which leads to an increase in training time. The reason is that the effectiveness of these methods is sensitive to the dimension. For example, DGI and GMI achieve the best accuracy at 512 dimensions. Third, previous work usually designs a discriminator (containing learnable parameters) for the objective function, which is computationally prohibitive. For example, DGI and MVGRL use a discriminator to measure the consistency of node embeddings and graph embeddings, which takes about 10%-30% of the training time. Obviously, it is necessary to find a UGRL method with low computational cost in training process and high representation quality. Summary of the invention

[0005] The purpose of the present invention is to overcome the deficiencies of the prior art and provide a new contrastive learning method, namely Simple Unsupervised Graph Representation learning (SUGRL), to achieve the effectiveness and scalability of representation learning.

[0006] We design a simple framework, namely SUGRL, to achieve effective and efficient contrastive learning. To achieve effectiveness, we design two triplet losses to explore the complementary information between structural information and neighbor information to enlarge the inter-class differences, and an upper limit loss to reduce the intra-class differences. To improve efficiency, our method is designed without GCNs for generating anchor and negative sample representations, as well as data augmentation and discriminators from previous graph contrastive learning. We conduct comprehensive experiments on various real-world datasets and experimental results show that our method consistently outperforms the state-of-the-art methods in terms of accuracy and scalability.

[0007] The present invention has the following beneficial effects:

[0008] 1) The present invention jointly considers structural information and neighbor information to explore their complementary information, aiming to expand the inter-class difference, and at the same time designs an upper limit loss to achieve small intra-class difference, ensuring effectiveness;

[0009] 2) In order to improve efficiency, the present invention removes data augmentation and discriminator from contrastive learning, which makes our method easy to achieve scalability on large-scale datasets;

[0010] 3) A comprehensive empirical study was conducted on 8 public benchmark datasets, and compared with 11 comparative methods, the effectiveness and efficiency of the present invention in node classification were verified. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 It is a flowchart of a specific implementation method of the simple unsupervised graph representation learning method of the present invention;

[0012] Figure 2 It is a structural flow chart of the SUGRL method of the present invention;

[0013] Figure 3 It is a module diagram for generating positive sample representation in the present invention; DETAILED DESCRIPTION

[0014] The specific implementation of the present invention is described below in conjunction with the accompanying drawings so that those skilled in the art can better understand the present invention. It should be noted that in the following description, when the detailed description of known functions and designs may dilute the main content of the present invention, these descriptions will be omitted here.

[0015] Example

[0016] Figure 1 This is a flowchart of a specific implementation method of the simple unsupervised graph representation learning method in the present invention. Figure 1 As shown, the specific steps of the simple unsupervised graph representation learning method in the present invention include:

[0017] S101: Input feature matrix X and corresponding graph structure:

[0018] Let G = (V, E) represent a graph, where V = {v1,v2,…,v N}and Denote the node set and edge set respectively. The feature matrix and adjacency matrix are expressed as and A∈{0,1} N×N , where x i is node v i The characteristics of i ,v j )∈E then a ij =1 otherwise a ij = 0. In addition, we assume that there exists a set of latent classes C on the representation space H, since both embeddings and samples are unlabeled in unsupervised learning.

[0019] S102: Constructing SUGRL learning model:

[0020] like Figure 2 As shown in the figure, a multi-layer perceptron (MLP) is first used to generate anchor points on the input representation with semantic information, and a graph convolutional network (GCN) and a neighbor sampling method are used to generate two different types of positive sample representations. Then, a row-by-row random permutation method is used on the anchor points to generate negative sample representations. The model further designs a new multi-loss model that reduces intra-class differences while expanding inter-class differences, forcing the anchor points to be close to the positive sample representation and away from the negative sample representation.

[0021] S103: Set the number of iterations t=1.

[0022] S104: Use efficient MLP to generate anchor and negative sample representations:

[0023] The MLP network on features with semantic information is used to generate anchor points, and the row random permutation method of the anchor points is used to generate negative sample representations.

[0024] X (l+1) = Dropout(σ(X (l) W (l) ))

[0025] H=X (l+1) W (l+1)

[0026] First, an MLP is used on the input X to generate anchor embeddings with semantic information, namely:

[0027] Where X (0) =X, σ is an activation function, W (l) is the weight of the lth layer.

[0028] For the generation of negative sample representations, the rows are randomly permuted to embed the anchors to obtain negative sample representations and further reduce the training time, i.e.

[0029] H - =Shuffle([h1,h2,...,h N ])

[0030] S105: Use GCN to generate positive sample representation:

[0031] This method obtains diversified information by generating two types of positive sample representations, namely structural embedding and neighbor embedding. Figure 3 As shown in Figure 2, we use graph convolutional networks (GCN) and neighbor sampling methods to generate them. The specific modules include:

[0032] S301: Obtain positive sample representation of structure information type:

[0033] In order to obtain the structural information of the graph, we apply the widely used graph convolutional network (GCN) as the basic encoder:

[0034]

[0035] Among them, H +(0) =X and H +(l) Represents the l-th layer feature. is a symmetric normalized adjacency matrix, yes The degree matrix, I N is the identity matrix. It is worth noting that our method directly shares weights between the MLP and GCN encoders (i.e., W (l) ) to further reduce time cost.

[0036] S302: Obtaining positive sample representation of neighbor information type:

[0037] In order to obtain a positive sample representation of the neighbor information type, we first store the neighbor embedding index of all nodes, then sample, and then calculate the average of the samples. In this way, the neighbor information of the node can be obtained efficiently:

[0038]

[0039] where m is the number of sampled neighbors, N i Represents node v i The set of 1-hop neighbors of .

[0040] In summary, structural embedding and neighbor embedding focus on all neighbors and a subset of neighbors, respectively. That is, structural embedding represents a general representation, while neighbor embedding is specific. Therefore, they interpret samples from different perspectives, so that considering them jointly may obtain their complementary information.

[0041] S106: Training SUGRL model:

[0042] The model further designs multiple loss functions to make the anchor points close to the positive sample representation and away from the negative sample representation.

[0043] Given an anchor, a positive representation, and a negative representation, the goal of contrastive learning is to make the positive pair (i.e., the anchor and the positive representation) close while keeping the negative pair (i.e., the anchor and the negative representation) far apart. In SUGRL, we consider the triplet loss as the basis and design an upper limit loss to remove the discriminator from our method (improve efficiency) and reduce the intra-class difference and increase the inter-class difference (improve effect). Specifically, the triplet loss of each sample can be expressed as:

[0044] α+d(h,h + )<d(h,h - )

[0045] where d(·) is a similarity measure (e.g., -norm distance), α is a non-negative value to ensure a "safe" distance between the positive sample representation and the negative sample representation. By summing up the losses of all negative samples, the above formula can be expanded to:

[0046]

[0047] in{·} + =max{·,0}, k is the number of negative samples.

[0048] To increase inter-class differences, we should push negative pairs away from positive pairs. To this end, we use the triplet loss on the two positive samples defined in S105:

[0049]

[0050] As mentioned in the previous section, structural embedding (i.e., h + ) and neighbor embedding (i.e. ) are different. According to the distance between the two positive sample representations and the anchor point, it can be divided into two cases, namely case 1: and case 2: if Then even if the loss term L N The corresponding term in is zero, L S {·} + The term may also be non-zero. In this case, the loss term L S Still valid, while the loss term L N Invalid. Therefore, negative samples will continue to pass through the loss term L S is pushed away from the anchor point, thereby increasing the difference between classes. Similar to case 1, case 2 can also increase the difference between classes. Based on the above analysis, either case 1 or case 2 can increase the difference between classes. In particular, if one of them is invalid, the other is still valid to further increase the difference between classes. Therefore, L S and L N It is possible to obtain complementary information from structure embedding and neighbor embedding so that they can increase the inter-class differences. triplet Require d(h,h + ) 2 and The distance between the anchor point and the positive sample representation should be greater than α, but it ignores the distance between the anchor point and the positive sample representation. If the distance between the anchor point and the positive sample representation is large, L triplet {·} + The term may also be non-zero. However, in this case, the intra-class variance may be large, which is not conducive to reducing the generalization error. To address this issue, we study an upper bound (i.e., α + β) for both negative and positive pairs through the following objective function:

[0051] α+d(h,h + )<d(h,h - )<d(h,h + )+α+β

[0052] where β is a non-negative tuning parameter. The upper bound α+β ensures that the distance between the negative sample representation and the anchor point is bounded, and therefore the distance between the positive sample representation and the anchor point based on the triplet loss of each sample is also bounded. Therefore, the intra-class variance is reduced. After summing up the losses of all negative samples, the proposed upper bound loss to reduce the intra-class variance is defined as follows:

[0053]

[0054] in{·} - =min{·,0} is the target in the above formula.

[0055] It is worth noting that the upper limit is not set arbitrarily, but is carefully designed by considering the importance of reducing the generalization error in contrastive learning. In addition, our method reduces the generalization error by reducing the intra-class variance and increasing the inter-class variance, thereby improving the generalization ability of the model.

[0056] Finally, the triple loss L S and L N Integrated with the upper bound loss on intra-class variance, our proposed multi-loss is formulated as:

[0057] L=ω1L S +ω2L N +L U

[0058] Where ω1 and ω2 are L S and L N The weight of .

[0059] S107: Determine whether t<t max , t max Indicates the predicted maximum number of iterations. If yes, go to step S108, otherwise go to step S109.

[0060] S108: Let t=t+1, update the model, and return to step S104.

[0061] S109: Output the final training model, which can generate low-dimensional, high-quality representations for various tasks.

[0062] In order to better illustrate the technical effect of the present invention, a specific example is used to experimentally verify the present invention. In this experimental verification, two citation network datasets (i.e., Cora and Citeseer) and one large-scale dataset (i.e., Ogbn-arxiv) are used. The comparison methods include one semi-supervised learning algorithm (i.e., GCN) and four unsupervised learning algorithms (i.e., DGI, GMI, MVGRL and GIC).

[0063] Table 1

[0064]

[0065] As shown in Table 1, the classification accuracy and execution time of various methods on real graph structure datasets. From the results, the present invention (SUGRL) outperforms all the comparison methods in terms of classification accuracy. Secondly, the present invention (SUGRL) has the best efficiency. Compared with other self-supervised methods, SUGRL is on average 122.4 times faster than the slowest comparison method GMI and 4.4 times faster than the fastest comparison method GIC. Therefore, for the node classification task, the present invention outperforms other comparison methods on almost all datasets in terms of both model performance and execution time.

[0066] Although the above describes the illustrative specific embodiments of the present invention to facilitate those skilled in the art to understand the present invention, it should be clear that the present invention is not limited to the scope of the specific embodiments. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations using the concept of the present invention are protected.

Claims

1. A simple unsupervised graph representation learning method, characterized in that: The following steps are involved: S1: Input feature matrix X and corresponding graph structure: Let G = (V, E) represent a graph, where V = {v1,v2,…,v N }and Denote the node set and edge set respectively. The feature matrix and adjacency matrix are expressed as and A∈{0,1} N×N , where x i is node v i The characteristics of i ,v j )∈E then a ij =1 otherwise a ij = 0. In addition, we assume that there exists a set of latent classes C on the representation space H, since both embeddings and samples are unlabeled in unsupervised learning. S2: Constructing SUGRL learning model: First, a multi-layer perceptron (MLP) is used to generate anchors on the input representation with semantic information, and a graph convolutional network (GCN) and a neighbor sampling method are used to generate two different types of positive sample representations. Then, a row-by-row random permutation method is used on the anchors to generate negative sample representations. The model further designs a new multi-loss to reduce intra-class differences while expanding inter-class differences, forcing the anchors to be close to the positive sample representation and away from the negative sample representation. S3: Let the number of iterations t=1. S4: Use efficient MLP to generate anchor and negative sample representations: The MLP network on features with semantic information is used to generate anchor points, and the row random permutation method of the anchor points is used to generate negative sample representations. X (l+1) =Dropout(σ(X (l) W (l) )) H=X (l+1) W (l+1) First, an MLP is used on the input X to generate anchor embeddings with semantic information, namely: Where X (0) =X, σ is an activation function, W (l) is the weight of the lth layer. For the generation of negative sample representations, the rows are randomly permuted to embed the anchors to obtain negative sample representations and further reduce the training time, i.e. H - =Shuffle([h1,h2,...,h N ]) S5: Use GCN to generate positive sample representation: Our method obtains diverse information by generating two types of positive sample representations, namely, structural embedding and neighbor embedding. S6: Training SUGRL model: The model further designs multiple loss functions to make the anchor points close to the positive sample representation and away from the negative sample representation. In SUGRL, the triplet loss is considered as the basis, and an upper limit loss is designed. The triplet loss of each sample can be expressed as: α+d(h,h + )<d(h,h - ) where d(·) is a similarity measure (e.g., l2-norm distance) and α is a non-negative value to ensure a “safe” distance between the representations of positive and negative samples. By summing up the losses of all negative samples, the equation can be expanded to: in{·} + =max{·,0}, k is the number of negative samples. We use triplet loss on the two types of positive samples defined in S5: We study an upper bound (i.e., α + β) for both negative and positive pairs via the following objective function: α+d(h,h + )<d(h,h - )<d(h,h + )+α+β where β is a non-negative tuning parameter. The upper bound α+β ensures that the distance between the negative sample representation and the anchor point is bounded, and the distance between the positive sample representation and the anchor point is also bounded. Therefore, the intra-class variance is reduced. After summing up the losses of all negative samples, the proposed upper bound loss to reduce the intra-class variance is defined as follows: in{·} - =min{·,0},d(h,h + ) 2 +α+β is a target in the equation. It is worth noting that the upper limit is not set arbitrarily, but is carefully designed by considering the importance of reducing the generalization error in contrastive learning. In addition, our method reduces the generalization error by reducing the intra-class variance and increasing the inter-class variance, thereby improving the generalization ability of the model. Finally, integrating the triplet loss with the upper limit loss, our proposed multi-loss is formulated as: L=ω1L S +ω2L N +L U Where ω1 and ω2 are L S and L N The weight of . S7: Determine whether t<t max , t max Indicates the predicted maximum number of iterations. If yes, go to step S8, otherwise go to step S9. S8: Let t=t+1, update the model, and return to step S3. S9: Outputs the final trained model, which can generate low-dimensional, high-quality representations for various tasks.

2. The method for generating positive sample representation using GCN according to claim 1, characterized in that: The two types of positive sample representations in step S5 are generated using the following method: In order to obtain the structural information of the graph, we apply the widely used graph convolutional network (GCN) as the basic encoder: Among them, H +(0) =X and H +(l) Represents the l-th layer feature. is a symmetric normalized adjacency matrix, yes The degree matrix, I N is the identity matrix. It is worth noting that our method directly shares weights between the MLP and GCN encoders (i.e., W (l) ) to further reduce time cost. In order to obtain a positive sample representation of the neighbor information type, we first store the neighbor embedding index of all nodes, then sample, and then calculate the average of the samples. In this way, the neighbor information of the node can be obtained efficiently: where m is the number of sampled neighbors, N i Represents node v i The set of 1-hop neighbors of . In summary, structural embedding and neighbor embedding focus on all neighbors and a subset of neighbors, respectively. That is, structural embedding represents a general representation, while neighbor embedding is specific. Therefore, they interpret samples from different perspectives, so that considering them jointly may obtain their complementary information.

Images