Graph representation learning method considering node diversity in directed graph

Abstract

The invention relates to a graph representation learning method considering node diversity in a directed graph. Many directed graph neural network (DGNNs) methods carry out the same processing on an out-edge neighbor set and an in-edge neighbor set when processing a neighbor set of each node, and do not consider the diversity of nodes in a directed graph, so that proper information cannot be adaptively acquired from neighbors in different directions generally. In order to solve the problem, a new method is researched, firstly, diversity of nodes is considered to carry out representation learning of a directed graph, namely neighbor diversity and degree diversity, then a new NDDGNN framework is provided, and the framework adaptively allocates weights for out-edge information and in-edge information on the node level. A large number of experiments on seven real world data sets verify the superior performance of the method in node classification and link prediction tasks, and the performance is better than that of the most advanced method.

Description

Technical Field

[0001] The present invention belongs to the field of graph learning, and more specifically, relates to a graph representation learning method that considers the diversity of nodes in a directed graph. Background Art

[0002] Graph Neural Networks (GNNs) have achieved great success by analyzing undirected graphs to extract representations of graph data. However, in many real-world applications, such as traffic networks and web networks, graphs are directed, where each edge is bidirectional. Therefore, many previous GNNs that consider undirected graphs ignore the direction information, resulting in the discarding of valuable information that is critical for downstream tasks. For example, in a web network, web pages with more links tend to have more traffic, and if the direction information of the web network is ignored, some web pages with few inbound edges and many outbound edges will be predicted as high-traffic web pages. Therefore, considering the direction information is practical for applying GNNs in directed graphs.

[0003] Previous GNN methods for directed graphs mainly focus on depicting directional information in graphs, including spectral DGNN methods and spatial DGNN methods. Spectral DGNN methods mainly focus on constructing Laplacian matrices with directional information. For example, DiGCN uses the transition probability of personalized PageRank to approximate the Laplacian matrix of directed graphs to consider directional information. MagNet designs a magnetic Laplacian matrix to capture directional information through a phase matrix. However, many directed spectral methods use the same mapping function to obtain node representations of outgoing and incoming information, so they may not be able to output discriminative representations by ignoring the difference between incoming and outgoing information.

[0004] Spatial DGNN methods aim to first learn different mapping functions to process input and output information, and then combine the two directional information to update node representations for downstream tasks. For example, DGCN learns mapping functions for directional information separately based on its second-order information. Dir-GNN first learns different mapping functions based on the first-order information of the node, and then manually assigns weights to fuse the two node representations obtained from different mapping functions. In summary, previous spatial DGNN methods consider the difference between the two directional information in two ways, namely different mapping functions and manually setting different weights on the node representations learned from different directional information, but they still have limitations that need to be addressed.

[0005] First, the representation of each node is affected by its neighbors, including the in-neighborhood set and out-neighborhood set in a directed graph. The label of each node is determined by the in-neighborhood set or the out-neighborhood set or even both. Each node in both the in-neighborhood and the out-neighborhood should have different weights, referred to as neighbor diversity. However, previous spatial DGNNs assigned them the same weight. Second, the degree can reflect the centrality of the node and provide information about the structure around it. In addition, degree information is very helpful for distinguishing two different nodes, especially in a directed graph that includes in-degree and out-degree. In particular, each node has different in-degree and out-degree, namely degree diversity. However, previous spatial DGNNs ignored the degree information in the directed graph, weakening the representation ability of the node. Obviously, both neighbor diversity and degree diversity show the characteristics of the node, namely node diversity. Summary of the invention

[0006] The purpose of the present invention is to overcome the deficiencies of the prior art and provide a new spatial directed graph neural network framework, namely, a node diversity directed graph neural network (NDDGNN), which considers node diversity to model the directional information in graph representation learning.

[0007] To do this, the present invention follows the directed space method to construct different mapping functions for outgoing information and incoming information. During the construction process, unlike previous spatial DGNNs that assign the same weight to the same neighbor set (i.e., outgoing neighbor set and incoming neighbor set) of each node, the present method assigns different weights to them by considering node diversity. Specifically, we consider neighbor diversity by defining in-Dirichlet energy and out-Dirichlet energy to encode the difference between each node and its neighbors, and introduce degree embedding to consider degree diversity to encode in-degree information and out-degree information.

[0008] The present invention has the following beneficial effects:

[0009] 1) This paper proposes a new directed GNN method that considers node diversity (including neighbor diversity and degree diversity), which can adaptively learn weights for each node;

[0010] 2) We conduct a comprehensive empirical study on real-world datasets and verify the superiority of our method over the state-of-the-art methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 It is a flowchart of a specific implementation method of the graph representation learning method considering the diversity of nodes in a directed graph of the present invention;

[0012] Figure 2 It is the structural flow chart of NDDGNN of the present invention;

[0013] Figure 3 It is a module diagram of computing node diversity in the present invention; DETAILED DESCRIPTION

[0014] The specific implementation of the present invention is described below in conjunction with the accompanying drawings so that those skilled in the art can better understand the present invention. It should be noted that in the following description, when the detailed description of known functions and designs may dilute the main content of the present invention, these descriptions will be omitted here.

[0015] Example

[0016] Figure 1 This is a flowchart of a specific implementation method of the directed graph representation learning method considering node diversity in the present invention. Figure 1 As shown, the specific steps of the present invention include:

[0017] S101: Input feature matrix and corresponding graph structure:

[0018] Let G = (V, E, X, Y) be a directed unweighted graph, where the node set, edge set, node feature matrix and label matrix are V = {v i |i=1,2,...,n},E={(v i ,v j )|v i ,v j ∈V},X∈R n×f and Y∈R n×c , respectively, A∈{0,1} n×n is the adjacency matrix of graph G, D = ∑ i A i,j is the degree matrix, is the normalized Laplacian matrix. By defining N i,→ For the i-th node v i The outgoing neighbor set, D → =Σ j (A i,j ) is the out-degree diagonal matrix, we obtain the symmetric normalized directed out-neighbor matrix and the row-normalized directed outgoing neighbor matrix Similarly, we can define N i,← , D ← ,, and

[0019] S102: Build NDDGNN learning model:

[0020] NDDGNN first maps the in-neighbors and out-neighbors of each node separately, and then calculates their neighbor diversity and degree diversity respectively. After that, NDDGNN uses the combined neighbor diversity and degree diversity to learn the weight of each node and normalizes the weights through Softmax. NDDGNN also studies consistency regularization to ensure that abnormal weights are not learned. Finally, these weights are used to fuse the representations of in-neighbors and out-neighbors to update the representation of node i.

[0021] S103: Set the number of iterations t=1.

[0022] S104: Computing Node Diversity:

[0023] Node diversity includes neighbor diversity, degree diversity, structural diversity, etc. The present invention studies node diversity by calculating neighbor diversity and degree diversity.

[0024] S301: Calculate neighbor diversity:

[0025] The present invention takes into account the diversity of neighbors and assigns different weights to them. The outgoing neighbor set of each node is different from the set of other nodes, and the incoming neighbor set of each node is also different from the set of other nodes. Therefore, it is reasonable to learn different weights for the outgoing neighbor set and the incoming neighbor set. The category of each node is related to the majority category in the neighbor set (i.e., the outgoing neighbor set or the incoming neighbor set). Specifically, if the category of each node is the same as the majority category in the neighbor set, the weight of the neighbor set should be larger to achieve small intra-class differences. Since we have label information for some nodes, the feature difference between the node and the node it connects to can be used to measure the intra-class difference, thereby determining the weight of the neighbor set. Considering that Dirichlet energy is one of the popular methods for measuring the feature differences of adjacent nodes, the present invention adopts it to guide the learning of weights. Specifically, given a node representation H = [h1, h2, ..., h n ]∈R n×f , the Dirichlet energy E(H (l) ) is defined as follows:

[0026]

[0027] where s i ' j Represents the weight of the edge.

[0028] Since different dimensions carry different information, it is necessary to assign different weights to different dimensions. This paper introduces learnable parameters in Dirichlet energy, so that the model can assign different weights to different dimensions and can be adaptively adjusted according to downstream tasks, which can be written as:

[0029]

[0030] where w (l) ∈R f×1 is a learnable parameter vector and ⊙ is the element-wise product.

[0031] Dirichlet energy is used to calculate the entire graph, and the present invention also focuses on the difference between each node and its neighbor set. In order to facilitate subsequent calculations and reduce complexity, we use the row-wise normalized adjacency matrix to represent s i ' j ,Right now

[0032] Therefore, for all nodes’ outgoing neighbor sets, the outgoing Dirichlet energy at layer l is It can be defined as follows:

[0033]

[0034] in Its ith element represents the outgoing Dirichlet energy of node i, and I is the identity matrix. Similarly, for the incoming Dirichlet energy of the incoming adjacent set The same is true for:

[0035]

[0036] On large-scale graphs, in order to reduce the time complexity, equation (2) can be rewritten as:

[0037]

[0038] This equation calculates the difference between each node and its two neighbors like equation (2) and ignores the variance within the neighborhood. Therefore, the Dirichlet energy in equation (3) can be written as The same is true for the in-Dirichlet energy. The extended Dirichlet energy adaptively characterizes the difference between each node in a directed graph and its in-neighbors and out-neighbors. However, there is a problem with using the corresponding Dirichlet energy as the weight of the two adjacency sets. Generally, the greater the difference in features between a node and its neighbors, the more likely they are to belong to different categories. Therefore, directly using the Dirichlet energy as a weight will result in assigning larger weights to adjacency sets with more nodes of different categories, and smaller weights to adjacency sets with more nodes of the same category. Therefore, the inverse is added in the invention to correct them.

[0039]

[0040] in and are the adaptive weights of each node to the two neighbor sets. Unlike the current spatial DGNNs that manually set weights, Equation (6) automatically learns by considering the difference between each node and its neighbors. and Therefore, for all nodes, neighbor sets with smaller feature differences (i.e., more similar categories) will be assigned larger weights compared to other neighbor sets, characterizing the neighborhood diversity of each node.

[0041] S302: Calculate degree diversity:

[0042] This paper proposes to learn degree embedding, integrating degree information with the representation learning process. ← ∈R n×f andDeg → ∈R n×f denote the learnable in-degree embedding and out-degree embedding respectively, and equation (6) becomes:

[0043]

[0044] Based on equation (7), nodes with the same in-degree have the same in-degree embedding. Similarly, nodes with the same out-degree also have the same out-degree embedding. Therefore, when two nodes share the same in / out degree and in / out neighbor difference, their weights during aggregation will be similar. In contrast, even if the features of two nodes themselves and their neighbors are very similar, as long as their degree information is different, they will not be mapped to the same point in the feature space (i.e., the model has the ability to distinguish between the two nodes).

[0045] S105: Update node representation:

[0046] In order to consider the node diversity in a directed graph, for a directed graph, the weight vector (i.e., ),in and are the weights of the incoming and outgoing neighbors of node i in layer l, respectively. Then, softmax normalization is used to ensure as follows:

[0047]

[0048] in Represents a diagonal matrix In addition, The equation also holds. Update the node representation H of the l+1th layer by the following method (l+1) :

[0049]

[0050] in is a diagonal matrix consisting of the lth layer As a diagonal element composition, including neighbor diversity and degree diversity, with Γ → In this way, each node can assign different weights to outgoing and incoming neighbors. By aggregating the representations of neighbors and the node representations of the previous layer, the node representation H at the l+1 layer in this method is (l+1) Updated via:

[0051]

[0052] Where α is a hyperparameter used to preserve the information of the previous layer.

[0053] S106: Training NDDGNN model:

[0054] The present invention decomposes the directional information into incoming information and outgoing information. It is expected that the weights of the learned incoming information follow the same distribution, and the weights of the learned outgoing information also follow the same distribution. To achieve this, consistency regularization is used to mitigate and avoid learning abnormal weights that deviate from the distribution. Specifically, the weight average of all incoming information or outgoing information is first calculated:

[0055]

[0056] Then by minimizing and The distance between and This is achieved by :

[0057]

[0058] In each training cycle, the cross entropy loss and consistency regularization loss are combined to perform representation learning. The final objective function of the method is as follows:

[0059] L=L sup +λL con (13)

[0060] Where L sup is a supervised learning loss, usually used for node classification tasks, and λ is a consistency regularization loss, which is used to ensure the consistency of the learned weight distribution.

[0061] The model uses an early stopping strategy for validation accuracy, optimizes all parameters through Adam, sets the learning rate range to {0.005, 0.01}, and sets the weight decay to 0, sets the number of model layers to {4, 5, 6}, the dropout range to {0.0, 0.35, 0.5, 0.6}, and the hidden unit size range to {32, 128, 256}, sets the hyperparameter α for retaining the representation of the previous layer to {0.0, 0.3, 0.5, 0.8, 1.0}, and sets the range of λ in our regularization term to {0.0, 0.1, 0.2, 0.9}.

[0062] S107 determines whether t<t max , t max Indicates the maximum number of iterations. If yes, go to step S108, otherwise go to step S109.

[0063] S108: Let t=t+1, update the model, and return to step S104.

[0064] S109: Output the final training model for various tasks.

[0065] In order to better illustrate the technical effect of the present invention, a specific example is used to experimentally verify the present invention. In this experimental verification, experiments were conducted on two homogeneous datasets (Cora-ML and Citeseer-Full) and one heterogeneous dataset (Roman-Empire). The comparison methods included one traditional undirected graph method (i.e., MLP), two state-of-the-art undirected graph methods (i.e., ACM-GCN, Grand.Gating), and one state-of-the-art directed graph method (i.e., Dir-GNN).

[0066] Table 1 Node classification accuracy on directed graph (%)

[0067]

[0068] As shown in Table 1, the classification accuracy of various methods on all datasets shows that this method has achieved significant improvements on all datasets. The node classification accuracy of this method on the dataset is better than the best undirected graph method Grand.Gating, and is 0.99% higher than the best directed graph method Dir-GNN on average, which shows that the proposed framework and directed graph method considering node diversity are effective.

[0069] Although the above describes the illustrative specific embodiments of the present invention to facilitate those skilled in the art to understand the present invention, it should be clear that the present invention is not limited to the scope of the specific embodiments. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations using the concept of the present invention are protected.

Claims

1. A graph representation learning method considering node diversity in a directed graph, characterized in that: The following steps are involved: S1: Input feature matrix and corresponding graph structure: Let G = (V, E, X, Y) be a directed unweighted graph, where the node set, edge set, node feature matrix and label matrix are V = {v i |i=1,2,...,n},E={(v i ,v j )|v i ,v j ∈V},X∈R n×f and Y∈R n×c , respectively, A∈{0,1} n×n is the adjacency matrix of graph G, D = Σ i A i,j is the degree matrix, is the normalized Laplacian matrix. By defining N i,→ For the i-th node v i The outgoing neighbor set, D → =Σ j (A i,j ) is the out-degree diagonal matrix, we obtain the symmetric normalized directed out-neighbor matrix and the row-normalized directed outgoing neighbor matrix Similarly, we can define N i,← , D ← ,, and S2: Build NDDGNN learning model: NDDGNN first maps the in-neighbors and out-neighbors of each node separately, and then calculates their neighbor diversity and degree diversity respectively. After that, NDDGNN uses the combined neighbor diversity and degree diversity to learn the weight of each node and normalizes the weights through Softmax. NDDGNN also studies consistency regularization to ensure that abnormal weights are not learned. Finally, these weights are used to fuse the representations of in-neighbors and out-neighbors to update the representation of node i. S3: Let the number of iterations t=1. S4: Computational Node Diversity: Node diversity includes neighbor diversity, degree diversity, structural diversity, etc. The present invention studies node diversity by calculating neighbor diversity and degree diversity. S5: Update node representation: In order to consider the node diversity in a directed graph, for a directed graph, the weight vector (i.e., ),in and are the weights of the incoming and outgoing neighbors of node i in layer l, respectively. Then, softmax normalization is used to ensure as follows: in Represents a diagonal matrix In addition, The equation also holds. Update the node representation H of the l+1th layer by the following method (l+1) : in is a diagonal matrix consisting of the lth layer As a diagonal element composition, including neighbor diversity and degree diversity, with Γ → In this way, each node can assign different weights to outgoing and incoming neighbors. By aggregating the representations of neighbors and the node representations of the previous layer, the node representation H at the l+1 layer in this method is (l+1) Updated via: Where α is a hyperparameter used to preserve the information of the previous layer. S6: Training NDDGNN model: First, calculate the weighted average of all incoming or outgoing information: Then by minimizing and The distance between and This is achieved by : In each training cycle, the cross entropy loss and consistency regularization loss are combined to perform representation learning. The final objective function of the method is as follows: L=L sup +λL con Where L sup is a supervised learning loss, usually used for node classification tasks, and λ is a consistency regularization loss, which is used to ensure the consistency of the learned weight distribution. S7: Determine whether t<t max , t max Indicates the predicted maximum number of iterations. If yes, go to step S8, otherwise go to step S9. S8: Let t=t+1, update the model, and return to step S4. S9: Output the final training model which can be used for various tasks.

2. The method for calculating node diversity according to claim 1, characterized in that: The two types of diversity in step S5 are calculated using the following method: Regarding neighbor diversity, considering that Dirichlet energy is one of the popular methods to measure the difference in features of adjacent nodes, the present invention uses it to guide learning weights. Specifically, given a node representation H = [h1,h2,...,h n ]∈R n×f , the Dirichlet energy E(H (l) ) is defined as follows: where s′ ij Represents the weight of the edge. The present invention introduces learnable parameters in Dirichlet energy, which enables the model to assign different weights to different dimensions and can be adaptively adjusted according to downstream tasks, which can be written as: where w (l) ∈R f×1 is a learnable parameter vector and ⊙ is the element-wise product. We use the row-normalized adjacency matrix to represent s′ ij ,Right now For all nodes’ outgoing neighbor sets, the outgoing Dirichlet energy of layer l is It can be defined as follows: in Its ith element represents the outgoing Dirichlet energy of node i, and I is the identity matrix. Similarly, for the incoming Dirichlet energy of the incoming adjacent set The same is true for: On large-scale graphs, in order to reduce the time complexity, rewrite the equation as: Directly using Dirichlet energy as weights will result in assigning larger weights to neighbor sets with more nodes of different categories, and the inverse is added in the invention to correct them. in and are the adaptive weights of each node to the two neighbor sets respectively. Therefore, for all nodes, neighbor sets with smaller feature differences (i.e., more similar categories) will be assigned larger weights compared to other neighbor sets, characterizing the neighborhood diversity of each node. For computing degree diversity, the invention proposes learning degree embedding, integrating degree information with the representation learning process, and using Deg ← ∈R n ×f andDeg → ∈R n×f They represent learnable in-degree embedding and out-degree embedding respectively, and are defined as follows: When two nodes share the same in / out degree and in / out neighbor difference, their weights during aggregation will be similar. Conversely, even if the features of two nodes themselves and their neighbors are very similar, as long as their degree information is different, they will not be mapped to the same point in the feature space (i.e., the model has the ability to distinguish between the two nodes).

Images