A network embedding pre-training method based on graph neural network hierarchical loss function

Abstract

The present invention discloses a network embedding pre-training method based on a graph neural network hierarchical loss function. The method comprises: representing the network as a graph, using the network embedding to learn a low-dimensional representation, and learning a set of shared weight matrices to aggregate information from node neighbors; constructing a set of walking sequences S for the input graph, where each sequence s∈S contains a set of nodes {v1,…,v |s|}; For the constructed walking sequence, a unit graph is constructed based on the nodes that co-occur in the sliding window, and the unit graph is generated based on the proximity of each target node to its one-hop and two-hop neighbors; Using the generated unit graph, a graph neural network model is trained with the designed hierarchical loss minimization as the optimization goal. The present invention combines a graph attention network with an effective hierarchical loss to preserve the margin between the target node and its neighbors, significantly improving the model performance.

Description

Technical Field

[0001] The present invention relates to the technical field of data mining and machine learning, and more specifically, to a network embedding pre-training method based on a graph neural network hierarchical loss function. Background Art

[0002] In a graph consisting of nodes and edges, the pre-trained network embedding is used to reasonably encode unlabeled nodes into a low-dimensional space, where nodes are close to their neighbors and far away from negative samples, so that they can show good performance in downstream tasks such as classification and clustering.

[0003] Graph structures are popular in real-world scenarios such as social networks and citation networks, and have attracted widespread attention from the research community in the past few years. Pre-training of network embeddings aims to map node relationships into a low-dimensional space without any supervised information (e.g., node labels). The learned embeddings can then be used to achieve subsequent tasks such as node classification and link prediction. Pre-training of network embeddings aims to preserve the proximity of unlabeled nodes in a low-dimensional space. To achieve this goal, current work can be divided into direct network embedding and deep neural network embedding.

[0004] For direct network embedding schemes, before the emergence of deep neural networks, direct embedding learning methods such as DeepWalk implemented node embedding by applying the SkipGram model in the generated random walks. Similar methods such as Line, node2vec, ACNE-ST, Adv-Caching, and HNS design biased random walk strategies or negative sampling modeling to explore different node relationships. In addition, Line and MNMF adopt matrix decomposition to learn node embedding. However, all direct embedding methods suffer from the problem of low computational efficiency because there are no shared parameters between nodes in the encoder. In addition, these methods usually lack strong generalization capabilities to new graphs.

[0005] For deep neural network embedding schemes, GNN (graph neural network) based methods have been recently proposed to aggregate information from graph structured data. For example, GCN (graph convolutional neural network) and its variants DGI and GAT have shown breakthrough performance in many tasks, including pre-training of network embedding. Compared with traditional models, GCN can jointly consider local connectivity and global consistency on the graph. Based on the basic theory of GCN, more variants have been proposed, including GraphSAGE, FastGCN, DGCN and HAN for large-scale datasets. Among them, PinSage achieves the maximum application of deep graph embedding by combining GraphSAGE designed for web-scale recommendation systems. In addition, MEIRec uses meta-path-guided GNN to simulate complex interactions of items for search intent recommendations.

[0006] Many pre-training methods for network embedding have been proposed. For example, DeepWalk is a graph embedding method that uses node pairs generated by random walks for representation learning. Later, similar methods such as Line, node2vec and TADW were proposed, but these methods suffer from low computational efficiency due to the lack of shared parameters between nodes in the encoder. Recently, graph neural networks (GNNs) have shown excellent performance in modeling graph-structured data, with typical works including GCN, GAT and GraphSAGE. Generally, the key to the success of GNNs is the parameter sharing of the encoder and the importance weights used to capture the relationship between the target node and its neighbors in the multi-layer structure. However, due to their inherent information aggregation mechanism, these GNN-based methods are prone to suboptimal results during pre-training of unlabeled nodes, which affects the prediction efficiency and accuracy of the model. Summary of the invention

[0007] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide a network embedding pre-training method based on a graph neural network hierarchical loss function. The method comprises the following steps:

[0008] Represent the network as a graph G = (V, E), where V is a set of vertices, represents a set of edges, for each v∈V, the network embedding is used to learn a low-dimensional representation V∈R d , and learn a set of shared weight matrices W l Used to aggregate information from node neighbors, where d represents the dimension of the space, d<<|V|, and l is the level of the graph neural network;

[0009] For the input graph, construct a set of walking sequences S, where each sequence s∈S contains a set of nodes {v1,…,v |s|};

[0010] For the constructed walking sequence, a unit graph is constructed based on the co-occurring nodes within the sliding window. The unit graph is generated based on the proximity of each target node to its one-hop and two-hop neighbors.

[0011] Using the generated unit graph, the graph neural network model is trained with the designed hierarchical loss minimization as the optimization goal. The hierarchical loss is expressed as:

[0012]

[0013] where v t is the target node, v p are direct neighbors in the unit graph, Represents node v p Neighbor, v n is a negative sample sampled from the entire vertex set, Indicates v n Neighbors, h∈R d represents the hidden representation of a node of dimension d, and They represent direct neighbor loss, hierarchical neighbor loss, and negative sampling loss respectively.

[0014] Compared with the prior art, the advantages of the present invention are that it fills the research gap of GNN's poor performance in unlabeled node pre-training, solves the problem that the edges of nodes cannot be distinguished during the aggregation of inherent node information; combines a graph attention network with an effective hierarchical loss to preserve the margins between the target node and its neighbors, and regards the two-hop pattern as a unit graph, which significantly improves the model performance.

[0015] Further features and advantages of the present invention will become apparent from the following detailed description of exemplary embodiments of the present invention with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the invention and, together with the description, serve to explain the principles of the invention.

[0017] Figure 1 is a schematic diagram of a learning embedding space in which a target and its neighbors are direct neighbors according to an embodiment of the present invention;

[0018] Figure 2 is a schematic diagram of a process of a network embedding pre-training method based on a graph neural network hierarchical loss function according to an embodiment of the present invention;

[0019] Figure 3 is a schematic diagram of clustering results using ACC and NMI metrics on different data sets according to an embodiment of the present invention;

[0020] Figure 4 is a schematic diagram of an ablation study using the Micro-F1 metric according to one embodiment of the present invention;

[0021] Figure 5 is a schematic diagram of an ablation study using the Macro-F1 metric according to another embodiment of the present invention;

[0022] Figure 6 is a schematic diagram of classification results of random walk parameters on Cora according to an embodiment of the present invention;

[0023] Figure 7 It is a schematic diagram of classification results of random walk parameters on Citeseer according to an embodiment of the present invention. DETAILED DESCRIPTION

[0024] Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that the relative arrangement of components and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present invention unless otherwise specifically stated.

[0025] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the invention, its application, or uses.

[0026] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered part of the specification.

[0027] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0028] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0029] To facilitate understanding of the present invention, the defects of the prior art are first explained, that is, due to the inherent information aggregation mechanism, the GNN-based method is prone to suboptimal results during the pre-training of unlabeled nodes. Figure 1 As shown, the left figure is an example of an input graph, the upper right figure is the embedding space learned by the existing GNNs, and the lower right figure is the embedding space learned by the method proposed in this invention, where node 1 is the target node, node 2 is the direct neighbor, and nodes 3-5 are two-hop neighbors. The main problem with existing GNN-based methods is that they model the distance between the target node and its neighbors in a coarse-grained manner. The edges of nodes are often difficult to distinguish, resulting in poor performance of downstream applications. Experimental verification has found that even in some cases of downstream classification and clustering tasks, direct embedding methods (such as DeepWalk) may outperform GNN-based methods (such as GCN, GAT, and GraphSAGE). We observe that the more discriminative representations are learned in the embedding space, the higher the performance of subsequent tasks can be achieved. In general, an ideal result is that the target node should be closer to its direct neighbors than the two-hop nodes. To solve this problem, the present invention proposes a simple and effective hierarchical loss graph attention network, called LlossNet, to preserve the margins of nodes in the embedding space.

[0030] Specifically, the present invention explicitly models the loss caused by direct neighbor loss (such as Figure 11 and 2 in ), multi-layer loss (node ​​1 and nodes 3-5), and margin constraints between the above two types of losses. This explicit modeling will help learn more discriminative node representations. However, it is not simple to determine the number of hops being considered. In addition, overly sophisticated multi-hop margin design may impair the ability of the model to generalize. Preferably, proximity preservation within two-hop neighbors is the best choice for the effectiveness and efficiency of LlossNet in experiments. In the present invention, this two-hop pattern is called a unit graph, and multi-hop margins can also be considered recursively.

[0031] Hereinafter, the existing network embedding pre-training, the network embedding pre-training method of the present invention and the experimental verification results will be described in detail.

[0032] 1. Pre-settings

[0033] Before introducing the proposed hierarchical loss, we proposed the current graph-based loss function in the pre-training of network embedding. According to the literature (W.Hamilton, Z.Ying, and J.Leskovec, "Inductive representation learning on large graphs," in Advances in Neural Information Processing Systems, 2017, pp. 1024–1034), in order to learn predictive representations in a completely unsupervised environment, the output representation of the graph-based nodes is applied to the loss function:

[0034]

[0035] where v t is the target node, v p is its direct neighbor, v n is a negative sample randomly drawn from all vertices, P NS (v) represents the negative sample distribution, h represents the output representation in the embedding space, and σ s is a sigmoid function, and N defines the number of negative samples. Note that given a node v, h v is the feature aggregated from its neighbors. For simplicity, the generation process of the node representation in the last layer is as follows:

[0036]

[0037] Where W is a trainable weight matrix, represents the neighbors of node v, CONCAT represents the concatenation operation, and AGGREGATE represents the aggregation function, including the types of mean, LSTM, maxpool, meanpool, and attention. According to equations (1) and (2), it can be concluded that the current graph-based loss only makes the output representations of the target node and its multi-hop neighbors close in the embedding space. Therefore, it cannot preserve the margins between these nodes, whereas the target node should be closer to its direct neighbors than the two-hop nodes.

[0038] 2. Model proposed by the present invention

[0039] A. Question format and symbols

[0040] Since the goal of the present invention is to learn unlabeled node representations, the network G = (V, E) can be represented, where V is a set of vertices, Represents a set of edges. For each v∈V, the network embedding aims to learn a low-dimensional representation V∈R d , to keep the network close, d represents the dimension of the space, d<<|V|. In addition, a set of shared weight matrices W are learned l is used to aggregate information from node neighbors, where l is the level of GNNs.

[0041] B. Phase 1: Unit Graph Generation

[0042] At this stage, the present invention introduces the unit graph generation of input data, such as Figure 2 As shown, Figure 2 (a) is the input graph; Figure 2 (b) Perform random walks to convert the graph into a sequence of nodes; Figure 2 (c) construct a new graph based on the set of positive context pairs in the sliding window; Figure 2 (d) Preserve the proximity of the target node v1, its direct neighbor v2, and its two-hop neighbors v3, v4, v5 in the embedding space; Figure 2 (e) illustrates the loss proposed in the present invention, including direct neighbor loss, hierarchical neighbor loss and negative sampling loss; n represents negative samples and h represents hidden representation.

[0043] Specifically, we first preprocess by performing random walks to form a set of walk sequences S, where each sequence s∈S contains a set of nodes {v1,…,v |s| Then, a new graph can be constructed based on the nodes that co-occur within the sliding window. The positive context pairs in s are given by Definition, where c represents the window size. Figure 2 (c) and Figure 2(d) as an example, node 2 in the sliding window is connected to nodes 3, 4, and 5 respectively. Please note that in this embodiment, only the proximity of each target node (e.g., node 1) to its one-hop and two-hop neighbors is considered. Node 1 is taken as the target node. In the sliding window, node 2 is its direct neighbor, and nodes 3, 4, and 5 are its two-hop neighbors. In this paper, Figure 2 (c) to Figure 2 The generated graph in (d) is called a unit graph. Based on this approach, multi-hop margins can be recursively considered. Through random walks, node relationships can be enriched, which will benefit the performance of pre-training. For fair comparison, random walks are used as pre-processing for all baselines in the experiment, with only major differences in subsequent models.

[0044] C. Second stage: layered loss optimization

[0045] like Figure 2 (d) and Figure 2 As shown in (e), using the generated unit map, the proposed hierarchical loss is defined as follows:

[0046]

[0047] where v t The target node (e.g. v1), v p is a direct neighbor in the unit graph (e.g. v2), Represents node v p neighbors (e.g. v3, v4, v5), v n Negative samples are sampled from the entire vertex set using the alias table method (AQ Li, A. Ahmed, S. Ravi, and AJSmola, "Reducing the sampling complexity of topic models," in Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining, 2014, pp. 891–900), using only O(1) time. Indicates v n Neighbors, h∈R d represents the hidden representation of a node of dimension d, and They represent direct neighbor loss, layer-wise loss, and negative sampling loss respectively. More details will be introduced below.

[0048] For the direct neighbor loss (the first part in Formula 3), without loss of generality, the target node is encouraged to be close to its direct neighbors in the embedding space, which is defined as follows:

[0049]

[0050] The hierarchical neighbor loss (the second part in Formula 3) is to make the target node close to its two-hop neighbors while maintaining the margins of two-hop neighbors and direct neighbors in the embedding space, for example, defined as follows:

[0051]

[0052] Agg is an aggregation function that aggregates the information of the central node and the messages transmitted from the neighbors. The condition (st) encourages the target node (i.e., v t ) is larger than its two-hop neighbor (i.e., v i ) is closer to its immediate neighbor (i.e., v p ). If the condition is not met, Set to 0. In other words, if the target node is closer to its two-hop neighbors than its direct neighbors, the second part of the optimization can be paused by setting the hierarchical neighbor loss to 0, while keeping the direct neighbor loss to make the target node close to its direct neighbors. In addition, before the message aggregation step of AGG, a message propagation step is required to process the messages passed from the neighboring nodes, see the following process.

[0053] 1) Message propagation step: Message propagation is at the central node v i and its neighbor node v j For example, encoding from a locally structured neighborhood, the message passing can be formulated as follows:

[0054]

[0055] Represents the slave node v j to v i The message transmitted has a message dimension ∈ R d . M v ∈R d×d is the transformation matrix, is the neighbor type (e.g., one-hop or multi-hop neighbor), h v ∈R d is the hidden representation of node v, Neighbor Type and neighbor nodes The representation of is taken as input and the output is the transformation matrix M v . Note that the neighbor type is a hot encoding. By concatenating and A multi-layer perceptron (MLP) is then used to learn the mapping. The detailed definition is as follows:

[0056]

[0057] in Represents a join operation.

[0058] 2) Message aggregation step: In this step, the goal is to aggregate the information of the central node (e.g., node 2) and the messages transmitted from its neighbors (e.g., nodes 3-5), such as Figure 2 (d) In order to reduce the noise information that may be propagated from neighbors, an attention aggregator is introduced by combining the attention mechanism to learn the importance weight of the node. The weight coefficient between two nodes It can be expressed by the following equation:

[0059]

[0060] W∈R d×d is a shared weight matrix used to map nodes to the same embedding space, Records a weight vector for learning the relationship between the central node and its neighbors, is node v i The neighbor set of r is the ReLU activation function.

[0061] Afterwards, using the learned weight coefficients and neighbors, node v i The aggregation function is expressed as follows:

[0062]

[0063] In summary, we have introduced how to formulate the direct neighbor loss and hierarchical neighbor loss of equation (3), and introduced the aggregation function of nodes, including message propagation and aggregation steps. Next, we will introduce the third part in equation 3, namely the negative sampling loss or negative sampling loss.

[0064] The negative sampling loss aims to make the target node v t Away from negative samples and their neighbors in the embedding space, which is defined as follows:

[0065]

[0066] Where P NS (v) represents the negative sampling distribution, v n is the negative sample extracted from it. Specifically, P NS (v) is defined by the following formula:

[0067]

[0068] where f vis the degree of node v in the graph, and β is the empirical degree power, which can be set to 3 / 4, for example.

[0069] D. LlossNet algorithm proposed in this invention

[0070] For greater clarity, the scheme proposed in the present invention is designed in the following algorithm (Algorithm 1). Specifically, a random walk is performed on the input graph to obtain a node sequence, and a new graph is generated using a sliding window (line 2). Then, in each epoch, a batch is sampled from the generated positive context pairs (line 5). Next, for each target node in the batch, its direct neighbor loss, two-hop neighbor loss, and negative sampling loss are constructed according to equations (4), (5), and (10), respectively (lines 7-9). Finally, the above losses are integrated by equation (3) and optimized using stochastic gradient descent (line 10). These steps (lines 5-10) are repeated until convergence is reached (the loss value becomes stable).

[0071] ***********************************************

[0072]

[0073] The pre-trained graph neural network of the present invention can be used in a variety of scenarios, such as recommendation scenarios. Taking the article content recommendation scenario as an example, the articles include news, social network topics, academic papers, etc. Each article is regarded as a data node, and the relationship between articles (such as citations, links, etc.) is regarded as an edge, which is input into the pre-trained graph neural network of the present invention. The output is a unique embedding vector learned for each node, which can reflect the correlation between nodes. When it is necessary to search for nodes related to a certain node, the inner product between nodes can be calculated, and the TOP K related nodes can be retrieved according to the size of the inner product.

[0074] Furthermore, in order to verify the effect of the present invention, simulation experiments were conducted. In the experiments, downstream tasks including node classification and clustering were used to evaluate the performance of pre-trained node representations on actual data sets.

[0075] A. Dataset

[0076] Experiments are conducted on four widely used network datasets, and their statistics are shown in Table 1.

[0077] Table 1: Dataset

[0078]

[0079] In Table 1, Cora is a research citation network constructed by the existing technology, which contains 2708 machine learning papers with 7 labels; Citeseer is another widely adopted research paper collection, which contains 3264 publications and 6 labels; Wiki is a language network containing 2405 web pages from 17 groups and 12761 edges between them. This dataset has been widely used in the evaluation of vertex classification tasks; DBLP is a computer science reference list constructed by the existing technology. In the experiment, conference paper lists are selected from 4 research fields: database, data mining, AI and CV.

[0080] B. Baseline Model

[0081] In the experiments, several existing state-of-the-art methods were used as baselines, which are designed for pre-training of network embeddings, including direct embedding and GNN-based embedding models. The description of the selected baselines is as follows:

[0082] DeepWalk is the first graph embedding method that applies representation learning to node pairs generated by random walks. It is a direct embedding learning method that does not share any parameters between nodes and uses the Skip-gram method to train node embeddings.

[0083] GCN introduces an efficient variant of convolutional neural networks that can run directly on graph-structured data and shows breakthrough performance in network embedding learning. In order to adapt to large-scale networks, an improved version of the GCN method is derived. This invention adopts this variant for a fairer comparison.

[0084] DGI introduces a general approach to learning network representations in an unsupervised manner. DGI relies on maximizing the mutual information between the graph-enhanced representation and the currently extracted graph information - both derived using the established graph convolutional network architecture. For the graph-enhanced representation, it is a subgraph generated from the target node. In general, it can be viewed as an improved version of the GCN approach.

[0085] GraphSAGE proposes an inductive way for computing vertex representations, which produces impressive performance on several large-scale baselines. For each vertex, GraphSAGE first samples a fixed-size set of neighbors and then performs an aggregation function on them to obtain the vertex representation. There are four types of aggregation functions, including SAGE-mean, SAGE-LSTM, SAGE-maxpool, and SAGE-meanpool (a variant of the maxpool aggregator). A major difference between meanpool and mean is that meanpool requires the vector of each neighbor to be independently input through a fully connected neural network.

[0086] GAT proposes a graph attention network that incorporates the attention mechanism into its propagation step. It computes the hidden representation of each node by paying attention to its neighbors, following a self-attention strategy to learn the importance weights of the neighbors of the central node.

[0087] C. Parameter settings

[0088] In the experiment, the present invention was implemented using the Tensorflow runtime environment. For all models, the embedding size was uniformly set to 128, the model parameters were randomly initialized with a Gaussian distribution, and the model was optimized using mini-batch Adam. The learning rate was set to 1e-3, the batch size was 512, and the maximum sequence length was set to 50. For the variants of GNN and the method of the present invention, GraphSAGE was followed by setting the number of hidden layers to 2. In addition, for fair comparison, random walks were used as preprocessing for all methods by setting the walk length, the number of walks per node, and the window size to 30, 5, and 5, respectively.

[0089] D. Evaluation Indicators

[0090] In order to evaluate the pre-training performance of the method in the downstream classification task, the Liblinear package with default settings is used to build the node classifier, and the Micro-F1 and Macro-F1 metrics are adopted. In addition, in the downstream clustering task, the results are reported using two widely used metrics, namely clustering accuracy (ACC) and normalized mutual information (NMI). When the clustering result matches the actual label exactly, the values ​​of ACC and NMI will be 1, otherwise, the value will be close to zero.

[0091] E. Evaluating Classification Performance

[0092] In this section, the performance of downstream classification tasks of various models on four real-world datasets is verified. Specifically, 20%, 40%, 60%, and 80% of the nodes are randomly selected as training sets, and the rest are test sets. See Tables 2, 3, 4, and 5 for Micro-F1 and Macro-F1 results.

[0093] Table 2: Cora’s node classification results

[0094]

[0095] Table 3: Citeseer node classification results

[0096]

[0097] Table 4: Wiki node classification results

[0098]

[0099] Table 5: Node classification results of DBLP

[0100]

[0101] The highest scores are in bold, followed by the underlined scores. The following observations can be made from the above table:

[0102] 1) In general, the proposed LLossNET outperforms the state-of-the-art models on all datasets with different training ratios, which demonstrates the effectiveness of the proposed method in considering the margin preservation between the target node and its neighbors.

[0103] 2) The overall performance of the baseline follows the following order: LlossNet>DeepWalk>DGI≈SAGE-meanpool≈SAGE-LSTM≈GAT>SAGE-maxpool≈SAGE-mean≈GCN. More specifically, in terms of the average training ratios of Cora, Citeseer, Wiki, and DBLP, the LLossNet of the present invention achieves 4.44%, 5.77%, 4.43%, and 3.59% performance improvements on Micro-F1, respectively, over the second-best model. In addition, in terms of Macro-F1, LLossNet outperforms the second-best model by 5.26%, 6.27%, 16.69%, and 5.07% on all four datasets, respectively. These improved results indicate that the present invention is more capable of learning discriminative embeddings in pre-training on graph-structured data.

[0104] 3) In the baseline results, DeepWalk outperforms GNN-based methods in most cases, such as on Cora, Citeseer, and Wiki datasets. This reveals the problem that GNNs perform poorly in pre-training on unlabeled nodes. Although GAT achieved the second best result on the DBLP dataset, its self-attention mechanism only considers the important weights of neighbors that are beneficial to embedding learning, while still ignoring the margins between multi-hop nodes.

[0105] It should be noted that the present invention trains all algorithms in an unsupervised manner, using Equation 3 as the objective function of the present invention and using Equation 1 as the baseline. In order to evaluate the performance of all methods after pre-training, classifiers are built by uniformly using the Liblinear package with default settings to perform downstream classification tasks, where the stopping criteria are tolerance parameter = 1e-4 or maxiter = 100. The present invention no longer divides the test nodes into validation and test sets, because the validation setting does not have much impact on the general performance trend of downstream classification.

[0106] F. Clustering performance evaluation

[0107] To further evaluate the pre-training performance of the present invention, a downstream clustering task is performed in the following way: All baselines are compared according to the ACC and NMI metrics.

[0108] Specifically, K-means was run uniformly with default settings on the pre-trained representations of the four datasets. Figure 3 As shown, the observation results are as follows:

[0109] 1) First, from Figure 3 (a) It can be seen that LLossNET outperforms the baseline in terms of ACC, which verifies the effectiveness of the present invention. In general, the order of ACC performance between baselines is similar to the classification results, where DeepWalk outperforms the GNN-based model except for DGI on Cora. ​​The present invention's LlossNet improves DeepWalk's performance by 26.11%, 3.42%, 3.28%, and 3.38% on Cora, Citeseer, Wiki, and DBLP, respectively. In addition, LlossNet improves DGI's performance by 2.88% in ACC performance.

[0110] 2) Going a step further, Figure 3 (b) It can be seen that LLossNet can also achieve consistent improvements compared to other models, which further confirms the superiority of the present invention for pre-training of unlabeled nodes. More specifically, LLossNET improves the NMI performance of the DeepWalk model by 14.66%, 13.64%, 4.21%, and 3.54% on four datasets. In addition, compared with the DGI method on Cora, Citerseer, Wiki, and DBLP, LLossNet obtains 7.60%, 1.17%, 80.94%, and 11.02% performance gains, respectively.

[0111] 3) The differences in ACC and NMI performance of DBLP for all methods are relatively small. This is because the number of labels is small, only 4 labels, so the differences are not obvious.

[0112] In addition, the precision and recall results are verified. In this section, the goal is to evaluate the performance of the precision and recall results of these methods. Two baselines, DeepWalk and GAT, are selected, which have performed well in previous classification experiments. The implemented metrics refer to scikit-learn, and 50% of the nodes are randomly selected as training sets, and the remaining nodes are used as test sets on Cora and Citeseer. The experimental results are reported in Table 6.

[0113] Table 6: Performance of Cola and Citeseer

[0114]

[0115] From Table 6, the following observations can be made:

[0116] 1) Compared with other baselines, the present invention can achieve better performance. Specifically, LlossNet is 4.78% and 19.03% higher than DeepWalk and GAT on Cora on average, respectively. Compared with DeepWalk and GAT on Citeser, LlossNet is 7.55% and 19.52% higher on average.

[0117] 2) Note that the performance of Precision micro and Recall micro metrics achieve the same results. In a single-label classification problem, if there are false positives, there are always false negatives, and vice versa, because one class is always predicted. Therefore, precision and recall are always the same when using the micro averaging scheme.

[0118] Furthermore, ablation experiments with different losses and hops are also conducted. In this part, the impact of different losses such as direct neighbor loss as Equation (4) and hierarchical neighbor loss as Equation (5) are studied. The performance of different hops with downstream classification tasks is also studied based on Micro-F1 and Macro-F1 metrics. In order to compare different losses, Figure 4 and Figure 5 The classification performance of LlossNet using direct loss, two-hop layer loss, and three-hop layer loss on four datasets with varying training ratios of {10%, 30%, 50%, 70%, 90%} is shown, where Figure 4 (a) to Figure 4 (d) Cora, Wiki, Citeseer and DBLP, respectively. Figure 5 (a) to Figure 5 (d) The same results for Cora, Wiki, Citeseer and DBLP. From these results, we can observe that:

[0119] 1) Compared with direct loss, both the two-hop layered loss and the three-hop layered loss of LlossNet can improve the classification performance, which verifies the effectiveness of the present invention. More specifically, for Micro-F1, the two-hop layered loss method outperforms the direct loss of Cora, Wiki, Citeseer, and DBLP by 5.60%, 5.14%, 6.51%, and 4.27%, respectively. On these datasets, the performance gains of Macro-F1 are 6.50%, 8.23%, 22.23%, and 6.54%, respectively.

[0120] 2) LlossNet with three-hop layered loss generally performs better than two-hop layered loss on Citeseer and DBLP, with gains of 0.46% and 0.23% for Micro-F1, and performs poorly on Cora and Wiki, with declines of 0.18% and 2.42%, respectively. In addition, for Macro-F1, three-hop layered loss outperforms two-hop layered loss on Citeseer and DBLP, with gains of 2.15% and 0.38%, respectively, while Cora and Wiki perform poorly, with declines of 1.09% and 3.81%, respectively. This is because an overly elaborate multi-hop margin design may impair the model's ability to generalize.

[0121] 3) It can also be observed that the performance of the proposed scheme shows stable performance in terms of different training ratios. Specifically, the Micro-F1 performance of the two-hop layered loss fluctuates by 78.80%, 54.60%, 62.68% and 79.91% on Cora, Citeseer, Wiki and DBLP respectively. And on these four datasets, the Macro-F1 performance of the two-hop layered loss fluctuates by 77.78%, 49.45%, 50.84% ​​and 74.13% accordingly.

[0122] In addition, the effectiveness of preprocessing is verified. When the network is too complex or not large enough to analyze, random walks on the graph can extract node semantics. In one embodiment, unbiased random walks are used to explore the network, and the probability of selecting the next hop node is equal. This process does not lead to bias in the network formation. In order to evaluate the impact of the random walk strategy on downstream tasks, the parameter values ​​of walk length, window size, and number of walks are changed in preprocessing. The classification results are shown in Figure 2. Figure 6 and Figure 7 shown.

[0123] Generally speaking, we change the walk length to {10, 15, 20, 25, 30}, the number of walks per node to {2, 3, 4, 5}, and the window size to {2, 3, 4, 5}. Figure 6 and Figure 7 In it can be observed that:

[0124] 1) The classification performance increases with the walk length and becomes stable when the length is between 25 and 30.

[0125] 2) On Cora, the performance fluctuates with the number of walks, while on Citeser, a larger number of walks can achieve better results.

[0126] 3) For the window size, the classification performance is improved when the size increases from 2 to 3. The larger the window size, the better the performance.

[0127] From these results, it can be concluded that this preprocessing of random walks is effective for exploring the network. It should be understood that the work of the present invention is scalable to other preprocessing techniques. The preprocessing of unbiased random walks aims to explore the network by searching for more potential node pairs using the above parameter settings. The method proposed in the present invention is independent of the preprocessing and can process the original node pairs. In addition, when the window size is set to 2, the preprocessing only considers the direct neighbors of the node, which is the same as the original network.

[0128] In summary, the present invention proposes a graph attention network with a carefully designed hierarchical loss, called LlossNet, for pre-training unlabeled nodes. Although GCN and its variants have demonstrated effective results in semi-supervised network embedding learning, they have obtained poor performance compared to direct embedding learning (such as Deepwalk on pre-training of unlabeled nodes), which is trained in a completely unsupervised manner. The present invention believes that during the aggregation of inherent information in GNN-based methods, the edges of nodes may be indistinguishable. To solve this problem, it is proposed to maintain the margins between the target node and its neighbors by constructing three types of losses, including direct neighbor loss, hierarchical neighbor loss, and negative sampling loss. Then, the loss is embedded in the graph attention network for demonstration. In addition, the present invention can be easily applied to other GNN-based models. And, by evaluating the pre-training performance of the baseline through downstream classification and clustering tasks, it is confirmed that the LlossNet proposed in the present invention can achieve impressive improvements on the most advanced models, which verifies the effectiveness of the proposed method.

[0129] The present invention may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present invention.

[0130] Computer readable storage medium can be a tangible device that can hold and store instructions used by an instruction execution device. Computer readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (non-exhaustive list) of computer readable storage medium include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, for example, a punch card or a convex structure in a groove on which instructions are stored, and any suitable combination thereof. The computer readable storage medium used here is not interpreted as a transient signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagated by a waveguide or other transmission medium (for example, a light pulse by an optical fiber cable), or an electrical signal transmitted by a wire.

[0131] Various aspects of the present invention are described herein with reference to the flow charts and / or block diagrams of the methods, devices (systems) and computer program products according to embodiments of the present invention. It should be understood that each box of the flow chart and / or block diagram and the combination of each box in the flow chart and / or block diagram can be implemented by computer-readable program instructions.

[0132] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0133] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0134] The flow charts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to multiple embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a part of a module, a program segment or an instruction, and a part of the module, a program segment or an instruction contains one or more executable instructions for realizing the specified logical function. In some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions. It is well known to those skilled in the art that it is equivalent to implement it by hardware, implement it by software, and implement it by combining software and hardware.

[0135] Embodiments of the present invention have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the marketplace, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein. The scope of the present invention is defined by the appended claims.

Claims

1. A network embedding pre-training method based on a graph neural network hierarchical loss function, comprising the following steps: Represent the network as a graph G = (V, E), where V is a set of vertices, represents a set of edges, for each v∈V, the network embedding is used to learn a low-dimensional representation V∈R d , and learn a set of shared weight matrices W l Used to aggregate information from node neighbors, where d represents the dimension of the space, d<<|V|, and l is the level of the graph neural network; For the input graph, construct a set of walking sequences S, where each sequence s∈S contains a set of nodes {v1,…,v |s| }; For the constructed walking sequence, a unit graph is constructed based on the co-occurring nodes within the sliding window. The unit graph is generated based on the proximity of each target node to its one-hop and two-hop neighbors. Using the generated unit graph, the graph neural network model is trained with the designed hierarchical loss minimization as the optimization goal. The hierarchical loss is expressed as: where v t is the target node, v p are direct neighbors in the unit graph, Represents node v p Neighbor, v n is a negative sample sampled from the entire vertex set, Indicates v n Neighbors, h∈R d represents the hidden representation of a node of dimension d, and They represent direct neighbor loss, hierarchical neighbor loss, and negative sampling loss respectively; The trained graph neural network model is used for article content recommendation, wherein the articles are taken as nodes and the relationships between articles are taken as edges, which are input into the graph neural network model to obtain a unique embedding vector learned by each node, and the embedding vector reflects the correlation between nodes, and then the inner product between nodes is calculated to obtain the articles associated with the recommended nodes according to the size of the inner product; Among them, the direct neighbor loss is used to encourage the target node to be close to its direct neighbors in the embedding space, expressed as: Among them, σ s is the sigmoid function; The hierarchical neighbor loss is used to make the target node close to its two-hop neighbors and maintain the margins of two-hop neighbors and direct neighbors in the embedding space, expressed as: Among them, σ s is the sigmoid function, Agg represents the aggregation function, and the condition represents the encouragement of the target node v t Compared to its two-hop neighbor v i Closer to its direct neighbor v p If the condition is not met, then Set to 0; Among them, the negative sampling loss is used to make the target node v t A negative sample and its neighbors far away in the embedding space are expressed as: where v n ~P NS (υ) Among them, P NS (v) represents the negative sampling distribution, v n is the negative sample extracted from it, f v is the degree of node v in the graph, and β is the empirical degree power.

2. The method according to claim 1, characterized in that Before message aggregation, message propagation is also performed to process messages transmitted from neighboring nodes, where the message propagation is performed at the central node v i and its neighbor node v j Between them, the message is delivered in the following way: Among them, MLP stands for multi-layer perceptron, Indicates a connection operation. Represents the slave node v j to v i The message transmitted has a message dimension ∈ R d , M v ∈R d×d is the transformation matrix, is the neighbor type, h v ∈R d is the hidden representation of node v, Neighbor Type and neighbor nodes The representation of is taken as input and the output is the transformation matrix M v , neighbor type is a one hot encoding.

3. The method according to claim 2, characterized in that The goal of message aggregation is to aggregate the information of the central node and the messages transmitted from its neighbors. The weight coefficient between the two nodes is It is expressed as: Using the learned weight coefficients and neighbors, node v i The aggregation function is expressed as: Where W∈R d×d is a shared weight matrix used to map nodes to the same embedding space, is a weight vector used to learn the relationship between the central node and its neighbors. is node v i The neighbor set of r is the ReLU activation function.

4. The method according to claim 1, characterized in that: The set of walking sequences S is obtained by performing random walks on the input graph representation.

5. The method according to claim 1, characterized in that The unit graph is constructed based on co-occurring nodes in a sliding window, where the positive context pairs in s are given by Definition, c represents the window size.

6. A computer-readable storage medium having a computer program stored thereon, wherein: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

7. A computer device comprising a memory and a processor, wherein a computer program capable of running on the processor is stored in the memory, characterized in that: When the processor executes the program, the steps of the method according to any one of claims 1 to 5 are implemented.

Images