A phased graph representation learning method based on citation information

Abstract

This invention discloses a phased graph representation learning method based on citation network information. It designs a clear phased pre-training strategy, performing subgraph-level and node-level similarity comparison learning on neighborhood sets respectively. In the first stage, the model is pre-trained on datasets from multiple different domains using subgraph comparison to obtain general structural features. In the second stage, the model is fine-tuned on two damaged views using node-level comparison, optimizing node-level neighborhood similarity according to a designed objective function. Finally, the embedding vector matrix of the original graph data is obtained and input into a classifier to classify citations to similar paper topics. Compared with existing technologies, this invention improves accuracy in node classification tasks by learning node neighborhood similarity through node-level comparison and achieves an unsupervised graph representation learning method.

Description

Technical Field

[0001] This invention belongs to the field of deep learning, specifically relating to a graph representation learning method based on a contrastive model for citation network data information. Background Technology

[0002] Graphs are an important form of representing relationships between objects in the real world. Various types of graphs exist in the real world; for example, this invention aims to improve graph representation learning technology for citation networks formed by the citation relationships between papers in academic platforms. In real-world applications, graph data is often sparse and non-uniform. These characteristics pose significant challenges to data mining tasks. Graph representation learning (or network representation learning) aims to map nodes in graph data into low-dimensional, dense vectors to preserve the attribute information of nodes in the original graph in the dimensionality-reduced space. These low-dimensional, dense vectors have strong normalization and serve as inputs for different downstream machine learning models, thereby completing tasks such as node classification, link prediction, and visualization. Nodes in graph data possess two important pieces of information: neighborhood similarity and structural similarity. Neighborhood similarity refers to the higher similarity a node should have with its neighboring nodes. Structural similarity refers to the higher similarity between two nodes with similar topological structures in the graph data. Existing graph representation learning methods typically cannot simultaneously consider both of these characteristics.

[0003] While the development of graph neural networks has greatly advanced the field of graph representation learning in recent years, most successful methods typically employ supervised learning, which is difficult to scale. This is because obtaining a large number of high-quality labels in the real world is often time-consuming and laborious. Furthermore, existing supervised representation learning methods often lead to models learning only label-related information, resulting in overfitting. Ideally, models should learn knowledge with strong transferability, reusability, and generality. In recent years, contrastive learning has made significant progress in natural language processing and computer vision. Inspired by the success of contrastive learning applications on images, recent research has proposed similar contrastive frameworks for graph representation learning to achieve unsupervised training on graph data. For training graphs, the goal of contrastive learning is to learn one or more encoders that make the representations of similar graph instances consistent, while the representations of dissimilar graph instances are inconsistent. Existing methods for constructing contrastive learning tasks on graph data can be unified into a general framework. First, view pairs are generated between the input graph and graph data augmentation methods. Then, a surrogate task is defined to specify positive and negative samples for contrastive learning, making the embedding vectors of positive samples more similar and the embedding vectors of negative samples more distinct. Therefore, researching how to effectively preserve neighborhood similarity and structural similarity in the original graph data in graph representation learning methods based on unsupervised methods is an important problem that urgently needs to be solved. Summary of the Invention

[0004] To address the problem that existing graph contrastive learning models cannot simultaneously consider both the neighborhood similarity and structural similarity of nodes, resulting in weak model generality, this invention proposes a phased graph representation learning method based on citation network information. A phased learning framework based on a contrastive model is designed to achieve phased learning of node neighborhood similarity and structural similarity, thereby achieving accurate classification in citation networks.

[0005] The present invention provides a staged graph representation learning method based on citation information, comprising:

[0006] Step 1: Take any node in the input graph as the center node v to obtain the extended citation information subgraph r-centersubgraph. The input graph is a graph structure dataset about citation information. The formal definition of r-center subgraph is graph G.

[0007] Step 2: Data augmentation is performed on any two citation information subgraphs to generate two sets of augmented subgraphs and corresponding subgraph embedding vectors, forming a similar instance pair and serving as a sample. Other similar instances are obtained through iterative graph sampling. All similar instance pairs are used for pre-training. By maximizing the subgraph-level contrast objective function L1, the pre-trained model can identify the structural similarity of the subgraph-level contrast learning nodes, thus obtaining the citation network embedding vector trained by the subgraph-level contrast learning model.

[0008] Step 3: Generate two damage view diagrams based on the input diagram. and Select any node and its corresponding node in another view as samples, and use a graph encoder to obtain a sampled graph. The node embedding vector U and sampling The node embedding vector V is used for pre-training, and the node-level contrastive objective function l(u) is maximized. i ,v i This enables the pre-trained model to identify the structural similarity of nodes in the node-level contrastive learning, thus obtaining the citation network embedding vector trained by the node-level contrastive learning model.

[0009] Step 4: Use a classifier to classify citations into similar paper topics based on the citation network embedding vectors trained in Step 2 and Step 3.

[0010] The beneficial effects and advantages of this invention are as follows:

[0011] (1) The phased learning framework proposed in this invention learns the structural similarity of nodes by comparing subgraphs and learns the neighborhood similarity of nodes by comparing nodes at the node level, thereby improving the accuracy in node classification tasks.

[0012] (2) It realizes an unsupervised graph representation learning method, which is beneficial to the expansion to various datasets. At the same time, it will also have a positive impact on commonly used downstream tasks such as node classification and link prediction. Attached Figure Description

[0013] Figure 1 This is an overall flowchart of a phased graph representation learning method for citation network information according to the present invention;

[0014] Figure 2 The concept of node structure similarity;

[0015] Figure 3 For the concept of node neighborhood similarity

[0016] Figure 4 A schematic diagram of the general framework used for comparison models;

[0017] Figure 5 This is a schematic diagram of commonly used graph data augmentation methods;

[0018] Figure 6 Example diagram of a second-order subgraph;

[0019] Figure 7 This is a schematic diagram of an embodiment of the present invention that learns the neighborhood similarity of nodes through subgraph-level comparison;

[0020] Figure 8 This is a schematic diagram of an embodiment of the present invention that learns the structural similarity of nodes through node-level comparison. Detailed Implementation

[0021] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0022] like Figure 1 The diagram shows the overall flowchart of a phased graph representation learning method (GRLWPT) for citation network information according to the present invention. The specific steps are as follows:

[0023] Step 1: Sampling Subgraph: Using any node v in the input graph as the anchor node v, obtain the subgraph r-center subgraph centered on the anchor node v. The input graph is a graph structure dataset of citations, the arbitrary node v is an article, and the subgraph r-center subgraph is the set of all articles centered on an article and whose intermediate citation count does not exceed r. The formal definition of r-center subgraph is graph G. In the citation information subgraph r-center subgraph, vertices represent identifying articles, and edge sets represent citing articles. The input graph is a graph structure dataset of citations, the nodes are articles, and the neighboring nodes are other articles that directly cite the article.

[0024] The formal definition of an r-center subgraph is G = {V, E}, where V represents the set of nodes in the graph. S represents the set of edges in the graph. G' represents the augmented graph after edge perturbation or node discarding of G. For a node V, its r-order neighborhood is defined as S. v ={u:d(u,v)≤r}, where d(u,v) represents the shortest distance between node U and node V. This will be referred to as, in S v Subgraph G of the middle sampling v G v This is also the r-center subgraph. For example... Figure 6 The example of a second-order subgraph shown provides an example of a 2-center subgraph.

[0025] Each G in the pre-training module v It is treated as an independent class and learns how to distinguish different instances, which is reflected in similar instances having similar embedding vectors.

[0026] The goal of the pre-training module is to learn general structural information across datasets. A subgraph is used as a comparison example, expanding node v into a subgraph centered on node v, defined as an r-center subgraph. The subgraph is obtained by selecting an anchor node and then randomly walking along edges from that anchor node. When the number of walks reaches a specified number of steps n, all nodes and edges traversed constitute the subgraph. An r-order neighborhood represents the set of all articles centered on a given article that have no more than r intermediate citations (meaning the article can be cited from r other articles).

[0027] Step 2: Perform graph sampling pre-training on the augmented subgraph using the embedding vectors:

[0028] In the field of computer vision, two random data augmentations of the same image are considered similar instance pairs. For example... Figure 7In the embodiment shown, which learns the neighborhood similarity of nodes through subgraph-level comparison, two augmented graphs G'1 and G'2 are generated from the same subgraph r-centersubgraph through two randomly generated citation subgraphs r-center subgraph1 and r-center subgraph2. Theoretically, any graph encoder can be used for encoding. This invention chooses GCN as the graph encoder. Two subgraphs are obtained through a selection function t (the difference in the number of nodes in the two subgraphs exceeds N). The pre-training module first performs data augmentation on these two subgraphs to generate x. q , x q , This represents an enhanced subgraph generated from two r-center subgraphs, such as... Figure 7 As shown. The two instances obtained from the same r-center subgraph are x. q , As positive samples, instances obtained from different r-center subgraphs x q , As a negative sample.

[0029] In the pre-training module, this invention treats two random data augmentation views within the same r-center subgraph as a similar instance pair. A similar instance pair refers to two citation subgraphs generated from the same r-center subgraph through random walks, where nodes represent articles and edges represent citations. Two similar instance pairs are treated as positive samples during comparative learning (the graph neural network makes the embedding vectors generated by the two positive samples more similar). In the pre-training module, this invention uses subgraph sampling as a graph data augmentation method. Graph sampling in the pre-training module consists of three steps: random walk, subgraph derivation, and anonymization.

[0030] a) Random walk: Starting from a central article v, a cited article is randomly selected as the next hop, and the walk continues from the selected article. Specifically, a random walk is performed on the subgraph G starting from the central node v. The probability of choosing a direction at each step is related to the edge weight; the greater the edge weight, the more likely the node is to walk along that edge. Simultaneously, at each step, there is a certain positive probability of returning to the initial node v; this probability can be set through hyperparameters.

[0031] b) Subgraph derivation: A subgraph is generated from all articles and citations involved in step a. Specifically, a subgraph centered on node v can be collected through random walks; this is referred to as... Then by Derived Called G v The enhanced image.

[0032] c. Anonymize the citations by removing their original IDs and re-randomizing them with numbers from 1 to n; specifically, this involves processing the sampled graphs... The nodes in the array are re-marked in any order. In order to preserve the underlying structural pattern, so as to hide the actual node index and prevent the encoder from learning unimportant information;

[0033] After each of the above steps, an instance is obtained. This instance is then abstracted into a dictionary matching task. Two instances obtained from the same r-center subgraph are considered positive samples, while instances obtained from different r-center subgraphs are considered negative samples. For example... Figure 6 As shown, instance x was obtained. q With dictionary instance group One of the instances in the dictionary instance group is related to x. q A matching pair is denoted as k. + To distinguish an instance x q Distinguishing it from other examples;

[0034] Then use two GCN encoders f q with f k The embedding vectors q, k0, k1, and k2 corresponding to the generated augmented graph are, from a dictionary perspective, formed by k0, k1, and k2, which constitute a dictionary of length 3, where x q , For positive sample x q , x q , These are negative samples. The contrastive loss function proposed in formula (1) prompts the model to... Instances identified as a pair of similar embedding vectors (generated embedding vectors are similar), while... and Instances identified as having dissimilar embedding vectors (resulting in different embedded vectors).

[0035] In summary, when learning common structural properties in graph data, the pre-training module of GRLWPT first generates multiple r-center subgraphs from different domain datasets, and then generates multiple instances through three steps: random walk with restart, subgraph induction, and anonymization. Next, it uses a GNN as a graph encoder to obtain the embedding vectors q, k0, k1, and k2 for different instances. Finally, it calculates the subgraph-level contrastive objective function L1 of the graph neural network, as shown in the following equation:

[0036]

[0037] in, k represents the embedding vector generated by the anchor subgraph. i k represents the embedding vector generated from the negative sample subgraph. + τ represents the embedding vector generated from the positive sample subgraph, and τ represents the temperature hyperparameter, which is used to control the randomness in the graph neural network.

[0038] The subgraph-level contrast objective function L achieves the structural similarity of nodes through subgraph-level contrast learning, and updates the parameters using stochastic gradient ascent to maximize L.

[0039] Step 3 involves learning node neighborhood similarity through node-level comparison. Specifically, the pre-trained module trained in Step 2 learns general node structure information from different domain datasets. This differs from learning node information through comparison between subgraph instances in the pre-trained module and from previous works that utilized local-global comparison. In Step 3, this invention learns the neighborhood similarity of nodes in a specific domain dataset by comparing the target at the node level.

[0040] Specifically, this invention first generates two corrupted graph views by randomly corrupting the original graph using graph data augmentation methods. Then, it uses a contrast target to force the embedding vectors of identical nodes and their neighboring nodes in the two different views to be similar, thus distinguishing them from other nodes and their neighbors. In learning the neighborhood similarity of nodes, this invention generates two corrupted graph views in each iteration, denoted as follows: and The corresponding embedding matrix definition of the two views and Where X * With A * These represent the adjacency matrix and feature matrix of the enhanced view, respectively.

[0041] Then, this invention distinguishes the embeddings of similar nodes from the embeddings of other nodes in different graph views by defining a comparison target. For any node V i The present invention assigns the corresponding node v in a view iAs anchor nodes. In order to learn the neighborhood similarity of nodes through node-level comparison, GRLWPT specifies that node v i The corresponding node u in another view i and v i with u i The neighboring nodes are positive samples. At the same time, this invention will also select a node different from V. i node V′ i This invention specifies v′ i and its corresponding node u′ in another view i , and all their neighboring nodes are negative samples.

[0042] In summary, this invention first uses graph data augmentation technology to generate two damaged graph views based on the original graph during each iteration. and Next, we will use a GNN as a graph encoder to obtain... The node embedding vector U and The node embedding vector V is then used. Finally, the parameters of the graph encoder are updated by maximizing the objective function in equation (2).

[0043]

[0044] in, This indicates the corresponding node (article) of the anchor node (article) in another view. This represents the neighboring nodes (referenced articles) of the anchor node (article) in the original graph. This indicates the domain node (referenced article) of the anchor node (article) in another view. τ represents a randomly selected node (article) that is different from the anchor node (article), τ represents the temperature hyperparameter used to control the randomness in the graph neural network, N represents the number of nodes (articles), and θ represents the cosine similarity function.

[0045] The effectiveness of this invention in classification tasks on four citation datasets: Cora, Citeseer, PubMed, and DBLP. The Cora dataset is a citation network dataset composed of machine learning papers, a popular dataset for graph deep learning in recent years. This dataset contains 2708 sample points and 5429 edges. Each sample point represents a scientific paper, and all sample points are divided into 7 categories. Each paper is represented by a bag-of-words model, where each word appears in the paper. Each sample point also has 1433 features, which are the TF-IDF values ​​from the bag-of-words model. The Citeseer dataset is a citation network dataset containing scientific papers from 6 different fields. This dataset contains 3327 sample points and 4732 edges. Each paper is represented by a bag-of-words model, where each word appears in the paper. Each sample point also has 3703 features, which are the TF-IDF values ​​from the bag-of-words model. The PubMed dataset is a biomedical citation network dataset containing 19,717 articles and 44,338 citation relationships. Each sample point represents a scientific paper, and all samples are divided into three categories. Each article is represented by a bag-of-words model, where each word is a word that appears in the article. Each sample point also has 500 features, which are the TF-IDF values ​​in the bag-of-words model. DBLP is a large computer science citation database containing a large amount of information on papers and authors in computer science and related fields. It provides a dataset containing all the bibliographic records that make up DBLP. The DBLP dataset contains various types of nodes, including authors, papers, terms, and conferences. DBLP contains 14,328 papers, 4,057 authors, 20 conferences, and 7,723 terms. The experimental procedure uses... Figure 6 and Figure 7 The proposed phased learning framework for node structural similarity and neighborhood similarity uses a graph neural network to embed graph data, obtaining embedding vectors. These embedding vectors are then fed into a classifier for classification. The validation process is shown below:

[0046] Step 1: Dataset Partitioning. In this citation dataset, the graph is constructed from links that cite each other between scientific papers. Specifically, nodes correspond to articles, and undirected edges represent citation links between articles. Furthermore, each node has an article type and a bag-of-words model, where the TF-IDF value represents the node's features. In the e-commerce dataset, nodes represent products, and edges represent two products that are frequently purchased together.

[0047] On these networks, this paper randomly selects 10% of the nodes as the training set, 10% of the nodes as the validation set, and the remaining nodes as the test set.

[0048] Step Two: Encoder Selection. For transductive learning tasks, this section uses a single-layer graph convolutional network model as the encoder, with the following propagation rule: in This is an adjacency matrix with self-loops (links between nodes). Let it be its corresponding degree matrix. For the nonlinear activator σ, this section chooses ReLU. Θ∈R F×F′ It is a linear change with learnable parameters, and it has F′ = 512 learnable parameters. Θ∈R F×F′ It can be applied to every node.

[0049] Step 3: Classifier Selection. The research objective of this paper is to improve the quality of embedding vectors obtained from unsupervised graph representation learning. Besides exploring embedding methods for graph representation models, it is also necessary to select a suitable node classification model to evaluate the quality of the obtained embedding vectors. Considering three points—the different dataset sizes used in the experiments, the relationships between different attributes of the node embedding vectors, and the fact that the experiments focus on improving the quality of the embedding vectors obtained from the graph representation learning model (rather than the performance of the classification model)—this paper ultimately chose the Logistic Regression method as the model for performing the node classification task in the experiments. An example of the node classification task is: a citation network consists of n articles, and these n articles belong to m different topics. A classifier is used to classify the embedding vectors obtained in this paper to obtain the labels of the topics to which all articles belong.

[0050] Step 4: Evaluate the model using accuracy metrics, i.e. The evaluation results are shown in Table 1. It can be seen that the method of the present invention (GRLWPT) has achieved a certain degree of improvement over other algorithms.

[0051] Table 1

[0052]

[0053]

[0054] Finally, it should be noted that the above content is only used to illustrate the technical solution of the present invention, and is not intended to limit the scope of protection of the present invention. Simple modifications or equivalent substitutions made by those skilled in the art to the technical solution of the present invention do not depart from the essence and scope of the technical solution of the present invention.

Claims

1. A phased graph representation learning method based on citation information, characterized in that, include: Step 1: Take any node in the input graph as the center node. Retrieve extended citation information subgraph The input graph is a graph-structured dataset containing citation information. The formal definition is a graph The arbitrary node in step 1 For the article, the citation information subgraph Let r be the number of articles that cite a given article as the center. The formal definition is a graph ; Step 2: Data augmentation is performed on any two citation information subgraphs to generate two sets of augmented subgraphs and their corresponding subgraph embedding vectors, forming a similar instance pair and serving as samples. Other similar instances are obtained through an iterative graph sampling process. Pre-training is performed using all similar instance pairs, and the subgraph-level contrast objective function is maximized. This enables the pre-trained model to identify the structural similarity of subgraph-level contrastive learning nodes, resulting in the citation network embedding vector trained by the subgraph-level contrastive learning model; the subgraph-level contrastive objective function in step 2 As shown in the following formula: in, This represents the embedding vector generated by the anchored subgraph. This represents the embedding vector generated from the negative sample subgraph. This represents the embedding vector generated from the positive sample subgraph. This represents the temperature hyperparameter, used to control the randomness in the graph neural network; Then, the parameters are updated using stochastic gradient ascent to obtain the objective function that maximizes the objective function. ; Step 3: Generate two damage view diagrams based on the input diagram. and Select any node and its corresponding node in another view as samples, and use a graph encoder to obtain a sampled graph. Node embedding vector and sampling Node embedding vector Pre-training is performed using node embedding vectors, and the node-level contrastive objective function is maximized. This enables the pre-trained model to identify the structural similarity of nodes in the node-level contrastive learning model, resulting in the citation network embedding vector trained by the node-level contrastive learning model; the subgraph-level contrastive objective function in step 3 The expression is as follows: in, This indicates the corresponding node of the anchor node in another view. This represents the neighboring nodes of the anchor node in the original graph. This indicates the domain node of the anchor node in another view, representing the corresponding node in that view. This represents a randomly selected node that is different from the anchor node. This represents the temperature hyperparameter, used to control the randomness in the graph neural network, where N represents the number of nodes. Represents the cosine similarity function; Step 4: Use a classifier to classify citations into similar paper topics based on the citation network embedding vectors trained in Step 2 and Step 3.

2. The staged graph representation learning method based on citation information according to claim 1, characterized in that, In step 2, the process of obtaining the subgraph embedding vector is as follows: It is obtained from any two citation information subgraphs.

1.

2. Perform data augmentation to generate two sets of augmented subgraphs, namely: , and , As a pair of similar instances, by the enhanced subgraph , and , Obtain the corresponding citation network embedding vector , , , The same citation information subgraph 1 Enhanced subgraph , As a positive sample, different citation information subgraphs 1 Enhanced subgraph , , As negative samples, the embedding vectors of the aforementioned similar instance pairs are used to train the enhanced subgraph by graph sampling based on the neighborhood similarity of nodes at the subgraph level. a) Starting from the center node v in the subgraph Perform a random walk on top, and then generate a subgraph from all the nodes and edges involved in the random walk. According to the subgraph Derive the sample diagram. Enhanced sample image c will enhance the sampled image. The nodes in the array are relabeled in any order. Iteratively execute a, b, c to obtain new pairs of similar instances.

3. The staged graph representation learning method based on citation information according to claim 1, characterized in that, In step 3, the process of obtaining the node embedding vector is as follows: selecting nodes. and its corresponding node in another view , and And their neighboring nodes are positive samples, and other nodes are selected. and its corresponding node in another view. , and And all their neighboring nodes are negative samples; In each iteration, two damaged view diagrams are first generated based on the input diagram. and Use a graph encoder to obtain a sampled graph. Node embedding vector and sampling Node embedding vector .

4. The staged graph representation learning method based on citation information according to claim 1, characterized in that, The formal definition is a graph and Enhanced Map ': Represents a graph, in which Represents a set of nodes. Represents the set of edges; 'By the pair It is obtained after edge perturbation or node discarding.

5. The staged graph representation learning method based on citation information according to claim 1, characterized in that, For a node The r-order neighborhood is defined as ,in, This represents the shortest distance between node u and node v. Is Subgraphs sampled in the middle .

6. The staged graph representation learning method based on citation information according to claim 1, characterized in that, The subgraph-level contrastive learning model contains subgraphs with the same citation information. 1 Enhanced subgraph , As a positive sample, different citation information subgraphs 1 Enhanced subgraph , , As a negative sample.

7. The staged graph representation learning method based on citation information according to claim 1, characterized in that, In the node-level contrastive learning model, there are arbitrary nodes Corresponding node in another view as well as and The neighboring nodes are positive samples, and a different one is selected. nodes , and the corresponding node in its other view All adjacent nodes are negative samples.

Images