Self-supervised message passing graph representation learning method based on contrastive generation

Abstract

The self-supervised message passing graph representation learning method based on contrast generation includes: step 1: data preprocessing, including: graph data processing and data partitioning; adding self-loops pointing to themselves to all isolated nodes in the original graph data set; if the edges in the data set are directed edges, converting all directed edges into undirected edges; dividing the nodes into training set, validation set and test set in a ratio of 1:1:8; step 2: contrast generation stage; in this stage, the embedding representation of the node is specifically learned through a contrast generation model; the contrast generation model consists of a contrast module and a generation module, which share an encoder structure and jointly optimize the embedding representation of the node based on the idea of ​​contrast and generation; step 3: contrast aggregation stage; the node embedding H of the topological graph G obtained by the contrast generation module is passed to the aggregation model, and the node embedding is aggregated by the contrast aggregation model, and the final node embedding of the model is obtained after training. The problem that the generative method is greatly affected by noise and the discriminability of input features is solved, and the robustness of the model is improved.

Description

Technical Field

[0001] The present invention relates to a representation learning method for graph structure data. Background Art

[0002] There are all kinds of intricate relationships in the real world, which constitute many huge relationship graphs. With the development of deep learning, people have been able to obtain the required information from complex relationship graphs of different scales. In order for computers to process the information contained in the graph, it is necessary to convert this data into vector representation, that is, graph representation learning method. Graph representation learning, also known as graph embedding, aims to obtain low-dimensional potential representations of nodes or edges in the graph, which can be widely used in social networks, traffic prediction, recommendation systems and other fields.

[0003] At present, many graph representation learning methods have been proposed to mine effective information from graph data. Existing graph representation learning methods can be divided into supervised and unsupervised categories according to whether the model needs to use manual labels during training. Supervised methods have shown excellent performance in many fields such as biomedicine, traffic prediction, recommendation, and abnormal diagnosis, but their performance is completely dependent on reliable labels for specific tasks, and it costs a huge amount of money to obtain enough labels in real life; while unsupervised methods still lag behind supervised methods in terms of accuracy, which limits their application in practical application scenarios such as medical diagnosis, risk prediction and other key fields.

[0004] Therefore, it is of great practical significance to develop a graph representation learning method that does not rely on label information and can accurately express the embedded representation of information. Summary of the invention

[0005] In order to overcome the problem that the embedding representation obtained by the existing unsupervised graph representation learning method has low accuracy in expressing information, the present invention provides a new self-supervised message passing graph representation learning method based on contrast generation to realize graph representation learning.

[0006] The present invention utilizes graph structure, converts raw data into latent layer embedding through contrastive generative model, uses contrastive aggregation model to optimize the embedding learned by contrastive generative model, sequentially performs learning of embedding representation and aggregation of embedding representation, thereby realizing graph representation learning.

[0007] The self-supervised message graph representation learning method based on contrastive generation is mainly divided into the contrastive generation stage and the contrastive aggregation stage; the specific steps are as follows:

[0008] Step 1: Data preprocessing;

[0009] Specifically include:

[0010] 1.1 Graph data processing and data partitioning; add self-loops pointing to themselves to all isolated nodes in the original graph data set; if the edges in the data set are directed edges, convert all directed edges to undirected edges; divide the nodes into training set, validation set and test set in a ratio of 1:1:8;

[0011] Step 2: Contrastive generation phase; in this phase, a contrastive generation model is used to specifically learn the embedded representation of the node; the contrastive generation model consists of a contrast module and a generation module, which share an encoder structure and jointly optimize the embedded representation of the node based on the idea of ​​contrast and generation;

[0012] 2.1 Constructing feature view: First, calculate the feature matrix by formula (1): The similarity matrix Select the first K similar nodes of each node as the neighbor nodes of this node, and finally use the K-nearest neighbor algorithm to construct the feature view G for the initial graph G = (A, X) f =(A f ,X); where N represents the number of nodes in the graph dataset, and d0 represents the initial dimension of the feature matrix X;

[0013]

[0014] where x i represents the initial feature vector of the i-th node, S i,j Indicates the similarity between the i-th node and the j-th node;

[0015] 2.2 Mask some nodes of the initial graph; randomly sample the node set V of the initial graph according to a certain proportion to obtain a node subset Pair Collection The node features within are masked, and the masked features are shown in (2);

[0016]

[0017] Among them, v i represents the i-th node in the initial graph;

[0018] 2.3 Training node embedding: Using the graph neural network GAT as the encoder, the masked initial graph and feature view G f =(A f ,X) is used as the input of the encoder, and after being embedded by the encoder, the masked initial graph node embedding matrix is ​​obtained And the feature view embedding matrix The details are shown in formula (3) and (4);

[0019]

[0020] H f =Encode(A f ,X) (4)

[0021] Where d is the dimension of the embedding vector, Encode(·) represents the encoder structure using the graph attention neural network;

[0022] 2.4 Multi-level contrastive learning: First, to achieve contrastive learning between nodes and subgraphs, it is necessary to generate subgraph-level embedding representations. Specifically, randomly sample the neighbor nodes of each node on the initial graph to generate a topological subgraph for each node; use the feature view to generate the feature subgraph of each node for all neighbor nodes; average the hidden layer embeddings of the two subgraphs of all nodes to obtain the hidden layer embeddings of the topological subgraphs of all nodes. and feature subgraph embedding The specific formula is as shown in formula (5) and (6):

[0023]

[0024]

[0025] in For node v i The topological subgraph set, N f,i For node v i The feature subgraph set of , Mean(·) represents the average readout function;

[0026] Contrastive learning is divided into three types of comparisons: node-subgraph, subgraph-subgraph, and cross-level comparisons. Cross-level contrastive learning requires constructing a joint representation of node embedding and subgraph embedding, connecting the hidden node vectors of the topological view and feature view with the subgraph vectors, respectively, to obtain the cross-level vectors of the topological view and feature view. and The details are shown in (7)(8);

[0027] c i =Concat(h i ,z i ) (7)

[0028] c f,i =Concat(h f,i ,z f,i ) (8)

[0029] where c i and c f,i Respectively represent the node v i The cross-level vectors of the topological view and feature view of , Concat(·) represents the connection operation;

[0030] For three types of contrastive learning, InfoNCE is used as the loss function, and the specific node and subgraph loss L N , subgraph loss L G And the cross-level loss L M As shown in formulas (9), (10), and (11):

[0031]

[0032]

[0033]

[0034] Where τ represents the temperature coefficient, which controls the strictness of positive and negative samples; Batch represents the node samples in the same batch; In formulas (10) and (11), 1 [i≠j] It is an indicator function that takes 1 when the condition is met and takes 0 when it is not met.

[0035] 2.5 Generate learning training; The decoder uses the node hidden layer embedding H of the topological graph and the adjacency matrix A to reconstruct the masked node features. The structure of the decoder is the same as that of the encoder. The specific reconstruction process is shown in formula (12):

[0036]

[0037] Where Encode(·) represents a 2-layer GAT decoder structure with an input dimension of d and an output dimension of d0; represents the reconstructed node feature matrix, Represents node v i The reconstructed feature vector is used to calculate the loss of reconstructing the masked node features through the SCE loss, as shown in formula (13):

[0038]

[0039] Loss function L AE The average reconstruction loss of all masked node features is calculated, where the scaling factor γ is a hyperparameter that can be adjusted on different datasets and can be regarded as an adaptive sample weight adjustment and scaling technique.

[0040] 2.6 Encoder optimization: The loss functions of the comparison module and the generation module are integrated through weight parameters, and the Adam algorithm is used to minimize the objective function L to optimize the parameters of the encoder Encode(·). The objective function calculation formula is shown in formula (14):

[0041] L=L AE +α·L M +β·L G +λ·LN (14)

[0042] Among them, α, β, and λ are the weight parameters that control the cross-level contrast loss, subgraph contrast loss, and node-subgraph contrast loss, respectively. After the contrast generation module is trained multiple times until convergence, the topological view node hidden layer embedding H obtained by the encoder at the final convergence is used as the input matrix of the next module.

[0043] Step 3: Comparison aggregation stage: The node embedding H of the topology graph G obtained by the comparison generation module is passed to the aggregation model, and the comparison aggregation model aggregates the node embeddings. After training, the final node embedding of this model is obtained;

[0044] 3.1 Aggregation model initialization; Randomly initialize the weight matrix W of the aggregation model Pre and W Nei , W Pre , Where W Pre is the weight matrix of node embedding, W Nei is the neighbor embedding weight matrix, heads is a hyperparameter used to control the number of aggregation heads, which must satisfy heads≤d and heads is a factor of d;

[0045] 3.2 Obtain node embedding and calculate neighbor embedding representation; node embedding matrix H Pre To compare the node embedding matrix (i.e. H) output by the generation module, the neighbor embedding matrix H Nei About node v i Neighbor embedding representation The calculation formula is shown in formula (15):

[0046]

[0047] Among them, Norm(·) represents vector normalization, N i is node v i The neighbor node set of , sim(·) is the similarity calculation function, where cosine similarity is used;

[0048] 3.3 Update aggregate embedding; aggregate embedding matrix H Agg The calculation formula is shown in formula (16):

[0049] H Agg =Norm(W Pre *H Pre +W Nei *H Nei ) (16)

[0050] Where * represents the element-wise product operation;

[0051] 3.4 Non-neighbor node sampling; for any node v in the topology graph i Randomly select a fixed number of non-neighbor nodes to generate a negative sample node set V i Neg , the number of nodes in the set can be set manually;

[0052] 3.5 Comparative aggregation model training; using the Adam algorithm to maximize the model comparison loss function L Agg , continuously optimize the weight matrix W of the aggregation model Pre and W Nei Until the model converges; aggregate model value function L Agg The calculation formula is shown in formula (17):

[0053]

[0054] Where N i For node v i The set of neighbor nodes, V i Neg For node v i The sampling non-neighbor node set, τ is the temperature coefficient, which controls the positive and negative samples (i.e., node v i The balance degree of neighbor nodes and non-neighbor nodes;

[0055] After the comparison aggregation module is trained to convergence, the final aggregation matrix H of the module Agg That is the final learned graph node embedding representation matrix;

[0056] This concludes the process of the self-supervised message passing graph representation learning method based on contrastive generation.

[0057] The present invention combines the above technologies to propose a self-supervised message passing graph representation learning method based on contrast generation; in order to solve the problem that the embedding representation obtained by the existing unsupervised graph representation learning method has low accuracy in expressing information, a contrast generation module is used to learn the embedding representation of the node, and an aggregation model is used to aggregate the embedding representation of the node, and the two are used in combination to obtain an embedding representation with higher information expression accuracy; in addition, introducing the contrast idea into the generative method and the aggregation module at the same time can effectively improve the discriminability of the final embedding and improve the robustness of the model.

[0058] The advantages of the present invention are: (1) the contrastive method is combined with the generative method, which solves the problem that the generative method is greatly affected by noise and the discriminability of input features, and improves the robustness of the model; (2) the contrastive aggregation model retains high-order neighborhood information for the learned embedding, while improving the discriminability of the final embedding matrix; (3) as a self-supervised graph representation learning method, there is no strict requirement for the data set, and the learned embedding representation is equal to or even exceeds the supervised learning method in terms of information expression accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 It is an overall flow chart of the method of the present invention.

[0060] Figure 2 It is a specific illustration of the method of the present invention. DETAILED DESCRIPTION

[0061] The specific implementation of the present invention is described below with reference to embodiments.

[0062] Suppose we perform graph representation learning on the Citeseer citation dataset. This dataset has 3327 nodes and 4552 edges, and the feature dimension of each node is 3703 dimensions.

[0063] The method of using the self-supervised message passing graph representation learning method based on contrast generation for literature recommendation is as follows: 1 Data preprocessing

[0064] 1.1 Graph data processing and dataset segmentation.

[0065] (1) Check the isolated nodes in the graph dataset and add edges pointing to themselves for all isolated nodes; if the edges in the graph dataset are directed edges, they are all converted into undirected edges.

[0066] (2) Divide the nodes in the data set into a ratio of training set / validation set / test set = 1:1:8.

[0067] 2. Contrast Generation Module Training

[0068] 2.1 Construct feature view.

[0069] (1) For the initial feature matrix Calculate the similarity matrix

[0070] (2) Set K of the K-nearest neighbor algorithm to 3, select the three nodes with the highest similarity in each node, add edges to construct the feature view G f =(X,A f ), G f It has 3327 nodes and 9981 edges.

[0071] 2.2 The 3327 nodes of the initial graph G are randomly masked at a ratio of 50%, and the input features of the masked nodes are all set to 0.

[0072] 2.3 Training Node Embeddings.

[0073] (1) Set the node embedding dimension d to 64.

[0074] (2) Randomly initialize the parameters of the 2-layer GAT encoder. The input dimension of the encoder is 3703 and the output dimension is 64.

[0075] (3) The masked initial image and feature view G f =(X,A f ) Input the encoder to get the initial graph embedding matrix

[0076] and the feature view embedding matrix

[0077] 2.4 Multi-level contrastive learning.

[0078] (1) Randomly sample the neighborhood of each node on the initial graph to generate the node's topological subgraph. All first-order neighbors of the node on the feature view are used as the node's feature subgraph. The embeddings of all nodes in the two subgraphs are averaged to obtain the embedding representation of the topological subgraph and the feature subgraph. and

[0079]

[0080] (2) Connect the embeddings of the topological subgraph and feature subgraph with the node embedding to obtain the cross-level representation of the topological view and feature view. and

[0081]

[0082] (3) Using the cross-level representation C of the topological view, the subgraph representation Z, the node hidden layer representation H, and the cross-level representation C of the topological view f , subgraph represents Z f , node hidden layer representation H f Calculate node and subgraph loss L N , subgraph loss L G and the cross-level loss L M .

[0083] 2.5 Generative Learning Training.

[0084] (1) The masked node features are reconstructed using the hidden layer embedding H of the initial graph and the adjacency matrix A. The structure of the decoder is symmetric to that of the encoder.

[0085] (2) Compare the reconstructed node features with the original features to calculate the reconstruction loss L AE .

[0086] 2.6 Compare the loss and the generated loss according to the weight set, use the Adam algorithm to minimize the loss function L, and optimize the encoder parameters.

[0087] 2.7 Repeat steps 2.2 to 2.7 until the contrast generation model converges or reaches the specified number of training times, and output the final embedding matrix H as the initial input of the next module.

[0088] 3. Comparative aggregation training.

[0089] 3.1 Randomly Initialize the Weight Matrix of the Aggregation Model and

[0090] 3.2 Initializing node embeddings using the final output of the contrast generation module

[0091] And calculate the neighbor embedding matrix H Nei .

[0092] 3.3 From the weight matrix W Pre and W Nei And initialize the node embedding H Pre and the neighbor embedding matrix H Nei The calculated aggregate embedding H Agg

[0093] 3.4 Set the number of negative samples for each node to 25, and randomly sample the non-neighboring nodes of the node to generate the negative sample set V Neg .

[0094] 3.5 Embedding H by Aggregation Agg And the negative sample set V Neg The comparative aggregation loss L is calculated Agg , use the Adam algorithm to minimize the contrast aggregation loss and optimize the weight matrix W Pre and W Nei .

[0095] 3.6 Repeat steps 3.2 to 3.5 until the model converges or reaches the specified number of training times; the final aggregate embedding H Agg This is the 64-dimensional embedding representation result of the Citeseer dataset node.

[0096] 4 Input the newly added document features into the model to realize document subject classification and related document recommendation.

[0097] When a newly published document is added to the citation database, the newly constructed graph structure data is input into the trained model to obtain the embedding results of the document nodes. The subject classification of the documents is carried out using the node embedding similarity between the documents, and high-quality related document recommendations are provided to researchers to help them discover the latest research results in the target field.

[0098] This concludes the specific implementation steps of graph representation learning on the Citeseer dataset using the self-supervised message passing graph representation learning method based on contrastive generation.

[0099] The contents described in the embodiments of this specification are merely an enumeration of the implementation forms of the inventive concept. The protection scope of the present invention should not be regarded as limited to the specific forms described in the embodiments. The protection scope of the present invention also extends to equivalent technical means that can be conceived by those skilled in the art based on the inventive concept.

Claims

1. Apply the method recommended by the literature on self-supervised message graph representation learning method based on contrastive generation, including the following steps: Step 1: Data preprocessing, including: Graph data processing and data partitioning; Add self-loops pointing to all isolated nodes in the original graph dataset; If the edges in the data set are directed edges, convert all directed edges to undirected edges; divide the nodes into training set, validation set and test set in a ratio of 1:1:8; Step 2: Contrastive generation phase; in this phase, a contrastive generation model is used to specifically learn the embedded representation of the node; the contrastive generation model consists of a contrast module and a generation module, which share an encoder structure and jointly optimize the embedded representation of the node based on the idea of ​​contrast and generation; Step 3: Comparison aggregation stage: The node embedding H of the topology graph G obtained by the comparison generation module is passed to the aggregation model, and the comparison aggregation model aggregates the node embeddings. After training, the final node embedding of this model is obtained; Step 4: Input the newly added document features into the model to achieve document subject classification and related document recommendation; when a newly published document is added to the citation database, the newly constructed graph structure data is input into the trained model to obtain the embedding results of the document nodes, and the node embedding similarity between the documents is used to classify the documents by subject, and provide related document recommendations.

2. The method for literature recommendation using a self-supervised message graph representation learning method based on contrast generation as claimed in claim 1, characterized in that: Step 2 specifically includes: 2.1 Constructing feature view: First, calculate the feature matrix by formula (1): The similarity matrix Select the first K similar nodes of each node as the neighbor nodes of this node, and finally use the K-nearest neighbor algorithm to construct the feature view G for the initial graph G = (A, X) f =(A f ,X); where N represents the number of nodes in the graph dataset, and d0 represents the initial dimension of the feature matrix X; where x i represents the initial feature vector of the i-th node, S i,j Indicates the similarity between the i-th node and the j-th node; 2.2 Mask some nodes of the initial graph; randomly sample the node set V of the initial graph according to a certain proportion to obtain a node subset Pair Collection The node features within are masked, and the masked features are shown in (2); Among them, v i represents the i-th node in the initial graph; 2.3 Training node embedding: Using the graph neural network GAT as the encoder, the masked initial graph and feature view G f =(A f ,X) is used as the input of the encoder, and after being embedded by the encoder, the masked initial graph node embedding matrix is ​​obtained And the feature view embedding matrix The details are shown in formula (3) and (4); H f =Encode(A f ,X) (4) Where d is the dimension of the embedding vector, Encode(·) represents the encoder structure using the graph attention neural network; 2.4 Multi-level contrastive learning: First, to achieve contrastive learning between nodes and subgraphs, it is necessary to generate an embedded representation at the subgraph level. Specifically, randomly sample the neighbor nodes of each node on the initial graph to generate a topological subgraph for each node. Use all neighbor nodes of each node on the feature view to generate the feature subgraph of the node. Average the hidden layer embeddings of the two subgraphs of all nodes to obtain the hidden layer embeddings of the topological subgraphs of all nodes. and feature subgraph embedding The specific formula is as shown in formula (5) and (6): in For node v i The topological subgraph set, N f,i For node v i The feature subgraph set of , Mean(·) represents the average readout function; Contrastive learning is divided into three types of comparisons: node-subgraph, subgraph-subgraph, and cross-level comparisons. Cross-level contrastive learning requires constructing a joint representation of node embedding and subgraph embedding, connecting the hidden node vectors of the topological view and feature view with the subgraph vectors, respectively, to obtain the cross-level vectors of the topological view and feature view. and The details are as shown in (7)(8); c i =Concat(h i ,z i ) (7) c f,i =Concat(h f,i ,z f,i ) (8) where c i and c f,i Respectively represent the node v i The cross-level vectors of the topological view and feature view of , Concat(·) represents the connection operation; For three types of contrastive learning, InfoNCE is used as the loss function, and the specific node and subgraph loss L N , subgraph loss L G And the cross-level loss L M As shown in formulas (9), (10), and (11): Where τ represents the temperature coefficient, which controls the strictness of positive and negative samples; Batch represents the node samples in the same batch; In formulas (10) and (11), 1 [i≠j] It is an indicator function, which takes 1 when the condition is met and 0 when it is not met; 2.5 Generate learning training; The decoder uses the node hidden layer embedding H of the topological graph and the adjacency matrix A to reconstruct the masked node features. The structure of the decoder is the same as that of the encoder. The specific reconstruction process is shown in formula (12): Where Decode(·) represents a 2-layer GAT decoder structure with an input dimension of d and an output dimension of d0; represents the reconstructed node feature matrix, Represents node v i The reconstructed feature vector is used to calculate the loss of reconstructing the masked node features through the SCE loss, as shown in formula (13): Loss function L AE The average reconstruction loss of all masked node features is calculated, where the scaling factor γ is a hyperparameter adjusted on different datasets and is considered as an adaptive sample weight adjustment and scaling technique; 2.6 Encoder optimization: The loss functions of the comparison module and the generation module are integrated through weight parameters, and the Adam algorithm is used to minimize the objective function L to optimize the parameters of the encoder Encode(·). The objective function calculation formula is shown in formula (14): L=L AE +α·L M +β·L G +λ·L N (14) Among them, α, β, and λ are the weight parameters that control the cross-level contrast loss, subgraph contrast loss, and node-subgraph contrast loss respectively. After the contrast generation module is trained multiple times until convergence, the topological view node hidden layer embedding H obtained by the encoder at the final convergence is used as the input matrix of the next module.

3. The method for literature recommendation using a self-supervised message graph representation learning method based on contrast generation as claimed in claim 1, characterized in that: Step 3 specifically includes: 3.1 Aggregation model initialization; Randomly initialize the weight matrix W of the aggregation model Pre and Where W Pre is the weight matrix of node embedding, W Nei is the neighbor embedding weight matrix, heads is a hyperparameter used to control the number of aggregation heads, which must satisfy heads≤d and heads is a factor of d; 3.2 Obtain node embedding and calculate neighbor embedding representation; node embedding matrix H Pre To compare the node embedding matrix H output by the generation module, the neighbor embedding matrix H Nei About node v i Neighbor embedding representation The calculation formula is shown in formula (15): Among them, Norm(·) represents vector normalization, N i is node v i The neighbor node set of , sim(·) is the similarity calculation function, where cosine similarity is used; 3.3 Update aggregate embedding; aggregate embedding matrix H Agg The calculation formula is shown in formula (16): H Agg =Norm(W Pre *H Pre +W Nei *H Nei ) (16) Where * represents the element-wise product operation; 3.4 Non-neighbor node sampling; for any node v in the topology graph i Randomly select a fixed number of non-neighbor nodes to generate a negative sample node set V i Neg , the number of nodes in the set can be set manually; 3.5 Comparative aggregation model training; using the Adam algorithm to maximize the model comparison loss function L Agg , continuously optimize the weight matrix W of the aggregation model Pre and W Nei Until the model converges; aggregate model value function L Agg The calculation formula is shown in formula (17): Where N i For node v i The set of neighbor nodes, V i Neg For node v i The sampling non-neighbor node set, τ is the temperature coefficient, which controls the positive and negative samples, i.e., the node v i The balance degree of neighbor nodes and non-neighbor nodes; After the comparison aggregation module is trained to convergence, the final aggregation matrix H of the module Agg This is the final learned graph node embedding representation matrix.

Images