Node classification method for large-scale self-supervised graph representation

Abstract

The invention discloses a node classification method for large-scale self-supervised graph representation, which relates to the technical field of electronic information, and comprises the following steps of: firstly, carrying out mask operation on node features, and reconstructing the masked node features by utilizing a part of observable node signals and an input adjacency matrix; designing a GCN encoder and an MLP encoder, and maximizing the correlation between the GCN encoder and the MLP encoder in an embedding space; node embedding representation is obtained through two encoders, node embedding with high representation capacity is obtained through feature fusion, node features are reconstructed based on a GCN decoder, and the correlation between the reconstructed features and original features is maximized. According to the node classification method represented by the large-scale self-supervised graph, only the MLP is used in the reasoning process, so that the reasoning speed is greatly increased, and deployment of sensitive applications is easy to delay. Gradient truncation is carried out on the MLP, and parameter updating comes from GCN, so that trainable parameters are reduced, and the training efficiency is improved.

Description

Technical Field

[0001] The invention relates to the field of electronic information technology, and in particular to a node classification method represented by a large-scale self-supervisory graph. Background Art

[0002] In graph data, entities are nodes and the relationships between entities are edges. Node classification is an important task in graph data mining, which aims to classify nodes into different categories based on the attribute characteristics of nodes and the structural information of the graph. Node classification technology has evolved from traditional methods based on manual features to graph neural networks based on deep learning, and has gradually introduced cutting-edge methods such as self-supervised learning, reducing the dependence on labeled data. Self-supervised methods capture graph structure and node attribute information by designing pre-training tasks (such as node attribute prediction, edge reconstruction, etc.), thereby achieving effective representation in node classification tasks and achieving better generalization ability. They have been widely used in applications such as social networks and recommendation systems. In the node classification task of self-supervised graph representation learning, there are two main methods: 1) Contrastive learning-based methods: By designing contrastive tasks, the model learns effective feature representations in an unsupervised environment. The key includes negative sampling, architecture design (such as momentum update or gradient stop), and data augmentation (such as feature masking and random walk). Although contrastive learning is effective in node classification, it faces the problems of sensitive negative sample selection and high consumption of computing resources. 2) Generative-based methods: Capture data structure and feature relationships by restoring the missing parts of the input data. The main methods include autoregressive models and autoencoder models. The latter is more suitable for graph data but performs poorly in node classification tasks. The emerging masked feature reconstruction method has shown some effect in graph representation learning, but has not yet surpassed contrastive learning. Summary of the invention

[0003] The purpose of the present invention is to provide a large-scale self-supervised graph representation node classification method, which adopts a graph autoencoder architecture to avoid the consumption of storage and computing resources in the contrastive learning process. At the same time, a multi-layer perceptron encoder is set up to align the output of the graph neural network encoder in the embedding space to speed up the reasoning speed without reducing the generalization performance.

[0004] To achieve the above object, the present invention provides a large-scale self-supervised graph representation node classification method, comprising the following steps:

[0005] Step 1: Graph enhancement: enhance the given graph and use a uniform random sampling strategy without replacement to obtain mask nodes;

[0006] Step 2: Graph alignment: construct two graph encoders, graph convolutional network (GCN) and multi-layer perceptron (MLP), and align the embedding space representations obtained by the two encoders.

[0007] Step 3: Parameter sharing: During the training process, the MLP parameters are not updated, the gradient of the MLP is truncated, and the GCN parameters are migrated to the MLP, so that the encoders of two different architectures share parameters.

[0008] Step 4: The representations obtained by the two encoders are averaged to perform feature fusion and sent to the decoder to reconstruct the features;

[0009] Step 5: Graph decoding: a single-layer GCN is used as a decoder to decode the reconstructed features.

[0010] Step 6: Use scaled cosine error (SCE) to measure the similarity between the reconstructed features and the original features.

[0011] Step 7: After the pre-training in steps 1 to 4 converges, only the MLP encoder is retained. In the inference phase, the MLP encoder does not perform any mask processing. After obtaining the final node embedding, a linear classifier is trained for node classification prediction.

[0012] Preferably, in the multi-layer perceptron MLP in step 2, the feed-forward layer is specifically expressed as:

[0013] H (l) =σ(H (l-1) W (l) +b (l) );

[0014] Among them, W (l) , H (l-1) and b (l) They are respectively represented as the weight matrix of the lth layer, the neuron output of the l-1th layer and the bias vector of the lth layer, and σ represents the activation function.

[0015] Preferably, the GCN graph convolution layer in step 2 is expressed as:

[0016]

[0017] in, The adjacency matrix A of the graph plus the identity matrix I represents the adjacency matrix with self-loops added. according to The calculated degree matrix (diagonal matrix) has elements represents the degree of node i, Symmetrically normalized adjacency matrix.

[0018] Preferably, the embedding representations in step 2 are aligned, and the difference between the embedding representations obtained by the two encoders is measured using KL divergence:

[0019]

[0020] Among them, D KL represents the loss function, HGCN and H MLP They are two representations of nodes in the embedding space.

[0021] Preferably, the parameters in step 3 are shared, the gradient of MLP is truncated, and the MLP parameters are updated according to the GCN parameters, that is:

[0022] θ MLP ←θ GCN ;

[0023] Among them, θ MLP represents the MLP parameters, θ GCN Represents GCN parameters.

[0024] Preferably, the feature fusion formula in step 4 is:

[0025] H fuse =(H MLP +H GCN ) / 2;

[0026] Among them, H GCN and H MLP are two representations of a point in the embedding space.

[0027] Preferably, the decoding process in step 5 is:

[0028]

[0029] Among them, Z rec represents the reconstructed node representation, W represents the weight matrix, and b represents the bias vector.

[0030] Preferably, the formula for measuring similarity in step 6 is:

[0031]

[0032] Among them, γ≥1, take γ=2, z i represents the reconstructed representation of node i.

[0033] Preferably, in step 6, the scaled cosine error plus KL divergence is used as the loss function of self-supervised pre-training, which is expressed as:

[0034]

[0035] Among them, λ is the weight coefficient, taking λ=0.8, D KL represents the KL divergence.

[0036] Preferably, in step 7, the connection layer FC is used as the classifier for node classification, and the multi-classification cross entropy loss is used as the target:

[0037]

[0038]

[0039] in, represents multi-classification cross entropy loss, N represents the number of samples, C represents the number of categories, and y ij represents the j-th category label of the i-th sample, It represents the predicted probability that the i-th sample belongs to the j-th category.

[0040] Therefore, the present invention adopts the above-mentioned large-scale self-supervised graph representation node classification method, proposes a new large-scale graph representation self-supervised pre-training framework, introduces the MLP module, and aligns the node embeddings generated by MLP and GCN in the embedding space so that MLP can learn not only the feature information of the nodes but also the structural information between the nodes. In the reasoning process, only MLP is used to greatly speed up the reasoning speed, which is easy to deploy delay-sensitive applications. By performing gradient truncation on MLP, its parameter update comes from GCN, which reduces the trainable parameters and improves the training efficiency.

[0041] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 The present invention is a schematic diagram of a node classification method represented by a large-scale self-supervised graph. DETAILED DESCRIPTION

[0043] The technical solution of the present invention is further described below through the accompanying drawings and embodiments.

[0044] Unless otherwise defined, the technical terms or scientific terms used in the present invention should be understood by people with ordinary skills in the field to which the present invention belongs. The words "first", "second" and similar words used in the present invention do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprise" and similar words mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0045] Example

[0046] See also Figure 1, the present invention provides a node classification method for large-scale self-supervised graph representation. The overall model includes the following structures: a graph enhancement module, a graph encoding and alignment module, a feature fusion and graph decoding module, and a graph reasoning and downstream task module. We take feature reconstruction as the training goal. First, we mask the node features and use some observable node signals and the input adjacency matrix to reconstruct the masked node features. In order to take into account the learning of contextual semantic information of the graph structure and the efficiency of reasoning, we design two encoders, GCN and MLP, and maximize the correlation between them in the embedding space. Based on the connection between GCN and MLP, we let the two encoders share trainable parameters to improve the training speed of the model. After obtaining the node embedding representation through these two different encoders, we fuse the features and then send them to the GCN-based decoder to reconstruct the node features and maximize the correlation between the reconstructed features and the original features. The whole process optimizes the target of the loss function through backpropagation.

[0047] Step 1: Graph enhancement. Given a graph We can use augmentation techniques to generate a new view Randomly select a subset from the node set V Each node feature is masked with a mask tag [MASK]. This tag is a learnable vector Therefore, in the mask feature matrix In the example, node v i The feature vector of ∈V It can be expressed as:

[0048]

[0049] Uniform random sampling without replacement is used to determine the nodes that need to be masked. In addition, a higher mask ratio helps to reduce redundancy in the attribute graph in most cases. In the present invention, the mask ratio is set to 0.65.

[0050] Step 2: We align the embedding space representations obtained by different encoders. We design two different encoders: Graph Convolutional Encoder (GCN) and Multi-layer Perceptron (MLP). MLP is based only on node features and is usually represented as a series of fully connected (feedforward) layers. The feedforward layer is specifically represented as follows:

[0051] H (l) =σ(H (l-1) W (l) +b (l) );

[0052] Among them, W (l) , H (l-1) and b (l)They are respectively represented as the weight matrix of the lth layer, the neuron output of the l-1th layer, and the bias vector of the lth layer, and σ represents the activation function. To further integrate the structural information, we designed a graph neural network encoder (GCN). The graph convolution layer is specifically represented as follows:

[0053]

[0054] in, The adjacency matrix A of the graph plus the identity matrix I represents the adjacency matrix with self-loops added. In this way, each node includes its own features in the convolution. according to The calculated degree matrix (diagonal matrix) has elements represents the degree of node i. The symmetric normalized adjacency matrix is ​​used to eliminate the influence of different node degrees and make feature aggregation more stable. In the present invention, the activation function is ELU, GCN and MLP are 2 layers, and the hidden layer dimension is 512.

[0055] After passing through two encoders GCN and MLP, we get two representations H of the node in the embedding space GCN and H MLP In order to use MLP to simulate the prediction of GNN, it is necessary to maximize the correlation between the two representations in the embedding space. Here, KL divergence is selected to maximize the fine-grained correlation between node features and graph structure. Its loss function is expressed as:

[0056]

[0057] Step 3: Parameter sharing: During the training process, the MLP parameters are not updated, the gradient of the MLP is truncated, and the GCN parameters are migrated to the MLP, so that the encoders of two different architectures share parameters.

[0058] The gradient of MLP is truncated, and the MLP parameters are updated according to the GCN parameters, that is:

[0059] θ MLP ←θ GCN ;

[0060] Among them, θ MLP represents the MLP parameters, θ GCN Represents GCN parameters.

[0061] Step 4: The representations obtained by the two encoders need to be subjected to feature fusion operation and then sent to the decoder to reconstruct the features. Common feature fusion operations include splicing, weighted summation, averaging, etc. The present invention uses a relatively simple averaging, which is expressed as:

[0062] H fuse =(H MLP+H GCN ) / 2;

[0063] Step 5, a more expressive single-layer GCN is used as its decoder. The GCN decoder can recover the input features based on a group of nodes instead of a single node, which helps the encoder learn high-level potential encoding. The graph decoding process is as follows:

[0064]

[0065] Among them, Z rec represents the reconstructed node representation, W represents the weight matrix, and b represents the bias vector.

[0066] Step 6: Use scaled cosine error (SCE) to measure the similarity between the reconstructed features and the original features. The formula is as follows:

[0067]

[0068] The contribution of simple samples in training is reduced by scaling the cosine error (exponent greater than or equal to 1), and in the present invention, γ=2.

[0069] The scaled cosine error plus KL divergence is used as the loss function for self-supervised pre-training, which is expressed as:

[0070]

[0071] Wherein, λ is a weight coefficient, and in the present invention, λ=0.85. The model parameters are updated using the adam optimizer to minimize the loss function.

[0072] Step 7: After multiple iterations of pre-training, only the MLP encoder is retained. In the inference phase, the MLP encoder is applied to the feature matrix X of the input graph without any masking:

[0073] Z = MLP(X);

[0074] Among them, Z represents the node embedding representation obtained after MLP.

[0075] After getting the embedded representation, a simple connection layer (FC) is trained as a classifier for node classification. Multi-classification cross entropy loss is used as the target:

[0076]

[0077]

[0078] Among them, N represents the number of samples, C represents the number of categories, and y ij represents the j-th category label of the i-th sample (one-hot encoding), It represents the predicted probability that the i-th sample belongs to the j-th category. The adam optimizer is also used to minimize the loss function value so that the classifier can better classify into the correct category.

[0079] The present invention is based on the pytorch deep learning framework and experiments are conducted on the citeseer dataset, but is not limited to this dataset. In this dataset, nodes represent scientific publications and edges represent citation relationships. Each node has a 3703-dimensional predefined feature. This dataset is used for node classification tasks, and the task is to predict the categories of certain publications. In the process of training a linear classifier, the dataset is divided into public partitions, with 120 training set samples, 500 validation set samples, and 1000 test set samples. Experiments show that the accuracy of the node classification prediction disclosed by the present invention reaches 72.2%.

[0080] Therefore, the present invention adopts the above-mentioned large-scale self-supervised graph representation node classification method, proposes a new large-scale graph representation self-supervised pre-training framework, introduces the MLP module, and aligns the node embeddings generated by MLP and GCN in the embedding space so that MLP can learn not only the feature information of the nodes but also the structural information between the nodes. In the reasoning process, only MLP is used to greatly speed up the reasoning speed, which is easy to deploy delay-sensitive applications. By performing gradient truncation on MLP, its parameter update comes from GCN, which reduces the trainable parameters and improves the training efficiency.

[0081] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solution of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solution to deviate from the spirit and scope of the technical solution of the present invention.

Claims

1. A large-scale self-supervised graph representation node classification method, characterized in that: The following steps are involved: Step 1: Graph enhancement: enhance the given graph and use a uniform random sampling strategy without replacement to obtain mask nodes; Step 2: Graph alignment: construct two graph encoders, graph convolutional network (GCN) and multi-layer perceptron (MLP), and align the embedding space representations obtained by the two encoders. Step 3: Parameter sharing: During the training process, the MLP parameters are not updated, the gradient of the MLP is truncated, and the GCN parameters are migrated to the MLP, so that the encoders of two different architectures share parameters. Step 4: The representations obtained by the two encoders are averaged to perform feature fusion and sent to the decoder to reconstruct the features; Step 5: Graph decoding: a single-layer GCN is used as a decoder to decode the reconstructed features. Step 6: Use scaled cosine error (SCE) to measure the similarity between the reconstructed features and the original features. Step 7: After the pre-training in steps 1 to 4 converges, only the MLP encoder is retained. In the inference phase, the MLP encoder does not perform any mask processing. After obtaining the final node embedding, a linear classifier is trained for node classification prediction.

2. The node classification method for large-scale self-supervised graph representation according to claim 1, characterized in that: In step 2, the multi-layer perceptron MLP, the feed-forward layer is specifically expressed as: H (l) =σ(H (l-1) W (l) +b (l) ); Among them, W (l) , H (l-1) and b (l) They are respectively represented as the weight matrix of the lth layer, the neuron output of the l-1th layer and the bias vector of the lth layer, and σ represents the activation function.

3. The node classification method for large-scale self-supervised graph representation according to claim 2, characterized in that: The GCN graph convolution layer in step 2 is expressed as: in, The adjacency matrix A of the graph plus the identity matrix I represents the adjacency matrix with self-loops added. according to The calculated degree matrix (diagonal matrix) has elements represents the degree of node i, Symmetrically normalized adjacency matrix.

4. The node classification method for large-scale self-supervised graph representation according to claim 3, characterized in that: The embedding representation alignment in step 2 uses KL divergence to measure the difference between the embedding representations obtained by the two encoders: Among them, D KL represents the loss function, H GCN and H MLP They are two representations of nodes in the embedding space.

5. The node classification method for large-scale self-supervised graph representation according to claim 4, characterized in that: The parameter sharing in step 3 performs gradient truncation on MLP, and the MLP parameters are updated according to the GCN parameters, namely: i MLP ←θ GCN ; Among them, θ MLP represents the MLP parameters, θ GCN Represents GCN parameters.

6. The node classification method for large-scale self-supervised graph representation according to claim 5, characterized in that: The feature fusion formula in step 4 is: H fuse =(H MLP +H GCN ) / 2; Among them, H GCN and H MLP are two representations of a point in the embedding space.

7. The node classification method for large-scale self-supervised graph representation according to claim 6, characterized in that: The decoding process in step 5 is: Among them, Z rec represents the reconstructed node representation, W represents the weight matrix, and b represents the bias vector.

8. The node classification method for large-scale self-supervised graph representation according to claim 7, characterized in that: The formula for measuring similarity in step 6 is: Among them, γ≥1, take γ=2, z i represents the reconstructed representation of node i.

9. The node classification method for large-scale self-supervised graph representation according to claim 8, characterized in that: In step 6, the scaled cosine error plus KL divergence is used as the loss function for self-supervised pre-training, expressed as: Among them, λ is the weight coefficient, taking λ=0.8, D KL represents the KL divergence.

10. The node classification method for large-scale self-supervised graph representation according to claim 9, characterized in that: In step 7, the connection layer FC is used as the classifier for node classification, and the multi-classification cross entropy loss is used as the target: in, represents multi-classification cross entropy loss, N represents the number of samples, C represents the number of categories, and y ij represents the j-th category label of the i-th sample, It represents the predicted probability that the i-th sample belongs to the j-th category.

Images