Graph mask auto-encoder generation type self-supervision graph pre-training method, equipment and product

Abstract

The invention discloses a graph mask auto-encoder generation type self-supervision graph pre-training method, equipment and a product. The method comprises the following steps: firstly, inputting a graph data set needing graph classification, a node feature set, a real label of a node, a proportion of a mask, the number of clustering sub-graphs and the number of layers of a neural network; then, node features in the graph are shielded by using a clustering mask method which transits from local to global, firstly, shielding is started from local nodes in a sub-graph, and all nodes in the whole sub-graph are shielded step by step, so that a graph mask self-encoder starts from local information learning in the graph; the global information of the cross-subgraph is gradually fused; and finally, training a graph mask auto-encoder by using a hybrid encoding mechanism for capturing global information. The graph mask auto-encoder combined with the global information provided by the invention aims to promote the graph mask auto-encoder to enhance the utilization of the global information in the graph, which is beneficial for improving the accuracy and wide applicability of a model in a graph classification scene in the real world.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer science and network data mining, and relates to a graph mask autoencoder training method, device and product combined with global information, and specifically to a graph mask autoencoder generative self-supervised graph pre-training method, device and product based on graph structure information and node feature information. Background Art

[0002] A graph is a data structure used to describe entities and their relationships. It uses nodes and edges to represent entities and complex relationships between entities. As a rich information carrier, graph data is different from traditional data formats such as images and texts. Graph data has no fixed structure and is highly sparse. It is becoming increasingly important in data mining and information retrieval research.

[0003] The great success of deep learning on graph-structured data often relies heavily on massive amounts of labeled data, and most work focuses on supervised or semi-supervised learning settings, that is, training models through specific downstream tasks and abundant labeled data. However, in existing research, sometimes labeled data for graph structures is extremely scarce, and high-quality labeled data usually requires a lot of time and resource costs, especially in the fields of bioinformatics, social networks, etc. The scarcity of labeled data and the difficulty of annotation limit the widespread application of supervised or semi-supervised learning methods in graph-structured data research.

[0004] To address this challenge, many techniques and methods have been proposed to make full use of the existing large amount of unlabeled data, aiming to enhance the training effect of graph learning models. Self-supervised graph pre-training methods have attracted much attention due to their effectiveness and have received a lot of attention in academia and industry. Self-supervised graph pre-training methods can be roughly summarized as contrastive self-supervised graph learning and generative self-supervised graph learning. In the research and development of generative self-supervised graph learning, the emergence of graph mask autoencoders marks a major progress in the field of self-supervised graph pre-training, especially the effect of graph mask autoencoders in downstream tasks such as graph classification. However, existing graph mask autoencoders mainly use local information to learn the representation of graphs through random masking methods, simple graph neural network encoders and node-level optimization objectives, thereby ignoring the rich global information in the graph. Global information is understood as information that reflects the semantics and structure of the entire graph, which is extracted by information fusion of long-distance nodes in the graph. Summary of the invention

[0005] In order to solve the above technical problems, the present invention combines a graph mask autoencoder with global information, and proposes a generative self-supervised graph pre-training method, device and product of a graph mask autoencoder based on graph structure information and node feature information.

[0006] The technical solution adopted by the method of the present invention is: a graph mask autoencoder generative self-supervised graph pre-training method, comprising the following steps:

[0007] Step 1: Input the graph dataset to be classified, the node feature set, the true label of the node, the ratio of the mask, the number of cluster subgraphs, and the number of neural network layers;

[0008] Step 2: Use a clustering masking method that transitions from local to global to mask the node features in the graph. First, mask the local nodes in a subgraph, and then gradually mask all the nodes in the entire subgraph. In this way, the graph mask autoencoder starts with learning local information in the graph and gradually integrates global information across subgraphs.

[0009] Step 3: Train the graph mask autoencoder using a hybrid encoding mechanism that captures global information.

[0010] Preferably, the specific implementation of step 2 includes the following sub-steps:

[0011] Step 2.1: Graph data preprocessing;

[0012] The graph data is divided into several non-intersecting cluster subgraphs using the METIS cluster subgraph algorithm, and each cluster subgraph does not intersect with each other;

[0013] Step 2.2: Cluster the node masks within the subgraph;

[0014] At the beginning of training, node features of a part of the node set inside each cluster subgraph are randomly selected for masking;

[0015] Step 2.3: Transition from within the cluster subgraph to between cluster subgraph masks;

[0016] As training progresses, the features of all nodes in some cluster subgraphs will be completely masked, while the features of some nodes in other remaining cluster subgraphs will continue to be masked;

[0017] Step 2.4: Node mask across cluster subgraphs;

[0018] In the final stage of training, all node features in the selected cluster subgraph will be completely masked out, and the learning goal of the model becomes to predict the node features in the completely masked subgraph based on the node information in other remaining subgraphs.

[0019] Preferably, the graph mask autoencoder in step 2 comprises a graph neural network encoding module, a Transformer encoding module and a decoder module; the graph neural network encoding module comprises N graph isomorphic networks arranged in series, so as to capture and fuse local, node neighborhood-level information in the input graph; the Transformer encoding module comprises N self-attention mechanism layers arranged in series, so that the graph mask autoencoder increases the ability to consider global graph information by increasing long-distance interactions in the entire graph; the decoder module comprises a graph isomorphic network or a multi-layer perceptron, so as to map the potential node embeddings generated by the encoder into reconstructed node features; wherein N is a preset value.

[0020] Preferably, the encoding process of the graph neural network is as follows:

[0021]

[0022] Where A represents the adjacency matrix of the input graph, H (l-1) represents the node embedding after passing through the l-1th layer of graph neural network encoder, Represents the l-th layer graph neural network encoder.

[0023] Preferably, the encoding process of the graph Transformer is as follows:

[0024]

[0025] Among them, H (l) is the potential node embedding after the l-th layer Transformer encoder, as well as Figure 3 shows the learnable parameter matrix of the self-attention layer in Transformer, σ represents the softmax function before the final output, and d represents the dimension of the potential node embedding.

[0026] Preferably, in the decoder part, the decoder f maps the potential node embedding H back to the node feature X; after the graph mask self-encoder outputs the potential representation of all nodes, the masked nodes are re-masked before being input to the decoder;

[0027]

[0028] in, is node v i The node latent representation input to the decoder, h m is the node representation after re-masking, and V represents the set of masked nodes;

[0029] After re-masking, the node representations of the graph are input into the decoder to reconstruct the input features of the nodes.

[0030] Preferably, in step 3, a multi-task optimization strategy is used to train the image mask autoencoder; wherein the multi-task optimization loss function L used is:

[0031] L = L1 + λL2;

[0032]

[0033] Among them, V m represents the set of masked nodes, x i represents the original feature of node i, h i represents the node feature of node i restored by the decoder; λ is the weight value; Z T represents the graph-level representation obtained from a large pre-trained teacher model, POOL represents the graph pooling operation, and f proj It represents the projection processing function. Represents node v i Node representations obtained from the student model.

[0034] The technical solution adopted by the device of the present invention is: a graph mask autoencoder generative self-supervised graph pre-training device, including a memory, a processor and a computer program stored on the memory and executable on the processor, and the processor implements a graph mask autoencoder generative self-supervised graph pre-training method when executing the program.

[0035] The technical solution adopted by the product of the present invention is: a software product, including a computer program, which implements a generative self-supervised graph pre-training method for a graph mask autoencoder when executed by a processor.

[0036] Compared with the prior art, the beneficial effects of the present invention include:

[0037] (1) The present invention uses a clustering masking method that transitions from local to global to mask the node features in the graph. It first starts with masking the local nodes in a subgraph, and then gradually masks all the nodes in the entire subgraph. In this way, the model starts from learning the local information in the graph and gradually integrates the global information across subgraphs, which promotes the graph mask autoencoder to perform a learning process from local to global.

[0038] (2) The present invention uses a hybrid coding mechanism that captures global information to train the graph mask autoencoder. The core of this coding mechanism is to utilize the advantages of the two architectures, graph neural network and graph transformer. Graph neural network is good at capturing local, node neighborhood level information, while on the other hand, the introduction of graph transformer increases the ability of the graph mask autoencoder to consider global graph information by increasing long-distance interactions in the entire graph.

[0039] (3) The present invention uses a multi-task optimization strategy that integrates global information to optimize the graph mask autoencoder. The core of this optimization strategy is not only to continue to use node feature reconstruction as the optimization focus of the graph mask autoencoder, but also to combine the auxiliary task of predicting global graph attributes. Using a larger pre-trained graph mask autoencoder as a teacher model to supplement knowledge, the graph mask autoencoder (student model) combined with global information can better capture the global information in the graph and conduct more extensive graph representation learning.

[0040] (4) The present invention designs a multi-task optimization strategy, which breaks the limitation of existing graph mask autoencoders that uses node feature reconstruction as the only optimization target. It not only continues to use node feature reconstruction as the focus of graph mask autoencoder optimization, but also combines the auxiliary task of predicting global graph attributes, so as to enable the model to better capture the global information in the graph, and endows the model with advanced capabilities in global information processing through knowledge distillation methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The technical solution of the present invention is further described below using embodiments and specific implementation methods. In addition, in the process of describing the technical solution, some drawings are also used. For those skilled in the art, other drawings and the intention of the present invention can also be obtained based on these drawings without paying creative work.

[0042] Figure 1 is a flow chart of a method according to an embodiment of the present invention;

[0043] Figure 2 Schematic diagram of cluster masking principle of an embodiment of the present invention;

[0044] Figure 3 2 is a structural diagram of a graph mask autoencoder according to an embodiment of the present invention;

[0045] Figure 4 Schematic diagram of a multi-task optimization strategy according to an embodiment of the present invention. DETAILED DESCRIPTION

[0046] In order to facilitate ordinary technicians in the field to understand and implement the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the implementation examples described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0047] Graph classification is one of the most widely used tasks for graph mask autoencoders on graph structured data. In recent years, many graph mask autoencoder models have been proposed and have made certain breakthroughs in graph classification tasks. However, existing models have limitations in the training process:

[0048] (1) Random masking is widely used. This strategy encourages the model to recover node features mainly through the neighboring nodes of the masked nodes. Although this method can effectively utilize the local structure of the graph, it essentially limits the learning focus of the model to local information interactions and may ignore the rich global context information provided by the graph.

[0049] (2) They mainly use graph neural networks as their encoders, which makes the model biased towards processing local information. Although graph neural networks are good at capturing node-level information interactions and complex details of neighborhood structures, this design limits the model from capturing more distant information in the graph and ignores the rich global information in the graph.

[0050] (3) The local center node is mainly used as the optimization target, and the existing models usually give priority to the reconstruction of the features of the masked nodes, which further strengthens the local center bias of the model. This model optimization method will promote the model to give priority to the restoration of the local neighborhood rather than the consistency of the global graph structure.

[0051] To address the above problems, the present invention proposes a graph mask autoencoder model that combines global information, and makes three improvements to existing research: a clustering mask method that transitions from local to global, a hybrid coding mechanism that captures global information, and a multi-task optimization strategy that integrates global information.

[0052] Please see Figure 1 , a graph mask autoencoder generative self-supervised graph pre-training method provided in this embodiment includes the following steps:

[0053] Step 1: Input the graph dataset to be classified, the node feature set, the true label of the node, the ratio of the mask, the number of cluster subgraphs, and the number of neural network layers;

[0054] Step 2: Use a clustering masking method that transitions from local to global to mask the node features in the graph. First, mask the local nodes in a subgraph, and then gradually mask all the nodes in the entire subgraph. In this way, the graph mask autoencoder starts with learning local information in the graph and gradually integrates global information across subgraphs.

[0055] In one embodiment, see Figure 2 ,The clustering mask method includes the following steps:

[0056] Step 2.1: Graph data preprocessing;

[0057] The graph data is divided into several non-intersecting cluster subgraphs using the METIS cluster subgraph algorithm, and each cluster subgraph does not intersect with each other;

[0058] Step 2.2: Cluster the node masks within the subgraph;

[0059] At the beginning of the graph mask autoencoder training, node features of a part of the node set inside each cluster subgraph are randomly selected for masking; this strategy enables the graph mask autoencoder combined with global information to learn the features of masked nodes based on the unmasked nodes in the same subgraph, thereby enhancing the model's ability to learn subgraph internal information.

[0060] Step 2.3: Transition from within the cluster subgraph to between cluster subgraph masks;

[0061] As the training of the graph mask autoencoder progresses, the features of all nodes in some cluster subgraphs will be completely masked, while the features of some nodes in other remaining cluster subgraphs will continue to be masked;

[0062] Step 2.4: Node mask across cluster subgraphs;

[0063] In the final stage of the training of the graph mask autoencoder, all node features in the selected cluster subgraph will be completely masked, and the learning goal of the graph mask autoencoder is transformed into predicting the node features in the fully masked subgraph based on the node information in other remaining subgraphs. This step is crucial to strengthen the model's ability to capture the relationship between subgraphs and improve its global prediction ability of the graph.

[0064] In one embodiment, see Figure 3 The graph mask autoencoder adopts a hybrid encoding mechanism by integrating graph neural network and graph Transformer.

[0065] The graph mask autoencoder generative self-supervised graph pre-training network structure includes a graph neural network encoding module, a Transformer encoding module and a decoder module. The graph neural network encoding module includes N graph isomorphic networks arranged in series, which are used to capture local, node neighborhood-level information in the fusion input graph. The Transformer encoding module includes N self-attention mechanism layers arranged in series, which increases the ability of the graph mask autoencoder to consider global graph information by increasing long-range interactions in the entire graph. The decoder module includes a graph isomorphic network or a multi-layer perceptron, which is used to map the potential node embeddings generated by the encoder into reconstructed node features.

[0066] Step 3: Train the graph mask autoencoder using a hybrid encoding mechanism that captures global information.

[0067] Please see Figure 3 In one implementation, hybrid coding is used to guide the image mask self-encoder to capture global information, specifically including:

[0068] (1) Processing of graph neural network encoder;

[0069] The graph mask autoencoder that combines global information must also capture local, node neighborhood-level information. Therefore, in the encoder part of the graph mask autoencoder, the graph with some node features masked will first be processed by the graph neural network encoder that focuses on the local information in the graph, and the feature set is used Represents. The processing process of graph neural network encoding is:

[0070]

[0071] Where A represents the adjacency matrix of the input graph, H (l-1) represents the node embedding after passing through the l-1th layer of graph neural network encoder, Represents the l-th layer graph neural network encoder.

[0072] (2) Processing of the Transformer encoder;

[0073] The graph mask autoencoder combined with global information introduces the graph transformer. By increasing the long-distance interactions in the entire graph, the graph mask autoencoder increases the ability to consider global graph information. The output of the previous part of the graph neural network encoding in the encoder becomes the input of the graph transformer encoding. The processing process of the graph transformer encoding is:

[0074]

[0075] Among them, H (l)is the potential node embedding after the l-th layer Transformer encoder, as well as Figure 3 shows the learnable parameter matrix of the self-attention layer in Transformer, σ represents the softmax function before the final output, and d represents the dimension of the potential node embedding.

[0076] (3) Decoder processing

[0077] In the decoder part, the decoder f maps the potential node embedding H back to the node feature X. The graph masked autoencoder uses a re-masking decoding method to process the potential node representation, so that a masked node will be forced to reconstruct its input features from the potential representations of adjacent or global unmasked nodes. After the encoder outputs the potential representations of all nodes, the masked nodes will be re-masked before being input to the decoder;

[0078]

[0079] in, is node v i The node latent representation input to the decoder, h m is the node representation after re-masking, and V represents the set of masked nodes;

[0080] After re-masking, the node representations of the graph are fed into the decoder to reconstruct the input features of the nodes.

[0081] Please see Figure 4 ,In one implementation ,a multi-task optimization strategy is adopted to train the graph mask autoencoder;

[0082] (1) Optimize the loss of node feature reconstruction;

[0083] The decoder of the graph masked autoencoder combined with global information reconstructs the features of the masked nodes based on the potential representation of the unmasked nodes. In order to optimize this training process, the present invention adopts the scaled cosine error.

[0084]

[0085] Among them, V m represents the set of masked nodes, x i represents the original feature of node i, h i Represents the node features of node i restored by the decoder.

[0086] (2) Optimize the knowledge distillation loss of global graph attributes;

[0087] The graph mask autoencoder with global information introduces an auxiliary task of predicting global graph attributes. While the graph mask autoencoder reconstructs node features, the optimization goal of predicting global graph attributes aligns the node embedding of the graph with the graph-level embedding obtained from a large pre-trained graph mask autoencoder to supplement the knowledge of the graph mask autoencoder.

[0088]

[0089] Among them, Z T represents the graph-level representation obtained from a large pre-trained teacher model, POOL represents the graph pooling operation, and f proj It represents the projection processing function. Represents node v i Node representations obtained from the student model.

[0090] (3) Multi-task optimization loss L;

[0091] L = L1 + λL2;

[0092] Among them, λ is the weight value;

[0093] The model of the present invention proposes and designs a multi-task optimization strategy in the target optimization part, breaking the limitation of the existing graph mask autoencoder that takes node feature reconstruction as the only optimization target. It not only continues to take node feature reconstruction as the focus of model optimization, but also combines the auxiliary task of predicting global graph properties, so as to enable the model to better capture the global information in the graph, and endow the model with advanced capabilities in global information processing through the method of knowledge distillation.

[0094] The present invention combines the graph mask autoencoder with global information to make three key improvements to existing research: First, cluster masking: Compared with the widely used random masking method, the cluster masking method is to gradually guide the model to learn the comprehensive information of the graph, especially the transition from local information in the graph to the global information in the graph. Second, hybrid coding: The model adopts a hybrid coding mechanism by integrating graph neural networks and graph transformers. This mechanism enables the graph mask autoencoder that mainly focuses on local information to transition to a model that can effectively capture local and global graph information. Third, multi-task optimization: By combining the task of predicting global graph attributes, the model breaks through the traditional graph mask autoencoder's single strategy of optimizing node reconstruction. This enables the model to not only ensure the accuracy of local reconstruction, but also emphasize the understanding and fusion of global graph features.

[0095] The graph mask autoencoder model combined with global information proposed in the present invention aims to enable the graph mask autoencoder to enhance the utilization of global information in the graph, which helps to improve the accuracy and wide applicability of the model in real-world graph classification scenarios.

[0096] It should be understood that the embodiments described above are part of the embodiments of the present invention, rather than all of the embodiments. In addition, the technical features in the various embodiments or single embodiments provided by the present invention can be combined with each other arbitrarily to form a feasible technical solution. Such combination is not restricted by the sequence of steps and / or the structural composition mode, but must be based on the ability of ordinary technicians in the field to implement. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such combination of technical solutions does not exist and is not within the protection scope required by the present invention.

[0097] It should be understood that the above description of the preferred embodiment is relatively detailed and cannot be regarded as limiting the scope of patent protection of the present invention. Under the enlightenment of the present invention, ordinary technicians in this field can also make substitutions or modifications without departing from the scope of protection of the claims of the present invention, which all fall within the scope of protection of the present invention. The scope of protection requested for the present invention shall be based on the attached claims.

Claims

1. A generative self-supervised graph pre-training method for a graph mask autoencoder, characterized in that: The following steps are involved: Step 1.1: Input the graph dataset to be classified, the node feature set, the true label of the node, the ratio of the mask, the number of cluster subgraphs, and the number of neural network layers; Step 1.2: Use the clustering masking method that transitions from local to global to mask the node features in the graph. First, start masking from the local nodes in a subgraph, and gradually mask all the nodes in the entire subgraph, so that the graph mask autoencoder starts from learning the local information in the graph and gradually integrates the global information across subgraphs. Step 1.3: Train a graph mask autoencoder using a hybrid encoding mechanism that captures global information.

2. The method for generative self-supervised graph pre-training of a graph mask autoencoder according to claim 1, characterized in that: The specific implementation of the local to global clustering mask method includes the following sub-steps: Step 2.1: Graph data preprocessing; The graph data is divided into several non-intersecting cluster subgraphs using the METIS cluster subgraph algorithm, and each cluster subgraph does not intersect with each other; Step 2.2: Cluster the node masks within the subgraph; At the beginning of training, node features of a part of the node set inside each cluster subgraph are randomly selected for masking; Step 2.3: Transition from within the cluster subgraph to between cluster subgraph masks; As training progresses, the features of all nodes in some cluster subgraphs will be completely masked, while the features of some nodes in other remaining cluster subgraphs will continue to be masked; Step 2.4: Node mask across cluster subgraphs; In the final stage of training, all node features in the selected cluster subgraph will be completely masked out, and the learning goal of the model becomes to predict the node features in the completely masked subgraph based on the node information in other remaining subgraphs.

3. The method for generative self-supervised graph pre-training of a graph mask autoencoder according to claim 1, characterized in that: The graph mask autoencoder in step 2 includes a graph neural network encoding module, a Transformer encoding module and a decoder module; the graph neural network encoding module includes N graph isomorphic networks arranged in series, which are used to capture the local, node neighborhood level information in the fusion input graph; the Transformer encoding module includes N self-attention mechanism layers arranged in series, which increases the ability of the graph mask autoencoder to consider global graph information by increasing long-distance interactions in the entire graph; the decoder module includes a graph isomorphic network or a multi-layer perceptron, which is used to map the potential node embeddings generated by the encoder into reconstructed node features; wherein N is a preset value.

4. The method for generative self-supervised graph pre-training of a graph mask autoencoder according to claim 3, characterized in that: The encoding process of the graph neural network encoder is as follows: Where A represents the adjacency matrix of the input graph, H (l-1) represents the node embedding after passing through the l-1th layer of graph neural network encoder, Represents the,th layer of graph neural network encoder.

5. The method for generative self-supervised graph pre-training of a graph mask autoencoder according to claim 3, characterized in that: The encoding process of the graph Transformer is as follows: Among them, H (l) is the potential node embedding after the ,th layer Transformer encoder, as well as Figure 1 shows the learnable parameter matrix of the self-attention layer in Transformer, σ represents the softmax function before the final output, and d represents the dimension of the potential node embedding.

6. The method for generative self-supervised graph pre-training of a graph mask autoencoder according to claim 3, characterized in that: In the decoder part, the decoder f maps the latent node embedding H back to the node feature X; after the graph mask encoder outputs the latent representation of all nodes, the masked nodes are re-masked before being input to the decoder; in, is node v i The node latent representation input to the decoder, h m is the node representation after re-masking, and V represents the set of masked nodes; After re-masking, the node representations of the graph are input into the decoder to reconstruct the input features of the nodes.

7. The method for generative self-supervised graph pre-training of a graph mask autoencoder according to any one of claims 1 to 6, characterized in that: In step 3, a multi-task optimization strategy is used to train the image mask autoencoder; the multi-task optimization loss function L used is: L = L1 + λL2; Among them, V m represents the set of masked nodes, x i represents the original feature of node i, h i represents the node feature of node i restored by the decoder; λ is the weight value; ZT represents the graph-level representation obtained from a large pre-trained teacher model, POOL represents the graph pooling operation, and f proj It represents the projection processing function. Represents node v i Node representations obtained from the student model.

8. A graph mask autoencoder generative self-supervised graph pre-training device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the generative self-supervised graph pre-training method of the graph mask autoencoder as described in any one of claims 1 to 7 is implemented.

9. A software product comprising a computer program, characterized in that When the computer program is executed by a processor, the method for generative self-supervised graph pre-training of a graph mask autoencoder as described in any one of claims 1 to 7 is implemented.

Images