Graph contrast learning network classification method based on extensible Transform module
By introducing the extensible Transformer module and Pivot-Enhanced Loss loss function in graph comparison learning network, the problems of information mining and global structure capture in heterogeneous graph processing are solved, and higher classification accuracy and lower computational complexity are achieved.
Patent Information
- Application Number
- CN202510016973.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-01-06
- Publication Date
- 2025-05-06
AI Technical Summary
The prior art is difficult to fully explore and utilize the graph structure and attribute information when dealing with heterogeneous graphs, resulting in poor performance on heterogeneous graphs and difficulty in capturing the global structure information of the graphs, and there are problems such as oversmoothing, overcompression and long-distance dependence.
The graph comparison learning network based on the extensible Transformer module is adopted. By introducing the self-attention mechanism and long-distance dependency capabilities of Transformer, combined with the Pivot-Enhanced Loss loss function, the node correlation degree in the graph neural network dataset is iteratively learned, and the Transformer is approximated through random features and reparameterization methods to reduce the time and space complexity.
The classification accuracy of homogeneous graphs and heterogeneous graphs is improved, the time and space complexity of Transformer is reduced, making it more advantageous on large-scale graphs, and significantly improving the expression ability and robustness of the model.
Smart Images
Figure CN119939342A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of graph neural network contrastive learning, and in particular to a classification method for a graph contrastive learning network based on an extensible Transformer module. Background Art
[0002] Graph data structure is a natural and common data structure. Nodes are used to represent different entities, and their corresponding edges represent mutual relationships. It is commonly found in social networks, biological molecular structures, transportation networks and other fields. Compared with traditional data such as image data, the challenge of graph data lies in the irregularity of the graph topology and the dynamic changes of its structure. These problems make the analysis and processing of graph data significantly more difficult. Currently, mainstream GNN can solve the above problems by operating on the topological structure of the graph and using the adjacency relationship between nodes to learn the feature representation of nodes, effectively capturing the local structural information of the graph. Among them, graph convolutional neural networks have shown very outstanding performance. Although many methods can use the characteristics related to the graph topology structure to perform effective information aggregation and feature extraction, they have achieved remarkable performance in a variety of graph-related tasks.
[0003] However, due to the lack of labels in the real world, GNNs based on supervised learning are often greatly restricted. Therefore, unsupervised learning is also a hot topic of research. It automatically discovers the characteristics of the data itself without relying on external label information. In unsupervised learning, contrastive learning is the learning paradigm that can best mine the characteristics of the data itself. Its basic idea is that similar data often have more similar features, and dissimilar data often have more different features. More precisely, contrastive learning maximizes the similarity between similar samples and minimizes the similarity between dissimilar samples by designing a contrastive loss function. Contrastive learning can enable the model to effectively learn the key features that distinguish nodes without the support of labeled data, so that various downstream tasks related to graphs can perform better with improved performance.
[0004] In graph contrastive learning, data augmentation techniques such as node or edge perturbation, feature masking, subgraph sampling, etc., not only improve the robustness of the model to noise, but also generate enhanced views to help the graph contrastive learning model understand the essential characteristics of the data from different perspectives. This learning paradigm can learn more about the intrinsic characteristics of the dataset itself, and significantly improve the generalization ability of the model. The above are the basic applications of contrastive learning in graph datasets. The technology proposed in this paper also focuses on some extended applications on graph datasets, especially in dealing with heterogeneous graphs, because heterogeneous graphs can represent more complex data structures and rich semantic information. Heterogeneous graphs often have greater local differences, making it more difficult to understand the features of their related nodes, which brings new challenges and opportunities to graph neural networks. How to fully mine and utilize the structural and attribute information in heterogeneous graphs is another hot topic for researchers. The model should not only be able to handle different types of nodes and edges, but also be able to improve the model's expressiveness and robustness. However, the current contrastive learning is not very expressive on heterogeneous graphs. How to design a better learning architecture to learn graph data has become a major research hotspot in graph contrastive learning.
[0005] In summary, although there are already many prediction methods for contrastive learning in the task of homogeneous graph node classification, such as representation learning methods based on graph embedding, GCN or methods based on adaptive sampling and aggregation, these methods each have some limitations, such as difficulty in capturing the global structural information of the graph, over-smoothing problems, over-compression problems, long-distance dependencies, heterogeneity and other problems. Summary of the invention
[0006] The purpose of the present invention is to improve the classification accuracy of homogeneous graphs and heterogeneous graphs while reducing the time and space complexity of transform, and improve the efficiency of classification tasks on large-scale graphs, and to provide a classification method based on a graph contrast learning network with an extensible Transformer module.
[0007] The purpose of the present invention can be achieved by the following technical solutions:
[0008] A classification method based on a graph contrastive learning network with an extensible Transformer module, characterized in that the method comprises the following steps:
[0009] S1. Constructing a graph contrastive learning framework based on an extensible Transformer module, wherein the framework includes an extensible Transformer module and a graph contrastive learning network;
[0010] S2. Keep the parameters of the scalable Transformer module unchanged, send the original features and structure encoding into the scalable Transformer module to obtain two enhanced graphs, input the enhanced graphs into the graph contrastive learning network, pass the enhanced graphs through the graph convolutional neural network and multilayer perceptron of the graph contrastive learning network to obtain the feature representation of the two nodes, calculate the graph contrastive learning loss function, and train the graph contrastive learning network;
[0011] S3, the parameters of the control graph comparison learning network remain unchanged, the scalable Transformer module is constructed by using random features and a reparameterized approximation method, a pivot enhancement objective function is calculated based on the feature representation output by the scalable Transformer module, and the scalable Transformer module is trained;
[0012] S4. Repeat S2 and S3 for iterative training to obtain the trained graph contrast learning network model and Transformer model. Input the actual graph data into the trained graph contrast learning network model and Transformer model respectively to obtain graph contrast learning features and Transformer features. Add the graph contrast learning features and Transformer features and input the sum obtained into the classification task model to obtain the classification result.
[0013] Furthermore, the feature representation of the output of the scalable Transformer module is:
[0014]
[0015] Among them, Z represents the input feature of the scalable Transformer module, Z ′ represents the feature representation of the output of the scalable Transformer module, β is a hyperparameter for residual connections, D represents the degree matrix of the graph, V represents the value vector, K represents the key vector, Q represents the query vector, represents the normalized query vector, represents the normalized key vector, and N represents the dimension of the vector.
[0016] Furthermore, the degree matrix D of the graph is:
[0017]
[0018] Furthermore, the normalized key vector for:
[0019]
[0020] Furthermore, the normalized query vector for:
[0021]
[0022] Furthermore, the specific steps of calculating the pivot enhancement objective function based on the feature representation output by the scalable Transformer module are:
[0023] For every homogeneous and heterogeneous edge e i,j Calculate the homogeneous sentinel enhancement loss and the heterogeneous sentinel enhancement loss, calculate the homogeneous loss function and the heterogeneous loss function, and the sum of the homogeneous loss function and the heterogeneous loss function is the pivot enhancement objective function.
[0024] Furthermore, the homogeneous sentinel enhancement loss and the heterogeneous sentinel enhancement loss are:
[0025]
[0026] Among them, e i,j represents the edge between node i and node j, Represents two randomly sampled nodes v i′ and node v j′ The similarity of Indicates that the edge i,j The similarity between the connected node pair node i and node j, θ represents the cosine similarity metric function, Z i ′ and Z j ′ respectively represent the node feature representation of node i and node j after the scalable Transformer module, represents the homogeneous sentinel enhancement loss, represents heterogeneous sentinel enhancement loss.
[0027] Furthermore, the homogeneous loss function and heterogeneous loss function are:
[0028]
[0029] in, represents the expected value, L homo represents the homogeneous loss function, L hete represents a heterogeneous loss function.
[0030] Furthermore, the graph contrast learning loss function is:
[0031]
[0032] Where V represents the vertex set, n = |V|, N i Represents node v i The set of neighbors of a section, v j For its positive sample pair, v k For its negative sample pair, z i ,zj It represents the feature representation of nodes i and j in the latent space after multi-layer perceptron MLP learning, that is, the feature representation of two nodes.
[0033] Furthermore, the specific steps for obtaining two enhanced images are:
[0034] The feature representation output by the scalable Transformer module is used to model the graph adjacency matrix and generate two enhanced graphs, namely the homogeneous enhanced view and the heterogeneous enhanced view.
[0035] Compared with the prior art, the present invention has the following beneficial effects:
[0036] This paper introduces the transformer's self-attention mechanism and the ability to capture long-distance dependencies into graph contrast learning, iteratively learns the correlation between different nodes in the graph neural network dataset, and proposes the Pivot-Enhanced Loss as a loss function to guide the learning of this module. By approximating the transformer, it reduces its time and space complexity, making it more advantageous on large-scale graphs. The pivot-enhanced objective function enables the framework to achieve better accuracy than the baseline method on both homogeneous and heterogeneous graphs. BRIEF DESCRIPTION OF THE DRAWINGS
[0037] Figure 1 It is a schematic diagram of the framework structure of the present invention. DETAILED DESCRIPTION
[0038] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.
[0039] In order to solve the problems existing in the prior art, a new graph contrastive learning framework (ST-GCL) based on an extensible Transformer module is proposed. Based on the contrastive learning paradigm, the present invention introduces an extensible transformer module to capture the similarity between different node pairs in the graph node classification data set. By capturing the similarity of all node pairs, the importance of neighbor nodes of different orders to the current central node can be judged, thereby obtaining better expressiveness in heterogeneous graphs. After each training is completed, the similarity between node pairs will be dynamically adjusted according to the objective function. The method proposed by the present invention comprises the following steps:
[0040] S1. Constructing a graph contrastive learning framework based on an extensible Transformer module, wherein the framework includes an extensible Transformer module and a graph contrastive learning network;
[0041] S2. Keep the parameters of the scalable Transformer module unchanged, send the original features and structure encoding into the scalable Transformer module to obtain two enhanced graphs, input the enhanced graphs into the graph contrastive learning network, pass the enhanced graphs through the graph convolutional neural network and multilayer perceptron of the graph contrastive learning network to obtain the feature representation of the two nodes, calculate the graph contrastive learning loss function, and train the graph contrastive learning network;
[0042] S3, the parameters of the control graph comparison learning network remain unchanged, the scalable Transformer module is constructed by using random features and a reparameterized approximation method, a pivot enhancement objective function is calculated based on the feature representation output by the scalable Transformer module, and the scalable Transformer module is trained;
[0043] S4. Repeat S2 and S3 for iterative training to obtain the trained graph contrast learning network model and Transformer model. Input the actual graph data into the trained graph contrast learning network model and Transformer model respectively to obtain graph contrast learning features and Transformer features. Add the graph contrast learning features and Transformer features and input the sum obtained into the classification task model to obtain the classification result.
[0044] The goal of this invention is to introduce the transformer's self-attention mechanism and ability to solve long-distance dependencies into graph contrast learning to solve heterogeneous graph problems. Figure 1 As shown in the figure, unlike most graph neural networks with graph contrast learning, ST-GCL does not use two enhanced views generated by the original image, but an enhanced view with full-node similarity measurement after being modified by the transformer module. At the same time, the encoder used by the graph contrast learning module is still GCN, and the objective function used is the InfoNCE loss function (contrast learning objective function). Compared with previous graph contrast learning, the improvements of the present invention are:
[0045] 1) Introduce the transformer's self-attention mechanism and ability to capture long-distance dependencies into graph contrastive learning, iteratively learn the correlation between different nodes in the graph neural network dataset, and propose Pivot-Enhanced Loss as the loss function to guide the learning of this module
[0046] 2) Propose a scalable transformer, which approximates the transformer by using methods such as Random Feature to reduce its time and space complexity, making it more advantageous on large-scale graphs.
[0047] 3) We conduct extensive experiments on 5 homogeneous graph datasets and 3 heterogeneous graph datasets to evaluate the proposed method. The results show that the proposed model outperforms most baseline methods on all datasets.
[0048] In general, the invention of the present invention utilizes the self-attention mechanism of transformer and its ability to capture long-distance dependencies, and develops a graph contrastive learning model based on an extensible Transformer module, and proposes a Pivot-Enhanced objective function to guide the learning of the module, which is an innovation that has not been seen before in graph contrastive learning. The model of the present invention achieves better accuracy than the baseline method on both homogeneous and heterogeneous graphs, further proving the superiority of the proposed method.
[0049] like Figure 1 As shown in the proposed model ST-GCL, represents the graph set after graph enhancement generated by the scalable Transformer module method, f(·), g(·) represent different neural networks, and GCN and MLP are used in the model respectively. i ,z i They represent the feature representation of node i respectively.
[0050] (1) Graph Contrastive Learning Network
[0051] For graph contrast learning, an original graph dataset is input, and two different views are generated through node perturbation, edge perturbation, feature masking, and graph reconstruction. The two generated views are sent to the graph convolutional neural network (GCN) to obtain the feature representation of each node in the graph dataset, and then the learned features are sent to the latent space for contrast learning. In graph contrast learning, the core encoder uses the graph convolutional neural network. The most common form of GCN is the spectral method based on the normalized Laplacian matrix. The main principles of the graph convolutional neural network are as follows: Among them, H (l) is the node feature matrix of the lth layer, is the adjacency matrix A with self-connection added, I is the identity matrix, yes The degree matrix, W l is the weight matrix of the layer, and σ is a nonlinear activation function, such as ReLU.
[0052] On the basis of graph contrastive learning, this paper adopts another transformer-based module to assist graph contrastive learning to help it perform better on heterogeneous graphs, and proposes a Pivot-Enhanced loss function to guide the learning of this module. As the scale of graph datasets continues to increase, the time and space consumption brought by transformers is nonlinear, which is not conducive to its scalability. Therefore, the transformer module is approximated by methods such as random feature and gumbel-softmax. The proposed model adopts the method:
[0053] Q=f Q (Z), K=f K (Z), V=f V (Z)
[0054]
[0055] Among them, f Q ,f K ,f V is a linear forward feedback layer, ‖·‖ represents the Fourier modulus, 1 is an N-dimensional full column vector, the diag operation converts the N-dimensional column vector into an N×N diagonal matrix, and β is a hyperparameter for residual connection.
[0056] (2) Model training and prediction
[0057] The first is the graph contrastive learning module. The contrastive learning paradigm generally uses InfoNCE as the objective function, and its formula is as follows:
[0058]
[0059] Where V represents the vertex set, n = |V|, N i Represents node v i The set of neighbors of a section, v j For its positive sample pair, v k is its negative sample pair. i ,z j Represents the feature representation of nodes i, j in the latent space after MLP learning.
[0060] Secondly, the proposed Pivot-Enhanced loss function guides the learning of the transformer module. Since contrastive learning is unsupervised learning during training, it is challenging to design a loss function to guide the learning of this module in an unsupervised environment. A pivot-enhanced loss function is designed to supervise the learning of this edge heterogeneity discrimination module. The goal is to distinguish homogeneous edges (connecting similar nodes) and heterogeneous edges (connecting dissimilar nodes). The main challenge lies in finding the boundary between "similar" and "dissimilar". To this end, the present invention proposes to use randomly sampled node pairs as the "pivot" for similarity measurement. Specifically, for each homogeneous and heterogeneous edge, e i,j ∈ε, the homogeneous sentinel enhancement loss and heterogeneous sentinel enhancement loss are recorded as:
[0061]
[0062] Where [x] + =max(x,0). The corresponding loss function is and The expected value of , the corresponding homogeneous and heterogeneous loss functions are:
[0063]
[0064] Then L t =L home +L hete .
[0065] During the training process of ST-GCL, the model parameters related to the transformer are kept unchanged, and the graph contrast learning module is trained first. Then, the graph contrast learning module is kept unchanged and the transformer module is trained. Thus, the two are trained crosswise. After the training is completed, the feature representation of each node output by the model is the sum of the graph contrast learning and transformer modules, and the obtained feature representation is used for downstream tasks. The downstream can be any neural network that can perform classification tasks such as predicting the properties of protein molecules. For example, in the field of advertising, users can be classified to predict which users are interested in specific advertisements, or in the field of malicious behavior detection, it can determine whether there are potential malicious actors.
[0066] The preferred specific embodiments of the present invention are described in detail above. It should be understood that a person skilled in the art can make many modifications and changes based on the concept of the present invention without creative work. Therefore, any technical solution that can be obtained by a person skilled in the art through logical analysis, reasoning or limited experiments based on the concept of the present invention on the basis of the prior art should be within the scope of protection determined by the claims.
Claims
1. A classification method based on a graph contrastive learning network with an extensible Transformer module, characterized in that: The method comprises the following steps: S1. Constructing a graph contrastive learning framework based on an extensible Transformer module, wherein the framework includes an extensible Transformer module and a graph contrastive learning network; S2. Keep the parameters of the scalable Transformer module unchanged, send the original features and structure encoding into the scalable Transformer module to obtain two enhanced graphs, input the enhanced graphs into the graph contrastive learning network, pass the enhanced graphs through the graph convolutional neural network and multilayer perceptron of the graph contrastive learning network to obtain the feature representation of the two nodes, calculate the graph contrastive learning loss function, and train the graph contrastive learning network; S3, the parameters of the control graph comparison learning network remain unchanged, the scalable Transformer module is constructed by using random features and a reparameterized approximation method, a pivot enhancement objective function is calculated based on the feature representation output by the scalable Transformer module, and the scalable Transformer module is trained; S4. Repeat S2 and S3 for iterative training to obtain the trained graph contrast learning network model and Transformer model. Input the actual graph data into the trained graph contrast learning network model and Transformer model respectively to obtain graph contrast learning features and Transformer features. Add the graph contrast learning features and Transformer features and input the sum obtained into the classification task model to obtain the classification result.
2. According to claim 1, a classification method based on a graph contrastive learning network with an extensible Transformer module is characterized in that: The feature representation of the output of the scalable Transformer module is: Where Z represents the input features of the scalable Transformer module, Z′ represents the feature representation of the output of the scalable Transformer module, β is a hyperparameter for residual connections, D represents the degree matrix of the graph, V represents the value vector, K represents the key vector, and Q represents the query vector. represents the normalized query vector, represents the normalized key vector, and N represents the dimension of the vector.
3. The classification method of a graph contrastive learning network based on an extensible Transformer module according to claim 2, characterized in that: The degree matrix D of the graph is:
4. The classification method of a graph contrastive learning network based on an extensible Transformer module according to claim 3, characterized in that: Normalized key vector for:
5. The classification method of a graph contrastive learning network based on an extensible Transformer module according to claim 4, characterized in that: Normalized query vector for:
6. The classification method of a graph contrastive learning network based on an extensible Transformer module according to claim 1, characterized in that: The specific steps of calculating the pivot enhancement objective function based on the feature representation output by the scalable Transformer module are: For every homogeneous and heterogeneous edge e i,j Calculate the homogeneous sentinel enhancement loss and the heterogeneous sentinel enhancement loss, calculate the homogeneous loss function and the heterogeneous loss function, and the sum of the homogeneous loss function and the heterogeneous loss function is the pivot enhancement objective function.
7. The classification method of a graph contrastive learning network based on an extensible Transformer module according to claim 1, characterized in that: The homogeneous sentinel enhancement loss and heterogeneous sentinel enhancement loss are: Among them, e i,j represents the edge between node i and node j, Represents two randomly sampled nodes v i′ and node v j′ The similarity of Indicates that the edge i,j The similarity between the connected node pair node i and node j, θ represents the cosine similarity metric function, Z′ i and Z′ j They represent the node feature representation of node i and node j after the scalable Transformer module, represents the homogeneous sentinel enhancement loss, represents heterogeneous sentinel enhancement loss.
8. The classification method of a graph contrastive learning network based on an extensible Transformer module according to claim 1, characterized in that: The homogeneous loss function and heterogeneous loss function are: in, represents the expected value, L homo represents the homogeneous loss function, L hete represents a heterogeneous loss function.
9. The classification method of a graph contrastive learning network based on an extensible Transformer module according to claim 1, characterized in that: The loss function for graph contrast learning is: Where V represents the vertex set, n = |V|, N i Represents node v i The set of neighbors of a section, v j For its positive sample pair, v k For its negative sample pair, z i ,z j It represents the feature representation of nodes i and j in the latent space after multi-layer perceptron MLP learning, that is, the feature representation of two nodes.
10. The classification method of a graph contrastive learning network based on an extensible Transformer module according to claim 1, characterized in that: The specific steps to obtain two enhanced images are: The feature representation output by the scalable Transformer module is used to model the graph adjacency matrix and generate two enhanced graphs, namely the homogeneous enhanced view and the heterogeneous enhanced view.