Method for revealing interactive learning ability of different geometric structures through two-way distillation diagram data

Through the bidirectional distillation method combined with knowledge distillation of manifold domain and Euclidean domain, the problem of graph neural network learning diversified graph structures in hyperbolic space is solved, the knowledge fusion across geometric models is realized, and the representation ability of students' models is improved and classification accuracy solve the problem of insufficient representation ability of students' models in the prior art.

CN118982046APending Publication Date: 2024-11-19NANJING NORMAL UNIVERSITY
View PDF 0 Cites 4 Cited by

Patent Information

Application Number
CN202411068907.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-06
Publication Date
2024-11-19

AI Technical Summary

Technical Problem

Existing graph neural network knowledge distillation technology is difficult to effectively learn diverse graph structures from hyperbolic space, and lacks mutual learning mechanisms to extract useful information across geometric models, resulting in insufficient representation ability of students' model and the ability of structural complexity. Scalability challenges are difficult to deploy in practical applications.

Method used

Using the bidirectional distillation method, a two-way distillation framework is established by constructing a positive knowledge distillation module of a hyperbolic graph neural network based on manifold domain and a negative knowledge distillation module of a Euclidean domain-based graph neural network, and using positive and negative directions. The combination of knowledge distillation loss functions eliminates the influence of geometric specific probability distribution and realizes knowledge fusion across geometric models.

Benefits of technology

A robust feature expression and low-latency lightweight framework model is realized, which improves the discrimination ability of students' models, ensures high classification accuracy and lightweight model design in the field of complex graph data, and avoids interference from geometric specific probability distribution.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118982046A_ABST
    Figure CN118982046A_ABST
Patent Text Reader

Abstract

The invention discloses a novel bidirectional knowledge distillation framework based on graph geometric data, and aims to establish an efficient multi-characteristic graph neural network teacher model for guiding a multi-layer perceptron (MLP) student network to learn. Specifically, HGNN and MLP with hyperbolic characteristics in a manifold domain are combined together through a forward knowledge distillation learning strategy to maintain geometric information of a hyperbolic manifold architecture; the stage can be regarded as training a primary student model by using HGNN; meanwhile, knowledge in the Euclidean domain GNN is distilled to the student MLP by adopting negative knowledge distillation to provide supplementary knowledge, and a high-level student model is trained; through geometric interaction learning of bidirectional distillation, complementary knowledge can be extracted from a multi-domain geometric model, and finally problem reasoning is realized by using a lightweight model; the method has good application potential on complex graph data.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of graph neural network knowledge distillation, and specifically relates to a method for revealing the interactive learning capabilities of different geometric structures of graph data through bidirectional distillation. Background Art

[0002] Background technologies involved: graph neural network, hyperbolic graph neural network, graph fusion learning, and graph neural network knowledge distillation.

[0003] 1) Graph Neural Network (GNN)

[0004] Early research on graph structure analysis attempted to obtain information from the adjacency matrix through factorization and random walks. The current mainstream research direction aims to update the characteristics of nodes by aggregating information about their neighbors. Graph neural networks usually adopt an information passing scheme to iteratively aggregate node neighbor information and learn node embeddings by reading in messages, updating, and reading out functions.

[0005] 2) Hyperbolic Graph Neural Network HGNN

[0006] In recent years, graph representation learning on non-Euclidean manifolds, especially on hyperbolic space, has attracted increasing attention due to its strong ability to capture complex graph structural relationships. The typical geometric property of hyperbolic space is that its volume grows exponentially with the radius, while Euclidean space grows polynomially. In order to learn the hierarchical characteristics of nodes, hyperbolic graph neural networks extend graph neural networks to hyperbolic space by embedding graphs into hyperbolic space with differentiable exponential mappings and logarithmic mappings, which can be viewed as smooth Riemann manifolds. Specifically, the hyperbolic graph neural network derives the core operations of the graph neural network in the hyperbolic surface model of hyperbolic space, converting the input features of the Euclidean space into hyperbolic embeddings. The advantages of hyperbolic representation learning span a large number of research topics and downstream applications.

[0007] 2) Graph Fusion Learning

[0008] Graph geometry interactive learning attempts to study the expressive power of graph neural networks from Euclidean and hyperbolic spaces to reflect complex data structures. Generally, it captures the available information between Euclidean space and hyperbolic space through dual-feature interactive learning and probabilistic assembly mechanisms, or learns multi-layer optimal combinations of Euclidean geometry and hyperbolic geometry to effectively encode various complex graph structures through contrastive learning. Although the above methods are effective, their structural complexity brings scalability challenges. Specifically, the complex mathematical operations involved in hyperbolic space and the need to switch between multiple frameworks pose obstacles to practical deployment in practical applications.

[0009] 3) Knowledge Distillation of Graph Neural Networks

[0010] There are many models that apply the knowledge distillation framework to graph neural networks to achieve better efficiency in different environments. In terms of the knowledge distillation framework, using the multi-layer perceptron model as a student model can combine the fast reasoning ability of the multi-layer perceptron model with the good performance of the graph neural network. Related research and practical applications have shown that the existing graph neural network-multi-layer perceptron knowledge distillation strategy still lacks the ability to learn diverse graph structures from hyperbolic space, and does not adopt a mutual learning mechanism to extract useful information and compensation information across geometric models to enhance the representation ability of the student model. Summary of the invention

[0011] The present invention discloses a method for revealing the interactive learning ability of different geometric structures of graph data through bidirectional distillation, and the method establishes a new graph network knowledge distillation framework. Specifically, the method includes first training a primary multi-layer perceptron through positive knowledge extraction to obtain geometric information based on a specific manifold domain structure. Then, based on the first-level multi-layer perceptron, an advanced multi-layer perceptron is trained by extracting negative knowledge to eliminate the influence of geometric specific probability distribution.

[0012] To achieve the above object, the technical solution of the present invention is as follows:

[0013] The present invention provides a method for revealing the interactive learning capabilities of different geometric structures of graph data through bidirectional distillation, comprising the following steps:

[0014] (1) Obtain graph data;

[0015] (2) Pre-training Hyperbolic Graph Neural Network (HGNN) and Graph Neural Network (GNN) teacher models;

[0016] (3) Construct a forward knowledge distillation module based on the manifold domain hyperbolic graph neural network HGNN;

[0017] (4) Construct a negative knowledge distillation module based on the Euclidean domain graph neural network (GNN);

[0018] (5) Construct a bidirectional distillation framework based on steps (3) and (4).

[0019] Furthermore, in step (3), a forward knowledge distillation loss function L of a hyperbolic graph neural network based on manifold domain is constructed. HGKL :

[0020]

[0021] Among them, V is the total sample, v is a single training sample, represents the soft label predicted by the student model, Soft labels predicted by the hyperbolic graph neural network teacher model representing the manifold domain, LKL is the Kullback-Leibler divergence, which calculates the difference between two probability distributions.

[0022] Furthermore, in step (4), a negative knowledge distillation loss function L based on the graph neural network in the Euclidean domain is constructed. NKD :

[0023]

[0024] in, Soft labels predicted by the graph neural network teacher model representing the Euclidean domain.

[0025] Furthermore, in step (5), a bidirectional knowledge distillation loss function L based on steps (3) and (4) is constructed. final :

[0026] L final =L label +λL HGKL +(1-λ)L NKD

[0027] in,

[0028] V train is the training set, y v represents the true label, L CE is the cross entropy, which calculates the difference between two probability distributions, and λ is a hyperparameter.

[0029] The beneficial effects of the present invention are:

[0030] This paper studies the interactive learning ability of graph geometry through bidirectional knowledge distillation, extracts valuable complementary knowledge from different domain geometric models to achieve robust feature expression, and finally constructs a lightweight framework model with low latency and cross-geometry fusion learning.

[0031] Compared with the positive knowledge distillation in traditional knowledge distillation tasks, this paper uses negative knowledge distillation to capture complementary knowledge to improve the discrimination ability of the student model. This two-way distillation strategy can effectively prevent the student model from being disturbed by geometric specific probability distributions.

[0032] The present invention has achieved good performance in complex graph data field recognition tasks, and ensures high classification accuracy while realizing lightweight model design. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 is a flow chart of the present invention;

[0034] Figure 2 This is a model architecture diagram of the present invention. DETAILED DESCRIPTION

[0035] The present invention will be further explained below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention.

[0036] This paper aims to design a new knowledge distillation method based on graph structure data that can integrate multiple model frameworks, which is fundamentally different from the traditional multi-teacher model and multi-frame fusion model. This paper innovatively starts from the perspective of knowledge distillation and uses a two-way guidance idea to distill useful knowledge of complex different geometric structures into a lightweight student model.

[0037] The method flow is as follows Figure 1 As shown in the figure, the model architecture is as follows Figure 2 As shown, the specific implementation is as follows:

[0038] Step (1): First, obtain data with graph characteristics in the real world, such as literature retrieval data in scientific research, multimedia data in social networks, human body movement bone joint data, industrial supply chain data, etc. It should be noted that any data type that can be described by a graph structure can be used as the input of the present invention. The input of the graph can be modeled as a set of points and edges G (V, E), "points (V)" represent entities, and "edges (E)" represent the relationship between entities; it should be noted that the acquired data needs to be divided into three parts: training set, validation set, and test set. The training set is used to train the model, the validation set is used to adjust the parameters, and the test set is used to verify the model performance.

[0039] Step (2): Pre-training the hyperbolic graph neural network (HGNN) and graph neural network teacher (GNN) models; knowledge distillation is essentially the process of transferring the knowledge possessed by a complex teacher model to a lightweight student model. The present invention requires training a large teacher model with complex expression capabilities based on the graph data in step (1) with the help of training set data, including a hyperbolic graph neural network that can characterize the hyperbolic characteristics of the manifold domain and a graph neural network that characterizes the characteristics of Euclidean domain data. The number of deep network layers in this step can be adaptively adjusted. For HGNN, hyperbolic space can embed low-distortion hierarchical structures because their volume grows exponentially with respect to the radius, that is, as the depth increases, the number of nodes in the tree increases from the origin outward, because the exponential growth characteristics involve complex manifold calculations in hyperbolic space. The main idea of ​​these operations is to use exponential and logarithmic mappings, and tangent space can be used to perform Euclidean transformations.

[0040] Step (3): Construct forward knowledge distillation based on hyperbolic graph neural network (HGNN); In this step, we focus on the geometric structure of hyperbolic manifolds, use the forward knowledge distillation strategy to design a hyperbolic graph network distillation model, and train a primary multi-layer perceptron student model (MLP). In this step, the student model can learn the knowledge characteristics of data on the manifold. The loss function L of the forward knowledge distillation based on the hyperbolic graph neural network constructed is HGKL , specifically:

[0041]

[0042] In the above formula, V is the total sample, v is a single training sample, represents the soft label predicted by the student model, represents the soft labels predicted by the HGNN teacher model, L KL is the Kullback-Leibler divergence, which calculates the difference between two probability distributions. The purpose of step (3) is to utilize the probability information of a specific class, that is, the similarity of the target object to a class label.

[0043] Step (4): Construct negative knowledge distillation based on graph neural network (GNN); In order to better fit the actual complex graph data containing hyperbolic and Euclidean geometric structures, a reverse knowledge distillation scheme is designed in this step to help the student model (MLP) learn the data features in the Euclidean space. The negative knowledge distillation loss function L based on the graph neural network is constructed. NKD , specifically:

[0044]

[0045] In the above formula, Represents the soft labels predicted by the hyperbolic graph neural network teacher model.

[0046] Traditional knowledge distillation methods achieve knowledge transfer by narrowing the gap between the probability distributions of the teacher model and the student model. This idea will conflict when faced with different probability distributions of multiple teacher models. Step (4) is to distill the geometric knowledge in the GNN into the student MLP through negative knowledge distillation. The key point is that the extracted knowledge in the GNN teacher's soft target reflects the degree of dislike (dissimilarity) of the input to a specific class. This strategy can reduce the feature differences caused by different inductive biases of teachers (HGNN and GNN) and provide compensation information for positive knowledge distillation learning.

[0047] Step (5): Construct a bidirectional (BD) distillation framework based on steps (3) and (4).

[0048] The total bidirectional knowledge distillation loss function L constructed by the present invention is final, specifically:

[0049] L final =L label +λL HGKL +(1-λ)L NKD ;

[0050] in,

[0051] V train is the training set, y v represents the true label, L CE is the cross entropy, which calculates the difference between two probability distributions, and λ is a hyperparameter used to balance the second HGKL term and the third NKD part.

[0052] In the bidirectional knowledge distillation framework, the second item transfers the hyperbolic GNN structure knowledge to the student MLP through HGKL, and the third item distills the knowledge of the Euclidean GNN teacher to the student model through NKD. Combining PKD and NKD, the performance gap between the teacher and student models is fully considered when deciding which knowledge to transfer. This operation eliminates the difference in cross-geometric distribution. This enables knowledge fusion across geometric models to fully represent complex graph data.

[0053] Step (6) Lightweight deployment and verification of student model.

[0054] Based on steps (1) to (5), the final multi-layer perceptron (MLP) student model with lightweight characteristics can be designed. This model can characterize the different geometric characteristics of the manifold domain and the Euclidean domain, and can learn more comprehensive knowledge information. In theory, this MLP model has the advantages of both speed and performance and can be quickly deployed to end devices.

[0055] In order to prove the good effect of the present invention in processing graph data, this embodiment tests and compares the present invention on a test set to make the effect more intuitive.

[0056] The test set uses seven general graph datasets: Cora, Citeseer, Pubmed, A-computer, A-photo, Disease, Arxiv. The comparison methods include various leading graph knowledge distillation methods and teacher model methods: GCN, HGCN, MLP, GLNN, NOSMOG, HGMK, GGIKD. As shown in Table 1, the method proposed in this paper performs outstandingly in classification effect and deployment time.

[0057] Table 1 Experimental verification results

[0058] method Cora Citeseer A-computer A-photo Pubmed Disease Arxiv GCN 81.77 70.58 85.52 92.03 78.03 80.63 70.68 HGCN 79.27 68.81 83.81 92.68 76.77 84.76 68.96 MLP 60.77 61.03 63.01 78.65 72.83 50.12 55.10 GLNN 82.10 71.42 85.56 92.70 79.70 80.83 59.60 NOSMOG 82.19 71.64 85.17 93.41 79.68 73.19 58.10 HGMK 80.34 72.02 83.92 92.85 79.32 83.62 59.01 GDJ 82.27 70.38 84.64 91.87 79.22 84.40 57.80 ours 82.81 72.32 85.69 93.46 80.61 85.55 60.45

[0059] This model can effectively learn node representations without sacrificing performance. In addition, the efficiency of this model in accelerating inference time is further analyzed. Specifically, experiments are conducted on 10 nodes of the test set randomly selected in each dataset. The inference speed of the student model is much faster than that of the teacher model. Compared with GCN, MLP eliminates the neighbor acquisition delay, and the inference speed of MLP is 15.34 times faster than GCN on large-scale datasets. Compared with HGCN, MLP has no hyperbolic operations, which makes the average inference speed of MLP 4.77 times faster than HGCN on 6 benchmark datasets.

[0060] It should be noted that the above content only illustrates the technical idea of ​​the present invention and cannot be used to limit the protection scope of the present invention. For ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications all fall within the protection scope of the claims of the present invention.

Claims

1. A method for revealing the interactive learning ability of different geometric structures of graph data through bidirectional distillation, characterized in that: The following steps are involved: (1) Obtain graph data; (2) Pre-training hyperbolic graph neural network and graph neural network teacher models; (3) Construct a forward knowledge distillation module based on the manifold domain hyperbolic graph neural network; (4) Construct a negative knowledge distillation module based on the graph neural network in the Euclidean domain; (5) Construct a bidirectional distillation framework based on steps (3) and (4).

2. The method for revealing the interactive learning ability of different geometric structures of graph data through bidirectional distillation according to claim 1, characterized in that: In step (3), construct the forward knowledge distillation loss function L of the hyperbolic graph neural network based on the manifold domain HGKL : Among them, V is the total sample, v is a single training sample, represents the soft label predicted by the student model, represents the soft labels predicted by the manifold domain hyperbolic graph neural network teacher model, L KL is the Kullback-Leibler divergence, which calculates the difference between two probability distributions.

3. The method for revealing the interactive learning ability of different geometric structures of graph data through bidirectional distillation according to claim 2, characterized in that: In step (4), a negative knowledge distillation loss function L is constructed based on the Euclidean domain graph neural network. NKD : in, Represents the soft labels predicted by the Euclidean graph neural network teacher model.

4. The method for revealing the interactive learning ability of different geometric structures of graph data through bidirectional distillation according to claim 3, characterized in that: In step (5), a bidirectional knowledge distillation loss function L is constructed based on steps (3) and (4). final : L final =L label +L HGKL +(1-λ)L NKD in, V train is the training set, y v represents the true label, L CE is the cross entropy, which calculates the difference between two probability distributions, and λ is a hyperparameter.

Citation Information

Cited By

  • Cross-domain small sample object image classification method and device for intelligent terminal

    CN122116010A

  • Cross-domain small sample object image classification method and device for intelligent terminal

    CN122116010B

  • Knowledge graph embedding hyperbolic mutual distillation method fusing topology and semantic information

    CN122489745A

  • Knowledge graph embedding hyperbolic mutual distillation method fusing topological and semantic information

    CN122489745B