Graph neural network distillation method, system, equipment and medium

Abstract

The invention discloses a graph neural network distillation method, system and device and a medium, and relates to the technical field of edge deployment, and the method comprises the steps: introducing a box-type feature aggregation feature extraction module between a graph neural network teacher model and a student model, extracting and aggregating graph structure information of multiple scales in the graph neural network model through a box type feature aggregation feature extraction module to obtain box aggregation loss, training the student model based on box aggregation distillation total loss formed by the box aggregation loss and predicted loss of the student model, and using the trained student model for a specified task; according to the method, the feature difference of the middle layer is reduced to the maximum extent, so that successful transfer of multi-scale knowledge and deep structure knowledge from a teacher model is promoted.

Description

Technical Field

[0001] The present invention relates to the field of edge deployment technology, and in particular to a graph neural network distillation method, system, device and medium. Background Art

[0002] Edge computing has attracted much attention due to its ability to perform complex computing tasks close to the data source. Edge computing reduces the burden on central servers and minimizes latency, making it particularly important for the Internet of Things and real-time applications.

[0003] To this end, various edge deployment techniques have been proposed. Graph Knowledge Distillation (GKD) is an efficient model compression technique that aims to effectively transfer knowledge from a complex, high-parameter teacher graph neural network (GNN) model to a simple, low-parameter student model. This process enhances the generalization ability of the student model, enabling it to reach or even exceed the performance of the teacher model. Hinton et al. first proposed the concept of knowledge distillation in 2015. They promoted knowledge transfer by regularizing the soft logarithmic output between the teacher and student models. On this basis, FitNet and Adversarial Training Distillation (AT) further improved the initial distillation method. FitNet adjusts the hidden features of the teacher and student models by adding a fully connected layer in the middle layer, making the learning of the student model more effective. At the same time, the AT method enhances the generalization ability of the student model through adversarial training. Yang et al. proposed a pairwise local structure preservation (LSP) module, marking the first application of knowledge distillation to graph neural networks (GNNs). This method captures the local topological structure of the network through the local structure preservation module and minimizes the distribution of knowledge transfer distances.

[0004] Currently popular distillation techniques mainly focus on adjusting the output layers and intermediate feature representations of the teacher model and the student model, often ignoring the rich information inherent in the graph structure. The graph structure not only includes the connections between nodes and edges, but also reflects the local and global topological characteristics of two-dimensional and three-dimensional space. Thoroughly exploring the complex information in the graph structure is crucial for effective knowledge transfer from the teacher model to the student model.

[0005] However, traditional graph distillation methods are mainly based on local structural features, usually using node neighborhood information, and rarely pay attention to the distillation of deep network structural information. Although significant progress has been made in GKD research, many methods focus on local neighborhood topology, often fail to fully consider the diversity and complexity of node features, and ignore the transmission of deep network information and high-level features, which easily limits the performance of student models in downstream tasks, resulting in the inability to effectively transmit deep collaborative information in actual applications, increasing the risk level when applying the model for specified tasks. Summary of the invention

[0006] In view of the shortcomings of the prior art that mainly focuses on local neighborhood topology and ignores the transmission of deep network information and high-level features, which easily limits the performance of student models in downstream tasks, the present invention proposes a graph neural network distillation method, system, device and medium, and introduces a box feature aggregator BFA, which can segment the network at different scales and aggregate the features in each box, thereby improving the feature representation of the central node, thereby solving the problems existing in the prior art.

[0007] A graph neural network distillation method, comprising the following steps:

[0008] The graph neural network model of the knowledge to be distilled is used as the teacher model; the graph neural network model of the knowledge to be received is used as the student model;

[0009] A box-feature aggregation feature extraction module is introduced between the teacher model and the student model to extract and aggregate graph structure information of multiple scales in the graph neural network model to obtain a box aggregation loss; the student model is trained based on the total box aggregation distillation loss formed by the box aggregation loss and the prediction loss of the student model, and the trained student model is applied to the specified task; wherein the generation process of the box aggregation loss specifically includes the following steps:

[0010] The quality of each node in the graph neural network structure is calculated step by step, the node with the highest quality in each round is selected as the central node, and all nodes with a radius of infection distance that can cover the entire graph structure are aggregated around the central node to obtain a box;

[0011] For the boxes obtained in each round of calculation, the output features of the intermediate layers of the teacher model and the student model are updated respectively; the intermediate layers of the teacher model and the student model are aligned, and the similarity of the output features of the two intermediate layers after the update is calculated to obtain the box aggregation loss.

[0012] Furthermore, the stepwise calculation of the quality of each node in the graph neural network structure specifically includes the following steps:

[0013] Initialize the graph Each The initial mass m i and candidate weight c i Set to 1, for each candidate central node i, the infection radius r b Starting from 1, gradually infect its neighbor nodes with a step size of 1. If a neighbor node has not been visited, mark it as visited, add it to the new infection set, and update the quality m i ; After multiple updates, until the infection radius r b Until the entire graph structure is covered, the mass m of each node after infection iThe calculation formula is expressed as:

[0014]

[0015] Where N(i,r b ) represents the infection radius r of node i b The set of all nodes in w ij Represents the contribution of node j to the quality of node i.

[0016] Furthermore, the node with the highest quality in each round is selected as the central node, wherein the central node h i It is expressed as:

[0017]

[0018] Among them, node i must satisfy condition c i =1, which indicates that the node is a candidate node for the central node; argmax indicates the maximum value parameter.

[0019] Furthermore, the aggregation takes the central node as the center and all nodes with the infection distance that can cover the entire graph structure as the radius to obtain a box; the generation process is expressed as:

[0020]

[0021] in Indicates that the central node h i The center is r and the radius is b The set of all nodes is represented as a box; the symbol \ represents the set difference operation; the function dist(h i ,j) represents the central node h i The shortest path distance between node j and node j; the shortest path distance is calculated using the breadth-first search BFS algorithm and is expressed as:

[0022]

[0023] in Indicates that from the central node h i The set of all paths to node j, P is a specific path, (u,v) represents an edge in path P, and w(u,v) is the weight of edge (u,v).

[0024] Furthermore, for each box obtained by calculation, the output features of the intermediate layers of the teacher model and the student model are updated respectively, which specifically includes the following steps:

[0025] Perform feature update on the intermediate output feature map of layer l, expressed as in and ε represent the set of nodes and edges respectively, and the matrix represents the node feature matrix, where N is the number of nodes, F l is the dimension of the intermediate features of layer l;

[0026] Box The node feature vector is expressed as Then the feature update of the center node of the i-th box is expressed as:

[0027]

[0028] in is the updated central node feature, represents the feature aggregation function used to integrate the features within the selected box, means selecting the top p% of boxes after sorting in descending order by the number of nodes.

[0029] Furthermore, the feature aggregation function for integrating the features in the selected box is expressed as:

[0030]

[0031] Further, the process of generating the total loss of the box aggregate distillation comprises the following steps:

[0032] Since the input graphs of the teacher model and the student model are the same, for a given intermediate feature graph Calculate the box features of the teacher model and the student model, denoted as BF t and BF s ;

[0033] For each central node h i , the similarity between the box features SIM i The calculation expression is:

[0034]

[0035] in represents the KL divergence between the feature distributions of the two models; by calculating the similarity between all box features, the box aggregation loss is obtained, which is expressed as:

[0036]

[0037] The prediction loss of the student model uses binary cross entropy loss, expressed as:

[0038]

[0039] where σ(z i ) represents the value applied to the model output z i Sigmoid function;

[0040] The total loss of box aggregation distillation is then expressed as:

[0041]

[0042] where μ is a hyperparameter used to balance the contribution of binary cross entropy loss and box aggregation loss.

[0043] The present invention also includes a graph neural network distillation system, comprising:

[0044] The model acquisition module is used to obtain the graph neural network model of the knowledge to be distilled as the teacher model; and to obtain the graph neural network model of the knowledge to be received as the student model;

[0045] A distillation module is used to extract and aggregate graph structure information of multiple scales in the graph neural network model by introducing a box feature aggregation feature extraction module between the teacher model and the student model to obtain a box aggregation loss; the student model is trained based on the total box aggregation distillation loss formed by the box aggregation loss and the prediction loss of the student model, and the trained student model is applied to a specified task; wherein the generation process of the box aggregation loss specifically includes:

[0046] The partitioning unit is used to gradually calculate the quality of each node in the graph structure of the graph neural network, select the node with the highest quality in each round as the central node, and aggregate all nodes with the infection distance that can cover the entire graph structure as the radius around the central node to obtain a box;

[0047] The loss calculation unit is used to update the output features of the middle layers of the teacher model and the student model for each round of calculated boxes; align the middle layers of the teacher model and the student model, calculate the similarity of the output features of the two middle layers after the update, and obtain the box aggregation loss.

[0048] The present invention also includes a graph neural network distillation computer device, including: a memory, a processor, and a computer program stored in the memory, and the processor implements the steps of the graph neural network distillation method when executing the computer program.

[0049] The present invention also includes a readable storage medium, which stores a computer program. The computer program includes program instructions, and when the program instructions are executed by a processor, they are used to execute the steps of the graph neural network distillation method.

[0050] The present invention provides a graph neural network distillation method, system, device and medium, which have the following beneficial effects:

[0051] The present invention introduces a box feature aggregator BFA, which aggregates all nodes with the infection distance of the entire graph structure as the radius, taking the node with the highest quality as the central node. It can segment the graph network structure at different scales and aggregate the features in each box, thereby improving the feature representation of the central node. This method can not only improve the feature aggregation efficiency by utilizing the fractal characteristics of the network, but also effectively transmit the deep structural information of the graph. At the same time, by calculating the similarity of the output features of the intermediate layers of the teacher model and the student model, the box aggregation loss is obtained, which can minimize the feature differences of the intermediate layers, thereby promoting the successful transfer of multi-scale knowledge and deep structural knowledge from the teacher model to the student model, and significantly improving the knowledge transmission efficiency from the teacher model to the student model. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 It is a flowchart of the neural network distillation method in the embodiment of the present invention;

[0053] Figure 2 A schematic diagram of a network renormalization process in an embodiment of the present invention;

[0054] Figure 3 The number of boxes and the reorganization radius r of the three network data sets in the embodiment of the present invention are b Schematic diagram of the logarithmic function trend;

[0055] Figure 4 Schematic diagram of F1 scores of different confidence levels in a PPI dataset in an embodiment of the present invention;

[0056] Figure 5 The different scales r in the PPI dataset in the embodiment of the present invention b Convergence comparison diagram of ;

[0057] Figure 6 Schematic diagram of the convergence trend of F1 scores of different baselines in three databases in the first 30 epochs in an embodiment of the present invention. DETAILED DESCRIPTION

[0058] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0059] This paper proposes a graph neural network distillation method, which aims to solve the limitations of traditional GNN knowledge distillation technology in processing node features and transmitting deep information. By proposing a box feature aggregation feature extraction module (BFA), box aggregation distillation (BAD) effectively captures and aggregates graph structure information at multiple scales, significantly improving the efficiency of knowledge transfer from the teacher model to the student model. Unlike traditional distillation methods that mainly focus on local neighborhood topology, BAD is good at retaining local information while ensuring efficient transmission of deep network information through multi-level feature aggregation; the renormalization process is applied to explore the fractal characteristics of the network, such as Figure 2 The network renormalization process shown.

[0060] set up is a graph, where represents the node set, and ε represents the edge set. Each node has a eigenvector f i ∈R F and a label y i ∈Y. The input data usually consists of a set of feature vectors, denoted by F = {f i ∈R F |i=1,2,…,n}, corresponding to n nodes, and graph

[0061] The core idea of ​​Graph Neural Networks (GNNs) is to update the representation of each node by aggregating its own features and the features of its k-hop neighbor nodes. The key equation of GNNs can be expressed as:

[0062]

[0063] in is the feature vector of node v at the (k+1)th layer. It is obtained by aggregating nodes and its neighbor node v's current representation vector The obtained result is then combined with the weight matrix W of the kth iteration using the aggregation function AGG. (k) Multiply, and the final result is activated by a nonlinear activation function σ. The initial representation is Given.

[0064] There are several common aggregation methods in graph neural networks, such as mean aggregation, weighted aggregation, and maximum pooling. Taking the Graph Attention Network (GAT) as an example, it introduces an attention mechanism that assigns different weights to different neighbor nodes, enabling the model to automatically learn the importance of each neighbor. The equation can be expressed as:

[0065]

[0066] The attention coefficient It is obtained by applying a linear transformation to the features of nodes v and u, followed by a nonlinear activation:

[0067]

[0068] in Represents the connection between the feature vectors of nodes v and u at the kth layer. (k) is the learned weight vector used to calculate the attention score, and L(.) represents the LeakyReLU activation function.

[0069] The graph neural network (GNN) at the kth layer can be expressed as in ε and denote the node set, edge set, and feature matrix of the kth layer respectively. These nodes and edges have been converted into learned embeddings. In addition, using Z = {z i ∈R F |i=1,2,…,n] to represent the feature vector of the intermediate feature map, where z i represents the feature vector of the i-th node, R F Represents the F-dimensional real space formed by all eigenvectors, with i indexing the nodes from 1 to n.

[0070] The local structure is represented by a d-dimensional real vector LS = {LS i ∈R d |i=1,2,…,n}, where d is the degree of node i. The formula for each element is as follows:

[0071]

[0072] in is defined as z i and z j The squared Euclidean L2 norm of the difference vectors between them is used to quantify their similarity in high-dimensional space. The eigenvalues ​​of all nodes pointing to the local central node are indexed and then normalized to convert into a probability distribution. This makes LS i It can accurately represent the local structure of each node i, as shown in equation (4).

[0073] The local structures of the teacher model and the student model are represented as LS s and LS t The similarity distribution between the local structures of the teacher model and the student model is measured using the KL divergence, which can be expressed as:

[0074]

[0075] Where S i Inversely proportional to the local structural distribution.

[0076] Therefore, by aggregating these similarity measures, an overall metric is obtained, called the Local Structure Preserving (LSP) loss, which is defined as:

[0077]

[0078] The composition of the total loss is as follows:

[0079]

[0080] in represents the cross entropy loss function between the predicted value of the student model and the true label, and λ is a hyperparameter used to balance the two losses. The standard back-propagation algorithm is combined with equation (7) to converge the model and update its weights.

[0081] like Figure 1 As shown, the method specifically comprises the following steps:

[0082] S1. Divide modules.

[0083] To initialize the graph Each The initial mass m i and candidate weight c i Set to 1. For each candidate central node i, the infection radius r b Starting from 1, gradually infect its neighbor nodes with a step size of 1. If a neighbor node has not been visited, mark it as visited, add it to the new infection set, and update the quality m i This process is repeated until the infection radius r b The infection radius is the radius r of the infected person (central node) directly connected to the infected person. b All people (nodes) within the radius r are infected. In each round of the cycle, a central node is found and put into a set, and after the round of the cycle, the central node and the radius r are added. b All nodes within the range are deleted from the graph structure, and the next central node is found in the remaining graph structure, and so on, until the entire network is empty. All central nodes in the above process are used for the next aggregation operation. For each round, the node with the highest quality is the candidate central node; the calculation formula for the quality after infection is:

[0084]

[0085] Where N(i,r b ) represents the radius r of node i b The set of all nodes in w ijrepresents the contribution of node j to the quality of node i. In most cases, w ij Set to 1.

[0086] Among all candidate central nodes, the node with the highest quality is selected as the central node (hub), as shown below:

[0087]

[0088] Among them, node i must satisfy condition c i =1, indicating that the node is a candidate node for the central node; argmax indicates the maximum value parameter.

[0089] By the central node h i and its radius is r b The subgraph consisting of neighbor nodes is called a box. Remove and update the number of unassigned nodes. This operation is expressed as:

[0090]

[0091] in Indicates that the central node h i The center is r and the radius is b The symbol \ represents the set difference operation, that is, removing the subgraph. The function dist(h i ,j) represents the central node h i The shortest path distance between node j and node j. This shortest path distance is calculated using the breadth-first search (BFS) algorithm and is defined as:

[0092]

[0093] in Indicates that from the central node h i The set of all possible paths to node j, P is a specific path, (u,v) represents an edge in path P, and w(u,v) is the weight of edge (u,v). In an unweighted graph (all edges in the graph have the same weight), weight w(u,v) = 1, which means that node h i The shortest path distance between and node j is the path with the fewest edges.

[0094] By aggregating all central nodes and their respective boxes, we obtain two sets, denoted as H = {h1,h2,…,h k}and Here, H represents the set of all central nodes, and B represents the set of boxes, each of which has a central node h. i The center is r and the radius is b .

[0095] S2. Polymerization extraction method of BFA.

[0096] The feature update is performed on the intermediate output feature map of layer l, expressed as in and ε represent the set of nodes and edges respectively. represents the node feature matrix, where N is the number of nodes, F l is the dimension of the intermediate features of layer l. Let The central node h i The node feature vector of a box centered on is expressed as The feature update of the center node of the i-th box is expressed as:

[0097]

[0098] in is the updated central node feature, represents the feature aggregation function used to integrate the features within the selected box. Specifically, It means that the first p% of boxes are selected after sorting in descending order by the number of nodes, where larger boxes (larger means the boxes with more nodes) have a greater impact on the central node and are therefore considered more important. The choice of the best confidence level depends on the neural network model. About aggregation function There are many ways to choose, and the aggregation strategies include:

[0099]

[0100] The three aggregation methods are sum aggregation (SUM), mean aggregation (MEAN) and graph neural network-based aggregation (GNN). The present invention uses the SUM method. Among them, the GNN-based aggregation method uses a single-layer convolutional graph convolutional network (GCN) for feature aggregation. In the experimental part, the present invention systematically compares these feature aggregation methods with methods that do not use feature aggregation. For weighted sum aggregation and mean aggregation methods, the present invention applies the strategy of applying aggregation to the central node or all nodes in the box.

[0101] After updating the features, the present invention uses represents the set of BFA feature vectors, where each feature vector can be expressed as:

[0102]

[0103] In this context, represents the Euclidean distance used to measure the similarity between nodes, where a smaller distance indicates a higher similarity between nodes. A negative exponential function is applied to emphasize this feature, ensuring that a smaller distance between adjacent nodes contributes more significantly to the feature update of the central node. The parameter λ is used as a tuning factor to control the sensitivity of the weight to distance. Equation (14) is used for the node neighborhood The weights in the box are normalized to ensure that the sum of all weights of the neighboring nodes of node i in the box is equal to 1.

[0104] Among them, the box-type aggregation distillation BAD method includes: since the input graphs of the teacher model and the student model are the same, for a given intermediate feature map The present invention calculates the box features of the teacher model and the student model, which are denoted as BF t and BF s For each central node h i , the similarity between their box features can be calculated:

[0105]

[0106] in Represents the Kulbak-Leibler divergence (KL divergence) between the feature distributions of the two models at the i-th node. This metric effectively captures the detailed differences in complex feature structures in high-dimensional feature space and facilitates local feature alignment. Lower SIM i A value of indicates a smaller difference between the two distributions in the feature space. By calculating the similarity of all nodes i, the box clustering loss can be expressed as:

[0107]

[0108] In addition to the box clustering loss, the supervision of the student network can also come from the node label predictions made by the student model. The binary cross entropy loss is used when training the model:

[0109]

[0110] where σ(z i ) represents the value applied to the model output z i Sigmoid function. Therefore, the total loss of box aggregation distillation can be expressed as:

[0111]

[0112] where μ is a hyperparameter used to balance the contribution of binary cross entropy loss and box aggregation loss.

[0113] S3. Train the student model based on the total box aggregation distillation loss formed by the box aggregation loss and the prediction loss of the student model, and use the trained student model for the specified task.

[0114] The present invention first converts the original image Clone to a new image This step ensures that subsequent operations will not affect the data in the original graph. For each node in the cloned graph, its initial mass is set to m i =1 and candidate coefficient c i = 1. Then traverse the entire clone graph, gradually calculate and update the quality m of each node i . According to the calculated node mass m i , select the node with the highest quality as the central node h i , and aggregate these nodes into "boxes" Next, for each box formed Update the intermediate features of the teacher GNN and the student GNN and Finally, using the box clustering loss Optimize model performance.

[0115] Experimental results show that the effectiveness of the BAD method is evaluated through node classification tasks on three benchmark datasets.

[0116] The proposed method is evaluated on three real-world networks, which are widely used in the field of graph learning and have become standard benchmark datasets for evaluating the performance of graph neural network models.

[0117] Cora is a graph-structured dataset that contains academic papers as nodes, and citation relationships form edges between nodes. Each node is represented by a feature vector extracted from the paper vocabulary, and the papers in the dataset are divided into 7 categories, corresponding to different research fields.

[0118] PubMed is similar to the Cora dataset, but larger in scale. It contains academic papers from the PubMed database, mainly related to diabetes research. The node features are constructed based on the bag-of-words model, and the edges represent the citation relationship between papers. The papers are classified into 3 categories, corresponding to different types of diabetes research.

[0119] Protein-Protein Interaction (PPI) consists of 24 independent graphs, each of which represents a protein interaction network in a specific human tissue. Nodes represent proteins and edges represent interactions between them. Each node has a feature vector that describes the biological properties of the protein. The dataset has 121 label dimensions and supports multi-label classification tasks, so that each protein node can belong to multiple functional categories at the same time.

[0120] A summary of the dataset statistics is shown in Table 1.

[0121] Table 1 Dataset statistics

[0122]

[0123] All datasets used in this paper are implemented using the GAT model under the PyTorch framework as the teacher model and student model in the experiment.

[0124] The baseline method used in this invention is as follows:

[0125] FitNet method: This method uses deep supervision, where intermediate layer features are set as targets to guide the student model to gradually learn the intermediate feature representations of the teacher model.

[0126] Attention Transfer (AT) Method: This method guides the student model to focus on key feature areas by transferring the information of salient areas in the attention mechanism of the teacher model.

[0127] LSP Method: The Local Structure Preserving (LSP) technique is used to transfer the local topological features of the teacher model to the student model, thereby improving its performance.

[0128] Since the task involves multi-label classification, binary cross entropy is selected as the loss function for all models. All models are trained through a 30-round warm-up phase, during which the first 30 rounds serve as an observation period for the initial convergence phase. This phase aims to leverage the rich knowledge embedded in the teacher model to provide a more stable learning signal during the early training of the student model, thereby preventing instability caused by insufficient learning in the early stages of training. After this initial phase, the teacher model is no longer used for supervision, maximizing the potential of the dataset and enabling the model to generalize more effectively to unseen data.

[0129] For all models, the learning rate is fixed to 0.005, the slope of the dead ReLU is set to 0.2, and the batch size is 2. For the Cora and PubMed datasets, the dropout rate of both the input and output layers is set to 0.6, and the weight decay is set to 0.0001. In contrast, for the PPI dataset, all three values ​​are set to 0. The teacher models all use a 3-layer architecture with 4 attention heads in the input layer, 6 attention heads in the output layer, and a hidden dimension of 256. Although the student models have a deeper architecture, they use fewer attention heads and a smaller hidden dimension. The specific configurations of the student models for different datasets are detailed in Table 2.

[0130] Table 2 Specific configuration of student models for different data sets

[0131]

[0132] The student models for Cora, PubMed, and PPI datasets are approximately 55.32, 800, and 19.92 times smaller than their corresponding teacher models, respectively. All experiments are performed on a single RTX2080 Ti GPU.

[0133] Fractal feature analysis: Due to the similarity of methods and results, the PPI dataset was used as an illustrative example. Five protein networks were randomly selected from the 24 protein networks for recalibration analysis and their fractal features were explored on a logarithmic scale. Figure 3 shown.

[0134] With the scale parameter (infection radius) r b Gradually increase (step size is 1), all networks in r b = 4 collapses to a single node. Linear distribution of data points on a logarithmic scale and its high goodness of fit R 2 The value indicates that these networks may have self-similar properties. According to the fractal dimension formula Calculated fractal dimension D f The range is between 1.7 and 1.8, which is consistent with the self-similar network theory proposed by Barabási et al., indicating that these networks have significant self-similarity and modular fractal structure.

[0135] The same method is applied to the Cora and PubMed datasets. Unlike PPI, Cora and PubMed are complete network graphs, so no sampling is required and the exploration of fractal structures is performed directly on the original graphs. Experiments show that these two datasets are b =9, the entire network is also in a box, such as Figure 3 The number of boxes and the reorganization radius r of the three network datasets b The logarithmic function trend of R 2 Indicates goodness of fit; (a) is the renormalized trend of the PPI dataset; (b) is the renormalized trend of the Cora dataset; (c) is the renormalized trend of the PubMed dataset. The calculated fractal dimension D f The fractal dimensions of the PubMed graph are 1.264 and 3.868 respectively, indicating that the two networks also exhibit significant fractal characteristics. It is worth noting that the higher fractal dimension of the PubMed graph indicates that it has a more complex fractal structure and stronger space filling ability.

[0136] Comparison of different aggregation methods: In the training process of all models, the teacher network and the student network were pre-trained on the same training set. The summation, averaging and GNN-based feature aggregation strategies were used to aggregate the features of the intermediate layer nodes of the PPI network. The experiment was conducted with the scale parameter r b= 1. The results presented in Table 3 show the trend of the F1 score, starting from the initial state (origin), through the warm-up phase (epochs 10 to 30), and finally reaching the final phase. GNNLayer1 and GNNLayer2 represent the aggregation of features using a single-layer and a double-layer GNN, respectively. The bold part identifies the best aggregation strategy.

[0137] Table 3 Performance of different aggregation methods evaluated by F1 score at each training stage of the PPI node classification task

[0138]

[0139] The experimental results show that in the initial stage of the PPI node classification task, the F1 scores of various aggregation methods are relatively low, about 30, reflecting the learning state of the model in the early stage of training. As the training progresses, the model performance gradually improves and begins to converge. Among these methods, the SUM aggregation method shows the fastest convergence rate, and its F1 score is significantly better than other methods in the first 30 rounds, and finally reaches the highest F1 score in the final evaluation stage. The average convergence rate in the first 30 rounds, the SUM method is 9.90% to 29.07% higher than other methods, indicating that the sum aggregation strategy better captures the fractal structure of the network, thereby achieving more effective feature representation learning.

[0140] In addition, the present invention also conducts comparative experiments on the performance of the SUM aggregation method at different scales to evaluate its performance in the PPI node classification task. The results are shown in Table 4.

[0141] Table 4. Evaluation of the performance of the SUM aggregation method at different scales at different training stages using the F1 score.

[0142]

[0143] Although the three scales of SUM aggregation method show good convergence and performance improvement, there are slight differences in performance at different training stages. Specifically, r b = 1 method has a slight advantage in the final F1 score, while r b =2 and r b =3 method showed faster convergence speed in some stages of training, and the average convergence rate was higher than r b = 1 is 3.9% and 2.3% faster. This shows that the fractal characteristics are b =2 has the greatest impact on the model. It is worth noting that b When =4, the PPI network collapses into a single node, so aggregation at this scale is no longer meaningful for evaluation.

[0144] Although this paper mainly introduces the experimental results of the PPI dataset, the same experimental method is also applied to the Cora and PubMed datasets to verify the versatility and robustness of the proposed method. The experimental results show that the SUM feature aggregation method shows significant advantages in all training stages of the Cora, PubMed and PPI datasets, including faster convergence speed and higher final F1 score.

[0145] Results of different confidence levels under the same scale: Due to the self-similarity of the network, during the renormalization process, the number and size of boxes follow a power-law distribution, that is, most boxes contain fewer nodes, while a few boxes contain more nodes. In this case, the central nodes of the smaller boxes have weaker aggregation capabilities and may not contribute much to the model performance. In order to evaluate the performance of the model under different confidence levels, the present invention extracts boxes of different confidence levels and sorts them from large to small by size. The classification task on the PPI network is then analyzed in increments of 10%, and the results are shown in Figure 2. Figure 4 As shown, (a) the middle scale r b = 1, (b) mesoscale r b =2. Figure 4 It shows that the learning ability of the model gradually improves as the training progresses. In the early stage (epochs 0 to 10), the F1 scores of different confidence levels increase steadily. However, in the middle stage (epochs 10 to 20), the performance of the model experiences large fluctuations, especially in r b =2, the model is more sensitive to changes in confidence, which indicates that different confidence levels have a significant impact on learning efficiency.

[0146] Figure 5 This further illustrates the convergence trend of the model. (a) The mesoscale r b = 1, (b) mesoscale r b =2. It is worth noting that in the scale (r b =2) and 50% confidence level, the model converges fastest and the final F1 score is the highest. However, when the scale increases to (r b =3), the number of renormalized boxes is reduced to only a few, making the confidence extraction method no longer applicable to analyzing networks at this scale. This result suggests that the choice of scale can significantly affect the learning performance of the model, especially at larger scales, where the stability and performance of the model may be limited by the limited number of boxes.

[0147] In addition, the present invention adopts the same confidence selection strategy for Cora graph and PubMed graph, and the results are similar to those when 100% box is selected. The root cause of this phenomenon is that the complexity and self-similarity of PPI network lead to obvious power law distribution in the renormalization process, while Cora and PubMed networks are relatively simple and show minimal fractal characteristics. Therefore, in these simpler networks, the choice of confidence box has little effect on model performance. This shows that the confidence strategy is more effective in complex networks, while it has limited impact in simple networks.

[0148] Comparison with Baselines: To comprehensively evaluate the performance improvement of the BAD method compared to existing baseline methods, we conducted extensive experiments on three datasets (Cora, PubMed, and PPI). Table 5 summarizes the F1 score improvement of the BAD method compared to each baseline method and the independent student model. SST represents the result of training the student model using only the true labels without the teacher knowledge. BAD refers to the method of the present invention, and the results are shown in bold. The best results of other methods are underlined. ΔF1 Cora ,ΔF1 PubMed andΔF1 PPI Represents the improvement of the BAD method compared to the baseline method.

[0149] Table 5 F1 scores of different knowledge distillation methods on Cora, PubMed and PPI datasets

[0150]

[0151] Experimental results show that the BAD method shows significant advantages on all datasets and consistently outperforms various baseline methods. For the AT method, the present invention uses the attention formula And LeakyReLU is applied as the attention activation function. The Cora dataset uses the scale r b = 1, and the PubMed dataset uses the scale r b =2, while the PPI dataset uses the scale r at the 50% confidence level. b =2.

[0152] Compared with the SST method, the standard deviation is improved by about 4.78%, indicating that the performance on different datasets is more stable, especially when dealing with complex graph structures in PubMed and PPI datasets. The method of the present invention shows greater performance improvement than other knowledge distillation methods, effectively extracting and transferring the local topological information of the teacher model, while capturing deep information and multi-scale structures, significantly enhancing the performance of the student model.

[0153] Convergence Analysis of Different Methods: Since the present invention uses the knowledge of the teacher model in the first 30 epochs for learning, the present invention analyzes the convergence of each method by monitoring the performance of the model throughout the warm-up phase of the three datasets, e.g. Figure 6 shown.

[0154] Experimental results show that the BAD model exhibits fast and stable convergence characteristics on all datasets. In the Cora dataset, Figure 6 (a) As shown, r b = 1, the average convergence speed of the BAD model in the warm-up phase is 2.96% to 4.19% faster than other baseline methods. Figure 6 (b) shows that in the PubMed dataset, in addition to having a similar convergence speed to the LSP method, r b = 1 improves the average convergence speed by 7.79% to 10.52% compared with other baseline methods. Figure 6 As shown in (c), in the PPI dataset, r b = 2 significantly improves the average convergence speed by 5.43% to 38.26%. These results show that the method of the present invention effectively utilizes the knowledge of the teacher model and achieves faster and more stable convergence on multiple data sets. Especially in complex data sets, the BAD model is able to quickly reach high performance in the early stages, demonstrating its advantage in processing complex structures, especially in networks with fractal structures.

[0155] This paper proposes a novel model distillation method based on fractal features, called Box Aggregation Distillation (BAD). The method aims to address the challenges of model deployment on edge devices, where small models have insufficient performance and large models have excessive demand for computing resources. By leveraging and transferring fractal knowledge and inherent network structure information, BAD significantly improves the performance of student models. Specifically, the fractal properties of the network are deeply analyzed to study how these features affect the performance of the model under different aggregation strategies. In addition, the superiority of the method in capturing fractal features is demonstrated by analyzing the observation differences at multiple scales. Knowledge transfer is achieved by minimizing the difference between the intermediate feature maps of the teacher model and the student model, combined with a designed confidence threshold strategy to reduce the negative impact of irrelevant features on the distillation effect. The significant advantage of BAD in handling complex graph structures is verified in three datasets with different fractal features and complexity, demonstrating its ability to effectively transfer deep and fractal structural information through multi-scale and multi-confidence aggregation strategies. This paper provides a novel solution for deploying models on lightweight edge devices. Experiments have shown that this integration and deep mining of multi-level information gives BAD excellent adaptability and flexibility in the distillation process, providing an innovative and efficient way for the distillation of graph models. At the same time, the effectiveness of BAD was verified on three classic datasets with different task difficulties (PPI, Cora, PubMed). Experimental results show that in terms of F1 score, the method of the present invention outperforms all baseline distillation methods. In addition, the early convergence speed of the model increased by 2.96% to 38.26%.

[0156] Based on the same inventive concept, the present invention also proposes a graph neural network distillation system, comprising:

[0157] The model acquisition module is used to obtain the graph neural network model of the knowledge to be distilled as the teacher model; and to obtain the graph neural network model of the knowledge to be received as the student model.

[0158] The distillation module is used to extract and aggregate graph structure information of multiple scales in the graph neural network model by introducing a box feature aggregation feature extraction module between the teacher model and the student model to obtain a box aggregation loss; the student model is trained based on the total box aggregation distillation loss formed by the box aggregation loss and the prediction loss of the student model, and the trained student model is used for the specified task; wherein the generation process of the box aggregation loss specifically includes:

[0159] The partitioning unit is used to gradually calculate the quality of each node in the graph structure of the graph neural network, select the node with the highest quality in each round as the central node, and aggregate all nodes with the central node as the center and the infection distance that can cover the entire graph structure as the radius to obtain a box.

[0160] The loss calculation unit is used to update the output features of the middle layers of the teacher model and the student model for each round of calculated boxes; align the middle layers of the teacher model and the student model, calculate the similarity of the output features of the two middle layers after the update, and obtain the box aggregation loss.

[0161] The present invention also proposes a graph neural network distillation computer device, comprising: a memory, a processor, and a computer program stored in the memory, and the processor implements the steps of the graph neural network distillation method when executing the computer program.

[0162] The present invention also proposes a readable storage medium, which stores a computer program. The computer program includes program instructions. When the program instructions are executed by a processor, they are used to execute the steps of the graph neural network distillation method.

[0163] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A graph neural network distillation method, characterized in that: The following steps are involved: The graph neural network model of the knowledge to be distilled is used as the teacher model, and the graph neural network model of the knowledge to be received is used as the student model; the graph structure information of multiple scales in the graph neural network model is extracted and aggregated through the box feature aggregation feature extraction module between the teacher model and the student model to obtain the box aggregation loss; the student model is trained based on the total box aggregation distillation loss formed by the box aggregation loss and the prediction loss of the student model, and the trained student model is applied to the specified task; wherein the generation process of the box aggregation loss specifically includes the following steps: The quality of each node in the graph neural network structure is calculated step by step, the node with the highest quality in each round is selected as the central node, and all nodes with a radius of infection distance that can cover the entire graph structure are aggregated around the central node to obtain a box; For the boxes obtained in each round of calculation, the output features of the intermediate layers of the teacher model and the student model are updated respectively; the intermediate layers of the teacher model and the student model are aligned, and the similarity of the output features of the two intermediate layers after the update is calculated to obtain the box aggregation loss.

2. A graph neural network distillation method according to claim 1, characterized in that: The stepwise calculation of the quality of each node in the graph neural network structure specifically includes the following steps: Initialize the graph Each The initial mass m i and candidate weight c i Set to 1, for each candidate central node i, the infection radius r b Starting from 1, gradually infect its neighbor nodes with a step size of 1. If a neighbor node has not been visited, it is marked as visited, added to the new infection set, and the quality m is updated. i ; After multiple updates, until the infection radius r b Until the entire graph structure is covered, the mass m of each node after infection i The calculation formula is expressed as: Where N(i,r b ) represents the infection radius r of node i b The set of all nodes in w ij Represents the contribution of node j to the quality of node i.

3. A graph neural network distillation method according to claim 2, characterized in that: The node with the highest quality in each round is selected as the central node, wherein the central node h i It is expressed as: Among them, node i must satisfy condition c i =1, which indicates that the node is a candidate node for the central node; argmax indicates the maximum value parameter.

4. A graph neural network distillation method according to claim 3, characterized in that: The aggregation takes the central node as the center and all nodes with the infection distance that can cover the entire graph structure as the radius to obtain a box; its generation process is expressed as: in Indicates that the central node h i The center is r and the radius is b The set of all nodes is represented as a box; the symbol \ represents the set difference operation; the function dist(h i ,j) represents the central node h i The shortest path distance between node j and node j; the shortest path distance is calculated using the breadth-first search BFS algorithm and is expressed as: in Indicates that from the central node h i The set of all paths to node j, P is a specific path, (u,v) represents an edge in path P, and w(u,v) is the weight of edge (u,v).

5. A graph neural network distillation method according to claim 4, characterized in that: For each box obtained by calculation, the output features of the intermediate layers of the teacher model and the student model are updated respectively, which specifically includes the following steps: Perform feature update on the intermediate output feature map of layer l, expressed as in and ε represent the set of nodes and edges respectively, and the matrix represents the node feature matrix, where N is the number of nodes, F l is the dimension of the intermediate features of layer l; Box The node feature vector of Then the feature update of the center node of the i-th box is expressed as: in is the updated central node feature, represents the feature aggregation function used to integrate the features within the selected box, means selecting the top p% of boxes after sorting in descending order by the number of nodes.

6. A graph neural network distillation method according to claim 5, characterized in that: The feature aggregation function for integrating the features in the selected box is expressed as:

7. A graph neural network distillation method according to claim 5, characterized in that: The process of generating the total loss of the box aggregate distillation comprises the following steps: Since the input graphs of the teacher model and the student model are the same, for a given intermediate feature graph Calculate the box features of the teacher model and the student model, denoted as BF t and BF s ; For each central node h i , the similarity between the box features SIM i The calculation expression is: in represents the KL divergence between the feature distributions of the two models; by calculating the similarity between all box features, the box aggregation loss is obtained, which is expressed as: The prediction loss of the student model uses binary cross entropy loss, expressed as: where σ(z i ) represents the value applied to the model output z i Sigmoid function; The total loss of box aggregation distillation is then expressed as: where μ is a hyperparameter used to balance the contribution of binary cross entropy loss and box aggregation loss.

8. A graph neural network distillation system, characterized in that: include: The model acquisition module is used to obtain the graph neural network model of the knowledge to be distilled as the teacher model; and to obtain the graph neural network model of the knowledge to be received as the student model; A distillation module is used to extract and aggregate graph structure information of multiple scales in the graph neural network model by introducing a box feature aggregation feature extraction module between the teacher model and the student model to obtain a box aggregation loss; the student model is trained based on the total box aggregation distillation loss formed by the box aggregation loss and the prediction loss of the student model, and the trained student model is applied to a specified task; wherein the generation process of the box aggregation loss specifically includes: The partitioning unit is used to gradually calculate the quality of each node in the graph structure of the graph neural network, select the node with the highest quality in each round as the central node, and aggregate all nodes with the infection distance that can cover the entire graph structure as the radius around the central node to obtain a box; The loss calculation unit is used to update the output features of the middle layers of the teacher model and the student model for each round of calculated boxes; align the middle layers of the teacher model and the student model, calculate the similarity of the output features of the two middle layers after the update, and obtain the box aggregation loss.

9. A graph neural network distillation computer device, characterized in that: include: A memory, a processor, and a computer program stored in the memory, wherein the processor implements the steps of the graph neural network distillation method according to any one of claims 1 to 7 when executing the computer program.

10. A readable storage medium, characterized in that: The readable storage medium stores a computer program, which includes program instructions. When the program instructions are executed by a processor, they are used to execute the steps of the graph neural network distillation method described in any one of claims 1 to 7.

Images