Graph neural network performance optimization method and device, storage medium and computer equipment

Abstract

The invention discloses a graph neural network performance optimization method and device, a storage medium and computer equipment, and the method comprises the steps: carrying out the linear interpolation of graph node features and labels in original graph structure data, and generating enhanced graph structure data; constructing a teacher graph neural network as a teacher model, and obtaining a soft label predicted by the teacher model for the enhanced graph structure data; and constructing a student graph neural network as a student model, migrating a soft label into the student model by using a knowledge distillation technology, training the student model based on enhanced graph structure data, and simultaneously guiding the student model by using knowledge distillation loss driven by cross entropy loss and the knowledge distillation technology until an optimal student model is trained. By performing linear interpolation on the node features and the labels, while an original graph topological structure is reserved, diversified training samples are generated, and the prediction performance of the student graph neural network can be improved.

Description

TECHNICAL FIELD

[0001] The application relates to the technical field of graph structure data mining, and in particular to a graph neural network performance optimization method and device, a storage medium and a computer device. BACKGROUND

[0002] As a deep learning model capable of directly acting on graph structure data, graph neural networks (GNNs) have shown strong performance in many fields in recent years, including social network analysis, recommendation systems, bioinformatics, and medical data analysis. In the medical field, graph structure data is widely used, such as disease-symptom association graphs, drug-target interaction graphs, and patient similarity networks. These graph structure data contains rich information that can effectively support drug development, patient stratification, and other important tasks.

[0003] However, existing graph neural networks face a series of challenges when processing medical graph structure data. Medical graph structure data is often highly complex and heterogeneous, with diverse node and edge types and complex data distribution, which puts higher demands on the model's expressive ability. At the same time, medical data annotation is costly and data is scarce, which can lead to overfitting and insufficient generalization ability of the model during training. Existing graph neural network models are difficult to meet actual needs in terms of performance. SUMMARY

[0004] Therefore, the application provides a graph neural network performance optimization method and device, a storage medium, and a computer device. To address the inherent sparsity and dependency challenges of graph structure data, the Mixup data augmentation technique is used to enhance the graph structure data while distilling the pre-trained teacher GNN (Graph Neural Network) model soft label knowledge to the student to improve the generalization ability of the student GNN model.

[0005] According to one aspect of the application, a graph neural network performance optimization method is provided, which includes:

[0006] Obtaining original graph structure data for training a graph neural network, wherein the original graph structure data includes a plurality of graph nodes, each graph node having a graph node feature and a label, the graph node feature representing attribute information of the graph node, and the label representing a category to which the graph node belongs;

[0007] Performing linear interpolation on the graph node features and labels in the original graph structure data to generate enhanced graph structure data;

[0008] construct a teacher graph neural network as a teacher model, and train the teacher model based on the enhanced graph structure data to obtain soft labels predicted by the teacher model for the enhanced graph structure data in a training process, wherein the soft labels are used to represent a prediction probability distribution of the teacher model for a category to which each graph node in the enhanced graph structure data belongs;

[0009] construct a student graph neural network as a student model, and migrate the soft labels to the student model by using a knowledge distillation technology, so that the student model learns the prediction ability of the teacher model, wherein when the soft labels are migrated to the student model by using the knowledge distillation technology, a knowledge distillation loss of the student model for the enhanced graph structure data is obtained, and the student model learns the soft label knowledge from the teacher model by minimizing the knowledge distillation loss;

[0010] train the student model based on the enhanced graph structure data, and simultaneously guide the student model by using a cross-entropy loss and a knowledge distillation loss driven by the knowledge distillation technology until an optimal student model is trained, wherein the knowledge distillation loss is used to measure a difference between the soft labels predicted by the student model and the teacher model for the enhanced graph structure data, and the cross-entropy loss is used to measure a difference between a prediction label predicted by the student model for the enhanced graph structure data and a real label in the enhanced graph structure data.

[0011] According to another aspect of the present application, a graph neural network performance optimization device is provided, and the device comprises:

[0012] An original graph structure data acquisition module is configured to acquire original graph structure data used for training a graph neural network, wherein the original graph structure data comprises a plurality of graph nodes, each graph node is associated with a graph node feature and a label, the graph node feature represents attribute information of the graph node, and the label represents a category to which the graph node belongs.

[0013] An original graph structure data enhancement module is configured to perform linear interpolation on the graph node features and the labels in the original graph structure data respectively to generate enhanced graph structure data.

[0014] A teacher model construction module is configured to construct a teacher graph neural network as a teacher model, and train the teacher model based on the enhanced graph structure data to obtain soft labels predicted by the teacher model for the enhanced graph structure data in a training process, wherein the soft labels are used to represent a prediction probability distribution of the teacher model for a category to which each graph node in the enhanced graph structure data belongs.

[0015] The student model construction module is configured to construct a student graph neural network as a student model, and to migrate the soft labels to the student model by using a knowledge distillation technique, so that the student model learns the prediction ability of the teacher model. When the soft labels are migrated to the student model by using the knowledge distillation technique, a knowledge distillation loss of the student model for the enhanced graph structure data is obtained, and the student model learns the soft label knowledge from the teacher model by minimizing the knowledge distillation loss.

[0016] The student model optimization training module is configured to train the student model based on the enhanced graph structure data, and to simultaneously guide the student model by using a cross-entropy loss and a knowledge distillation loss driven by the knowledge distillation technique, until an optimal student model is trained. The knowledge distillation loss is used to measure the difference between the soft labels predicted by the student model and the teacher model for the enhanced graph structure data, and the cross-entropy loss is used to measure the difference between the prediction labels predicted by the student model for the enhanced graph structure data and the real labels in the enhanced graph structure data.

[0017] According to yet another aspect of the present application, a storage medium having a computer program stored thereon is provided. The program, when executed by a processor, implements the graph neural network performance optimization method described above.

[0018] According to still another aspect of the present application, a computer device is provided, which includes a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor. The processor implements the graph neural network performance optimization method described above when executing the program.

[0019] By means of the above technical solutions, the graph neural network performance optimization method and device, storage medium, and computer device provided by the present application perform linear interpolation on the node features and labels in the original graph structure data to generate enhanced graph structure data, construct a teacher graph neural network as a teacher model and obtain soft labels predicted by the teacher model for the enhanced graph structure data, construct a student graph neural network as a student model, migrate the soft labels to the student model by using a knowledge distillation technique, train the student model based on the enhanced graph structure data, and simultaneously guide the student model by using a cross-entropy loss and a knowledge distillation loss driven by the knowledge distillation technique, until an optimal student model is trained. By performing linear interpolation on the node features and labels, diversified training samples are generated while the original graph topology structure is retained, and the prediction performance of the student graph neural network can be improved.

[0020] The above description is only a summary of the technical solutions of the present application. In order to more clearly understand the technical means of the present application, the specific embodiments of the present application can be implemented in accordance with the content of the description, and in order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the following specific embodiments of the present application are described. BRIEF DESCRIPTION OF DRAWINGS

[0021] The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiments of the application and together with the description serve to explain the application. In the drawings:

[0022] Figure 1 A flowchart of a graph neural network performance optimization method provided by an embodiment of the application is shown;

[0023] Figure 2 A flowchart of original graph structure data enhancement provided by an embodiment of the application is shown;

[0024] Figure 3 A structural diagram of a graph neural network performance optimization device provided by an embodiment of the application is shown. DETAILED DESCRIPTION

[0025] The application will be described in detail below with reference to the drawings and in conjunction with embodiments. It should be noted that the embodiments in the application and the features in the embodiments can be combined with each other without conflict.

[0026] In this embodiment, a graph neural network performance optimization method is provided, as shown in Figure 1 The method comprises the following steps.

[0027] Step 101, obtaining original graph structure data for training a graph neural network, wherein the original graph structure data comprises a plurality of graph nodes, and each graph node corresponds to graph node features and a label, the graph node features representing attribute information of the graph node, and the label representing a category to which the graph node belongs.

[0028] A graph neural network (GNN) is a deep learning model designed for processing graph structure data, and its core goal is to learn low-dimensional embedding representation of nodes, edges or subgraphs in graph structure through a neural network, and use it for downstream tasks such as node classification, graph classification and link prediction. Unlike traditional convolutional neural networks (CNN) or recurrent neural networks (RNN), GNN can directly process data in non-Euclidean space (such as social networks, molecular structures, knowledge graphs, etc.), and capture the dependency between nodes through the way of aggregating neighbor information. The core idea and working principle of GNN are as follows:

[0029] 1. Message passing mechanism:

[0030] The core of GNN is to aggregate the information of neighbor nodes to update the representation of the target node. The specific process can be divided into the following steps:

[0031] Aggregate: Collect the neighbor information (e.g., node features) of the target node.

[0032] Update: Update the representation of the target node by combining its own features and the aggregated neighbor information.

[0033] Iterate: Repeat the above process until the node representation converges or reaches a preset number of layers.

[0034] 2. Mathematical expression:

[0035] For example, in a classic Graph Convolutional Network (GCN), the update formula for node representation can be expressed as:

[0036]

[0037] where, is the adjacency matrix with self-loops, is the degree matrix, H (l) is the node representation matrix at the l-th layer, W (l) is a learnable parameter. σ is an activation function (e.g., ReLU).

[0038] Further, the application scenarios of GNNs include, for example:

[0039] 1. Node classification, examples: user interest prediction in social networks, protein function classification.

[0040] 2. Graph classification, examples: molecular toxicity prediction, document topic classification.

[0041] 3. Link prediction, examples: friend recommendation in social networks, knowledge graph completion.

[0042] 4. Community discovery, examples: user group division in social networks, congestion area detection in transportation networks.

[0043] 5. Physical system simulation, examples: molecular dynamics simulation, crystal structure prediction in materials science.

[0044] To this end, graph neural networks extend neural networks to graph-structured data through a message passing mechanism, providing a powerful tool for handling complex relational data. Its core advantage is to directly model the dependence between nodes, making it suitable for a variety of scenarios (such as social networks, molecular design, and traffic prediction).

[0045] Currently, graph-structured data, involving complex relationships between nodes and edges, is prevalent in various fields, including social networks, bioinformatics, and recommendation systems. Graph Neural Networks (GNNs) have become the mainstream method for processing graph-structured data, effectively capturing structural information and exhibiting superior performance in tasks such as node classification, graph classification, and link prediction. However, as the scale and complexity of graph-structured data continue to increase, the generalization ability of graph neural networks is limited.

[0046] Knowledge Distillation (KD) has become an effective technique to enhance the performance of graph neural networks. The core idea of graph knowledge distillation is to transfer knowledge from a pre-trained teacher model to a student model, thereby guiding the training of the student model. Existing graph knowledge distillation methods can be broadly divided into two categories:

[0047] Graph Neural Networks to Graph Neural Networks, and Graph Neural Networks to Multilayer Perceptron, the former involves transferring knowledge from a teacher graph neural network model to a student graph neural network student model with the same model architecture, referred to as GNN-to-GNN distillation method. Considering the inference delay problem of graph neural networks, the latter focuses on transferring knowledge from a teacher graph neural network model directly to a lightweight multilayer perceptron student model, i.e., GNN-to-MLP distillation method.

[0048] In the above embodiments of the present application, the focus is on the former (GNN-to-GNN distillation method) to achieve performance enhancement of graph neural network models. Specifically, it can be applied in the medical field, such as disease-symptom association networks, drug-target interaction networks, patient similarity networks, etc. The original graph-structured data in the medical field can be obtained from various sources, such as biomedical literature, extracting disease-symptom, drug-disease relationships from literature through natural language processing techniques, and constructing knowledge graphs. Biological databases: such as Gene Ontology (GO), DrugBank, etc., providing interaction data between genes, proteins, and drugs. Medical image data: extracting image features through image processing techniques to construct image-disease association graphs.

[0049] Graph nodes can represent different entities in the medical field, such as:

[0050] Patient nodes: each node represents a patient, containing the patient's clinical features (such as age, gender, medical history, etc.).

[0051] Disease nodes: each node represents a disease, containing the description information of the disease (such as symptoms, causes, etc.).

[0052] Drug nodes: each node represents a drug, containing the pharmacological properties of the drug (such as chemical structure, target, etc.).

[0053] Gene / protein node: Each node represents a gene or protein, containing its biological function or expression level information.

[0054] Next, extract graph node features, which are vectors representing the attribute information of graph nodes. They can be extracted in the following ways:

[0055] Text features: Extract disease, drug, and symptom text features from biomedical literature using natural language processing techniques (such as BERT).

[0056] Image features: Use deep learning models (such as CNN) to extract features from medical images as image node features.

[0057] Biological features: Expression levels, sequence characteristics, and other features of genes or proteins can be used as biological node features.

[0058] Graph node labels are used to define the categories to which the nodes belong, for example:

[0059] Disease node label: Can be the classification of the disease (such as infectious disease, chronic disease, etc.).

[0060] Drug node label: Can be the therapeutic area of the drug (such as anticancer drug, antibiotic, etc.).

[0061] Image node label: Can be the type of disease displayed in the image.

[0062] Graph structure connects different nodes through edges, representing the relationship between nodes, for example:

[0063] Drug-target association: The edge between the drug node and the protein node represents the action of the drug on a specific target.

[0064] Disease-symptom association: The edge between the disease node and the symptom node represents the relationship between the disease and the symptom.

[0065] To this end, the original graph structure data obtained can be used to train graph neural networks to support medical tasks such as drug development.

[0066] Step 102: Linearly interpolate the graph node features and labels in the original graph structure data to generate enhanced graph structure data.

[0067] Then, the Mixup strategy can be used to generate enhanced graph structure data by linearly interpolating the features and labels of the nodes in the original graph structure data. By generating virtual node features and label combinations through linear interpolation, the distribution range of the training data is expanded, allowing the model to be exposed to more diverse samples, thereby reducing the risk of overfitting. The interpolated data is similar to samples that may exist in reality but have not been collected, which can help the model learn more robust feature representations and improve its generalization ability for unseen data.

[0068] In the original graph structure data, some nodes or edges may have insufficient information, and linear interpolation can generate virtual nodes between known nodes to fill in the data gaps and enhance the connectivity of the graph. For nodes with uneven feature distribution, interpolation can smooth the feature space, allowing the model to learn more continuous feature representations. The slight perturbation introduced during the interpolation process can simulate noise in the data, allowing the model to learn more robust features during training and improving its tolerance for noisy data. By generating diverse augmented data, the model can learn more comprehensive features, reduce its sensitivity to specific adversarial samples, and improve its security.

[0069] To this end, the soft labels (probability distribution) generated by linear interpolation contain more information than hard labels (single class), which can provide the student model with richer supervision signals and improve the effectiveness of knowledge distillation. The augmented graph structure data can help the student model better simulate the prediction behavior of the teacher model, improving the performance of the distilled student model. Through the interpolation-generated data, the graph neural network can learn more smooth feature representations, reduce the sparsity of the feature space, and improve the accuracy of node classification or link prediction. In graph neural networks, message passing between nodes depends on the connectivity of the graph, and the interpolation-generated virtual nodes can enhance the connectivity of the graph and improve the effectiveness of message passing.

[0070] Furthermore, for a patient-disease association graph, for example, where patient node features include age, blood sugar level, etc., and the label is whether the patient has diabetes. Through linear interpolation, virtual patient features between two known patients are generated, such as the average of age and blood sugar level. Then, a soft label is generated between "healthy" and "diabetes", such as a probability of 0.7 that the patient has diabetes.

[0071] These augmented data can help the model learn more smooth feature representations and reduce its sensitivity to extreme values. By learning more rich information through soft labels, the model's prediction ability for diabetes is improved. The problem of data sparsity is alleviated, especially for minority patient groups such as young diabetic patients.

[0072] Optionally, as Figure 2As shown, for step 102, “linearly interpolating the graph node features and labels in the original graph structure data respectively to generate enhanced graph structure data”, specifically includes:

[0073] Step 1021, determining a graph node pair based on any two graph nodes in the original graph structure data, and linearly interpolating the respective graph node features of the two graph nodes in the graph node pair using the node feature linear interpolation formula to obtain a virtual graph node feature linearly interpolated based on the graph node pair.

[0074] Step 1022, linearly interpolating the respective labels of the two graph nodes in the graph node pair using the label linear interpolation formula to obtain a virtual label linearly interpolated based on the graph node pair.

[0075] Step 1023, obtaining enhanced graph structure data based on the virtual graph node features and virtual labels linearly interpolated by each graph node pair, wherein the virtual labels in the enhanced graph structure data are used as real labels to train the graph neural network, and the virtual graph node features and virtual labels are represented as:

[0076]

[0077] And are virtual graph node features and virtual labels, respectively, x i and x j are two graph node features in the original graph structure data, y i and y j are labels corresponding to x i and x j respectively, and λ is a random number between 0 and 1, subject to Beta distribution Beta(α, α), λ is used to represent the weight parameter of linear interpolation, and α is a parameter controlling the shape of λ distribution.

[0078] In the above embodiments of the present application, the input original graph structure data G is introduced and the Mixup strategy is used to linearly interpolate the node features and corresponding labels. This strategy not only enables the model to learn a smoother decision boundary, but also serves as a regularization mechanism to enhance the generalization ability of the model. Therefore, the Mixup is used to generate enhanced graph structure data to achieve diversified graph structure data. The virtual data generated by Mixup not only enhances the robustness of the model to graph structure data and reduces the dependence of graph distillation on original graph structure data, but also effectively increases the diversity of training data and alleviates the sparsity problem of graph structure data.

[0079] Specifically, for linear interpolation of node features, specifically, the goal is to generate virtual graph node features, the formula is node feature linear interpolation formula, more specifically, two graph node features xi and x j , randomly generate interpolation coefficient λ (for example, λ ~ Beta (α, α), α is used to control the interpolation intensity, α may be 0.4, for example). Then, calculate the virtual graph node feature according to the formula.

[0080] For linear interpolation of labels, the goal is to generate virtual labels (need to adapt to the task type), scene classification, for example:

[0081] Classification task (label is one-hot encoding), use soft label interpolation.

[0082] Regression task (label is continuous value), direct linear interpolation.

[0083] More specifically, obtain two graph node features x i and x j , and their respective labels y i and y j , select the interpolation method according to the task type, and calculate the virtual label using the same λ.

[0084] For constructing enhanced graph structure data, it can include isolated node enhancement and connecting original graph nodes, for the isolated node enhancement method, the virtual graph node feature and the virtual label can be added to the dataset as independent samples, without connecting any edges. It is suitable for dynamically generating enhanced data during batch training. For the method of connecting original graph nodes, edges are added for connection.

[0085] In particular, for the interpolation coefficient λ, using Beta distribution (such as α = 0.4) can avoid overfitting and enhance diversity. In the classification task, the soft label needs to cooperate with the loss function that supports the soft label (such as KL divergence).

[0086] Step 103, constructing a teacher graph neural network as a teacher model, and training the teacher model based on the enhanced graph structure data, obtaining the soft label predicted by the teacher model for the enhanced graph structure data in the training process, wherein the soft label is used to represent the prediction probability distribution of the teacher model for the class to which each graph node in the enhanced graph structure data belongs.

[0087] Soft labels provide more information than hard labels (traditional one-hot encoding or discrete labels). It reflects the teacher's confidence in each node belonging to each class. By learning these soft labels, the student model can better understand the internal structure and uncertainty of the data during knowledge distillation, thereby enhancing the generalization ability to unseen data. Soft labels contain complex patterns and relationships learned by the teacher model during training. When predicting, this information can help the student model more accurately determine the class to which the node belongs, especially when the data has noise or the class boundary is ambiguous, soft labels can provide more delicate decision basis.

[0088] The enhanced graph structure data is generated by transforming the original graph structure data (node feature interpolation), which increases the diversity of the data. The teacher model is trained on these enhanced data and generates soft labels, which can mine the hidden information and patterns in the data, so that the value of the data is more fully utilized.

[0089] In the case of high cost of graph structure data labeling or limited amount of labeled data, soft labels can serve as an additional supervision signal. Even without a large amount of hard label data, the soft labels generated by the teacher model can provide valuable guidance for the student model, helping it to learn better feature representation.

[0090] The teacher model is usually a larger and better-performing model, and the knowledge and experience learned by the teacher model on the enhanced graph structure data can be passed to the student model through the soft labels. The student model can learn the generalization ability and decision logic of the teacher model with the help of these soft labels, so as to obtain performance close to the teacher model while maintaining small scale and high computational efficiency. The student model is trained under the guidance of soft labels, which can maintain good performance while reducing the number of parameters and computational complexity.

[0091] In addition, in graph structure data, there may be a class imbalance problem, that is, the number of nodes in some classes is much larger than that in other classes. Soft labels can provide more balanced supervision information, because the teacher model considers the features and relationships of all classes during training, thereby helping the student model to better handle the class imbalance situation.

[0092] Optionally, each virtual graph node feature corresponds to a virtual graph node, and for step 103, "training the teacher model based on the enhanced graph structure data", specifically includes:

[0093] Step 1031, when training the teacher model based on the enhanced graph structure data, the cross-entropy loss of each virtual graph node in the enhanced graph structure data is summed and averaged by an average cross-entropy loss function to obtain an average cross-entropy loss, and the training of the teacher model is guided based on the average cross-entropy loss, wherein the average cross-entropy loss function is:

[0094]

[0095] L t is the average cross-entropy loss of the teacher model, L CE is the cross-entropy loss function, N is the total number of virtual graph nodes in the enhanced graph structure data, is the virtual graph node feature, is the embedding representation predicted by the teacher model T for the virtual graph node feature ​is the virtual graph node feature The corresponding virtual label, Used to calculate the cross entropy loss between the embedding representation predicted by the teacher model T and the true label.

[0096] In the above embodiment of the present application, a GNN teacher model is constructed based on the enhanced graph structure data, and the GNN teacher model is trained. Specifically, for each virtual graph node in the enhanced graph structure data, the virtual graph node feature is Input into the teacher model T to obtain the teacher model T for the virtual graph node feature Predicted embedding representation

[0097] Use the cross entropy loss function L CE ,calculate With the virtual graph node characteristics Corresponding virtual label The cross entropy loss between The cross entropy loss function is used to measure the probability distribution of the model prediction (given by Converted) and the true label probability distribution The difference between.

[0098] Sum the cross entropy losses of all N virtual graph nodes in the enhanced graph structure data, and then take the average to get the average cross entropy loss L T , the calculated average cross entropy loss L T As the optimization target, the gradient of the loss function with respect to the teacher model parameters is calculated through the back-propagation algorithm.

[0099] An optimization algorithm (such as stochastic gradient descent and its variants, such as the Adam optimizer) is used to update the parameters of the teacher model according to the calculated gradients, so that the difference between the model's prediction results and the true labels gradually decreases.

[0100] Repeat the above steps, continuously input the enhanced graph structure data into the teacher model, calculate the loss, backpropagate and update the parameters until the model converges, that is, the average cross entropy loss L T No longer significantly reduce or reach the preset number of training rounds.

[0101] Through the above steps, the teacher graph neural network model is trained based on the enhanced graph structure data and the average cross entropy loss function, so that the teacher model can learn the effective features and patterns in the graph structure data, thereby providing valuable soft label information in subsequent tasks (knowledge distillation).

[0102] At step 104, the student graph neural network is constructed as a student model, and the soft label is migrated to the student model by using the knowledge distillation technology, so that the student model learns the prediction ability of the teacher model. When the soft label is migrated to the student model by using the knowledge distillation technology, the knowledge distillation loss of the student model for the enhanced graph structure data is obtained, and the student model learns the soft label knowledge from the teacher model by minimizing the knowledge distillation loss.

[0103] At step 105, the student model is trained based on the enhanced graph structure data, and the cross-entropy loss and the knowledge distillation loss driven by the knowledge distillation technology are used to guide the student model at the same time until the optimal student model is trained. The knowledge distillation loss is used to measure the difference between the soft labels predicted by the student model and the teacher model for the enhanced graph structure data, and the cross-entropy loss is used to measure the difference between the prediction label predicted by the student model for the enhanced graph structure data and the real label in the enhanced graph structure data.

[0104] Then, the structure of the student graph neural network model S can be similar to that of the teacher model, but the number of layers, the number of neurons and other hyperparameters are usually less to ensure the lightweight of the model. The parameters of the student model are initialized, for example, random initialization or other parameter initialization methods can be used.

[0105] The virtual graph node features in the enhanced graph structure data are input into the teacher model T and the student model S respectively, and the embedding representation output by the teacher model T and the embedding representation output by the student model are obtained.

[0106] A suitable loss function is used to measure the difference between the output of the student model and the output of the teacher model (soft label), that is, the knowledge distillation loss. The loss function is, for example, the Kullback-Leibler divergence loss function, and the formula is which can measure the distance between two probability distributions.

[0107] In the knowledge distillation process, the cross-entropy loss between the output of the student model and the real label (hard label) is also combined. The total loss (optimization loss L s ) can be the weighted sum of the knowledge distillation loss and the cross-entropy loss, that is, where β is the weight coefficient for balancing the two losses, and the value range is between 0 and 1.

[0108] The calculated total loss L sAs an optimization objective, the gradient of the loss function with respect to the student model parameters is calculated by a backpropagation algorithm. Using an optimization algorithm (such as stochastic gradient descent and its variants, such as the Adam optimizer), the parameters of the student model are updated according to the calculated gradient, so that the difference between the prediction results of the student model and the soft labels of the teacher model and the real labels gradually decreases.

[0109] The above steps are repeated, and the enhanced graph structure data is constantly input into the student model, the total loss is calculated, the backpropagation is calculated, and the parameters are updated until the student model converges, i.e. the total loss L s No longer significantly reduced or reached the preset number of training rounds.

[0110] Through the above steps, the student model constantly learns the soft label knowledge generated by the teacher model in the training process, and combines the information of the real label, so as to be able to imitate the prediction ability of the teacher model, while maintaining a small scale and high computational efficiency, and obtain a performance close to the teacher model.

[0111] In particular, the soft label (Soft Label) is a probability distribution predicted by the teacher graph neural network (Teacher GNN) for the enhanced graph structure data. It reflects the probability distribution of the teacher model (teacher graph neural network) for each graph node belonging to a class, for example, it can be a vector containing multiple class probabilities (for example, for a three-class task, the soft label can be [0.1, 0.7, 0.2]).

[0112] As a knowledge carrier of the teacher model, the soft label is transmitted to the student graph neural network (Student GNN) through the knowledge distillation technology, helping the student model to learn the prediction ability and generalization ability of the teacher model.

[0113] Compared with the hard label (real label), the soft label contains more information about the relationship between classes (for example, similarity or uncertainty between classes), which helps the student model to learn more detailed decision boundaries.

[0114] The real label (Hard Label / Ground Truth Label) is the explicit annotation of the class to which each graph node in the original graph structure data belongs, for example, it can be a one-hot vector (for example, for a three-class task, the real label can be [0, 1, 0], indicating that the node belongs to the second class).

[0115] As a supervision signal, the real label guides the student graph neural network to learn the correct class prediction. During training, the difference between the prediction of the student model and the real label is measured by the cross-entropy loss function to ensure that the student model does not deviate from the task goal.

[0116] For the relationship between soft labels and real labels, specifically, in the knowledge distillation process, soft labels and real labels jointly guide the training of student graph neural networks:

[0117] Soft labels measure the difference between the predictions of student models and teacher models through knowledge distillation loss (such as KL divergence), prompting student models to learn the generalization ability of teacher models.

[0118] Real labels measure the difference between student models and real labels through cross-entropy loss, ensuring the accuracy of student models on task goals.

[0119] By simultaneously using soft labels and real labels, student graph neural networks can learn the generalization ability and robustness of teacher models while maintaining accuracy on task goals, thereby improving overall performance.

[0120] Optionally, for step 105, "training a student model based on enhanced graph structure data", specifically includes:

[0121] Step 1051, in the process of training a student model based on enhanced graph structure data, calculate the average KL divergence loss of the student model on the enhanced graph structure data, take the calculated average KL divergence loss as the knowledge distillation loss, and use the way of minimizing the knowledge distillation loss to guide the training of the student model, wherein the average KL divergence loss is used to measure the difference between the embedding representation predicted by the student model for the enhanced graph structure data and the node embedding representation predicted by the teacher model for the same data, the calculation formula of the knowledge distillation loss is:

[0122]

[0123] For the knowledge distillation loss of the student model S, is the embedding representation predicted by the student model S for the virtual graph node feature , is the embedding representation predicted by the teacher model T for the virtual graph node feature , L KL is the KL divergence (Kullback-Leibler divergence) loss function, and N is the total number of virtual graph nodes in the enhanced graph structure data.

[0124] In the above embodiments of the present application, the KL divergence loss measures the difference between the output probability distributions of the student model and the teacher model. By minimizing the average KL divergence loss, the student model can learn the complex patterns and generalization ability learned by the teacher model on the augmented graph structured data. This enables the student model to make more accurate predictions when facing unseen data, thereby enhancing the generalization ability of the model. The teacher model is usually a larger and better-performing model that has learned rich feature representations and knowledge on the augmented graph structured data. By minimizing the KL divergence loss with the output of the teacher model, the student model can approximate the prediction results of the teacher model, thereby obtaining more accurate prediction probability distributions and improving the prediction accuracy. As a knowledge distillation loss function, the KL divergence loss can transfer the "soft knowledge" (i.e., the prediction probability distribution of different classes by the teacher model) in the teacher model to the student model. This soft knowledge contains more rich information than hard labels (true labels), which can help the student model better understand the internal structure of the data and the relationship between classes, and achieve effective knowledge transfer.

[0125] The student model usually has a simpler structure and fewer parameters than the teacher model. Through knowledge distillation, the student model can learn most of the knowledge and performance of the teacher model while maintaining a smaller scale, achieving model compression. This enables the student model to have higher computational efficiency and lower resource requirements in practical applications.

[0126] Optionally, the teacher model and the student model each have multiple layers, each layer uses a ReLU activation function, and the embedding dimension is 128.

[0127] In the above embodiments of the present application, the multi-layer structure allows the model to gradually extract features at different levels in the input graph structure data. In a graph neural network, each layer can aggregate information about node neighbors. As the number of layers increases, the model can capture more global and abstract graph structure information. For example, the shallow layer may learn the local neighborhood features of the node, while the deep layer can capture the topological structure and long-distance dependencies of the entire graph. The multi-layer structure increases the complexity of the model, enabling it to fit more complex functions. This is very helpful for processing complex graph structure data and tasks, such as node classification, graph classification, etc., and can better learn nonlinear relationships in the data. The introduction of the nonlinear ReLU (Rectified Linear Unit) activation function f(x) = max(0,x) introduces nonlinearity to the model. In a graph neural network, nonlinearity allows the model to learn complex patterns and decision boundaries in the data. The ReLU activation function is very computationally efficient and can accelerate the convergence of the model during training. Compared with other activation functions (such as Sigmoid and Tanh), ReLU does not have the problem of gradient disappearance (the gradient is 1 in the positive interval), which enables the model to adjust parameters faster during training and improve training efficiency.

[0128] An embedding dimension of 128 provides sufficient dimensions for each node or graph to represent its features. In graph neural networks, node embedding maps the node to a low-dimensional vector space. This vector contains the contextual information and feature information of the node in the graph. The 128-dimensional embedding space can accommodate rich information, allowing the model to distinguish different nodes and capture the similarities and differences between them. In addition, the 128-dimensional embedding dimension achieves a good balance between computational cost and model performance. If the embedding dimension is too low, the model may not be able to fully express the characteristics of the data, resulting in performance degradation; if the embedding dimension is too high, it will increase the computational complexity and storage requirements of the model, and may also lead to overfitting.

[0129] To this end, a multi-layer teacher model using ReLU activation and an embedding dimension of 128 can learn rich graph structure knowledge and feature representations. This knowledge can be effectively transferred to the student model through knowledge distillation techniques. During training, the student model can imitate the teacher model's multi-layer feature extraction process, nonlinear transformations, and embedding representations, achieving performance close to that of the teacher model.

[0130] Optionally, the knowledge distillation technology also includes matching intermediate layer features of the student model and the teacher model, and using mean square error as the intermediate layer distillation loss.

[0131] In the above-described embodiments of the present application, the intermediate-layer features of the teacher model contain rich implicit information learned during training. This information not only contributes to the final prediction results but also involves the transformation and abstraction of data at different levels. By matching the intermediate-layer features, the student model can capture this implicit knowledge, thereby more comprehensively learning the internal workings of the teacher model, rather than simply imitating the final output. The intermediate-layer features reflect the model's feature extraction and understanding of the input data at different stages. By learning the intermediate-layer features of the teacher model, the student model can gain a deeper understanding of the data's feature representation, helping to improve the student's ability to capture the data's inherent structure and patterns, thereby enhancing the model's performance in various tasks. Compared to using only the distillation loss of the final output layer, the intermediate-layer distillation loss provides a more detailed supervisory signal. During training, this additional supervisory information can help the student model converge faster, reducing training time and data requirements. Because the intermediate-layer features can be viewed as "intermediate results" of the model at different stages, matching these results can guide the student model to achieve the performance of the teacher model as closely as possible at each layer. The intermediate-layer distillation loss also acts as a regularizer, helping to stabilize the student model's training process. It can prevent the student model from overfitting or gradient instability in the early stages of training, making the model parameter update smoother and more reasonable.

[0132] While existing GNN-to-GNN distillation methods designed for GNNs have achieved remarkable results in improving model performance, they still have some limitations when working with graph-structured data. For one thing, the sparsity of graph-structured data can lead to insufficient information transfer during the distillation process. Furthermore, traditional graph knowledge distillation techniques often rely heavily on data and may not fully capture all features and relationships in graph-structured data, thus limiting the potential for improving model performance.

[0133] By applying the technical solution of this embodiment, we generate richer augmented data by linearly interpolating node features and labels while preserving the structural properties of the graph. Using this generated graph structure data, we implement knowledge distillation by aligning the output logarithmic probabilities between the pre-trained teacher and student graph neural network models. This integrates the original graph structure data with the graph structure data generated by Mixup, achieving comprehensive supervised knowledge transfer and improving the representation and generalization capabilities of the student model.

[0134] Further, as Figure 1 The specific implementation of the method, the embodiment of the present application provides a graph neural network performance optimization device, such as Figure 3 As shown, the device includes:

[0135] The original graph structure data acquisition module 201 is configured to acquire original graph structure data for training a graph neural network, wherein the original graph structure data comprises a plurality of graph nodes, the graph nodes correspond to graph node features and labels, the graph node features represent attribute information of the graph nodes, and the labels represent categories to which the graph nodes belong.

[0136] The original graph structure data enhancement module 202 is configured to perform linear interpolation on the graph node features and the labels in the original graph structure data respectively to generate enhanced graph structure data.

[0137] The teacher model construction module 203 is configured to construct a teacher graph neural network as a teacher model, train the teacher model based on the enhanced graph structure data, and acquire soft labels predicted by the teacher model for the enhanced graph structure data in a training process, wherein the soft labels represent prediction probability distributions of the teacher model for categories to which graph nodes in the enhanced graph structure data belong.

[0138] The student model construction module 204 is configured to construct a student graph neural network as a student model, migrate the soft labels to the student model by using a knowledge distillation technology, and enable the student model to learn prediction ability of the teacher model, wherein when the soft labels are migrated to the student model by using the knowledge distillation technology, a knowledge distillation loss of the student model for the enhanced graph structure data is acquired, and the student model learns the soft label knowledge from the teacher model by minimizing the knowledge distillation loss.

[0139] The student model optimization training module 205 is configured to train the student model based on the enhanced graph structure data, and simultaneously guide the student model by using a cross-entropy loss and a knowledge distillation loss driven by the knowledge distillation technology until an optimal student model is trained, wherein the knowledge distillation loss is used to measure differences between soft labels predicted by the student model and the teacher model for the enhanced graph structure data, and the cross-entropy loss is used to measure differences between prediction labels predicted by the student model for the enhanced graph structure data and real labels in the enhanced graph structure data.

[0140] It should be noted that other corresponding descriptions of the functions of the graph neural network performance optimization device provided in the embodiments of the present application can be referred to the corresponding descriptions in the method, which will not be described here. Figures 1 to 2

[0141] Based on the method as shown in Figures 1 to 2 Accordingly, the embodiments of the present application also provide a storage medium having a computer program stored thereon, and the computer program is executed by a processor to implement the graph neural network performance optimization method as shown in Figures 1 to 2

[0142] ​​Based on such understanding, the technical scheme of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a U disk, a mobile hard disk, etc.), and includes a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the method described in various implementation scenarios of the present application.

[0143] Based on the method as shown in Figures 1 to 2 and the virtual device embodiment as shown in Figure 3 In order to achieve the above-mentioned purposes, the embodiments of the present application also provide a computer device, which can be a personal computer, a server, a network device, etc., and the computer device includes a storage medium and a processor; the storage medium is used to store a computer program; and the processor is used to execute the computer program to implement the graph neural network performance optimization method as shown in Figures 1 to 2 .

[0144] Optionally, the computer device can further include a user interface, a network interface, a camera, a radio frequency (RF) circuit, a sensor, an audio circuit, a WI-FI module, etc. The user interface can include a display screen, an input unit such as a keyboard, etc. The optional user interface can further include a USB interface, a card reader interface, etc. The network interface can optionally include a standard wired interface, a wireless interface (such as a Bluetooth interface, a WI-FI interface), etc.

[0145] Those skilled in the art can understand that the computer device structure provided by the embodiments does not constitute a limitation on the computer device, and can include more or fewer components, or combine certain components, or different component arrangements.

[0146] The storage medium can further include an operating system and a network communication module. The operating system is a program for managing and saving computer device hardware and software resources, and supports the running of information processing programs and other software and / or programs. The network communication module is used to realize the communication between the components in the storage medium, and the communication with other hardware and software in the entity device.

[0147] Through the description of the above implementation methods, those skilled in the art can clearly understand that the present application can be implemented by means of software plus the necessary general hardware platform, or by means of hardware to perform linear interpolation on the graph node features and labels in the original graph structure data to generate enhanced graph structure data; construct a teacher graph neural network as a teacher model, and obtain the soft labels predicted by the teacher model for the enhanced graph structure data; construct a student graph neural network as a student model, use knowledge distillation technology to migrate the soft labels to the student model, train the student model based on the enhanced graph structure data, and use cross entropy loss and knowledge distillation loss driven by knowledge distillation technology to guide the student model at the same time until the optimal student model is trained. By performing linear interpolation on node features and labels, while retaining the original graph topology, a variety of training samples are generated, which can improve the prediction performance of the student graph neural network.

[0148] Those skilled in the art will understand that the accompanying drawings are only schematic diagrams of a preferred implementation scenario, and the modules or processes in the accompanying drawings are not necessarily required to implement the present application. Those skilled in the art will understand that the modules in the devices in the implementation scenario can be distributed in the devices of the implementation scenario according to the implementation scenario description, or can be changed accordingly and located in one or more devices different from the implementation scenario. The modules of the above-mentioned implementation scenario can be combined into one module, or can be further split into multiple sub-modules.

[0149] The serial numbers of the above application are for descriptive purposes only and do not represent the advantages or disadvantages of the implementation scenarios. The above disclosures are only a few specific implementation scenarios of the present application, but the present application is not limited thereto, and any changes that can be made by those skilled in the art should fall within the scope of protection of the present application.

Claims

1. A method for optimizing performance of a graph neural network, the method comprising: The method comprises: obtaining original graph structure data for training a graph neural network, wherein the original graph structure data comprises a plurality of graph nodes, the graph nodes correspond to graph node features and labels, the graph node features represent attribute information of the graph nodes, and the labels represent categories to which the graph nodes belong; performing linear interpolation on the graph node features and the labels in the original graph structure data respectively to generate enhanced graph structure data; constructing a teacher graph neural network as a teacher model, training the teacher model based on the enhanced graph structure data, and obtaining soft labels predicted by the teacher model for the enhanced graph structure data during the training, wherein the soft labels represent prediction probability distributions of the teacher model for categories to which each graph node in the enhanced graph structure data belongs; constructing a student graph neural network as a student model, migrating the soft labels to the student model by using a knowledge distillation technology, and enabling the student model to learn the prediction capability of the teacher model, wherein when the knowledge distillation technology is used to migrate the soft labels to the student model, a knowledge distillation loss of the student model for the enhanced graph structure data is obtained, and the student model learns the soft label knowledge from the teacher model by minimizing the knowledge distillation loss; training the student model based on the enhanced graph structure data, and simultaneously guiding the student model by using a cross-entropy loss and a knowledge distillation loss driven by the knowledge distillation technology until an optimal student model is trained, wherein the knowledge distillation loss is used to measure differences between the soft labels predicted by the student model and the teacher model for the enhanced graph structure data, and the cross-entropy loss is used to measure differences between prediction labels predicted by the student model for the enhanced graph structure data and real labels in the enhanced graph structure data.

2. The method of claim 1, wherein, The method comprises: determining a graph node pair based on any two graph nodes in the original graph structure data, performing linear interpolation on the graph node features of the two graph nodes in the graph node pair by using a node feature linear interpolation formula to obtain virtual graph node features linearly interpolated based on the graph node pair; performing linear interpolation on the labels of the two graph nodes in the graph node pair by using a label linear interpolation formula to obtain virtual labels linearly interpolated based on the graph node pair; obtaining the enhanced graph structure data based on the virtual graph node features and the virtual labels linearly interpolated by each graph node pair, wherein the virtual labels in the enhanced graph structure data are used as real labels to train the graph neural network, and the virtual graph node features and the virtual labels are represented as: and are virtual graph node features and virtual labels, respectively, x i and x j are two graph node features in the original graph structure data, respectively, y i and y j are the respective labels of x i and x j , and λ is a random number between 0 and 1, subject to Beta distribution Beta(α, α), λ is used to represent the weight parameter of linear interpolation, and α is a parameter for controlling the shape of the distribution of λ.

3. The method of claim 2, wherein, each virtual graph node feature corresponds to a virtual graph node, and the training of the teacher model based on the enhanced graph structure data comprises: when the teacher model is trained based on the enhanced graph structure data, summing and averaging cross-entropy losses of each virtual graph node in the enhanced graph structure data by using an average cross-entropy loss function to obtain an average cross-entropy loss, and guiding the training of the teacher model based on the average cross-entropy loss, wherein the average cross-entropy loss function is: L T is the average cross-entropy loss for the teacher model, CE is the cross-entropy loss function, N is the total number of virtual graph nodes in the augmented graph structure data, is the virtual graph node feature, is the embedding representation predicted by the teacher model T for the virtual graph node feature is the embedding representation predicted by the teacher model T for the virtual graph node feature is the virtual graph node feature, is the corresponding virtual label, is used to calculate the cross-entropy loss between the embedding representation predicted by the teacher model T and the real label.

4. The method of claim 1, wherein, the training of the student model based on the enhanced graph structure data comprises: In the process of training the student model based on the enhanced graph structure data, the average KL divergence loss of the student model on the enhanced graph structure data is calculated, and the calculated average KL divergence loss is used as the knowledge distillation loss. The training of the student model is guided by minimizing the knowledge distillation loss, wherein the average KL divergence loss is used to measure the difference between the embedding representation predicted by the student model for the enhanced graph structure data and the node embedding representation predicted by the teacher model for the same data. The calculation formula of the knowledge distillation loss is: is a knowledge distillation loss for the student model S, is an embedding representation predicted by the student model S for a virtual graph node feature is an embedding representation predicted by the student model S for a virtual graph node feature is an embedding representation predicted by the teacher model T for a virtual graph node feature is an embedding representation predicted by the teacher model T for a virtual graph node feature, L KL is a KL divergence (Kullback-Leibler divergence) loss function, and N is the total number of virtual graph nodes in the augmented graph structure data.

5. The method of claim 1, wherein, The cross entropy loss and the knowledge distillation loss driven by the knowledge distillation technology guide the student model simultaneously until the optimal student model is trained, including: Combining the cross entropy loss and the knowledge distillation loss driven by the knowledge distillation technology, an optimization loss is constructed. The optimization loss is minimized in the process of training the student model based on the enhanced graph structure data until the optimal student model is obtained. The optimization loss is: L s To optimize the loss, β is the weight coefficient for balancing the cross-entropy loss and the knowledge distillation loss .

6. The method according to any one of claims 1 to 5, characterized in that, The teacher model and the student model are respectively multi-layered, each layer uses a ReLU activation function, and the embedding dimension is 128.

7. The method of claim 6, wherein, The knowledge distillation technology also includes matching the intermediate layer features of the student model and the teacher model, and using the mean square error as the intermediate layer distillation loss.

8. A graph neural network performance optimization apparatus, comprising: The device comprises: An original graph structure data acquisition module is used to acquire original graph structure data for training a graph neural network, wherein the original graph structure data includes multiple graph nodes, each of which has a graph node feature and a label. The graph node feature represents the attribute information of the graph node, and the label represents the category to which the graph node belongs. The original graph structure data enhancement module is used to perform linear interpolation on the graph node features and labels in the original graph structure data to generate enhanced graph structure data; A teacher model construction module is used to construct a teacher graph neural network as a teacher model, train the teacher model based on the enhanced graph structure data, and obtain soft labels predicted by the teacher model for the enhanced graph structure data during training, wherein the soft labels are used to represent the teacher model's predicted probability distribution of the category to which each graph node in the enhanced graph structure data belongs; The student model construction module is used to construct a student graph neural network as the student model and use knowledge distillation technology to migrate soft labels to the student model so that the student model can learn the predictive ability of the teacher model. When using knowledge distillation technology to migrate soft labels to the student model, the knowledge distillation loss of the student model for the enhanced graph structure data is obtained. By minimizing the knowledge distillation loss, the student model learns the soft label knowledge from the teacher model. The student model optimization training module is used to train the student model based on the enhanced graph structure data. It uses the cross entropy loss and the knowledge distillation loss driven by the knowledge distillation technology to guide the student model at the same time until the optimal student model is trained. Among them, the knowledge distillation loss is used to measure the difference between the soft labels predicted by the student model and the teacher model for the enhanced graph structure data, and the cross entropy loss is used to measure the difference between the predicted labels predicted by the student model for the enhanced graph structure data and the true labels in the enhanced graph structure data.

9. A storage medium having stored thereon a computer program, characterized in that The computer program, when executed by a processor, implements the method for optimizing performance of a graph neural network according to any one of claims 1 to 7.

10. A computer device comprising a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, characterized in that, The processor, when executing the computer program, implements the method for optimizing performance of a graph neural network according to any one of claims 1 to 7.

Images