A graph incremental learning method based on denoising efficient graph compression
By employing methods of denoising, efficient graph compression, and balanced training, the problems of data imbalance and forgetting in incremental graph learning are solved, improving the model's generalization ability and stability while reducing computational resource consumption and enhancing privacy protection.
Patent Information
- Application Number
- CN202411743520.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-30
- Publication Date
- 2025-11-21
- Estimated Expiration
- 2044-11-30
AI Technical Summary
Existing graph incremental learning methods face the problems of data imbalance and forgetting in dynamic graph environments. In particular, replay-based methods have difficulty maintaining knowledge of old tasks during training and require a large amount of replay memory.
We employ a noise reduction and efficient graph compression technique to compress the new task graph into a smaller graph. We then generate a synthetic graph using relevant entropy regularization loss and complementary loss to reduce the impact of noise. Simultaneously, we perform balanced training in the memory library to ensure that the model learns a balanced learning curve for both new and old tasks.
It effectively solves the problem of data imbalance, improves the generalization and stability of the model, maintains model performance, reduces computing resource consumption, and enhances privacy protection capabilities.
Smart Images
Figure CN119600364B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application belongs to the field of pattern recognition and machine learning, and particularly relates to graph network learning and incremental learning technology, and in particular to a graph incremental learning method based on denoising efficient graph compression. BACKGROUND
[0002] In recent years, with the rapid development of artificial intelligence technology, graph neural networks have played an important role in social network analysis, bioinformatics, knowledge graph construction and transportation planning as a powerful modeling tool. For example, in social networks, GNN can efficiently capture the relationship between users and optimize the recommendation system; in bioinformatics, GNN is used to predict protein interactions; in knowledge graphs, GNN can mine potential relationships and perform knowledge reasoning. However, the success of traditional graph neural networks is due to their excellent modeling ability for static graph data, but their performance in dynamic graph environments is limited. Therefore, there is an urgent need for an effective technology to adapt to changing new data and knowledge.
[0003] Incremental learning as an important machine learning method, with lower computational and storage overhead, realizes the adaptation of the model to new data while retaining the ability of historical task knowledge, making incremental learning an important research direction of machine learning. Therefore, graph incremental learning has attracted more and more attention.
[0004] With the continuous development of the Internet and information technology, massive graph data is widely used in various practical scenarios. In social networks, user interactions form dynamic graphs, which can be used to recommend friends and identify fake accounts; in bioinformatics, complex molecular structures and gene relationships can also be modeled as graphs for drug research and disease diagnosis; in urban transportation, road networks and their dynamically changing attribute data are crucial for path planning and traffic prediction. Many graph incremental learning methods have emerged to cope with the characteristics of evolving graph data. Recent research has proposed three different methods: replay-based methods, regularization-based methods, and network architecture modification-based methods. Replay-based methods address the forgetting problem by retaining representative data from previous tasks and training the model with new data. Regularization-based methods prevent the model from over-adjusting previously learned knowledge when learning new tasks by adding additional constraints to the loss function. Network architecture modification-based methods face the forgetting problem by modifying the specific architecture, activation function or algorithm layer of the network.
[0005] While replay-based methods are considered the most intuitive and effective incremental graph learning strategies, they still face several challenges in practical applications. First, when replay space is limited, these replay data methods struggle to represent the distribution of historical data. This is because current replay methods often focus on using only a portion of the graph data to build memory, i.e., sampling and storing only a single node or a portion of the subgraph for replay. Second, due to memory constraints, new task graph data typically dominates the training data, leading to a data imbalance between replay data and new task graph data. When using this imbalanced data to update the graph neural network model, the model may become biased towards the current task data, resulting in the forgetting of knowledge from previously learned tasks. Summary of the Invention
[0006] To address the problems of existing methods, this invention proposes a novel graph incremental learning method based on playback. Graph compression effectively reduces training data from large graphs to smaller graphs while maintaining model performance. The compressed graph in this invention maintains similar distribution characteristics to the original large graph by learning a small graph composed of synthetic nodes. During the generation of the compressed graph, this invention also introduces a relevance entropy regularization loss to reduce the impact of noise during the compression process. This compressed graph effectively captures the overall structure and attribute features of the original large graph. Furthermore, the compressed graph is composed of new nodes, which not only preserves the feature distribution of the original data but also improves the privacy protection capability of the graph data to a certain extent. To address the problem of imbalance between the scale of new task data and playback data during training, when new task data arrives, we compress the new task data and playback data to the same scale for training. The novel graph incremental learning training method proposed in this invention can effectively solve the data imbalance problem and enhance the generalization ability of the model.
[0007] The technical solution steps of the present invention are as follows:
[0008] Step 1: During the training phase, the graph of the new task is first compressed into a smaller graph using a noise reduction and efficient graph compression technique. The influence of noise and outliers in the graph dataset is mitigated using a relevant entropy flow loss, and a complementary loss is used to make the distribution of the compressed graph approximate the distribution of the new task graph. During this process, small-scale graphs that retain the original graph information are obtained through the constraints of the two losses. Then, the compressed graph is placed into a memory database, and compression is performed on all tasks within the memory database. Figure One The final model is obtained through training.
[0009] Step 2, the testing phase: for the test nodes, the final classification result is obtained through the network.
[0010] Step 1 includes the following steps:
[0011] Step 1.1: Incremental segmentation of the dataset by class:
[0012] Step 1.2, denoising efficient graph compression of the new task to obtain a compressed graph:
[0013] Step 1.3, placing the compressed graph of the new task into the memory library:
[0014] Step 1.4, using the compressed graph of the memory library to balance train the model:
[0015] The step 2 includes the following steps:
[0016] Step 2.1, inputting a test data set for testing DETAILED DESCRIPTION
[0017] Class-incremental segmentation processing of the data set: in order to be closer to the scattered data flow in the incremental learning scene, the present application divides the graph data set into different sub-data sets according to the categories, and each sub-data set represents a sub-task at different time. For the CoraFull data set, it has a graph data set of 70 category nodes, and then the data set is divided according to that each task contains two categories. The present application uses three data sets for experiments, CoraFull, OGB-Arxiv, Reddit and OGB-Products data sets. For the four data sets, each task contains two categories, and the categories of different tasks do not overlap.
[0018] Denoising efficient graph compression method: among the three major graph incremental learning methods, the replay-based method has better anti-forgetting ability, because it retains the data of the old task, and when learning the new task, the new task and the old task are trained together to effectively improve the plasticity and stability of the model. Although this method well solves the forgetting problem, in practical application, the replay method based on sampling needs a large replay memory to remember old knowledge, and also produces the problem of imbalance between the scale of new task data and replay data. Compared with using the sampling replay method, using the compressed replay method can significantly improve the model performance.
[0019] Data set compression technology has achieved remarkable results in the field of computer vision. By compressing the size of the image data set, the computer vision model can maintain high recognition accuracy and robustness while reducing the consumption of computing resources. The purpose of graph compression is to combine large graphs Compressing into a small graph to reduce the training cost, and can achieve the same test accuracy as the model trained on the entire
[0020] Data set, and the compression target is as follows:
[0021]
[0022] where is a loss function related to the task, and θ is the parameter of GNN.
[0023] The present application proposes a new graph compression method, called denoising efficient graph compression. The method of the present application synthesizes synthetic data close to the distribution of the real training data based on embedding representation. However, it is difficult to accurately estimate the distribution of the data set due to the high dimension of the embedding representation. Therefore, a series of randomly initialized neural networks are used to obtain different embedding representations to represent the training data set, and each individual network provides a partial explanation of the data set. The set of randomly initialized networks produces a more comprehensive representation. In practical applications, there is often some noise and outliers in graph data. Therefore, the present application adds a related entropy regularization loss to suppress the influence of noise.
[0024] For a task T k , the incoming graph is The desired synthetic graph is generated by graph compression The initial synthetic graph can be obtained by random noise, random sampling or clustering algorithm. In the distribution matching scheme, the objective function of graph compression can be represented as:
[0025]
[0026] where the Dist(·,·) function is used to calculate the distance between the original graph and the compressed graph, and the two graphs are encoded by the same graph encoder GNN θ The distance is measured in the embedding space. Specifically, the optimal compressed graph should be as close as possible to the original graph in distribution.
[0027]
[0028] The noise and outliers of the real graph data will affect the estimation of the distribution of the real data. Therefore, the present application introduces a related entropy loss when generating the compressed graph, and uses the current data set to construct a predefined supervised neighbor graph G s , where the element in the ith row and jth column is defined as:
[0029]
[0030] where represents the sample x j The set of homogeneous second-order neighbors, G s The construction of G not only considers the local relationship, but also uses the label information. By giving the training sample set and the newly constructed adjacency matrix, the related entropy regularization is represented as:
[0031]
[0032] The correlation entropy loss adopts a correlation entropy measure instead of a traditional Euclidean distance to measure the distance between the output responses of a pair of samples. i j The range of the correlation entropy measure is 0 to 1, while the range of the Euclidean distance is the entire real number space. In a noisy environment, if there is an abnormal point in a sample pair, the Euclidean distance measure value can be very large, and minimizing this regular term can cause the model to pay excessive attention to abnormal points. Due to the difference in the value range, the correlation entropy loss can improve the generalization ability and robustness of the model to some extent, especially when the sample pair contains noise or abnormal points. In order to enhance the similarity between the new task graph and the synthesized graph in the embedding space, this paper also introduces a complementary loss as a regularization term, which uses the well-known maximum mean difference loss to calculate the distribution distance between the two graphs.
[0033]
[0034] where is the set of node classes in the graph , E k,c and are the embedding matrices of the input graph and the compressed graph respectively, c k is the label of the k-class node, is the class ratio of c k classes. |·| is the number of rows in the matrix, and Mean(·) is the average vector of node embedding.
[0035] Finally, the invention uses the stochastic gradient descent method to complete the following optimization problem to learn the compressed dataset:
[0036]
[0037] where λ is the loss balance parameter, and the invention assigns a fixed label to each synthesized node, which remains unchanged during training.
[0038] Memory bank update: put the compressed graph of the new task into the memory bank
[0039]
[0040] Balanced training: in graph incremental learning, the number of nodes in the new task is much larger than the number of nodes in the replay data, which can cause class imbalance problems during training. This imbalance problem can cause the model to tend to learn the knowledge of the new task and easily forget the knowledge of the old task. Previous methods have focused on scaling the loss function coefficients, which generally take the following form:
[0041]
[0042] In this formula, most methods focus on optimizing the parameters a and b, which will inevitably affect the overall performance of the model. The present application compresses the graph of the new task without directly using the entire new task graph for training. Specifically, the new task graph is compressed into small graphs and saved to the replay memory, and finally the model is updated using the data in the replay memory.
[0043]
[0044] Network test: input test data set, four data sets are divided into multiple tasks, each task is composed of two different classes in the data set. For each task, 60% of the nodes in the data set are used for the training set, 20% of the nodes are used for the validation set, and the remaining 20% of the nodes are used for the test set. The budget ratio represents the proportion of the number of nodes in the replay memory to the total number of nodes in the entire training set, and the budget is evenly distributed in each task. By default, the budget ratio of the joint baseline is 1, because it stores all the training data in memory. While for other replay-based methods, the default budget ratio is 0.01, which means the size of the replay memory is only 1% of the entire training data. BRIEF DESCRIPTION OF DRAWINGS
[0045] Figure 1 The specific implementation flowchart of the present application.
[0046] Figure 2 The precision matrix diagram of the method of the present application on four data sets. DETAILED DESCRIPTION
[0048] Table 1 is the experimental result of graph class incremental learning on four data sets. Compared with different graph incremental learning methods, the final performance is measured by average accuracy (AA) and average forgetting rate (AF).
[0049] Table 1 Experimental results on four data sets under class incremental setting
[0050]
[0051]
[0052] Table 2 is the ablation study result on the CoraFull data set and the OGB-Arxiv data set.
[0053] Table 2 Ablation experiment
[0054]
[0055] Quantitative evaluation: Table 1 shows the quantitative results on four data sets, AA average accuracy represents the average classification accuracy of the model after learning all tasks, which is used to evaluate the learning ability of the model; AF average forgetting rate represents the average forgetting degree of the model to each task, which is used to evaluate the anti-forgetting ability of the model. DEGC represents the method proposed in the application, from the table, it can be seen that DEGC performs significantly on four data sets, and the average accuracy is 66.1%, 67.1%, 97.9% and 71.9% respectively. In addition, DEGC surpasses the most advanced method Cat, and the accuracy is improved by 1.6%, 1.1%, 0.2% and 1.0%.
[0056] Table 2 shows the results of the ablation study on the CoraFull data set and the OGB-Arxiv data set, which are the correlation entropy loss module and the balanced training module. Not containing the correlation entropy loss module means not using the loss function L C Not containing the balanced training module Bmt means using the typical replay-based scheme, that is, using the entire input graph for training. It can be seen from Table 2 that both modules play a role and make a good performance improvement. Careful observation can find that the correlation entropy loss works on both Corafull and Arxiv data sets, indicating that there are noise points in the real graph data, and using the correlation entropy loss can eliminate the influence of these abnormal points and noise in the process of compressing the graph data. When using the balanced training method, the performance of the model is significantly improved. This shows that there is a problem of data imbalance in the data set, and the traditional training method may not fully consider the difference in data distribution, while the balanced training ensures that the model can learn the data of different tasks sufficiently, enhancing the generalization performance of the model.
[0057] Figure 2 The precision matrix of the method of the application on four data sets is given to help understand the learning process in depth. The colored block in the pth row and qth column represents the classification accuracy of task T p after training the model on T q Each row of the task matrix represents the change of the classification performance of the learned task after the model learns a new task. Each column shows the classification performance change process of a specific task in the order of all task learning. Light color represents high classification performance, and dark color represents low classification performance. During the entire training process, DEGC shows significant stability, and the accuracy of each task decreases by the lowest.
Claims
1. A graph incremental learning method based on denoising and efficient graph compression, characterized in that: Step 1: During the training phase, the graph of the new task is first compressed into a smaller graph using a noise reduction and efficient graph compression technique. The influence of noise points and outliers in the graph dataset is mitigated by the relevant entropy flow loss. The distribution of the compressed graph is made closer to the distribution of the graph of the new task by the complementary loss. During this period, a small-scale graph that can retain the information of the original graph is obtained by constraining the two losses. Then, the compressed graph is put into the memory library, and the final model is obtained by training together with the compressed graphs of all tasks in the memory library. Step 2, Testing Phase: For test nodes, the final classification result is obtained through the model; For task T k The input image is Generate the desired composite image through image compression. The initial synthetic graph is obtained through random noise, random sampling, or clustering algorithms; under the distribution matching scheme, the objective function for graph compression is expressed as: The Dist(·,·) function is used to calculate the distance between the original image and the compressed image. Both images are encoded by the same graph encoder GNN. θ Encoding, distance measured in the embedding space; optimal compressed graph The distribution should be as close as possible to the original image; Noise and outliers in real graph data can affect the estimation of the real data distribution. Therefore, a correlation entropy loss is introduced during the compressed graph generation process, and a predefined supervised nearest neighbor graph G is constructed using the current dataset. s The element in the i-th row and j-th column is defined as: in Indicates sample x j G is a set of second-order neighbors of the same kind. s The construction not only considers local relations but also utilizes label information; given a set of training samples and a newly constructed adjacency matrix, the relevance entropy regularization is expressed as: The correlation entropy loss uses correlation entropy as a metric, rather than the traditional Euclidean distance, to measure the distance between paired sample output responses; for second-order neighbor sample pairs (x i ,x j The range of the relevant entropy metric is from 0 to 1, while the range of the Euclidean distance is the entire real number space. In noisy environments, if outliers exist in a sample pair, the Euclidean distance metric may be very large, and minimizing this regularization term may cause the model to overemphasize outliers. Due to the different value ranges, the correlation entropy loss can improve the generalization ability and robustness of the model to a certain extent, especially when the sample pair contains noise or outliers. To enhance the similarity between the new task graph and the synthetic graph in the embedding space, a complementarity loss is introduced as a regularization term, and the well-known maximum average difference loss is used to calculate the distribution distance between the two graphs. in It is a picture A collection of node classes, E k,c and These are the embedding matrices of the input graph and the compressed graph, respectively, c k It is the label of the k-category node. It is C k The class ratio of the class; |·| is the row number in the matrix, and Mean(·) is the average vector of node embeddings; Finally, we use stochastic gradient descent to solve the following optimization problem to learn how to compress the dataset: Where λ is the loss balance parameter, which assigns a fixed label to each synthetic node and remains unchanged during training; Balanced training is used to address the problem of imbalance between new and old data during model training. When the size of the graph for a new task is much larger than the size of the graph in the replay library, it is difficult for the model to learn knowledge from the historical graph and the incoming graph in a balanced way. The new task graph is compressed and put into the memory library, and the compressed graphs of all tasks in the memory library are used to train the model.
2. The graph incremental learning method based on denoising and efficient graph compression according to claim 1, characterized in that: Step 1 includes the following steps: Step 1.1: Incremental segmentation of the dataset by class: Step 1.2: Perform noise reduction and efficient image compression on the new task to obtain a compressed image: Step 1.3: Add the compressed image of the new task to the memory library: Step 1.4: Perform balanced training on the model using the compressed graph from the memory library: Step 2.1: Input the test dataset for testing.
3. The graph incremental learning method based on denoising and efficient graph compression according to claim 1, characterized in that: The network architecture is implemented using PyTorch. For the first graph compression stage, which is the efficient denoising and graph compression part, it is trained for 200 epochs with an initial learning rate of 10. -3 Every 20 rounds, the learning rate was multiplied by 0.95; the selected GNN encoder was a GCN, and all encoders had a 256-dimensional hidden layer and a 128-dimensional output layer; during training the prediction model, this stage involved 200 rounds of training with an initial learning rate of 10. -4 The learning rate is multiplied by 0.9 every 20 rounds. The selected GNN encoder is GCN, and the hidden layer dimension is 256.