Multi-teacher self-training method based on noise label semi-supervised node classification

Abstract

A graph neural network (GNN) obtains a favorite result in a semi-supervised learning task on graph structure data. However, most existing methods assume that training data have correct labels, but in the real world, graph structure data often have noise labels so as to reduce the effectiveness of GNN. In order to solve the problem, the invention provides a new label correction method which is called a multi-teacher self-training method for semi-supervised node classification based on noise labels, and semi-supervised node classification is carried out on noisy labels. Specifically, firstly, model training parameters in early iteration are stored as a teacher model, and then in later iteration of a training process of semi-supervised node classification, the teacher model is used for guiding the processes of model training, noise label removal, pseudo label selection and the like. Therefore, based on the guidance of a teacher model, the provided method realizes the effectiveness of the model by solving the overfitting problem, and improves the accuracy of noise label removal and the quality of pseudo label selection. Wide experimental results on a real data set show that compared with the most advanced method at present, the method achieves the best effect.

Description

Technical Field

[0001] The present invention belongs to the technical field of data analysis, and more specifically, relates to a multi-teacher self-training method for semi-supervised node classification based on noise labels. Background Art

[0002] Graph neural networks have been widely used in practical applications for tasks such as node classification, graph classification, link prediction, and graph clustering, such as traffic networks, social networks, and molecular protein interface prediction. The success of graph neural networks lies in that they are usually designed to learn representations of graph-structured data and to preserve the local topological structure of these graph-structured data in the context of semi-supervised learning. In practical applications, it is easy to obtain a large amount of unlabeled data, but it is difficult to obtain sufficient labeled data. In addition, some labels usually contain noise. Therefore, noisy labels can affect the effectiveness of graph neural networks. In particular, graph neural networks will incorrectly aggregate information from noisy data to further degrade the quality of graph neural network models. Therefore, it is essential to deal with noisy labels to process graph-structured data.

[0003] Existing methods for dealing with noisy labels can be divided into two main categories, loss correction methods and sample selection methods. Loss correction methods reduce the adverse effects of noisy labels by correcting the training loss, such as first selecting several prototypes for each class, and then using the average of the similarities between the sample and these prototypes to correct the training loss. Or train two networks at the same time, first predict the labels and noise of all training samples, and then correct the training loss by using different loss functions for different samples. Sample selection methods aim to select clean samples during training. For example, train two networks at the same time and let them select clean data for each other. Or distinguish clean data from noisy data through three steps, namely negative learning, selective negative learning, and selective positive learning. Generally, these methods are not directly applied to deal with noisy labels on graph-structured data. First, they usually require a large amount of labeled data, while graphs are usually sparsely labeled, with only a small number of labeled nodes. Second, many existing noisy label works focus on independent and identically distributed data (such as images), but they have difficulty in handling information propagation of noisy labels on graph-structured data.

[0004] Recently, some methods have been proposed to address the noisy label problem on graph-structured data. For example, attempts have been made to reduce the misleading effect of noisy labels on the network by generating label weights. Or the supervisory information is enhanced by obtaining pseudo labels, thereby mitigating the impact of noisy labels. However, there are several problems that need to be addressed in existing methods: 1) Previous methods did not directly consider removing noisy labels, assigning pseudo labels to nodes with noisy labels, but retaining the original noisy labels, causing the model to acquire more and more erroneous information. 2) The quality of the selected pseudo labels is difficult to guarantee. If low-quality pseudo labels are selected, these methods will lead to misleading results and their classification accuracy will be greatly reduced. Therefore, it is crucial to simultaneously consider removing noisy labels and selecting high-quality pseudo labels for semi-supervised node classification with noisy labels.

[0005] Solving the above problems of previous methods is often challenging. On the one hand, graph-structured data are often sparsely labeled, with only limited labeled nodes. If the correct labels are accidentally removed, the supervision information is reduced, resulting in degraded model performance. Therefore, it is crucial to accurately select noisy labels. On the other hand, model effectiveness affects the selection of pseudo labels. If the model is affected by noisy labels and has poor generalization performance, it will be difficult to select high-quality pseudo labels. Therefore, selecting high-quality pseudo labels is challenging. Summary of the invention

[0006] The purpose of the present invention is to overcome the shortcomings of the prior art and propose a new method, namely a multi-teacher self-training method for semi-supervised node classification based on noisy labels, which is used to perform semi-supervised node classification on a data set with noisy labels, including three steps: model training, noisy label removal, and pseudo-label selection. To this end, we first perform a certain round of training process on semi-supervised node classification to make it contain generalization capabilities, and then regard these models as teacher models to guide the three steps of semi-supervised node classification in subsequent iterations of the training process. Specifically, in model training, we introduce multi-teacher guided losses to guide model learning, aiming to alleviate the overfitting problem. In noisy label removal, we first select suspicious nodes from the labeled nodes, and then use the teacher model to select noisy labels from the suspicious nodes. This effectively ensures the accuracy of noisy label selection. In pseudo-label selection, the knowledge of the teacher model in each iteration is accumulated to form multi-teacher predictions, which are used to guide the selection of pseudo-labels in the current iteration, thereby effectively improving the quality of the selected pseudo-labels.

[0007] Compared with the previous methods, the present invention has the following beneficial effects:

[0008] 1) The proposed method adopts a teacher model to guide the learning of model training, noisy label removal and pseudo-label selection for semi-supervised node classification with noisy labels, which solves the problems of previous methods.

[0009] 2) The present invention improves the quality of pseudo labels by correcting the pseudo labels, thereby improving the learning effect;

[0010] 3) To the best of our knowledge, this method is the first to perform noisy label removal on graph-structured data with noisy labels. Extensive experiments on four datasets demonstrate the effectiveness of this method in the semi-supervised node classification problem with noisy labels. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 It is a flowchart of a specific implementation method of the multi-teacher self-training method for semi-supervised node classification based on noise labels of the present invention;

[0012] Figure 2 It is a structural diagram of the module in the present invention; DETAILED DESCRIPTION

[0013] The specific implementation of the present invention is described below in conjunction with the accompanying drawings so that those skilled in the art can better understand the present invention. It should be noted that in the following description, when the detailed description of known functions and designs may dilute the main content of the present invention, these descriptions will be omitted here.

[0014] Example

[0015] Figure 1 1 is a flowchart of a specific implementation of the multi-teacher self-training method for semi-supervised node classification based on noise labels of the present invention. Figure 1 As shown, the specific steps of the multi-teacher self-training method for semi-supervised node classification based on noise labels of the present invention include:

[0016] S101: Train and save the teacher network:

[0017] The following formula is used to determine the embedding Z of all nodes in the i-th layer of the graph convolutional network: (i) And the probability matrix P:

[0018]

[0019] Among them, W (i) is the trainable parameter of the i-th layer, σ is the activation function, in particular, in I is the unit matrix, is a diagonal matrix, After obtaining the probability matrix P, the model is trained using the cross entropy loss, which is determined by the following formula:

[0020]

[0021] Where S represents the set of nodes with labels; after the training process converges, the node loss set l = {l1,l2,…,l s}, where l i represents the cross entropy loss of the i-th node. Next, determine the threshold θ of the loss value and distinguish the noise label from all labels by the following formula:

[0022]

[0023] Where sort(·) means sorting the vector in descending order. represents the upper limit function, β represents the percentage of deleted labels in the total number of labels; given θ, the labels of nodes with loss values ​​higher than θ are considered as noise labels and deleted;

[0024] After removing the noise labels, the prediction confidence of all unlabeled nodes is obtained through the probability matrix P, that is, h = {h1,h2,…,h u}, where h i =max(p i ), u is the number of unlabeled nodes; then, pseudo labels are assigned to the m nodes with the highest prediction confidence for each class by the following formula:

[0025]

[0026] Where k (k≤c) represents the kth class, τ k is the prediction confidence of the mth node in the kth class. Use arg max(·) to select the class with the highest prediction probability. After deleting and augmenting the label data, retrain the model on the updated dataset.

[0027] We then repeat the above process until all unlabeled data are labeled or a predetermined number of iterations is reached.

[0028] Compared with the loss correction methods commonly used in the noisy label problem, the above method has at least two advantages. First, the above method directly reduces the noise rate of the dataset by filtering and enhancing the labeled data, effectively reducing the impact of the noisy labels, and does not need to be indirectly processed through complex loss functions. Second, the above method can increase the supervision information in the dataset by selecting pseudo labels, thereby effectively improving the effectiveness of the model. Previous loss correction methods usually require a large amount of labeled data, so it is difficult to handle datasets with less label information.

[0029] S102: Using multi-teacher models to guide self-distillation:

[0030] Studies have shown that models can easily lead to overfitting problems on datasets with noisy labels. The reason is that traditional loss functions are not robust to noisy labels, for example, the cross entropy loss in the equation commonly used in classification tasks. In fact, the cross entropy loss can make the model prediction results close to the sample labels. However, if the sample contains an error label, the cross entropy loss will make the model prediction results close to the sample with incorrect labels, resulting in a decrease in generalization ability.

[0031] The present invention solves the overfitting problem by performing knowledge distillation during the model training process. The main idea of ​​knowledge distillation is to transfer knowledge from the teacher model to the student model to make the student model more effective. For example, the model parameters in the first few iterations are saved as teachers and used for multi-teacher self-distillation in subsequent iterations to guide model training in subsequent iterations. Specifically, the model is trained in early iterations using a cross entropy loss function. After the model gains generalization ability in several iterations, the model parameters are saved as a teacher model. In subsequent iterations of the training process, the prediction results of the teacher model are used as soft labels, and then the multi-teacher guided loss is defined, which is defined by the following formula:

[0032]

[0033] Where w represents the number of iterations at which multi-teacher self-distillation starts, t represents the current iteration, and T is the distillation temperature. Combining the multi-teacher guidance loss with the cross entropy loss, we get the multi-teacher self-distillation loss, which is defined by the following formula:

[0034] L ms =L ce +αL mg

[0035] Where α is a hyperparameter used to adjust the weight of the multi-teacher guidance loss in the total loss, and the multi-teacher self-distillation loss enables the model to learn knowledge from labeled nodes and models saved from previous iterations;

[0036] By using multi-teacher self-distillation loss to train the model, the overfitting problem is effectively alleviated.

[0037] S103: Using multiple teachers to guide noise label removal:

[0038] After the training process converges, the noisy labels are filtered out from the data set to reduce the erroneous label information. Existing methods do not consider removing noisy labels during the training process because it is difficult to accurately select noisy labels. To address this problem, the present invention proposes a noisy label selection method guided by multiple teachers to ensure the accuracy of noisy label selection.

[0039] Specifically, calculate the loss set l between the probability matrix P predicted by the current model and the node label set Y, and determine the threshold θce To select suspicious nodes, the selection method is defined by the following formula:

[0040]

[0041] Where ε is a hyperparameter that represents the percentage of suspicious nodes in the total number of labeled nodes, and the loss value is higher than θ ce Nodes are considered suspicious nodes;

[0042] Use the teacher model saved in each iteration of S101 to calculate the multi-teacher loss L of all suspicious nodes mt , defined by the following formula:

[0043]

[0044] Where N represents the set of suspicious nodes, the multi-teacher loss comprehensively considers the knowledge of all teacher models and allows the model saved in the previous iteration to provide guidance information for the selection of noisy labels in subsequent iterations; mt Represents the multi-teacher loss set and determines the threshold θ of the multi-teacher loss mt , the noise node is selected by the following formula:

[0045]

[0046] Where n is the number of suspicious nodes, is a hyperparameter indicating the percentage of selected noise nodes among suspicious nodes; the multi-teacher loss is higher than θ mt Nodes with are considered as noise nodes and their labels are removed;

[0047] The proposed multi-teacher guided noisy label selection effectively ensures the accuracy of noise sample selection and effectively reduces the wrong label information in the labeled dataset by removing the noisy labels.

[0048] S104: Using multi-teacher guided pseudo-label selection:

[0049] Define multi-teacher prediction δ to accumulate information provided by models trained in different iterations, as follows:

[0050] δ t =δ t-1 +t λ P (t)

[0051] Where t represents the current iteration, and λ is a hyperparameter; the weight of information is determined by adjusting λ; if λ>0, the influence of the model trained in each iteration on δ increases with the number of iterations, which indicates that we pay more attention to the information provided by the later model for selecting pseudo labels; if λ<0, the influence of the model trained in each iteration on δ decreases with the number of iterations, which means that we pay more attention to the information provided by the early model to select pseudo labels; if λ=0, δ just adds up all network predictions;

[0052] The pseudo labels are assigned to the m nodes with the highest prediction confidence for each class by the following formula:

[0053]

[0054] Where k (k≤c) represents the kth class, ρ k is the prediction confidence of the mth node in the kth class; secondly, by limiting the prediction confidence of the sample, avoid selecting samples with too low prediction confidence. Specifically, φk is used to represent the highest prediction confidence of the node of the prediction class k, and the selected pseudo-label node is required to satisfy the following formula:

[0055] max(k i )>φ argmax (k i )·μ

[0056] Where μ∈(0,1) is a hyperparameter that further ensures the quality of the pseudo-label. Finally, the pseudo-labels are assigned to the unlabeled nodes that satisfy both of the above formulas.

[0057] In the proposed multi-teacher guided label correction, our method improves the accuracy of selecting noisy labels through effective noisy label selection and improves the quality of pseudo labels by using multi-teacher predictions.

[0058] S105: Node classification using the final result:

[0059] Through the above steps, each node has been assigned a new label, forming a new graph data structure. This new graph can be used as a classification result and can also be used by downstream tasks.

[0060] In order to better illustrate the technical effect of the present invention, a specific example is used to experimentally verify the present invention. In this experimental verification, the effectiveness of the present invention is evaluated using four node classification benchmark datasets in the field of data analysis, including two citation datasets (i.e., Cora and Citeseer) and two commercial datasets (i.e., Computers and Photo).

[0061] Five comparison methods were set up in this experimental verification, including a baseline method (i.e., GCN), a semi-supervised node classification method (i.e., Self-training), and two methods for dealing with noisy label problems (i.e., Co-teaching and NRGNN).

[0062] This experiment verifies that different noise rates are added to the dataset. Specifically, given a noise rate r, the labels of the training set are corrupted with two types of noise: Uniform noise: There is a probability that the labels will be uniformly flipped to other classes. In the experiment, the noise rate is set to 20% and 40% to achieve uniform noise. Paired noise: It is assumed that the labels only make mistakes in the most similar pair of classes. More specifically, the labels have a probability of flipping to their paired classes. In the experiment, the noise rate of the pair noise is set to 20% and 40%. In the experiment, we randomly generate 10 noises for the training process and report the average results and the corresponding standard deviations in this article.

[0063] Table 1 is a statistical table of classification accuracy of the present invention and the comparative method under different noises in this embodiment.

[0064]

[0065] Table 1

[0066] As shown in Table 1, it can be seen from the results in Table 1 that the present invention achieves the best results under the two noise conditions of the four data sets, thereby verifying the effectiveness of the present invention.

[0067] Although the above describes the illustrative specific embodiments of the present invention to facilitate those skilled in the art to understand the present invention, it should be clear that the present invention is not limited to the scope of the specific embodiments. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations using the concept of the present invention are protected.

Claims

1. A multi-teacher self-training method for semi-supervised node classification based on noisy labels, characterized in that: The following steps are involved: S1: Using graph convolutional networks as the model backbone, semi-supervised node classification with noisy labels is performed, and the parameters trained in the early iterations of the model are saved as the teacher model. The specific method is: S1.1: The embedding Z of all nodes in the i-th layer of the graph convolutional network is determined using the following formula (i) And the probability matrix P: Among them, W (i) is the trainable parameter of the i-th layer, σ is the activation function, in particular, I is the unit matrix, is a diagonal matrix, S1.2: After obtaining the probability matrix P, the model is trained using the cross entropy loss, which is determined by the following formula: Where S represents the set of nodes with labels; after the training process converges, the node loss set l = {l1,l2,…,l s }, determine the threshold θ of the loss value, and distinguish the noise label from all labels by the following formula: Where sort(·) means sorting the vector in descending order. represents the upper limit function, β represents the percentage of deleted labels in the total number of labels; the labels of nodes with loss values ​​higher than θ are regarded as noise labels and deleted; S1.3: After removing the noise labels, the prediction confidence of all unlabeled nodes is obtained through the probability matrix P, that is, h = {h1,h2,…,h u }, where h i =max(p i ), u is the number of unlabeled nodes; then, pseudo labels are assigned to the m nodes with the highest prediction confidence for each class by the following formula: Where k (k≤c) represents the kth class, τ k is the prediction confidence of the mth node in the kth class. Use arg max(·) to select the class with the highest prediction probability. After deleting and augmenting the label data, retrain the model on the updated dataset. S1.4: Repeat the process from S1.1 to S1.3 until all unlabeled nodes are marked or the predetermined number of iterations is reached; save the model parameters of each iteration as the teacher model; S2: Use the teacher model of step S1 to guide multi-teacher self-distillation. In subsequent training, use a combination of cross entropy loss and multi-teacher guided loss to train the model; the multi-teacher guided loss uses the prediction results of the teacher model as soft labels and is defined by the following formula: Where w represents the number of iterations at which multi-teacher self-distillation starts, t represents the current iteration, and T is the distillation temperature. Combining the multi-teacher guidance loss with the cross entropy loss, we get the multi-teacher self-distillation loss, which is defined by the following formula: L ms =L ce +αL mg Where α is a hyperparameter used to adjust the weight of the multi-teacher guidance loss in the total loss, and the multi-teacher self-distillation loss enables the model to learn knowledge from labeled nodes and models saved from previous iterations; S3: Use the teacher model of step S1 to guide the removal of noise labels. The specific steps include: S3.1: Calculate the loss set l between the probability matrix P predicted by the current model and the node label set Y, and determine the threshold θ ce To select suspicious nodes, the selection method is defined by the following formula: Where ε is a hyperparameter that represents the percentage of suspicious nodes in the total number of labeled nodes, and the loss value is higher than θ ce Nodes are considered suspicious nodes; S3.2: Using the teacher model saved in each iteration of step S1, calculate the multi-teacher loss L of all suspicious nodes mt , defined by the following formula: Where N represents the set of suspicious nodes, the multi-teacher loss comprehensively considers the knowledge of all teacher models and allows the model saved in the previous iteration to provide guidance information for the selection of noisy labels in subsequent iterations; mt Represents the multi-teacher loss set and determines the threshold θ of the multi-teacher loss mt , the noise node is selected by the following formula: Where n is the number of suspicious nodes, is a hyperparameter indicating the percentage of selected noise nodes among suspicious nodes; the multi-teacher loss is higher than θ mt The nodes with are considered as noise nodes and their labels are removed; S4: Use the teacher model of step S1 to guide the selection of pseudo labels, thereby effectively improving the quality of the selected pseudo labels. The specific method is as follows: S4.1: Define multi-teacher prediction δ to accumulate information provided by models trained in different iterations, as follows: d t =d t-1 +t λ P (t) Where t represents the current iteration, and λ is a hyperparameter; the weight of information is determined by adjusting λ; if λ>0, the influence of the model trained in each iteration on δ increases with the number of iterations, which indicates that we pay more attention to the information provided by the later model for selecting pseudo labels; if λ<0, the influence of the model trained in each iteration on δ decreases with the number of iterations, which means that we pay more attention to the information provided by the early model to select pseudo labels; if λ=0, δ just adds up all network predictions; S4.2: Pseudo labels are assigned to the m nodes with the highest prediction confidence for each class by the following formula: Where k (k≤c) represents the kth class, ρ k is the prediction confidence of the mth node in the kth class; secondly, by limiting the prediction confidence of the sample, avoid selecting samples with too low prediction confidence. Specifically, φk is used to represent the highest prediction confidence of the node of the prediction class k, and the selected pseudo-label node is required to satisfy the following formula: max(k i )>φ argmax (k i )·m Among them, μ∈(0,1) is a hyperparameter, which further ensures the quality of pseudo labels. Finally, the pseudo labels are assigned to the unlabeled nodes that satisfy both of the above formulas; S5: Use the final output graph structure data to classify nodes and guide further data mining.

2. The multi-teacher self-training method for semi-supervised node classification based on noise labels according to claim 1 is characterized in that: Each class has m nodes with the highest prediction confidence, m≥10.

3. The multi-teacher self-training method for semi-supervised node classification based on noise labels according to claim 1, characterized in that: The percentage of the selected noise nodes in the suspicious nodes

Images