A Label Noise Estimation Method Based on Manifold Regularized Transfer Matrix

Abstract

A label noise estimation method based on a manifold regularization transfer matrix provided by the present invention pre-trains a first network in a second network, and after distilling a data set, inputs the obtained sub-data set into the second network to obtain the probability of the class to which the data instances in the sub-data set belong and obtain a transfer matrix related to the data instances; further calculates the cross-entropy loss of the second network according to the data instance labels, and combines an association matrix expressing the consistency of the data instances belonging to the same manifold and a penalty matrix of the data instances belonging to different manifolds to calculate the loss function of the second network; adjusts the loss function to reduce the training of the second network to obtain a trained second network, thereby completing the estimation of the class to which the data instances belong. The present invention can reduce the estimation error without affecting the approximation error of the transfer matrix, and experiments prove that the present invention can achieve excellent performance in label noise learning.

Description

Technical Field

[0001] The present invention belongs to the technical field of artificial intelligence, and particularly relates to a method for estimating label noise based on a manifold regularization transfer matrix. Background Art

[0002] Learning with noisy labels has received increasing attention in the field of deep learning. The main reason is that accurately annotating large-scale datasets is very expensive and often infeasible in many cases. An effective method is to collect such large-scale datasets from multiple platforms or through web crawlers, which inevitably generates low-quality and noisy data. Therefore, mitigating the side effects of noisy labels has become a very popular topic.

[0003] Methods for dealing with noisy labels can be divided into two categories: classifier algorithms with statistically inconsistent and consistent ones. In the first category of methods, the distribution of label noise is not explicitly modeled. This method usually uses some heuristic methods to reduce the side effects of label noise. Although these methods usually work well, the classifiers learned from noisy data are not statistically consistent, and their reliability cannot be guaranteed. While statistically consistent classifiers can solve this problem, in the algorithms of statistically consistent classifiers, learning the transfer matrix plays an important role in constructing a statistically consistent classifier for learning with noisy labels. Using the learned transfer matrix can explicitly model the generation process of noisy labels.

[0004] However, in the process of explicitly modeling the noisy label using the learned transfer matrix, the existing technology only uses the classifier learned from noisy data to obtain the instance-related learned transfer matrix (IDTM) T(x) as a function of the instance x. In this process, without any constraints, the learned transfer matrix is not identifiable under instance-dependent noise (IDN). Therefore, for noisy labels, obtaining the learned transfer matrix IDTM to complete identification and classification is a very challenging problem. Summary of the Invention

[0005] In order to solve the above problems existing in the prior art, the present invention provides a method for estimating label noise based on a manifold regularization transfer matrix. The technical problems to be solved by the present invention are realized through the following technical solutions:

[0006] A method for estimating label noise based on a manifold regularization transfer matrix provided by the present invention includes:

[0007] Obtain a dataset carrying labels;

[0008] Wherein, the dataset includes data instances carrying noisy labels and data instances not carrying noisy labels;

[0009] Input the data instances in the dataset into the first network, so that the first network sequentially extracts features from and classifies the data instances to obtain the probability of the class to which the data instance belongs;

[0010] Calculate the loss function of the first network according to the probability of the class to which the data instance belongs, and pre-train the first network in the direction of decreasing the loss function to obtain a pre-trained first network;

[0011] Distill the data instances in the dataset using the distillation method to distill out the data instances of the clean class to form a sub-dataset;

[0012] Input the sub-dataset into the second network to obtain the probability of the class to which the data instance in the sub-dataset belongs and obtain the transfer matrix related to the data instance;

[0013] Based on the probability of the class to which the data instance in the sub-dataset belongs, the transfer matrix, and the data instance label in the sub-dataset, obtain the cross-entropy loss of the second network;

[0014] Based on the transfer matrix, the constructed association matrix representing the consistency of data instances belonging to the same manifold, the penalty matrix for data instances belonging to different manifolds, and the cross-entropy loss, calculate the loss function of the second network;

[0015] Adjust the loss function to reduce and train the second network to obtain a trained second network;

[0016] Use the second network to estimate the class to which the data instance belongs.

[0017] Among them, the first network is a partial network of the second network. The first network includes a backbone neural network and a classifier. The second network includes: a backbone neural network, a transfer neural network, and a classifier. The output of the backbone neural network is respectively connected to the input of the classifier and the transfer neural network.

[0018] Optionally, the step of inputting the data instances in the dataset into the first network so that the first network sequentially extracts features from and classifies the data instances to obtain the probability of the class to which the data instance belongs includes:

[0019] Input the data instances in the dataset into the backbone neural network so that the backbone neural network extracts features from the data instances in the dataset to obtain feature vectors, and input the feature vectors into the classifier to obtain the probability of the class to which the data instance belongs.

[0020] Optionally, the step of inputting the sub-dataset into the second network to obtain the probability of the class to which the data instance in the sub-dataset belongs and obtain the transfer matrix related to the data instance includes:

[0021] Input the sub-dataset into the backbone neural network so that the backbone neural network extracts features from the data instances in the sub-dataset to obtain a feature vector with the same dimension as the number of categories to which the data instances belong;

[0022] Input the feature vectors into a classifier and a transfer network respectively to obtain the category probabilities of each data instance in the sub-dataset output by the classifier and the transfer matrix related to the data instance output by the transfer network.

[0023] Optionally, distilling the data instances in the dataset using the distillation method to distill out the data instances of the clean class to form a sub-dataset includes:

[0024] Input each data instance in the dataset into a pre-trained first network in sequence to estimate the probability of the category to which each data instance belongs;

[0025] Judge the magnitude relationship between the probability of the category to which each data instance belongs and the probability threshold. If it is greater, determine that the data instance is a clean class data instance without a noisy label;

[0026] Form a sub-dataset with the clean class data instances.

[0027] Optionally, obtaining the cross-entropy loss of the second network based on the category probabilities of the data instances in the sub-dataset, the transfer matrix, and the data instance labels in the sub-dataset includes:

[0028] Multiply the category probabilities of the data instances in the sub-dataset by the transfer matrix, and perform cross-entropy on the multiplication result and the data instance labels in the sub-dataset to obtain the cross-entropy loss of the second network.

[0029] Optionally, calculating the loss function of the second network based on the transfer matrix, the constructed association matrix representing the consistency of data instances belonging to the same manifold, the penalty matrix for data instances belonging to different manifolds, and the cross-entropy loss includes:

[0030] Construct a first manifold loss based on the transfer matrix and the constructed association matrix representing the consistency of data instances belonging to the same manifold;

[0031] Construct a second manifold loss based on the transfer matrix and the constructed penalty matrix representing the dispersion of data instances belonging to different manifolds;

[0032] Subtract the second manifold loss from the first manifold loss to obtain a manifold loss;

[0033] Sum the manifold loss and the cross-entropy loss as the loss function of the second network.

[0034] Among them, the probability of the class to which the data instance belongs is expressed as:

[0035]

[0036] Among them, f(a i , w) represents the probability that the feature a i corresponding to the data instance x i belongs to class j, and C represents the number of classes.

[0037] Among them, the loss function of the first network is expressed as:

[0038]

[0039] Among them, N is the number of data instances with noisy labels, represents the label of the data instance, a i represents the feature obtained by feature extraction of the data instance x i after feature extraction.

[0040] Among them, the first manifold loss is expressed as:

[0041]

[0042] The second manifold loss is expressed as:

[0043]

[0044] Among them,

[0045] Among them, represents the association matrix, represents the penalty matrix, T(a i ) represents the transition matrix corresponding to the feature a i extracted from the data instance x i , T(a j ) represents the transition matrix corresponding to the feature a j of the data instance x j ;

[0046] The manifold loss is expressed as:

[0047] L(θ) = L1 - l2

[0048] The loss function of the second network is expressed as:

[0049] L total = L(w) + L(θ).

[0050] A label noise estimation method based on a manifold regularization transfer matrix provided by the present invention pre-trains a first network in a second network, and after distilling a data set, inputs the obtained sub-data set into the second network to obtain the probability of the class to which the data instances in the sub-data set belong and obtain a transfer matrix related to the data instances; further calculates the cross-entropy loss of the second network according to the data instance labels, and combines the constructed association matrix representing the consistency of data instances belonging to the same manifold and the penalty matrix of data instances belonging to different manifolds to calculate the loss function of the second network; adjusts the loss function to reduce the training of the second network to obtain a trained second network, thereby completing the estimation of the class to which the data instances belong. The present invention can reduce the estimation error without affecting the approximation error of the transfer matrix, and experiments prove that the present invention can achieve excellent performance in label noise learning.

[0051] The following will further describe the present invention in detail with reference to the drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 is a schematic flowchart of a label noise estimation method based on a manifold regularization transfer matrix provided by an embodiment of the present invention;

[0053] Figure 2 is a schematic diagram of a network model framework provided by an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0054] The following further describes the present invention in detail with reference to specific embodiments, but the embodiments of the present invention are not limited thereto.

[0055] As Figure 1 shown, a label noise estimation method based on a manifold regularization transfer matrix provided by the present invention includes:

[0056] S1, obtaining a data set carrying labels;

[0057] Among them, the data set includes data instances carrying noisy labels and data instances not carrying noisy labels;

[0058] S2, inputting the data instances in the data set into the first network, so that the first network sequentially extracts features and classifies the data instances to obtain the probability of the class to which the data instances belong;

[0059] Referring to Figure 2 shown, the first network is a partial network of the second network. The first network includes a backbone neural network and a classifier. The second network includes: a backbone neural network, a transfer neural network, and a classifier. The output of the backbone neural network is respectively connected to the input of the classifier and the input of the transfer neural network.

[0060] As an alternative embodiment of the present invention, step S2 includes: inputting data instances in the dataset into the backbone neural network so that the backbone neural network extracts features from the data instances in the dataset to obtain feature vectors, and inputting the feature vectors into a classifier to obtain the probability of the class to which the data instance belongs.

[0061] Given data with noisy labels where N is the number of data with noisy labels, x n is an instance, is the class of the data with noisy labels. First, input x n into the neural network backbone for feature extraction to obtain a 1×C vector. Exemplarily, assuming the feature vector: [1 2 10], the vector contains the features of the sample, and the size of the vector depends on the number of classes; where C is the number of classes, and then the output of the backbone is obtained through the softmax function. The probability of the class to which the data instance belongs is expressed as:

[0062]

[0063] where f(a i , w) represents the probability that the data instance x i corresponding to the feature a i belongs to class j, and C represents the number of classes.

[0064] S3. Calculate the loss function of the first network according to the probability of the class to which the data instance belongs, and pre-train the first network in the direction of reducing the loss function to obtain a pre-trained first network;

[0065] Pre-train the first network by minimizing the empirical risk L, that is, the loss function, as shown in the following formula:

[0066]

[0067] where w is the parameter of the backbone neural network, f(x i , w) is the output of the classifier, and L(w) is the loss function of the classifier plus the backbone neural network; N is the number of data instances with noisy labels, represents the label of the data instance, and a i represents the feature obtained by feature extraction of the data instance x i after feature extraction.

[0068] S4. Distill the data instances in the dataset using the distillation method to distill out clean-class data instances to form a sub-dataset;

[0069] As an alternative embodiment of the present invention, step S4 of the present invention includes:

[0070] S41: Input each data instance in the dataset into the pre-trained first network in sequence to estimate the probability of each data instance belonging to a certain class;

[0071] S42: Judge the magnitude relationship between the probability of each data instance belonging to a certain class and the probability threshold. If it is greater, determine that this data instance is a clean class data instance without noise labels;

[0072] S43: Compose the data instances of the clean class into a sub-dataset.

[0073] In the present invention, the instance distillation method in Shuo Yang, Erkun Yang, Bo Han, Yang Liu, Min Xu, Gang Niu, and Tongliang Liu. Estimating instance - dependent label - noise transition matrix using dnns. arXiv preprint arXiv:2105.13001, 2021. can be adopted to extract reliable clean instances where N clean is the number of distilled clean classes. This method can extract a sub - dataset with theoretically guaranteed Bayesian - optimal labels from the noisy dataset. The method of the present invention is not limited to the above - mentioned distillation - based example extraction method, and many other sample screening methods can also be used.

[0074] S5: Input the sub - dataset into the second network to obtain the probability of the data instances in the sub - dataset belonging to a certain class and obtain the transition matrix related to the data instances;

[0075] As an optional implementation manner of the present invention, step S5 of the present invention includes:

[0076] S51: Input the sub - dataset into the backbone neural network so that the backbone neural network extracts features from the data instances in the sub - dataset to obtain a feature vector with the same dimension as the number of classes to which the data instances belong;

[0077] S52: Input the feature vectors into the classifier and the transition network respectively to obtain the class probability of each data instance in the sub - dataset output by the classifier and the transition matrix related to the data instances output by the transition network.

[0078] The second network of the present invention models the probability of a given input instance x and its corresponding estimated potential clean label feature vector, observing the noise label is expressed as where T ij (x,θ)∈Rk×k , is the label of the extracted clean data.

[0079] The 1xC vector obtained from the backbone neural network is input into the transfer neural network to obtain the CxC transfer matrix T(a). Then, the classification probability p obtained by the backbone neural network through the classifier is multiplied by T(a) to obtain p'. The loss function is constructed using p' and the true noise label, and training is performed by minimizing the loss function. p is the label obtained by the network (which the present invention considers to be correct), and p' is the probability label obtained by p through the transfer matrix (which the present invention considers to be noisy). The loss function is constructed using the noisy probability label and the true noise label.

[0080] S6. Based on the probability of the data instance belonging to a class in the subset data, the transfer matrix, and the data instance label in the subset data, obtain the cross-entropy loss of the second network;

[0081] In this step, the probability of the data instance belonging to a class in the subset data can be multiplied by the transfer matrix, and the multiplication result is subjected to cross-entropy with the data instance label in the subset data to obtain the cross-entropy loss of the second network.

[0082] S7. Based on the transfer matrix, the constructed association matrix representing the consistency of data instances belonging to the same manifold, the penalty matrix for data instances belonging to different manifolds, and the cross-entropy loss, calculate the loss function of the second network;

[0083] It should be noted that: Manifold learning generally aims to preserve the intrinsic proximity structure in a potential low-dimensional feature space. Classical manifold learning techniques, such as Sam T Roweis and Lawrence K Saul. Nonlinear dimensionality reduction by locally linear embedding. science, 290(5500):2323–2326, 2000, estimate the local manifold through reasonable assumptions. In this patent, we adopt manifold embedding technology to implement the practical assumption we proposed, "the closer two instances in the same class are, the more similar their corresponding transfer matrices are", so that the IDTM T(x) can actually be learned. By introducing manifold regularization, although we do not directly reduce the complexity of T(a) because we do not further model, we still effectively reduce the degrees of freedom of the linear system so that T(a) can be stably estimated. At the same time, T(a) can be considered to be actually stable because adding such a smoothing assumption can prevent T(a) from jumping up and down in a very small neighborhood. The present invention constructs an association matrix to describe the consistency of the same manifold and a penalty matrix to describe the dispersion between different manifolds.

[0084] As an alternative embodiment of the present invention, step S7 of the present invention includes:

[0085] S71: Based on the transfer matrix and the association matrix that constructs the consistency of the already constructed expression data instances belonging to the same manifold, construct the first manifold loss;

[0086] S72: Based on the transfer matrix and the penalty matrix that constructs the dispersion of the already constructed expression data instances belonging to different manifolds, construct the second manifold loss;

[0087] S73: Subtract the second manifold loss from the first manifold loss to obtain the manifold loss;

[0088] Among them, the first manifold loss is expressed as:

[0089]

[0090] The second manifold loss is expressed as:

[0091]

[0092] Among them,

[0093] Among them, represents the association matrix, represents the penalty matrix, T(a i ) represents the transfer matrix corresponding to the feature a i extracted from the data instance x i T(a j ) represents the transfer matrix corresponding to the feature a j extracted from the data instance x j corresponding to.

[0094] Based on the above, the manifold regularization on the IDTM T(a i , θ), that is, the manifold loss is expressed as:

[0095] L(θ) = L1 - L2

[0096] S74: Sum the manifold loss and the cross-entropy loss as the loss function of the second network.

[0097] The loss function of the second network is expressed as:

[0098] L total = L(w) + L(θ).

[0099] S8, Obtain the trained second network by adjusting the loss function to train the second network;

[0100] S9, Use the second network to estimate the category to which the data instance belongs.

[0101] The effects of the present invention can be further illustrated by the following simulation experiments.

[0102] 1. Simulation conditions

[0103] The present invention conducts simulations on a central processing unit of Inter(R) Core(TM) i7-4790 3.60GHz CPU, NVIDIA GeForce RTX 3090 GPU, and Ubuntu 18.04 operating system, using the open-source pytorch1.6 of Facebook, Inc. in the United States. Four image classification databases, F-MNIST, SVHN, CIFAR-10, and CIFAR-100, are used in the database.

[0104] 2. The methods compared in the experiment are as follows:

[0105] One is to robustly train a deep neural network based on highly noisy labels, denoted as co-teaching in the experiment. References: Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. arXiv preprint arXiv:1804.06872, 2018. The other is a joint training method for combating noisy labels based on protocols, denoted as Jocor in the experiment. References: Hongxin Wei, Lei Feng, Xiangyu Chen, and Bo An. Combating noisy labels by agreement: A joint training method with co-regularization. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, pages 13726–13735, 2020. The third is to estimate the instance-dependent label-noise transition matrix based on DNNS, denoted as TMDNN in the experiment. References: Shuo Yang, Erkun Yang, Bo Han, Yang Liu, Min Xu, Gang Niu, and Tongliang Liu. Estimating instance-dependent label-noise transition matrix using dnns. arXiv preprint arXiv:2105.13001, 2021.

[0106] 3. Result display

[0107] The comparison results between the present invention and the above three methods are shown in the following table:

[0108] Table 1 Experimental results on the F-MNIST dataset

[0109]

[0110] Table 2 Experimental results on the CIFAR10 dataset

[0111]

[0112] Experimental Results on the SVHN Dataset in Table 3

[0113]

[0114]

[0115] Experimental Results on the CIFAR100 Dataset in Table 4

[0116]

[0117] Based on the proposed hypothesis that the closer two instances are, the more similar their corresponding transition matrices are, the present invention constructs manifold regularization to effectively reduce the degrees of freedom of T(x) and make it stably estimable. A large number of experimental results show that the present method is superior to the existing methods in dealing with IDN, especially at high noise rates. In addition, the present method is a plug-and-play module that helps to improve other methods. The above content is a further detailed description of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is only limited to these descriptions. For those of ordinary skill in the technical field to which the present invention pertains, without departing from the concept of the present invention, several simple deductions or substitutions can still be made, and all should be regarded as belonging to the protection scope of the present invention.

Claims

1. A label noise estimation method based on a manifold regularization transfer matrix, which is applied to an image classification task, and is characterized in that, Including: Obtaining a dataset with labels; the dataset is an image dataset; Wherein, the dataset includes data instances with noisy labels and data instances without noisy labels; the data instances are image data; Inputting the data instances in the dataset into a first network, so that the first network sequentially performs feature extraction and data instance classification on the data instances to obtain the probability of the class to which the data instances belong; Calculating the loss function of the first network according to the probability of the class to which the data instances belong, and pre-training the first network in the direction of reducing the loss function to obtain a pre-trained first network; Distilling the data instances in the dataset using the distillation method to distill out data instances of clean classes to form a sub-dataset; Inputting the sub-dataset into a second network to obtain the probability of the class to which the data instances in the sub-dataset belong and obtain a transfer matrix related to the data instances; Based on the probability of the class to which the data instances in the sub-dataset belong, the transfer matrix, and the data instance labels in the sub-dataset, obtaining the cross-entropy loss of the second network; Based on the transfer matrix, the constructed association matrix representing the consistency of data instances belonging to the same manifold, the penalty matrix for data instances belonging to different manifolds, and the cross-entropy loss, calculating the loss function of the second network; Training the second network by adjusting the loss function to decrease to obtain a trained second network; Using the second network to estimate the class to which the data instances belong.

2. The label noise estimation method based on the manifold regularization transfer matrix according to claim 1, wherein, The first network is a partial network of the second network. The first network includes a backbone neural network and a classifier. The second network includes: a backbone neural network, a transfer neural network, and a classifier. The output of the backbone neural network is respectively connected to the input of the classifier and the input of the transfer neural network.

3. The method for estimating label noise based on a manifold regularization transfer matrix according to claim 2, wherein The step of inputting the data instances in the dataset into the first network, so that the first network sequentially performs feature extraction and data instance classification on the data instances to obtain the probability of the class to which the data instances belong includes: Inputting the data instances in the dataset into the backbone neural network, so that the backbone neural network performs feature extraction on the data instances in the dataset to obtain feature vectors, and inputting the feature vectors into the classifier to obtain the probability of the class to which the data instances belong.

4. The method for estimating label noise based on a manifold regularization transfer matrix according to claim 2, wherein The step of inputting the sub-dataset into the second network to obtain the probability of the class to which the data instances in the sub-dataset belong and obtain a transfer matrix related to the data instances includes: Inputting the sub-dataset into the backbone neural network, so that the backbone neural network performs feature extraction on the data instances in the sub-dataset to obtain feature vectors with the same dimension as the number of classes to which the data instances belong; Inputting the feature vectors into the classifier and the transfer network respectively to obtain the class probabilities of each data instance in the sub-dataset output by the classifier, and the transfer matrix related to the data instances output by the transfer network.

5. The method for estimating label noise based on a manifold regularization transfer matrix according to claim 1, wherein The step of distilling the data instances in the dataset using the distillation method to distill out data instances of clean classes to form a sub-dataset includes: Each data instance in the dataset is sequentially input into the pre-trained first network to estimate the probability of each data instance belonging to a certain class; Judge the magnitude relationship between the probability of each data instance belonging to a certain class and the probability threshold. If it is greater, determine that the data instance is a clean class data instance without noisy labels; The data instances of the clean class are composed into a sub-dataset.

6. The label noise estimation method based on the manifold regularization transfer matrix according to claim 1, characterized in that The obtaining of the cross-entropy loss of the second network based on the probability of the data instances in the sub-dataset belonging to a certain class, the transition matrix, and the labels of the data instances in the sub-dataset includes: Multiply the probability of the data instances in the sub-dataset belonging to a certain class by the transition matrix, and perform cross-entropy on the multiplication result and the labels of the data instances in the sub-dataset to obtain the cross-entropy loss of the second network.

7. The method for estimating label noise based on a manifold regularization transfer matrix according to claim 1, wherein The calculation of the loss function of the second network based on the transition matrix, the constructed association matrix representing the consistency of data instances belonging to the same manifold, the penalty matrix for data instances belonging to different manifolds, and the cross-entropy loss includes: Based on the transition matrix and the constructed association matrix representing the consistency of data instances belonging to the same manifold, construct the first manifold loss; Based on the transition matrix and the constructed penalty matrix representing the dispersion of data instances belonging to different manifolds, construct the second manifold loss; Subtract the second manifold loss from the first manifold loss to obtain the manifold loss; Sum the manifold loss and the cross-entropy loss as the loss function of the second network.

8. The method for estimating label noise based on a manifold regularization transfer matrix according to claim 1, wherein The probability of the data instance belonging to a certain class is expressed as: Among them, f(a i , w) represents the probability that the data instance x i corresponding to the feature a i belongs to the j-th class, and C represents the number of classes.

9. The method for estimating label noise based on a manifold regularization transfer matrix according to claim 8, wherein The loss function of the first network is expressed as: where N is the number of data instances carrying noisy labels, denotes the label of the data instance, a i denotes the data instance x i is the feature obtained after feature extraction.

10. The label noise estimation method based on the manifold regularization transition matrix according to claim 9, wherein The first manifold loss is expressed as: The second manifold loss is expressed as: Among them, Among them, represents the association matrix, represents the penalty matrix, T(a i ) represents the feature a i extracted from the data instance x i corresponding transition matrix, T(a j ) represents the data instance x j feature a j corresponding transition matrix; The manifold loss is expressed as: L(θ) = L1 - L2 The loss function of the second network is expressed as: L total = L(w) + L(θ).

Images