An image classification method for noisy labels

Abstract

The present invention relates to an image classification method for label-noisy situations, and belongs to the field of data processing technology. The present invention can be applied to the classification task of label-noisy images to improve the classification accuracy. Different from the existing methods, the method used in the present invention does not rely on the specific distribution of the image data set, nor does it need to identify each clean image sample, which greatly saves data annotation costs and classification speed. In addition, the method used in the present invention can be used as a general post-processing strategy, suitable for various popular networks pre-trained on label-noisy image data sets to improve their classification performance. Through experiments on typical public data sets, it is proved that the method used in the present invention achieves accurate image sample statistics estimation and high classification accuracy.

Description

Technical Field

[0001] The invention relates to an image classification method for a label-noise situation, which is applied to a multi-classification task of noise data and belongs to the technical field of machine learning. Background Art

[0002] Real-world data may contain a large number of samples with noisy labels, which usually mislead the training algorithm and lead to the degradation of classification performance of test data. Therefore, label noise learning has become an important task in the field of machine learning. Existing label noise processing methods can be roughly divided into three categories, namely, methods based on sample selection, robust loss function design methods, and methods based on statistical estimation. One of the popular research trends is to estimate critical statistics (such as sample mean and sample covariance), which aims to restore the clean data distribution by estimating some key statistics, such as the transfer matrix, the centroid of the data set, and the mean / covariance of the data.

[0003] At present, only two studies have attempted to estimate the unbiased statistics (sample mean and covariance) of each class of samples. Gao et al. proposed a method for estimating noisy sample statistics based on clustering methods. This method first uses the k-means algorithm to cluster the corrupted data, and then estimates the noise from the clusters based on the first-order and second-order statistics. However, this method is difficult to handle multi-classification. Lee et al. proposed using a robust generative classifier to estimate the statistics of each class from the perspective of outlier removal. It assumes that the clean examples and noisy examples of each class obey an isotropic Gaussian distribution, and the noisy examples are more widely scattered than the clean examples. A generative classifier is introduced on top of the hidden feature space of the pre-trained deep neural network to obtain a more robust decision boundary. By estimating each class of statistics using the minimum covariance determinant, neither the deep model needs to be retrained nor the structure of the deep model needs to be changed. However, the assumption of isotropic Gaussian distribution may not hold in many real-world scenarios, and the instance selection process of the robust generative classifier is unreliable because the selected instances may still contain noisy labels, resulting in biased estimates of the mean and covariance.

[0004] In terms of patents, the Chinese invention patent application with application number CN202310513686.3 provides a "deep classification network noise label modeling and correction method". This method proposes to construct multiple label sets and assign different weights to different label samples in the constructed label sets, and obtain the corresponding designed weighted loss function to train the classification network. The Chinese invention patent application with application number CN202310509397.6 provides a "neural network model training method with noisy multi-label data". This method is based on the statistical consistency of label noise learning loss, and uses a meta-learning algorithm to simultaneously learn the instance feature-dependent label noise transfer matrix network parameters, data imbalance parameters and multi-label classification neural network parameters. The Chinese invention patent application with application number CN202310243322.8 provides a "method for processing multi-type label noise based on symmetric KLD adaptive dynamic threshold". This method realizes semi-supervised learning by fusing the obtained model with the ability to learn and identify different labels with pseudo labels, and at the same time performs comparative learning based on unlabeled data sets to obtain the minimum loss value of the model for processing noisy labels.

[0005] Although the existing label noise learning methods based on statistical estimation have achieved relatively good results under specific conditions, the isotropic Gaussian distribution of the data set is not satisfied in many cases, and the label noise problem in multi-classification is difficult to handle. Therefore, the present invention establishes an algebraic relationship between the noise sample statistics and the clean sample statistics, calculates the unbiased clean sample statistics, obtains the class posteriori of the clean data, and obtains the weights of each layer by optimizing the negative log-likelihood loss. Summary of the invention

[0006] The present invention provides an image classification method for label noise situations, which is used for classification tasks of noisy label data sets to improve classification accuracy. The present invention reveals the mathematical relationship between noise data statistics and clean data statistics in multi-classification label noise problems, so that unbiased estimates of first-order and second-order statistics of clean data can be obtained based on observable noise data, and then the posterior of the clean class can be inferred; the purpose of the present invention is to estimate unbiased clean class statistics for noise data sets of arbitrary distribution and number of categories, obtain layer weights of ensemble learning, and thus obtain a robust generative classifier that adapts to noise situations, and achieve accurate classification of test data.

[0007] The technical solution of the present invention is: an image classification method for the situation where labels are noisy, comprising:

[0008] Input label noisy image data;

[0009] Initialize layer weights and train the neural network using labeled noisy image data;

[0010] Calculate various estimators for labeled noisy image samples and derive corresponding clean estimators;

[0011] Use various clean estimators of the sample to calculate the class posteriori, optimize the loss, and learn the weights;

[0012] Output the optimal parameters of the pre-trained network and the optimal set of layer weights;

[0013] Use the trained network to perform image classification.

[0014] Further, the steps include:

[0015] The first step is to input the labeled noisy image training set and validation set Neural Network θ and layer collection Where n is the sample size of the training data set; x i Represents the characteristics of the i-th training sample; represents the noisy label of the i-th training sample; Represents the characteristics of the i-th sample in the validation set; represents the noisy label of the i-th sample in the validation set; n val Indicates the size of the validation set, that is, the number of samples; layer set Indicates the network used θ A collection of some network layers;

[0016] The second step is to initialize the layer weights Based on the labeled noisy image training set Use any existing method to pre-train the neural network f θ ;

[0017] The third step is to solve the equation to obtain the noise transfer matrix estimate And calculate the noise class prior and clean class priors And use the obtained and Derive the coefficient matrix estimate

[0018] The fourth step is to perform two layers of nested loops, the first layer is the layer collection Loop, in this layer, features are extracted from the labeled noise image training set The second layer loops through each class and first calculates the first-order statistics of each class of noise samples. and the second-order statistics Then calculate the centroid of each type of clean sample and covariance Loop through all layers and classes until the end;

[0019] Step 5: Use the clean sample statistics obtained in step 4 to calculate the class posterior P(Y|X) of the clean data and minimize the validation set The negative logarithmic loss on , gets the optimal layer weight;

[0020] Step 6: Output the optimal pre-trained network parameters θ * and layer set weights Use the trained network to perform image classification.

[0021] Furthermore, in the second step, the neural network is pre-trained using any existing method, including Co-teaching and JoCoR.

[0022] Furthermore, in the third step, the coefficient matrix from the noise sample statistics to the clean sample statistics is calculated The derivation formula of the coefficient matrix is:

[0023]

[0024] in is a permutation matrix, i.e., a matrix obtained by swapping the i-th and j-th rows of the identity matrix; is the estimated value of the prior probability of the i-th category; C represents the total number of categories in the classification task; Represents the transfer matrix obtained in the third step The (i,j)th element of .

[0025] Furthermore, in the actual learning process of the second and third steps, a label-free image dataset is defined For distribution The n samples sampled in , assuming that the image dataset with label noise is Is the distribution The n samples sampled in , the corresponding validation set is where n val represents the size of the validation set, where is the dataset S after labeling noise pollution. In addition, the parameterized neural network output is f θ (·), the set of layers for ensemble learning is

[0026] First, solve the following equation to get the estimated value of the noise transfer matrix

[0027]

[0028] Among them, x i , They represent the features of the i-th sample in the training data set and its noisy label respectively; represents the cross entropy loss function; Represents the matrix The logarithm of the determinant is used to reduce The volume of the space spanned; λ is a trade-off hyperparameter;

[0029] Secondly, in order to estimate the clean class prior The following system of equations needs to be solved:

[0030]

[0031] in is the noisy class prior probability of the ith class, is the random variable corresponding to the noisy label. In order to solve the equations in (3), we use the empirical estimate Instead of T, T ij and Respectively represent the matrix T, The (i,j)th element of , where the noise class prior probability It is estimated by the following formula:

[0032]

[0033] Then, according to formulas (2), (3) and (4), the empirical estimate of the coefficient matrix is ​​expressed as:

[0034]

[0035] Where K i→j represents the permutation matrix, that is, the matrix obtained by swapping the i-th and j-th rows of the identity matrix; is the analytical solution of (3); C represents the total number of categories in the classification task;

[0036] Furthermore, the fourth step and the fifth step include:

[0037] Derive various statistics of the noise-free data set S; assuming φ l (x) represents the output of the network layer l when the input is x. Formula (6) gives the estimation of various first-order and second-order statistics of labeled noisy image data:

[0038]

[0039] in Represents labeled noisy image data The number of samples of the i-th category in , and then the unbiased estimates of various first-order and second-order statistics of clean data are derived based on the above formula, which are expressed as

[0040]

[0041]

[0042] Based on this, the unbiased estimates of the covariances of various types are:

[0043]

[0044] Next, the overall covariance of the noise-free image data can be expressed as Substituting formulas (7), (8) and (9) into the class posterior probability formula, we get

[0045]

[0046] The clean class posterior distribution corresponding to the network layer l can be obtained; in order to construct a more robust posterior probability, the network f is integrated using ensemble learning θ Corresponding to the last two layers (denoted as ) is weighted to obtain the final class posterior distribution, that is,

[0047]

[0048] The layer weights The optimal value is obtained by Minimizing the negative log-likelihood function is obtained, which is expressed as

[0049] In the test phase, based on the Bayesian criterion, the class posterior probability of any sample x can be obtained by formulas (10) and (11).

[0050] Compared with the existing technology, the present invention has the following three significant advantages:

[0051] (1) Universality. The proposed method can be considered as a general post-processing strategy that can improve the classification performance of many deep neural networks pre-trained on labeled noisy images. In addition, compared with existing methods, it can also handle multi-classification problems.

[0052] (2) Reliability. The various statistical estimation methods proposed in the present invention guarantee convergence in theory, and the estimation accuracy of some key statistical quantities has been proven.

[0053] (3) Practicality. The estimation method in the present invention does not rely on the error-prone clean sample selection process. In addition, the method of the present invention does not contain any tuning hyperparameters. Therefore, it can be easily implemented in various practical scenarios;

[0054] (4) The present invention is used for classification tasks of noisy label data sets to improve classification accuracy. The present invention reveals the mathematical relationship between noise data statistics and clean data statistics in the multi-classification label noise problem, so that based on the observable noise data, unbiased estimates of the first-order and second-order statistics of the clean data can be obtained, and then the posterior of the clean class can be inferred; the purpose of the present invention is to estimate the unbiased clean class statistics for noise data sets of arbitrary distribution and number of categories, obtain the layer weights of ensemble learning, and thus obtain a robust generative classifier that adapts to noise situations and achieves accurate classification of test data. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 It is a flow chart of the present invention. DETAILED DESCRIPTION

[0056] Example 1: Figure 1 As shown, a method for image classification in the case of label noise includes the following steps:

[0057] The first step is to input the labeled noisy image training set and validation set Neural Network θ and layer collection Where n is the sample size of the training data set; x i Represents the characteristics of the i-th training sample; represents the noisy label of the i-th training sample; Represents the characteristics of the i-th sample in the validation set; represents the noisy label of the i-th sample in the validation set; n val Indicates the size of the validation set, that is, the number of samples; layer set Indicates the network used θ A collection of some network layers;

[0058] The second step is to initialize the layer weights Based on the labeled noisy image training set Use any existing method to pre-train the neural network f θ ;

[0059] The third step is to solve equation (2) to obtain the noise transfer matrix estimate And calculate the noise class prior and clean class priors And use the obtained and The coefficient matrix is ​​estimated by formula (1) or (5)

[0060] The fourth step is to perform two layers of nested loops, the first layer is the layer collection Loop, in this layer, features are extracted from the labeled noise image training set The second layer loops over each class, first using formula (6) to calculate the first-order statistics of each class of noise samples: and the second-order statistics Then use formulas (7) and (9) to calculate the centroid of each type of clean sample and covariance Loop through all layers and classes until the end;

[0061] In the fifth step, the clean sample statistics obtained in the fourth step are used to calculate the class posterior P(Y|X) of the clean data using formula (11), and the validation set is minimized by The negative logarithmic loss on , gets the optimal layer weight;

[0062] Step 6: Output the optimal pre-trained network parameters θ * and layer set weights Use the trained network to perform image classification.

[0063] Furthermore, in the second step, the neural network is pre-trained using any existing method, including Co-teaching and JoCoR.

[0064] Furthermore, in the third step, the coefficient matrix from the noise sample statistics to the clean sample statistics is calculated The derivation formula of the coefficient matrix is:

[0065]

[0066] in is a permutation matrix, i.e., a matrix obtained by swapping the i-th and j-th rows of the identity matrix; is the estimated value of the prior probability of the i-th category; C represents the total number of categories in the classification task; Represents the transfer matrix obtained in the third step The (i,j)th element of .

[0067] Furthermore, in the actual learning process of the second and third steps, a label-free image dataset is defined For distribution The n samples sampled in , assuming that the image dataset with label noise is Is the distribution The n samples sampled in , the corresponding validation set is where n val represents the size of the validation set, where is the dataset S after labeling noise pollution. In addition, the parameterized neural network output is f θ(·), the set of layers for ensemble learning is

[0068] First, solve the following equation to get the estimated value of the noise transfer matrix

[0069]

[0070] Among them, x i , They represent the features of the i-th sample in the training data set and its noisy label respectively; represents the cross entropy loss function; Represents the matrix The logarithm of the determinant is used to reduce The volume of the space spanned; λ is a trade-off hyperparameter;

[0071] Secondly, in order to estimate the clean class prior The following system of equations needs to be solved:

[0072]

[0073] in is the noisy class prior probability of the ith class, is the random variable corresponding to the noisy label. In order to solve the equations in (3), we use the empirical estimate Instead of T, T ij and Respectively represent the matrix T, The (i,j)th element of , where the noise class prior probability It is estimated by the following formula:

[0074]

[0075] Then, according to formulas (2), (3) and (4), the empirical estimate of the coefficient matrix is ​​expressed as:

[0076]

[0077] Where K i→j represents the permutation matrix, that is, the matrix obtained by swapping the i-th and j-th rows of the identity matrix; is the analytical solution of (3); C represents the total number of categories in the classification task;

[0078] Furthermore, the fourth step and the fifth step include:

[0079] Derive various statistics of the noise-free data set S; assuming φ l(x) represents the output of the network layer l when the input is x. Formula (6) gives the estimation of various first-order and second-order statistics of labeled noisy image data:

[0080]

[0081] in Represents labeled noisy image data The number of samples of the i-th category in , and then the unbiased estimates of various first-order and second-order statistics of clean data are derived based on the above formula, which are expressed as

[0082]

[0083]

[0084] Based on this, the unbiased estimates of the covariances of various types are:

[0085]

[0086] Next, the overall covariance of the noise-free image data can be expressed as Substituting formulas (7), (8) and (9) into the class posterior probability formula, we get

[0087]

[0088] The clean class posterior distribution corresponding to the network layer l can be obtained; in order to construct a more robust posterior probability, the network f is integrated using ensemble learning θ Corresponding to the last two layers (denoted as ) is weighted to obtain the final class posterior distribution, that is,

[0089]

[0090] The layer weights The optimal value is obtained by Minimizing the negative log-likelihood function is obtained, which is expressed as

[0091] In the test phase, based on the Bayesian criterion, the class posterior probability of any sample x can be obtained by formulas (10) and (11).

[0092] This method establishes a mathematical relationship between the statistics of labeled noisy image data and the statistics of labeled clean image data, and derives the unbiased mean and variance of labeled clean image data from the observable labeled noisy image data, thereby restoring the true distribution of the image data and obtaining a clean class posteriori. In this way, the optimal ensemble learning layer weights are obtained by optimizing the negative log-likelihood function, and a classifier that is robust to noisy data is constructed.

[0093] The method used in the present invention is compared with the mainstream method on a typical public data set, and it is actually verified that it has excellent estimation ability for the statistics of label noisy image data, and can greatly improve the accuracy of neural network classification of label noisy images. In Table 1, it can be found that when the pre-training method is cross entropy (i.e., non-robust pre-training method), the method used in the present invention greatly improves the classification performance of the pre-trained deep neural network. Specifically, on the CIFAR-10 data set, the method used in the present invention improves the classification accuracy of the deep neural network pre-trained with cross entropy loss by 12.57%. The corresponding improvement on the CIFAR-100 data set is 8.89%. It can be clearly observed that in all noise scenes, the method used in the present invention achieves better performance than the benchmark method on both data sets.

[0094] Table 1

[0095]

[0096] The specific implementation modes of the present invention are described in detail above in conjunction with the accompanying drawings, but the present invention is not limited to the above implementation modes, and various changes can be made within the knowledge scope of ordinary technicians in this field without departing from the purpose of the present invention.

Claims

1. A method for image classification in the case of label noise, characterized in that: include: Input label noisy image data; Initialize layer weights and train the neural network using labeled noisy image data; Calculate various estimators for labeled noisy image samples and derive corresponding clean estimators; Use various clean estimators of the sample to calculate the class posteriori, optimize the loss, and learn the weights; Output the optimal parameters of the pre-trained network and the optimal set of layer weights; Use the trained network to classify images; The steps include: The first step is to input the labeled noisy image training set and validation set Neural Network θ and layer collection Where n is the sample size of the training data set; x i Represents the characteristics of the i-th training sample; represents the noisy label of the i-th training sample; Represents the characteristics of the i-th sample in the validation set; represents the noisy label of the i-th sample in the validation set; n val Indicates the size of the validation set, that is, the number of samples; layer set Indicates the network used θ A collection of some network layers; The second step is to initialize the layer weights Based on the labeled noisy image training set Use any existing method to pre-train the neural network f θ ; The third step is to solve the equation to obtain the noise transfer matrix estimate And calculate the noise class prior and clean class priors And use the obtained and Derive the coefficient matrix estimate The fourth step is to perform two layers of nested loops, the first layer is the layer collection Loop, in this layer, features are extracted from the labeled noise image training set The second layer loops through each class and first calculates the first-order statistics of each class of noise samples. and the second-order statistics Then calculate the centroid of each type of clean sample and covariance Loop through all layers and classes until the end; Step 5: Use the clean sample statistics obtained in step 4 to calculate the class posterior P(Y|X) of the clean data and minimize the validation set The negative logarithmic loss on , gets the optimal layer weight; Step 6: Output the optimal pre-trained network parameters θ * and layer set weights Use the trained network to classify images; In the third step, the coefficient matrix from the noisy sample statistics to the clean sample statistics is calculated The derivation formula of the coefficient matrix is: Where K c→c′ is a permutation matrix, i.e., a matrix obtained by swapping the c-th and c′-th rows of the identity matrix; is the estimated value of the prior probability of the cth class; C represents the total number of categories in the classification task; The resulting transfer matrix is ​​represented by the (c,c′)th element of ; In the actual learning process of the second and third steps, define a dataset of images without label noise For distribution The n samples sampled in , assuming that the image dataset with label noise is Is the distribution The n samples sampled in , the corresponding validation set is where n val represents the size of the validation set, where is the dataset S after labeling noise pollution. In addition, the parameterized neural network output is f θ (·), the set of layers for ensemble learning is First, solve the following equation to get the estimated value of the noise transfer matrix Among them, x i , They represent the features of the i-th sample in the training data set and its noisy label respectively; represents the cross entropy loss function; Represents the matrix The logarithm of the determinant is used to reduce The volume of the space spanned; λ is a trade-off hyperparameter; Secondly, in order to estimate the clean class prior The following system of equations needs to be solved: in is the noisy class prior probability of the cth class, is the random variable corresponding to the noisy label. In order to solve the equations in (3), we use the empirical estimate Instead of T, T cc′ and They represent the matrix T, The (c,c′)th element of , where the noise class prior probability It is estimated by the following formula: Then, according to formulas (2), (3) and (4), the empirical estimate of the coefficient matrix is ​​expressed as: Where K c→c′ represents the permutation matrix, that is, the matrix obtained by exchanging the c-th row and the c′-th row of the identity matrix; is the analytical solution of (3); C represents the total number of categories in the classification task; The fourth and fifth steps include: Derive various statistics of the noise-free data set S; assuming φ l (x) represents the output of the network layer l when the input is x. Formula (6) gives the estimation of various first-order and second-order statistics of labeled noisy image data: in Represents labeled noisy image data The number of samples of the i-th category in , and then the unbiased estimates of various first-order and second-order statistics of clean data are derived based on the above formula, which are expressed as Based on this, the unbiased estimates of the covariances of various types are: Next, the overall covariance of the noise-free image data can be expressed as Substituting formulas (7), (8) and (9) into the class posterior probability formula, we get The clean class posterior distribution corresponding to the network layer l can be obtained; in order to construct a more robust posterior probability, the network f is integrated using ensemble learning θ The posterior probabilities corresponding to the last two layers are weighted to obtain the final class posterior distribution, that is, The layer weights The optimal value is obtained by Minimizing the negative log-likelihood function is obtained, which is expressed as In the test phase, based on the Bayesian criterion, the class posterior probability of any sample x can be obtained by formulas (10) and (11).

2. The image classification method for label-noisy situations according to claim 1, characterized in that: In the second step, the neural network is pre-trained using any existing method, including Co-teaching and JoCoR.

Images