A long-tail learning approach for image classification with in-distribution and out-of-distribution noisy labels

Abstract

The present invention belongs to an image classification method, and specifically discloses a long-tail learning image classification method with in-distribution and out-of-distribution noise labels. This method adopts a long-tail learning framework with in-distribution and out-of-distribution noise labels in an open world to build a deep learning model to solve the challenging learning problem of simultaneously processing ID and OOD noise samples on long-tail data. Specifically, the present invention first uses model consistency prediction to filter OOD noise samples and assign uniform labels, thereby enhancing the model's detection ability for OOD noise samples; at the same time, JS divergence is used to separate ID noise samples and reduce the misclassification of clean samples in the distribution, especially in the tail categories; in addition, additional semantic classifiers are introduced to reduce the bias of pseudo labels on majority categories; finally, strongly enhanced consistency regularization is introduced to further improve the generalization performance and robustness of the model, and enhance the classification performance of the model on multi-biased datasets.

Description

Technical Field

[0001] The present invention belongs to the technical field of image classification, and in particular relates to a long-tail learning image classification method with in-distribution and out-of-distribution noise labels. Background Art

[0002] Noisy label learning is a key research area in deep learning, focusing on how to effectively train models in the presence of inaccurate labels. Long-tail data learning addresses the problem of uneven class distribution in datasets, particularly in the field of image recognition using deep learning. In a long-tail distribution, the number of samples from a small number of classes, known as the head classes, dominates, while the number of samples from a large number of classes, known as the tail classes, is sparse. This situation leads to overfitting of common classes during model training, while insufficient recognition of rare classes. Long-tail data learning aims to improve the model's recognition of tail classes through various strategies, such as data augmentation, resampling, and class reweighting, thereby enhancing overall performance.

[0003] Deep neural networks have achieved great success in image classification using large-scale labeled datasets. However, due to the scarcity of certain data and the high cost of labeling, it is challenging to obtain large-scale well-labeled samples for training. A common alternative is to collect samples from the Internet, which often leads to data bias in the constructed training datasets, such as Figure 2 As shown in

[15] , images crawled from the web usually exhibit unbalanced category distribution and long-tail characteristics, and samples obtained through query search may also have noisy labels that are inconsistent with the actual categories.

[0004] This type of noise label can be divided into out-of-distribution (OOD) noise labels where the sample belongs to an unknown category, and in-distribution (ID) noise labels where the sample comes from a known category but has the wrong label.

[0005] In long-tail datasets, the feature space is heavily influenced by the majority of head categories, causing classifiers trained on these distributions to favor the majority and neglect the minority tail categories. Furthermore, deep neural networks are prone to overfitting noisy labels, resulting in degraded generalization performance. Therefore, image classification using deep neural networks faces a complex scenario where datasets suffer from multiple biases, including long-tail class imbalance, ID noise, and OOD noise. Traditional methods focus on addressing either class imbalance or noisy labels, but rarely consider both simultaneously. This is particularly true when both ID and OOD noise coexist. The true labels for OOD noise are outside the dataset, and incorrectly assigning labels within the OOD noise distribution can degrade the classification performance of the neural network. Summary of the Invention

[0006] The purpose of this invention is to propose a long-tail learning image classification method with in-distribution and out-of-distribution noise labels, which is beneficial to solving the problem of the simultaneous existence of ID noise and OOD noise samples in long-tail data, improving the generalization performance and robustness of the deep learning model, and enhancing the classification performance of the model on multi-bias datasets.

[0007] In order to achieve the above-mentioned purpose, the present invention adopts the following technical solutions:

[0008] A long-tail learning image classification method with in-distribution and out-of-distribution noisy labels includes the following steps:

[0009] Step 1. Obtain a training dataset with in-distribution and out-of-distribution noise in real-world scenarios ;

[0010] in, Represented by the image i.e. sample With the corresponding label The training data set consists of is the total number of samples in the dataset, , Indicates the preset label of the dataset;

[0011] , Indicates the total number of data set categories and preset labels There are noisy labels in ;

[0012] Step 2. Build a deep learning model and ;

[0013] in and Represents the deep learning model Representation extractor and classifier, and Represents the deep learning model Representation extractor and classifier, The model parameters are expressed as , The model parameters are expressed as ;

[0014] Build a representation storage queue for each category in the dataset , Indicates the category Determine the maximum number of training rounds And the maximum number of rounds of model parameter warm-up initialization , initialize the current round =1;

[0015] Step 3. Perform label smoothing to obtain smooth labels ; Determine the current round Is it less than or equal to ; If the current round Less than or equal to , then go to step 4; otherwise, if the current round Greater than , then go to step 5;

[0016] Step 4. Use All samples in and the corresponding smooth labels Calculating cross entropy loss , and go to step 11;

[0017] Step 5. Calculate the threshold for filtering clean samples in the distribution in the current round ;

[0018] Step 6. Use the deep learning model and Separate predictions The labels of the training dataset are filtered out of the distribution noise samples according to the consistency of the labels. The remaining samples in the training dataset are used as the distribution samples. The noise samples outside the distribution are mapped to the smooth labels. Each component on Set to , and independently calculate the out-of-distribution sample loss ;

[0019] Step 7. Use Jensen-Shannon divergence to calculate the predicted probability distribution of samples within the distribution and the corresponding smoothed labels Likelihood , get the probability that each sample in the distribution is a non-noise label, that is, a clean sample in the distribution , the probability is greater than the threshold The samples within the distribution of are regarded as clean samples within the distribution, and the samples within other distributions are regarded as noise samples within the distribution;

[0020] Step 8. Extract representations of clean samples within the distribution and pre-set labels based on the dataset , add the representation of the extracted clean sample in the distribution to the corresponding representation storage queue If the number of elements in the characterization storage queue reaches the upper limit, the element that was first added to the characterization storage queue will be removed and then added;

[0021] Step 9. Store the queue by characterization Construct a class prototype for each category. For in-distribution noise samples, select high-confidence noise samples in the distribution. Use a semantic classifier to obtain semantic pseudo labels by comparing the cosine similarity between the high-confidence noise sample representation in the distribution and each class prototype. ,pass Predictions for high-confidence noise samples within the distribution yield linear pseudo labels ; Semantic pseudo-label With linear pseudo-labels Add the weights to get the pseudo labels of the noise samples within the distribution;

[0022] Step 10. Calculate the cross entropy loss of clean samples within the distribution and high confidence noise samples within the distribution , calculate the consistency regularization term , get the final loss ; and represents the weight of the loss term;

[0023] Step 11. By minimizing the loss , update the model parameters , through the updated renew ;

[0024] Step 12. Update the current round The value of ;

[0025] Determine the current round Is it less than or equal to the maximum round ; If yes, go to step 3; otherwise, the training ends;

[0026] Step 13. Use the trained deep learning model Make classification predictions for the input image.

[0027] The present invention has the following advantages:

[0028] As shown above, the present invention relates to a long-tail learning image classification method with in-distribution and out-of-distribution noise labels. In each training round, for each batch of iterations, the method of the present invention first uses the model's consistency prediction to separate OOD noise samples and assigns uniform labels to the OOD noise samples, thereby enhancing the model's ability to distinguish between ID samples and OOD noise samples. For ID noise samples, the method of the present invention uses JS divergence as the noise separation indicator for ID samples. The recall rate and precision of tail samples are better than those of loss-based methods, effectively preventing tail samples with large losses from being misclassified as ID noise samples. In order to further address the bias of pseudo-labels towards the majority class in unbalanced datasets, the method of the present invention additionally introduces a semantic classifier; specifically, the representations of clean samples in the distribution are stored in a representation storage queue according to the preset labels of the dataset, and then class prototypes are constructed from the representation storage queue and compared with the representations of ID noise samples to generate semantic pseudo-labels, which are then mixed with linear pseudo-labels to restore the true labels of ID noise samples. In addition, the method of the present invention also applies data augmentation and consistency regularization to further improve the model's ability to learn meaningful representations. Actual experimental results show that data augmentation and consistency regularization can effectively enhance the classification performance of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 is a flowchart of a long-tail learning image classification method with in-distribution and out-of-distribution noise labels in an embodiment of the present invention;

[0030] Figure 2 Schematic diagram of the distribution of a long-tail data set containing in-distribution noise samples and out-of-distribution noise samples in an embodiment of the present invention;

[0031] Figure 3 It is a structural diagram of the deep learning model constructed in an embodiment of the present invention. DETAILED DESCRIPTION

[0032] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments:

[0033] In this embodiment, a long-tail learning image classification method with in-distribution and out-of-distribution noise labels is proposed. The method adopts a long-tail learning framework with in-distribution and out-of-distribution noise labels in an open world to build a deep learning model to solve the challenging learning problem of simultaneously processing ID and OOD noise samples on long-tail data. First, the method of the present invention separates OOD noise samples based on prediction consistency, while taking into account category imbalance, and assigns uniform labels to OOD noise samples to enhance the discriminative ability of the model. Next, the JS divergence between the predicted probability distribution and the corresponding label is used to filter the ID noise samples. For ID noise, an additional semantic classifier is mixed to generate category-balanced pseudo labels. In addition, consistency regularization is introduced to encourage the model to produce consistent predictions under different enhancements, thereby improving its generalization ability.

[0034] like Figure 1 As shown in Figure 1, the long-tail learning image classification method with in-distribution and out-of-distribution noisy labels includes the following steps:

[0035] Step 1. Obtain a training dataset with in-distribution and out-of-distribution noise in real-world scenarios .

[0036] in, Represented by the image i.e. sample With the corresponding label The training data set consists of is the total number of samples in the dataset, ; , Indicates the total number of categories in the dataset, Indicates category.

[0037] There are noisy labels in the dataset, that is, there are preset labels and The actual label does not match.

[0038] Step 2. Build a deep learning model and .

[0039] in, and Respectively Representation extractor and classifier, and Respectively Representation extractor and classifier, The model parameters are expressed as , The model parameters are expressed as .

[0040] Build a representation storage queue for each category in the dataset , Indicates the category Determine the maximum number of training rounds. And the maximum number of rounds of model parameter warm-up initialization , initialize the current round =1.

[0041] Step 3. Perform label smoothing to obtain smooth labels .

[0042] Specifically, the smoothing labels Is the length of vector, Each component on Expressed as:

[0043] ;

[0044] in, is a hyperparameter used to control the degree of label smoothing, Indicates the preset label of the dataset.

[0045] Determine the current round Is it less than or equal to ; If the current round Less than or equal to , then go to step 4, otherwise, if the current round Greater than , go to step 5.

[0046] Step 4. Use all samples of the training dataset and the corresponding smooth labels Calculating cross entropy loss , go to step 11.

[0047] For cross entropy loss The calculation is expressed as:

[0048] .

[0049] in, represents the predicted probability distribution, , express logits, represents the Softmax activation function, , express No. A portion.

[0050] is the cross entropy loss function, .

[0051] Expresses the expectation of loss, .

[0052] Step 5. Calculate the threshold for filtering clean samples in the distribution in the current round .

[0053] for Calculation and current round Related, the formula is as follows:

[0054] ;

[0055] in, represents the upper threshold, express The initial threshold value.

[0056] Step 6. Use the deep learning model and Separate predictions .

[0057] Filter samples based on label consistency The out-of-distribution noise samples in The remaining samples are regarded as samples within the distribution, and the noise samples outside the distribution are labeled Each component on Set to , and calculate the out-of-distribution sample loss .

[0058] Here, label consistency means that for the same ,use and The label predictions obtained separately should be consistent.

[0059] Specifically, for the input and of The obtained logits are and Taking into account the class imbalance, the prior probability of each class is used Debiasing is performed, and the logits after debiasing are:

[0060] and .

[0061] The probability that the sample is a noise sample outside the distribution for:

[0062] .

[0063] Obviously ,Will The samples of the distribution are regarded as out-of-distribution noise, and the out-of-distribution noise samples are corresponding to the labels Each component on Set to .

[0064] Further calculate the out-of-distribution sample loss : .

[0065] Step 7. Use Jensen-Shannon divergence to calculate the predicted probability distribution of samples within the distribution and the corresponding labels Likelihood , get the probability that each sample in the distribution is a non-noise label, that is, a clean sample in the distribution , the probability is greater than The samples in the distribution are regarded as clean samples in the distribution, and the remaining samples, i.e., the probability is not greater than The in-distribution samples of are used as in-distribution noise samples.

[0066] Specifically, the predicted probability distribution With smooth label JS divergence between The calculation method is:

[0067] .

[0068] in, represents the Kullback-Leibler divergence;

[0069] , .

[0070] The probability of a clean sample within the distribution is expressed as:

[0071] , ,Will The sample of is regarded as a clean sample within the distribution, The other in-distribution samples are used as in-distribution noise samples.

[0072] Step 8. Extract representations for clean samples within the distribution according to the preset labels , add the representation of the clean sample in the distribution to the corresponding representation storage queue If the number of elements in the representation storage queue reaches the upper limit, the element that was first added to the representation storage queue will be removed and then added.

[0073] Specifically, the characterization formula for extracting clean samples within the distribution is as follows:

[0074] .

[0075] in, Indicates passing Obtained characterization, express Representation extractor.

[0076] according to Preset tags for ,Will Store in the corresponding queue ,in .

[0077] Step 9. Pass Construct a class prototype for each category. For in-distribution noise samples, select high-confidence noise samples in the distribution. Compare the cosine similarity between the high-confidence noise sample representation in the distribution and each class prototype to obtain semantic pseudo labels. ,pass Predictions for high-confidence noise samples within the distribution yield linear pseudo labels . With linear pseudo-labels Add them according to the weights to get the pseudo labels of the noise samples within the distribution.

[0078] Specifically, for each category The class prototype .

[0079] in Indicates the category Characterization Queue No. The elements are , the set of class prototypes is represented as .

[0080] The conditions for selecting high confidence noise samples within the distribution are:

[0081] and ;

[0082] in , Represents the predicted probability distribution The maximum weight, is a hyperparameter, Represents the threshold for filtering high-confidence noise samples within the distribution. The set of high-confidence noise samples within the distribution is expressed as .

[0083] Compute semantic pseudo-labels of in-distribution high-confidence noise samples via class prototypes :

[0084] .

[0085] in represents the cosine similarity, Representation of high confidence noise samples within the distribution , express The representation extractor, is a hyperparameter; .

[0086] In this embodiment The value is 0.05.

[0087] Deep Learning Models Linear pseudo-labels obtained by predicting high-confidence noise samples within the distribution Expressed as:

[0088] .

[0089] Semantic pseudo-labels With linear pseudo-labels Add by weight to get the pseudo labels of noise samples in the final distribution:

[0090] ;in represents the pseudo label weight.

[0091] Step 10. Calculate the cross entropy loss of clean samples within the distribution and high confidence noise samples within the distribution , calculate the consistency regularization term , and get the final loss .

[0092] Specifically, the classification loss The calculation formula is as follows:

[0093] .

[0094] in , Indicates the sample Perform strong data image enhancement, express logits, represents the Softmax activation function; express and The cross entropy loss, express and Cross entropy loss; For a clean sample set within the distribution, use the set represents all clean samples in the distribution, is a set of high confidence noise samples within the distribution; express The expected cross entropy loss of a sample.

[0095] Consistency regularization term Expressed as:

[0096] .

[0097] in Represents the control weight, when hour, ,otherwise , express and The KL divergence of express and KL divergence of .

[0098] Final loss ,in and Represents the loss term weight.

[0099] Step 11. By minimizing the loss , update the model parameters , through the updated renew .

[0100] Step 12. Update the current round The value of .

[0101] Determine the current round Is it less than or equal to the maximum round ; If so, go to step 3; otherwise, the training ends.

[0102] Step 13. Use the trained deep learning model Make classification predictions for the input image.

[0103] Compared with traditional methods, the present invention first uses the consistency prediction of the model to separate OOD noise samples, and assigns uniform labels to the separated OOD noise samples. The results of each component on the uniform label obey the uniform distribution, which enhances the OOD noise detection ability of the model. For ID samples, the present invention uses Jensen-Shannon divergence, namely JS divergence, as the noise separation indicator of ID samples. Taking into account the problem that the model tends to be biased towards the head category due to category imbalance in long-tail datasets, an additional semantic classifier is introduced, which is mixed with the linear pseudo-labels generated by the model prediction according to the weight to reduce the category imbalance bias. Finally, a strongly enhanced consistency regularization is introduced to further improve the generalization performance of the model. The method of the present invention solves the dual-type noise problem in unbalanced datasets well, and effectively improves the image classification performance of deep neural network models trained on multi-bias datasets.

[0104] Of course, the above description is only a preferred embodiment of the present invention, and the present invention is not limited to the above-mentioned embodiments. It should be noted that all equivalent substitutions and obvious deformation forms made by any technician familiar with this field under the guidance of this specification fall within the substantive scope of this specification and should be protected by the present invention.

Claims

1. A long-tail learning image classification method with in-distribution and out-of-distribution noisy labels, characterized by: The steps include: Step 1. Obtain a training dataset with in-distribution and out-of-distribution noise in real-world scenarios ; in, Represented by the image i.e. sample With the corresponding label The training data set consists of is the total number of samples in the dataset, , Indicates the preset label of the dataset; , Indicates the total number of data set categories and preset labels There are noisy labels in ; Step 2. Build a deep learning model and ; in and Represents deep learning models Representation extractor and classifier, and Represents deep learning models Representation extractor and classifier, The model parameters are expressed as , The model parameters are expressed as ; Build a representation storage queue for each category in the dataset , Indicates the category Determine the maximum number of training rounds And the maximum number of rounds of model parameter warm-up initialization , initialize the current round =1; Step 3. Perform label smoothing to obtain smooth labels ; Determine the current round Is it less than or equal to If the current round Less than or equal to , then go to step 4; otherwise, if the current round Greater than , then go to step 5; Step 4. Use All samples in and the corresponding smooth labels Calculating cross entropy loss , and go to step 11; Step 5. Calculate the threshold for filtering clean samples in the distribution in the current round ; Step 6. Use the deep learning model and Separate predictions The labels of the training dataset are filtered out of the distribution noise samples according to the consistency of the labels. The remaining samples in the training dataset are used as the distribution samples. The noise samples outside the distribution are mapped to the smooth labels. Each component on Set to , and independently calculate the out-of-distribution sample loss ; Step 7. Use Jensen-Shannon divergence to calculate the predicted probability distribution of samples within the distribution and the corresponding smoothed labels Likelihood , get the probability that each sample in the distribution is a non-noise label, that is, a clean sample in the distribution , the probability is greater than the threshold The samples within the distribution of are regarded as clean samples within the distribution, and the samples within other distributions are regarded as noise samples within the distribution; Step 8. Extract representations of clean samples within the distribution and pre-set labels based on the dataset , add the representation of the extracted clean sample in the distribution to the corresponding representation storage queue If the number of elements in the characterization storage queue reaches the upper limit, the element that was first added to the characterization storage queue will be removed and then added; Step 9. Store the queue by characterization Construct a class prototype for each category. For in-distribution noise samples, select high-confidence noise samples in the distribution. Use a semantic classifier to obtain semantic pseudo labels by comparing the cosine similarity between the high-confidence noise sample representation in the distribution and each class prototype. ,pass Predictions for high-confidence noise samples within the distribution yield linear pseudo labels ; Semantic pseudo-label With linear pseudo-labels Add the weights to get the pseudo labels of the noise samples within the distribution; Step 10. Calculate the cross entropy loss of clean samples within the distribution and high confidence noise samples within the distribution , calculate the consistency regularization term , get the final loss ; and represents the weight of the loss term; Step 11. By minimizing the loss , update the model parameters , through the updated renew ; Step 12. Update the current round The value of ; Determine the current round Is it less than or equal to the maximum round ; If yes, go to step 3; otherwise, the training ends; Step 13. Use the trained deep learning model Make classification predictions for the input image.

2. The long-tail learning image classification method with in-distribution and out-of-distribution noisy labels according to claim 1, characterized in that In step 3, smooth the label Is the length of vector, Each component on Expressed as: ； in, is a hyperparameter used to control the degree of label smoothing.

3. The long-tail learning image classification method with in-distribution and out-of-distribution noisy labels according to claim 1, characterized in that In step 4, the cross entropy loss The calculation is expressed as: ； in, represents the predicted probability distribution, , Represents a sample logits, represents the Softmax activation function, , express No. Quantity is the cross entropy loss function, ; Expresses the expectation of loss, .

4. The long-tail learning image classification method with in-distribution and out-of-distribution noisy labels according to claim 1, characterized in that In step 5, the threshold The calculation of the current round Related, specifically as follows: ； in, represents the upper threshold, express The initial threshold value.

5. The long-tail learning image classification method with in-distribution and out-of-distribution noisy labels according to claim 1, characterized in that The step 6 is specifically as follows: For input deep learning models and of The obtained logits are and ; Then use the prior probability of each category Debiasing is performed, and the logits after debiasing are: and ; Calculation sample is the probability of a noise sample outside the distribution for: ； in, ,Will The samples are regarded as out-of-distribution noise samples, and the out-of-distribution noise samples are labeled Each component on Set to , Represents the index of the largest component in a vector; Computing out-of-distribution sample loss , the formula is as follows: .

6. The long-tail learning image classification method with in-distribution and out-of-distribution noisy labels according to claim 1, characterized in that The step 7 is specifically as follows: Predicted probability distribution With smooth label JS divergence between The calculation method is: ； in, represents the Kullback-Leibler divergence; ， ； The probability of a clean sample within the distribution is expressed as: , ,Will The sample of is regarded as a clean sample within the distribution, The other in-distribution samples are used as in-distribution noise samples.

7. The long-tail learning image classification method with in-distribution and out-of-distribution noisy labels according to claim 1, characterized in that In step 8, the characterization formula for extracting clean samples within the distribution is as follows: ； in, Indicates passing The obtained sample The representation of express Representation extractor for According to the characterization Preset tags for ,Will Store in the corresponding queue ,in .

8. The long-tail learning image classification method with in-distribution and out-of-distribution noisy labels according to claim 1, characterized in that In step 9, calculate each category The class prototype : ； in Indicates the category Characterization Queue No. The elements are , the set of class prototypes is represented as ; Compute semantic pseudo-labels of in-distribution high-confidence noise samples via class prototypes : ； in represents the cosine similarity, Representation of high confidence noise samples within the distribution , express The representation extractor, is a hyperparameter; ; Deep Learning Models Linear pseudo-labels obtained by predicting high-confidence noise samples within the distribution Expressed as: ； Semantic pseudo-labels With linear pseudo-labels Add by weight to get the pseudo labels of noise samples in the final distribution: ;in represents the pseudo label weight.

9. The long-tail learning image classification method with in-distribution and out-of-distribution noisy labels according to claim 1, characterized in that In step 9, the conditions for selecting high-confidence noise samples within the distribution are: and ; in , Represents the predicted probability distribution The maximum weight, is a hyperparameter, Represents the threshold for filtering high-confidence noise samples within the distribution. The set of high-confidence noise samples within the distribution is expressed as .

10. The long-tail learning image classification method with in-distribution and out-of-distribution noisy labels according to claim 1, characterized in that The step 10 is specifically as follows: Classification loss The calculation formula is as follows: ； in , Indicates the sample Perform strong data image enhancement, express logits, Represents the Softmax activation function; express and The cross entropy loss, express and Cross entropy loss; is a set of clean samples within the distribution, is a set of high confidence noise samples within the distribution; express The expectation of the sample cross entropy loss; Consistency regularization term Expressed as: ； in Represents the control weight, when hour, ,otherwise , express and The KL divergence of express and KL divergence of Final loss ; and Represents the loss term weight.

Images