Long-tail learning image classification method with in-distribution and out-distribution noise labels

Abstract

The invention belongs to an image classification method, and particularly discloses a long-tail learning image classification method with in-distribution and out-distribution noise labels. According to the method, a long-tail learning framework with in-distribution and out-of-distribution noise tags in an open world is adopted to build a deep learning model, and the deep learning model is used for solving the challenging learning problem of simultaneously processing ID and OOD noise samples on long-tail data. Specifically, according to the method, firstly, OOD noise samples are filtered and uniform labels are distributed by utilizing model consistency prediction, so that the detection capability of the model on the OOD noise samples is enhanced; meanwhile, ID noise samples are separated by using JS divergence, and misclassification of clean samples in distribution is reduced, especially in a tail category; in addition, an additional semantic classifier is introduced to reduce the bias of the pseudo labels to majority categories; and finally, strong enhanced consistency regularization is introduced, so that the generalization performance and robustness of the model are further improved, and the classification performance of the model on a multi-bias data set is enhanced.

Description

Technical Field

[0001] The present invention belongs to the technical field of image classification, and in particular relates to a long-tail learning image classification method with in-distribution and out-of-distribution noise labels. Background Art

[0002] Noisy label learning is an important research area in deep learning, which focuses on how to effectively train models in the presence of inaccurate labels. Long-tail data learning targets the problem of uneven distribution of categories in data sets, especially in the field of image recognition in deep learning. In the long-tail distribution, the number of samples of a few categories, namely the head categories, dominates, while the number of samples of a large number of categories, namely the tail categories, is scarce. This situation makes it easy for the model to overfit common categories during training, while the recognition ability of rare categories is insufficient. Long-tail data learning aims to improve the model's recognition ability of tail categories through various strategies, such as data augmentation, resampling, and category reweighting, thereby improving the overall performance.

[0003] Deep neural networks have achieved great success in image classification using large-scale labeled datasets. However, due to the scarcity of certain data and the high cost of labeling, it is challenging to obtain large-scale well-labeled samples for training. A common alternative is to collect samples from the Internet, which often leads to data bias in the constructed training datasets, such as Figure 2 As shown in Figure 2, images crawled from the web usually exhibit unbalanced category distribution and long-tail characteristics, and samples obtained through query search may also have noisy labels that are inconsistent with the actual categories.

[0004] This kind of noisy labels can be divided into Out Of Distribution (OOD) noisy labels where the samples belong to unknown categories, and In Distribution (ID) noisy labels where the samples come from known categories but have wrong labels.

[0005] In long-tail datasets, the feature space is largely influenced by the majority of head categories, causing classifiers trained on these distributions to favor the majority and ignore the minority of tail categories. In addition, deep neural networks are prone to overfitting noisy labels, resulting in decreased generalization performance. Therefore, image classification with deep neural networks faces a complex scenario, where the dataset has multiple biases such as long-tail category imbalance, ID noise, and OOD noise. Traditional methods focus on solving category imbalance or noisy labels, but rarely consider both at the same time, especially when ID and OOD noise exist at the same time. The true label of OOD noise is outside the dataset, and incorrectly assigning labels within the distribution to OOD noise will reduce the classification performance of the neural network. Summary of the invention

[0006] The purpose of the present invention is to propose a long-tail learning image classification method with in-distribution and out-of-distribution noise labels, which is beneficial to solve the problem of the simultaneous existence of ID noise and OOD noise samples in long-tail data, improve the generalization performance and robustness of the deep learning model, and enhance the classification performance of the model on multi-biased data sets.

[0007] In order to achieve the above object, the present invention adopts the following technical scheme:

[0008] A long-tail learning image classification method with in-distribution and out-of-distribution noisy labels comprises the following steps:

[0009] Step 1. Obtain a training dataset with in-distribution and out-of-distribution noise in real-world scenarios ;

[0010] in, Represented by the image, i.e., the sample With the corresponding label The training data set consists of is the total number of samples in the dataset, , Indicates the preset label of the dataset;

[0011] , Indicates the total number of data set categories and preset labels There are noisy labels in ;

[0012] Step 2. Build a deep learning model and ;

[0013] in and Represents the deep learning model Representation extractor and classifier of and Represents the deep learning model Representation extractor and classifier of The model parameters are expressed as , The model parameters are expressed as ;

[0014] Construct a representation storage queue for each category in the dataset , Indicates the category The representation storage queue at the time; determine the maximum number of training rounds And the maximum number of rounds of model parameter warm-up initialization , initialize the current round =1;

[0015] Step 3. Perform label smoothing to obtain smooth labels ; Determine the current round Is it less than or equal to ; If the current round Less than or equal to , then go to step 4; otherwise, if the current round Greater than , then go to step 5;

[0016] Step 4. Use All samples in and the corresponding smooth label Calculating cross entropy loss , and go to step 11;

[0017] Step 5. Calculate the threshold for filtering clean samples in the distribution in the current round ;

[0018] Step 6. Use the deep learning model and Separate predictions The labels of the training dataset are filtered out according to the consistency of the labels. The remaining samples in the training dataset are used as the samples in the distribution. The noise samples in the distribution are mapped to the smooth labels. Each component on Set to , and independently calculate the out-of-distribution sample loss ;

[0019] Step 7. Use Jensen-Shannon divergence to calculate the predicted probability distribution of samples within the distribution and the corresponding smoothed labels The likelihood , get the probability that each sample in the distribution is a non-noise label, that is, a clean sample in the distribution , the probability is greater than the threshold The samples within the distribution of are regarded as clean samples within the distribution, and the samples within other distributions are regarded as noise samples within the distribution;

[0020] Step 8. Extract the representation of clean samples within the distribution and preset labels according to the dataset , add the representation of the clean sample in the extracted distribution to the corresponding representation storage queue If the number of elements in the characterization storage queue reaches the upper limit, remove the earliest element added to the characterization storage queue and then add it;

[0021] Step 9. Store the queue by characterization Construct a class prototype for each category. For noise samples within the distribution, select high-confidence noise samples within the distribution. Use a semantic classifier to obtain semantic pseudo labels by comparing the cosine similarity between the high-confidence noise sample representation within the distribution and each class prototype. ,pass Predictions for high-confidence noise samples within the distribution yield linear pseudo labels ; Semantic pseudo-label With linear pseudo-labels Add the weights to get the pseudo labels of noise samples within the distribution;

[0022] Step 10. Calculate the cross entropy loss of clean samples within the distribution and high confidence noise samples within the distribution , calculate the consistency regularization term , get the final loss ; and represents the weight of the loss term;

[0023] Step 11. By minimizing the loss , update the model parameters , through the updated renew ;

[0024] Step 12. Update the current round The value of ;

[0025] Determine the current round Is it less than or equal to the maximum round? ; If yes, go to step 3; otherwise, the training ends;

[0026] Step 13. Use the trained deep learning model Make classification predictions for the input image.

[0027] The present invention has the following advantages:

[0028] As shown above, the present invention relates to a long-tail learning image classification method with in-distribution and out-of-distribution noise labels. In each training round, for each batch of iterations, the method of the present invention first uses the consistency prediction of the model to separate the OOD noise samples, and assigns uniform labels to the OOD noise samples, thereby enhancing the model's ability to distinguish between ID samples and OOD noise samples. For ID noise samples, the method of the present invention uses JS divergence as the noise separation indicator for ID samples, and the recall rate and precision of the tail samples are better than the loss-based method, effectively preventing the tail samples with large losses from being misclassified as ID noise samples. In order to further solve the bias of pseudo-labels to the majority category in unbalanced data sets, the method of the present invention additionally introduces a semantic classifier; specifically, the representation of the clean samples in the distribution is stored in the representation storage queue according to the preset labels of the data set, and then the class prototype is constructed from the representation storage queue and compared with the representation of the ID noise samples to generate semantic pseudo-labels, which are then mixed with linear pseudo-labels to restore the true labels of the ID noise samples. In addition, the method of the present invention also applies data enhancement and consistency regularization to further improve the model's ability to learn meaningful representations. Actual experimental results show that data augmentation and consistency regularization can effectively enhance the classification performance of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 is a flow chart of a long-tail learning image classification method with in-distribution and out-of-distribution noise labels in an embodiment of the present invention;

[0030] Figure 2 is a schematic diagram of the distribution of a long-tail data set including in-distribution noise samples and out-of-distribution noise samples in an embodiment of the present invention;

[0031] Figure 3 It is a structural diagram of the deep learning model constructed in an embodiment of the present invention. DETAILED DESCRIPTION

[0032] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments:

[0033] In this embodiment, a long-tail learning image classification method with in-distribution and out-of-distribution noise labels is proposed. The method adopts a long-tail learning framework with in-distribution and out-of-distribution noise labels in the open world, and builds a deep learning model to solve the challenging learning problem of simultaneously processing ID and OOD noise samples on long-tail data. First, the method of the present invention separates OOD noise samples based on prediction consistency, while considering class imbalance, and assigns uniform labels to OOD noise samples to enhance the discrimination ability of the model. Next, the JS divergence between the predicted probability distribution and the corresponding label is used to filter the ID noise samples. For ID noise, an additional semantic classifier is mixed to generate class-balanced pseudo-labels. In addition, consistency regularization is introduced to encourage the model to produce consistent predictions under different enhancements, thereby improving its generalization ability.

[0034] like Figure 1 As shown in FIG. 1 , the long-tail learning image classification method with in-distribution and out-of-distribution noisy labels includes the following steps:

[0035] Step 1. Obtain a training dataset with in-distribution and out-of-distribution noise in real-world scenarios .

[0036] in, Represented by the image, i.e., the sample With the corresponding label The training data set consists of is the total number of samples in the dataset, ; , Represents the total number of data set categories, Indicates category.

[0037] There are noisy labels in the dataset, that is, there are preset labels and The actual label does not match.

[0038] Step 2. Build a deep learning model and .

[0039] in, and Respectively Representation extractor and classifier of and Respectively Representation extractor and classifier of The model parameters are expressed as , The model parameters are expressed as .

[0040] Construct a representation storage queue for each category in the dataset , Indicates the category Determine the maximum number of training rounds And the maximum number of rounds of model parameter warm-up initialization , initialize the current round =1.

[0041] Step 3. Perform label smoothing to obtain smooth labels .

[0042] Specifically, the smoothing label Is the length The vector of Each component on It is expressed as:

[0043] ;

[0044] in, is a hyperparameter used to control the degree of label smoothing, Indicates the preset label of the dataset.

[0045] Determine the current round Is it less than or equal to ; If the current round Less than or equal to , then go to step 4, otherwise, if the current round Greater than , go to step 5.

[0046] Step 4. Use all samples of the training dataset and the corresponding smooth label Calculating cross entropy loss , go to step 11.

[0047] For the cross entropy loss The calculation is expressed as:

[0048] .

[0049] in, represents the predicted probability distribution, , express logits, represents the Softmax activation function, , express No. A quantity.

[0050] is the cross entropy loss function, .

[0051] Indicates the expectation of loss, .

[0052] Step 5. Calculate the threshold for filtering clean samples in the distribution in the current round .

[0053] for Calculation and current round Related, the formula is as follows:

[0054] ;

[0055] in, represents the upper threshold, express The initial threshold value.

[0056] Step 6. Use the deep learning model and Separate predictions .

[0057] Filter samples based on label consistency The out-of-distribution noise samples in The remaining samples in the distribution are regarded as samples within the distribution, and the noise samples outside the distribution are corresponding to labels Each component on Set to , and calculate the out-of-distribution sample loss .

[0058] Here, label consistency means that for the same ,use and The label predictions obtained separately should be consistent.

[0059] Specifically, for the input and of The obtained logits are and Considering the class imbalance, the prior probability of each class is used After debiasing, the logits are:

[0060] and .

[0061] The probability that the sample is a noise sample outside the distribution for:

[0062] .

[0063] Apparently ,Will The samples of out-of-distribution noise are regarded as out-of-distribution noise samples, and the corresponding labels Each component on Set to .

[0064] Further calculation of out-of-distribution sample loss : .

[0065] Step 7. Use Jensen-Shannon divergence to calculate the predicted probability distribution of samples in the distribution and the corresponding labels The likelihood , get the probability that each sample in the distribution is a non-noise label, that is, a clean sample in the distribution , the probability is greater than The samples in the distribution are regarded as clean samples in the distribution, and the remaining samples, i.e., the probability is not greater than The samples within the distribution of are used as the noise samples within the distribution.

[0066] Specifically, the predicted probability distribution With smooth label JS divergence between The calculation method is:

[0067] .

[0068] in, represents the Kullback-Leibler divergence;

[0069] , .

[0070] The probability of a clean sample within the distribution is expressed as:

[0071] , ,Will The sample of is regarded as a clean sample within the distribution, The other in-distribution samples of are used as in-distribution noise samples.

[0072] Step 8. Extract representations for clean samples within the distribution according to preset labels , add the representation of the clean sample in the distribution to the corresponding representation storage queue If the number of elements in the characterization storage queue reaches the upper limit, the earliest element added to the characterization storage queue is removed and then added.

[0073] Specifically, the characterization formula for extracting clean samples within the distribution is as follows:

[0074] .

[0075] in, Indicates that What you get Characterization, express Characterization extractor.

[0076] according to Preset tags for ,Will Store in the corresponding queue ,in .

[0077] Step 9. Pass Construct a class prototype for each category. For noise samples within the distribution, select high-confidence noise samples within the distribution. Compare the cosine similarity between the representation of high-confidence noise samples within the distribution and each class prototype to obtain semantic pseudo labels. ,pass Predictions for high-confidence noise samples within the distribution yield linear pseudo labels . With linear pseudo-labels Add them by weight to get the pseudo labels of noise samples within the distribution.

[0078] Specifically, for each category The class prototype .

[0079] in Indicates the category Characterization Cohort No. The element is , the set of class prototypes is represented as .

[0080] The conditions for selecting high confidence noise samples within the distribution are:

[0081] and ;

[0082] in , Represents the predicted probability distribution The maximum weight, is a hyperparameter, represents the threshold for filtering high-confidence noise samples in the distribution. The set of high-confidence noise samples in the distribution is expressed as .

[0083] Compute semantic pseudo labels of high-confidence noise samples within the distribution through class prototypes :

[0084] .

[0085] in represents the cosine similarity, Characterization of high confidence noise samples within the distribution , express The representation extractor of is a hyperparameter; .

[0086] In this embodiment The value is 0.05.

[0087] Deep Learning Models Linear pseudo labels obtained by predicting high-confidence noise samples within the distribution It is expressed as:

[0088] .

[0089] Semantic pseudo-labels With linear pseudo-labels Add by weight to get the pseudo labels of noise samples in the final distribution:

[0090] ;in represents the pseudo label weight.

[0091] Step 10. Calculate the cross entropy loss of clean samples within the distribution and high confidence noise samples within the distribution , calculate the consistency regularization term , and the final loss .

[0092] Specifically, the classification loss The calculation formula is as follows:

[0093] .

[0094] in , Indicates the sample Perform strong data image enhancement, express logits, Represents the Softmax activation function; express and The cross entropy loss is express and The cross entropy loss of For a set of clean samples within the distribution, use the set represents all clean samples in the distribution, is a set of high confidence noise samples within the distribution; express The expectation of the sample cross entropy loss.

[0095] Consistency regularization term It is expressed as:

[0096] .

[0097] in represents the control weight, when hour, ,otherwise , express and The KL divergence of express and The KL divergence of .

[0098] Final loss ,in and Represents the weight of the loss term.

[0099] Step 11. By minimizing the loss , update the model parameters , through the updated renew .

[0100] Step 12. Update the current round The value of .

[0101] Determine the current round Is it less than or equal to the maximum round? ; If so, go to step 3; otherwise, the training ends.

[0102] Step 13. Use the trained deep learning model Make classification predictions for the input image.

[0103] Compared with the traditional method, the present invention first uses the consistency prediction of the model to separate the OOD noise samples, and assigns uniform labels to the separated OOD noise samples. The result of each component on the uniform label obeys the uniform distribution, which enhances the OOD noise detection ability of the model. For ID samples, the present invention adopts Jensen-Shannon divergence, namely JS divergence, as the noise separation index of ID samples. Taking into account the problem that the model tends to the head category due to category imbalance in long-tail data sets, an additional semantic classifier is introduced, which is mixed with the linear pseudo-labels generated by the model prediction according to the weight to reduce the category imbalance bias. Finally, a strongly enhanced consistency regularization is introduced to further improve the generalization performance of the model. The method of the present invention solves the problem of dual-type noise in unbalanced data sets well, and effectively improves the image classification performance of deep neural network models trained on multi-bias data sets.

[0104] Of course, the above description is only a preferred embodiment of the present invention, and the present invention is not limited to the above embodiments. It should be noted that all equivalent substitutions and obvious deformation forms made by any technician familiar with the field under the guidance of this specification fall within the essential scope of this specification and should be protected by the present invention.

Claims

1. A long-tail learning image classification method with in-distribution and out-of-distribution noisy labels, characterized in that The steps include: Step 1. Obtain a training dataset with in-distribution and out-of-distribution noise in real-world scenarios ; in, Represented by the image, i.e., the sample With the corresponding label The training data set consists of is the total number of samples in the dataset, , Indicates the preset label of the dataset; , Indicates the total number of data set categories and preset labels There are noisy labels in ; Step 2. Build a deep learning model and ; in and Represents the deep learning model Representation extractor and classifier of and Represents the deep learning model Representation extractor and classifier of The model parameters are expressed as , The model parameters are expressed as ; Construct a representation storage queue for each category in the dataset , Indicates the category Determine the maximum number of training rounds And the maximum number of rounds of model parameter warm-up initialization , initialize the current round =1; Step 3. Perform label smoothing to obtain smooth labels ; Determine the current round Is it less than or equal to ; If the current round Less than or equal to , then go to step 4; otherwise, if the current round Greater than , then go to step 5; Step 4. Use All samples in and the corresponding smooth label Calculating cross entropy loss , and go to step 11; Step 5. Calculate the threshold for filtering clean samples in the distribution in the current round ; Step 6. Use the deep learning model and Separate predictions The labels of the training dataset are filtered out according to the consistency of the labels. The remaining samples in the training dataset are used as the samples in the distribution. The noise samples in the distribution are mapped to the smooth labels. Each component on Set to , and independently calculate the out-of-distribution sample loss ; Step 7. Use Jensen-Shannon divergence to calculate the predicted probability distribution of samples within the distribution and the corresponding smoothed labels The likelihood , get the probability that each sample in the distribution is a non-noise label, that is, a clean sample in the distribution , the probability is greater than the threshold The samples within the distribution of are regarded as clean samples within the distribution, and the samples within other distributions are regarded as noise samples within the distribution; Step 8. Extract the representation of clean samples within the distribution and preset labels according to the dataset , add the representation of the clean sample in the extracted distribution to the corresponding representation storage queue If the number of elements in the characterization storage queue reaches the upper limit, remove the earliest element added to the characterization storage queue and then add it; Step 9. Store the queue by characterization Construct a class prototype for each category. For noise samples within the distribution, select high-confidence noise samples within the distribution. Use a semantic classifier to obtain semantic pseudo labels by comparing the cosine similarity between the high-confidence noise sample representation within the distribution and each class prototype. ,pass Predictions for high-confidence noise samples within the distribution yield linear pseudo labels ; Semantic pseudo-label With linear pseudo-labels Add the weights to get the pseudo labels of noise samples within the distribution; Step 10. Calculate the cross entropy loss of clean samples within the distribution and high confidence noise samples within the distribution , calculate the consistency regularization term , get the final loss ; and represents the weight of the loss term; Step 11. By minimizing the loss , update the model parameters , through the updated renew ; Step 12. Update the current round The value of ; Determine the current round Is it less than or equal to the maximum round? ; If yes, go to step 3; otherwise, the training ends; Step 13. Use the trained deep learning model Make classification predictions for the input image.

2. The long-tail learning image classification method with in-distribution and out-of-distribution noise labels according to claim 1, characterized in that: In step 3, smooth the label Is the length The vector of Each component on It is expressed as: ； in, is a hyperparameter used to control the degree of label smoothing.

3. The long-tail learning image classification method with in-distribution and out-of-distribution noise labels according to claim 1, characterized in that: In step 4, the cross entropy loss The calculation is expressed as: ； in, represents the predicted probability distribution, , Representation sample logits, represents the Softmax activation function, , express No. Quantity; is the cross entropy loss function, ; Indicates the expectation of loss, .

4. The long-tail learning image classification method with in-distribution and out-of-distribution noise labels according to claim 1, characterized in that: In step 5, the threshold The calculation of the current round Related, as follows: ； in, represents the upper threshold, express The initial threshold value.

5. The long-tail learning image classification method with in-distribution and out-of-distribution noise labels according to claim 1, characterized in that: The step 6 is specifically as follows: For input deep learning models and of The obtained logits are and ; Then use the prior probability of each category After debiasing, the logits are: and ; Calculation Sample is the probability of a noise sample outside the distribution for: ； in, ,Will The samples are regarded as out-of-distribution noise samples, and the out-of-distribution noise samples are corresponding to labels Each component on Set to , Represents the index of the largest component in a vector; Computing out-of-distribution sample loss , the formula is as follows: .

6. The long-tail learning image classification method with in-distribution and out-of-distribution noise labels according to claim 1, characterized in that: The step 7 is specifically as follows: Predicted probability distribution With smooth label JS divergence between The calculation method is: ； in, represents the Kullback-Leibler divergence; ， ； The probability of a clean sample within the distribution is expressed as: , ,Will The sample of is regarded as a clean sample within the distribution, The other in-distribution samples of are used as in-distribution noise samples.

7. The long-tail learning image classification method with in-distribution and out-of-distribution noise labels according to claim 1, characterized in that: In step 8, the characterization formula for extracting clean samples in the distribution is as follows: ； in, Indicates that The sample obtained The representation of express Characterization extractor for According to the characterization Preset tags for ,Will Store in the corresponding queue ,in .

8. The long-tail learning image classification method with in-distribution and out-of-distribution noise labels according to claim 1, characterized in that: In step 9, calculate each category The class prototype : ； in Indicates the category Characterization Cohort No. The element is , the set of class prototypes is represented as ; Compute semantic pseudo labels of high-confidence noise samples within the distribution through class prototypes : ； in represents the cosine similarity, Characterization of high confidence noise samples within the distribution , express The representation extractor of is a hyperparameter; ; Deep Learning Models Linear pseudo-labels obtained by predicting high-confidence noise samples within the distribution It is expressed as: ； Semantic pseudo-labels With linear pseudo-labels Add by weight to get the pseudo labels of noise samples in the final distribution: ;in represents the pseudo label weight.

9. The long-tail learning image classification method with in-distribution and out-of-distribution noise labels according to claim 1, characterized in that: In step 9, the condition for selecting high confidence noise samples within the distribution is: and ; in , Represents the predicted probability distribution The maximum weight, is a hyperparameter, represents the threshold for filtering high-confidence noise samples in the distribution. The set of high-confidence noise samples in the distribution is expressed as .

10. The long-tail learning image classification method with in-distribution and out-of-distribution noise labels according to claim 1, characterized in that: The step 10 is specifically as follows: Classification Loss The calculation formula is as follows: ； in , Indicates the sample Perform strong data image enhancement, express logits, Represents the Softmax activation function; express and The cross entropy loss is express and The cross entropy loss of is a set of clean samples within the distribution, is a set of high confidence noise samples within the distribution; express The expectation of the sample cross entropy loss; Consistency regularization term It is expressed as: ； in represents the control weight, when hour, ,otherwise , express and The KL divergence of express and KL divergence of Final loss ; and Represents the weight of the loss term.

Images