Image classification method for long-tail category distribution

Abstract

The invention discloses a long-tail category distribution-oriented image classification method. The method comprises the following steps of: collecting training data and category labels thereof by a user; performing linear interpolation on the training data by using a data mixing technology, and constructing a new label sample pair; training three different expert models by using the newly constructed label sample pair; the optimal integration weight of the expert model on the test distribution is learned by using the test data without labels, and the generalization performance during the multi-expert test is improved; using the optimally integrated expert model to predict the test data again to obtain a final classification result; and if the user is satisfied with the prediction result, ending, otherwise, collecting more training data and labels, and returning to execute the second step. According to the method, the generalization effect of the multi-expert integration model on unknown test distribution is improved, and a new scheme is provided for solving the long tail problem in a real scene.

Description

Technical Field

[0001] The present invention relates to the field of image classification, and in particular to an image classification method oriented to long-tail category distribution. Background Art

[0002] Real-world visual recognition datasets usually present a long-tail category distribution, that is, a small number of categories contain a large number of samples, while most other categories contain only a small number of samples. Due to the imbalanced category distribution, the trained model is easily biased towards the head class with more samples, and performs poorly on the tail class.

[0003] Long-tail learning is one of the mainstream research directions to solve the problem of imbalanced category distribution. Most current long-tail research only considers that the training data follows the long-tail category distribution and assumes that the test data follows the uniform category distribution. However, this assumption ignores the fact that test data in real scenarios often have unknown distribution shifts, which makes existing methods fail to solve the actual long-tail problem. Summary of the invention

[0004] The purpose of the present invention is to provide an image classification method for long-tail category distribution. The method is suitable for image classification scenarios where the training set obeys the long-tail category distribution and the test set obeys the unknown category distribution, so as to solve the long-tail problem in real scenarios.

[0005] To achieve the above functions, the present invention designs an image classification method for long-tail category distribution, and executes the following steps S1 to S6 to complete the classification prediction of the image:

[0006] Step S1: Collect original images as training samples, annotate corresponding category labels, and construct a long-tail training data set;

[0007] Step S2: Using data mixing technology, linearly interpolate the original images and category labels in the long-tail training data set constructed in step S1 to construct new images and category labels;

[0008] Step S3: construct a multi-expert model, wherein the multi-expert model has expert models with different prediction biases, and each expert model is based on a loss function with a different logical adjustment coefficient, including expert models for positive long-tail distribution, uniform distribution, and reverse long-tail distribution. The differences between the expert models are ensured by retaining independent expert model network parameters. The multi-expert model is trained using the images and category labels obtained in step S2.

[0009] Step S4: construct a test set using unlabeled images as test samples, and input the images in the test set into the multi-expert model respectively to test the generalization performance of the multi-expert model in an unknown test distribution and obtain the optimal integration weight of the multi-expert model;

[0010] Step S5: Use the optimal integrated multi-expert model to predict the images in the test set again to obtain the final classification result;

[0011] Step S6: If the user is satisfied with the final classification result, the process ends; otherwise, more training samples and labels are collected and step S2 is executed.

[0012] Beneficial effects: Compared with the prior art, the advantages of the present invention include:

[0013] The general long-tail setting only considers the long-tail distribution of the training set and the balanced distribution of the test set. In real-world applications, the distribution of the test set may be diverse, which may cause the long-tail method designed for the balanced distribution of the test set to perform poorly. In order to consider the uncertainty of the distribution of test data, the present invention first uses the training data to learn multiple expert models with different prediction biases. Then, pseudo-labels of test data are generated through the expert model, and an adaptive threshold is designed to screen high-confidence pseudo-labels to optimize consistency regularization, automatically learn the optimal linear combination of each expert model, and improve the generalization effect of the multi-expert integrated model on the unknown test distribution, providing a new solution to solve the long-tail problem in real scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 is a flowchart of an image classification method for long-tail category distribution provided according to an embodiment of the present invention;

[0015] Figure 2 It is a flowchart of screening unlabeled test data for model optimization according to an embodiment of the present invention. DETAILED DESCRIPTION

[0016] The present invention will be further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the protection scope of the present invention.

[0017] An image classification method for long-tail category distribution provided by an embodiment of the present invention refers to Figure 1 First, the user collects the original images and their category labels as training data, and each category in the training data contains a different number of samples. Then, the mixed data is used to train three experts with different prediction biases through different loss functions, and different experts can cope with different test distributions. Next, the test data is input, and the existing expert model is used to generate pseudo labels for the test data, and then the adaptive threshold method proposed in the present invention is used to screen out pseudo labels with high confidence, and the expert integration weights are updated through consistency regularization. After the update is completed, the test data is predicted again using the optimal integrated expert model to obtain the final classification result; if the user is not satisfied with the prediction result, the training data and labels are collected again to train the expert model.

[0018] The specific steps are as follows:

[0019] Step S1: Collect original images as training samples, annotate corresponding category labels, and construct a long-tail training data set;

[0020] The constructed long-tail training dataset is Among them, x i represents the i-th training sample, y i Represents the training sample x i The category label, n s Represents the sum of training samples of each category in the long-tail training dataset.

[0021] Step S2: Using data mixing technology, linearly interpolate the original images and category labels in the long-tail training data set constructed in step S1 to construct new images and category labels;

[0022] Step S2 method is as follows:

[0023] For any two training samples and class labels (x i ,y i ) and (x j ,y j ), first sample the mixed weight λ from the Beta distribution, then perform linear interpolation according to the weight to construct a new pair of training samples and category labels

[0024] λ~Beta(α,α)

[0025] λ=max(λ,1-λ)

[0026]

[0027] Among them, Beta distribution is a continuous probability density distribution, α is the Beta distribution parameter, and the mixing weight λ is randomly sampled from the Beta distribution, λ∈[0,1].

[0028] Step S3: construct a multi-expert model, wherein the multi-expert model has expert models with different prediction biases, and each expert model is based on a loss function with a different logical adjustment coefficient, including expert models for positive long-tail distribution, uniform distribution, and reverse long-tail distribution. The differences between the expert models are ensured by retaining independent expert model network parameters. The multi-expert model is trained using the images and category labels obtained in step S2.

[0029] The multi-expert model constructed in step S3 is as follows:

[0030] The expert models are named expert 0, expert 1, and expert 2 respectively; expert 0 is a positive expert model, and its prediction bias is a positive long-tail distribution; at this time, the model assumes that the training data category distribution p s (y) and test data category distribution p t (y) is the same, that is, p t (y) = p s (y); where the loss function l0 of the model optimization at this time is the standard cross entropy loss;

[0031] Expert 1 is a balanced expert model, which is used to process the test set with the same distribution as the long-tail training set. Its prediction bias is uniform distribution, which is expressed as p t (y) = 1 / C, and its loss function l1 is the balanced cross entropy loss;

[0032] Expert 2 is a reverse expert model, which is used to process the test set with the opposite distribution to the long-tail training set. Its prediction bias is the reverse long-tail distribution, which is expressed as p t (y) = inverse(p s (y)), its loss function l2 is the inverse cross entropy loss; the loss functions of the three expert models are specifically expressed as follows:

[0033]

[0034] Among them, D s represents the long-tail training data set, n s represents the sum of training samples of each category in the long-tail training dataset, θ represents the shared backbone network, and f E represents the independent network of the expert model E, E∈{0,1,2}, and σ represents the softmax function.

[0035] Step S4: construct a test set using unlabeled images as test samples, and input the images in the test set into the multi-expert model respectively to test the generalization performance of the multi-expert model in an unknown test distribution and obtain the optimal integration weight of the multi-expert model;

[0036] The specific steps of step S4 are as follows:

[0037] Step S4.1: Filter high-confidence test samples through adaptive thresholds, optimize consistency regularization, and learn the optimal integration weights of the expert model; for the screening of test samples, design adaptive global thresholds and local thresholds. The global threshold should take into account the time difference of learning difficulty during testing, which should be related to the model's prediction confidence for unlabeled test data and reflect the overall learning state. Therefore, the global threshold τ is set to the average prediction confidence of the model for the test data, and the momentum is updated using the exponential moving average method.

[0038] Figure 2A detailed description of using adaptive thresholds to filter high confidence test samples is given. Let B represent the number of test samples in each batch, then the test samples in the test set are divided into batch, Indicates the subscript of the batch; the screening of each batch of test samples constitutes a loop body, and b is initialized to 1. Specifically, for each loop, first determine whether each batch of samples has been traversed. If so, end. If not, calculate the global threshold of the current batch of samples to consider the difference in thresholds during training time. k : k∈(1,…,B) represents the subscript of the test sample in a batch, q k Represents the test sample x k Prediction probability distribution after weak enhancement;

[0039] Step S4.2: The global threshold is expressed as follows:

[0040]

[0041] In the formula, τ b represents the global threshold, m∈(0,1), is the momentum parameter, and max(·) represents the highest prediction confidence of the test sample;

[0042] Step S4.3: The local threshold should take into account the differences in learning difficulty between categories during testing. The threshold for well-learned categories should be correspondingly higher, while a lower threshold should be used for screening difficult categories. Similarly, the expectation of the model's prediction for each category c is calculated in an exponential moving average manner to estimate the category-specific learning state; considering the differences in thresholds between categories, the local threshold is expressed as follows:

[0043]

[0044] In the formula, p b (c) represents the local threshold, c represents the category, and C represents the total number of categories;

[0045] Step S4.3: Integrate the global threshold and the local threshold with the maximum normalization of the local threshold, then the adaptive threshold τ of the b-th batch of test samples in category c is b (c) is:

[0046]

[0047] Step S4.3: Filter the pseudo labels of the test samples according to the threshold, retain the samples and pseudo labels above the threshold, and optimize the model weights through consistency regularization.

[0048] The category corresponding to the highest prediction confidence of the weakly enhanced image is used as the pseudo-label, and the prediction of the strongly enhanced image is restricted to be consistent with the pseudo-label through the cross entropy loss, so as to optimize the expert integration weight. represents the learnable expert ensemble weights, initialized with w0=w1=w2=1 / 3, and after each update, the weights are kept to always satisfy ∑ E w E =1, E∈{0,1,2}; then the prediction probability p of the integrated multi-expert model for the test sample x DE (x) is expressed as:

[0049]

[0050] Step S4.4: Define weak enhancement α(·) and strong enhancement Α(·). Then the predicted probability of the test sample x after weak and strong enhancement is expressed as:

[0051] q=p DE (α(x))

[0052] Q=p DE (A(x))

[0053] In the formula, q represents the predicted probability of the test sample x after weak enhancement, and Q represents the predicted probability of the test sample x after strong enhancement;

[0054] The consistency regularization loss is expressed as the cross entropy loss of q and Q

[0055]

[0056] in, Indicates that the predicted probability distribution of q is converted into a one-hot encoding, c represents the category, and C represents the total number of categories;

[0057] Step S4.5: Consistency regularization loss function l after testing samples are screened during testing t It is expressed as:

[0058]

[0059] In the formula, It is an indicator function that returns 1 if the condition in the parentheses is met, otherwise it returns 0.

[0060] Step S5: Use the optimal integrated multi-expert model to predict the images in the test set again to obtain the final classification result;

[0061] Step S6: If the user is satisfied with the final classification result, the process ends; otherwise, more training samples and labels are collected and step S2 is executed.

[0062] In summary, the present invention proposes a multi-expert long-tail learning algorithm for agnostic test distribution. The algorithm is mainly divided into two stages. The first stage uses long-tail training data to learn multiple expert classification models with different prediction biases; the second stage uses the integrated expert model to generate pseudo labels for test data, and uses adaptive thresholds to filter out confident samples for training, so as to learn the optimal combination of expert models under the current test distribution.

[0063] The embodiments of the present invention are described in detail above with reference to the accompanying drawings, but the present invention is not limited to the above embodiments, and various changes can be made within the knowledge scope of ordinary technicians in this field without departing from the purpose of the present invention.

Claims

1. An image classification method for long-tail category distribution, characterized in that: Execute the following steps S1 to S6 to complete the classification prediction of the image: Step S1: Collect original images as training samples, annotate corresponding category labels, and construct a long-tail training data set; Step S2: Using data mixing technology, linearly interpolate the original images and category labels in the long-tail training data set constructed in step S1 to construct new images and category labels; Step S3: construct a multi-expert model, wherein the multi-expert model has expert models with different prediction biases, and each expert model is based on a loss function with a different logical adjustment coefficient, including expert models for positive long-tail distribution, uniform distribution, and reverse long-tail distribution. The differences between the expert models are ensured by retaining independent expert model network parameters. The multi-expert model is trained using the images and category labels obtained in step S2. Step S4: construct a test set using unlabeled images as test samples, and input the images in the test set into the multi-expert model respectively to test the generalization performance of the multi-expert model in an unknown test distribution and obtain the optimal integration weight of the multi-expert model; Step S5: Use the optimal integrated multi-expert model to predict the images in the test set again to obtain the final classification result; Step S6: If the user is satisfied with the final classification result, the process ends; otherwise, more training samples and labels are collected and step S2 is executed.

2. The image classification method for long-tail category distribution according to claim 1, characterized in that: The long-tail training dataset constructed in step S1 is Among them, x i represents the i-th training sample, y i Represents the training sample x i The category label, n s Represents the sum of training samples of each category in the long-tail training dataset.

3. The image classification method for long-tail category distribution according to claim 1, characterized in that: Step S2 method is as follows: For any two training samples and class labels (x i ,y i ) and (x j ,y j ), first sample the mixed weight λ from the Beta distribution, then perform linear interpolation according to the weight to construct a new pair of training samples and category labels λ~Beta(α,α) λ=max(λ,1-λ) Among them, Beta distribution is a continuous probability density distribution, α is the Beta distribution parameter, and the mixing weight λ is randomly sampled from the Beta distribution, λ∈[0,1].

4. The image classification method for long-tail category distribution according to claim 1, characterized in that: The multi-expert model constructed in step S3 is as follows: The expert models are named expert 0, expert 1, and expert 2 respectively; expert 0 is a positive expert model, and its prediction bias is a positive long-tail distribution; at this time, the model assumes that the training data category distribution p s (y) and test data category distribution p t (y) is the same, that is, p t (y) = p s (y); where the loss function l0 of the model optimization at this time is the standard cross entropy loss; Expert 1 is a balanced expert model, and its prediction bias is uniform distribution, expressed as p t (y) = 1 / C, where C represents the total number of categories and its loss function l1 is the balanced cross entropy loss; Expert 2 is a reverse expert model, whose prediction bias is reverse long-tail distribution, expressed as p t (y) = inverse(p s (y)), its loss function l2 is the inverse cross entropy loss; the loss functions of the three expert models are specifically expressed as follows: Among them, D s represents the long-tail training data set, n s represents the sum of training samples of each category in the long-tail training dataset, θ represents the shared backbone network, and f E represents the independent network of the expert model E, E∈{0,1,2}, and σ represents the softmax function.

5. The image classification method for long-tail category distribution according to claim 1, characterized in that: The specific steps of step S4 are as follows: Step S4.1: Design adaptive global and local thresholds for the screening of test samples. Let B represent the number of test samples in each batch. Then the test samples in the test set are divided into batch, Indicates the subscript of the batch; x k : k∈(1,…,B) represents the subscript of the test sample in a batch, q k Represents the test sample x k Prediction probability distribution after weak enhancement; Step S4.2: The global threshold is expressed as follows: In the formula, τ b represents the global threshold, m∈(0,1), is the momentum parameter, and max(·) represents the highest prediction confidence of the test sample; Step S4.3: The local threshold is expressed as follows: In the formula, p b (c) represents the local threshold, c represents the category, and C represents the total number of categories; Step S4.3: Integrate the global threshold and the local threshold with the maximum normalization of the local threshold, then the adaptive threshold τ of the b-th batch of test samples in category c is b (c) is: Step S4.3: Let represents the learnable expert ensemble weights, initialized with w0=w1=w2=1 / 3, and after each update, the weights are kept to always satisfy ∑ E w E =1, E∈{0,1,2}; then the prediction probability p of the integrated multi-expert model for the test sample x DE (x) is expressed as: Step S4.4: Define weak enhancement α(·) and strong enhancement Α(·). Then the predicted probability of the test sample x after weak and strong enhancement is expressed as: q=p DE (α(x)) Q=p DE (A(x)) In the formula, q represents the predicted probability of the test sample x after weak enhancement, and Q represents the predicted probability of the test sample x after strong enhancement; The consistency regularization loss is expressed as the cross entropy loss of q and Q in, Indicates that the predicted probability distribution of q is converted into a one-hot encoding, c represents the category, and C represents the total number of categories; Step S4.5: Consistency regularization loss function l after testing samples are screened during testing t It is expressed as: In the formula, It is an indicator function that returns 1 if the condition in the parentheses is met, otherwise it returns 0.

Images