A Mixup-based method for long-tail data augmentation

Abstract

This invention discloses a long-tail data augmentation method based on Mixup. When training the model, the original long-tail dataset (x) is first input. i ,y i The process initializes a confusion probability matrix cf to 0. In each batch, a parameter θ following a Beta distribution is randomly obtained. New samples are generated through Mixup data augmentation and used as input to the model. Since each use of Mixup introduces the original imbalanced distribution, the virtual samples generated by Mixup follow a more imbalanced long-tailed distribution, exacerbating the imbalance problem of the dataset. To address this, this invention introduces θ and the confusion probability matrix cf into the loss function to mitigate the imbalance introduced by Mixup. The total loss of the model is obtained through a balanced loss function, and the model parameters are updated until the number of learning iterations exceeds a specified number. This invention is applicable to imbalanced image recognition scenarios such as autonomous driving scene recognition and fine-grained image recognition.

Description

Technical Field

[0001] This invention relates to image recognition technology in deep learning, to long-tail datasets augmented with Mixup data, and particularly to a long-tail data augmentation method based on Mixup. Background Technology

[0002] Training deep neural network models relies heavily on massive amounts of data. The datasets we typically use are usually manually selected and balanced. However, in the real world, collected data is often imbalanced, meaning the head class constitutes the vast majority of samples, while the tail class comprises only a small fraction. For example, in images collected for autonomous driving scenarios, there are far more images of people standing than jumping. Similarly, in fine-grained image recognition, the number of images for the plant superclass is far greater than that of protozoa. This causes the model to incorrectly favor the head class during training, resulting in significantly lower accuracy for the tail class compared to the head class—this is the long-tail problem. Forcibly collecting tail class samples would greatly increase time and financial costs. Therefore, how to solve the long-tail problem and improve the prediction accuracy of the tail class has become a key issue in deep learning.

[0003] From a data augmentation perspective, Mixup is a simple and efficient data augmentation method that combines two randomly selected samples using a parameter θ to generate new samples. In recent years, many methods have applied Mixup to long-tail problems to increase the diversity and robustness of tail class samples. However, since both random samples selected by Mixup come from long-tailed datasets, the newly generated samples are dominated by the head class, exacerbating the imbalance problem during training. Most methods address this issue from the input perspective, correcting the generated samples to include more tail classes. For example: 1) When selecting two samples, one is chosen through random sampling, and the other is chosen based on the number of samples in each class; the more samples in a class, the lower the probability of selecting that class; 2) Adjusting the parameter θ based on the class of the sample to give tail classes a larger proportion in the newly generated samples; the smaller the number of samples in each class, the larger the parameter θ. The problem with these methods is that too many tail class samples during training can lead to overfitting in the tail class. Recently, the MiSLAS method has improved the performance of Mixup on long-tailed datasets by decoupling the representation and classifier. However, this method is not end-to-end and has the problem of being difficult to apply.

[0004] This invention corrects the model's bias from the output perspective, avoiding overfitting of tail classes. It can be applied to scenarios such as autonomous driving scene recognition and fine-grained image recognition. Furthermore, this invention is an end-to-end model, which can be easily and conveniently applied to different long-tail datasets simply by modifying the loss function. Summary of the Invention

[0005] Purpose of the invention: In order to solve the problem that the data augmentation method Mixup exacerbates the imbalance in long-tail datasets, this invention provides a long-tail data augmentation processing method based on Mixup. Furthermore, this invention can be combined with the Delayed Reweighting (DRW) method to simultaneously mitigate the imbalance introduced by Mixup and the imbalance inherent in the long-tail dataset itself.

[0006] Technical Solution: A long-tail data augmentation method based on Mixup, wherein the method includes the following steps in the process of training a model on long-tail data:

[0007] (1) Input the original long-tail dataset S;

[0008] (2) Initialize the confusion probability matrix cf to 0;

[0009] (3) Obtain the parameters by θ ~ Beta(α,α), and Mixup uses the parameters θ to generate new sample data. Use new samples as input to the model;

[0010] (4) Using θ and the confusion probability matrix cf, apply a balanced loss function l BaMix The total loss of the model is calculated as follows:

[0011]

[0012]

[0013] (5) Update the model parameters and use the validation set to verify the model accuracy;

[0014] (6) Calculate the confusion matrix and update the confusion probability matrix cf;

[0015] (7) Update the number of learning sessions;

[0016] (8) If the number of learning iterations is greater than the specified number of learning iterations, end the model training; otherwise, repeat step (3) to continue training the model and update the model parameters.

[0017] Furthermore, for the long-tailed distributed dataset input in step (1), new samples are generated in step (2) through a Mixup transformation, specifically including the following calculations:

[0018]

[0019]

[0020] In the formula, (x i ,y i ), (xj ,y j )∈S, y i Is input x i The true label, θ~Beta(α,α), α∈(0,+∞), (x i ,y i ), (x j ,y j ) represents two samples randomly selected by Mixup, which are combined using the parameter θ to generate a new sample.

[0021] Furthermore, new samples It serves as the input to the model in step (3).

[0022] For the parameter θ in step (3) which is between 0 and 1, the model f used is generally a deep neural network model, including network models such as ResNet-32 and ResNet-50.

[0023] To further analyze the imbalance introduced by Mixup, it is proven that using the mean to hide sample j,

[0024]

[0025]

[0026] in, n is the total number of samples. Each time Mixup is used, the sample mean is introduced. In imbalanced datasets, It is dominated by the head class, which leads to the virtual samples generated by Mixup. The bias towards the head class exacerbates the imbalance in the original dataset. Therefore, to address the imbalance introduced by Mixup, this invention employs a novel balanced loss function. BaMix .

[0027] Furthermore, the specific formula for the balance loss function in step (4) is as follows:

[0028]

[0029]

[0030]

[0031] The above formula is derived by first proving the difference between the output of the original sample and the output of the mixed sample, as shown below:

[0032]

[0033] and These are the conditional probabilities of the original sample prediction and the conditional probabilities of the mixed-up sample prediction, respectively:

[0034]

[0035]

[0036] and The link function is:

[0037]

[0038] To further calculate the difference between the output of the original sample and the output of the mixed sample, i.e. This invention introduces a confusion probability matrix.

[0039] Assumption For the original sample x i Predicted as label y i The probability, Virtual samples after using Mixup Predicted as y i The conditional probability, cf(class1,class2) is the probability that class1 is identified as class2, n t Let be the total number of samples of class t, t∈{1,…,k}, and k be the number of classes, then:

[0040]

[0041] It can be observed that, by The influence, that is, category y i The larger the sample size, The larger the value, the more clearly the model's predictions are influenced by the number of samples in each class, showing a bias towards the head class. To reduce the model's bias towards the head class, let... To achieve balanced predictions, the model output is subtracted from B(y,θ), and the denominator term n·cf(y,y) is incremented by 1 to avoid a denominator of 0. Experiments revealed that the value of B(y,θ) may approach negative infinity. To ensure B(y,θ) is greater than 0, this invention amplifies B(y,θ) to... The final output of the model is obtained.

[0042] Furthermore, in step (6), the confusion probability matrix cf is updated by the confusion matrix. Through the validation of the validation set, we obtain the predicted label and the true label of the current validation set. By statistically analyzing the predicted label and the true label, we obtain the confusion matrix. The confusion probability matrix cf only needs to be obtained by dividing each row of the confusion matrix by the total number of samples in that row (i.e., the number of samples in the validation set of that category). The confusion probability matrix will be updated after each round of training.

[0043] Beneficial Effects: This invention addresses the problem of long-tail data processing in image recognition technology, primarily solving the issue that Mixup data augmentation exacerbates dataset imbalance in long-tail datasets. The invention introduces θ and a confusion probability matrix cf into the loss function to mitigate the imbalance introduced by Mixup. The confusion probability matrix cf is initially a matrix of all zeros, and is calculated from the confusion matrix with each learning round. The total loss of the model is obtained through a balanced loss function, and the model parameters are updated until the number of learning rounds exceeds a specified number. Furthermore, this invention corrects model bias from the output perspective, avoiding overfitting of tail classes. It is an end-to-end model that can be easily and conveniently applied to different long-tail datasets, making it suitable for image recognition in various imbalanced scenarios. Attached Figure Description

[0044] Figure 1 This is a flowchart illustrating the implementation of the method described in this invention;

[0045] Figure 2 These are virtual samples generated after using Mixup;

[0046] Figure 3 This describes the training process using a balanced loss function. Detailed Implementation

[0047] To illustrate the technical solutions disclosed in this invention in detail, further descriptions are provided below in conjunction with the accompanying drawings.

[0048] First, this invention provides a long-tail data augmentation processing method based on Mixup. Please refer to... Figure 1 The process shown in this invention focuses on constructing a new balanced loss function for the Mixup long-tail dataset. The loss function proposed by this method can effectively adapt to the long-tail dataset using Mixup and reduce the imbalance introduced by Mixup.

[0049] Figure 2 It demonstrates how Mixup generates virtual samples, combining... Figure 2 Assuming the sample (x) selected by Mixup i ,y i ), (x j ,yj The samples belong to the head class and the tail class, respectively. However, in practical applications, since Mixup is randomly sampled, the probability of selecting a tail class sample is much smaller than that of the head class. This results in the virtual samples generated by Mixup following a more unbalanced long-tail distribution.

[0050] Existing methods, such as BBN, Remix, and UniMix, all address the issue from the input perspective, adjusting the virtual samples generated by Mixup to increase the proportion of tail class samples. However, increasing tail class samples can lead to overfitting of the tail class. To address this problem, this invention takes an output-oriented approach, using relevant theories to demonstrate how Mixup causes the model output to be biased towards the head class, and then corrects this bias by balancing the loss function. The entire process is as follows: Figure 3 As shown, compared with other methods, the method described in this invention only requires modification of the model's loss function, is simple to operate, easy to apply, and applicable to various imbalanced image recognition methods.

[0051] The specific steps of implementing this invention are further described below. A long-tail data augmentation processing method based on Mixup includes the following steps:

[0052] Step 1: Input the original long-tailed distributed dataset S;

[0053] Step 2: Initialize the confusion probability matrix cf to 0;

[0054] Step 3: Obtain a parameter θ that follows a Beta distribution, and use Mixup to generate new samples as input to the model;

[0055] Step 4: Calculate the total loss of the model using a balanced loss function;

[0056] Step 5: Update model parameters and verify model accuracy;

[0057] Step 6: Update the confusion probability matrix cf;

[0058] Step 7: Update the number of learning iterations. If the number of learning iterations is greater than the specified number of learning iterations, training stops; otherwise, repeat step 3.

[0059] To further explain the above steps, step 1 involves inputting the original long-tailed distribution dataset. Common long-tailed datasets include CIFAR10-LT, CIFAR100-LT, ImageNet-LT, and iNaturalist. From step 1, you can obtain the specific number of samples for each category.

[0060] In step 2, the confusion probability matrix is ​​initially 0. After each training round, it will be updated by the confusion matrix. The confusion matrix is ​​obtained by statistically analyzing the predicted labels and the true labels. The confusion probability matrix cf can be obtained by dividing each row of the confusion matrix by the total number of samples in that row (i.e., the number of samples in the validation set for that category).

[0061] In step 3, θ ~ Beta(α,α), α∈(0,+∞), and α is generally set to around 1.0. In the CIFAR10-LT, CIFAR100-LT, and ImageNet-LT datasets, this invention sets α to 1.0; in the iNaturalist dataset, this invention sets α to 0.4. In each batch, a randomly generated θ is obtained. For the samples (x) in a batch... i ,y i ), randomly shuffled to obtain (x) j ,y j Mixup uses the following formula to generate new samples.

[0062]

[0063]

[0064] Sample j can be hidden using the following theorem.

[0065] Theorem 1. Assume θ ~ Beta [1 / 2,1] (α,α), α∈(0,+∞), j~Unif(1,n), (x i ,y i After using Mixup, Given:

[0066]

[0067]

[0068] Let θ be the mean. (δ i ,ε i () is a random disturbance with a mean of 0:

[0069]

[0070]

[0071] Similarly, θ~Beta [0,1 / 2] For (α,α), the above formula still holds true.

[0072] To simplify the analysis, the perturbation (δ) is ignored.i ,ε i ), and use specific θ~Beta [0,1] (α,α), simplified sample generated by Mixup for:

[0073]

[0074]

[0075] It is important to note that The virtual samples used are simplified mixed-up samples for ease of analysis. In actual experiments, this invention still uses... As a virtual sample after Mixup.

[0076] The balancing loss function in step 4 is derived as follows.

[0077] Suppose we have a dataset S = {(x1, y1), ..., (x n ,y n )}, where n is the total number of samples, n t Let x be the number of samples of class t, where t∈{1,…,k}, and x be the number of samples of class t. i ,y i )∈S, y i Is input x i The labels are and the model is f; The dataset after Mixup z y Let f(x) be the y-th output of the model. y , for

[0078] make For conditional probability,

[0079]

[0080] The link function is

[0081]

[0082] The proof of Theorem 2 is as follows:

[0083] Theorem 2. Is input x i Predicted as category y i The probability, It is input Predicted as category y i The probability of is then,

[0084]

[0085] prove. Add to both sides simultaneously get:

[0086]

[0087] Right now because

[0088]

[0089]

[0090] Will Substitute We obtain Theorem 2, that is

[0091] Furthermore, regarding the input after Mixup... Approximately obtained

[0092]

[0093] but,

[0094]

[0095] First calculate In part,

[0096]

[0097] cf(class1, class2) represents the probability that class 1 is predicted as class 2, initially set to 0, and subsequently calculated from the confusion matrix. Clearly, Subject to sample size Impact, causing minus Reduce prediction bias towards the head class, due to cf(y i ,y i The denominator may be 0. To ensure the denominator is not 0, this invention uses n·cf(y) i ,y i Add 1 to this denominator, because in the experiment... It may be close to 0, we will Enlarged to The balance term is then expressed as The bias towards the head class in the model is corrected by a balancing term. B(y) iB(y,θ) can be represented in a more general form as B(y,θ) (where y can be either y,θ) i It can also be y j The modified loss function is further called the balanced loss function l. BaMix The format is:

[0098]

[0099]

[0100]

[0101] In summary, the balanced loss function proposed in this invention, by introducing a balance term B(y,θ), can effectively adapt to Mixup data augmentation, reduce the imbalance introduced by Mixup, improve classification accuracy, and achieve optimal accuracy on multiple long-tail datasets.

[0102] This invention is applicable to image recognition in various imbalanced scenarios, such as scene recognition and fine-grained image recognition in autonomous driving. Images collected by autonomous driving systems are often imbalanced; people standing and walking are common scenarios, while jumping is rare. This lack of rare scenarios leads to decreased prediction accuracy for tail classes, causing the autonomous driving system to make incorrect decisions in special situations. In fine-grained image recognition, some superclasses have a much larger number of images than others; for example, the number of images for the plant superclass is much larger than for protozoa, causing the model to predict biased towards plants, even if the predicted image does not belong to a plant. This invention corrects the model's output; only the loss function needs to be modified during application, making it simple and easy to apply. The following sections demonstrate the results of this invention's application on four long-tailed datasets: CIFAR10-LT, CIFAR100-LT, ImageNet-LT, and iNaturalist, and compare the prediction accuracy of this invention with other methods.

[0103] Table 1. Precision (%) of datasets CIFAR10-LT and CIFAR100-LT on ResNet-32.

[0104]

[0105]

[0106] Table 1 shows a comparison of the accuracy of this invention with some common long-tail methods such as BBN, LDAM-DRW, and remix-DRW on the long-tail datasets CIFAR10-LT and CIFAR100-LT. CIFAR10-LT and CIFAR100-LT are long-tail datasets from CIFAR10 and CIFAR100, respectively. The number of samples in each class decreases exponentially with the increase of the class index. The imbalance degree ρ = max i {n i} / min i {n i The larger the value of ρ, the higher the degree of imbalance. In the experiments, all validation sets were evenly distributed. In Table 1, ρ was set to 200, 100, 50, and 10. Among these methods, LDAM-DRW is a classic rebalanced loss function method without using Mixup. BBN, remix-DRW, UniMix, CMO-DRW, and MiSLAS are methods that apply Mixup to long-tail problems. All methods were trained on a ResNet-32 network. Among all these methods, the classification accuracy achieved by this invention is the highest, which is 0.15% to 2.10% higher than previous methods.

[0107] Table 2. Accuracy (%) of the long-tailed dataset ImageNet-LT on networks ResNet-10 and ResNet-50.

[0108]

[0109] Table 2 compares the accuracy of various methods on the long-tailed dataset ImageNet-LT using ResNet-10 / ResNet-50 networks. ImageNet-LT is a long-tailed dataset from the large ImageNet dataset, with the largest category containing 1280 samples and the smallest containing 5 samples. Among the compared methods, LDAM-DRW is a traditional rebalanced loss function method, while the other methods are combined with Mixup. BBN and UniMix balance the Mixup from the input perspective, while this invention better adapts to the Mixup method from the output perspective. On ResNet-10, this invention achieves 1.43% higher accuracy than the previous highest-accuracy method, and on ResNet-50, it achieves 2.22% higher accuracy than the previous highest-accuracy method.

[0110] Table 3. Accuracy (%) of the long-tail dataset iNaturalist 2018 on ResNet-50.

[0111] Dataset iNaturalist2018 method ResNet-50 LDAM-DRW 68.00 BBN 69.62 Remix-DRW 70.49 UniMix 69.15 CRT+Mixup 70.20 LWS+Mixup 70.90 MiSLAS 71.60 This invention 71.97

[0112] Table 3 compares the accuracy results of each method on the long-tailed dataset iNaturalist 2018. iNaturalist is a large and extremely imbalanced dataset containing 8142 classes, 437513 training images, 24426 validation images, and 149394 test images. Among these methods, the present invention also achieved the highest accuracy of 71.97%.

[0113] As can be seen from the validation of various long-tail datasets, this invention has achieved significant advantages compared with both traditional rebalance loss functions and long-tail methods using Mixup. Furthermore, this invention is an end-to-end method, which only requires modification of the loss function and can be easily and effectively applied to various long-tail datasets, making it suitable for image recognition under various imbalanced conditions.

Claims

1. A method for augmenting long-tail data based on Mixup, characterized in that: The method for training long-tail data includes the following steps: (1) Input the original long-tail dataset The long-tail dataset mentioned For image datasets; (2) Initialize the confusion probability matrix cf to 0; (3) Through Mixup uses the parameters obtained. Generate new sample data The new sample is used as the input to the model; Data augmentation of long-tail datasets using Mixup, new sample data. The generation includes the following calculations: in, , , It is input The true label, , , , These are two samples randomly selected by Mixup, using parameters... Combine the two samples to generate a new sample. ; (4) Utilization And the confusion probability matrix cf, using a balanced loss function The total loss of the model is calculated as follows: ； (5) Update the model parameters and use the validation set to verify the model accuracy; (6) Calculate the confusion matrix and update the confusion probability matrix cf; (7) Update the number of learning sessions; (8) If the number of learning iterations is greater than the specified number of learning iterations, end the model training; otherwise, repeat step (3) to continue training the model and update the model parameters.

2. The long-tail data augmentation method based on Mixup according to claim 1, characterized in that: Step (4) The calculation process for the total loss of the model is as follows: Assumption Original sample Predicted as a label The probability, For virtual samples after using Mixup Predicted as The conditional probability, For category Identified as The probability, For category The total number of samples, , To determine the number of categories, we have: Among them are: ； In the formula, For input The One output, For input The Each output, after using Mixup, is affected by the model's output. The influence is biased towards the head class. To avoid bias in model predictions towards the head class, a balancing term is introduced. ,in, It might be a number close to 0, leading to To approach negative infinity, in order to Greater than 0, will Enlarged to And in order to make If the denominator is not zero, add 1 to the denominator term to balance the terms. The final result is: The total loss of the model is calculated as follows: 。 3. The long-tail data augmentation method based on Mixup according to claim 1, characterized in that: Step (6) updates the confusion probability matrix cf using the confusion matrix, as follows: By validating the validation set, we obtain the predicted and true labels of the current validation set. By statistically analyzing the predicted and true labels, we obtain the confusion matrix. The confusion probability matrix (cf) is obtained by dividing each row of the confusion matrix by the total number of rows. The confusion probability matrix is ​​updated after each round of training.

Images