A long-tail learning data enhancement method based on Mosaic fusion

Abstract

The present invention discloses a long-tail learning data enhancement method based on Mosaic fusion. First, a center segmentation point is set in a long-tail data image and each image is divided into four regions. Then, the order of image samples within the batch is shuffled to obtain four sets of image samples in different orders. All image sample categories are divided into majority and minority classes according to the number of categories. The image samples of the two categories are subjected to Mosaic fusion data enhancement using different strategies. Finally, the image label information is adjusted and the loss function is calculated based on the fusion status of each image. The present invention takes into account the characteristics of long-tail data and introduces Mosaic fusion enhancement into the training data. By leveraging the rich information of the majority class samples, the diversity of the minority class is increased, thereby improving the classification performance of the deep learning network model and the classification effect of the model on long-tail data sets.

Description

Technical Field

[0001] The present invention relates to the technical field of long-tail data recognition, and in particular to a long-tail learning data enhancement method based on Mosaic fusion. Background Art

[0002] Machine learning is the study of how computers use data and algorithms to learn and automatically improve their performance. With the rapid development of information technology in recent years, vast amounts of data have been generated and stored in fields such as the internet and industry. Machine learning techniques have also been widely developed and applied across various fields. However, in practical applications of machine learning, we often encounter the phenomenon of "long tail" in datasets, where the number of samples in some categories is small, while the number of samples in other categories is large. This unbalanced data distribution can cause trained deep neural network models to poorly classify samples from minority categories, affecting model performance. The problem of long tail data arises widely in various fields, such as image recognition, natural language processing, medical diagnosis, and network security. Therefore, research on this problem has become a hot research topic in machine learning.

[0003] Traditional machine learning methods are prone to performance degradation when training long-tail image datasets. Due to the small number of samples in the minority category, the classifier cannot fully learn the characteristics and patterns of these categories, resulting in poor discrimination ability. At the same time, the majority class samples have a large amount of data and are given higher weight when training the classifier, while the minority class data is given lower weight. If the classifier does not consider the importance of minority class data, it will lead to performance degradation. This unbalanced distribution ultimately leads to the classifier being more inclined to classify samples into the more frequent category and ignore the minority category. This often results in poor classification of the minority category, which limits the accuracy and generalization ability of the classifier.

[0004] To address the long-tail problem, over the past few decades, methods such as oversampling, undersampling, and cost-sensitive methods have been used in long-tail datasets. Undersampling balances the dataset by reducing the number of samples from frequently occurring categories, but this approach can lead to data loss and overfitting. Oversampling increases the number of samples from less frequently occurring categories, but this approach can lead to data duplication and overfitting. Other researchers have proposed weighting-based methods, the most common of which is weighting by sample size. However, class-weighted methods require manual weighting, are inflexible, and fail to address the problem of insufficient minority class features. Data augmentation, on the other hand, increases the number and diversity of minority class samples, helping the model better learn minority class features and thus improving classification performance. Among current data augmentation methods, two commonly used methods, Mixup and Cutmix, while effective in addressing the problem of insufficient minority class data, do not address the characteristics of long-tail data. The Remix method improves on the Mixup method by assigning labels that favor the less-popular category when mixing two samples. However, this method is prone to noise generation and still lacks sufficient enhancement of minority class samples. Summary of the Invention

[0005] The purpose of this invention is to improve the problem of insufficient diversity of minority class samples in traditional methods and the problem of insufficient model training due to the imbalance of sample number in long-tail datasets. A long-tail learning data enhancement method based on Mosaic fusion is proposed. This method uses the information of majority class samples to improve the classification ability of the model on minority class samples, thereby improving the performance of deep neural network models on long-tail datasets.

[0006] To achieve the above objectives, the present invention provides a technical solution: a long-tail learning data enhancement method based on Mosaic fusion, comprising the following steps:

[0007] Step 1: Obtain long-tail data images, set the range of the image center segmentation point, and divide each image into four regions according to the horizontal and vertical directions of the segmentation point; shuffle the label order of the image samples in the training batch to obtain four sets of image samples in different orders for subsequent Mosaic fusion;

[0008] Step 2: Classify the image samples into majority and minority classes based on the number of image samples in different categories. Perform data augmentation on each sample in the batch. If the sample belongs to the majority class, perform mosaic fusion on the sample with the samples from other image sample sets obtained in step 1 and fill it in the segmented area of ​​the image. If the sample belongs to the minority class, keep the original image unchanged and do not perform fusion.

[0009] Step 3: To reflect the contribution of each image in the fusion process, adjust the image label information based on the mosaic fusion results of each image. Perform data enhancement by utilizing the rich information of the majority class samples to improve the recognition effect of the minority class samples. Finally, input the images and labels into the deep neural network model to calculate the loss function to complete the long-tail learning data enhancement.

[0010] Furthermore, in step 1, a range of image center segmentation points is first set, and each image is divided into four parts a, b, c, and d along the horizontal and vertical directions of the center segmentation points. The center segmentation points of the images are randomly selected within the set range; then, an image sample set X1 within a training batch is obtained, and the order of the image samples in the batch is randomly shuffled three times to obtain three sets of image sample sets X2, X3, and X4 with different orders for subsequent Mosaic fusion.

[0011] Furthermore, in step 2, a hyperparameter is selected to determine the number of image sample categories. The image sample categories are divided into majority and minority categories according to the number of samples contained in the image sample categories. The category with a sample number greater than the hyperparameter is the majority category, and the category with a sample number less than the hyperparameter is the minority category. The image samples in the batch are traversed in a loop, and different strategies are used for majority and minority samples. If the image sample belongs to the majority category, the sample will have a probability of p to perform Mosaic fusion with a random image sample from the three groups of image sample sets obtained in step 1, and will also perform Mosaic fusion with a random image sample from one of the groups of image sample sets with a probability of 1-p. Mosaic fusion is used to utilize the rich sample information of the majority category to improve the recognition effect of the minority category samples, while adding regularization to the model. If it is a minority category, the sample information of the minority category is kept intact. The image fusion formula is as follows:

[0012]

[0013] Where M a ,M b ,M c ,M d Respectively represent the image masks of the corresponding positions of the four regions a, b, c, and d divided in step 1; M rand The image mask represents the corresponding position of a randomly selected area among the four areas a, b, c, and d; represents the synthesized image sample; x1, x2, x3, x4 represent random image samples from the image sample sets X1, X2, X3, X4 obtained in step 1 respectively; ⊙ represents element-by-element multiplication; n represents the number of categories of the image sample x1, τ represents the hyperparameter for determining the number of categories, r is a random number in the range of (0, 1), and p is a hyperparameter in the range of (0, 1).

[0014] Furthermore, in step 3, after completing the mosaic fusion of the image samples, the label values ​​of the image samples are adjusted. According to the fusion status of each image, the label information of the image is adjusted and the loss function is calculated. The label fusion formula is as follows:

[0015]

[0016] Where S a ,S b ,S c ,S d They represent the area ratio of the a, b, c, and d regions divided in step 1 to the area of ​​the entire image, respectively. rand It represents the area ratio of a randomly selected area in the four areas a, b, c, and d to the area of ​​the entire image. Represents the label information of the synthesized image sample, y1, y2, y3, y4 represent the label information of the image samples x1, x2, x3, x4 respectively.

[0017] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0018] 1. The present invention can effectively increase the diversity of minority class samples and further improve classification accuracy. Traditional classification methods often have difficulty effectively classifying minority class image data when faced with long-tail distributed image data. The method of the present invention uses Mosaic fusion technology to enhance long-tail data. It performs multiple Mosaic fusions of majority class samples with other samples to generate new samples, enriching the feature diversity of minority class samples while improving the linearization of the sample space.

[0019] 2. The method is simple and easy to implement, without incurring additional costs. The method only requires fusion data augmentation on the training dataset, enriching the information of minority class samples with information from majority class samples, without requiring additional data sampling. This makes the method more feasible and cost-effective for real-world datasets.

[0020] 3. The method of the present invention is flexible and adaptable to different long-tail datasets. The method of the present invention uses Mosaic fusion technology for data augmentation, which is flexible and adaptable to different types of long-tail datasets. By adjusting the method and ratio of fusion data augmentation, it can be optimized for the characteristics of different long-tail datasets, making the method of the present invention more applicable and generalizable. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 4 is a flowchart of the method of the present invention.

[0022] Figure 2This is a flow chart of the implementation of the method of the present invention based on Mosaic fusion (i.e., Mosaicmix).

[0023] Figure 3 Schematic diagram of Mosaic fusion (i.e., Mosaicmix) of CIFAR-100-LT long-tail data images. DETAILED DESCRIPTION

[0024] The present invention will be described in further detail below with reference to the embodiments and drawings, but the embodiments of the present invention are not limited thereto.

[0025] First, you can obtain a long-tail image dataset online, which includes image samples from multiple different categories. Commonly used long-tail datasets with imbalanced categories include the iNaturalist dataset, ImageNet-LT, CIFAR-10-LT, and CIFAR-100-LT datasets. Alternatively, you can use a custom dataset. Such datasets typically contain a large number of samples in the top few categories, while most categories at the end have very few samples, resulting in a long-tail distribution. This example primarily uses the CIFAR-10-LT and CIFAR-100-LT datasets.

[0026] The CIFAR-10-LT and CIFAR-100-LT datasets are sampled from the CIFAR-10 and CIFAR-100 datasets. Both the CIFAR-10 and CIFAR-100 datasets contain 60,000 images of size 32×32. The CIFAR-10 dataset contains 10 classes, while the CIFAR-100 dataset contains 100 classes. In order to simulate the long-tail distribution in the imbalanced dataset, the CIFAR-10 and CIFAR-100 datasets are resampled to artificially reduce the training sample size of each class except the first class to obtain the CIFAR-10-LT and CIFAR-100-LT datasets. In this process, the test dataset is kept unchanged and still has a balanced sample distribution. For the CIFAR-10-LT and CIFAR100-LT datasets, experiments are conducted on these two datasets using two different imbalance ratios ρ∈{100, 50}. The parameter ρ represents the maximum number of sample categories n in the dataset i and the minimum number of sample categories n j The imbalance ratio between them, that is, ρ = n i / n j .

[0027] Feature extraction of long-tail image datasets requires the use of a deep neural network model. ResNet32 is used as the backbone feature extraction network. The model is trained using the SGD optimizer. In terms of parameter settings, the momentum is set to 0.9, the weight decay is set to 0.0002, the basic learning rate is set to 0.1, and the epoch is uniformly set to 400.

[0028] After determining the dataset and deep neural network model, such as Figure 1 As shown, the long-tail learning data enhancement method based on Mosaic fusion provided in this embodiment specifically includes the following steps:

[0029] Step 1: Obtain long-tail data images and set a selection range for the image center segmentation point. Each image is divided into four parts, a, b, c, and d, along the horizontal and vertical directions of the center segmentation point. The image center segmentation point is randomly selected within the set range. Then, obtain a training batch of image sample set X1 and randomly shuffle the order of the image samples in this batch to obtain three sets of image sample sets with different orders, X2, X3, and X4. A total of four sets of image sample sets with different orders are obtained for subsequent mosaic fusion.

[0030] Step 2: Select a hyperparameter to determine the number of image sample categories. According to the number of samples of each category of the long-tail data image, the image sample categories are divided into majority and minority categories. The category with more image sample categories than the hyperparameter is the majority category, and the category with less than the hyperparameter is the minority category. Each image sample in the batch is looped through, and different strategies are used for fusion data enhancement for majority and minority samples respectively. The specific implementation steps of fusion enhancement are as follows: Figure 2 shown.

[0031] If the image sample belongs to the majority class sample, the sample will have a probability of p to perform Mosaic fusion with the random image sample from the three groups of image sample sets obtained in step 1, that is, the corresponding parts of the original image sample and the other three random image samples, a total of four image samples, are filled into the four parts a, b, c, and d of the segmentation to form the following: Figure 3 In addition, a random image sample from one of the image sample sets obtained in step 1 is mosaic-fused with the probability of 1-p. To enhance the robustness of the model, the randomly selected image sample is inserted into a random area in the four parts a, b, c, and d of the original image sample, while the other areas remain the original image samples and are fused into a new image sample.

[0032] If it is a minority class, the original image is kept unchanged and no mosaic fusion is performed.

[0033] Mosaic fusion uses the rich information of majority class samples to improve the recognition effect of minority class samples, while also improving the linearity of the sample space. However, mosaic fusion of four images can easily lead to overfitting, so mosaic fusion of four images is combined with mosaic fusion of two images for mixed data enhancement. The image fusion formula is as follows:

[0034]

[0035] Where M a ,M b ,M c ,M d Respectively represent the image masks of the corresponding positions of the four regions a, b, c, and d divided in step 1; M rand Represents the image mask of the corresponding position of a randomly selected area among the four areas a, b, c, and d; represents the synthesized image sample; x1, x2, x3, x4 represent random image samples from the image sample sets X1, X2, X3, X4 obtained in step 1 respectively; ⊙ represents element-by-element multiplication; n represents the number of categories of the image sample x1, τ represents the hyperparameter for determining the number of categories, r is a random number in the range of (0, 1), and p is a hyperparameter in the range of (0, 1). In the experiment, setting this parameter to 1 / 3 can achieve better results.

[0036] Step 3: After completing the Mosaic fusion of the image samples, adjust the label value of the image samples and adjust the label information of the image according to the fusion status of each image. The label fusion formula is as follows:

[0037]

[0038] Where S a ,S b ,S c ,S d They represent the area ratio of the a, b, c, and d regions divided in step 1 to the area of ​​the entire image; S rand It represents the area ratio of a randomly selected area in the four areas a, b, c, and d to the area of ​​the entire image. Represents the label information of the synthesized image sample; y1, y2, y3, y4 represent the label information of the image samples x1, x2, x3, x4 respectively.

[0039] In this paper, we introduce Mosaic Fusion technology to enhance the dataset. By fusing information from multiple samples to generate new training samples, we effectively increase the number of minority class samples and diversity, alleviating the problem of insufficient minority class samples in long-tail datasets. Furthermore, Mosaic Fusion of majority class samples also helps improve the robustness and generalization ability of majority class samples.

[0040] The present invention uses the ResNet-32 backbone network to conduct experiments on two long-tail distribution datasets, CIFAR-10-LT and CIFAR-100-LT. The experimental results are shown in Table 1.

[0041] Table 1 Accuracy of ResNet-32 on long-tail datasets CIFAR-10-LT and CIFAR-100-LT (%)

[0042]

[0043] Table 1 compares the accuracy of various long-tail learning data augmentation methods on the long-tail datasets CIFAR-10-LT and CIFAR-100-LT. Mixup and Cutmix are common regularized data augmentation methods, while LDAM_DRW balances the long-tail problem from a loss function and two-stage perspective. Remix and CMO build on Mixup and Cutmix to enhance long-tail data. When compared on the CIFAR-100-LT dataset with an imbalance ratio of 100, our method improves accuracy by 1.22% compared to the previously most accurate CMO method. It also performs quite well on other datasets and is worthy of promotion.

[0044] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be considered as equivalent replacement methods and are included in the scope of protection of the present invention.

Claims

1. A long-tail learning data enhancement method based on Mosaic fusion, characterized in that: The following steps are involved: Step 1: Obtain long-tail data images, set the range of the image center segmentation point, and divide each image into four regions according to the horizontal and vertical directions of the segmentation point; shuffle the label order of the image samples in the training batch to obtain four sets of image samples in different orders for subsequent Mosaic fusion; Step 2: Classify the image samples into majority and minority classes based on the number of image samples in different categories. Perform data augmentation on each sample in the batch. If the sample belongs to the majority class, perform mosaic fusion on the sample with the samples from other image sample sets obtained in step 1 and fill it in the segmented area of ​​the image. If the sample belongs to the minority class, keep the original image unchanged and do not perform fusion. Step 3: To reflect the contribution of each image in the fusion process, adjust the image label information based on the mosaic fusion results of each image. Perform data enhancement by utilizing the rich information of the majority class samples to improve the recognition effect of the minority class samples. Finally, input the images and labels into the deep neural network model to calculate the loss function to complete the long-tail learning data enhancement.

2. A long-tail learning data enhancement method based on Mosaic fusion according to claim 1, characterized in that: In step 1, we first set a range for the center segmentation point of the image, and divide each image into four parts a, b, c, and d along the horizontal and vertical directions of the center segmentation point. The center segmentation point of the image is randomly selected within the set range; then, we obtain a set of image samples X1 within a training batch, and randomly shuffle the order of the image samples in the batch three times to obtain three sets of image sample sets X2, X3, and X4 with different orders for subsequent Mosaic fusion.

3. A long-tail learning data enhancement method based on Mosaic fusion according to claim 2, characterized in that: In step 2, a hyperparameter is selected to determine the number of image sample categories. The image sample categories are divided into majority and minority categories according to the number of samples contained in the image sample categories. The category with a sample number greater than the hyperparameter is the majority category, and the category with a sample number less than the hyperparameter is the minority category. The image samples in the batch are traversed in a loop, and different strategies are used for majority and minority samples. If the image sample belongs to the majority class sample, the sample will have a probability of p to perform Mosaic fusion with a random image sample from the three groups of image sample sets obtained in step 1. In addition, the sample will have a probability of 1-p to perform Mosaic fusion with a random image sample from one of the groups of image sample sets. Mosaic fusion is used to utilize the rich sample information of the majority class to improve the recognition effect of the minority class samples, while adding regularization to the model. If it is a minority class, the sample information of the minority class is kept intact. The image fusion formula is as follows: Where M a ,M b ,M c ,M d Respectively represent the image masks of the corresponding positions of the four regions a, b, c, and d divided in step 1; M rand The image mask represents the corresponding position of a randomly selected area among the four areas a, b, c, and d; represents the synthesized image sample; x1, x2, x3, x4 represent random image samples from the image sample sets X1, X2, X3, X4 obtained in step 1 respectively; ⊙ represents element-by-element multiplication; n represents the number of categories of the image sample x1, τ represents the hyperparameter for determining the number of categories, r is a random number in the range of (0, 1), and p is a hyperparameter in the range of (0, 1).

4. A long-tail learning data enhancement method based on Mosaic fusion according to claim 3, characterized in that: In step 3, after completing the mosaic fusion of the image samples, adjust the label values ​​of the image samples. According to the fusion status of each image, adjust the label information of the image and calculate the loss function. The label fusion formula is as follows: Where S a ,S b ,S c ,S d They represent the area ratio of the a, b, c, and d regions divided in step 1 to the area of ​​the entire image, respectively. rand It represents the area ratio of a randomly selected area in the four areas a, b, c, and d to the area of ​​the entire image. Represents the label information of the synthesized image sample, y1, y2, y3, y4 represent the label information of the image samples x1, x2, x3, x4 respectively.

Images