A long-tail image data classification method based on mixed samples

Abstract

The present invention belongs to the field of image classification and designs a long-tail image data classification method based on mixed samples. The present invention proposes a new solution to the long-tail training set problem encountered in image classification research. It aims to use three experts with specialized knowledge to jointly assist the algorithm in making the final decision, avoiding problems such as excessive deviation of the model classifier weight caused by a single model. The present invention is suitable for business scenarios of image classification with long-tail data distribution. By designing multiple experts with specific field knowledge, the classification performance of the model for all frequency distribution types is improved without losing the accuracy of the head class classification. It provides a solution for the actual engineering application of image classification when the data has a long-tail distribution, alleviates problems such as data collection difficulties, improves the overfitting of the algorithm model to the head class data, and improves the learning ability of the tail class data.

Description

Technical Field

[0001] The present invention belongs to the field of image classification, and in particular relates to a long-tail image data classification method based on mixed samples. Background Art

[0002] Long-tail distribution refers to the imbalanced distribution of sample sizes across training sets in image classification tasks. Due to the difficulty in collecting data for certain categories, the resulting dataset contains a large number of samples for a few categories (the head categories), while many categories (the tail categories) are underrepresented. Training machine learning models with datasets with long-tail distributions results in an imbalance in the model's ability to learn features across different categories, making it unable to meet application requirements. In image classification, long-tail distribution of training data can easily lead to the model's learning process being dominated by the head categories, which have abundant sample data. This limits the model's ability to learn and model the tail categories, resulting in less feature information to be learned and a highly chaotic feature space, which poses challenges to final classification accuracy. The problem posed by long-tail data imbalance in classification is essentially an imbalance in the final classifier weights. Current solutions can be broadly categorized as addressing the data, algorithm, or a combination of the two. On a micro level, research on the long-tail data distribution problem can be summarized as focusing on category rebalancing, information enhancement, and improving model modules.

[0003] While the specially designed single-model network structure of conventional long-tail classification algorithms can alleviate classification imbalance to a certain extent, it still cannot effectively balance classification accuracy across all categories. Furthermore, for certain extremely small tail categories, repeated sampling can lead to overfitting, or the sampling frequency is too low, making it impossible to learn good feature representations. While most previous methods have achieved good results, this can be attributed to sacrificing the classification performance of the head categories, which contain a large amount of data, in order to improve the classification performance of the tail categories, where data is scarce. This results in the model's increasing confidence in the head categories and a loss of the ability to distinguish between different categories. This is unacceptable in real-world applications such as face recognition, species classification, medical image diagnosis, and drone detection. Summary of the Invention

[0004] In view of the shortcomings of the existing technology, the present invention designs a long-tail image data classification method based on mixed samples.

[0005] A long-tail image data classification method based on mixed samples specifically includes the following steps:

[0006] Step 1: Use conventional sampling and class balanced sampling methods to sample the image data set to obtain and Then use the Mixup mixed sample method to obtain image data mixed samples X={x i,y i} then all samples in the training set are generated by mixed sampling, where x i ,y i are the vector data and labels of a single sample, i∈{1,2,...,N}; according to the Mixup method, the mixed sample is:

[0007]

[0008] Among them, λ~Beta(α, α) is the random value of the mixed strategy; the mixed sample obtained Used for training experts E1 and E2, while E3 uses conventionally sampled samples Perform training; according to step 1, and Then, the shared model parameters are used to calculate and obtain and The model uses ResNet as the backbone network, uses different network layers to represent different experts, and the loss function of each expert is different;

[0009] Step 2: Based on the results obtained from the sampling in step 1, the outputs of experts E1, E2 and E3 are obtained respectively;

[0010] Step 2.1: According to x mix 、 Get the output z1 of E1;

[0011] Will As the input of the E1 deep network, after calculation by the network module and the fully connected layer, we get That is, the output of E1 is expressed as:

[0012]

[0013] Where C represents the number of categories in the training set;

[0014] Step 2.2: According to x mix 、 Get the output z2 of E2;

[0015] Will As the input of the E2 deep network, after the network module and the fully connected layer calculation, we get That is, the output of E2 is expressed as:

[0016]

[0017] Step 2.3: According to x uni 、 Get the output z3 of E3;

[0018] Will As the input of E3, after the network module and the fully connected layer calculation, we get Therefore, the output of E3 is expressed as:

[0019]

[0020] Then it is the output vector representation of experts E1, E2 and E3;

[0021] Step 3: Based on the outputs of experts E1, E2, and E3 obtained in step 2, calculate the losses of experts E1, E2, and E3 respectively, and update experts E1, E2, and E3;

[0022] Step 3.1: Calculate the loss of E1 based on the output of E1 obtained in step 2

[0023] For E1, its positioning is to retain its original learning method and focus more on the extraction of head class information to avoid the destruction of the good learning ability of the head class after modifying the loss function. Therefore, the most basic cross entropy loss function combined with the hybrid strategy is used as the loss. The loss function of E1 is:

[0024]

[0025] where λ comes from the mixed sample strategy, is the cross entropy loss function;

[0026] Step 3.2: Calculate the loss of E2 based on the output of E2 obtained in step 2

[0027] For E2, while maintaining a good learning ability of the head class features, it also takes into account other samples to balance the classification weights; LDAM is used as the core loss function of E2; since the input samples of E2 are mixed samples, the class balanced sampling samples are not suitable for the LDAM loss function, so The cross entropy loss function is still used when calculating the loss, and LDAM is only used for part, then the loss function of E2 is:

[0028]

[0029] Step 3.3: Calculate the loss of E3 based on the output of E3 obtained in step 2

[0030] By balancing the classification center of gravity through E3, the output z3 is adjusted by introducing the number of categories, and then Softmax is used for regression, so that E3 pays more attention to the rare categories; therefore, the loss function of E3 is:

[0031]

[0032] Step 4: Update the model parameters based on the losses of experts E1, E2, and E3;

[0033] The total loss of the model is:

[0034]

[0035] After the model loss is calculated, back propagation is performed to update the model parameters;

[0036] Step 5: Based on the model whose parameters have been updated, perform model output inference; in the inference stage, the sampling samples of experts E1, E2 and E3 are all (x i ,y i ), the arithmetic mean of the output vectors of each expert is the final output of the inference phase, specifically:

[0037]

[0038] Where m is the number of expert models, that is, m = 3, so the final TOP1 prediction category of the inference stage model is argmax(z final (x i )), that is, z final The index of the maximum value in .

[0039] Beneficial technical effects of the present invention:

[0040] Based on the characteristics of long-tail distributed data, the present invention proposes a long-tail image data classification method based on mixed samples, which provides a new solution to the long-tail training set problem encountered in image classification research. Suitable for business scenarios of image classification with long-tail distributed data, the method can improve the classification performance of the model for all frequency distribution types without sacrificing the accuracy of the head class classification by designing multiple expert models with specific domain knowledge. This provides a solution for the practical engineering application of image classification when data has a long-tail distribution, alleviates problems such as data collection difficulties, improves the overfitting of the algorithm model for the head class data, and enhances the learning ability of the tail class data. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 The embodiment of the present invention is based on a framework diagram of a long-tail image classification model of diversified experts. DETAILED DESCRIPTION

[0042] The present invention will be further described below with reference to the accompanying drawings and embodiments;

[0043] like Figure 1As shown, this paper proposes a method for classifying long-tail image data based on mixed samples. By designing multiple experts with specific domain knowledge, the model's classification performance for all frequency distribution types is improved without sacrificing accuracy in head-class classification. The experts share parameters in the early network layers to reduce computational effort, and use different loss functions to update their parameters. This encourages each expert to focus on training samples from categories with different frequency distributions. The resulting integrated model achieves superior classification performance in long-tail data distributions.

[0044] The goal is to use three experts with specialized knowledge to jointly assist the algorithm in making the final decision, avoiding problems such as excessive bias in the model classifier weights caused by a single model. The model's three experts, E1, E2, and E3, are abstracted using different network layers and loss functions. Each focuses on different dataset partitioning situations and uses different training data. Some experts use samples based on hybrid strategies for training, and different loss functions are designed to update the expert model's expertise. The final classification result is aggregated.

[0045] The model uses ResNet as its backbone network, with different layers representing different experts. Each expert has a different loss function. During training, the loss is calculated based on the output of each expert. During inference, the model's final output is the average of the three experts' outputs. The output is then processed using the Softmax function to produce the classification result for each category.

[0046] A long-tail image data classification method based on mixed samples specifically includes the following steps:

[0047] Step 1: Use conventional sampling and class balanced sampling methods to sample the image data set to obtain and Then use the Mixup mixed sample method to obtain image data mixed samples X={x i ,y i} then all samples in the training set are generated by mixed sampling, where x i ,y i are the vector data and labels of a single sample, i∈{1, 2, ..., N}; according to the Mixup method, the mixed sample is:

[0048]

[0049] Among them, λ~Beta(α, α) is the random value of the mixed strategy; the mixed sample obtained Used for training experts E1 and E2, while E3 uses conventionally sampled samples Perform training; according to step 1, and Then, the shared model parameters are used to calculate and obtain and The model uses ResNet as the backbone network, uses different network layers to represent different experts, and the loss function of each expert is different;

[0050] Step 2: Based on the results obtained from the sampling in step 1, the outputs of experts E1, E2 and E3 are obtained respectively;

[0051] Step 2.1: According to x mix 、 Get the output z1 of E1;

[0052] Will As the input of the E1 deep network, after calculation by the network module and the fully connected layer, we get That is, the output of E1 is expressed as:

[0053]

[0054] Where C represents the number of categories in the training set;

[0055] Step 2.2: According to x mix 、 Get the output z2 of E2;

[0056] Will As the input of the E2 deep network, after the network module and the fully connected layer calculation, we get That is, the output of E2 is expressed as:

[0057]

[0058] Step 2.3: According to x uni 、 Get the output z3 of E3;

[0059] Will As the input of E3, after the network module and the fully connected layer calculation, we get Therefore, the output of E3 is expressed as:

[0060]

[0061] Then it is the output vector representation of experts E1, E2 and E3;

[0062] Step 3: Based on the outputs of experts E1, E2, and E3 obtained in step 2, calculate the losses of experts E1, E2, and E3 respectively; and update experts E1, E2, and E3.

[0063] Step 3.1: Calculate the loss of E1 based on the output of E1 obtained in step 2

[0064] For E1, we retain its original learning method and focus more on extracting head class information to avoid destroying the good learning ability of the head class after modifying the loss function. Therefore, we use the most basic cross entropy loss function combined with a hybrid strategy as the loss. The loss function of E1 is:

[0065]

[0066] where λ comes from the mixed sample strategy, is the cross entropy loss function;

[0067] Step 3.2: Calculate the loss of E2 based on the output of E2 obtained in step 2

[0068] For E2, while maintaining a good learning ability of the head class features, it also takes into account other samples to balance the classification weights; LDAM is used as the core loss function of E2; since the input samples of E2 are mixed samples, the class balanced sampling samples are not suitable for the LDAM loss function, so The cross entropy loss function is still used when calculating the loss, and LDAM is only used for part, then the loss function of E2 is:

[0069]

[0070] Step 3.3: Calculate the loss of E3 based on the output of E3 obtained in step 2

[0071] By balancing the classification center of gravity through E3, the output z3 is adjusted by introducing the number of categories, and then Softmax is used for regression, so that E3 pays more attention to the rare categories; therefore, the loss function of E3 is:

[0072]

[0073] Step 4: Update the model parameters based on the losses of experts E1, E2, and E3;

[0074] The total loss of the model is:

[0075]

[0076] After the model loss is calculated, back propagation is performed to update the model parameters;

[0077] Step 5: Based on the model whose parameters have been updated, perform model output inference; in the inference stage, the sampling samples of experts E1, E2 and E3 are all (x i ,y i ), the arithmetic mean of the output vectors of each expert is the final output of the inference phase, specifically:

[0078]

[0079] Where m is the number of expert models, that is, m = 3, so the final TOP1 prediction category of the inference stage model is argmax(z final (x i )), that is, z final The index of the maximum value in .

[0080] The following is the algorithm for implementing the functions of the present invention:

[0081]

[0082]

[0083] The experiments show the classification effects of different algorithms on Many-Shot, Medium-Shot and Few-Shot in the CIFAR100-LT dataset with ρ=100, as shown in the table.

[0084] Top-1 accuracy (%) of each part of CIFAR100-LT (ρ=100)

[0085]

[0086] This method improves medium-shot and few-shot classification by 6.5% and 12.3%, respectively. This is due to the combined effect of mixed samples and the expert model E3, which frees the model classifier from being dominated by the head class, which contains a large number of samples. Multiple experts with specific domain knowledge improve the model's classification performance across all frequency distribution types without sacrificing accuracy for the head class.

Claims

1. A long-tail image data classification method based on mixed samples, characterized in that: The specific steps include: Step 1: Use conventional sampling and class balanced sampling methods to sample the image data set to obtain and Then use the Mixup mixed sample method to obtain image data mixed samples X={x i ,y i } then all samples in the training set are generated by mixed sampling, where x i ,y i are the vector data and label of a single sample, i∈{1,2,...,N}; Step 2: Based on the results obtained from the sampling in step 1, the outputs of experts E1, E2 and E3 are obtained respectively; Step 3: Based on the outputs of experts E1, E2, and E3 obtained in step 2, calculate the losses of experts E1, E2, and E3 respectively, and update experts E1, E2, and E3; Step 4: Update the model parameters based on the losses of experts E1, E2, and E3; Step 5: Based on the model whose parameters have been updated, perform model output inference; in the inference stage, the sampling samples of experts E1, E2 and E3 are all (x i ,y i ), the arithmetic mean of the output vectors of each expert is the final output of the reasoning stage; Among them, step 1 is based on the Mixup method, and the mixed sample is: Among them, λ~Beta(α,α) is the random value of the mixed strategy; the mixed sample obtained Used for training experts E1 and E2, while E3 uses conventionally sampled samples Perform training; according to step 1, and Then, the shared model parameters are used to calculate and obtain and The model uses ResNet as the backbone network, uses different network layers to represent different experts, and the loss function of each expert is different; Among them, step 3 is specifically as follows: Step 3.1: Calculate the loss of E1 based on the output of E1 obtained in step 2 Step 3.2: Calculate the loss of E2 based on the output of E2 obtained in step 2 Step 3.3: Calculate the loss of E3 based on the output of E3 obtained in step 2 Among them, step 3.1 calculates the loss of E1 Specifically: For E1, its positioning is to retain its original learning method and focus more on the extraction of head class information to avoid the destruction of the good learning ability of the head class after modifying the loss function. Therefore, the most basic cross entropy loss function combined with the hybrid strategy is used as the loss. The loss function of E1 is: where λ comes from the mixed sample strategy, is the cross entropy loss function; Among them, step 3.2 calculates the loss of E2 Specifically: For E2, while maintaining a good learning ability of the head class features, it also takes into account other samples to balance the classification weights; LDAM is used as the core loss function of E2; since the input samples of E2 are mixed samples, the class balanced sampling samples are not suitable for the LDAM loss function, so The cross entropy loss function is still used when calculating the loss, and LDAM is only used for part, then the loss function of E2 is: Among them, step 3.3 calculates the loss of E3 Specifically: By balancing the classification center of gravity through E3, the output z3 is adjusted by introducing the number of categories, and then Softmax is used for regression, so that E3 pays more attention to the rare categories; therefore, the loss function of E3 is:

2. The long-tail image data classification method based on mixed samples according to claim 1 is characterized in that: Step 2 is as follows: Step 2.1: According to x mix 、 Get the output z1 of E1; Will As the input of the E1 deep network, after calculation by the network module and the fully connected layer, we get That is, the output of E1 is expressed as: Where C represents the number of categories in the training set; Step 2.2: According to x mix 、 Get the output z2 of E2; Will As the input of the E2 deep network, after the network module and the fully connected layer calculation, we get That is, the output of E2 is expressed as: Step 2.3: According to x uni 、 Get the output z3 of E3; Will As the input of E3, after the network module and the fully connected layer calculation, we get Therefore, the output of E3 is expressed as: It is the output vector representation of experts E1, E2 and E3.

3. The long-tail image data classification method based on mixed samples according to claim 1, characterized in that: Step 4 obtains the total model loss as: After the model loss is calculated, backpropagation is performed to update the model parameters.

4. The long-tail image data classification method based on mixed samples according to claim 1, characterized in that: In step 5, the arithmetic mean of the output vectors of each expert is taken as the final output of the inference phase, which is: Where m is the number of expert models, that is, m = 3, so the final TOP1 prediction category of the inference stage model is argmax(z final (x i )), that is, z final The index of the maximum value in .

Images