Long-tail image recognition method based on representation data augmentation and loss rebalancing

Abstract

The application discloses a long-tail image recognition method based on characteristic data enhancement and loss rebalancing, and comprises the following steps: long-tail image preprocessing; then, different data sets are constructed into training sets and test sets according to different unbalanced proportions; a random data enhancement list is constructed; the sampling weight of each category is obtained, the number of samples from the head category to the tail category is added to the scaling factor of the weight, and then the weight is converted into a Tensor vector; the weight is used as a parameter to construct an anti-sampling training set and the weight of a weighted loss; a long-tail data classification model is established; a one-stage training paradigm is constructed, and a bilateral branch structure is used to adaptively change from an unweighted branch to a weighted branch; in the initial training, the unweighted branch learns the characteristics of the long-tail image, and gradually changes to using the reweighted classification branch to learn how to classify the long-tail data as the training round increases, so that the original data learns the representation of the long-tail image and the long-tail image is recognized.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of computer vision, and particularly relates to a long-tail image recognition method based on representation data enhancement and loss rebalancing. BACKGROUND

[0002] With the application and development of artificial intelligence in recent years, the traditional image classification method using artificial is gradually replaced by artificial intelligence technology. However, all data in nature is long-tail data distribution, so there are still problems to be solved if the artificial intelligence technology is applied to real scene image classification. For example, in wild animal image classification, it takes a very long time to collect a photo of a rare animal. For some endangered animals, it may take several years to get a photo. However, the identification of rare new animals is more valuable than the protection of animals with large quantities. If the image classification framework can only work well on animal categories with large quantities, the method will never have practical applicability.

[0003] Therefore, long-tail image recognition is a problem to be solved in actual image classification, which mainly manifests in that (1) the uneven distribution of the number of samples in each category makes the system pay more attention to the categories with large quantities of samples and ignore the categories with small quantities of samples, resulting in very low recognition accuracy of the categories with small quantities of samples; and (2) if the system pays too much attention to the categories with small quantities of samples, the accuracy of the categories with small quantities of samples is increased to a certain extent, but the accuracy of the categories with large quantities of samples is greatly damaged. The above two problems show the “treadmill” characteristics of long-tail image classification accuracy.

[0004] In long-tail image data distribution, the categories with large quantities of samples are defined as head categories, and the categories with small quantities of samples are defined as tail categories. Due to the lack of diversity of tail data, the training of the entire classification system is dominated by head categories, and finally, the system has a certain bias for the classification of head categories.

[0005] The related research work of long-tail distribution recognition is divided into three categories in this paper: data resampling strategy, loss reweighting and data enhancement strategy.

[0006] (1) The data resampling strategy is a preprocessing operation for solving the long-tail distribution directly from the data level. At present, there are a large number of resampling techniques, which can be further divided into: tail class upsampling and head class downsampling methods.

[0007] Tail class oversampling means increasing the sampling probability of tail class data during model training to alleviate the imbalance between classes. However, this method simply repeats the sampling of a small amount of tail data and does not introduce additional sample information, so it is very limited in improving the recognition accuracy of tail classes, and is prone to overfitting of the model to tail classes.

[0008] Head class undersampling means reducing the sampling probability of head class data during model training to alleviate the imbalance between classes. This method also has drawbacks, that is, while undersampling head class data, the data of the head class is lost, thus losing important information contained in these samples. To solve this problem, some methods propose to develop rules or strategies to selectively remove redundant head class samples.

[0009] We explore the classification framework for image data representation learning and the learning of classifiers in the classification framework for long-tail data classification in real-world scenarios. We experiment with various sampling methods for long-tail data in the classification framework. Finally, we find that the main reason for poor long-tail recognition accuracy in the long-tail distribution-based image classification framework is the learning of classifiers, and the resampling strategy affects the learning of image features by the classification framework for long-tail data.

[0010] (2) Loss reweighting strategy is a preprocessing operation for long-tail distribution from the perspective of data attention of the classification framework. There are already a large number of loss reweighting techniques. Loss reweighting is a method for solving long-tail recognition at the algorithm level. By designing different loss sizes for different data, a loss cost matrix is constructed. Loss reweighting integrates different error classification values of each class into the classification decision, with the purpose of guiding the classifier to adjust the weight of the tail class for error classification. Specifically, since the tail class data is less, tail data is often easily ignored and easily misclassified, therefore, the tail class should be given a greater loss weight to make the classification framework pay more attention to the tail class. Compared with data resampling strategy, loss reweighting can fully utilize all data, and this method does not introduce any artificially constructed samples, so it does not introduce noise in training.

[0011] (3) Data augmentation strategy: The purpose of data augmentation is to use a set of data augmentation techniques to enhance the size and quality of the system training data set. In long-tail learning, two data augmentation methods have been explored, including transfer-based augmentation and conventional (non-transfer) augmentation.

[0012] Although researchers have made some achievements in the field of long-tail data identification, due to the extreme imbalance of long-tail data and the lack of tail class samples, the existing technology is not mature. There are mainly the following defects. It is extremely difficult to improve the identification efficiency of the tail class without affecting the identification performance of the model for the head class and extracting the extremely difficult discriminative features of the tail class. (1) In long-tail image classification, using loss reweighting and sample resampling in the initial training stage of the model will change the original spatial distribution of the original data, cause distribution distortion, and lead to model overfitting. (2) The multi-stage training method is difficult and complex, which is difficult for model deployment in the later stage and is not conducive to the reuse of the model. (3) Most long-tail image recognition tasks are based on the model of the multi-stage training paradigm, and the performance of the representation learning based on the cross-entropy loss still has a huge room for improvement.

[0013] Therefore, how to develop a simple long-tail image classification framework that can effectively improve the identification accuracy of the tail class and maintain the identification accuracy of the head class is a problem to be solved. SUMMARY

[0014] The purpose of the embodiment of the application is to provide a long-tail image identification method based on representation data enhancement and loss rebalancing, which can effectively solve the problem that the system pays too much attention to the head class, resulting in a decrease in the identification accuracy of the tail class, and also can maintain the identification accuracy of the head class while improving the identification accuracy of the tail class.

[0015] To solve the above technical problems, the technical solution adopted by the application is a long-tail image identification method based on representation data enhancement and loss rebalancing, which is performed according to the following steps:

[0016] Step S1, long-tail image preprocessing:

[0017] Step S2, different data sets are constructed into training sets and test sets according to different imbalance ratios, the imbalance ratio of the training set in the long-tail data distribution is [500, 256, 100, 50, 10], and different proportions of the training set are constructed using python, and the test set is kept balanced;

[0018] Step S3, construct a random data enhancement list, use the transform toolkit in python to construct the random enhancement list needed, and the order of the data enhancement list is random cropping of the picture, random flipping of the picture, changing the attributes of the image, and finally normalizing;

[0019] Step S4, obtain the sampling weight of each class, add the weight scaling factor to the number of samples flipped from the head class to the tail class, and then convert the weight into a Tensor vector; wherein the weight expression of each class is as follows:

[0020]

[0021]

[0022] wherein, N represents a scaling factor of the weight, W represents the weight of the category; formula (1) is used to calculate the weight of the number of samples of each category in the total number of samples and assign a value to a, which is used in formula (2) to calculate the weight of each category;

[0023] Step S5, using the weight as a parameter, constructing the weight of the anti-sampling training set and the weighted loss by using the pytorch toolkit;

[0024] Step S6, establishing a long-tail data classification model;

[0025] Step S7, constructing a one-stage training paradigm, and using a double-branch structure to adaptively change from an unweighted branch to a weighted branch; initially, the unweighted branch learns the features of the long-tail image, and gradually changes to use the reweighted classification branch to learn how to classify the long-tail data as the training round increases, thereby completing the original data learning the representation of the long-tail image and identifying the long-tail image.

[0026] Further, the step S1 specifically comprises: calling the Pytorch toolkit, adjusting the size of the long-tail image to 224x224, performing normalization processing on the adjusted image, and taking the image as an image data set.

[0027] Further, the step S6 specifically comprises:

[0028] S61, using ResNet as a backbone network, and referring to the content in contrast learning, adding a prediction head and a contrast head on the ResNet;

[0029] S62, performing random twice data augmentation on the constructed training set and the anti-sampling training set to generate four training sets;

[0030] S63, first performing Mix-up data mixing on the pictures and labels of the training set and the pictures and labels of the anti-sampling training set, and then performing Cut-mix data mixing on the pictures and labels of the training set and the pictures and labels of the anti-sampling training set, and finally generating two groups of image mixed samples using different mixing types;

[0031] S64, constructing a self-supervised learning branch, constructing two groups of pictures by using global picture mixing data augmentation Mix-up and local picture mixing data augmentation Cut-mix, and then obtaining the whole group of picture feature mappings h g and h lThen the contrastive head h g and h l represents the converted output u g and u l , the negative cosine similarity of which is minimized, expressed as follows:

[0032]

[0033] where ||·|| is the L2 normalization, h g represents the features extracted by the feature extractor for the entire set of images enhanced by Mix-up, h l represents the features extracted by the feature extractor for the entire set of images enhanced by Cut-mix, u g and u l respectively represent the image features after linear spatial mapping of the features h g and h l ;

[0034] Following the architecture of Sim-Siam, the stop gradient operation is used to prevent collapse, and the SimSiam loss function is defined as:

[0035]

[0036] where sg(h l ), sg(h g ) are the outputs after the features extracted by the feature extractor for the images enhanced by Cut-mix and Mix-up respectively, and the extracted features are input to the mapping head for feature mapping;

[0037] Finally, two sets of images using the training set and the anti-sampling set are constructed;

[0038] The loss of the unweighted classification branch and the weighted classification branch are both based on cross-entropy loss, the difference is that the class weights obtained by formulas (1) and (2) are weighted to the cross-entropy loss in the weighted branch; while the unweighted branch only uses cross-entropy to calculate the cross-entropy loss of each class; therefore, when calculating the cross-entropy loss of the unweighted classification branch c , the following formula is used:

[0039]

[0040] where n represents the batch size of each training of the sample, f(·) represents the predicted probability and u are the label probabilities of the i-th image after Mix-up and Cut-mix respectively;

[0041] S65, construct class rebalancing branch, introduce a weighting factor inversely proportional to label frequency, then force the model to learn the tail classes and suppress the head classes, the weight factor is set to:

[0042]

[0043] where r i is the sampling frequency of the i-th class of the training dataset, k is a hyperparameter measuring the gap between head and tail classes, and c is the total number of classes;

[0044] Given a training dataset The rebalancing weighted classification branch cross-entropy loss cb can be written as:

[0045]

[0046] where and represent the predicted probability of the mixed image and the weight factor to which the image should be weighted, respectively.

[0047] Further, in the S61, the prediction head and the contrast head maintain the structure consistent with Sim-Siam in contrast learning.

[0048] Further, the S63 specific process is:

[0049] (X i , P i ) represents the picture and label of the training set, (X j , P j ) represents the picture and label of the inverse sampling training set, and the mixed image and label probability after Mix-up are denoted as The mixed image and label probability after Cut-mix are denoted as

[0050] The mixing expression of Mix-up is as follows:

[0051] λ ~ Beta(β, β) (3)

[0052]

[0053]

[0054] where β is a self-set parameter with a value range of (0, 1), Beta(β, β) represents a sampling value obtained by Beta sampling using the β value, and the sampling value is assigned to λ, x i , p i is the picture and label of the training set, x j , pj For the picture and label of the anti-sampling training set, For the image and label probability mixed using Mix-up;

[0055] The mixing expression of Cut-mix is as follows:

[0056]

[0057] Cut-Mix combines another image by replacing the local area of the image with the local area of another training image, where M∈{0,1} Width×Height is a binary mask, is the position of the patch to be cropped from the two images, is the element-wise multiplication, Width and Height represent the width and height of the image x i , respectively, is the picture after Cut-mix data enhancement;

[0058] Where the image data mixing of Cut-mix is achieved by sampling the bounding box coordinates of the image, B=(r x ,r y ,r w ,r h ) represents the area to be cropped in the picture X i and X j , and the coordinates of the sampling area are realized by the following formula:

[0059]

[0060]

[0061] r x ,r y ,r w ,r h respectively represent the x coordinate, y coordinate, width of the cropped area, and height of the cropped area, r x ,r y are obtained by uniform sampling, Width and Height represent the width and height of the image x i , respectively, and λ is calculated by formula (3). The mixed label and weight are the same as Mix-Up.

[0062] Further, the step S7 specifically comprises: constructing a one-stage training paradigm, using a double-branch structure to adaptively learn a rebalancing branch, multiplying the cross-entropy loss￡ c of the unweighted classification branch by α, and multiplying the cross-entropy loss￡ cb of the rebalanced weighted classification branch by (1-α), and α is automatically reduced as the current training round T increases:

[0063]

[0064] where T max represents the total training epoch, and the meaning of a is the model attention weight; specifically, a is the weight of the cross-entropy loss of the unweighted classification branch c , and (1-a) is the weight of the cross-entropy loss of the re-balanced weighted classification branch cb , and the model will gradually transfer the attention from the unweighted classification branch to the re-balanced weighted classification branch through the change of the value of a as the training epoch increases.

[0065] The total loss total is defined as the combination of the loss c , the loss cb and the loss sim :

[0066] ￡ total = a￡ c + (1-a)￡ cb + g￡ sim (16)

[0067] where g is a hyper-parameter that controls the sim loss.

[0068] The beneficial effects of the present application are: to realize the construction of a long-tail image classifier with strong generalization ability and balance, and at the same time to fully improve the feature extractor's ability to extract discriminative features; to solve the problem of the prior art that improving the tail class leads to the decrease of the accuracy of the head class, the problem of the complexity of the multi-stage training model steps and the sensitivity to parameters leading to significant decline in overall performance. And it also has the following characteristics:

[0069] (1) The present application proposes a progressive global and local data mixed consistent classification framework for long-tail images, which can effectively improve the overall classification accuracy of long-tail images.

[0070] (2) The present application uses an effective progressive re-balancing learning strategy to integrate learning image features and classifier classification boundary learning into one step, reducing the complexity of multiple training of the classification framework, and is more suitable for solving the long-tail data distribution problem in real environment.

[0071] (3) The long-tail data classification framework of the present application does not require additional data operations and can train the classification framework with small batches of data compared to other classification frameworks, so the reusability of the classification framework is higher.

[0072] (4) The training algorithm does not require negative sample pairs and larger training batches, and has high flexibility and small usage restrictions in actual application. BRIEF DESCRIPTION OF DRAWINGS

[0073] In order to make the technical solutions in the embodiments of the present application or the prior art clearer, the accompanying drawings needed in the embodiments or prior art description will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present application, and other accompanying drawings can be obtained by those of ordinary skill in the art without any creative effort.

[0074] Figure 1 is a distribution diagram of long-tail data;

[0075] Figure 2 is a network structure diagram of contrast learning;

[0076] Figure 3 is a long-tail recognition method based on representation data enhancement and loss rebalancing;

[0077] Figure 4 is a diagram of the region of interest for the head class and tail class model. DETAILED DESCRIPTION

[0078] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without any creative effort are within the scope of protection of the present application.

[0079] The long-tail image recognition method based on representation data enhancement and loss rebalancing comprises the following steps:

[0080] Step S1, long-tail image preprocessing: calling the Pytorch toolkit, adjusting the size of the long-tail image to 224x224, and normalizing the adjusted image as an image dataset.

[0081] Step S2, constructing training set and test set according to different imbalance ratios of different datasets, the imbalance rate of the training set in the long-tail data distribution is [500, 256, 100, 50, 10], and different proportions of training sets are constructed by using python, and the test set is kept balanced.

[0082] Step S3, constructing a random data enhancement list, using the transform toolkit in python to construct the random enhancement list needed. The order of the data enhancement list is random cropping of pictures, random flipping of pictures, changing the properties of the image, and finally normalizing.

[0083] Step S4, obtain the sampling weight of each category, by flipping the sample number of the head class to the tail class plus the scaling factor of the weight, and then converting the weight into a Tensor vector. Wherein, the weight expression of each category is calculated as follows:

[0084]

[0085]

[0086] Wherein, N represents the scaling factor of the weight, and W represents the weight of the category. Formula (1) calculates the weight of the number of samples of each category in the total number of samples and assigns a to formula (2) to calculate the weight of each category.

[0087] Step S5, use the weight as a parameter to build the anti-sampling training set and the weight of the weighted loss using the pytorch toolkit.

[0088] Step S6, establish a long-tail data classification model;

[0089] S61, use ResNet as the backbone network, and on this basis, refer to the content in contrast learning, add a prediction head and a contrast head to ResNet. Wherein the prediction head and the contrast head maintain the same structure as Sim-Siam in contrast learning;

[0090] S62, randomly perform twice data augmentation on the constructed training set and anti-sampling training set to generate four training sets;

[0091] S63, first mix the pictures and labels of the training set and the pictures and labels of the anti-sampling training set using Mix-up data, and then mix the pictures and labels of the training set and the pictures and labels of the anti-sampling training set using Cut-mix data, finally generate two groups of image mixed samples using different mixed types.

[0092] Use (X i ,P i ) to represent the picture and label of the training set, (X j ,P j ) to represent the picture and label of the anti-sampling training set. The image and label probability after Mix-up mixing are denoted as The image and label probability after Cut-mix mixing are denoted as

[0093] The mixing expression of Mix-up is as follows:

[0094] λ~Beta(β,β) (3)

[0095]

[0096]

[0097] where β is a self-set parameter with a value range of (0, 1), Beta(β, β) represents obtaining a sampling value by Beta sampling using the value of β, and assigning the sampling value to λ.x i , p i is the picture and label of the training set, x j , p j is the picture and label of the inverse sampling training set, is the image and label probability after using Mix-up mixing.

[0098] The mixing expression of Cut-mix is as follows:

[0099]

[0100] Cut-Mix combines another image by replacing the local area of the image with the local area of another training image. Where M∈{0,1} Width×Height is expressed as a binary mask, which can be understood as the patch position to be cropped from the two images, and is element-wise multiplication. Width and Height represent the width and height of the image x i , respectively. is the picture after Cut-mix data enhancement.

[0101] Specifically, we sample the coordinates of the bounding box of the image, B=(r x ,r y ,r w ,r h ) represents the area to be cropped in the picture X i and X j . The coordinates of the sampling area are realized by the following formula.

[0102]

[0103]

[0104] r x ,r y ,r w ,r h respectively represent the x coordinate and y coordinate of the area to be cropped, the width of the cropped area, and the height of the cropped area. r x ,r y are obtained by uniform sampling. Width and Height represent the width and height of the image x i , respectively. Where λ is calculated by formula (3), and the mixed label and weight are the same as Mix-Up.

[0105] S64, constructing the self-supervised learning branch. In the above step S61, the contrastive head and the mapping head are constructed, and here our goal is to maximize the cosine similarity of the global and local mixture in the feature space to obtain the contrast consistency. Specifically, the two types of augmented images are converted by the mapping head of the feature extractor to obtain the h g and h j , and then the contrastive head converts the two representations into outputs u g and u l . The negative cosine similarity of them is minimized, and the expression is as follows

[0106]

[0107] where ||·|| is the L2 normalization. h g represents the features of the whole set of pictures after Mix-up augmentation extracted by the feature extractor, h l represents the features of the whole set of pictures after Cut-mix augmentation extracted by the feature extractor, u g and u l respectively represent the picture features after linear space mapping of the features h g and h l .

[0108] To minimize the negative cosine similarity of the augmented images, follow the architecture of Sim-Siam, and use the stop gradient operation to prevent collapse. The SimSiam loss function is defined as:

[0109]

[0110] where sg(h l ), sg(h g ) are the outputs after the features extracted by the feature extractor after Cut-mix and Mix-up data mixing, respectively.

[0111] The above steps finally construct two groups of pictures using the training set and the anti-sampling set, and calculate the cross-entropy loss of the unweighted classification branch c by the following formula:

[0112]

[0113] where n represents the batch size of each training of the sample, and f(·) represents the predicted probability and h are the i-th picture after Mix-up and Cut-mix, respectively, and h

[0114] While a set of images are mixed by Mix-up and Cut-mix respectively, thus the actual batch size will be twice the sampled batch size.

[0115] S65, constructing the class re-balancing branch: the design principle of class re-weighting is to introduce a weighting factor inversely proportional to the label frequency, and then force the model to learn the tail classes and suppress the head classes. Therefore, the weight factor is set to:

[0116]

[0117] where r i is the sampling frequency of the i-th class of the training dataset, k is a hyperparameter measuring the gap between head classes and tail classes, and c is the total number of classes.

[0118] Note that k = 0 corresponds to no re-weighting, and k = 1 corresponds to the class-balanced method. Change the scalar weight to one-hot vector form, and mix the weight vectors of the two images.

[0119] Given a training dataset The re-balancing weighted classification branch cross-entropy loss cb can be written as:

[0120]

[0121] where and represent the prediction probability of the mixed image and the weight factor that the image should be weighted.

[0122] Note that the global and local mixed images have the same mixing weight.

[0123] Step S7, construct a one-stage training paradigm, and use the double-branch structure to adaptively learn the re-balancing branch. Multiply the cross-entropy loss c of the unweighted classification branch by a, and multiply the cross-entropy loss cb of the re-balancing weighted classification branch by (1-a). a decreases automatically as the current training round T increases:

[0124]

[0125] where T max represents the total training round. The total loss total is defined as the combination of the loss c , the loss cb and the loss sim :

[0126] total = a c + (1-a)​cb + gamma sim (16)

[0127] where gamma is a hyperparameter controlling the sim loss, with a default value of 10.

[0128] Embodiments

[0129] The long-tail recognition method based on representation data augmentation and loss rebalancing disclosed by the present application is well verified on the long-tail dataset. As shown in FIG. 1, the long-tail dataset is divided into three parts, which are head classes, middle classes and tail classes. The head classes are specifically defined as the sample number greater than 100, the middle classes are defined as the sample number greater than 20 and less than 100, and the tail classes are the sample number less than 20. Figure 1 As can be seen from FIG. 1, on the dataset based on the long-tail distribution, some head classes have thousands of samples, while some tail classes have only 2-5 samples, showing extreme data imbalance. If a neural network without any processing is used for model training, the performance of the trained model will be very poor, because the limited number of parameters of the model cannot effectively express all classes, resulting in that the model has good performance for the head classes or middle classes with more samples, but the tail classes are completely not learned. However, in real application scenarios, the data of the tail classes often has more research value than the data of the head classes, such as rare animals in wild animal recognition, which have more research significance than common animals. If the model completely focuses on the tail classes, the performance of the head classes and middle classes will decrease sharply, so that the model loses good robustness. Therefore, the classification method of the present application needs to be used for accurate classification. Figure 1 The specific process is as follows:

[0130]

[0131] ​1. Image preprocessing: Each dataset in the long-tail dataset has different image sizes, so when these image data are obtained, preprocessing is first needed. In the embodiment of the present application, the preprocessing can be divided into three steps: 1) adjusting the image size: calling the Pytorch toolkit to perform the resizing operation on the long-tail dataset, and in this paper, the long-tail data image size is uniformly processed to 224x224 size; 2) data normalization processing: in many image processing processes, it is necessary to normalize the image as a preprocessing process. The so-called normalization processing is to make the value of the point distributed between 0-255 of the image pixel value distributed between [0, 1]. After processing, the speed of model parameter convergence can be accelerated. Here, we also use the normalization operation on the long-tail data to make the network parameters converge quickly in training and save training time; 3) dividing the dataset: dividing the long-tail dataset into a long-tail data training set and a long-tail data test set. The long-tail data training set is used to train the long-tail data classification model of the present application, and the long-tail data test set is used to test the performance of the image classification model in the long-tail data. In particular, the training set has five unbalanced probabilities of [500, 256, 100, 50, 10], and the test set is completely balanced data distribution. The imbalance rate here = the total number of samples of the most numerous category / the total number of samples of the least numerous category.

[0132] 2. Establishing a long-tail data classification modeling In order to capture the discriminative features of each class of long-tail data and improve the generalization ability of the model, the present application proposes a long-tail recognition method based on representation data enhancement and loss rebalancing. The model of this method can be divided into three parts: feature extractor learning, classifier learning and cumulative learning.

[0133] As shown in Figure 4 , the method proves that it can focus on the discriminative features of the head class and the tail class, visualize the model performance, and prove that the model performance is good.

[0134] a. First, as described in S5, S6 above, as shown in Figure 3 , the encoder part performs the inverse sampling operation on the input training dataset to obtain the inverse sampling dataset. The training dataset and the inverse sampling dataset are combined with each other by using the data mixing method of Mix-up and Cut-mix, and are input into the feature extractor to extract the discriminative features of each class. Finally, the features are input into the classifier.

[0135] b. The classifier learning part inputs the result x obtained in the feature extractor into the classification head and the rebalancing head of the classifier, as well as the prediction head and the contrast head, respectively. The classification head is a self-provided ResNet, and a copy of the classification head is named as the rebalancing head. The logits output from the classification head are used to calculate the cross-entropy loss with the real label, and the cross-entropy loss expression is as follows:

[0136]

[0137] The logits outputted from the re-balancing head and the real label compute the weighted cross-entropy loss, and the weight of the loss is obtained by the following expression:

[0138]

[0139]

[0140] Therefore, the formula of the weighted loss is:

[0141]

[0142] And the outputs of the mapping head and the contrast head are calculated for cosine similarity between each other, as shown in the architecture of Figure 2 The mapping head is used to fit the output of the mapping head more, and the specific calculation formula is:

[0143]

[0144] By calculating the similarity, it is converted into a loss back to the feature extractor for optimization, so the calculation formula of the contrast loss is:

[0145]

[0146] c. Cumulative learning

[0147] In order to learn better representation, the model uses non-weighted cross-entropy loss at the beginning, and the model uses re-weighting and re-sampling for classifier learning in the latter half stage, so a method is needed to connect the two learning methods, so the invention uses the method of cumulative learning, which calculates the overall loss by time weighting. The specific weighting expression is as follows:

[0148]

[0149] Therefore, the expression of the final model loss is as follows:

[0150] ￡ total = α￡ c + (1-α)￡ cb + γ￡ sim (7)

[0151] 3. Prediction: In order to evaluate the classification performance of the long-tail data classification model of the present application, experiments need to be conducted on the long-tail image data test set. At the same time, in order to verify the performance difference of the classification model of the present application and other classification models in the classification of long-tail images, the long-tail image classification method of the present application and other classification methods are compared in terms of classification accuracy based on the long-tail image test set. The classification accuracy is expressed by the classification accuracy formula as follows:

[0152]

[0153] Wherein, f is the long-tail image classification method based on the image feature encoder, D is the image test set of the long-tail distribution, N is the number of images in the long-tail image test set, x i is the i-th image in the long-tail image test set D, y i is the true class label of the i-th image in the long-tail image test set D, f(x i ) is the predicted class label of the long-tail image classification method after feature learning of the long-tail distribution image.

[0154] As can be seen from the experimental results, the classification accuracy of the present application on different long-tail distribution image test sets is higher than that of the current mainstream algorithm, and is obviously higher than that of other algorithm models, as shown in Table 1. It is fully proved that the classification method of the present application has high accuracy in different long-tail distribution image classification.

[0155] Table 1 Performance of the algorithm of the present application on different data sets

[0156] Dataset Accuracy Cifar-10-LT (IF: 100) 60.29％ Cifar-100-LT (IF: 100) 56.23％ Image-Net-LT (IF: 256) 56.28％

[0157] In addition, in order to effectively compare the performance of other algorithms and the present algorithm, the Cifar-100-LT (IF:100) data set is selected as the performance comparison, and the methods based on contrast learning and cross-entropy loss and the two-stage training method are selected for performance comparison. As shown in Table 2.

[0158] Table 2 Comparison of the present application and other algorithms on the data set Cifar-100-LT (IF:100)

[0159]

[0160]

[0161] Each embodiment in the specification is described in a related manner, and the same or similar parts between each embodiment can be referred to each other. Each embodiment focuses on the difference from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the related parts can be referred to the part of the method embodiment.

[0162] The above descriptions are only the preferable embodiments of the present application, not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A long-tail image recognition method based on characterization data enhancement and loss rebalancing, characterized in that, The following steps are performed: Step S1, long tail image preprocessing: Step S2, different data sets are constructed according to different unbalanced proportions to construct training sets and test sets, and the unbalanced rate of the training set in the long tail data distribution is [500, 256, 100, 50, 10], and different proportions of training sets are constructed using python, and the test set is balanced; Step S3, construct a random data augmentation list, use the transform toolkit in python to construct the random augmentation list needed, and the order of the data augmentation list is random cropping of the picture, random flipping of the picture, changing the properties of the image, and finally normalizing; Step S4, obtain the sampling weight of each class, add the sample number of the head class to the tail class by flipping and the scaling factor of the weight, and then convert the weight to a Tensor vector; wherein, the weight expression of each class is as follows: wherein, N a scaling factor representing a weight, W a weight representing a category; formula (1) calculates a weight of each category sample quantity accounting for the total sample quantity and assigns the weight to a , used for formula (2) to calculate the weight of each category; Step S5, use pytorch toolkit to build the weight of the anti-sampling training set and the weighted loss as a parameter; Step S6, establish a long tail data classification model; Step S7, build a one-stage training paradigm, use a double-branch structure to adaptively change from an unweighted branch to a weighted branch; initially, the unweighted branch learns the characteristics of the long tail image, and gradually changes to use the weighted classification branch to learn how to classify the long tail data as the training round increases, thereby completing the original data learning of the long tail image representation and identifying the long tail image.

2. The long-tail image recognition method based on characterization data enhancement and loss rebalancing according to claim 1, characterized in that, The step S1 is specifically: calling the Pytorch toolkit, adjusting the size of the long tail image to 224x224, normalizing the adjusted image, and taking the image dataset.

3. The long-tail image recognition method based on characterization data enhancement and loss rebalancing according to claim 1, characterized in that, The step S6 specifically includes: S61, use ResNet as the backbone network, refer to the content in contrast learning, add a prediction head and a contrast head to ResNet; S62, randomly enhance the training set and the anti-sampling training set twice to generate four training sets; S63, first mix the training set and the anti-sampling training set, and then mix the training set and the anti-sampling training set, and finally generate two groups of image mixed samples using different mixing types; S64, constructing a self-supervised learning branch, constructing two groups of pictures through global picture mixed data enhancement Mix-up and local picture mixed data enhancement Cut-mix, and obtaining the picture feature mapping of the whole group of pictures after Mix-up enhancement and the picture feature mapping of the whole group of pictures after Cut-mix enhancement by the mapping head of the feature extractor respectively and Cut-mix enhancement Then, the contrast head compares and to express the conversion into output and , minimizes the negative cosine similarity of them, and the expression is as follows: wherein is normalized, represents the feature extracted by the feature extractor from the whole group of pictures after Mix-up enhancement, represents the feature extracted by the feature extractor from the whole group of pictures after Cut-mix enhancement, and respectively represent the picture features after linear space mapping of the features and ​​ Following the architecture of Sim-Siam, the stop gradient operation is used to prevent collapse, and the SimSiam loss function is defined as: where are the outputs of the feature mapping head after the extracted features are input into the mapping head for feature mapping, respectively. Finally, two groups of pictures using training sets and anti-sampling sets are constructed; In calculating the loss of the unweighted classification branch and the weighted classification branch, both are based on cross-entropy loss, the difference is that the cross-entropy of the weighted branch is weighted to the cross-entropy loss using the class weights derived from formulas (1) and (2); while the unweighted branch only uses cross-entropy to calculate the cross-entropy loss of each class; therefore, when calculating the cross-entropy loss of the unweighted classification branch The following formula is used: Where n represents the batch size of each training of the sample, represents the predicted probability, respectively, the label probability of the first picture after Mix-up and Cut-mix, respectively, the label probability of the first picture after Mix-up and Cut-mix. S65, build a class rebalancing branch, introduce a weighting factor inversely proportional to the label frequency, then force the model to learn the tail classes and suppress the head classes learning, the weight factor is set to: where, is the sampling frequency of the i class of the training data set, is a hyperparameter that measures the gap between the head class and the tail class, is the total number of classes; Given a training dataset , the re-balanced weighted classification branch cross-entropy loss can be written as where and denote the predicted probability of a mixed image and the weight factor by which the image should be weighted, respectively.

4. The long-tail image recognition method based on characteristic data enhancement and loss rebalancing according to claim 3, characterized in that, In the S61, the prediction head and the contrast head maintain the same structure as Sim-Siam in contrast learning.

5. The long-tail image recognition method based on characterization data enhancement and loss rebalancing according to claim 3, characterized in that, The specific process of S63 is: Using denote the images and labels of the training set, denote the images and labels of the anti-sampled training set, the image and label probability after mixing using Mix-up are denoted as , the image and label probability after mixing using Cut-mix are denoted as , The mixing expression for Mix-up is as follows: in It is a user-defined parameter with a value range of (0, 1). Indicates use value A sampled value is obtained by sampling, and this sampled value is assigned to... Images and labels for the training set. Images and labels for the offsampled training set. The image and label probabilities after using Mix-up; The mixing expression of Cut-mix is as follows: Cut-Mix combines another image by locally replacing the image region with another training image’s local, where is represented as a binary mask, which is the patch position that needs to be cut from two images, is element-wise multiplication, and denote the width and height of the image respectively, is the picture after Cut-mix data augmentation; Cut-mix image data mixing involves sampling the bounding box coordinates of the image. In the image and The coordinates of the sampling area, which needs to be cropped, are determined using the following formula: These represent the regions to be cropped. x coordinate, y Coordinates, width of the clipping region, height of the clipping region, All were obtained through uniform sampling. and Representing images respectively Width and height, The mixed labels and weights are the same as those of Mix-Up, as calculated by formula (3).

6. The long-tail image recognition method based on characterization data enhancement and loss rebalancing according to claim 1, characterized in that, The step S7 specifically comprises: constructing a one-stage training paradigm, using a bilateral branch structure to adaptively learn a rebalancing branch, multiplying the cross-entropy loss of the unweighted classification branch by the cross-entropy loss of the rebalanced weighted classification branch by automatically decreases with the increase of the current training round T: wherein represents the total training round, the meaning is a model attention weight; total loss defined as loss loss and loss combination: where is a hyperparameter controlling loss.

Images