A long tail identification method with strong and weak double-branch network with difficult sample perception

Abstract

The application discloses a long-tail identification method of a strong-weak double-branch network with difficult sample perception, and is used for solving the problem of long-tail distribution data identification, and the method comprises the following steps: constructing a strong branch and a weak branch of residual network structures, and the difference between the two branches lies in processing input data, one branch uses strong enhanced data, and the other branch uses weak enhanced data. Meanwhile, the application increases the difference between the two branches by maximizing the KL divergence. In addition, the application proposes a new learning strategy for the network structure to shift the focus of learning from the difference to classification, and through the constructed difficult sample perception loss function, the difficult samples can be more effectively paid attention to, and the prediction accuracy is further improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of long-tailed distribution visual recognition, and particularly relates to a long-tailed recognition method with strong and weak dual-branch networks with difficult sample perception. BACKGROUND

[0002] Real-world data often presents a long-tailed distribution, where most examples come from only a few classes. In this case, the number of training instances per class varies greatly, from a few samples for tail classes to thousands of samples for head classes. The performance of state-of-the-art classification models is usually biased towards the majority classes, resulting in poor generalization performance for rare classes.

[0003] Early work alleviates the degradation of long-tailed training data by rebalancing the data distribution. These methods can effectively reduce the dominance of head classes during training, but due to the distortion of the original data distribution, they often have the risk of overfitting to tail classes. Recently, many two-stage methods have made significant improvements compared to one-stage rebalancing methods. Delayed re-sampling and delayed re-weighting methods first train the network with a long-tailed distribution in the first stage, and then adjust the network using different re-sampling or re-weighting strategies in the second stage. This simple two-stage training scheme has been shown to be able to handle overfitting and set new state-of-the-art performance on many long-tailed benchmarks. There are also many ensemble methods that train long-tailed data in groups, such as LFME and BBN. These methods usually train a model for each group, and then combine individual models in a multi-branch framework. Specifically, LFME divides the dataset into multiple classes, medium classes and few classes, and assigns separate branches to learn feature representations for each subset. BBN proposes a double-sided branch network that simulates the delayed re-sampling (DRS) process by dynamically combining an instance balance sampler and an inverse balance sampler. SUMMARY

[0004] The application aims to solve the problem of inaccurate identification of tail classes and the problem of difficult samples being difficult to learn in long-tailed data recognition, and provides a long-tailed recognition method with strong and weak dual-branch networks with difficult sample perception.

[0005] Technical solution: A long-tailed recognition method with strong and weak dual-branch networks with difficult sample perception, comprising the following steps:

[0006] (1) Construct a dual-branch network structure: the dual-branch network structure is the same residual network structure, including a weak branch and a strong branch, wherein: the weak branch is based on a traditional data enhancement strategy, and the strong branch is randomly added with gray, blur and color distortion processing, including adding a regularization term to improve the difference between the two branches;

[0007] (2) Learning strategy from difference to classification: obtain as different branches as possible through difference loss in the early stage of training, and gradually shift the focus of training to classification as the training progresses;

[0008] (3) Constructing a difficult sample awareness loss function, in the long-tail data recognition, when and only when the training approaches the end and the accuracy of each class is stable, replace the classification loss LDAM-DRW with the difficult sample awareness loss, the expression of the difficult sample awareness loss function is as follows:

[0009]

[0010] Wherein is the difficult sample awareness loss weight of the c-th class, M is the number of training examples, C is the number of classes, w c is the weight of the c-th class, is the target label of the c-th class of the training example m, calculated by Softmax(z); The setting rules of are as follows:

[0011]

[0012] Wherein represents the weight of the c-th class in the e-th round of iteration, m is a momentum factor, is the accuracy of the c-th class on the validation set in the e-th round of iteration.

[0013] Further, the total loss of the method for image long-tail data recognition is defined as follows:

[0014] L SDN =λ(L cls (y,P s )+L cls (y,P w ))+(1-λ)L diff (P s ||P w )

[0015] Wherein λ and 1-λ are the weights of the classification loss and the difference loss respectively; λ is defined as follows:

[0016]

[0017] In the above formula, E max is the total number of training iterations, and E is the current iteration.

[0018] Further, the training of the double-branch network structure includes the following calculation process:

[0019] (1.1) Let x represent the training sample, y e {1, 2, …, C} is its label, where C is the number of classes, respectively, apply strong enhancement and weak enhancement strategy to "strong branch" and "weak branch", then obtain two enhanced samples (x_s, y) and (x_w, y) as input data, where (x_s, y) represents strong branch, (x_w, y) represents weak branch;

[0020] (1.2) Send the two samples obtained in step (1.1) to their respective branches to obtain feature vectors f_s and f_w;

[0021] (1.3) Send the two feature vectors obtained in step (1.2) to the classifiers W_s and W_w respectively; the output log formula is as follows:

[0022]

[0023] Where z s , z w respectively represent the predicted output of the strong branch and the weak branch;

[0024] Where the class probability is calculated by the softmax function as follows:

[0025]

[0026] In addition, the method increases a regularization term to ensure the difference between the two branches.

[0027] Where step (1) includes maximizing the KL divergence of the classification probability of the two branches on a total of C categories as follows:

[0028]

[0029] Where L diff represents the difference loss.

[0030] Further, the method introduces a hard sample aware loss function from the cross-entropy loss function for single-label classification, which is as follows:

[0031]

[0032] Where M is the number of training examples, C is the number of classes, is the target label of the c-th class of training example m, is the estimated probability of the c-th class of training example m, is calculated by Softmax(z).

[0033] Further, the reweighted loss formula is represented as:

[0034]

[0035] where w c is the weight of the c-th class, and LDAM-DRW loss is referenced as L WCE .

[0036] Finally, the difficult sample-aware loss function is formulated as:

[0037]

[0038] where the setting rules of are as follows:

[0039] where w c e is the weight of the c-th class in the e-th iteration, m is the momentum factor, and acc c e is the accuracy of the c-th class on the validation set in the e-th iteration.

[0040] Beneficial effects: The effectiveness of the double-branch network structure and the difficult sample loss function constructed by the present application is obviously improved compared with existing long-tail data recognition methods, including superior performance in the imbalance rate, greatly improving the accuracy in the long-tail data recognition process, and improving the recognition processing capability of the long-tail data.

[0041] BRIEF DESCRIPTION OF DRAWINGS

[0042] Figure 1 is the implementation flowchart of the method of the present application;

[0043] Figure 2 is the overall framework of the strong and weak double-branch network with difficult sample-aware loss of the present application. DETAILED DESCRIPTION

[0044] To further illustrate the technical solutions disclosed by the present application, the following further elaboration is made in conjunction with the drawings of the specification.

[0045] The existing multi-branch method essentially increases the difference between branches, so the present application first proposes a simple and effective method to increase the difference between branches and achieves good results. In addition, it is found that the accuracy of the validation set is not positively correlated with the number of training samples. Some classes have a large number of pictures, but the accuracy is not high, while some classes have a small number of pictures, but the accuracy is high. In view of this phenomenon, in the method of the present application, a new difficult sample-aware loss function is further provided, which dynamically adjusts the weights of difficult classes and simple classes during the training process, so that the model pays more attention to difficult samples.

[0046] ​The method disclosed by the application is used to solve the problem of long-tail distribution data identification, and is particularly related to the field of image processing, and the application designs two branches for strong representation learning and weak representation learning. The difference between the two branches is that when processing input data, one branch uses strong augmented data, and the other branch uses weak augmented data. At the same time, the difference between the two branches is increased by maximizing the KL divergence. In addition, the method includes designing a new learning strategy for the network structure to shift the focus of learning from difference to classification. A new difficult sample-aware loss function is designed for difficult samples, which can more effectively focus on difficult samples and further improve the accuracy of prediction.

[0047] Embodiment 1

[0048] In combination Figure 1 And Figure 2 , Figure 2 The overall framework of the strong-weak dual-branch network with difficult sample-aware loss is shown. The implementation process of the method disclosed by the application is as follows:

[0049] (1) Constructing a strong-weak dual-branch network structure

[0050] The two branches use the same residual network structure, which are respectively called weak branch and strong branch. For the weak branch, the traditional data augmentation strategy is followed, while for the strong branch, the training and the difference of the results are improved by randomly adding gray, blur and color distortion.

[0051] Let x represent the training sample, and y in y is its label, where C is the number of classes. Strong augmentation and weak augmentation strategies are applied to the strong branch and the weak branch respectively, and then two augmented samples (x_s, y) and (x_w, y) are obtained as input data, where (x_s, y) represents the strong branch and (x_w, y) represents the weak branch. The two samples are sent to their corresponding branches to obtain feature vectors f_s and f_w. Then, the two feature vectors will be sent to the classifiers W_s and W_w respectively. The output log formula is as follows:

[0052]

[0053] Where z s , z w represent the prediction output of the strong branch and the weak branch respectively. The probability of the class is calculated by the softmax function as follows:

[0054]

[0055] In addition, the application adds a regularization term to ensure the difference between the two branches. The KL divergence of the classification probability of the two branches on the total C categories is maximized as:

[0056]

[0057] where L diff represents the disparity loss.

[0058] The classification loss is defined as L cls with the final loss defined as:

[0059] L SDN = λ(L cls (y, P s )+ L cls (y, P w )) + (1 - λ) L diff (P s || P w )

[0060] where λ and 1 - λ are the weights of the classification loss and the disparity loss, respectively. The specific description of λ is given in step (2).

[0061] (2) Constructing a learning strategy from disparity to classification

[0062] This step proposes a new learning strategy to shift the focus of learning from disparity to classification. Specifically, it is hoped that in the early stages of training, the branches will be as different as possible through the disparity loss, and gradually shift the focus of training to classification as the training progresses. For this purpose, λ is defined as follows:

[0063]

[0064] where E max is the total number of training iterations, and E is the current iteration. Accordingly, it can be seen that λ is automatically generated according to the number of training iterations, and will gradually increase λ to control the speed of the learning strategy from disparity to classification as the number of training iterations increases.

[0065] (3) Constructing a new hard sample loss function

[0066] Based on a large number of prior art and experiments, it is shown that in the long-tail data recognition task, the classes with small number of samples are not necessarily difficult to learn, and similarly, the classes with large number of samples are not necessarily easy to learn. However, the common re-weighted loss function only assigns different weights to different classes according to the number of classes, i.e. the larger the number of samples, the smaller the weight of the class. This will lead to insufficient learning of some difficult classes with large number of samples, and over-learning of some simple classes with small number of samples. In order to solve the above problems, step (3) constructs a new loss function, called hard sample aware loss (HL).

[0067] First, the hard sample aware loss function is introduced from the cross-entropy loss function for single-label classification.

[0068]

[0069] where M is the number of training examples, C is the number of classes, is the target label of the c-th class of training example m, is the estimated probability of the c-th class of training example m.Here is computed by Softmax(z).

[0070] The common re-weighted loss can be formulated as:

[0071]

[0072] where w c is the weight of the c-th class.This embodiment uses LDAM-DRW loss reference as L WCE .

[0073] Formally, a weight term is introduced in the re-weighted loss function to obtain the hard sample aware loss function described in step (3):

[0074]

[0075] where is the hard sample aware loss weight of the c-th class.Set

[0076]

[0077] where denotes the weight of the c-th class in the e-th round of iteration, m is the momentum factor, is the accuracy of the c-th class on the validation set in the e-th round of iteration.

[0078] Note that hard sample aware loss is used to replace classification loss LDAM-DRW only when and if training is close to the end and the accuracy of each class is stable.In addition, the momentum factor also provides stability for the change of hard sample aware weight.

[0079] In summary, a simple and effective structure called strong-weak dual branch network is introduced, which has a special difference-to-classification learning strategy for long-tailed visual recognition.In addition, a new hard sample aware loss is proposed for hard-to-learn samples.A large number of experiments show that this method is superior to previous work and has great advantages on long-tailed benchmark tests.

[0080] Table 1.top-1 accuracy of ResNet-32 on Long-tailed CIFAR-10 / 100

[0081]

[0082] Table 1 reports the top-1 accuracy of cifar-10-LT and cifar-100-LT using ResNet-32. The imbalance ratios are 200, 100, 50, and 20. The method described in the present application performs best in all the datasets, which demonstrates the generality of the method. Compared with previous reweighting and multi-branch methods, it can be seen that the method described in the present application is significantly better than other prior art, which indicates the effectiveness of the loss function and architecture in the present application. Compared with strong augmentation methods, the present application also surpasses them, which indicates that the improvement of the method is not only due to data augmentation. Compared with other recent competitive methods, it is also superior to prior art in all different imbalance rates. It can be seen that the improvement of CIFAR-100-LT is greater than that of CIFAR-10-LT, which indicates that the method described in the present application can obtain better results when facing more difficult datasets. The present embodiment also reports the accuracy on the multi-class, medium-class and few-class according to the number of training samples of each class.

[0083] Table 2. Top-1 accuracy of ResNet-10 / ResNet-50 on ImageNet-LT and iNaturalist 2018

[0084]

[0085] The effectiveness of the method described in the present application is further verified on ImageNet LT and iNaturalist 2018. The results of the two large-scale imbalanced datasets are reported in Table 2. On ImageNet LT, the present application is 2.2% (ResNet-10) and 1.9% (ResNet-50) higher than RIDE, respectively, and on iNaturalist 2018, the present application is 1.1% (ResNet-50) higher than RIDE, which indicates that the present application can be effectively promoted to large-scale datasets.

[0086] Embodiment 2

[0087] Further combination Figure 1 The application of the present application is described below through an algorithmic embodiment.

[0088] The dataset uses ImageNet-LT, the programming language is Python3, and the framework uses Pytorch1.7.1.

[0089] Prepare the dataset ImageNet-LT, which has a total of 1000 categories and presents a long-tail distribution overall. The training set contains more than 110,000 pictures, with a maximum of 1280 pictures per category and a minimum of only 5 pictures; the test set contains 50,000 pictures, with 50 pictures per category. Prepare the feature extraction network ResNet-50, with an output feature dimension of 512, and the classifier uses a fully connected layer with an input feature dimension of 512 and an output feature dimension of 1000. Randomly initialize the parameters of the neural network.

[0090] Training phase: A picture after random cropping, random horizontal flipping, automatic augmentation and normalization is used as a weakly augmented picture. On the basis of weak augmentation, we add pictures with random addition of grayscale, blur and color distortion as strongly augmented pictures, and send them into the dual-branch network for training. During training, we maximize the KL divergence and minimize the classification loss to ensure the difference between the two branches and the accuracy of classification. At the same time, we use the learning strategy from difference to classification during training, and gradually shift the focus from difference to classification. At the end of training, we replace the classification loss function with the newly proposed difficult sample-aware loss function to increase the weight of difficult samples for fine-tuning.

[0091] The batch size of training is 128, and there are a total of 200 iterations, of which the last 20 iterations use the difficult sample-aware loss function. The initial learning rate is 0.05, and the learning rate is decayed to 0.1 and 0.01 at 120 and 160 rounds, respectively.

[0092] Test phase: Construct the ImageNet-LT test set, which has 50 pictures per category. After scaling, center cropping and normalization, the pictures are sent into the network for prediction, and the outputs of the dual-branch are integrated as the final result. The accuracy on the entire test set reaches 56.31%, achieving the best result so far.

Claims

1. An image long-tail data recognition method with strong and weak double-branch network with difficult sample perception, characterized in that: The method comprises the following steps: (1) constructing a double-branch network structure: the double-branch network structure is a residual network structure, comprising a weak branch and a strong branch, wherein the weak branch is based on a conventional data enhancement strategy, and the strong branch is based on the weak branch and randomly adds gray, blur and color distortion processing, and a regularization term is added to improve the difference between the two branches; The training of the double-branch network structure comprises the following calculation process: (1.1) Let x represent a training sample, y e {1, 2, …, C} is its label, where C is the class number, respectively apply strong enhancement and weak enhancement strategy to strong branch and weak branch, then obtain two enhanced samples (x s , y) and (x w , y) as input data, where (x s , y) represents strong branch, (x w , y) represents weak branch; (1.2) sending the two samples obtained in step (1.1) to their respective branches to obtain feature vectors and ; (1.3) The two eigenvectors obtained in step (1.2) are sent to the classifier W s and W w ; the corresponding calculation formula is as follows: ， ， wherein , respectively denote the prediction outputs of the strong and weak branches. Wherein the probability of the class is calculated by the softmax function as follows: ， ， In addition, the method increases the regularization term to ensure the difference between the two branches; (2) learning strategy from difference to classification: in the early stage of training, the branches are obtained as different as possible through the difference loss, and the training focus is gradually transferred to classification as the training progresses; (3) constructing a difficult sample perception loss function, in the long-tail data recognition, only when the training is close to the end and the accuracy of each class is stable, the difficult sample perception loss replaces the classification loss LDAM-DRW, and the expression of the difficult sample perception loss function is as follows: wherein is a difficult sample perception loss weight for the c-th class, M is the number of training examples, C is the number of classes, is a weight for the c-th class, is a target label for the c-th class of the training example m, by Softmax (z) calculation; The setting rules are as follows: ， wherein represents the c class in the e weight of the m is a momentum factor, is the accuracy of the e iteration on the validation set.

2. The image long-tail data recognition method with strong and weak double-branch networks with difficult sample perception according to claim 1, characterized in that: The total loss of the method for image long-tail data recognition is defined as follows: ， wherein and are the weights of the classification loss and the discrepancy loss, respectively; are defined as follows: ， In the above formulae, is the total number of training iterations, E is the current iteration number.

3. The image long-tail data recognition method with strong and weak double-branch networks with difficult sample perception according to claim 2, characterized in that: The method includes branching two at a total of C The KL divergence maximization of the classification probability on the category is: ， wherein, represents the difference loss.

4. The image long-tail data recognition method with strong and weak double-branch networks with difficult sample perception according to claim 1, characterized in that: The method introduces the difficult sample perception loss function from the cross-entropy loss function for single-label classification, and the cross-entropy loss function is as follows: ， where, M is the number of training examples, C is the number of classes, is the target label for the i-th class of the i-th training example, m is the estimated probability for the i-th class of the i-th training example, c m c is computed by Softmax(z);​​​​ The corresponding reweighted loss formula is represented as: ， wherein, is the weight of the c class, the LDAM-DRW loss in the method references as .

Images